synapse-product/synapse/rest/client
Eric Eastwood fef2e792be
Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114)
Fix https://github.com/matrix-org/synapse/issues/11091
Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`)

 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`)
    - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)).
    - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
       - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date.
       - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order.
 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764
 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls.
    - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793


Before | After
--- | ---
![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png)



#### Why aren't we sorting topologically when receiving backfill events?

> The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway.
> 
> As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering.
>
> -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138

See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
2022-02-07 15:54:13 -06:00
..
__init__.py Flatten the synapse.rest.client package (#10600) 2021-08-17 11:57:58 +00:00
_base.py Add support for /_matrix/client/v3 APIs (#11318) 2021-11-16 14:47:58 +01:00
account_data.py Make the get_global_account_data_by_type_for_user cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
account_validity.py Addtional type hints for the REST servlets. (#10665) 2021-08-23 08:14:17 -04:00
account.py Use direct references for configuration variables (part 7). (#10959) 2021-10-04 07:18:54 -04:00
auth.py Require direct references to configuration variables. (#10985) 2021-10-06 10:47:41 -04:00
capabilities.py Include io.element.thread capability for MSC3440. (#11690) 2022-01-05 12:08:03 -05:00
devices.py Make get_device return None if the device doesn't exist rather than raising an exception. (#11565) 2021-12-13 15:39:43 +00:00
directory.py Addtional type hints for the REST servlets. (#10665) 2021-08-23 08:14:17 -04:00
events.py Bundle aggregations outside of the serialization method. (#11612) 2022-01-07 09:10:46 -05:00
filter.py Use direct references for configuration variables (part 6). (#10916) 2021-09-29 06:44:15 -04:00
groups.py Additional type hints for client REST servlets (part 5) (#10736) 2021-09-03 09:22:22 -04:00
initial_sync.py Additional type hints for REST servlets (part 2). (#10674) 2021-08-26 11:53:52 +00:00
keys.py Add support for /_matrix/client/v3 APIs (#11318) 2021-11-16 14:47:58 +01:00
knock.py Additional type hints for client REST servlets (part 4) (#10728) 2021-09-01 11:59:32 -04:00
login.py Stabilise support for MSC2918 refresh tokens as they have now been merged into the Matrix specification. (#11435) 2021-12-06 19:11:43 +00:00
logout.py Additional type hints for REST servlets (part 2). (#10674) 2021-08-26 11:53:52 +00:00
notifications.py Bundle aggregations outside of the serialization method. (#11612) 2022-01-07 09:10:46 -05:00
openid.py Use direct references for some configuration variables (#10798) 2021-09-13 13:07:12 -04:00
password_policy.py Use direct references for some configuration variables (part 3) (#10885) 2021-09-23 07:13:34 -04:00
presence.py Additional type hints for REST servlets (part 2). (#10674) 2021-08-26 11:53:52 +00:00
profile.py Use direct references for configuration variables (part 6). (#10916) 2021-09-29 06:44:15 -04:00
push_rule.py Revert experimental push rules from #7997. (#11884) 2022-02-02 09:49:31 -05:00
pusher.py Addtional type hints for the REST servlets. (#10665) 2021-08-23 08:14:17 -04:00
read_marker.py Add a constant for receipt types (m.read). (#11531) 2021-12-08 12:26:29 -05:00
receipts.py Add a constant for receipt types (m.read). (#11531) 2021-12-08 12:26:29 -05:00
register.py Stabilise MSC3231 (Token Based Registration) (#11867) 2022-02-04 12:15:13 +00:00
relations.py Include whether the requesting user has participated in a thread. (#11577) 2022-01-18 11:38:57 -05:00
report_event.py Additional type hints for client REST servlets (part 4) (#10728) 2021-09-01 11:59:32 -04:00
room_batch.py Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
room_keys.py Additional type hints for client REST servlets (part 4) (#10728) 2021-09-01 11:59:32 -04:00
room_upgrade_rest_servlet.py Addtional type hints for the REST servlets. (#10665) 2021-08-23 08:14:17 -04:00
room.py Improvements to bundling aggregations. (#11815) 2022-01-26 08:27:04 -05:00
sendtodevice.py Additional type hints for client REST servlets (part 4) (#10728) 2021-09-01 11:59:32 -04:00
shared_rooms.py Use direct references for configuration variables (part 6). (#10916) 2021-09-29 06:44:15 -04:00
sync.py Improvements to bundling aggregations. (#11815) 2022-01-26 08:27:04 -05:00
tags.py Addtional type hints for the REST servlets. (#10665) 2021-08-23 08:14:17 -04:00
thirdparty.py Addtional type hints for the REST servlets. (#10665) 2021-08-23 08:14:17 -04:00
tokenrefresh.py Addtional type hints for the REST servlets. (#10665) 2021-08-23 08:14:17 -04:00
transactions.py Additional type hints for client REST servlets (part 5) (#10736) 2021-09-03 09:22:22 -04:00
user_directory.py Use direct references for configuration variables (part 5). (#10897) 2021-09-24 07:25:21 -04:00
versions.py Add MSC2716 and MSC3030 to /versions -> unstable_features (#11582) 2021-12-16 11:25:37 -06:00
voip.py Fix getTurnServer response: return an integer ttl (#10922) 2021-09-30 08:04:55 -04:00