anonymousland-synapse/synapse/handlers
Eric Eastwood fef2e792be
Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114)
Fix https://github.com/matrix-org/synapse/issues/11091
Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`)

 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`)
    - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)).
    - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
       - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date.
       - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order.
 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764
 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls.
    - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793


Before | After
--- | ---
![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png)



#### Why aren't we sorting topologically when receiving backfill events?

> The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway.
> 
> As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering.
>
> -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138

See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
2022-02-07 15:54:13 -06:00
..
ui_auth Stabilise MSC3231 (Token Based Registration) (#11867) 2022-02-04 12:15:13 +00:00
__init__.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
account_data.py Fix docstring on add_account_data_for_user. (#11716) 2022-01-10 15:38:22 +00:00
account_validity.py Require direct references to configuration variables. (#10985) 2021-10-06 10:47:41 -04:00
admin.py Remove the 'password_hash' from the Users Admin API endpoint response dictionary (#11576) 2022-01-14 14:53:33 +00:00
appservice.py Remove optional state of ApplicationService.is_interested's store parameter (#11911) 2022-02-07 18:26:42 +00:00
auth.py Add a module callback to set username at registration (#11790) 2022-01-26 14:21:13 +00:00
cas.py Use direct references for some configuration variables (part 3) (#10885) 2021-09-23 07:13:34 -04:00
deactivate_account.py Remove account data (including client config, push rules and ignored users) upon user deactivation. (#11621) 2022-01-24 13:37:00 +00:00
device.py Cache empty responses from /user/devices (#11587) 2022-01-05 13:33:28 +00:00
devicemessage.py Clarifications and small fixes to to-device related code (#11247) 2021-11-09 14:31:15 +00:00
directory.py Convert all namedtuples to attrs. (#11665) 2021-12-30 18:47:12 +00:00
e2e_keys.py Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
e2e_room_keys.py Add type hints to synapse/storage/databases/main/e2e_room_keys.py (#11549) 2021-12-14 17:46:47 +00:00
event_auth.py Check *all* auth events for room id and rejection (#11009) 2021-10-18 18:28:30 +01:00
events.py Remove log_function and its uses (#11761) 2022-01-18 13:06:04 +00:00
federation_event.py Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
federation.py Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
groups_local.py Require type hints in the handlers module. (#10831) 2021-09-20 08:56:23 -04:00
identity.py Remove legacy code related to deprecated trust_identity_server_for_password_resets config flag (#11333) 2021-11-18 10:56:32 -08:00
initial_sync.py Bundle aggregations outside of the serialization method. (#11612) 2022-01-07 09:10:46 -05:00
message.py Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
oidc.py Save the OIDC session ID (sid) with the device on login (#11482) 2021-12-06 12:43:06 -05:00
pagination.py Include whether the requesting user has participated in a thread. (#11577) 2022-01-18 11:38:57 -05:00
password_policy.py Use direct references for some configuration variables (part 3) (#10885) 2021-09-23 07:13:34 -04:00
presence.py Remove log_function and its uses (#11761) 2022-01-18 13:06:04 +00:00
profile.py Configurable limits on avatars (#11846) 2022-01-28 15:41:33 +01:00
read_marker.py Remove the deprecated BaseHandler. (#11005) 2021-10-08 07:44:43 -04:00
receipts.py Add a constant for receipt types (m.read). (#11531) 2021-12-08 12:26:29 -05:00
register.py Add a config flag to inhibit M_USER_IN_USE during registration (#11743) 2022-01-26 13:02:54 +01:00
room_batch.py Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
room_list.py Convert all namedtuples to attrs. (#11665) 2021-12-30 18:47:12 +00:00
room_member_worker.py Implement knock feature (#6739) 2021-06-09 19:39:51 +01:00
room_member.py Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
room_summary.py Fix a bug that corrupted the cache of federated space hierarchies (#11775) 2022-01-20 11:03:42 +00:00
room.py Improvements to bundling aggregations. (#11815) 2022-01-26 08:27:04 -05:00
saml.py Remove the deprecated BaseHandler. (#11005) 2021-10-08 07:44:43 -04:00
search.py Improvements to bundling aggregations. (#11815) 2022-01-26 08:27:04 -05:00
send_email.py Pass str to twisted's IReactorTCP (#10895) 2021-09-30 12:51:47 +01:00
set_password.py Remove the deprecated BaseHandler. (#11005) 2021-10-08 07:44:43 -04:00
sso.py Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
state_deltas.py Easy refactors of the user directory (#10789) 2021-09-10 10:54:38 +01:00
stats.py Fix AssertionErrors after purging events (#11642) 2022-01-04 16:36:33 +00:00
sync.py Send to-device messages to application services (#11215) 2022-02-01 14:13:38 +00:00
typing.py Clean up an indirect reference to the homeserver datastore (#11914) 2022-02-07 13:25:09 +00:00
user_directory.py Fix AssertionErrors after purging events (#11642) 2022-01-04 16:36:33 +00:00