synapse-product/synapse/storage/databases/main
Eric Eastwood fef2e792be
Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114)
Fix https://github.com/matrix-org/synapse/issues/11091
Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`)

 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`)
    - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)).
    - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
       - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date.
       - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order.
 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764
 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls.
    - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793


Before | After
--- | ---
![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png)



#### Why aren't we sorting topologically when receiving backfill events?

> The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway.
> 
> As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering.
>
> -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138

See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
2022-02-07 15:54:13 -06:00
..
__init__.py Remove redundant get_current_events_token (#11643) 2022-01-04 16:10:27 +00:00
account_data.py Add a background database update to purge account data for deactivated users. (#11655) 2022-02-02 11:37:18 +00:00
appservice.py Send to-device messages to application services (#11215) 2022-02-01 14:13:38 +00:00
cache.py Invalidate the get_users_in_room{_with_profile} caches only when necessary. (#11878) 2022-02-02 12:24:07 -05:00
censor_events.py Type hint the constructors of the data store classes (#11555) 2021-12-13 17:05:00 +00:00
client_ips.py Type hint the constructors of the data store classes (#11555) 2021-12-13 17:05:00 +00:00
deviceinbox.py Send to-device messages to application services (#11215) 2022-02-01 14:13:38 +00:00
devices.py Debug for device lists updates (#11760) 2022-01-20 13:38:44 +00:00
directory.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
e2e_room_keys.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
end_to_end_keys.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
event_federation.py Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
event_push_actions.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
events_bg_updates.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
events_forward_extremities.py Attempt to annotate events_forward_extremities (#11314) 2021-11-12 15:58:17 +00:00
events_worker.py Remove redundant get_current_events_token (#11643) 2022-01-04 16:10:27 +00:00
events.py Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
filtering.py Improve type hints in storage classes. (#11652) 2021-12-29 13:04:28 +00:00
group_server.py Type hint the constructors of the data store classes (#11555) 2021-12-13 17:05:00 +00:00
keys.py Add missing type hints to synapse.util (#9982) 2021-05-24 15:32:01 -04:00
lock.py Type hint the constructors of the data store classes (#11555) 2021-12-13 17:05:00 +00:00
media_repository.py Improve type hints in storage classes. (#11652) 2021-12-29 13:04:28 +00:00
metrics.py Remove redundant COALESCE()s around COUNT()s in database queries (#11570) 2021-12-14 12:34:30 +00:00
monthly_active_users.py Remove redundant COALESCE()s around COUNT()s in database queries (#11570) 2021-12-14 12:34:30 +00:00
openid.py Add type hints to some storage classes (#11307) 2021-11-11 08:47:31 -05:00
presence.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
profile.py Database storage profile passes mypy (#11342) 2021-11-15 12:59:33 +00:00
purge_events.py Drop unused table public_room_list_stream. (#11795) 2022-01-21 09:19:56 +00:00
push_rule.py Revert experimental push rules from #7997. (#11884) 2022-02-02 09:49:31 -05:00
pusher.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
receipts.py Type hint the constructors of the data store classes (#11555) 2021-12-13 17:05:00 +00:00
registration.py Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
rejections.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
relations.py Remove the obsolete MSC1849 configuration flag. (#11843) 2022-01-31 10:13:32 -05:00
room_batch.py Correct type hint for room_batch.py (#11310) 2021-11-11 16:49:28 +00:00
room.py Make pagination of rooms in admin api stable (#11737) 2022-01-17 11:42:51 +00:00
roommember.py Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
search.py Convert all namedtuples to attrs. (#11665) 2021-12-30 18:47:12 +00:00
session.py Run pyupgrade --py37-plus --keep-percent-format on Synapse (#11685) 2022-01-05 09:53:05 -08:00
signatures.py Stop reading from event_reference_hashes (#11794) 2022-01-21 09:18:10 +00:00
state_deltas.py Improve log messages for stream ids (#11536) 2021-12-08 14:15:14 +00:00
state.py Better error messages from get_create_event_for_room (#11638) 2022-01-04 16:10:05 +00:00
stats.py Add type hints to synapse/storage/databases/main/stats.py (#11653) 2021-12-29 08:01:13 -05:00
stream.py Improvements to bundling aggregations. (#11815) 2022-01-26 08:27:04 -05:00
tags.py Add type hints to synapse/storage/databases/main/account_data.py (#11546) 2021-12-13 16:28:10 +00:00
transactions.py Add admin API to get a list of federated rooms (#11658) 2022-01-25 16:11:40 +00:00
ui_auth.py Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
user_directory.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
user_erasure_store.py Annotations for user_erasure_store (#11313) 2021-11-11 19:22:19 +00:00