anonymousland-synapse/synapse
Eric Eastwood fef2e792be
Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114)
Fix https://github.com/matrix-org/synapse/issues/11091
Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`)

 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`)
    - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)).
    - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
       - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date.
       - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order.
 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764
 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls.
    - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793


Before | After
--- | ---
![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png)



#### Why aren't we sorting topologically when receiving backfill events?

> The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway.
> 
> As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering.
>
> -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138

See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
2022-02-07 15:54:13 -06:00
..
_scripts Add a flag to the synapse_review_recent_signups script to ignore and filter appservice users. (#11675) 2022-01-17 16:43:25 +00:00
api Stabilise MSC3231 (Token Based Registration) (#11867) 2022-02-04 12:15:13 +00:00
app Remove support for the webclient listener. (#11895) 2022-02-03 18:36:49 +00:00
appservice Remove optional state of ApplicationService.is_interested's store parameter (#11911) 2022-02-07 18:26:42 +00:00
config Remove support for the webclient listener. (#11895) 2022-02-03 18:36:49 +00:00
crypto Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
events Improvements to bundling aggregations. (#11815) 2022-01-26 08:27:04 -05:00
federation Improve opentracing for federation requests (#11870) 2022-02-03 12:29:16 +00:00
groups Add type annotations to synapse.metrics (#10847) 2021-11-17 19:07:02 +00:00
handlers Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
http Don't print HTTPStatus.* in "Processed..." logs (#11827) 2022-01-26 12:47:34 +00:00
logging Improve opentracing for federation requests (#11870) 2022-02-03 12:29:16 +00:00
metrics Fix type errors introduced by new annotations in the Prometheus Client library. (#11832) 2022-02-02 16:51:00 +00:00
module_api Allow modules to retrieve server and worker names (#11868) 2022-02-01 16:23:55 +01:00
push Revert experimental push rules from #7997. (#11884) 2022-02-02 09:49:31 -05:00
replication Stop reading from event_reference_hashes (#11794) 2022-01-21 09:18:10 +00:00
res Allow configuration of the oEmbed URLs. (#10714) 2021-08-31 18:37:07 -04:00
rest Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
server_notices disallow-untyped-defs for synapse.server_notices (#11021) 2021-10-07 19:55:15 +01:00
spam_checker_api Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
state Remove log_function and its uses (#11761) 2022-01-18 13:06:04 +00:00
static Display an error page during failure of fallback UIA. (#10561) 2021-08-18 08:13:35 -04:00
storage Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114) 2022-02-07 15:54:13 -06:00
streams Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
util Describe prune_unread_entries in docstrings (#11876) 2022-02-02 15:11:23 +00:00
__init__.py 1.52.0rc1 2022-02-01 11:04:17 +00:00
event_auth.py Check *all* auth events for room id and rejection (#11009) 2021-10-18 18:28:30 +01:00
notifier.py Send to-device messages to application services (#11215) 2022-02-01 14:13:38 +00:00
py.typed Mark Module API error imports as re-exported and mark Synapse as containing type annotations (#11054) 2021-10-13 08:42:41 +01:00
python_dependencies.py Fix type errors introduced by new annotations in the Prometheus Client library. (#11832) 2022-02-02 16:51:00 +00:00
server.py Bundle aggregations outside of the serialization method. (#11612) 2022-01-07 09:10:46 -05:00
types.py Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
visibility.py Make the get_global_account_data_by_type_for_user cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00