synapse-product/synapse/handlers
Richard van der Hoff af2248f8bf
Optimise missing prev_event handling (#9601)
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from 
the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events,
we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote
server for the state at those prev_events, and then we may need to then ask the remote server for any events
in that state which we don't already have, as well as the auth events for those missing state events, so that we
can auth them.

This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state
events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the
observation that we are currently loading all of those auth events into memory at the start of the operation, but
we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and
leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're
actually going to need, but it doesn't.)

The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about
60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number
of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache
churn. (NB I've already tripled the size of the cache from its default of 10K).

Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into
a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be
returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different
(ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting
that we clean it up.
2021-03-15 13:51:02 +00:00
..
ui_auth UI Auth via SSO: redirect the user to an appropriate SSO. (#9081) 2021-01-12 17:38:03 +00:00
__init__.py Remove the deprecated Handlers object (#8494) 2020-10-09 07:24:34 -04:00
_base.py Simplify the flow for SSO UIA (#8881) 2020-12-08 14:03:38 +00:00
account_data.py Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
account_validity.py Add type hints for account validity handler (#8620) 2020-10-26 14:17:31 -04:00
acme_issuing_service.py Add type hints to various handlers. (#9223) 2021-01-26 10:50:21 -05:00
acme.py Create a SynapseReactor type which incorporates the necessary reactor interfaces. (#9528) 2021-03-08 08:25:43 -05:00
admin.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
appservice.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
auth.py Convert Requester to attrs (#9586) 2021-03-10 18:15:56 +00:00
cas_handler.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
deactivate_account.py Ensure pushers are deleted for deactivated accounts (#9285) 2021-02-25 13:56:55 +00:00
device.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
devicemessage.py Ratelimit cross-user key sharing requests. (#8957) 2021-02-19 13:20:34 -05:00
directory.py Allow spam-checker modules to be provide async methods. (#8890) 2020-12-11 14:05:15 -05:00
e2e_keys.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
e2e_room_keys.py Add type hints to E2E handler. (#9232) 2021-01-28 08:34:19 -05:00
events.py Ratelimit cross-user key sharing requests. (#8957) 2021-02-19 13:20:34 -05:00
federation.py Optimise missing prev_event handling (#9601) 2021-03-15 13:51:02 +00:00
groups_local.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
identity.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
initial_sync.py Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
message.py Add configs to make profile data more private (#9203) 2021-02-19 09:50:41 +00:00
oidc_handler.py Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
pagination.py Fix additional type hints. (#9543) 2021-03-09 07:41:32 -05:00
password_policy.py Add type hints to some handlers (#8505) 2020-10-09 07:20:51 -04:00
presence.py Prevent presence background jobs from running when presence is disabled (#9530) 2021-03-03 10:21:46 +00:00
profile.py Fix style checking due to updated black. 2021-02-19 08:19:54 -05:00
read_marker.py Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
receipts.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
register.py Fix the auth provider on the logins metric (#9573) 2021-03-10 18:15:03 +00:00
room_list.py Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
room_member_worker.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
room_member.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
room.py Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
saml_handler.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
search.py Add type hints to various handlers. (#9223) 2021-01-26 10:50:21 -05:00
set_password.py Add type hints to various handlers. (#9223) 2021-01-26 10:50:21 -05:00
sso.py Prometheus metrics for logins and registrations (#9511) 2021-03-04 16:39:27 +00:00
state_deltas.py Add type hints to various handlers. (#9223) 2021-01-26 10:50:21 -05:00
stats.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
sync.py Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
typing.py Update black, and run auto formatting over the codebase (#9381) 2021-02-16 22:32:34 +00:00
user_directory.py Add back the guard against the user directory stream position not existing. (#9428) 2021-02-18 08:44:19 -05:00