synapse-product/synapse
Eric Eastwood 40bb37eb27
Stop getting missing prev_events after we already know their signature is invalid (#13816)
While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place.

Related to
 - https://github.com/matrix-org/synapse/issues/13622
    - https://github.com/matrix-org/synapse/pull/13635
 - https://github.com/matrix-org/synapse/issues/13676

Part of https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures.

With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now.

For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761

To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid.

 - `backfill`
    - `outgoing-federation-request` `/backfill`
    - `_check_sigs_and_hash_and_fetch`
       - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill
          -  `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
          -  `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
   - `_process_pulled_events`
      - `_process_pulled_event` for each validated event
         -  Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it
            - `_get_state_ids_after_missing_prev_event`
               - `outgoing-federation-request` `/state_ids`
               -  `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again
               -  `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
2022-10-15 00:36:49 -05:00
..
_scripts Add an API for listing threads in a room. (#13394) 2022-10-13 08:02:11 -04:00
api Stop getting missing prev_events after we already know their signature is invalid (#13816) 2022-10-15 00:36:49 -05:00
app Mark /relations endpoint as usable on workers. (#14028) 2022-10-12 06:46:13 -04:00
appservice Send the appservice access token as a header. (#13996) 2022-10-04 07:06:41 -04:00
config Merge remote-tracking branch 'origin/release-v1.69' into develop 2022-10-14 14:11:27 -04:00
crypto Instrument _check_sigs_and_hash_and_fetch to trace time spent in child concurrent calls (#13588) 2022-08-23 21:53:37 -05:00
events fix: Push notifications for invite over federation (#13719) 2022-09-28 12:31:53 +00:00
federation Don't require optional invite_room_state field on fed v2 invite (#14083) 2022-10-14 13:48:33 +01:00
handlers Stop getting missing prev_events after we already know their signature is invalid (#13816) 2022-10-15 00:36:49 -05:00
http Indicate what endpoint came back with a JSON response we were unable to parse (#14097) 2022-10-07 11:39:45 -05:00
logging Update mypy and mypy-zope, attempt 3 (#13993) 2022-09-30 17:36:28 +01:00
metrics Catch BrokenPipeError from metrics server, and log as a warning (#14072) 2022-10-07 13:35:44 +01:00
module_api Deprecate the generate_short_term_login_token method in favor of an async create_login_token method in the Module API. (#13842) 2022-10-06 10:22:36 +00:00
push Return the main timeline for events which are not part of a thread. (#14140) 2022-10-12 12:15:52 -04:00
replication Merge remote-tracking branch 'origin/release-v1.69' into develop 2022-10-14 14:11:27 -04:00
res Add viewport directive to HTML templates to optimise for mobile (#13493) 2022-08-11 14:04:20 +01:00
rest Accept threaded receipts for events related to the root event. (#14174) 2022-10-14 18:05:25 +00:00
server_notices Directly lookup local membership instead of getting all members in a room first (get_users_in_room mis-use) (#13608) 2022-08-24 14:13:12 -05:00
spam_checker_api Fix import in module_api module and docs on the new check_event_for_spam signature (#12918) 2022-05-31 12:04:53 +02:00
state Prepatory work for batching events to send (#13487) 2022-09-28 10:39:03 +01:00
static Add viewport directive to HTML templates to optimise for mobile (#13493) 2022-08-11 14:04:20 +01:00
storage Stop getting missing prev_events after we already know their signature is invalid (#13816) 2022-10-15 00:36:49 -05:00
streams Do not allow a None-limit on PaginationConfig. (#14146) 2022-10-14 12:30:05 +00:00
util Making parse_server_name more consistent (#14007) 2022-10-11 12:42:11 +00:00
__init__.py Parse SYNAPSE_ASYNC_IO_REACTOR env variable & log the reactor on startup (#14092) 2022-10-07 15:19:59 +01:00
event_auth.py Rename the EventFormatVersions enum values so that they line up with room version numbers. (#13706) 2022-09-07 11:08:20 +01:00
notifier.py Batch up notifications after event persistence (#14033) 2022-10-05 10:12:48 -07:00
py.typed Mark Module API error imports as re-exported and mark Synapse as containing type annotations (#11054) 2021-10-13 08:42:41 +01:00
server.py Fix error in is_mine_id when encountering a malformed ID (#13746) 2022-09-08 15:54:36 +01:00
types.py Accept & store thread IDs for receipts (implement MSC3771). (#13782) 2022-09-23 14:33:28 +00:00
visibility.py Skip filtering during push if there are no push actions (#13992) 2022-09-30 17:40:33 +01:00