synapse-product/tests/storage
Eric Eastwood 40bb37eb27
Stop getting missing prev_events after we already know their signature is invalid (#13816)
While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place.

Related to
 - https://github.com/matrix-org/synapse/issues/13622
    - https://github.com/matrix-org/synapse/pull/13635
 - https://github.com/matrix-org/synapse/issues/13676

Part of https://github.com/matrix-org/synapse/issues/13356

Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures.

With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now.

For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761

To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid.

 - `backfill`
    - `outgoing-federation-request` `/backfill`
    - `_check_sigs_and_hash_and_fetch`
       - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill
          -  `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
          -  `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)`
   - `_process_pulled_events`
      - `_process_pulled_event` for each validated event
         -  Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it
            - `_get_state_ids_after_missing_prev_event`
               - `outgoing-federation-request` `/state_ids`
               -  `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again
               -  `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
2022-10-15 00:36:49 -05:00
..
databases Fix have_seen_event cache not being invalidated (#13863) 2022-09-27 15:55:43 -05:00
util Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
__init__.py Reference Matrix Home Server 2014-08-12 15:10:52 +01:00
test__base.py Optimise _update_client_ips_batch_txn to batch together database operations. (#12252) 2022-04-08 15:29:13 +01:00
test_account_data.py Use the ignored_users table to test event visibility & sync. (#12225) 2022-03-15 14:06:05 -04:00
test_appservice.py Remove code which updates application_services_state.last_txn (#12680) 2022-05-17 11:07:18 +01:00
test_background_update.py Add type hints to tests files. (#12256) 2022-03-21 09:43:16 -04:00
test_base.py Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
test_cleanup_extrems.py Default to private room visibility rather than public when a client does not specify one, according to spec. (#12350) 2022-04-01 15:55:09 +01:00
test_client_ips.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_database.py Handle cancellation in DatabasePool.runInteraction() (#12199) 2022-03-16 15:07:41 +00:00
test_devices.py Additional constants for EDU types. (#12884) 2022-05-27 07:14:36 -04:00
test_directory.py Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
test_e2e_room_keys.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_end_to_end_keys.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_event_chain.py Persist CreateRoom events to DB in a batch (#13800) 2022-09-28 10:11:48 +00:00
test_event_federation.py Stop getting missing prev_events after we already know their signature is invalid (#13816) 2022-10-15 00:36:49 -05:00
test_event_metrics.py Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
test_event_push_actions.py Use threaded receipts when fetching events for push. (#13878) 2022-10-04 12:07:02 -04:00
test_events.py Faster room joins: avoid blocking when pulling events with missing prevs (#13355) 2022-07-26 12:39:23 +01:00
test_id_generators.py Remove redundant get_success calls in test code (#12346) 2022-04-01 16:10:31 +01:00
test_keys.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_main.py Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
test_monthly_active_users.py Update mypy and mypy-zope, attempt 3 (#13993) 2022-09-30 17:36:28 +01:00
test_profile.py Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
test_purge.py Async get event cache prep (#13242) 2022-07-15 09:30:46 +00:00
test_receipts.py Clarify that a method returns only unthreaded receipts. (#13937) 2022-09-29 07:07:31 -04:00
test_redaction.py Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
test_registration.py Allow admins to require a manual approval process before new accounts can be used (using MSC3866) (#13556) 2022-09-29 15:23:24 +02:00
test_relations.py Accept threaded receipts for events related to the root event. (#14174) 2022-10-14 18:05:25 +00:00
test_rollback_worker.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_room_search.py Rename storage classes (#12913) 2022-05-31 12:17:50 +00:00
test_room.py Remove obsolete RoomEventsStoreTestCase (#13200) 2022-07-07 13:47:26 +01:00
test_roommember.py Fix that user cannot /forget rooms after the last member has left (#13546) 2022-08-30 09:58:38 +00:00
test_state.py Make DictionaryCache have better expiry properties (#13292) 2022-07-21 17:13:44 +01:00
test_stream.py Generate historic pagination token for /messages when no ?from token provided (#12370) 2022-04-06 11:40:28 +01:00
test_transactions.py Remove HomeServer.get_datastore() (#12031) 2022-02-23 11:04:02 +00:00
test_txn_limit.py Require direct references to configuration variables. (#10985) 2021-10-06 10:47:41 -04:00
test_unsafe_locale.py Refuse to start if DB has an unsafe locale (#12262) 2022-03-23 10:23:05 -07:00
test_user_directory.py Remove remaining bits of groups code. (#12936) 2022-06-01 09:41:25 -04:00