synapse-product/synapse/storage/databases/main
David Robertson 80d44060c9
Faster joins: omit partial rooms from eager syncs until the resync completes (#14870)
* Allow `AbstractSet` in `StrCollection`

Or else frozensets are excluded. This will be useful in an upcoming
commit where I plan to change a function that accepts `List[str]` to
accept `StrCollection` instead.

* `rooms_to_exclude` -> `rooms_to_exclude_globally`

I am about to make use of this exclusion mechanism to exclude rooms for
a specific user and a specific sync. This rename helps to clarify the
distinction between the global config and the rooms to exclude for a
specific sync.

* Better function names for internal sync methods

* Track a list of excluded rooms on SyncResultBuilder

I plan to feed a list of partially stated rooms for this sync to ignore

* Exclude partial state rooms during eager sync

using the mechanism established in the previous commit

* Track un-partial-state stream in sync tokens

So that we can work out which rooms have become fully-stated during a
given sync period.

* Fix mutation of `@cached` return value

This was fouling up a complement test added alongside this PR.
Excluding a room would mean the set of forgotten rooms in the cache
would be extended. This means that room could be erroneously considered
forgotten in the future.

Introduced in #12310, Synapse 1.57.0. I don't think this had any
user-visible side effects (until now).

* SyncResultBuilder: track rooms to force as newly joined

Similar plan as before. We've omitted rooms from certain sync responses;
now we establish the mechanism to reintroduce them into future syncs.

* Read new field, to present rooms as newly joined

* Force un-partial-stated rooms to be newly-joined

for eager incremental syncs only, provided they're still fully stated

* Notify user stream listeners to wake up long polling syncs

* Changelog

* Typo fix

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Unnecessary list cast

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Rephrase comment

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Another comment

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Fixup merge(?)

* Poke notifier when receiving un-partial-stated msg over replication

* Fixup merge whoops

Thanks MV :)

Co-authored-by: Mathieu Velen <mathieuv@matrix.org>

Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-01-23 15:44:39 +00:00
..
__init__.py Merge/remove Slaved* stores into WorkerStores (#14375) 2022-11-11 10:51:49 +00:00
account_data.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
appservice.py Use device_one_time_keys_count to match MSC3202 (#14565) 2022-11-28 16:17:29 +00:00
cache.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
censor_events.py Safe async event cache (#13308) 2022-07-19 11:25:29 +00:00
client_ips.py Optimise _update_client_ips_batch_txn to batch together database operations. (#12252) 2022-04-08 15:29:13 +01:00
deviceinbox.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
devices.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
directory.py Replace uses of simple_insert_many with simple_insert_many_values. (#11742) 2022-01-13 19:44:18 -05:00
e2e_room_keys.py Remove redundant types from comments. (#14412) 2022-11-16 15:25:24 +00:00
end_to_end_keys.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
event_federation.py Remove option to skip locking of tables during emulated upserts (#14469) 2022-11-28 13:42:06 +00:00
event_push_actions.py Delete event_push_summary_unique_index again. (#14669) 2022-12-14 09:25:33 -05:00
events_bg_updates.py Add index to improve performance of the /timestamp_to_event endpoint used for jumping to a specific date in the timeline of a room. (#14799) 2023-01-11 12:29:13 +00:00
events_forward_extremities.py Fix returned count of delete extremities admin API (#12496) 2022-04-19 16:49:45 +01:00
events_worker.py Skip processing stats for broken rooms. (#14873) 2023-01-23 11:36:20 +00:00
events.py Batch fetch bundled references (#14508) 2022-11-22 09:41:09 -05:00
filtering.py Merge/remove Slaved* stores into WorkerStores (#14375) 2022-11-11 10:51:49 +00:00
keys.py Add some type hints to datastore (#12485) 2022-04-27 13:05:00 +01:00
lock.py Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
media_repository.py Replace noop background updates with DELETE. (#12954) 2022-06-13 14:06:27 -04:00
metrics.py Add some type hints to datastore (#12717) 2022-05-17 15:29:06 +01:00
monthly_active_users.py Remove redundant types from comments. (#14412) 2022-11-16 15:25:24 +00:00
openid.py Add type hints to some storage classes (#11307) 2021-11-11 08:47:31 -05:00
presence.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
profile.py Remove remaining pieces of groups code. (#12966) 2022-06-06 13:20:05 -04:00
purge_events.py Add support to purge rows from MSC2716 and other tables when purging a room (#13825) 2022-09-16 10:56:56 -05:00
push_rule.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
pusher.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
receipts.py Always notify replication when a stream advances (#14877) 2023-01-20 18:02:18 +00:00
registration.py Remove redundant types from comments. (#14412) 2022-11-16 15:25:24 +00:00
rejections.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
relations.py Faster joins: omit partial rooms from eager syncs until the resync completes (#14870) 2023-01-23 15:44:39 +00:00
room_batch.py Remove option to skip locking of tables during emulated upserts (#14469) 2022-11-28 13:42:06 +00:00
room.py Faster joins: omit partial rooms from eager syncs until the resync completes (#14870) 2023-01-23 15:44:39 +00:00
roommember.py Faster joins: omit partial rooms from eager syncs until the resync completes (#14870) 2023-01-23 15:44:39 +00:00
search.py Drop support for Postgres 10 in full text search code. (#14397) 2022-11-09 09:55:34 -05:00
session.py Run pyupgrade --py37-plus --keep-percent-format on Synapse (#11685) 2022-01-05 09:53:05 -08:00
signatures.py remove constantly lib use and switch to enums. (#12624) 2022-05-04 11:26:11 +00:00
state_deltas.py Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
state.py Enable Faster Remote Room Joins against worker-mode Synapse. (#14752) 2023-01-22 21:10:11 +00:00
stats.py Skip processing stats for broken rooms. (#14873) 2023-01-23 11:36:20 +00:00
stream.py Fix race calling /members?at= (#14817) 2023-01-12 10:29:09 +00:00
tags.py Merge account data streams (#14826) 2023-01-13 14:57:43 +00:00
transactions.py Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
ui_auth.py Add some type hints to datastore (#12485) 2022-04-27 13:05:00 +01:00
user_directory.py Add optional ICU support for user search (#14464) 2022-12-12 13:21:17 +01:00
user_erasure_store.py Annotations for user_erasure_store (#11313) 2021-11-11 19:22:19 +00:00