anonymousland-synapse/synapse/storage
Erik Johnston f112cfe5bb
Fix MultiWriteIdGenerator's handling of restarts. (#8374)
On startup `MultiWriteIdGenerator` fetches the maximum stream ID for
each instance from the table and uses that as its initial "current
position" for each writer. This is problematic as a) it involves either
a scan of events table or an index (neither of which is ideal), and b)
if rows are being persisted out of order elsewhere while the process
restarts then using the maximum stream ID is not correct. This could
theoretically lead to race conditions where e.g. events that are
persisted out of order are not sent down sync streams.

We fix this by creating a new table that tracks the current positions of
each writer to the stream, and update it each time we finish persisting
a new entry. This is a relatively small overhead when persisting events.
However for the cache invalidation stream this is a much bigger relative
overhead, so instead we note that for invalidation we don't actually
care about reliability over restarts (as there's no caches to
invalidate) and simply don't bother reading and writing to the new table
in that particular case.
2020-09-24 16:53:51 +01:00
..
databases Fix MultiWriteIdGenerator's handling of restarts. (#8374) 2020-09-24 16:53:51 +01:00
engines Use SequenceGenerator for state group ID allocation 2020-07-16 11:25:08 +01:00
schema Only run one background update at a time 2020-03-31 17:43:58 +01:00
util Fix MultiWriteIdGenerator's handling of restarts. (#8374) 2020-09-24 16:53:51 +01:00
__init__.py Make StreamToken.room_key be a RoomStreamToken instance. (#8281) 2020-09-11 12:22:55 +01:00
_base.py Be stricter about JSON that is accepted by Synapse (#8106) 2020-08-19 07:26:03 -04:00
background_updates.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
database.py Catch-up after Federation Outage (split, 1) (#8230) 2020-09-04 12:22:23 +01:00
keys.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
persist_events.py Add EventStreamPosition type (#8388) 2020-09-24 13:24:17 +01:00
prepare_database.py Use slots in attrs classes where possible (#8296) 2020-09-14 12:50:06 -04:00
purge_events.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
push_rule.py Move storage classes into a main "data store". 2019-10-21 16:05:06 +01:00
relations.py Use slots in attrs classes where possible (#8296) 2020-09-14 12:50:06 -04:00
roommember.py Add EventStreamPosition type (#8388) 2020-09-24 13:24:17 +01:00
state.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
types.py isort 5 compatibility (#7786) 2020-07-05 16:32:02 +01:00