synapse-product/synapse
Erik Johnston f112cfe5bb
Fix MultiWriteIdGenerator's handling of restarts. (#8374)
On startup `MultiWriteIdGenerator` fetches the maximum stream ID for
each instance from the table and uses that as its initial "current
position" for each writer. This is problematic as a) it involves either
a scan of events table or an index (neither of which is ideal), and b)
if rows are being persisted out of order elsewhere while the process
restarts then using the maximum stream ID is not correct. This could
theoretically lead to race conditions where e.g. events that are
persisted out of order are not sent down sync streams.

We fix this by creating a new table that tracks the current positions of
each writer to the stream, and update it each time we finish persisting
a new entry. This is a relatively small overhead when persisting events.
However for the cache invalidation stream this is a much bigger relative
overhead, so instead we note that for invalidation we don't actually
care about reliability over restarts (as there's no caches to
invalidate) and simply don't bother reading and writing to the new table
in that particular case.
2020-09-24 16:53:51 +01:00
..
_scripts Remove obsolete __future__ imports (#8337) 2020-09-17 08:37:01 -04:00
api Don't push if an user account has expired (#8353) 2020-09-23 16:06:28 +01:00
app Simplify super() calls to Python 3 syntax. (#8344) 2020-09-18 09:56:44 -04:00
appservice Add type annotations to SimpleHttpClient (#8372) 2020-09-24 15:47:20 +01:00
config Simplify super() calls to Python 3 syntax. (#8344) 2020-09-18 09:56:44 -04:00
crypto Add a comment re #1691 2020-09-21 12:34:06 +01:00
events Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
federation Synapse 1.20.0rc5 (2020-09-18) 2020-09-18 11:17:58 -04:00
groups Simplify super() calls to Python 3 syntax. (#8344) 2020-09-18 09:56:44 -04:00
handlers Add EventStreamPosition type (#8388) 2020-09-24 13:24:17 +01:00
http Add type annotations to SimpleHttpClient (#8372) 2020-09-24 15:47:20 +01:00
logging Simplify super() calls to Python 3 syntax. (#8344) 2020-09-18 09:56:44 -04:00
metrics Use slots in attrs classes where possible (#8296) 2020-09-14 12:50:06 -04:00
module_api Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
push Don't push if an user account has expired (#8353) 2020-09-23 16:06:28 +01:00
replication Fix MultiWriteIdGenerator's handling of restarts. (#8374) 2020-09-24 16:53:51 +01:00
res/templates Improve SAML error messages (#8248) 2020-09-14 09:05:36 -04:00
rest Add type annotations to SimpleHttpClient (#8372) 2020-09-24 15:47:20 +01:00
server_notices Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
spam_checker_api Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
state Improve logging of state resolution (#8371) 2020-09-23 16:42:44 +01:00
static Clean-up the fallback login code. (#7657) 2020-06-10 09:50:39 -04:00
storage Fix MultiWriteIdGenerator's handling of restarts. (#8374) 2020-09-24 16:53:51 +01:00
streams Clean up types for PaginationConfig (#8250) 2020-09-08 15:00:17 +01:00
util Simplify super() calls to Python 3 syntax. (#8344) 2020-09-18 09:56:44 -04:00
__init__.py 1.20.0 2020-09-22 15:18:31 +01:00
event_auth.py Convert events worker database to async/await. (#8071) 2020-08-18 16:20:49 -04:00
notifier.py Add EventStreamPosition type (#8388) 2020-09-24 13:24:17 +01:00
python_dependencies.py Do not check lint/test dependencies at runtime. (#8377) 2020-09-23 11:45:37 +01:00
secrets.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
server.py Fix ratelimiting for federation /send requests. (#8342) 2020-09-18 10:49:29 +01:00
types.py Add EventStreamPosition type (#8388) 2020-09-24 13:24:17 +01:00
visibility.py Convert groups and visibility code to async / await. (#7951) 2020-07-27 12:32:08 -04:00