280 Commits

Author	SHA1	Message	Date
Erik Johnston	cdf2707678	Fix bug in wait for stream position (#14872) ... This caused some requests to fail. This caused some requests to fail. This really only started causing issues due to #14856	2023-01-19 22:19:56 +00:00
Erik Johnston	9187fd940e	Wait for streams to catch up when processing HTTP replication. (#14820) ... This should hopefully mitigate a class of races where data gets out of sync due a HTTP replication request racing with the replication streams.	2023-01-18 19:35:29 +00:00
Erik Johnston	316590d1ea	Fix bug in wait_for_stream_position (#14856) ... We were incorrectly checking if the *local* token had been advanced, rather than the token for the remote instance. In practice, I don't think this has caused any bugs due to where we use `wait_for_stream_position`, as critically we don't use it on instances that also write to the given streams (and so the local token will lag behind all remote tokens).	2023-01-17 09:58:22 +00:00
Erik Johnston	2b084c5b71	Merge device list replication streams (#14833)	2023-01-17 09:29:58 +00:00
Erik Johnston	73ff493dfb	Merge account data streams (#14826)	2023-01-13 14:57:43 +00:00
Nick Mills-Barrett	db1cfe9c80	Update all stream IDs after processing replication rows (#14723) ... This creates a new store method, `process_replication_position` that is called after `process_replication_rows`. By moving stream ID advances here this guarantees any relevant cache invalidations will have been applied before the stream is advanced. This avoids race conditions where Python switches between threads mid way through processing the `process_replication_rows` method where stream IDs may be advanced before caches are invalidated due to class resolution ordering. See this comment/issue for further discussion: https://github.com/matrix-org/synapse/issues/14158#issuecomment-1344048703	2023-01-04 11:49:26 +00:00
reivilibre	2888d7ec83	Faster remote room joins: invalidate caches and unblock requests when receiving un-partial-stated event notifications over replication. [rei:frrj/streams/unpsr] (#14546)	2022-12-19 14:57:51 +00:00
reivilibre	fb60cb16fe	Faster remote room joins: stream the un-partial-stating of events over replication. [rei:frrj/streams/unpsr] (#14545)	2022-12-14 14:47:11 +00:00
reivilibre	9e82caac45	Faster remote room joins: unblock tasks waiting for full room state when the un-partial-stating of that room is received over the replication stream. [rei:frrj/streams/unpsr] (#14474)	2022-12-06 15:48:42 +00:00
reivilibre	501f62d1a6	Faster remote room joins: stream the un-partial-stating of rooms over replication. [rei:frrj/streams/unpsr] (#14473)	2022-12-05 13:07:55 +00:00
Andrew Morgan	e7132c3f81	Fix check to ignore blank lines in incoming TCP replication (#14449)	2022-11-17 16:09:56 +00:00
Shay	7b7478e8b6	Batch up notifications after event persistence (#14033)	2022-10-05 10:12:48 -07:00
Patrick Cloke	efd108b45d	Accept & store thread IDs for receipts (implement MSC3771). (#13782) ... Updates the `/receipts` endpoint and receipt EDU handler to parse a `thread_id` from the body and insert it in the database.	2022-09-23 14:33:28 +00:00
Brendan Abolivier	8ae42ab8fa	Support enabling/disabling pushers (from MSC3881) (#13799) ... Partial implementation of MSC3881	2022-09-21 14:39:01 +00:00
Patrick Cloke	32fc3b7ba4	Remove configuration options for direct TCP replication. (#13647) ... Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.	2022-09-06 07:50:02 +00:00
Šimon Brandner	0e99f07952	Remove support for unstable private read receipts (#13653) ... Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>	2022-09-01 13:31:54 +01:00
Šimon Brandner	ab18441573	Support stable identifiers for MSC2285: private read receipts. (#13273) ... This adds support for the stable identifiers of MSC2285 while continuing to support the unstable identifiers behind the configuration flag. These will be removed in a future version.	2022-08-05 11:09:33 -04:00
David Robertson	b977867358	Rate limit joins per-room (#13276)	2022-07-19 11:45:17 +00:00
Erik Johnston	f721f1baba	Revert "Make all process_replication_rows methods async (#13304)" (#13312) ... This reverts commit 5d4028f217.	2022-07-18 14:28:14 +01:00
Nick Mills-Barrett	5d4028f217	Make all process_replication_rows methods async (#13304) ... More prep work for asyncronous caching, also makes all process_replication_rows methods consistent (presence handler already is so). Signed off by Nick @ Beeper (@Fizzadar)	2022-07-17 22:19:43 +01:00
Patrick Cloke	cf05258f76	Remove groups replication code. (#12900) ... The replication logic for groups is no longer used, so the message passing infrastructure can be removed.	2022-05-31 13:04:08 -04:00
reivilibre	39dee30f01	Send USER_IP commands on a different Redis channel, in order to reduce traffic to workers that do not process these commands. (#12809)	2022-05-20 15:28:23 +01:00
reivilibre	177b884ad7	Lay some foundation work to allow workers to only subscribe to some kinds of messages, reducing replication traffic. (#12672)	2022-05-19 16:29:08 +01:00
Andrew Morgan	83be72d76c	Add StreamKeyType class and replace string literals with constants (#12567)	2022-05-16 15:35:31 +00:00
Shay	d80a7ab151	Update replication.md with info on TCP module structure (#12621)	2022-05-09 14:46:43 -07:00
Šimon Brandner	ef86cf3d28	Update _on_new_receipts() to work with MSC2285 changes. (#12636)	2022-05-05 13:25:51 +00:00
Erik Johnston	c0379d6e5b	Reduce log spam when running multiple event persisters (#12610)	2022-05-05 10:20:23 +01:00
Erik Johnston	d1cd96ce29	Add opentracing spans to calls to external cache (#12380)	2022-04-07 13:18:29 +01:00
Sean Quah	800ba87cc8	Refactor and convert Linearizer to async (#12357) ... Refactor and convert `Linearizer` to async. This makes a `Linearizer` cancellation bug easier to fix. Also refactor to use an async context manager, which eliminates an unlikely footgun where code that doesn't immediately use the context manager could forget to release the lock. Signed-off-by: Sean Quah <seanq@element.io>	2022-04-05 15:43:52 +01:00
reivilibre	f871222880	Move update_client_ip background job from the main process to the background worker. (#12251)	2022-04-01 13:08:55 +01:00
reivilibre	4a53f35737	Improve code documentation for the typing stream over replication. (#12211)	2022-03-11 14:00:15 +00:00
Patrick Cloke	3e4af36bc8	Rename get_tcp_replication to get_replication_command_handler. (#12192) ... Since the object it returns is a ReplicationCommandHandler. This is clean-up from adding support to Redis where the command handler was added as an additional layer of abstraction from the TCP protocol.	2022-03-10 13:01:56 +00:00
Patrick Cloke	d8bab6793c	Fix incorrect type hints for txredis. (#12042) ... Some properties were marked as RedisProtocol instead of ConnectionHandler, which wraps RedisProtocol instance(s).	2022-03-08 07:26:05 -05:00
Erik Johnston	423cca9efe	Spread out sending device lists to remote hosts (#12132)	2022-03-04 11:48:15 +00:00
Richard van der Hoff	e24ff8ebe3	Remove HomeServer.get_datastore() (#12031) ... The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733	2022-02-23 11:04:02 +00:00
Patrick Cloke	d0e78af35e	Add missing type hints to synapse.replication. (#11938)	2022-02-08 11:03:08 -05:00
Patrick Cloke	6c0984e3f0	Remove unnecessary ignores due to Twisted upgrade. (#11939) ... Twisted 22.1.0 fixed some internal type hints, allowing Synapse to remove ignore calls for parameters to connectTCP.	2022-02-08 09:15:59 -05:00
Patrick Cloke	10a88ba91c	Use auto_attribs/native type hints for attrs classes. (#11692)	2022-01-13 13:49:28 +00:00
Patrick Cloke	cbd82d0b2d	Convert all namedtuples to attrs. (#11665) ... To improve type hints throughout the code.	2021-12-30 18:47:12 +00:00
Sean Quah	ffd858aa68	Add type hints to synapse/storage/databases/main/events_worker.py (#11411) ... Also refactor the stream ID trackers/generators a bit and try to document them better.	2021-11-26 18:41:31 +00:00
Patrick Cloke	5cace20bf1	Add missing type hints to synapse.app. (#11287)	2021-11-10 15:06:54 -05:00
Nick Barrett	af54167516	Enable passing typing stream writers as a list. (#11237) ... This makes the typing stream writer config match the other stream writers that only currently support a single worker.	2021-11-03 14:25:47 +00:00
Brendan Abolivier	c7a5e49664	Implement an on_new_event callback (#11126) ... Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2021-10-26 15:17:36 +02:00
Sean Quah	2b82ec425f	Add type hints for most HomeServer parameters (#11095)	2021-10-22 18:15:41 +01:00
Sean Quah	6a67f3786a	Fix logging context warnings when losing replication connection (#10984) ... Instead of triggering `__exit__` manually on the replication handler's logging context, use it as a context manager so that there is an `__enter__` call to balance the `__exit__`.	2021-10-15 13:10:58 +01:00
Patrick Cloke	f4b1a9a527	Require direct references to configuration variables. (#10985) ... This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).	2021-10-06 10:47:41 -04:00
David Robertson	29364145b2	Pass str to twisted's IReactorTCP (#10895) ... This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job. Until that change is in a twisted release, we'll have to ignore the type of the `host` argument. I've raised #10899 to remind us to review the issue in a few months' time.	2021-09-30 12:51:47 +01:00
Patrick Cloke	94b620a5ed	Use direct references for configuration variables (part 6). (#10916)	2021-09-29 06:44:15 -04:00
Patrick Cloke	bb7fdd821b	Use direct references for configuration variables (part 5). (#10897)	2021-09-24 07:25:21 -04:00
Patrick Cloke	01c88a09cd	Use direct references for some configuration variables (#10798) ... Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).	2021-09-13 13:07:12 -04:00