synapse-product/synapse
Erik Johnston 52a50e8686
Use vector clocks for room stream tokens. (#8439)
Currently when using multiple event persisters we (in the worst case) don't tell clients about events until all event persisters have persisted new events after the original event. This is a suboptimal, especially if one of the event persisters goes down.

To handle this, we encode the position of each event persister in the room tokens so that we can send events to clients immediately. To reduce the size of the token we do two things:

1. We create a unique immutable persistent mapping between instance names and a generated small integer ID, which we can encode in the tokens instead of the instance name; and
2. We encode the "persisted upto position" of the room token and then only explicitly include instances that have positions strictly greater than that.

The new tokens look something like: `m3478~1.3488~2.3489`, where the first number is the min position, and the subsequent `-` separated pairs are the instance ID to positions map. (We use `.` and `~` as separators as they're URL safe and not already used by `StreamToken`).
2020-10-07 15:15:33 +01:00
..
_scripts Remove obsolete __future__ imports (#8337) 2020-09-17 08:37:01 -04:00
api Do not assume that account data is of the correct form. (#8454) 2020-10-05 09:28:05 -04:00
app Combine SpamCheckerApi with the more generic ModuleApi. (#8464) 2020-10-07 12:03:26 +01:00
appservice Add type annotations to SimpleHttpClient (#8372) 2020-09-24 15:47:20 +01:00
config Update default room version to 6 (#8461) 2020-10-05 21:40:51 +01:00
crypto Fix occasional "Re-starting finished log context" from keyring (#8398) 2020-09-25 12:29:54 +01:00
events Combine SpamCheckerApi with the more generic ModuleApi. (#8464) 2020-10-07 12:03:26 +01:00
federation Remove stream ordering from Metadata dict (#8452) 2020-10-05 14:43:14 +01:00
groups Simplify super() calls to Python 3 syntax. (#8344) 2020-09-18 09:56:44 -04:00
handlers Add typing information to the device handler. (#8407) 2020-10-07 08:58:21 -04:00
http Don't bother responding to client requests that have already disconnected (#8465) 2020-10-06 10:03:39 +01:00
logging Enable mypy checking for unreachable code and fix instances. (#8432) 2020-10-01 08:09:18 -04:00
metrics Rewrite BucketCollector 2020-09-30 16:49:15 +01:00
module_api Combine SpamCheckerApi with the more generic ModuleApi. (#8464) 2020-10-07 12:03:26 +01:00
push Enable mypy checking for unreachable code and fix instances. (#8432) 2020-10-01 08:09:18 -04:00
replication Add unit test for event persister sharding (#8433) 2020-10-02 09:57:12 +01:00
res/templates Convert additional templates to Jinja (#8444) 2020-10-02 11:15:53 +01:00
rest Add support for MSC2697: Dehydrated devices (#8380) 2020-10-07 08:00:17 -04:00
server_notices Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
spam_checker_api Combine SpamCheckerApi with the more generic ModuleApi. (#8464) 2020-10-07 12:03:26 +01:00
state Enable mypy checking for unreachable code and fix instances. (#8432) 2020-10-01 08:09:18 -04:00
static Clean-up the fallback login code. (#7657) 2020-06-10 09:50:39 -04:00
storage Use vector clocks for room stream tokens. (#8439) 2020-10-07 15:15:33 +01:00
streams Make token serializing/deserializing async (#8427) 2020-09-30 20:29:19 +01:00
util Expose a get_resource_usage method in Measure 2020-09-29 17:35:20 +01:00
__init__.py 1.21.0rc2 2020-10-02 12:46:58 +01:00
event_auth.py Ensure that event.redacts is the proper type before handling it (#8457) 2020-10-05 10:24:17 -04:00
notifier.py Various clean ups to room stream tokens. (#8423) 2020-09-29 21:48:33 +01:00
python_dependencies.py Drop support for ancient prometheus_client (#8426) 2020-09-30 16:42:05 +01:00
secrets.py Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
server.py Combine SpamCheckerApi with the more generic ModuleApi. (#8464) 2020-10-07 12:03:26 +01:00
types.py Use vector clocks for room stream tokens. (#8439) 2020-10-07 15:15:33 +01:00
visibility.py Do not assume that account data is of the correct form. (#8454) 2020-10-05 09:28:05 -04:00