Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2025-05-08 04:54:57 -04:00
Code Issues Projects Releases Packages Wiki Activity

Make event persisters periodically announce position over replication. (#8499)

Browse source 
Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress.

This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.
This commit is contained in:
Erik Johnston 2020-10-12 15:51:41 +01:00 • committed by GitHub
parent 6905f5751a
commit 8de3703d21
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 128 additions and 34 deletions
Download patch file Download diff file Expand all files Collapse all files

36
synapse/replication/tcp/commands.py
View file

@ -141,15 +141,23 @@ class RdataCommand(Command):
class PositionCommand(Command):
"""Sent by the server to tell the client the stream position without
needing to send an RDATA.
"""Sent by an instance to tell others the stream position without needing to
send an RDATA.
Two tokens are sent, the new position and the last position sent by the
instance (in an RDATA or other POSITION). The tokens are chosen so that *no*
rows were written by the instance between the `prev_token` and `new_token`.
(If an instance hasn't sent a position before then the new position can be
used for both.)
Format::
POSITION <stream_name> <instance_name> <token>
POSITION <stream_name> <instance_name> <prev_token> <new_token>
On receipt of a POSITION command clients should check if they have missed
any updates, and if so then fetch them out of band.
On receipt of a POSITION command instances should check if they have missed
any updates, and if so then fetch them out of band. Instances can check this
by comparing their view of the current token for the sending instance with
the included `prev_token`.
The `<instance_name>` is the process that sent the command and is the source
of the stream.
@ -157,18 +165,26 @@ class PositionCommand(Command):
NAME = "POSITION"
def __init__(self, stream_name, instance_name, token):
def __init__(self, stream_name, instance_name, prev_token, new_token):
self.stream_name = stream_name
self.instance_name = instance_name
self.token = token
self.prev_token = prev_token
self.new_token = new_token
@classmethod
def from_line(cls, line):
stream_name, instance_name, token = line.split(" ", 2)
return cls(stream_name, instance_name, int(token))
stream_name, instance_name, prev_token, new_token = line.split(" ", 3)
return cls(stream_name, instance_name, int(prev_token), int(new_token))
def to_line(self):
return " ".join((self.stream_name, self.instance_name, str(self.token)))
return " ".join(
(
self.stream_name,
self.instance_name,
str(self.prev_token),
str(self.new_token),
)
)
class ErrorCommand(_SimpleCommand):