Commit Graph

40 Commits

Author SHA1 Message Date
Erik Johnston
11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Erik Johnston
458b6f4733 Only invalidate membership caches based on the cache stream
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston
3670025e64 Rename func 2017-01-30 14:11:31 +00:00
Erik Johnston
252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston
f32fb65552 Add new API appservice specific public room list 2016-12-06 16:12:27 +00:00
Erik Johnston
26072df6af Ensure only main or federation_sender process can send federation traffic 2016-11-23 14:09:47 +00:00
Erik Johnston
4c79a63fd7 Explicit federation ack 2016-11-23 10:40:44 +00:00
Erik Johnston
90565d015e Invalidate retry cache in both directions 2016-11-22 17:45:44 +00:00
Erik Johnston
f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston
668f91d707 Fix check of wrong variable 2016-10-11 13:57:22 +01:00
Erik Johnston
748d8fdc7b Reduce DB hits for replication
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.

This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
Erik Johnston
211786ecd6 Stream public room changes down replication 2016-09-15 11:47:23 +01:00
Erik Johnston
a4339de9de Correctly handle typing stream id resetting 2016-09-09 16:44:26 +01:00
Erik Johnston
ab80d5e0a9 Drop replication log levels 2016-09-09 14:56:50 +01:00
Erik Johnston
44982606ee Merge pull request #1060 from matrix-org/erikj/state_ids
Assign state groups in state handler.
2016-09-01 14:20:42 +01:00
Mark Haines
1aa3e1d287 Add a replication stream for direct to device messages 2016-08-31 10:38:58 +01:00
Erik Johnston
3e784eff74 Remove state replication stream 2016-08-30 16:51:36 +01:00
Erik Johnston
64e7e11853 Implement cache replication stream 2016-08-15 11:16:45 +01:00
Mark Haines
70599ce925 Allow external processes to mark a user as syncing. (#812)
* Add infrastructure to the presence handler to track sync requests in external processes

* Expire stale entries for dead external processes

* Add an http endpoint for making users as syncing

Add some docstrings and comments.

* Fixes
2016-06-02 15:20:15 +01:00
Mark Haines
0cb441fedd Move typing handler out of the Handlers object 2016-05-17 15:58:46 +01:00
Mark Haines
3b86ecfa79 Move the presence handler out of the Handlers object 2016-05-16 18:56:37 +01:00
Mark Haines
0e792e7903 Log the stream IDs in an order that makes sense 2016-05-13 11:54:44 +01:00
Erik Johnston
8940281d1b Don't warn 2016-05-05 15:10:03 +01:00
Erik Johnston
5d8a93a10e Add some log information at returned replication streams 2016-05-05 10:29:21 +01:00
Mark Haines
8d7ad44331 Report per request metrics for all of the things using request_handler 2016-04-28 10:57:49 +01:00
Mark Haines
8a65666454 Fix backfill replication to advance the stream correctly 2016-04-27 15:38:43 +01:00
Mark Haines
cfe1ff4bdb Add a replication endpoint for deleting pushers 2016-04-21 16:33:05 +01:00
Mark Haines
1d4deff25a Separate generating the replication response...
from doing the http request parsing to make it easier
to write unit tests for replication.
2016-04-05 11:23:57 +01:00
Mark Haines
62e395f0e3 Merge pull request #676 from matrix-org/markjh/replicate_stateIII
Add replication streams for ex outliers and current state resets
2016-03-31 11:20:57 +01:00
Mark Haines
2ec5426035 Use a namedtuple rather than tuple unpacking 2016-03-31 10:33:02 +01:00
Matthew Hodgson
9113316b0e typo 2016-03-30 17:29:42 +01:00
Mark Haines
1fbb094c6f Add replication streams for ex outliers and current state resets 2016-03-30 17:19:56 +01:00
Mark Haines
31a9eceda5 Add a replication stream for state groups 2016-03-30 16:01:58 +01:00
Mark Haines
b6e8420aee Add replication stream for pushers 2016-03-15 17:33:10 +00:00
Mark Haines
ebcbb23226 s/stream_ordering/event_stream_ordering/ in push 2016-03-04 16:15:23 +00:00
Mark Haines
2223204eba Hook push rules up to the replication API 2016-03-02 17:26:20 +00:00
Mark Haines
60a0f81c7a Add a /replication API for extracting the updates that happened on
synapse

This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.

This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.

Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.

Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.

The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00