Erik Johnston
a5c401bd12
Merge pull request #2097 from matrix-org/erikj/repl_tcp_client
...
Move to using TCP replication
2017-04-05 09:36:21 +01:00
Erik Johnston
a76886726b
Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix
...
Advance replication streams even if nothing is listening
2017-04-04 15:40:51 +01:00
Erik Johnston
4264ceb31c
Fiddle tcp replication logging
2017-04-04 14:14:03 +01:00
Erik Johnston
023ee197be
Advance replication streams even if nothing is listening
...
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
2017-04-04 13:19:26 +01:00
Erik Johnston
3a1f3f8388
Change slave storage to use new replication interface
...
As the TCP replication uses a slightly different API and streams than
the HTTP replication.
This breaks HTTP replication.
2017-04-03 15:34:19 +01:00
Erik Johnston
52bfa604e1
Add basic replication client handler and factory
2017-04-03 15:34:13 +01:00
Erik Johnston
0a6a966e2b
Always advance stream tokens
2017-04-03 15:22:56 +01:00
Erik Johnston
1df7c28661
Use callbacks to notify tcp replication rather than deferreds
2017-03-31 15:42:51 +01:00
Erik Johnston
36d2b66f90
Add a timestamp to USER_SYNC command
...
This timestamp is used to indicate when the user last sync'd
2017-03-31 15:42:22 +01:00
Erik Johnston
bfcf016714
Fix up docs
2017-03-31 11:19:24 +01:00
Erik Johnston
4d7fc7f977
Add server side resource for tcp replication
2017-03-30 13:24:45 +01:00
Erik Johnston
7450693435
Initial TCP protocol implementation
...
This defines the low level TCP replication protocol
2017-03-30 12:54:46 +01:00
Erik Johnston
8da6f0be48
Define the various streams we will replicate
2017-03-30 12:54:46 +01:00
Erik Johnston
11880103b1
Make federation send queue take the current position
2017-03-30 12:54:36 +01:00
Erik Johnston
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
Erik Johnston
09f79aaad0
Use presence replication stream to invalidate cache
...
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
2017-03-24 13:21:08 +00:00
Erik Johnston
d58b1ffe94
Replace some calls to cursor_to_dict
...
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
2017-03-24 11:07:02 +00:00
Erik Johnston
aac6d1fc9b
PEP8
2017-03-20 13:47:56 +00:00
Erik Johnston
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
Richard van der Hoff
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
Erik Johnston
45c7f12d2a
Add new storage function to slave store
2017-03-13 16:26:44 +00:00
Erik Johnston
8f267fa8a8
Fix it for the workers
2017-03-10 11:22:25 +00:00
Erik Johnston
e933a2712d
Don't log unknown cache warnings in workers
2017-02-28 16:22:41 +00:00
Erik Johnston
095b45c165
Aggregate event push actions
2017-02-14 13:39:41 +00:00
Erik Johnston
9e617cd4c2
Cache get_presence storage
2017-02-13 13:50:03 +00:00
Erik Johnston
6bba80241c
Merge pull request #1912 from matrix-org/markjh/roominitialsync
...
Add db functions needed for room initial sync to slave
2017-02-13 12:20:21 +01:00
Mark Haines
3a46280ca3
Add db functions needed for room initial sync to slave
2017-02-13 11:16:53 +00:00
Erik Johnston
0f3e296cb7
Fix replication
2017-02-02 15:02:03 +00:00
Erik Johnston
458b6f4733
Only invalidate membership caches based on the cache stream
...
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston
3670025e64
Rename func
2017-01-30 14:11:31 +00:00
Erik Johnston
252b503fc8
Hook device list updates to replication
2017-01-27 14:31:35 +00:00
Erik Johnston
a55fa2047f
Insert delta of current_state_events to be more efficient
2017-01-20 17:10:18 +00:00
Erik Johnston
09cbcb78d3
Add cache to get_public_room_ids_at_stream_id
2016-12-12 14:41:51 +00:00
Erik Johnston
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
Erik Johnston
26072df6af
Ensure only main or federation_sender process can send federation traffic
2016-11-23 14:09:47 +00:00
Erik Johnston
4c79a63fd7
Explicit federation ack
2016-11-23 10:40:44 +00:00
Erik Johnston
90565d015e
Invalidate retry cache in both directions
2016-11-22 17:45:44 +00:00
Erik Johnston
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
Erik Johnston
668f91d707
Fix check of wrong variable
2016-10-11 13:57:22 +01:00
Erik Johnston
748d8fdc7b
Reduce DB hits for replication
...
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.
This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
Erik Johnston
995f2f032f
Fix public room pagination for client_reader app
2016-09-16 14:48:21 +01:00
Erik Johnston
418bcd4309
Add new storage function to slave store
2016-09-16 08:37:39 +01:00
Erik Johnston
cb3edec6af
Use stream_change cache to make get_forward_extremeties_for_room cache more effective
2016-09-15 14:28:13 +01:00
Erik Johnston
55e6fc917c
Add cache to get_forward_extremeties_for_room
2016-09-15 14:04:28 +01:00
Erik Johnston
211786ecd6
Stream public room changes down replication
2016-09-15 11:47:23 +01:00
Erik Johnston
a4339de9de
Correctly handle typing stream id resetting
2016-09-09 16:44:26 +01:00
Erik Johnston
ab80d5e0a9
Drop replication log levels
2016-09-09 14:56:50 +01:00
Mark Haines
6a6cbfcf1e
Track the max_stream_device_id in a separate table, since we delete from the inbox table
2016-09-09 11:48:23 +01:00
