Commit Graph

125 Commits

Author SHA1 Message Date
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Richard van der Hoff
29ed09e80a Fix assertion to stop transaction queue getting wedged
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
Erik Johnston
45c7f12d2a Add new storage function to slave store 2017-03-13 16:26:44 +00:00
Erik Johnston
8f267fa8a8 Fix it for the workers 2017-03-10 11:22:25 +00:00
Erik Johnston
e933a2712d Don't log unknown cache warnings in workers 2017-02-28 16:22:41 +00:00
Erik Johnston
095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston
9e617cd4c2 Cache get_presence storage 2017-02-13 13:50:03 +00:00
Erik Johnston
6bba80241c Merge pull request #1912 from matrix-org/markjh/roominitialsync
Add db functions needed for room initial sync to slave
2017-02-13 12:20:21 +01:00
Mark Haines
3a46280ca3 Add db functions needed for room initial sync to slave 2017-02-13 11:16:53 +00:00
Erik Johnston
0f3e296cb7 Fix replication 2017-02-02 15:02:03 +00:00
Erik Johnston
458b6f4733 Only invalidate membership caches based on the cache stream
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston
252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston
a55fa2047f Insert delta of current_state_events to be more efficient 2017-01-20 17:10:18 +00:00
Erik Johnston
09cbcb78d3 Add cache to get_public_room_ids_at_stream_id 2016-12-12 14:41:51 +00:00
Erik Johnston
90565d015e Invalidate retry cache in both directions 2016-11-22 17:45:44 +00:00
Erik Johnston
7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston
995f2f032f Fix public room pagination for client_reader app 2016-09-16 14:48:21 +01:00
Erik Johnston
418bcd4309 Add new storage function to slave store 2016-09-16 08:37:39 +01:00
Erik Johnston
cb3edec6af Use stream_change cache to make get_forward_extremeties_for_room cache more effective 2016-09-15 14:28:13 +01:00
Erik Johnston
55e6fc917c Add cache to get_forward_extremeties_for_room 2016-09-15 14:04:28 +01:00
Erik Johnston
211786ecd6 Stream public room changes down replication 2016-09-15 11:47:23 +01:00
Mark Haines
6a6cbfcf1e Track the max_stream_device_id in a separate table, since we delete from the inbox table 2016-09-09 11:48:23 +01:00
Mark Haines
fa9d36e050 Merge branch 'develop' into markjh/direct_to_device_federation 2016-09-08 13:43:43 +01:00
Mark Haines
2a0159b8ae Fix the stream change cache to work over replication 2016-09-07 15:58:00 +01:00
Erik Johnston
a99e933550 Add upgrade script that will slowly prune state_groups_state entries 2016-09-05 10:05:36 +01:00
Erik Johnston
44982606ee Merge pull request #1060 from matrix-org/erikj/state_ids
Assign state groups in state handler.
2016-09-01 14:20:42 +01:00
Erik Johnston
826ca61745 Add storage function to SlaveStore 2016-08-31 14:45:04 +01:00
Mark Haines
1aa3e1d287 Add a replication stream for direct to device messages 2016-08-31 10:38:58 +01:00
Erik Johnston
55fc17cf4b Merge pull request #1049 from matrix-org/erikj/presence_users_in_room
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-30 10:50:37 +01:00
Erik Johnston
41788bba50 Add to slave store 2016-08-30 09:55:17 +01:00
Erik Johnston
873f870e5a Add new direct message storage functions to slave store 2016-08-30 09:40:32 +01:00
Erik Johnston
bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Erik Johnston
4daa397a00 Add is_host_joined to slave storage 2016-08-26 13:10:56 +01:00
Erik Johnston
50943ab942 Add new state storage funcs to replication 2016-08-26 09:57:32 +01:00
Erik Johnston
07229bbdae Add appservice worker 2016-08-18 14:59:55 +01:00
Erik Johnston
a2427981b7 Use cached get_user_by_access_token in slaves 2016-08-16 11:24:32 +01:00
Erik Johnston
d9664344ec Rename table. Add docs. 2016-08-15 11:45:57 +01:00
Erik Johnston
0be963472b Use cached version of get_aliases_for_room 2016-08-15 11:24:12 +01:00
Erik Johnston
64e7e11853 Implement cache replication stream 2016-08-15 11:16:45 +01:00
Erik Johnston
5aa024e501 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader 2016-07-29 11:24:56 +01:00
Mark Haines
0a7d3cd00f Create separate methods for getting messages to push
for the email and http pushers rather than trying to make a single
method that will work with their conflicting requirements.

The http pusher needs to get the messages in ascending stream order, and
doesn't want to miss a message.

The email pusher needs to get the messages in descending timestamp order,
and doesn't mind if it misses messages.
2016-07-28 20:24:24 +01:00
Erik Johnston
ec8b217722 Add destination retry to slave store 2016-07-28 17:35:53 +01:00
Erik Johnston
76b89d0edb Add slace storage functions for public room list 2016-07-28 17:03:40 +01:00
Erik Johnston
0fcbca531f Add get_auth_chain to slave store 2016-07-28 16:36:28 +01:00
Erik Johnston
6ede23ff1b Add more key storage funcs into slave store 2016-07-28 15:41:26 +01:00
Erik Johnston
aede7248ab Split out a FederationReader process 2016-07-21 17:37:44 +01:00
David Baker
46b7362304 pep8 2016-06-24 11:44:57 +01:00
David Baker
870c45913e Use similar naming we use in email notifs for push
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
Erik Johnston
05e01f21d7 Remove event fetching from DB threads 2016-06-03 17:22:13 +01:00
Mark Haines
20eccd84d4 Merge pull request #827 from matrix-org/markjh/more_slaved_methods
Add methods to events, account data and receipt slaves
2016-06-03 11:46:21 +01:00
Mark Haines
b09f348530 Merge pull request #824 from matrix-org/markjh/slaved_presence_store
Add a slaved store for presence
2016-06-03 11:26:33 +01:00
Mark Haines
81cf449daa Add methods to events, account data and receipt slaves
Adds the methods needed by /sync to the slaved events,
account data and receipt stores.
2016-06-03 11:19:27 +01:00
Mark Haines
3ae915b27e Add a slaved store for presence 2016-06-03 11:05:53 +01:00
Mark Haines
f88d747f79 Add a comment explaining why the filter cache doesn't need exipiring 2016-06-03 11:03:10 +01:00
Mark Haines
0eae075723 Add slaved stores for filters, tokens, and push rules 2016-06-03 10:58:03 +01:00
Mark Haines
0466454b00 Assert that stream replicated stream positions are ints 2016-05-13 17:33:44 +01:00
Mark Haines
1f71f386f6 Merge branch 'develop' into dbkr/email_notifs_on_pusher 2016-05-13 16:59:56 +01:00
Mark Haines
206eb9fd94 Shift some of the state_group methods into the SlavedEventStore 2016-05-13 16:58:14 +01:00
Mark Haines
3abab26458 Add a slaved datastore for account data 2016-05-13 15:34:06 +01:00
Mark Haines
3547e66bc6 Make sure we advance our stream position 2016-05-13 11:53:00 +01:00
David Baker
5f46be19a7 Pass through get_events to pusher too 2016-05-10 18:43:40 +02:00
David Baker
d46b18a00f Pass through _get_event_txn 2016-05-10 18:27:06 +02:00
Mark Haines
8a65666454 Fix backfill replication to advance the stream correctly 2016-04-27 15:38:43 +01:00
Mark Haines
78741cf025 Merge pull request #743 from matrix-org/markjh/slave_pushers
Replicate the pushers
2016-04-21 17:21:29 +01:00
Mark Haines
d4823efad9 Replicate the pushers 2016-04-21 16:18:00 +01:00
Mark Haines
c0d8e0eb63 Replicate push actions 2016-04-21 15:25:58 +01:00
Mark Haines
f52dd35ac3 Merge pull request #738 from matrix-org/markjh/slaved_receipts
Add a slaved receipts store
2016-04-19 17:31:59 +01:00
Mark Haines
5bbd424ee0 Add a slaved receipts store 2016-04-19 17:14:08 +01:00
Mark Haines
e99365f601 Replicate get_invited_rooms_for_user 2016-04-19 15:22:14 +01:00
Mark Haines
ceb599e789 Add tests for redactions 2016-04-07 16:52:07 +01:00
Mark Haines
60ec9793fb Add tests for get_latest_event_ids_in_room and get_current_state 2016-04-07 13:17:56 +01:00
Mark Haines
6bfec56796 Test that room membership is replicated 2016-04-06 16:20:13 +01:00
Mark Haines
75fb9ac1be Add a slaved events store class
Add a test to check that get_room_names_and_aliases does the same
thing on both the master and on the slave data store.
2016-04-06 14:18:35 +01:00