Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
10,851 Commits 5 Branches 723 Tags 124 MiB
85ba83eb51
Commit Graph

234 Commits

Author SHA1 Message Date
Erik Johnston
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers 
Split PusherStore
 2018-02-23 11:23:35 +00:00
Erik Johnston
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store 
Split PushRulesStore
 2018-02-23 11:23:23 +00:00
Erik Johnston
70349872c2 Update copyright 2018-02-23 11:14:35 +00:00
Erik Johnston
eba93b05bf Split EventsWorkerStore into separate file 2018-02-23 11:01:21 +00:00
Erik Johnston
bf8a36e080 Update copyright 2018-02-23 10:52:10 +00:00
Erik Johnston
c2ecfcc3a4 Update copyright 2018-02-23 10:41:34 +00:00
Erik Johnston
7e6cf89dc2 Update copyright 2018-02-23 10:39:19 +00:00
Erik Johnston
26d37f7a63 Update copyright 2018-02-23 10:33:55 +00:00
Erik Johnston
bb73f55fc6 Use absolute imports 2018-02-23 10:31:16 +00:00
Erik Johnston
faeb369f15 Fix missing invalidations for receipt storage 2018-02-21 15:19:54 +00:00
Erik Johnston
3dec9c66b3 Split out RoomMemberStore 2018-02-21 12:07:26 +00:00
Erik Johnston
46244b2759 Split AS stores 2018-02-21 11:49:34 +00:00
Erik Johnston
27b094f382 Split out get_events and co into a worker store 2018-02-21 11:41:48 +00:00
Erik Johnston
d15d237b0d Split out EventPushActionWorkerStore 2018-02-21 11:01:13 +00:00
Erik Johnston
6f72765371 Split PusherStore 2018-02-21 10:54:21 +00:00
Erik Johnston
cbaad969f9 Split PushRulesStore 2018-02-21 10:43:31 +00:00
Erik Johnston
ca9b9d9703 Split AccountDataStore and TagStore 2018-02-21 10:15:04 +00:00
Erik Johnston
95e4cffd85 Fix comment 2018-02-20 17:58:40 +00:00
Erik Johnston
e316bbb4c0 Use abstract base class to access stream IDs 2018-02-20 17:43:57 +00:00
Erik Johnston
f5ac4dc2d4 Split ReceiptsStore 2018-02-20 16:28:28 +00:00
Erik Johnston
106906a65e Don't serialize current state over replication 2018-02-15 13:53:18 +00:00
Erik Johnston
ef344b10e5 Don't log errors propogated from send_event 2018-02-15 11:03:49 +00:00
Erik Johnston
8ec2e638be Add event_creator worker 2018-02-07 10:32:32 +00:00
Erik Johnston
24dd73028a Add replication http endpoint for event sending 2018-02-07 10:32:32 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events (#2784) 
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
 2018-02-06 14:31:24 +00:00
Richard van der Hoff
5c3c32f16f Metrics for number of RDATA commands received 
I found myself wishing we had this.
 2018-01-15 17:45:55 +00:00
Richard van der Hoff
0edf085b68 Fix some logcontext leaks in replication resource 
The @measure_func annotations rely on the wrapped function respecting the
logcontext rules. Add the necessary yields to make this work.
 2017-11-23 23:19:43 +00:00
Richard van der Hoff
35a4b63240 Pull out bits of StateStore to a mixin 
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.

fixes https://github.com/matrix-org/synapse/issues/2655.
 2017-11-14 11:43:58 +00:00
Richard van der Hoff
6cfee09be9 Make __init__ consitstent across Store heirarchy 
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
 2017-11-13 10:46:07 +00:00
Richard van der Hoff
eaaabc6c4f replace 'except:' with 'except Exception:' 
what could possibly go wrong
 2017-10-23 15:52:32 +01:00
hera
f807f7f804 log when we get an exception handling replication updates 2017-10-12 11:51:24 +01:00
Erik Johnston
2cc998fed8 Fix replication. And notify 2017-07-20 17:13:18 +01:00
Erik Johnston
925b3638ff Reduce log levels in tcp replication 2017-07-11 10:04:21 +01:00
Erik Johnston
27f26e48b7 Serialize user ip command as json 2017-06-27 16:25:38 +01:00
Erik Johnston
8c23221666 Fix up 2017-06-27 15:53:45 +01:00
Erik Johnston
78cefd78d6 Make workers report to master for user ip updates 2017-06-27 14:58:10 +01:00
Erik Johnston
dae9a00a28 Initialise exclusive_user_regex 2017-06-21 14:19:33 +01:00
Erik Johnston
8177563ebe Fix for workers 2017-06-21 13:57:49 +01:00
Erik Johnston
6aa5bc8635 Initial worker impl 2017-06-16 11:47:11 +01:00
Erik Johnston
d53fe399eb Add cache for is_host_joined 2017-06-13 09:56:18 +01:00
Erik Johnston
a837765e8c Merge pull request #2266 from matrix-org/erikj/host_in_room 
Change is_host_joined to use current_state table
 2017-06-12 09:49:51 +01:00
Erik Johnston
8060974344 Fix replication 2017-06-09 16:40:52 +01:00
Erik Johnston
2cac7623a5 Add missing notifier 2017-06-09 11:24:41 +01:00
Erik Johnston
298d83b340 Fix replication 2017-06-09 11:01:28 +01:00
Erik Johnston
dfbda5e025 Faster cache for get_joined_hosts 2017-05-25 17:24:44 +01:00
Erik Johnston
f85a415279 Add missing storage function to slave store 2017-05-22 16:31:24 +01:00
Erik Johnston
9ac263ed1b Add new storage functions to slave store 2017-05-04 14:29:03 +01:00
Erik Johnston
e4f3431116 Remove unused cache 2017-04-24 13:27:38 +01:00
Erik Johnston
247c736b9b Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl 
Reduce federation replication traffic
 2017-04-12 11:07:13 +01:00
Erik Johnston
9c712a366f Move get_presence_list_* to SlaveStore 2017-04-11 16:07:33 +01:00
Erik Johnston
28a4649785 Remove HTTP replication APIs 2017-04-11 09:52:11 +01:00
Erik Johnston
29574fd5b3 Reduce federation presence replication traffic 
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
 2017-04-10 16:48:30 +01:00
Erik Johnston
2e6f5a4910 Typo 2017-04-10 16:17:40 +01:00
Erik Johnston
efcb6db688 Merge pull request #2109 from matrix-org/erikj/send_queue_fix 
Fix up federation SendQueue and document types
 2017-04-10 13:09:25 +01:00
Erik Johnston
0364d23210 Up replication ping timeout 2017-04-10 11:32:05 +01:00
Erik Johnston
ab904caf33 Comments 2017-04-10 10:02:17 +01:00
Erik Johnston
98ce212093 Merge pull request #2103 from matrix-org/erikj/no-double-encode 
Don't double encode replication data
 2017-04-07 09:39:52 +01:00
Erik Johnston
ad544c803a Document types of the replication streams 2017-04-06 13:28:52 +01:00
Erik Johnston
69b3fd485d Fix incorrect type when using InvalidateCacheCommand 2017-04-06 09:36:38 +01:00
Erik Johnston
fcc803b2bf Add log lines 2017-04-05 17:13:44 +01:00
Erik Johnston
3f213d908d Rearrange metrics 2017-04-05 14:15:09 +01:00
Erik Johnston
1ca0e78ca1 Fix typo 2017-04-05 13:43:39 +01:00
Erik Johnston
b43d3267e2 Fixup some metrics for tcp repl 2017-04-05 13:34:54 +01:00
Erik Johnston
a5c401bd12 Merge pull request #2097 from matrix-org/erikj/repl_tcp_client 
Move to using TCP replication
 2017-04-05 09:36:21 +01:00
Erik Johnston
a76886726b Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix 
Advance replication streams even if nothing is listening
 2017-04-04 15:40:51 +01:00
Erik Johnston
4264ceb31c Fiddle tcp replication logging 2017-04-04 14:14:03 +01:00
Erik Johnston
023ee197be Advance replication streams even if nothing is listening 
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
 2017-04-04 13:19:26 +01:00
Erik Johnston
3a1f3f8388 Change slave storage to use new replication interface 
As the TCP replication uses a slightly different API and streams than
the HTTP replication.

This breaks HTTP replication.
 2017-04-03 15:34:19 +01:00
Erik Johnston
52bfa604e1 Add basic replication client handler and factory 2017-04-03 15:34:13 +01:00
Erik Johnston
0a6a966e2b Always advance stream tokens 2017-04-03 15:22:56 +01:00
Erik Johnston
1df7c28661 Use callbacks to notify tcp replication rather than deferreds 2017-03-31 15:42:51 +01:00
Erik Johnston
36d2b66f90 Add a timestamp to USER_SYNC command 
This timestamp is used to indicate when the user last sync'd
 2017-03-31 15:42:22 +01:00
Erik Johnston
bfcf016714 Fix up docs 2017-03-31 11:19:24 +01:00
Erik Johnston
4d7fc7f977 Add server side resource for tcp replication 2017-03-30 13:24:45 +01:00
Erik Johnston
7450693435 Initial TCP protocol implementation 
This defines the low level TCP replication protocol
 2017-03-30 12:54:46 +01:00
Erik Johnston
8da6f0be48 Define the various streams we will replicate 2017-03-30 12:54:46 +01:00
Erik Johnston
11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication 
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
 2017-03-30 11:48:35 +01:00
Erik Johnston
09f79aaad0 Use presence replication stream to invalidate cache 
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
 2017-03-24 13:21:08 +00:00
Erik Johnston
d58b1ffe94 Replace some calls to cursor_to_dict 
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
 2017-03-24 11:07:02 +00:00
Erik Johnston
aac6d1fc9b PEP8 2017-03-20 13:47:56 +00:00
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Richard van der Hoff
29ed09e80a Fix assertion to stop transaction queue getting wedged 
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
 2017-03-15 12:16:55 +00:00
Erik Johnston
45c7f12d2a Add new storage function to slave store 2017-03-13 16:26:44 +00:00
Erik Johnston
8f267fa8a8 Fix it for the workers 2017-03-10 11:22:25 +00:00
Erik Johnston
e933a2712d Don't log unknown cache warnings in workers 2017-02-28 16:22:41 +00:00
Erik Johnston
095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston
9e617cd4c2 Cache get_presence storage 2017-02-13 13:50:03 +00:00
Erik Johnston
6bba80241c Merge pull request #1912 from matrix-org/markjh/roominitialsync 
Add db functions needed for room initial sync to slave
 2017-02-13 12:20:21 +01:00
Mark Haines
3a46280ca3 Add db functions needed for room initial sync to slave 2017-02-13 11:16:53 +00:00
Erik Johnston
0f3e296cb7 Fix replication 2017-02-02 15:02:03 +00:00
Erik Johnston
458b6f4733 Only invalidate membership caches based on the cache stream 
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
 2017-01-31 16:09:03 +00:00
Erik Johnston
3670025e64 Rename func 2017-01-30 14:11:31 +00:00
Erik Johnston
252b503fc8 Hook device list updates to replication 2017-01-27 14:31:35 +00:00
Erik Johnston
a55fa2047f Insert delta of current_state_events to be more efficient 2017-01-20 17:10:18 +00:00
Erik Johnston
09cbcb78d3 Add cache to get_public_room_ids_at_stream_id 2016-12-12 14:41:51 +00:00
Erik Johnston
f32fb65552 Add new API appservice specific public room list 2016-12-06 16:12:27 +00:00
Erik Johnston
26072df6af Ensure only main or federation_sender process can send federation traffic 2016-11-23 14:09:47 +00:00
Erik Johnston
4c79a63fd7 Explicit federation ack 2016-11-23 10:40:44 +00:00
Erik Johnston
90565d015e Invalidate retry cache in both directions 2016-11-22 17:45:44 +00:00