Richard van der Hoff
0ca459ea33
Basic support for room versioning
...
This is the first tranche of support for room versioning. It includes:
* setting the default room version in the config file
* new room_version param on the createRoom API
* storing the version of newly-created rooms in the m.room.create event
* fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
Erik Johnston
cb298ff623
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet
2018-08-03 09:25:15 +01:00
Richard van der Hoff
01e93f48ed
Kill off MatrixCodeMessageException
...
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).
Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
Erik Johnston
443da003bc
Use new helper base class for membership requests
2018-07-31 14:32:23 +01:00
Erik Johnston
729b672823
Use new helper base class for ReplicationSendEventRestServlet
2018-07-31 14:32:23 +01:00
Erik Johnston
d81602b75a
Add helper base class for generating new replication endpoints
...
This will hopefully reduce the boiler plate required to implement new
internal HTTP requests.
2018-07-31 14:32:20 +01:00
Richard van der Hoff
f59be4eb0e
Fix unit tests
...
on_notifier_poke no longer runs synchonously, so we have to do a different hack
to make sure that the replication data has been sent. Let's actually listen for
its arrival.
2018-07-25 10:30:36 +01:00
Richard van der Hoff
371da42ae4
Wrap a number of things that run in the background
...
This will reduce the number of "Starting db connection from sentinel context"
warnings, and will help with our metrics.
2018-07-25 09:41:12 +01:00
Erik Johnston
0faa3223cd
Fix missing attributes on workers.
...
This was missed during the transition from attribute to getter for
getting state from context.
2018-07-23 16:28:00 +01:00
Erik Johnston
05f5dabc10
Use stream cache in get_linearized_receipts_for_room
...
This avoids us from uncessarily hitting the database when there has been
no change for the room
2018-07-10 17:22:42 +01:00
Amber Brown
49af402019
run isort
2018-07-09 16:09:20 +10:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
Erik Johnston
33fdcfa957
Merge pull request #3441 from matrix-org/erikj/redo_erasure
...
Fix user erasure and re-enable
2018-06-25 14:37:01 +01:00
Erik Johnston
eb50c44eaf
Add UserErasureWorkerStore to workers
2018-06-25 14:22:24 +01:00
Amber Brown
07cad26d65
Remove all global reactor imports & pass it around explicitly ( #3424 )
2018-06-25 14:08:28 +01:00
Amber Brown
77ac14b960
Pass around the reactor explicitly ( #3385 )
2018-06-22 09:37:10 +01:00
Amber Brown
99b77aa829
Fix tcp protocol metrics naming ( #3410 )
2018-06-21 09:39:27 +01:00
Richard van der Hoff
b7e7fd2d0e
Fix replication metrics
...
fix bug introduced in #3256
2018-06-04 16:23:05 +01:00
Amber Brown
754826a830
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-28 18:57:23 +10:00
Amber Brown
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
...
replace some iteritems with six
2018-05-24 13:04:07 -05:00
Amber Brown
b6063631c3
more cleanup
2018-05-22 17:36:20 -05:00
Amber Brown
228f1f584e
fix the test failures
2018-05-22 15:02:38 -05:00
Amber Brown
8f5a688d42
cleanups, self-registration
2018-05-22 10:56:03 -05:00
Amber Brown
a8990fa2ec
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-22 10:50:26 -05:00
Richard van der Hoff
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Amber Brown
fcc525b0b7
rest of the changes
2018-05-21 19:48:57 -05:00
Amber Brown
df9f72d9e5
replacing portions
2018-05-21 19:47:37 -05:00
Adrian Tschira
933bf2dd35
replace some iteritems with six
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Adrian Tschira
57b58e2174
make imports local
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-28 13:41:41 +02:00
Richard van der Hoff
b78395b7fe
Refactor ResponseCache usage
...
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.
This will be largely non-functional, but does include the following functional
changes:
* federation_server.on_context_state_request: drops use of _server_linearizer
which looked redundant and could cause incorrect cache misses by yielding
between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
on production.
2018-04-12 13:02:15 +01:00
Richard van der Hoff
b3384232a0
Add metrics for ResponseCache
2018-04-10 23:14:47 +01:00
Richard van der Hoff
3ee4ad09eb
Fix json encoding bug in replication
...
json encoders have an encode method, not a dumps method.
2018-04-03 15:09:48 +01:00
Richard van der Hoff
05630758f2
Use static JSONEncoders
...
using json.dumps with custom options requires us to create a new JSONEncoder on
each call. It's more efficient to create one upfront and reuse it.
2018-03-29 23:13:33 +01:00
Erik Johnston
9aa5a0af51
Explicitly use simplejson
2018-03-20 09:58:13 +00:00
Erik Johnston
610accbb7f
Fix replication after switch to simplejson
...
Turns out that simplejson serialises namedtuple's as dictionaries rather
than tuples by default.
2018-03-19 16:12:48 +00:00
Erik Johnston
fa72803490
Merge branch 'master' of github.com:matrix-org/synapse into develop
2018-03-19 11:41:01 +00:00
Erik Johnston
926ba76e23
Replace ujson with simplejson
2018-03-15 23:43:31 +00:00
Erik Johnston
57db62e554
Merge pull request #2992 from matrix-org/erikj/implement_member_workre
...
Implement RoomMemberWorkerHandler
2018-03-14 14:29:33 +00:00
Erik Johnston
0011ede3b0
Fix imports
2018-03-14 14:19:23 +00:00
Erik Johnston
62ad701326
s/join/joined/ in notify_user_membership_change
2018-03-14 14:17:43 +00:00
Erik Johnston
b27320b550
Implement RoomMemberWorkerHandler
2018-03-13 18:26:00 +00:00
Erik Johnston
3518d0ea8f
Split up ProfileStore
2018-03-13 17:36:50 +00:00
Erik Johnston
d0fcc48f9d
extra_users is actually a list of UserIDs
2018-03-13 11:20:06 +00:00
Erik Johnston
2e223163ff
Split Directory store
2018-03-05 15:11:30 +00:00
Erik Johnston
fafa3e7114
Split registration store
2018-03-02 13:48:27 +00:00
Erik Johnston
1a6c7cdf54
Merge pull request #2928 from matrix-org/erikj/read_marker_caches
...
Fix typo in getting replication account data processing
2018-03-01 17:56:14 +00:00
Erik Johnston
89b7232ff8
Fix typo in getting replication account data processing
2018-03-01 17:50:30 +00:00
Erik Johnston
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
...
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
Erik Johnston
65cf454fd1
Remove unused DataStore
2018-03-01 17:27:53 +00:00
Erik Johnston
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
...
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
Erik Johnston
a83c514d1f
Improve caching for read_marker API
...
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
Erik Johnston
33bebb63f3
Add some caches to help read marker API
2018-03-01 17:08:17 +00:00
Erik Johnston
2ad4d5b5bb
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed
2018-03-01 16:59:39 +00:00
Erik Johnston
64346be26d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store
2018-03-01 16:26:42 +00:00
Erik Johnston
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
...
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
Erik Johnston
f793bc3877
Split out stream store
2018-03-01 15:13:08 +00:00
Erik Johnston
6411f725be
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 14:20:53 +00:00
Erik Johnston
a9a2d66cdd
Split out SignatureStore and EventFederationStore
2018-03-01 14:17:53 +00:00
Erik Johnston
0c8ba5dd1c
Split up RoomStore
2018-03-01 14:01:19 +00:00
Erik Johnston
126b9bf96f
Log in the correct places
2018-03-01 12:05:33 +00:00
Erik Johnston
157298f986
Don't do preserve_fn for every request
2018-03-01 11:59:45 +00:00
Erik Johnston
89f90d808a
Add some logging
2018-03-01 11:59:16 +00:00
Erik Johnston
8ded8ba2c7
Make repl send_event idempotent and retry on timeouts
...
If we treated timeouts as failures on the worker we would attempt to
clean up e.g. push actions while the master might still process the
event.
2018-03-01 11:20:34 +00:00
Erik Johnston
6b8604239f
Correctly send ratelimit and extra_users params
2018-03-01 10:08:39 +00:00
Erik Johnston
28e973ac11
Calculate push actions on worker
2018-02-28 18:02:30 +00:00
Erik Johnston
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
Erik Johnston
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
Erik Johnston
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
Erik Johnston
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
Erik Johnston
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
Erik Johnston
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
Erik Johnston
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
Erik Johnston
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
Erik Johnston
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
Erik Johnston
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
Erik Johnston
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
Erik Johnston
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
Erik Johnston
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
Erik Johnston
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
Erik Johnston
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
Erik Johnston
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
Erik Johnston
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
Erik Johnston
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
Erik Johnston
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
Erik Johnston
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
Erik Johnston
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
Erik Johnston
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
Erik Johnston
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
Erik Johnston
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
Erik Johnston
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
Erik Johnston
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
Erik Johnston
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
Erik Johnston
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
Erik Johnston
ef344b10e5
Don't log errors propogated from send_event
2018-02-15 11:03:49 +00:00
Erik Johnston
8ec2e638be
Add event_creator worker
2018-02-07 10:32:32 +00:00
Erik Johnston
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff
5c3c32f16f
Metrics for number of RDATA commands received
...
I found myself wishing we had this.
2018-01-15 17:45:55 +00:00
Richard van der Hoff
0edf085b68
Fix some logcontext leaks in replication resource
...
The @measure_func annotations rely on the wrapped function respecting the
logcontext rules. Add the necessary yields to make this work.
2017-11-23 23:19:43 +00:00
Richard van der Hoff
35a4b63240
Pull out bits of StateStore to a mixin
...
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.
fixes https://github.com/matrix-org/synapse/issues/2655 .
2017-11-14 11:43:58 +00:00
Richard van der Hoff
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Richard van der Hoff
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
hera
f807f7f804
log when we get an exception handling replication updates
2017-10-12 11:51:24 +01:00
Erik Johnston
2cc998fed8
Fix replication. And notify
2017-07-20 17:13:18 +01:00
Erik Johnston
925b3638ff
Reduce log levels in tcp replication
2017-07-11 10:04:21 +01:00
Erik Johnston
27f26e48b7
Serialize user ip command as json
2017-06-27 16:25:38 +01:00
Erik Johnston
8c23221666
Fix up
2017-06-27 15:53:45 +01:00
Erik Johnston
78cefd78d6
Make workers report to master for user ip updates
2017-06-27 14:58:10 +01:00
Erik Johnston
dae9a00a28
Initialise exclusive_user_regex
2017-06-21 14:19:33 +01:00
Erik Johnston
8177563ebe
Fix for workers
2017-06-21 13:57:49 +01:00
Erik Johnston
6aa5bc8635
Initial worker impl
2017-06-16 11:47:11 +01:00
Erik Johnston
d53fe399eb
Add cache for is_host_joined
2017-06-13 09:56:18 +01:00
Erik Johnston
a837765e8c
Merge pull request #2266 from matrix-org/erikj/host_in_room
...
Change is_host_joined to use current_state table
2017-06-12 09:49:51 +01:00
Erik Johnston
8060974344
Fix replication
2017-06-09 16:40:52 +01:00
Erik Johnston
2cac7623a5
Add missing notifier
2017-06-09 11:24:41 +01:00
Erik Johnston
298d83b340
Fix replication
2017-06-09 11:01:28 +01:00
Erik Johnston
dfbda5e025
Faster cache for get_joined_hosts
2017-05-25 17:24:44 +01:00
Erik Johnston
f85a415279
Add missing storage function to slave store
2017-05-22 16:31:24 +01:00
Erik Johnston
9ac263ed1b
Add new storage functions to slave store
2017-05-04 14:29:03 +01:00
Erik Johnston
e4f3431116
Remove unused cache
2017-04-24 13:27:38 +01:00
Erik Johnston
247c736b9b
Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
...
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
Erik Johnston
9c712a366f
Move get_presence_list_* to SlaveStore
2017-04-11 16:07:33 +01:00
Erik Johnston
28a4649785
Remove HTTP replication APIs
2017-04-11 09:52:11 +01:00
Erik Johnston
29574fd5b3
Reduce federation presence replication traffic
...
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
Erik Johnston
2e6f5a4910
Typo
2017-04-10 16:17:40 +01:00
Erik Johnston
efcb6db688
Merge pull request #2109 from matrix-org/erikj/send_queue_fix
...
Fix up federation SendQueue and document types
2017-04-10 13:09:25 +01:00
Erik Johnston
0364d23210
Up replication ping timeout
2017-04-10 11:32:05 +01:00
Erik Johnston
ab904caf33
Comments
2017-04-10 10:02:17 +01:00
Erik Johnston
98ce212093
Merge pull request #2103 from matrix-org/erikj/no-double-encode
...
Don't double encode replication data
2017-04-07 09:39:52 +01:00
Erik Johnston
ad544c803a
Document types of the replication streams
2017-04-06 13:28:52 +01:00
Erik Johnston
69b3fd485d
Fix incorrect type when using InvalidateCacheCommand
2017-04-06 09:36:38 +01:00
Erik Johnston
fcc803b2bf
Add log lines
2017-04-05 17:13:44 +01:00
Erik Johnston
3f213d908d
Rearrange metrics
2017-04-05 14:15:09 +01:00
Erik Johnston
1ca0e78ca1
Fix typo
2017-04-05 13:43:39 +01:00
Erik Johnston
b43d3267e2
Fixup some metrics for tcp repl
2017-04-05 13:34:54 +01:00
Erik Johnston
a5c401bd12
Merge pull request #2097 from matrix-org/erikj/repl_tcp_client
...
Move to using TCP replication
2017-04-05 09:36:21 +01:00
Erik Johnston
a76886726b
Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix
...
Advance replication streams even if nothing is listening
2017-04-04 15:40:51 +01:00
Erik Johnston
4264ceb31c
Fiddle tcp replication logging
2017-04-04 14:14:03 +01:00
Erik Johnston
023ee197be
Advance replication streams even if nothing is listening
...
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
2017-04-04 13:19:26 +01:00
Erik Johnston
3a1f3f8388
Change slave storage to use new replication interface
...
As the TCP replication uses a slightly different API and streams than
the HTTP replication.
This breaks HTTP replication.
2017-04-03 15:34:19 +01:00
Erik Johnston
52bfa604e1
Add basic replication client handler and factory
2017-04-03 15:34:13 +01:00
Erik Johnston
0a6a966e2b
Always advance stream tokens
2017-04-03 15:22:56 +01:00
Erik Johnston
1df7c28661
Use callbacks to notify tcp replication rather than deferreds
2017-03-31 15:42:51 +01:00
Erik Johnston
36d2b66f90
Add a timestamp to USER_SYNC command
...
This timestamp is used to indicate when the user last sync'd
2017-03-31 15:42:22 +01:00
Erik Johnston
bfcf016714
Fix up docs
2017-03-31 11:19:24 +01:00
Erik Johnston
4d7fc7f977
Add server side resource for tcp replication
2017-03-30 13:24:45 +01:00
Erik Johnston
7450693435
Initial TCP protocol implementation
...
This defines the low level TCP replication protocol
2017-03-30 12:54:46 +01:00
Erik Johnston
8da6f0be48
Define the various streams we will replicate
2017-03-30 12:54:46 +01:00
Erik Johnston
11880103b1
Make federation send queue take the current position
2017-03-30 12:54:36 +01:00
Erik Johnston
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
