Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
8,603 Commits 5 Branches 723 Tags 124 MiB
b9557064bf
Commit Graph

6145 Commits

Author SHA1 Message Date
Erik Johnston
4264ceb31c Fiddle tcp replication logging 2017-04-04 14:14:03 +01:00
Erik Johnston
023ee197be Advance replication streams even if nothing is listening 
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
 2017-04-04 13:19:26 +01:00
Erik Johnston
d1605794ad Remove unused worker config option 2017-04-04 11:17:00 +01:00
Erik Johnston
3376f16012 Shuffle and comment synchrotron presence 2017-04-04 11:14:16 +01:00
Erik Johnston
6ce6bbedcb Move where we ack federation 2017-04-04 11:02:44 +01:00
Erik Johnston
62b89daac6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server 2017-04-04 09:46:16 +01:00
Richard van der Hoff
773e64cc1a Merge pull request #2095 from matrix-org/rav/cull_log_preserves 
Cull spurious PreserveLoggingContexts
 2017-04-03 17:02:25 +01:00
Richard van der Hoff
2d05eb3cf5 Merge remote-tracking branch 'origin/release-v0.20.0' into develop 2017-04-03 16:23:23 +01:00
Richard van der Hoff
30bcbf775a Accept join events from all servers 
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.

(Fixes #1893).
 2017-04-03 15:58:07 +01:00
Richard van der Hoff
7eb9f34cc3 Remove spurious yield 
In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`:
it doesn't return anything anyway.
 2017-04-03 15:44:19 +01:00
Richard van der Hoff
0b08c48fc5 Remove more spurious PreserveLoggingContexts 
Remove `PreserveLoggingContext` around calls to `Notifier.on_new_room_event`;
there is no problem if the logcontext is set when calling it.
 2017-04-03 15:43:37 +01:00
Richard van der Hoff
65e1683680 Remove spurious PreserveLoggingContext 
In `on_new_room_event`, remove `PreserveLoggingContext` - we can call its
subroutines with the logcontext set.
 2017-04-03 15:42:38 +01:00
Richard van der Hoff
feb496056e preserve_fn some deferred-returning things 
In `Notifier._on_new_room_event`, `preserve_fn` around its subroutines which
return deferreds, so that it is safe to call it with an active logcontext.
 2017-04-03 15:41:17 +01:00
Richard van der Hoff
e2eebf1696 Fix fixme in preserve_fn 
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
 2017-04-03 15:38:02 +01:00
Erik Johnston
36c28bc467 Update all the workers and master to use TCP replication 2017-04-03 15:35:52 +01:00
Erik Johnston
3a1f3f8388 Change slave storage to use new replication interface 
As the TCP replication uses a slightly different API and streams than
the HTTP replication.

This breaks HTTP replication.
 2017-04-03 15:34:19 +01:00
Erik Johnston
52bfa604e1 Add basic replication client handler and factory 2017-04-03 15:34:13 +01:00
Erik Johnston
0a6a966e2b Always advance stream tokens 2017-04-03 15:22:56 +01:00
Richard van der Hoff
773e1c6d68 Remove spurious @preserve_fn decorators 
Remove `@preserve_fn` decorators on `on_new_room_event`,
`_notify_pending_new_room_events`, `_on_new_room_event`, `on_new_event`, and
`on_new_replication_data` - none of these functions return a deferred, and the
decorator does nothing unless the wrapped function returns a deferred, so the
decorator was a no-op.
 2017-04-03 15:14:11 +01:00
Erik Johnston
0d1c85e643 Merge branch 'release-v0.20.0' of github.com:matrix-org/synapse into develop 2017-04-03 14:58:14 +01:00
Erik Johnston
1df7c28661 Use callbacks to notify tcp replication rather than deferreds 2017-03-31 15:42:51 +01:00
Erik Johnston
36d2b66f90 Add a timestamp to USER_SYNC command 
This timestamp is used to indicate when the user last sync'd
 2017-03-31 15:42:22 +01:00
Erik Johnston
8a240e4f9c Merge pull request #2078 from APwhitehat/assertuserfriendly 
add user friendly report of assertion error in synctl.py
 2017-03-31 14:41:49 +01:00
Erik Johnston
142b6b4abf Merge pull request #2011 from matrix-org/matthew/turn_allow_guests 
add setting (on by default) to support TURN for guests
 2017-03-31 14:37:09 +01:00
Erik Johnston
2a06b44be2 Merge pull request #1986 from matrix-org/matthew/enable_guest_3p 
enable guest access for the 3pl/3pid APIs
 2017-03-31 14:36:03 +01:00
Matthew Hodgson
9ee397b440 switch to allow_guest=True for authing 3Ps as per PR feedback 2017-03-31 13:54:26 +01:00
Erik Johnston
9d0170ac6c Fix up presence 2017-03-31 11:36:32 +01:00
Erik Johnston
bfcf016714 Fix up docs 2017-03-31 11:19:24 +01:00
Erik Johnston
9cee0ce7db Merge pull request #2075 from matrix-org/erikj/cache_speed 
Speed up cached function access
 2017-03-31 10:10:56 +01:00
Erik Johnston
350333a09a Merge pull request #2076 from matrix-org/erikj/as_perf 
Make AS's faster
 2017-03-31 09:43:10 +01:00
Erik Johnston
4d17add8de Remove unused instance variable 2017-03-31 09:38:27 +01:00
Erik Johnston
27b1b4a2c9 Speed up copy_and_replace 2017-03-30 17:50:31 +01:00
Erik Johnston
5b5b171f3e Docs 2017-03-30 17:05:53 +01:00
Erik Johnston
b282fe7170 Revert log context change 2017-03-30 17:03:59 +01:00
Erik Johnston
9ff4e0e91b Bump version and changelog 2017-03-30 16:37:40 +01:00
Erik Johnston
0834d1a70c Merge pull request #2079 from matrix-org/erikj/push_regex_cache 
Cache glob to regex at a higher level for push
 2017-03-30 16:28:18 +01:00
Erik Johnston
63fcc42990 Remove user from process_presence when stops syncing 2017-03-30 14:26:08 +01:00
Erik Johnston
6194a64ae9 Doc new instance variables 2017-03-30 14:19:10 +01:00
Erik Johnston
014fee93b3 Manually calculate cache key as getcallargs is expensive 
This is because getcallargs recomputes the getargspec, amongst other
things, which we don't need to do as its already been done
 2017-03-30 14:14:46 +01:00
Erik Johnston
86780a8bc3 Don't convert to deferreds when not necessary 2017-03-30 14:14:36 +01:00
Erik Johnston
3ba2859e0c Add tcp replication listener type and hook it up 2017-03-30 13:31:10 +01:00
Erik Johnston
e9dd8370b0 Add functions to presence to support remote syncs 
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
 2017-03-30 13:25:14 +01:00
Erik Johnston
4d7fc7f977 Add server side resource for tcp replication 2017-03-30 13:24:45 +01:00
Richard van der Hoff
f9b4bb05e0 Fix the logcontext handling in the cache wrappers (#2077) 
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
 2017-03-30 13:22:24 +01:00
Erik Johnston
7450693435 Initial TCP protocol implementation 
This defines the low level TCP replication protocol
 2017-03-30 12:54:46 +01:00
Erik Johnston
8da6f0be48 Define the various streams we will replicate 2017-03-30 12:54:46 +01:00
Erik Johnston
11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston
7984708a55 Add a simple hook to wait for replication traffic 2017-03-30 11:57:52 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication 
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
 2017-03-30 11:48:35 +01:00
Erik Johnston
30348c924c Use txn.fetchall() so we can reuse txn 2017-03-30 10:30:05 +01:00
Anant Prakash
6cdca71079 synctl.py: wait for synapse to stop before restarting (#2020) 2017-03-29 19:20:13 +01:00
Anant Prakash
305d16d612 add user friendly report of assertion error in synctl.py 
Signed-off-by: Anant Prakash <anantprakashjsr@gmail.com>
 2017-03-29 20:41:39 +05:30
Erik Johnston
a3810136fe Cache glob to regex at a higher level for push 2017-03-29 15:53:14 +01:00
Erik Johnston
3ce8d59176 Increase cache size for _get_state_group_for_event 2017-03-29 14:31:46 +01:00
Erik Johnston
b9c2ae6788 Merge pull request #1849 from matrix-org/erikj/state_typo 
Fix bug where current_state_events renamed to current_state_ids
 2017-03-29 11:53:42 +01:00
Erik Johnston
85be3dde81 Bail early if remote wouldn't be retried (#2064) 
* Bail early if remote wouldn't be retried

* Don't always return true

* Just use get_retry_limiter

* Spelling
 2017-03-29 11:48:27 +01:00
Erik Johnston
2f8b580b64 Merge pull request #2053 from matrix-org/erikj/e2e_one_time_upsert 
Don't user upsert to persist new one time keys
 2017-03-29 11:44:23 +01:00
Erik Johnston
c5b0bdd542 Merge pull request #2067 from matrix-org/erikj/notify_on_fed 
Notify on new federation traffic
 2017-03-29 11:41:37 +01:00
Erik Johnston
e4df0e189d Decrank last commit 2017-03-29 11:02:35 +01:00
Erik Johnston
4ad613f6be Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert 2017-03-29 10:57:19 +01:00
Erik Johnston
ac6bc55512 Correctly look up key 2017-03-29 10:56:26 +01:00
Erik Johnston
69efd77749 Add comment 2017-03-29 09:50:05 +01:00
Erik Johnston
51b156d48a Cache whether an AS is interested based on members 2017-03-28 13:27:21 +01:00
Erik Johnston
30f5ffdca2 Remove param and cast at call site 2017-03-28 13:27:21 +01:00
Erik Johnston
650f0e69f2 Compile the regex's used in ASes 2017-03-28 13:27:21 +01:00
Erik Johnston
d28db583da Merge pull request #2063 from matrix-org/erikj/device_list_batch 
Batch sending of device list pokes
 2017-03-28 11:35:41 +01:00
Erik Johnston
58a35366be The algorithm is part of the key id 2017-03-28 11:34:37 +01:00
Erik Johnston
bac9bf1b12 Typo 2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f Short circuit if all new events have same state group 2017-03-27 18:00:47 +01:00
Erik Johnston
35b4aa04be Notify on new federation traffic 2017-03-27 14:07:47 +01:00
Erik Johnston
2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Erik Johnston
281553afe6 Merge pull request #2062 from matrix-org/erikj/presence_replication 
Use presence replication stream to invalidate cache
 2017-03-24 13:57:45 +00:00
Erik Johnston
987f4945b4 Actually call invalidate 2017-03-24 13:28:20 +00:00
Erik Johnston
31d56c3fb5 Merge pull request #2061 from matrix-org/erikj/add_transaction_store 
Add slave transaction store to workers who send federation requests
 2017-03-24 13:24:27 +00:00
Erik Johnston
09f79aaad0 Use presence replication stream to invalidate cache 
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
 2017-03-24 13:21:08 +00:00
Erik Johnston
23e0ff840a Merge pull request #2060 from matrix-org/erikj/cache_hosts_in_room 
Cache hosts in room
 2017-03-24 13:07:22 +00:00
Erik Johnston
48e7697911 Add slave transaction store 2017-03-24 13:05:30 +00:00
Richard van der Hoff
f136c89d5e Merge pull request #2058 from matrix-org/rav/logcontext_leaks_2 
try not to drop context after federation requests
 2017-03-24 12:47:26 +00:00
Richard van der Hoff
01fc847f7f Merge pull request #2057 from matrix-org/rav/missing_yield_2 
Add another missing yield on check_device_registered
 2017-03-24 12:46:43 +00:00
Erik Johnston
7fc1f1e2b6 Cache hosts in room 2017-03-24 11:46:24 +00:00
Erik Johnston
57cfa513f5 Merge pull request #2054 from matrix-org/erikj/user_iter_cursor 
Reduce some CPU work on DB threads
 2017-03-24 11:40:08 +00:00
Erik Johnston
d58b1ffe94 Replace some calls to cursor_to_dict 
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
 2017-03-24 11:07:02 +00:00
Erik Johnston
e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
David Baker
e36950dec5 Merge pull request #2055 from matrix-org/dbkr/fix_add_msisdn_requestToken 
Fix token request for addition of phone numbers
 2017-03-24 10:32:57 +00:00
David Baker
f902e89d4b Merge pull request #2056 from matrix-org/dbkr/fix_invite_reject 
Fix rejection of invites not reaching sync
 2017-03-24 10:32:51 +00:00
Richard van der Hoff
a380f041c2 try not to drop context after federation requests 
preserve_context_over_fn uses a ContextPreservingDeferred, which only restores
context for the duration of its callbacks, which isn't really correct, and
means that subsequent operations in the same request can end up without their
logcontexts.
 2017-03-23 22:36:21 +00:00
Richard van der Hoff
9397edb28b Merge pull request #2050 from matrix-org/rav/federation_backoff 
push federation retry limiter down to matrixfederationclient
 2017-03-23 22:27:01 +00:00
Richard van der Hoff
06ce7335e9 Merge pull request #2052 from matrix-org/rav/time_bound_deferred 
Fix time_bound_deferred to throw the right exception
 2017-03-23 22:22:54 +00:00
Richard van der Hoff
13c8749ac9 Add another missing yield on check_device_registered 2017-03-23 22:18:53 +00:00
David Baker
e1f1784f99 Fix rejection of invites not reaching sync 
Always allow the user to see their own leave events, otherwise
they won't see the event if they reject an invite for a room whose
history visibility is set such that they cannot see events before
joining.
 2017-03-23 18:50:31 +00:00
David Baker
86e865d7d2 Oops, remove unintentional change 2017-03-23 18:48:30 +00:00
David Baker
a2dfab12c5 Fix token request for addition of phone numbers 2017-03-23 18:46:17 +00:00
Erik Johnston
00957d1aa4 User Cursor.__iter__ instead of fetchall 
This prevents unnecessary construction of lists
 2017-03-23 17:53:49 +00:00
pik
250ce11ab9 Add jsonschema to python_dependencies.py 
Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:47 -03:00
pik
566641a0b5 use jsonschema.FormatChecker for RoomID and UserID strings 
* use a valid filter in rest/client/v2_alpha test

Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:41 -03:00
pik
acafcf1c5b Add valid filter tests, flake8, fix typo 
Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:10 -03:00
pik
e56c79c114 check_valid_filter using JSONSchema 
* add invalid filter tests

Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:07 -03:00
Erik Johnston
6ebe2d23b1 Raise a more helpful exception 2017-03-23 13:48:30 +00:00
Erik Johnston
e64655c25d Don't user upsert to persist new one time keys 
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
 2017-03-23 13:17:00 +00:00
Richard van der Hoff
5a16cb4bf0 Ignore backoff history for invites, aliases, and roomdirs 
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
 2017-03-23 12:23:22 +00:00