Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
8,909 Commits 5 Branches 723 Tags 124 MiB
1a81a1898e
Commit Graph

6407 Commits

Author SHA1 Message Date
David Baker
5fd12dce01 Remove debugging 2017-04-26 12:36:26 +01:00
David Baker
82ae0238f9 Revert accidental commit 2017-04-26 11:43:16 +01:00
David Baker
81804909d3 Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error 2017-04-26 11:31:55 +01:00
David Baker
c366276056 Fix get_json 2017-04-26 10:07:01 +01:00
David Baker
1a9255c12e Use CodeMessageException subclass instead 
Parse json errors from get_json client methods and throw special
errors.
 2017-04-25 19:30:55 +01:00
Erik Johnston
f053a1409e Make state caches cache in ascii 2017-04-25 17:22:55 +01:00
Erik Johnston
acb58bfb6a fix up 2017-04-25 15:39:19 +01:00
Erik Johnston
f7181615f2 Don't specify default as dict 2017-04-25 15:22:59 +01:00
Erik Johnston
f144365281 Comment 2017-04-25 15:18:26 +01:00
Erik Johnston
d9aa645f86 Reduce size of joined_user cache 
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.

We also try converting the string to ascii to further reduce the size.
 2017-04-25 14:38:51 +01:00
Erik Johnston
22f3d3ae76 Reduce _get_state_group_for_event cache size 2017-04-25 11:43:03 +01:00
Erik Johnston
b4da08cad8 Merge pull request #2158 from matrix-org/erikj/reduce_cache_size 
Reduce cache size by not storing deferreds
 2017-04-25 11:18:07 +01:00
Erik Johnston
efab1dadde Remove DEBUG_CACHES 2017-04-25 10:54:09 +01:00
Erik Johnston
119cb9bbcf Reduce cache size by not storing deferreds 
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.

However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.

This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.

As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
 2017-04-25 10:23:11 +01:00
Mark Haines
e6e2627636 Fix code for reporting old verify keys in synapse 2017-04-24 18:51:25 +01:00
Richard van der Hoff
30f7bfa121 Merge pull request #2145 from matrix-org/rav/reject_invite_to_unreachable_server 
Fix rejection of invites to unreachable servers
 2017-04-24 15:20:52 +01:00
Erik Johnston
7af825bae4 Merge pull request #2155 from matrix-org/erikj/string_intern 
Only intern ascii strings
 2017-04-24 14:28:41 +01:00
Erik Johnston
d134d0935e Only intern ascii strings 2017-04-24 14:07:48 +01:00
Erik Johnston
e4f3431116 Remove unused cache 2017-04-24 13:27:38 +01:00
David Baker
a46982cee9 Need the HTTP status code 2017-04-21 16:20:12 +01:00
David Baker
70caf49914 Do the same for get_json 2017-04-21 16:09:03 +01:00
David Baker
a1595cec78 Don't error for 3xx responses 2017-04-21 11:51:17 +01:00
David Baker
2e165295b7 Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error 2017-04-21 11:35:52 +01:00
David Baker
a90a0f5c8a Propagate errors sensibly from proxied IS requests 
When we're proxying Matrix endpoints, parse out Matrix error
responses and turn them into SynapseErrors so they can be
propagated sensibly upstream.
 2017-04-21 11:32:48 +01:00
Richard van der Hoff
91b3981800 Try harder when sending leave events 
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
 2017-04-21 01:50:36 +01:00
Richard van der Hoff
0cdb32fc43 Remove redundant try/except clauses 
The `except SynapseError` clauses were pointless because the wrapped functions
would never throw a `SynapseError` (they either throw a `CodeMessageException`
or a `RuntimeError`).

The `except CodeMessageException` is now also pointless because the caller
treats all exceptions equally, so we may as well just throw the
`CodeMessageException`.
 2017-04-21 01:32:01 +01:00
Richard van der Hoff
838810b76a Broaden the conditions for locally_rejecting invites 
The logic for marking invites as locally rejected was all well and good, but
didn't happen when the remote server returned a 500, or wasn't reachable, or
had no DNS, or whatever.

Just expand the except clause to catch everything.

Fixes https://github.com/matrix-org/synapse/issues/761.
 2017-04-21 01:31:37 +01:00
Richard van der Hoff
736b9a4784 Remove redundant function 
inline `reject_remote_invite`, which only existed to make tracing the callflow
more difficult.
 2017-04-21 01:31:09 +01:00
Richard van der Hoff
4903ccf159 Fix some lies, and other clarifications, in docstrings 
The documentation on get_json has been wrong ever since the very first commit
to synapse...
 2017-04-21 01:31:09 +01:00
Matthew Wolff
d4040e9e28
Queried CONDITIONAL_REQUIREMENTS 2017-04-18 16:19:48 -05:00
Luke Barnard
3fb8784c92 m.read_marker -> m.fully_read (#2128) 
Also:
 - change the REST endpoint to have a "S" on the end (so it's now /read_markers)
 - change the content of the m.read_up_to event to have the key "event_id" instead of "marker".
 2017-04-18 17:46:15 +01:00
Matthew Wolff
54f9a4cb59
Fixed travis build failure 
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
 2017-04-17 01:38:27 -05:00
Matthew Wolff
8e780b113d
web_server_root documentation fix 
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
 2017-04-17 00:49:11 -05:00
Anant Prakash
574d573ac2 Check that requested room_id exists 2017-04-14 23:50:59 +05:30
Luke Barnard
78f0ddbfad Merge pull request #2120 from matrix-org/luke/read-markers 
Implement Read Marker API
 2017-04-13 14:21:31 +01:00
Luke Barnard
6a70647d45 Correct logic in is_event_after 2017-04-13 13:46:17 +01:00
Anant Prakash
c1f52a321d synctl.py: Check if synapse is already running 2017-04-13 18:00:02 +05:30
Luke Barnard
b9557064bf Simplify is_event_after logic 2017-04-12 14:36:20 +01:00
Luke Barnard
cf6121e3da More null-guard changes 2017-04-12 14:02:03 +01:00
Erik Johnston
247c736b9b Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl 
Reduce federation replication traffic
 2017-04-12 11:07:13 +01:00
Paul Evans
8fbc0d29ee Merge pull request #2121 from matrix-org/paul/sent-transactions-metric 
Add a counter metric for successfully-sent transactions
 2017-04-12 11:04:31 +01:00
Erik Johnston
c06c00190f Merge pull request #2116 from matrix-org/erikj/dedupe_federation_repl2 
Dedupe KeyedEdu and Devices federation repl traffic
 2017-04-12 10:57:24 +01:00
Luke Barnard
c0aba0a23e Remove Unused ref to hs 2017-04-12 10:52:11 +01:00
Luke Barnard
b9676a75f6 Move a space 2017-04-12 10:51:17 +01:00
Luke Barnard
69a18514e9 Only notify user, not entire room 2017-04-12 10:50:37 +01:00
Luke Barnard
122cd52ce4 Remove comment, simplify null-guard 2017-04-12 10:48:32 +01:00
Erik Johnston
26ae5178a4 Add some comments 2017-04-12 10:36:29 +01:00
Erik Johnston
bf9060156a Merge pull request #2117 from matrix-org/erikj/remove_http_replication 
Remove HTTP replication APIs
 2017-04-12 10:21:42 +01:00
Erik Johnston
1745069543 Comment 2017-04-12 10:17:10 +01:00
Erik Johnston
c7ddb5ef7a Reuse get_interested_parties 2017-04-12 10:16:26 +01:00
Erik Johnston
7b41013102 Merge pull request #2118 from matrix-org/erikj/no_devices 
Fix getting latest device IP for user with no devices
 2017-04-12 10:14:32 +01:00
Luke Barnard
77fb2b72ae Handle no previous RM 2017-04-12 09:47:29 +01:00
Luke Barnard
7f94709066 travis flake8.. 2017-04-11 18:35:45 +01:00
Luke Barnard
867822fa1e flake8 2017-04-11 17:36:04 +01:00
Luke Barnard
73880268ef Refactor event ordering check to events store 2017-04-11 17:34:09 +01:00
Luke Barnard
131485ef66 Copyright 2017-04-11 17:33:51 +01:00
Paul "LeoNerd" Evans
11dbceb761 Add a counter metric for successfully-sent transactions 2017-04-11 17:16:12 +01:00
Luke Barnard
0127423027 flake8 2017-04-11 17:07:07 +01:00
Erik Johnston
85657eedf8 Bail on where clause instead 2017-04-11 16:24:31 +01:00
Erik Johnston
b48045a8f5 Don't bother with outer check for now 2017-04-11 16:23:24 +01:00
Erik Johnston
9c712a366f Move get_presence_list_* to SlaveStore 2017-04-11 16:07:33 +01:00
Erik Johnston
a8c8e4efd4 Comment 2017-04-11 15:35:49 +01:00
Erik Johnston
414522aed5 Move get_interested_parties 2017-04-11 15:33:26 +01:00
Erik Johnston
2be8a281d2 Comments 2017-04-11 15:28:24 +01:00
Erik Johnston
6308ac45b0 Move get_interested_remotes back to presence handler 2017-04-11 15:19:26 +01:00
Erik Johnston
b9b72bc6e2 Comments 2017-04-11 15:15:34 +01:00
Luke Barnard
d892079844 Finish implementing RM endpoint 
- This change causes a 405 to be sent if "m.read_marker" is set via /account_data
 - This also fixes-up the RM endpoint so that it actually Works.
 2017-04-11 15:01:39 +01:00
lukebarnard
e263c26690 Initial commit of RM server-side impl 
(See https://docs.google.com/document/d/1UWqdS-e1sdwkLDUY0wA4gZyIkRp-ekjsLZ8k6g_Zvso/edit#heading=h.lndohpg8at5u)
 2017-04-11 11:55:30 +01:00
Erik Johnston
f3cf3ff8b6 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-04-11 11:13:32 +01:00
Erik Johnston
34840cdcef Fix getting latest device IP for user with no devices 2017-04-11 09:56:54 +01:00
Erik Johnston
28a4649785 Remove HTTP replication APIs 2017-04-11 09:52:11 +01:00
Matthew Hodgson
7c551ec445 trust a hypothetical future riot.im IS 2017-04-10 17:58:36 +01:00
Erik Johnston
84fbb80c8f Use generators 2017-04-10 16:55:56 +01:00
Erik Johnston
40453b3f84 Dedupe KeyedEdu and Devices federation repl traffic 2017-04-10 16:49:51 +01:00
Erik Johnston
29574fd5b3 Reduce federation presence replication traffic 
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
 2017-04-10 16:48:30 +01:00
Erik Johnston
2e6f5a4910 Typo 2017-04-10 16:17:40 +01:00
David Baker
405ba4178a Merge pull request #2102 from DanielDent/add-auth-email 
Support authenticated SMTP
 2017-04-10 15:42:16 +01:00
Erik Johnston
efcb6db688 Merge pull request #2109 from matrix-org/erikj/send_queue_fix 
Fix up federation SendQueue and document types
 2017-04-10 13:09:25 +01:00
Erik Johnston
0018491af2 Rename variable 2017-04-10 12:44:43 +01:00
Erik Johnston
0364d23210 Up replication ping timeout 2017-04-10 11:32:05 +01:00
Erik Johnston
8c5f03cec7 Revert to sending the same data type as before 2017-04-10 10:07:18 +01:00
Erik Johnston
f8434db549 Change name 2017-04-10 10:03:07 +01:00
Erik Johnston
ab904caf33 Comments 2017-04-10 10:02:17 +01:00
Richard van der Hoff
64765e5199 When we do an invite rejection, save the signed leave event to the db 
During a rejection of an invite received over federation, we ask a remote
server to make us a `leave` event, then sign it, then send that with
`send_leave`.

We were saving the *unsigned* version of the event (which has a different event
id to the signed version) to our db (and sending it to the clients), whereas
other servers in the room will have seen the *signed* version. We're not aware
of any actual problems that caused, except that it makes the database confusing
to look at and generally leaves the room in a weird state.
 2017-04-07 14:39:32 +01:00
Erik Johnston
2a3e822f44 Comment 2017-04-07 13:47:04 +01:00
Erik Johnston
a828a64b75 Comment 2017-04-07 11:54:03 +01:00
Erik Johnston
d4d176e5d0 Add logging 2017-04-07 11:51:28 +01:00
Erik Johnston
449d1297ca Fix up federation SendQueue and document types 2017-04-07 11:48:33 +01:00
Erik Johnston
d72667fcce Speed up get_current_state_ids 
Using _simple_select_list is fairly expensive for functions that return
a lot of rows and/or get called a lot. (This is because it carefully
constructs a list of dicts).

get_current_state_ids gets called a lot on startup and e.g. when the IRC
bridge decided to send tonnes of joins/leaves (as it invalidates the
cache). We therefore replace it with a custon txn function that builds
up the final result dict without building up and intermediate
representation.
 2017-04-07 10:10:49 +01:00
Erik Johnston
a41fe500d6 Bump version and changelog 2017-04-07 10:03:48 +01:00
Erik Johnston
54f59bd7d4 Merge pull request #2107 from HarHarLinks/patch-1 
fix typo in synctl help
 2017-04-07 09:54:37 +01:00
Erik Johnston
98ce212093 Merge pull request #2103 from matrix-org/erikj/no-double-encode 
Don't double encode replication data
 2017-04-07 09:39:52 +01:00
Kim Brose
8a1137ceab fix typo in synctl help 2017-04-06 17:10:20 +02:00
Erik Johnston
877c029c16 Use iteritems 2017-04-06 15:51:22 +01:00
Erik Johnston
944692ef69 Merge pull request #2106 from matrix-org/erikj/reduce_user_sync 
Reduce rate of USER_SYNC repl commands
 2017-04-06 13:35:31 +01:00
Erik Johnston
391712a4f9 Comment 2017-04-06 13:35:00 +01:00
Erik Johnston
ad544c803a Document types of the replication streams 2017-04-06 13:28:52 +01:00
Erik Johnston
dbf87282d3 Docs 2017-04-06 13:11:21 +01:00
Erik Johnston
69b3fd485d Fix incorrect type when using InvalidateCacheCommand 2017-04-06 09:36:38 +01:00
Daniel Dent
5058292537 Support authenticated SMTP 
Closes (SYN-714) #1385

Signed-off-by: Daniel Dent <matrixcontrib@contactdaniel.net>
 2017-04-05 21:01:08 -07:00
Erik Johnston
fcc803b2bf Add log lines 2017-04-05 17:13:44 +01:00
Erik Johnston
3f213d908d Rearrange metrics 2017-04-05 14:15:09 +01:00
Erik Johnston
1ca0e78ca1 Fix typo 2017-04-05 13:43:39 +01:00
Erik Johnston
b43d3267e2 Fixup some metrics for tcp repl 2017-04-05 13:34:54 +01:00
Erik Johnston
b5cb6347a4 Don't immediately notify the master about users whose syncs have gone away 2017-04-05 13:25:40 +01:00
Erik Johnston
96b9b6c127 Don't double json encode typing replication data 2017-04-05 11:34:20 +01:00
Erik Johnston
f10ce8944b Don't double json encode federation replication data 2017-04-05 11:10:28 +01:00
Erik Johnston
a5c401bd12 Merge pull request #2097 from matrix-org/erikj/repl_tcp_client 
Move to using TCP replication
 2017-04-05 09:36:21 +01:00
Erik Johnston
d1d5362267 Add comment 2017-04-04 16:41:03 +01:00
Erik Johnston
9f26d3b75b Deduplicate new deviceinbox rows for replication 2017-04-04 16:21:21 +01:00
Erik Johnston
a76886726b Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix 
Advance replication streams even if nothing is listening
 2017-04-04 15:40:51 +01:00
Erik Johnston
ac66e11f2b Add the appropriate amount of preserve_fn 2017-04-04 15:22:54 +01:00
Erik Johnston
4264ceb31c Fiddle tcp replication logging 2017-04-04 14:14:03 +01:00
Erik Johnston
023ee197be Advance replication streams even if nothing is listening 
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
 2017-04-04 13:19:26 +01:00
Erik Johnston
d1605794ad Remove unused worker config option 2017-04-04 11:17:00 +01:00
Erik Johnston
3376f16012 Shuffle and comment synchrotron presence 2017-04-04 11:14:16 +01:00
Erik Johnston
6ce6bbedcb Move where we ack federation 2017-04-04 11:02:44 +01:00
Erik Johnston
62b89daac6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server 2017-04-04 09:46:16 +01:00
Richard van der Hoff
773e64cc1a Merge pull request #2095 from matrix-org/rav/cull_log_preserves 
Cull spurious PreserveLoggingContexts
 2017-04-03 17:02:25 +01:00
Richard van der Hoff
2d05eb3cf5 Merge remote-tracking branch 'origin/release-v0.20.0' into develop 2017-04-03 16:23:23 +01:00
Richard van der Hoff
30bcbf775a Accept join events from all servers 
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.

(Fixes #1893).
 2017-04-03 15:58:07 +01:00
Richard van der Hoff
7eb9f34cc3 Remove spurious yield 
In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`:
it doesn't return anything anyway.
 2017-04-03 15:44:19 +01:00
Richard van der Hoff
0b08c48fc5 Remove more spurious PreserveLoggingContexts 
Remove `PreserveLoggingContext` around calls to `Notifier.on_new_room_event`;
there is no problem if the logcontext is set when calling it.
 2017-04-03 15:43:37 +01:00
Richard van der Hoff
65e1683680 Remove spurious PreserveLoggingContext 
In `on_new_room_event`, remove `PreserveLoggingContext` - we can call its
subroutines with the logcontext set.
 2017-04-03 15:42:38 +01:00
Richard van der Hoff
feb496056e preserve_fn some deferred-returning things 
In `Notifier._on_new_room_event`, `preserve_fn` around its subroutines which
return deferreds, so that it is safe to call it with an active logcontext.
 2017-04-03 15:41:17 +01:00
Richard van der Hoff
e2eebf1696 Fix fixme in preserve_fn 
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
 2017-04-03 15:38:02 +01:00
Erik Johnston
36c28bc467 Update all the workers and master to use TCP replication 2017-04-03 15:35:52 +01:00
Erik Johnston
3a1f3f8388 Change slave storage to use new replication interface 
As the TCP replication uses a slightly different API and streams than
the HTTP replication.

This breaks HTTP replication.
 2017-04-03 15:34:19 +01:00
Erik Johnston
52bfa604e1 Add basic replication client handler and factory 2017-04-03 15:34:13 +01:00
Erik Johnston
0a6a966e2b Always advance stream tokens 2017-04-03 15:22:56 +01:00
Richard van der Hoff
773e1c6d68 Remove spurious @preserve_fn decorators 
Remove `@preserve_fn` decorators on `on_new_room_event`,
`_notify_pending_new_room_events`, `_on_new_room_event`, `on_new_event`, and
`on_new_replication_data` - none of these functions return a deferred, and the
decorator does nothing unless the wrapped function returns a deferred, so the
decorator was a no-op.
 2017-04-03 15:14:11 +01:00
Erik Johnston
0d1c85e643 Merge branch 'release-v0.20.0' of github.com:matrix-org/synapse into develop 2017-04-03 14:58:14 +01:00
Erik Johnston
1df7c28661 Use callbacks to notify tcp replication rather than deferreds 2017-03-31 15:42:51 +01:00
Erik Johnston
36d2b66f90 Add a timestamp to USER_SYNC command 
This timestamp is used to indicate when the user last sync'd
 2017-03-31 15:42:22 +01:00
Erik Johnston
8a240e4f9c Merge pull request #2078 from APwhitehat/assertuserfriendly 
add user friendly report of assertion error in synctl.py
 2017-03-31 14:41:49 +01:00
Erik Johnston
142b6b4abf Merge pull request #2011 from matrix-org/matthew/turn_allow_guests 
add setting (on by default) to support TURN for guests
 2017-03-31 14:37:09 +01:00
Erik Johnston
2a06b44be2 Merge pull request #1986 from matrix-org/matthew/enable_guest_3p 
enable guest access for the 3pl/3pid APIs
 2017-03-31 14:36:03 +01:00
Matthew Hodgson
9ee397b440 switch to allow_guest=True for authing 3Ps as per PR feedback 2017-03-31 13:54:26 +01:00
Erik Johnston
9d0170ac6c Fix up presence 2017-03-31 11:36:32 +01:00
Erik Johnston
bfcf016714 Fix up docs 2017-03-31 11:19:24 +01:00
Erik Johnston
9cee0ce7db Merge pull request #2075 from matrix-org/erikj/cache_speed 
Speed up cached function access
 2017-03-31 10:10:56 +01:00
Erik Johnston
350333a09a Merge pull request #2076 from matrix-org/erikj/as_perf 
Make AS's faster
 2017-03-31 09:43:10 +01:00
Erik Johnston
4d17add8de Remove unused instance variable 2017-03-31 09:38:27 +01:00
Erik Johnston
27b1b4a2c9 Speed up copy_and_replace 2017-03-30 17:50:31 +01:00
Erik Johnston
5b5b171f3e Docs 2017-03-30 17:05:53 +01:00
Erik Johnston
b282fe7170 Revert log context change 2017-03-30 17:03:59 +01:00
Erik Johnston
9ff4e0e91b Bump version and changelog 2017-03-30 16:37:40 +01:00
Erik Johnston
0834d1a70c Merge pull request #2079 from matrix-org/erikj/push_regex_cache 
Cache glob to regex at a higher level for push
 2017-03-30 16:28:18 +01:00
Erik Johnston
63fcc42990 Remove user from process_presence when stops syncing 2017-03-30 14:26:08 +01:00
Erik Johnston
6194a64ae9 Doc new instance variables 2017-03-30 14:19:10 +01:00
Erik Johnston
014fee93b3 Manually calculate cache key as getcallargs is expensive 
This is because getcallargs recomputes the getargspec, amongst other
things, which we don't need to do as its already been done
 2017-03-30 14:14:46 +01:00
Erik Johnston
86780a8bc3 Don't convert to deferreds when not necessary 2017-03-30 14:14:36 +01:00
Erik Johnston
3ba2859e0c Add tcp replication listener type and hook it up 2017-03-30 13:31:10 +01:00
Erik Johnston
e9dd8370b0 Add functions to presence to support remote syncs 
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
 2017-03-30 13:25:14 +01:00
Erik Johnston
4d7fc7f977 Add server side resource for tcp replication 2017-03-30 13:24:45 +01:00
Richard van der Hoff
f9b4bb05e0 Fix the logcontext handling in the cache wrappers (#2077) 
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
 2017-03-30 13:22:24 +01:00
Erik Johnston
7450693435 Initial TCP protocol implementation 
This defines the low level TCP replication protocol
 2017-03-30 12:54:46 +01:00
Erik Johnston
8da6f0be48 Define the various streams we will replicate 2017-03-30 12:54:46 +01:00
Erik Johnston
11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston
7984708a55 Add a simple hook to wait for replication traffic 2017-03-30 11:57:52 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication 
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
 2017-03-30 11:48:35 +01:00
Erik Johnston
30348c924c Use txn.fetchall() so we can reuse txn 2017-03-30 10:30:05 +01:00
Anant Prakash
6cdca71079 synctl.py: wait for synapse to stop before restarting (#2020) 2017-03-29 19:20:13 +01:00
Anant Prakash
305d16d612 add user friendly report of assertion error in synctl.py 
Signed-off-by: Anant Prakash <anantprakashjsr@gmail.com>
 2017-03-29 20:41:39 +05:30
Erik Johnston
a3810136fe Cache glob to regex at a higher level for push 2017-03-29 15:53:14 +01:00
Erik Johnston
3ce8d59176 Increase cache size for _get_state_group_for_event 2017-03-29 14:31:46 +01:00
Erik Johnston
b9c2ae6788 Merge pull request #1849 from matrix-org/erikj/state_typo 
Fix bug where current_state_events renamed to current_state_ids
 2017-03-29 11:53:42 +01:00
Erik Johnston
85be3dde81 Bail early if remote wouldn't be retried (#2064) 
* Bail early if remote wouldn't be retried

* Don't always return true

* Just use get_retry_limiter

* Spelling
 2017-03-29 11:48:27 +01:00
Erik Johnston
2f8b580b64 Merge pull request #2053 from matrix-org/erikj/e2e_one_time_upsert 
Don't user upsert to persist new one time keys
 2017-03-29 11:44:23 +01:00
Erik Johnston
c5b0bdd542 Merge pull request #2067 from matrix-org/erikj/notify_on_fed 
Notify on new federation traffic
 2017-03-29 11:41:37 +01:00
Erik Johnston
e4df0e189d Decrank last commit 2017-03-29 11:02:35 +01:00
Erik Johnston
4ad613f6be Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert 2017-03-29 10:57:19 +01:00
Erik Johnston
ac6bc55512 Correctly look up key 2017-03-29 10:56:26 +01:00
Erik Johnston
69efd77749 Add comment 2017-03-29 09:50:05 +01:00
Erik Johnston
51b156d48a Cache whether an AS is interested based on members 2017-03-28 13:27:21 +01:00
Erik Johnston
30f5ffdca2 Remove param and cast at call site 2017-03-28 13:27:21 +01:00
Erik Johnston
650f0e69f2 Compile the regex's used in ASes 2017-03-28 13:27:21 +01:00
Erik Johnston
d28db583da Merge pull request #2063 from matrix-org/erikj/device_list_batch 
Batch sending of device list pokes
 2017-03-28 11:35:41 +01:00
Erik Johnston
58a35366be The algorithm is part of the key id 2017-03-28 11:34:37 +01:00
Erik Johnston
bac9bf1b12 Typo 2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f Short circuit if all new events have same state group 2017-03-27 18:00:47 +01:00
Erik Johnston
35b4aa04be Notify on new federation traffic 2017-03-27 14:07:47 +01:00
Erik Johnston
2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Erik Johnston
281553afe6 Merge pull request #2062 from matrix-org/erikj/presence_replication 
Use presence replication stream to invalidate cache
 2017-03-24 13:57:45 +00:00
Erik Johnston
987f4945b4 Actually call invalidate 2017-03-24 13:28:20 +00:00
Erik Johnston
31d56c3fb5 Merge pull request #2061 from matrix-org/erikj/add_transaction_store 
Add slave transaction store to workers who send federation requests
 2017-03-24 13:24:27 +00:00
Erik Johnston
09f79aaad0 Use presence replication stream to invalidate cache 
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
 2017-03-24 13:21:08 +00:00
Erik Johnston
23e0ff840a Merge pull request #2060 from matrix-org/erikj/cache_hosts_in_room 
Cache hosts in room
 2017-03-24 13:07:22 +00:00
Erik Johnston
48e7697911 Add slave transaction store 2017-03-24 13:05:30 +00:00
Richard van der Hoff
f136c89d5e Merge pull request #2058 from matrix-org/rav/logcontext_leaks_2 
try not to drop context after federation requests
 2017-03-24 12:47:26 +00:00
Richard van der Hoff
01fc847f7f Merge pull request #2057 from matrix-org/rav/missing_yield_2 
Add another missing yield on check_device_registered
 2017-03-24 12:46:43 +00:00
Erik Johnston
7fc1f1e2b6 Cache hosts in room 2017-03-24 11:46:24 +00:00
Erik Johnston
57cfa513f5 Merge pull request #2054 from matrix-org/erikj/user_iter_cursor 
Reduce some CPU work on DB threads
 2017-03-24 11:40:08 +00:00
Erik Johnston
d58b1ffe94 Replace some calls to cursor_to_dict 
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
 2017-03-24 11:07:02 +00:00
Erik Johnston
e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
David Baker
e36950dec5 Merge pull request #2055 from matrix-org/dbkr/fix_add_msisdn_requestToken 
Fix token request for addition of phone numbers
 2017-03-24 10:32:57 +00:00
David Baker
f902e89d4b Merge pull request #2056 from matrix-org/dbkr/fix_invite_reject 
Fix rejection of invites not reaching sync
 2017-03-24 10:32:51 +00:00
Richard van der Hoff
a380f041c2 try not to drop context after federation requests 
preserve_context_over_fn uses a ContextPreservingDeferred, which only restores
context for the duration of its callbacks, which isn't really correct, and
means that subsequent operations in the same request can end up without their
logcontexts.
 2017-03-23 22:36:21 +00:00
Richard van der Hoff
9397edb28b Merge pull request #2050 from matrix-org/rav/federation_backoff 
push federation retry limiter down to matrixfederationclient
 2017-03-23 22:27:01 +00:00
Richard van der Hoff
06ce7335e9 Merge pull request #2052 from matrix-org/rav/time_bound_deferred 
Fix time_bound_deferred to throw the right exception
 2017-03-23 22:22:54 +00:00
Richard van der Hoff
13c8749ac9 Add another missing yield on check_device_registered 2017-03-23 22:18:53 +00:00
David Baker
e1f1784f99 Fix rejection of invites not reaching sync 
Always allow the user to see their own leave events, otherwise
they won't see the event if they reject an invite for a room whose
history visibility is set such that they cannot see events before
joining.
 2017-03-23 18:50:31 +00:00
David Baker
86e865d7d2 Oops, remove unintentional change 2017-03-23 18:48:30 +00:00
David Baker
a2dfab12c5 Fix token request for addition of phone numbers 2017-03-23 18:46:17 +00:00
Erik Johnston
00957d1aa4 User Cursor.__iter__ instead of fetchall 
This prevents unnecessary construction of lists
 2017-03-23 17:53:49 +00:00
pik
250ce11ab9 Add jsonschema to python_dependencies.py 
Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:47 -03:00
pik
566641a0b5 use jsonschema.FormatChecker for RoomID and UserID strings 
* use a valid filter in rest/client/v2_alpha test

Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:41 -03:00
pik
acafcf1c5b Add valid filter tests, flake8, fix typo 
Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:10 -03:00
pik
e56c79c114 check_valid_filter using JSONSchema 
* add invalid filter tests

Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:07 -03:00
Erik Johnston
6ebe2d23b1 Raise a more helpful exception 2017-03-23 13:48:30 +00:00
Erik Johnston
e64655c25d Don't user upsert to persist new one time keys 
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
 2017-03-23 13:17:00 +00:00
Richard van der Hoff
5a16cb4bf0 Ignore backoff history for invites, aliases, and roomdirs 
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
 2017-03-23 12:23:22 +00:00
Richard van der Hoff
b88a323ffb Fix time_bound_deferred to throw the right exception 
Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred
would throw a CancelledError when the deferred timed out, which was rather
confusing.
 2017-03-23 12:07:11 +00:00
Richard van der Hoff
55366814a6 Merge pull request #2048 from matrix-org/rav/missing_yield 
Add a missing yield in device key upload
 2017-03-23 09:29:44 +00:00
Richard van der Hoff
4bd597d9fc push federation retry limiter down to matrixfederationclient 
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
 2017-03-23 09:28:46 +00:00
Richard van der Hoff
ad8a26e361 MatrixFederationHttpClient: clean up 
rename _create_request to _request, and push ascii-encoding of `destination`
and `path` down into it
 2017-03-23 00:27:04 +00:00
Richard van der Hoff
19b9366d73 Fix a couple of logcontext leaks 
Use preserve_fn to correctly manage the logcontexts around things we don't want
to yield on.
 2017-03-23 00:17:46 +00:00
Richard van der Hoff
e08f81d96a Add a missing yield in device key upload 
(this would only very very rarely actually be a useful thing, so the main
problem was the logcontext leak...)
 2017-03-23 00:16:43 +00:00
Richard van der Hoff
35e0cfb54d Merge pull request #2044 from matrix-org/rav/crypto_docs 
fix up some key verif docstrings
 2017-03-22 17:12:55 +00:00
Richard van der Hoff
7b67848042 Merge pull request #2042 from matrix-org/rav/fix_key_caching 
Fix caching of remote servers' signature keys
 2017-03-22 17:11:55 +00:00
Richard van der Hoff
95f21c7a66 Fix caching of remote servers' signature keys 
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.

By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
 2017-03-22 15:11:30 +00:00
Matthew Hodgson
d101488c5f Merge branch 'master' into develop 2017-03-21 22:48:21 +01:00
Richard van der Hoff
64778693be fix up some key verif docstrings 2017-03-21 13:27:50 +00:00
Erik Johnston
37a187bfab Merge pull request #2033 from matrix-org/erikj/repl_speed 
Don't send the full event json over replication
 2017-03-21 13:11:15 +00:00
Erik Johnston
633dcc316c Bump changelog and version 2017-03-20 16:37:14 +00:00
Richard van der Hoff
c36d15d2de Add some debug to help diagnose weird federation issue 2017-03-20 15:36:14 +00:00
Erik Johnston
aac6d1fc9b PEP8 2017-03-20 13:47:56 +00:00
Richard van der Hoff
bd08ee7a46 Merge pull request #2026 from matrix-org/rav/logcontext_docs 
Logcontext docs
 2017-03-20 12:05:21 +00:00
Richard van der Hoff
eddce9d74a Merge pull request #2027 from matrix-org/rav/logcontext_leaks 
A few fixes to logcontext things
 2017-03-20 11:53:36 +00:00
Richard van der Hoff
f40c2db05a Stop preserve_fn leaking context into the reactor 
Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the
reactor, and add a test for it.

Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do
the same thing but had its own, different, set of bugs.
 2017-03-18 00:07:43 +00:00
Richard van der Hoff
067b00d49d Run the reactor with the sentinel logcontext 
This fixes a class of 'Unexpected logcontext' messages, which were happening
because the logcontext was somewhat arbitrarily swapping between the sentinel
and the `run` logcontext.
 2017-03-18 00:07:43 +00:00
Richard van der Hoff
994d7ae7c5 Remove broken use of clock.call_later 
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.

We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
 2017-03-18 00:01:37 +00:00
Richard van der Hoff
d2d146a314 Logcontext docs 2017-03-17 23:59:28 +00:00
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae Avoid resetting state on rejected events 
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.

This should fix https://github.com/matrix-org/synapse/issues/1935.
 2017-03-17 15:06:08 +00:00
Richard van der Hoff
5068fb16a5 Refactoring and cleanups 
A few non-functional changes:

* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
  readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
  mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
  mind-bending conditional.
 2017-03-17 15:06:07 +00:00
Richard van der Hoff
2abe85d50e Merge pull request #2016 from matrix-org/rav/queue_pdus_during_join 
Queue up federation PDUs while a room join is in progress
 2017-03-17 11:32:44 +00:00
Erik Johnston
da146657c9 Comments 2017-03-16 13:04:07 +00:00
Erik Johnston
a158c36a8a Comment 2017-03-16 11:57:45 +00:00
Erik Johnston
6957bfdca6 Don't recreate so many sets 2017-03-16 11:54:26 +00:00
Erik Johnston
2ccf3b241c Implement no op for room stream in sync 2017-03-16 11:06:41 +00:00
Richard van der Hoff
9ce53a3861 Queue up federation PDUs while a room join is in progress 
This just takes the existing `room_queues` logic and moves it out to
`on_receive_pdu` instead of `_process_received_pdu`, which ensures that we
don't start trying to fetch prev_events and whathaveyou until the join has
completed.
 2017-03-15 18:01:11 +00:00
Erik Johnston
54d2b7e596 Merge pull request #2014 from Half-Shot/hs/fix-appservice-presence 
Add fallback to last_active_ts if it beats the last sync time on a presence timeout.
 2017-03-15 17:37:15 +00:00
Erik Johnston
a8f96c63aa Comment 2017-03-15 16:01:01 +00:00
Erik Johnston
f83ac78201 Cache set of users whose presence the other user should see 2017-03-15 15:29:19 +00:00
Will Hunt
e6032054bf Add a great comment to handle_timeout for active vs sync times. 2017-03-15 15:24:48 +00:00
Will Hunt
ebf5a6b14c Add fallback to last_active_ts if it beats the last sync time. 2017-03-15 15:17:16 +00:00
Erik Johnston
e892457a03 Comment 2017-03-15 15:01:39 +00:00
Erik Johnston
a297155a97 Remove unused import 2017-03-15 14:49:25 +00:00
Erik Johnston
6c82de5100 Format presence events on the edges instead of reformatting them multiple times 2017-03-15 14:27:34 +00:00