Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-20 19:24:21 -05:00
Code Issues Packages Projects Releases Wiki Activity
8,867 Commits 4 Branches 609 Tags 60 MiB
5dbaa520a5
Commit Graph

6367 Commits

Author SHA1 Message Date
Erik Johnston
54f59bd7d4 Merge pull request #2107 from HarHarLinks/patch-1 
fix typo in synctl help
 2017-04-07 09:54:37 +01:00
Erik Johnston
98ce212093 Merge pull request #2103 from matrix-org/erikj/no-double-encode 
Don't double encode replication data
 2017-04-07 09:39:52 +01:00
Kim Brose
8a1137ceab fix typo in synctl help 2017-04-06 17:10:20 +02:00
Erik Johnston
877c029c16 Use iteritems 2017-04-06 15:51:22 +01:00
Erik Johnston
944692ef69 Merge pull request #2106 from matrix-org/erikj/reduce_user_sync 
Reduce rate of USER_SYNC repl commands
 2017-04-06 13:35:31 +01:00
Erik Johnston
391712a4f9 Comment 2017-04-06 13:35:00 +01:00
Erik Johnston
ad544c803a Document types of the replication streams 2017-04-06 13:28:52 +01:00
Erik Johnston
dbf87282d3 Docs 2017-04-06 13:11:21 +01:00
Erik Johnston
69b3fd485d Fix incorrect type when using InvalidateCacheCommand 2017-04-06 09:36:38 +01:00
Daniel Dent
5058292537 Support authenticated SMTP 
Closes (SYN-714) #1385

Signed-off-by: Daniel Dent <matrixcontrib@contactdaniel.net>
 2017-04-05 21:01:08 -07:00
Erik Johnston
fcc803b2bf Add log lines 2017-04-05 17:13:44 +01:00
Erik Johnston
3f213d908d Rearrange metrics 2017-04-05 14:15:09 +01:00
Erik Johnston
1ca0e78ca1 Fix typo 2017-04-05 13:43:39 +01:00
Erik Johnston
b43d3267e2 Fixup some metrics for tcp repl 2017-04-05 13:34:54 +01:00
Erik Johnston
b5cb6347a4 Don't immediately notify the master about users whose syncs have gone away 2017-04-05 13:25:40 +01:00
Erik Johnston
96b9b6c127 Don't double json encode typing replication data 2017-04-05 11:34:20 +01:00
Erik Johnston
f10ce8944b Don't double json encode federation replication data 2017-04-05 11:10:28 +01:00
Erik Johnston
a5c401bd12 Merge pull request #2097 from matrix-org/erikj/repl_tcp_client 
Move to using TCP replication
 2017-04-05 09:36:21 +01:00
Erik Johnston
d1d5362267 Add comment 2017-04-04 16:41:03 +01:00
Erik Johnston
9f26d3b75b Deduplicate new deviceinbox rows for replication 2017-04-04 16:21:21 +01:00
Erik Johnston
a76886726b Merge pull request #2098 from matrix-org/erikj/repl_tcp_fix 
Advance replication streams even if nothing is listening
 2017-04-04 15:40:51 +01:00
Erik Johnston
ac66e11f2b Add the appropriate amount of preserve_fn 2017-04-04 15:22:54 +01:00
Erik Johnston
4264ceb31c Fiddle tcp replication logging 2017-04-04 14:14:03 +01:00
Erik Johnston
023ee197be Advance replication streams even if nothing is listening 
Otherwise the streams don't advance and steadily fall behind, so when a
worker does connect either a) they'll be streamed lots of old updates or
b) the connection will fail as the streams are too far behind.
 2017-04-04 13:19:26 +01:00
Erik Johnston
d1605794ad Remove unused worker config option 2017-04-04 11:17:00 +01:00
Erik Johnston
3376f16012 Shuffle and comment synchrotron presence 2017-04-04 11:14:16 +01:00
Erik Johnston
6ce6bbedcb Move where we ack federation 2017-04-04 11:02:44 +01:00
Erik Johnston
62b89daac6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server 2017-04-04 09:46:16 +01:00
Richard van der Hoff
773e64cc1a Merge pull request #2095 from matrix-org/rav/cull_log_preserves 
Cull spurious PreserveLoggingContexts
 2017-04-03 17:02:25 +01:00
Richard van der Hoff
2d05eb3cf5 Merge remote-tracking branch 'origin/release-v0.20.0' into develop 2017-04-03 16:23:23 +01:00
Richard van der Hoff
30bcbf775a Accept join events from all servers 
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.

(Fixes #1893).
 2017-04-03 15:58:07 +01:00
Richard van der Hoff
7eb9f34cc3 Remove spurious yield 
In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`:
it doesn't return anything anyway.
 2017-04-03 15:44:19 +01:00
Richard van der Hoff
0b08c48fc5 Remove more spurious PreserveLoggingContexts 
Remove `PreserveLoggingContext` around calls to `Notifier.on_new_room_event`;
there is no problem if the logcontext is set when calling it.
 2017-04-03 15:43:37 +01:00
Richard van der Hoff
65e1683680 Remove spurious PreserveLoggingContext 
In `on_new_room_event`, remove `PreserveLoggingContext` - we can call its
subroutines with the logcontext set.
 2017-04-03 15:42:38 +01:00
Richard van der Hoff
feb496056e preserve_fn some deferred-returning things 
In `Notifier._on_new_room_event`, `preserve_fn` around its subroutines which
return deferreds, so that it is safe to call it with an active logcontext.
 2017-04-03 15:41:17 +01:00
Richard van der Hoff
e2eebf1696 Fix fixme in preserve_fn 
`preserve_fn` is no longer used as a decorator anywhere, so we can safely fix a
fixme therein.
 2017-04-03 15:38:02 +01:00
Erik Johnston
36c28bc467 Update all the workers and master to use TCP replication 2017-04-03 15:35:52 +01:00
Erik Johnston
3a1f3f8388 Change slave storage to use new replication interface 
As the TCP replication uses a slightly different API and streams than
the HTTP replication.

This breaks HTTP replication.
 2017-04-03 15:34:19 +01:00
Erik Johnston
52bfa604e1 Add basic replication client handler and factory 2017-04-03 15:34:13 +01:00
Erik Johnston
0a6a966e2b Always advance stream tokens 2017-04-03 15:22:56 +01:00
Richard van der Hoff
773e1c6d68 Remove spurious @preserve_fn decorators 
Remove `@preserve_fn` decorators on `on_new_room_event`,
`_notify_pending_new_room_events`, `_on_new_room_event`, `on_new_event`, and
`on_new_replication_data` - none of these functions return a deferred, and the
decorator does nothing unless the wrapped function returns a deferred, so the
decorator was a no-op.
 2017-04-03 15:14:11 +01:00
Erik Johnston
0d1c85e643 Merge branch 'release-v0.20.0' of github.com:matrix-org/synapse into develop 2017-04-03 14:58:14 +01:00
Erik Johnston
1df7c28661 Use callbacks to notify tcp replication rather than deferreds 2017-03-31 15:42:51 +01:00
Erik Johnston
36d2b66f90 Add a timestamp to USER_SYNC command 
This timestamp is used to indicate when the user last sync'd
 2017-03-31 15:42:22 +01:00
Erik Johnston
8a240e4f9c Merge pull request #2078 from APwhitehat/assertuserfriendly 
add user friendly report of assertion error in synctl.py
 2017-03-31 14:41:49 +01:00
Erik Johnston
142b6b4abf Merge pull request #2011 from matrix-org/matthew/turn_allow_guests 
add setting (on by default) to support TURN for guests
 2017-03-31 14:37:09 +01:00
Erik Johnston
2a06b44be2 Merge pull request #1986 from matrix-org/matthew/enable_guest_3p 
enable guest access for the 3pl/3pid APIs
 2017-03-31 14:36:03 +01:00
Matthew Hodgson
9ee397b440 switch to allow_guest=True for authing 3Ps as per PR feedback 2017-03-31 13:54:26 +01:00
Erik Johnston
9d0170ac6c Fix up presence 2017-03-31 11:36:32 +01:00
Erik Johnston
bfcf016714 Fix up docs 2017-03-31 11:19:24 +01:00
Erik Johnston
9cee0ce7db Merge pull request #2075 from matrix-org/erikj/cache_speed 
Speed up cached function access
 2017-03-31 10:10:56 +01:00
Erik Johnston
350333a09a Merge pull request #2076 from matrix-org/erikj/as_perf 
Make AS's faster
 2017-03-31 09:43:10 +01:00
Erik Johnston
4d17add8de Remove unused instance variable 2017-03-31 09:38:27 +01:00
Erik Johnston
27b1b4a2c9 Speed up copy_and_replace 2017-03-30 17:50:31 +01:00
Erik Johnston
5b5b171f3e Docs 2017-03-30 17:05:53 +01:00
Erik Johnston
b282fe7170 Revert log context change 2017-03-30 17:03:59 +01:00
Erik Johnston
9ff4e0e91b Bump version and changelog 2017-03-30 16:37:40 +01:00
Erik Johnston
0834d1a70c Merge pull request #2079 from matrix-org/erikj/push_regex_cache 
Cache glob to regex at a higher level for push
 2017-03-30 16:28:18 +01:00
Erik Johnston
63fcc42990 Remove user from process_presence when stops syncing 2017-03-30 14:26:08 +01:00
Erik Johnston
6194a64ae9 Doc new instance variables 2017-03-30 14:19:10 +01:00
Erik Johnston
014fee93b3 Manually calculate cache key as getcallargs is expensive 
This is because getcallargs recomputes the getargspec, amongst other
things, which we don't need to do as its already been done
 2017-03-30 14:14:46 +01:00
Erik Johnston
86780a8bc3 Don't convert to deferreds when not necessary 2017-03-30 14:14:36 +01:00
Erik Johnston
3ba2859e0c Add tcp replication listener type and hook it up 2017-03-30 13:31:10 +01:00
Erik Johnston
e9dd8370b0 Add functions to presence to support remote syncs 
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
 2017-03-30 13:25:14 +01:00
Erik Johnston
4d7fc7f977 Add server side resource for tcp replication 2017-03-30 13:24:45 +01:00
Richard van der Hoff
f9b4bb05e0 Fix the logcontext handling in the cache wrappers (#2077) 
The cache wrappers had a habit of leaking the logcontext into the reactor while
the lookup function was running, and then not restoring it correctly when the
lookup function had completed. It's all the fault of
`preserve_context_over_{fn,deferred}` which are basically a bit broken.
 2017-03-30 13:22:24 +01:00
Erik Johnston
7450693435 Initial TCP protocol implementation 
This defines the low level TCP replication protocol
 2017-03-30 12:54:46 +01:00
Erik Johnston
8da6f0be48 Define the various streams we will replicate 2017-03-30 12:54:46 +01:00
Erik Johnston
11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston
7984708a55 Add a simple hook to wait for replication traffic 2017-03-30 11:57:52 +01:00
Erik Johnston
24d35ab47b Add new storage functions for new replication 
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
 2017-03-30 11:48:35 +01:00
Erik Johnston
30348c924c Use txn.fetchall() so we can reuse txn 2017-03-30 10:30:05 +01:00
Anant Prakash
6cdca71079 synctl.py: wait for synapse to stop before restarting (#2020) 2017-03-29 19:20:13 +01:00
Anant Prakash
305d16d612 add user friendly report of assertion error in synctl.py 
Signed-off-by: Anant Prakash <anantprakashjsr@gmail.com>
 2017-03-29 20:41:39 +05:30
Erik Johnston
a3810136fe Cache glob to regex at a higher level for push 2017-03-29 15:53:14 +01:00
Erik Johnston
3ce8d59176 Increase cache size for _get_state_group_for_event 2017-03-29 14:31:46 +01:00
Erik Johnston
b9c2ae6788 Merge pull request #1849 from matrix-org/erikj/state_typo 
Fix bug where current_state_events renamed to current_state_ids
 2017-03-29 11:53:42 +01:00
Erik Johnston
85be3dde81 Bail early if remote wouldn't be retried (#2064) 
* Bail early if remote wouldn't be retried

* Don't always return true

* Just use get_retry_limiter

* Spelling
 2017-03-29 11:48:27 +01:00
Erik Johnston
2f8b580b64 Merge pull request #2053 from matrix-org/erikj/e2e_one_time_upsert 
Don't user upsert to persist new one time keys
 2017-03-29 11:44:23 +01:00
Erik Johnston
c5b0bdd542 Merge pull request #2067 from matrix-org/erikj/notify_on_fed 
Notify on new federation traffic
 2017-03-29 11:41:37 +01:00
Erik Johnston
e4df0e189d Decrank last commit 2017-03-29 11:02:35 +01:00
Erik Johnston
4ad613f6be Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert 2017-03-29 10:57:19 +01:00
Erik Johnston
ac6bc55512 Correctly look up key 2017-03-29 10:56:26 +01:00
Erik Johnston
69efd77749 Add comment 2017-03-29 09:50:05 +01:00
Erik Johnston
51b156d48a Cache whether an AS is interested based on members 2017-03-28 13:27:21 +01:00
Erik Johnston
30f5ffdca2 Remove param and cast at call site 2017-03-28 13:27:21 +01:00
Erik Johnston
650f0e69f2 Compile the regex's used in ASes 2017-03-28 13:27:21 +01:00
Erik Johnston
d28db583da Merge pull request #2063 from matrix-org/erikj/device_list_batch 
Batch sending of device list pokes
 2017-03-28 11:35:41 +01:00
Erik Johnston
58a35366be The algorithm is part of the key id 2017-03-28 11:34:37 +01:00
Erik Johnston
bac9bf1b12 Typo 2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f Short circuit if all new events have same state group 2017-03-27 18:00:47 +01:00
Erik Johnston
35b4aa04be Notify on new federation traffic 2017-03-27 14:07:47 +01:00
Erik Johnston
2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Erik Johnston
281553afe6 Merge pull request #2062 from matrix-org/erikj/presence_replication 
Use presence replication stream to invalidate cache
 2017-03-24 13:57:45 +00:00
Erik Johnston
987f4945b4 Actually call invalidate 2017-03-24 13:28:20 +00:00
Erik Johnston
31d56c3fb5 Merge pull request #2061 from matrix-org/erikj/add_transaction_store 
Add slave transaction store to workers who send federation requests
 2017-03-24 13:24:27 +00:00
Erik Johnston
09f79aaad0 Use presence replication stream to invalidate cache 
Instead of using the cache invalidation replication stream to invalidate
the _get_presence_cache, we can instead rely on the presence replication
stream. This reduces the amount of replication traffic considerably.
 2017-03-24 13:21:08 +00:00
Erik Johnston
23e0ff840a Merge pull request #2060 from matrix-org/erikj/cache_hosts_in_room 
Cache hosts in room
 2017-03-24 13:07:22 +00:00
Erik Johnston
48e7697911 Add slave transaction store 2017-03-24 13:05:30 +00:00
Richard van der Hoff
f136c89d5e Merge pull request #2058 from matrix-org/rav/logcontext_leaks_2 
try not to drop context after federation requests
 2017-03-24 12:47:26 +00:00
Richard van der Hoff
01fc847f7f Merge pull request #2057 from matrix-org/rav/missing_yield_2 
Add another missing yield on check_device_registered
 2017-03-24 12:46:43 +00:00
Erik Johnston
7fc1f1e2b6 Cache hosts in room 2017-03-24 11:46:24 +00:00
Erik Johnston
57cfa513f5 Merge pull request #2054 from matrix-org/erikj/user_iter_cursor 
Reduce some CPU work on DB threads
 2017-03-24 11:40:08 +00:00
Erik Johnston
d58b1ffe94 Replace some calls to cursor_to_dict 
cursor_to_dict can be surprisinglh expensive for large result sets, so lets
only call it when we need to.
 2017-03-24 11:07:02 +00:00
Erik Johnston
e71940aa64 Use iter(items|values) 2017-03-24 10:57:02 +00:00
David Baker
e36950dec5 Merge pull request #2055 from matrix-org/dbkr/fix_add_msisdn_requestToken 
Fix token request for addition of phone numbers
 2017-03-24 10:32:57 +00:00
David Baker
f902e89d4b Merge pull request #2056 from matrix-org/dbkr/fix_invite_reject 
Fix rejection of invites not reaching sync
 2017-03-24 10:32:51 +00:00
Richard van der Hoff
a380f041c2 try not to drop context after federation requests 
preserve_context_over_fn uses a ContextPreservingDeferred, which only restores
context for the duration of its callbacks, which isn't really correct, and
means that subsequent operations in the same request can end up without their
logcontexts.
 2017-03-23 22:36:21 +00:00
Richard van der Hoff
9397edb28b Merge pull request #2050 from matrix-org/rav/federation_backoff 
push federation retry limiter down to matrixfederationclient
 2017-03-23 22:27:01 +00:00
Richard van der Hoff
06ce7335e9 Merge pull request #2052 from matrix-org/rav/time_bound_deferred 
Fix time_bound_deferred to throw the right exception
 2017-03-23 22:22:54 +00:00
Richard van der Hoff
13c8749ac9 Add another missing yield on check_device_registered 2017-03-23 22:18:53 +00:00
David Baker
e1f1784f99 Fix rejection of invites not reaching sync 
Always allow the user to see their own leave events, otherwise
they won't see the event if they reject an invite for a room whose
history visibility is set such that they cannot see events before
joining.
 2017-03-23 18:50:31 +00:00
David Baker
86e865d7d2 Oops, remove unintentional change 2017-03-23 18:48:30 +00:00
David Baker
a2dfab12c5 Fix token request for addition of phone numbers 2017-03-23 18:46:17 +00:00
Erik Johnston
00957d1aa4 User Cursor.__iter__ instead of fetchall 
This prevents unnecessary construction of lists
 2017-03-23 17:53:49 +00:00
pik
250ce11ab9 Add jsonschema to python_dependencies.py 
Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:47 -03:00
pik
566641a0b5 use jsonschema.FormatChecker for RoomID and UserID strings 
* use a valid filter in rest/client/v2_alpha test

Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:41 -03:00
pik
acafcf1c5b Add valid filter tests, flake8, fix typo 
Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:10 -03:00
pik
e56c79c114 check_valid_filter using JSONSchema 
* add invalid filter tests

Signed-off-by: pik <alexander.maznev@gmail.com>
 2017-03-23 11:42:07 -03:00
Erik Johnston
6ebe2d23b1 Raise a more helpful exception 2017-03-23 13:48:30 +00:00
Erik Johnston
e64655c25d Don't user upsert to persist new one time keys 
Instead we no-op duplicate one time key uploads, an error if the key_id
already exists but encodes a different key.
 2017-03-23 13:17:00 +00:00
Richard van der Hoff
5a16cb4bf0 Ignore backoff history for invites, aliases, and roomdirs 
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
 2017-03-23 12:23:22 +00:00
Richard van der Hoff
b88a323ffb Fix time_bound_deferred to throw the right exception 
Due to a failure to instantiate DeferredTimedOutError, time_bound_deferred
would throw a CancelledError when the deferred timed out, which was rather
confusing.
 2017-03-23 12:07:11 +00:00
Richard van der Hoff
55366814a6 Merge pull request #2048 from matrix-org/rav/missing_yield 
Add a missing yield in device key upload
 2017-03-23 09:29:44 +00:00
Richard van der Hoff
4bd597d9fc push federation retry limiter down to matrixfederationclient 
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
 2017-03-23 09:28:46 +00:00
Richard van der Hoff
ad8a26e361 MatrixFederationHttpClient: clean up 
rename _create_request to _request, and push ascii-encoding of `destination`
and `path` down into it
 2017-03-23 00:27:04 +00:00
Richard van der Hoff
19b9366d73 Fix a couple of logcontext leaks 
Use preserve_fn to correctly manage the logcontexts around things we don't want
to yield on.
 2017-03-23 00:17:46 +00:00
Richard van der Hoff
e08f81d96a Add a missing yield in device key upload 
(this would only very very rarely actually be a useful thing, so the main
problem was the logcontext leak...)
 2017-03-23 00:16:43 +00:00
Richard van der Hoff
35e0cfb54d Merge pull request #2044 from matrix-org/rav/crypto_docs 
fix up some key verif docstrings
 2017-03-22 17:12:55 +00:00
Richard van der Hoff
7b67848042 Merge pull request #2042 from matrix-org/rav/fix_key_caching 
Fix caching of remote servers' signature keys
 2017-03-22 17:11:55 +00:00
Richard van der Hoff
95f21c7a66 Fix caching of remote servers' signature keys 
The `@cached` decorator on `KeyStore._get_server_verify_key` was missing
its `num_args` parameter, which meant that it was returning the wrong key for
any server which had more than one recorded key.

By way of a fix, change the default for `num_args` to be *all* arguments. To
implement that, factor out a common base class for `CacheDescriptor` and `CacheListDescriptor`.
 2017-03-22 15:11:30 +00:00
Matthew Hodgson
d101488c5f Merge branch 'master' into develop 2017-03-21 22:48:21 +01:00
Richard van der Hoff
64778693be fix up some key verif docstrings 2017-03-21 13:27:50 +00:00
Erik Johnston
37a187bfab Merge pull request #2033 from matrix-org/erikj/repl_speed 
Don't send the full event json over replication
 2017-03-21 13:11:15 +00:00
Erik Johnston
633dcc316c Bump changelog and version 2017-03-20 16:37:14 +00:00
Richard van der Hoff
c36d15d2de Add some debug to help diagnose weird federation issue 2017-03-20 15:36:14 +00:00
Erik Johnston
aac6d1fc9b PEP8 2017-03-20 13:47:56 +00:00
Richard van der Hoff
bd08ee7a46 Merge pull request #2026 from matrix-org/rav/logcontext_docs 
Logcontext docs
 2017-03-20 12:05:21 +00:00
Richard van der Hoff
eddce9d74a Merge pull request #2027 from matrix-org/rav/logcontext_leaks 
A few fixes to logcontext things
 2017-03-20 11:53:36 +00:00
Richard van der Hoff
f40c2db05a Stop preserve_fn leaking context into the reactor 
Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the
reactor, and add a test for it.

Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do
the same thing but had its own, different, set of bugs.
 2017-03-18 00:07:43 +00:00
Richard van der Hoff
067b00d49d Run the reactor with the sentinel logcontext 
This fixes a class of 'Unexpected logcontext' messages, which were happening
because the logcontext was somewhat arbitrarily swapping between the sentinel
and the `run` logcontext.
 2017-03-18 00:07:43 +00:00
Richard van der Hoff
994d7ae7c5 Remove broken use of clock.call_later 
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.

We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
 2017-03-18 00:01:37 +00:00
Richard van der Hoff
d2d146a314 Logcontext docs 2017-03-17 23:59:28 +00:00
Erik Johnston
61f471f779 Don't send the full event json over replication 2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae Avoid resetting state on rejected events 
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.

This should fix https://github.com/matrix-org/synapse/issues/1935.
 2017-03-17 15:06:08 +00:00
Richard van der Hoff
5068fb16a5 Refactoring and cleanups 
A few non-functional changes:

* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
  readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
  mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
  mind-bending conditional.
 2017-03-17 15:06:07 +00:00
Richard van der Hoff
2abe85d50e Merge pull request #2016 from matrix-org/rav/queue_pdus_during_join 
Queue up federation PDUs while a room join is in progress
 2017-03-17 11:32:44 +00:00
Erik Johnston
da146657c9 Comments 2017-03-16 13:04:07 +00:00
Erik Johnston
a158c36a8a Comment 2017-03-16 11:57:45 +00:00
Erik Johnston
6957bfdca6 Don't recreate so many sets 2017-03-16 11:54:26 +00:00
Erik Johnston
2ccf3b241c Implement no op for room stream in sync 2017-03-16 11:06:41 +00:00
Richard van der Hoff
9ce53a3861 Queue up federation PDUs while a room join is in progress 
This just takes the existing `room_queues` logic and moves it out to
`on_receive_pdu` instead of `_process_received_pdu`, which ensures that we
don't start trying to fetch prev_events and whathaveyou until the join has
completed.
 2017-03-15 18:01:11 +00:00
Erik Johnston
54d2b7e596 Merge pull request #2014 from Half-Shot/hs/fix-appservice-presence 
Add fallback to last_active_ts if it beats the last sync time on a presence timeout.
 2017-03-15 17:37:15 +00:00
Erik Johnston
a8f96c63aa Comment 2017-03-15 16:01:01 +00:00
Erik Johnston
f83ac78201 Cache set of users whose presence the other user should see 2017-03-15 15:29:19 +00:00
Will Hunt
e6032054bf Add a great comment to handle_timeout for active vs sync times. 2017-03-15 15:24:48 +00:00
Will Hunt
ebf5a6b14c Add fallback to last_active_ts if it beats the last sync time. 2017-03-15 15:17:16 +00:00
Erik Johnston
e892457a03 Comment 2017-03-15 15:01:39 +00:00
Erik Johnston
a297155a97 Remove unused import 2017-03-15 14:49:25 +00:00
Erik Johnston
6c82de5100 Format presence events on the edges instead of reformatting them multiple times 2017-03-15 14:27:34 +00:00
David Baker
0ad44acb5a Merge pull request #1997 from matrix-org/dbkr/cas_partialdownload 
Handle PartialDownloadError in CAS login
 2017-03-15 13:52:34 +00:00
Matthew Hodgson
0970e0307e typo 2017-03-15 12:40:42 +00:00
Matthew Hodgson
5aa42d4292 set default for turn_allow_guests correctly 2017-03-15 12:40:13 +00:00
Matthew Hodgson
e0ff66251f add setting (on by default) to support TURN for guests 2017-03-15 12:22:18 +00:00
Richard van der Hoff
29ed09e80a Fix assertion to stop transaction queue getting wedged 
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
 2017-03-15 12:16:55 +00:00
Erik Johnston
872e75a3d5 Add some metrics on notifier 2017-03-15 10:56:51 +00:00
Erik Johnston
7827251daf Merge pull request #1994 from matrix-org/dbkr/msisdn_signin_2 
Phone number registration / login support v2
 2017-03-15 09:59:54 +00:00
Richard van der Hoff
b5d1c68beb Implement reset_context_after_deferred 
to correctly reset the context when we fire off a deferred we aren't going to
wait for.
 2017-03-15 02:21:07 +00:00
Richard van der Hoff
f2ed64eaaf Merge pull request #1992 from matrix-org/rav/fix_media_loop 
Fix routing loop when fetching remote media
 2017-03-14 23:40:35 +00:00
Erik Johnston
bad72b0b8e Merge pull request #2002 from matrix-org/erikj/dont_sync_by_default 
Reduce number of spurious sync result generations.
 2017-03-14 16:58:45 +00:00
Erik Johnston
fd2eef49c8 Reduce spurious calls to generate sync 2017-03-14 16:33:45 +00:00
Richard van der Hoff
1d09586599 Address review comments 
- don't blindly proxy all HTTPRequestExceptions
- log unexpected exceptions at error
- avoid `isinstance`
- improve docs on `from_http_response_exception`
 2017-03-14 14:15:37 +00:00
Richard van der Hoff
7f237800e9 re-refactor exception heirarchy 
Give CodeMessageException back its `msg` attribute, and use that to hold the
HTTP status message for HttpResponseException.
 2017-03-14 14:15:37 +00:00
David Baker
1ece06273e Handle PartialDownloadError in CAS login 2017-03-14 13:37:36 +00:00
Erik Johnston
bb256ac96f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed 2017-03-14 11:35:05 +00:00
Erik Johnston
cc7a294e2e Fix current_state_events table to not lie 
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
 2017-03-14 10:57:43 +00:00
David Baker
7b6ed9871e Use extend instead of += 2017-03-14 10:49:55 +00:00
David Baker
d79a687d85 Oops, remove print 2017-03-14 10:40:20 +00:00
Luke Barnard
f29d85d9e4 Merge pull request #1993 from matrix-org/luke/delete-devices 
Implement delete_devices API
 2017-03-14 10:02:56 +00:00
Luke Barnard
bbeeb97f75 Implement _simple_delete_many_txn, use it to delete devices 
(But this doesn't implement the same for deleting access tokens or e2e keys.

Also respond to code review.
 2017-03-13 17:53:23 +00:00
David Baker
0a9945220e Fix registration for broken clients 
Only offer msisdn flows if the x_show_msisdn option is given.
 2017-03-13 17:29:38 +00:00
David Baker
73a5f06652 Support registration / login with phone number 
Changes from https://github.com/matrix-org/synapse/pull/1971
 2017-03-13 17:27:51 +00:00
Luke Barnard
c077c3277b Flake 2017-03-13 16:45:38 +00:00
Richard van der Hoff
31f3ca1b2b Merge pull request #1990 from matrix-org/rav/log_config_comments 
Add helpful texts to logger config options
 2017-03-13 16:42:12 +00:00
Luke Barnard
c81f33f73d Implement delete_devices API 
This implements the proposal here https://docs.google.com/document/d/1C-25Gqz3TXy2jIAoeOKxpNtmme0jI4g3yFGqv5GlAAk for deleting multiple devices at once in a single request.
 2017-03-13 16:33:51 +00:00
Richard van der Hoff
170ccc9de5 Fix routing loop when fetching remote media 
When we proxy a media request to a remote server, add a query-param, which will
tell the remote server to 404 if it doesn't recognise the server_name.

This should fix a routing loop where the server keeps forwarding back to
itself.

Also improves the error handling on remote media fetches, so that we don't
always return a rather obscure 502.
 2017-03-13 16:30:36 +00:00
Erik Johnston
45c7f12d2a Add new storage function to slave store 2017-03-13 16:26:44 +00:00
Richard van der Hoff
6037a9804c Add helpful texts to logger config options 2017-03-13 12:33:35 +00:00
Richard van der Hoff
3c69f32402 Merge remote-tracking branch 'origin/develop' into rav/refactor_received_pdu 2017-03-13 12:20:47 +00:00
Richard van der Hoff
6bfe8e32b5 Merge pull request #1983 from matrix-org/rav/no_redirect_stdio 
Add an option to disable stdio redirect
 2017-03-13 12:20:07 +00:00
Richard van der Hoff
5fc9261929 Merge pull request #1982 from matrix-org/rav/sighup_for_logconfig 
Reread log config on SIGHUP
 2017-03-13 12:19:54 +00:00
Erik Johnston
0162994983 Comments 2017-03-13 11:53:26 +00:00
Erik Johnston
254b7c5b15 Bump changelog and versions 2017-03-13 10:02:58 +00:00
Erik Johnston
672dcf59d3 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.19.3 2017-03-13 09:59:54 +00:00
Erik Johnston
7eae6eaa2f Revert "Support registration & login with phone number" 2017-03-13 09:59:33 +00:00
Erik Johnston
79926e016e Assume rooms likely haven't changed 2017-03-13 09:50:10 +00:00
Matthew Hodgson
a61dd408ed enable guest access for the 3pl/3pid APIs 2017-03-12 19:30:45 +00:00
Erik Johnston
8ffbe43ba1 Get current state by using current_state_events table 2017-03-10 17:39:35 +00:00
Richard van der Hoff
bcfa5cd00c Add an option to disable stdio redirect 
This makes it tractable to run synapse under pdb.
 2017-03-10 15:38:29 +00:00
Richard van der Hoff
d84bd51e95 Refactor logger config for workers 
- to make it easier to add more config options.
 2017-03-10 15:34:01 +00:00
Richard van der Hoff
9072a8c627 Reread log config on SIGHUP 
When we are using a log_config file, reread it on SIGHUP.
 2017-03-10 15:29:55 +00:00
Erik Johnston
8f267fa8a8 Fix it for the workers 2017-03-10 11:22:25 +00:00
Erik Johnston
64d62e41b8 Noop repated delete device inbox calls from /sync 2017-03-10 10:36:43 +00:00
Erik Johnston
3545e17f43 Add setdefault key to ExpiringCache 2017-03-10 10:30:49 +00:00
Richard van der Hoff
29235901b8 Move FederationServer._handle_new_pdu to FederationHandler 
Unfortunately this significantly increases the size of the already-rather-big
FederationHandler, but the code fits more naturally here, and it paves the way
for the tighter integration that I need between handling incoming PDUs and
doing the join dance.

Other than renaming the existing `FederationHandler.on_receive_pdu` to
`_process_received_pdu` to make way for it, this just consists of the move, and
replacing `self.handler` with `self` and `self` with `self.replication_layer`.
 2017-03-09 16:20:13 +00:00
Richard van der Hoff
e8b1721290 Move sig check out of _handle_new_pdu 
When we receive PDUs via `get_missing_events`, we have already checked their
sigs, so there is no need to do it again.
 2017-03-09 15:50:44 +00:00
Richard van der Hoff
3406333a58 Factor _get_missing_events_for_pdu out of _handle_new_pdu 
This should be functionally identical: it just seeks to improve readability by
reducing indentation.
 2017-03-09 15:50:44 +00:00
Richard van der Hoff
45d173a59a Fix docstring 2017-03-09 15:50:29 +00:00
David Baker
663396e45d Merge pull request #1971 from matrix-org/dbkr/msisdn_signin 
Support registration & login with phone number
 2017-03-09 10:18:51 +00:00
David Baker
ece7e00048 Comment when our 3pids would be incomplete 2017-03-08 19:07:18 +00:00
David Baker
9d0d40fc15 Docs 2017-03-08 19:05:29 +00:00
David Baker
3edc57296d Incorrectly copied copyright 
This file post-dates OM
 2017-03-08 19:00:51 +00:00
David Baker
727124a762 Not any more, it doesn't 2017-03-08 19:00:23 +00:00
Richard van der Hoff
6ad71cc29d Remove spurious SQL logging (#1972) 
looks like the upsert function was accidentally sending sql logging to the
general logger. We already log the sql in `txn.execute`.
 2017-03-08 18:00:44 +00:00
David Baker
d4d3629aaf Better error message 2017-03-08 17:01:26 +00:00
Erik Johnston
3170c56e07 Bump changelog and version 2017-03-08 13:00:47 +00:00
David Baker
1c99934b28 pep8 2017-03-08 11:58:20 +00:00
David Baker
a9e2b9ec16 Add msisdn util file 2017-03-08 11:53:36 +00:00
David Baker
85bb322333 Pull out datastore in initialiser 2017-03-08 11:51:25 +00:00
David Baker
65d43f3ca5 Minor fixes from PR feedback 2017-03-08 11:48:43 +00:00
David Baker
0e0aee25c4 Fix log line 2017-03-08 11:46:22 +00:00
David Baker
82c5e7de25 Typos 2017-03-08 11:42:44 +00:00
David Baker
2e27339add Refector out assert_params_in_request 
and replace requestEmailToken where we meant requestMsisdnToken
 2017-03-08 11:37:34 +00:00
David Baker
88df6c0c9a Factor out msisdn canonicalisation 
Plus a couple of other minor fixes
 2017-03-08 11:03:39 +00:00
David Baker
402a7bf63d Fix pep8 2017-03-08 09:33:40 +00:00
David Baker
00466e2feb Support new login format 
https://docs.google.com/document/d/1-6ZSSW5YvCGhVFDyD2QExAUAdpCWjccvJT5xiyTTG2Y/edit#
 2017-03-07 16:37:23 +00:00
Erik Johnston
ac5491f563 Select distinct devices from DB 
Otherwise we might pull out tonnes of duplicate user_ids and this can
make synapse sad.
 2017-03-06 11:10:14 +00:00
David Baker
b0effa2160 Add msisdns as 3pids during registration 
and support binding them with the bind_msisdn param
 2017-03-03 18:34:39 +00:00
Erik Johnston
82f7f1543b Merge pull request #1964 from matrix-org/erikj/device_list_update_fix 
Fix device list update to not constantly resync
 2017-03-03 16:18:16 +00:00
Erik Johnston
f2581ee8b8 Don't keep around old stream IDs forever 2017-03-03 16:02:53 +00:00
Erik Johnston
9834367eea Spelling 2017-03-03 15:31:57 +00:00
Erik Johnston
da52d3af31 Fix up 2017-03-03 15:29:13 +00:00
David Baker
ad882cd54d Just return the deferred straight off 
defer.returnValue doth not maketh a generator: it would need a
yield to be a generator, and this doesn't need a yield.
 2017-03-01 18:08:51 +00:00
David Baker
3557cf34dc Merge remote-tracking branch 'origin/develop' into dbkr/msisdn_signin 2017-03-01 17:20:37 +00:00
Erik Johnston
d766343668 Add index to device_lists_stream 2017-03-01 15:56:30 +00:00
Erik Johnston
36be39b8b3 Fix device list update to not constantly resync 2017-03-01 14:12:11 +00:00
Erik Johnston
3365117151 Clobber old device list stream entries 2017-03-01 10:21:30 +00:00
Erik Johnston
e933a2712d Don't log unknown cache warnings in workers 2017-02-28 16:22:41 +00:00
Erik Johnston
d638a7484b Merge pull request #1959 from matrix-org/erikj/intern_once 
Intern table column names once
 2017-02-28 15:15:16 +00:00
Erik Johnston
b84907bdbb Intern table column names once 2017-02-28 14:38:16 +00:00
Erik Johnston
e4919b9329 Add stream_id index to device_lists_outbound_pokes 
As this is used for replication streaming
 2017-02-28 11:19:06 +00:00
Erik Johnston
8a12b6f1eb Fix up txn name 2017-02-28 10:15:50 +00:00
Erik Johnston
848cf95ea0 Pop with default value to stop throwing 2017-02-28 10:02:54 +00:00
Erik Johnston
eda96586ca Merge pull request #1955 from matrix-org/erikj/current_state_query_bypass 
Don't fetch current state in common case
 2017-02-27 19:15:59 +00:00
Erik Johnston
64a2cef9bb Pop rather than del from dict 2017-02-27 19:15:36 +00:00
Erik Johnston
a41dce8f8a Remove needless check 2017-02-27 18:54:43 +00:00
Erik Johnston
c0d6045776 It should be all 2017-02-27 18:45:24 +00:00
Erik Johnston
49f4bc4709 Don't fetch current state in common case 
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
 2017-02-27 18:33:41 +00:00
Erik Johnston
fc5d876dba Merge pull request #1954 from matrix-org/erikj/cache_device2 
Cache get_user_devices_from_cache
 2017-02-27 16:36:40 +00:00
Erik Johnston
f58dbb02a6 Cache get_user_devices_from_cache 2017-02-27 16:22:12 +00:00
Erik Johnston
acf6d4d2e3 Merge pull request #1945 from jkolo/fix_dynamic_thumbnails_aspect 
Fix #1677 (dynamic thumbnails aspect)
 2017-02-27 09:51:52 +00:00
Jurek
aea5461488 Fix dynamic thumbnails aspect 2017-02-24 22:43:27 +01:00
Erik Johnston
1a4f8022e6 Strip newlines from SQL queries 2017-02-23 11:15:31 +00:00
Erik Johnston
b2d20e94fa Remove lock from rotate notifs 2017-02-22 14:24:02 +00:00
Erik Johnston
7455ba436a Ensure we pass positive ints to delay function 2017-02-22 12:08:14 +00:00
Erik Johnston
b7442c3e2b Store looping call 2017-02-21 13:59:25 +00:00
Erik Johnston
a3708a1885 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-02-21 13:46:27 +00:00
Erik Johnston
30ecfef5a3 Bump version and changelog 2017-02-21 13:43:36 +00:00
Richard van der Hoff
0c4cf9372b Fix a race in transaction queue 
It was theoretically possible for a PDU to get queued and not sent for ages. On
closer inspection I think there were bigger problems elsewhere, but we might as
well fix this since it's easy.
 2017-02-20 16:46:25 +00:00
Erik Johnston
6226a27bf8 Remove unused param 2017-02-20 16:01:54 +00:00
Erik Johnston
efff39c030 Fix /context/ visibiltiy rules 2017-02-20 16:01:49 +00:00
Erik Johnston
17673404fb Remove unused param 2017-02-20 15:02:01 +00:00
Erik Johnston
7f026792e1 Fix /context/ visibiltiy rules 2017-02-20 14:54:50 +00:00
Erik Johnston
699be7d1be Fix up notif rotation 2017-02-18 14:42:39 +00:00
David Baker
ec8d7603e6 Merge pull request #1925 from matrix-org/dbkr/pushers_lang_lengthen 
Make the pushers lang field column longer
 2017-02-17 11:29:06 +00:00
David Baker
8c87bb550e Merge pull request #1922 from matrix-org/dbkr/allow_forget_for_ban 
Allow forgetting rooms you're banned from
 2017-02-17 10:52:30 +00:00
David Baker
4aa29508af Use TEXT rather than VARCHAR 
While we're changing anyway
 2017-02-17 10:51:49 +00:00
David Baker
b4017539d4 Make the pushers lang field column longer 
To accommodate things like zh-Hans-CN

Fixes https://github.com/vector-im/riot-ios/issues/1031
 2017-02-17 10:42:57 +00:00
Erik Johnston
138e030cfe Comment 2017-02-16 15:03:36 +00:00
Erik Johnston
502ae6c663 Comment 2017-02-16 14:47:11 +00:00
Erik Johnston
e6acf0c399 Store the default push actions in a more efficient manner 2017-02-16 14:40:24 +00:00
Erik Johnston
04eca2589d Merge pull request #1916 from matrix-org/erikj/push_actions_delete 
Aggregate event push actions
 2017-02-16 15:28:58 +01:00
David Baker
474c9aadbe Allow forgetting rooms you're banned from 2017-02-15 19:32:20 +00:00
Richard van der Hoff
7dcbcca68c Merge pull request #1921 from matrix-org/rav/fix_key_changes 
Fix bugs in the /keys/changes api
 2017-02-15 11:25:16 +00:00
David Baker
fa467e62a9 Merge pull request #1917 from matrix-org/dbkr/make_ban_reasons_work 
Make kick & ban reasons work
 2017-02-14 16:10:06 +00:00
David Baker
355d62c499 Make kick & ban reasons work 
We somehow specced APIs with reason strings, preserve the content
in the events  and even have the clients display them, but failed
to actually pass the parameter through to the event content.
 2017-02-14 15:10:55 +00:00
David Baker
ce3e583d94 WIP support for msisdn 3pid proxy methods 2017-02-14 15:05:55 +00:00
Richard van der Hoff
fc2f29c1d0 Fix bugs in the /keys/changes api 
* `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were
  passing a `RoomStreamToken`, which meant that it returned the *current*
  extremities, rather than those corresponding to the `from_token`. However:
* `get_state_ids_for_events` required a second ('types') parameter; this meant
  that a `TypeError` was thrown and we ended up acting as though there was *no*
  prev state.
* `get_state_ids_for_events` actually returns a map from event_id to state
  dictionary - just looking up the state keys in it again meant that we acted
  as though there was no prev state. We now check if each member's state has
  changed since *any* of the extremities.

Also add/fix some comments.
 2017-02-14 13:59:50 +00:00
Erik Johnston
ce3c8df6df Less aggressive timers 2017-02-14 13:41:24 +00:00
Erik Johnston
095b45c165 Aggregate event push actions 2017-02-14 13:39:41 +00:00
Erik Johnston
795f8e3fe7 Merge pull request #1873 from matrix-org/erikj/delete_push_actions 
Be more agressive about purging old room event_push_actions
 2017-02-14 14:29:04 +01:00
Erik Johnston
d7457c7661 Merge pull request #1914 from matrix-org/erikj/cache_presence 
Cache get_presence storage
 2017-02-13 16:59:19 +01:00
Erik Johnston
9e617cd4c2 Cache get_presence storage 2017-02-13 13:50:03 +00:00
Kegan Dougal
d0497425f8 Ordering is important on errbacks so add the cleanup func before creating an ObservableDeferred 2017-02-13 13:49:44 +00:00
Kegan Dougal
808ddf0ae7 Pop the txn from the map in case it has already been deleted somehow 2017-02-13 13:36:15 +00:00
Kegan Dougal
feb15dc99f Don't cache errors at all 2017-02-13 13:33:12 +00:00
Kegan Dougal
ecd7e36047 http txns: Do not cache error responses 
Previously we did. This meant that, amongst other errors, rate-limiting errors
would be cached and prevent messages with that txn ID being sent.
 2017-02-13 13:16:48 +00:00
Erik Johnston
6bba80241c Merge pull request #1912 from matrix-org/markjh/roominitialsync 
Add db functions needed for room initial sync to slave
 2017-02-13 12:20:21 +01:00
Mark Haines
3a46280ca3 Add db functions needed for room initial sync to slave 2017-02-13 11:16:53 +00:00
Tyler Smith
df4407d665 Fix typo in config comments. 
Signed-off-by: Tyler Smith <tylersmith.me@gmail.com>
 2017-02-11 23:02:57 -08:00
Erik Johnston
1beda9c8a7 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-02-09 10:46:58 +00:00
Erik Johnston
505bfd82bb Update version and changelog 2017-02-09 10:41:02 +00:00
Erik Johnston
52cd019a54 Make None check explicit 2017-02-08 16:04:29 +00:00
Erik Johnston
f20cd34858 Merge pull request #1892 from matrix-org/erikj/rejection_fwd_extrem 
Ignore new rejected events when working out forward extremities.
 2017-02-08 16:59:06 +01:00
Erik Johnston
7723b4caa4 Ignore new rejected events when working out forward extremeties. 2017-02-08 14:48:06 +00:00
David Baker
063a1251a9 Remove a few aspirational but unused constants 
from the Kegan era
 2017-02-08 11:36:08 +00:00
Erik Johnston
af6da6db2d Merge pull request #1784 from morteza-araby/user-admin 
Administration functionalities
 2017-02-06 16:21:10 +01:00
Erik Johnston
38434a7fbb Bump changelog and version 2017-02-04 08:27:51 +00:00
Erik Johnston
84f600b2ee Bump changelog and version 2017-02-02 18:58:33 +00:00
Erik Johnston
aec1708c53 Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.19.0 2017-02-02 18:57:05 +00:00