Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-14 20:44:19 -05:00
Code Issues Packages Projects Releases Wiki Activity
16,800 Commits 4 Branches 609 Tags 60 MiB
5d64fefd6c
Commit Graph

811 Commits

Author SHA1 Message Date
Richard van der Hoff
704c3e6239 Merge branch 'master' into develop 2018-08-02 15:43:30 +01:00
Richard van der Hoff
0a65450d04 Validation for events/rooms in fed requests 
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.

(patch supplied by @turt2live)
 2018-08-02 13:48:40 +01:00
Richard van der Hoff
018d75a148 Refactor code for turning HttpResponseException into SynapseError 
This commit replaces SynapseError.from_http_response_exception with
HttpResponseException.to_synapse_error.

The new method actually returns a ProxiedRequestError, which allows us to pass
through additional metadata from the API call.
 2018-08-01 16:02:46 +01:00
Richard van der Hoff
fa7dc889f1 Be more careful which errors we send back over the C-S API 
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.

That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.

The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
 2018-08-01 16:02:38 +01:00
Richard van der Hoff
c82ccd3027 Factor out exception handling in federation_client 
Factor out the error handling from make_membership_event, send_join, and
send_leave, so that it can be shared.
 2018-08-01 16:01:04 +01:00
Amber Brown
da7785147d
Python 3: Convert some unicode/bytes uses (#3569) 2018-08-02 00:54:06 +10:00
Travis Ralston
37be52ac34
limt -> limit 2018-07-31 16:29:09 -06:00
Travis Ralston
e908b86832 Remove pdu_failures from transactions 
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
 2018-07-30 16:28:47 -06:00
Travis Ralston
7d32f0d745
Update the send_leave path to be an event_id 
It's still not used, however the parameter is an event ID not a transaction ID.
 2018-07-26 14:41:59 -06:00
Richard van der Hoff
ce0c18dec5 Improve logging for exceptions handling PDUs 
when we get an exception handling a federation PDU, log the whole stacktrace.
 2018-07-23 22:13:19 +01:00
Richard van der Hoff
667fba68f3 Run things as background processes 
This fixes #3518, and ensures that we get useful logs and metrics for lots of
things that happen in the background.

(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
 2018-07-18 20:55:05 +01:00
Richard van der Hoff
6e3fc657b4 Resource tracking for background processes 
This introduces a mechanism for tracking resource usage by background
processes, along with an example of how it will be used.

This will help address #3518, but more importantly will give us better insights
into things which are happening but not being shown up by the request metrics.

We *could* do this with Measure blocks, but:
 - I think having them pulled out as a completely separate metric class will
   make it easier to distinguish top-level processes from those which are
   nested.

 - I want to be able to report on in-flight background processes, and I don't
   think we want to do this for *all* Measure blocks.
 2018-07-18 10:50:33 +01:00
Krombel
3366b9c534 rename assert_params_in_request to assert_params_in_dict 
the method "assert_params_in_request" does handle dicts and not
requests. A request body has to be parsed to json before this method
can be used
 2018-07-13 21:53:01 +02:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
3cf3e08a97 Implementation of server_acls 
... as described at
https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw.
 2018-07-04 19:06:20 +01:00
Richard van der Hoff
546bc9e28b More server_name validation 
We need to do a bit more validation when we get a server name, but don't want
to be re-doing it all over the shop, so factor out a separate
parse_and_validate_server_name, and do the extra validation.

Also, use it to verify the server name in the config file.
 2018-07-04 18:59:51 +01:00
Richard van der Hoff
508196e08a
Reject invalid server names (#3480) 
Make sure that server_names used in auth headers are sane, and reject them with
a sensible error code, before they disappear off into the depths of the system.
 2018-07-03 14:36:14 +01:00
Erik Johnston
e3b4043800
Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking 
Check the state of prev_events a bit more thoroughly when coming over federation
 2018-06-29 13:55:02 +01:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Amber Brown
77078d6c8e handle federation not telling us about prev_events 2018-06-27 11:27:32 +01:00
Erik Johnston
b4a5d767a9
Merge pull request #3428 from matrix-org/erikj/persisted_pdu 
Simplify get_persisted_pdu
 2018-06-22 14:47:43 +01:00
Amber Brown
c2eff937ac
Populate synapse_federation_client_sent_pdu_destinations:count again (#3386) 2018-06-21 09:39:58 +01:00
Amber Brown
a61738b316
Remove run_on_reactor (#3395) 2018-06-14 18:27:37 +10:00
Richard van der Hoff
9fc5b74b24 simplify get_persisted_pdu 
it doesn't make much sense to use get_persisted_pdu on the receive path: just
get the event straight from the store.
 2018-06-12 09:51:31 +01:00
Ivan Shapovalov
c88d50aa8f federation/send_queue.py: fix usage of sortedcontainers.SortedDict 2018-06-06 02:45:18 +03:00
Amber Brown
f7869f8f8b
Port to sortedcontainers (with tests!) (#3332) 2018-06-06 00:13:57 +10:00
Ivan Shapovalov
7d9d75e4e8 federation/send_queue.py: fix usage of LaterGauge 
Fixes a startup crash due to commit df9f72d9e5
"replacing portions".
 2018-06-03 14:16:17 +03:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Amber Brown
e987079037 fixes 2018-05-23 13:03:51 -05:00
Amber Brown
53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown
071206304d cleanup pep8 errors 2018-05-22 16:54:22 -05:00
Amber Brown
85ba83eb51 fixes 2018-05-22 16:28:23 -05:00
Amber Brown
df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Richard van der Hoff
23e2dfe940
Merge pull request #3209 from damir-manapov/master 
transaction_id, destination defined twice
 2018-05-11 00:35:13 +01:00
Damir Manapov
db18d854cd transaction_id, destination twice 2018-05-10 22:13:31 +03:00
Richard van der Hoff
ca7211104e Merge branch 'release-v0.28.1' into develop 2018-05-01 18:16:57 +01:00
Richard van der Hoff
33f469ba19 Apply some limits to depth to counter abuse 
* When creating a new event, cap its depth to 2^63 - 1
* When receiving events, reject any without a sensible depth

As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI
 2018-05-01 17:54:19 +01:00
Richard van der Hoff
db75c86e84
Merge branch 'develop' into py3-xrange-1 2018-04-30 01:02:25 +01:00
Adrian Tschira
d82b6ea9e6 Move more xrange to six 
plus a bonus next()

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-28 13:57:00 +02:00
Richard van der Hoff
fc149b4eeb Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background 2018-04-27 14:31:23 +01:00
Richard van der Hoff
2a13af23bc Use run_in_background in preference to preserve_fn 
While I was going through uses of preserve_fn for other PRs, I converted places
which only use the wrapped function once to use run_in_background, to avoid
creating the function object.
 2018-04-27 12:55:51 +01:00
Richard van der Hoff
9255a6cb17 Improve exception handling for background processes 
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.

This is unsatisfactory for a number of reasons:
 - logging on garbage collection is best-effort and may happen some time after
   the error, if at all
 - it can be hard to figure out where the error actually happened.
 - it is logged as a scary CRITICAL error which (a) I always forget to grep for
   and (b) it's not really CRITICAL if a background process we don't care about
   fails.

So this is an attempt to add exception handling to everything we fire off into
the background.
 2018-04-27 11:07:40 +01:00
Richard van der Hoff
77ebef9d43
Merge pull request #3118 from matrix-org/rav/reject_prev_events 
Reject events which have lots of prev_events
 2018-04-23 17:51:38 +01:00
Richard van der Hoff
dc875d2712
Merge pull request #3106 from NotAFile/py3-six-itervalues-1 
Use six.itervalues in some places
 2018-04-20 15:43:52 +01:00
Richard van der Hoff
11a67b7c9d
Merge pull request #3093 from matrix-org/rav/response_cache_wrap 
Refactor ResponseCache usage
 2018-04-20 11:31:17 +01:00
Richard van der Hoff
0c280d4d99 Reinstate linearizer for federation_server.on_context_state_request 2018-04-20 11:10:04 +01:00
Richard van der Hoff
b1dfbc3c40 Refactor store.have_events 
It turns out that most of the time we were calling have_events, we were only
using half of the result. Replace have_events with have_seen_events and
get_rejection_reasons, so that we can see what's going on a bit more clearly.
 2018-04-20 10:25:56 +01:00
Richard van der Hoff
1f4b498b73 Add some comments 2018-04-18 00:15:36 +01:00
Adrian Tschira
36c59ce669 Use six.itervalues in some places 
There's more where that came from

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-04-15 20:39:43 +02:00
Matthew Hodgson
78a9698650 fix federation_domain_whitelist 
we were checking the wrong server_name on inbound requests
 2018-04-13 15:47:43 +01:00
Matthew Hodgson
25b0ba30b1 revert last to PR properly 2018-04-13 15:46:37 +01:00
Matthew Hodgson
f8d46cad3c correctly auth inbound federation_domain_whitelist reqs 2018-04-13 15:41:52 +01:00
Richard van der Hoff
d3347ad485 Revert "Use sortedcontainers instead of blist" 
This reverts commit 9fbe70a7dc.

It turns out that sortedcontainers.SortedDict is not an exact match for
blist.sorteddict; in particular, `popitem()` removes things from the opposite
end of the dict.

This is trivial to fix, but I want to add some unit tests, and potentially some
more thought about it, before we do so.
 2018-04-13 11:16:43 +01:00
Richard van der Hoff
b78395b7fe Refactor ResponseCache usage 
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.

This will be largely non-functional, but does include the following functional
changes:

* federation_server.on_context_state_request: drops use of _server_linearizer
  which looked redundant and could cause incorrect cache misses by yielding
  between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
  on production.
 2018-04-12 13:02:15 +01:00
Richard van der Hoff
d5c74b9f6c
Merge pull request #3092 from matrix-org/rav/response_cache_metrics 
Add metrics for ResponseCache
 2018-04-12 12:59:36 +01:00
Erik Johnston
f67e906e18 Set all metrics at the same time 2018-04-12 11:18:19 +01:00
Erik Johnston
4dae4a97ed Track last processed event received_ts 2018-04-11 14:27:09 +01:00
Erik Johnston
92e34615c5 Track where event stream processing have gotten up to 2018-04-11 12:13:40 +01:00
Richard van der Hoff
233699c42e
Merge pull request #2760 from Valodim/pypy 
Synapse on PyPy
 2018-04-11 11:20:01 +01:00
Richard van der Hoff
b3384232a0 Add metrics for ResponseCache 2018-04-10 23:14:47 +01:00
Erik Johnston
9daf82278f
Merge pull request #3078 from matrix-org/erikj/federation_sender 
Send federation events concurrently
 2018-04-10 16:43:48 +01:00
Erik Johnston
a060dfa132 Use run_in_background instead 2018-04-10 14:25:11 +01:00
Erik Johnston
1246d23710 Preserve log contexts correctly 2018-04-10 12:04:32 +01:00
Erik Johnston
d49cbf712f Log event ID on exception 2018-04-10 12:03:41 +01:00
Erik Johnston
11d2609da7 Ensure slashes are escaped 2018-04-10 11:24:40 +01:00
Erik Johnston
dab87b84a3 URL quote path segments over federation 2018-04-10 11:16:08 +01:00
Vincent Breitmoser
9fbe70a7dc Use sortedcontainers instead of blist 
This commit drop-in replaces blist with SortedContainers. They are
written in pure python so work with pypy, but perform as good as
native implementations, at least in a couple benchmarks:

http://www.grantjenks.com/docs/sortedcontainers/performance.html
 2018-04-10 11:29:51 +02:00
Erik Johnston
6e025a97b4 Handle all events in a room correctly 2018-04-09 16:02:48 +01:00
Erik Johnston
11974f3787 Send federation events concurrently 2018-04-09 11:47:10 +01:00
Erik Johnston
145d14656b Handle exceptions in get_hosts_for_room when sending events over federation 2018-04-09 11:47:01 +01:00
Luke Barnard
112c2253e2 pep8 2018-04-06 15:43:27 +01:00
Luke Barnard
f8d1917fce Fix federation client set_group_joinable typo 2018-04-06 15:43:27 +01:00
David Baker
b370fe61c0 Implement group join API 2018-04-06 15:43:27 +01:00
Krombel
1d71f484d4 use PUT instead of POST for federating groups/m.join_policy 2018-04-06 12:54:09 +02:00
Luke Barnard
104c0bc1d5 Use "/settings/" (plural) 2018-04-05 14:07:16 +01:00
Luke Barnard
eb8d8d6f57 Use join_policy API instead of joinable 
The API is now under
 /groups/$group_id/setting/m.join_policy

and expects a JSON blob of the shape

```json
{
  "m.join_policy": {
    "type": "invite"
  }
}
```

where "invite" could alternatively be "open".
 2018-04-03 16:16:40 +01:00
David Baker
32260baa41 pep8 2018-03-28 14:29:42 +01:00
David Baker
79452edeee Add joinability for groups 
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
 2018-03-28 14:03:37 +01:00
Erik Johnston
95cb401ae0
Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer 
Remove ReplicationLayer and user Client/Server directly
 2018-03-13 15:45:08 +00:00
Erik Johnston
56e709857c
Merge pull request #2979 from matrix-org/erikj/no_handlers 
Don't build handlers on workers unnecessarily
 2018-03-13 13:46:38 +00:00
Erik Johnston
cea462e285 s/replication_server/federation_server 2018-03-13 13:22:21 +00:00
Erik Johnston
9a2d9b4789
Merge pull request #2977 from matrix-org/erikj/replication_move_props 
Move property setting from ReplicationLayer to base classes
 2018-03-13 11:45:25 +00:00
Erik Johnston
f43b6d6d9b Fix docstring types 2018-03-13 11:29:35 +00:00
Erik Johnston
ea7b3c4b1b Remove unused ReplicationLayer 2018-03-13 11:00:04 +00:00
Erik Johnston
265b993b8a Split replication layer into two 2018-03-13 10:55:47 +00:00
Erik Johnston
e05bf34117 Move property setting from ReplicationLayer to FederationBase 2018-03-13 10:51:30 +00:00
Erik Johnston
c3f79c9da5 Split out edu/query registration to a separate class 2018-03-13 10:24:27 +00:00
Matthew Hodgson
ab9f844aaf
Add federation_domain_whitelist option (#2820) 
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
 2018-01-22 19:11:18 +01:00
Richard van der Hoff
a027c2af8d Metrics for events processed in appservice and fed sender 
More metrics I wished I'd had
 2018-01-15 18:23:24 +00:00
Richard van der Hoff
bd91857028 Check missing fields in event_from_pdu_json 
Return a 400 rather than a 500 when somebody messes up their send_join
 2017-12-30 18:40:19 +00:00
Richard van der Hoff
3079f80d4a Factor out event_from_pdu_json 
turns out we have two copies of this, and neither needs to be an instance
method
 2017-12-30 18:40:19 +00:00
Richard van der Hoff
65abc90fb6 federation_server: clean up imports 2017-12-30 18:40:19 +00:00
Richard van der Hoff
a7b726ad18 federation_client: clean up imports 2017-12-30 18:40:19 +00:00
Richard van der Hoff
d4fb4f7c52 Clear logcontext before starting fed txn queue runner 
These processes take a long time compared to the request, so there is lots of
"Entering|Restoring dead context" in the logs. Let's try to shut it up a bit.
 2017-11-28 15:26:14 +00:00
Richard van der Hoff
7e6fa29cb5 Remove preserve_context_over_{fn, deferred} 
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
 2017-11-14 11:22:42 +00:00
Erik Johnston
82e4bfb53d Add brackets 2017-11-09 10:06:42 +00:00
Erik Johnston
e8814410ef Have an explicit API to update room config 2017-11-08 16:13:27 +00:00
Erik Johnston
94ff2cda73
Revert "Modify group room association API to allow modification of is_public" 2017-11-08 15:43:34 +00:00
Luke Barnard
207fabbc6a Update docs for updating room group association 2017-11-01 09:35:15 +00:00
Luke Barnard
13b3d7b4a0 Flake8 2017-10-31 17:20:11 +00:00
Luke Barnard
20fe347906 Modify group room association API to allow modification of is_public 
also includes renamings to make things more consistent.
 2017-10-31 17:04:28 +00:00
Erik Johnston
2a7e9faeec Do logcontexts outside ResponseCache 2017-10-25 15:21:08 +01:00
Erik Johnston
39dc52157d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/group_fed_update_profile 2017-10-24 09:16:20 +01:00
Richard van der Hoff
eaaabc6c4f replace 'except:' with 'except Exception:' 
what could possibly go wrong
 2017-10-23 15:52:32 +01:00
Erik Johnston
ce6d4914f4 Correctly wire in update group profile over federation 2017-10-23 15:21:24 +01:00
Erik Johnston
011d03a0f6 Fix typo 2017-10-19 11:22:48 +01:00
Erik Johnston
9ab859f27b Fix typo in group attestation handling 2017-10-19 10:55:52 +01:00
Richard van der Hoff
582bd19ee9 Fix 500 error when we get an error handling a PDU 
FederationServer doesn't have a send_failure (and nor does its subclass,
ReplicationLayer), so this was failing.

I'm not really sure what the idea behind send_failure is, given (a) we don't do
anything at the other end with it except log it, and (b) we also send back the
failure via the transaction response. I suspect there's a whole lot of dead
code around it, but for now I'm just removing the broken bit.
 2017-10-17 20:52:40 +01:00
Luke Barnard
85f5674e44 Delint 2017-10-16 15:52:17 +01:00
Luke Barnard
2c5972f87f Implement GET /groups/$groupId/invited_users 2017-10-16 15:31:11 +01:00
Richard van der Hoff
8dd0c85ac5 Merge pull request #2529 from matrix-org/rav/fix_transaction_failure_handling 
log pdu_failures from incoming transactions
 2017-10-11 17:29:14 +01:00
Erik Johnston
271f5601f3 Fix typo in invite to group 2017-10-11 14:45:33 +01:00
Richard van der Hoff
b75d443caf log pdu_failures from incoming transactions 
... even if we have no EDUs.

This appears to have been introduced in
476899295f.
 2017-10-11 14:36:13 +01:00
Erik Johnston
535cc49f27 Merge pull request #2466 from matrix-org/erikj/groups_merged 
Initial Group Implementation
 2017-10-11 13:20:07 +01:00
Richard van der Hoff
6a6cc27aee fed server: process PDUs for different rooms in parallel 
With luck, this will give a real-time improvement when there are many rooms and
the server ends up calling out to fetch missing events.
 2017-10-09 18:30:31 +01:00
Richard van der Hoff
4c7c4d4061 Fed server: use a linearizer for ongoing transactions 
We don't want to process the same transaction multiple times concurrently, so
use a linearizer.
 2017-10-09 18:30:10 +01:00
Richard van der Hoff
ba5b9b80a5 fed server: refactor on_incoming_transaction 
Move as much as possible to after the have_responded check, and reduce the
number of times we iterate over the pdu list.
 2017-10-09 18:10:53 +01:00
Richard van der Hoff
a6e3222fe5 Fed server: Move origin-check code to _handle_received_pdu 
The response-building code expects there to be an entry in the `results` list
for each entry in the pdu_list, so the early `continue` was messing this
up. That doesn't really matter, because all that the federation client does is
log any errors, but it's pretty poor form.
 2017-10-09 17:53:32 +01:00
Richard van der Hoff
01bbacf3c4 Fix up logcontext handling in (federation) TransactionQueue 
Avoid using preserve_context_over_function, which has problems with respect to
logcontexts.
 2017-10-06 22:39:25 +01:00
David Baker
27955056e0 Merge branch 'develop' into erikj/groups_merged 2017-10-02 16:20:41 +01:00
David Baker
6cd5fcd536 Make the spam checker a module 2017-09-26 19:20:23 +01:00
Erik Johnston
17b8e2bd02 Add remove room API 2017-09-26 15:52:41 +01:00
Richard van der Hoff
6de74ea6d7 Fix logcontexts in _check_sigs_and_hashes 2017-09-20 01:32:42 +01:00
Richard van der Hoff
fcf2c0fd1a Remove redundant preserve_fn 
preserve_fn is a no-op unless the wrapped function returns a
Deferred. verify_json_objects_for_server returns a list, so this is doing
nothing.
 2017-09-20 01:32:42 +01:00
Richard van der Hoff
5ed109d59f PoC for filtering spammy events (#2456) 
Demonstration of how you might add some hooks to filter out spammy events.
 2017-09-19 12:20:11 +01:00
Erik Johnston
ef8e578677 Add bulk group publicised lookup API 2017-08-09 13:36:22 +01:00
Erik Johnston
6f443a74cf Add update group profile API 2017-07-20 09:46:33 +01:00
Erik Johnston
14a34f12d7 Comments 2017-07-18 17:28:42 +01:00
Erik Johnston
3431ec55dc Comments 2017-07-18 17:23:50 +01:00
Erik Johnston
94ecd871a0 Fix typos 2017-07-18 16:38:54 +01:00
Erik Johnston
12ed4ee48e Correctly parse query params 2017-07-18 15:33:09 +01:00
Erik Johnston
e5ea6dd021 Add client apis 2017-07-18 14:37:06 +01:00
Erik Johnston
cccfcfa7b9 Comments 2017-07-18 10:35:18 +01:00
Erik Johnston
2f9eafdd36 Add local group server support 2017-07-17 12:03:49 +01:00
Erik Johnston
cb3aee8219 Ensure category and role ids are non-null 2017-07-14 14:06:55 +01:00
Erik Johnston
8575e3160f Comments 2017-07-13 13:52:41 +01:00
Erik Johnston
7a39da8cc6 Add summary APIs to federation 2017-07-13 11:13:19 +01:00
Erik Johnston
a62406aaa5 Add group summary APIs 2017-07-12 11:36:15 +01:00
Erik Johnston
fe4e885f54 Add federation API for adding room to group 2017-07-11 14:35:07 +01:00
Erik Johnston
6322fbbd41 Comment 2017-07-11 11:52:03 +01:00
Erik Johnston
b8ca494ee9 Initial group server implementation 2017-07-10 15:44:15 +01:00
Erik Johnston
5e49a57ecc Separate federation servlet into different lists 2017-07-05 14:32:24 +01:00
Erik Johnston
6e2a7ee1bc Remove spurious log lines 2017-06-07 11:05:17 +01:00
Erik Johnston
dfbda5e025 Faster cache for get_joined_hosts 2017-05-25 17:24:44 +01:00
Erik Johnston
ec5c4499f4 Make presence use cached users/hosts in room 2017-05-16 16:01:43 +01:00
Richard van der Hoff
de042b3b88 Do some logging when one-time-keys get claimed 
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
 2017-05-09 19:04:56 +01:00
Erik Johnston
27c28eaa27 Merge pull request #2190 from matrix-org/erikj/mark_remote_as_back_more 
Always mark remotes as up if we receive a signed request from them
 2017-05-05 14:08:12 +01:00
Erik Johnston
310b1ccdc1 Use preserve_fn and add logs 2017-05-05 13:41:19 +01:00
Erik Johnston
db7d0c3127 Always mark remotes as up if we receive a signed request from them 2017-05-05 10:34:53 +01:00
Erik Johnston
7166854f41 Add cache for get_current_hosts_in_room 2017-05-02 10:36:35 +01:00
Richard van der Hoff
91b3981800 Try harder when sending leave events 
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
 2017-04-21 01:50:36 +01:00
Richard van der Hoff
4903ccf159 Fix some lies, and other clarifications, in docstrings 
The documentation on get_json has been wrong ever since the very first commit
to synapse...
 2017-04-21 01:31:09 +01:00
Erik Johnston
247c736b9b Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl 
Reduce federation replication traffic
 2017-04-12 11:07:13 +01:00
Paul Evans
8fbc0d29ee Merge pull request #2121 from matrix-org/paul/sent-transactions-metric 
Add a counter metric for successfully-sent transactions
 2017-04-12 11:04:31 +01:00
Erik Johnston
26ae5178a4 Add some comments 2017-04-12 10:36:29 +01:00
Erik Johnston
1745069543 Comment 2017-04-12 10:17:10 +01:00
Erik Johnston
c7ddb5ef7a Reuse get_interested_parties 2017-04-12 10:16:26 +01:00
Paul "LeoNerd" Evans
11dbceb761 Add a counter metric for successfully-sent transactions 2017-04-11 17:16:12 +01:00
Erik Johnston
a8c8e4efd4 Comment 2017-04-11 15:35:49 +01:00
Erik Johnston
2be8a281d2 Comments 2017-04-11 15:28:24 +01:00
Erik Johnston
6308ac45b0 Move get_interested_remotes back to presence handler 2017-04-11 15:19:26 +01:00
Erik Johnston
b9b72bc6e2 Comments 2017-04-11 15:15:34 +01:00
Erik Johnston
84fbb80c8f Use generators 2017-04-10 16:55:56 +01:00
Erik Johnston
40453b3f84 Dedupe KeyedEdu and Devices federation repl traffic 2017-04-10 16:49:51 +01:00
Erik Johnston
29574fd5b3 Reduce federation presence replication traffic 
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
 2017-04-10 16:48:30 +01:00
Erik Johnston
0018491af2 Rename variable 2017-04-10 12:44:43 +01:00
Erik Johnston
8c5f03cec7 Revert to sending the same data type as before 2017-04-10 10:07:18 +01:00
Erik Johnston
f8434db549 Change name 2017-04-10 10:03:07 +01:00
Erik Johnston
ab904caf33 Comments 2017-04-10 10:02:17 +01:00
Erik Johnston
a828a64b75 Comment 2017-04-07 11:54:03 +01:00
Erik Johnston
d4d176e5d0 Add logging 2017-04-07 11:51:28 +01:00
Erik Johnston
449d1297ca Fix up federation SendQueue and document types 2017-04-07 11:48:33 +01:00
Erik Johnston
f10ce8944b Don't double json encode federation replication data 2017-04-05 11:10:28 +01:00
Erik Johnston
62b89daac6 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server 2017-04-04 09:46:16 +01:00
Richard van der Hoff
30bcbf775a Accept join events from all servers 
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.

(Fixes #1893).
 2017-04-03 15:58:07 +01:00
Erik Johnston
11880103b1 Make federation send queue take the current position 2017-03-30 12:54:36 +01:00
Erik Johnston
85be3dde81 Bail early if remote wouldn't be retried (#2064) 
* Bail early if remote wouldn't be retried

* Don't always return true

* Just use get_retry_limiter

* Spelling
 2017-03-29 11:48:27 +01:00
Erik Johnston
c5b0bdd542 Merge pull request #2067 from matrix-org/erikj/notify_on_fed 
Notify on new federation traffic
 2017-03-29 11:41:37 +01:00
Erik Johnston
35b4aa04be Notify on new federation traffic 2017-03-27 14:07:47 +01:00
Erik Johnston
2a28b79e04 Batch sending of device list pokes 2017-03-24 14:44:49 +00:00
Richard van der Hoff
5a16cb4bf0 Ignore backoff history for invites, aliases, and roomdirs 
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
 2017-03-23 12:23:22 +00:00
Richard van der Hoff
4bd597d9fc push federation retry limiter down to matrixfederationclient 
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
 2017-03-23 09:28:46 +00:00
Richard van der Hoff
29ed09e80a Fix assertion to stop transaction queue getting wedged 
... and update some docstrings to correctly reflect the types being used.

get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
 2017-03-15 12:16:55 +00:00
Richard van der Hoff
29235901b8 Move FederationServer._handle_new_pdu to FederationHandler 
Unfortunately this significantly increases the size of the already-rather-big
FederationHandler, but the code fits more naturally here, and it paves the way
for the tighter integration that I need between handling incoming PDUs and
doing the join dance.

Other than renaming the existing `FederationHandler.on_receive_pdu` to
`_process_received_pdu` to make way for it, this just consists of the move, and
replacing `self.handler` with `self` and `self` with `self.replication_layer`.
 2017-03-09 16:20:13 +00:00
Richard van der Hoff
e8b1721290 Move sig check out of _handle_new_pdu 
When we receive PDUs via `get_missing_events`, we have already checked their
sigs, so there is no need to do it again.
 2017-03-09 15:50:44 +00:00
Richard van der Hoff
3406333a58 Factor _get_missing_events_for_pdu out of _handle_new_pdu 
This should be functionally identical: it just seeks to improve readability by
reducing indentation.
 2017-03-09 15:50:44 +00:00
Richard van der Hoff
45d173a59a Fix docstring 2017-03-09 15:50:29 +00:00
Richard van der Hoff
0c4cf9372b Fix a race in transaction queue 
It was theoretically possible for a PDU to get queued and not sent for ages. On
closer inspection I think there were bigger problems elsewhere, but we might as
well fix this since it's easy.
 2017-02-20 16:46:25 +00:00
Erik Johnston
df4ecff5a9 Correctly raise exceptions for ratelimitng. Ratelimit on 401 2017-02-01 15:42:19 +00:00
Erik Johnston
ae7a132f38 Better handle 404 response for federation /send/ 2017-01-31 13:40:09 +00:00
Erik Johnston
c974116f19 Implement device key caching over federation 2017-01-26 16:07:24 +00:00
Erik Johnston
51e9fe36e4 Fix up sending of m.device_list_update edus 2017-01-25 16:55:21 +00:00
Erik Johnston
2367c5568c Add basic implementation of local device list changes 2017-01-25 14:27:27 +00:00
Erik Johnston
fc7cae8aa3 Merge pull request #1824 from matrix-org/erikj/retry_host_log 
Lower the not retrying host log line to debug
 2017-01-18 11:07:51 +00:00
Erik Johnston
f878f64f43 Lower the not retrying host log line to debug 2017-01-17 17:20:39 +00:00
Erik Johnston
e6153e1bd1 Fix couple of federation state bugs 2017-01-17 13:22:34 +00:00
Erik Johnston
f7085ac84f Name linearizer's for better logs 2017-01-09 17:17:10 +00:00
Matthew
4304e7e593 do the discard check in the right place to avoid grabbing dependent events 2017-01-07 03:44:18 +00:00
Matthew
e10c527930 Discard PDUs from invalid origins due to #1753 in 0.18.[56] 2017-01-07 02:13:14 +00:00
Matthew Hodgson
468749c9fc fix comment 2017-01-05 12:00:11 +00:00
Matthew Hodgson
eedf400d05 limit total timeout for get_missing_events to 10s 2017-01-05 11:58:15 +00:00
Mark Haines
f784980d2b Only send events that originate on this server. 
Or events that are sent via the federation "send_join" API.

This should match the behaviour from before v0.18.5 and #1635 landed.
 2017-01-05 11:26:30 +00:00
Mark Haines
e02bdaf08b Get the destinations from the state from before the event 
Rather than the state after then event.
 2017-01-04 15:17:15 +00:00
Mark Haines
b6b67715ed Send ALL membership events to the server that was affected. 
Send all membership changes to the server that was affected.
This ensures that if the last member of a room on a server
was kicked or banned they get told about it.
 2017-01-04 13:56:20 +00:00
Mark Haines
62ce3034f3 s/aquire/acquire/g 2016-12-30 20:04:44 +00:00
Mark Haines
0aff09f6c9 Add more useful logging when we block fetching events 2016-12-30 20:00:44 +00:00
Mark Haines
cc50b1ae53 Remove fallback from get_missing_events. 
get_missing_events used to fallback to fetching the missing events
individually requesting from every server in the room, one by one.e

This could be unacceptably slow, possibly causing #1732
 2016-12-30 18:13:15 +00:00
Erik Johnston
f32fb65552 Add new API appservice specific public room list 2016-12-06 16:12:27 +00:00
Erik Johnston
aaecffba3a Correctly handle 500's and 429 on federation 2016-11-24 15:04:49 +00:00
Erik Johnston
feec718265 Shuffle receipt handler around so that worker apps don't need to load it 2016-11-23 15:14:24 +00:00
Erik Johnston
4c79a63fd7 Explicit federation ack 2016-11-23 10:40:44 +00:00
Erik Johnston
51e89709aa Comments 2016-11-21 17:59:39 +00:00
Erik Johnston
88d85ebae1 Add some metrics 2016-11-21 17:36:05 +00:00
Erik Johnston
50934ce460 Comments 2016-11-21 16:55:23 +00:00
Erik Johnston
9687e039e7 Remove explicit calls to send_pdu 2016-11-21 14:48:51 +00:00
Erik Johnston
524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston
1587b5a033 Add initial cut of federation send queue 2016-11-16 14:47:52 +00:00
Erik Johnston
59ef517e6b Use new federation_sender DI 2016-11-16 14:47:52 +00:00
Erik Johnston
847d5db1d1 Add transaction queue and transport layer to DI 2016-11-16 14:47:52 +00:00
Erik Johnston
daec6fc355 Move logic into transaction_queue 2016-11-16 14:47:52 +00:00
Erik Johnston
0e830d3770 Rename transaction queue functions to send_* 2016-11-16 14:47:52 +00:00
Erik Johnston
22578545a0 Time out typing over federation 2016-09-23 14:00:52 +01:00
Erik Johnston
23b6701a28 Support filtering remote room lists 2016-09-16 10:24:15 +01:00
Erik Johnston
5810cffd33 Pass since/from parameters over federation 2016-09-15 10:36:19 +01:00
Mark Haines
76b09c29b0 Merge pull request #1110 from matrix-org/markjh/e2e_timeout 
Add a timeout parameter for end2end key queries.
 2016-09-13 10:50:45 +01:00
Mark Haines
949c2c5435 Add a timeout parameter for end2end key queries. 
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.

Fetch keys from different servers in parallel rather than in series.

Set the default timeout to 10s.
 2016-09-12 18:17:09 +01:00
Erik Johnston
706b5d76ed Fix backfill when cannot find an event. 
`get_pdu` can succeed but return None.
 2016-09-12 14:59:51 +01:00
Erik Johnston
af4701b311 Fix incorrect attribute name 2016-09-09 17:36:56 +01:00
Erik Johnston
464ffd1b5e Comment 2016-09-09 17:17:23 +01:00
Erik Johnston
327425764e Add edu.type as part of key. Remove debug logging 2016-09-09 17:13:30 +01:00
Erik Johnston
52b2318777 Clobber EDUs in send queue 2016-09-09 15:59:08 +01:00
Erik Johnston
ab80d5e0a9 Drop replication log levels 2016-09-09 14:56:50 +01:00
Erik Johnston
a6c6750166 Check if destination is ready for retry earlier 2016-09-09 13:46:05 +01:00
Erik Johnston
4598682b43 Fix tightloop on sending transaction 2016-09-09 13:12:53 +01:00
Erik Johnston
d2688d7f03 Correctly guard against multiple concurrent transactions 2016-09-09 11:44:36 +01:00
Erik Johnston
b390756150 Update last_device_stream_id_by_dest if there is nothing to send 2016-09-09 11:00:15 +01:00
Mark Haines
43954d000e Add a new method to enqueue the device messages rather than sending a dummy EDU 2016-09-07 16:10:51 +01:00
Mark Haines
cb98ac261b Move the check for federated device_messages. 
Move the check into _attempt_new_transaction.
Only delete messages if there were messages to delete.
 2016-09-07 15:39:13 +01:00
Mark Haines
31a07d2335 Add stream change caches for device messages 2016-09-07 15:27:07 +01:00
Mark Haines
d4a35ada28 Send device messages over federation 2016-09-06 18:16:20 +01:00
Erik Johnston
2854ee2a52 Only pull out IDs from DB for /state_ids/ request 2016-09-02 10:53:36 +01:00
Erik Johnston
f4778d4cd9 Merge branch 'erikj/pdu_check' of github.com:matrix-org/synapse into develop 2016-09-01 14:40:36 +01:00
Erik Johnston
516a272aca Ensure we only return a validated pdu in get_pdu 2016-09-01 10:55:02 +01:00
Erik Johnston
bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Erik Johnston
9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Erik Johnston
c315922b5f PEP8 2016-08-10 16:34:10 +01:00
Erik Johnston
ca8abfbf30 Clean up TransactionQueue 2016-08-10 16:24:16 +01:00
Erik Johnston
5aeadb7414 Merge pull request #999 from matrix-org/erikj/measure_more 
Measure federation send transaction resources
 2016-08-10 14:16:14 +01:00
Erik Johnston
487bc49bf8 Don't stop on 4xx series errors 2016-08-10 13:39:12 +01:00
Erik Johnston
ea8c4094db Also pull out rejected events 2016-08-10 13:26:13 +01:00
Erik Johnston
f91df1f761 Store if we fail to fetch an event from a destination 2016-08-10 11:31:46 +01:00
Erik Johnston
3bc9629be5 Measure federation send transaction resources 2016-08-10 10:56:38 +01:00
Erik Johnston
f5deaff424 Merge pull request #991 from matrix-org/erikj/retry_make 
Retry joining via other servers if first one failed. Fix some other bugs.
 2016-08-05 18:21:27 +01:00
Erik Johnston
5f360182c6 Fix a couple of python bugs 2016-08-05 18:08:32 +01:00
Erik Johnston
46453bfc2f Retry joining via other servers if first one failed 2016-08-05 18:02:03 +01:00
Erik Johnston
24f36469bc Add federation /version API 2016-08-05 16:36:07 +01:00
Erik Johnston
597c79be10 Change the way we specify if we require auth or not 2016-08-05 16:17:04 +01:00
Erik Johnston
2d4de61fb7 Fix typo 2016-08-05 10:48:56 +01:00
Erik Johnston
fccadb7719 Check if we already have the events returned by /state/ 2016-08-05 10:43:47 +01:00
Erik Johnston
1515d1b581 Fallback to /state/ on both 400 and 404 2016-08-05 10:24:23 +01:00
Benjamin Saunders
a2b7102eea Tweak integrity error recovery to work as intended 2016-08-04 20:38:08 -07:00
Erik Johnston
257c41cc2e Fix typos. 2016-08-04 14:05:45 +01:00
Erik Johnston
b4e2290d89 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids_api 2016-08-04 14:04:35 +01:00
Erik Johnston
e3ee63578f Tidy up get_events 2016-08-04 14:01:18 +01:00
Erik Johnston
edb33eb163 Rename fields to _ids 2016-08-03 17:19:15 +01:00
Erik Johnston
bcc9cda8ca Fix copy + paste fails 2016-08-03 17:17:26 +01:00
Erik Johnston
4c56bedee3 Actually call get_room_state 2016-08-03 15:04:29 +01:00
Erik Johnston
520ee9bd2c Fix syntax error 2016-08-03 15:03:15 +01:00
Erik Johnston
a60a2eaa02 Comment 2016-08-03 14:52:43 +01:00
Erik Johnston
e3a720217a Add /state_ids federation API 
The new API only returns the event_ids for the state, as most
requesters will already have the vast majority of the events already.
 2016-08-03 14:47:37 +01:00
Mark Haines
921f17f938 Merge branch 'develop' into rav/refactor_device_query 2016-08-03 11:12:47 +01:00
Richard van der Hoff
1efee2f52b E2E keys: Make federation query share code with client query 
Refactor the e2e query handler to separate out the local query, and then make
the federation handler use it.
 2016-08-02 18:12:00 +01:00
Erik Johnston
c9154b970c Don't double wrap 200 2016-08-02 16:45:53 +01:00
Erik Johnston
b3d5c4ad9d Fix response cache 2016-08-02 16:42:21 +01:00
Erik Johnston
248e6770ca Cache federation state responses 2016-07-21 10:30:12 +01:00
Erik Johnston
8f4a9bbc16 Linearize some federation endpoints based on (origin, room_id) 2016-06-17 16:43:45 +01:00
Erik Johnston
9ba2bf1570 Merge pull request #878 from matrix-org/erikj/ujson 
Disable responding with canonical json for federation
 2016-06-17 16:22:12 +01:00
Erik Johnston
120c238705 Disable responding with canonical json for federation 2016-06-17 16:10:37 +01:00
Erik Johnston
2884712ca7 Only re-sign our own events 2016-06-17 14:47:33 +01:00
Erik Johnston
d41a1a91d3 Linearize fetching of gaps on incoming events 
This potentially stops the server from doing multiple requests for the
same data.
 2016-06-15 15:16:14 +01:00
Erik Johnston
d88faf92d1 Fix up federation PublicRoomList 2016-06-08 14:39:31 +01:00
Erik Johnston
958c968d02 Merge pull request #856 from matrix-org/erikj/fed_pub_rooms 
Enable auth on /publicRoom endpoints
 2016-06-08 14:36:09 +01:00
Erik Johnston
1fd6eb695d Enable auth on federation PublicRoomList 2016-06-08 14:15:18 +01:00
Erik Johnston
17aab5827a Add some logging for when servers ask for missing events 2016-06-08 11:55:31 +01:00
David Baker
6ecb2ca4ec pep8 2016-06-01 09:48:55 +01:00
David Baker
963e3ed282 Apparently I am not permitted to have two blank lines here 2016-05-31 17:22:53 +01:00
David Baker
d240796ded Basic, un-cached support for secondary_directory_servers 2016-05-31 17:20:07 +01:00
David Baker
70ecb415f5 Fix c+p fail 2016-05-31 12:00:54 +01:00
David Baker
e1625d62a8 Add federation room list servlet 2016-05-31 11:55:57 +01:00
Erik Johnston
f6ebaf4a32 Run transaction queue on reactor 
This ensures that any CPU work that happens doesn't block message
sending.
 2016-05-09 10:10:06 +01:00
Mark Haines
9c272da05f Add an openidish mechanism for proving to third parties that you own a given user_id 2016-05-05 13:42:44 +01:00
Erik Johnston
62b51b8452 Fix typo in event_auth servlet path 2016-04-29 12:00:51 +01:00
Erik Johnston
914f1eafac Lower timeout for make_membership_event 
Calls to make_membership_event are done in response to client requests,
and so should not be retried over long timeframes.
 2016-04-15 11:22:23 +01:00
Erik Johnston
acdfef7b14 Intern all the things 2016-03-23 16:25:54 +00:00
Mark Haines
5244c0b48e Remove unused backfilled parameter from persist_event 2016-03-21 18:06:08 +00:00
Erik Johnston
9adf0e92bc Catch exceptions from EDU handling 2016-03-18 15:12:50 +00:00
Erik Johnston
3c5f25507b Yield on EDU handling 2016-03-18 13:55:16 +00:00