Richard van der Hoff
fc0f13dd03
Fix incorrect truncation in get_missing_events
...
It's quite important that get_missing_events returns the *latest* events in the
room; however we were pulling event ids out of the database until we got *at
least* 10, and then taking the *earliest* of the results.
We also shouldn't really be relying on depth, and should be checking the
room_id.
2018-10-16 21:10:04 +01:00
Richard van der Hoff
4a15a3e4d5
Include eventid in log lines when processing incoming federation transactions ( #3959 )
...
when processing incoming transactions, it can be hard to see what's going on,
because we process a bunch of stuff in parallel, and because we may end up
recursively working our way through a chain of three or four events.
This commit creates a way to use logcontexts to add the relevant event ids to
the log lines.
2018-09-27 11:25:34 +01:00
Richard van der Hoff
703de4ec13
Comments and interface cleanup for on_receive_pdu
...
Add some informative comments about what's going on here.
Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart
from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of
`get_missing` and use `sent_to_us_directly` consistently.
2018-09-20 13:06:55 +01:00
Richard van der Hoff
4f8baab0c4
Merge branch 'master' into develop
2018-09-06 13:05:22 +01:00
Amber Brown
2d2828dcbc
Port http/ to Python 3 ( #3771 )
2018-09-06 00:10:47 +10:00
Richard van der Hoff
c127c8d042
Fix origin handling for pushed transactions
...
Use the actual origin for push transactions, rather than whatever the remote
server claimed.
2018-09-05 13:08:07 +01:00
Erik Johnston
ef184caf30
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-15 14:25:46 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
Erik Johnston
72d1902bbe
Fixup doc comments
2018-08-09 10:23:49 +01:00
Erik Johnston
5785b93711
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation
2018-08-09 10:16:16 +01:00
Erik Johnston
a3f5bf79a0
Add EDU/query handling over replication
2018-08-06 15:23:31 +01:00
Richard van der Hoff
0d63d93ca8
Enforce compatibility when processing make_join requests
...
Reject make_join requests from servers which do not support the room version.
Also include the room version in the response.
2018-08-03 16:08:32 +01:00
Richard van der Hoff
704c3e6239
Merge branch 'master' into develop
2018-08-02 15:43:30 +01:00
Richard van der Hoff
0a65450d04
Validation for events/rooms in fed requests
...
When we get a federation request which refers to an event id, make sure that
said event is in the room the caller claims it is in.
(patch supplied by @turt2live)
2018-08-02 13:48:40 +01:00
Travis Ralston
e908b86832
Remove pdu_failures from transactions
...
The field is never read from, and all the opportunities given to populate it are not utilized. It should be very safe to remove this.
2018-07-30 16:28:47 -06:00
Richard van der Hoff
ce0c18dec5
Improve logging for exceptions handling PDUs
...
when we get an exception handling a federation PDU, log the whole stacktrace.
2018-07-23 22:13:19 +01:00
Amber Brown
49af402019
run isort
2018-07-09 16:09:20 +10:00
Richard van der Hoff
3cf3e08a97
Implementation of server_acls
...
... as described at
https://docs.google.com/document/d/1EttUVzjc2DWe2ciw4XPtNpUpIl9lWXGEsy2ewDS7rtw .
2018-07-04 19:06:20 +01:00
Erik Johnston
e3b4043800
Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking
...
Check the state of prev_events a bit more thoroughly when coming over federation
2018-06-29 13:55:02 +01:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy ( #3462 )
2018-06-28 14:49:57 +01:00
Amber Brown
77078d6c8e
handle federation not telling us about prev_events
2018-06-27 11:27:32 +01:00
Richard van der Hoff
9fc5b74b24
simplify get_persisted_pdu
...
it doesn't make much sense to use get_persisted_pdu on the receive path: just
get the event straight from the store.
2018-06-12 09:51:31 +01:00
Amber Brown
071206304d
cleanup pep8 errors
2018-05-22 16:54:22 -05:00
Amber Brown
df9f72d9e5
replacing portions
2018-05-21 19:47:37 -05:00
Richard van der Hoff
77ebef9d43
Merge pull request #3118 from matrix-org/rav/reject_prev_events
...
Reject events which have lots of prev_events
2018-04-23 17:51:38 +01:00
Richard van der Hoff
dc875d2712
Merge pull request #3106 from NotAFile/py3-six-itervalues-1
...
Use six.itervalues in some places
2018-04-20 15:43:52 +01:00
Richard van der Hoff
0c280d4d99
Reinstate linearizer for federation_server.on_context_state_request
2018-04-20 11:10:04 +01:00
Richard van der Hoff
1f4b498b73
Add some comments
2018-04-18 00:15:36 +01:00
Adrian Tschira
36c59ce669
Use six.itervalues in some places
...
There's more where that came from
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-04-15 20:39:43 +02:00
Richard van der Hoff
b78395b7fe
Refactor ResponseCache usage
...
Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a
(get, set) pair, and then use it throughout the codebase.
This will be largely non-functional, but does include the following functional
changes:
* federation_server.on_context_state_request: drops use of _server_linearizer
which looked redundant and could cause incorrect cache misses by yielding
between the get and the set.
* RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks
* the wrap function includes some logging. I'm hoping this won't be too noisy
on production.
2018-04-12 13:02:15 +01:00
Richard van der Hoff
b3384232a0
Add metrics for ResponseCache
2018-04-10 23:14:47 +01:00
Erik Johnston
56e709857c
Merge pull request #2979 from matrix-org/erikj/no_handlers
...
Don't build handlers on workers unnecessarily
2018-03-13 13:46:38 +00:00
Erik Johnston
9a2d9b4789
Merge pull request #2977 from matrix-org/erikj/replication_move_props
...
Move property setting from ReplicationLayer to base classes
2018-03-13 11:45:25 +00:00
Erik Johnston
f43b6d6d9b
Fix docstring types
2018-03-13 11:29:35 +00:00
Erik Johnston
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
Erik Johnston
e05bf34117
Move property setting from ReplicationLayer to FederationBase
2018-03-13 10:51:30 +00:00
Erik Johnston
c3f79c9da5
Split out edu/query registration to a separate class
2018-03-13 10:24:27 +00:00
Richard van der Hoff
3079f80d4a
Factor out event_from_pdu_json
...
turns out we have two copies of this, and neither needs to be an instance
method
2017-12-30 18:40:19 +00:00
Richard van der Hoff
65abc90fb6
federation_server: clean up imports
2017-12-30 18:40:19 +00:00
Erik Johnston
2a7e9faeec
Do logcontexts outside ResponseCache
2017-10-25 15:21:08 +01:00
Richard van der Hoff
582bd19ee9
Fix 500 error when we get an error handling a PDU
...
FederationServer doesn't have a send_failure (and nor does its subclass,
ReplicationLayer), so this was failing.
I'm not really sure what the idea behind send_failure is, given (a) we don't do
anything at the other end with it except log it, and (b) we also send back the
failure via the transaction response. I suspect there's a whole lot of dead
code around it, but for now I'm just removing the broken bit.
2017-10-17 20:52:40 +01:00
Richard van der Hoff
8dd0c85ac5
Merge pull request #2529 from matrix-org/rav/fix_transaction_failure_handling
...
log pdu_failures from incoming transactions
2017-10-11 17:29:14 +01:00
Richard van der Hoff
b75d443caf
log pdu_failures from incoming transactions
...
... even if we have no EDUs.
This appears to have been introduced in
476899295f
2017-10-11 14:36:13 +01:00
Richard van der Hoff
6a6cc27aee
fed server: process PDUs for different rooms in parallel
...
With luck, this will give a real-time improvement when there are many rooms and
the server ends up calling out to fetch missing events.
2017-10-09 18:30:31 +01:00
Richard van der Hoff
4c7c4d4061
Fed server: use a linearizer for ongoing transactions
...
We don't want to process the same transaction multiple times concurrently, so
use a linearizer.
2017-10-09 18:30:10 +01:00
Richard van der Hoff
ba5b9b80a5
fed server: refactor on_incoming_transaction
...
Move as much as possible to after the have_responded check, and reduce the
number of times we iterate over the pdu list.
2017-10-09 18:10:53 +01:00
Richard van der Hoff
a6e3222fe5
Fed server: Move origin-check code to _handle_received_pdu
...
The response-building code expects there to be an entry in the `results` list
for each entry in the pdu_list, so the early `continue` was messing this
up. That doesn't really matter, because all that the federation client does is
log any errors, but it's pretty poor form.
2017-10-09 17:53:32 +01:00
Richard van der Hoff
de042b3b88
Do some logging when one-time-keys get claimed
...
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
2017-05-09 19:04:56 +01:00
Richard van der Hoff
30bcbf775a
Accept join events from all servers
...
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.
(Fixes #1893 ).
2017-04-03 15:58:07 +01:00
Richard van der Hoff
29235901b8
Move FederationServer._handle_new_pdu to FederationHandler
...
Unfortunately this significantly increases the size of the already-rather-big
FederationHandler, but the code fits more naturally here, and it paves the way
for the tighter integration that I need between handling incoming PDUs and
doing the join dance.
Other than renaming the existing `FederationHandler.on_receive_pdu` to
`_process_received_pdu` to make way for it, this just consists of the move, and
replacing `self.handler` with `self` and `self` with `self.replication_layer`.
2017-03-09 16:20:13 +00:00
Richard van der Hoff
e8b1721290
Move sig check out of _handle_new_pdu
...
When we receive PDUs via `get_missing_events`, we have already checked their
sigs, so there is no need to do it again.
2017-03-09 15:50:44 +00:00
Richard van der Hoff
3406333a58
Factor _get_missing_events_for_pdu out of _handle_new_pdu
...
This should be functionally identical: it just seeks to improve readability by
reducing indentation.
2017-03-09 15:50:44 +00:00
Erik Johnston
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
Erik Johnston
f7085ac84f
Name linearizer's for better logs
2017-01-09 17:17:10 +00:00
Matthew
4304e7e593
do the discard check in the right place to avoid grabbing dependent events
2017-01-07 03:44:18 +00:00
Matthew
e10c527930
Discard PDUs from invalid origins due to #1753 in 0.18.[56]
2017-01-07 02:13:14 +00:00
Matthew Hodgson
468749c9fc
fix comment
2017-01-05 12:00:11 +00:00
Matthew Hodgson
eedf400d05
limit total timeout for get_missing_events to 10s
2017-01-05 11:58:15 +00:00
Mark Haines
62ce3034f3
s/aquire/acquire/g
2016-12-30 20:04:44 +00:00
Mark Haines
0aff09f6c9
Add more useful logging when we block fetching events
2016-12-30 20:00:44 +00:00
Mark Haines
d4a35ada28
Send device messages over federation
2016-09-06 18:16:20 +01:00
Erik Johnston
2854ee2a52
Only pull out IDs from DB for /state_ids/ request
2016-09-02 10:53:36 +01:00
Erik Johnston
b4e2290d89
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids_api
2016-08-04 14:04:35 +01:00
Erik Johnston
edb33eb163
Rename fields to _ids
2016-08-03 17:19:15 +01:00
Erik Johnston
bcc9cda8ca
Fix copy + paste fails
2016-08-03 17:17:26 +01:00
Erik Johnston
e3a720217a
Add /state_ids federation API
...
The new API only returns the event_ids for the state, as most
requesters will already have the vast majority of the events already.
2016-08-03 14:47:37 +01:00
Mark Haines
921f17f938
Merge branch 'develop' into rav/refactor_device_query
2016-08-03 11:12:47 +01:00
Richard van der Hoff
1efee2f52b
E2E keys: Make federation query share code with client query
...
Refactor the e2e query handler to separate out the local query, and then make
the federation handler use it.
2016-08-02 18:12:00 +01:00
Erik Johnston
c9154b970c
Don't double wrap 200
2016-08-02 16:45:53 +01:00
Erik Johnston
b3d5c4ad9d
Fix response cache
2016-08-02 16:42:21 +01:00
Erik Johnston
248e6770ca
Cache federation state responses
2016-07-21 10:30:12 +01:00
Erik Johnston
8f4a9bbc16
Linearize some federation endpoints based on (origin, room_id)
2016-06-17 16:43:45 +01:00
Erik Johnston
2884712ca7
Only re-sign our own events
2016-06-17 14:47:33 +01:00
Erik Johnston
d41a1a91d3
Linearize fetching of gaps on incoming events
...
This potentially stops the server from doing multiple requests for the
same data.
2016-06-15 15:16:14 +01:00
Erik Johnston
17aab5827a
Add some logging for when servers ask for missing events
2016-06-08 11:55:31 +01:00
Mark Haines
9c272da05f
Add an openidish mechanism for proving to third parties that you own a given user_id
2016-05-05 13:42:44 +01:00
Mark Haines
5244c0b48e
Remove unused backfilled parameter from persist_event
2016-03-21 18:06:08 +00:00
Erik Johnston
9adf0e92bc
Catch exceptions from EDU handling
2016-03-18 15:12:50 +00:00
Erik Johnston
3c5f25507b
Yield on EDU handling
2016-03-18 13:55:16 +00:00
Daniel Wagner-Hall
577951b032
Allow third_party_signed to be specified on /join
2016-02-23 15:11:25 +00:00
Erik Johnston
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
Matthew Hodgson
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
2cebe53545
Exchange 3pid invites for m.room.member invites
2015-11-05 16:43:19 +00:00
Daniel Wagner-Hall
137fafce4e
Allow rejecting invites
...
This is done by using the same /leave flow as you would use if you had
already accepted the invite and wanted to leave.
2015-10-20 11:58:58 +01:00
Daniel Wagner-Hall
0c38e8637f
Remove unnecessary class-wrapping
2015-10-13 18:00:38 +01:00
Daniel Wagner-Hall
5b3e9713dd
Implement third party identifier invites
2015-10-01 17:49:52 +01:00
Mark Haines
0cceb2ac92
Add a few strategic new lines to break up the on_query_client_keys and on_claim_client_keys methods in federation_server.py
2015-08-13 17:27:46 +01:00
Mark Haines
62c010283d
Add federation support for end-to-end key requests
2015-07-23 16:03:38 +01:00
Erik Johnston
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
Erik Johnston
23c639ff32
Split a storage function in two so that we don't have to do extra work.
2015-05-01 10:17:19 +01:00
Paul "LeoNerd" Evans
2e4f0b2bd7
Replace the @metrics.counted annotations in federation with specifically-written counters and distributions
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
9470412316
Initial attempt at sprinkling some @metrics.counted decorations around the federation code
2015-03-12 16:24:50 +00:00
Erik Johnston
9708f49abf
Docs
2015-03-05 16:35:16 +00:00
Erik Johnston
ae702d161a
Handle if get_missing_pdu returns 400 or not all events.
2015-03-05 16:08:02 +00:00
Erik Johnston
0a036944bd
Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu
2015-03-02 13:53:30 +00:00
Erik Johnston
29481690c5
If we're yielding don't add errback
2015-03-02 11:50:43 +00:00
Erik Johnston
23d9bd1d74
Process transactions serially.
...
Since the events received in a transaction are ordered, later events
might depend on earlier events and so we shouldn't blindly process them
in parellel.
2015-03-02 11:39:57 +00:00
Erik Johnston
db215b7e00
Implement and use new batched get missing pdu
2015-02-23 13:58:02 +00:00
Erik Johnston
0ac2a79faa
Initial stab at implementing a batched get_missing_pdus request
2015-02-19 17:24:14 +00:00
Erik Johnston
676e8ee78a
Remove debug raise
2015-02-17 15:22:45 +00:00
