146 Commits

Author	SHA1	Message	Date
Richard van der Hoff	9fc5b74b24	simplify get_persisted_pdu ... it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.	2018-06-12 09:51:31 +01:00
Amber Brown	071206304d	cleanup pep8 errors	2018-05-22 16:54:22 -05:00
Amber Brown	df9f72d9e5	replacing portions	2018-05-21 19:47:37 -05:00
Richard van der Hoff	77ebef9d43	Merge pull request #3118 from matrix-org/rav/reject_prev_events ... Reject events which have lots of prev_events	2018-04-23 17:51:38 +01:00
Richard van der Hoff	dc875d2712	Merge pull request #3106 from NotAFile/py3-six-itervalues-1 ... Use six.itervalues in some places	2018-04-20 15:43:52 +01:00
Richard van der Hoff	0c280d4d99	Reinstate linearizer for federation_server.on_context_state_request	2018-04-20 11:10:04 +01:00
Richard van der Hoff	1f4b498b73	Add some comments	2018-04-18 00:15:36 +01:00
Adrian Tschira	36c59ce669	Use six.itervalues in some places ... There's more where that came from Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:39:43 +02:00
Richard van der Hoff	b78395b7fe	Refactor ResponseCache usage ... Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.	2018-04-12 13:02:15 +01:00
Richard van der Hoff	b3384232a0	Add metrics for ResponseCache	2018-04-10 23:14:47 +01:00
Erik Johnston	56e709857c	Merge pull request #2979 from matrix-org/erikj/no_handlers ... Don't build handlers on workers unnecessarily	2018-03-13 13:46:38 +00:00
Erik Johnston	9a2d9b4789	Merge pull request #2977 from matrix-org/erikj/replication_move_props ... Move property setting from ReplicationLayer to base classes	2018-03-13 11:45:25 +00:00
Erik Johnston	f43b6d6d9b	Fix docstring types	2018-03-13 11:29:35 +00:00
Erik Johnston	265b993b8a	Split replication layer into two	2018-03-13 10:55:47 +00:00
Erik Johnston	e05bf34117	Move property setting from ReplicationLayer to FederationBase	2018-03-13 10:51:30 +00:00
Erik Johnston	c3f79c9da5	Split out edu/query registration to a separate class	2018-03-13 10:24:27 +00:00
Richard van der Hoff	3079f80d4a	Factor out event_from_pdu_json ... turns out we have two copies of this, and neither needs to be an instance method	2017-12-30 18:40:19 +00:00
Richard van der Hoff	65abc90fb6	federation_server: clean up imports	2017-12-30 18:40:19 +00:00
Erik Johnston	2a7e9faeec	Do logcontexts outside ResponseCache	2017-10-25 15:21:08 +01:00
Richard van der Hoff	582bd19ee9	Fix 500 error when we get an error handling a PDU ... FederationServer doesn't have a send_failure (and nor does its subclass, ReplicationLayer), so this was failing. I'm not really sure what the idea behind send_failure is, given (a) we don't do anything at the other end with it except log it, and (b) we also send back the failure via the transaction response. I suspect there's a whole lot of dead code around it, but for now I'm just removing the broken bit.	2017-10-17 20:52:40 +01:00
Richard van der Hoff	8dd0c85ac5	Merge pull request #2529 from matrix-org/rav/fix_transaction_failure_handling ... log pdu_failures from incoming transactions	2017-10-11 17:29:14 +01:00
Richard van der Hoff	b75d443caf	log pdu_failures from incoming transactions ... ... even if we have no EDUs. This appears to have been introduced in 476899295f.	2017-10-11 14:36:13 +01:00
Richard van der Hoff	6a6cc27aee	fed server: process PDUs for different rooms in parallel ... With luck, this will give a real-time improvement when there are many rooms and the server ends up calling out to fetch missing events.	2017-10-09 18:30:31 +01:00
Richard van der Hoff	4c7c4d4061	Fed server: use a linearizer for ongoing transactions ... We don't want to process the same transaction multiple times concurrently, so use a linearizer.	2017-10-09 18:30:10 +01:00
Richard van der Hoff	ba5b9b80a5	fed server: refactor on_incoming_transaction ... Move as much as possible to after the have_responded check, and reduce the number of times we iterate over the pdu list.	2017-10-09 18:10:53 +01:00
Richard van der Hoff	a6e3222fe5	Fed server: Move origin-check code to _handle_received_pdu ... The response-building code expects there to be an entry in the `results` list for each entry in the pdu_list, so the early `continue` was messing this up. That doesn't really matter, because all that the federation client does is log any errors, but it's pretty poor form.	2017-10-09 17:53:32 +01:00
Richard van der Hoff	de042b3b88	Do some logging when one-time-keys get claimed ... might help us figure out if https://github.com/vector-im/riot-web/issues/3868 has happened.	2017-05-09 19:04:56 +01:00
Richard van der Hoff	30bcbf775a	Accept join events from all servers ... Make sure that we accept join events from any server, rather than just the origin server, to make the federation join dance work correctly. (Fixes #1893).	2017-04-03 15:58:07 +01:00
Richard van der Hoff	29235901b8	Move FederationServer._handle_new_pdu to FederationHandler ... Unfortunately this significantly increases the size of the already-rather-big FederationHandler, but the code fits more naturally here, and it paves the way for the tighter integration that I need between handling incoming PDUs and doing the join dance. Other than renaming the existing `FederationHandler.on_receive_pdu` to `_process_received_pdu` to make way for it, this just consists of the move, and replacing `self.handler` with `self` and `self` with `self.replication_layer`.	2017-03-09 16:20:13 +00:00
Richard van der Hoff	e8b1721290	Move sig check out of _handle_new_pdu ... When we receive PDUs via `get_missing_events`, we have already checked their sigs, so there is no need to do it again.	2017-03-09 15:50:44 +00:00
Richard van der Hoff	3406333a58	Factor _get_missing_events_for_pdu out of _handle_new_pdu ... This should be functionally identical: it just seeks to improve readability by reducing indentation.	2017-03-09 15:50:44 +00:00
Erik Johnston	c974116f19	Implement device key caching over federation	2017-01-26 16:07:24 +00:00
Erik Johnston	f7085ac84f	Name linearizer's for better logs	2017-01-09 17:17:10 +00:00
Matthew	4304e7e593	do the discard check in the right place to avoid grabbing dependent events	2017-01-07 03:44:18 +00:00
Matthew	e10c527930	Discard PDUs from invalid origins due to #1753 in 0.18.[56]	2017-01-07 02:13:14 +00:00
Matthew Hodgson	468749c9fc	fix comment	2017-01-05 12:00:11 +00:00
Matthew Hodgson	eedf400d05	limit total timeout for get_missing_events to 10s	2017-01-05 11:58:15 +00:00
Mark Haines	62ce3034f3	s/aquire/acquire/g	2016-12-30 20:04:44 +00:00
Mark Haines	0aff09f6c9	Add more useful logging when we block fetching events	2016-12-30 20:00:44 +00:00
Mark Haines	d4a35ada28	Send device messages over federation	2016-09-06 18:16:20 +01:00
Erik Johnston	2854ee2a52	Only pull out IDs from DB for /state_ids/ request	2016-09-02 10:53:36 +01:00
Erik Johnston	b4e2290d89	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids_api	2016-08-04 14:04:35 +01:00
Erik Johnston	edb33eb163	Rename fields to _ids	2016-08-03 17:19:15 +01:00
Erik Johnston	bcc9cda8ca	Fix copy + paste fails	2016-08-03 17:17:26 +01:00
Erik Johnston	e3a720217a	Add /state_ids federation API ... The new API only returns the event_ids for the state, as most requesters will already have the vast majority of the events already.	2016-08-03 14:47:37 +01:00
Mark Haines	921f17f938	Merge branch 'develop' into rav/refactor_device_query	2016-08-03 11:12:47 +01:00
Richard van der Hoff	1efee2f52b	E2E keys: Make federation query share code with client query ... Refactor the e2e query handler to separate out the local query, and then make the federation handler use it.	2016-08-02 18:12:00 +01:00
Erik Johnston	c9154b970c	Don't double wrap 200	2016-08-02 16:45:53 +01:00
Erik Johnston	b3d5c4ad9d	Fix response cache	2016-08-02 16:42:21 +01:00
Erik Johnston	248e6770ca	Cache federation state responses	2016-07-21 10:30:12 +01:00
Erik Johnston	8f4a9bbc16	Linearize some federation endpoints based on (origin, room_id)	2016-06-17 16:43:45 +01:00
Erik Johnston	2884712ca7	Only re-sign our own events	2016-06-17 14:47:33 +01:00
Erik Johnston	d41a1a91d3	Linearize fetching of gaps on incoming events ... This potentially stops the server from doing multiple requests for the same data.	2016-06-15 15:16:14 +01:00
Erik Johnston	17aab5827a	Add some logging for when servers ask for missing events	2016-06-08 11:55:31 +01:00
Mark Haines	9c272da05f	Add an openidish mechanism for proving to third parties that you own a given user_id	2016-05-05 13:42:44 +01:00
Mark Haines	5244c0b48e	Remove unused backfilled parameter from persist_event	2016-03-21 18:06:08 +00:00
Erik Johnston	9adf0e92bc	Catch exceptions from EDU handling	2016-03-18 15:12:50 +00:00
Erik Johnston	3c5f25507b	Yield on EDU handling	2016-03-18 13:55:16 +00:00
Daniel Wagner-Hall	577951b032	Allow third_party_signed to be specified on /join	2016-02-23 15:11:25 +00:00
Erik Johnston	2c1fbea531	Fix up logcontexts	2016-02-08 14:26:45 +00:00
Matthew Hodgson	6c28ac260c	copyrights	2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall	2cebe53545	Exchange 3pid invites for m.room.member invites	2015-11-05 16:43:19 +00:00
Daniel Wagner-Hall	137fafce4e	Allow rejecting invites ... This is done by using the same /leave flow as you would use if you had already accepted the invite and wanted to leave.	2015-10-20 11:58:58 +01:00
Daniel Wagner-Hall	0c38e8637f	Remove unnecessary class-wrapping	2015-10-13 18:00:38 +01:00
Daniel Wagner-Hall	5b3e9713dd	Implement third party identifier invites	2015-10-01 17:49:52 +01:00
Mark Haines	0cceb2ac92	Add a few strategic new lines to break up the on_query_client_keys and on_claim_client_keys methods in federation_server.py	2015-08-13 17:27:46 +01:00
Mark Haines	62c010283d	Add federation support for end-to-end key requests	2015-07-23 16:03:38 +01:00
Erik Johnston	476899295f	Change the way we do logging contexts so that they survive divergences	2015-05-08 16:32:18 +01:00
Erik Johnston	23c639ff32	Split a storage function in two so that we don't have to do extra work.	2015-05-01 10:17:19 +01:00
Paul "LeoNerd" Evans	2e4f0b2bd7	Replace the @metrics.counted annotations in federation with specifically-written counters and distributions	2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans	9470412316	Initial attempt at sprinkling some @metrics.counted decorations around the federation code	2015-03-12 16:24:50 +00:00
Erik Johnston	9708f49abf	Docs	2015-03-05 16:35:16 +00:00
Erik Johnston	ae702d161a	Handle if get_missing_pdu returns 400 or not all events.	2015-03-05 16:08:02 +00:00
Erik Johnston	0a036944bd	Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu	2015-03-02 13:53:30 +00:00
Erik Johnston	29481690c5	If we're yielding don't add errback	2015-03-02 11:50:43 +00:00
Erik Johnston	23d9bd1d74	Process transactions serially. ... Since the events received in a transaction are ordered, later events might depend on earlier events and so we shouldn't blindly process them in parellel.	2015-03-02 11:39:57 +00:00
Erik Johnston	db215b7e00	Implement and use new batched get missing pdu	2015-02-23 13:58:02 +00:00
Erik Johnston	0ac2a79faa	Initial stab at implementing a batched get_missing_pdus request	2015-02-19 17:24:14 +00:00
Erik Johnston	676e8ee78a	Remove debug raise	2015-02-17 15:22:45 +00:00
Erik Johnston	c02da58a9d	Merge branch 'develop' of github.com:matrix-org/synapse into failures	2015-02-17 15:15:07 +00:00
Erik Johnston	659ead082f	Format the response of transaction request in a nicer way	2015-02-17 15:11:44 +00:00
Erik Johnston	c82e26ad4b	Actually respond with JSON to incoming transaction	2015-02-17 13:24:13 +00:00
Erik Johnston	02bfa889de	Handle recieving failures in transactions	2015-02-17 13:13:14 +00:00
Erik Johnston	72a4de2ce6	Use consumeErrors=True on all DeferredLists. ... This is so that the DeferredLists actually consume the error instead of propogating down the non-existent errback chain. This should reduce the number of unhandled errors we are seeing.	2015-02-17 10:07:01 +00:00
Erik Johnston	91fc5eef1d	Mark old events as outliers. ... This is to fix the issue where if a remote server sends an event that references a really "old" event, then the local server will pull that in and send to all clients. We decide if an event is old if its depth is less than the minimum depth of the room.	2015-02-16 14:27:40 +00:00
Erik Johnston	6138584651	Don't return anything from _handle_new_pdu, since we ignore the return value anyway	2015-02-16 14:08:02 +00:00
Erik Johnston	963256638d	Correctly handle all the places that can throw exceptions	2015-02-12 18:17:11 +00:00
Erik Johnston	697ab75a34	Sign auth_chains when returned by /state/ requests	2015-02-10 15:46:24 +00:00
Erik Johnston	8dae5c8108	Remove unused imports	2015-02-03 15:01:12 +00:00
Erik Johnston	0dd3aea319	Keep around the old (buggy) version of the prune_event function so that we can use it to check signatures for events on old servers	2015-02-03 14:58:30 +00:00
Erik Johnston	776ac820f9	Briefly doc structure of query_auth API.	2015-01-30 15:58:28 +00:00
Erik Johnston	a70a801184	Fix bug where we superfluously asked for current state. Change API of /query_auth/ so that we don't duplicate events in the response.	2015-01-30 13:34:01 +00:00
Erik Johnston	c1d860870b	Fix regression where we no longer correctly handled the case of gaps in our event graph	2015-01-30 10:48:47 +00:00
Erik Johnston	78015948a7	Initial implementation of auth conflict resolution	2015-01-29 16:52:33 +00:00
Erik Johnston	c92d64a6c3	Make it the responsibility of the replication layer to check signature and hashes.	2015-01-26 14:33:11 +00:00
Erik Johnston	7b88619241	Split up replication_layer module into client, server and transaction queue	2015-01-26 10:45:24 +00:00