Commit Graph

448 Commits

Author SHA1 Message Date
Erik Johnston
50934ce460 Comments 2016-11-21 16:55:23 +00:00
Erik Johnston
9687e039e7 Remove explicit calls to send_pdu 2016-11-21 14:48:51 +00:00
Erik Johnston
524d61bf7e Fix tests 2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245 Store federation stream positions in the database 2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a Handle sending events and device messages over federation 2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e Hook up the send queue and create a federation sender worker 2016-11-16 17:34:44 +00:00
Erik Johnston
1587b5a033 Add initial cut of federation send queue 2016-11-16 14:47:52 +00:00
Erik Johnston
59ef517e6b Use new federation_sender DI 2016-11-16 14:47:52 +00:00
Erik Johnston
847d5db1d1 Add transaction queue and transport layer to DI 2016-11-16 14:47:52 +00:00
Erik Johnston
daec6fc355 Move logic into transaction_queue 2016-11-16 14:47:52 +00:00
Erik Johnston
0e830d3770 Rename transaction queue functions to send_* 2016-11-16 14:47:52 +00:00
Erik Johnston
22578545a0 Time out typing over federation 2016-09-23 14:00:52 +01:00
Erik Johnston
23b6701a28 Support filtering remote room lists 2016-09-16 10:24:15 +01:00
Erik Johnston
5810cffd33 Pass since/from parameters over federation 2016-09-15 10:36:19 +01:00
Mark Haines
76b09c29b0 Merge pull request #1110 from matrix-org/markjh/e2e_timeout
Add a timeout parameter for end2end key queries.
2016-09-13 10:50:45 +01:00
Mark Haines
949c2c5435 Add a timeout parameter for end2end key queries.
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.

Fetch keys from different servers in parallel rather than in series.

Set the default timeout to 10s.
2016-09-12 18:17:09 +01:00
Erik Johnston
706b5d76ed Fix backfill when cannot find an event.
`get_pdu` can succeed but return None.
2016-09-12 14:59:51 +01:00
Erik Johnston
af4701b311 Fix incorrect attribute name 2016-09-09 17:36:56 +01:00
Erik Johnston
464ffd1b5e Comment 2016-09-09 17:17:23 +01:00
Erik Johnston
327425764e Add edu.type as part of key. Remove debug logging 2016-09-09 17:13:30 +01:00
Erik Johnston
52b2318777 Clobber EDUs in send queue 2016-09-09 15:59:08 +01:00
Erik Johnston
ab80d5e0a9 Drop replication log levels 2016-09-09 14:56:50 +01:00
Erik Johnston
a6c6750166 Check if destination is ready for retry earlier 2016-09-09 13:46:05 +01:00
Erik Johnston
4598682b43 Fix tightloop on sending transaction 2016-09-09 13:12:53 +01:00
Erik Johnston
d2688d7f03 Correctly guard against multiple concurrent transactions 2016-09-09 11:44:36 +01:00
Erik Johnston
b390756150 Update last_device_stream_id_by_dest if there is nothing to send 2016-09-09 11:00:15 +01:00
Mark Haines
43954d000e Add a new method to enqueue the device messages rather than sending a dummy EDU 2016-09-07 16:10:51 +01:00
Mark Haines
cb98ac261b Move the check for federated device_messages.
Move the check into _attempt_new_transaction.
Only delete messages if there were messages to delete.
2016-09-07 15:39:13 +01:00
Mark Haines
31a07d2335 Add stream change caches for device messages 2016-09-07 15:27:07 +01:00
Mark Haines
d4a35ada28 Send device messages over federation 2016-09-06 18:16:20 +01:00
Erik Johnston
2854ee2a52 Only pull out IDs from DB for /state_ids/ request 2016-09-02 10:53:36 +01:00
Erik Johnston
f4778d4cd9 Merge branch 'erikj/pdu_check' of github.com:matrix-org/synapse into develop 2016-09-01 14:40:36 +01:00
Erik Johnston
516a272aca Ensure we only return a validated pdu in get_pdu 2016-09-01 10:55:02 +01:00
Erik Johnston
bed10f9880 Use state handler instead of get_users_in_room/get_joined_hosts 2016-08-26 14:54:30 +01:00
Erik Johnston
9219139351 Preserve some logcontexts 2016-08-24 11:58:40 +01:00
Erik Johnston
c315922b5f PEP8 2016-08-10 16:34:10 +01:00
Erik Johnston
ca8abfbf30 Clean up TransactionQueue 2016-08-10 16:24:16 +01:00
Erik Johnston
5aeadb7414 Merge pull request #999 from matrix-org/erikj/measure_more
Measure federation send transaction resources
2016-08-10 14:16:14 +01:00
Erik Johnston
487bc49bf8 Don't stop on 4xx series errors 2016-08-10 13:39:12 +01:00
Erik Johnston
ea8c4094db Also pull out rejected events 2016-08-10 13:26:13 +01:00
Erik Johnston
f91df1f761 Store if we fail to fetch an event from a destination 2016-08-10 11:31:46 +01:00
Erik Johnston
3bc9629be5 Measure federation send transaction resources 2016-08-10 10:56:38 +01:00
Erik Johnston
f5deaff424 Merge pull request #991 from matrix-org/erikj/retry_make
Retry joining via other servers if first one failed. Fix some other bugs.
2016-08-05 18:21:27 +01:00
Erik Johnston
5f360182c6 Fix a couple of python bugs 2016-08-05 18:08:32 +01:00
Erik Johnston
46453bfc2f Retry joining via other servers if first one failed 2016-08-05 18:02:03 +01:00
Erik Johnston
24f36469bc Add federation /version API 2016-08-05 16:36:07 +01:00
Erik Johnston
597c79be10 Change the way we specify if we require auth or not 2016-08-05 16:17:04 +01:00
Erik Johnston
2d4de61fb7 Fix typo 2016-08-05 10:48:56 +01:00
Erik Johnston
fccadb7719 Check if we already have the events returned by /state/ 2016-08-05 10:43:47 +01:00
Erik Johnston
1515d1b581 Fallback to /state/ on both 400 and 404 2016-08-05 10:24:23 +01:00
Benjamin Saunders
a2b7102eea Tweak integrity error recovery to work as intended 2016-08-04 20:38:08 -07:00
Erik Johnston
257c41cc2e Fix typos. 2016-08-04 14:05:45 +01:00
Erik Johnston
b4e2290d89 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids_api 2016-08-04 14:04:35 +01:00
Erik Johnston
e3ee63578f Tidy up get_events 2016-08-04 14:01:18 +01:00
Erik Johnston
edb33eb163 Rename fields to _ids 2016-08-03 17:19:15 +01:00
Erik Johnston
bcc9cda8ca Fix copy + paste fails 2016-08-03 17:17:26 +01:00
Erik Johnston
4c56bedee3 Actually call get_room_state 2016-08-03 15:04:29 +01:00
Erik Johnston
520ee9bd2c Fix syntax error 2016-08-03 15:03:15 +01:00
Erik Johnston
a60a2eaa02 Comment 2016-08-03 14:52:43 +01:00
Erik Johnston
e3a720217a Add /state_ids federation API
The new API only returns the event_ids for the state, as most
requesters will already have the vast majority of the events already.
2016-08-03 14:47:37 +01:00
Mark Haines
921f17f938 Merge branch 'develop' into rav/refactor_device_query 2016-08-03 11:12:47 +01:00
Richard van der Hoff
1efee2f52b E2E keys: Make federation query share code with client query
Refactor the e2e query handler to separate out the local query, and then make
the federation handler use it.
2016-08-02 18:12:00 +01:00
Erik Johnston
c9154b970c Don't double wrap 200 2016-08-02 16:45:53 +01:00
Erik Johnston
b3d5c4ad9d Fix response cache 2016-08-02 16:42:21 +01:00
Erik Johnston
248e6770ca Cache federation state responses 2016-07-21 10:30:12 +01:00
Erik Johnston
8f4a9bbc16 Linearize some federation endpoints based on (origin, room_id) 2016-06-17 16:43:45 +01:00
Erik Johnston
9ba2bf1570 Merge pull request #878 from matrix-org/erikj/ujson
Disable responding with canonical json for federation
2016-06-17 16:22:12 +01:00
Erik Johnston
120c238705 Disable responding with canonical json for federation 2016-06-17 16:10:37 +01:00
Erik Johnston
2884712ca7 Only re-sign our own events 2016-06-17 14:47:33 +01:00
Erik Johnston
d41a1a91d3 Linearize fetching of gaps on incoming events
This potentially stops the server from doing multiple requests for the
same data.
2016-06-15 15:16:14 +01:00
Erik Johnston
d88faf92d1 Fix up federation PublicRoomList 2016-06-08 14:39:31 +01:00
Erik Johnston
958c968d02 Merge pull request #856 from matrix-org/erikj/fed_pub_rooms
Enable auth on /publicRoom endpoints
2016-06-08 14:36:09 +01:00
Erik Johnston
1fd6eb695d Enable auth on federation PublicRoomList 2016-06-08 14:15:18 +01:00
Erik Johnston
17aab5827a Add some logging for when servers ask for missing events 2016-06-08 11:55:31 +01:00
David Baker
6ecb2ca4ec pep8 2016-06-01 09:48:55 +01:00
David Baker
963e3ed282 Apparently I am not permitted to have two blank lines here 2016-05-31 17:22:53 +01:00
David Baker
d240796ded Basic, un-cached support for secondary_directory_servers 2016-05-31 17:20:07 +01:00
David Baker
70ecb415f5 Fix c+p fail 2016-05-31 12:00:54 +01:00
David Baker
e1625d62a8 Add federation room list servlet 2016-05-31 11:55:57 +01:00
Erik Johnston
f6ebaf4a32 Run transaction queue on reactor
This ensures that any CPU work that happens doesn't block message
sending.
2016-05-09 10:10:06 +01:00
Mark Haines
9c272da05f Add an openidish mechanism for proving to third parties that you own a given user_id 2016-05-05 13:42:44 +01:00
Erik Johnston
62b51b8452 Fix typo in event_auth servlet path 2016-04-29 12:00:51 +01:00
Erik Johnston
914f1eafac Lower timeout for make_membership_event
Calls to make_membership_event are done in response to client requests,
and so should not be retried over long timeframes.
2016-04-15 11:22:23 +01:00
Erik Johnston
acdfef7b14 Intern all the things 2016-03-23 16:25:54 +00:00
Mark Haines
5244c0b48e Remove unused backfilled parameter from persist_event 2016-03-21 18:06:08 +00:00
Erik Johnston
9adf0e92bc Catch exceptions from EDU handling 2016-03-18 15:12:50 +00:00
Erik Johnston
3c5f25507b Yield on EDU handling 2016-03-18 13:55:16 +00:00
Mark Haines
e9c1cabac2 Use parse_json_object_from_request to parse JSON out of request bodies 2016-03-11 16:41:03 +00:00
Erik Johnston
ed61a49169 Add profile information to invites 2016-03-04 14:35:02 +00:00
Daniel Wagner-Hall
577951b032 Allow third_party_signed to be specified on /join 2016-02-23 15:11:25 +00:00
Erik Johnston
2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Erik Johnston
9959d9ece8 Remove redundated BaseHomeServer 2016-01-26 13:52:29 +00:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
14d7acfad4 Host /unstable and /r0 versions of r0 APIs 2015-12-01 17:34:32 +00:00
Erik Johnston
cf4ef5f3c7 Only retry federation requests for a long time for background requests 2015-11-17 18:26:50 +00:00
Paul "LeoNerd" Evans
aca6e5bf46 Don't complain if /make_join response lacks 'prev_state' list (SYN-517) 2015-11-13 17:27:25 +00:00
Erik Johnston
c0b3554401 Fix missing profile data in federation joins
There was a regression where we stopped including profile data in
initial joins for rooms joined over federation.
2015-11-12 16:19:55 +00:00
Daniel Wagner-Hall
2cebe53545 Exchange 3pid invites for m.room.member invites 2015-11-05 16:43:19 +00:00
Erik Johnston
97d792b28f Don't rearrange transaction_queue 2015-11-03 16:31:08 +00:00
Erik Johnston
7ce264ce5f Fix broken cache for getting retry times. This meant we retried remote destinations way more frequently than we should 2015-11-03 16:24:03 +00:00
Daniel Wagner-Hall
137fafce4e Allow rejecting invites
This is done by using the same /leave flow as you would use if you had
already accepted the invite and wanted to leave.
2015-10-20 11:58:58 +01:00
Daniel Wagner-Hall
643b5fcdc8 Look for keys on the right objects 2015-10-15 13:10:30 +01:00
Daniel Wagner-Hall
0c38e8637f Remove unnecessary class-wrapping 2015-10-13 18:00:38 +01:00
Daniel Wagner-Hall
17dffef5ec Move event contents into third_party_layout field 2015-10-13 15:48:12 +01:00
Daniel Wagner-Hall
fcd9ba8802 Fix lint errors 2015-10-06 10:13:05 -05:00
Daniel Wagner-Hall
5b3e9713dd Implement third party identifier invites 2015-10-01 17:49:52 +01:00
Mark Haines
8899df13bf Merge pull request #208 from matrix-org/markjh/end-to-end-key-federation
Federation for end-to-end key requests.
2015-08-18 09:12:54 +01:00
Mark Haines
0cceb2ac92 Add a few strategic new lines to break up the on_query_client_keys and on_claim_client_keys methods in federation_server.py 2015-08-13 17:27:46 +01:00
Erik Johnston
2df8dd9b37 Move all the caches into their own package, synapse.util.caches 2015-08-11 18:00:59 +01:00
Mark Haines
2da3b1e60b Get the end-to-end key federation working 2015-07-24 18:26:46 +01:00
Mark Haines
62c010283d Add federation support for end-to-end key requests 2015-07-23 16:03:38 +01:00
Erik Johnston
b5f55a1d85 Implement bulk verify_signed_json API 2015-06-26 10:39:34 +01:00
Erik Johnston
cee69441d3 Log more when we have processed the request 2015-06-15 17:11:44 +01:00
Erik Johnston
8047fd2434 Merge pull request #176 from matrix-org/erikj/backfill_auth
Improve backfill.
2015-06-03 17:25:37 +01:00
Erik Johnston
3483b78d1a Log where a request came from in federation 2015-06-02 18:15:13 +01:00
Erik Johnston
d3ded420b1 Rephrase log line 2015-06-02 16:30:52 +01:00
Erik Johnston
22716774d5 Don't about JSON when warning about content tampering 2015-06-02 16:30:52 +01:00
Erik Johnston
6eadbfbea0 Remove redundant for loop 2015-05-22 16:12:20 +01:00
Erik Johnston
74b7de83ec Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-22 16:10:42 +01:00
Mark Haines
36317f3dad Merge pull request #156 from matrix-org/erikj/join_perf
Make joining #matrix:matrix.org over federation quicker
2015-05-22 16:09:54 +01:00
Erik Johnston
284f55a7fb Add doc strings 2015-05-22 15:18:04 +01:00
Erik Johnston
b21d015c55 Log origin and stats of incoming transactions 2015-05-22 14:44:25 +01:00
Erik Johnston
e70e8e053e Add txn_id to some log lines 2015-05-22 14:44:02 +01:00
Erik Johnston
1b446a5d85 Log less lines at INFO level, but include more helpful information 2015-05-22 14:29:57 +01:00
Erik Johnston
c2633907c5 Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-21 14:58:47 +01:00
Erik Johnston
ebfdd2eb5b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-21 14:54:52 +01:00
Erik Johnston
20814fabdd Actually fetch state for new backwards extremeties when backfilling. 2015-05-20 11:59:02 +01:00
Erik Johnston
3a653515ec Add None check 2015-05-19 15:27:09 +01:00
Erik Johnston
5b1631a4a9 Add a timeout param to get_event 2015-05-19 14:53:32 +01:00
Erik Johnston
d5cea26d45 Remove pointless newline 2015-05-18 10:16:45 +01:00
Erik Johnston
c71176858b Newline, remove debug logging 2015-05-18 10:11:14 +01:00
Erik Johnston
f8bd4de87d Remove debug logging 2015-05-18 09:58:03 +01:00
Erik Johnston
a2c4f3f150 Fix daedlock 2015-05-15 10:54:04 +01:00
Erik Johnston
95dedb866f Unwrap defer.gatherResults failures 2015-05-12 13:14:29 +01:00
Erik Johnston
476899295f Change the way we do logging contexts so that they survive divergences 2015-05-08 16:32:18 +01:00
Erik Johnston
0cf7e480b4 And use buffer(...) there as well 2015-05-05 18:20:01 +01:00
Erik Johnston
977338a7af Use buffer(...) when inserting into bytea column 2015-05-05 18:12:53 +01:00
Erik Johnston
3d5a955e08 Missed events are not outliers 2015-05-05 17:36:57 +01:00
Erik Johnston
1942382246 Don't log enqueue_ 2015-05-01 16:14:25 +01:00
Erik Johnston
23c639ff32 Split a storage function in two so that we don't have to do extra work. 2015-05-01 10:17:19 +01:00
Erik Johnston
c178e4e6ca Add missing servlet to list 2015-03-19 17:48:21 +00:00
Paul "LeoNerd" Evans
128cf2daf7 Appease pep8 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
c782e893ec Neater metrics from TransactionQueue 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
2e4f0b2bd7 Replace the @metrics.counted annotations in federation with specifically-written counters and distributions 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
1748605c5d Count incoming HTTP requests per servlet that responds 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
b0cf867319 Use _ instead of . as a metric namespacing separator, for Prometheus 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f9478e475b Rename Metrics' "keys" to "labels" 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
094803cf82 Put vector gauges on transaction queue pending PDU and EDU dicts 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
9470412316 Initial attempt at sprinkling some @metrics.counted decorations around the federation code 2015-03-12 16:24:50 +00:00