Richard van der Hoff
91b3981800
Try harder when sending leave events
...
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
2017-04-21 01:50:36 +01:00
Richard van der Hoff
4903ccf159
Fix some lies, and other clarifications, in docstrings
...
The documentation on get_json has been wrong ever since the very first commit
to synapse...
2017-04-21 01:31:09 +01:00
Erik Johnston
247c736b9b
Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
...
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
Paul Evans
8fbc0d29ee
Merge pull request #2121 from matrix-org/paul/sent-transactions-metric
...
Add a counter metric for successfully-sent transactions
2017-04-12 11:04:31 +01:00
Erik Johnston
26ae5178a4
Add some comments
2017-04-12 10:36:29 +01:00
Erik Johnston
1745069543
Comment
2017-04-12 10:17:10 +01:00
Erik Johnston
c7ddb5ef7a
Reuse get_interested_parties
2017-04-12 10:16:26 +01:00
Paul "LeoNerd" Evans
11dbceb761
Add a counter metric for successfully-sent transactions
2017-04-11 17:16:12 +01:00
Erik Johnston
a8c8e4efd4
Comment
2017-04-11 15:35:49 +01:00
Erik Johnston
2be8a281d2
Comments
2017-04-11 15:28:24 +01:00
Erik Johnston
6308ac45b0
Move get_interested_remotes back to presence handler
2017-04-11 15:19:26 +01:00
Erik Johnston
b9b72bc6e2
Comments
2017-04-11 15:15:34 +01:00
Erik Johnston
84fbb80c8f
Use generators
2017-04-10 16:55:56 +01:00
Erik Johnston
40453b3f84
Dedupe KeyedEdu and Devices federation repl traffic
2017-04-10 16:49:51 +01:00
Erik Johnston
29574fd5b3
Reduce federation presence replication traffic
...
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
Erik Johnston
0018491af2
Rename variable
2017-04-10 12:44:43 +01:00
Erik Johnston
8c5f03cec7
Revert to sending the same data type as before
2017-04-10 10:07:18 +01:00
Erik Johnston
f8434db549
Change name
2017-04-10 10:03:07 +01:00
Erik Johnston
ab904caf33
Comments
2017-04-10 10:02:17 +01:00
Erik Johnston
a828a64b75
Comment
2017-04-07 11:54:03 +01:00
Erik Johnston
d4d176e5d0
Add logging
2017-04-07 11:51:28 +01:00
Erik Johnston
449d1297ca
Fix up federation SendQueue and document types
2017-04-07 11:48:33 +01:00
Erik Johnston
f10ce8944b
Don't double json encode federation replication data
2017-04-05 11:10:28 +01:00
Erik Johnston
62b89daac6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server
2017-04-04 09:46:16 +01:00
Richard van der Hoff
30bcbf775a
Accept join events from all servers
...
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.
(Fixes #1893 ).
2017-04-03 15:58:07 +01:00
Erik Johnston
11880103b1
Make federation send queue take the current position
2017-03-30 12:54:36 +01:00
Erik Johnston
85be3dde81
Bail early if remote wouldn't be retried ( #2064 )
...
* Bail early if remote wouldn't be retried
* Don't always return true
* Just use get_retry_limiter
* Spelling
2017-03-29 11:48:27 +01:00
Erik Johnston
c5b0bdd542
Merge pull request #2067 from matrix-org/erikj/notify_on_fed
...
Notify on new federation traffic
2017-03-29 11:41:37 +01:00
Erik Johnston
35b4aa04be
Notify on new federation traffic
2017-03-27 14:07:47 +01:00
Erik Johnston
2a28b79e04
Batch sending of device list pokes
2017-03-24 14:44:49 +00:00
Richard van der Hoff
5a16cb4bf0
Ignore backoff history for invites, aliases, and roomdirs
...
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
Richard van der Hoff
4bd597d9fc
push federation retry limiter down to matrixfederationclient
...
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
Richard van der Hoff
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
Richard van der Hoff
29235901b8
Move FederationServer._handle_new_pdu to FederationHandler
...
Unfortunately this significantly increases the size of the already-rather-big
FederationHandler, but the code fits more naturally here, and it paves the way
for the tighter integration that I need between handling incoming PDUs and
doing the join dance.
Other than renaming the existing `FederationHandler.on_receive_pdu` to
`_process_received_pdu` to make way for it, this just consists of the move, and
replacing `self.handler` with `self` and `self` with `self.replication_layer`.
2017-03-09 16:20:13 +00:00
Richard van der Hoff
e8b1721290
Move sig check out of _handle_new_pdu
...
When we receive PDUs via `get_missing_events`, we have already checked their
sigs, so there is no need to do it again.
2017-03-09 15:50:44 +00:00
Richard van der Hoff
3406333a58
Factor _get_missing_events_for_pdu out of _handle_new_pdu
...
This should be functionally identical: it just seeks to improve readability by
reducing indentation.
2017-03-09 15:50:44 +00:00
Richard van der Hoff
45d173a59a
Fix docstring
2017-03-09 15:50:29 +00:00
Richard van der Hoff
0c4cf9372b
Fix a race in transaction queue
...
It was theoretically possible for a PDU to get queued and not sent for ages. On
closer inspection I think there were bigger problems elsewhere, but we might as
well fix this since it's easy.
2017-02-20 16:46:25 +00:00
Erik Johnston
df4ecff5a9
Correctly raise exceptions for ratelimitng. Ratelimit on 401
2017-02-01 15:42:19 +00:00
Erik Johnston
ae7a132f38
Better handle 404 response for federation /send/
2017-01-31 13:40:09 +00:00
Erik Johnston
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
Erik Johnston
51e9fe36e4
Fix up sending of m.device_list_update edus
2017-01-25 16:55:21 +00:00
Erik Johnston
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
Erik Johnston
fc7cae8aa3
Merge pull request #1824 from matrix-org/erikj/retry_host_log
...
Lower the not retrying host log line to debug
2017-01-18 11:07:51 +00:00
Erik Johnston
f878f64f43
Lower the not retrying host log line to debug
2017-01-17 17:20:39 +00:00
Erik Johnston
e6153e1bd1
Fix couple of federation state bugs
2017-01-17 13:22:34 +00:00
Erik Johnston
f7085ac84f
Name linearizer's for better logs
2017-01-09 17:17:10 +00:00
Matthew
4304e7e593
do the discard check in the right place to avoid grabbing dependent events
2017-01-07 03:44:18 +00:00
Matthew
e10c527930
Discard PDUs from invalid origins due to #1753 in 0.18.[56]
2017-01-07 02:13:14 +00:00
Matthew Hodgson
468749c9fc
fix comment
2017-01-05 12:00:11 +00:00
Matthew Hodgson
eedf400d05
limit total timeout for get_missing_events to 10s
2017-01-05 11:58:15 +00:00
Mark Haines
f784980d2b
Only send events that originate on this server.
...
Or events that are sent via the federation "send_join" API.
This should match the behaviour from before v0.18.5 and #1635 landed.
2017-01-05 11:26:30 +00:00
Mark Haines
e02bdaf08b
Get the destinations from the state from before the event
...
Rather than the state after then event.
2017-01-04 15:17:15 +00:00
Mark Haines
b6b67715ed
Send ALL membership events to the server that was affected.
...
Send all membership changes to the server that was affected.
This ensures that if the last member of a room on a server
was kicked or banned they get told about it.
2017-01-04 13:56:20 +00:00
Mark Haines
62ce3034f3
s/aquire/acquire/g
2016-12-30 20:04:44 +00:00
Mark Haines
0aff09f6c9
Add more useful logging when we block fetching events
2016-12-30 20:00:44 +00:00
Mark Haines
cc50b1ae53
Remove fallback from get_missing_events.
...
get_missing_events used to fallback to fetching the missing events
individually requesting from every server in the room, one by one.e
This could be unacceptably slow, possibly causing #1732
2016-12-30 18:13:15 +00:00
Erik Johnston
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
Erik Johnston
aaecffba3a
Correctly handle 500's and 429 on federation
2016-11-24 15:04:49 +00:00
Erik Johnston
feec718265
Shuffle receipt handler around so that worker apps don't need to load it
2016-11-23 15:14:24 +00:00
Erik Johnston
4c79a63fd7
Explicit federation ack
2016-11-23 10:40:44 +00:00
Erik Johnston
51e89709aa
Comments
2016-11-21 17:59:39 +00:00
Erik Johnston
88d85ebae1
Add some metrics
2016-11-21 17:36:05 +00:00
Erik Johnston
50934ce460
Comments
2016-11-21 16:55:23 +00:00
Erik Johnston
9687e039e7
Remove explicit calls to send_pdu
2016-11-21 14:48:51 +00:00
Erik Johnston
524d61bf7e
Fix tests
2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
Erik Johnston
1587b5a033
Add initial cut of federation send queue
2016-11-16 14:47:52 +00:00
Erik Johnston
59ef517e6b
Use new federation_sender DI
2016-11-16 14:47:52 +00:00
Erik Johnston
847d5db1d1
Add transaction queue and transport layer to DI
2016-11-16 14:47:52 +00:00
Erik Johnston
daec6fc355
Move logic into transaction_queue
2016-11-16 14:47:52 +00:00
Erik Johnston
0e830d3770
Rename transaction queue functions to send_*
2016-11-16 14:47:52 +00:00
Erik Johnston
22578545a0
Time out typing over federation
2016-09-23 14:00:52 +01:00
Erik Johnston
23b6701a28
Support filtering remote room lists
2016-09-16 10:24:15 +01:00
Erik Johnston
5810cffd33
Pass since/from parameters over federation
2016-09-15 10:36:19 +01:00
Mark Haines
76b09c29b0
Merge pull request #1110 from matrix-org/markjh/e2e_timeout
...
Add a timeout parameter for end2end key queries.
2016-09-13 10:50:45 +01:00
Mark Haines
949c2c5435
Add a timeout parameter for end2end key queries.
...
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.
Fetch keys from different servers in parallel rather than in series.
Set the default timeout to 10s.
2016-09-12 18:17:09 +01:00
Erik Johnston
706b5d76ed
Fix backfill when cannot find an event.
...
`get_pdu` can succeed but return None.
2016-09-12 14:59:51 +01:00
Erik Johnston
af4701b311
Fix incorrect attribute name
2016-09-09 17:36:56 +01:00
Erik Johnston
464ffd1b5e
Comment
2016-09-09 17:17:23 +01:00
Erik Johnston
327425764e
Add edu.type as part of key. Remove debug logging
2016-09-09 17:13:30 +01:00
Erik Johnston
52b2318777
Clobber EDUs in send queue
2016-09-09 15:59:08 +01:00
Erik Johnston
ab80d5e0a9
Drop replication log levels
2016-09-09 14:56:50 +01:00
Erik Johnston
a6c6750166
Check if destination is ready for retry earlier
2016-09-09 13:46:05 +01:00
Erik Johnston
4598682b43
Fix tightloop on sending transaction
2016-09-09 13:12:53 +01:00
Erik Johnston
d2688d7f03
Correctly guard against multiple concurrent transactions
2016-09-09 11:44:36 +01:00
Erik Johnston
b390756150
Update last_device_stream_id_by_dest if there is nothing to send
2016-09-09 11:00:15 +01:00
Mark Haines
43954d000e
Add a new method to enqueue the device messages rather than sending a dummy EDU
2016-09-07 16:10:51 +01:00
Mark Haines
cb98ac261b
Move the check for federated device_messages.
...
Move the check into _attempt_new_transaction.
Only delete messages if there were messages to delete.
2016-09-07 15:39:13 +01:00
Mark Haines
31a07d2335
Add stream change caches for device messages
2016-09-07 15:27:07 +01:00
Mark Haines
d4a35ada28
Send device messages over federation
2016-09-06 18:16:20 +01:00
Erik Johnston
2854ee2a52
Only pull out IDs from DB for /state_ids/ request
2016-09-02 10:53:36 +01:00
Erik Johnston
f4778d4cd9
Merge branch 'erikj/pdu_check' of github.com:matrix-org/synapse into develop
2016-09-01 14:40:36 +01:00
Erik Johnston
516a272aca
Ensure we only return a validated pdu in get_pdu
2016-09-01 10:55:02 +01:00
Erik Johnston
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
Erik Johnston
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
Erik Johnston
c315922b5f
PEP8
2016-08-10 16:34:10 +01:00
Erik Johnston
ca8abfbf30
Clean up TransactionQueue
2016-08-10 16:24:16 +01:00
Erik Johnston
5aeadb7414
Merge pull request #999 from matrix-org/erikj/measure_more
...
Measure federation send transaction resources
2016-08-10 14:16:14 +01:00
Erik Johnston
487bc49bf8
Don't stop on 4xx series errors
2016-08-10 13:39:12 +01:00
Erik Johnston
ea8c4094db
Also pull out rejected events
2016-08-10 13:26:13 +01:00
Erik Johnston
f91df1f761
Store if we fail to fetch an event from a destination
2016-08-10 11:31:46 +01:00
Erik Johnston
3bc9629be5
Measure federation send transaction resources
2016-08-10 10:56:38 +01:00
Erik Johnston
f5deaff424
Merge pull request #991 from matrix-org/erikj/retry_make
...
Retry joining via other servers if first one failed. Fix some other bugs.
2016-08-05 18:21:27 +01:00
Erik Johnston
5f360182c6
Fix a couple of python bugs
2016-08-05 18:08:32 +01:00
Erik Johnston
46453bfc2f
Retry joining via other servers if first one failed
2016-08-05 18:02:03 +01:00
Erik Johnston
24f36469bc
Add federation /version API
2016-08-05 16:36:07 +01:00
Erik Johnston
597c79be10
Change the way we specify if we require auth or not
2016-08-05 16:17:04 +01:00
Erik Johnston
2d4de61fb7
Fix typo
2016-08-05 10:48:56 +01:00
Erik Johnston
fccadb7719
Check if we already have the events returned by /state/
2016-08-05 10:43:47 +01:00
Erik Johnston
1515d1b581
Fallback to /state/ on both 400 and 404
2016-08-05 10:24:23 +01:00
Benjamin Saunders
a2b7102eea
Tweak integrity error recovery to work as intended
2016-08-04 20:38:08 -07:00
Erik Johnston
257c41cc2e
Fix typos.
2016-08-04 14:05:45 +01:00
Erik Johnston
b4e2290d89
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_ids_api
2016-08-04 14:04:35 +01:00
Erik Johnston
e3ee63578f
Tidy up get_events
2016-08-04 14:01:18 +01:00
Erik Johnston
edb33eb163
Rename fields to _ids
2016-08-03 17:19:15 +01:00
Erik Johnston
bcc9cda8ca
Fix copy + paste fails
2016-08-03 17:17:26 +01:00
Erik Johnston
4c56bedee3
Actually call get_room_state
2016-08-03 15:04:29 +01:00
Erik Johnston
520ee9bd2c
Fix syntax error
2016-08-03 15:03:15 +01:00
Erik Johnston
a60a2eaa02
Comment
2016-08-03 14:52:43 +01:00
Erik Johnston
e3a720217a
Add /state_ids federation API
...
The new API only returns the event_ids for the state, as most
requesters will already have the vast majority of the events already.
2016-08-03 14:47:37 +01:00
Mark Haines
921f17f938
Merge branch 'develop' into rav/refactor_device_query
2016-08-03 11:12:47 +01:00
Richard van der Hoff
1efee2f52b
E2E keys: Make federation query share code with client query
...
Refactor the e2e query handler to separate out the local query, and then make
the federation handler use it.
2016-08-02 18:12:00 +01:00
Erik Johnston
c9154b970c
Don't double wrap 200
2016-08-02 16:45:53 +01:00
Erik Johnston
b3d5c4ad9d
Fix response cache
2016-08-02 16:42:21 +01:00
Erik Johnston
248e6770ca
Cache federation state responses
2016-07-21 10:30:12 +01:00
Erik Johnston
8f4a9bbc16
Linearize some federation endpoints based on (origin, room_id)
2016-06-17 16:43:45 +01:00
Erik Johnston
9ba2bf1570
Merge pull request #878 from matrix-org/erikj/ujson
...
Disable responding with canonical json for federation
2016-06-17 16:22:12 +01:00
Erik Johnston
120c238705
Disable responding with canonical json for federation
2016-06-17 16:10:37 +01:00
Erik Johnston
2884712ca7
Only re-sign our own events
2016-06-17 14:47:33 +01:00
Erik Johnston
d41a1a91d3
Linearize fetching of gaps on incoming events
...
This potentially stops the server from doing multiple requests for the
same data.
2016-06-15 15:16:14 +01:00
Erik Johnston
d88faf92d1
Fix up federation PublicRoomList
2016-06-08 14:39:31 +01:00
Erik Johnston
958c968d02
Merge pull request #856 from matrix-org/erikj/fed_pub_rooms
...
Enable auth on /publicRoom endpoints
2016-06-08 14:36:09 +01:00
Erik Johnston
1fd6eb695d
Enable auth on federation PublicRoomList
2016-06-08 14:15:18 +01:00
Erik Johnston
17aab5827a
Add some logging for when servers ask for missing events
2016-06-08 11:55:31 +01:00
David Baker
6ecb2ca4ec
pep8
2016-06-01 09:48:55 +01:00
David Baker
963e3ed282
Apparently I am not permitted to have two blank lines here
2016-05-31 17:22:53 +01:00
David Baker
d240796ded
Basic, un-cached support for secondary_directory_servers
2016-05-31 17:20:07 +01:00
David Baker
70ecb415f5
Fix c+p fail
2016-05-31 12:00:54 +01:00
David Baker
e1625d62a8
Add federation room list servlet
2016-05-31 11:55:57 +01:00
Erik Johnston
f6ebaf4a32
Run transaction queue on reactor
...
This ensures that any CPU work that happens doesn't block message
sending.
2016-05-09 10:10:06 +01:00
Mark Haines
9c272da05f
Add an openidish mechanism for proving to third parties that you own a given user_id
2016-05-05 13:42:44 +01:00
Erik Johnston
62b51b8452
Fix typo in event_auth servlet path
2016-04-29 12:00:51 +01:00
Erik Johnston
914f1eafac
Lower timeout for make_membership_event
...
Calls to make_membership_event are done in response to client requests,
and so should not be retried over long timeframes.
2016-04-15 11:22:23 +01:00
Erik Johnston
acdfef7b14
Intern all the things
2016-03-23 16:25:54 +00:00
Mark Haines
5244c0b48e
Remove unused backfilled parameter from persist_event
2016-03-21 18:06:08 +00:00
Erik Johnston
9adf0e92bc
Catch exceptions from EDU handling
2016-03-18 15:12:50 +00:00
Erik Johnston
3c5f25507b
Yield on EDU handling
2016-03-18 13:55:16 +00:00
Mark Haines
e9c1cabac2
Use parse_json_object_from_request to parse JSON out of request bodies
2016-03-11 16:41:03 +00:00
Erik Johnston
ed61a49169
Add profile information to invites
2016-03-04 14:35:02 +00:00
Daniel Wagner-Hall
577951b032
Allow third_party_signed to be specified on /join
2016-02-23 15:11:25 +00:00
Erik Johnston
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
Erik Johnston
9959d9ece8
Remove redundated BaseHomeServer
2016-01-26 13:52:29 +00:00
Matthew Hodgson
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
14d7acfad4
Host /unstable and /r0 versions of r0 APIs
2015-12-01 17:34:32 +00:00
Erik Johnston
cf4ef5f3c7
Only retry federation requests for a long time for background requests
2015-11-17 18:26:50 +00:00
Paul "LeoNerd" Evans
aca6e5bf46
Don't complain if /make_join response lacks 'prev_state' list (SYN-517)
2015-11-13 17:27:25 +00:00
Erik Johnston
c0b3554401
Fix missing profile data in federation joins
...
There was a regression where we stopped including profile data in
initial joins for rooms joined over federation.
2015-11-12 16:19:55 +00:00
Daniel Wagner-Hall
2cebe53545
Exchange 3pid invites for m.room.member invites
2015-11-05 16:43:19 +00:00
Erik Johnston
97d792b28f
Don't rearrange transaction_queue
2015-11-03 16:31:08 +00:00
Erik Johnston
7ce264ce5f
Fix broken cache for getting retry times. This meant we retried remote destinations way more frequently than we should
2015-11-03 16:24:03 +00:00
Daniel Wagner-Hall
137fafce4e
Allow rejecting invites
...
This is done by using the same /leave flow as you would use if you had
already accepted the invite and wanted to leave.
2015-10-20 11:58:58 +01:00
Daniel Wagner-Hall
643b5fcdc8
Look for keys on the right objects
2015-10-15 13:10:30 +01:00
Daniel Wagner-Hall
0c38e8637f
Remove unnecessary class-wrapping
2015-10-13 18:00:38 +01:00
Daniel Wagner-Hall
17dffef5ec
Move event contents into third_party_layout field
2015-10-13 15:48:12 +01:00
Daniel Wagner-Hall
fcd9ba8802
Fix lint errors
2015-10-06 10:13:05 -05:00
Daniel Wagner-Hall
5b3e9713dd
Implement third party identifier invites
2015-10-01 17:49:52 +01:00
Mark Haines
8899df13bf
Merge pull request #208 from matrix-org/markjh/end-to-end-key-federation
...
Federation for end-to-end key requests.
2015-08-18 09:12:54 +01:00
Mark Haines
0cceb2ac92
Add a few strategic new lines to break up the on_query_client_keys and on_claim_client_keys methods in federation_server.py
2015-08-13 17:27:46 +01:00
Erik Johnston
2df8dd9b37
Move all the caches into their own package, synapse.util.caches
2015-08-11 18:00:59 +01:00
Mark Haines
2da3b1e60b
Get the end-to-end key federation working
2015-07-24 18:26:46 +01:00
Mark Haines
62c010283d
Add federation support for end-to-end key requests
2015-07-23 16:03:38 +01:00
Erik Johnston
b5f55a1d85
Implement bulk verify_signed_json API
2015-06-26 10:39:34 +01:00
Erik Johnston
cee69441d3
Log more when we have processed the request
2015-06-15 17:11:44 +01:00
Erik Johnston
8047fd2434
Merge pull request #176 from matrix-org/erikj/backfill_auth
...
Improve backfill.
2015-06-03 17:25:37 +01:00
Erik Johnston
3483b78d1a
Log where a request came from in federation
2015-06-02 18:15:13 +01:00
Erik Johnston
d3ded420b1
Rephrase log line
2015-06-02 16:30:52 +01:00
Erik Johnston
22716774d5
Don't about JSON when warning about content tampering
2015-06-02 16:30:52 +01:00
Erik Johnston
6eadbfbea0
Remove redundant for loop
2015-05-22 16:12:20 +01:00
Erik Johnston
74b7de83ec
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-22 16:10:42 +01:00
Mark Haines
36317f3dad
Merge pull request #156 from matrix-org/erikj/join_perf
...
Make joining #matrix:matrix.org over federation quicker
2015-05-22 16:09:54 +01:00
Erik Johnston
284f55a7fb
Add doc strings
2015-05-22 15:18:04 +01:00
Erik Johnston
b21d015c55
Log origin and stats of incoming transactions
2015-05-22 14:44:25 +01:00
Erik Johnston
e70e8e053e
Add txn_id to some log lines
2015-05-22 14:44:02 +01:00
Erik Johnston
1b446a5d85
Log less lines at INFO level, but include more helpful information
2015-05-22 14:29:57 +01:00
Erik Johnston
c2633907c5
Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes
2015-05-21 14:58:47 +01:00
Erik Johnston
ebfdd2eb5b
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf
2015-05-21 14:54:52 +01:00
Erik Johnston
20814fabdd
Actually fetch state for new backwards extremeties when backfilling.
2015-05-20 11:59:02 +01:00
Erik Johnston
3a653515ec
Add None check
2015-05-19 15:27:09 +01:00
Erik Johnston
5b1631a4a9
Add a timeout param to get_event
2015-05-19 14:53:32 +01:00
Erik Johnston
d5cea26d45
Remove pointless newline
2015-05-18 10:16:45 +01:00
Erik Johnston
c71176858b
Newline, remove debug logging
2015-05-18 10:11:14 +01:00
Erik Johnston
f8bd4de87d
Remove debug logging
2015-05-18 09:58:03 +01:00
Erik Johnston
a2c4f3f150
Fix daedlock
2015-05-15 10:54:04 +01:00
Erik Johnston
95dedb866f
Unwrap defer.gatherResults failures
2015-05-12 13:14:29 +01:00
Erik Johnston
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
Erik Johnston
0cf7e480b4
And use buffer(...) there as well
2015-05-05 18:20:01 +01:00
Erik Johnston
977338a7af
Use buffer(...) when inserting into bytea column
2015-05-05 18:12:53 +01:00
Erik Johnston
3d5a955e08
Missed events are not outliers
2015-05-05 17:36:57 +01:00
Erik Johnston
1942382246
Don't log enqueue_
2015-05-01 16:14:25 +01:00
Erik Johnston
23c639ff32
Split a storage function in two so that we don't have to do extra work.
2015-05-01 10:17:19 +01:00
Erik Johnston
c178e4e6ca
Add missing servlet to list
2015-03-19 17:48:21 +00:00
Paul "LeoNerd" Evans
128cf2daf7
Appease pep8
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
c782e893ec
Neater metrics from TransactionQueue
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
2e4f0b2bd7
Replace the @metrics.counted annotations in federation with specifically-written counters and distributions
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
1748605c5d
Count incoming HTTP requests per servlet that responds
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
094803cf82
Put vector gauges on transaction queue pending PDU and EDU dicts
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
9470412316
Initial attempt at sprinkling some @metrics.counted decorations around the federation code
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
120b689284
Delete pointless (and unreachable) __init__ method from FederationClient
2015-03-12 16:24:50 +00:00
Erik Johnston
abaf47bbb6
Fix bug in logging.
2015-03-10 10:28:29 +00:00
Erik Johnston
98b867f7b7
Fix bug in logging.
2015-03-10 10:16:09 +00:00
Erik Johnston
019422ebba
Merge pull request #101 from matrix-org/neaten-federation-servlets
...
Neaten federation servlets
2015-03-09 17:39:06 +00:00
Paul "LeoNerd" Evans
d79d91a4a7
Appease pep8
2015-03-05 20:55:40 +00:00
Paul "LeoNerd" Evans
5eab2549ab
Append a $ on PATH at registration time, meaning each PATH attribute doesn't need it
2015-03-05 20:36:05 +00:00
Paul "LeoNerd" Evans
7644cb79b2
Slightly neater(?) arrangement of authentication wrapper for HTTP servlet methods
2015-03-05 20:33:16 +00:00
Paul "LeoNerd" Evans
ba8ac996f9
Remove the dead 'rate_limit_origin' method from TransportLayerServer
2015-03-05 19:43:17 +00:00
Paul "LeoNerd" Evans
a901ed16b5
Move federation API responding code out of weird mix of lambdas into Servlet-style methods on instances
2015-03-05 19:10:57 +00:00
Erik Johnston
9708f49abf
Docs
2015-03-05 16:35:16 +00:00
Erik Johnston
96fee64421
Remove unecessary check
2015-03-05 16:31:47 +00:00
Erik Johnston
39aa968a76
Respect min_depth argument
2015-03-05 16:31:32 +00:00
Erik Johnston
6dfd8c73fc
Docs.
2015-03-05 16:31:13 +00:00
Erik Johnston
ae702d161a
Handle if get_missing_pdu returns 400 or not all events.
2015-03-05 16:08:02 +00:00
Erik Johnston
0a036944bd
Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu
2015-03-02 13:53:30 +00:00
Erik Johnston
4195e55ccc
Merge branch 'develop' of github.com:matrix-org/synapse into federation_rate_limit
2015-03-02 13:39:22 +00:00
Erik Johnston
29481690c5
If we're yielding don't add errback
2015-03-02 11:50:43 +00:00
Erik Johnston
23d9bd1d74
Process transactions serially.
...
Since the events received in a transaction are ordered, later events
might depend on earlier events and so we shouldn't blindly process them
in parellel.
2015-03-02 11:39:57 +00:00
Erik Johnston
9d9b230501
Make the federation server ratelimiting configurable.
2015-03-02 11:33:45 +00:00
Erik Johnston
0554d07082
Move federation rate limiting out of transport layer
2015-02-27 15:41:52 +00:00
Erik Johnston
9dc9118e55
Document FederationRateLimiter
2015-02-27 15:16:47 +00:00
Erik Johnston
93d90765c4
Initial implementation of federation server rate limiting
2015-02-26 16:15:26 +00:00
Erik Johnston
59362454dd
Must update pending_transactions map before yield'ing
2015-02-26 15:47:35 +00:00
Erik Johnston
db215b7e00
Implement and use new batched get missing pdu
2015-02-23 13:58:02 +00:00
Erik Johnston
0ac2a79faa
Initial stab at implementing a batched get_missing_pdus request
2015-02-19 17:24:14 +00:00
Erik Johnston
a7925259a1
Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.7.1
2015-02-18 13:57:55 +00:00
Erik Johnston
446ef58992
Add errback to all deferreds in transaction_queue
2015-02-18 12:03:26 +00:00
Mark Haines
2462aacd77
Restrict the destinations that synapse can talk to
2015-02-18 11:52:51 +00:00
Erik Johnston
b68e4a729f
Discard destination 'localhost'
2015-02-18 11:32:39 +00:00
Erik Johnston
47d3ff4cf8
Don't send failure to self
2015-02-18 11:30:37 +00:00
Erik Johnston
e482541e1d
Fix pyflakes
2015-02-18 10:44:22 +00:00
Erik Johnston
1be67eca8a
Merge branch 'keyclient_retry_scheme' of github.com:matrix-org/synapse into develop
2015-02-18 10:34:40 +00:00
Erik Johnston
fe672a04f7
Merge pull request #77 from matrix-org/failures
...
Failures
2015-02-18 10:29:29 +00:00
Erik Johnston
08f804208b
Merge pull request #79 from matrix-org/get_pdu_limiting
...
Get pdu limiting
2015-02-18 10:29:10 +00:00
Erik Johnston
ec847059f3
Rename _fail_fetch_pdu_cache to _get_pdu_cache
2015-02-18 10:14:10 +00:00
Erik Johnston
9371019133
Try to only back off if we think we failed to connect to the remote
2015-02-17 18:13:34 +00:00
Erik Johnston
2b8f1a956c
Add per server retry limiting.
...
Factor out the pre destination retry logic from TransactionQueue so it
can be reused in both get_pdu and crypto.keyring
2015-02-17 17:20:56 +00:00
Erik Johnston
676e8ee78a
Remove debug raise
2015-02-17 15:22:45 +00:00
Erik Johnston
fa6c93bd26
Merge branch 'consumeErrors' of github.com:matrix-org/synapse into develop
2015-02-17 15:18:17 +00:00
Erik Johnston
c02da58a9d
Merge branch 'develop' of github.com:matrix-org/synapse into failures
2015-02-17 15:15:07 +00:00
Erik Johnston
659ead082f
Format the response of transaction request in a nicer way
2015-02-17 15:11:44 +00:00
Erik Johnston
c82e26ad4b
Actually respond with JSON to incoming transaction
2015-02-17 13:24:13 +00:00
Erik Johnston
02bfa889de
Handle recieving failures in transactions
2015-02-17 13:13:14 +00:00
Erik Johnston
72a4de2ce6
Use consumeErrors=True on all DeferredLists.
...
This is so that the DeferredLists actually consume the error instead of
propogating down the non-existent errback chain. This should reduce the
number of unhandled errors we are seeing.
2015-02-17 10:07:01 +00:00
Erik Johnston
0194e71e99
Merge branch 'develop' of github.com:matrix-org/synapse into get_pdu_limiting
2015-02-17 09:48:23 +00:00
Erik Johnston
baa5b9a975
Cache results of get_pdu.
2015-02-16 18:02:39 +00:00
Erik Johnston
91fc5eef1d
Mark old events as outliers.
...
This is to fix the issue where if a remote server sends an event
that references a really "old" event, then the local server will pull
that in and send to all clients.
We decide if an event is old if its depth is less than the minimum depth
of the room.
2015-02-16 14:27:40 +00:00
Erik Johnston
6138584651
Don't return anything from _handle_new_pdu, since we ignore the return value anyway
2015-02-16 14:08:02 +00:00
Erik Johnston
789251afa7
Fix logging
2015-02-12 19:29:43 +00:00
Erik Johnston
58d848adc0
Parrellize fetching of events
2015-02-12 18:35:36 +00:00
Erik Johnston
963256638d
Correctly handle all the places that can throw exceptions
2015-02-12 18:17:11 +00:00
Erik Johnston
c52e8d395b
Merge pull request #61 from matrix-org/timeout-federation-requests
...
Timeout federation requests
2015-02-11 17:10:33 +00:00
Erik Johnston
ef276e8770
Fix so timing out connections to actually work.
2015-02-11 16:48:05 +00:00
Erik Johnston
ddb816cf60
Don't unfreeze when using FreezeEvent.get_dict, as we are using a JSONEncoder that understands FrozenDict
2015-02-11 15:44:28 +00:00
Erik Johnston
4ebbaf0d43
Blunty replace json with simplejson
2015-02-11 14:23:10 +00:00
Erik Johnston
c8e1da930d
Log all the exits from _attempt_new_transaction
2015-02-10 17:30:46 +00:00
Erik Johnston
697ab75a34
Sign auth_chains when returned by /state/ requests
2015-02-10 15:46:24 +00:00
Erik Johnston
e9c85a4d5a
Connection errors in twisted aren't RuntimeErrors
2015-02-05 13:50:15 +00:00
Erik Johnston
e1515c3e91
Pass through list of room hosts from room alias query to federation so that it can retry against different room hosts
2015-02-05 13:44:42 +00:00
Erik Johnston
ae46f10fc5
Apply sanity to the transport client interface. Convert 'make_join' and 'send_join' to accept iterables of destinations
2015-02-04 16:28:12 +00:00
Erik Johnston
ff78eded01
Retry make_join
2015-02-04 13:55:10 +00:00
Erik Johnston
3c39f42a05
New line
2015-02-03 16:14:19 +00:00
Erik Johnston
9bace3a367
Actually, the old prune_event function was non-deterministic, so no point keeping it around :(
2015-02-03 15:32:17 +00:00
Erik Johnston
8dae5c8108
Remove unused imports
2015-02-03 15:01:12 +00:00
Erik Johnston
7b810e136e
Add new FederationBase
2015-02-03 15:00:42 +00:00
Erik Johnston
0dd3aea319
Keep around the old (buggy) version of the prune_event function so that we can use it to check signatures for events on old servers
2015-02-03 14:58:30 +00:00
Erik Johnston
0f48e22ef6
PEP8
2015-02-03 10:43:29 +00:00
Erik Johnston
40c6fe1b81
Don't bother requesting PDUs with bad signatures from the same server
2015-02-02 17:06:37 +00:00
Erik Johnston
941f59101b
Don't fail an entire request if one of the returned events fails a signature check. If an event does fail a signature check, look in the local database and request it from the originator.
2015-02-02 16:56:01 +00:00
Erik Johnston
776ac820f9
Briefly doc structure of query_auth API.
2015-01-30 15:58:28 +00:00
Erik Johnston
a70a801184
Fix bug where we superfluously asked for current state. Change API of /query_auth/ so that we don't duplicate events in the response.
2015-01-30 13:34:01 +00:00
Erik Johnston
c1d860870b
Fix regression where we no longer correctly handled the case of gaps in our event graph
2015-01-30 10:48:47 +00:00
Erik Johnston
78015948a7
Initial implementation of auth conflict resolution
2015-01-29 16:52:33 +00:00
Erik Johnston
0ef5bfd6a9
Start implementing auth conflict res
2015-01-28 16:16:53 +00:00
Erik Johnston
c92d64a6c3
Make it the responsibility of the replication layer to check signature and hashes.
2015-01-26 14:33:11 +00:00
Erik Johnston
7b88619241
Split up replication_layer module into client, server and transaction queue
2015-01-26 10:45:24 +00:00
Erik Johnston
ca65a9d03e
Split out TransactionQueue from replication layer
2015-01-22 16:37:08 +00:00
Mark Haines
5fed042640
Finish renaming "context" to "room_id" in federation codebase
2015-01-16 19:01:03 +00:00
Mark Haines
2408c4b0a4
Fold _do_request_for_transaction into the methods that called it since it was a trivial wrapper around client.get_json
2015-01-16 19:01:03 +00:00
Mark Haines
602684eac5
Split transport layer into client and server parts
2015-01-16 19:01:03 +00:00
Matrix
0529a7e2e9
Add some logging for when we are sending transactions.
2015-01-06 14:06:25 +00:00
Mark Haines
adb04b1e57
Update copyright notices
2015-01-06 13:21:39 +00:00
Mark Haines
041ac476a5
Supply auth_chain along with current state in '/state/', fetch auth events from a remote server if we are missing some of them
2014-12-18 18:47:13 +00:00
Erik Johnston
52f99243ab
Use is_outlier() so that we don't get AttributeError
2014-12-16 18:33:50 +00:00
Erik Johnston
3c77d13aa5
Kill off synapse.api.events.*
2014-12-16 11:29:05 +00:00
Erik Johnston
aae8a37e63
Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor
2014-12-10 13:18:40 +00:00
Erik Johnston
95aa903ffa
Try and figure out how and why signatures are being changed.
2014-12-10 11:37:47 +00:00
Erik Johnston
b8d30899b1
Code style.
2014-12-10 10:16:09 +00:00
Matthew Hodgson
71da2bed55
plateau retries after 1h
2014-12-10 00:18:44 +00:00
Matthew Hodgson
faf12b64f8
add errbacks to enqueue_pdu deferreds; change logging for failed federation sends to warn rather than exception
2014-12-10 00:12:51 +00:00
Matthew Hodgson
2b1acb7671
squidge to 79 columns as per pep8
2014-12-10 00:03:55 +00:00
Matthew Hodgson
8529fba02d
fix a million stupid bugs and make it actually work
2014-12-08 19:34:51 +00:00
Erik Johnston
ee3df06183
More bug fixes
2014-12-08 14:50:48 +00:00
Erik Johnston
d044121168
Various typos and bug fixes.
2014-12-08 09:08:26 +00:00
Matthew Hodgson
9c43b258ec
actually reset retry schedule if we can successfuly talk to it
2014-12-08 00:17:12 +00:00
Matthew Hodgson
5cd43d4b9f
fix stupid syntax thinkos
2014-12-07 23:44:16 +00:00
Matthew Hodgson
aed62a3583
track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover.
2014-12-07 02:26:07 +00:00
Erik Johnston
c31dba86ec
Convert rest and handlers to use new event structure
2014-12-04 15:50:01 +00:00
Erik Johnston
5d7c9ab789
Begin converting things to use the new Event structure
2014-12-04 11:27:59 +00:00
Erik Johnston
d06dfc70b0
Fix bug where did not always resolve all the deferreds in _attempt_new_transaction
2014-12-03 11:50:23 +00:00
Erik Johnston
07699b5871
Change the way we get missing auth and state events
2014-11-27 14:31:43 +00:00
Erik Johnston
3598c11c8d
Correctly handle the case where we get an event for an unknown room, which turns out we are actually in
2014-11-26 10:41:08 +00:00
Erik Johnston
64fc859dac
Fix bugs in invite/join dances.
...
We now do more implement more of the auth on the events so that we
don't reject valid events.
2014-11-25 17:59:49 +00:00
Erik Johnston
4961a4fab1
Mark the auth events as possible outlier
2014-11-24 13:55:49 +00:00
Erik Johnston
2bca242fdc
Ask for any auth events that we don't have
2014-11-24 13:46:41 +00:00
Erik Johnston
4bd0ab76c6
We don't always want to Auth get_persisted_pdu
2014-11-24 12:56:17 +00:00
Mark Haines
db9ce032a4
Fix pep8 codestyle warnings
2014-11-20 17:26:36 +00:00
Mark Haines
32090aee16
Add a few missing yields, Move deferred lists inside PreserveLoggingContext because they don't interact well with the logging contexts
2014-11-20 16:24:00 +00:00
Erik Johnston
4eada9a908
Fix backfill request
2014-11-19 17:22:37 +00:00
Mark Haines
c5eabe3143
replace user_id with sender
2014-11-19 16:38:40 +00:00
Mark Haines
428581dd05
SYN-144: Remove bad keys from pdu json objects, convert age_ts to age
...
for all pdus sent.
2014-11-18 19:20:25 +00:00
Mark Haines
db7e8b5619
SYN-141: Decode the query params as UTF-8
2014-11-18 17:17:57 +00:00
Mark Haines
cb4b6c844a
Merge PDUs and Events into one object
2014-11-14 21:25:02 +00:00
Mark Haines
8d8a133c89
SYN-103: Remove "origin" and "destination" keys from edus
2014-11-13 15:49:03 +00:00
Erik Johnston
3db0efa69f
Fix pyflake warnings and add a FIXME comment to deal with auth_chains received when joining
2014-11-12 11:27:02 +00:00
Erik Johnston
6fea478d2e
Fix bugs with invites/joins across federatiom.
...
Both in terms of auth and not trying to fetch missing PDUs for invites,
joins etc.
2014-11-12 11:24:11 +00:00
Erik Johnston
092979b8cc
Fix bugs which broke federation due to changes in function signatures.
2014-11-11 14:19:13 +00:00
Erik Johnston
5d439b127b
PEP8
2014-11-10 13:46:44 +00:00
Erik Johnston
c46088405a
Remove useless comments
2014-11-10 13:39:33 +00:00
Erik Johnston
003668cfaa
Add auth to the various server-server APIs
2014-11-10 13:37:24 +00:00
Erik Johnston
6447db063a
Fix backfill to work. Add auth to backfill request
2014-11-10 11:59:51 +00:00
Erik Johnston
1c06806f90
Finish redaction algorithm.
2014-11-10 10:21:32 +00:00
Erik Johnston
02c3b1c9e2
Add '/event_auth/' federation api
2014-11-07 15:35:53 +00:00
Erik Johnston
d2fb2b8095
Implement invite part of invite join dance
2014-11-07 13:41:00 +00:00
Erik Johnston
328dab2463
Remove /context/ request
2014-11-07 11:40:38 +00:00
Erik Johnston
3b4dec442d
Return auth chain when handling send_join
2014-11-07 11:22:12 +00:00
Erik Johnston
a5a4ef3fd7
Fix bug in replication
2014-11-04 15:16:43 +00:00
Erik Johnston
fc7b2b11a2
PEP8
2014-11-04 15:09:34 +00:00
Erik Johnston
440cbd5235
Add support for sending failures
2014-11-04 14:17:55 +00:00
Erik Johnston
d7412c4df1
Remove unused interface
2014-11-04 14:16:19 +00:00
Erik Johnston
aa76bf39ab
Remove unused imports
2014-11-04 14:14:02 +00:00
Erik Johnston
68698e0ac8
Fix bugs in generating event signatures and hashing
2014-11-03 17:51:42 +00:00
Erik Johnston
ad6eacb3e9
Rename PDU fields to match that of events.
2014-11-03 13:06:58 +00:00
Erik Johnston
d59aa6af25
For now, don't store txn -> pdu mappings.
2014-11-03 11:35:19 +00:00
Erik Johnston
2f39dc19a2
Remove more references to dead PDU tables
2014-10-31 14:27:14 +00:00
Erik Johnston
bfa36a72b9
Remove PDU tables.
2014-10-31 14:00:32 +00:00
Erik Johnston
21fe249d62
Actually don't store any PDUs
2014-10-31 10:47:34 +00:00
