Erik Johnston
ce6d4914f4
Correctly wire in update group profile over federation
2017-10-23 15:21:24 +01:00
Erik Johnston
011d03a0f6
Fix typo
2017-10-19 11:22:48 +01:00
Erik Johnston
9ab859f27b
Fix typo in group attestation handling
2017-10-19 10:55:52 +01:00
Richard van der Hoff
582bd19ee9
Fix 500 error when we get an error handling a PDU
...
FederationServer doesn't have a send_failure (and nor does its subclass,
ReplicationLayer), so this was failing.
I'm not really sure what the idea behind send_failure is, given (a) we don't do
anything at the other end with it except log it, and (b) we also send back the
failure via the transaction response. I suspect there's a whole lot of dead
code around it, but for now I'm just removing the broken bit.
2017-10-17 20:52:40 +01:00
Luke Barnard
85f5674e44
Delint
2017-10-16 15:52:17 +01:00
Luke Barnard
2c5972f87f
Implement GET /groups/$groupId/invited_users
2017-10-16 15:31:11 +01:00
Richard van der Hoff
8dd0c85ac5
Merge pull request #2529 from matrix-org/rav/fix_transaction_failure_handling
...
log pdu_failures from incoming transactions
2017-10-11 17:29:14 +01:00
Erik Johnston
271f5601f3
Fix typo in invite to group
2017-10-11 14:45:33 +01:00
Richard van der Hoff
b75d443caf
log pdu_failures from incoming transactions
...
... even if we have no EDUs.
This appears to have been introduced in
476899295f
2017-10-11 14:36:13 +01:00
Erik Johnston
535cc49f27
Merge pull request #2466 from matrix-org/erikj/groups_merged
...
Initial Group Implementation
2017-10-11 13:20:07 +01:00
Richard van der Hoff
6a6cc27aee
fed server: process PDUs for different rooms in parallel
...
With luck, this will give a real-time improvement when there are many rooms and
the server ends up calling out to fetch missing events.
2017-10-09 18:30:31 +01:00
Richard van der Hoff
4c7c4d4061
Fed server: use a linearizer for ongoing transactions
...
We don't want to process the same transaction multiple times concurrently, so
use a linearizer.
2017-10-09 18:30:10 +01:00
Richard van der Hoff
ba5b9b80a5
fed server: refactor on_incoming_transaction
...
Move as much as possible to after the have_responded check, and reduce the
number of times we iterate over the pdu list.
2017-10-09 18:10:53 +01:00
Richard van der Hoff
a6e3222fe5
Fed server: Move origin-check code to _handle_received_pdu
...
The response-building code expects there to be an entry in the `results` list
for each entry in the pdu_list, so the early `continue` was messing this
up. That doesn't really matter, because all that the federation client does is
log any errors, but it's pretty poor form.
2017-10-09 17:53:32 +01:00
Richard van der Hoff
01bbacf3c4
Fix up logcontext handling in (federation) TransactionQueue
...
Avoid using preserve_context_over_function, which has problems with respect to
logcontexts.
2017-10-06 22:39:25 +01:00
David Baker
27955056e0
Merge branch 'develop' into erikj/groups_merged
2017-10-02 16:20:41 +01:00
David Baker
6cd5fcd536
Make the spam checker a module
2017-09-26 19:20:23 +01:00
Erik Johnston
17b8e2bd02
Add remove room API
2017-09-26 15:52:41 +01:00
Richard van der Hoff
6de74ea6d7
Fix logcontexts in _check_sigs_and_hashes
2017-09-20 01:32:42 +01:00
Richard van der Hoff
fcf2c0fd1a
Remove redundant preserve_fn
...
preserve_fn is a no-op unless the wrapped function returns a
Deferred. verify_json_objects_for_server returns a list, so this is doing
nothing.
2017-09-20 01:32:42 +01:00
Richard van der Hoff
5ed109d59f
PoC for filtering spammy events ( #2456 )
...
Demonstration of how you might add some hooks to filter out spammy events.
2017-09-19 12:20:11 +01:00
Erik Johnston
ef8e578677
Add bulk group publicised lookup API
2017-08-09 13:36:22 +01:00
Erik Johnston
6f443a74cf
Add update group profile API
2017-07-20 09:46:33 +01:00
Erik Johnston
14a34f12d7
Comments
2017-07-18 17:28:42 +01:00
Erik Johnston
3431ec55dc
Comments
2017-07-18 17:23:50 +01:00
Erik Johnston
94ecd871a0
Fix typos
2017-07-18 16:38:54 +01:00
Erik Johnston
12ed4ee48e
Correctly parse query params
2017-07-18 15:33:09 +01:00
Erik Johnston
e5ea6dd021
Add client apis
2017-07-18 14:37:06 +01:00
Erik Johnston
cccfcfa7b9
Comments
2017-07-18 10:35:18 +01:00
Erik Johnston
2f9eafdd36
Add local group server support
2017-07-17 12:03:49 +01:00
Erik Johnston
cb3aee8219
Ensure category and role ids are non-null
2017-07-14 14:06:55 +01:00
Erik Johnston
8575e3160f
Comments
2017-07-13 13:52:41 +01:00
Erik Johnston
7a39da8cc6
Add summary APIs to federation
2017-07-13 11:13:19 +01:00
Erik Johnston
a62406aaa5
Add group summary APIs
2017-07-12 11:36:15 +01:00
Erik Johnston
fe4e885f54
Add federation API for adding room to group
2017-07-11 14:35:07 +01:00
Erik Johnston
6322fbbd41
Comment
2017-07-11 11:52:03 +01:00
Erik Johnston
b8ca494ee9
Initial group server implementation
2017-07-10 15:44:15 +01:00
Erik Johnston
5e49a57ecc
Separate federation servlet into different lists
2017-07-05 14:32:24 +01:00
Erik Johnston
6e2a7ee1bc
Remove spurious log lines
2017-06-07 11:05:17 +01:00
Erik Johnston
dfbda5e025
Faster cache for get_joined_hosts
2017-05-25 17:24:44 +01:00
Erik Johnston
ec5c4499f4
Make presence use cached users/hosts in room
2017-05-16 16:01:43 +01:00
Richard van der Hoff
de042b3b88
Do some logging when one-time-keys get claimed
...
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
2017-05-09 19:04:56 +01:00
Erik Johnston
27c28eaa27
Merge pull request #2190 from matrix-org/erikj/mark_remote_as_back_more
...
Always mark remotes as up if we receive a signed request from them
2017-05-05 14:08:12 +01:00
Erik Johnston
310b1ccdc1
Use preserve_fn and add logs
2017-05-05 13:41:19 +01:00
Erik Johnston
db7d0c3127
Always mark remotes as up if we receive a signed request from them
2017-05-05 10:34:53 +01:00
Erik Johnston
7166854f41
Add cache for get_current_hosts_in_room
2017-05-02 10:36:35 +01:00
Richard van der Hoff
91b3981800
Try harder when sending leave events
...
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
2017-04-21 01:50:36 +01:00
Richard van der Hoff
4903ccf159
Fix some lies, and other clarifications, in docstrings
...
The documentation on get_json has been wrong ever since the very first commit
to synapse...
2017-04-21 01:31:09 +01:00
Erik Johnston
247c736b9b
Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl
...
Reduce federation replication traffic
2017-04-12 11:07:13 +01:00
Paul Evans
8fbc0d29ee
Merge pull request #2121 from matrix-org/paul/sent-transactions-metric
...
Add a counter metric for successfully-sent transactions
2017-04-12 11:04:31 +01:00
Erik Johnston
26ae5178a4
Add some comments
2017-04-12 10:36:29 +01:00
Erik Johnston
1745069543
Comment
2017-04-12 10:17:10 +01:00
Erik Johnston
c7ddb5ef7a
Reuse get_interested_parties
2017-04-12 10:16:26 +01:00
Paul "LeoNerd" Evans
11dbceb761
Add a counter metric for successfully-sent transactions
2017-04-11 17:16:12 +01:00
Erik Johnston
a8c8e4efd4
Comment
2017-04-11 15:35:49 +01:00
Erik Johnston
2be8a281d2
Comments
2017-04-11 15:28:24 +01:00
Erik Johnston
6308ac45b0
Move get_interested_remotes back to presence handler
2017-04-11 15:19:26 +01:00
Erik Johnston
b9b72bc6e2
Comments
2017-04-11 15:15:34 +01:00
Erik Johnston
84fbb80c8f
Use generators
2017-04-10 16:55:56 +01:00
Erik Johnston
40453b3f84
Dedupe KeyedEdu and Devices federation repl traffic
2017-04-10 16:49:51 +01:00
Erik Johnston
29574fd5b3
Reduce federation presence replication traffic
...
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
Erik Johnston
0018491af2
Rename variable
2017-04-10 12:44:43 +01:00
Erik Johnston
8c5f03cec7
Revert to sending the same data type as before
2017-04-10 10:07:18 +01:00
Erik Johnston
f8434db549
Change name
2017-04-10 10:03:07 +01:00
Erik Johnston
ab904caf33
Comments
2017-04-10 10:02:17 +01:00
Erik Johnston
a828a64b75
Comment
2017-04-07 11:54:03 +01:00
Erik Johnston
d4d176e5d0
Add logging
2017-04-07 11:51:28 +01:00
Erik Johnston
449d1297ca
Fix up federation SendQueue and document types
2017-04-07 11:48:33 +01:00
Erik Johnston
f10ce8944b
Don't double json encode federation replication data
2017-04-05 11:10:28 +01:00
Erik Johnston
62b89daac6
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server
2017-04-04 09:46:16 +01:00
Richard van der Hoff
30bcbf775a
Accept join events from all servers
...
Make sure that we accept join events from any server, rather than just the
origin server, to make the federation join dance work correctly.
(Fixes #1893 ).
2017-04-03 15:58:07 +01:00
Erik Johnston
11880103b1
Make federation send queue take the current position
2017-03-30 12:54:36 +01:00
Erik Johnston
85be3dde81
Bail early if remote wouldn't be retried ( #2064 )
...
* Bail early if remote wouldn't be retried
* Don't always return true
* Just use get_retry_limiter
* Spelling
2017-03-29 11:48:27 +01:00
Erik Johnston
c5b0bdd542
Merge pull request #2067 from matrix-org/erikj/notify_on_fed
...
Notify on new federation traffic
2017-03-29 11:41:37 +01:00
Erik Johnston
35b4aa04be
Notify on new federation traffic
2017-03-27 14:07:47 +01:00
Erik Johnston
2a28b79e04
Batch sending of device list pokes
2017-03-24 14:44:49 +00:00
Richard van der Hoff
5a16cb4bf0
Ignore backoff history for invites, aliases, and roomdirs
...
Add a param to the federation client which lets us ignore historical backoff
data for federation queries, and set it for a handful of operations.
2017-03-23 12:23:22 +00:00
Richard van der Hoff
4bd597d9fc
push federation retry limiter down to matrixfederationclient
...
rather than having to instrument everywhere we make a federation call,
make the MatrixFederationHttpClient manage the retry limiter.
2017-03-23 09:28:46 +00:00
Richard van der Hoff
29ed09e80a
Fix assertion to stop transaction queue getting wedged
...
... and update some docstrings to correctly reflect the types being used.
get_new_device_msgs_for_remote can return a long under some circumstances,
which was being stored in last_device_list_stream_id_by_dest, and was then
upsetting things on the next loop.
2017-03-15 12:16:55 +00:00
Richard van der Hoff
29235901b8
Move FederationServer._handle_new_pdu to FederationHandler
...
Unfortunately this significantly increases the size of the already-rather-big
FederationHandler, but the code fits more naturally here, and it paves the way
for the tighter integration that I need between handling incoming PDUs and
doing the join dance.
Other than renaming the existing `FederationHandler.on_receive_pdu` to
`_process_received_pdu` to make way for it, this just consists of the move, and
replacing `self.handler` with `self` and `self` with `self.replication_layer`.
2017-03-09 16:20:13 +00:00
Richard van der Hoff
e8b1721290
Move sig check out of _handle_new_pdu
...
When we receive PDUs via `get_missing_events`, we have already checked their
sigs, so there is no need to do it again.
2017-03-09 15:50:44 +00:00
Richard van der Hoff
3406333a58
Factor _get_missing_events_for_pdu out of _handle_new_pdu
...
This should be functionally identical: it just seeks to improve readability by
reducing indentation.
2017-03-09 15:50:44 +00:00
Richard van der Hoff
45d173a59a
Fix docstring
2017-03-09 15:50:29 +00:00
Richard van der Hoff
0c4cf9372b
Fix a race in transaction queue
...
It was theoretically possible for a PDU to get queued and not sent for ages. On
closer inspection I think there were bigger problems elsewhere, but we might as
well fix this since it's easy.
2017-02-20 16:46:25 +00:00
Erik Johnston
df4ecff5a9
Correctly raise exceptions for ratelimitng. Ratelimit on 401
2017-02-01 15:42:19 +00:00
Erik Johnston
ae7a132f38
Better handle 404 response for federation /send/
2017-01-31 13:40:09 +00:00
Erik Johnston
c974116f19
Implement device key caching over federation
2017-01-26 16:07:24 +00:00
Erik Johnston
51e9fe36e4
Fix up sending of m.device_list_update edus
2017-01-25 16:55:21 +00:00
Erik Johnston
2367c5568c
Add basic implementation of local device list changes
2017-01-25 14:27:27 +00:00
Erik Johnston
fc7cae8aa3
Merge pull request #1824 from matrix-org/erikj/retry_host_log
...
Lower the not retrying host log line to debug
2017-01-18 11:07:51 +00:00
Erik Johnston
f878f64f43
Lower the not retrying host log line to debug
2017-01-17 17:20:39 +00:00
Erik Johnston
e6153e1bd1
Fix couple of federation state bugs
2017-01-17 13:22:34 +00:00
Erik Johnston
f7085ac84f
Name linearizer's for better logs
2017-01-09 17:17:10 +00:00
Matthew
4304e7e593
do the discard check in the right place to avoid grabbing dependent events
2017-01-07 03:44:18 +00:00
Matthew
e10c527930
Discard PDUs from invalid origins due to #1753 in 0.18.[56]
2017-01-07 02:13:14 +00:00
Matthew Hodgson
468749c9fc
fix comment
2017-01-05 12:00:11 +00:00
Matthew Hodgson
eedf400d05
limit total timeout for get_missing_events to 10s
2017-01-05 11:58:15 +00:00
Mark Haines
f784980d2b
Only send events that originate on this server.
...
Or events that are sent via the federation "send_join" API.
This should match the behaviour from before v0.18.5 and #1635 landed.
2017-01-05 11:26:30 +00:00
Mark Haines
e02bdaf08b
Get the destinations from the state from before the event
...
Rather than the state after then event.
2017-01-04 15:17:15 +00:00
Mark Haines
b6b67715ed
Send ALL membership events to the server that was affected.
...
Send all membership changes to the server that was affected.
This ensures that if the last member of a room on a server
was kicked or banned they get told about it.
2017-01-04 13:56:20 +00:00
Mark Haines
62ce3034f3
s/aquire/acquire/g
2016-12-30 20:04:44 +00:00
Mark Haines
0aff09f6c9
Add more useful logging when we block fetching events
2016-12-30 20:00:44 +00:00
Mark Haines
cc50b1ae53
Remove fallback from get_missing_events.
...
get_missing_events used to fallback to fetching the missing events
individually requesting from every server in the room, one by one.e
This could be unacceptably slow, possibly causing #1732
2016-12-30 18:13:15 +00:00
Erik Johnston
f32fb65552
Add new API appservice specific public room list
2016-12-06 16:12:27 +00:00
Erik Johnston
aaecffba3a
Correctly handle 500's and 429 on federation
2016-11-24 15:04:49 +00:00
Erik Johnston
feec718265
Shuffle receipt handler around so that worker apps don't need to load it
2016-11-23 15:14:24 +00:00
Erik Johnston
4c79a63fd7
Explicit federation ack
2016-11-23 10:40:44 +00:00
Erik Johnston
51e89709aa
Comments
2016-11-21 17:59:39 +00:00
Erik Johnston
88d85ebae1
Add some metrics
2016-11-21 17:36:05 +00:00
Erik Johnston
50934ce460
Comments
2016-11-21 16:55:23 +00:00
Erik Johnston
9687e039e7
Remove explicit calls to send_pdu
2016-11-21 14:48:51 +00:00
Erik Johnston
524d61bf7e
Fix tests
2016-11-21 11:53:02 +00:00
Erik Johnston
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
Erik Johnston
1587b5a033
Add initial cut of federation send queue
2016-11-16 14:47:52 +00:00
Erik Johnston
59ef517e6b
Use new federation_sender DI
2016-11-16 14:47:52 +00:00
Erik Johnston
847d5db1d1
Add transaction queue and transport layer to DI
2016-11-16 14:47:52 +00:00
Erik Johnston
daec6fc355
Move logic into transaction_queue
2016-11-16 14:47:52 +00:00
Erik Johnston
0e830d3770
Rename transaction queue functions to send_*
2016-11-16 14:47:52 +00:00
Erik Johnston
22578545a0
Time out typing over federation
2016-09-23 14:00:52 +01:00
Erik Johnston
23b6701a28
Support filtering remote room lists
2016-09-16 10:24:15 +01:00
Erik Johnston
5810cffd33
Pass since/from parameters over federation
2016-09-15 10:36:19 +01:00
Mark Haines
76b09c29b0
Merge pull request #1110 from matrix-org/markjh/e2e_timeout
...
Add a timeout parameter for end2end key queries.
2016-09-13 10:50:45 +01:00
Mark Haines
949c2c5435
Add a timeout parameter for end2end key queries.
...
Add a timeout parameter for controlling how long synapse will wait
for responses from remote servers. For servers that fail include how
they failed to make it easier to debug.
Fetch keys from different servers in parallel rather than in series.
Set the default timeout to 10s.
2016-09-12 18:17:09 +01:00
Erik Johnston
706b5d76ed
Fix backfill when cannot find an event.
...
`get_pdu` can succeed but return None.
2016-09-12 14:59:51 +01:00
Erik Johnston
af4701b311
Fix incorrect attribute name
2016-09-09 17:36:56 +01:00
Erik Johnston
464ffd1b5e
Comment
2016-09-09 17:17:23 +01:00
Erik Johnston
327425764e
Add edu.type as part of key. Remove debug logging
2016-09-09 17:13:30 +01:00
Erik Johnston
52b2318777
Clobber EDUs in send queue
2016-09-09 15:59:08 +01:00
Erik Johnston
ab80d5e0a9
Drop replication log levels
2016-09-09 14:56:50 +01:00
Erik Johnston
a6c6750166
Check if destination is ready for retry earlier
2016-09-09 13:46:05 +01:00
Erik Johnston
4598682b43
Fix tightloop on sending transaction
2016-09-09 13:12:53 +01:00
Erik Johnston
d2688d7f03
Correctly guard against multiple concurrent transactions
2016-09-09 11:44:36 +01:00
Erik Johnston
b390756150
Update last_device_stream_id_by_dest if there is nothing to send
2016-09-09 11:00:15 +01:00
Mark Haines
43954d000e
Add a new method to enqueue the device messages rather than sending a dummy EDU
2016-09-07 16:10:51 +01:00
Mark Haines
cb98ac261b
Move the check for federated device_messages.
...
Move the check into _attempt_new_transaction.
Only delete messages if there were messages to delete.
2016-09-07 15:39:13 +01:00
Mark Haines
31a07d2335
Add stream change caches for device messages
2016-09-07 15:27:07 +01:00
Mark Haines
d4a35ada28
Send device messages over federation
2016-09-06 18:16:20 +01:00
Erik Johnston
2854ee2a52
Only pull out IDs from DB for /state_ids/ request
2016-09-02 10:53:36 +01:00
Erik Johnston
f4778d4cd9
Merge branch 'erikj/pdu_check' of github.com:matrix-org/synapse into develop
2016-09-01 14:40:36 +01:00
Erik Johnston
516a272aca
Ensure we only return a validated pdu in get_pdu
2016-09-01 10:55:02 +01:00
Erik Johnston
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
Erik Johnston
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
Erik Johnston
c315922b5f
PEP8
2016-08-10 16:34:10 +01:00
Erik Johnston
ca8abfbf30
Clean up TransactionQueue
2016-08-10 16:24:16 +01:00
Erik Johnston
5aeadb7414
Merge pull request #999 from matrix-org/erikj/measure_more
...
Measure federation send transaction resources
2016-08-10 14:16:14 +01:00
Erik Johnston
487bc49bf8
Don't stop on 4xx series errors
2016-08-10 13:39:12 +01:00
Erik Johnston
ea8c4094db
Also pull out rejected events
2016-08-10 13:26:13 +01:00
Erik Johnston
f91df1f761
Store if we fail to fetch an event from a destination
2016-08-10 11:31:46 +01:00
