Commit Graph

111 Commits

Author SHA1 Message Date
David Robertson
c071cd5a0e
Ensure fed-sender catchup does not block for full state ()
* Reproduce bad scenario in test
* Avoid catchup optimisation for partial state rooms
2023-03-13 12:31:19 +00:00
David Robertson
4bb26c95a9
Refactor filter_events_for_server ()
* Tweak docstring and type hint

* Flip logic and provide better name

* Separate decision from action

* Track a set of strings, not EventBases

* Require explicit boolean options from callers

* Add explicit option for partial state rooms

* Changelog

* Rename param
2023-03-10 15:31:25 +00:00
Sean Quah
0a686d1d13
Faster joins: Refactor handling of servers in room ()
Ensure that the list of servers in a partial state room always contains
the server we joined off.

Also refactor `get_partial_state_servers_at_join` to return `None` when
the given room is no longer partial stated, to explicitly indicate when
the room has partial state. Otherwise it's not clear whether an empty
list means that the room has full state, or the room is partial stated,
but the server we joined off told us that there are no servers in the
room.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-02-03 15:39:59 +00:00
Richard van der Hoff
cb59e08062
Improve logging and opentracing for to-device message handling ()
A batch of changes intended to make it easier to trace to-device messages through the system.

The intention here is that a client can set a property org.matrix.msgid in any to-device message it sends. That ID is then included in any tracing or logging related to the message. (Suggestions as to where this field should be documented welcome. I'm not enthusiastic about speccing it - it's very much an optional extra to help with debugging.)

I've also generally improved the data we send to opentracing for these messages.
2022-12-06 09:52:55 +00:00
Mathieu Velten
4569eda944
Use servers list approx to send read receipts when in partial state ()
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-11-30 13:39:47 +01:00
Patrick Cloke
d748bbc8f8
Include thread information when sending receipts over federation. ()
Include the thread_id field when sending read receipts over
federation. This might result in the same user having multiple
read receipts per-room, meaning multiple EDUs must be sent
to encapsulate those receipts.

This restructures the PerDestinationQueue APIs to support
multiple receipt EDUs, queue_read_receipt now becomes linear
time in the number of queued threaded receipts in the room for
the given user, it is expected this is a small number since receipt
EDUs are sent as filler in transactions.
2022-11-28 14:40:17 +00:00
Mathieu Velten
39cde585bf
Faster joins: use initial list of servers if we don't have the full state yet ()
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-11-24 18:09:47 +01:00
Mathieu Velten
1526ff389f
Faster joins: filter out non local events when a room doesn't have its full state ()
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-11-21 16:46:14 +01:00
Erik Johnston
09b588854e
Fix TypeError: 'dict_keys' object is not reversible () 2022-10-24 13:05:14 +01:00
Shay
a86b2f6837
Fix a bug where redactions were not being sent over federation if we did not have the original event. () 2022-10-11 11:18:45 -07:00
Erik Johnston
299b00d968
Prioritize outbound to-device over device list updates ()
Otherwise device list changes for large accounts can temporarily delay to-device messages.
2022-09-27 15:17:41 +01:00
reivilibre
526f84bc2e
Fix Prometheus recording rules to not use legacy metric names. () 2022-09-08 15:01:42 +01:00
Erik Johnston
2318603772
Add some logging to help track down () 2022-09-01 13:54:52 +01:00
Nick Mills-Barrett
21eeacc995
Federation Sender & Appservice Pusher Stream Optimisations ()
* Replace `get_new_events_for_appservice` with `get_all_new_events_stream`

The functions were near identical and this brings the AS worker closer
to the way federation senders work which can allow for multiple workers
to handle AS traffic.

* Pull received TS alongside events when processing the stream

This avoids an extra query -per event- when both federation sender
and appservice pusher process events.
2022-07-15 09:36:56 +01:00
Erik Johnston
a7e506ddee
Reduce amount of state we pull out when attempting to send catchup PDUs. ()
* Don't pull out state for catchup

* Newsfile

* Merge newsfile
2022-06-07 14:35:56 +01:00
Erik Johnston
44de53bb79
Reduce state pulled from DB due to sending typing and receipts over federation ()
Reducing the amount of state we pull from the DB is useful as fetching state is expensive in terms of DB, CPU and memory.
2022-06-06 16:46:11 +01:00
Patrick Cloke
c52abc1cfd
Additional constants for EDU types. ()
Instead of hard-coding strings in many places.
2022-05-27 07:14:36 -04:00
Patrick Cloke
b5707ceaba
Avoid attempting to delete push actions for remote users. ()
Remote users will never have push actions, so we can avoid a database
round-trip/transaction completely.
2022-05-26 07:09:16 -04:00
Dirk Klimpel
6edefef602
Add some type hints to datastore () 2022-05-17 15:29:06 +01:00
Richard van der Hoff
d66d68f917
Add extra debug logging to federation sender ()
... in order to debug some problems we've been having with certain events not
being sent when expected.
2022-05-03 16:32:40 +01:00
Richard van der Hoff
db2edf5a65
Exclude OOB memberships from the federation sender ()
As the comment says, there is no need to process such events, and indeed we
need to avoid doing so.

Fixes .
2022-05-03 12:47:56 +00:00
Erik Johnston
423cca9efe
Spread out sending device lists to remote hosts () 2022-03-04 11:48:15 +00:00
Richard van der Hoff
e24ff8ebe3
Remove HomeServer.get_datastore() ()
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of 
2022-02-23 11:04:02 +00:00
Richard van der Hoff
a85dde3445
Minor typing fixes ()
These started failing in
https://github.com/matrix-org/synapse/pull/12031... I'm a bit mystified by how
they ever worked.
2022-02-21 18:37:04 +00:00
David Robertson
f160fe18e3
Debug for device lists updates ()
Debug for .

I'm having a hard time tracking down what's going wrong in that issue.
In the reported example, I could see server A sending federation traffic
to server B and all was well. Yet B reports out-of-sync device updates
from A.

I couldn't see what was _in_ the events being sent from A to B. So I
have added some crude logging to track

- when we have updates to send to a remote HS
- the edus we actually accumulate to send
- when a federation transaction includes a device list update edu
- when such an EDU is received

This is a bit of a sledgehammer.
2022-01-20 13:38:44 +00:00
Patrick Cloke
10a88ba91c
Use auto_attribs/native type hints for attrs classes. () 2022-01-13 13:49:28 +00:00
Patrick Cloke
d2279f471b
Add most of the missing type hints to synapse.federation. ()
This skips a few methods which are difficult to type.
2021-12-02 16:18:10 +00:00
reivilibre
75ca0a6168
Annotate log_function decorator ()
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-27 17:27:23 +01:00
Andrew Morgan
aa2c027792
Remove unnecessary parentheses around tuples returned from methods () 2021-09-23 11:59:07 +01:00
Patrick Cloke
8c7a531e27
Use direct references for some configuration variables (part 2) () 2021-09-15 08:34:52 -04:00
Patrick Cloke
01c88a09cd
Use direct references for some configuration variables ()
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
reivilibre
524b8ead77
Add types to synapse.util. () 2021-09-10 17:03:18 +01:00
Patrick Cloke
1de26b3467
Convert Transaction and Edu object to attrs ()
Instead of wrapping the JSON into an object, this creates concrete
instances for Transaction and Edu. This allows for improved type
hints and simplified code.
2021-08-06 09:39:59 -04:00
Erik Johnston
ac5c221208
Stagger send presence to remotes ()
This is to help with performance, where trying to connect to thousands
of hosts at once can consume a lot of CPU (due to TLS etc).

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-07-15 11:52:56 +01:00
Jonathan de Jong
bf72d10dbf
Use inline type hints in various other places (in synapse/) () 2021-07-15 11:02:43 +01:00
Richard van der Hoff
b378d98c8f
Add debug logging for issue ()
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
2021-05-11 11:04:03 +01:00
Andrew Morgan
4e0fd35bc9 Revert "Experimental Federation Speedup ()"
This reverts commit 05e8c70c05.
2021-04-28 11:38:33 +01:00
Richard van der Hoff
294c675033
Remove synapse.types.Collection ()
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22 16:43:50 +01:00
Erik Johnston
db70435de7
Fix bug where we sent remote presence states to remote servers () 2021-04-20 13:37:54 +01:00
Erik Johnston
2b7dd21655
Don't send normal presence updates over federation replication stream () 2021-04-19 10:50:49 +01:00
Richard van der Hoff
5a153772c1
remove HomeServer.get_config ()
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
2021-04-14 19:09:08 +01:00
Jonathan de Jong
05e8c70c05
Experimental Federation Speedup ()
This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max).

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-14 17:19:02 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ()
Part of 

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Erik Johnston
3a569fb200 Fix sharded federation sender sometimes using 100% CPU.
We pull all destinations requiring catchup from the DB in batches.
However, if all those destinations get filtered out (due to the
federation sender being sharded), then the `last_processed` destination
doesn't get updated, and we keep requesting the same set repeatedly.
2021-04-08 17:34:07 +01:00
Andrew Morgan
04819239ba
Add a Synapse Module for configuring presence update routing ()
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.

This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.

A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. 

The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:

* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06 14:38:30 +01:00
Erik Johnston
33548f37aa
Improve tracing for to device messages () 2021-04-01 17:08:21 +01:00
Patrick Cloke
da75d2ea1f
Add type hints for the federation sender. ()
Includes an abstract base class which both the FederationSender
and the FederationRemoteSendQueue must implement.
2021-03-29 11:43:20 -04:00
Erik Johnston
c602ba8336
Fixed undefined variable error in catchup ()
Broke in 

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-24 16:12:47 +00:00
Erik Johnston
dd71eb0f8a
Make federation catchup send last event from any server. ()
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.

Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
2021-03-18 15:52:26 +00:00
Erik Johnston
026503fa3b
Don't go into federation catch up mode so easily ()
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. .

Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
2021-03-15 14:42:40 +00:00