Commit Graph

198 Commits

Author SHA1 Message Date
Shay
7b7478e8b6
Batch up notifications after event persistence () 2022-10-05 10:12:48 -07:00
David Robertson
0eb7e69768
Notifier: accept callbacks to fire on room joins () 2022-07-13 19:48:24 +01:00
Erik Johnston
e3163e2e11
Reduce the amount of state we pull from the DB () 2022-06-06 09:24:12 +01:00
Erik Johnston
1e453053cb
Rename storage classes () 2022-05-31 12:17:50 +00:00
Patrick Cloke
c52abc1cfd
Additional constants for EDU types. ()
Instead of hard-coding strings in many places.
2022-05-27 07:14:36 -04:00
Andrew Morgan
83be72d76c
Add StreamKeyType class and replace string literals with constants () 2022-05-16 15:35:31 +00:00
Erik Johnston
8dd3e0e084
Immediately retry any requests that have backed off when a server comes back online. ()
Otherwise it can take up to a minute for any in-flight `/send` requests to be retried.
2022-05-10 10:39:54 +01:00
Richard van der Hoff
e24ff8ebe3
Remove HomeServer.get_datastore() ()
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of 
2022-02-23 11:04:02 +00:00
Andrew Morgan
87f2005713
Add some tests for propagation of device list changes between local users () 2022-02-15 11:27:56 +00:00
Patrick Cloke
a121507cfe
Adds misc missing type hints () 2022-02-11 07:20:16 -05:00
Andrew Morgan
64ec45fc1b
Send to-device messages to application services ()
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-02-01 14:13:38 +00:00
Richard van der Hoff
251b5567ec
Remove log_function and its uses ()
I've never found this terribly useful. I think it was added in the early days
of Synapse, without much thought as to what would actually be useful to log,
and has just been cargo-culted ever since.

Rather, it tends to clutter up debug logs with useless information.
2022-01-18 13:06:04 +00:00
Patrick Cloke
10a88ba91c
Use auto_attribs/native type hints for attrs classes. () 2022-01-13 13:49:28 +00:00
Patrick Cloke
323151b787
Convert EventStreamResult to attrs. () 2021-12-15 11:10:02 -05:00
Andrew Morgan
c9c3aea9b1
Fix providing a RoomStreamToken instance to _notify_app_services_ephemeral ()
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-02 10:39:02 +00:00
Brendan Abolivier
c7a5e49664
Implement an on_new_event callback ()
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-10-26 15:17:36 +02:00
Andrew Morgan
6408372234
Improve docstrings for methods related to sending EDUs to application services () 2021-10-21 17:42:25 +01:00
Patrick Cloke
4054dfa409
Add type hints for event streams. () 2021-09-21 13:34:26 -04:00
Richard van der Hoff
858363d0b7
Generics for ObservableDeferred ()
Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to
follow suit.
2021-07-28 19:55:50 +00:00
Jonathan de Jong
bf72d10dbf
Use inline type hints in various other places (in synapse/) () 2021-07-15 11:02:43 +01:00
Richard van der Hoff
1d143074c5
Improve opentracing annotations for Notifier ()
The existing tracing reports an error each time there is a timeout, which isn't
really representative.

Additionally, we log things about the way `wait_for_events` works
(eg, the result of the callback) to the *parent* span, which is confusing.
2021-06-03 16:01:30 +01:00
Richard van der Hoff
b378d98c8f
Add debug logging for issue ()
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
2021-05-11 11:04:03 +01:00
Richard van der Hoff
294c675033
Remove synapse.types.Collection ()
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22 16:43:50 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ()
Part of 

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes ()
Part of 

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Jonathan de Jong
e2b8a90897
Update mypy configuration: no_implicit_optional = True () 2021-04-05 09:10:18 -04:00
Erik Johnston
33548f37aa
Improve tracing for to device messages () 2021-04-01 17:08:21 +01:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase ()
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Erik Johnston
a03d71dc9d
Fix "Starting metrics collection from sentinel context" errors () 2021-01-08 14:33:53 +00:00
Patrick Cloke
be2db93b3c
Do not assume that the contents dictionary includes history_visibility. () 2020-12-16 08:46:37 -05:00
Erik Johnston
a6ea1a957e
Don't pull event from DB when handling replication traffic. ()
I was trying to make it so that we didn't have to start a background task when handling RDATA, but that is a bigger job (due to all the code in `generic_worker`). However I still think not pulling the event from the DB may help reduce some DB usage due to replication, even if most workers will simply go and pull that event from the DB later anyway.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-10-28 12:11:45 +00:00
Erik Johnston
2b7c180879
Start fewer opentracing spans ()
 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.

This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.

I don't really know how much this will help.
2020-10-26 09:30:19 +00:00
Patrick Cloke
34a5696f93
Fix typos and spelling errors. () 2020-10-23 12:38:40 -04:00
Will Hunt
c276bd9969
Send some ephemeral events to appservices ()
Optionally sends typing, presence, and read receipt information to appservices.
2020-10-15 12:33:28 -04:00
Erik Johnston
921a3f8a59
Fix not sending events over federation when using sharded event persisters ()
* Fix outbound federaion with multiple event persisters.

We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.

Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.

* Change some interfaces to use RoomStreamToken.

By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
2020-10-14 13:27:51 +01:00
Patrick Cloke
a93f3121f8
Add type hints to some handlers () 2020-10-09 07:20:51 -04:00
Erik Johnston
ea70f1c362
Various clean ups to room stream tokens. () 2020-09-29 21:48:33 +01:00
Erik Johnston
ac11fcbbb8
Add EventStreamPosition type ()
The idea is to remove some of the places we pass around `int`, where it can represent one of two things:

1. the position of an event in the stream; or
2. a token that partitions the stream, used as part of the stream tokens.

The valid operations are then:

1. did a position happen before or after a token;
2. get all events that happened before or after a token; and
3. get all events between two tokens.

(Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
2020-09-24 13:24:17 +01:00
Patrick Cloke
aec294ee0d
Use slots in attrs classes where possible ()
slots use less memory (and attribute access is faster) while slightly
limiting the flexibility of the class attributes. This focuses on objects
which are instantiated "often" and for short periods of time.
2020-09-14 12:50:06 -04:00
Erik Johnston
fe8ed1b46f
Make StreamToken.room_key be a RoomStreamToken instance. () 2020-09-11 12:22:55 +01:00
Erik Johnston
5d3e306d9f
Clean up Notifier.on_new_room_event code path ()
The idea here is that we pass the `max_stream_id` to everything, and only use the stream ID of the particular event to figure out *when* the max stream position has caught up to the event and we can notify people about it.

This is to maintain the distinction between the position of an item in the stream (i.e. event A has stream ID 513) and a token that can be used to partition the stream (i.e. give me all events after stream ID 352). This distinction becomes important when the tokens are more complicated than a single number, which they will be once we start tracking the position of multiple writers in the tokens.

The valid operations here are:

1. Is a position before or after a token
2. Fetching all events between two tokens
3. Merging multiple tokens to get the "max", i.e. `C = max(A, B)` means that for all positions P where P is before A *or* before B, then P is before C.

Future PR will change the token type to a dedicated type.
2020-09-10 13:24:43 +01:00
Erik Johnston
0f545e6b96
Clean up types for PaginationConfig ()
This removes `SourcePaginationConfig` and `get_pagination_rows`. The reasoning behind this is that these generic classes/functions erased the types of the IDs it used (i.e. instead of passing around `StreamToken` it'd pass in e.g. `token.room_key`, which don't have uniform types).
2020-09-08 15:00:17 +01:00
Patrick Cloke
c619253db8
Stop sub-classing object () 2020-09-04 06:54:56 -04:00
Erik Johnston
9d1e4942ab
Fix typing for notifier () 2020-08-12 14:03:08 +01:00
Erik Johnston
a1e9bb9eae
Add typing info to Notifier () 2020-08-11 19:40:02 +01:00
Patrick Cloke
e19de43eb5
Convert streams to async. () 2020-08-04 07:21:47 -04:00
Patrick Cloke
38e1fac886
Fix some spelling mistakes / typos. () 2020-07-09 09:52:58 -04:00
Erik Johnston
1a1da60ad2
Fix new flake8 errors () 2020-05-12 11:20:48 +01:00
Patrick Cloke
b0cbc57375
Convert the synapse.notifier module to async/await. () 2020-05-01 15:14:49 -04:00
Erik Johnston
3eab76ad43
Don't relay REMOTE_SERVER_UP cmds to same conn. ()
For direct TCP connections we need the master to relay REMOTE_SERVER_UP
commands to the other connections so that all instances get notified
about it. The old implementation just relayed to all connections,
assuming that sending back to the original sender of the command was
safe. This is not true for redis, where commands sent get echoed back to
the sender, which was causing master to effectively infinite loop
sending and then re-receiving REMOTE_SERVER_UP commands that it sent.

The fix is to ensure that we only relay to *other* connections and not
to the connection we received the notification from.

Fixes .
2020-04-29 14:10:59 +01:00