Commit Graph

736 Commits

Author SHA1 Message Date
Shay
8ab16a92ed
Persist CreateRoom events to DB in a batch () 2022-09-28 10:11:48 +00:00
Mathieu Velten
41461fd4d6
typing: check origin server of typing event against room's servers ()
This is also using the partial state approximation if needed so we do
not block here during a fast join.

Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-09-26 17:33:32 +02:00
Patrick Cloke
efd108b45d
Accept & store thread IDs for receipts (implement MSC3771). ()
Updates the `/receipts` endpoint and receipt EDU handler to parse a
`thread_id` from the body and insert it in the database.
2022-09-23 14:33:28 +00:00
reivilibre
c06b2b7142
Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. () 2022-09-23 11:47:16 +01:00
Erik Johnston
42d261c32f
Port the push rule classes to Rust. () 2022-09-20 12:10:31 +01:00
Sean Quah
b73cbb8215
Avoid putting rejected events in room state ()
Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-16 12:45:04 +01:00
Eric Eastwood
957e3d74fc
Keep track when we try and fail to process a pulled event ()
We can follow-up this PR with:

 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622
 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623
 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700

Fix https://github.com/matrix-org/synapse/issues/13621

Part of https://github.com/matrix-org/synapse/issues/13356

Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)
2022-09-14 13:57:50 -05:00
Sean Quah
c73774467e
Fix bug in device list caching when remote users leave rooms ()
When a remote user leaves the last room shared with the homeserver, we
have to mark their device list as unsubscribed, otherwise we would hold
on to a stale device list in our cache. Crucially, the device list would
remain cached even after the remote user rejoined the room, which could
lead to E2EE failures until the next change to the remote user's device
list.

Fixes .

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-09-14 10:42:57 +01:00
Šimon Brandner
0e99f07952
Remove support for unstable private read receipts ()
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-09-01 13:31:54 +01:00
Dirk Klimpel
682dfcfc0d
Fix that user cannot /forget rooms after the last member has left () 2022-08-30 09:58:38 +00:00
Quentin Gliech
3dd175b628
synapse.api.auth.Auth cleanup: make permission-related methods use Requester instead of the UserID ()
Part of 

This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.

It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
2022-08-22 14:17:59 +01:00
Andrew Morgan
40e3e68cd7
Register homeserver modules when creating test homeserver () 2022-08-19 16:52:20 +01:00
Patrick Cloke
f3fba4914d
Reduce the number of tests using TCP replication. ()
Uses Redis replication in additional test cases (instead of
TCP replication). A small step towards dropping TCP replication.
2022-08-19 08:25:24 -04:00
reivilibre
3a245f6cfe
Fix validation problem that occurs when a user tries to deactivate their account or change their password. () 2022-08-19 11:03:29 +00:00
Erik Johnston
5442891cbc
Make push rules use proper structures. ()
This improves load times for push rules:

| Version              | Time per user | Time for 1k users | 
| -------------------- | ------------- | ----------------- |
| Before               |       138 µs  |             138ms |
| Now (with custom)    |       2.11 µs |            2.11ms |
| Now (without custom) |       49.7 ns |           0.05 ms |

This therefore has a large impact on send times for rooms
with large numbers of local users in the room.
2022-08-16 12:22:17 +01:00
Šimon Brandner
ab18441573
Support stable identifiers for MSC2285: private read receipts. ()
This adds support for the stable identifiers of MSC2285 while
continuing to support the unstable identifiers behind the configuration
flag. These will be removed in a future version.
2022-08-05 11:09:33 -04:00
Dirk Klimpel
e2ed1b7155
Use literals in place of HTTPStatus constants in tests () 2022-08-05 16:59:09 +02:00
reivilibre
e17e5c97e0
Faster Room Joins: don't leave a stuck room partial state flag if the join fails. () 2022-08-01 16:45:39 +00:00
Sean Quah
224d792dd7
Refactor _resolve_state_at_missing_prevs to return an EventContext ()
Previously, `_resolve_state_at_missing_prevs` returned the resolved
state before an event and a partial state flag. These were unwieldy to
carry around would only ever be used to build an event context. Build
the event context directly instead.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-08-01 13:53:56 +01:00
Patrick Cloke
922b771337
Add missing type hints for tests.unittest. () 2022-07-27 17:18:41 +00:00
Sean Quah
335ebb21cc
Faster room joins: avoid blocking when pulling events with missing prevs ()
Avoid blocking on full state in `_resolve_state_at_missing_prevs` and
return a new flag indicating whether the resolved state is partial.
Thread that flag around so that it makes it into the event context.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-07-26 12:39:23 +01:00
Jan Schär
e8519e0ed2
Support Implicit TLS for sending emails ()
Previously, TLS could only be used with STARTTLS.
Add a new option `force_tls`, where TLS is used from the start.
Implicit TLS is recommended over STARTLS,
see https://datatracker.ietf.org/doc/html/rfc8314

Fixes .

Signed-off-by: Jan Schär <jan@jschaer.ch>
2022-07-25 16:27:19 +01:00
David Robertson
b977867358
Rate limit joins per-room () 2022-07-19 11:45:17 +00:00
Dirk Klimpel
96cf81e312
Use HTTPStatus constants in place of literals in tests. () 2022-07-15 19:31:27 +00:00
Nick Mills-Barrett
cc21a431f3
Async get event cache prep ()
Some experimental prep work to enable external event caching based on  & . Doesn't actually move the cache at all, just lays the groundwork for async implemented caches.

Signed off by Nick @ Beeper (@Fizzadar)
2022-07-15 09:30:46 +00:00
Nick Mills-Barrett
21eeacc995
Federation Sender & Appservice Pusher Stream Optimisations ()
* Replace `get_new_events_for_appservice` with `get_all_new_events_stream`

The functions were near identical and this brings the AS worker closer
to the way federation senders work which can allow for multiple workers
to handle AS traffic.

* Pull received TS alongside events when processing the stream

This avoids an extra query -per event- when both federation sender
and appservice pusher process events.
2022-07-15 09:36:56 +01:00
Richard van der Hoff
fe15a865a5
Rip out auth-event reconciliation code ()
There is a corner in `_check_event_auth` (long known as "the weird corner") where, if we get an event with auth_events which don't match those we were expecting, we attempt to resolve the diffence between our state and the remote's with a state resolution.

This isn't specced, and there's general agreement we shouldn't be doing it.

However, it turns out that the faster-joins code was relying on it, so we need to introduce something similar (but rather simpler) for that.
2022-07-14 21:52:26 +00:00
David Robertson
52a0c8f2f7
Rename test case method to add_hashes_and_signatures_from_other_server () 2022-07-12 18:46:32 +00:00
Richard van der Hoff
d4b1c0d800
Fix inconsistencies in event validation () 2022-06-17 16:30:59 +01:00
David Robertson
a4ae1406d1
Fix typechecks against twisted trunk () 2022-06-15 11:49:58 +01:00
Quentin Gliech
fe1daad672
Move the "email unsubscribe" resource, refactor the macaroon generator & simplify the access token verification logic. ()
This simplifies the access token verification logic by removing the `rights`
parameter which was only ever used for the unsubscribe link in email
notifications. The latter has been moved under the `/_synapse` namespace,
since it is not a standard API.

This also makes the email verification link more secure, by embedding the
app_id and pushkey in the macaroon and verifying it. This prevents the user
from tampering the query parameters of that unsubscribe link.

Macaroon generation is refactored:

- Centralised all macaroon generation and verification logic to the
  `MacaroonGenerator`
- Moved to `synapse.utils`
- Changed the constructor to require only a `Clock`, hostname, and a secret key
  (instead of a full `Homeserver`).
- Added tests for all methods.
2022-06-14 09:12:08 -04:00
Quentin Gliech
92103cb2c8
Decouple synapse.api.auth_blocking.AuthBlocking from synapse.api.auth.Auth. () 2022-06-14 09:51:15 +01:00
Patrick Cloke
53b77b203a
Replace noop background updates with DELETE. ()
Removes the `register_noop_background_update` and deletes the background
updates directly in a delta file.
2022-06-13 14:06:27 -04:00
Patrick Cloke
81608490e3
Stop depending on room_id to be returned for children state in the hierarchy response. ()
The `room_id` field was removed from MSC2946 before
it was accepted. It was initially kept for backwards compatibility
and should be removed now that the stable form of the API
is used.

This change only stops Synapse from validating that it is returned,
a future PR will remove returning it as part of the response.
2022-06-10 07:15:51 -04:00
Patrick Cloke
9dc3293e0b
Consolidate the logic of delete_device/delete_devices. ()
By always using delete_devices and sometimes passing a list
with a single device ID.

Previously these methods had gotten out of sync with each
other and it seems there's little benefit to the single-device
variant.
2022-06-07 07:43:35 -04:00
Erik Johnston
44de53bb79
Reduce state pulled from DB due to sending typing and receipts over federation ()
Reducing the amount of state we pull from the DB is useful as fetching state is expensive in terms of DB, CPU and memory.
2022-06-06 16:46:11 +01:00
Erik Johnston
e3163e2e11
Reduce the amount of state we pull from the DB () 2022-06-06 09:24:12 +01:00
Erik Johnston
888a29f412
Wait for lazy join to complete when getting current state () 2022-06-01 16:02:53 +01:00
Patrick Cloke
7bc08f3201
Remove remaining bits of groups code. ()
* Update worker docs to remove group endpoints.
* Removes an unused parameter to `ApplicationService`.
* Break dependency between media repo and groups.
* Avoid copying `m.room.related_groups` state events during room upgrades.
2022-06-01 09:41:25 -04:00
Erik Johnston
1e453053cb
Rename storage classes () 2022-05-31 12:17:50 +00:00
Patrick Cloke
c52abc1cfd
Additional constants for EDU types. ()
Instead of hard-coding strings in many places.
2022-05-27 07:14:36 -04:00
Patrick Cloke
1885ee0113
Remove unstable APIs for /hierarchy. ()
Removes the unstable endpoint as well as a duplicated field
which was modified during stabilization.
2022-05-26 07:10:28 -04:00
Erik Johnston
b83bc5fab5
Pull out less state when handling gaps mk2 () 2022-05-26 09:48:12 +00:00
Erik Johnston
4660d9fdcf
Fix up state_store naming () 2022-05-25 12:59:04 +01:00
Dirk Klimpel
50ae4eafe1
Add some type hints to event_federation datastore ()
Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-05-18 16:02:10 +01:00
David Robertson
d4713d3e33
Discard null-containing strings before updating the user directory () 2022-05-18 11:28:14 +01:00
Richard van der Hoff
24b590de32
Remove code which updates application_services_state.last_txn ()
This column is unused as of , so let's stop writing to it.
2022-05-17 11:07:18 +01:00
Erik Johnston
fcf951d5dc
Track in memory events using weakrefs () 2022-05-17 10:34:27 +01:00
Šimon Brandner
3ce15cc7be
Avoid unnecessary copies when filtering private read receipts. ()
A minor optimization to avoid unnecessary copying/building
identical dictionaries when filtering private read receipts.

Also clarifies comments and cleans-up some tests.
2022-05-16 15:06:23 +00:00
Erik Johnston
c72d26c1e1
Refactor EventContext ()
Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing.

The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). 

One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used.


Part of 
2022-05-10 19:43:13 +00:00