Commit Graph

4726 Commits

Author SHA1 Message Date
Eric Eastwood
a5485437cf
Add is_encrypted filtering to Sliding Sync /sync (#17281)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-17 12:06:18 -05:00
Eric Eastwood
e5b8a3e37f
Add stream_ordering sort to Sliding Sync /sync (#17293)
Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
2024-06-17 11:27:14 -05:00
Richard van der Hoff
2c36a679ae
Include user membership on events (#17282)
MSC4115 has now completed FCP, so we can enable it by default and switch
to the stable identifier.
2024-06-13 21:45:54 +00:00
Eric Eastwood
c12ee0d5ba
Add is_dm filtering to Sliding Sync /sync (#17277)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-13 13:56:58 -05:00
Eric Eastwood
8aaff851b1
Fix newly_left rooms not appearing if we returned early (Sliding Sync) (#17301)
Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`. 

Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync)

The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`.

Found while working on https://github.com/element-hq/synapse/pull/17293
2024-06-13 11:36:57 -05:00
Eric Eastwood
8c58eb7f17
Add event.internal_metadata.instance_name (#17300)
Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`.

`instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`.

This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293

Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see 09609cb0db (diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410)
2024-06-13 11:32:50 -05:00
Eric Eastwood
dad1559721
Reorganize Pydantic models and types used in handlers (#17279)
Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues.

 - Moved Pydantic parsing models to `synapse/types/rest`
 - Moved handler types to `synapse/types/handlers`
2024-06-10 15:03:50 -05:00
reivilibre
17d6c28285
Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. (#17266)
Fixes: #17013

Add logging for whether room keys are replaced
This is motivated by the Crypto team who need to diagnose crypto issues.

The existing opentracing logging is not enough because it is not enabled
for all users.
2024-06-07 12:01:21 +01:00
Eric Eastwood
4a7c58642c
Add Sliding Sync /sync endpoint (initial implementation) (#17187)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync

This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).

Rooms appear in the Sliding sync response based on:

 - `invite`, `join`, `knock`, `ban` membership events
 - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
 - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
 - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.

### Example request

`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync`

```json
{
  "lists": {
    "foo-list": {
      "ranges": [ [0, 99] ],
      "sort": [ "by_notification_level", "by_recency", "by_name" ],
      "required_state": [
        ["m.room.join_rules", ""],
        ["m.room.history_visibility", ""],
        ["m.space.child", "*"]
      ],
      "timeline_limit": 100
    }
  }
}
```

Response:
```json
{
  "next_pos": "s58_224_0_13_10_1_1_16_0_1",
  "lists": {
    "foo-list": {
      "count": 1,
      "ops": [
        {
          "op": "SYNC",
          "range": [0, 99],
          "room_ids": [
            "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
          ]
        }
      ]
    }
  },
  "rooms": {},
  "extensions": {}
}
```
2024-06-06 14:44:32 -05:00
Erik Johnston
ce9385819b
Handle OTK uploads off master (#17271)
And fallback keys uploads. Only device keys need handling on master
2024-06-06 17:47:02 +01:00
Erik Johnston
a963f579de
Don't try and resync devices for down hosts (#17273)
It's just a waste of time if we won't even query the remote host as its
marked as down.
2024-06-06 17:46:52 +01:00
Erik Johnston
3f06bbc0ac
Always return OTK counts (#17275)
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-06 17:10:58 +01:00
Eric Eastwood
7d8f0ef351
Use fully-qualified PersistedEventPosition when returning RoomsForUser (#17265)
Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation.

Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
2024-06-04 12:58:03 -05:00
Erik Johnston
5c2a837e3c
Fix bug where typing replication breaks (#17252)
This can happen on restarts of the service, due to old rooms being
pruned.
2024-05-31 16:07:05 +01:00
Erik Johnston
64f5a4a353
Fix logging errors when receiving invalid User ID for key querys (#17250) 2024-05-31 11:27:56 +01:00
Erik Johnston
5624c8b961
In sync wait for worker to catch up since token (#17215)
Otherwise things will get confused.

An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
2024-05-30 14:03:49 +01:00
Erik Johnston
4e3868dc46
Fix deduplicating of membership events to not create unused state groups. (#17164)
We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.

The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.

Fixes #3791
2024-05-30 11:33:48 +00:00
Erik Johnston
726006cdf2
Don't invalidate all get_relations_for_event on history purge (#17083)
This is a tree cache already, so may as well move the room ID to the
front and use that
2024-05-29 12:57:10 +01:00
Erik Johnston
d7198dfb95
Ignore attempts to send to-device messages to bad users (#17240)
Currently sending a to-device message to a user ID with a dodgy
destination is accepted, but then ends up spamming the logs when we try
and send to the destination.

An alternative would be to reject the request, but I'm slightly nervous
that could break things.
2024-05-29 11:52:48 +01:00
Erik Johnston
94ef2f4f5d
Handle duplicate OTK uploads racing (#17241)
Currently this causes one of then to 500.
2024-05-29 11:16:00 +01:00
Eric Eastwood
c97251d5ba
Add Sliding Sync /sync/e2ee endpoint for To-Device messages (#17167)
This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync  (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa.

Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought.

Based on:

 - https://github.com/matrix-org/matrix-spec-proposals/pull/3575
 - https://github.com/matrix-org/matrix-spec-proposals/pull/3885
 - https://github.com/matrix-org/matrix-spec-proposals/pull/3884
2024-05-23 12:06:16 -05:00
reivilibre
7e2412265d
Log exceptions when failing to auto-join new user according to the auto_join_rooms option. (#17176)
Would have been useful for tracking down #16878.

Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
2024-05-22 14:22:33 +01:00
Erik Johnston
b71d277438
Reduce work of calculating outbound device pokes (#17211) 2024-05-22 13:55:18 +01:00
devonh
6a9a641fb8
Bring auto-accept invite logic into Synapse (#17147)
This PR ports the logic from the
[synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite)
module into synapse.

I went with the naive approach of injecting the "module" next to where
third party modules are currently loaded. If there is a better/preferred
way to handle this, I'm all ears. It wasn't obvious to me if there was a
better location to add this logic that would cleanly apply to all
incoming invite events.

Relies on https://github.com/element-hq/synapse/pull/17166 to fix linter
errors.
2024-05-21 20:09:17 +00:00
Erik Johnston
b5facbac0f
Improve perf of sync device lists (#17216)
Re-introduces #17191, and includes #17197 and #17214

The basic idea is to stop calling `get_rooms_for_user` everywhere, and
instead use the table `device_lists_changes_in_room`.

Commits reviewable one-by-one.
2024-05-21 16:48:20 +01:00
Eric Eastwood
c856ae4724
Refactor SyncResultBuilder assembly to its own function (#17202)
We will re-use `get_sync_result_builder(...)` in
https://github.com/element-hq/synapse/pull/17167

Split out from https://github.com/element-hq/synapse/pull/17167
2024-05-16 13:05:31 -05:00
Eric Eastwood
fe07995e69
Fix joined_rooms/joined_room_ids usage (#17208)
This change was introduced in
https://github.com/element-hq/synapse/pull/17203

But then https://github.com/element-hq/synapse/pull/17207 was reverted
which brought back usage `joined_rooms` that needed to be updated.
Wasn't caught because `develop` wasn't up to date before merging.
2024-05-16 17:27:38 +00:00
Eric Eastwood
52a649580f
Rename to be obvious: joined_rooms -> joined_room_ids (#17203)
Split out from https://github.com/element-hq/synapse/pull/17167
2024-05-16 11:55:51 -05:00
Eric Eastwood
28a948f04f
Removed request_key from the SyncConfig (moved outside as its own function parameter) (#17201)
Removed `request_key` from the `SyncConfig` (moved outside as its own function parameter) so it doesn't have to flow into `_generate_sync_entry_for_xxx` methods. This way we can separate the concerns of caching from generating the response and reuse the `_generate_sync_entry_for_xxx` functions as we see fit. Plus caching doesn't really have anything to do with the config of sync.

Split from https://github.com/element-hq/synapse/pull/17167

Spawning from https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279
2024-05-16 11:54:46 -05:00
Erik Johnston
fd12003441
Revert "Improve perf of sync device lists" (#17207)
Reverts element-hq/synapse#17191
2024-05-16 16:07:54 +01:00
Erik Johnston
5e892671a7
Fix bug where push rules would be empty in /sync (#17142)
Fixes #16987

Some old accounts seem to have an entry in global account data table for
push rules, which we should ignore
2024-05-16 15:04:14 +01:00
Eric Eastwood
d2d48cce85
Refactor Sync handler to be able to return different sync responses (SyncVersion) (#17200)
Refactor Sync handler to be able to be able to return different sync
responses (`SyncVersion`). Preparation to be able support sync v2 and a
new Sliding Sync `/sync/e2ee` endpoint which returns a subset of sync
v2.

Split upon request:
https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279

Split from https://github.com/element-hq/synapse/pull/17167 where we
will add `SyncVersion.E2EE_SYNC` and a new type of sync response.
2024-05-16 11:36:54 +01:00
Erik Johnston
0b91ccce47
Improve perf of sync device lists (#17191)
It's almost always more efficient to query the rooms that have device
list changes, rather than looking at the list of all users whose devices
have changed and then look for shared rooms.
2024-05-14 14:39:04 +01:00
Aurélien Grimpard
7d82987b27
Allows CAS SSO flow to provide user IDs composed of numbers only (#17098) 2024-05-14 13:55:32 +01:00
devonh
393429d692
Fix undiscovered linter errors (#17166)
Linter errors are showing up in #17147 that are unrelated to that PR.
The errors do not currently show up on develop.

This PR aims to resolve the linter errors separately from #17147.
2024-05-08 14:57:32 +00:00
Shay
37558d5e4c
Add support for MSC3823 - Account Suspension (#17051) 2024-05-01 17:45:17 +01:00
devonh
7ab0f630da
Apply user email & picture during OIDC registration if present & selected (#17120)
This change will apply the `email` & `picture` provided by OIDC to the
new user account when registering a new user via OIDC. If the user is
directed to the account details form, this change makes sure they have
been selected before applying them, otherwise they are omitted. In
particular, this change ensures the values are carried through when
Synapse has consent configured, and the redirect to the consent form/s
are followed.

I have tested everything manually. Including: 
- with/without consent configured
- allowing/not allowing the use of email/avatar (via
`sso_auth_account_details.html`)
- with/without automatic account detail population (by un/commenting the
`localpart_template` option in synapse config).

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [X] Pull request is based on the develop branch
* [X] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [X] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2024-04-29 15:23:05 +00:00
Richard van der Hoff
b548f7803a
Add support for MSC4115 (#17104)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-04-29 15:22:13 +01:00
Richard van der Hoff
c897ac63e9
Ensure that incoming to-device messages are not dropped (#17127)
... when workers are unreachable, etc.

Fixes https://github.com/element-hq/synapse/issues/17117.

The general principle is just to make sure that we propagate any
exceptions to the JsonResource, so that we return an error code to the
sending server. That means that the sending server no longer considers
the message safely sent, so it will retry later.

In the issue, Erik mentions that an alternative solution would be to
persist the to-device messages into a table so that they can be retried.
This might be an improvement for performance, but even if we did that,
we still need this mechanism, since we might be unable to reach the
database. So, if we want to do that, it can be a later follow-up.

---------

Co-authored-by: Erik Johnston <erik@matrix.org>
2024-04-29 14:11:00 +01:00
Till
47773232b0
Redact membership events if the user requested erasure upon deactivating (#17076)
Fixes #15355 by redacting all membership events before leaving rooms.
2024-04-25 14:25:31 +01:00
Kegan Dougal
259442fa4c
bugfix: make msc3967 idempotent (#16943)
MSC3967 was updated recently to make it more robust to network failures:

> there is an existing cross-signing master key and it exactly matches
the cross-signing master key provided in the request body. If there are
any additional keys provided in the request (self signing key, user
signing key) they MUST also match the existing keys stored on the
server. In other words, the request contains no new keys. If there are
new keys, UIA MUST be performed.


https://github.com/matrix-org/matrix-spec-proposals/blob/hughns/device-signing-upload-uia/proposals/3967-device-signing-upload-uia.md#proposal

This covers the case where the 200 OK is lost in transit so the client
retries the upload, only to then get UIA'd.

Complement tests: https://github.com/matrix-org/complement/pull/713 -
passing example
https://github.com/element-hq/synapse/actions/runs/7976948122/job/21778795094?pr=16943#step:7:8820

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-04-15 10:57:56 +00:00
Erik Johnston
89f1092284
Also check if first event matches the last in prev batch (#17066)
Refinement of #17064 

cc @richvdh
2024-04-09 14:01:12 +00:00
Mathieu Velten
e363881592
Fix PR #16677, a parameter was missing in a function call (#17033)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-04-09 14:06:46 +01:00
Erik Johnston
4d10a8fb18
Fixups to #17064 (#17065)
Forget a line, and an empty batch is trivially linear.

c.f. #17064
2024-04-08 14:55:19 +01:00
Erik Johnston
1f8f991d51
Add back fast path for non-gappy syncs (#17064)
PR #16942 removed an invalid optimisation that avoided pulling out state
for non-gappy syncs. This causes a large increase in DB usage. c.f.
#16941 for why that optimisation was wrong.

However, we can still optimise in the simple case where the events in
the timeline are a linear chain without any branching/merging of the
DAG.

cc. @richvdh
2024-04-08 14:25:28 +01:00
Richard van der Hoff
0e68e9b7f4
Fix bug in calculating state for non-gappy syncs (#16942)
Unfortunately, the optimisation we applied here for non-gappy syncs is
not actually valid.

Fixes https://github.com/element-hq/synapse/issues/16941.

~~Based on https://github.com/element-hq/synapse/pull/16930.~~
Requires https://github.com/matrix-org/sytest/pull/1374.
2024-04-04 16:15:35 +00:00
Richard van der Hoff
230b709d9d
/sync: fix bug in calculating state response (#16930)
Fix a long-standing issue which could cause state to be omitted from the
sync response if the last event was filtered out.

Fixes: https://github.com/element-hq/synapse/issues/16928
2024-04-04 12:14:24 +00:00
Richard van der Hoff
05957ac70f
Fix bug in /sync response for archived rooms (#16932)
This PR fixes a very, very niche edge-case, but I've got some more work
coming which will otherwise make the problem worse.

The bug happens when the syncing user leaves a room, and has a sync
filter which includes "left" rooms, but sets the timeline limit to 0. In
that case, the state returned in the `state` section is calculated
incorrectly.

The fix is to pass a token corresponding to the point that the user
leaves the room through to `compute_state_delta`.
2024-04-04 12:47:59 +01:00
Erik Johnston
fd48fc4585
Fixups to new push stream (#17038)
Follow on from #17037
2024-03-28 16:29:23 +00:00
Erik Johnston
ea6bfae0fc
Add support for moving /push_rules off of main process (#17037) 2024-03-28 15:44:07 +00:00