Commit Graph

492 Commits

Author SHA1 Message Date
Erik Johnston
06953bc193 Always return OTK counts (#17275)
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-10 15:53:45 +01:00
Erik Johnston
5624c8b961
In sync wait for worker to catch up since token (#17215)
Otherwise things will get confused.

An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
2024-05-30 14:03:49 +01:00
Eric Eastwood
c97251d5ba
Add Sliding Sync /sync/e2ee endpoint for To-Device messages (#17167)
This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync  (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa.

Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought.

Based on:

 - https://github.com/matrix-org/matrix-spec-proposals/pull/3575
 - https://github.com/matrix-org/matrix-spec-proposals/pull/3885
 - https://github.com/matrix-org/matrix-spec-proposals/pull/3884
2024-05-23 12:06:16 -05:00
Erik Johnston
b5facbac0f
Improve perf of sync device lists (#17216)
Re-introduces #17191, and includes #17197 and #17214

The basic idea is to stop calling `get_rooms_for_user` everywhere, and
instead use the table `device_lists_changes_in_room`.

Commits reviewable one-by-one.
2024-05-21 16:48:20 +01:00
Eric Eastwood
c856ae4724
Refactor SyncResultBuilder assembly to its own function (#17202)
We will re-use `get_sync_result_builder(...)` in
https://github.com/element-hq/synapse/pull/17167

Split out from https://github.com/element-hq/synapse/pull/17167
2024-05-16 13:05:31 -05:00
Eric Eastwood
fe07995e69
Fix joined_rooms/joined_room_ids usage (#17208)
This change was introduced in
https://github.com/element-hq/synapse/pull/17203

But then https://github.com/element-hq/synapse/pull/17207 was reverted
which brought back usage `joined_rooms` that needed to be updated.
Wasn't caught because `develop` wasn't up to date before merging.
2024-05-16 17:27:38 +00:00
Eric Eastwood
52a649580f
Rename to be obvious: joined_rooms -> joined_room_ids (#17203)
Split out from https://github.com/element-hq/synapse/pull/17167
2024-05-16 11:55:51 -05:00
Eric Eastwood
28a948f04f
Removed request_key from the SyncConfig (moved outside as its own function parameter) (#17201)
Removed `request_key` from the `SyncConfig` (moved outside as its own function parameter) so it doesn't have to flow into `_generate_sync_entry_for_xxx` methods. This way we can separate the concerns of caching from generating the response and reuse the `_generate_sync_entry_for_xxx` functions as we see fit. Plus caching doesn't really have anything to do with the config of sync.

Split from https://github.com/element-hq/synapse/pull/17167

Spawning from https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279
2024-05-16 11:54:46 -05:00
Erik Johnston
fd12003441
Revert "Improve perf of sync device lists" (#17207)
Reverts element-hq/synapse#17191
2024-05-16 16:07:54 +01:00
Erik Johnston
5e892671a7
Fix bug where push rules would be empty in /sync (#17142)
Fixes #16987

Some old accounts seem to have an entry in global account data table for
push rules, which we should ignore
2024-05-16 15:04:14 +01:00
Eric Eastwood
d2d48cce85
Refactor Sync handler to be able to return different sync responses (SyncVersion) (#17200)
Refactor Sync handler to be able to be able to return different sync
responses (`SyncVersion`). Preparation to be able support sync v2 and a
new Sliding Sync `/sync/e2ee` endpoint which returns a subset of sync
v2.

Split upon request:
https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279

Split from https://github.com/element-hq/synapse/pull/17167 where we
will add `SyncVersion.E2EE_SYNC` and a new type of sync response.
2024-05-16 11:36:54 +01:00
Erik Johnston
0b91ccce47
Improve perf of sync device lists (#17191)
It's almost always more efficient to query the rooms that have device
list changes, rather than looking at the list of all users whose devices
have changed and then look for shared rooms.
2024-05-14 14:39:04 +01:00
Richard van der Hoff
b548f7803a
Add support for MSC4115 (#17104)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-04-29 15:22:13 +01:00
Erik Johnston
89f1092284
Also check if first event matches the last in prev batch (#17066)
Refinement of #17064 

cc @richvdh
2024-04-09 14:01:12 +00:00
Erik Johnston
4d10a8fb18
Fixups to #17064 (#17065)
Forget a line, and an empty batch is trivially linear.

c.f. #17064
2024-04-08 14:55:19 +01:00
Erik Johnston
1f8f991d51
Add back fast path for non-gappy syncs (#17064)
PR #16942 removed an invalid optimisation that avoided pulling out state
for non-gappy syncs. This causes a large increase in DB usage. c.f.
#16941 for why that optimisation was wrong.

However, we can still optimise in the simple case where the events in
the timeline are a linear chain without any branching/merging of the
DAG.

cc. @richvdh
2024-04-08 14:25:28 +01:00
Richard van der Hoff
0e68e9b7f4
Fix bug in calculating state for non-gappy syncs (#16942)
Unfortunately, the optimisation we applied here for non-gappy syncs is
not actually valid.

Fixes https://github.com/element-hq/synapse/issues/16941.

~~Based on https://github.com/element-hq/synapse/pull/16930.~~
Requires https://github.com/matrix-org/sytest/pull/1374.
2024-04-04 16:15:35 +00:00
Richard van der Hoff
230b709d9d
/sync: fix bug in calculating state response (#16930)
Fix a long-standing issue which could cause state to be omitted from the
sync response if the last event was filtered out.

Fixes: https://github.com/element-hq/synapse/issues/16928
2024-04-04 12:14:24 +00:00
Richard van der Hoff
05957ac70f
Fix bug in /sync response for archived rooms (#16932)
This PR fixes a very, very niche edge-case, but I've got some more work
coming which will otherwise make the problem worse.

The bug happens when the syncing user leaves a room, and has a sync
filter which includes "left" rooms, but sets the timeline limit to 0. In
that case, the state returned in the `state` section is calculated
incorrectly.

The fix is to pass a token corresponding to the point that the user
leaves the room through to `compute_state_delta`.
2024-04-04 12:47:59 +01:00
Shay
8fb5b0f335
Improve event validation (#16908)
As the title states.
2024-03-19 17:52:53 +00:00
Richard van der Hoff
52f456a822
/sync: Fix edge-case in calculating the "device_lists" response (#16949)
Fixes https://github.com/element-hq/synapse/issues/16948. If the `join`
and the `leave` are in the same sync response, we need to count them as
a "left" user.
2024-03-14 17:34:19 +00:00
Richard van der Hoff
6d5bafb2c8
Split up SyncHandler.compute_state_delta (#16929)
This is a huge method, which melts my brain.

This is a non-functional change which lays some groundwork for future
work in this area.
2024-03-14 17:18:48 +00:00
dependabot[bot]
1e68b56a62
Bump black from 23.10.1 to 24.2.0 (#16936) 2024-03-13 16:46:44 +00:00
Erik Johnston
23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Erik Johnston
cbe8a80d10
Faster load recents for sync (#16783)
This hopefully reduces the amount of state we need to keep in memory
2024-01-10 15:11:59 +00:00
Erik Johnston
7469fa7585
Simplify internal metadata class. (#16762)
We remove these fields as they're just duplicating data the event
already stores, and (for reasons 🤫) I'd like to simplify
the class to only store simple types.

I'm not entirely convinced that we shouldn't instead add helper methods
to the event class to generate stream tokens, but I don't really think
that's where they belong either
2024-01-05 13:03:20 +00:00
Patrick Cloke
8e1e62c9e0 Update license headers 2023-11-21 15:29:58 -05:00
Patrick Cloke
f2f2c7c1f0
Use full GitHub links instead of bare issue numbers. (#16637) 2023-11-15 08:02:11 -05:00
Patrick Cloke
85e5f2dc25
Add a new module API to update user presence state. (#16544)
This adds a module API which allows a module to update a user's
presence state/status message. This is useful for controlling presence
from an external system.

To fully control presence from the module the presence.enabled config
parameter gains a new state of "untracked" which disables internal tracking
of presence changes via user actions, etc. Only updates from the module will
be persisted and sent down sync properly).
2023-10-26 15:11:24 -04:00
Erik Johnston
ba47fea528
Allow multiple workers to write to receipts stream. (#16432)
Fixes #16417
2023-10-25 16:16:19 +01:00
Erik Johnston
e9069c9f91
Mark sync as limited if there is a gap in the timeline (#16485)
This splits thinsg into two queries, but most of the time we won't have
new event backwards extremities so this shouldn't actually add an extra
RTT for the majority of cases.

Note this removes the check for events with no prev events, but that was
part of MSC2716 work that has since been removed.
2023-10-19 15:04:18 +01:00
Mathieu Velten
eee6474bce
Remove useless async job to delete device messages on sync (#16491) 2023-10-16 11:06:27 +01:00
Erik Johnston
009b47badf
Factor out MultiWriter token from RoomStreamToken (#16427) 2023-10-05 10:46:28 +01:00
Patrick Cloke
d7c89c5908
Return immutable objects for cachedList decorators (#16350) 2023-09-19 15:26:44 -04:00
Patrick Cloke
c1e244c8f7
Make cached account data/tags/admin types immutable (#16325) 2023-09-18 09:55:04 -04:00
Erik Johnston
e9addf6a01
Don't schedule an async task on every sync (#16312) 2023-09-13 11:59:44 +01:00
Mathieu Velten
4f1840a88a
Delete device messages asynchronously and in staged batches (#16240) 2023-09-06 09:30:53 +02:00
Matthew Ma
8a4fb7a6ba
Disable caching in /sync corner case (#16080)
Fixes #15502
2023-08-17 10:22:50 +01:00
Patrick Cloke
ad3f43be9a
Run pyupgrade for python 3.7 & 3.8. (#16110) 2023-08-15 08:11:20 -04:00
Patrick Cloke
4b4e0dc3ce
Error if attempting to set m.push_rules account data, per MSC4010. (#15555)
m.push_rules, like m.fully_read, is a special account data type that cannot
be set using the normal /account_data endpoint. Return an error instead
of allowing data that will not be used to be stored.
2023-05-09 10:34:10 -04:00
Patrick Cloke
2bfe3f0b81
Use account data constants in more places. (#15554) 2023-05-09 07:23:27 -04:00
Sean Quah
89a71e7390
Fix a rare bug where initial /syncs would fail (#15383)
This change fixes a rare bug where initial /syncs would fail with a
`KeyError` under the following circumstances:
 1. A user fast joins a remote room.
 2. The user is kicked from the room before the room's full state has
    been synced.
 3. A second local user fast joins the room.
 4. Events are backfilled into the room with a higher topological
    ordering than the original user's leave. They are assigned a
    negative stream ordering. It's not clear how backfill happened here,
    since it is expected to be equivalent to syncing the full state.
 5. The second local user leaves the room before the room's full state
    has been synced. The homeserver does not complete the sync.
 6. The original user performs an initial /sync with lazy_load_members
    enabled.
     * Because they were kicked from the room, the room is included in
       the /sync response even though the include_leave option is not
       specified.
     * To populate the room's timeline, `_load_filtered_recents` /
       `get_recent_events_for_room` fetches events with a lower stream
       ordering than the leave event and picks the ones with the highest
       topological orderings (which are most recent). This captures the
       backfilled events after the leave, since they have a negative
       stream ordering. These events are filtered out of the timeline,
       since the user was not in the room at the time and cannot view
       them. The sync code ends up with an empty timeline for the room
       that notably does not include the user's leave event.
       This seems buggy, but at least we don't disclose events the user
       isn't allowed to see.
     * Normally, `compute_state_delta` would fetch the state at the
       start and end of the room's timeline to generate the sync
       response. Since the timeline is empty, it fetches the state at
       `min(now, last event in the room)`, which corresponds with the
       second user's leave. The state during the entirety of the second
       user's membership does not include the membership for the first
       user because of partial state.
       This part is also questionable, since we are fetching state from
       outside the bounds of the user's membership.
     * `compute_state_delta` then tries and fails to find the user's
       membership in the auth events of timeline events. Because there
       is no timeline event whose auth events are expected to contain
       the user's membership, a `KeyError` is raised.

Also contains a drive-by fix for a separate unlikely race condition.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-04-04 13:10:25 +01:00
Sean Quah
caf43c3d7c
Faster joins: Fix spurious errors on incremental sync (#15232)
When pushing events in partial state rooms down incremental /sync, we
try to find the `m.room.member` state event for their senders by digging
through their auth events, so that we can present the membership to the
client. Events usually have a membership event in their auth events,
with the exception of the `m.room.create` event and a user's first join
into the room.

When implementing #13477, we took the case of a user's first join into
account, but forgot to handle the `m.room.create` case. This change
fixes that.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-03-09 14:18:39 +00:00
dependabot[bot]
9bb2eac719
Bump black from 22.12.0 to 23.1.0 (#15103) 2023-02-22 15:29:09 -05:00
Sean Quah
463c19ac36
Faster joins: Omit device list updates from partial state rooms in /sync (#15069)
...when lazy loading of members is not enabled. It's weird to notify
a client that another user's device list has changed when the client
doesn't think that they share a room.

Note that when a room is un-partial stated, device list updates are
emitted for every member in that room over /sync.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-02-14 12:32:19 +00:00
Erik Johnston
f09db5c991
Skip calculating unread push actions in /sync when enable_push is false. (#14980) 2023-02-14 11:10:29 +00:00
Sean Quah
d0c713cc85
Return read-only collections from @cached methods (#13755)
It's important that collections returned from `@cached` methods are not
modified, otherwise future retrievals from the cache will return the
modified collection.

This applies to the return values from `@cached` methods and the values
inside the dictionaries returned by `@cachedList` methods. It's not
necessary for the dictionaries returned by `@cachedList` methods
themselves to be read-only.

Signed-off-by: Sean Quah <seanq@matrix.org>
Co-authored-by: David Robertson <davidr@element.io>
2023-02-10 23:29:00 +00:00
Patrick Cloke
cf5233b783
Avoid fetching unused account data in sync. (#14973)
The per-room account data is no longer unconditionally
fetched, even if all rooms will be filtered out.

Global account data will not be fetched if it will all be
filtered out.
2023-02-10 14:22:16 +00:00
Sean Quah
b95407908d
Avoid mutating cached values in _generate_sync_entry_for_account_data (#15047) 2023-02-10 08:11:20 -05:00
Patrick Cloke
da05b70af5
Skip unused calculations in sync handler. (#14908)
If a sync request does not need to calculate per-room entries &
is not generating presence & is not generating device list data
(e.g. during initial sync) avoid the expensive calculation of room
specific data.

This is a micro-optimisation for clients syncing simply to receive
to-device information.
2023-02-02 13:45:12 -05:00