Commit Graph

23878 Commits

Author SHA1 Message Date
Tulir Asokan
7ce30d6375 Merge remote-tracking branch 'upstream/release-v1.109' 2024-06-17 15:46:24 +03:00
Quentin Gliech
edfb7aad3a
1.109.0rc3 2024-06-17 14:07:49 +02:00
Quentin Gliech
f983a77ab0
Set our own stream position from the current sequence value on startup (#17309) 2024-06-17 11:50:00 +00:00
Quentin Gliech
12d7303707
Use the release branch for sytest in release-branch PRs (#17306) 2024-06-17 11:21:51 +00:00
Erik Johnston
a3cb244755
Automatically apply SQL for inconsistent sequence (#17305)
Rather than forcing the server operator to apply the SQL manually.

This should be safe, as there should be only one writer for these
sequences.
2024-06-14 16:40:29 +01:00
Tulir Asokan
d21cd494ca Merge remote-tracking branch 'upstream/release-v1.109' 2024-06-11 17:55:37 +03:00
Quentin Gliech
e6816babf6
CHANGES.md: s/OTKs/one-time-keys/ 2024-06-11 15:39:30 +02:00
Quentin Gliech
a8069e9739
1.109.0rc2 2024-06-11 15:22:21 +02:00
Erik Johnston
8c4937b216
Fix bug where device lists would break sync (#17292)
If the stream ID in the unconverted table is ahead of the device lists
ID gen, then it can break all /sync requests that had an ID from ahead
of the table.

The fix is to make sure we add the unconverted table to the list of
tables we check at start up.

Broke in https://github.com/element-hq/synapse/pull/17229
2024-06-10 15:56:57 +01:00
Erik Johnston
b84e31375b Update changelog 2024-06-10 15:55:42 +01:00
Erik Johnston
06953bc193 Always return OTK counts (#17275)
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-10 15:53:45 +01:00
Tulir Asokan
896b1ea2ba Merge remote-tracking branch 'upstream/release-v1.109' 2024-06-05 11:21:19 +03:00
Andrew Morgan
e2f8476044 Fix typo in CHANGES.md 2024-06-04 09:47:28 +01:00
Andrew Morgan
18c1196893 1.109.0rc1 2024-06-04 09:46:09 +01:00
dependabot[bot]
8a3270075b
Bump types-pyopenssl from 24.0.0.20240311 to 24.1.0.20240425 (#17260)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:26:41 +01:00
dependabot[bot]
f458dff16d
Bump mypy-zope from 1.0.3 to 1.0.4 (#17262)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:23:29 +01:00
dependabot[bot]
6b709c512d
Bump lxml from 5.2.1 to 5.2.2 (#17261)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-03 14:21:20 +01:00
Erik Johnston
5c2a837e3c
Fix bug where typing replication breaks (#17252)
This can happen on restarts of the service, due to old rooms being
pruned.
2024-05-31 16:07:05 +01:00
Nick Mills-Barrett
966a886b36 Include stream ordering in the unsigned event dict 2024-05-31 15:31:24 +03:00
Tulir Asokan
535e97da7b Fix wildcard user and room ID filters
Signed-off-by: Tulir Asokan <tulir@maunium.net>
2024-05-31 14:30:58 +03:00
Erik Johnston
64f5a4a353
Fix logging errors when receiving invalid User ID for key querys (#17250) 2024-05-31 11:27:56 +01:00
Erik Johnston
7dd14fadb1
Fix sentry default tags (#17251)
This was broken by the sentry 2.0 upgrade

Broke in v1.108.0
2024-05-31 11:27:47 +01:00
Erik Johnston
5624c8b961
In sync wait for worker to catch up since token (#17215)
Otherwise things will get confused.

An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
2024-05-30 14:03:49 +01:00
Erik Johnston
4e3868dc46
Fix deduplicating of membership events to not create unused state groups. (#17164)
We try and deduplicate in two places: 1) really early on, and 2) just
before we persist the event. The first case was broken due to it
occuring before the profile information was added, and so it thought the
event contents were different.

The second case did catch it and handle it correctly, however doing so
creates a redundant state group leading to bloat.

Fixes #3791
2024-05-30 11:33:48 +00:00
Erik Johnston
d16910ca02
Replaces all usages of StreamIdGenerator with MultiWriterIdGenerator (#17229)
Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`, which is safer.
2024-05-30 11:07:32 +00:00
Erik Johnston
225f378ffa
Clean out invalid destinations from outbox (#17242)
We started ensuring we only insert valid destinations:
https://github.com/element-hq/synapse/pull/17240
2024-05-30 11:25:24 +01:00
Erik Johnston
8bd9ff0783
Ensure we delete media if we reject due to spam check (#17246)
Fixes up #17239

We need to keep the spam check within the `try/except` block. Also makes
it so that we don't enter the top span twice.

Also also ensures that we get the right thumbnail length.
2024-05-30 11:22:19 +01:00
Erik Johnston
466f344547
Move towards using MultiWriterIdGenerator everywhere (#17226)
There is a problem with `StreamIdGenerator` where it can go backwards
over restarts when a stream ID is requested but then not inserted into
the DB. This is problematic if we want to land #17215, and is generally
a potential cause for all sorts of nastiness.

Instead of trying to fix `StreamIdGenerator`, we may as well move to
`MultiWriterIdGenerator` that does not suffer from this problem (the
latest positions are stored in `stream_positions` table). This involves
adding SQLite support to the class.

This only changes id generators that were already using
`MultiWriterIdGenerator` under postgres, a separate PR will move the
rest of the uses of `StreamIdGenerator` over.
2024-05-29 12:19:10 +00:00
Erik Johnston
726006cdf2
Don't invalidate all get_relations_for_event on history purge (#17083)
This is a tree cache already, so may as well move the room ID to the
front and use that
2024-05-29 12:57:10 +01:00
Erik Johnston
967b6948b0
Change allow_unsafe_locale to also apply on new databases (#17238)
We relax this as there are use cases where this is safe, though it is
still highly recommended that people avoid using it.
2024-05-29 12:04:13 +01:00
Erik Johnston
d7198dfb95
Ignore attempts to send to-device messages to bad users (#17240)
Currently sending a to-device message to a user ID with a dodgy
destination is accepted, but then ends up spamming the logs when we try
and send to the destination.

An alternative would be to reject the request, but I'm slightly nervous
that could break things.
2024-05-29 11:52:48 +01:00
Erik Johnston
94ef2f4f5d
Handle duplicate OTK uploads racing (#17241)
Currently this causes one of then to 500.
2024-05-29 11:16:00 +01:00
Erik Johnston
bb5a692946
Fix slipped logging context when media rejected (#17239)
When a module rejects a piece of media we end up trying to close the
same logging context twice.

Instead of fixing the existing code we refactor to use an async context
manager, which is easier to write correctly.
2024-05-29 11:14:42 +01:00
Olivier 'reivilibre
ad179b0136 Merge branch 'master' into develop 2024-05-28 13:34:19 +01:00
dependabot[bot]
5147ce294a
Bump phonenumbers from 8.13.35 to 8.13.37 (#17235) 2024-05-28 13:26:37 +01:00
Tulir Asokan
acfc71a2de Merge remote-tracking branch 'upstream/release-v1.108' 2024-05-28 14:04:47 +03:00
Olivier 'reivilibre
f35bc08d39 1.108.0 2024-05-28 11:54:28 +01:00
dependabot[bot]
f2616edb73
Bump pyicu from 2.13 to 2.13.1 (#17236) 2024-05-28 11:28:58 +01:00
dependabot[bot]
86a2a0258f
Bump pyopenssl from 24.0.0 to 24.1.0 (#17234) 2024-05-28 11:28:32 +01:00
dependabot[bot]
0893ee9af8
Bump prometheus-client from 0.19.0 to 0.20.0 (#17233) 2024-05-28 11:28:16 +01:00
dependabot[bot]
887f773472
Bump serde from 1.0.202 to 1.0.203 (#17232) 2024-05-28 11:27:51 +01:00
Shay
9edb725ebc
Support MSC3916 by adding unstable media endpoints to _matrix/client (#17213)
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/rav/authentication-for-media/proposals/3916-authentication-for-media.md)
adds new media endpoints under `_matrix/client`. This PR adds the
`/preview_url`, `/config`, and `/thumbnail` endpoints. `/download` will
be added in a follow-up PR once the work for the federation `/download`
endpoint is complete (see
https://github.com/element-hq/synapse/pull/17172).

Should be reviewable commit-by-commit.
2024-05-24 09:47:37 +01:00
Eric Eastwood
c97251d5ba
Add Sliding Sync /sync/e2ee endpoint for To-Device messages (#17167)
This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync  (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa.

Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought.

Based on:

 - https://github.com/matrix-org/matrix-spec-proposals/pull/3575
 - https://github.com/matrix-org/matrix-spec-proposals/pull/3885
 - https://github.com/matrix-org/matrix-spec-proposals/pull/3884
2024-05-23 12:06:16 -05:00
reivilibre
7e2412265d
Log exceptions when failing to auto-join new user according to the auto_join_rooms option. (#17176)
Would have been useful for tracking down #16878.

Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
2024-05-22 14:22:33 +01:00
reivilibre
7ef00b7628
Add logging to tasks managed by the task scheduler, showing CPU and database usage. (#17219)
The log format is the same as the request log format, except:

- fields that are specific to HTTP requests have been removed
- the task's params are included at the end of the log line.

These log lines are emitted:
- when the task function finishes — both completion and failure (and I
suppose it is possible for a task to become schedulable again?)
- every 5 minutes whilst it is running

Closes #17217.

---------

Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
2024-05-22 14:12:58 +01:00
Erik Johnston
b71d277438
Reduce work of calculating outbound device pokes (#17211) 2024-05-22 13:55:18 +01:00
Yadd
a547b49773
Update Lemonldap-NG OIDC config (#17204)
Update OIDC documentation: by default Matrix doesn't query userinfo endpoint, then claims should be put on id_token.
2024-05-22 12:29:31 +00:00
devonh
6a9a641fb8
Bring auto-accept invite logic into Synapse (#17147)
This PR ports the logic from the
[synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite)
module into synapse.

I went with the naive approach of injecting the "module" next to where
third party modules are currently loaded. If there is a better/preferred
way to handle this, I'm all ears. It wasn't obvious to me if there was a
better location to add this logic that would cleanly apply to all
incoming invite events.

Relies on https://github.com/element-hq/synapse/pull/17166 to fix linter
errors.
2024-05-21 20:09:17 +00:00
Erik Johnston
b5facbac0f
Improve perf of sync device lists (#17216)
Re-introduces #17191, and includes #17197 and #17214

The basic idea is to stop calling `get_rooms_for_user` everywhere, and
instead use the table `device_lists_changes_in_room`.

Commits reviewable one-by-one.
2024-05-21 16:48:20 +01:00
Erik Johnston
b250ca5df2 Merge branch 'release-v1.108' into develop 2024-05-21 12:21:26 +01:00