Commit Graph

24032 Commits

Author SHA1 Message Date
Andrew Morgan
199223062a
Revert "Support MSC3916 by adding a federation /download endpoint" (#17325) 2024-06-18 16:54:19 +01:00
Jörg Thalheim
97c3d98816
register_new_matrix_user: add password-file flag (#17294)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
2024-06-18 16:21:51 +01:00
Quentin Gliech
fa3adc896a
Merge branch 'master' into develop 2024-06-18 12:33:43 +02:00
Johannes Marbach
79767a1108
Add support for via query parameter from MSC4156 (#17322)
This adds support for the `via` query parameter from
https://github.com/matrix-org/matrix-spec-proposals/pull/4156.
2024-06-18 10:03:39 +00:00
Quentin Gliech
4af654f0da
1.109.0 2024-06-18 11:46:34 +02:00
Dirk Klimpel
1c7d85fdfe
fix missing quotes for exclude_rooms_from_sync (#17308)
We tried to configure rooms `exclude_rooms_from_sync`. If we do not
quote we get an error.
The example should be valid.
2024-06-18 10:37:02 +01:00
Quentin Gliech
5a65e8a0d1
Switch to macOS 12 runners to fix building of wheels for macOS (#17319)
This changes the release artefacts workflow to use `macos-12` runners
instead of `macos-11`, as the latter will be fully deprecated in a few
days.

This also updates `cibuildwheel` to a newer version, as it would not
'repair' the macOS wheels correctly

The difference is that now instead of outputting a macOS 11+ compatible
wheel, we output a macOS 12+ compatible one. This is fine, as macOS 11
is considered EOL since September 2023.

We can also expect that macOS 12 will be considered EOL in September
2024, as Apple usually supports the last 3 macOS version, and macOS 15
is scheduled to be released around that time.
2024-06-18 11:26:36 +02:00
dependabot[bot]
088992a484
Bump types-pyyaml from 6.0.12.12 to 6.0.12.20240311 (#17316) 2024-06-18 10:01:34 +01:00
dependabot[bot]
d17d931a53
Bump docker/build-push-action from 5 to 6 (#17312) 2024-06-18 09:57:58 +01:00
dependabot[bot]
334123f0cd
Bump dawidd6/action-download-artifact from 5 to 6 (#17313) 2024-06-18 09:57:42 +01:00
dependabot[bot]
d8e81f67eb
Bump types-netaddr from 1.2.0.20240219 to 1.3.0.20240530 (#17314) 2024-06-18 09:57:24 +01:00
dependabot[bot]
19a3d5b606
Bump phonenumbers from 8.13.37 to 8.13.39 (#17315) 2024-06-18 09:57:07 +01:00
dependabot[bot]
52813a8d94
Bump msgpack from 1.0.7 to 1.0.8 (#17317) 2024-06-18 09:56:56 +01:00
Eric Eastwood
a5485437cf
Add is_encrypted filtering to Sliding Sync /sync (#17281)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-17 12:06:18 -05:00
Eric Eastwood
e5b8a3e37f
Add stream_ordering sort to Sliding Sync /sync (#17293)
Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
2024-06-17 11:27:14 -05:00
Quentin Gliech
e88332b5f4
Merge branch 'release-v1.109' into develop 2024-06-17 15:51:16 +02:00
Quentin Gliech
edfb7aad3a
1.109.0rc3 2024-06-17 14:07:49 +02:00
Quentin Gliech
f983a77ab0
Set our own stream position from the current sequence value on startup (#17309) 2024-06-17 11:50:00 +00:00
Quentin Gliech
12d7303707
Use the release branch for sytest in release-branch PRs (#17306) 2024-06-17 11:21:51 +00:00
Erik Johnston
a3cb244755
Automatically apply SQL for inconsistent sequence (#17305)
Rather than forcing the server operator to apply the SQL manually.

This should be safe, as there should be only one writer for these
sequences.
2024-06-14 16:40:29 +01:00
Richard van der Hoff
3aae60f17b
Enable cross-signing key upload without UIA (#17284)
Per MSC3967, which is now stable, we should not require UIA when
uploading cross-signing keys for the first time.

Fixes: #17227
2024-06-14 11:14:56 +01:00
Richard van der Hoff
2c36a679ae
Include user membership on events (#17282)
MSC4115 has now completed FCP, so we can enable it by default and switch
to the stable identifier.
2024-06-13 21:45:54 +00:00
Eric Eastwood
c12ee0d5ba
Add is_dm filtering to Sliding Sync /sync (#17277)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
2024-06-13 13:56:58 -05:00
Eric Eastwood
8aaff851b1
Fix newly_left rooms not appearing if we returned early (Sliding Sync) (#17301)
Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`. 

Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync)

The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`.

Found while working on https://github.com/element-hq/synapse/pull/17293
2024-06-13 11:36:57 -05:00
Eric Eastwood
8c58eb7f17
Add event.internal_metadata.instance_name (#17300)
Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`.

`instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`.

This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293

Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see 09609cb0db (diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410)
2024-06-13 11:32:50 -05:00
Eric Eastwood
ebdce69f6a
Fix get_last_event_in_room_before_stream_ordering(...) finding the wrong last event (#17295)
PR where this was introduced: https://github.com/matrix-org/synapse/pull/14817

### What does this affect?

`get_last_event_in_room_before_stream_ordering(...)` is used in Sync v2 in a lot of different state calculations.

`get_last_event_in_room_before_stream_ordering(...)`  is also used in `/rooms/{roomId}/members`
2024-06-13 11:00:52 -05:00
Andrew Morgan
c6eb99c878
Bump mypy from 1.8.0 to 1.9.0 (#17297)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-13 15:50:10 +01:00
Quentin Gliech
5db3eec5bc
Clarify that MSC4151 is enabled on matrix.org (#17296)
This clarifies in the comments that the MSC is being used in matrix.org

See #17270
2024-06-13 13:49:00 +00:00
Travis Ralston
f1c4dfb08b
Add report room API (MSC4151) (#17270)
https://github.com/matrix-org/matrix-spec-proposals/pull/4151

This is intended to be enabled by default for immediate use. When FCP is
complete, the unstable endpoint will be dropped and stable endpoint
supported instead - no backwards compatibility is expected for the
unstable endpoint.
2024-06-12 12:27:46 +02:00
dependabot[bot]
0edf1cacf7
Bump types-jsonschema from 4.21.0.20240311 to 4.22.0.20240610 (#17288) 2024-06-12 10:52:26 +01:00
dependabot[bot]
d0f90bd04e
Bump jinja2 from 3.1.3 to 3.1.4 (#17287) 2024-06-12 10:52:14 +01:00
Quentin Gliech
0248ed70a9
Merge branch 'release-v1.109' into develop 2024-06-11 16:25:26 +02:00
Quentin Gliech
e6816babf6
CHANGES.md: s/OTKs/one-time-keys/ 2024-06-11 15:39:30 +02:00
Quentin Gliech
a8069e9739
1.109.0rc2 2024-06-11 15:22:21 +02:00
dependabot[bot]
863578bfcf
Bump regex from 1.10.4 to 1.10.5 (#17290) 2024-06-11 10:50:13 +01:00
dependabot[bot]
9e59d18022
Bump dawidd6/action-download-artifact from 3.1.4 to 5 (#17289) 2024-06-11 10:50:03 +01:00
dependabot[bot]
491365f199
Bump types-pillow from 10.2.0.20240423 to 10.2.0.20240520 (#17285) 2024-06-11 10:47:03 +01:00
Eric Eastwood
dad1559721
Reorganize Pydantic models and types used in handlers (#17279)
Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues.

 - Moved Pydantic parsing models to `synapse/types/rest`
 - Moved handler types to `synapse/types/handlers`
2024-06-10 15:03:50 -05:00
Erik Johnston
8c4937b216
Fix bug where device lists would break sync (#17292)
If the stream ID in the unconverted table is ahead of the device lists
ID gen, then it can break all /sync requests that had an ID from ahead
of the table.

The fix is to make sure we add the unconverted table to the list of
tables we check at start up.

Broke in https://github.com/element-hq/synapse/pull/17229
2024-06-10 15:56:57 +01:00
Erik Johnston
b84e31375b Update changelog 2024-06-10 15:55:42 +01:00
Erik Johnston
06953bc193 Always return OTK counts (#17275)
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-10 15:53:45 +01:00
Eric Eastwood
265ee88f34
Wrong retention policy being used when filtering events (lint ControlVarUsedAfterBlockViolation WPS441) (#17272)
Fix loop var being used outside block.

Before this change, we were always using the last room_id's retention policy for all events being filtered.

I found this bug with the [new lint rule, `ControlVarUsedAfterBlockViolation` `WPS441`](https://github.com/astral-sh/ruff/pull/11769), that I re-implemented in `ruff`. Shout-out to @reivilibre for all the help in the beginning! 

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2024-06-10 09:48:38 -05:00
Shay
ab94bce02c
Support MSC3916 by adding a federation /download endpoint (#17172) 2024-06-07 13:54:28 +01:00
reivilibre
17d6c28285
Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. (#17266)
Fixes: #17013

Add logging for whether room keys are replaced
This is motivated by the Crypto team who need to diagnose crypto issues.

The existing opentracing logging is not enough because it is not enabled
for all users.
2024-06-07 12:01:21 +01:00
Eric Eastwood
4a7c58642c
Add Sliding Sync /sync endpoint (initial implementation) (#17187)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync

This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).

Rooms appear in the Sliding sync response based on:

 - `invite`, `join`, `knock`, `ban` membership events
 - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
 - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
 - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.

### Example request

`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync`

```json
{
  "lists": {
    "foo-list": {
      "ranges": [ [0, 99] ],
      "sort": [ "by_notification_level", "by_recency", "by_name" ],
      "required_state": [
        ["m.room.join_rules", ""],
        ["m.room.history_visibility", ""],
        ["m.space.child", "*"]
      ],
      "timeline_limit": 100
    }
  }
}
```

Response:
```json
{
  "next_pos": "s58_224_0_13_10_1_1_16_0_1",
  "lists": {
    "foo-list": {
      "count": 1,
      "ops": [
        {
          "op": "SYNC",
          "range": [0, 99],
          "room_ids": [
            "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
          ]
        }
      ]
    }
  },
  "rooms": {},
  "extensions": {}
}
```
2024-06-06 14:44:32 -05:00
Erik Johnston
ce9385819b
Handle OTK uploads off master (#17271)
And fallback keys uploads. Only device keys need handling on master
2024-06-06 17:47:02 +01:00
Erik Johnston
a963f579de
Don't try and resync devices for down hosts (#17273)
It's just a waste of time if we won't even query the remote host as its
marked as down.
2024-06-06 17:46:52 +01:00
Erik Johnston
3f06bbc0ac
Always return OTK counts (#17275)
Broke in https://github.com/element-hq/synapse/pull/17215
2024-06-06 17:10:58 +01:00
Shay
fcbc79bb87
Ratelimiting of remote media downloads (#17256) 2024-06-05 13:43:36 +01:00
Erik Johnston
aabf577166
Handle hyphens in user dir search porperly (#17254)
c.f. #16675
2024-06-05 10:40:34 +01:00