Commit Graph

983 Commits

Author SHA1 Message Date
Erik Johnston
ceb3686dcd
Fixup sliding sync comment (#17531)
c.f.
https://github.com/element-hq/synapse/pull/17529#discussion_r1705780925
2024-08-07 10:32:36 +01:00
Erik Johnston
c270355349
SS: Reset connection if token is unrecognized (#17529)
This triggers the client to start a new sliding sync connection. If we
don't do this and the client asks for the full range of rooms, we end up
sending down all rooms and their state from scratch (which can be very
slow)

This causes things like
https://github.com/element-hq/element-x-ios/issues/3115 after we restart
the server

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-06 10:39:11 +01:00
Eric Eastwood
46de0ee16b
Sliding Sync: Update filters to be robust against remote invite rooms (#17450)
Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to
be robust when dealing with remote invite rooms in Sliding Sync.

Part of
[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575):
Sliding Sync

Follow-up to https://github.com/element-hq/synapse/pull/17434

We now take into account current state, fallback to stripped state
for invite/knock rooms, then historical state. If we can't determine
the info needed to filter a room (either from state or stripped state),
it is filtered out.
2024-07-30 13:20:29 -05:00
Erik Johnston
d3f9afd8d9
Add a cache on get_rooms_for_local_user_where_membership_is (#17460)
As it gets used in sliding sync.

We basically invalidate it in all the same places as
`get_rooms_for_user`. Most of the changes are due to needing the
arguments you pass in to be hashable (which lists aren't)
2024-07-19 16:19:15 +01:00
Travis Ralston
677142b6a9
Fix docs on record_action to clarify the actions are applied (#17426)
This looks like a copy/paste error: the function doesn't reject
anything, but instead allows the action count to go through regardless.
The remainder of the function's documentation appears correct.
2024-07-11 14:03:13 +01:00
Eric Eastwood
3fef535ff2
Add rooms.bump_stamp to Sliding Sync /sync for easier client-side sorting (#17395)
`bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves.

Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies).

In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types.

---

In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for.

The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622
2024-07-08 13:17:08 -05:00
Quentin Gliech
c896030f67
MSC3861: allow overriding the introspection endpoint (#17406)
This makes it easier to go through an internal endpoint instead of the
public facing URL when introspecting tokens, reducing latency.
2024-07-08 14:08:11 +02:00
Erik Johnston
8e9e6f1a0a
Allow enabling sliding sync per-user (#17393)
Based on #17392
2024-07-05 13:04:27 +01:00
Shay
8f890447b0
Support MSC3916 by adding _matrix/client/v1/media/download endpoint (#17365) 2024-07-02 14:07:04 +01:00
Richard van der Hoff
2c36a679ae
Include user membership on events (#17282)
MSC4115 has now completed FCP, so we can enable it by default and switch
to the stable identifier.
2024-06-13 21:45:54 +00:00
Eric Eastwood
4a7c58642c
Add Sliding Sync /sync endpoint (initial implementation) (#17187)
Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync

This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering).

Rooms appear in the Sliding sync response based on:

 - `invite`, `join`, `knock`, `ban` membership events
 - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`)
 - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`)
 - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range.

### Example request

`POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync`

```json
{
  "lists": {
    "foo-list": {
      "ranges": [ [0, 99] ],
      "sort": [ "by_notification_level", "by_recency", "by_name" ],
      "required_state": [
        ["m.room.join_rules", ""],
        ["m.room.history_visibility", ""],
        ["m.space.child", "*"]
      ],
      "timeline_limit": 100
    }
  }
}
```

Response:
```json
{
  "next_pos": "s58_224_0_13_10_1_1_16_0_1",
  "lists": {
    "foo-list": {
      "count": 1,
      "ops": [
        {
          "op": "SYNC",
          "range": [0, 99],
          "room_ids": [
            "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server"
          ]
        }
      ]
    }
  },
  "rooms": {},
  "extensions": {}
}
```
2024-06-06 14:44:32 -05:00
Erik Johnston
52af16c561
Add a short sleep if the request is rate-limited (#17210)
This helps prevent clients from "tight-looping" retrying their request.
2024-05-18 12:03:30 +01:00
Richard van der Hoff
b548f7803a
Add support for MSC4115 (#17104)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-04-29 15:22:13 +01:00
Shay
8fb5b0f335
Improve event validation (#16908)
As the title states.
2024-03-19 17:52:53 +00:00
dependabot[bot]
1e68b56a62
Bump black from 23.10.1 to 24.2.0 (#16936) 2024-03-13 16:46:44 +00:00
Patrick Cloke
696cc9e802
Stabilize support for Retry-After header (MSC4014) (#16947) 2024-03-08 09:33:46 +00:00
Erik Johnston
23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Erik Johnston
eaad9bb156 Merge remote-tracking branch 'gitlab/clokep/license-license' into new_develop 2023-12-13 15:11:56 +00:00
David Robertson
32a59a6495
Keep track of user_ips and monthly_active_users when delegating auth (#16672)
* Describe `insert_client_ip`
* Pull out client_ips and MAU tracking to BaseAuth
* Define HAS_AUTHLIB once in tests

sick of copypasting

* Track ips and token usage when delegating auth
* Test that we track MAU and user_ips
* Don't track `__oidc_admin`
2023-11-23 12:35:37 +00:00
Patrick Cloke
8e1e62c9e0 Update license headers 2023-11-21 15:29:58 -05:00
Sumner Evans
999bd77d3a
Asynchronous Uploads (#15503)
Support asynchronous uploads as defined in MSC2246.
2023-11-15 09:19:24 -05:00
Christoph
32fd9bc673
Fix possible AttributeError when account-api is called over unix socket (#16404)
Fixes #16396
2023-10-09 10:16:07 +01:00
Patrick Cloke
06bbf1029c
Convert simple_select_list_paginate_txn to return tuples. (#16433) 2023-10-06 11:41:57 -04:00
Patrick Cloke
7ec0a141b4
Convert more cached return values to immutable types (#16356) 2023-09-20 07:48:55 -04:00
Erik Johnston
954921736b
Refactor get_user_by_id (#16316) 2023-09-14 12:46:30 +01:00
Patrick Cloke
32fb264120 Merge remote-tracking branch 'origin/release-v1.92' into develop 2023-09-06 13:08:22 -04:00
Patrick Cloke
55c20da4a3 Merge remote-tracking branch 'origin/release-v1.91' into release-v1.92 2023-09-06 11:25:28 -04:00
Quentin Gliech
1940d990a3
Revert MSC3861 introspection cache, admin impersonation and account lock (#16258) 2023-09-06 15:19:51 +01:00
Patrick Cloke
a2b8814d64
Fix incorrect docstring for Ratelimiter. (#16255) 2023-09-05 12:11:05 -04:00
Patrick Cloke
ea75346f6a
Track presence state per-device and combine to a user state. (#16066)
Tracks presence on an individual per-device basis and combine
the per-device state into a per-user state. This should help in
situations where a user has multiple devices with conflicting status
(e.g. one is syncing with unavailable and one is syncing with online).

The tie-breaking is done by priority:

    BUSY > ONLINE > UNAVAILABLE > OFFLINE
2023-09-05 09:58:51 -04:00
David Robertson
62a1a9be52
Describe which rate limiter was hit in logs (#16135) 2023-08-30 00:39:39 +01:00
Will Hunt
0538e3e2db
Add Retry-After to M_LIMIT_EXCEEDED error responses (#16136)
Implements MSC4041 behind an experimental configuration flag.
2023-08-24 10:40:26 -04:00
Shay
69048f7b48
Add an admin endpoint to allow authorizing server to signal token revocations (#16125) 2023-08-22 14:15:34 +00:00
Mathieu Velten
2d15e39684
MSC3861: allow impersonation by an admin using a query param (#16132) 2023-08-18 15:46:46 +02:00
Erik Johnston
6130afb862
Add response time metrics for introspection requests (#16131)
See #16119
2023-08-18 12:16:00 +01:00
Shay
54a51ff6c1
Cache token introspection response from OIDC provider (#16117) 2023-08-17 10:53:10 -07:00
Patrick Cloke
ad3f43be9a
Run pyupgrade for python 3.7 & 3.8. (#16110) 2023-08-15 08:11:20 -04:00
Mathieu Velten
dac97642e4
Implements admin API to lock an user (MSC3939) (#15870) 2023-08-10 09:10:55 +00:00
Patrick Cloke
6d81aec09f
Support room version 11 (#15912)
And fix a bug in the implementation of the updated redaction
format (MSC2174) where the top-level redacts field was not
properly added for backwards-compatibility.
2023-07-18 08:44:59 -04:00
Eric Eastwood
1c802de626
Re-introduce the outbound federation proxy (#15913)
Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`).

This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
2023-07-18 09:49:21 +01:00
Eric Eastwood
0f02f0b4da
Remove experimental MSC2716 implementation to incrementally import history into existing rooms (#15748)
Context for why we're removing the implementation:

 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010
 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734

Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737

Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
2023-06-16 14:12:24 -05:00
Patrick Cloke
f880e64b11
Stabilize support for MSC3952: Intentional mentions. (#15520) 2023-06-06 09:11:07 +01:00
Shay
d0c4257f14
N + 3: Read from column full_user_id rather than user_id of tables profiles and user_filters (#15649) 2023-06-02 17:24:13 -07:00
Patrick Cloke
c01343de43
Add stricter mypy options (#15694)
Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any,
and disallow_incomplete_defs.
2023-05-31 07:18:29 -04:00
Quentin Gliech
ceb3dd77db Enforce that an admin token also has the basic Matrix API scope 2023-05-30 09:43:06 -04:00
Quentin Gliech
f739bde962 Reject tokens with multiple device scopes 2023-05-30 09:43:06 -04:00
Quentin Gliech
98afc57d59 Make OIDC scope constants 2023-05-30 09:43:06 -04:00
Quentin Gliech
14a5be9c4d Handle errors when introspecting tokens
This returns a proper 503 when the introspection endpoint is not working
for some reason, which should avoid logging out clients in those cases.
2023-05-30 09:43:06 -04:00
Quentin Gliech
4d0231b364 Make AS tokens work & allow ASes to /register 2023-05-30 09:43:06 -04:00
Quentin Gliech
c008b44b4f Add an admin token for MAS -> Synapse calls 2023-05-30 09:43:06 -04:00