Commit Graph

986 Commits

Author SHA1 Message Date
Patrick Cloke
6d7523ef14
Batch fetch bundled references (#14508)
Avoid an n+1 query problem and fetch the bundled aggregations for
m.reference relations in a single query instead of a query per event.

This applies similar logic for as was previously done for edits in
8b309adb43 (#11660; threads
in b65acead42 (#11752); and
annotations in 1799a54a54 (#14491).
2022-11-22 09:41:09 -05:00
Patrick Cloke
1799a54a54
Batch fetch bundled annotations (#14491)
Avoid an n+1 query problem and fetch the bundled aggregations for
m.annotation relations in a single query instead of a query per event.

This applies similar logic for as was previously done for edits in
8b309adb43 (#11660) and threads
in b65acead42 (#11752).
2022-11-22 07:26:11 -05:00
Ashish Kumar
a3623af74e
Add an Admin API endpoint for looking up users based on 3PID (#14405) 2022-11-11 15:38:17 +00:00
Shay
7894251bce
Correctly create power level event during initial room creation (#14361) 2022-11-07 13:38:50 -08:00
Tulir Asokan
a4b1f64562
Fix /refresh endpoint version (#14364) 2022-11-04 16:43:51 +00:00
Brendan Abolivier
86c5a710d8
Implement MSC3912: Relation-based redactions (#14260)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-11-03 16:21:31 +00:00
David Robertson
dbfc9b803e
Fix dehydrated device REST checks (#14336) 2022-10-31 20:31:43 +00:00
Quentin Gliech
cc3a52b33d
Support OIDC backchannel logouts (#11414)
If configured an OIDC IdP can log a user's session out of
Synapse when they log out of the identity provider.

The IdP sends a request directly to Synapse (and must be
configured with an endpoint) when a user logs out.
2022-10-31 13:07:30 -04:00
Olivier Wilkinson (reivilibre)
1335367ca7 Merge branch 'master' into develop 2022-10-28 15:59:51 +01:00
reivilibre
6a6e1e8c07
Fix room creation being rate limited too aggressively since Synapse v1.69.0. (#14314)
* Introduce a test for the old behaviour which we want to restore

* Reintroduce the old behaviour in a simpler way

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Use 1 credit instead of 2 for creating a room: be more lenient than before

Notably, the UI in Element Web was still broken after restoring to prior behaviour.

After discussion, we agreed that it would be sensible to increase the limit.

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-10-28 11:53:34 +01:00
Olivier Wilkinson (reivilibre)
85fcbba595 Merge branch 'release-v1.70' into develop 2022-10-25 15:39:35 +01:00
Quentin Gliech
9192d74b0b
Refactor OIDC tests to better mimic an actual OIDC provider. (#13910)
This implements a fake OIDC server, which intercepts calls to the HTTP client.
Improves accuracy of tests by covering more internal methods.

One particular example was the ID token validation, which previously mocked.

This uncovered an incorrect dependency: Synapse actually requires at least
authlib 0.15.1, not 0.14.0.
2022-10-25 14:25:02 +00:00
Patrick Cloke
4dd7aa371b
Properly update the threads table when thread events are redacted. (#14248)
When the last event in a thread is redacted we need to update
the threads table:

* Find the new latest event in the thread and store it into the table; or
* Remove the thread from the table if it is no longer a thread (i.e. all
  events in the thread were redacted).
2022-10-21 09:11:19 -04:00
Tadeusz Sośnierz
1433b5d5b6
Show erasure status when listing users in the Admin API (#14205)
* Show erasure status when listing users in the Admin API

* Use USING when joining erased_users

* Add changelog entry

* Revert "Use USING when joining erased_users"

This reverts commit 30bd2bf106415caadcfdbdd1b234ef2b106cc394.

* Make the erased check work on postgres

* Add a testcase for showing erased user status

* Appease the style linter

* Explicitly convert `erased` to bool to make SQLite consistent with Postgres

This also adds us an easy way in to fix the other accidentally integered columns.

* Move erasure status test to UsersListTestCase

* Include user erased status when fetching user info via the admin API

* Document the erase status in user_admin_api

* Appease the linter and mypy

* Signpost comments in tests

Co-authored-by: Tadeusz Sośnierz <tadeusz@sosnierz.com>
Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
2022-10-21 13:52:44 +01:00
Patrick Cloke
755bfeee3a
Use servlets for /key/ endpoints. (#14229)
To fix the response for unknown endpoints under that prefix.

See MSC3743.
2022-10-20 11:32:47 -04:00
Eric Eastwood
fa8616e65c
Fix MSC3030 /timestamp_to_event returning outliers that it has no idea whether are near a gap or not (#14215)
Fix MSC3030 `/timestamp_to_event` endpoint returning `outliers` that it has no idea whether are near a gap or not (and therefore unable to determine whether it's actually the closest event). The reason Synapse doesn't know whether an `outlier` is next to a gap is because our gap checks rely on entries in the `event_edges`, `event_forward_extremeties`, and `event_backward_extremities` tables which is [not the case for `outliers`](2c63cdcc3f/docs/development/room-dag-concepts.md (outliers)).

Also fixes MSC3030 Complement `can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint` test flake.  Although this acted flakey in Complement, if `sync_partial_state` raced and beat us before `/timestamp_to_event`, then even if we retried the failing `/context` request it wouldn't work until we made this Synapse change. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.

Fix  https://github.com/matrix-org/synapse/issues/13944


### Why did this fail before? Why was it flakey?

Sleuthing the server logs on the [CI failure](https://github.com/matrix-org/synapse/actions/runs/3149623842/jobs/5121449357#step:5:5805), it looks like `hs2:/timestamp_to_event` found `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` event locally. Then when we went and asked for it via `/context`, since it's an `outlier`, it was filtered out of the results -> `You don't have permission to access that event.`

This is reproducible when `sync_partial_state` races and persists `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` before we evaluate `get_event_for_timestamp(...)`. To consistently reproduce locally, just add a delay at the [start of `get_event_for_timestamp(...)`](cb20b885cb/synapse/handlers/room.py (L1470-L1496)) so it always runs after `sync_partial_state` completes.

```py
from twisted.internet import task as twisted_task
d = twisted_task.deferLater(self.hs.get_reactor(), 3.5)
await d
```

In a run where it passes, on `hs2`, `get_event_for_timestamp(...)` finds a different event locally which is next to a gap and we request from a closer one from `hs1` which gets backfilled. And since the backfilled event is not an `outlier`, it's returned as expected during `/context`.

With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.
2022-10-18 19:46:25 -05:00
Hugh Nimmo-Smith
4eaf3eb840
Implementation of HTTP 307 response for MSC3886 POST endpoint (#14018)
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Andrew Morgan <andrewm@element.io>
2022-10-18 15:52:25 +00:00
Patrick Cloke
4283bd1cf9
Support filtering the /messages API by relation type (MSC3874). (#14148)
Gated behind an experimental configuration flag.
2022-10-17 11:32:11 -04:00
Patrick Cloke
126a15794c
Do not allow a None-limit on PaginationConfig. (#14146)
The callers either set a default limit or manually handle a None-limit
later on (by setting a default value).

Update the callers to always instantiate PaginationConfig with a default
limit and then assume the limit is non-None.
2022-10-14 12:30:05 +00:00
Patrick Cloke
c3e4edb4d6
Stabilize the threads API. (#14175)
Stabilize the threads API (MSC3856) by supporting (only) the v1
path for the endpoint.

This also marks the API as safe for workers since it is a read-only
API.
2022-10-14 07:16:50 -04:00
Patrick Cloke
3bbe532abb
Add an API for listing threads in a room. (#13394)
Implement the /threads endpoint from MSC3856.

This is currently unstable and behind an experimental configuration
flag.

It includes a background update to backfill data, results from
the /threads endpoint will be partial until that finishes.
2022-10-13 08:02:11 -04:00
Patrick Cloke
00c93d2e7e
Be more lenient in the oEmbed response parsing. (#14089)
Attempt to parse any valid information from an oEmbed response
(instead of bailing at the first unexpected data). This should allow
for more partial oEmbed data to be returned, resulting in better /
more URL previews, even if those URL previews are only partial.
2022-10-07 09:29:43 -04:00
Patrick Cloke
0b037d6c91
Fix handling of public rooms filter with a network tuple. (#14053)
Fixes two related bugs:

* The handling of `[null]` for a `room_types` filter was incorrect.
* The ordering of arguments when providing both a network tuple
  and room type field was incorrect.
2022-10-05 12:49:52 +00:00
Patrick Cloke
b706111b78
Do not return unspecced original_event field when using the stable /relations endpoint. (#14025)
Keep the old behavior (of including the original_event field) for any
requests to the /unstable version of the endpoint, but do not include
the field when the /v1 version is used.

This should avoid new clients from depending on this field, but will
not help with current dependencies.
2022-10-03 16:47:15 +00:00
lukasdenk
719488dda8
Add query parameter ts to allow appservices set the origin_server_ts for state events. (#11866)
MSC3316 declares that both /rooms/{roomId}/send and /rooms/{roomId}/state
should accept a ts parameter for appservices. This change expands support
to /state and adds tests.
2022-10-03 13:30:45 +00:00
Erik Johnston
535f8c8f7d
Skip filtering during push if there are no push actions (#13992) 2022-09-30 17:40:33 +01:00
Brendan Abolivier
be76cd8200
Allow admins to require a manual approval process before new accounts can be used (using MSC3866) (#13556) 2022-09-29 15:23:24 +02:00
Hugh Nimmo-Smith
e5fdf16d46
Expose MSC3882 only be under an unstable endpoint. (#13868) 2022-09-29 07:22:27 -04:00
Shay
a2cf66a94d
Prepatory work for batching events to send (#13487)
This PR begins work on batching up events during the creation of a room. The PR splits out the creation and sending/persisting of the events. The first three events in the creation of the room-creating the room, joining the creator to the room, and the power levels event are sent sequentially, while the subsequent events are created and collected to be sent at the end of the function. This is currently done by appending them to a list and then iterating over the list to send, the next step (after this PR) would be to send and persist the collected events as a batch.
2022-09-28 10:39:03 +01:00
Patrick Cloke
87fe9db467
Support the stable dir parameter for /relations. (#13920)
Since MSC3715 has passed FCP, the stable parameter can be used.

This currently falls back to the unstable parameter if the stable
parameter is not provided (and MSC3715 support is enabled in
the configuration).
2022-09-27 10:47:34 -04:00
Patrick Cloke
b7272b73aa
Properly paginate forward in the /relations API. (#13840)
This fixes a bug where the `/relations` API with `dir=f` would
skip the first item of each page (except the first page), causing
incomplete data to be returned to the client.
2022-09-22 12:47:49 +00:00
Hugh Nimmo-Smith
0fd2f2d460
Implementation of MSC3882 login token request (#13722) 2022-09-21 15:12:29 +00:00
Brendan Abolivier
8ae42ab8fa
Support enabling/disabling pushers (from MSC3881) (#13799)
Partial implementation of MSC3881
2022-09-21 14:39:01 +00:00
Quentin Gliech
74f60cec92
Add an admin API endpoint to find a user based on its external ID in an auth provider. (#13810) 2022-09-16 12:29:03 +00:00
David Robertson
742f9f9d78
A third batch of Pydantic validation for rest/client/account.py (#13736) 2022-09-15 18:36:02 +01:00
Andrew Morgan
918c74bfb5
Add a MXCUri class to make working with mxc uri's easier. (#13162) 2022-09-15 12:57:16 +00:00
Dirk Klimpel
f799eac7ea
Add timestamp to user's consent (#13741)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-09-08 15:41:48 +00:00
reivilibre
d3d9ca156e
Cancel the processing of key query requests when they time out. (#13680) 2022-09-07 12:03:32 +01:00
Connor Davis
bb5b47b62a
Add Admin API to Fetch Messages Within a Particular Window (#13672)
This adds two new admin APIs that allow us to fetch messages from a room within a particular time.
2022-09-07 10:54:44 +01:00
Šimon Brandner
0e99f07952
Remove support for unstable private read receipts (#13653)
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-09-01 13:31:54 +01:00
Jacek Kuśnierz
84ddcd7bbf
Drop support for calling /_matrix/client/v3/rooms/{roomId}/invite without an id_access_token (#13241)
Fixes #13206

Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
2022-08-31 12:10:25 +00:00
David Robertson
a160406d24
Fix admin List Room API return type on sqlite (#13509) 2022-08-31 10:38:16 +00:00
Eric Eastwood
d58615c82c
Directly lookup local membership instead of getting all members in a room first (get_users_in_room mis-use) (#13608)
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:13:12 -05:00
David Robertson
956e015413
Drop support for delegating email validation, round 2 (#13596) 2022-08-23 11:40:00 +00:00
Dirk Klimpel
37f329c9ad
Fix that sending server notices fail if avatar is None (#13566)
Indroduced in #11846.
2022-08-23 09:48:35 +01:00
Quentin Gliech
3dd175b628
synapse.api.auth.Auth cleanup: make permission-related methods use Requester instead of the UserID (#13024)
Part of #13019

This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.

It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
2022-08-22 14:17:59 +01:00
Andrew Morgan
f9f03426de
Implement MSC3852: Expose last_seen_user_agent to users for their own devices; also expose to Admin API (#13549) 2022-08-19 16:17:10 +00:00
reivilibre
8bdf2bd31e
Fix a bug in the /event_reports Admin API which meant that the total count could be larger than the number of results you can actually query for. (#13525)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-08-17 18:08:23 +00:00
Dirk Klimpel
d75512d19e
Add forgotten status to Room Details API (#13503) 2022-08-17 09:42:01 +00:00
David Robertson
d642ce4b32
Use Pydantic to systematically validate a first batch of endpoints in synapse.rest.client.account. (#13188) 2022-08-15 19:05:57 +00:00