Commit Graph

686 Commits

Author SHA1 Message Date
Shay
1c95ddd09b
Batch up storing state groups when creating new room (#14918) 2023-02-24 13:15:29 -08:00
dependabot[bot]
9bb2eac719
Bump black from 22.12.0 to 23.1.0 (#15103) 2023-02-22 15:29:09 -05:00
Patrick Cloke
c9b9143655
Fix-up type hints in tests/server.py. (#15084)
This file was being ignored by mypy, we remove that
and add the missing type hints & deal with any fallout.
2023-02-17 18:19:38 +00:00
David Robertson
ffc2ee521d
Use mypy 1.0 (#15052)
* Update mypy and mypy-zope
* Remove unused ignores

These used to suppress

```
synapse/storage/engines/__init__.py:28: error: "__new__" must return a
class instance (got "NoReturn")  [misc]
```

and

```
synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons"  [attr-defined]
```

(note that we check `hasattr(e, "reasons")` above)

* Avoid empty body warnings, sometimes by marking methods as abstract

E.g.

```
tests/handlers/test_register.py:58: error: Missing return statement  [empty-body]
tests/handlers/test_register.py:108: error: Missing return statement  [empty-body]
```

* Suppress false positive about `JaegerConfig`

Complaint was

```
synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context  [truthy-function]
```

* Fix not calling `is_state()`

Oops!

```
tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context  [truthy-function]
```

* Suppress false positives from ParamSpecs

````
synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]"  [arg-type]
````

* Drive-by improvement to `wrapping_logic` annotation

* Workaround false "unreachable" positives

See https://github.com/Shoobx/mypy-zope/issues/91

```
tests/http/test_proxyagent.py:626: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:762: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:826: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:838: error: Statement is unreachable  [unreachable]
tests/http/test_proxyagent.py:845: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable  [unreachable]
tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:60: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:93: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:127: error: Statement is unreachable  [unreachable]
tests/logging/test_remote_handler.py:152: error: Statement is unreachable  [unreachable]
```

* Changelog

* Tweak DBAPI2 Protocol to be accepted by mypy 1.0

Some extra context in:
- https://github.com/matrix-org/python-canonicaljson/pull/57
- https://github.com/python/mypy/issues/6002
- https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected

* Pull in updated canonicaljson lib

so the protocol check just works

* Improve comments in opentracing

I tried to workaround the ignores but found it too much trouble.

I think the corresponding issue is
https://github.com/python/mypy/issues/12909. The mypy repo has a PR
claiming to fix this (https://github.com/python/mypy/pull/14677) which
might mean this gets resolved soon?

* Better annotation for INTERACTIVE_AUTH_CHECKERS

* Drive-by AUTH_TYPE annotation, to remove an ignore
2023-02-16 16:09:11 +00:00
Patrick Cloke
42aea0d8af
Add final type hint to tests.unittest. (#15072)
Adds a return type to HomeServerTestCase.make_homeserver and deal
with any variables which are no longer Any.
2023-02-14 14:03:35 -05:00
reivilibre
e9b1ff9f31
Prevent clients from reporting nonexistent events. (#13779) 2023-02-14 15:50:59 +00:00
Patrick Cloke
fc35e0673f
Add missing type hints in tests (#14879)
* FIx-up type hints in tests.logging.
* Add missing type hints to test_transactions.
2023-01-26 14:45:24 -05:00
Andrew Morgan
836c592f15
Fix type hints in knocking tests. (#14887) 2023-01-25 14:38:20 -05:00
David Robertson
80d44060c9
Faster joins: omit partial rooms from eager syncs until the resync completes (#14870)
* Allow `AbstractSet` in `StrCollection`

Or else frozensets are excluded. This will be useful in an upcoming
commit where I plan to change a function that accepts `List[str]` to
accept `StrCollection` instead.

* `rooms_to_exclude` -> `rooms_to_exclude_globally`

I am about to make use of this exclusion mechanism to exclude rooms for
a specific user and a specific sync. This rename helps to clarify the
distinction between the global config and the rooms to exclude for a
specific sync.

* Better function names for internal sync methods

* Track a list of excluded rooms on SyncResultBuilder

I plan to feed a list of partially stated rooms for this sync to ignore

* Exclude partial state rooms during eager sync

using the mechanism established in the previous commit

* Track un-partial-state stream in sync tokens

So that we can work out which rooms have become fully-stated during a
given sync period.

* Fix mutation of `@cached` return value

This was fouling up a complement test added alongside this PR.
Excluding a room would mean the set of forgotten rooms in the cache
would be extended. This means that room could be erroneously considered
forgotten in the future.

Introduced in #12310, Synapse 1.57.0. I don't think this had any
user-visible side effects (until now).

* SyncResultBuilder: track rooms to force as newly joined

Similar plan as before. We've omitted rooms from certain sync responses;
now we establish the mechanism to reintroduce them into future syncs.

* Read new field, to present rooms as newly joined

* Force un-partial-stated rooms to be newly-joined

for eager incremental syncs only, provided they're still fully stated

* Notify user stream listeners to wake up long polling syncs

* Changelog

* Typo fix

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Unnecessary list cast

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Rephrase comment

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Another comment

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

* Fixup merge(?)

* Poke notifier when receiving un-partial-stated msg over replication

* Fixup merge whoops

Thanks MV :)

Co-authored-by: Mathieu Velen <mathieuv@matrix.org>

Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2023-01-23 15:44:39 +00:00
Catalan Lover
e8f2bf5c40
Change default room version to 10. Implements MSC3904 (#14111)
* Change Documentation to have v10 as default room version

* Change Default Room version to 10

* Add changelog entry for default room version swap

* Add changelog entry for v10 default room version in docs

* Clarify doc changelog entry

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Improve Documentation changes.

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Update Changelog entry to have correct format

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>

* Update Spec Version to 1.5

* Only need 1 changelog.

* Fix test.

* Update "Changed in" line

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-01-18 18:59:48 +00:00
Richard van der Hoff
06ab64f201
Implement MSC3925: changes to bundling of edits (#14811)
Two parts to this:

 * Bundle the whole of the replacement with any edited events. This is backwards-compatible so I haven't put it behind a flag.
 * Optionally, inhibit server-side replacement of edited events. This has scope to break things, so it is currently disabled by default.
2023-01-10 16:31:28 +00:00
Jeyachandran Rathnam
5e0888076f
Disable sending confirmation email when 3pid is disabled #14682 (#14725)
* Fixes #12277 :Disable sending confirmation email when 3pid is disabled

* Fix test_add_email_if_disabled test case to reflect changes to enable_3pid_changes flag

* Add changelog file

* Rename newsfragment.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-01-09 11:12:03 +00:00
Patrick Cloke
9d8a3234ba
Respond with proper error responses on unknown paths. (#14621)
Returns a proper 404 with an errcode of M_RECOGNIZED for
unknown endpoints per MSC3743.
2022-12-08 11:37:05 -05:00
Nick Mills-Barrett
60c3fea327
Reject receipt requests with invalid room or event IDs. (#14632)
If the room or event IDs are empty or of an invalid form they
should be rejected.
2022-12-07 17:35:41 +00:00
Eric Eastwood
8f10c8b054
Move MSC3030 /timestamp_to_event endpoint to stable v1 location (#14471)
Fix https://github.com/matrix-org/synapse/issues/14390

 - Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`
 - Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`

Complement test changes: https://github.com/matrix-org/complement/pull/559
2022-11-28 15:54:18 -06:00
Patrick Cloke
6d7523ef14
Batch fetch bundled references (#14508)
Avoid an n+1 query problem and fetch the bundled aggregations for
m.reference relations in a single query instead of a query per event.

This applies similar logic for as was previously done for edits in
8b309adb43 (#11660; threads
in b65acead42 (#11752); and
annotations in 1799a54a54 (#14491).
2022-11-22 09:41:09 -05:00
Patrick Cloke
1799a54a54
Batch fetch bundled annotations (#14491)
Avoid an n+1 query problem and fetch the bundled aggregations for
m.annotation relations in a single query instead of a query per event.

This applies similar logic for as was previously done for edits in
8b309adb43 (#11660) and threads
in b65acead42 (#11752).
2022-11-22 07:26:11 -05:00
Shay
7894251bce
Correctly create power level event during initial room creation (#14361) 2022-11-07 13:38:50 -08:00
Tulir Asokan
a4b1f64562
Fix /refresh endpoint version (#14364) 2022-11-04 16:43:51 +00:00
Brendan Abolivier
86c5a710d8
Implement MSC3912: Relation-based redactions (#14260)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-11-03 16:21:31 +00:00
David Robertson
dbfc9b803e
Fix dehydrated device REST checks (#14336) 2022-10-31 20:31:43 +00:00
Quentin Gliech
cc3a52b33d
Support OIDC backchannel logouts (#11414)
If configured an OIDC IdP can log a user's session out of
Synapse when they log out of the identity provider.

The IdP sends a request directly to Synapse (and must be
configured with an endpoint) when a user logs out.
2022-10-31 13:07:30 -04:00
Olivier Wilkinson (reivilibre)
1335367ca7 Merge branch 'master' into develop 2022-10-28 15:59:51 +01:00
reivilibre
6a6e1e8c07
Fix room creation being rate limited too aggressively since Synapse v1.69.0. (#14314)
* Introduce a test for the old behaviour which we want to restore

* Reintroduce the old behaviour in a simpler way

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Use 1 credit instead of 2 for creating a room: be more lenient than before

Notably, the UI in Element Web was still broken after restoring to prior behaviour.

After discussion, we agreed that it would be sensible to increase the limit.

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-10-28 11:53:34 +01:00
Olivier Wilkinson (reivilibre)
85fcbba595 Merge branch 'release-v1.70' into develop 2022-10-25 15:39:35 +01:00
Quentin Gliech
9192d74b0b
Refactor OIDC tests to better mimic an actual OIDC provider. (#13910)
This implements a fake OIDC server, which intercepts calls to the HTTP client.
Improves accuracy of tests by covering more internal methods.

One particular example was the ID token validation, which previously mocked.

This uncovered an incorrect dependency: Synapse actually requires at least
authlib 0.15.1, not 0.14.0.
2022-10-25 14:25:02 +00:00
Patrick Cloke
4dd7aa371b
Properly update the threads table when thread events are redacted. (#14248)
When the last event in a thread is redacted we need to update
the threads table:

* Find the new latest event in the thread and store it into the table; or
* Remove the thread from the table if it is no longer a thread (i.e. all
  events in the thread were redacted).
2022-10-21 09:11:19 -04:00
Eric Eastwood
fa8616e65c
Fix MSC3030 /timestamp_to_event returning outliers that it has no idea whether are near a gap or not (#14215)
Fix MSC3030 `/timestamp_to_event` endpoint returning `outliers` that it has no idea whether are near a gap or not (and therefore unable to determine whether it's actually the closest event). The reason Synapse doesn't know whether an `outlier` is next to a gap is because our gap checks rely on entries in the `event_edges`, `event_forward_extremeties`, and `event_backward_extremities` tables which is [not the case for `outliers`](2c63cdcc3f/docs/development/room-dag-concepts.md (outliers)).

Also fixes MSC3030 Complement `can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint` test flake.  Although this acted flakey in Complement, if `sync_partial_state` raced and beat us before `/timestamp_to_event`, then even if we retried the failing `/context` request it wouldn't work until we made this Synapse change. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.

Fix  https://github.com/matrix-org/synapse/issues/13944


### Why did this fail before? Why was it flakey?

Sleuthing the server logs on the [CI failure](https://github.com/matrix-org/synapse/actions/runs/3149623842/jobs/5121449357#step:5:5805), it looks like `hs2:/timestamp_to_event` found `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` event locally. Then when we went and asked for it via `/context`, since it's an `outlier`, it was filtered out of the results -> `You don't have permission to access that event.`

This is reproducible when `sync_partial_state` races and persists `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` before we evaluate `get_event_for_timestamp(...)`. To consistently reproduce locally, just add a delay at the [start of `get_event_for_timestamp(...)`](cb20b885cb/synapse/handlers/room.py (L1470-L1496)) so it always runs after `sync_partial_state` completes.

```py
from twisted.internet import task as twisted_task
d = twisted_task.deferLater(self.hs.get_reactor(), 3.5)
await d
```

In a run where it passes, on `hs2`, `get_event_for_timestamp(...)` finds a different event locally which is next to a gap and we request from a closer one from `hs1` which gets backfilled. And since the backfilled event is not an `outlier`, it's returned as expected during `/context`.

With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.
2022-10-18 19:46:25 -05:00
Hugh Nimmo-Smith
4eaf3eb840
Implementation of HTTP 307 response for MSC3886 POST endpoint (#14018)
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Andrew Morgan <andrewm@element.io>
2022-10-18 15:52:25 +00:00
Patrick Cloke
4283bd1cf9
Support filtering the /messages API by relation type (MSC3874). (#14148)
Gated behind an experimental configuration flag.
2022-10-17 11:32:11 -04:00
Patrick Cloke
126a15794c
Do not allow a None-limit on PaginationConfig. (#14146)
The callers either set a default limit or manually handle a None-limit
later on (by setting a default value).

Update the callers to always instantiate PaginationConfig with a default
limit and then assume the limit is non-None.
2022-10-14 12:30:05 +00:00
Patrick Cloke
c3e4edb4d6
Stabilize the threads API. (#14175)
Stabilize the threads API (MSC3856) by supporting (only) the v1
path for the endpoint.

This also marks the API as safe for workers since it is a read-only
API.
2022-10-14 07:16:50 -04:00
Patrick Cloke
3bbe532abb
Add an API for listing threads in a room. (#13394)
Implement the /threads endpoint from MSC3856.

This is currently unstable and behind an experimental configuration
flag.

It includes a background update to backfill data, results from
the /threads endpoint will be partial until that finishes.
2022-10-13 08:02:11 -04:00
Patrick Cloke
0b037d6c91
Fix handling of public rooms filter with a network tuple. (#14053)
Fixes two related bugs:

* The handling of `[null]` for a `room_types` filter was incorrect.
* The ordering of arguments when providing both a network tuple
  and room type field was incorrect.
2022-10-05 12:49:52 +00:00
Patrick Cloke
b706111b78
Do not return unspecced original_event field when using the stable /relations endpoint. (#14025)
Keep the old behavior (of including the original_event field) for any
requests to the /unstable version of the endpoint, but do not include
the field when the /v1 version is used.

This should avoid new clients from depending on this field, but will
not help with current dependencies.
2022-10-03 16:47:15 +00:00
lukasdenk
719488dda8
Add query parameter ts to allow appservices set the origin_server_ts for state events. (#11866)
MSC3316 declares that both /rooms/{roomId}/send and /rooms/{roomId}/state
should accept a ts parameter for appservices. This change expands support
to /state and adds tests.
2022-10-03 13:30:45 +00:00
Erik Johnston
535f8c8f7d
Skip filtering during push if there are no push actions (#13992) 2022-09-30 17:40:33 +01:00
Brendan Abolivier
be76cd8200
Allow admins to require a manual approval process before new accounts can be used (using MSC3866) (#13556) 2022-09-29 15:23:24 +02:00
Hugh Nimmo-Smith
e5fdf16d46
Expose MSC3882 only be under an unstable endpoint. (#13868) 2022-09-29 07:22:27 -04:00
Shay
a2cf66a94d
Prepatory work for batching events to send (#13487)
This PR begins work on batching up events during the creation of a room. The PR splits out the creation and sending/persisting of the events. The first three events in the creation of the room-creating the room, joining the creator to the room, and the power levels event are sent sequentially, while the subsequent events are created and collected to be sent at the end of the function. This is currently done by appending them to a list and then iterating over the list to send, the next step (after this PR) would be to send and persist the collected events as a batch.
2022-09-28 10:39:03 +01:00
Patrick Cloke
87fe9db467
Support the stable dir parameter for /relations. (#13920)
Since MSC3715 has passed FCP, the stable parameter can be used.

This currently falls back to the unstable parameter if the stable
parameter is not provided (and MSC3715 support is enabled in
the configuration).
2022-09-27 10:47:34 -04:00
Patrick Cloke
b7272b73aa
Properly paginate forward in the /relations API. (#13840)
This fixes a bug where the `/relations` API with `dir=f` would
skip the first item of each page (except the first page), causing
incomplete data to be returned to the client.
2022-09-22 12:47:49 +00:00
Hugh Nimmo-Smith
0fd2f2d460
Implementation of MSC3882 login token request (#13722) 2022-09-21 15:12:29 +00:00
David Robertson
742f9f9d78
A third batch of Pydantic validation for rest/client/account.py (#13736) 2022-09-15 18:36:02 +01:00
reivilibre
d3d9ca156e
Cancel the processing of key query requests when they time out. (#13680) 2022-09-07 12:03:32 +01:00
Šimon Brandner
0e99f07952
Remove support for unstable private read receipts (#13653)
Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
2022-09-01 13:31:54 +01:00
Jacek Kuśnierz
84ddcd7bbf
Drop support for calling /_matrix/client/v3/rooms/{roomId}/invite without an id_access_token (#13241)
Fixes #13206

Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
2022-08-31 12:10:25 +00:00
Eric Eastwood
d58615c82c
Directly lookup local membership instead of getting all members in a room first (get_users_in_room mis-use) (#13608)
See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
2022-08-24 14:13:12 -05:00
David Robertson
956e015413
Drop support for delegating email validation, round 2 (#13596) 2022-08-23 11:40:00 +00:00
Quentin Gliech
3dd175b628
synapse.api.auth.Auth cleanup: make permission-related methods use Requester instead of the UserID (#13024)
Part of #13019

This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it.

It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
2022-08-22 14:17:59 +01:00