Commit Graph

4274 Commits

Author SHA1 Message Date
Sean Quah
5627182788
Use ParamSpec in type hints for synapse.logging.context (#12150)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-08 15:58:14 +00:00
Patrick Cloke
ca9234a9eb
Do not return allowed_room_ids from /hierarchy response. (#12175)
This field is only to be used in the Server-Server API, and not the
Client-Server API, but was being leaked when a federation response
was used in the /hierarchy API.
2022-03-08 08:09:11 -05:00
Erik Johnston
423cca9efe
Spread out sending device lists to remote hosts (#12132) 2022-03-04 11:48:15 +00:00
Richard van der Hoff
8533c8b03d
Avoid generating state groups for local out-of-band leaves (#12154)
If we locally generate a rejection for an invite received over federation, it
is stored as an outlier (because we probably don't have the state for the
room). However, currently we still generate a state group for it (even though
the state in that state group will be nonsense).

By setting the `outlier` param on `create_event`, we avoid the nonsensical
state.
2022-03-03 19:58:08 +00:00
Andrew Morgan
fb0ffa9676
Rename various ApplicationServices interested methods (#11915) 2022-03-03 18:14:09 +00:00
Patrick Cloke
1d11b452b7
Use the proper serialization format when bundling aggregations. (#12090)
This ensures that the `latest_event` field of the bundled aggregation
for threads uses the same format as the other events in the response.
2022-03-03 10:43:06 -05:00
Erik Johnston
61fd2a8f59
Limit the size of the aggregation_key (#12101)
There's no reason to let people use long keys.
2022-03-03 10:52:35 +00:00
Patrick Cloke
1103c5fe8a
Check if instances are lists, not sequences. (#12128)
As a str is a sequence, the checks were not granular
enough and would allow lists or strings, when only
lists were valid.
2022-03-02 13:18:51 +00:00
Brendan Abolivier
300ed0b8a6
Add module callbacks called for reacting to deactivation status change and profile update (#12062) 2022-03-01 15:00:03 +00:00
Richard van der Hoff
e2e1d90a5e
Faster joins: persist to database (#12012)
When we get a partial_state response from send_join, store information in the
database about it:
 * store a record about the room as a whole having partial state, and stash the
   list of member servers too.
 * flag the join event itself as having partial state
 * also, for any new events whose prev-events are partial-stated, note that
   they will *also* be partial-stated.

We don't yet make any attempt to interpret this data, so API calls (and a bunch
of other things) are just going to get incorrect data.
2022-03-01 12:49:54 +00:00
Richard van der Hoff
9d11fee8f2
Improve exception handling for concurrent execution (#12109)
* fix incorrect unwrapFirstError import

this was being imported from the wrong place

* Refactor `concurrently_execute` to use `yieldable_gather_results`

* Improve exception handling in `yieldable_gather_results`

Try to avoid swallowing so many stack traces.

* mark unwrapFirstError deprecated

* changelog
2022-03-01 09:34:30 +00:00
Patrick Cloke
7754af24ab
Remove the unstable /spaces endpoint. (#12073)
...and various code supporting it.

The /spaces endpoint was from an old version of MSC2946 and included
both a Client-Server and Server-Server API. Note that the unstable
/hierarchy endpoint (from the final version of MSC2946) is not yet
removed.
2022-02-28 18:33:00 +00:00
Erik Johnston
a711ae78a8
Add logging to /sync for debugging #11916 (#12068) 2022-02-23 14:22:22 +00:00
Richard van der Hoff
5b2b36809f
Remove more references to get_datastore (#12067)
These have snuck in since #12031 was started.

Also a couple of other cleanups while we're in the area.
2022-02-23 12:35:53 +00:00
Richard van der Hoff
e24ff8ebe3
Remove HomeServer.get_datastore() (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
Brendan Abolivier
250104d357
Implement account status endpoints (MSC3720) (#12001)
See matrix-org/matrix-doc#3720

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-02-22 15:10:10 +00:00
Richard van der Hoff
81364db49b
Run _handle_queued_pdus as a background process (#12041)
... to ensure it gets a proper log context, mostly.
2022-02-22 13:33:22 +00:00
Richard van der Hoff
3070af4809
remote join processing: get create event from state, not auth_chain (#12039)
A follow-up to #12005, in which I apparently missed that there are a bunch of other places that assume the create event is in the auth chain.
2022-02-21 19:27:35 +00:00
Richard van der Hoff
a85dde3445
Minor typing fixes (#12034)
These started failing in
https://github.com/matrix-org/synapse/pull/12031... I'm a bit mystified by how
they ever worked.
2022-02-21 18:37:04 +00:00
Dirk Klimpel
7c82da27aa
Add type hints to synapse/storage/databases/main (#11984) 2022-02-21 16:03:06 +00:00
Sean Quah
e6acd3cf4f
Upgrade mypy to version 0.931 (#12030)
Upgrade mypy to 0.931, mypy-zope to 0.3.5 and fix new complaints.
2022-02-18 15:57:26 +00:00
Brendan Abolivier
707049c6ff
Allow modules to set a display name on registration (#12009)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-17 16:54:16 +00:00
Richard van der Hoff
696acd3515
send_join response: get create event from state, not auth_chain (#12005)
msc3706 proposes changing the `/send_join` response:

> Any events returned within `state` can be omitted from `auth_chain`.

Currently, we rely on `m.room.create` being returned in `auth_chain`, but since
the `m.room.create` event must necessarily be part of the state, the above
change will break this.

In short, let's look for `m.room.create` in `state` rather than `auth_chain`.
2022-02-17 11:59:26 +00:00
Brendan Abolivier
130fd45393
Limit concurrent AS joins (#11996)
Initially introduced in matrix-org-hotfixes by e5537cf (and tweaked by later commits).

Fixes #11995

See also #4826
2022-02-16 11:16:48 +00:00
Erik Johnston
2b5643b3af
Optimise calculating device_list changes in /sync. (#11974)
For users with large accounts it is inefficient to calculate the set of
users they share a room with (and takes a lot of space in the cache).
Instead we can look at users whose devices have changed since the last
sync and check if they share a room with the syncing user.
2022-02-15 15:01:00 +00:00
Richard van der Hoff
bab2394aa9
_auth_and_persist_outliers: drop events we have already seen (#11994)
We already have two copies of this code, in 2/3 of the callers of
`_auth_and_persist_outliers`. Before I add a third, let's push it down.
2022-02-15 14:33:28 +00:00
Patrick Cloke
e44f91d678
Refactor search code to reduce function size. (#11991)
Splits the search code into a few logical functions instead of a single
unreadable function.

There are also a few additional changes for readability.

After refactoring it was clear to see there were some unused and
unnecessary variables, which were simplified.
2022-02-15 13:47:05 +00:00
Andrew Morgan
55113dd5e8
Notify users, rather than rooms, of device list updates (#11905)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-12 09:33:49 -05:00
Brendan Abolivier
0171fa5226
Remove deprecated user_may_create_room_with_invites callback (#11950)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-11 13:58:11 +00:00
Patrick Cloke
a121507cfe
Adds misc missing type hints (#11953) 2022-02-11 07:20:16 -05:00
Patrick Cloke
8c94b3abe9
Experimental support to include bundled aggregations in search results (MSC3666) (#11837) 2022-02-08 09:21:20 -05:00
Patrick Cloke
6c0984e3f0
Remove unnecessary ignores due to Twisted upgrade. (#11939)
Twisted 22.1.0 fixed some internal type hints, allowing Synapse
to remove ignore calls for parameters to connectTCP.
2022-02-08 09:15:59 -05:00
Brendan Abolivier
0640f8ebaa
Add a callback to allow modules to deny 3PID (#11854)
Part of the Tchap Synapse mainlining.

This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
2022-02-08 11:20:32 +01:00
Eric Eastwood
fef2e792be
Fix historical messages backfilling in random order on remote homeservers (MSC2716) (#11114)
Fix https://github.com/matrix-org/synapse/issues/11091
Fix https://github.com/matrix-org/synapse/issues/10764 (side-stepping the issue because we no longer have to deal with `fake_prev_event_id`)

 1. Made the `/backfill` response return messages in `(depth, stream_ordering)` order (previously only sorted by `depth`)
    - Technically, it shouldn't really matter how `/backfill` returns things but I'm just trying to make the `stream_ordering` a little more consistent from the origin to the remote homeservers in order to get the order of messages from `/messages` consistent ([sorted by `(topological_ordering, stream_ordering)`](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)).
    - Even now that we return backfilled messages in order, it still doesn't guarantee the same `stream_ordering` (and more importantly the [`/messages` order](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)) on the other server. For example, if a room has a bunch of history imported and someone visits a permalink to a historical message back in time, their homeserver will skip over the historical messages in between and insert the permalink as the next message in the `stream_order` and totally throw off the sort.
       - This will be even more the case when we add the [MSC3030 jump to date API endpoint](https://github.com/matrix-org/matrix-doc/pull/3030) so the static archives can navigate and jump to a certain date.
       - We're solving this in the future by switching to [online topological ordering](https://github.com/matrix-org/gomatrixserverlib/issues/187) and [chunking](https://github.com/matrix-org/synapse/issues/3785) which by its nature will apply retroactively to fix any inconsistencies introduced by people permalinking
 2. As we're navigating `prev_events` to return in `/backfill`, we order by `depth` first (newest -> oldest) and now also tie-break based on the `stream_ordering` (newest -> oldest). This is technically important because MSC2716 inserts a bunch of historical messages at the same `depth` so it's best to be prescriptive about which ones we should process first. In reality, I think the code already looped over the historical messages as expected because the database is already in order.
 3. Making the historical state chain and historical event chain float on their own by having no `prev_events` instead of a fake `prev_event` which caused backfill to get clogged with an unresolvable event. Fixes https://github.com/matrix-org/synapse/issues/11091 and https://github.com/matrix-org/synapse/issues/10764
 4. We no longer find connected insertion events by finding a potential `prev_event` connection to the current event we're iterating over. We now solely rely on marker events which when processed, add the insertion event as an extremity and the federating homeserver can ask about it when time calls.
    - Related discussion, https://github.com/matrix-org/synapse/pull/11114#discussion_r741514793


Before | After
--- | ---
![](https://user-images.githubusercontent.com/558581/139218681-b465c862-5c49-4702-a59e-466733b0cf45.png) | ![](https://user-images.githubusercontent.com/558581/146453159-a1609e0a-8324-439d-ae44-e4bce43ac6d1.png)



#### Why aren't we sorting topologically when receiving backfill events?

> The main reason we're going to opt to not sort topologically when receiving backfill events is because it's probably best to do whatever is easiest to make it just work. People will probably have opinions once they look at [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) which could change whatever implementation anyway.
> 
> As mentioned, ideally we would do this but code necessary to make the fake edges but it gets confusing and gives an impression of “just whyyyy” (feels icky). This problem also dissolves with online topological ordering.
>
> -- https://github.com/matrix-org/synapse/pull/11114#discussion_r741517138

See https://github.com/matrix-org/synapse/pull/11114#discussion_r739610091 for the technical difficulties
2022-02-07 15:54:13 -06:00
Andrew Morgan
cf06783d54
Remove optional state of ApplicationService.is_interested's store parameter (#11911) 2022-02-07 18:26:42 +00:00
Andrew Morgan
e03dde259b
Clean up an indirect reference to the homeserver datastore (#11914) 2022-02-07 13:25:09 +00:00
Jonathan de Jong
02632b3504
Stabilise MSC3231 (Token Based Registration) (#11867) 2022-02-04 12:15:13 +00:00
Brendan Abolivier
d80d39b035
Add a ratelimiter for 3pid invite (#11892) 2022-02-03 14:28:15 +01:00
Andrew Morgan
64ec45fc1b
Send to-device messages to application services (#11215)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-02-01 14:13:38 +00:00
Brendan Abolivier
bf60da1a60
Configurable limits on avatars (#11846)
Only allow files which file size and content types match configured
limits to be set as avatar.

Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
2022-01-28 15:41:33 +01:00
Brendan Abolivier
2d3bd9aa67
Add a module callback to set username at registration (#11790)
This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step).
This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not.

Co-authored-by: David Robertson <davidr@element.io>
2022-01-26 14:21:13 +00:00
Patrick Cloke
2897fb6b4f
Improvements to bundling aggregations. (#11815)
This is some odds and ends found during the review of #11791
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
Brendan Abolivier
95b3f952fa
Add a config flag to inhibit M_USER_IN_USE during registration (#11743)
This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work).

This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique.

More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
2022-01-26 13:02:54 +01:00
reivilibre
df54c8485a
Remove account data (including client config, push rules and ignored users) upon user deactivation. (#11621)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-01-24 13:37:00 +00:00
reivilibre
4c2096599c
Make the get_global_account_data_by_type_for_user cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
Brendan Abolivier
bfe6d5553a
Correctly await on_logged_out callbacks (#11786) 2022-01-20 19:19:40 +01:00
Andrew Morgan
7bf2d6c268
Partially revert #11675; prevent attempting to create pushers on workers (#11770) 2022-01-20 09:37:34 -05:00
Sean Quah
af13a3be29
Fix a bug that corrupted the cache of federated space hierarchies (#11775)
`FederationClient.get_room_hierarchy()` caches its return values, so
refactor the code to avoid modifying the returned room summary.
2022-01-20 11:03:42 +00:00
Patrick Cloke
68acb0a29d
Include whether the requesting user has participated in a thread. (#11577)
Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
2022-01-18 11:38:57 -05:00
Richard van der Hoff
251b5567ec
Remove log_function and its uses (#11761)
I've never found this terribly useful. I think it was added in the early days
of Synapse, without much thought as to what would actually be useful to log,
and has just been cargo-culted ever since.

Rather, it tends to clutter up debug logs with useless information.
2022-01-18 13:06:04 +00:00
lukasdenk
d8be9924ef
Add a flag to the synapse_review_recent_signups script to ignore and filter appservice users. (#11675) 2022-01-17 16:43:25 +00:00
Andrew Morgan
18862f20b5
Remove the 'password_hash' from the Users Admin API endpoint response dictionary (#11576) 2022-01-14 14:53:33 +00:00
Patrick Cloke
0c40c619aa
Include bundled aggregations in the sync response cache. (#11659) 2022-01-13 10:45:28 -05:00
Patrick Cloke
10a88ba91c
Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
reivilibre
ffd227c382
Fix docstring on add_account_data_for_user. (#11716) 2022-01-10 15:38:22 +00:00
Patrick Cloke
8e57584a58
Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)
By returning all of the m.space.child state of the space, not just
the first 50. The number of rooms returned is still capped at 50.

For the federation API this implies that the requesting server will
need to individually query for any other rooms it is not joined to.
2022-01-07 19:27:58 -05:00
Patrick Cloke
6bf81a7a61
Bundle aggregations outside of the serialization method. (#11612)
This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.

Overall this should cause no change in behavior, but is prep work
for other improvements.
2022-01-07 09:10:46 -05:00
lukasdenk
2ef1fea8d2
Make room creations denied by user_may_create_room cause an M_FORBIDDEN error to be returned, not M_UNKNOWN (#11672)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-06 13:16:42 +00:00
David Robertson
c9eb678b73
Remove debug logging for #4422 (#11693)
as per
https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
2022-01-06 12:44:36 +00:00
Travis Ralston
feb3e006d7
Fix space hierarchy endpoint to match MSC2946 (#11667)
Fixes minor discrepancies between the /hierarchy endpoint described
in MSC2946 and the implementation.

Note that the changes impact the stable and unstable /hierarchy and
unstable /spaces endpoints for both client and federation APIs.
2022-01-05 20:33:43 +00:00
David Robertson
88a78c6577
Cache empty responses from /user/devices (#11587)
If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
2022-01-05 13:33:28 +00:00
Richard van der Hoff
0fb3dd0830
Refactor the way we set outlier (#11634)
* `_auth_and_persist_outliers`: mark persisted events as outliers

Mark any events that get persisted via `_auth_and_persist_outliers` as, well,
outliers.

Currently this will be a no-op as everything will already be flagged as an
outlier, but I'm going to change that.

* `process_remote_join`: stop flagging as outlier

The events are now flagged as outliers later on, by `_auth_and_persist_outliers`.

* `send_join`: remove `outlier=True`

The events created here are returned in the result of `send_join` to
`FederationHandler.do_invite_join`. From there they are passed into
`FederationEventHandler.process_remote_join`, which passes them to
`_auth_and_persist_outliers`... which sets the `outlier` flag.

* `get_event_auth`: remove `outlier=True`

stop flagging the events returned by `get_event_auth` as outliers. This method
is only called by `_get_remote_auth_chain_for_event`, which passes the results
into `_auth_and_persist_outliers`, which will flag them as outliers.

* `_get_remote_auth_chain_for_event`: remove `outlier=True`

we pass all the events into `_auth_and_persist_outliers`, which will now flag
the events as outliers.

* `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter

This param is now never set to True, so we can remove it.

* `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param

This is no longer set anywhere, so we can remove it.

* `get_pdu`: remove unused `outlier` parameter

... and chase it down into `get_pdu_from_destination_raw`.

* `event_from_pdu_json`: remove redundant `outlier` param

This is never set to `True`, so can be removed.

* changelog

* update docstring
2022-01-05 12:26:11 +00:00
Richard van der Hoff
b38bdae3a2
Fix AssertionErrors after purging events (#11642)
* Fix AssertionErrors after purging events

If you purged a bunch of events from your database, and then restarted synapse
without receiving more events, then you would get a bunch of AssertionErrors on
restart.

This fixes the situation by rewinding the stream processors.

* `check-newsfragment`: ignore deleted newsfiles
2022-01-04 16:36:33 +00:00
Richard van der Hoff
878aa55293
FederationClient.backfill: stop flagging events as outliers (#11632)
Events returned by `backfill` should not be flagged as outliers.

Fixes:

```
AssertionError: null
  File "synapse/handlers/federation.py", line 313, in try_backfill
    dom, room_id, limit=100, extremities=extremities
  File "synapse/handlers/federation_event.py", line 517, in backfill
    await self._process_pulled_events(dest, events, backfilled=True)
  File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events
    await self._process_pulled_event(origin, ev, backfilled=backfilled)
  File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event
    assert not event.internal_metadata.is_outlier()
```

See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992

Fixes #8894.
2022-01-04 16:31:32 +00:00
Richard van der Hoff
2359ee3864
Remove redundant get_current_events_token (#11643)
* Push `get_room_{min,max_stream_ordering}` into StreamStore

Both implementations of this are identical, so we may as well push it down and
get rid of the abstract base class nonsense.

* Remove redundant `StreamStore` class

This is empty now

* Remove redundant `get_current_events_token`

This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid
of it.

* newsfile
2022-01-04 16:10:27 +00:00
Patrick Cloke
cbd82d0b2d
Convert all namedtuples to attrs. (#11665)
To improve type hints throughout the code.
2021-12-30 18:47:12 +00:00
Patrick Cloke
f58b300d27
Do not attempt to bundled aggregations for /members and /state. (#11623)
Both of those APIs return state events, which will not have bundled
aggregations added anyway.
2021-12-29 08:02:03 -05:00
Patrick Cloke
b6102230a7
Add type hints to event_push_actions. (#11594) 2021-12-21 13:25:34 +00:00
Richard van der Hoff
2215954147
Various opentracing enhancements (#11619)
* Wrap `auth.get_user_by_req` in an opentracing span

give `get_user_by_req` its own opentracing span, since it can result in a
non-trivial number of sub-spans which it is useful to group together.

This requires a bit of reorganisation because it also sets some tags (and may
force tracing) on the servlet span.

* Emit opentracing span for encoding json responses

This can be a significant time sink.

* Rename all sync spans with a prefix

* Write an opentracing span for encoding sync response

* opentracing span to group generate_room_entries

* opentracing spans within sync.encode_response

* changelog

* Use the `trace` decorator instead of context managers
2021-12-21 11:10:36 +00:00
Patrick Cloke
dd47788752
Do not bundle aggregations for APIs which shouldn't include them. (#11592)
And make bundling aggregations opt-in, instead of opt-out to avoid
having APIs to include extraneous data (and being much heavier than
necessary).
2021-12-20 14:14:38 -05:00
Richard van der Hoff
c3e38b88f2
Improve opentracing support for ResponseCache (#11607)
This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
2021-12-20 18:12:08 +00:00
Sean Quah
c7fe32edb4
Add type hints to synapse/storage/databases/main/room.py (#11575) 2021-12-15 18:00:48 +00:00
Patrick Cloke
323151b787
Convert EventStreamResult to attrs. (#11574) 2021-12-15 11:10:02 -05:00
Sean Quah
ecfcd9bbbe
Add type hints to synapse/storage/databases/main/e2e_room_keys.py (#11549) 2021-12-14 17:46:47 +00:00
Sean Quah
0147b3de20
Add missing type hints to synapse.logging.context (#11556) 2021-12-14 17:35:28 +00:00
reivilibre
e5cdb9e233
Make get_device return None if the device doesn't exist rather than raising an exception. (#11565)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-12-13 15:39:43 +00:00
Eric Eastwood
aa8708ebed
Allow events to be created with no prev_events (MSC2716) (#11243)
The event still needs to have `auth_events` defined to be valid.

Split out from https://github.com/matrix-org/synapse/pull/11114
2021-12-10 23:08:51 -06:00
David Robertson
fd2dadb815
Adjust _get_rooms_changed comments (#11550)
C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
2021-12-10 19:19:48 +00:00
Hubert Chathi
b47d10dc46
Support unprefixed versions of fallback key property names. (#11541) 2021-12-09 11:41:27 +00:00
Patrick Cloke
d93362d87f
Add a constant for receipt types (m.read). (#11531)
And expand some type hints in the receipts storage module.
2021-12-08 12:26:29 -05:00
Hubert Chathi
8541809cb9
Send and handle cross-signing messages using the stable prefix. (#10520) 2021-12-08 10:01:38 +00:00
David Robertson
14d593f72d
Refactors in _generate_sync_entry_for_rooms (#11515)
* Move sync_token up to the top
* Pull out _get_ignored_users
* Try to signpost the body of `_generate_sync_entry_for_rooms`
* Pull out _calculate_user_changes

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-07 12:42:05 +00:00
David Robertson
2a3ec6facf
Correctly register shutdown handler for presence workers (#11518)
Fixes #11517
2021-12-07 12:34:38 +00:00
Richard van der Hoff
b1ecd19c5d
Fix 'delete room' admin api to work on incomplete rooms (#11523)
If, for some reason, we don't have the create event, we should still be able to
purge a room.
2021-12-07 11:37:54 +00:00
David Robertson
9c55dedc8c
Correctly ignore invites from ignored users (#11511) 2021-12-07 11:24:31 +00:00
Quentin Gliech
a15a893df8
Save the OIDC session ID (sid) with the device on login (#11482)
As a step towards allowing back-channel logout for OIDC.
2021-12-06 12:43:06 -05:00
Patrick Cloke
494ebd7347
Include bundled aggregations in /sync and related fixes (#11478)
Due to updates to MSC2675 this includes a few fixes:

* Include bundled aggregations for /sync.
* Do not include bundled aggregations for /initialSync and /events.
* Do not bundle aggregations for state events.
* Clarifies comments and variable names.
2021-12-06 15:51:15 +00:00
reivilibre
637df95de6
Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445) 2021-12-03 16:42:44 +00:00
David Robertson
d26808dd85
Comments on the /sync tentacles (#11494)
This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-02 20:58:32 +00:00
Eric Eastwood
a6f1a3abec
Add MSC3030 experimental client and federation API endpoints to get the closest event to a given timestamp (#9445)
MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030

Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about.
```
GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}
```

Federation API endpoint:
```
GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}
```

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-12-02 01:02:20 -06:00
Patrick Cloke
a4521ce0a8
Support the stable /hierarchy endpoint from MSC2946 (#11329)
This also makes additional updates where the implementation
had drifted from the approved MSC.

Unstable endpoints will be removed at a later data.
2021-11-29 14:32:20 -05:00
reivilibre
a82b90ab32
Add type annotations to some of the configuration surrounding refresh tokens. (#11428) 2021-11-29 13:34:14 +00:00
reivilibre
1d8b80b334
Support expiry of refresh tokens and expiry of the overall session when refresh tokens are in use. (#11425) 2021-11-26 14:27:14 +00:00
reivilibre
f25c75d376
Rename unstable access_token_lifetime configuration option to refreshable_access_token_lifetime to make it clear it only concerns refreshable access tokens. (#11388) 2021-11-23 17:01:34 +00:00
Patrick Cloke
6a5dd485bd
Refactor the code to inject bundled relations during serialization. (#11408) 2021-11-23 06:43:56 -05:00
Patrick Cloke
7ae559944a
Fix checking whether a room can be published on creation. (#11392)
If `room_list_publication_rules` was configured with a rule with a
non-wildcard alias and a room was created with an alias then an
internal server error would have been thrown.

This fixes the error and properly applies the publication rules
during room creation.
2021-11-19 15:19:32 +00:00
Eric Eastwood
7ffddd819c
Prevent historical state from being pushed to an application service via /transactions (MSC2716) (#11265)
Mark historical state from the MSC2716 `/batch_send` endpoint as `historical` which makes it `backfilled` and have a negative `stream_ordering` so it doesn't get queried by `/transactions`.

Fix https://github.com/matrix-org/synapse/issues/11241

Complement tests: https://github.com/matrix-org/complement/pull/221
2021-11-18 14:16:08 -06:00
Shay
92b75388f5
Remove legacy code related to deprecated trust_identity_server_for_password_resets config flag (#11333)
* remove code legacy code related to deprecated config flag "trust_identity_server_for_password_resets" from synapse/config/emailconfig.py

* remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/config/registration.py

* remove legacy code supporting depreciated config flag "trust_identity_server_for_password_resets" from synapse/handlers/identity.py

* add tests to ensure config error is thrown and synapse refuses to start when depreciated config flag is found

* add changelog

* slightly change behavior to only check for deprecated flag if set to 'true'

* Update changelog.d/11333.misc

Co-authored-by: reivilibre <oliverw@matrix.org>

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-18 10:56:32 -08:00
reivilibre
433ee159cb
Rename get_refresh_token_for_user_id to create_refresh_token_for_user_id (#11370) 2021-11-18 14:45:38 +00:00
Patrick Cloke
4bd54b263e
Do not allow MSC3440 threads to fork threads (#11161)
Adds validation to the Client-Server API to ensure that
the potential thread head does not relate to another event
already. This results in not allowing a thread to "fork" into
other threads.

If the target event is unknown for some reason (maybe it isn't
visible to your homeserver), but is the target of other events
it is assumed that the thread can be created from it. Otherwise,
it is rejected as an unknown event.
2021-11-18 13:43:09 +00:00
Sean Quah
84fac0f814
Add type annotations to synapse.metrics (#10847) 2021-11-17 19:07:02 +00:00
reivilibre
0d86f6334a
Rename get_access_token_for_user_id method to create_access_token_for_user_id (#11369) 2021-11-17 14:10:57 +00:00
Sean Quah
88375beeaa
Avoid sharing room hierarchy responses between users (#11355)
Different users may be allowed to see different rooms within a space,
so sharing responses between users is inadvisable.
2021-11-16 15:40:47 +00:00
reivilibre
3a1462f7e0
Properly register all callback hooks for legacy password authentication providers (#11340)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-11-16 12:53:31 +00:00
David Robertson
bea815cec8
Test room alias deletion (#11327)
* Prefer `HTTPStatus` over plain `int`

This is an Opinion that no-one has seemed to object to yet.

* `--disallow-untyped-defs` for `tests.rest.client.test_directory`
* Improve synapse's annotations for deleting aliases
* Test case for deleting a room alias
* Changelog
2021-11-12 19:56:00 +00:00
Dirk Klimpel
8840a7b7f1
Convert delete room admin API to async endpoint (#11223)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-11-12 12:35:31 +00:00
Patrick Cloke
5cace20bf1
Add missing type hints to synapse.app. (#11287) 2021-11-10 15:06:54 -05:00
Richard van der Hoff
73cbb284b9
Remove redundant parameters on _check_event_auth (#11292)
as of #11012, these parameters are unused.
2021-11-10 14:16:06 +00:00
Andrew Morgan
a026695083
Clarifications and small fixes to to-device related code (#11247)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-11-09 14:31:15 +00:00
David Robertson
b6f4d122ef
Allow admins to proactively block rooms (#11228)
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-11-09 13:11:47 +00:00
Patrick Cloke
a19d01c3d9
Support filtering by relations per MSC3440 (#11236)
Adds experimental support for `relation_types` and `relation_senders`
fields for filters.
2021-11-09 08:10:58 -05:00
Erik Johnston
af784644c3
Include cross-signing signatures when syncing remote devices for the first time (#11234)
When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results.

c.f. #11159
2021-11-09 11:45:36 +00:00
Eric Eastwood
84f235aea4
Rename to more clear get_insertion_event_id_by_batch_id (MSC2716) (#11244)
`get_insertion_event_by_batch_id` -> `get_insertion_event_id_by_batch_id`

Split out from https://github.com/matrix-org/synapse/pull/11114
2021-11-08 21:21:10 -06:00
Richard van der Hoff
86a497efaa
Default value for public_baseurl (#11210)
We might as well use a default value for `public_baseurl` based on
`server_name` - in many cases, it will be correct.
2021-11-08 14:13:10 +00:00
reivilibre
499c44d696
Make minor correction to type of auth_checkers callbacks (#11253) 2021-11-04 17:10:11 +00:00
Nick Barrett
a271e233e9
Add a linearizer on (appservice, stream) when handling ephemeral events. (#11207)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-11-03 16:51:00 +00:00
Nick Barrett
af54167516
Enable passing typing stream writers as a list. (#11237)
This makes the typing stream writer config match the other stream writers
that only currently support a single worker.
2021-11-03 14:25:47 +00:00
Patrick Cloke
c01bc5f43d
Add remaining type hints to synapse.events. (#11098) 2021-11-02 09:55:52 -04:00
Andrew Morgan
c9c3aea9b1
Fix providing a RoomStreamToken instance to _notify_app_services_ephemeral (#11137)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-02 10:39:02 +00:00
Rafael Gonçalves
0e16b418f6
Add knock information in admin exported data (#11171)
Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
2021-10-28 18:54:38 +01:00
reivilibre
75ca0a6168
Annotate log_function decorator (#10943)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-27 17:27:23 +01:00
Patrick Cloke
19d5dc6931
Refactor Filter to handle fields according to data being filtered. (#11194)
This avoids filtering against fields which cannot exist on an
event source. E.g. presence updates don't have a room.
2021-10-27 11:26:30 -04:00
Patrick Cloke
a930da3291
Include the stable identifier for MSC3288. (#11187)
Includes both the stable and unstable identifier to store-invite
calls to the identity server. In the future we should remove the
unstable identifier.
2021-10-27 14:19:19 +00:00
Brendan Abolivier
c7a5e49664
Implement an on_new_event callback (#11126)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-10-26 15:17:36 +02:00
Brendan Abolivier
c1510c97b5
Fix cyclic import in the module API (#11180)
Introduced in #10548

See https://github.com/matrix-org/synapse-email-account-validity/runs/3979337154?check_suite_focus=true for an example of a module's CI choking over this issue.
2021-10-25 16:45:19 +00:00
AndrewFerr
4387b791e0
Don't set new room alias before potential 403 (#10930)
Fixes: #10929 

Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-10-25 15:24:49 +01:00
Richard van der Hoff
da957a60e8
Ensure that we correctly auth events returned by send_join (#11012)
This is the final piece of the jigsaw for #9595. As with other changes before this one (eg #10771), we need to make sure that we auth the auth events in the right order, and actually check that their predecessors haven't been rejected.

To do this I've reused the existing code we use when persisting outliers elsewhere.

I've removed the code for attempting to fetch missing auth_events - the events should have been present in the send_join response, so the likely reason they are missing is that we couldn't verify them, so requesting them again is unlikely to help. Instead, we simply drop any state which relies on those auth events, as we do at a backwards-extremity. See also matrix-org/complement#216 for a test for this.
2021-10-25 15:21:09 +01:00
David Robertson
2d91b6256e
Fix adding excluded users to the private room sharing tables when joining a room (#11143)
* We only need to fetch users in private rooms

* Filter out `user_id` at the top

* Discard excluded users in the top loop

We weren't doing this in the "First, if they're our user" branch so this
is a bugfix.

* The caller must check that `user_id` is included

This is in the docstring. There are two call sites:
- one in `_handle_room_publicity_change`, which explicitly checks before calling;
- and another in `_handle_room_membership_event`, which returns early if
  the user is excluded.

So this change is safe.

* Test joining a private room with an excluded user

* Tweak an existing test

* Changelog

* test docstring

* lint
2021-10-21 17:48:59 +01:00
Andrew Morgan
6408372234
Improve docstrings for methods related to sending EDUs to application services (#11138) 2021-10-21 17:42:25 +01:00
Robert Edström
62db603fa0
Consider IP whitelist for identity server resolution (#11120)
Signed-off-by: Robert Edström <github@legogris.se>
2021-10-20 18:43:49 +01:00
Richard van der Hoff
0930e9ae12
Clean up _update_auth_events_and_context_for_auth (#11122)
Remove some redundant code, and generally simplify.
2021-10-20 18:22:40 +01:00
Sean Quah
ee2cee5f52 Merge branch 'master' into develop 2021-10-20 13:43:02 +01:00
Sean Quah
df95d3aec2
Revert change to counting of deactivated users towards the monthly active users limit (#11127)
Temporarily revert "Add functionality to remove deactivated users from the monthly_active_users table (#10947)".

This reverts commit eda8c88b84.
2021-10-20 11:05:29 +01:00
Richard van der Hoff
f3efa0036b
Move _persist_auth_tree into FederationEventHandler (#11115)
This is just a lift-and-shift, because it fits more naturally here. We do
rename it to `process_remote_join` at the same time though.
2021-10-19 10:24:09 +01:00
Richard van der Hoff
0170774b19
Rename _auth_and_persist_fetched_events (#11116)
... to `_auth_and_persist_outliers`, since that reflects its purpose better.
2021-10-19 10:23:55 +01:00
Patrick Cloke
3ab55d43bd
Add missing type hints to synapse.api. (#11109)
* Convert UserPresenceState to attrs.
* Remove args/kwargs from error classes and explicitly pass msg/errorcode.
2021-10-18 15:01:10 -04:00
Richard van der Hoff
cc33d9eee2
Check auth on received events' auth_events (#11001)
Currently, when we receive an event whose auth_events differ from those we expect, we state-resolve between the two state sets, and check that the event passes auth based on the resolved state.

This means that it's possible for us to accept events which don't pass auth at their declared auth_events (or where the auth events themselves were rejected), leading to problems down the line like #10083.

This change means we will:

 * ignore any events where we cannot find the auth events
 * reject any events whose auth events were rejected
 * reject any events which do not pass auth at their declared auth_events.

Together with a whole raft of previous work, this is a partial fix to #9595.

Fixes #6643.

Based on #11009.
2021-10-18 18:29:37 +01:00
Richard van der Hoff
a5d2ea3d08
Check *all* auth events for room id and rejection (#11009)
This fixes a bug where we would accept an event whose `auth_events` include
rejected events, if the rejected event was shadowed by another `auth_event`
with same `(type, state_key)`.

The approach is to pass a list of auth events into
`check_auth_rules_for_event` instead of a dict, which of course means updating
the call sites.

This is an extension of #10956.
2021-10-18 18:28:30 +01:00
Richard van der Hoff
e8f24b6c35
_run_push_actions_and_persist_event: handle no min_depth (#11014)
Make sure that we correctly handle rooms where we do not yet have a
`min_depth`, and also add some comments and logging.
2021-10-18 17:17:15 +01:00
David Robertson
37b845dabc
Don't remove local users from dir when the leave their last room (#11103) 2021-10-18 13:20:04 +00:00
David Robertson
e09be0c87a
Correctly exclude users when making a room public or private (#11075)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-15 15:53:05 +01:00
Patrick Cloke
5573133348
Move experimental & retention config out of the server module. (#11070) 2021-10-15 14:30:48 +00:00
Eric Eastwood
daf498e099
Fix 500 error on /messages when we accumulate more than 5 backward extremities (#11027)
Found while working on the Gitter backfill script and noticed
it only happened after we sent 7 batches, https://gitlab.com/gitterHQ/webapp/-/merge_requests/2229#note_665906390

When there are more than 5 backward extremities for a given depth,
backfill will throw an error because we sliced the extremity list
to 5 but then try to iterate over the full list. This causes
us to look for state that we never fetched and we get a `KeyError`.

Before when calling `/messages` when there are more than 5 backward extremities:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 258, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "/usr/local/lib/python3.8/site-packages/synapse/http/server.py", line 446, in _async_render
    callback_return = await raw_callback_return
  File "/usr/local/lib/python3.8/site-packages/synapse/rest/client/room.py", line 580, in on_GET
    msgs = await self.pagination_handler.get_messages(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/pagination.py", line 396, in get_messages
    await self.hs.get_federation_handler().maybe_backfill(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 133, in maybe_backfill
    return await self._maybe_backfill_inner(room_id, current_depth, limit)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 386, in _maybe_backfill_inner
    likely_extremeties_domains = get_domains_from_state(states[e_id])
KeyError: '$zpFflMEBtZdgcMQWTakaVItTLMjLFdKcRWUPHbbSZJl'
```
2021-10-14 18:53:45 -05:00
Eric Eastwood
35d6b914eb
Resolve and share state_groups for all historical events in batch (MSC2716) (#10975)
Resolve and share `state_groups` for all historical events in batch.  This also helps for showing the appropriate avatar/displayname in Element and will work whenever `/messages` has one of the historical messages as the first message in the batch.

This does have the flaw where if you just insert a single historical event somewhere, it probably won't resolve the state correctly from `/messages` or `/context` since it will grab a non historical event above or below with resolved state which never included the historical state back then. For the same reasions, this also does not work in Element between the transition from actual messages to historical messages. In the Gitter case, this isn't really a problem since all of the historical messages are in one big lump at the beginning of the room.

For a future iteration, might be good to look at `/messages` and `/context` to additionally add the `state` for any historical messages in that batch.

---

How are the `state_groups` shared? To illustrate the `state_group` sharing, see this example:


**Before** (new `state_group` for every event 😬, very inefficient):
```
# Tests from https://github.com/matrix-org/complement/pull/206
$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events
create_new_client_event m.room.member event=$_JXfwUDIWS6xKGG4SmZXjSFrizhARM7QblhATVWWUcA state_group=None
create_new_client_event org.matrix.msc2716.insertion event=$1ZBfmBKEjg94d-vGYymKrVYeghwBOuGJ3wubU1-I9y0 state_group=9
create_new_client_event org.matrix.msc2716.insertion event=$Mq2JvRetTyclPuozRI682SAjYp3GqRuPc8_cH5-ezPY state_group=10
create_new_client_event m.room.message event=$MfmY4rBQkxrIp8jVwVMTJ4PKnxSigpG9E2cn7S0AtTo state_group=11
create_new_client_event m.room.message event=$uYOv6V8wiF7xHwOMt-60d1AoOIbqLgrDLz6ZIQDdWUI state_group=12
create_new_client_event m.room.message event=$PAbkJRMxb0bX4A6av463faiAhxkE3FEObM1xB4D0UG4 state_group=13
create_new_client_event org.matrix.msc2716.batch event=$Oy_S7AWN7rJQe_MYwGPEy6RtbYklrI-tAhmfiLrCaKI state_group=14
```

**After** (all events in batch sharing `state_group=10`) (the base insertion event has `state_group=8` which matches the `prev_event` we're inserting next to):

```
# Tests from https://github.com/matrix-org/complement/pull/206
$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events
create_new_client_event m.room.member event=$PWomJ8PwENYEYuVNoG30gqtybuQQSZ55eldBUSs0i0U state_group=None
create_new_client_event org.matrix.msc2716.insertion event=$e_mCU7Eah9ABF6nQU7lu4E1RxIWccNF05AKaTT5m3lw state_group=9
create_new_client_event org.matrix.msc2716.insertion event=$ui7A3_GdXIcJq0C8GpyrF8X7B3DTjMd_WGCjogax7xU state_group=10
create_new_client_event m.room.message event=$EnTIM5rEGVezQJiYl62uFBl6kJ7B-sMxWqe2D_4FX1I state_group=10
create_new_client_event m.room.message event=$LGx5jGONnBPuNhAuZqHeEoXChd9ryVkuTZatGisOPjk state_group=10
create_new_client_event m.room.message event=$wW0zwoN50lbLu1KoKbybVMxLbKUj7GV_olozIc5i3M0 state_group=10
create_new_client_event org.matrix.msc2716.batch event=$5ZB6dtzqFBCEuMRgpkU201Qhx3WtXZGTz_YgldL6JrQ state_group=10
```
2021-10-13 17:44:00 -05:00
David Robertson
317e9e415c
Rearrange the user_directory's _handle_deltas function (#11035)
* Pull out `_handle_room_membership_event`
* Discard excluded users early
* Rearrange logic so the change is membership is effectively switched over. See PR for rationale.
2021-10-13 12:50:00 +00:00
Patrick Cloke
1f9d0b8a7a
Add type hints to synapse.events.*. (#11066)
Except `synapse/events/__init__.py`, which will be done in a follow-up.
2021-10-13 07:24:07 -04:00
Azrenbeth
cdd308845b
Port the Password Auth Providers module interface to the new generic interface (#10548)
Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-13 11:21:52 +00:00
Sean Quah
36224e056a
Add type hints to synapse.storage.databases.main.client_ips (#10972) 2021-10-12 13:50:34 +01:00
Eric Eastwood
a7d22c36db
Refactor MSC2716 /batch_send endpoint into separate handler functions (#10974) 2021-10-08 18:35:00 -05:00
David Robertson
670a8d9a1e
Fix overwriting profile when making room public (#11003)
This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
2021-10-08 12:52:48 +01:00
Patrick Cloke
eb9ddc8c2e
Remove the deprecated BaseHandler. (#11005)
The shared ratelimit function was replaced with a dedicated
RequestRatelimiter class (accessible from the HomeServer
object).

Other properties were copied to each sub-class that inherited
from BaseHandler.
2021-10-08 07:44:43 -04:00
David Robertson
e0bf34dada
Don't alter directory entries for local users when setting a per-room nickname (#11002)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-10-07 13:26:11 +01:00
Richard van der Hoff
96fe77c254
Improve the logging in _auth_and_persist_outliers (#11010)
Include the event ids being peristed
2021-10-07 11:43:25 +00:00
Richard van der Hoff
86af6b2f0e
Add a comment in _process_received_pdu (#11011) 2021-10-07 12:20:03 +01:00
Brendan Abolivier
4e51621064
Add a spamchecker method to allow or deny 3pid invites (#10894)
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.

Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
2021-10-06 17:18:13 +02:00
Patrick Cloke
f4b1a9a527
Require direct references to configuration variables. (#10985)
This removes the magic allowing accessing configurable
variables directly from the config object. It is now required
that a specific configuration class is used (e.g. `config.foo`
must be replaced with `config.server.foo`).
2021-10-06 10:47:41 -04:00
Brendan Abolivier
829f2a82b0
Add a spamchecker callback to allow or deny room joins (#10910)
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-10-06 14:32:16 +00:00
David Robertson
370bca32e6
Don't drop user dir deltas when server leaves room (#10982)
Fix a long-standing bug where a batch of user directory changes would be
silently dropped if the server left a room early in the batch.

* Pull out `wait_for_background_update` in tests

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-06 12:56:45 +00:00
Eric Eastwood
392863fbf1
Fix logic flaw preventing tracking of MSC2716 events in existing room versions (#10962)
We correctly allowed using the MSC2716 batch endpoint for
the room creator in existing room versions but accidentally didn't track
the events because of a logic flaw.

This prevented you from connecting subsequent chunks together because it would
throw the unknown batch ID error.

We only want to process MSC2716 events when:

 - The room version supports MSC2716
 - Any room where the homeserver has the `msc2716_enabled` experimental feature enabled and the event is from the room creator
2021-10-05 11:51:57 -05:00
Richard van der Hoff
787af4a106
Host cache_joined_hosts_for_event to caller (#10986)
`_check_event_auth` is only called in two places, and only one of those sets
`send_on_behalf_of`. Warming the cache isn't really part of auth anyway, so
moving it out makes a lot more sense.
2021-10-05 13:01:41 +01:00
Richard van der Hoff
d099535deb
_update_auth_events_and_context_for_auth: add some comments (#10987)
Add some more comments about wtf is going on here.
2021-10-05 12:50:38 +01:00
Richard van der Hoff
cb88ed912b
_check_event_auth: move event validation earlier (#10988)
There's little point in doing a fancy state reconciliation dance if the event
itself is invalid.

Likewise, there's no point checking it again in `_check_for_soft_fail`.
2021-10-05 12:50:07 +01:00
Hillery Shay
eda8c88b84
Add functionality to remove deactivated users from the monthly_active_users table (#10947)
* add test

* add function to remove user from monthly active table in deactivate code

* add function to remove user from monthly active table

* add changelog entry

* update changelog number

* requested changes

* update docstring on new function

* fix lint error

* Update synapse/storage/databases/main/monthly_active_users.py

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-04 08:34:42 -07:00
David Robertson
f7b034a24b
Consistently exclude from user_directory (#10960)
* Introduce `should_include_local_users_in_dir`

We exclude three kinds of local users from the user_directory tables. At
present we don't consistently exclude all three in the same places. This
commit introduces a new function to gather those exclusion conditions
together. Because we have to handle local and remote users in different
ways, I've made that function only consider the case of remote users.
It's the caller's responsibility to make the local versus remote
distinction clear and correct.

A test fixup is required. The test now hits a path which makes db
queries against the users table. The expected rows were missing, because
we were using a dummy user that hadn't actually been registered.

We also add new test cases to covert the exclusion logic.

----

By my reading this makes these changes:

* When an app service user registers or changes their profile, they will
  _not_ be added to the user directory. (Previously only support and
  deactivated users were excluded). This is consistent with the logic that
  rebuilds the user directory. See also [the discussion
  here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548).
* When rebuilding the directory, exclude support and disabled users from
  room sharing tables. Previously only appservice users were excluded.
* Exclude all three categories of local users when rebuilding the
  directory. Previously `_populate_user_directory_process_users` didn't do
  any exclusion.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-04 11:45:51 +00:00
Patrick Cloke
a0f48ee89d
Use direct references for configuration variables (part 7). (#10959) 2021-10-04 07:18:54 -04:00
Patrick Cloke
9e5a429c8b
Clean-up registration tests (#10945)
Uses `override_config` and fixes test_auto_create_auto_join_where_no_consent
to properly configure auto-join rooms.
2021-09-30 14:06:02 -04:00
Patrick Cloke
d1bf5f7c9d
Strip "join_authorised_via_users_server" from join events which do not need it. (#10933)
This fixes a "Event not signed by authorising server" error when
transition room member from join -> join, e.g. when updating a
display name or avatar URL for restricted rooms.
2021-09-30 11:13:59 -04:00
Olivier Wilkinson (reivilibre)
44dee1fe8c Synapse 1.44.0rc2 (2021-09-30)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint to return a 500 error. ([\#10938](https://github.com/matrix-org/synapse/issues/10938))
 - Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\#10944](https://github.com/matrix-org/synapse/issues/10944))
 
 Improved Documentation
 ----------------------
 
 - Minor updates to the installation instructions. ([\#10919](https://github.com/matrix-org/synapse/issues/10919))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmFVpBcACgkQLS76LzL7
 4EeM6Q/8D853gaGsZGzKPpXus7AZ3/sgOQlRrRKWFjIb2JX6fLvTyZRobxebHhQx
 YrE/cwhs7NZsmmi3YNMngpLez9EB8YtxQZCpKZnD+G4OijLMbZuRlVOM5kE69zX5
 gOtcYPK9CmMsp6ex9wexzF3LH6ZyFh1YwUPrGJoXUTubFuirLjtEtFDQdEBeyuUz
 TpRWzTCjfItNSpIJXoucUZjbZCOzB5+iY+MwgASeiFqVShLiUpTzUE8LRo1tTlQS
 i4hm4XZwVNOSRHTVj7qd+3Mg3ncEbpL9BiBNOq1IjoMjNiqv43MCR4iXH5hQiixz
 XIuqcd9nGLeCKOGrMEK6cOojq37eAhOnYnEUaACgyfQ8Bhq6wzZ7q47q2mFMySlh
 EjtExZmpGCMfH2NcnbQ1YSaNzXZuZnyrqou7l86I+c65dwCEf2U7rFh5/Egs/LMJ
 zjJpiHaogwnzYDZdWKz1R7hEcAwRPyvwy7WYavit9022w1LueW5DPTgFXwAYJzqE
 7VJXb2imAn1he+wIVL+Ye9i7ptFbBfq53T5pnhqZ3bDDzDXA0N6zQ8aOB3RvF0ni
 KcTX8R6FvVHO4C1fpHH4J+PkShOL3InrYxb4Tlf0GD8HsznL0qDT1yiiNRGmdqcA
 8tAe2O8uTkNX6oR/qTMe8/vExRRBZQlIQkO4YX1Cv6YYOWdgfc8=
 =aEUS
 -----END PGP SIGNATURE-----

Merge tag 'v1.44.0rc2' into develop

Synapse 1.44.0rc2 (2021-09-30)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.44.0rc1 which caused the experimental [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint to return a 500 error. ([\#10938](https://github.com/matrix-org/synapse/issues/10938))
- Fix a bug introduced in v1.44.0rc1 which prevented sending presence events to application services. ([\#10944](https://github.com/matrix-org/synapse/issues/10944))

Improved Documentation
----------------------

- Minor updates to the installation instructions. ([\#10919](https://github.com/matrix-org/synapse/issues/10919))
2021-09-30 13:40:24 +01:00
David Robertson
29364145b2
Pass str to twisted's IReactorTCP (#10895)
This follows a correction made in twisted/twisted#1664 and should fix our Twisted Trial CI job.

Until that change is in a twisted release, we'll have to ignore the type
of the `host` argument. I've raised #10899 to remind us to review the
issue in a few months' time.
2021-09-30 12:51:47 +01:00
Eric Eastwood
c4bf48ee6f Fix event context for outliers in important MSC2716 spot (#10938)
Fix event context for outlier causing failures in all of the MSC2716
Complement tests.

The `EventContext.for_outlier` refactor happened in
https://github.com/matrix-org/synapse/pull/10883
and this spot was left out.
2021-09-30 11:34:44 +01:00
reivilibre
a03ed5e6ae
Fix issue causing sending presence to ASes to fail (due to incomplete type annotations) (#10944) 2021-09-30 11:06:47 +01:00
Richard van der Hoff
428174f902
Split event_auth.check into two parts (#10940)
Broadly, the existing `event_auth.check` function has two parts:
 * a validation section: checks that the event isn't too big, that it has the rught signatures, etc. 
   This bit is independent of the rest of the state in the room, and so need only be done once 
   for each event.
 * an auth section: ensures that the event is allowed, given the rest of the state in the room.
   This gets done multiple times, against various sets of room state, because it forms part of
   the state res algorithm.

Currently, this is implemented with `do_sig_check` and `do_size_check` parameters, but I think
that makes everything hard to follow. Instead, we split the function in two and call each part
separately where it is needed.
2021-09-29 18:59:15 +01:00
Patrick Cloke
94b620a5ed
Use direct references for configuration variables (part 6). (#10916) 2021-09-29 06:44:15 -04:00
Richard van der Hoff
5279b9161b
Use RoomVersion objects (#10934)
Various refactors to use `RoomVersion` objects instead of room version identifiers.
2021-09-29 10:57:10 +01:00
Eric Eastwood
9fd057b8c5
Ensure (room_id, next_batch_id) is unique to avoid cross-talk/conflicts between batches (MSC2716) (#10877)
Part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)

Part of https://github.com/matrix-org/synapse/issues/10737
2021-09-28 21:23:16 -05:00
Richard van der Hoff
2622b28c5c
Inline _check_event_auth for outliers (#10926)
* Inline `_check_event_auth` for outliers

When we are persisting an outlier, most of `_check_event_auth` is redundant:

 * `_update_auth_events_and_context_for_auth` does nothing, because the
   `input_auth_events` are (now) exactly the event's auth_events,
   which means that `missing_auth` is empty.

 * we don't care about soft-fail, kicking guest users or `send_on_behalf_of`
   for outliers

... so the only thing that matters is the auth itself, so let's just do that.

* `_auth_and_persist_fetched_events_inner`: de-async `prep`

`prep` no longer calls any `async` methods, so let's make it synchronous.

* Simplify `_check_event_auth`

We no longer need to support outliers here, which makes things rather simpler.

* changelog

* lint
2021-09-28 15:25:07 +01:00
Patrick Cloke
eb2c7e51c4
Clean-up type hints in server config (#10915)
By using attrs instead of dicts to store configuration.

Also updates some of the attrs classes to use proper type
hints and auto_attribs.
2021-09-28 09:24:40 -04:00
Brendan Abolivier
b10257e879
Add a spamchecker callback to allow or deny room creation based on invites (#10898)
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.

This adds a callback that's fairly similar to user_may_create_room except it also allows processing based on the invites sent at room creation.
2021-09-24 16:38:23 +02:00
Richard van der Hoff
0420d4e6a5
Stop trying to auth/persist events whose auth events we do not have. (#10907) 2021-09-24 14:01:45 +01:00
Patrick Cloke
bb7fdd821b
Use direct references for configuration variables (part 5). (#10897) 2021-09-24 07:25:21 -04:00
Richard van der Hoff
85551b7a85
Factor out common code for persisting fetched auth events (#10896)
* Factor more stuff out of `_get_events_and_persist`

It turns out that the event-sorting algorithm in `_get_events_and_persist` is
also useful in other circumstances. Here we move the current
`_auth_and_persist_fetched_events` to `_auth_and_persist_fetched_events_inner`,
and then factor the sorting part out to `_auth_and_persist_fetched_events`.

* `_get_remote_auth_chain_for_event`: remove redundant `outlier` assignment

`get_event_auth` returns events with the outlier flag already set, so this is
redundant (though we need to update a test where `get_event_auth` is mocked).

* `_get_remote_auth_chain_for_event`: move existing-event tests earlier

Move a couple of tests outside the loop. This is a bit inefficient for now, but
a future commit will make it better. It should be functionally identical.

* `_get_remote_auth_chain_for_event`: use `_auth_and_persist_fetched_events`

We can use the same codepath for persisting the events fetched as part of an
auth chain as for those fetched individually by `_get_events_and_persist` for
building the state at a backwards extremity.

* `_get_remote_auth_chain_for_event`: use a dict for efficiency

`_auth_and_persist_fetched_events` sorts the events itself, so we no longer
need to care about maintaining the ordering from `get_event_auth` (and no
longer need to sort by depth in `get_event_auth`).

That means that we can use a map, making it easier to filter out events we
already have, etc.

* changelog

* `_auth_and_persist_fetched_events`: improve docstring
2021-09-24 11:56:33 +01:00
Richard van der Hoff
261c9763c4
Simplify _auth_and_persist_fetched_events (#10901)
Combine the two loops over the list of events, and hence get rid of
`_NewEventInfo`. Also pass the event back alongside the context, so that it's
easier to process the result.
2021-09-24 11:56:13 +01:00
Richard van der Hoff
a7304adc7d
Factor out _get_remote_auth_chain_for_event from _update_auth_events_and_context_for_auth (#10884)
* Reload auth events from db after fetching and persisting

In `_update_auth_events_and_context_for_auth`, when we fetch the remote auth
tree and persist the returned events: load the missing events from the database
rather than using the copies we got from the remote server.

This is mostly in preparation for additional refactors, but does have an
advantage in that if we later get around to checking the rejected status, we'll
be able to make use of it.

* Factor out `_get_remote_auth_chain_for_event` from `_update_auth_events_and_context_for_auth`

* changelog
2021-09-23 17:34:33 +01:00
Patrick Cloke
47854c71e9
Use direct references for configuration variables (part 4). (#10893) 2021-09-23 12:03:01 -04:00
David Robertson
dcfd864970
Fix reactivated users not being added to the user directory (#10782)
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
Co-authored-by: reivilibre <olivier@librepush.net>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-23 12:02:13 +00:00
Patrick Cloke
e584534403
Use direct references for some configuration variables (part 3) (#10885)
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.

It also improves type hints since mypy can now resolve the
types of the configuration variables.
2021-09-23 07:13:34 -04:00
Andrew Morgan
aa2c027792
Remove unnecessary parentheses around tuples returned from methods (#10889) 2021-09-23 11:59:07 +01:00
Richard van der Hoff
26f2bfedbf
Factor out a separate EventContext.for_outlier (#10883)
Constructing an EventContext for an outlier is actually really simple, and
there's no sense in going via an `async` method in the `StateHandler`.

This also means that we can resolve a bunch of FIXMEs.
2021-09-22 17:58:57 +01:00
Richard van der Hoff
8f2a52766b
Ensure we mark sent knocks as outliers (#10873) 2021-09-22 15:20:18 +01:00
Eric Eastwood
51e2db3598
Rename MSC2716 things from chunk to batch to match /batch_send endpoint (#10838)
See https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684574497

Dropping support for older MSC2716 room versions so we don't have to worry about
supporting both chunk and batch events.
2021-09-21 15:06:28 -05:00
Patrick Cloke
4054dfa409
Add type hints for event streams. (#10856) 2021-09-21 13:34:26 -04:00
David Robertson
60453315bd
Always add local users to the user directory (#10796)
It's a simplification, but one that'll help make the user directory logic easier
to follow with the other changes upcoming. It's not strictly required for those
changes, but this will help simplify the resulting logic that listens for
`m.room.member` events and generally make the logic easier to follow.

This means the config option `search_all_users` ends up controlling the
search query only, and not the data we store. The cost of doing so is an
extra row in the `user_directory` and `user_directory_search` tables for
each local user which

- belongs to no public rooms
- belongs to no private rooms of size ≥ 2

I think the cost of this will be marginal (since they'll already have entries
 in `users` and `profiles` anyway).

As a small upside, a homeserver whose directory was built with this
change can toggle `search_all_users` without having to rebuild their
directory.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-21 12:02:34 +00:00
Aaron Raimist
6a751ff5e0
Allow sending a membership event to unban a user (#10807)
* Allow membership event to unban user

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-09-21 10:23:34 +00:00
Patrick Cloke
b3590614da
Require type hints in the handlers module. (#10831)
Adds missing type hints to methods in the synapse.handlers
module and requires all methods to have type hints there.

This also removes the unused construct_auth_difference method
from the FederationHandler.
2021-09-20 08:56:23 -04:00
David Robertson
b4c1af8cea Merge branch 'release-v1.43' into develop 2021-09-17 12:15:17 +01:00
Erik Johnston
9a6f4a684f
Cleanup opentracing logging for syncs (#10828)
We added a bunch of spans in #10704, but this ended up adding a lot of
redundant spans for rooms where nothing changed, so instead we only
start the span if there might be something interesting going on.
2021-09-15 17:14:25 +01:00
Patrick Cloke
8c7a531e27
Use direct references for some configuration variables (part 2) (#10812) 2021-09-15 08:34:52 -04:00
David Robertson
319b8b6bef
Name the type of token in "Invalid token" messages (#10815)
I had one of these error messages yesterday and assumed it was an
invalid auth token (because that was an HTTP query parameter in the
test) I was working on. In fact, it was an invalid next batch token for
syncing.
2021-09-14 11:25:05 +01:00
Patrick Cloke
01c88a09cd
Use direct references for some configuration variables (#10798)
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
reivilibre
524b8ead77
Add types to synapse.util. (#10601) 2021-09-10 17:03:18 +01:00