Commit Graph

4083 Commits

Author SHA1 Message Date
Patrick Cloke
2897fb6b4f
Improvements to bundling aggregations. (#11815)
This is some odds and ends found during the review of #11791
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
Brendan Abolivier
95b3f952fa
Add a config flag to inhibit M_USER_IN_USE during registration (#11743)
This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work).

This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique.

More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
2022-01-26 13:02:54 +01:00
reivilibre
df54c8485a
Remove account data (including client config, push rules and ignored users) upon user deactivation. (#11621)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-01-24 13:37:00 +00:00
reivilibre
4c2096599c
Make the get_global_account_data_by_type_for_user cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
Brendan Abolivier
bfe6d5553a
Correctly await on_logged_out callbacks (#11786) 2022-01-20 19:19:40 +01:00
Andrew Morgan
7bf2d6c268
Partially revert #11675; prevent attempting to create pushers on workers (#11770) 2022-01-20 09:37:34 -05:00
Sean Quah
af13a3be29
Fix a bug that corrupted the cache of federated space hierarchies (#11775)
`FederationClient.get_room_hierarchy()` caches its return values, so
refactor the code to avoid modifying the returned room summary.
2022-01-20 11:03:42 +00:00
Patrick Cloke
68acb0a29d
Include whether the requesting user has participated in a thread. (#11577)
Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
2022-01-18 11:38:57 -05:00
Richard van der Hoff
251b5567ec
Remove log_function and its uses (#11761)
I've never found this terribly useful. I think it was added in the early days
of Synapse, without much thought as to what would actually be useful to log,
and has just been cargo-culted ever since.

Rather, it tends to clutter up debug logs with useless information.
2022-01-18 13:06:04 +00:00
lukasdenk
d8be9924ef
Add a flag to the synapse_review_recent_signups script to ignore and filter appservice users. (#11675) 2022-01-17 16:43:25 +00:00
Andrew Morgan
18862f20b5
Remove the 'password_hash' from the Users Admin API endpoint response dictionary (#11576) 2022-01-14 14:53:33 +00:00
Patrick Cloke
0c40c619aa
Include bundled aggregations in the sync response cache. (#11659) 2022-01-13 10:45:28 -05:00
Patrick Cloke
10a88ba91c
Use auto_attribs/native type hints for attrs classes. (#11692) 2022-01-13 13:49:28 +00:00
reivilibre
ffd227c382
Fix docstring on add_account_data_for_user. (#11716) 2022-01-10 15:38:22 +00:00
Patrick Cloke
8e57584a58
Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)
By returning all of the m.space.child state of the space, not just
the first 50. The number of rooms returned is still capped at 50.

For the federation API this implies that the requesting server will
need to individually query for any other rooms it is not joined to.
2022-01-07 19:27:58 -05:00
Patrick Cloke
6bf81a7a61
Bundle aggregations outside of the serialization method. (#11612)
This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.

Overall this should cause no change in behavior, but is prep work
for other improvements.
2022-01-07 09:10:46 -05:00
lukasdenk
2ef1fea8d2
Make room creations denied by user_may_create_room cause an M_FORBIDDEN error to be returned, not M_UNKNOWN (#11672)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-06 13:16:42 +00:00
David Robertson
c9eb678b73
Remove debug logging for #4422 (#11693)
as per
https://github.com/matrix-org/synapse/pull/11532#discussion_r769123269
2022-01-06 12:44:36 +00:00
Travis Ralston
feb3e006d7
Fix space hierarchy endpoint to match MSC2946 (#11667)
Fixes minor discrepancies between the /hierarchy endpoint described
in MSC2946 and the implementation.

Note that the changes impact the stable and unstable /hierarchy and
unstable /spaces endpoints for both client and federation APIs.
2022-01-05 20:33:43 +00:00
David Robertson
88a78c6577
Cache empty responses from /user/devices (#11587)
If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
2022-01-05 13:33:28 +00:00
Richard van der Hoff
0fb3dd0830
Refactor the way we set outlier (#11634)
* `_auth_and_persist_outliers`: mark persisted events as outliers

Mark any events that get persisted via `_auth_and_persist_outliers` as, well,
outliers.

Currently this will be a no-op as everything will already be flagged as an
outlier, but I'm going to change that.

* `process_remote_join`: stop flagging as outlier

The events are now flagged as outliers later on, by `_auth_and_persist_outliers`.

* `send_join`: remove `outlier=True`

The events created here are returned in the result of `send_join` to
`FederationHandler.do_invite_join`. From there they are passed into
`FederationEventHandler.process_remote_join`, which passes them to
`_auth_and_persist_outliers`... which sets the `outlier` flag.

* `get_event_auth`: remove `outlier=True`

stop flagging the events returned by `get_event_auth` as outliers. This method
is only called by `_get_remote_auth_chain_for_event`, which passes the results
into `_auth_and_persist_outliers`, which will flag them as outliers.

* `_get_remote_auth_chain_for_event`: remove `outlier=True`

we pass all the events into `_auth_and_persist_outliers`, which will now flag
the events as outliers.

* `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter

This param is now never set to True, so we can remove it.

* `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param

This is no longer set anywhere, so we can remove it.

* `get_pdu`: remove unused `outlier` parameter

... and chase it down into `get_pdu_from_destination_raw`.

* `event_from_pdu_json`: remove redundant `outlier` param

This is never set to `True`, so can be removed.

* changelog

* update docstring
2022-01-05 12:26:11 +00:00
Richard van der Hoff
b38bdae3a2
Fix AssertionErrors after purging events (#11642)
* Fix AssertionErrors after purging events

If you purged a bunch of events from your database, and then restarted synapse
without receiving more events, then you would get a bunch of AssertionErrors on
restart.

This fixes the situation by rewinding the stream processors.

* `check-newsfragment`: ignore deleted newsfiles
2022-01-04 16:36:33 +00:00
Richard van der Hoff
878aa55293
FederationClient.backfill: stop flagging events as outliers (#11632)
Events returned by `backfill` should not be flagged as outliers.

Fixes:

```
AssertionError: null
  File "synapse/handlers/federation.py", line 313, in try_backfill
    dom, room_id, limit=100, extremities=extremities
  File "synapse/handlers/federation_event.py", line 517, in backfill
    await self._process_pulled_events(dest, events, backfilled=True)
  File "synapse/handlers/federation_event.py", line 642, in _process_pulled_events
    await self._process_pulled_event(origin, ev, backfilled=backfilled)
  File "synapse/handlers/federation_event.py", line 669, in _process_pulled_event
    assert not event.internal_metadata.is_outlier()
```

See https://sentry.matrix.org/sentry/synapse-matrixorg/issues/231992

Fixes #8894.
2022-01-04 16:31:32 +00:00
Richard van der Hoff
2359ee3864
Remove redundant get_current_events_token (#11643)
* Push `get_room_{min,max_stream_ordering}` into StreamStore

Both implementations of this are identical, so we may as well push it down and
get rid of the abstract base class nonsense.

* Remove redundant `StreamStore` class

This is empty now

* Remove redundant `get_current_events_token`

This was an exact duplicate of `get_room_max_stream_ordering`, so let's get rid
of it.

* newsfile
2022-01-04 16:10:27 +00:00
Patrick Cloke
cbd82d0b2d
Convert all namedtuples to attrs. (#11665)
To improve type hints throughout the code.
2021-12-30 18:47:12 +00:00
Patrick Cloke
f58b300d27
Do not attempt to bundled aggregations for /members and /state. (#11623)
Both of those APIs return state events, which will not have bundled
aggregations added anyway.
2021-12-29 08:02:03 -05:00
Patrick Cloke
b6102230a7
Add type hints to event_push_actions. (#11594) 2021-12-21 13:25:34 +00:00
Richard van der Hoff
2215954147
Various opentracing enhancements (#11619)
* Wrap `auth.get_user_by_req` in an opentracing span

give `get_user_by_req` its own opentracing span, since it can result in a
non-trivial number of sub-spans which it is useful to group together.

This requires a bit of reorganisation because it also sets some tags (and may
force tracing) on the servlet span.

* Emit opentracing span for encoding json responses

This can be a significant time sink.

* Rename all sync spans with a prefix

* Write an opentracing span for encoding sync response

* opentracing span to group generate_room_entries

* opentracing spans within sync.encode_response

* changelog

* Use the `trace` decorator instead of context managers
2021-12-21 11:10:36 +00:00
Patrick Cloke
dd47788752
Do not bundle aggregations for APIs which shouldn't include them. (#11592)
And make bundling aggregations opt-in, instead of opt-out to avoid
having APIs to include extraneous data (and being much heavier than
necessary).
2021-12-20 14:14:38 -05:00
Richard van der Hoff
c3e38b88f2
Improve opentracing support for ResponseCache (#11607)
This adds some opentracing annotations to ResponseCache, to make it easier to see what's going on; in particular, it adds a link back to the initial trace which is actually doing the work of generating the response.
2021-12-20 18:12:08 +00:00
Sean Quah
c7fe32edb4
Add type hints to synapse/storage/databases/main/room.py (#11575) 2021-12-15 18:00:48 +00:00
Patrick Cloke
323151b787
Convert EventStreamResult to attrs. (#11574) 2021-12-15 11:10:02 -05:00
Sean Quah
ecfcd9bbbe
Add type hints to synapse/storage/databases/main/e2e_room_keys.py (#11549) 2021-12-14 17:46:47 +00:00
Sean Quah
0147b3de20
Add missing type hints to synapse.logging.context (#11556) 2021-12-14 17:35:28 +00:00
reivilibre
e5cdb9e233
Make get_device return None if the device doesn't exist rather than raising an exception. (#11565)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-12-13 15:39:43 +00:00
Eric Eastwood
aa8708ebed
Allow events to be created with no prev_events (MSC2716) (#11243)
The event still needs to have `auth_events` defined to be valid.

Split out from https://github.com/matrix-org/synapse/pull/11114
2021-12-10 23:08:51 -06:00
David Robertson
fd2dadb815
Adjust _get_rooms_changed comments (#11550)
C.f. https://github.com/matrix-org/synapse/pull/11494#pullrequestreview-827780886
2021-12-10 19:19:48 +00:00
Hubert Chathi
b47d10dc46
Support unprefixed versions of fallback key property names. (#11541) 2021-12-09 11:41:27 +00:00
Patrick Cloke
d93362d87f
Add a constant for receipt types (m.read). (#11531)
And expand some type hints in the receipts storage module.
2021-12-08 12:26:29 -05:00
Hubert Chathi
8541809cb9
Send and handle cross-signing messages using the stable prefix. (#10520) 2021-12-08 10:01:38 +00:00
David Robertson
14d593f72d
Refactors in _generate_sync_entry_for_rooms (#11515)
* Move sync_token up to the top
* Pull out _get_ignored_users
* Try to signpost the body of `_generate_sync_entry_for_rooms`
* Pull out _calculate_user_changes

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-07 12:42:05 +00:00
David Robertson
2a3ec6facf
Correctly register shutdown handler for presence workers (#11518)
Fixes #11517
2021-12-07 12:34:38 +00:00
Richard van der Hoff
b1ecd19c5d
Fix 'delete room' admin api to work on incomplete rooms (#11523)
If, for some reason, we don't have the create event, we should still be able to
purge a room.
2021-12-07 11:37:54 +00:00
David Robertson
9c55dedc8c
Correctly ignore invites from ignored users (#11511) 2021-12-07 11:24:31 +00:00
Quentin Gliech
a15a893df8
Save the OIDC session ID (sid) with the device on login (#11482)
As a step towards allowing back-channel logout for OIDC.
2021-12-06 12:43:06 -05:00
Patrick Cloke
494ebd7347
Include bundled aggregations in /sync and related fixes (#11478)
Due to updates to MSC2675 this includes a few fixes:

* Include bundled aggregations for /sync.
* Do not include bundled aggregations for /initialSync and /events.
* Do not bundle aggregations for state events.
* Clarifies comments and variable names.
2021-12-06 15:51:15 +00:00
reivilibre
637df95de6
Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445) 2021-12-03 16:42:44 +00:00
David Robertson
d26808dd85
Comments on the /sync tentacles (#11494)
This mainly consists of docstrings and inline comments. There are one or two type annotations and variable renames thrown in while I was here.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-02 20:58:32 +00:00
Eric Eastwood
a6f1a3abec
Add MSC3030 experimental client and federation API endpoints to get the closest event to a given timestamp (#9445)
MSC3030: https://github.com/matrix-org/matrix-doc/pull/3030

Client API endpoint. This will also go and fetch from the federation API endpoint if unable to find an event locally or we found an extremity with possibly a closer event we don't know about.
```
GET /_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}
```

Federation API endpoint:
```
GET /_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>
{
    "event_id": ...
    "origin_server_ts": ...
}
```

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-12-02 01:02:20 -06:00
Patrick Cloke
a4521ce0a8
Support the stable /hierarchy endpoint from MSC2946 (#11329)
This also makes additional updates where the implementation
had drifted from the approved MSC.

Unstable endpoints will be removed at a later data.
2021-11-29 14:32:20 -05:00