Commit Graph

2926 Commits

Author SHA1 Message Date
Dirk Klimpel
d666fc02fa
Add type hints to some tests files (#12371) 2022-04-05 13:54:41 +01:00
reivilibre
708d88b1a2
Allow specifying the Postgres database's port when running unit tests with Postgres. (#12376) 2022-04-05 12:44:48 +01:00
Erik Johnston
5c9e39e619
Track device list updates per room. (#12321)
This is a first step in dealing with #7721.

The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things:

1. we can defer calculating the set of remote servers that need to be poked about the change; and
2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed.

However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled.

There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly).

Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
2022-04-04 15:25:20 +01:00
Richard van der Hoff
f0b03186d9
Add type hints for tests/unittest.py. (#12347)
In particular, add type hints for get_success and friends, which are then helpful in a bunch of places.
2022-04-01 16:04:16 +00:00
Richard van der Hoff
33ebee47e4
Remove redundant get_success calls in test code (#12346)
There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
2022-04-01 16:10:31 +01:00
reivilibre
c4cf916ed7
Default to private room visibility rather than public when a client does not specify one, according to spec. (#12350) 2022-04-01 15:55:09 +01:00
Nick Mills-Barrett
993d90f82b
Use a sequence to generate AS transaction IDs, drop last_txn AS state (#12209)
Switching to a sequence means there's no need to track `last_txn` on the
AS state table to generate new TXN IDs. This also means that there is
no longer contention between the AS scheduler and AS handler on updates
to the `application_services_state` table, which will prevent serialization
errors during the complete AS txn transaction.
2022-04-01 13:33:25 +00:00
Richard van der Hoff
b7762b0c9f
Move single-use methods out of TestCase (#12348)
These methods are only used by a single testcase, so they shouldn't be
cluttering up the base `TestCase` class.
2022-04-01 12:48:08 +00:00
Richard van der Hoff
319a805cd3
Raise an exception when getting state at an outlier (#12191)
It seems like calling `_get_state_group_for_events` for an event where the
state is unknown is an error. Accordingly, let's raise an exception rather than
silently returning an empty result.
2022-04-01 13:01:49 +01:00
Richard van der Hoff
9b43df1f7b
Optimise _get_state_after_missing_prev_event: use /state (#12040)
If we're missing most of the events in the room state, then we may as well call the /state endpoint, instead of individually requesting each and every event.
2022-04-01 12:53:42 +01:00
Brendan Abolivier
e4409301ba
Add a module callback to react to account data changes (#12327)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2022-04-01 11:22:48 +02:00
Amanda Graven
4e900ece42
Add set_user_admin function to the module API (#12341) 2022-04-01 09:31:30 +01:00
Patrick Cloke
bebf994ee8
Move MSC2654 support behind an experimental configuration flag. (#12295)
To match the current thinking on disabling experimental features by default.
2022-03-31 15:05:13 -04:00
Richard van der Hoff
6927d87254
Handle outliers in /federation/v1/event (#12332)
The intention here is to avoid doing state lookups for outliers in
`/_matrix/federation/v1/event`. Unfortunately that's expanded into something of
a rewrite of `filter_events_for_server`, which ended up trying to do that
operation in a couple of places.
2022-03-31 17:39:34 +00:00
Brendan Abolivier
5e88143dff
Add a callback to react to 3PID associations (#12302) 2022-03-31 18:27:21 +02:00
Brendan Abolivier
f96b85eca8
Ensure the type of URL attributes is always str when matching against preview blacklist (#12333) 2022-03-31 11:49:49 +02:00
Patrick Cloke
c31c1091d4
Remove the unused and unstable /aggregations endpoint. (#12293)
This endpoint was removed from MSC2675 before it was approved.
It is currently unspecified (even in any MSCs) and therefore subject to
removal. It is not implemented by any known clients.

This also changes the bundled aggregation format for `m.annotation`,
which previously included pagination tokens for the `/aggregations`
endpoint, which are no longer useful.
2022-03-30 11:45:32 -04:00
Andrew Morgan
d8d0271977
Send device list updates to application services (MSC3202) - part 1 (#11881)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-30 14:39:27 +01:00
Andrew Morgan
19a1d6a42a
Remove references to "msc2403" (#12165) 2022-03-30 11:15:47 +00:00
Andrew Morgan
c8cbd66d3b
Start application service stream token tracking from 1 (#12193)
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-03-30 11:10:12 +00:00
Brendan Abolivier
437a8ed9ef
Add a configuration to exclude rooms from sync response (#12310) 2022-03-30 09:43:04 +00:00
David Robertson
e0bb268134
Fix typechecker problems exposed by signedjson 1.1.2 (#12326) 2022-03-29 21:37:50 +00:00
David Robertson
a2b00a4486
Bump black and click versions (#12320) 2022-03-29 10:41:19 +00:00
Shay
3c41d87b67
Add restrictions by default to open registration in Synapse (#12091) 2022-03-25 10:11:01 -07:00
David Robertson
fffb3c4c8f
Always allow the empty string as an avatar_url. (#12261)
Hopefully this fixes #12257.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-25 13:28:42 +00:00
Patrick Cloke
1800bd47a8
Remove unused test code. (#12291)
The `MockHttpResource` and `MockKey` objects were unused
since #9396 (74af356baf).
2022-03-24 19:25:52 +00:00
Patrick Cloke
4df10d3214
Do not consider events by ignored users for relations (#12285)
Filter the events returned from `/relations` for the requester's ignored users
in a similar way to `/messages` (and `/sync`).
2022-03-24 10:25:42 -04:00
Brendan Abolivier
5436b014f4
Optionally include account validity in MSC3720 account status responses (#12266) 2022-03-24 11:19:41 +01:00
Shay
e78d4f61fc
Refuse to start if DB has an unsafe locale (#12262) 2022-03-23 10:23:05 -07:00
Nicolas Werner
692b82838e
Allow registering admin users using the module API (#12250)
Signed-off-by: Nicolas Werner <n.werner@famedly.com>
2022-03-23 12:48:47 +01:00
Jonathan de Jong
516d092ff9
Rename shared_rooms to mutual_rooms (#12036)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-03-23 11:19:20 +00:00
Dirk Klimpel
9d21ecf7ce
Add type hints to tests files. (#12256) 2022-03-21 09:43:16 -04:00
David Robertson
bf9d549e3a
Try to detect borked package installations. (#12244)
* Try to detect borked package installations.

Fixes #12223.

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-03-18 19:03:46 +00:00
Patrick Cloke
80e0e1f35e
Only fetch thread participation for events with threads. (#12228)
We fetch the thread summary in two phases:

1. The summary that is shared by all users (count of messages and latest event).
2. Whether the requesting user has participated in the thread.

There's no use in attempting step 2 for events which did not return a summary
from step 1.
2022-03-18 17:15:45 +00:00
Dirk Klimpel
9e06e22064
Add type hints to more tests files. (#12240) 2022-03-17 07:25:50 -04:00
Patrick Cloke
3f7cfbc9e5 Merge branch 'release-v1.55' into develop 2022-03-16 12:22:07 -04:00
Patrick Cloke
96274565ff
Fix bundling aggregations if unsigned is not a returned event field. (#12234)
An error occured if a filter was supplied with `event_fields` which did not include
`unsigned`.

In that case, bundled aggregations are still added as the spec states it is allowed
for servers to add additional fields.
2022-03-16 12:17:39 -04:00
Sean Quah
6121056740
Handle cancellation in DatabasePool.runInteraction() (#12199)
To handle cancellation, we ensure that `after_callback`s and
`exception_callback`s are always run, since the transaction will
complete on another thread regardless of cancellation.

We also wait until everything is done before releasing the
`CancelledError`, so that logging contexts won't get used after they
have been finished.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-16 15:07:41 +00:00
Patrick Cloke
1da0f79d54
Refactor relations tests (#12232)
* Moves the relation pagination tests to a separate class.
* Move the assertion of the response code into the `_send_relation` helper.
* Moves some helpers into the base-class.
2022-03-16 09:20:57 -04:00
Patrick Cloke
4587b35929
Clean-up logic for rebasing URLs during URL preview. (#12219)
By using urljoin from the standard library and reducing the number
of places URLs are rebased.
2022-03-16 07:21:36 -04:00
Patrick Cloke
dda9b7fc4d
Use the ignored_users table to test event visibility & sync. (#12225)
Instead of fetching the raw account data and re-parsing it. The
ignored_users table is a denormalised version of the account data
for quick searching.
2022-03-15 14:06:05 -04:00
Sean Quah
dea577998f
Add tests for database transaction callbacks (#12198)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-15 15:40:34 +00:00
Dirk Klimpel
5dd949bee6
Add type hints to some tests/handlers files. (#12224) 2022-03-15 09:16:37 -04:00
Sean Quah
2fcf4b3f6c
Add cancellation support to @cached and @cachedList decorators (#12183)
These decorators mostly support cancellation already. Add cancellation
tests and fix use of finished logging contexts by delaying cancellation,
as suggested by @erikjohnston.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 19:04:29 +00:00
Sean Quah
605d161d7d
Add cancellation support to ReadWriteLock (#12120)
Also convert `ReadWriteLock` to use async context managers.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 18:49:07 +00:00
Sean Quah
90b2327066
Add delay_cancellation utility function (#12180)
`delay_cancellation` behaves like `stop_cancellation`, except it
delays `CancelledError`s until the original `Deferred` resolves.
This is handy for unifying cleanup paths and ensuring that uncancelled
coroutines don't use finished logcontexts.

Signed-off-by: Sean Quah <seanq@element.io>
2022-03-14 17:52:15 +00:00
Shay
ef3619e61d
Add config settings for background update parameters (#11980) 2022-03-11 10:46:45 -08:00
Nick Mills-Barrett
735e89bd3a
Add an additional HTTP pusher + push rule tests. (#12188)
And rename the field used for caching from _id to _cache_key.
2022-03-11 08:45:26 -05:00
Dirk Klimpel
32c828d0f7
Add type hints to tests/rest. (#12208)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-11 12:42:22 +00:00
Patrick Cloke
e10a2fe0c2
Add some type hints to the tests.handlers module. (#12207) 2022-03-11 07:07:15 -05:00
Patrick Cloke
bc9dff1d95
Remove unnecessary pass statements. (#12206) 2022-03-11 07:06:21 -05:00
Richard van der Hoff
483f2aa2ec
Retention test: avoid relying on state at purged events (#12202)
This test was relying on poking events which weren't in the database into
filter_events_for_client.
2022-03-11 10:33:49 +00:00
Patrick Cloke
ea27528b5d
Support stable identifiers for MSC3440: Threading (#12151)
The unstable identifiers are still supported if the experimental configuration
flag is enabled. The unstable identifiers will be removed in a future release.
2022-03-10 15:36:13 +00:00
Patrick Cloke
88cd6f9378
Allow retrieving the relations of a redacted event. (#12130)
This is allowed per MSC2675, although the original implementation did
not allow for it and would return an empty chunk / not bundle aggregations.

The main thing to improve is that the various caches get cleared properly
when an event is redacted, and that edits must not leak if the original
event is redacted (as that would presumably leak something similar to
the original event content).
2022-03-10 09:03:59 -05:00
Patrick Cloke
3e4af36bc8
Rename get_tcp_replication to get_replication_command_handler. (#12192)
Since the object it returns is a ReplicationCommandHandler.

This is clean-up from adding support to Redis where the command handler
was added as an additional layer of abstraction from the TCP protocol.
2022-03-10 13:01:56 +00:00
Sean Quah
a4c1fdb44a
Remove dead code in tests/storage/test_database.py (#12197)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-09 18:45:21 +00:00
Will Hunt
15382b1afa
Add third_party module callbacks to check if a user can delete a room and deactivate a user (#12028)
* Add check_can_deactivate_user

* Add check_can_shutdown_rooms

* Documentation

* callbacks, not functions

* Various suggested tweaks

* Add tests for test_check_can_shutdown_room and test_check_can_deactivate_user

* Update check_can_deactivate_user to not take a Requester

* Fix check_can_shutdown_room docs

* Renegade and use `by_admin` instead of `admin_user_id`

* fix lint

* Update docs/modules/third_party_rules_callbacks.md

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Update docs/modules/third_party_rules_callbacks.md

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Update docs/modules/third_party_rules_callbacks.md

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

* Update docs/modules/third_party_rules_callbacks.md

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-03-09 18:23:57 +00:00
Patrick Cloke
690cb4f3b3
Allow for ignoring some arguments when caching. (#12189)
* `@cached` can now take an `uncached_args` which is an iterable of names to not use in the cache key.
* Requires `@cached`, @cachedList` and `@lru_cache` to use keyword arguments for clarity.
* Asserts that keyword-only arguments in cached functions are not accepted. (I tested this briefly and I don't believe this works properly.)
2022-03-09 18:07:41 +00:00
Olivier Wilkinson (reivilibre)
0dc9c5653c Merge branch 'master' into develop 2022-03-08 15:37:35 +00:00
Patrick Cloke
ca9234a9eb
Do not return allowed_room_ids from /hierarchy response. (#12175)
This field is only to be used in the Server-Server API, and not the
Client-Server API, but was being leaked when a federation response
was used in the /hierarchy API.
2022-03-08 08:09:11 -05:00
reivilibre
2eef234ae3
Fix a bug introduced in 1.54.0rc1 which meant that Synapse would refuse to start if pre-release versions of dependencies were installed. (#12177)
* Add failing test to characterise the regression #12176

* Permit pre-release versions of specified packages

* Newsfile (bugfix)

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-03-08 10:47:28 +00:00
Shay
26211fec24
Fix a bug in background updates wherein background updates are never run using the default batch size (#12157) 2022-03-07 09:44:33 -08:00
Patrick Cloke
f63bedef07
Invalidate caches when an event with a relation is redacted. (#12121)
The caches for the target of the relation must be cleared
so that the bundled aggregations are re-calculated after
the redaction is processed.
2022-03-07 14:00:05 +00:00
Richard van der Hoff
00a67f831a Merge remote-tracking branch 'origin/release-v1.54' into develop 2022-03-04 22:40:51 +00:00
Sean Quah
158e0937eb
Add test for ObservableDeferred's cancellation behaviour (#12149)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-04 13:10:05 +00:00
Patrick Cloke
cd1ae3d0b4
Remove backwards compatibility with RelationPaginationToken. (#12138) 2022-03-04 07:10:10 -05:00
Erik Johnston
423cca9efe
Spread out sending device lists to remote hosts (#12132) 2022-03-04 11:48:15 +00:00
Richard van der Hoff
87c230c27c
Update client-visibility filtering for outlier events (#12155)
Avoid trying to get the state for outliers, which isn't a sensible thing to do.
2022-03-04 10:31:19 +00:00
Andrew Morgan
fb0ffa9676
Rename various ApplicationServices interested methods (#11915) 2022-03-03 18:14:09 +00:00
Dirk Klimpel
7e91107be1
Add type hints to tests/rest (#12146)
* Add type hints to `tests/rest`

* newsfile

* change import from `SigningKey`
2022-03-03 16:05:44 +00:00
Patrick Cloke
1d11b452b7
Use the proper serialization format when bundling aggregations. (#12090)
This ensures that the `latest_event` field of the bundled aggregation
for threads uses the same format as the other events in the response.
2022-03-03 10:43:06 -05:00
David Robertson
cea1b58c4a
Don't impose version checks on dev extras at runtime (#12129)
* Fix incorrect argument in test case

* Add copyright header

* Docstring and __all__

* Exclude dev depenencies

* Use changelog from #12088

* Include version in error messages

This will hopefully distinguish between the version of the source code
and the version of the distribution package that is installed.

* Linter script is your friend
2022-03-03 12:47:55 +00:00
David Robertson
106959b3cf
Remove unused mocks from test_typing (#12136)
* Remove unused mocks from `test_typing`

It's not clear what these do. `get_user_by_access_token` has the wrong
signature, including the return type. Tests all pass without these. I
think we should nuke them.

* Changelog

* Fixup imports
2022-03-02 17:24:52 +00:00
Dirk Klimpel
2ffaf30803
Add type hints to tests/rest/client (#12108)
* Add type hints to `tests/rest/client`

* newsfile

* fix imports

* add `test_account.py`

* Remove one type hint in `test_report_event.py`

* change `on_create_room` to `async`

* update new functions in `test_third_party_rules.py`

* Add `test_filter.py`

* add `test_rooms.py`

* change to `assertEquals` to `assertEqual`

* lint
2022-03-02 16:34:14 +00:00
Patrick Cloke
6adb89ff00
Improve and refactor the tests for relations. (#12113)
* Modernizes code (f-strings, etc.)
* Fixes incorrect comments.
* Splits the test case into two.
* Factors out some duplicated code.
2022-03-02 06:56:16 -05:00
reivilibre
c7b2f1ccdc
Back out in-flight state caching changes. (#12126) 2022-03-02 10:37:04 +00:00
lukasdenk
8e56a1b73c
Make get_room_version use cached get_room_version_id. (#11808) 2022-03-02 10:35:34 +00:00
David Robertson
313581e4e9
Use importlib.metadata to read requirements (#12088)
* Pull runtime dep checks into their own module
* Reimplement `check_requirements` using `importlib`

I've tried to make this clearer. We start by working out which of
Synapse's requirements we need to be installed here and now. I was
surprised that there wasn't an easier way to see which packages were
installed by a given extra.

I've pulled out the error messages into functions that deal with "is
this for an extra or not". And I've rearranged the loop over two
different sets of requirements into one loop with a "must be instaled"
flag.

I hope you agree that this is clearer.

* Test cases
2022-03-01 17:44:41 +00:00
Sean Quah
4d6b6c17c8
Fix rare error in ReadWriteLock when writers complete immediately (#12105)
Signed-off-by: Sean Quah <seanq@element.io>
2022-03-01 15:27:15 +00:00
Brendan Abolivier
300ed0b8a6
Add module callbacks called for reacting to deactivation status change and profile update (#12062) 2022-03-01 15:00:03 +00:00
Sean Quah
91bc15c772
Add stop_cancellation utility function (#12106) 2022-03-01 13:51:03 +00:00
reivilibre
c893632319
Order in-flight state group queries in biggest-first order (#11610)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-03-01 13:41:57 +00:00
Richard van der Hoff
e2e1d90a5e
Faster joins: persist to database (#12012)
When we get a partial_state response from send_join, store information in the
database about it:
 * store a record about the room as a whole having partial state, and stash the
   list of member servers too.
 * flag the join event itself as having partial state
 * also, for any new events whose prev-events are partial-stated, note that
   they will *also* be partial-stated.

We don't yet make any attempt to interpret this data, so API calls (and a bunch
of other things) are just going to get incorrect data.
2022-03-01 12:49:54 +00:00
Richard van der Hoff
5458eb8551
Fix 'Unhandled error in Deferred' (#12089)
* Fix 'Unhandled error in Deferred'

Fixes a CRITICAL "Unhandled error in Deferred" log message which happened when
a function wrapped with `@cachedList` failed

* Minor optimisation to cachedListDescriptor

we can avoid re-using `missing`, which saves looking up entries in
`deferreds_map`, and means we don't need to copy it.

* Improve type annotation on CachedListDescriptor
2022-03-01 09:51:38 +00:00
Richard van der Hoff
9d11fee8f2
Improve exception handling for concurrent execution (#12109)
* fix incorrect unwrapFirstError import

this was being imported from the wrong place

* Refactor `concurrently_execute` to use `yieldable_gather_results`

* Improve exception handling in `yieldable_gather_results`

Try to avoid swallowing so many stack traces.

* mark unwrapFirstError deprecated

* changelog
2022-03-01 09:34:30 +00:00
Dirk Klimpel
952efd0bca
Add type hints to tests/rest/client (#12094)
* Add type hints to `tests/rest/client`

* update `mypy.ini`

* newsfile

* add `test_register.py`
2022-02-28 18:59:00 +00:00
Patrick Cloke
7754af24ab
Remove the unstable /spaces endpoint. (#12073)
...and various code supporting it.

The /spaces endpoint was from an old version of MSC2946 and included
both a Client-Server and Server-Server API. Note that the unstable
/hierarchy endpoint (from the final version of MSC2946) is not yet
removed.
2022-02-28 18:33:00 +00:00
Dirk Klimpel
1901cb1d4a
Add type hints to tests/rest/client (#12084) 2022-02-28 17:47:37 +00:00
Richard van der Hoff
6c0b44a3d7
Fix PushRuleEvaluator and Filter to work on frozendicts (#12100)
* Fix `PushRuleEvaluator` to work on frozendicts

frozendicts do not (necessarily) inherit from dict, so this needs to handle
them correctly.

* Fix event filtering for frozen events

Looks like this one was introduced by #11194.
2022-02-28 17:40:24 +00:00
David Robertson
5565f454e1
Actually fix bad debug logging rejecting device list & signing key transactions (#12098) 2022-02-28 14:10:36 +00:00
Patrick Cloke
02d708568b
Replace assertEquals and friends with non-deprecated versions. (#12092) 2022-02-28 07:12:29 -05:00
Dirk Klimpel
54e74cc15f
Add type hints to tests/rest/client (#12072) 2022-02-24 13:56:38 -05:00
reivilibre
2cc5ea933d
Add support for MSC3202: sending one-time key counts and fallback key usage states to Application Services. (#11617)
Co-authored-by: Erik Johnston <erik@matrix.org>
2022-02-24 17:55:45 +00:00
Sean Quah
41cf4c2cf6
Fix non-strings in the event_search table (#12037)
Don't attempt to add non-string `value`s to `event_search` and add a
background update to clear out bad rows from `event_search` when
using sqlite.

Signed-off-by: Sean Quah <seanq@element.io>
2022-02-24 11:52:28 +00:00
Dirk Klimpel
64c73c6ac8
Add type hints to tests/rest/client (#12066) 2022-02-23 08:33:19 -05:00
Richard van der Hoff
5b2b36809f
Remove more references to get_datastore (#12067)
These have snuck in since #12031 was started.

Also a couple of other cleanups while we're in the area.
2022-02-23 12:35:53 +00:00
Richard van der Hoff
e24ff8ebe3
Remove HomeServer.get_datastore() (#12031)
The presence of this method was confusing, and mostly present for backwards
compatibility. Let's get rid of it.

Part of #11733
2022-02-23 11:04:02 +00:00
Brendan Abolivier
250104d357
Implement account status endpoints (MSC3720) (#12001)
See matrix-org/matrix-doc#3720

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-02-22 15:10:10 +00:00
reivilibre
dcb6a37837
Cap the number of in-flight requests for state from a single group (#11608) 2022-02-22 14:24:31 +00:00
reivilibre
235d2916ce
Fix slow performance of /logout in some cases where refresh tokens are in use. The slowness existed since the initial implementation of refresh tokens. (#12056) 2022-02-22 13:29:04 +00:00
Richard van der Hoff
7273011f60
Faster joins: Support for calling /federation/v1/state (#12013)
This is an endpoint that we have server-side support for, but no client-side support. It's going to be useful for resyncing partial-stated rooms, so let's introduce it.
2022-02-22 12:17:10 +00:00
reivilibre
546b9c9e64
Add more tests for in-flight state query duplication. (#12033) 2022-02-22 11:44:11 +00:00
reivilibre
284ea2025a
Track and deduplicate in-flight requests to _get_state_for_groups. (#10870)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-18 17:23:31 +00:00
reivilibre
eb609c65d0
Fix bug in StateFilter.return_expanded() and add some tests. (#12016) 2022-02-18 14:54:31 +00:00
Travis Ralston
19bd9cff1a
Use stable MSC3069 is_guest flag on /whoami. (#12021)
Keeping backwards compatibility with the unstable flag for now.
2022-02-18 07:48:23 -05:00
Brendan Abolivier
707049c6ff
Allow modules to set a display name on registration (#12009)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-17 16:54:16 +00:00
Richard van der Hoff
da0e9f8efd
Faster joins: parse msc3706 fields in send_join response (#12011)
Part of my work on #11249: add code to handle the new fields added in MSC3706.
2022-02-17 16:11:59 +00:00
lukasdenk
4077177390
Prevent duplicate push notifications for room reads (#11835) 2022-02-17 10:23:54 +00:00
Erik Johnston
dc9fe61050
Fix incorrect get_rooms_for_user for remote user (#11999)
When the server leaves a room the `get_rooms_for_user` cache is not
correctly invalidated for the remote users in the room. This means that
subsequent calls to `get_rooms_for_user` for the remote users would
incorrectly include the room (it shouldn't be included because the
server no longer knows anything about the room).
2022-02-15 14:26:28 +00:00
Patrick Cloke
45f45404de
Fix incorrect thread summaries when the latest event is edited. (#11992)
If the latest event in a thread was edited than the original
event content was included in bundled aggregation for
threads instead of the edited event content.
2022-02-15 08:26:57 -05:00
Andrew Morgan
87f2005713
Add some tests for propagation of device list changes between local users (#11972) 2022-02-15 11:27:56 +00:00
Richard van der Hoff
63c46349c4
Implement MSC3706: partial state in /send_join response (#11967)
* Make `get_auth_chain_ids` return a Set

It has a set internally, and a set is often useful where it gets used, so let's
avoid converting to an intermediate list.

* Minor refactors in `on_send_join_request`

A little bit of non-functional groundwork

* Implement MSC3706: partial state in /send_join response
2022-02-12 10:44:16 +00:00
Patrick Cloke
4d7e74b2e5
Support the MSC3715 for /relations. (#11941)
This adds an unstable org.matrix.msc3715.dir parameter
which acts like dir on /mesages.
2022-02-11 11:20:27 -05:00
Brendan Abolivier
0171fa5226
Remove deprecated user_may_create_room_with_invites callback (#11950)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-11 13:58:11 +00:00
Erik Johnston
79fb64e417
Fix to-device being dropped in limited sync in SQLite. (#11966)
If ther are more than 100 to-device messages pending for a device
`/sync` will only return the first 100, however the next batch token was
incorrectly calculated and so all other pending messages would be
dropped.

This is due to `txn.rowcount` only returning the number of rows that
*changed*, rather than the number *selected* in SQLite.
2022-02-11 13:38:05 +00:00
Patrick Cloke
a121507cfe
Adds misc missing type hints (#11953) 2022-02-11 07:20:16 -05:00
Richard van der Hoff
c3db7a0b59
Tests: replace mocked Authenticator with the real thing (#11913)
If we prepopulate the test homeserver with a key for a remote homeserver, we
can make federation requests to it without having to stub out the
authenticator. This has two advantages:

 * means that what we are testing is closer to reality (ie, we now have
   complete tests for the incoming-request-authorisation flow)

 * some tests require that other objects be signed by the remote server (eg,
   the event in `/send_join`), and doing that would require a whole separate
   set of mocking out. It's much simpler just to use real keys.
2022-02-11 12:06:02 +00:00
Dirk Klimpel
d36943c4df
Support the stable API endpoint for MSC3283: new settings in /capabilities endpoint (#11933) 2022-02-11 09:32:11 +00:00
Patrick Cloke
df36945ff0
Support pagination tokens from /sync and /messages in the relations API. (#11952) 2022-02-10 10:52:48 -05:00
Denis Kasak
337f38cac3
Implement a content type allow list for URL previews (#11936)
This implements an allow list for content types for which Synapse will attempt URL preview. If a URL resolves to a resource with a content type which isn't in the list, the download will terminate immediately.

This makes sense given that Synapse would never successfully generate a URL preview for such files in the first place, and helps prevent issues with streaming media servers, such as #8302.

Signed-off-by: Denis Kasak dkasak@termina.org.uk
2022-02-10 15:43:01 +00:00
Patrick Cloke
d0e78af35e
Add missing type hints to synapse.replication. (#11938) 2022-02-08 11:03:08 -05:00
Patrick Cloke
8c94b3abe9
Experimental support to include bundled aggregations in search results (MSC3666) (#11837) 2022-02-08 09:21:20 -05:00
Brendan Abolivier
0640f8ebaa
Add a callback to allow modules to deny 3PID (#11854)
Part of the Tchap Synapse mainlining.

This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
2022-02-08 11:20:32 +01:00
Andrew Morgan
cf06783d54
Remove optional state of ApplicationService.is_interested's store parameter (#11911) 2022-02-07 18:26:42 +00:00
Andrew Morgan
0c4878caf2
Add a unit test for users receiving their own device list updates (#11909) 2022-02-07 13:21:19 +00:00
Jonathan de Jong
02632b3504
Stabilise MSC3231 (Token Based Registration) (#11867) 2022-02-04 12:15:13 +00:00
Patrick Cloke
119edf51eb
Remove support for the webclient listener. (#11895)
Also remove support for non-HTTP(S) web_client_location.
2022-02-03 18:36:49 +00:00
reivilibre
833247553f
Allow specifying the application service-specific user_id parameter in the join test helper. (#11616) 2022-02-03 13:09:22 +00:00
Richard van der Hoff
31b554c297
Fixes for opentracing scopes (#11869)
`start_active_span` was inconsistent as to whether it would activate the span
immediately, or wait for `scope.__enter__` to happen (it depended on whether
the current logcontext already had an associated scope). The inconsistency was
rather confusing if you were hoping to set up a couple of separate spans before
activating either.

Looking at the other implementations of opentracing `ScopeManager`s, the
intention is that it *should* be activated immediately, as the name
implies. Indeed, the idea is that you don't have to use the scope as a
contextmanager at all - you can just call `.close` on the result. Hence, our
cleanup has to happen in `.close` rather than `.__exit__`.

So, the main change here is to ensure that `start_active_span` does activate
the span, and that `scope.close()` does close the scope.

We also add some tests, which requires a `tracer` param so that we don't have
to rely on the global variable in unit tests.
2022-02-02 22:41:57 +00:00
reivilibre
af795173be
Add a background database update to purge account data for deactivated users. (#11655) 2022-02-02 11:37:18 +00:00
reivilibre
513913cc6b
Expose the registered device ID from the register_appservice_user test helper. (#11615) 2022-02-02 09:59:55 +00:00
Andrew Morgan
3f72c2a322
Convert ApplicationServiceTestCase to use simple_async_mock (#11880) 2022-02-01 17:45:13 +00:00
Andrew Morgan
64ec45fc1b
Send to-device messages to application services (#11215)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-02-01 14:13:38 +00:00
Dirk Klimpel
901b264c0c
Add type hints to tests/rest/admin (#11851) 2022-01-31 14:20:05 -05:00
Brendan Abolivier
bf60da1a60
Configurable limits on avatars (#11846)
Only allow files which file size and content types match configured
limits to be set as avatar.

Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
2022-01-28 15:41:33 +01:00
Brendan Abolivier
2d3bd9aa67
Add a module callback to set username at registration (#11790)
This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step).
This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not.

Co-authored-by: David Robertson <davidr@element.io>
2022-01-26 14:21:13 +00:00
Patrick Cloke
2897fb6b4f
Improvements to bundling aggregations. (#11815)
This is some odds and ends found during the review of #11791
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
Brendan Abolivier
95b3f952fa
Add a config flag to inhibit M_USER_IN_USE during registration (#11743)
This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work).

This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique.

More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
2022-01-26 13:02:54 +01:00
Dirk Klimpel
6a72c910f1
Add admin API to get a list of federated rooms (#11658) 2022-01-25 16:11:40 +00:00
Dirk Klimpel
0d6cfea9b8
Add admin API to reset connection timeouts for remote server (#11639)
* Fix get federation status of destination if no error occured
2022-01-25 12:06:29 +00:00
Patrick Cloke
807efd26ae
Support rendering previews with data: URLs in them (#11767)
Images which are data URLs will no longer break URL
previews and will properly be "downloaded" and
thumbnailed.
2022-01-24 08:58:18 -05:00
Andrew Morgan
c3040dd5cc Synapse 1.51.0rc2 (2022-01-24)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmHum5QTHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9G6vD/9Dw6V0687vzahnU6aYWaecRSO1sbww
 EtcCiXOh0r8HXPAwEIXJYomSTTbPl0eAwP8T1the1WZVQArvRW2VvMzDoBheO8bt
 dCm4CTKFCGUsI/GrlFDPtjAEd6kruAETmpHZn7bAFSqtIpissD6FjUwg/ND/NJfM
 zVM/bMgM0+js6eD9J5/k16E1ZWj7Lbp+/fKN+qTeQrXzIeT13WQZ4Nz1o/cqe/21
 o/coI3FmYq8CzKpfA6qZMDd/OtYpYqwwr7otSmW+6qHeZI/yqeoxlpzfgSZGpRUe
 mtXqbQllZQeqrbm8oK96GNmKcThy6awTwJPoD46b1AOkmFdf/lGSqO0lnTQRRPqR
 hyPPdrx6Lt2t7DDbVVyUElzkqLPhVJtrItPDC8669sWvmSgsPQdUqIsRmhF+8aJe
 ffjRvKbGRICnNZkT+qGf1HwuMHajZyAIHAS/kyFDKUZCvau3VQ1wlyOqVQ1hWr7+
 3k3CobjSx4y1bYKXncAK6hWH6lE8M319jaTnVfYXocDLWRonyFf7o286Q+c8WBGF
 tY0FzvUPb5S2kktC4WLwcqtcTWK1cu5MI6GfD69EqL7iifJRVyKDeUoD7tiUvgzN
 O2wBl2soJIsAU+8y16WQu7p+k2nZIomCCAySnW3C8mlxvv7CKQu0Url8J7IH8uZE
 ZVN2MMe7H+ZHJQ==
 =Fpsa
 -----END PGP SIGNATURE-----

Merge tag 'v1.51.0rc2' into develop

Synapse 1.51.0rc2 (2022-01-24)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
2022-01-24 13:55:03 +00:00
reivilibre
df54c8485a
Remove account data (including client config, push rules and ignored users) upon user deactivation. (#11621)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-01-24 13:37:00 +00:00
Andrew Morgan
dc671d3ea7
Fix logic for dropping old events in fed queue (#11806)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Richard van der Hoff <richard@matrix.org>
2022-01-24 12:20:01 +00:00
Richard van der Hoff
2aa37a4250
Add state_key and rejection_reason to events (#11792)
... and start populating them for new events
2022-01-21 12:21:28 +00:00
Patrick Cloke
7a11509d17 Do not try to serialize raw aggregations dict. (#11791) 2022-01-21 10:40:34 +00:00
Patrick Cloke
b784299cbc
Do not try to serialize raw aggregations dict. (#11791) 2022-01-21 10:31:31 +00:00
Richard van der Hoff
9f2016e96e
Drop unused table public_room_list_stream. (#11795)
This is a follow-up to #10565.
2022-01-21 09:19:56 +00:00
reivilibre
4c2096599c
Make the get_global_account_data_by_type_for_user cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
Brendan Abolivier
bfe6d5553a
Correctly await on_logged_out callbacks (#11786) 2022-01-20 19:19:40 +01:00