Commit Graph

2698 Commits

Author SHA1 Message Date
Richard van der Hoff
31b554c297
Fixes for opentracing scopes (#11869)
`start_active_span` was inconsistent as to whether it would activate the span
immediately, or wait for `scope.__enter__` to happen (it depended on whether
the current logcontext already had an associated scope). The inconsistency was
rather confusing if you were hoping to set up a couple of separate spans before
activating either.

Looking at the other implementations of opentracing `ScopeManager`s, the
intention is that it *should* be activated immediately, as the name
implies. Indeed, the idea is that you don't have to use the scope as a
contextmanager at all - you can just call `.close` on the result. Hence, our
cleanup has to happen in `.close` rather than `.__exit__`.

So, the main change here is to ensure that `start_active_span` does activate
the span, and that `scope.close()` does close the scope.

We also add some tests, which requires a `tracer` param so that we don't have
to rely on the global variable in unit tests.
2022-02-02 22:41:57 +00:00
reivilibre
af795173be
Add a background database update to purge account data for deactivated users. (#11655) 2022-02-02 11:37:18 +00:00
reivilibre
513913cc6b
Expose the registered device ID from the register_appservice_user test helper. (#11615) 2022-02-02 09:59:55 +00:00
Andrew Morgan
3f72c2a322
Convert ApplicationServiceTestCase to use simple_async_mock (#11880) 2022-02-01 17:45:13 +00:00
Andrew Morgan
64ec45fc1b
Send to-device messages to application services (#11215)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-02-01 14:13:38 +00:00
Dirk Klimpel
901b264c0c
Add type hints to tests/rest/admin (#11851) 2022-01-31 14:20:05 -05:00
Brendan Abolivier
bf60da1a60
Configurable limits on avatars (#11846)
Only allow files which file size and content types match configured
limits to be set as avatar.

Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
2022-01-28 15:41:33 +01:00
Brendan Abolivier
2d3bd9aa67
Add a module callback to set username at registration (#11790)
This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step).
This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not.

Co-authored-by: David Robertson <davidr@element.io>
2022-01-26 14:21:13 +00:00
Patrick Cloke
2897fb6b4f
Improvements to bundling aggregations. (#11815)
This is some odds and ends found during the review of #11791
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
Brendan Abolivier
95b3f952fa
Add a config flag to inhibit M_USER_IN_USE during registration (#11743)
This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work).

This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique.

More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
2022-01-26 13:02:54 +01:00
Dirk Klimpel
6a72c910f1
Add admin API to get a list of federated rooms (#11658) 2022-01-25 16:11:40 +00:00
Dirk Klimpel
0d6cfea9b8
Add admin API to reset connection timeouts for remote server (#11639)
* Fix get federation status of destination if no error occured
2022-01-25 12:06:29 +00:00
Patrick Cloke
807efd26ae
Support rendering previews with data: URLs in them (#11767)
Images which are data URLs will no longer break URL
previews and will properly be "downloaded" and
thumbnailed.
2022-01-24 08:58:18 -05:00
Andrew Morgan
c3040dd5cc Synapse 1.51.0rc2 (2022-01-24)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmHum5QTHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9G6vD/9Dw6V0687vzahnU6aYWaecRSO1sbww
 EtcCiXOh0r8HXPAwEIXJYomSTTbPl0eAwP8T1the1WZVQArvRW2VvMzDoBheO8bt
 dCm4CTKFCGUsI/GrlFDPtjAEd6kruAETmpHZn7bAFSqtIpissD6FjUwg/ND/NJfM
 zVM/bMgM0+js6eD9J5/k16E1ZWj7Lbp+/fKN+qTeQrXzIeT13WQZ4Nz1o/cqe/21
 o/coI3FmYq8CzKpfA6qZMDd/OtYpYqwwr7otSmW+6qHeZI/yqeoxlpzfgSZGpRUe
 mtXqbQllZQeqrbm8oK96GNmKcThy6awTwJPoD46b1AOkmFdf/lGSqO0lnTQRRPqR
 hyPPdrx6Lt2t7DDbVVyUElzkqLPhVJtrItPDC8669sWvmSgsPQdUqIsRmhF+8aJe
 ffjRvKbGRICnNZkT+qGf1HwuMHajZyAIHAS/kyFDKUZCvau3VQ1wlyOqVQ1hWr7+
 3k3CobjSx4y1bYKXncAK6hWH6lE8M319jaTnVfYXocDLWRonyFf7o286Q+c8WBGF
 tY0FzvUPb5S2kktC4WLwcqtcTWK1cu5MI6GfD69EqL7iifJRVyKDeUoD7tiUvgzN
 O2wBl2soJIsAU+8y16WQu7p+k2nZIomCCAySnW3C8mlxvv7CKQu0Url8J7IH8uZE
 ZVN2MMe7H+ZHJQ==
 =Fpsa
 -----END PGP SIGNATURE-----

Merge tag 'v1.51.0rc2' into develop

Synapse 1.51.0rc2 (2022-01-24)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
2022-01-24 13:55:03 +00:00
reivilibre
df54c8485a
Remove account data (including client config, push rules and ignored users) upon user deactivation. (#11621)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-01-24 13:37:00 +00:00
Andrew Morgan
dc671d3ea7
Fix logic for dropping old events in fed queue (#11806)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
Co-authored-by: Richard van der Hoff <richard@matrix.org>
2022-01-24 12:20:01 +00:00
Richard van der Hoff
2aa37a4250
Add state_key and rejection_reason to events (#11792)
... and start populating them for new events
2022-01-21 12:21:28 +00:00
Patrick Cloke
7a11509d17 Do not try to serialize raw aggregations dict. (#11791) 2022-01-21 10:40:34 +00:00
Patrick Cloke
b784299cbc
Do not try to serialize raw aggregations dict. (#11791) 2022-01-21 10:31:31 +00:00
Richard van der Hoff
9f2016e96e
Drop unused table public_room_list_stream. (#11795)
This is a follow-up to #10565.
2022-01-21 09:19:56 +00:00
reivilibre
4c2096599c
Make the get_global_account_data_by_type_for_user cache be a tree-cache whose key is prefixed with the user ID (#11788) 2022-01-21 08:38:36 +00:00
Brendan Abolivier
bfe6d5553a
Correctly await on_logged_out callbacks (#11786) 2022-01-20 19:19:40 +01:00
Andrew Morgan
121b9e2475
Add a regression test for using both webclient and client resources simultaneously (#11765) 2022-01-20 09:47:29 -05:00
Sean Quah
af13a3be29
Fix a bug that corrupted the cache of federated space hierarchies (#11775)
`FederationClient.get_room_hierarchy()` caches its return values, so
refactor the code to avoid modifying the returned room summary.
2022-01-20 11:03:42 +00:00
Patrick Cloke
68acb0a29d
Include whether the requesting user has participated in a thread. (#11577)
Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
2022-01-18 11:38:57 -05:00
Daniel Sonck
6b241f5286
Make pagination of rooms in admin api stable (#11737)
Always add state.room_id after the configurable ORDER BY. Otherwise,
for any sort, certain pages can contain results from
other pages. (Especially when sorting by creator, since there may
be many rooms by the same creator)

* Document different order direction of numerical fields

"joined_members", "joined_local_members", "version" and "state_events"
are ordered in descending direction by default (dir=f). Added a note
in tests to explain the differences in ordering.

Signed-off-by: Daniël Sonck <daniel@sonck.nl>
2022-01-17 11:42:51 +00:00
Olivier Wilkinson (reivilibre)
e7da1ced24 Merge branch 'release-v1.50' into develop 2022-01-14 15:25:16 +00:00
Andrew Morgan
18862f20b5
Remove the 'password_hash' from the Users Admin API endpoint response dictionary (#11576) 2022-01-14 14:53:33 +00:00
Patrick Cloke
3e0536cd2a
Replace uses of simple_insert_many with simple_insert_many_values. (#11742)
This should be (slightly) more efficient and it is simpler
to have a single method for inserting multiple values.
2022-01-13 19:44:18 -05:00
reivilibre
b602ba194b
Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could fail because too many EDUs were produced for device updates. (#11730)
Co-authored-by: David Robertson <davidr@element.io>
2022-01-13 18:12:18 +00:00
Patrick Cloke
0c40c619aa
Include bundled aggregations in the sync response cache. (#11659) 2022-01-13 10:45:28 -05:00
Jason Robinson
2560b1b6b2
Allow tracking puppeted users for MAU (#11561)
Currently when puppeting another user, the user doing the puppeting is
tracked for client IPs and MAU (if configured).

When tracking MAU is important, it becomes necessary to be possible to
also track the client IPs and MAU of puppeted users. As an example a
client that manages user creation and creation of tokens via the Synapse
admin API, passing those tokens for the client to use.

This PR adds optional configuration to enable tracking of puppeted users
into monthly active users. The default behaviour stays the same.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2022-01-12 16:09:36 +00:00
reivilibre
22abfca8d9
Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not be sent to remote homeservers if there were too many to send at once. (#11729)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-01-12 15:21:13 +00:00
David Robertson
1b1aed38e3 Deal with mypy errors w/ type-hinted pynacl 1.5.0 (#11714)
* Deal with mypy errors w/ type-hinted pynacl 1.5.0

Fixes #11644.

I really don't like that we're monkey patching pynacl SignedKey
instances with alg and version objects. But I'm too scared to make the
changes necessary right now.

(Ideally I would replace `signedjson.types.SingingKey` with a runtime class which
wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16
2022-01-12 14:54:06 +00:00
David Robertson
c43dd4d01b
Deal with mypy errors w/ type-hinted pynacl 1.5.0 (#11714)
* Deal with mypy errors w/ type-hinted pynacl 1.5.0

Fixes #11644.

I really don't like that we're monkey patching pynacl SignedKey
instances with alg and version objects. But I'm too scared to make the
changes necessary right now.

(Ideally I would replace `signedjson.types.SingingKey` with a runtime class which
wraps or inherits from `nacl.signing.SigningKey`.) C.f. https://github.com/matrix-org/python-signedjson/issues/16
2022-01-10 13:40:46 +00:00
Patrick Cloke
8e57584a58
Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)
By returning all of the m.space.child state of the space, not just
the first 50. The number of rooms returned is still capped at 50.

For the federation API this implies that the requesting server will
need to individually query for any other rooms it is not joined to.
2022-01-07 19:27:58 -05:00
David Robertson
d3cf0730f8
Optionally use an on-disk sqlite db in tests (#11702)
* Optionally use an on-disk sqlite db in tests

When debugging a test it is sometimes useful to inspect the state of the
DB. This is not easy when the db is in-memory: one cannot attach the
sqlite CLI to another process's DB.

With this change, if SYNAPSE_TEST_PERSIST_SQLITE_DB is set, we use
 `_trial_temp/test.db` as our sqlite database. One can then use 
`sqlite3 _trial_temp/test.db` and query to your heart's content.

The DB is destroyed and recreated between different test cases.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-01-07 19:13:41 +00:00
David Robertson
2bb4bd1269
Test that bans win a join against a race when computing /sync response (#11701) 2022-01-07 16:43:21 +00:00
Patrick Cloke
6bf81a7a61
Bundle aggregations outside of the serialization method. (#11612)
This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.

Overall this should cause no change in behavior, but is prep work
for other improvements.
2022-01-07 09:10:46 -05:00
Patrick Cloke
6c68e874b1
Remove the /send_relation endpoint. (#11682)
This was removed from MSC2674 before that was approved
and is not used by any known clients.
2022-01-06 19:00:34 +00:00
Shay
70ce9aea71
Strip unauthorized fields from unsigned object in events received over federation (#11530)
* add some tests to verify we are stripping unauthorized fields out of unsigned

* add function to strip unauthorized fields from the unsigned object of event

* newsfragment

* update newsfragment number

* add check to on_send_membership_event

* refactor tests

* fix lint error

* slightly refactor tests and add some comments

* slight refactor

* refactor tests

* fix import error

* slight refactor

* remove unsigned filtration code from synapse/handlers/federation_event.py

* lint

* move unsigned filtering code to event base

* refactor tests

* update newsfragment

* requested changes

* remove unused retun values
2022-01-06 09:09:30 -08:00
Dirk Klimpel
3b51c763ba
Fix get federation status of destination if no error occured (#11593) 2022-01-05 14:46:50 -05:00
David Robertson
88a78c6577
Cache empty responses from /user/devices (#11587)
If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
2022-01-05 13:33:28 +00:00
Richard van der Hoff
0fb3dd0830
Refactor the way we set outlier (#11634)
* `_auth_and_persist_outliers`: mark persisted events as outliers

Mark any events that get persisted via `_auth_and_persist_outliers` as, well,
outliers.

Currently this will be a no-op as everything will already be flagged as an
outlier, but I'm going to change that.

* `process_remote_join`: stop flagging as outlier

The events are now flagged as outliers later on, by `_auth_and_persist_outliers`.

* `send_join`: remove `outlier=True`

The events created here are returned in the result of `send_join` to
`FederationHandler.do_invite_join`. From there they are passed into
`FederationEventHandler.process_remote_join`, which passes them to
`_auth_and_persist_outliers`... which sets the `outlier` flag.

* `get_event_auth`: remove `outlier=True`

stop flagging the events returned by `get_event_auth` as outliers. This method
is only called by `_get_remote_auth_chain_for_event`, which passes the results
into `_auth_and_persist_outliers`, which will flag them as outliers.

* `_get_remote_auth_chain_for_event`: remove `outlier=True`

we pass all the events into `_auth_and_persist_outliers`, which will now flag
the events as outliers.

* `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter

This param is now never set to True, so we can remove it.

* `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param

This is no longer set anywhere, so we can remove it.

* `get_pdu`: remove unused `outlier` parameter

... and chase it down into `get_pdu_from_destination_raw`.

* `event_from_pdu_json`: remove redundant `outlier` param

This is never set to `True`, so can be removed.

* changelog

* update docstring
2022-01-05 12:26:11 +00:00
Dirk Klimpel
7a1cefc6e3
Add admin API to get users' account data (#11664)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-05 11:49:06 +00:00
reivilibre
84bfe47b01
Re-apply: Move glob_to_regex and re_word_boundary to matrix-python-common #11505 (#11687)
Co-authored-by: Sean Quah <seanq@element.io>
2022-01-05 11:41:49 +00:00
Patrick Cloke
8422a7f7f6
Include the topic event in the prejoin state, per MSC3173. (#11666)
Invites and knocks will now include the topic in the stripped state
send to clients before joining the room.
2022-01-04 11:08:08 -05:00
Patrick Cloke
cbd82d0b2d
Convert all namedtuples to attrs. (#11665)
To improve type hints throughout the code.
2021-12-30 18:47:12 +00:00
reivilibre
2c7f5e74e5
Fix a type annotation in test_account_data.py and remove it from the Mypy exclusion list. (#11657)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-12-29 15:12:30 +00:00
reivilibre
e6897e7383
Refactor tests.util.setup_test_homeserver and tests.server.setup_test_homeserver. (#11503) 2021-12-21 16:12:05 +00:00