Commit Graph

854 Commits

Author SHA1 Message Date
Brendan Abolivier
0171fa5226
Remove deprecated user_may_create_room_with_invites callback (#11950)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-02-11 13:58:11 +00:00
Erik Johnston
79fb64e417
Fix to-device being dropped in limited sync in SQLite. (#11966)
If ther are more than 100 to-device messages pending for a device
`/sync` will only return the first 100, however the next batch token was
incorrectly calculated and so all other pending messages would be
dropped.

This is due to `txn.rowcount` only returning the number of rows that
*changed*, rather than the number *selected* in SQLite.
2022-02-11 13:38:05 +00:00
Richard van der Hoff
c3db7a0b59
Tests: replace mocked Authenticator with the real thing (#11913)
If we prepopulate the test homeserver with a key for a remote homeserver, we
can make federation requests to it without having to stub out the
authenticator. This has two advantages:

 * means that what we are testing is closer to reality (ie, we now have
   complete tests for the incoming-request-authorisation flow)

 * some tests require that other objects be signed by the remote server (eg,
   the event in `/send_join`), and doing that would require a whole separate
   set of mocking out. It's much simpler just to use real keys.
2022-02-11 12:06:02 +00:00
Dirk Klimpel
d36943c4df
Support the stable API endpoint for MSC3283: new settings in /capabilities endpoint (#11933) 2022-02-11 09:32:11 +00:00
Patrick Cloke
df36945ff0
Support pagination tokens from /sync and /messages in the relations API. (#11952) 2022-02-10 10:52:48 -05:00
Denis Kasak
337f38cac3
Implement a content type allow list for URL previews (#11936)
This implements an allow list for content types for which Synapse will attempt URL preview. If a URL resolves to a resource with a content type which isn't in the list, the download will terminate immediately.

This makes sense given that Synapse would never successfully generate a URL preview for such files in the first place, and helps prevent issues with streaming media servers, such as #8302.

Signed-off-by: Denis Kasak dkasak@termina.org.uk
2022-02-10 15:43:01 +00:00
Patrick Cloke
8c94b3abe9
Experimental support to include bundled aggregations in search results (MSC3666) (#11837) 2022-02-08 09:21:20 -05:00
Andrew Morgan
0c4878caf2
Add a unit test for users receiving their own device list updates (#11909) 2022-02-07 13:21:19 +00:00
Jonathan de Jong
02632b3504
Stabilise MSC3231 (Token Based Registration) (#11867) 2022-02-04 12:15:13 +00:00
reivilibre
833247553f
Allow specifying the application service-specific user_id parameter in the join test helper. (#11616) 2022-02-03 13:09:22 +00:00
reivilibre
513913cc6b
Expose the registered device ID from the register_appservice_user test helper. (#11615) 2022-02-02 09:59:55 +00:00
Dirk Klimpel
901b264c0c
Add type hints to tests/rest/admin (#11851) 2022-01-31 14:20:05 -05:00
Brendan Abolivier
bf60da1a60
Configurable limits on avatars (#11846)
Only allow files which file size and content types match configured
limits to be set as avatar.

Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
2022-01-28 15:41:33 +01:00
Patrick Cloke
2897fb6b4f
Improvements to bundling aggregations. (#11815)
This is some odds and ends found during the review of #11791
and while continuing to work in this code:

* Return attrs classes instead of dictionaries from some methods
  to improve type safety.
* Call `get_bundled_aggregations` fewer times.
* Adds a missing assertion in the tests.
* Do not return empty bundled aggregations for an event (preferring
  to not include the bundle at all, as the docstring states).
2022-01-26 08:27:04 -05:00
Brendan Abolivier
95b3f952fa
Add a config flag to inhibit M_USER_IN_USE during registration (#11743)
This is mostly motivated by the tchap use case, where usernames are automatically generated from the user's email address (in a way that allows figuring out the email address from the username). Therefore, it's an issue if we respond to requests on /register and /register/available with M_USER_IN_USE, because it can potentially leak email addresses (which include the user's real name and place of work).

This commit adds a flag to inhibit the M_USER_IN_USE errors that are raised both by /register/available, and when providing a username early into the registration process. This error will still be raised if the user completes the registration process but the username conflicts. This is particularly useful when using modules (https://github.com/matrix-org/synapse/pull/11790 adds a module callback to set the username of users at registration) or SSO, since they can ensure the username is unique.

More context is available in the PR that introduced this behaviour to synapse-dinsic: matrix-org/synapse-dinsic#48 - as well as the issue in the matrix-dinsic repo: matrix-org/matrix-dinsic#476
2022-01-26 13:02:54 +01:00
Dirk Klimpel
6a72c910f1
Add admin API to get a list of federated rooms (#11658) 2022-01-25 16:11:40 +00:00
Dirk Klimpel
0d6cfea9b8
Add admin API to reset connection timeouts for remote server (#11639)
* Fix get federation status of destination if no error occured
2022-01-25 12:06:29 +00:00
Patrick Cloke
807efd26ae
Support rendering previews with data: URLs in them (#11767)
Images which are data URLs will no longer break URL
previews and will properly be "downloaded" and
thumbnailed.
2022-01-24 08:58:18 -05:00
Patrick Cloke
b784299cbc
Do not try to serialize raw aggregations dict. (#11791) 2022-01-21 10:31:31 +00:00
Richard van der Hoff
9f2016e96e
Drop unused table public_room_list_stream. (#11795)
This is a follow-up to #10565.
2022-01-21 09:19:56 +00:00
Patrick Cloke
68acb0a29d
Include whether the requesting user has participated in a thread. (#11577)
Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
2022-01-18 11:38:57 -05:00
Daniel Sonck
6b241f5286
Make pagination of rooms in admin api stable (#11737)
Always add state.room_id after the configurable ORDER BY. Otherwise,
for any sort, certain pages can contain results from
other pages. (Especially when sorting by creator, since there may
be many rooms by the same creator)

* Document different order direction of numerical fields

"joined_members", "joined_local_members", "version" and "state_events"
are ordered in descending direction by default (dir=f). Added a note
in tests to explain the differences in ordering.

Signed-off-by: Daniël Sonck <daniel@sonck.nl>
2022-01-17 11:42:51 +00:00
Andrew Morgan
18862f20b5
Remove the 'password_hash' from the Users Admin API endpoint response dictionary (#11576) 2022-01-14 14:53:33 +00:00
Patrick Cloke
3e0536cd2a
Replace uses of simple_insert_many with simple_insert_many_values. (#11742)
This should be (slightly) more efficient and it is simpler
to have a single method for inserting multiple values.
2022-01-13 19:44:18 -05:00
Patrick Cloke
0c40c619aa
Include bundled aggregations in the sync response cache. (#11659) 2022-01-13 10:45:28 -05:00
David Robertson
2bb4bd1269
Test that bans win a join against a race when computing /sync response (#11701) 2022-01-07 16:43:21 +00:00
Patrick Cloke
6bf81a7a61
Bundle aggregations outside of the serialization method. (#11612)
This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.

Overall this should cause no change in behavior, but is prep work
for other improvements.
2022-01-07 09:10:46 -05:00
Patrick Cloke
6c68e874b1
Remove the /send_relation endpoint. (#11682)
This was removed from MSC2674 before that was approved
and is not used by any known clients.
2022-01-06 19:00:34 +00:00
Dirk Klimpel
3b51c763ba
Fix get federation status of destination if no error occured (#11593) 2022-01-05 14:46:50 -05:00
Dirk Klimpel
7a1cefc6e3
Add admin API to get users' account data (#11664)
Co-authored-by: reivilibre <olivier@librepush.net>
2022-01-05 11:49:06 +00:00
Richard van der Hoff
8e4083e2f6 Merge remote-tracking branch 'origin/release-v1.49' into develop 2021-12-20 16:34:01 +00:00
Richard van der Hoff
76aa5537ad
Disable aggregation bundling on /sync responses (#11583)
* Disable aggregation bundling on `/sync` responses

A partial revert of #11478. This turns out to have had a significant CPU impact
on initial-sync handling. For now, let's disable it, until we find a more
efficient way of achieving this.

* Fix tests.

Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2021-12-20 16:33:35 +00:00
Dirk Klimpel
8428ef66c7
Add type hints to synapse/tests/rest/admin (#11590) 2021-12-16 14:59:56 -05:00
Patrick Cloke
eb39da6782
Move HTML parsing to a separate file for URL previews. (#11566)
* Splits the logic for parsing HTML from the resource handling code.
* Fix a circular import in the oEmbed code (which uses the HTML parsing code).
* Renames some of the HTML parsing methods to:
  * Make it clear which methods are "internal" to the module.
  * Clarify what the methods do.
2021-12-13 17:55:07 +00:00
Eric Eastwood
8391bd6ab5
Test to ensure we share the same state_group across the whole historical batch (MSC2716) (#11487)
Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

We did some work on making sure the `state_groups` were shared in
https://github.com/matrix-org/synapse/pull/10975
2021-12-10 20:59:20 -06:00
Patrick Cloke
9562f0c2f1
Ensure emails are canonicalized before fetching associated user. (#11547)
This should fix pushers with an email in non-canonical form is used as
the pushkey.
2021-12-10 07:17:28 -05:00
Patrick Cloke
3b8872299a
Do not allow cross-room relations, per MSC2674. (#11516) 2021-12-09 13:16:01 -05:00
Dirk Klimpel
b3bcacf3c1
Add missing errcode to parse_string and parse_boolean (#11542) 2021-12-09 11:23:34 +00:00
Dirk Klimpel
7ecaa3b976
Clean up synapse.rest.admin (#11535) 2021-12-08 16:59:40 +00:00
reivilibre
365e9482fe
Use HTTPStatus constants in place of literals in tests.rest.client.test_auth. (#11520) 2021-12-08 14:54:47 +00:00
Richard van der Hoff
b1ecd19c5d
Fix 'delete room' admin api to work on incomplete rooms (#11523)
If, for some reason, we don't have the create event, we should still be able to
purge a room.
2021-12-07 11:37:54 +00:00
reivilibre
2d42e586a8
Fix the test breakage introduced by #11435 as a result of concurrent PRs (#11522) 2021-12-07 10:49:39 +00:00
reivilibre
2f053f3f82
Stabilise support for MSC2918 refresh tokens as they have now been merged into the Matrix specification. (#11435) 2021-12-06 19:11:43 +00:00
Dirk Klimpel
8b4b153c9e
Add admin API to get some information about federation status (#11407) 2021-12-06 16:59:50 +00:00
Patrick Cloke
494ebd7347
Include bundled aggregations in /sync and related fixes (#11478)
Due to updates to MSC2675 this includes a few fixes:

* Include bundled aggregations for /sync.
* Do not include bundled aggregations for /initialSync and /events.
* Do not bundle aggregations for state events.
* Clarifies comments and variable names.
2021-12-06 15:51:15 +00:00
reivilibre
637df95de6
Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445) 2021-12-03 16:42:44 +00:00
Dirk Klimpel
e5f426cd54
Add type hints to synapse/tests/rest/admin (#11501) 2021-12-03 13:57:13 +00:00
Sean Quah
858d80bf0f
Fix media repository failing when media store path contains symlinks (#11446) 2021-12-02 16:05:24 +00:00
Patrick Cloke
379f2650cf
Bundle relations of relations into the /relations result. (#11284)
Per updates to MSC2675 which now states that bundled
aggregations should be included from the `/relations`
endpoint.
2021-11-30 11:33:33 -05:00
Dirk Klimpel
432a174bc1
Remove unnecessary json.dumps from tests.rest.admin (#11461)
The tests helpers automatically convert dictionaries to
JSON payloads, no need to do it manually for each
test.
2021-11-30 14:51:04 +00:00
Dirk Klimpel
35b1900f00
Convert status codes to HTTPStatus in tests.rest.admin (#11455) 2021-11-30 09:53:54 +00:00
Erik Johnston
d08ef6f155
Make background updates controllable via a plugin (#11306)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-11-29 17:57:06 +01:00
reivilibre
1b6691dce4
Update MSC2918 refresh token support to confirm with the latest revision: accept the refresh_tokens parameter in the request body rather than in the URL parameters. (#11430) 2021-11-26 19:06:16 +00:00
reivilibre
1d8b80b334
Support expiry of refresh tokens and expiry of the overall session when refresh tokens are in use. (#11425) 2021-11-26 14:27:14 +00:00
Brendan Abolivier
7f9841bdec
Lower minumum batch size to 1 for background updates (#11422)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-24 19:21:44 +00:00
reivilibre
f25c75d376
Rename unstable access_token_lifetime configuration option to refreshable_access_token_lifetime to make it clear it only concerns refreshable access tokens. (#11388) 2021-11-23 17:01:34 +00:00
Sean Quah
454c3d7694 Merge branch 'master' into develop 2021-11-23 13:06:56 +00:00
Kostas
1035663833
Add config for customizing the claim used for JWT logins. (#11361)
Allows specifying a different claim (from the default "sub") to use
when calculating the localpart of the Matrix ID used during the
JWT login.
2021-11-22 13:01:03 -05:00
Patrick Cloke
3d893b8cf2
Store arbitrary relations from events. (#11391)
Instead of only known relation types. This also reworks the background
update for thread relations to crawl events and search for any relation
type, not just threaded relations.
2021-11-22 12:01:47 -05:00
Dirk Klimpel
ea20937084
Add an admin API to run background jobs. (#11352)
Instead of having admins poke into the database directly.

Can currently run jobs to populate stats and to populate
the user directory.
2021-11-19 19:39:46 +00:00
Sean Quah
91f2bd0907 Prevent the media store from writing outside of the configured directory
Also tighten validation of server names by forbidding invalid characters
in IPv6 addresses and empty domain labels.
2021-11-19 13:39:15 +00:00
Dirk Klimpel
81b18fe5c0
Add dedicated admin API for blocking a room (#11324) 2021-11-18 17:43:49 +00:00
Patrick Cloke
4bd54b263e
Do not allow MSC3440 threads to fork threads (#11161)
Adds validation to the Client-Server API to ensure that
the potential thread head does not relate to another event
already. This results in not allowing a thread to "fork" into
other threads.

If the target event is unknown for some reason (maybe it isn't
visible to your homeserver), but is the target of other events
it is assumed that the thread can be created from it. Otherwise,
it is rejected as an unknown event.
2021-11-18 13:43:09 +00:00
reivilibre
0d86f6334a
Rename get_access_token_for_user_id method to create_access_token_for_user_id (#11369) 2021-11-17 14:10:57 +00:00
Patrick Cloke
24b61f379a
Add ability to un-shadow-ban via the admin API. (#11347) 2021-11-16 12:43:53 +00:00
David Robertson
0dda1a7968
Misc typing fixes for tests, part 2 of N (#11330) 2021-11-16 10:41:35 +00:00
Tulir Asokan
6f862c5c28
Add support for the stable version of MSC2778 (#11335)
* Add support for the stable version of MSC2778

Signed-off-by: Tulir Asokan <tulir@maunium.net>

* Expect m.login.application_service in login and password provider tests

Signed-off-by: Tulir Asokan <tulir@maunium.net>
2021-11-15 10:31:22 +00:00
David Robertson
bea815cec8
Test room alias deletion (#11327)
* Prefer `HTTPStatus` over plain `int`

This is an Opinion that no-one has seemed to object to yet.

* `--disallow-untyped-defs` for `tests.rest.client.test_directory`
* Improve synapse's annotations for deleting aliases
* Test case for deleting a room alias
* Changelog
2021-11-12 19:56:00 +00:00
David Robertson
4c96ce396e
Misc typing fixes for tests, part 1 of N (#11323)
* Annotate HomeserverTestCase.servlets
* Correct annotation of federation_auth_origin
* Use AnyStr custom_headers instead of a Union

This allows (str, str) and (bytes, bytes).
This disallows (str, bytes) and (bytes, str)

* DomainSpecificString.SIGIL is a ClassVar
2021-11-12 15:50:54 +00:00
Dirk Klimpel
8840a7b7f1
Convert delete room admin API to async endpoint (#11223)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-11-12 12:35:31 +00:00
David Robertson
b6f4d122ef
Allow admins to proactively block rooms (#11228)
Co-authored-by: Dirk Klimpel <5740567+dklimpel@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-11-09 13:11:47 +00:00
Patrick Cloke
a19d01c3d9
Support filtering by relations per MSC3440 (#11236)
Adds experimental support for `relation_types` and `relation_senders`
fields for filters.
2021-11-09 08:10:58 -05:00
rogersheu
820337e6a4
Require body for read receipts with user-agent exceptions (#11157)
Co-authored-by: reivilibre <olivier@librepush.net>
2021-11-09 10:26:07 +00:00
Erik Johnston
4ee71b9637
Add some background update admin APIs (#11263)
Fixes #11259
2021-11-08 16:08:02 +00:00
Richard van der Hoff
86a497efaa
Default value for public_baseurl (#11210)
We might as well use a default value for `public_baseurl` based on
`server_name` - in many cases, it will be correct.
2021-11-08 14:13:10 +00:00
Dirk Klimpel
7537201840
Add search by room ID and room alias to List Room admin API (#11099)
Fixes: #10874
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-11-02 10:01:13 +00:00
Shay
f5c6a80886
Handle missing Content-Type header when accessing remote media (#11200)
* add code to handle missing content-type header and a test to verify that it works

* add handling for missing content-type in the /upload endpoint as well

* slightly refactor test code to put private method in approriate place

* handle possible null value for content-type when pulling from the local db

* add changelog

* refactor test and add code to handle missing content-type in cached remote media

* requested changes

* Update changelog.d/11200.bugfix

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-11-01 10:26:02 -07:00
Dirk Klimpel
caa706d825
Fix a bug in unit test test_block_room_and_not_purge (#11226) 2021-11-01 16:10:09 +00:00
reivilibre
69ab3dddbc
Make check_event_allowed module API callback not fail open (accept events) when an exception is raised (#11033) 2021-11-01 15:45:56 +00:00
Dirk Klimpel
66bdca3e31
Remove deprecated delete room admin API (#11213)
Remove deprecated delete room admin API,
`POST /_synapse/admin/v1/rooms/<room_id>/delete`
2021-11-01 15:11:24 +00:00
Richard van der Hoff
71f9966f27
Support for serving server well-known files (#11211)
Fixes https://github.com/matrix-org/synapse/issues/8308
2021-11-01 15:10:16 +00:00
Rafael Gonçalves
0e16b418f6
Add knock information in admin exported data (#11171)
Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
2021-10-28 18:54:38 +01:00
Brendan Abolivier
c7a5e49664
Implement an on_new_event callback (#11126)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-10-26 15:17:36 +02:00
Jason Robinson
63cbdd8af0
Enable changing user type via users admin API (#11174)
Users admin API can now also modify user
type in addition to allowing it to be
set on user creation.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-10-26 09:01:06 +00:00
Patrick Cloke
ba00e20234
Add a thread relation type per MSC3440. (#11088)
Adds experimental support for MSC3440's `io.element.thread` relation
type (and the aggregation for it).
2021-10-21 14:39:16 -04:00
Dirk Klimpel
ef7fe09778
Fix setting a user's external_id via the admin API returns 500 and deletes users existing external mappings if that external ID is already mapped (#11051)
Fixes #10846
2021-10-21 09:52:32 +01:00
Aaron R
2c61a318cc
Show error when timestamp in seconds is provided to the /purge_media_cache API (#11101) 2021-10-20 14:41:48 +00:00
Patrick Cloke
1609ccf8fe
Fix-up some type hints in the relations tests. (#11076) 2021-10-14 09:19:35 -04:00
reivilibre
50d8601581
Add a test for a workaround concerning the behaviour of third-party rule modules and SynapseErrors. (#11071) 2021-10-14 14:14:15 +01:00
Patrick Cloke
732bbf6737
Be more lenient when parsing the version for oEmbed responses. (#11065) 2021-10-13 07:00:07 -04:00
Dirk Klimpel
988de0afb0
Simplify the user admin API tests (#11048) 2021-10-12 20:38:48 +01:00
Sean Quah
84f5d83257
Add tests for MediaFilePaths (#11057) 2021-10-12 18:19:35 +01:00
Patrick Cloke
1b112840d2
Autodiscover oEmbed endpoint from returned HTML (#10822)
Searches the returned HTML for an oEmbed endpoint using the
autodiscovery mechanism (`<link rel=...>`), and will request it
to generate the preview.
2021-10-08 14:14:42 -04:00
Brendan Abolivier
4e51621064
Add a spamchecker method to allow or deny 3pid invites (#10894)
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.

Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
2021-10-06 17:18:13 +02:00
Brendan Abolivier
829f2a82b0
Add a spamchecker callback to allow or deny room joins (#10910)
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-10-06 14:32:16 +00:00
AndrewFerr
30f0240401
Make is_public Optional[bool] for create_room_as test util (#10951) (#10963)
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-10-04 14:43:03 +00:00
David Robertson
f7b034a24b
Consistently exclude from user_directory (#10960)
* Introduce `should_include_local_users_in_dir`

We exclude three kinds of local users from the user_directory tables. At
present we don't consistently exclude all three in the same places. This
commit introduces a new function to gather those exclusion conditions
together. Because we have to handle local and remote users in different
ways, I've made that function only consider the case of remote users.
It's the caller's responsibility to make the local versus remote
distinction clear and correct.

A test fixup is required. The test now hits a path which makes db
queries against the users table. The expected rows were missing, because
we were using a dummy user that hadn't actually been registered.

We also add new test cases to covert the exclusion logic.

----

By my reading this makes these changes:

* When an app service user registers or changes their profile, they will
  _not_ be added to the user directory. (Previously only support and
  deactivated users were excluded). This is consistent with the logic that
  rebuilds the user directory. See also [the discussion
  here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548).
* When rebuilding the directory, exclude support and disabled users from
  room sharing tables. Previously only appservice users were excluded.
* Exclude all three categories of local users when rebuilding the
  directory. Previously `_populate_user_directory_process_users` didn't do
  any exclusion.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-04 11:45:51 +00:00
Patrick Cloke
a0f48ee89d
Use direct references for configuration variables (part 7). (#10959) 2021-10-04 07:18:54 -04:00
David Robertson
e46ac85d67
type-hint HomeserverTestcase.setup_test_homeserver (#10961)
* type-hint `HomeserverTestcase.setup_test_homeserver`

For better IDE completion. A small drive-by.
2021-10-01 12:22:47 +01:00
Patrick Cloke
94b620a5ed
Use direct references for configuration variables (part 6). (#10916) 2021-09-29 06:44:15 -04:00