Commit Graph

619 Commits

Author SHA1 Message Date
Patrick Cloke
b784299cbc
Do not try to serialize raw aggregations dict. (#11791) 2022-01-21 10:31:31 +00:00
Patrick Cloke
68acb0a29d
Include whether the requesting user has participated in a thread. (#11577)
Per updates to MSC3440.

This is implement as a separate method since it needs to be cached
on a per-user basis, instead of a per-thread basis.
2022-01-18 11:38:57 -05:00
Patrick Cloke
0c40c619aa
Include bundled aggregations in the sync response cache. (#11659) 2022-01-13 10:45:28 -05:00
David Robertson
2bb4bd1269
Test that bans win a join against a race when computing /sync response (#11701) 2022-01-07 16:43:21 +00:00
Patrick Cloke
6bf81a7a61
Bundle aggregations outside of the serialization method. (#11612)
This makes the serialization of events synchronous (and it no
longer access the database), but we must manually calculate and
provide the bundled aggregations.

Overall this should cause no change in behavior, but is prep work
for other improvements.
2022-01-07 09:10:46 -05:00
Patrick Cloke
6c68e874b1
Remove the /send_relation endpoint. (#11682)
This was removed from MSC2674 before that was approved
and is not used by any known clients.
2022-01-06 19:00:34 +00:00
Richard van der Hoff
8e4083e2f6 Merge remote-tracking branch 'origin/release-v1.49' into develop 2021-12-20 16:34:01 +00:00
Richard van der Hoff
76aa5537ad
Disable aggregation bundling on /sync responses (#11583)
* Disable aggregation bundling on `/sync` responses

A partial revert of #11478. This turns out to have had a significant CPU impact
on initial-sync handling. For now, let's disable it, until we find a more
efficient way of achieving this.

* Fix tests.

Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2021-12-20 16:33:35 +00:00
Eric Eastwood
8391bd6ab5
Test to ensure we share the same state_group across the whole historical batch (MSC2716) (#11487)
Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

We did some work on making sure the `state_groups` were shared in
https://github.com/matrix-org/synapse/pull/10975
2021-12-10 20:59:20 -06:00
Patrick Cloke
3b8872299a
Do not allow cross-room relations, per MSC2674. (#11516) 2021-12-09 13:16:01 -05:00
reivilibre
365e9482fe
Use HTTPStatus constants in place of literals in tests.rest.client.test_auth. (#11520) 2021-12-08 14:54:47 +00:00
reivilibre
2d42e586a8
Fix the test breakage introduced by #11435 as a result of concurrent PRs (#11522) 2021-12-07 10:49:39 +00:00
reivilibre
2f053f3f82
Stabilise support for MSC2918 refresh tokens as they have now been merged into the Matrix specification. (#11435) 2021-12-06 19:11:43 +00:00
Patrick Cloke
494ebd7347
Include bundled aggregations in /sync and related fixes (#11478)
Due to updates to MSC2675 this includes a few fixes:

* Include bundled aggregations for /sync.
* Do not include bundled aggregations for /initialSync and /events.
* Do not bundle aggregations for state events.
* Clarifies comments and variable names.
2021-12-06 15:51:15 +00:00
reivilibre
637df95de6
Support configuring the lifetime of non-refreshable access tokens separately to refreshable access tokens. (#11445) 2021-12-03 16:42:44 +00:00
Patrick Cloke
379f2650cf
Bundle relations of relations into the /relations result. (#11284)
Per updates to MSC2675 which now states that bundled
aggregations should be included from the `/relations`
endpoint.
2021-11-30 11:33:33 -05:00
reivilibre
1b6691dce4
Update MSC2918 refresh token support to confirm with the latest revision: accept the refresh_tokens parameter in the request body rather than in the URL parameters. (#11430) 2021-11-26 19:06:16 +00:00
reivilibre
1d8b80b334
Support expiry of refresh tokens and expiry of the overall session when refresh tokens are in use. (#11425) 2021-11-26 14:27:14 +00:00
reivilibre
f25c75d376
Rename unstable access_token_lifetime configuration option to refreshable_access_token_lifetime to make it clear it only concerns refreshable access tokens. (#11388) 2021-11-23 17:01:34 +00:00
Kostas
1035663833
Add config for customizing the claim used for JWT logins. (#11361)
Allows specifying a different claim (from the default "sub") to use
when calculating the localpart of the Matrix ID used during the
JWT login.
2021-11-22 13:01:03 -05:00
Patrick Cloke
3d893b8cf2
Store arbitrary relations from events. (#11391)
Instead of only known relation types. This also reworks the background
update for thread relations to crawl events and search for any relation
type, not just threaded relations.
2021-11-22 12:01:47 -05:00
Patrick Cloke
4bd54b263e
Do not allow MSC3440 threads to fork threads (#11161)
Adds validation to the Client-Server API to ensure that
the potential thread head does not relate to another event
already. This results in not allowing a thread to "fork" into
other threads.

If the target event is unknown for some reason (maybe it isn't
visible to your homeserver), but is the target of other events
it is assumed that the thread can be created from it. Otherwise,
it is rejected as an unknown event.
2021-11-18 13:43:09 +00:00
reivilibre
0d86f6334a
Rename get_access_token_for_user_id method to create_access_token_for_user_id (#11369) 2021-11-17 14:10:57 +00:00
David Robertson
0dda1a7968
Misc typing fixes for tests, part 2 of N (#11330) 2021-11-16 10:41:35 +00:00
Tulir Asokan
6f862c5c28
Add support for the stable version of MSC2778 (#11335)
* Add support for the stable version of MSC2778

Signed-off-by: Tulir Asokan <tulir@maunium.net>

* Expect m.login.application_service in login and password provider tests

Signed-off-by: Tulir Asokan <tulir@maunium.net>
2021-11-15 10:31:22 +00:00
David Robertson
bea815cec8
Test room alias deletion (#11327)
* Prefer `HTTPStatus` over plain `int`

This is an Opinion that no-one has seemed to object to yet.

* `--disallow-untyped-defs` for `tests.rest.client.test_directory`
* Improve synapse's annotations for deleting aliases
* Test case for deleting a room alias
* Changelog
2021-11-12 19:56:00 +00:00
David Robertson
4c96ce396e
Misc typing fixes for tests, part 1 of N (#11323)
* Annotate HomeserverTestCase.servlets
* Correct annotation of federation_auth_origin
* Use AnyStr custom_headers instead of a Union

This allows (str, str) and (bytes, bytes).
This disallows (str, bytes) and (bytes, str)

* DomainSpecificString.SIGIL is a ClassVar
2021-11-12 15:50:54 +00:00
Patrick Cloke
a19d01c3d9
Support filtering by relations per MSC3440 (#11236)
Adds experimental support for `relation_types` and `relation_senders`
fields for filters.
2021-11-09 08:10:58 -05:00
rogersheu
820337e6a4
Require body for read receipts with user-agent exceptions (#11157)
Co-authored-by: reivilibre <olivier@librepush.net>
2021-11-09 10:26:07 +00:00
Richard van der Hoff
86a497efaa
Default value for public_baseurl (#11210)
We might as well use a default value for `public_baseurl` based on
`server_name` - in many cases, it will be correct.
2021-11-08 14:13:10 +00:00
reivilibre
69ab3dddbc
Make check_event_allowed module API callback not fail open (accept events) when an exception is raised (#11033) 2021-11-01 15:45:56 +00:00
Rafael Gonçalves
0e16b418f6
Add knock information in admin exported data (#11171)
Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
2021-10-28 18:54:38 +01:00
Brendan Abolivier
c7a5e49664
Implement an on_new_event callback (#11126)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-10-26 15:17:36 +02:00
Patrick Cloke
ba00e20234
Add a thread relation type per MSC3440. (#11088)
Adds experimental support for MSC3440's `io.element.thread` relation
type (and the aggregation for it).
2021-10-21 14:39:16 -04:00
Patrick Cloke
1609ccf8fe
Fix-up some type hints in the relations tests. (#11076) 2021-10-14 09:19:35 -04:00
reivilibre
50d8601581
Add a test for a workaround concerning the behaviour of third-party rule modules and SynapseErrors. (#11071) 2021-10-14 14:14:15 +01:00
Brendan Abolivier
4e51621064
Add a spamchecker method to allow or deny 3pid invites (#10894)
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.

Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
2021-10-06 17:18:13 +02:00
Brendan Abolivier
829f2a82b0
Add a spamchecker callback to allow or deny room joins (#10910)
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-10-06 14:32:16 +00:00
AndrewFerr
30f0240401
Make is_public Optional[bool] for create_room_as test util (#10951) (#10963)
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-10-04 14:43:03 +00:00
David Robertson
f7b034a24b
Consistently exclude from user_directory (#10960)
* Introduce `should_include_local_users_in_dir`

We exclude three kinds of local users from the user_directory tables. At
present we don't consistently exclude all three in the same places. This
commit introduces a new function to gather those exclusion conditions
together. Because we have to handle local and remote users in different
ways, I've made that function only consider the case of remote users.
It's the caller's responsibility to make the local versus remote
distinction clear and correct.

A test fixup is required. The test now hits a path which makes db
queries against the users table. The expected rows were missing, because
we were using a dummy user that hadn't actually been registered.

We also add new test cases to covert the exclusion logic.

----

By my reading this makes these changes:

* When an app service user registers or changes their profile, they will
  _not_ be added to the user directory. (Previously only support and
  deactivated users were excluded). This is consistent with the logic that
  rebuilds the user directory. See also [the discussion
  here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548).
* When rebuilding the directory, exclude support and disabled users from
  room sharing tables. Previously only appservice users were excluded.
* Exclude all three categories of local users when rebuilding the
  directory. Previously `_populate_user_directory_process_users` didn't do
  any exclusion.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-10-04 11:45:51 +00:00
Patrick Cloke
a0f48ee89d
Use direct references for configuration variables (part 7). (#10959) 2021-10-04 07:18:54 -04:00
David Robertson
e46ac85d67
type-hint HomeserverTestcase.setup_test_homeserver (#10961)
* type-hint `HomeserverTestcase.setup_test_homeserver`

For better IDE completion. A small drive-by.
2021-10-01 12:22:47 +01:00
Patrick Cloke
94b620a5ed
Use direct references for configuration variables (part 6). (#10916) 2021-09-29 06:44:15 -04:00
Travis Ralston
8cef1ab2ac
Implement MSC3069: Guest support on whoami (#9655) 2021-09-29 11:32:45 +01:00
Brendan Abolivier
b10257e879
Add a spamchecker callback to allow or deny room creation based on invites (#10898)
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.

This adds a callback that's fairly similar to user_may_create_room except it also allows processing based on the invites sent at room creation.
2021-09-24 16:38:23 +02:00
Erik Johnston
50022cff96
Add reactor to SynapseRequest and fix up types. (#10868) 2021-09-24 11:01:25 +01:00
Patrick Cloke
47854c71e9
Use direct references for configuration variables (part 4). (#10893) 2021-09-23 12:03:01 -04:00
Patrick Cloke
e584534403
Use direct references for some configuration variables (part 3) (#10885)
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.

It also improves type hints since mypy can now resolve the
types of the configuration variables.
2021-09-23 07:13:34 -04:00
Patrick Cloke
4054dfa409
Add type hints for event streams. (#10856) 2021-09-21 13:34:26 -04:00
Brendan Abolivier
2843058a8b
Test that state events sent by modules correctly end up in the room's state (#10835)
Test for #10830

Ideally the test would also make sure the new state event comes down sync, but this is probably good enough.
2021-09-21 17:40:20 +02:00
David Robertson
60453315bd
Always add local users to the user directory (#10796)
It's a simplification, but one that'll help make the user directory logic easier
to follow with the other changes upcoming. It's not strictly required for those
changes, but this will help simplify the resulting logic that listens for
`m.room.member` events and generally make the logic easier to follow.

This means the config option `search_all_users` ends up controlling the
search query only, and not the data we store. The cost of doing so is an
extra row in the `user_directory` and `user_directory_search` tables for
each local user which

- belongs to no public rooms
- belongs to no private rooms of size ≥ 2

I think the cost of this will be marginal (since they'll already have entries
 in `users` and `profiles` anyway).

As a small upside, a homeserver whose directory was built with this
change can toggle `search_all_users` without having to rebuild their
directory.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-09-21 12:02:34 +00:00
Aaron Raimist
6a751ff5e0
Allow sending a membership event to unban a user (#10807)
* Allow membership event to unban user

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-09-21 10:23:34 +00:00
Patrick Cloke
63f28e4a0c
Handle room upgrades for spaces (#10774)
By copying the `room_type` field of the create event and
migrating any non-empty `m.space.child` events to the
new room that is created.
2021-09-10 07:30:05 -04:00
Sean
273b6861f2
Remove unstable MSC2858 API, including experimental.msc2858_enabled config option (#10693)
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-09 17:59:59 +01:00
Richard van der Hoff
5d9e7e0c71 Merge branch 'master' into develop 2021-08-31 14:09:48 +01:00
Richard van der Hoff
8c26f16c76
Fix up unit tests (#10723)
These were broken in an incorrect merge of GHSA-jj53-8fmw-f2w2 (cb35df9)
2021-08-31 12:56:22 +01:00
reivilibre
cb35df940a
Merge pull request from GHSA-jj53-8fmw-f2w2 2021-08-31 11:24:09 +01:00
reivilibre
52c7a51cfc
Merge pull request from GHSA-3x4c-pq33-4w3q
* Add some tests to characterise the problem

Some failing. Current states:

  RoomsMemberListTestCase
test_get_member_list ...
[OK]
test_get_member_list_mixed_memberships ...
[OK]
test_get_member_list_no_permission ...
[OK]
test_get_member_list_no_permission_former_member ...
[OK]
test_get_member_list_no_permission_former_member_with_at_token ...
[FAIL]
test_get_member_list_no_room ...
[OK]
test_get_member_list_no_permission_with_at_token ...
[FAIL]

* Correct the tests

* Check user is/was member before divulging room membership

* Pull out only the 1 membership event we want.

* Update tests/rest/client/v1/test_rooms.py

Co-authored-by: Erik Johnston <erik@matrix.org>

* Fixup tests (following apply review suggestion)

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-08-31 10:09:58 +01:00
Aaron Raimist
40f619eaa5
Validate new m.room.power_levels events (#10232)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-08-26 17:07:58 +01:00
Erik Johnston
cd22fb568a Merge branch 'master' into develop 2021-08-24 17:13:29 +01:00
Callum Brown
947dbbdfd1
Implement MSC3231: Token authenticated registration (#10142)
Signed-off-by: Callum Brown <callum@calcuode.com>

This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
2021-08-21 22:14:43 +01:00
David Robertson
ecd823d766
Flatten tests/rest/client/{v1,v2_alpha} too (#10667) 2021-08-20 17:50:44 +01:00
David Robertson
7862d704fd Follow-up: format changelog, add licence (#10593)
Merged before approval; these comments from @clokep on that PR.
2021-08-20 16:33:52 +01:00
David Robertson
ee3b2ac59a
Validate device_keys for C-S /keys/query requests (#10593)
* Validate device_keys for C-S /keys/query requests

Closes #10354

A small, not particularly critical fix. I'm interested in seeing if we
can find a more systematic approach though. #8445 is the place for any discussion.
2021-08-20 15:47:03 +01:00
Patrick Cloke
5cda75fede
Set room version 8 as preferred for restricted rooms. (#10571) 2021-08-20 07:17:50 -04:00
Dirk Klimpel
b5fef6054a
Support MSC3283: Expose enable_set_displayname in capabilities (#10452) 2021-08-19 09:40:40 +00:00
reivilibre
642a42edde
Flatten the synapse.rest.client package (#10600) 2021-08-17 11:57:58 +00:00
Patrick Cloke
fab352ac2c
Fix type hints in space summary tests. (#10575)
And ensure that the file is checked via mypy.
2021-08-11 10:43:40 -04:00
Erik Johnston
60f0534b6e
Fix exceptions in logs when failing to get remote room list (#10541) 2021-08-06 14:05:41 +01:00
Richard van der Hoff
74d7336686
Add a setting to disable TLS for sending email (#10546)
This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.
2021-08-06 10:13:34 +00:00
Brendan Abolivier
e8a3e81402
Don't fail on empty bodies when sending out read receipts (#10531)
Fixes a bug introduced in rc1 that would cause Synapse to 400 on read receipts requests with empty bodies.

Broken in #10413
2021-08-04 16:13:24 +02:00
Šimon Brandner
c3b037795a
Support for MSC2285 (hidden read receipts) (#10413)
Implementation of matrix-org/matrix-doc#2285
2021-07-28 10:05:11 +02:00
Travis Ralston
b3a757eb3b
Support MSC2033: Device ID on whoami (#9918)
* Fix no-access-token bug in deactivation tests
* Support MSC2033: Device ID on whoami
* Test for appservices too

MSC: https://github.com/matrix-org/matrix-doc/pull/2033

The MSC has passed FCP, which means stable endpoints can be used.
2021-07-27 05:28:20 +00:00
Michael Telatynski
69226c1ab4
MSC3244 room capabilities implementation (#10283) 2021-07-20 12:59:23 +01:00
Brendan Abolivier
a743bf4694
Port the ThirdPartyEventRules module interface to the new generic interface (#10386)
Port the third-party event rules interface to the generic module interface introduced in v1.37.0
2021-07-20 12:39:46 +02:00
reivilibre
4e340412c0
Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric (#10332)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-19 16:11:34 +01:00
Jonathan de Jong
93729719b8
Use inline type hints in tests/ (#10350)
This PR is tantamount to running:

    python3.8 -m com2ann -v 6 tests/

(com2ann requires python 3.8 to run)
2021-07-13 11:52:58 +01:00
Jonathan de Jong
89cfc3dd98
[pyupgrade] tests/ (#10347) 2021-07-13 11:43:15 +01:00
Patrick Cloke
f55836929d
Do not recurse into non-spaces in the spaces summary. (#10256)
Previously m.child.room events in non-space rooms would be
treated as part of the room graph, but this is no longer
supported.
2021-06-29 12:00:04 -04:00
Quentin Gliech
bd4919fb72
MSC2918 Refresh tokens implementation (#9450)
This implements refresh tokens, as defined by MSC2918

This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235

The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one.

Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-06-24 14:33:20 +01:00
Andrew Morgan
394673055d
Re-introduce "Leave out optional keys from /sync" change (#10214)
Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.
2021-06-23 15:57:41 +01:00
Richard van der Hoff
fcf3c7032b
Ensure that we do not cache empty sync responses after a timeout (#10158)
Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
2021-06-17 16:23:11 +01:00
Marcus
8070b893db
update black to 21.6b0 (#10197)
Reformat all files with the new version.

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
2021-06-17 15:20:06 +01:00
Patrick Cloke
9e5ab6dd58
Remove the experimental flag for knocking and use stable prefixes / endpoints. (#10167)
* Room version 7 for knocking.
* Stable prefixes and endpoints (both client and federation) for knocking.
* Removes the experimental configuration flag.
2021-06-15 07:45:14 -04:00
Sorunome
d936371b69
Implement knock feature (#6739)
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
Travis Ralston
5325f0308c
r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224)
[MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. 

This is progress towards https://github.com/matrix-org/synapse/issues/8334
2021-06-03 13:50:49 +01:00
Callum Brown
8fb9af570f
Make reason and score optional for report_event (#10077)
Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414
See #8551 

Signed-off-by: Callum Brown <callum@calcuode.com>
2021-05-27 18:42:23 +01:00
Brendan Abolivier
a683028d81
Correctly ratelimit invites when creating a room (#9968)
* Correctly ratelimit invites when creating a room

Also allow ratelimiting for more than one action at a time.
2021-05-12 16:05:28 +02:00
Richard van der Hoff
63fb220e5f
Tests for to-device messages (#9965) 2021-05-11 18:01:11 +01:00
Erik Johnston
eba431c539
Revert "Leave out optional keys from /sync (#9919)" (#9940)
This reverts commit e9eb3549d3.
2021-05-06 15:06:35 +01:00
DeepBlueV7.X
e9eb3549d3
Leave out optional keys from /sync (#9919)
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)

Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2021-05-05 14:37:56 +01:00
Erik Johnston
9d25a0ae65
Split presence out of master (#9820) 2021-04-23 12:21:55 +01:00
Erik Johnston
177dae2704
Limit length of accepted email addresses (#9855) 2021-04-22 17:49:11 +01:00
Jonathan de Jong
495b214f4f
Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
Andrew Morgan
71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Will Hunt
e300ef64b1
Require AppserviceRegistrationType (#9548)
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
2021-04-12 15:13:55 +01:00
Patrick Cloke
0b3112123d
Use mock from the stdlib. (#9772) 2021-04-09 13:44:38 -04:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Jonathan de Jong
e2b8a90897
Update mypy configuration: no_implicit_optional = True (#9742) 2021-04-05 09:10:18 -04:00
Andrew Morgan
7b06f85c0e
Ensure we use a copy of the event content dict before modifying it in serialize_event (#9585)
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)) - will raise a 500 if you try to a edit a reply to a message.

`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.

To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
2021-03-17 16:51:55 +00:00
Dirk Klimpel
8000cf1315
Return m.change_password.enabled=false if local database is disabled (#9588)
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
2021-03-16 11:44:25 -04:00
Richard van der Hoff
dd69110d95
Add support for stable MSC2858 API (#9617)
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
2021-03-16 11:21:26 +00:00
Patrick Cloke
33a02f0f52
Fix additional type hints from Twisted upgrade. (#9518) 2021-03-03 15:47:38 -05:00
Richard van der Hoff
15090de850
SSO: redirect to public URL before setting cookies (#9436)
... otherwise, we don't get the cookie back.
2021-02-26 14:02:06 +00:00
Andrew Morgan
0a363f9ca4
Remove cache for get_shared_rooms_for_users (#9416)
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.

The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately.

Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
2021-02-22 16:52:45 +00:00
Rishabh Arya
e17553e185
Parse ui_auth.session_timeout as a duration (instead of treating it as ms) (#9426) 2021-02-18 09:18:14 -05:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke
731e08c63a
Handle missing data in power levels events during room upgrade. (#9395) 2021-02-16 08:31:39 -05:00
Patrick Cloke
74af356baf
Convert additional test-cases to homeserver test case. (#9396)
And convert some inlineDeferreds to async-friendly functions.
2021-02-16 08:04:15 -05:00
Patrick Cloke
2c9b4a5f16 Synapse 1.27.0rc2 (2021-02-11)
==============================
 
 Features
 --------
 
 - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
 
 Bugfixes
 --------
 
 - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
 - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
 
 Improved Documentation
 ----------------------
 
 - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR
 MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS
 /tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN
 309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL
 GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb
 /QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP
 xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn
 ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B
 iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4
 awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU
 2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H
 lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0=
 =NSyp
 -----END PGP SIGNATURE-----

Merge tag 'v1.27.0rc2' into develop

Synapse 1.27.0rc2 (2021-02-11)
==============================

Features
--------

- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))

Bugfixes
--------

- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))

Improved Documentation
----------------------

- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
2021-02-11 11:56:03 -05:00
Patrick Cloke
e40d88cff3
Backout changes for automatically calculating the public baseurl. (#9313)
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
2021-02-11 11:16:54 -05:00
Patrick Cloke
8a33d217bd
Convert some test cases to use HomeserverTestCase. (#9377)
This has the side-effect of being able to remove use of `inlineCallbacks`
in the test-cases for cleaner tracebacks.
2021-02-11 10:29:09 -05:00
Richard van der Hoff
e288499c60
Social login UI polish (#9301) 2021-02-03 20:31:23 +00:00
Richard van der Hoff
18ab35284a Merge branch 'social_login' into develop 2021-02-01 17:28:37 +00:00
Richard van der Hoff
4167494c90
Replace username picker with a template (#9275)
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.

I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
2021-02-01 15:52:50 +00:00
Richard van der Hoff
9c715a5f19
Fix SSO on workers (#9271)
Fixes #8966.

* Factor out build_synapse_client_resource_tree

Start a function which will mount resources common to all workers.

* Move sso init into build_synapse_client_resource_tree

... so that we don't have to do it for each worker

* Fix SSO-login-via-a-worker

Expose the SSO login endpoints on workers, like the documentation says.

* Update workers config for new endpoints

Add documentation for endpoints recently added (#8942, #9017, #9262)

* remove submit_token from workers endpoints list

this *doesn't* work on workers (yet).

* changelog

* Add a comment about the odd path for SAML2Resource
2021-02-01 15:47:59 +00:00
Richard van der Hoff
f78d07bf00
Split out a separate endpoint to complete SSO registration (#9262)
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
2021-02-01 13:15:51 +00:00
Erik Johnston
f2c1560eca
Ratelimit invites by room and target user (#9258) 2021-01-29 16:38:29 +00:00
Erik Johnston
4b73488e81
Ratelimit 3PID /requestToken API (#9238) 2021-01-28 17:39:21 +00:00
Richard van der Hoff
7fa1346f93 Merge branch 'social_login' into develop 2021-01-27 17:27:24 +00:00
Richard van der Hoff
a737cc2713
Implement MSC2858 support (#9183)
Fixes #8928.
2021-01-27 12:41:24 +00:00
Patrick Cloke
4a55d267ee
Add an admin API for shadow-banning users. (#9209)
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.

A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.

Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
2021-01-25 14:49:39 -05:00
Richard van der Hoff
7447f19702
Prefix idp_id with "oidc-" (#9189)
... to avoid clashes with other SSO mechanisms
2021-01-21 12:25:02 +00:00
Richard van der Hoff
02070c69fa
Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128)
* Factor out a common TestHtmlParser

Looks like I'm doing this in a few different places.

* Improve OIDC login test

Complete the OIDC login flow, rather than giving up halfway through.

* Ensure that OIDC login works with multiple OIDC providers

* Fix bugs in handling clientRedirectUrl

 - don't drop duplicate query-params, or params with no value
 - allow utf-8 in query-params
2021-01-18 14:52:49 +00:00
Richard van der Hoff
b5dea8702d Fix test failure due to bad merge
0dd2649c1 (#9112) changed the signature of `auth_via_oidc`. Meanwhile,
26d10331e (#9091) introduced a new test which relied on the old signature of
`auth_via_oidc`. The two branches were never tested together until they landed
in develop.
2021-01-15 18:03:33 +00:00
Richard van der Hoff
0dd2649c12
Improve UsernamePickerTestCase (#9112)
* make the OIDC bits of the test work at a higher level - via the REST api instead of poking the OIDCHandler directly.
* Move it to test_login.py, where I think it fits better.
2021-01-15 13:45:13 +00:00
Richard van der Hoff
26d10331e5 Add a test for wrong user returned by SSO 2021-01-13 20:22:41 +00:00
Richard van der Hoff
233c8b9fce
Add a test for UI-Auth-via-SSO (#9082)
* Add complete test for UI-Auth-via-SSO.

* review comments
2021-01-13 20:21:55 +00:00
Dirk Klimpel
7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932)
This only applies if the user's data is to be erased.
2021-01-12 16:30:15 -05:00
Richard van der Hoff
8a910f97a4 Add some tests for the IDP picker flow 2021-01-07 14:56:42 +00:00
Richard van der Hoff
bbd04441ed Fix type hints in test_login.py 2021-01-07 14:56:42 +00:00
Richard van der Hoff
3fc2399dbe black-format tests/rest/client/v1/test_login.py
black seems to want to reformat this, despite `black --check` being happy with
it :/
2021-01-07 12:17:37 +00:00
Richard van der Hoff
8d3d264052
Skip unit tests which require optional dependencies (#9031)
If we are lacking an optional dependency, skip the tests that rely on it.
2021-01-07 11:41:28 +00:00
Richard van der Hoff
d2c616a413
Combine the SSO Redirect Servlets (#9015)
* Implement CasHandler.handle_redirect_request

... to make it match OidcHandler and SamlHandler

* Clean up interface for OidcHandler.handle_redirect_request

Make it accept `client_redirect_url=None`.

* Clean up interface for `SamlHandler.handle_redirect_request`

... bring it into line with CAS and OIDC by making it take a Request parameter,
move the magic for `client_redirect_url` for UIA into the handler, and fix the
return type to be a `str` rather than a `bytes`.

* Define a common protocol for SSO auth provider impls

* Give SsoIdentityProvider an ID and register them

* Combine the SSO Redirect servlets

Now that the SsoHandler knows about the identity providers, we can combine the
various *RedirectServlets into a single implementation which delegates to the
right IdP.

* changelog
2021-01-04 18:13:49 +00:00
Patrick Cloke
a802606475
Support PyJWT v2.0.0. (#8986)
Tests were broken due to an API changing. The code used in Synapse
proper should be compatible with both versions already.
2020-12-22 13:00:14 -05:00
Patrick Cloke
5d4c330ed9
Allow re-using a UI auth validation for a period of time (#8970) 2020-12-18 07:33:57 -05:00
Richard van der Hoff
7a332850e6
Merge pull request #8951 from matrix-org/rav/username_picker_2
More preparatory refactoring of the OidcHandler tests
2020-12-16 14:53:26 +00:00
Richard van der Hoff
2dd2e90e2b Test get_extra_attributes fallback
despite the warnings saying "don't implement get_extra_attributes", we had
implemented it, so the tests weren't doing what we thought they were.
2020-12-15 23:10:59 +00:00
Richard van der Hoff
c9dd47d668 lint 2020-12-15 22:35:50 +00:00
Richard van der Hoff
394516ad1b Remove spurious "SynapseRequest" result from `make_request"
This was never used, so let's get rid of it.
2020-12-15 22:35:40 +00:00
Richard van der Hoff
ac2acf1524 Remove redundant reading of SynapseRequest.args
this didn't seem to be doing a lot, so remove it.
2020-12-15 22:35:03 +00:00
Richard van der Hoff
0378581c13 remove 'response' result from _get_shared_rooms 2020-12-15 22:34:20 +00:00
Richard van der Hoff
7eebe4b3fc Replace request.code with channel.code
The two are equivalent, but really we want to check the HTTP result that got
returned to the channel, not the code that the Request object *intended* to
return to the channel.
2020-12-15 22:32:12 +00:00
Erik Johnston
1d55c7b567
Don't ratelimit autojoining of rooms (#8921)
Fixes #8866
2020-12-11 10:17:49 +00:00
Richard van der Hoff
cf3b8156be
Fix errorcode for disabled registration (#8867)
The spec says we should return `M_FORBIDDEN` when someone tries to register and
registration is disabled.
2020-12-03 15:41:19 +00:00
Richard van der Hoff
ed5172852a
Merge pull request #8858 from matrix-org/rav/sso_uia
UIA: offer only available auth flows
2020-12-02 20:06:53 +00:00
Richard van der Hoff
0bac276890 UIA: offer only available auth flows
During user-interactive auth, do not offer password auth to users with no
password, nor SSO auth to users with no SSO.

Fixes #7559.
2020-12-02 18:54:15 +00:00
Patrick Cloke
30fba62108
Apply an IP range blacklist to push and key revocation requests. (#8821)
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
Andrew Morgan
edb3d3f827
Allow specifying room version in 'RestHelper.create_room_as' and add typing (#8854)
This PR adds a `room_version` argument to the `RestHelper`'s `create_room_as` function for tests. I plan to use this for testing knocking, which currently uses an unstable room version.
2020-12-02 10:38:18 +00:00