Commit Graph

2507 Commits

Author SHA1 Message Date
Erik Johnston
37623e3382
Increase perf of handling presence when joining large rooms. (#9916) 2021-05-05 17:27:05 +01:00
DeepBlueV7.X
e9eb3549d3
Leave out optional keys from /sync (#9919)
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)

Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2021-05-05 14:37:56 +01:00
ThibF
0085dc5abc
Delete room endpoint (#9889)
Support the delete of a room through DELETE request and mark
previous request as deprecated through documentation.

Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
2021-04-29 10:31:45 +01:00
Erik Johnston
e4ab8676b4
Fix tight loop handling presence replication. (#9900)
Only affects workers. Introduced in #9819.

Fixes #9899.
2021-04-28 14:42:50 +01:00
Andrew Morgan
fe604a022a
Remove various bits of compatibility code for Python <3.6 (#9879)
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
2021-04-27 13:13:07 +01:00
Richard van der Hoff
3ff2251754
Improved validation for received requests (#9817)
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
Richard van der Hoff
84936e2264
Kill off _PushHTTPChannel. (#9878)
First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation.

Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
2021-04-23 18:40:57 +01:00
Richard van der Hoff
59d24c5bef
pass a reactor into SynapseSite (#9874) 2021-04-23 17:06:47 +01:00
Andrew Morgan
ceaa76970f
Remove room and user invite ratelimits in default unit test config (#9871) 2021-04-23 13:37:48 +01:00
Erik Johnston
9d25a0ae65
Split presence out of master (#9820) 2021-04-23 12:21:55 +01:00
Richard van der Hoff
51a20914a8
Limit the size of HTTP responses read over federation. (#9833) 2021-04-23 11:08:41 +01:00
Erik Johnston
177dae2704
Limit length of accepted email addresses (#9855) 2021-04-22 17:49:11 +01:00
Patrick Cloke
683d6f75af
Rename handler and config modules which end in handler/config. (#9816) 2021-04-20 14:55:20 -04:00
Andrew Morgan
6982db9651 Merge branch 'master' into develop 2021-04-20 14:55:16 +01:00
Patrick Cloke
b076bc276e
Always use the name as the log ID. (#9829)
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.

This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20 14:19:00 +01:00
Erik Johnston
de0d088adc
Add presence federation stream (#9819) 2021-04-20 14:11:24 +01:00
Jonathan de Jong
495b214f4f
Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
Andrew Morgan
71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
Richard van der Hoff
5a153772c1
remove HomeServer.get_config (#9815)
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
2021-04-14 19:09:08 +01:00
Patrick Cloke
936e69825a
Separate creating an event context from persisting it in the federation handler (#9800)
This refactoring allows adding logic that uses the event context
before persisting it.
2021-04-14 12:35:28 -04:00
Patrick Cloke
e8816c6ace Revert "Check for space membership during a remote join of a restricted room. (#9763)"
This reverts commit cc51aaaa7a.

The PR was prematurely merged and not yet approved.
2021-04-14 12:33:37 -04:00
Patrick Cloke
cc51aaaa7a
Check for space membership during a remote join of a restricted room. (#9763)
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join
rules.
    
This only applies to an experimental room version, as defined in MSC3083.
2021-04-14 12:32:20 -04:00
Erik Johnston
00a6db9676
Move some replication processing out of generic_worker (#9796)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-04-14 17:06:06 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Dan Callahan
1d5f0e3529
Bump black configuration to target py36 (#9781)
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13 10:41:34 +01:00
Dirk Klimpel
1fc97ee876
Add an admin API to manage ratelimit for a specific user (#9648) 2021-04-13 10:26:37 +01:00
Will Hunt
e300ef64b1
Require AppserviceRegistrationType (#9548)
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
2021-04-12 15:13:55 +01:00
Patrick Cloke
0b3112123d
Use mock from the stdlib. (#9772) 2021-04-09 13:44:38 -04:00
Dirk Klimpel
48a1f4db31
Remove old admin API GET /_synapse/admin/v1/users/<user_id> (#9401)
Related: #8334
Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25)

`GET /_synapse/admin/v1/users/<user_id>` has no
- unit tests
- documentation

API in v2 is available (#5925 - 12/2019, v1.7.0).
API is misleading. It expects `user_id` and returns a list of all users.

Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-04-09 09:44:40 +01:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682)
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Richard van der Hoff
9e167d9c53 Merge remote-tracking branch 'origin/develop' into rav/drop_py35 2021-04-08 18:30:38 +01:00
Richard van der Hoff
24c58ebfc9 remove unused param on make_tuple_comparison_clause 2021-04-08 18:29:57 +01:00
Richard van der Hoff
3ada9b4264 Drop support for sqlite<3.22 as well 2021-04-08 16:42:32 +01:00
Patrick Cloke
48d44ab142
Record more information into structured logs. (#9654)
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
2021-04-08 08:01:14 -04:00
Andrew Morgan
04819239ba
Add a Synapse Module for configuring presence update routing (#9491)
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.

This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.

A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. 

The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:

* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06 14:38:30 +01:00
Andrew Morgan
024f121b74
Fix reported bugbear: too broad exception assertion (#9753) 2021-04-06 13:48:22 +01:00
Patrick Cloke
e7b769aea1
Convert storage test cases to HomeserverTestCase. (#9736) 2021-04-06 07:21:02 -04:00
Jonathan de Jong
e2b8a90897
Update mypy configuration: no_implicit_optional = True (#9742) 2021-04-05 09:10:18 -04:00
Dirk Klimpel
bb0fe02a52
Add order_by to list user admin API (#9691) 2021-04-01 11:28:53 +01:00
Patrick Cloke
35c5ef2d24
Add an experimental room version to support restricted join rules. (#9717)
Per MSC3083.
2021-03-31 16:39:08 -04:00
Richard van der Hoff
f02663c4dd
Replace room_invite_state_types with room_prejoin_state (#9700)
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
2021-03-30 12:12:44 +01:00
Erik Johnston
963f4309fe
Make RateLimiter class check for ratelimit overrides (#9711)
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited.

We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits.

Fixes #9663
2021-03-30 12:06:09 +01:00
Patrick Cloke
01dd90b0f0
Add type hints to DictionaryCache and TTLCache. (#9442) 2021-03-29 12:15:33 -04:00
Andrew Morgan
5fdff97719
Fix CI by ignore type for None module import (#9709) 2021-03-29 14:42:38 +01:00
Erik Johnston
b5efcb577e
Make it possible to use dmypy (#9692)
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
2021-03-26 16:49:46 +00:00
Jonathan de Jong
4c3827f2c1
Enable addtional flake8-bugbear linting checks. (#9659) 2021-03-24 09:34:30 -04:00
Andrew Morgan
5b268997bd
Allow providing credentials to HTTPS_PROXY (#9657)
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70

This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.

Also added some type hints.
2021-03-22 17:20:47 +00:00
Brendan Abolivier
592d6305fd
Merge branch 'develop' into babolivier/msc3026 2021-03-19 16:12:40 +01:00
Dirk Klimpel
8dd2ea65a9
Consistently check whether a password may be set for a user. (#9636) 2021-03-18 12:54:08 -04:00
Erik Johnston
dd71eb0f8a
Make federation catchup send last event from any server. (#9640)
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.

Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
2021-03-18 15:52:26 +00:00
Brendan Abolivier
405aeb0b2c
Implement MSC3026: busy presence state 2021-03-18 16:34:47 +01:00
Andrew Morgan
7b06f85c0e
Ensure we use a copy of the event content dict before modifying it in serialize_event (#9585)
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)) - will raise a 500 if you try to a edit a reply to a message.

`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.

To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
2021-03-17 16:51:55 +00:00
Jonathan de Jong
27d2820c33
Enable flake8-bugbear, but disable most checks. (#9499)
* Adds B00 to ignored checks.
* Fixes remaining issues.
2021-03-16 14:19:27 -04:00
Hubbe
dd5e5dc1d6
Add SSO attribute requirements for OIDC providers (#9609)
Allows limiting who can login using OIDC via the claims
made from the IdP.
2021-03-16 11:46:07 -04:00
Dirk Klimpel
8000cf1315
Return m.change_password.enabled=false if local database is disabled (#9588)
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
2021-03-16 11:44:25 -04:00
Andrew Morgan
847ecdd8fa
Pass SSO IdP information to spam checker's registration function (#9626)
Fixes https://github.com/matrix-org/synapse/issues/9572

When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.

This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
2021-03-16 12:41:41 +00:00
Richard van der Hoff
dd69110d95
Add support for stable MSC2858 API (#9617)
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
2021-03-16 11:21:26 +00:00
Patrick Cloke
d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. (#9608) 2021-03-15 11:14:39 -04:00
Erik Johnston
026503fa3b
Don't go into federation catch up mode so easily (#9561)
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.

Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
2021-03-15 14:42:40 +00:00
Patrick Cloke
55da8df078
Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
Patrick Cloke
e55bd0e110
Add tests for blacklisting reactor/agent. (#9563) 2021-03-11 09:15:22 -05:00
Patrick Cloke
2a99cc6524
Use the chain cover index in get_auth_chain_ids. (#9576)
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
2021-03-10 09:57:59 -05:00
Richard van der Hoff
eaada74075
JWT OIDC secrets for Sign in with Apple (#9549)
Apple had to be special. They want a client secret which is generated from an EC key.

Fixes #9220. Also fixes #9212 while I'm here.
2021-03-09 15:03:37 +00:00
Patrick Cloke
7fdc6cefb3
Fix additional type hints. (#9543)
Type hint fixes due to Twisted 21.2.0 adding type hints.
2021-03-09 07:41:32 -05:00
Patrick Cloke
075c16b410
Handle image transparency better when thumbnailing. (#9473)
Properly uses RGBA mode for 1- and 8-bit images with transparency
(instead of RBG mode).
2021-03-09 07:37:09 -05:00
Jonathan de Jong
d6196efafc
Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
Patrick Cloke
cb7fc7523e
Add a basic test for purging rooms. (#9541)
Unfortunately this doesn't test re-joining the room since
that requires having another homeserver to query over
federation, which isn't easily doable in unit tests.
2021-03-08 09:21:36 -05:00
Richard van der Hoff
7eb6e39a8f
Record the SSO Auth Provider in the login token (#9510)
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
2021-03-04 14:44:22 +00:00
Patrick Cloke
33a02f0f52
Fix additional type hints from Twisted upgrade. (#9518) 2021-03-03 15:47:38 -05:00
Patrick Cloke
16ec8c3272
(Hopefully) stop leaking file descriptors in media repo. (#9497)
By consuming the response if the headers imply that the
content is too large.
2021-03-01 12:45:00 -05:00
Tim Leung
ddb240293a
Add support for no_proxy and case insensitive env variables (#9372)
### Changes proposed in this PR

- Add support for the `no_proxy` and `NO_PROXY` environment variables
  - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519))
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment

This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`

Related to #9306 which also uses `ProxyAgent`

Signed-off-by: Timothy Leung tim95@hotmail.co.uk
2021-02-26 17:37:57 +00:00
Richard van der Hoff
15090de850
SSO: redirect to public URL before setting cookies (#9436)
... otherwise, we don't get the cookie back.
2021-02-26 14:02:06 +00:00
Erik Johnston
2566dc57ce
Test that we require validated email for email pushers (#9496) 2021-02-25 15:35:14 +00:00
Erik Johnston
2927921942
Clean up ShardedWorkerHandlingConfig (#9466)
* Split ShardedWorkerHandlingConfig

This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).

* Remove special cases in ShardedWorkerHandlingConfig.

`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.

To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
2021-02-24 13:23:18 +00:00
Richard van der Hoff
1b2d6d55c5
Remove vestiges of uploads_path config (#9462)
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
2021-02-22 19:54:49 +00:00
Dirk Klimpel
71c9f8de6d
Add an order_by field to list users' media admin API. (#8978) 2021-02-22 14:38:51 -05:00
Andrew Morgan
0a363f9ca4
Remove cache for get_shared_rooms_for_users (#9416)
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.

The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately.

Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
2021-02-22 16:52:45 +00:00
Erik Johnston
179c0953ff
Regenerate exact thumbnails if missing (#9438) 2021-02-19 17:09:57 +00:00
Erik Johnston
3a2fe5054f Add test 2021-02-19 15:52:04 +00:00
Patrick Cloke
c4a55ac4a4 Fix style checking due to updated black. 2021-02-19 08:19:54 -05:00
Andrew Morgan
8bcfc2eaad
Be smarter about which hosts to send presence to when processing room joins (#9402)
This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually.

---

When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed.

It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw.

This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence:

* If it was a local user join, send that user's latest presence to all servers in the room
* If it was a remote user join, send the presence for all local users in the room to that homeserver

We deduplicate by inserting all of those pending updates into a dictionary of the form:

```
{
  server_name1: {presence_update1, ...},
  server_name2: {presence_update1, presence_update2, ...}
}
```

Only after building this dict do we then start sending out presence updates.
2021-02-19 11:37:29 +00:00
Andrew Morgan
13e9029f44
Add a config option to prioritise local users in user directory search results (#9383)
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.

Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.

This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
2021-02-19 11:02:03 +00:00
AndrewFerr
9bc74743d5
Add configs to make profile data more private (#9203)
Add off-by-default configuration settings to:
- disable putting an invitee's profile info in invite events
- disable profile lookup via federation

Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-02-19 09:50:41 +00:00
Rishabh Arya
e17553e185
Parse ui_auth.session_timeout as a duration (instead of treating it as ms) (#9426) 2021-02-18 09:18:14 -05:00
Patrick Cloke
8ec2217103
Reduce the memory usage of previewing media files. (#9421)
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.

It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
2021-02-18 09:01:29 -05:00
Dirk Klimpel
c8d9383cfb
Add the shadow-banning status to the display user admin API. (#9400) 2021-02-17 15:19:23 -05:00
Richard van der Hoff
e1071fd625
Support for form_post in OIDC responses (#9376)
Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
2021-02-17 10:15:14 +00:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Richard van der Hoff
3b754aea27
Clean up caching/locking of OIDC metadata load (#9362)
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
2021-02-16 16:27:38 +00:00
Patrick Cloke
731e08c63a
Handle missing data in power levels events during room upgrade. (#9395) 2021-02-16 08:31:39 -05:00
Patrick Cloke
74af356baf
Convert additional test-cases to homeserver test case. (#9396)
And convert some inlineDeferreds to async-friendly functions.
2021-02-16 08:04:15 -05:00
Patrick Cloke
2c9b4a5f16 Synapse 1.27.0rc2 (2021-02-11)
==============================
 
 Features
 --------
 
 - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
 
 Bugfixes
 --------
 
 - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
 - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
 
 Improved Documentation
 ----------------------
 
 - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR
 MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS
 /tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN
 309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL
 GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb
 /QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP
 xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn
 ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B
 iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4
 awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU
 2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H
 lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0=
 =NSyp
 -----END PGP SIGNATURE-----

Merge tag 'v1.27.0rc2' into develop

Synapse 1.27.0rc2 (2021-02-11)
==============================

Features
--------

- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))

Bugfixes
--------

- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))

Improved Documentation
----------------------

- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
2021-02-11 11:56:03 -05:00
Patrick Cloke
e40d88cff3
Backout changes for automatically calculating the public baseurl. (#9313)
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
2021-02-11 11:16:54 -05:00
Patrick Cloke
8a33d217bd
Convert some test cases to use HomeserverTestCase. (#9377)
This has the side-effect of being able to remove use of `inlineCallbacks`
in the test-cases for cleaner tracebacks.
2021-02-11 10:29:09 -05:00
Patrick Cloke
6dade80048
Combine the CAS & SAML implementations for required attributes. (#9326) 2021-02-11 10:05:15 -05:00
Patrick Cloke
0963d39ea6
Handle additional errors when previewing URLs. (#9333)
* Handle the case of lxml not finding a document tree.
* Parse the document encoding from the XML tag.
2021-02-08 12:33:30 -05:00
David Teller
b0b2cac057
Merge pull request #9150 from Yoric/develop-context
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
2021-02-08 15:53:44 +01:00
Erik Johnston
adc96d4236 Merge branch 'erikj/media_spam_checker' into develop 2021-02-04 17:01:59 +00:00
Erik Johnston
7e8083eb48 Add check_media_file_for_spam spam checker hook 2021-02-04 17:01:30 +00:00
Patrick Cloke
792263c97c
Handle empty rooms when generating email notifications. (#9257)
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
2021-02-04 10:18:25 -05:00
Richard van der Hoff
3f534d3fdf Merge branch 'social_login_hotfixes' into develop 2021-02-03 20:34:27 +00:00
Richard van der Hoff
17f2a512f3 Merge remote-tracking branch 'origin/release-v1.27.0' into social_login_hotfixes 2021-02-03 20:33:32 +00:00
Richard van der Hoff
e288499c60
Social login UI polish (#9301) 2021-02-03 20:31:23 +00:00
Patrick Cloke
4ca054a4ea
Convert blacklisted IPv4 addresses to compatible IPv6 addresses. (#9240)
Also add a few more IP ranges to the default blacklist.
2021-02-03 07:13:46 -05:00
Erik Johnston
ff55300b91
Honour ratelimit flag for application services for invite ratelimiting (#9302) 2021-02-03 10:17:37 +00:00
Travis Ralston
b60bb28bbc
Add an admin API to get the current room state (#9168)
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.

This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
2021-02-02 11:16:29 +00:00
Richard van der Hoff
846b9d3df0
Put OIDC callback URI under /_synapse/client. (#9288) 2021-02-01 22:56:01 +00:00
Patrick Cloke
5d38a3c97f
Refactor email summary generation. (#9260)
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
  was tweaked to depend on whether there was 1 sender or multiple senders,
  not based on if there was 1 room or multiple rooms.
2021-02-01 13:09:39 -05:00
Richard van der Hoff
18ab35284a Merge branch 'social_login' into develop 2021-02-01 17:28:37 +00:00
Richard van der Hoff
4167494c90
Replace username picker with a template (#9275)
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.

I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
2021-02-01 15:52:50 +00:00
Richard van der Hoff
8aed29dc61
Improve styling and wording of SSO redirect confirm template (#9272) 2021-02-01 15:50:56 +00:00
Richard van der Hoff
9c715a5f19
Fix SSO on workers (#9271)
Fixes #8966.

* Factor out build_synapse_client_resource_tree

Start a function which will mount resources common to all workers.

* Move sso init into build_synapse_client_resource_tree

... so that we don't have to do it for each worker

* Fix SSO-login-via-a-worker

Expose the SSO login endpoints on workers, like the documentation says.

* Update workers config for new endpoints

Add documentation for endpoints recently added (#8942, #9017, #9262)

* remove submit_token from workers endpoints list

this *doesn't* work on workers (yet).

* changelog

* Add a comment about the odd path for SAML2Resource
2021-02-01 15:47:59 +00:00
Richard van der Hoff
f78d07bf00
Split out a separate endpoint to complete SSO registration (#9262)
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
2021-02-01 13:15:51 +00:00
Erik Johnston
f2c1560eca
Ratelimit invites by room and target user (#9258) 2021-01-29 16:38:29 +00:00
Erik Johnston
4b73488e81
Ratelimit 3PID /requestToken API (#9238) 2021-01-28 17:39:21 +00:00
David Teller
31d072aea0 FIXUP: linter 2021-01-28 16:53:40 +01:00
David Teller
b859919acc FIXUP: Now testing that the user is admin! 2021-01-28 12:31:07 +01:00
David Teller
10332c175c New API /_synapse/admin/rooms/{roomId}/context/{eventId}
Signed-off-by: David Teller <davidt@element.io>
2021-01-28 12:29:49 +01:00
Richard van der Hoff
7fa1346f93 Merge branch 'social_login' into develop 2021-01-27 17:27:24 +00:00
Richard van der Hoff
a737cc2713
Implement MSC2858 support (#9183)
Fixes #8928.
2021-01-27 12:41:24 +00:00
Erik Johnston
dd8da8c5f6
Precompute joined hosts and store in Redis (#9198) 2021-01-26 13:57:31 +00:00
Patrick Cloke
4937fe3d6b
Try to recover from unknown encodings when previewing media. (#9164)
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
2021-01-26 07:32:17 -05:00
Patrick Cloke
4a55d267ee
Add an admin API for shadow-banning users. (#9209)
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.

A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.

Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
2021-01-25 14:49:39 -05:00
Patrick Cloke
e448dbbf5b Synapse 1.26.0rc2 (2021-01-25)
==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
 MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
 +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
 BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
 d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
 oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
 O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
 VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
 g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
 =Qc7+
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc2' into develop

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
2021-01-25 08:51:45 -05:00
Patrick Cloke
6f7417c3db
Handle missing content keys when calculating presentable names. (#9165)
Treat the content as untrusted and do not assume it is of
the proper form.
2021-01-25 07:27:16 -05:00
Erik Johnston
056327457f
Fix chain cover update to handle events with duplicate auth events (#9210) 2021-01-22 19:44:08 +00:00
Patrick Cloke
a7882f9887
Return a 404 if no valid thumbnail is found. (#9163)
If no thumbnail of the requested type exists, return a 404 instead
of erroring. This doesn't quite match the spec (which does not define
what happens if no thumbnail can be found), but is consistent with
what Synapse already does.
2021-01-21 14:53:58 -05:00
Dirk Klimpel
c55e62548c
Add tests for List Users Admin API (#9045) 2021-01-21 09:18:46 -05:00
Richard van der Hoff
7447f19702
Prefix idp_id with "oidc-" (#9189)
... to avoid clashes with other SSO mechanisms
2021-01-21 12:25:02 +00:00
Richard van der Hoff
0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Richard van der Hoff
fa50e4bf4d
Give public_baseurl a default value (#9159) 2021-01-20 12:30:41 +00:00
Erik Johnston
6633a4015a
Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
Richard van der Hoff
02070c69fa
Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128)
* Factor out a common TestHtmlParser

Looks like I'm doing this in a few different places.

* Improve OIDC login test

Complete the OIDC login flow, rather than giving up halfway through.

* Ensure that OIDC login works with multiple OIDC providers

* Fix bugs in handling clientRedirectUrl

 - don't drop duplicate query-params, or params with no value
 - allow utf-8 in query-params
2021-01-18 14:52:49 +00:00
Richard van der Hoff
b5dea8702d Fix test failure due to bad merge
0dd2649c1 (#9112) changed the signature of `auth_via_oidc`. Meanwhile,
26d10331e (#9091) introduced a new test which relied on the old signature of
`auth_via_oidc`. The two branches were never tested together until they landed
in develop.
2021-01-15 18:03:33 +00:00
Erik Johnston
350d9923cd
Make chain cover index bg update go faster (#9124)
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
2021-01-15 17:18:37 +00:00
Richard van der Hoff
9de6b94117
Land support for multiple OIDC providers (#9110)
This is the final step for supporting multiple OIDC providers concurrently.

First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before:

    oidc_config:
       enabled: true
       issuer: "https://oidc_provider"
       # etc

After:

    oidc_providers:
     - idp_id: prov1
       issuer: "https://oidc_provider"

     - idp_id: prov2
       issuer: "https://another_oidc_provider"

The old format is still grandfathered in.

With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
2021-01-15 16:55:29 +00:00
Patrick Cloke
3e4cdfe5d9
Add an admin API endpoint to protect media. (#9086)
Protecting media stops it from being quarantined when
e.g. all media in a room is quarantined. This is useful
for sticker packs and other media that is uploaded by
server administrators, but used by many people.
2021-01-15 11:18:09 -05:00
Patrick Cloke
74dd906041
Avoid raising the body exceeded error multiple times. (#9108)
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
2021-01-15 11:00:13 -05:00
Richard van der Hoff
0dd2649c12
Improve UsernamePickerTestCase (#9112)
* make the OIDC bits of the test work at a higher level - via the REST api instead of poking the OIDCHandler directly.
* Move it to test_login.py, where I think it fits better.
2021-01-15 13:45:13 +00:00
Richard van der Hoff
4575ad0b1e
Store an IdP ID in the OIDC session (#9109)
Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP.

As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
2021-01-15 13:22:12 +00:00
Richard van der Hoff
14950a45d6
Merge pull request #9091 from matrix-org/rav/error_on_bad_sso
Give the user a better error when they present bad SSO creds
2021-01-15 00:27:13 +00:00
Erik Johnston
1a08e0cdab
Fix event chain bg update. (#9118)
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
2021-01-14 18:57:32 +00:00
Erik Johnston
7036e24e98
Add background update for add chain cover index (#9029) 2021-01-14 15:18:27 +00:00
Richard van der Hoff
21a296cd5a
Split OidcProvider out of OidcHandler (#9107)
The idea here is that we will have an instance of OidcProvider for each
configured IdP, with OidcHandler just doing the marshalling of them.

For now it's still hardcoded with a single provider.
2021-01-14 13:29:17 +00:00
Richard van der Hoff
26d10331e5 Add a test for wrong user returned by SSO 2021-01-13 20:22:41 +00:00
Richard van der Hoff
233c8b9fce
Add a test for UI-Auth-via-SSO (#9082)
* Add complete test for UI-Auth-via-SSO.

* review comments
2021-01-13 20:21:55 +00:00
Patrick Cloke
98a64b7f7f
Add basic domain validation for DomainSpecificString.is_valid. (#9071)
This checks that the domain given to `DomainSpecificString.is_valid` (e.g.
`UserID`, `RoomAlias`, etc.) is of a valid form. Previously some validation
was done on the localpart (e.g. the sigil), but not the domain portion.
2021-01-13 07:05:16 -05:00
Erik Johnston
aa4d8c1f9a Merge branch 'master' into develop 2021-01-13 10:36:55 +00:00
Richard van der Hoff
bc4bf7b384
Preparatory refactors of OidcHandler (#9067)
Some light refactoring of OidcHandler, in preparation for bigger things:

  * remove inheritance from deprecated BaseHandler
  * add an object to hold the things that go into a session cookie
  * factor out a separate class for manipulating said cookies
2021-01-13 10:26:12 +00:00
Dirk Klimpel
7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932)
This only applies if the user's data is to be erased.
2021-01-12 16:30:15 -05:00
Marcus
e385c8b473
Don't apply the IP range blacklist to proxy connections (#9084)
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
2021-01-12 12:20:30 -05:00
Patrick Cloke
723b19748a
Handle bad JSON data being returned from the federation API. (#9070) 2021-01-12 11:07:01 -05:00
Richard van der Hoff
2ec8ca5e60
Remove SynapseRequest.get_user_agent (#9069)
SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests",
which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the 
LoggingContext and write the right entries to the request log).

Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a
SynapseRequest when there is nothing synapse-specific about the Request at all, and any old
twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult.

In short: move get_user_agent out to a utility function.
2021-01-12 12:34:16 +00:00
David Teller
b161528fcc
Also support remote users on the joined_rooms admin API. (#8948)
For remote users, only the rooms which the server knows about are returned.
Local users have all of their joined rooms returned.
2021-01-11 14:32:17 -05:00
Dirk Klimpel
42d3a28d8b
Removes unnecessary declarations in the tests for the admin API. (#9063) 2021-01-11 11:15:54 -05:00
Erik Johnston
1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00
Richard van der Hoff
12f79da587
Merge pull request #9036 from matrix-org/rav/multi_idp/tests
Add tests for the IdP picker
2021-01-08 14:24:41 +00:00
Richard van der Hoff
d32870ffa5
Fix validate_config on nested objects (#9054) 2021-01-08 14:23:04 +00:00
Richard van der Hoff
8a910f97a4 Add some tests for the IDP picker flow 2021-01-07 14:56:42 +00:00
Richard van der Hoff
bbd04441ed Fix type hints in test_login.py 2021-01-07 14:56:42 +00:00
Patrick Cloke
23d701864f
Improve the performance of calculating ignored users in large rooms (#9024)
This allows for efficiently finding which users ignore a particular
user.

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-01-07 13:03:38 +00:00
Richard van der Hoff
3fc2399dbe black-format tests/rest/client/v1/test_login.py
black seems to want to reformat this, despite `black --check` being happy with
it :/
2021-01-07 12:17:37 +00:00
Richard van der Hoff
8d3d264052
Skip unit tests which require optional dependencies (#9031)
If we are lacking an optional dependency, skip the tests that rely on it.
2021-01-07 11:41:28 +00:00
Patrick Cloke
1b4d5d6acf
Empty iterables should count towards cache usage. (#9028) 2021-01-06 12:33:20 -05:00
Patrick Cloke
9dde9c9f01
Implement MSC2176: Updated redaction rules (#8984)
An experimental room version ("org.matrix.msc2176") contains
the new redaction rules for testing.
2021-01-05 07:41:48 -05:00
Richard van der Hoff
d2c616a413
Combine the SSO Redirect Servlets (#9015)
* Implement CasHandler.handle_redirect_request

... to make it match OidcHandler and SamlHandler

* Clean up interface for OidcHandler.handle_redirect_request

Make it accept `client_redirect_url=None`.

* Clean up interface for `SamlHandler.handle_redirect_request`

... bring it into line with CAS and OIDC by making it take a Request parameter,
move the magic for `client_redirect_url` for UIA into the handler, and fix the
return type to be a `str` rather than a `bytes`.

* Define a common protocol for SSO auth provider impls

* Give SsoIdentityProvider an ID and register them

* Combine the SSO Redirect servlets

Now that the SsoHandler knows about the identity providers, we can combine the
various *RedirectServlets into a single implementation which delegates to the
right IdP.

* changelog
2021-01-04 18:13:49 +00:00
Patrick Cloke
1c9a850562
Add type hints to the crypto module. (#8999) 2021-01-04 10:04:50 -05:00
Patrick Cloke
0eccf53146
Use the SSO handler helpers for CAS registration/login. (#8856) 2021-01-03 16:25:44 +00:00
Andrew Morgan
168ba00d01
Fix RoomDirectoryFederationTests and make them actually run (#8998)
The `RoomDirectoryFederationTests` tests were not being run unless explicitly called as an `__init__.py` file was not present in `tests/federation/transport/`. Thus the folder was not a python module, and `trial` did not look inside for any test cases to run. This was found while working on #6739.

This PR adds a `__init__.py` and also fixes the test in a couple ways:

- Switch to subclassing `unittest.FederatingHomeserverTestCase` instead, which sets up federation endpoints for us.
- Supply a `federation_auth_origin` to `make_request` in order to more act like the request is coming from another server, instead of just an unauthenicated client requesting a federation endpoint.

I found that the second point makes no difference to the test passing, but felt like the right thing to do if we're testing over federation.
2020-12-30 19:27:32 +00:00
Patrick Cloke
d0c3c24eb2
Drop the unused local_invites table. (#8979)
This table has been unused since Synapse v1.17.0.
2020-12-29 07:26:29 -05:00
Patrick Cloke
a802606475
Support PyJWT v2.0.0. (#8986)
Tests were broken due to an API changing. The code used in Synapse
proper should be compatible with both versions already.
2020-12-22 13:00:14 -05:00
Erik Johnston
d781a81e69
Allow server admin to get admin bit in rooms where local user is an admin (#8756)
This adds an admin API that allows a server admin to get power in a room if a local user has power in a room. Will also invite the user if they're not in the room and its a private room. Can specify another user (rather than the admin user) to be granted power.

Co-authored-by: Matthew Hodgson <matthew@matrix.org>
2020-12-18 15:37:19 +00:00
Erik Johnston
5e7d75daa2
Fix mainline ordering in state res v2 (#8971)
This had two effects 1) it'd give the wrong answer and b) would iterate
*all* power levels in the auth chain of each event. The latter of which
can be *very* expensive for certain types of IRC bridge rooms that have
large numbers of power level changes.
2020-12-18 15:00:34 +00:00
Richard van der Hoff
28877fade9
Implement a username picker for synapse (#8942)
The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow
`UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser
back to the client, we redirect to a username-picker resource, which allows the user to enter a username.
We *then* complete the SSO flow (including doing the client permission checks).

The static resources for the username picker itself (in 
https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker)
are essentially lifted wholesale from
https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. 
As the comment says, we might want to think about making them customisable, but that can be a follow-up. 

Fixes #8876.
2020-12-18 14:19:46 +00:00
Patrick Cloke
5d4c330ed9
Allow re-using a UI auth validation for a period of time (#8970) 2020-12-18 07:33:57 -05:00
Erik Johnston
a7a913918c Merge remote-tracking branch 'origin/erikj/as_mau_block' into develop 2020-12-18 09:51:56 +00:00
Erik Johnston
70586aa63e
Try and drop stale extremities. (#8929)
If we see stale extremities while persisting events, and notice that
they don't change the result of state resolution, we drop them.
2020-12-18 09:49:18 +00:00
Erik Johnston
14eab1b4d2
Update tests/test_mau.py
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-12-17 16:14:13 +00:00
Richard van der Hoff
c9c1c9d82f
Fix UsersListTestCase (#8964) 2020-12-17 10:46:40 -05:00
Brendan Abolivier
f2783fc201
Use the simple dictionary in full text search for the user directory (#8959)
* Use the simple dictionary in fts for the user directory

* Clarify naming
2020-12-17 14:42:30 +01:00
Erik Johnston
4c33796b20 Correctly handle AS registerations and add test 2020-12-17 12:55:21 +00:00
Dirk Klimpel
c07022303e
Fix a bug that deactivated users appear in the directory (#8933)
Fixes a bug that deactivated users appear in the directory when their profile information was updated.

To change profile information of deactivated users is neccesary for example you will remove displayname or avatar.
But they should not appear in directory. They are deactivated.



Co-authored-by: Erik Johnston <erikj@jki.re>
2020-12-17 12:05:39 +00:00
Dirk Klimpel
06006058d7
Make search statement in List Room and User Admin API case-insensitive (#8931) 2020-12-17 10:43:37 +00:00
Patrick Cloke
ff5c4da128
Add a maximum size for well-known lookups. (#8950) 2020-12-16 17:25:24 -05:00
Richard van der Hoff
e1b8e37f93
Push login completion down into SsoHandler (#8941)
This is another part of my work towards fixing #8876. It moves some of the logic currently in the SAML and OIDC handlers - in particular the call to `AuthHandler.complete_sso_login` down into the `SsoHandler`.
2020-12-16 20:01:53 +00:00
Patrick Cloke
bd30cfe86a
Convert internal pusher dicts to attrs classes. (#8940)
This improves type hinting and should use less memory.
2020-12-16 11:25:30 -05:00
Richard van der Hoff
7a332850e6
Merge pull request #8951 from matrix-org/rav/username_picker_2
More preparatory refactoring of the OidcHandler tests
2020-12-16 14:53:26 +00:00
Richard van der Hoff
8388a7fb3a Make _make_callback_with_userinfo async
... so that we can test its behaviour when it raises.

Also pull it out to the top level so that I can use it from other test classes.
2020-12-15 23:10:59 +00:00
Richard van der Hoff
c1883f042d Remove spurious mocking of complete_sso_login
The tests that need this all do it already.
2020-12-15 23:10:59 +00:00
Richard van der Hoff
2dd2e90e2b Test get_extra_attributes fallback
despite the warnings saying "don't implement get_extra_attributes", we had
implemented it, so the tests weren't doing what we thought they were.
2020-12-15 23:10:59 +00:00
Richard van der Hoff
c9dd47d668 lint 2020-12-15 22:35:50 +00:00
Richard van der Hoff
394516ad1b Remove spurious "SynapseRequest" result from `make_request"
This was never used, so let's get rid of it.
2020-12-15 22:35:40 +00:00
Richard van der Hoff
ac2acf1524 Remove redundant reading of SynapseRequest.args
this didn't seem to be doing a lot, so remove it.
2020-12-15 22:35:03 +00:00
Richard van der Hoff
5bcf6e8289 Skip redundant check on request.args 2020-12-15 22:35:03 +00:00
Richard van der Hoff
0378581c13 remove 'response' result from _get_shared_rooms 2020-12-15 22:34:20 +00:00
Richard van der Hoff
7eebe4b3fc Replace request.code with channel.code
The two are equivalent, but really we want to check the HTTP result that got
returned to the channel, not the code that the Request object *intended* to
return to the channel.
2020-12-15 22:32:12 +00:00
Richard van der Hoff
01333681bc
Preparatory refactoring of the SamlHandlerTestCase (#8938)
* move simple_async_mock to test_utils

... so that it can be re-used

* Remove references to `SamlHandler._map_saml_response_to_user` from tests

This method is going away, so we can no longer use it as a test point. Instead,
factor out a higher-level method which takes a SAML object, and verify correct
behaviour by mocking out `AuthHandler.complete_sso_login`.

* changelog
2020-12-15 20:56:10 +00:00
Richard van der Hoff
6d02eb22df
Fix startup failure with localdb_enabled: False (#8937) 2020-12-14 20:42:03 +00:00
Patrick Cloke
1619802228
Various clean-ups to the logging context code (#8935) 2020-12-14 14:19:47 -05:00
Richard van der Hoff
895e04319b
Preparatory refactoring of the OidcHandlerTestCase (#8911)
* Remove references to handler._auth_handler

(and replace them with hs.get_auth_handler)

* Factor out a utility function for building Requests

* Remove mocks of `OidcHandler._map_userinfo_to_user`

This method is going away, so mocking it out is no longer a valid approach.

Instead, we mock out lower-level methods (eg _remote_id_from_userinfo), or
simply allow the regular implementation to proceed and update the expectations
accordingly.

* Remove references to `OidcHandler._map_userinfo_to_user` from tests

This method is going away, so we can no longer use it as a test point. Instead
we build mock "callback" requests which we pass into `handle_oidc_callback`,
and verify correct behaviour by mocking out `AuthHandler.complete_sso_login`.
2020-12-14 11:38:50 +00:00
David Teller
f14428b25c
Allow spam-checker modules to be provide async methods. (#8890)
Spam checker modules can now provide async methods. This is implemented
in a backwards-compatible manner.
2020-12-11 14:05:15 -05:00
Patrick Cloke
3af0672350
Improve tests for structured logging. (#8916) 2020-12-11 07:25:01 -05:00
Dirk Klimpel
0a34cdfc66
Add number of local devices to Room Details Admin API (#8886) 2020-12-11 10:42:47 +00:00
Erik Johnston
1d55c7b567
Don't ratelimit autojoining of rooms (#8921)
Fixes #8866
2020-12-11 10:17:49 +00:00
Patrick Cloke
344ab0b53a
Default to blacklisting reserved IP ranges and add a whitelist. (#8870)
This defaults `ip_range_blacklist` to reserved IP ranges and also adds an
`ip_range_whitelist` setting to override it.
2020-12-09 13:56:06 -05:00
Patrick Cloke
6ff34e00d9
Skip the SAML tests if xmlsec1 isn't available. (#8905) 2020-12-09 12:23:30 -05:00
Aaron Raimist
cd9e72b185
Add X-Robots-Tag header to stop crawlers from indexing media (#8887)
Fixes / related to: https://github.com/matrix-org/synapse/issues/6533

This should do essentially the same thing as a robots.txt file telling robots to not index the media repo. https://developers.google.com/search/reference/robots_meta_tag

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2020-12-08 22:51:03 +00:00
Will Hunt
ff1f0ee094
Call set_avatar_url with target_user, not user_id (#8872)
* Call set_avatar_url with target_user, not user_id

Fixes https://github.com/matrix-org/synapse/issues/8871

* Create 8872.bugfix

* Update synapse/rest/admin/users.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Testing

* Update changelog.d/8872.bugfix

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-12-07 19:13:07 +00:00
Patrick Cloke
1f3748f033
Do not raise a 500 exception when previewing empty media. (#8883) 2020-12-07 10:00:08 -05:00
Patrick Cloke
96358cb424
Add authentication to replication endpoints. (#8853)
Authentication is done by checking a shared secret provided
in the Synapse configuration file.
2020-12-04 10:56:28 -05:00
Erik Johnston
df4b1e9c74
Pass room_id to get_auth_chain_difference (#8879)
This is so that we can choose which algorithm to use based on the room ID.
2020-12-04 15:52:49 +00:00
Patrick Cloke
b774c555d8
Add additional validation to pusher URLs. (#8865)
Pusher URLs now must end in `/_matrix/push/v1/notify` per the
specification.
2020-12-04 10:51:56 -05:00
Patrick Cloke
112f6bd49e Synapse 1.24.0rc2 (2020-12-04)
==============================
 
 Bugfixes
 --------
 
 - Fix a regression in v1.24.0rc1 which failed to allow SAML mapping providers which were unable to redirect users to an additional page. ([\#8878](https://github.com/matrix-org/synapse/issues/8878))
 
 Internal Changes
 ----------------
 
 - Add support for the `prometheus_client` newer than 0.9.0. Contributed by Jordan Bancino. ([\#8875](https://github.com/matrix-org/synapse/issues/8875))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl/KQ6MACgkQM/xY9qcR
 MEjqOg/9Fd4Ow0QdnK3O6J0bmXP6+g2GfjJCAKlYlF/nwX1ez9jZVed00D2h9/hD
 YCUJCEv+XZpbv5F4usiYwoLGithbueo7AKEOQJQMEuQBHU1/E6TJ9iiKR3lTBNNw
 ccuyMMqqEatYxoa4DIO/lrwm2mhwRNdMPt0r1DPlML13bb/TVbeXXykbOCyZSKm1
 YuXpxYDngRPL9o46I77l7/KNTSpYYeZi2qwp2orl6GBfa0KSvR2Oi6uohKYzuoHC
 +XLvHSFMpHAZkaZsIMxU5hRVH6jtkb/D37CABuKQsdlvHcooGK4Sdt6aoUyhPtoh
 6YwajSazWKEJ/9YO5JV22qkWFPi9T6s/lPZfjOSu/euwMw7DDQJ/8t8Sm6N7sJDS
 A/5FctOAd1jd9BiAaqkHB2zLHZSomIHQpJJfDyE3UgAGQQ6fm2Wg7SkLByDx/MYC
 4D74dBuVBPENM2VxNoyjHbpVSeQi8t0RgWb44/PUAOtpwm3f3eVRuS8zY6uR2bZr
 YZUbJylqKhRW6dBdZH+EzVolRGvBgLkc27IzRCQ3kGUEo2Wem05s6MuKMvLxoIvN
 NHLWNClUWITT8FgkfFP3c6CK40dwIGJzYryI+71qPW/R7snf2fazuoHlH1pAmFh0
 E0/nBMTw1deixOYXbG1ah63AhJ1NjdTfimDsKmWZLqVc6g7g4d4=
 =is1Y
 -----END PGP SIGNATURE-----

Merge tag 'v1.24.0rc2' into develop

Synapse 1.24.0rc2 (2020-12-04)
==============================

Bugfixes
--------

- Fix a regression in v1.24.0rc1 which failed to allow SAML mapping providers which were unable to redirect users to an additional page. ([\#8878](https://github.com/matrix-org/synapse/issues/8878))

Internal Changes
----------------

- Add support for the `prometheus_client` newer than 0.9.0. Contributed by Jordan Bancino. ([\#8875](https://github.com/matrix-org/synapse/issues/8875))
2020-12-04 09:14:31 -05:00
Patrick Cloke
22c6c19f91
Fix a regression that mapping providers should be able to redirect users. (#8878)
This was broken in #8801.
2020-12-04 08:25:15 -05:00
Richard van der Hoff
cf3b8156be
Fix errorcode for disabled registration (#8867)
The spec says we should return `M_FORBIDDEN` when someone tries to register and
registration is disabled.
2020-12-03 15:41:19 +00:00
Richard van der Hoff
269ba1bc84 Merge remote-tracking branch 'origin/develop' into rav/remove_unused_mocks 2020-12-02 20:08:46 +00:00
Richard van der Hoff
ed5172852a
Merge pull request #8858 from matrix-org/rav/sso_uia
UIA: offer only available auth flows
2020-12-02 20:06:53 +00:00
Richard van der Hoff
f347f0cd58
remove unused FakeResponse (#8864) 2020-12-02 18:58:25 +00:00
Richard van der Hoff
0bac276890 UIA: offer only available auth flows
During user-interactive auth, do not offer password auth to users with no
password, nor SSO auth to users with no SSO.

Fixes #7559.
2020-12-02 18:54:15 +00:00
Richard van der Hoff
b751624ff8 remove unused DeferredMockCallable 2020-12-02 18:38:29 +00:00
Richard van der Hoff
c834f1d67a remove unused resource_for_federation
This is now only used in `test_typing`, so move it there.
2020-12-02 18:38:29 +00:00
Richard van der Hoff
76469898ee Factor out FakeResponse from test_oidc 2020-12-02 18:30:29 +00:00
Richard van der Hoff
90cf1eec44 Remove redundant mocking 2020-12-02 17:53:38 +00:00
Richard van der Hoff
7ea85302f3 fix up various test cases
A few test cases were relying on being able to mount non-client servlets on the
test resource. it's better to give them their own Resources.
2020-12-02 16:30:01 +00:00
Patrick Cloke
30fba62108
Apply an IP range blacklist to push and key revocation requests. (#8821)
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
Erik Johnston
c5b6abd53d
Correctly handle unpersisted events when calculating auth chain difference. (#8827)
We do state res with unpersisted events when calculating the new current state of the room, so that should be the only thing impacted. I don't think this is tooooo big of a deal as:

1. the next time a state event happens in the room the current state should correct itself;
2. in the common case all the unpersisted events' auth events will be pulled in by other state, so will still return the correct result (or one which is sufficiently close to not affect the result); and
3. we mostly use the state at an event to do important operations, which isn't affected by this.
2020-12-02 15:22:37 +00:00
Richard van der Hoff
693516e756 Add create_resource_dict method to HomeserverTestCase
Rather than using a single JsonResource, construct a resource tree, as we do in
the prod code, and allow testcases to add extra resources by overriding
`create_resource_dict`.
2020-12-02 15:21:00 +00:00
Patrick Cloke
8388384a64
Fix a regression when grandfathering SAML users. (#8855)
This was broken in #8801 when abstracting code shared with OIDC.

After this change both SAML and OIDC have a concept of
grandfathering users, but with different implementations.
2020-12-02 07:45:42 -05:00
Patrick Cloke
c21bdc813f
Add basic SAML tests for mapping users. (#8800) 2020-12-02 07:09:21 -05:00
Richard van der Hoff
d3ed93504b
Create a PasswordProvider wrapper object (#8849)
The idea here is to abstract out all the conditional code which tests which
methods a given password provider has, to provide a consistent interface.
2020-12-02 10:38:50 +00:00
Andrew Morgan
edb3d3f827
Allow specifying room version in 'RestHelper.create_room_as' and add typing (#8854)
This PR adds a `room_version` argument to the `RestHelper`'s `create_room_as` function for tests. I plan to use this for testing knocking, which currently uses an unstable room version.
2020-12-02 10:38:18 +00:00
Richard van der Hoff
4d9496559d
Support "identifier" dicts in UIA (#8848)
The spec requires synapse to support `identifier` dicts for `m.login.password`
user-interactive auth, which it did not (instead, it required an undocumented
`user` parameter.)

To fix this properly, we need to pull the code that interprets `identifier`
into `AuthHandler.validate_login` so that it can be called from the UIA code.

Fixes #5665.
2020-12-01 17:42:26 +00:00
Richard van der Hoff
89f7930730
Don't offer password login when it is disabled (#8835)
Fix a minor bug where we would offer "m.login.password" login if a custom auth provider supported it, even if password login was disabled.
2020-12-01 13:04:03 +00:00
Richard van der Hoff
ddc4343683
Add some tests for password_auth_providers (#8819)
These things seemed to be completely untested, so I added a load of tests for
them.
2020-12-01 11:10:42 +00:00
Richard van der Hoff
09ac0569fe
Fix broken testcase (#8851)
This test was broken by #8565. It doesn't need to set set `self.clock`
here anyway - that is done by `setUp`.
2020-12-01 11:04:57 +00:00
Andrew Morgan
17fa58bdd1
Add a config option to change whether unread push notification counts are per-message or per-room (#8820)
This PR adds a new config option to the `push` section of the homeserver config, `group_unread_count_by_room`. By default Synapse will group push notifications by room (so if you have 1000 unread messages, if they lie in 55 rooms, you'll see an unread count on your phone of 55).

However, it is also useful to be able to send out the true count of unread messages if desired. If `group_unread_count_by_room` is set to `false`, then with the above example, one would see an unread count of 1000 (email anyone?).
2020-11-30 18:43:54 +00:00
Jonathan de Jong
ca60822b34
Simplify the way the HomeServer object caches its internal attributes. (#8565)
Changes `@cache_in_self` to use underscore-prefixed attributes.
2020-11-30 13:28:44 -05:00
Dirk Klimpel
3f0ff53158
Remove deprecated /_matrix/client/*/admin endpoints (#8785)
These are now only available via `/_synapse/admin/v1`.
2020-11-25 16:26:11 -05:00
Patrick Cloke
4fd222ad70
Support trying multiple localparts for OpenID Connect. (#8801)
Abstracts the SAML and OpenID Connect code which attempts to regenerate
the localpart of a matrix ID if it is already in use.
2020-11-25 10:04:22 -05:00
Patrick Cloke
79bfe966e0
Improve error checking for OIDC/SAML mapping providers (#8774)
Checks that the localpart returned by mapping providers for SAML and
OIDC are valid before registering new users.

Extends the OIDC tests for existing users and invalid data.
2020-11-19 14:25:17 -05:00
Richard van der Hoff
950bb0305f
Consistently use room_id from federation request body (#8776)
* Consistently use room_id from federation request body

Some federation APIs have a redundant `room_id` path param (see
https://github.com/matrix-org/matrix-doc/issues/2330). We should make sure we
consistently use either the path param or the body param, and the body param is
easier.

* Kill off some references to "context"

Once upon a time, "rooms" were known as "contexts". I think this kills of the
last references to "contexts".
2020-11-19 10:05:33 +00:00
Erik Johnston
03e392f787
Fix tests on develop (#8777)
This was broken due to #8617 and #8761.
2020-11-18 15:43:11 +00:00
Richard van der Hoff
deff8f628d
Merge pull request #8761 from matrix-org/rav/test_request_rendering
Make `make_request` actually render the request
2020-11-17 15:17:04 +00:00
Patrick Cloke
ee382025b0
Abstract shared SSO code. (#8765)
De-duplicates code between the SAML and OIDC implementations.
2020-11-17 09:46:23 -05:00
Erik Johnston
f737368a26
Add admin API for logging in as a user (#8617) 2020-11-17 10:51:25 +00:00
Richard van der Hoff
acfe3b3065 Remove redundant HomeserverTestCase.render 2020-11-16 18:24:08 +00:00
Richard van der Hoff
be8fa65d0b Remove redundant calls to render() 2020-11-16 18:24:08 +00:00
Richard van der Hoff
129ae841e5 Make make_request actually render the request
remove the stubbing out of `request.process`, so that `requestReceived` also renders the request via the appropriate resource.

Replace render() with a stub for now.
2020-11-16 18:24:00 +00:00
Richard van der Hoff
1f41422c98 Fix the URL in the URL preview tests
the preview resource is mointed at preview_url, not url_preview
2020-11-16 18:24:00 +00:00
Richard van der Hoff
3dc1871219
Merge pull request #8757 from matrix-org/rav/pass_site_to_make_request
Pass a Site into `make_request`
2020-11-16 18:22:24 +00:00