Commit Graph

2288 Commits

Author SHA1 Message Date
Patrick Cloke
16ec8c3272
(Hopefully) stop leaking file descriptors in media repo. (#9497)
By consuming the response if the headers imply that the
content is too large.
2021-03-01 12:45:00 -05:00
Tim Leung
ddb240293a
Add support for no_proxy and case insensitive env variables (#9372)
### Changes proposed in this PR

- Add support for the `no_proxy` and `NO_PROXY` environment variables
  - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519))
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment

This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`

Related to #9306 which also uses `ProxyAgent`

Signed-off-by: Timothy Leung tim95@hotmail.co.uk
2021-02-26 17:37:57 +00:00
Richard van der Hoff
15090de850
SSO: redirect to public URL before setting cookies (#9436)
... otherwise, we don't get the cookie back.
2021-02-26 14:02:06 +00:00
Erik Johnston
2566dc57ce
Test that we require validated email for email pushers (#9496) 2021-02-25 15:35:14 +00:00
Erik Johnston
2927921942
Clean up ShardedWorkerHandlingConfig (#9466)
* Split ShardedWorkerHandlingConfig

This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).

* Remove special cases in ShardedWorkerHandlingConfig.

`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.

To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
2021-02-24 13:23:18 +00:00
Richard van der Hoff
1b2d6d55c5
Remove vestiges of uploads_path config (#9462)
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
2021-02-22 19:54:49 +00:00
Dirk Klimpel
71c9f8de6d
Add an order_by field to list users' media admin API. (#8978) 2021-02-22 14:38:51 -05:00
Andrew Morgan
0a363f9ca4
Remove cache for get_shared_rooms_for_users (#9416)
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.

The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately.

Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
2021-02-22 16:52:45 +00:00
Erik Johnston
179c0953ff
Regenerate exact thumbnails if missing (#9438) 2021-02-19 17:09:57 +00:00
Erik Johnston
3a2fe5054f Add test 2021-02-19 15:52:04 +00:00
Patrick Cloke
c4a55ac4a4 Fix style checking due to updated black. 2021-02-19 08:19:54 -05:00
Andrew Morgan
8bcfc2eaad
Be smarter about which hosts to send presence to when processing room joins (#9402)
This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually.

---

When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed.

It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw.

This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence:

* If it was a local user join, send that user's latest presence to all servers in the room
* If it was a remote user join, send the presence for all local users in the room to that homeserver

We deduplicate by inserting all of those pending updates into a dictionary of the form:

```
{
  server_name1: {presence_update1, ...},
  server_name2: {presence_update1, presence_update2, ...}
}
```

Only after building this dict do we then start sending out presence updates.
2021-02-19 11:37:29 +00:00
Andrew Morgan
13e9029f44
Add a config option to prioritise local users in user directory search results (#9383)
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.

Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.

This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
2021-02-19 11:02:03 +00:00
AndrewFerr
9bc74743d5
Add configs to make profile data more private (#9203)
Add off-by-default configuration settings to:
- disable putting an invitee's profile info in invite events
- disable profile lookup via federation

Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-02-19 09:50:41 +00:00
Rishabh Arya
e17553e185
Parse ui_auth.session_timeout as a duration (instead of treating it as ms) (#9426) 2021-02-18 09:18:14 -05:00
Patrick Cloke
8ec2217103
Reduce the memory usage of previewing media files. (#9421)
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.

It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
2021-02-18 09:01:29 -05:00
Dirk Klimpel
c8d9383cfb
Add the shadow-banning status to the display user admin API. (#9400) 2021-02-17 15:19:23 -05:00
Richard van der Hoff
e1071fd625
Support for form_post in OIDC responses (#9376)
Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
2021-02-17 10:15:14 +00:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Richard van der Hoff
3b754aea27
Clean up caching/locking of OIDC metadata load (#9362)
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
2021-02-16 16:27:38 +00:00
Patrick Cloke
731e08c63a
Handle missing data in power levels events during room upgrade. (#9395) 2021-02-16 08:31:39 -05:00
Patrick Cloke
74af356baf
Convert additional test-cases to homeserver test case. (#9396)
And convert some inlineDeferreds to async-friendly functions.
2021-02-16 08:04:15 -05:00
Patrick Cloke
2c9b4a5f16 Synapse 1.27.0rc2 (2021-02-11)
==============================
 
 Features
 --------
 
 - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
 
 Bugfixes
 --------
 
 - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
 - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
 
 Improved Documentation
 ----------------------
 
 - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR
 MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS
 /tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN
 309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL
 GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb
 /QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP
 xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn
 ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B
 iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4
 awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU
 2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H
 lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0=
 =NSyp
 -----END PGP SIGNATURE-----

Merge tag 'v1.27.0rc2' into develop

Synapse 1.27.0rc2 (2021-02-11)
==============================

Features
--------

- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))

Bugfixes
--------

- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))

Improved Documentation
----------------------

- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
2021-02-11 11:56:03 -05:00
Patrick Cloke
e40d88cff3
Backout changes for automatically calculating the public baseurl. (#9313)
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
2021-02-11 11:16:54 -05:00
Patrick Cloke
8a33d217bd
Convert some test cases to use HomeserverTestCase. (#9377)
This has the side-effect of being able to remove use of `inlineCallbacks`
in the test-cases for cleaner tracebacks.
2021-02-11 10:29:09 -05:00
Patrick Cloke
6dade80048
Combine the CAS & SAML implementations for required attributes. (#9326) 2021-02-11 10:05:15 -05:00
Patrick Cloke
0963d39ea6
Handle additional errors when previewing URLs. (#9333)
* Handle the case of lxml not finding a document tree.
* Parse the document encoding from the XML tag.
2021-02-08 12:33:30 -05:00
David Teller
b0b2cac057
Merge pull request #9150 from Yoric/develop-context
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
2021-02-08 15:53:44 +01:00
Erik Johnston
adc96d4236 Merge branch 'erikj/media_spam_checker' into develop 2021-02-04 17:01:59 +00:00
Erik Johnston
7e8083eb48 Add check_media_file_for_spam spam checker hook 2021-02-04 17:01:30 +00:00
Patrick Cloke
792263c97c
Handle empty rooms when generating email notifications. (#9257)
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
2021-02-04 10:18:25 -05:00
Richard van der Hoff
3f534d3fdf Merge branch 'social_login_hotfixes' into develop 2021-02-03 20:34:27 +00:00
Richard van der Hoff
17f2a512f3 Merge remote-tracking branch 'origin/release-v1.27.0' into social_login_hotfixes 2021-02-03 20:33:32 +00:00
Richard van der Hoff
e288499c60
Social login UI polish (#9301) 2021-02-03 20:31:23 +00:00
Patrick Cloke
4ca054a4ea
Convert blacklisted IPv4 addresses to compatible IPv6 addresses. (#9240)
Also add a few more IP ranges to the default blacklist.
2021-02-03 07:13:46 -05:00
Erik Johnston
ff55300b91
Honour ratelimit flag for application services for invite ratelimiting (#9302) 2021-02-03 10:17:37 +00:00
Travis Ralston
b60bb28bbc
Add an admin API to get the current room state (#9168)
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.

This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
2021-02-02 11:16:29 +00:00
Richard van der Hoff
846b9d3df0
Put OIDC callback URI under /_synapse/client. (#9288) 2021-02-01 22:56:01 +00:00
Patrick Cloke
5d38a3c97f
Refactor email summary generation. (#9260)
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
  was tweaked to depend on whether there was 1 sender or multiple senders,
  not based on if there was 1 room or multiple rooms.
2021-02-01 13:09:39 -05:00
Richard van der Hoff
18ab35284a Merge branch 'social_login' into develop 2021-02-01 17:28:37 +00:00
Richard van der Hoff
4167494c90
Replace username picker with a template (#9275)
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.

I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
2021-02-01 15:52:50 +00:00
Richard van der Hoff
8aed29dc61
Improve styling and wording of SSO redirect confirm template (#9272) 2021-02-01 15:50:56 +00:00
Richard van der Hoff
9c715a5f19
Fix SSO on workers (#9271)
Fixes #8966.

* Factor out build_synapse_client_resource_tree

Start a function which will mount resources common to all workers.

* Move sso init into build_synapse_client_resource_tree

... so that we don't have to do it for each worker

* Fix SSO-login-via-a-worker

Expose the SSO login endpoints on workers, like the documentation says.

* Update workers config for new endpoints

Add documentation for endpoints recently added (#8942, #9017, #9262)

* remove submit_token from workers endpoints list

this *doesn't* work on workers (yet).

* changelog

* Add a comment about the odd path for SAML2Resource
2021-02-01 15:47:59 +00:00
Richard van der Hoff
f78d07bf00
Split out a separate endpoint to complete SSO registration (#9262)
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
2021-02-01 13:15:51 +00:00
Erik Johnston
f2c1560eca
Ratelimit invites by room and target user (#9258) 2021-01-29 16:38:29 +00:00
Erik Johnston
4b73488e81
Ratelimit 3PID /requestToken API (#9238) 2021-01-28 17:39:21 +00:00
David Teller
31d072aea0 FIXUP: linter 2021-01-28 16:53:40 +01:00
David Teller
b859919acc FIXUP: Now testing that the user is admin! 2021-01-28 12:31:07 +01:00
David Teller
10332c175c New API /_synapse/admin/rooms/{roomId}/context/{eventId}
Signed-off-by: David Teller <davidt@element.io>
2021-01-28 12:29:49 +01:00
Richard van der Hoff
7fa1346f93 Merge branch 'social_login' into develop 2021-01-27 17:27:24 +00:00
Richard van der Hoff
a737cc2713
Implement MSC2858 support (#9183)
Fixes #8928.
2021-01-27 12:41:24 +00:00
Erik Johnston
dd8da8c5f6
Precompute joined hosts and store in Redis (#9198) 2021-01-26 13:57:31 +00:00
Patrick Cloke
4937fe3d6b
Try to recover from unknown encodings when previewing media. (#9164)
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
2021-01-26 07:32:17 -05:00
Patrick Cloke
4a55d267ee
Add an admin API for shadow-banning users. (#9209)
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.

A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.

Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
2021-01-25 14:49:39 -05:00
Patrick Cloke
e448dbbf5b Synapse 1.26.0rc2 (2021-01-25)
==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
 MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
 +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
 BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
 d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
 oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
 O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
 VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
 g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
 =Qc7+
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc2' into develop

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
2021-01-25 08:51:45 -05:00
Patrick Cloke
6f7417c3db
Handle missing content keys when calculating presentable names. (#9165)
Treat the content as untrusted and do not assume it is of
the proper form.
2021-01-25 07:27:16 -05:00
Erik Johnston
056327457f
Fix chain cover update to handle events with duplicate auth events (#9210) 2021-01-22 19:44:08 +00:00
Patrick Cloke
a7882f9887
Return a 404 if no valid thumbnail is found. (#9163)
If no thumbnail of the requested type exists, return a 404 instead
of erroring. This doesn't quite match the spec (which does not define
what happens if no thumbnail can be found), but is consistent with
what Synapse already does.
2021-01-21 14:53:58 -05:00
Dirk Klimpel
c55e62548c
Add tests for List Users Admin API (#9045) 2021-01-21 09:18:46 -05:00
Richard van der Hoff
7447f19702
Prefix idp_id with "oidc-" (#9189)
... to avoid clashes with other SSO mechanisms
2021-01-21 12:25:02 +00:00
Richard van der Hoff
0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Richard van der Hoff
fa50e4bf4d
Give public_baseurl a default value (#9159) 2021-01-20 12:30:41 +00:00
Erik Johnston
6633a4015a
Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
Richard van der Hoff
02070c69fa
Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128)
* Factor out a common TestHtmlParser

Looks like I'm doing this in a few different places.

* Improve OIDC login test

Complete the OIDC login flow, rather than giving up halfway through.

* Ensure that OIDC login works with multiple OIDC providers

* Fix bugs in handling clientRedirectUrl

 - don't drop duplicate query-params, or params with no value
 - allow utf-8 in query-params
2021-01-18 14:52:49 +00:00
Richard van der Hoff
b5dea8702d Fix test failure due to bad merge
0dd2649c1 (#9112) changed the signature of `auth_via_oidc`. Meanwhile,
26d10331e (#9091) introduced a new test which relied on the old signature of
`auth_via_oidc`. The two branches were never tested together until they landed
in develop.
2021-01-15 18:03:33 +00:00
Erik Johnston
350d9923cd
Make chain cover index bg update go faster (#9124)
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
2021-01-15 17:18:37 +00:00
Richard van der Hoff
9de6b94117
Land support for multiple OIDC providers (#9110)
This is the final step for supporting multiple OIDC providers concurrently.

First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before:

    oidc_config:
       enabled: true
       issuer: "https://oidc_provider"
       # etc

After:

    oidc_providers:
     - idp_id: prov1
       issuer: "https://oidc_provider"

     - idp_id: prov2
       issuer: "https://another_oidc_provider"

The old format is still grandfathered in.

With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
2021-01-15 16:55:29 +00:00
Patrick Cloke
3e4cdfe5d9
Add an admin API endpoint to protect media. (#9086)
Protecting media stops it from being quarantined when
e.g. all media in a room is quarantined. This is useful
for sticker packs and other media that is uploaded by
server administrators, but used by many people.
2021-01-15 11:18:09 -05:00
Patrick Cloke
74dd906041
Avoid raising the body exceeded error multiple times. (#9108)
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
2021-01-15 11:00:13 -05:00
Richard van der Hoff
0dd2649c12
Improve UsernamePickerTestCase (#9112)
* make the OIDC bits of the test work at a higher level - via the REST api instead of poking the OIDCHandler directly.
* Move it to test_login.py, where I think it fits better.
2021-01-15 13:45:13 +00:00
Richard van der Hoff
4575ad0b1e
Store an IdP ID in the OIDC session (#9109)
Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP.

As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
2021-01-15 13:22:12 +00:00
Richard van der Hoff
14950a45d6
Merge pull request #9091 from matrix-org/rav/error_on_bad_sso
Give the user a better error when they present bad SSO creds
2021-01-15 00:27:13 +00:00
Erik Johnston
1a08e0cdab
Fix event chain bg update. (#9118)
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
2021-01-14 18:57:32 +00:00
Erik Johnston
7036e24e98
Add background update for add chain cover index (#9029) 2021-01-14 15:18:27 +00:00
Richard van der Hoff
21a296cd5a
Split OidcProvider out of OidcHandler (#9107)
The idea here is that we will have an instance of OidcProvider for each
configured IdP, with OidcHandler just doing the marshalling of them.

For now it's still hardcoded with a single provider.
2021-01-14 13:29:17 +00:00
Richard van der Hoff
26d10331e5 Add a test for wrong user returned by SSO 2021-01-13 20:22:41 +00:00
Richard van der Hoff
233c8b9fce
Add a test for UI-Auth-via-SSO (#9082)
* Add complete test for UI-Auth-via-SSO.

* review comments
2021-01-13 20:21:55 +00:00
Patrick Cloke
98a64b7f7f
Add basic domain validation for DomainSpecificString.is_valid. (#9071)
This checks that the domain given to `DomainSpecificString.is_valid` (e.g.
`UserID`, `RoomAlias`, etc.) is of a valid form. Previously some validation
was done on the localpart (e.g. the sigil), but not the domain portion.
2021-01-13 07:05:16 -05:00
Erik Johnston
aa4d8c1f9a Merge branch 'master' into develop 2021-01-13 10:36:55 +00:00
Richard van der Hoff
bc4bf7b384
Preparatory refactors of OidcHandler (#9067)
Some light refactoring of OidcHandler, in preparation for bigger things:

  * remove inheritance from deprecated BaseHandler
  * add an object to hold the things that go into a session cookie
  * factor out a separate class for manipulating said cookies
2021-01-13 10:26:12 +00:00
Dirk Klimpel
7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932)
This only applies if the user's data is to be erased.
2021-01-12 16:30:15 -05:00
Marcus
e385c8b473
Don't apply the IP range blacklist to proxy connections (#9084)
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
2021-01-12 12:20:30 -05:00
Patrick Cloke
723b19748a
Handle bad JSON data being returned from the federation API. (#9070) 2021-01-12 11:07:01 -05:00
Richard van der Hoff
2ec8ca5e60
Remove SynapseRequest.get_user_agent (#9069)
SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests",
which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the 
LoggingContext and write the right entries to the request log).

Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a
SynapseRequest when there is nothing synapse-specific about the Request at all, and any old
twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult.

In short: move get_user_agent out to a utility function.
2021-01-12 12:34:16 +00:00
David Teller
b161528fcc
Also support remote users on the joined_rooms admin API. (#8948)
For remote users, only the rooms which the server knows about are returned.
Local users have all of their joined rooms returned.
2021-01-11 14:32:17 -05:00
Dirk Klimpel
42d3a28d8b
Removes unnecessary declarations in the tests for the admin API. (#9063) 2021-01-11 11:15:54 -05:00
Erik Johnston
1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00
Richard van der Hoff
12f79da587
Merge pull request #9036 from matrix-org/rav/multi_idp/tests
Add tests for the IdP picker
2021-01-08 14:24:41 +00:00
Richard van der Hoff
d32870ffa5
Fix validate_config on nested objects (#9054) 2021-01-08 14:23:04 +00:00
Richard van der Hoff
8a910f97a4 Add some tests for the IDP picker flow 2021-01-07 14:56:42 +00:00
Richard van der Hoff
bbd04441ed Fix type hints in test_login.py 2021-01-07 14:56:42 +00:00
Patrick Cloke
23d701864f
Improve the performance of calculating ignored users in large rooms (#9024)
This allows for efficiently finding which users ignore a particular
user.

Co-authored-by: Erik Johnston <erik@matrix.org>
2021-01-07 13:03:38 +00:00
Richard van der Hoff
3fc2399dbe black-format tests/rest/client/v1/test_login.py
black seems to want to reformat this, despite `black --check` being happy with
it :/
2021-01-07 12:17:37 +00:00
Richard van der Hoff
8d3d264052
Skip unit tests which require optional dependencies (#9031)
If we are lacking an optional dependency, skip the tests that rely on it.
2021-01-07 11:41:28 +00:00
Patrick Cloke
1b4d5d6acf
Empty iterables should count towards cache usage. (#9028) 2021-01-06 12:33:20 -05:00
Patrick Cloke
9dde9c9f01
Implement MSC2176: Updated redaction rules (#8984)
An experimental room version ("org.matrix.msc2176") contains
the new redaction rules for testing.
2021-01-05 07:41:48 -05:00
Richard van der Hoff
d2c616a413
Combine the SSO Redirect Servlets (#9015)
* Implement CasHandler.handle_redirect_request

... to make it match OidcHandler and SamlHandler

* Clean up interface for OidcHandler.handle_redirect_request

Make it accept `client_redirect_url=None`.

* Clean up interface for `SamlHandler.handle_redirect_request`

... bring it into line with CAS and OIDC by making it take a Request parameter,
move the magic for `client_redirect_url` for UIA into the handler, and fix the
return type to be a `str` rather than a `bytes`.

* Define a common protocol for SSO auth provider impls

* Give SsoIdentityProvider an ID and register them

* Combine the SSO Redirect servlets

Now that the SsoHandler knows about the identity providers, we can combine the
various *RedirectServlets into a single implementation which delegates to the
right IdP.

* changelog
2021-01-04 18:13:49 +00:00
Patrick Cloke
1c9a850562
Add type hints to the crypto module. (#8999) 2021-01-04 10:04:50 -05:00
Patrick Cloke
0eccf53146
Use the SSO handler helpers for CAS registration/login. (#8856) 2021-01-03 16:25:44 +00:00
Andrew Morgan
168ba00d01
Fix RoomDirectoryFederationTests and make them actually run (#8998)
The `RoomDirectoryFederationTests` tests were not being run unless explicitly called as an `__init__.py` file was not present in `tests/federation/transport/`. Thus the folder was not a python module, and `trial` did not look inside for any test cases to run. This was found while working on #6739.

This PR adds a `__init__.py` and also fixes the test in a couple ways:

- Switch to subclassing `unittest.FederatingHomeserverTestCase` instead, which sets up federation endpoints for us.
- Supply a `federation_auth_origin` to `make_request` in order to more act like the request is coming from another server, instead of just an unauthenicated client requesting a federation endpoint.

I found that the second point makes no difference to the test passing, but felt like the right thing to do if we're testing over federation.
2020-12-30 19:27:32 +00:00