Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-12-30 15:26:12 -05:00
Code Issues Packages Projects Releases Wiki Activity
19,933 Commits 4 Branches 615 Tags 63 MiB
cbd82d0b2d
Commit Graph

13983 Commits

Author SHA1 Message Date
Jonathan de Jong
95e47b2e78
[pyupgrade] synapse/ (#10348) 
This PR is tantamount to running 
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```

Part of #9744
 2021-07-19 15:28:05 +01:00
Eric Eastwood
7387d6f624
Remove unused events_by_room (#10421) 
It looks like it was first used and introduced in 5130d80d79 (diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8R305) but the 

But the usage was removed in 4c6a31cd6e (diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8)
 2021-07-19 10:16:46 +01:00
Jonathan de Jong
323452944e
One last inline type hint (for the whole repo) (#10418) 2021-07-16 20:12:56 +01:00
Jonathan de Jong
98aec1cc9d
Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
Brendan Abolivier
36dc15412d
Add a module type for account validity (#9884) 
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity.

To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
 2021-07-16 18:11:53 +02:00
Patrick Cloke
d427f64724
Do not include signatures/hashes in make_{join,leave,knock} responses. (#10404) 
These signatures would end up invalid since the joining/leaving/knocking
server would modify the response before calling send_{join,leave,knock}.
 2021-07-16 10:36:38 -04:00
Jonathan de Jong
bdfde6dca1
Use inline type hints in http/federation/, storage/ and util/ (#10381) 2021-07-15 12:46:54 -04:00
Erik Johnston
3acf85c85f
Reduce likelihood of Postgres table scanning state_groups_state. (#10359) 
The postgres statistics collector sometimes massively underestimates the
number of distinct state groups are in the `state_groups_state`, which
can cause postgres to use table scans for queries for multiple state
groups.

We fix this by manually setting `n_distinct` on the column.
 2021-07-15 16:02:12 +01:00
Erik Johnston
ac5c221208
Stagger send presence to remotes (#10398) 
This is to help with performance, where trying to connect to thousands
of hosts at once can consume a lot of CPU (due to TLS etc).

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2021-07-15 11:52:56 +01:00
Jonathan de Jong
bf72d10dbf
Use inline type hints in various other places (in synapse/) (#10380) 2021-07-15 11:02:43 +01:00
Dirk Klimpel
c7603af1d0
Allow providing credentials to http_proxy (#10360) 2021-07-15 10:37:08 +01:00
Erik Johnston
7695ca0618
Fix a number of logged errors caused by remote servers being down. (#10400) 2021-07-15 10:35:46 +01:00
Patrick Cloke
eb3beb8f12
Add type hints and comments to event auth code. (#10393) 2021-07-14 14:13:40 +01:00
Eric Eastwood
0d5b08ac7a
Fix messages from multiple senders in historical chunk (MSC2716) (#10276) 
Fix messages from multiple senders in historical chunk. This also means that an app service does not need to define `?user_id` when using this endpoint.

Follow-up to https://github.com/matrix-org/synapse/pull/9247

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
 2021-07-13 14:12:33 -05:00
Patrick Cloke
30b56f6925
Add type hints to get_domain_from_id and get_localpart_from_id. (#10385) 2021-07-13 12:08:47 -04:00
Patrick Cloke
2d16e69b4b
Show all joinable rooms in the spaces summary. (#10298) 
Previously only world-readable rooms were shown. This means that
rooms which are public, knockable, or invite-only with a pending invitation,
are included in a space summary. It also applies the same logic to
the experimental room version from MSC3083 -- if a user has access
to the proper allowed rooms then it is shown in the spaces summary.

This change is made per MSC3173 allowing stripped state of a room to
be shown to any potential room joiner.
 2021-07-13 08:59:27 -04:00
Richard van der Hoff
475fcb0f20 Merge branch 'master' into develop 2021-07-13 13:30:16 +01:00
Richard van der Hoff
c647c2a9ac 1.38.0 2021-07-13 13:19:06 +01:00
Richard van der Hoff
f7bfa694ae 1.38.0rc3 2021-07-13 11:57:55 +01:00
Dirk Klimpel
d9b3637e44
Bugfix make_room_admin fails for users that have left a private room (#10367) 
Fixes: #10338
 2021-07-13 11:53:45 +01:00
Erik Johnston
879d8c1ee1
Fix federation inbound age metric. (#10355) 
We should be reporting the age rather than absolute timestamp.
 2021-07-13 11:33:15 +01:00
Richard van der Hoff
c2c364f27f
Replace room_depth.min_depth with a BIGINT (#10289) 
while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth
with a BIGINT.
 2021-07-12 17:22:54 +01:00
reivilibre
ca9dface8c
Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. (#10344) 
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
 2021-07-09 14:12:47 +01:00
Richard van der Hoff
751372fa61
Switch application_services_txns.txn_id to BIGINT (#10349) 2021-07-09 13:01:11 +01:00
Erik Johnston
251cfc4e09 Synapse 1.38.0rc2 (2021-07-09) 
==============================
 
 Bugfixes
 --------
 
 - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336))
 
 Improved Documentation
 ----------------------
 
 - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287))
 - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmDoH+4QHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCXxYCACneuRvkdvYqiH+PhPe8tXqhhJIifH1LecY
 FlJqp4OJPR2VFzio1btsgpRPQyLBLHZkJ9pgWsXAETbYOO+hSeOc4nIHsyqlSJhe
 v01sCUE4sle3DBrw15fG4XpercsiM3TFMyR9pV9laq9nIn8j+CY5K6W5t12/mYGy
 asHS0IKilCMhJlFwgE3eBr6P6fywi0JoIrr8EpfIs4eC2qDFpUlsrAQSkbE1JvdP
 O4BGZJKVysg3a6WYSWdJytqLYe942k8qUF4B4h4VmQi0xbuKSsTLiK/cFC8ohRMv
 E+O5O/KgwqwE/XOcukbsjlHxuiiFZTq6154PwLxXUpNnsMNn2/ph
 =6iBw
 -----END PGP SIGNATURE-----

Merge tag 'v1.38.0rc2' into develop

Synapse 1.38.0rc2 (2021-07-09)
==============================

Bugfixes
--------

- Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336))

Improved Documentation
----------------------

- Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287))
- Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))
 2021-07-09 11:26:17 +01:00
Andreas Rammhold
e3e73e181b
Upsert redactions in case they already exists (#10343) 
* Upsert redactions in case they already exists

Occasionally, in combination with retention, redactions aren't deleted
from the database whenever they are due for deletion. The server will
eventually try to backfill the deleted events and trip over the already
existing redaction events.

Switching to an UPSERT for those events allows us to recover from there
situations. The retention code still needs fixing but that is outside of
my current comfort zone on this code base.

This is related to #8707 where the error was discussed already.

Signed-off-by: Andreas Rammhold <andreas@rammhold.de>

* Also purge redactions when purging events

Previously redacints where left behind leading to backfilling issues
when the server stumbled across the already existing yet to be
backfilled redactions.

This issues has been discussed in #8707.

Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
 2021-07-09 11:03:02 +01:00
Erik Johnston
717a07b73f 1.38.0rc2 2021-07-09 10:59:28 +01:00
Erik Johnston
1579fdd54a
Ensure we always drop the federation inbound lock (#10336) 2021-07-09 10:16:54 +01:00
Eric Eastwood
d26094e92c
Add base starting insertion event when no chunk ID is provided (MSC2716) (#10250) 
* Add base starting insertion point when no chunk ID is provided

This is so we can have the marker event point to this initial
insertion event and be able to traverse the events in the first chunk.
 2021-07-08 20:25:59 -05:00
Cristina
f6767abc05
Remove functionality associated with unused historical stats tables (#9721) 
Fixes #9602
 2021-07-08 16:57:13 +01:00
reivilibre
aa78064869
Minor changes to user_daily_visits (#10324) 
* Use fake time in tests in _get_start_of_day.

* Change the inequality of last_seen in user_daily_visits

Co-authored-by: Erik Johnston <erik@matrix.org>
 2021-07-08 14:27:12 +01:00
Brendan Abolivier
225be77787
Rebuild event auth when rebuilding an event after a call to a ThirdPartyEventRules module (#10316) 
Because modules might send extra state events when processing an event (e.g. matrix-org/synapse-dinsic#100), and in some cases these extra events might get dropped if we don't recalculate the initial event's auth.
 2021-07-08 13:00:05 +02:00
Richard van der Hoff
240b3ce253 Merge remote-tracking branch 'origin/release-v1.38' into develop 
merge @dklimpel's fixes to the sample config
 2021-07-07 12:37:10 +01:00
Dirk Klimpel
56fd5fa8e1
Update links to documentation in sample config (#10287) 
Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2021-07-07 12:35:45 +01:00
Richard van der Hoff
4358f51bb6 Merge branch 'release-v1.38' into develop 2021-07-07 11:09:49 +01:00
Brendan Abolivier
9ad8455895
ANALYZE new stream ordering column (#10326) 
Fixes #10325
 2021-07-07 11:56:17 +02:00
Dagfinn Ilmari Mannsåker
47e28b4031
Ignore EDUs for rooms we're not in (#10317) 2021-07-06 14:31:13 +01:00
Brendan Abolivier
37da9db082 1.38.0rc1 2021-07-06 13:54:23 +01:00
Dirk Klimpel
bcb0962a72
Fix deactivate a user if he does not have a profile (#10252) 2021-07-06 13:08:53 +01:00
Erik Johnston
6655ea5587
Add script for getting info about recently registered users (#10290) 2021-07-06 13:03:16 +01:00
Erik Johnston
c65067d673
Handle old staged inbound events (#10303) 
We might have events in the staging area if the service was restarted while there were unhandled events in the staging area.

Fixes #10295
 2021-07-06 13:02:37 +01:00
Erik Johnston
7a5873277e
Add support for evicting cache entries based on last access time. (#10205) 2021-07-05 16:32:12 +01:00
Patrick Cloke
8d609435c0
Move methods involving event authentication to EventAuthHandler. (#10268) 
Instead of mixing them with user authentication methods.
 2021-07-01 14:25:37 -04:00
Richard van der Hoff
0aab50c772
fix ordering of bg update (#10291) 
this was a typo introduced in #10282. We don't want to end up doing the
`replace_stream_ordering_column` update after anything that comes up in
migration 60/03.
 2021-07-01 18:45:55 +01:00
Dirk Klimpel
6c02cca95f
Add SSO external_ids to Query User Account admin API (#10261) 
Related to #10251
 2021-07-01 11:26:24 +02:00
Erik Johnston
76addadd7c
Add some metrics to staging area (#10284) 2021-07-01 10:18:25 +01:00
Richard van der Hoff
b6dbf89fae
Change more stream_ordering columns to BIGINT (#10286) 2021-06-30 17:27:20 +01:00
Richard van der Hoff
859dc05b36
Rebuild other indexes using stream_ordering (#10282) 
We need to rebuild *all* of the indexes that use the current `stream_ordering`
column.
 2021-06-30 15:01:24 +01:00
Erik Johnston
e6f5b9359f Merge branch 'master' into develop 2021-06-30 14:57:33 +01:00
Erik Johnston
f193034d59 1.37.1 2021-06-30 12:24:13 +01:00
Patrick Cloke
aaf7d1acb8
Correct type hints for synapse.event_auth. (#10253) 2021-06-30 07:08:42 -04:00
Erik Johnston
329ef5c715
Fix the inbound PDU metric (#10279) 
This broke in #10272
 2021-06-30 12:07:16 +01:00
Richard van der Hoff
bc5589a1bb Merge branch 'release-v1.37' into develop 2021-06-29 21:48:24 +01:00
Richard van der Hoff
d561367c18 1.37.1rc1 2021-06-29 21:39:30 +01:00
Richard van der Hoff
785bceef72 Merge branch 'release-v1.37' into develop 2021-06-29 20:25:47 +01:00
Richard van der Hoff
f99e9cc2da v1.37.1a1 2021-06-29 19:58:25 +01:00
Richard van der Hoff
c0bebd00ef Merge remote-tracking branch 'origin/erikj/async_federation_base_branch' into release-v1.37 2021-06-29 19:55:55 +01:00
Erik Johnston
c54db67d0e
Handle inbound events from federation asynchronously (#10272) 
Fixes #9490

This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous.

Two drawbacks:

    Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately.
    We now only process one event per room at a time. This can be fixed up further down the line.
 2021-06-29 19:55:22 +01:00
Erik Johnston
85d237eba7
Add a distributed lock (#10269) 
This adds a simple best effort locking mechanism that works cross workers.
 2021-06-29 19:15:47 +01:00
Patrick Cloke
f55836929d
Do not recurse into non-spaces in the spaces summary. (#10256) 
Previously m.child.room events in non-space rooms would be
treated as part of the room graph, but this is no longer
supported.
 2021-06-29 12:00:04 -04:00
Richard van der Hoff
7647b0337f
Fix populate_stream_ordering2 background job (#10267) 
It was possible for us not to find any rows in a batch, and hence conclude that
we had finished. Let's not do that.
 2021-06-29 12:43:36 +01:00
Richard van der Hoff
60efc51a2b
Migrate stream_ordering to a bigint (#10264) 
* Move background update names out to a separate class

`EventsBackgroundUpdatesStore` gets inherited and we don't really want to
further pollute the namespace.

* Migrate stream_ordering to a bigint

* changelog
 2021-06-29 11:25:34 +01:00
Richard van der Hoff
a0ed0f363e
Soft-fail spammy events received over federation (#10263) 2021-06-29 11:08:06 +01:00
Brendan Abolivier
3d370efc6d Merge branch 'master' into develop 2021-06-29 10:20:38 +01:00
Brendan Abolivier
cdf569e468 1.37.0 2021-06-29 10:15:34 +01:00
Patrick Cloke
0555d7b0dc
Add additional types to the federation transport server. (#10213) 2021-06-28 07:36:41 -04:00
Richard van der Hoff
8165ba48b1
Return errors from send_join etc if the event is rejected (#10243) 
Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.
 2021-06-24 16:00:08 +01:00
Richard van der Hoff
6e8fb42be7
Improve validation for send_{join,leave,knock} (#10225) 
The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
 2021-06-24 15:30:49 +01:00
Quentin Gliech
bd4919fb72
MSC2918 Refresh tokens implementation (#9450) 
This implements refresh tokens, as defined by MSC2918

This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235

The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one.

Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
 2021-06-24 14:33:20 +01:00
Brendan Abolivier
763dba77ef Synapse 1.37.0rc1 (2021-06-24) 
==============================
 
 This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface.
 
 This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings.
 
 Features
 --------
 
 - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227))
 - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247))
 - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206))
 - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108))
 - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191))
 
 Bugfixes
 --------
 
 - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115))
 - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154))
 - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158))
 - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175))
 - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184))
 - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195))
 - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208))
 - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221))
 
 Improved Documentation
 ----------------------
 
 - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436))
 - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122))
 - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180))
 - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198))
 
 Deprecations and Removals
 -------------------------
 
 - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238))
 - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161))
 - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194))
 
 Internal Changes
 ----------------
 
 - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933))
 - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080))
 - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193))
 - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143))
 - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144))
 - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145))
 - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148))
 - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155))
 - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156))
 - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160))
 - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164))
 - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168))
 - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183))
 - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188))
 - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189))
 - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190))
 - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197))
 - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUW7UACgkQHgFcFF8Z
 Fs1ZfQ//bSyzg9mB7F43JdVBH9cpGups3Ur6nmsqE608/i/OEtomx9kTZsoN/9AO
 j6WoM1bhJ3j9hoE63w4DIPVhBrojtwmYWeFrRvpRVlwTzZPhfij1nIH9pPrNVwd6
 /LE0uA1kW5sDZCTiOhrz84BBBz6knjBH4otl+MDDdrszYixoP3i3SrrZVJAEYyGz
 Dpw2OudpoOtMdGyzFMr2ewf2mQm5UFEPs9MT0AqhQJMjyM1lJYzrq3Obv1oTdCJq
 BdLx6kMhK8+MbZPrcqIMXgSMHLYeQYeJOrdTzuCozFo0DSblThpW//aa21+sZtPx
 dnIE0PpjAsm2Qg1II3EUX+YycYpI5KPhMq0ydEmUD7vTuGgJCKqAV3FBRwMlF2VE
 9SOr8Mjd1ip03dHz2vlg38OlrFJ52P0e1RDZwlDtWfeJjZGZ4W6VjMhT7H/Z5USg
 2J1FRPTD8E9w8wBOkjVcl76Lj6AY1KU36/ZLoYUbtrt58XYfkD4OzZmAxdJrqbMg
 na5T9bd9iM8vWditU+vncxU+c7Bkn55GFyDVMQYl8QtXzBvnbv8gQilGwJLkncZJ
 MreQ3OIBPbLNcHml56nmmY+ab+AtrIxuFl0UcZu+LGW8+lN+UZ9G9q0NFO3HbAWN
 /LuhCVpO42SLAuLlIhnO3mcHM5jzZodKIa6oXpcMS2jv0nylQWo=
 =MtOo
 -----END PGP SIGNATURE-----
gpgsig -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUXG8ACgkQHgFcFF8Z
 Fs2buxAAg7X4mYKHwlWLn2n2W6bUKY24HAnjtzMrNW76xIHXBJJiJMnhQD3MSKVd
 uiUdolc+Tt3drJYOJyurP8R3YI0OuEVbSL0Krs8f3l6jNOU0Gy7YQzc6XUHnA42J
 rb6mU8gMdCusk9RHHr1r+sBefwzmxTk4naCkw2n5i+r4ZfNparJK7H9yXCPoedUN
 VBZi9f09jaUIm1SiHgK24sW9XfE6ifjdVg+E/r599v2eHJxtCTHUasH9TCAFD8KL
 g17NtAP95+SQWPGwUXJFZYcVee+pGbylXuwP0RXXmMvyOWVKloI0LE6phGyLQ5cH
 rMRagIkvVLMdYagaX8Tf47hNLVCXRJpuryxMgY3+5fx0O7lCMXlC3mDfs9zze+Fj
 KXmJtVy/vpd+zpjQw+HHPnL1dDzsKnDqLFXFCj9Sjv6A/ruckxwWofuh0z83dYtv
 1Abp8k7gVTVGZqLGPo8gRoqI2jQhWDvmJd+crb05qp86SzCz1sCma2ujWQtv3c1t
 6aQpfmGMhFPp+hS2lS8AcyxK4vq+GudtXUOgzdqCNvIs+eJAV5cq6uufLIy6c2XR
 FAR7UowoAoDvqTdD7+i5nHEjbudjjN5KgPXlO3a9+gG5jOR3IsOLZ8FVN0ML6qIJ
 wzl0wM9YIlxyleMlWb0l4CFNDKrGJVhWgOupruHS+D5jfZQsZ3g=
 =AjRf
 -----END PGP SIGNATURE-----

Merge tag 'v1.37.0rc1' into develop

Synapse 1.37.0rc1 (2021-06-24)
==============================

This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface.

This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings.

Features
--------

- Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227))
- Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247))
- Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206))
- Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108))
- Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191))

Bugfixes
--------

- Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115))
- Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154))
- Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158))
- Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175))
- Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184))
- Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195))
- Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208))
- Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221))

Improved Documentation
----------------------

- Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436))
- Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122))
- Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180))
- Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198))

Deprecations and Removals
-------------------------

- The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238))
- Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161))
- Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194))

Internal Changes
----------------

- Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933))
- Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080))
- Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193))
- Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143))
- Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144))
- Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145))
- Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148))
- Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155))
- Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156))
- Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160))
- Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164))
- Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168))
- Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183))
- Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188))
- Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189))
- Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190))
- Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197))
- Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))
 2021-06-24 11:20:28 +01:00
Brendan Abolivier
c955e37868
Fix wrapping of legacy check_registration_for_spam (#10238) 
Fixes #10234
 2021-06-23 17:22:08 +02:00
Andrew Morgan
394673055d
Re-introduce "Leave out optional keys from /sync" change (#10214) 
Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.
 2021-06-23 15:57:41 +01:00
Brendan Abolivier
e19e3d452d
Improve the reliability of auto-joining remote rooms (#10237) 
If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do.

Fixes #10233 (at least to some extent)
 2021-06-23 16:14:52 +02:00
Richard van der Hoff
8beead66ae
Send out invite rejections and knocks over federation (#10223) 
ensure that events sent via `send_leave` and `send_knock` are sent on to
the rest of the federation.
 2021-06-23 12:54:50 +01:00
Michael[tm] Smith
27c06a6e06
Drop Origin & Accept from Access-Control-Allow-Headers value (#10114) 
* Drop Origin & Accept from Access-Control-Allow-Headers value

This change drops the Origin and Accept header names from the value of the
Access-Control-Allow-Headers response header sent by Synapse. Per the CORS
protocol, it’s not necessary or useful to include those header names.

Details:

Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name, Origin
is a “forbidden header name” set by the browser and that frontend
JavaScript code is never allowed to set.

So the value of Access-Control-Allow-Headers isn’t relevant to Origin or
in general to other headers set by the browser itself — the browser
never ever consults the Access-Control-Allow-Headers value to confirm
that it’s OK for the request to include an Origin header.

And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header,
Accept is a “CORS-safelisted request-header”, which means that browsers
allow requests to contain the Accept header regardless of whether the
Access-Control-Allow-Headers value contains "Accept".

So it’s unnecessary for the Access-Control-Allow-Headers to explicitly
include Accept. Browsers will not perform a CORS preflight for requests
containing an Accept request header.

Related: https://github.com/matrix-org/matrix-doc/pull/3225

Signed-off-by: Michael[tm] Smith <mike@w3.org>
 2021-06-23 11:25:03 +01:00
Brendan Abolivier
9ec45aca1f 1.37.0rc1 2021-06-23 09:38:27 +01:00
Erik Johnston
33701dc116
Fix schema delta to not take as long on large servers (#10227) 
Introduced in #6739
 2021-06-22 12:00:45 +01:00
Brendan Abolivier
34db6bb9f5
Warn users trying to use the deprecated spam checker interface (#10210) 
So admins aren't surprised if things break when we remove this code in a couple of months.
 2021-06-22 12:24:10 +02:00
Eric Eastwood
96f6293de5
Add endpoints for backfilling history (MSC2716) (#9247) 
Work on https://github.com/matrix-org/matrix-doc/pull/2716
 2021-06-22 10:02:53 +01:00
jkanefendt
756fd513df
Implement config option sso.update_profile_information (#10108) 
Implemented config option sso.update_profile_information to keep user's display name in sync with the SSO displayname.

Signed-off-by: Johannes Kanefendt <johannes.kanefendt@krzn.de>
 2021-06-21 23:48:57 +01:00
Erik Johnston
a5cd05beee
Fix performance of responding to user key requests over federation (#10221) 
We were repeatedly looking up a config option in a loop (using the
unclassed config style), which is expensive enough that it can cause
large CPU usage.
 2021-06-21 14:38:59 +01:00
Andrew Morgan
182147195b
Check third party rules before persisting knocks over federation (#10212) 
An accidental mis-ordering of operations during #6739 technically allowed an incoming knock event over federation in before checking it against any configured Third Party Access Rules modules.

This PR corrects that by performing the TPAR check *before* persisting the event.
 2021-06-21 11:57:09 +01:00
Richard van der Hoff
107c06081f
Ensure that errors during startup are written to the logs and the console. (#10191) 
* Defer stdio redirection until we are about to start the reactor

* Catch and handle exceptions during startup
 2021-06-21 11:41:25 +01:00
Patrick Cloke
0bd968921c
Fix a missing await when in the spaces summary. (#10208) 
This could cause a minor data leak if someone defined a non-restricted join rule
with an allow key or used a restricted join rule in an older room version, but this is
unlikely.

Additionally this starts adding unit tests to the spaces summary handler.
 2021-06-18 18:41:33 +01:00
Brendan Abolivier
1b3e398bea
Standardise the module interface (#10062) 
This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
 2021-06-18 12:15:52 +01:00
Richard van der Hoff
91fa9cca99
Expose opentracing trace id in response headers (#10199) 
Fixes: #9480
 2021-06-18 11:43:22 +01:00
Brendan Abolivier
08c8469322
Remove support for ACME v1 (#10194) 
Fixes #9778

ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
 2021-06-17 18:56:48 +01:00
Patrick Cloke
8c97d5863f
Update MSC3083 support per changes in the MSC. (#10189) 
Adds a "type" field and generalize "space" to "room_id".
 2021-06-17 12:53:27 -04:00
Richard van der Hoff
fcf3c7032b
Ensure that we do not cache empty sync responses after a timeout (#10158) 
Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
 2021-06-17 16:23:11 +01:00
Richard van der Hoff
9cf6e0eae7
Rip out the DNS lookup limiter (#10190) 
As I've written in various places in the past (#7113, #9865) I'm pretty sure this is doing nothing useful at all.
 2021-06-17 16:22:41 +01:00
Marcus
8070b893db
update black to 21.6b0 (#10197) 
Reformat all files with the new version.

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
 2021-06-17 15:20:06 +01:00
Andrew Morgan
6f1a28de19
Fix incorrect time magnitude on delayed call (#10195) 
Fixes https://github.com/matrix-org/synapse/issues/10030.

We were expecting milliseconds where we should have provided a value in seconds.

The impact of this bug isn't too bad. The code is intended to count the number of remote servers that the homeserver can see and report that as a metric. This metric is supposed to run initially 1 second after server startup, and every 60s as well. Instead, it ran 1,000 seconds after server startup, and every 60s after startup.

This fix allows for the correct metrics to be collected immediately, as well as preventing a random collection 1,000s in the future after startup.
 2021-06-17 15:04:26 +01:00
Eric Eastwood
a911dd768b
Add fields to better debug where events are being soft_failed (#10168) 
Follow-up to https://github.com/matrix-org/synapse/pull/10156#discussion_r650292223
 2021-06-17 14:59:45 +01:00
Richard van der Hoff
52c60bd0a9
Fix persist_events to stop leaking opentracing contexts (#10193) 2021-06-17 11:21:53 +01:00
Patrick Cloke
18edc9ab06
Improve comments in the structured logging code. (#10188) 2021-06-16 19:18:02 +01:00
Patrick Cloke
76f9c701c3
Always require users to re-authenticate for dangerous operations. (#10184) 
Dangerous actions means deactivating an account, modifying an account
password, or adding a 3PID.

Other actions (deleting devices, uploading keys) can re-use the same UI
auth session if ui_auth.session_timeout is configured.
 2021-06-16 11:07:28 -04:00
Erik Johnston
36c426e294
Add debug logging when we enter/exit Measure block (#10183) 
It can be helpful to know when trying to track down slow requests.
 2021-06-16 13:29:54 +01:00
Lukas Lihotzki
2c240213f4
Fix requestOpenIdToken response: integer expires_in (#10175) 
`expires_in` must be an integer according to the OpenAPI spec:
https://github.com/matrix-org/matrix-doc/blob/master/data/api/client-server/definitions/openid_token.yaml#L32

True division (`/`) returns a float instead (`"expires_in": 3600.0`).
Floor division (`//`) returns an integer, so the response is spec compliant.

Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>
 2021-06-16 13:16:35 +01:00
Richard van der Hoff
9e405034e5
Make opentracing trace into event persistence (#10134) 
* Trace event persistence

When we persist a batch of events, set the parent opentracing span to the that
from the request, so that we can trace all the way in.

* changelog

* When we force tracing, set a baggage item

... so that we can check again later.

* Link in both directions between persist_events spans
 2021-06-16 11:41:15 +01:00
Erik Johnston
d09e24a52d Merge branch 'master' into develop 2021-06-15 15:52:24 +01:00
Erik Johnston
1c8045f674 1.36.0 2021-06-15 15:42:02 +01:00
Patrick Cloke
4911f7931d
Remove support for unstable MSC1772 prefixes. (#10161) 
The stable prefixes have been supported since v1.34.0. The unstable
prefixes are not supported by any known clients.
 2021-06-15 08:03:17 -04:00
Patrick Cloke
9e5ab6dd58
Remove the experimental flag for knocking and use stable prefixes / endpoints. (#10167) 
* Room version 7 for knocking.
* Stable prefixes and endpoints (both client and federation) for knocking.
* Removes the experimental configuration flag.
 2021-06-15 07:45:14 -04:00
Michael Kutzner
aac2c49b9b
Fix 'ip_range_whitelist' not working for federation servers (#10115) 
Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers.

Signed-off-by: Michael Kutzner 1mikure@gmail.com
 2021-06-15 08:53:55 +01:00
Richard van der Hoff
1dfdc87b9b
Refactor EventPersistenceQueue (#10145) 
some cleanup, pulled out of #10134.
 2021-06-14 11:59:27 +01:00
Richard van der Hoff
d7808a2dde
Extend ResponseCache to pass a context object into the callback (#10157) 
This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug.

The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
 2021-06-14 10:26:09 +01:00
Erik Johnston
29966a285d Synapse 1.36.0rc2 (2021-06-11) 
==============================
 
 Bugfixes
 --------
 
 - Fix a bug which caused  presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149))
 - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163))
 - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmDDdrcQHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCTzfB/4qaTqW2mBiwjf52SmOu6HNyd8uQd6nLIAZ
 mJC218Wakh2tT0W4iVkKwUgpuHFbtcy0rSNTlXtW4kG8XzhpTvT56RH9qls99aD3
 SGKqYpOv6NkWZibN6NdVvLDW85ixficDTXco3BljRCIMlORhY0swy+LWLwksdjWj
 6kQ+Gi/QAtKP3Pt5epYs0Ix5o1T94DfZOWE//mqBhG5cMDAw/K/G/c8tRfEjclt9
 wACBjmt2fw/Lbn9j3b0feNVp+xnFcFNuAK2bSEd8Y3yph1mhjdsIszULnM7IFNsR
 Q8zg+i7PJKNq8pQjei8j8T/aKscPTPH5XGqOSLlizj15snsiwlkz
 =C6lV
 -----END PGP SIGNATURE-----

Merge tag 'v1.36.0rc2' into develop

Synapse 1.36.0rc2 (2021-06-11)
==============================

Bugfixes
--------

- Fix a bug which caused  presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149))
- Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163))
- Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165))
 2021-06-11 15:46:38 +01:00
Erik Johnston
fb10a73e85 1.36.0rc2 2021-06-11 15:21:34 +01:00
Erik Johnston
cdd985c64f Only send a presence state to a destination once (#10165) 
It turns out that we were sending the same presence state to a remote
potentially multiple times.
 2021-06-11 15:21:08 +01:00
Erik Johnston
5e0b4719ea Fix sending presence over federation when using workers (#10163) 
When using a federation sender we'd send out all local presence updates over
federation even when they shouldn't be.

Fixes #10153.
 2021-06-11 15:20:54 +01:00
Erik Johnston
c955f22e2c Fix bug when running presence off master (#10149) 
Hopefully fixes #10027.
 2021-06-11 15:20:45 +01:00
Erik Johnston
968f8283b4
Only send a presence state to a destination once (#10165) 
It turns out that we were sending the same presence state to a remote
potentially multiple times.
 2021-06-11 15:19:42 +01:00
Richard van der Hoff
c1b9922498
Support for database schema version ranges (#9933) 
This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
 2021-06-11 14:45:53 +01:00
Erik Johnston
c8dd4db9eb
Fix sending presence over federation when using workers (#10163) 
When using a federation sender we'd send out all local presence updates over
federation even when they shouldn't be.

Fixes #10153.
 2021-06-11 13:08:30 +01:00
Andrew Morgan
a15a046c93
Clean up a broken import in admin_cmd.py (#10154) 2021-06-11 11:34:40 +01:00
Erik Johnston
d26d15ba3d
Fix bug when running presence off master (#10149) 
Hopefully fixes #10027.
 2021-06-11 10:27:12 +01:00
Eric Eastwood
b31daac01c
Add metrics to track how often events are soft_failed (#10156) 
Spawned from missing messages we were seeing on `matrix.org` from a
federated Gtiter bridged room, https://gitlab.com/gitterHQ/webapp/-/issues/2770.
The underlying issue in Synapse is tracked by https://github.com/matrix-org/synapse/issues/10066
where the message and join event race and the message is `soft_failed` before the
`join` event reaches the remote federated server.

Less soft_failed events = better and usually this should only trigger for events
where people are doing bad things and trying to fuzz and fake everything.
 2021-06-11 10:12:35 +01:00
Aaron Raimist
e6245e6d48
Mention that you need to configure max upload size in reverse proxy as well (#10122) 
Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2021-06-10 11:40:24 +01:00
Andrew Morgan
a7a37437bc
Integrate knock rooms with the public rooms directory (#9359) 
This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403.

Specifically, it:

* Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory
* Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room

Based on https://github.com/matrix-org/synapse/issues/6739. Complement tests for this change: https://github.com/matrix-org/complement/pull/72
 2021-06-09 20:31:31 +01:00
Sorunome
d936371b69
Implement knock feature (#6739) 
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
 2021-06-09 19:39:51 +01:00
Patrick Cloke
11846dff8c
Limit the number of in-flight /keys/query requests from a single device. (#10144) 2021-06-09 07:05:32 -04:00
Richard van der Hoff
1bf83a191b
Clean up the interface for injecting opentracing over HTTP (#10143) 
* Remove unused helper functions

* Clean up the interface for injecting opentracing over HTTP

* changelog
 2021-06-09 11:33:00 +01:00
Patrick Cloke
c7f3fb2745
Add type hints to the federation server transport. (#10080) 2021-06-08 11:19:25 -04:00
Andrew Morgan
8df9941cc2 1.36.0rc1 2021-06-08 14:09:00 +01:00
Erik Johnston
1092718cac
Fix logging context when opening new DB connection (#10141) 
Fixes #10140
 2021-06-08 13:49:29 +01:00
Patrick Cloke
9e4610cc27
Correct type hints for parse_string(s)_from_args. (#10137) 2021-06-08 08:30:48 -04:00
Erik Johnston
c842c581ed
When joining a remote room limit the number of events we concurrently check signatures/hashes for (#10117) 
If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.
 2021-06-08 11:07:46 +01:00
Erik Johnston
a0101fc021
Handle /backfill returning no events (#10133) 
Fixes #10123
 2021-06-08 10:37:01 +01:00
Richard van der Hoff
0acb5010ec
More database opentracing (#10136) 
Add a couple of extra logs/spans, to give a bit of a better idea.
 2021-06-07 18:01:32 +01:00
Richard van der Hoff
b2557cbf42
opentracing: use a consistent name for background processes (#10135) 
... otherwise we tend to get a namespace clash between the bg process and the
functions that it calls.
 2021-06-07 17:57:49 +01:00
14mRh4X0r
8942e23a69
Always update AS last_pos, even on no events (#10107) 
Fixes #1834.

`get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether.

Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list.


Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>
 2021-06-07 15:42:05 +01:00
Dirk Klimpel
d558292548
Add missing type hints to the admin API servlets (#10105) 2021-06-07 15:12:34 +01:00
Richard van der Hoff
fa1db8f156
Delete completes to-device messages earlier in /sync (#10124) 
I hope this will improve
https://github.com/matrix-org/synapse/issues/9564.
 2021-06-07 09:19:06 +01:00
Erik Johnston
a0cd8ae8cb
Don't try and backfill the same room in parallel. (#10116) 
If backfilling is slow then the client may time out and retry, causing
Synapse to start a new `/backfill` before the existing backfill has
finished, duplicating work.
 2021-06-04 10:47:58 +01:00
Erik Johnston
c96ab31dff
Limit number of events in a replication request (#10118) 
Fixes #9956.
 2021-06-04 10:35:47 +01:00
Richard van der Hoff
d8be7d493d
Enable Prometheus metrics for the jaeger client library (#10112) 2021-06-04 09:25:33 +01:00
Richard van der Hoff
9eea4646be
Add OpenTracing for database activity. (#10113) 
This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels:

 * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions.
 * emit a span while we hold a database connection open
 * emit a span for each database transaction - actual actual transaction.
 * emit a span for each database query.

I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
 2021-06-03 16:31:56 +01:00
Richard van der Hoff
1d143074c5
Improve opentracing annotations for Notifier (#10111) 
The existing tracing reports an error each time there is a timeout, which isn't
really representative.

Additionally, we log things about the way `wait_for_events` works
(eg, the result of the callback) to the *parent* span, which is confusing.
 2021-06-03 16:01:30 +01:00
Travis Ralston
5325f0308c
r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224) 
[MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. 

This is progress towards https://github.com/matrix-org/synapse/issues/8334
 2021-06-03 13:50:49 +01:00
Patrick Cloke
d7a646abca Merge branch 'master' into develop 2021-06-03 08:43:38 -04:00
Patrick Cloke
57c01dca29 1.35.1 2021-06-03 08:18:22 -04:00
Patrick Cloke
36a7ff0c86 Do not show invite-only rooms in spaces summary (unless joined/invited). (#10109) 2021-06-03 08:04:01 -04:00
Dirk Klimpel
0284d2a297
Add new admin APIs to remove media by media ID from quarantine. (#10044) 
Related to: #6681, #5956, #10040

Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2021-06-02 18:50:35 +01:00
Erik Johnston
fc3d2dc269
Rewrite the KeyRing (#10035) 2021-06-02 16:37:59 +01:00
Patrick Cloke
3cf6b34b4e
Do not show invite-only rooms in spaces summary (unless joined/invited). (#10109) 2021-06-02 11:31:41 -04:00
Erik Johnston
4deaebfe00
Make /sync do less state res (#10102) 2021-06-02 15:48:17 +01:00
Andrew Morgan
3ff6fe2851 Merge branch 'master' into develop 2021-06-01 13:47:27 +01:00
Andrew Morgan
a8372ad591 1.35.0 2021-06-01 13:23:55 +01:00
Richard van der Hoff
b4b2fd2ece
add a cache to have_seen_event (#9953) 
Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
 2021-06-01 12:04:47 +01:00
Brad Murray
10e6d2abce
Fix opentracing inject to use the SpanContext, not the Span (#10074) 
Signed-off-by: Brad Murray brad@beeper.com
 2021-06-01 08:40:26 +01:00
Erik Johnston
258a9a9e8b 1.35.0rc3 2021-05-28 17:06:05 +01:00
Erik Johnston
6b6c6a02db Merge remote-tracking branch 'origin/release-v1.35.0' into release-v1.35 2021-05-28 17:03:14 +01:00
Brendan Abolivier
9408b86f5c Limit the number of events sent over replication when persisting events. (#10082) 2021-05-28 16:29:09 +01:00
Erik Johnston
1641c5c707 Log method and path when dropping request due to size limit (#10091) 2021-05-28 16:29:09 +01:00
Erik Johnston
84cf3e47a0
Allow response of /send_join to be larger. (#10093) 
Fixes #10087.
 2021-05-28 16:28:01 +01:00
Richard van der Hoff
ed53bf314f
Set opentracing priority before setting other tags (#10092) 
... because tags on spans which aren't being sampled get thrown away.
 2021-05-28 16:14:08 +01:00
Erik Johnston
3f96dbbda7
Log method and path when dropping request due to size limit (#10091) 2021-05-28 15:57:53 +01:00
Eric Eastwood
ac3e02d089
Add parse_strings_from_args to get prev_events array (#10048) 
Split out from https://github.com/matrix-org/synapse/pull/9247

Strings:

 - `parse_string`
 - `parse_string_from_args`
 - `parse_strings_from_args`

For comparison with ints:

 - `parse_integer`
 - `parse_integer_from_args`

Previous discussions:

 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687
 - https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156
 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791

Signed-off-by: Eric Eastwood <erice@element.io>
 2021-05-28 14:19:06 +01:00
Erik Johnston
5eed6348ce
Move some more endpoints off master (#10084) 2021-05-27 22:45:43 +01:00
Callum Brown
8fb9af570f
Make reason and score optional for report_event (#10077) 
Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414
See #8551 

Signed-off-by: Callum Brown <callum@calcuode.com>
 2021-05-27 18:42:23 +01:00
Brendan Abolivier
f828a70be3
Limit the number of events sent over replication when persisting events. (#10082) 2021-05-27 17:10:58 +01:00
Erik Johnston
8e132fe64e Synapse 1.35.0rc2 (2021-05-27) 
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](https://github.com/matrix-org/synapse/issues/10079))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmCvpZMQHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCdPwCACQIlIWd6eIoXLUc+wDcrd+k5xL376EdYah
 x7ABswiYSm+9C4xr58gJD3xc6eiD2PCIWdZN0rsQDLIOfSXW6x1lyKD+Ds0HySok
 MaVpsoxbb9o/Zf9qtXF2bLSArZUQwfoNaA45NLgNzfUIijf1e+bd2wNEgHlRSoMz
 m10GggOFU0Ds/CCYZpxZw/eXDbLWL7eaHR30vw/jQ1cEsV+S4ucnUmLHFCF7YCyI
 Np80pnywH5cYKerecldFWenL4YZJswVbx+AW9e3lBzq5jOrRZkmLkaHg10mCR2f6
 CV03ie65Ce+7x5UU6v6nHA0DYUTQGIjlJBtyCN3tFglUduQ8Gpu0
 =Mfsf
 -----END PGP SIGNATURE-----

Merge tag 'v1.35.0rc2' into develop

Synapse 1.35.0rc2 (2021-05-27)
==============================

Bugfixes
--------

- Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](https://github.com/matrix-org/synapse/issues/10079))
 2021-05-27 14:59:46 +01:00
Erik Johnston
b1bc26a909 1.35.0rc2 2021-05-27 14:46:24 +01:00
Erik Johnston
78b5102ae7
Fix up BatchingQueue (#10078) 
Fixes #10068
 2021-05-27 14:32:31 +01:00
Patrick Cloke
8e15c92c2f
Pass the origin when calculating the spaces summary over GET. (#10079) 
Fixes a bug due to conflicting PRs which were merged. (One added a new caller to
a method, the other added a new parameter to the same method.)
 2021-05-27 08:52:28 -04:00
Richard van der Hoff
fe5dad46b0
Remove redundant code to reload tls cert (#10054) 
we don't need to reload the tls cert if we don't have any tls listeners.

Follow-up to #9280.
 2021-05-27 10:34:24 +01:00
Richard van der Hoff
224f2f949b
Combine LruCache.invalidate and invalidate_many (#9973) 
* Make `invalidate` and `invalidate_many` do the same thing

... so that we can do either over the invalidation replication stream, and also
because they always confused me a bit.

* Kill off `invalidate_many`

* changelog
 2021-05-27 10:33:56 +01:00
Patrick Cloke
f42e4c4eb9
Remove the experimental spaces enabled flag. (#10063) 
In lieu of just always enabling the unstable spaces endpoint and
unstable room version.
 2021-05-26 14:35:16 -04:00
Dirk Klimpel
65e6c64d83
Add an admin API for unprotecting local media from quarantine (#10040) 
Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2021-05-26 11:19:47 +01:00
Erik Johnston
557635f69a 1.35.0rc1 2021-05-25 11:00:13 +01:00
Patrick Cloke
7adcb20fc0
Add missing type hints to synapse.util (#9982) 2021-05-24 15:32:01 -04:00
Jerin J Titus
057ce7b754
Remove tls_fingerprints option (#9280) 
Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
 2021-05-24 17:43:30 +01:00
Richard van der Hoff
c0df6bae06
Remove keylen from LruCache. (#9993) 
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.

The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).

Commits should be independently reviewable.
 2021-05-24 14:02:01 +01:00
Patrick Cloke
316f89e87f
Enable experimental spaces by default. (#10011) 
The previous spaces_enabled flag now defaults to true and
is exposed in the sample config.
 2021-05-24 08:57:14 -04:00
Eric Eastwood
5f1198a67e
Fix get_state_ids_for_event return type typo to match what the function actually does (#10050) 
It looks like a typo copy/paste from `get_state_for_event` above.
 2021-05-24 10:43:33 +01:00
Erik Johnston
3e831f24ff
Don't hammer the database for destination retry timings every ~5mins (#10036) 2021-05-21 17:57:08 +01:00
Michael Telatynski
e8ac9ac8ca
Fix /upload 500'ing when presented a very large image (#10029) 
* Fix /upload 500'ing when presented a very large image

Catch DecompressionBombError and re-raise as ThumbnailErrors

* Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml

to get it to bomb out quicker, to load less into memory
in the case of super large images

* Add changelog entry for 10029
 2021-05-21 18:31:59 +02:00
Patrick Cloke
c5413d0e9e
Remove unused properties from the SpaceSummaryHandler. (#10038) 2021-05-21 12:02:01 -04:00
Marek Matys
6a8643ff3d
Fixed removal of new presence stream states (#10014) 
Fixes: https://github.com/matrix-org/synapse/issues/9962

This is a fix for above problem.

I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts.

Signed-off-by: Marek Matys <themarcq@gmail.com>
 2021-05-21 12:02:06 +01:00
Erik Johnston
7958eadcd1
Add a batching queue implementation. (#10017) 2021-05-21 11:20:51 +01:00
Erik Johnston
1c6a19002c
Add Keyring.verify_events_for_server and reduce memory usage (#10018) 
Also add support for giving a callback to generate the JSON object to
verify. This should reduce memory usage, as we no longer have the event
in memory in dict form (which has a large memory footprint) for extend
periods of time.
 2021-05-20 16:25:11 +01:00
Erik Johnston
64887f06fc
Use ijson to parse the response to /send_join, reducing memory usage. (#9958) 
Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
 2021-05-20 16:11:48 +01:00
Patrick Cloke
551d2c3f4b
Allow a user who could join a restricted room to see it in spaces summary. (#9922) 
This finishes up the experimental implementation of MSC3083 by showing
the restricted rooms in the spaces summary (from MSC2946).
 2021-05-20 11:10:36 -04:00
Patrick Cloke
ac6bfcd52f
Refactor checking restricted join rules (#10007) 
To be more consistent with similar code. The check now automatically
raises an AuthError instead of passing back a boolean. It also absorbs
some shared logic between callers.
 2021-05-18 12:17:04 -04:00
Andrew Morgan
4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823) 2021-05-18 14:13:45 +01:00
Patrick Cloke
206a7b5f12
Fix the allowed range of valid ordering characters for spaces. (#10002) 
\x7F was meant to be \0x7E (~) this was originally incorrect
in MSC1772.
 2021-05-17 09:59:17 -04:00
Patrick Cloke
9752849e2b
Clarify comments in the space summary handler. (#9974) 2021-05-17 09:01:19 -04:00
Erik Johnston
653fe2f3cd Merge branch 'master' into develop 2021-05-17 12:14:58 +01:00
Erik Johnston
afb6dcf806 1.34.0 2021-05-17 11:34:39 +01:00
Brendan Abolivier
41ac128fd3
Split multiplart email sending into a dedicated handler (#9977) 
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2021-05-17 12:33:38 +02:00
Richard van der Hoff
5090f26b63
Minor @cachedList enhancements (#9975) 
- use a tuple rather than a list for the iterable that is passed into the
  wrapped function, for performance

- test that we can pass an iterable and that keys are correctly deduped.
 2021-05-14 11:12:36 +01:00
Dan Callahan
52ed9655ed
Remove unnecessary SystemRandom from SQLBaseStore (#9987) 
It's not obvious that instances of SQLBaseStore each need their own
instances of random.SystemRandom(); let's just use random directly.

Introduced by 52839886d6

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-14 10:59:10 +01:00
Dan Callahan
ebdef256b3
Remove superfluous call to bool() (#9986) 
Our strtobool already returns a bool, so no need to re-cast here

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-14 10:58:57 +01:00
Dan Callahan
bd918d874f
Simplify exception handling in is_ascii. (#9985) 
We can get away with just catching UnicodeError here.

    ⋮
    +-- ValueError
    |    +-- UnicodeError
    |         +-- UnicodeDecodeError
    |         +-- UnicodeEncodeError
    |         +-- UnicodeTranslateError
    ⋮

https://docs.python.org/3/library/exceptions.html#exception-hierarchy

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-14 10:58:52 +01:00
Dan Callahan
498084228b
Use Python's secrets module instead of random (#9984) 
Functionally identical, but more obviously cryptographically secure.
...Explicit is better than implicit?

Avoids needing to know that SystemRandom() implies a CSPRNG, and
complies with the big scary red box on the documentation for random:

> Warning:
>   The pseudo-random generators of this module should not be used for
>   security purposes. For security or cryptographic uses, see the
>   secrets module.

https://docs.python.org/3/library/random.html

Signed-off-by: Dan Callahan <danc@element.io>
 2021-05-14 10:58:46 +01:00
Richard van der Hoff
c14f99be46
Support enabling opentracing by user (#9978) 
Add a config option which allows enabling opentracing by user id, eg for
debugging requests made by a test user.
 2021-05-14 10:51:08 +01:00
Brendan Abolivier
47806b0869 1.34.0rc1 2021-05-12 16:59:46 +01:00
Brendan Abolivier
a683028d81
Correctly ratelimit invites when creating a room (#9968) 
* Correctly ratelimit invites when creating a room

Also allow ratelimiting for more than one action at a time.
 2021-05-12 16:05:28 +02:00
Richard van der Hoff
7562d887e1
Change the format of access tokens away from macaroons (#5588) 2021-05-12 15:04:51 +01:00
Erik Johnston
affaffb0ab
Run cache_joined_hosts_for_event in background (#9951) 2021-05-12 13:17:11 +01:00
Patrick Cloke
27c375f812
Sort child events according to MSC1772 for the spaces summary API. (#9954) 
This should help ensure that equivalent results are achieved between
homeservers querying for the summary of a space.

This implements modified MSC1772 rules, according to MSC2946.
The different is that the origin_server_ts of the m.room.create event
is not used as a tie-breaker since this might not be known if the
homeserver is not part of the room.
 2021-05-11 12:57:39 -04:00
Patrick Cloke
f4833e0c06
Support fetching the spaces summary via GET over federation. (#9947) 
Per changes in MSC2946, the C-S and S-S APIs for spaces summary
should use GET requests.

Until this is stable, the POST endpoints still exist.

This does not switch federation requests to use the GET version yet
since it is newly added and already deployed servers might not support
it. When switching to the stable endpoint we should switch to GET
requests.
 2021-05-11 12:21:43 -04:00
Patrick Cloke
28c6841102
Send the m.room.create stripped event with invites (support MSC1772). (#9966) 
MSC1772 specifies the m.room.create event should be sent as part
of the invite_state. This was done optionally behind an experimental
flag, but is now done by default due to MSC1772 being approved.
 2021-05-11 10:58:58 -04:00
Brendan Abolivier
652a6b094d Merge branch 'master' into develop 2021-05-11 14:15:30 +01:00
Aaron Raimist
dc6366a9bd
Add config option to hide device names over federation (#9945) 
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2021-05-11 14:03:23 +01:00
Brendan Abolivier
86fb71431c
1.33.2 2021-05-11 14:01:32 +01:00
Richard van der Hoff
b378d98c8f
Add debug logging for issue #9533 (#9959) 
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
 2021-05-11 11:04:03 +01:00
Richard van der Hoff
7967b36efe
Fix m.room_key_request to-device messages (#9961) 
fixes #9960
 2021-05-11 11:02:56 +01:00
Richard van der Hoff
03318a766c
Merge pull request from GHSA-x345-32rc-8h85 
* tests for push rule pattern matching

* tests for acl pattern matching

* factor out common `re.escape`

* Factor out common re.compile

* Factor out common anchoring code

* add word_boundary support to `glob_to_regex`

* Use `glob_to_regex` in push rule evaluator

NB that this drops support for character classes. I don't think anyone ever
used them.

* Improve efficiency of globs with multiple wildcards

The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.

* add assertion on regex pattern

* Fix mypy

* Simplify glob_to_regex

* Inline the glob_to_regex helper function

Signed-off-by: Dan Callahan <danc@element.io>

* Moar comments

Signed-off-by: Dan Callahan <danc@element.io>

Co-authored-by: Dan Callahan <danc@element.io>
 2021-05-11 11:47:23 +02:00
Erik Johnston
2b2985b5cf
Improve performance of backfilling in large rooms. (#9935) 
We were pulling the full auth chain for the room out of the DB each time
we backfilled, which can be *huge* for large rooms and is totally
unnecessary.
 2021-05-10 13:29:02 +01:00
Erik Johnston
6c84778549
Always cache 'event_to_prev_state_group' (#9950) 
Fixes regression in send PDU times introduced in #9905.
 2021-05-07 14:54:09 +01:00
Erik Johnston
b65ecaff9b Merge remote-tracking branch 'origin/release-v1.33.2' into develop 2021-05-07 13:27:19 +01:00
Erik Johnston
4df26abf28
Unpin attrs dep after new version has been released (#9946) 
c.f. #9936
 2021-05-07 12:57:21 +01:00
Richard van der Hoff
25f43faa70
Reorganise the database schema directories (#9932) 
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.

It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
 2021-05-07 10:22:05 +01:00
Erik Johnston
8771b1337d
Export jemalloc stats to prometheus when used (#9882) 2021-05-06 15:54:07 +01:00
Erik Johnston
eba431c539
Revert "Leave out optional keys from /sync (#9919)" (#9940) 
This reverts commit e9eb3549d3.
 2021-05-06 15:06:35 +01:00
Erik Johnston
a8803e2b6e Merge remote-tracking branch 'origin/master' into develop 2021-05-06 14:36:59 +01:00
Erik Johnston
ac88aca7f7 1.33.1 2021-05-06 14:06:38 +01:00
Erik Johnston
24f07a83e6
Pin attrs to <21.1.0 (#9937) 
Fixes #9936
 2021-05-06 14:06:06 +01:00
Patrick Cloke
70f0ffd2fc Follow-up to #9915 to correct the identifier for room types. 2021-05-05 16:31:16 -04:00
Patrick Cloke
d783880083
Include the time of the create event in Spaces Summary. (#9928) 
This is an update based on changes to MSC2946. The origin_server_ts
of the m.room.create event is copied into the creation_ts field for each
room returned from the spaces summary.
 2021-05-05 13:33:05 -04:00
Erik Johnston
37623e3382
Increase perf of handling presence when joining large rooms. (#9916) 2021-05-05 17:27:05 +01:00
Patrick Cloke
e2a443550e
Support stable MSC1772 spaces identifiers. (#9915) 
Support both the unstable and stable identifiers. A future release
will disable the unstable identifiers.
 2021-05-05 11:56:51 -04:00
Erik Johnston
ef889c98a6
Optionally track memory usage of each LruCache (#9881) 
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
 2021-05-05 16:54:36 +01:00
Erik Johnston
1fb9a2d0bf
Limit how often GC happens by time. (#9902) 
Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.

This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.

Adds a `gc_min_seconds_between` config option to override the defaults.

Fixes #9890.
 2021-05-05 16:53:45 +01:00
Erik Johnston
de8f0a03a3
Don't set the external cache if its been done recently (#9905) 2021-05-05 16:53:22 +01:00
Erik Johnston
d0aee697ac
Use get_current_users_in_room from store and not StateHandler (#9910) 2021-05-05 16:49:34 +01:00
DeepBlueV7.X
e9eb3549d3
Leave out optional keys from /sync (#9919) 
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)

Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
 2021-05-05 14:37:56 +01:00
Brendan Abolivier
a61b13c0a1 Merge branch 'master' into develop 2021-05-05 14:25:49 +01:00
Brendan Abolivier
0644ac0989 1.33.0 2021-05-05 14:15:54 +01:00
Erik Johnston
e3bc4617fc
Time external cache response time (#9904) 2021-05-04 15:14:22 +01:00
Andrew Morgan
b85821aca2
Add port parameter to the sample config for psycopg2 args (#9911) 
Adds the `port` option with the default value to the sample config file.
 2021-05-04 13:28:59 +01:00
Andrew Morgan
4d624f467a Synapse 1.33.0rc2 (2021-04-29) 
==============================
 
 Bugfixes
 --------
 
 - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCKtVATHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ok1D/40l93bxGBFtSNEHlqem7lZRLY2On9j
 AUnzIMqtJTLUgXf67vKIYqtTWkRjTdLIhIuINwhRmB0K23vcRrONnnyRLdhUC4Hl
 v6qLlPBHzupAJQPRcy3FGIVwsloAZ3JqVhV8AG0gXBRA0Dc34qYXh1DlA8EQ8mW/
 l2umQIJI562xLNss7c5tpbCQxi7mq6OmTZ+Nwh45VRDavKEQQzSb9sPfs5w8IGLA
 ViYosUoXoXEWus/j60E38LSA/v1YcEE+Uyf5+LieV4b2c305xKj8gIbZqjLHR2K5
 U9IrUp4QlYpPiebfGn7EmiPnOnlWdR9soIVZMzJ/m29tlyRSL5MBagtWA2fFlXo6
 PywW1R/VsnLTdOuWY/t+et11iI4lVa7VSpywEcbMKGgmGKexDnfB10K7jHw0YICq
 0pyLgy8JWyGUk/xpH2W1pvak8vyyN/zI/H09bDgGu721Qr/EPiHWF6RADNASpUVW
 w10SnO1udVc8ajugNwpX2oOQA7i5uiEuTdyFUHYfZ/aSpaCvdxM+rL3v0K955bNc
 /AGjSbs9ITe+JJQDiAJsWCYc+XLkR4h5KHVaZm2XlusbYb4iobK1DflkWXnQFjsi
 rlTcUH3sIFot3i9mjBBvXdvmzuI9qyx81/Sh8gRunPYT5TPhwaXKfyMBKE93/kif
 4bKoHIXVcJvAqA==
 =sboq
 -----END PGP SIGNATURE-----

Merge tag 'v1.33.0rc2' into develop

Synapse 1.33.0rc2 (2021-04-29)
==============================

Bugfixes
--------

- Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))
 2021-04-29 14:35:14 +01:00
Patrick Cloke
bb4b11846f
Add missing type hints to handlers and fix a Spam Checker type hint. (#9896) 
The user_may_create_room_alias method on spam checkers
declared the room_alias parameter as a str when in reality it is
passed a RoomAlias object.
 2021-04-29 07:17:28 -04:00
Andrew Morgan
e9444cc74d 1.33.0rc2 2021-04-29 11:45:37 +01:00
ThibF
0085dc5abc
Delete room endpoint (#9889) 
Support the delete of a room through DELETE request and mark
previous request as deprecated through documentation.

Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
 2021-04-29 10:31:45 +01:00
Erik Johnston
802560211a Merge remote-tracking branch 'origin/release-v1.33.0' into develop 2021-04-28 14:43:10 +01:00
Erik Johnston
e4ab8676b4
Fix tight loop handling presence replication. (#9900) 
Only affects workers. Introduced in #9819.

Fixes #9899.
 2021-04-28 14:42:50 +01:00
Patrick Cloke
10a08ab88a
Use the parent's logging context name for runWithConnection. (#9895) 
This fixes a regression where the logging context for runWithConnection
was reported as runWithConnection instead of the connection name,
e.g. "POST-XYZ".
 2021-04-28 07:44:52 -04:00
Andrew Morgan
fa6679e794 Synapse 1.33.0rc1 (2021-04-28) 
==============================
 
 Features
 --------
 
 - Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814))
 - Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850))
 - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832))
 
 Bugfixes
 --------
 
 - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726))
 - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788))
 - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802))
 - Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833))
 - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867))
 - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868))
 
 Improved Documentation
 ----------------------
 
 - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801))
 
 Internal Changes
 ----------------
 
 - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162))
 - Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786))
 - Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796))
 - Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815))
 - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816))
 - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817))
 - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821))
 - Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825))
 - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838))
 - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845))
 - Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855))
 - Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856))
 - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858))
 - Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871))
 - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874))
 - Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875))
 - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876))
 - Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878))
 - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879))
 - Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCJQmgTHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9EKWD/9MZG4ngjECrk3QMBlYqaipGl/l7wcl
 8vBOTxiEqIgVJxHLCRFBbtFOItdxB4YkrorIo2fqshkk+lv4CRtTD9jEKHRvS4T6
 7p1icRdRTv2K94OkJ8R9jMFlmywZFU87oHPfI2xUYg6hvOKrR+RwPvIjA7c24UZt
 6MJqDhgGDlZD7/hQdQof9O4oOJzzIgLJPk6o8E42y6c0bLlPPKgH3sh0vlenDLfE
 15thRCOeiP237YYvXSdbr7G3PI66Efhq4BwQowSrgFg+B0BR68l4747iSIeWmLQJ
 Ow6QLLFPCOhgAPC4amp9PdaaV/9NYiBcvNxlOvyQAVl/+ioEiATNHnzdNtDouCQo
 nDdSHw0Mt9D3i+rxpu5Pf0gZN9dXRGiczqnq5QKL8+EvT/4FLYeluEeduuy3rE+G
 o5OTCd3EajxynzjftuopeysNAw6zeDpbulZoTCeCumxBL2+wAod2PsyY5Ei9Gapn
 iJvExNOJX4OlkFc67jO2CK8o3sUTNEDeIDWCQ0fVKAwIt7T45ebTA/UzDNWg7YzN
 EyUp+3NZcUZBskgMB5hpcijPJoXNYzWZPews73vMPV7AfQRxDzU5xrM9AdbadBDd
 Idv6wT9ssDUA+M0aKnafvoSJZ+qE85mi2x0rRsueZNd9uO9/QHIXKIb+4PvVpj3C
 BJnZf34m568AGQ==
 =smAk
 -----END PGP SIGNATURE-----

Merge tag 'v1.33.0rc1' into develop

Synapse 1.33.0rc1 (2021-04-28)
==============================

Features
--------

- Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814))
- Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850))
- Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832))

Bugfixes
--------

- Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726))
- Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788))
- Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802))
- Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833))
- Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867))
- Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868))

Improved Documentation
----------------------

- Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801))

Internal Changes
----------------

- Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162))
- Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786))
- Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796))
- Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815))
- Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816))
- Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817))
- Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821))
- Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838))
- Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845))
- Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855))
- Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856))
- Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858))
- Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871))
- Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874))
- Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875))
- Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876))
- Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878))
- Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879))
- Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
 2021-04-28 12:12:29 +01:00
Erik Johnston
391bfe9a7b
Reduce memory footprint of caches (#9886) 2021-04-28 11:59:28 +01:00
Andrew Morgan
787de3190f 1.33.0rc1 2021-04-28 11:43:33 +01:00
Andrew Morgan
4e0fd35bc9 Revert "Experimental Federation Speedup (#9702)" 
This reverts commit 05e8c70c05.
 2021-04-28 11:38:33 +01:00
Erik Johnston
dd2d32dcdb
Add type hints to presence handler (#9885) 2021-04-28 11:07:47 +01:00
Andrew Morgan
fe604a022a
Remove various bits of compatibility code for Python <3.6 (#9879) 
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
 2021-04-27 13:13:07 +01:00
Patrick Cloke
1350b053da
Pass errors back to the client when trying multiple federation destinations. (#9868) 
This ensures that something like an auth error (403) will be
returned to the requester instead of attempting to try more
servers, which will likely result in the same error, and then
passing back a generic 400 error.
 2021-04-27 07:30:34 -04:00
Erik Johnston
0ffa5fb935
Use current state table for presence.get_interested_remotes (#9887) 
This should be a lot quicker than asking the state handler.
 2021-04-27 10:09:41 +01:00
Richard van der Hoff
3ff2251754
Improved validation for received requests (#9817) 
* Simplify `start_listening` callpath

* Correctly check the size of uploaded files
 2021-04-23 19:20:44 +01:00
Andrew Morgan
695b73c861
Allow OIDC cookies to work on non-root public baseurls (#9726) 
Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574.

As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.
 2021-04-23 18:22:47 +01:00
Richard van der Hoff
59d24c5bef
pass a reactor into SynapseSite (#9874) 2021-04-23 17:06:47 +01:00
Patrick Cloke
e83627926f
Add type hints to auth and auth_blocking. (#9876) 2021-04-23 12:02:16 -04:00
Erik Johnston
a15c003e5b
Make DomainSpecificString an attrs class (#9875) 2021-04-23 15:46:29 +01:00
Erik Johnston
9d25a0ae65
Split presence out of master (#9820) 2021-04-23 12:21:55 +01:00
Patrick Cloke
d924827da1
Check for space membership during a remote join of a restricted room (#9814) 
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join rules.

This only applies to an experimental room version, as defined in MSC3083.
 2021-04-23 07:05:51 -04:00
Erik Johnston
3853a7edfc
Only store data in caches, not "smart" objects (#9845) 2021-04-23 11:47:07 +01:00
Richard van der Hoff
51a20914a8
Limit the size of HTTP responses read over federation. (#9833) 2021-04-23 11:08:41 +01:00
Erik Johnston
177dae2704
Limit length of accepted email addresses (#9855) 2021-04-22 17:49:11 +01:00
Richard van der Hoff
69018acbd2
Clear the resync bit after resyncing device lists (#9867) 
Fixes #9866.
 2021-04-22 16:53:24 +01:00
Richard van der Hoff
294c675033
Remove synapse.types.Collection (#9856) 
This is no longer required, since we have dropped support for Python 3.5.
 2021-04-22 16:43:50 +01:00
Andrew Morgan
3186324260 Merge branch 'master' into develop 2021-04-22 11:23:56 +01:00
Andrew Morgan
55159c48e3 1.32.2 2021-04-21 18:45:39 +01:00
Richard van der Hoff
d9bd62f9d1
Make LoggingContext's name optional (#9857) 
Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55
 2021-04-21 16:39:34 +01:00
Andrew Morgan
4b2217ace2 Merge branch 'master' into develop 2021-04-21 14:55:06 +01:00
Andrew Morgan
a745531c10 1.32.1 2021-04-21 14:01:12 +01:00
Richard van der Hoff
5d281c10dd
Stop BackgroundProcessLoggingContext making new prometheus timeseries (#9854) 
This undoes part of b076bc276e.
 2021-04-21 10:03:31 +01:00
Patrick Cloke
683d6f75af
Rename handler and config modules which end in handler/config. (#9816) 2021-04-20 14:55:20 -04:00
Andrew Morgan
6982db9651 Merge branch 'master' into develop 2021-04-20 14:55:16 +01:00
Andrew Morgan
e031c7e0cc 1.32.0 2021-04-20 14:31:27 +01:00
Patrick Cloke
b076bc276e
Always use the name as the log ID. (#9829) 
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.

This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
 2021-04-20 14:19:00 +01:00
Erik Johnston
de0d088adc
Add presence federation stream (#9819) 2021-04-20 14:11:24 +01:00
Erik Johnston
db70435de7
Fix bug where we sent remote presence states to remote servers (#9850) 2021-04-20 13:37:54 +01:00
Jonathan de Jong
495b214f4f
Fix (final) Bugbear violations (#9838) 2021-04-20 11:50:49 +01:00
Andrew Morgan
71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) 
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
 2021-04-19 19:16:34 +01:00
Denis Kasak
e694a598f8
Sanity check identity server passed to bind/unbind. (#9802) 
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
 2021-04-19 17:21:46 +01:00
Erik Johnston
2b7dd21655
Don't send normal presence updates over federation replication stream (#9828) 2021-04-19 10:50:49 +01:00
Andrew Morgan
c571736c6c
User directory: use calculated room membership state instead (#9821) 
Fixes: #9797.

Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
 2021-04-16 18:17:18 +01:00
Erik Johnston
601b893352
Small speed up joining large remote rooms (#9825) 
There are a couple of points in `persist_events` where we are doing a
query per event in series, which we can replace.
 2021-04-16 14:44:55 +01:00
Richard van der Hoff
5a153772c1
remove HomeServer.get_config (#9815) 
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
 2021-04-14 19:09:08 +01:00
Patrick Cloke
936e69825a
Separate creating an event context from persisting it in the federation handler (#9800) 
This refactoring allows adding logic that uses the event context
before persisting it.
 2021-04-14 12:35:28 -04:00
Patrick Cloke
e8816c6ace Revert "Check for space membership during a remote join of a restricted room. (#9763)" 
This reverts commit cc51aaaa7a.

The PR was prematurely merged and not yet approved.
 2021-04-14 12:33:37 -04:00
Patrick Cloke
cc51aaaa7a
Check for space membership during a remote join of a restricted room. (#9763) 
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join
rules.
    
This only applies to an experimental room version, as defined in MSC3083.
 2021-04-14 12:32:20 -04:00
Jonathan de Jong
05e8c70c05
Experimental Federation Speedup (#9702) 
This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max).

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
 2021-04-14 17:19:02 +01:00
Erik Johnston
00a6db9676
Move some replication processing out of generic_worker (#9796) 
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2021-04-14 17:06:06 +01:00
rkfg
c9a2b5d402
More robust handling of the Content-Type header for thumbnail generation (#9788) 
Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
 2021-04-14 16:30:59 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786) 
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
 2021-04-14 15:34:27 +01:00
Andrew Morgan
3efd98aa1c 1.32.0rc1 2021-04-13 14:23:43 +01:00
Dan Callahan
1d5f0e3529
Bump black configuration to target py36 (#9781) 
Signed-off-by: Dan Callahan <danc@element.io>
 2021-04-13 10:41:34 +01:00
Dirk Klimpel
1fc97ee876
Add an admin API to manage ratelimit for a specific user (#9648) 2021-04-13 10:26:37 +01:00
Will Hunt
e300ef64b1
Require AppserviceRegistrationType (#9548) 
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
 2021-04-12 15:13:55 +01:00
Richard van der Hoff
f946450184
Fix duplicate logging of exceptions in transaction processing (#9780) 
There's no point logging this twice.
 2021-04-09 18:12:15 +01:00
Dirk Klimpel
48a1f4db31
Remove old admin API GET /_synapse/admin/v1/users/<user_id> (#9401) 
Related: #8334
Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25)

`GET /_synapse/admin/v1/users/<user_id>` has no
- unit tests
- documentation

API in v2 is available (#5925 - 12/2019, v1.7.0).
API is misleading. It expects `user_id` and returns a list of all users.

Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2021-04-09 09:44:40 +01:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682) 
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
 2021-04-08 22:38:54 +01:00
Richard van der Hoff
9e167d9c53 Merge remote-tracking branch 'origin/develop' into rav/drop_py35 2021-04-08 18:30:38 +01:00
Richard van der Hoff
24c58ebfc9 remove unused param on make_tuple_comparison_clause 2021-04-08 18:29:57 +01:00
Erik Johnston
be0e722fe1 Merge branch 'erikj/fix_stalled_catchup' into develop 2021-04-08 18:05:35 +01:00
Erik Johnston
3a569fb200 Fix sharded federation sender sometimes using 100% CPU. 
We pull all destinations requiring catchup from the DB in batches.
However, if all those destinations get filtered out (due to the
federation sender being sharded), then the `last_processed` destination
doesn't get updated, and we keep requesting the same set repeatedly.
 2021-04-08 17:34:07 +01:00
Richard van der Hoff
3ada9b4264 Drop support for sqlite<3.22 as well 2021-04-08 16:42:32 +01:00
Richard van der Hoff
abade34633 Require py36 and Postgres 9.6 2021-04-08 16:42:32 +01:00
Patrick Cloke
452991527a
MSC3083: Check for space membership during a local join of restricted rooms. (#9735) 
When joining a room with join rules set to 'restricted', check if the
user is a member of the spaces defined in the 'allow' key of the join rules.

This only applies to an experimental room version, as defined in MSC3083.
 2021-04-08 08:28:32 -04:00
Patrick Cloke
48d44ab142
Record more information into structured logs. (#9654) 
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
 2021-04-08 08:01:14 -04:00
Andrew Morgan
0d87c6bd12
Don't report anything from GaugeBucketCollector metrics until data is present (#8926) 
This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.
 2021-04-06 16:32:04 +01:00
Andrew Morgan
04819239ba
Add a Synapse Module for configuring presence update routing (#9491) 
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.

This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.

A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. 

The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:

* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
 2021-04-06 14:38:30 +01:00
Patrick Cloke
44bb881096
Add type hints to expiring cache. (#9730) 2021-04-06 08:58:18 -04:00
Richard van der Hoff
0ef321ff3b
Remove outdated constraint on remote_media_cache_thumbnails (#9725) 
The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key`
constraint is superceded by
`remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds
`thumbnail_method` to the unique key).

PR #7124 made an attempt to remove the old constraint, but got the name wrong,
so it didn't work. Here we update the bg update and rerun it.

Fixes #8649.
 2021-04-06 13:36:05 +01:00
Erik Johnston
5688a74cf3 Merge branch 'master' into develop 2021-04-06 13:29:29 +01:00
Erik Johnston
1d8863c67d 1.31.0 2021-04-06 13:09:56 +01:00
Patrick Cloke
d959d28730
Add type hints to the federation handler and server. (#9743) 2021-04-06 07:21:57 -04:00
Jonathan de Jong
e2b8a90897
Update mypy configuration: no_implicit_optional = True (#9742) 2021-04-05 09:10:18 -04:00
Erik Johnston
33548f37aa
Improve tracing for to device messages (#9686) 2021-04-01 17:08:21 +01:00
Dirk Klimpel
bb0fe02a52
Add order_by to list user admin API (#9691) 2021-04-01 11:28:53 +01:00
Patrick Cloke
35c5ef2d24
Add an experimental room version to support restricted join rules. (#9717) 
Per MSC3083.
 2021-03-31 16:39:08 -04:00
Denis Kasak
5ff8eb97c6
Make sample config allowed_local_3pids regex stricter. (#9719) 
The regex should be terminated so that subdomain matches of another
domain are not accepted. Just ensuring that someone doesn't shoot
themselves in the foot by copying our example.

Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
 2021-03-31 12:27:20 +00:00
Cristina
670564446c
Deprecate imp (#9718) 
Fixes #9642.

Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
 2021-03-31 12:04:27 +01:00
Richard van der Hoff
4dabcf026e
Include m.room.create in invite_room_state for Spaces (#9710) 2021-03-30 14:03:17 +01:00
Richard van der Hoff
f02663c4dd
Replace room_invite_state_types with room_prejoin_state (#9700) 
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
 2021-03-30 12:12:44 +01:00
Erik Johnston
963f4309fe
Make RateLimiter class check for ratelimit overrides (#9711) 
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited.

We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits.

Fixes #9663
 2021-03-30 12:06:09 +01:00
Erik Johnston
78e48f61bf 1.31.0rc1 2021-03-30 11:19:21 +01:00
Patrick Cloke
01dd90b0f0
Add type hints to DictionaryCache and TTLCache. (#9442) 2021-03-29 12:15:33 -04:00
Patrick Cloke
da75d2ea1f
Add type hints for the federation sender. (#9681) 
Includes an abstract base class which both the FederationSender
and the FederationRemoteSendQueue must implement.
 2021-03-29 11:43:20 -04:00
Richard van der Hoff
4bbd535450
Update the OIDC sample config (#9695) 
I've reiterated the advice about using `oidc` to migrate, since I've seen a few
people caught by this.

I've also removed a couple of the examples as they are duplicating the OIDC
documentation, and I think they might be leading people astray.
 2021-03-29 15:40:11 +01:00
Jonathan de Jong
fc53a606e4
Fix re.Pattern mypy error on 3.6 (#9703) 2021-03-29 09:40:45 -04:00
Richard van der Hoff
ad8690a26c
Fix the suggested pip incantation for cryptography (#9699) 
If you have the wrong version of `cryptography` installed, synapse suggests:

```
To install run:
    pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6''
```

However, the use of ' inside '...' doesn't work, so when you run this, you get
an error.
 2021-03-29 11:55:33 +01:00
Richard van der Hoff
7c8402ddb8
Suppress CryptographyDeprecationWarning (#9698) 
This warning is somewhat confusing to users, so let's suppress it
 2021-03-26 17:33:55 +00:00
Erik Johnston
b5efcb577e
Make it possible to use dmypy (#9692) 
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
 2021-03-26 16:49:46 +00:00
Erik Johnston
019010964d Merge branch 'master' into develop 2021-03-26 12:26:58 +00:00
Erik Johnston
c6f8e8086c 1.30.1 2021-03-26 12:03:29 +00:00
Erik Johnston
12d6184713
Explicitly upgrade openssl in docker file and enforce new version of cryptography (#9697) 2021-03-26 12:00:25 +00:00
Andrew Morgan
fae81f2f68
Add a storage method for returning all current presence from all users (#9650) 
Split off from https://github.com/matrix-org/synapse/pull/9491

Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: b588f16e39/synapse/handlers/presence.py (L1131-L1133)

Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
 2021-03-25 10:34:23 +00:00
Erik Johnston
c602ba8336
Fixed undefined variable error in catchup (#9664) 
Broke in #9640

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-03-24 16:12:47 +00:00
Jonathan de Jong
4c3827f2c1
Enable addtional flake8-bugbear linting checks. (#9659) 2021-03-24 09:34:30 -04:00
Richard van der Hoff
c73cc2c2ad
Spaces summary: call out to other servers (#9653) 
When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data.

Fixes: #9447
 2021-03-24 12:45:39 +00:00
Patrick Cloke
af387cf52a
Add type hints to misc. files. (#9676) 2021-03-24 06:49:01 -04:00
Patrick Cloke
7e8dc9934e
Add a type hints for service notices to the HomeServer object. (#9675) 2021-03-24 06:48:46 -04:00
Erik Johnston
e550ab17ad
Increase default join burst ratelimiting (#9674) 
It's legitimate behaviour to try and join a bunch of rooms at once.
 2021-03-23 14:52:20 +00:00
Jonathan de Jong
0caf2a338e
Fix federation stall on concurrent access errors (#9639) 2021-03-23 13:52:30 +00:00
Richard van der Hoff
4ecba9bd5c
Federation API for Space summary (#9652) 
Builds on the work done in #9643 to add a federation API for space summaries.

There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
 2021-03-23 11:51:12 +00:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. (#9665) 2021-03-23 07:12:48 -04:00
Andrew Morgan
5b268997bd
Allow providing credentials to HTTPS_PROXY (#9657) 
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70

This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.

Also added some type hints.
 2021-03-22 17:20:47 +00:00
Ankit Dobhal
d66f9070cd
Fixed code misc. quality issues (#9649) 
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
 2021-03-22 11:18:13 -04:00
Erik Johnston
d600d4506b Merge branch 'master' into develop 2021-03-22 13:36:36 +00:00
Erik Johnston
e2904f720d 1.30.0 2021-03-22 13:15:55 +00:00
Brendan Abolivier
b6ed4f55ac
Incorporate review 2021-03-19 18:19:50 +01:00
Brendan Abolivier
592d6305fd
Merge branch 'develop' into babolivier/msc3026 2021-03-19 16:12:40 +01:00
Brendan Abolivier
0b56481caa
Fix lint 2021-03-19 16:11:08 +01:00
Richard van der Hoff
066068f034 fix mypy 2021-03-19 12:20:11 +00:00
Patrick Cloke
9b0e3009fa Fix type-hints from bad merge. 2021-03-18 14:40:56 -04:00
Richard van der Hoff
004234f03a
Initial spaces summary API (#9643) 
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
 2021-03-18 18:24:16 +00:00
Brendan Abolivier
066c703729
Move support for MSC3026 behind an experimental flag 2021-03-18 18:37:19 +01:00
Dirk Klimpel
8dd2ea65a9
Consistently check whether a password may be set for a user. (#9636) 2021-03-18 12:54:08 -04:00
Erik Johnston
dd71eb0f8a
Make federation catchup send last event from any server. (#9640) 
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.

Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
 2021-03-18 15:52:26 +00:00
Brendan Abolivier
405aeb0b2c
Implement MSC3026: busy presence state 2021-03-18 16:34:47 +01:00
Andrew Morgan
7b06f85c0e
Ensure we use a copy of the event content dict before modifying it in serialize_event (#9585) 
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)) - will raise a 500 if you try to a edit a reply to a message.

`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.

To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
 2021-03-17 16:51:55 +00:00
Patrick Cloke
cc324d53fe
Fix up types for the typing handler. (#9638) 
By splitting this to two separate methods the callers know
what methods they can expect on the handler.
 2021-03-17 11:30:21 -04:00
Hubert Chathi
73dbce5523
only save remote cross-signing keys if they're different from the current ones (#9634) 
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-03-17 11:04:57 -04:00
Erik Johnston
ad721fc559
Fix bad naming of storage function (#9637) 
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
 2021-03-17 13:20:08 +00:00
Richard van der Hoff
567f88f835
Prep work for removing outlier from internal_metadata (#9411) 
* Populate `internal_metadata.outlier` based on `events` table

Rather than relying on `outlier` being in the `internal_metadata` column,
populate it based on the `events.outlier` column.

* Move `outlier` out of InternalMetadata._dict

Ultimately, this will allow us to stop writing it to the database. For now, we
have to grandfather it back in so as to maintain compatibility with older
versions of Synapse.
 2021-03-17 12:33:18 +00:00
Patrick Cloke
b449af0379
Add type hints to the room member handler. (#9631) 2021-03-17 07:14:39 -04:00
Jonathan de Jong
27d2820c33
Enable flake8-bugbear, but disable most checks. (#9499) 
* Adds B00 to ignored checks.
* Fixes remaining issues.
 2021-03-16 14:19:27 -04:00
Hubbe
dd5e5dc1d6
Add SSO attribute requirements for OIDC providers (#9609) 
Allows limiting who can login using OIDC via the claims
made from the IdP.
 2021-03-16 11:46:07 -04:00
Dirk Klimpel
8000cf1315
Return m.change_password.enabled=false if local database is disabled (#9588) 
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
 2021-03-16 11:44:25 -04:00
Andrew Morgan
d315e96443 1.30.0rc1 2021-03-16 13:45:46 +00:00
Andrew Morgan
847ecdd8fa
Pass SSO IdP information to spam checker's registration function (#9626) 
Fixes https://github.com/matrix-org/synapse/issues/9572

When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.

This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
 2021-03-16 12:41:41 +00:00
Patrick Cloke
1383508f29
Handle an empty cookie as an invalid macaroon. (#9620) 
* Handle an empty cookie as an invalid macaroon.

* Newsfragment
 2021-03-16 11:29:35 +00:00
Richard van der Hoff
dd69110d95
Add support for stable MSC2858 API (#9617) 
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
 2021-03-16 11:21:26 +00:00
Richard van der Hoff
5b5bc188cf
Clean up config settings for stats (#9604) 
... and complain if people try to turn it off.
 2021-03-16 10:57:54 +00:00
Andrew Morgan
1b0eaed21f
Prevent bundling aggregations for state events (#9619) 
There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
 2021-03-16 10:27:51 +00:00
Richard van der Hoff
1c8a2541da
Fix Internal Server Error on GET /saml2/authn_response (#9623) 
* Fix Internal Server Error on `GET /saml2/authn_response`

Seems to have been introduced in #8765 (Synapse 1.24.0)

* Fix newsfile
 2021-03-16 10:20:20 +00:00
Patrick Cloke
d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. (#9608) 2021-03-15 11:14:39 -04:00
Erik Johnston
026503fa3b
Don't go into federation catch up mode so easily (#9561) 
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.

Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
 2021-03-15 14:42:40 +00:00
Richard van der Hoff
af2248f8bf
Optimise missing prev_event handling (#9601) 
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from 
the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events,
we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote
server for the state at those prev_events, and then we may need to then ask the remote server for any events
in that state which we don't already have, as well as the auth events for those missing state events, so that we
can auth them.

This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state
events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the
observation that we are currently loading all of those auth events into memory at the start of the operation, but
we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and
leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're
actually going to need, but it doesn't.)

The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about
60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number
of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache
churn. (NB I've already tripled the size of the cache from its default of 10K).

Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into
a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be
returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different
(ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting
that we clean it up.
 2021-03-15 13:51:02 +00:00
Patrick Cloke
55da8df078
Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
Richard van der Hoff
1e67bff833
Reject concurrent transactions (#9597) 
If more transactions arrive from an origin while we're still processing the
first one, reject them.

Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489
 2021-03-12 15:14:55 +00:00
Richard van der Hoff
2b328d7e02
Improve logging when processing incoming transactions (#9596) 
Put the room id in the logcontext, to make it easier to understand what's going on.
 2021-03-12 15:08:03 +00:00
Richard van der Hoff
464e5da7b2
Add logging for redis connection setup (#9590) 2021-03-11 18:35:09 +00:00
Patrick Cloke
e55bd0e110
Add tests for blacklisting reactor/agent. (#9563) 2021-03-11 09:15:22 -05:00
Dirk Klimpel
70d1b6abff
Re-Activating account when local passwords are disabled (#9587) 
Fixes: #8393
 2021-03-11 13:52:32 +00:00
Richard van der Hoff
a7a3790066
Convert Requester to attrs (#9586) 
... because namedtuples suck

Fix up a couple of other annotations to keep mypy happy.
 2021-03-10 18:15:56 +00:00
Richard van der Hoff
1107214a1d
Fix the auth provider on the logins metric (#9573) 
We either need to pass the auth provider over the replication api, or make sure
we report the auth provider on the worker that received the request. I've gone
with the latter.
 2021-03-10 18:15:03 +00:00
Patrick Cloke
2a99cc6524
Use the chain cover index in get_auth_chain_ids. (#9576) 
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
 2021-03-10 09:57:59 -05:00
Patrick Cloke
918f6ed827
Fix a bug in the background task for purging chain cover. (#9583) 2021-03-10 08:55:52 -05:00
Patrick Cloke
dc51d8ffaf
Add a background task to purge unused chain IDs. (#9542) 
This is a companion change to apply the fix in #9498 /
922788c604 to previously
purged rooms.
 2021-03-09 11:22:25 -05:00
Richard van der Hoff
eaada74075
JWT OIDC secrets for Sign in with Apple (#9549) 
Apple had to be special. They want a client secret which is generated from an EC key.

Fixes #9220. Also fixes #9212 while I'm here.
 2021-03-09 15:03:37 +00:00
Erik Johnston
9cd18cc588
Retry 5xx errors in federation client (#9567) 
Fixes #8915
 2021-03-09 13:15:12 +00:00
Patrick Cloke
7fdc6cefb3
Fix additional type hints. (#9543) 
Type hint fixes due to Twisted 21.2.0 adding type hints.
 2021-03-09 07:41:32 -05:00
Patrick Cloke
075c16b410
Handle image transparency better when thumbnailing. (#9473) 
Properly uses RGBA mode for 1- and 8-bit images with transparency
(instead of RBG mode).
 2021-03-09 07:37:09 -05:00
Jonathan de Jong
9898470e7d
Add logging to ObservableDeferred callbacks (#9523) 2021-03-09 11:09:31 +00:00
Matthew Hodgson
0764d0c6e5 quick config comment tweak to clarify allow_profile_lookup_over_federation 2021-03-08 21:52:04 +00:00
Jonathan de Jong
d6196efafc
Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
Dirk Klimpel
7076eee4b9
Add type hints to purge room and server notice admin API. (#9520) 2021-03-08 10:34:38 -05:00
Erik Johnston
b988b07bb0 Merge branch 'master' into develop 2021-03-08 14:06:35 +00:00
Erik Johnston
15c788e22d 1.29.0 2021-03-08 13:52:13 +00:00
Patrick Cloke
58114f8a17
Create a SynapseReactor type which incorporates the necessary reactor interfaces. (#9528) 
This helps fix some type hints when running with Twisted 21.2.0.
 2021-03-08 08:25:43 -05:00
Richard van der Hoff
8a4b3738f3
Replace last_*_pdu_age metrics with timestamps (#9540) 
Following the advice at
https://prometheus.io/docs/practices/instrumentation/#timestamps-not-time-since,
it's preferable to export unix timestamps, not ages.

There doesn't seem to be any particular naming convention for timestamp
metrics.
 2021-03-04 16:40:18 +00:00
Richard van der Hoff
df425c2c63
Prometheus metrics for logins and registrations (#9511) 
Add prom metrics for number of users successfully registering and logging in, by SSO provider.
 2021-03-04 16:39:27 +00:00
Richard van der Hoff
7eb6e39a8f
Record the SSO Auth Provider in the login token (#9510) 
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
 2021-03-04 14:44:22 +00:00
Erik Johnston
3f49d80dcf 1.29.0rc1 2021-03-04 10:12:53 +00:00
Patrick Cloke
33a02f0f52
Fix additional type hints from Twisted upgrade. (#9518) 2021-03-03 15:47:38 -05:00
Richard van der Hoff
4db07f9aef
Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539) 
Should fix some remaining warnings
 2021-03-03 18:49:08 +00:00
Erik Johnston
a4fa044c00
Fix 'rejected_events_metadata' background update (#9537) 
Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.
 2021-03-03 16:04:24 +00:00
Patrick Cloke
922788c604
Purge chain cover tables when purging events. (#9498) 2021-03-03 11:04:08 -05:00
Dirk Klimpel
d790d0d314
Add type hints to user admin API. (#9521) 2021-03-03 08:09:39 -05:00
Patrick Cloke
0c330423bc
Bump the mypy and mypy-zope versions. (#9529) 2021-03-03 07:19:19 -05:00
Erik Johnston
16f9f93eb7
Make deleting stale pushers a background update (#9536) 2021-03-03 12:08:16 +00:00
Aaron Raimist
0279e0e086
Prevent presence background jobs from running when presence is disabled (#9530) 
Prevent presence background jobs from running when presence is disabled

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2021-03-03 10:21:46 +00:00
Patrick Cloke
aee10768d8 Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)" 
This reverts commit f5c93fc993.

This is being backed out due to a regression (#9507) and additional
review feedback being provided.
 2021-03-02 09:43:34 -05:00
Erik Johnston
7f5d753d06
Re-run rejected metadata background update. (#9503) 
It landed in schema version 58 after 59 had been created, causing some
servers to not run it. The main effect of was that not all rooms had
their chain cover calculated correctly. After the BG updates complete
the chain covers will get fixed when a new state event in the affected
rooms is received.
 2021-03-02 14:31:23 +00:00
Erik Johnston
16108c579d
Fix SQL delta file taking a long time to run (#9516) 
Fixes #9504
 2021-03-02 14:05:01 +00:00
Dirk Klimpel
f00c4e7af0
Add type hints to device and event report admin API (#9519) 2021-03-02 09:31:12 +00:00
Patrick Cloke
ad8589d392
Fix a bug when a room alias is given to the admin join endpoint (#9506) 2021-03-01 13:59:01 -05:00
Patrick Cloke
16ec8c3272
(Hopefully) stop leaking file descriptors in media repo. (#9497) 
By consuming the response if the headers imply that the
content is too large.
 2021-03-01 12:45:00 -05:00
Patrick Cloke
a0bc9d387e
Use the proper Request in type hints. (#9515) 
This also pins the Twisted version in the mypy job for CI until
proper type hints are fixed throughout Synapse.
 2021-03-01 12:23:46 -05:00
Jonathan de Jong
e12077a78a
Allow bytecode again (#9502) 
In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again.

Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
 2021-02-26 18:30:54 +00:00
Tim Leung
ddb240293a
Add support for no_proxy and case insensitive env variables (#9372) 
### Changes proposed in this PR

- Add support for the `no_proxy` and `NO_PROXY` environment variables
  - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519))
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment

This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`

Related to #9306 which also uses `ProxyAgent`

Signed-off-by: Timothy Leung tim95@hotmail.co.uk
 2021-02-26 17:37:57 +00:00
Richard van der Hoff
15090de850
SSO: redirect to public URL before setting cookies (#9436) 
... otherwise, we don't get the cookie back.
 2021-02-26 14:02:06 +00:00
Erik Johnston
2566dc57ce
Test that we require validated email for email pushers (#9496) 2021-02-25 15:35:14 +00:00
Erik Johnston
1e62d9ee8c
Ensure pushers are deleted for deactivated accounts (#9285) 2021-02-25 13:56:55 +00:00
Erik Johnston
1efdcc3e87 Merge branch 'master' into develop 2021-02-25 10:53:31 +00:00
Erik Johnston
b5c4fe1971 1.28.0 2021-02-25 10:22:07 +00:00
Richard van der Hoff
d8e95e5452
Add support for X-Forwarded-Proto (#9472) 
rewrite XForwardedForRequest to set `isSecure()` based on
`X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
 2021-02-24 18:11:33 +00:00
Erik Johnston
7cc571510b
Add SQL delta for deleting stale pushers (#9479) 2021-02-24 17:21:10 +00:00
Jonathan de Jong
f5c93fc993
Fix #8518 (sync requests being cached wrongly on timeout) (#9358) 
This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false".

This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
 2021-02-24 13:57:00 +00:00
Erik Johnston
2927921942
Clean up ShardedWorkerHandlingConfig (#9466) 
* Split ShardedWorkerHandlingConfig

This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).

* Remove special cases in ShardedWorkerHandlingConfig.

`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.

To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
 2021-02-24 13:23:18 +00:00
Erik Johnston
0b5c967813
Refactor to ensure we call check_consistency (#9470) 
The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
 2021-02-24 10:13:53 +00:00
Patrick Cloke
7292b7c0eb
Add back the deprecated SAML endpoint. (#9474) 2021-02-23 12:57:37 -05:00
Richard van der Hoff
713145d3de
Add a comment about systemd-python. (#9464) 
This confused me for a while.
 2021-02-23 13:42:36 +00:00
Patrick Cloke
65a9eb8994
Include newly added sequences in the port DB script. (#9449) 
And ensure the consistency of `event_auth_chain_id`.
 2021-02-23 07:33:24 -05:00
Erik Johnston
66f4949e7f
Fix deleting pushers when using sharded pushers. (#9465) 2021-02-22 21:14:42 +00:00
Richard van der Hoff
1b2d6d55c5
Remove vestiges of uploads_path config (#9462) 
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
 2021-02-22 19:54:49 +00:00
Dirk Klimpel
71c9f8de6d
Add an order_by field to list users' media admin API. (#8978) 2021-02-22 14:38:51 -05:00
Andrew Morgan
0a363f9ca4
Remove cache for get_shared_rooms_for_users (#9416) 
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.

The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately.

Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
 2021-02-22 16:52:45 +00:00
Andrew Morgan
e22b71810e
Clean up the user directory sample config section (#9385) 
The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).

This PR cleans that up a bit.
 2021-02-22 11:44:31 +00:00
Patrick Cloke
fc8b3d8809
Ratelimit cross-user key sharing requests. (#8957) 2021-02-19 13:20:34 -05:00
Erik Johnston
179c0953ff
Regenerate exact thumbnails if missing (#9438) 2021-02-19 17:09:57 +00:00
Patrick Cloke
a1901abd6b
Add documentation and type hints to parse_duration. (#9432) 2021-02-19 08:32:21 -05:00
Patrick Cloke
c4a55ac4a4 Fix style checking due to updated black. 2021-02-19 08:19:54 -05:00
Patrick Cloke
d9f1dccba9 Synapse 1.28.0rc1 (2021-02-19) 
==============================
 
 Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).
 
 This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.
 
 Removal warning
 ---------------
 
 The v1 list accounts API is deprecated and will be removed in a future release.
 This API was undocumented and misleading. It can be replaced by the
 [v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
 which has been available since Synapse 1.7.0 (2019-12-13).
 
 Please check if you're using any scripts which use the admin API and replace
 `GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.
 
 Features
 --------
 
 - New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150))
 - Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301))
 - Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311))
 - Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376))
 - Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400))
 
 Bugfixes
 --------
 
 - Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257))
 - Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296))
 - Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393))
 - Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333))
 - Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361))
 - Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391))
 - Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395))
 - Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421))
 - Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425))
 - The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426))
 - Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428))
 
 Updates to the Docker image
 ---------------------------
 
 - Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433))
 
 Improved Documentation
 ----------------------
 
 - Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281))
 - Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291))
 - Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308))
 - Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317))
 - Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322))
 - Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390))
 - Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404))
 - Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407))
 - Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434))
 
 Deprecations and Removals
 -------------------------
 
 - Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429))
 
 Internal Changes
 ----------------
 
 - Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003))
 - Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123))
 - Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240))
 - Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299))
 - Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305))
 - Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307))
 - Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326))
 - Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362))
 - Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396))
 - Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381))
 - Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384))
 - Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394))
 - Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAvtsUACgkQM/xY9qcR
 MEgEmA/+O3fCMKdyh4soBirnEZa88m1HL4Jyd+jYl7+IuFChZz0k+RYLdQ376GaC
 tmT4aGQISdu1wYUwO6zpN5ercvFCJ5OaSJzHinEaHZMnEfWe9SyMNB1oD9h6UYSO
 8r7OT2U28pTTAbj/VpSNK8osbx/F9T5o1a7v3PLkB1kJcK9j9oqNrqpdsSW2Cz1B
 rGlBKRSICKR4bzfk3dR9Wr4Fcyb+gINn0LTtYYqMWxP1EtRC+VI3NEWOk3OD2dzk
 UztvUn0F7rKFmtcBRTXsxuGiGI9IGrd9tyKc7ilBz24eSA95psperUtHNKGwSUaO
 kt+bdMfEKM2cMnydMuiCYtdl/M59Rf3ShRE3OZd7TK4S4NoPy6whFsAJf7mBGxfp
 RcMwO/7mWRX5qjDRG6x5nM+aCqO0Y8h8WNF4gZtiN2XGyxlU0TBQMBwBM66qAoXO
 zCPt4tMlLsOBCABNkWT3W5Ww5KxNeMp2qEA9UnmjezA5XHtSx/DVN+6D5E4Rxw2v
 PzS278Sq3YO/JK6TfXsFBkZvDFPmCDdTQfKpoDbT6Hv16CXTz66XdiEg5BPZYIvp
 bNurfxr22/RiScxiA73cXbRBHq2qAPTxZo1O40R2LnhX0wQPQ5wfUW8oxZOnuhZ3
 bWw4PvhWsZDt2SB9EIKf4RQtSxNlar88C6tpHSC89ScmE6PPWnI=
 =Srwm
 -----END PGP SIGNATURE-----

Merge tag 'v1.28.0rc1' into develop

Synapse 1.28.0rc1 (2021-02-19)
==============================

Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).

This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.

Removal warning
---------------

The v1 list accounts API is deprecated and will be removed in a future release.
This API was undocumented and misleading. It can be replaced by the
[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
which has been available since Synapse 1.7.0 (2019-12-13).

Please check if you're using any scripts which use the admin API and replace
`GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.

Features
--------

- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150))
- Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301))
- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311))
- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376))
- Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400))

Bugfixes
--------

- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257))
- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296))
- Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393))
- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333))
- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361))
- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391))
- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395))
- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421))
- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425))
- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426))
- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428))

Updates to the Docker image
---------------------------

- Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433))

Improved Documentation
----------------------

- Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281))
- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291))
- Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308))
- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317))
- Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322))
- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390))
- Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404))
- Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407))
- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434))

Deprecations and Removals
-------------------------

- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429))

Internal Changes
----------------

- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003))
- Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123))
- Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240))
- Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299))
- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305))
- Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307))
- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326))
- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362))
- Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396))
- Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381))
- Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384))
- Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394))
- Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))
 2021-02-19 08:10:19 -05:00
Patrick Cloke
b114a45f5f
Support not providing an IdP icon when choosing a username. (#9440) 2021-02-19 07:48:46 -05:00
Andrew Morgan
8bcfc2eaad
Be smarter about which hosts to send presence to when processing room joins (#9402) 
This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually.

---

When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed.

It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw.

This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence:

* If it was a local user join, send that user's latest presence to all servers in the room
* If it was a remote user join, send the presence for all local users in the room to that homeserver

We deduplicate by inserting all of those pending updates into a dictionary of the form:

```
{
  server_name1: {presence_update1, ...},
  server_name2: {presence_update1, presence_update2, ...}
}
```

Only after building this dict do we then start sending out presence updates.
 2021-02-19 11:37:29 +00:00
Andrew Morgan
13e9029f44
Add a config option to prioritise local users in user directory search results (#9383) 
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.

Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.

This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
 2021-02-19 11:02:03 +00:00
Erik Johnston
3d2acc930f Return a 404 if we don't have the original file 2021-02-19 10:46:18 +00:00
AndrewFerr
9bc74743d5
Add configs to make profile data more private (#9203) 
Add off-by-default configuration settings to:
- disable putting an invitee's profile info in invite events
- disable profile lookup via federation

Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
 2021-02-19 09:50:41 +00:00
Patrick Cloke
1381cd05b0 1.28.0rc1 2021-02-18 12:32:49 -05:00
Erik Johnston
b106080fb4 Regenerate exact thumbnails if missing 2021-02-18 17:05:32 +00:00
Patrick Cloke
9ee3b9775f
Remove deprecated SAML2 callback URL since it does not work. (#9434) 
Updates documentation from #9289 and removes a deprecated
endpoint which didn't work as expected.
 2021-02-18 11:20:33 -05:00
Rishabh Arya
e17553e185
Parse ui_auth.session_timeout as a duration (instead of treating it as ms) (#9426) 2021-02-18 09:18:14 -05:00
Patrick Cloke
8ec2217103
Reduce the memory usage of previewing media files. (#9421) 
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.

It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
 2021-02-18 09:01:29 -05:00
Patrick Cloke
43f1c82457
Add back the guard against the user directory stream position not existing. (#9428) 
As the comment says, this guard was there for when the
initial user directory update has yet to happen.
 2021-02-18 08:44:19 -05:00
Dirk Klimpel
c8d9383cfb
Add the shadow-banning status to the display user admin API. (#9400) 2021-02-17 15:19:23 -05:00
Andrew Morgan
a25661b2eb
Remove dead notify_for_states presence method (#9408) 2021-02-17 17:32:26 +00:00
Andrew Morgan
3e5749b99f
Fix only handling the last presence state for each user (#9425) 
This is a small bug that I noticed while working on #8956.

We have a for-loop which attempts to strip all presence changes for each user except for the final one, as we don't really care about older presence:

9e19c6aab4/synapse/handlers/presence.py (L368-L371)

`new_states_dict` stores this stripped copy of latest presence state for each user, before it is... put into a new variable `new_state`, which is just overridden by the subsequent for loop.

I believe this was instead meant to override `new_states`. Without doing so, it effectively meant:

1. The for loop had no effect.
2. We were still processing old presence state for users.
 2021-02-17 17:31:37 +00:00
Patrick Cloke
d2f0ec12d5
Add type hints to groups code. (#9393) 2021-02-17 08:41:47 -05:00
Richard van der Hoff
e1071fd625
Support for form_post in OIDC responses (#9376) 
Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
 2021-02-17 10:15:14 +00:00
Richard van der Hoff
33f64ca7d6
Allow OIDC config to override discovered values (#9384) 
Fixes #9347
 2021-02-16 22:33:09 +00:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381) 
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
 2021-02-16 22:32:34 +00:00
Richard van der Hoff
3b754aea27
Clean up caching/locking of OIDC metadata load (#9362) 
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
 2021-02-16 16:27:38 +00:00
Erik Johnston
0ad087273c Merge branch 'master' into develop 2021-02-16 13:39:30 +00:00
Patrick Cloke
731e08c63a
Handle missing data in power levels events during room upgrade. (#9395) 2021-02-16 08:31:39 -05:00
Erik Johnston
a27c1fd74b 1.27.0 2021-02-16 13:12:02 +00:00
Andrew Morgan
594f2853e0
Remove dead handled_events set in invite_join (#9394) 
This PR removes a set that was created and [initially used](1d2a0040cf (diff-0bc92da3d703202f5b9be2d3f845e375f5b1a6bc6ba61705a8af9be1121f5e42R435-R436)), but is no longer today.

May help cut down a bit on the time it takes to accept invites.
 2021-02-12 22:15:50 +00:00
Patrick Cloke
7950aa8a27 Fix some typos. 2021-02-12 11:14:12 -05:00
Patrick Cloke
2c9b4a5f16 Synapse 1.27.0rc2 (2021-02-11) 
==============================
 
 Features
 --------
 
 - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
 
 Bugfixes
 --------
 
 - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
 - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
 
 Improved Documentation
 ----------------------
 
 - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR
 MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS
 /tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN
 309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL
 GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb
 /QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP
 xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn
 ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B
 iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4
 awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU
 2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H
 lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0=
 =NSyp
 -----END PGP SIGNATURE-----

Merge tag 'v1.27.0rc2' into develop

Synapse 1.27.0rc2 (2021-02-11)
==============================

Features
--------

- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))

Bugfixes
--------

- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))

Improved Documentation
----------------------

- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 2021-02-11 11:56:03 -05:00
Patrick Cloke
40de534238 1.27.0rc2 2021-02-11 11:22:29 -05:00
Patrick Cloke
e40d88cff3
Backout changes for automatically calculating the public baseurl. (#9313) 
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
 2021-02-11 11:16:54 -05:00
Erik Johnston
6aa87f8ce3
Ensure that we never stop reconnecting to redis (#9391) 2021-02-11 16:06:29 +00:00
Patrick Cloke
6dade80048
Combine the CAS & SAML implementations for required attributes. (#9326) 2021-02-11 10:05:15 -05:00
Eric Eastwood
80d6dc9783
Remove conflicting sqlite tables that are "reserved" (shadow fts4 tables) (#9003) 
Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests.

Fix #8996
 2021-02-10 20:12:57 +00:00
Brendan Abolivier
29ae04af3b
Remove unneeded type constraints on 3rd party protocol lookup responses 2021-02-09 17:50:25 +01:00
Patrick Cloke
3f58fc848d
Type hints and validation improvements. (#9321) 
* Adds type hints to the groups servlet and stringutils code.
* Assert the maximum length of some input values for spec compliance.
 2021-02-08 13:59:54 -05:00
Patrick Cloke
0963d39ea6
Handle additional errors when previewing URLs. (#9333) 
* Handle the case of lxml not finding a document tree.
* Parse the document encoding from the XML tag.
 2021-02-08 12:33:30 -05:00
David Teller
b0b2cac057
Merge pull request #9150 from Yoric/develop-context 
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
 2021-02-08 15:53:44 +01:00
Jonathan de Jong
d882fbca38
Update type hints for Cursor to match PEP 249. (#9299) 2021-02-05 15:39:19 -05:00
Erik Johnston
adc96d4236 Merge branch 'erikj/media_spam_checker' into develop 2021-02-04 17:01:59 +00:00
Erik Johnston
7e8083eb48 Add check_media_file_for_spam spam checker hook 2021-02-04 17:01:30 +00:00
Patrick Cloke
792263c97c
Handle empty rooms when generating email notifications. (#9257) 
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
 2021-02-04 10:18:25 -05:00
Patrick Cloke
2ab6e67ab7
Fix escaping of braces in OIDC sample config. (#9317) 
This fixes the Jinja2 templates for the mapping provider.
 2021-02-04 09:06:20 -05:00
Jonathan de Jong
2814028ce5
Add experimental support for PyPy. (#9123) 
* Adds proper dependencies.
* Minor fixes in database layer.
 2021-02-04 08:29:47 -05:00
Marcus
b0f4119b8b
Add debug logging to DNS SRV requests. (#9305) 2021-02-03 16:47:30 -05:00
Richard van der Hoff
3f534d3fdf Merge branch 'social_login_hotfixes' into develop 2021-02-03 20:34:27 +00:00
Richard van der Hoff
17f2a512f3 Merge remote-tracking branch 'origin/release-v1.27.0' into social_login_hotfixes 2021-02-03 20:33:32 +00:00
Richard van der Hoff
e288499c60
Social login UI polish (#9301) 2021-02-03 20:31:23 +00:00
Patrick Cloke
afa18f1baa
Clarify documentation about escaping URLs in templates. (#9310) 2021-02-03 14:51:38 -05:00
Richard van der Hoff
ce669863b9
Add debug for OIDC flow (#9307) 2021-02-03 19:45:34 +00:00
Richard van der Hoff
7a0dcea3e5
social login Fix username validation javascript (#9297) 
* fix validation and don't use built-in validation UI

Co-authored-by: Bruno Windels <brunow@element.io>
 2021-02-03 17:52:55 +00:00
Richard van der Hoff
f20dadb649
Fix formatting for "bad session" error during sso registration flow (#9296) 2021-02-03 16:13:09 +00:00
dykstranet
e4cdecb310
config: Add detail to auto_join_rooms comment (#9291) 
config: Add detail to auto_join_rooms comment

Signed-off-by: Gary Dykstra <gary@dykstranet.com>
 2021-02-03 15:21:30 +00:00
Tim Gates
e1943d1353
Typo fix in a comment: subequently -> subsequently. (#8988) 2021-02-03 07:24:53 -05:00
Patrick Cloke
4ca054a4ea
Convert blacklisted IPv4 addresses to compatible IPv6 addresses. (#9240) 
Also add a few more IP ranges to the default blacklist.
 2021-02-03 07:13:46 -05:00
Erik Johnston
ff55300b91
Honour ratelimit flag for application services for invite ratelimiting (#9302) 2021-02-03 10:17:37 +00:00
Richard van der Hoff
96e460df2e
social login: add noopener to terms link (#9300) 2021-02-02 18:35:28 +00:00
Erik Johnston
2610930721 1.27.0rc1 2021-02-02 13:32:05 +00:00
Travis Ralston
b60bb28bbc
Add an admin API to get the current room state (#9168) 
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.

This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
 2021-02-02 11:16:29 +00:00
Richard van der Hoff
8f75bf1df7
Put SAML callback URI under /_synapse/client. (#9289) 2021-02-02 09:43:50 +00:00
Richard van der Hoff
846b9d3df0
Put OIDC callback URI under /_synapse/client. (#9288) 2021-02-01 22:56:01 +00:00
Richard van der Hoff
8fee6a3ab2 Merge branch 'social_login' into develop 2021-02-01 18:48:11 +00:00
Richard van der Hoff
351845452c fix broken HTML tag 2021-02-01 18:47:01 +00:00
Richard van der Hoff
5963426b95 Merge branch 'social_login' into develop 2021-02-01 18:46:12 +00:00
Bruno Windels
f30c3a99be make primary button not wider than viewport 2021-02-01 18:39:17 +00:00
Richard van der Hoff
c543bf87ec
Collect terms consent from the user during SSO registration (#9276) 2021-02-01 18:37:41 +00:00
Richard van der Hoff
e5d70c8a82
Improve styling and wording of SSO UIA templates (#9286) 
fixes #9171
 2021-02-01 18:36:04 +00:00
Patrick Cloke
5d38a3c97f
Refactor email summary generation. (#9260) 
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
  was tweaked to depend on whether there was 1 sender or multiple senders,
  not based on if there was 1 room or multiple rooms.
 2021-02-01 13:09:39 -05:00
Richard van der Hoff
419313b06a
Improve styling and wording of SSO error templates (#9287) 2021-02-01 18:01:15 +00:00
Richard van der Hoff
85c56b5a67
Make importing display name and email optional (#9277) 2021-02-01 17:30:42 +00:00
Richard van der Hoff
18ab35284a Merge branch 'social_login' into develop 2021-02-01 17:28:37 +00:00
Jan Christian Grünhage
43dd93bb26
Add phone home stats for encrypted messages. (#9283) 
Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
 2021-02-01 17:06:22 +00:00
Andrew Morgan
a800603561
Prevent email UIA failures from raising a LoginError (#9265) 
Context, Fixes: https://github.com/matrix-org/synapse/issues/9263

In the past to fix an issue with old Riots re-requesting threepid validation tokens, we raised a `LoginError` during UIA instead of `InteractiveAuthIncompleteError`. This is now breaking the way Tchap logs in - which isn't standard, but also isn't disallowed by the spec.

An easy fix is just to remove the 4 year old workaround.
 2021-02-01 15:54:39 +00:00
Richard van der Hoff
4167494c90
Replace username picker with a template (#9275) 
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.

I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
 2021-02-01 15:52:50 +00:00
Richard van der Hoff
8aed29dc61
Improve styling and wording of SSO redirect confirm template (#9272) 2021-02-01 15:50:56 +00:00
Richard van der Hoff
9c715a5f19
Fix SSO on workers (#9271) 
Fixes #8966.

* Factor out build_synapse_client_resource_tree

Start a function which will mount resources common to all workers.

* Move sso init into build_synapse_client_resource_tree

... so that we don't have to do it for each worker

* Fix SSO-login-via-a-worker

Expose the SSO login endpoints on workers, like the documentation says.

* Update workers config for new endpoints

Add documentation for endpoints recently added (#8942, #9017, #9262)

* remove submit_token from workers endpoints list

this *doesn't* work on workers (yet).

* changelog

* Add a comment about the odd path for SAML2Resource
 2021-02-01 15:47:59 +00:00
Richard van der Hoff
f78d07bf00
Split out a separate endpoint to complete SSO registration (#9262) 
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
 2021-02-01 13:15:51 +00:00
Ivan Shapovalov
13c7ab8181
Fixes for PyPy compatibility (#9270) 
* synapse.app.base: only call gc.freeze() on CPython

gc.freeze() is an implementation detail of CPython garbage collector,
and notably does not exist on PyPy.

Rather than playing whack-a-mole and skipping the call when under PyPy,
simply restrict it to CPython because the whole gc module is
implementation-defined.

Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
 2021-01-30 17:22:05 +00:00
Erik Johnston
f2c1560eca
Ratelimit invites by room and target user (#9258) 2021-01-29 16:38:29 +00:00
Richard van der Hoff
0d81a6fa3e Merge branch 'social_login' into develop 2021-01-28 22:08:11 +00:00
Erik Johnston
4b73488e81
Ratelimit 3PID /requestToken API (#9238) 2021-01-28 17:39:21 +00:00
Erik Johnston
54a6afeee3
Cache config options in SSL verification (#9255) 
Reading from the config object is *slow*.
 2021-01-28 17:38:59 +00:00
David Teller
31d072aea0 FIXUP: linter 2021-01-28 16:53:40 +01:00
Patrick Cloke
a78016dadf
Add type hints to E2E handler. (#9232) 
This finishes adding type hints to the `synapse.handlers` module.
 2021-01-28 08:34:19 -05:00
David Teller
93f84e0373 FIXUP: Making get_event_context a bit more paranoid 2021-01-28 12:31:07 +01:00
David Teller
b755f60ce2 FIXUP: Removing awaitable 2021-01-28 12:31:07 +01:00
David Teller
a764869623 FIXUP: Doc 2021-01-28 12:31:07 +01:00
David Teller
b859919acc FIXUP: Now testing that the user is admin! 2021-01-28 12:31:07 +01:00
David Teller
de7f049527 FIXUP: Don't filter events at all for admin/v1/rooms/.../context/... 2021-01-28 12:31:07 +01:00
David Teller
10332c175c New API /_synapse/admin/rooms/{roomId}/context/{eventId} 
Signed-off-by: David Teller <davidt@element.io>
 2021-01-28 12:29:49 +01:00
Richard van der Hoff
a083aea396
Add 'brand' field to MSC2858 response (#9242) 
We've decided to add a 'brand' field to help clients decide how to style the
buttons.

Also, fix up the allowed characters for idp_id, while I'm in the area.
 2021-01-27 21:31:45 +00:00
Richard van der Hoff
869667760f
Support for scraping email addresses from OIDC providers (#9245) 2021-01-27 21:28:59 +00:00
Pankaj Yadav
2e537a0280
Check if a user is in the room before sending a PowerLevel event on their behalf (#9235) 2021-01-27 17:38:08 +00:00
Richard van der Hoff
fbd9de6d1f Synapse 1.26.0 (2021-01-27) 
===========================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes and for general upgrade guidance.
 
 No significant changes since 1.26.0rc2.
 
 Synapse 1.26.0rc2 (2021-01-25)
 ==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 
 Synapse 1.26.0rc1 (2021-01-20)
 ==============================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes and for general upgrade guidance.
 
 Features
 --------
 
 - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
 - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
 - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
 - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
 - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
 - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
 - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
 - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
 - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
 - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
 - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
 - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))
 
 Bugfixes
 --------
 
 - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
 - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
 - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
 - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
 - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
 - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
 - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
 - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
 - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
 - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
 - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
 - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
 - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
 - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))
 
 Improved Documentation
 ----------------------
 
 - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
 - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
 - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
 - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
 - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))
 
 Deprecations and Removals
 -------------------------
 
 - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))
 
 Internal Changes
 ----------------
 
 - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
 - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
 - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
 - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
 - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
 - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
 - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
 - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
 - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
 - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
 - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
 - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
 - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
 - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
 - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
 - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
 - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
 - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
 - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
 - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
 - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
 - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
 - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmARjqYACgkQM/xY9qcR
 MEiknRAAtga+KXEG40xg7w78XXQk6nXIhhi30nYue9/SeHMTCmLmd1+f0yTySrRz
 JMc/xe9VP1th5DThHZeHJf0tLIVz/VFA5gr/fmk44FFEFvc77IhttVHSXbYZLuM8
 komOzLTJy7ueVSpyF9xVfEEz6UIRc098b2KHjuBh3SGlZAOmFuFHUMVBeEuYo80E
 ayVWXPNP+7U5y1qLA9uY/oxMsFjl6bPvjak3YDyGWCr2Y4nOhM24AwccErs0yAtu
 kCRm9JR0yD72uRLyaGyMZ9Atg4o01BD7hu0ilxhbRIdAvI5Cdk0E7rZshoLr33wk
 Ls4XlK9SxJSV4vEu44OJBCRnDjQS9HQnwYk13z+H5HrJcxXRyIeCwRD7aofzNY8O
 kNg/1oasBDiW2iVoLJ4Y0nWLHPcNlWJxUzkkjlF8BmEUQjURNVdh0n3y3/ST+T1S
 scex0J9Z95HNoLfRYID0adrYqWTOl5GLzy9C1AwqlLWY79y6YsB4LDmmfQgoTKCM
 VICSYtEKWph6WXdpd/kJsTgHyG3UFAMDraWVez7TxXpc7lpvOMkld+0oQnHWAXC3
 XRc+63jTs6pFJ74M+ziqOrC/1ORg1DK8OLIm/TskLoYnn6L9Dh8ssXeabLv63H8o
 cEqETUbZ77rZro0Nz2H1OJG09uTFUzFEy5Ecya7g4DqtRT/Fs9Q=
 =I9CI
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0' into social_login

Synapse 1.26.0 (2021-01-27)
===========================

This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.

No significant changes since 1.26.0rc2.

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))

Synapse 1.26.0rc1 (2021-01-20)
==============================

This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.

Features
--------

- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))

Bugfixes
--------

- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))

Improved Documentation
----------------------

- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))

Deprecations and Removals
-------------------------

- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))

Internal Changes
----------------

- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 2021-01-27 17:27:58 +00:00
Richard van der Hoff
7fa1346f93 Merge branch 'social_login' into develop 2021-01-27 17:27:24 +00:00
Patrick Cloke
17b713850f Merge branch 'master' into develop 2021-01-27 11:13:21 -05:00
Patrick Cloke
e54746bdf7
Clean-up the template loading code. (#9200) 
* Enables autoescape by default for HTML files.
* Adds a new read_template method for reading a single template.
* Some logic clean-up.
 2021-01-27 10:59:50 -05:00
Patrick Cloke
73ed289bd2 1.26.0 2021-01-27 10:50:37 -05:00
Richard van der Hoff
a737cc2713
Implement MSC2858 support (#9183) 
Fixes #8928.
 2021-01-27 12:41:24 +00:00
Andrew Morgan
a64c29926e
Pass a dict, instead of None, to modules if a None config is specified in the homeserver config (#9229) 
If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead.

This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](5310808d3b/docs/sample_config.yaml (L1816-L1845)) states) then `None` is not passed to the module. An empty dict is passed instead.

This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.
 2021-01-27 11:49:31 +00:00
Patrick Cloke
1baab20352
Add type hints to various handlers. (#9223) 
With this change all handlers except the e2e_* ones have
type hints enabled.
 2021-01-26 10:50:21 -05:00
Patrick Cloke
26837d5dbe
Do not require the CAS service URL setting (use public_baseurl instead). (#9199) 
The current configuration is handled for backwards compatibility,
but is considered deprecated.
 2021-01-26 10:49:25 -05:00
Erik Johnston
dd8da8c5f6
Precompute joined hosts and store in Redis (#9198) 2021-01-26 13:57:31 +00:00
Patrick Cloke
4937fe3d6b
Try to recover from unknown encodings when previewing media. (#9164) 
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
 2021-01-26 07:32:17 -05:00
Jason Robinson
e5b659e9e1
Merge pull request #9062 from matrix-org/jaywink/admin-forward-extremities 
Add forward extremities endpoint to rooms admin API
 2021-01-26 12:57:38 +02:00
Erik Johnston
a1ff1e967f
Periodically send pings to detect dead Redis connections (#9218) 
This is done by creating a custom `RedisFactory` subclass that
periodically pings all connections in its pool.

We also ensure that the `replyTimeout` param is non-null, so that we
timeout waiting for the reply to those pings (and thus triggering a
reconnect).
 2021-01-26 10:54:54 +00:00
Jason Robinson
4936fc59fc Fix get forward extremities query 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-26 10:21:02 +02:00
Jason Robinson
cee4010f94 Merge branch 'develop' into jaywink/admin-forward-extremities 
# Conflicts:
#	synapse/rest/admin/__init__.py
 2021-01-26 10:15:32 +02:00
Jason Robinson
e20f18a766
Make natural join inner join 
Co-authored-by: Erik Johnston <erik@matrix.org>
 2021-01-26 10:13:35 +02:00
Patrick Cloke
5b857b77f7
Don't error if deleting a non-existent pusher. (#9121) 2021-01-25 14:52:30 -05:00
Patrick Cloke
4a55d267ee
Add an admin API for shadow-banning users. (#9209) 
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.

A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.

Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
 2021-01-25 14:49:39 -05:00
Richard van der Hoff
65fb3b2e25 Synapse 1.26.0rc2 (2021-01-25) 
==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
 MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
 +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
 BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
 d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
 oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
 O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
 VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
 g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
 =Qc7+
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc2' into social_login

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 2021-01-25 19:37:58 +00:00
Jason Robinson
fe18882bb5 Merge remote-tracking branch 'origin/develop' into jaywink/admin-forward-extremities 2021-01-25 15:55:54 +02:00
Patrick Cloke
e448dbbf5b Synapse 1.26.0rc2 (2021-01-25) 
==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
 MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
 +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
 BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
 d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
 oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
 O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
 VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
 g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
 =Qc7+
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc2' into develop

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 2021-01-25 08:51:45 -05:00
Patrick Cloke
a01605c136 1.26.0rc2 2021-01-25 08:25:40 -05:00
Patrick Cloke
6f7417c3db
Handle missing content keys when calculating presentable names. (#9165) 
Treat the content as untrusted and do not assume it is of
the proper form.
 2021-01-25 07:27:16 -05:00
Jason Robinson
8965b6cfec Merge branch 'develop' into jaywink/admin-forward-extremities 2021-01-23 21:41:35 +02:00
Jason Robinson
930ba00971 Add depth and received_ts to forward_extremities admin API response 
Also add a warning on the admin API documentation.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-23 21:34:32 +02:00
Erik Johnston
056327457f
Fix chain cover update to handle events with duplicate auth events (#9210) 2021-01-22 19:44:08 +00:00
Erik Johnston
28f255d5f3
Bump psycopg2 version (#9204) 
As we use `execute_values` with the `fetch` parameter.
 2021-01-22 11:14:49 +00:00
Patrick Cloke
a7882f9887
Return a 404 if no valid thumbnail is found. (#9163) 
If no thumbnail of the requested type exists, return a 404 instead
of erroring. This doesn't quite match the spec (which does not define
what happens if no thumbnail can be found), but is consistent with
what Synapse already does.
 2021-01-21 14:53:58 -05:00
Erik Johnston
758ed5f1bc
Speed up chain cover calculation (#9176) 2021-01-21 17:00:12 +00:00
Erik Johnston
12ec55bfaa
Increase perf of handling concurrent use of StreamIDGenerators. (#9190) 
We have seen a failure mode here where if there are many in flight
unfinished IDs then marking an ID as finished takes a lot of CPU (as
calling deque.remove iterates over the list)
 2021-01-21 16:31:51 +00:00
Erik Johnston
939ef657ce Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 16:05:13 +00:00
Erik Johnston
ccfafac882
Add schema update to fix existing DBs affected by #9193 (#9195) 2021-01-21 16:03:25 +00:00
Erik Johnston
b249f002b8 Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 15:09:30 +00:00
Erik Johnston
2506074ef0
Fix receipts or account data not being sent down sync (#9193) 
Introduced in #9104 

This wasn't picked up by the tests as this is all fine the first time you run Synapse (after upgrading), but then when you restart the wrong value is pulled from `stream_positions`.
 2021-01-21 15:09:09 +00:00
Erik Johnston
7a43482f19
Use execute_batch in more places (#9188) 
* Use execute_batch in more places

* Newsfile
 2021-01-21 14:44:12 +00:00
Dirk Klimpel
c55e62548c
Add tests for List Users Admin API (#9045) 2021-01-21 09:18:46 -05:00
Richard van der Hoff
42a8e81370
Add a check for duplicate IdP ids (#9184) 2021-01-21 13:20:58 +00:00
Richard van der Hoff
b5120f09f1 Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 13:17:07 +00:00
Richard van der Hoff
7447f19702
Prefix idp_id with "oidc-" (#9189) 
... to avoid clashes with other SSO mechanisms
 2021-01-21 12:25:02 +00:00
Erik Johnston
eee6fcf5fa
Use execute_batch instead of executemany in places (#9181) 
`execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.
 2021-01-21 10:22:53 +00:00
Patrick Cloke
f81d02d75b Synapse 1.26.0rc1 (2021-01-20) 
==============================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 verious is not trivial.  Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes  and for general upgrade guidance.
 
 Features
 --------
 
 - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
 - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
 - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
 - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
 - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
 - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
 - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
 - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
 - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
 - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
 - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
 - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))
 
 Bugfixes
 --------
 
 - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
 - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
 - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
 - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
 - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
 - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
 - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
 - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
 - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
 - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
 - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
 - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
 - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
 - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))
 
 Improved Documentation
 ----------------------
 
 - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
 - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
 - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
 - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
 - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))
 
 Deprecations and Removals
 -------------------------
 
 - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))
 
 Internal Changes
 ----------------
 
 - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
 - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
 - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
 - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
 - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
 - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
 - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
 - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
 - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
 - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
 - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
 - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
 - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
 - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
 - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
 - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
 - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
 - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
 - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
 - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
 - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
 - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
 - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAIVFkACgkQM/xY9qcR
 MEhLfQ//Z4xnP5Icx732k2xpdR+JaXX6heNVFm/UuLNd6wq+uXjO/Dgc9IKQ5LXD
 kfG+/OnNGDr/eUk4mQ5s/ccBvaTrOid2IIemJd4lUBy/Q5LvPvfKIp08QFr72WJT
 7Y4Ma54nNSSMUAdESgj/aFAHjATbfHhxPOZ5OdQHGLAuJ/OUwR2ksasVNCuINbgh
 8vrxrzjeYn1Zl3UTBrsdCcat7AFqYjxK/Y0i6JxfSgRwDZjUbW4M2C4OGWZJPUrU
 2YTPHangxd+22+HObE2KDEzaHDdCR/Kj4dhCiJBynZ5iucuHSS+mgf39RtbjqWML
 01cmXXUV+FH7rnCilmF9rYdmHZ6L9BJxtrQDQU4dB106BX9G7hKR6rYHc9Z6tjOh
 3tg0cKWYMXOsTNnvRqiFNh2q0yBKS228HRAwyGosW/6NiYj43ArtYP6knX7dQyBX
 0mYNkfry2dyX6Kj5el0i9MOTHYQxfc8apsBm2M3OTNYukgQKA0NhbAquibaAxEEW
 2qIqbSp4CUlGPvM7+u+ZGTu0hEbVKKNjqMfXhuY8A4/BWiV2mSqKcEirR5cE/LqY
 mq5Ac7vbO4Uh+1xiRw/G9ITi1dqAjYIzhBawlhdUPOh+aINWTUikYYzjUSmm4PbN
 H9BTPLA9iAdVRfuWJ9um2G0DdS8Qpx/aIRh3MScXVly6cGCC6Do=
 =kgqn
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc1' into develop

Synapse 1.26.0rc1 (2021-01-20)
==============================

This release brings a new schema version for Synapse and rolling back to a previous
verious is not trivial.  Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes  and for general upgrade guidance.

Features
--------

- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))

Bugfixes
--------

- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))

Improved Documentation
----------------------

- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))

Deprecations and Removals
-------------------------

- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))

Internal Changes
----------------

- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 2021-01-20 11:27:39 -05:00
Richard van der Hoff
e51b2f3f91
Tighten the restrictions on idp_id (#9177) 2021-01-20 08:55:14 -05:00
Richard van der Hoff
0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Patrick Cloke
620ecf13b0
Various improvements to the federation client. (#9129) 
* Type hints for `FederationClient`.
* Using `async` functions instead of returning `Awaitable` instances.
 2021-01-20 07:59:18 -05:00
Richard van der Hoff
6c0dfd2e8e Merge remote-tracking branch 'origin/develop' into release-v1.26.0 2021-01-20 12:33:05 +00:00
Richard van der Hoff
fa50e4bf4d
Give public_baseurl a default value (#9159) 2021-01-20 12:30:41 +00:00
Patrick Cloke
72822e60be 1.26.0rc1 2021-01-20 07:14:25 -05:00
Patrick Cloke
fa842a9866
Use the account handler in additional places. (#9166) 2021-01-20 10:44:52 +00:00
Patrick Cloke
47d48a5853
Validate the server name for the /publicRooms endpoint. (#9161) 
If a remote server name is provided, ensure it is something reasonable
before making remote connections to it.
 2021-01-19 14:21:59 -05:00
Richard van der Hoff
73b03722f4
Fix error messages from OIDC config parsing (#9153) 
Make sure we report the correct config path for errors in the OIDC configs.
 2021-01-19 14:56:54 +00:00
Erik Johnston
6633a4015a
Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
Erik Johnston
f08ef64926
Enforce all replication HTTP clients calls use kwargs (#9144) 2021-01-18 15:24:04 +00:00
Patrick Cloke
2b467d0b61
Properly raise an exception when the body exceeds the max size. (#9145) 
...instead of just creating the exception object and doing nothing with it.
 2021-01-18 10:21:42 -05:00
Richard van der Hoff
02070c69fa
Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128) 
* Factor out a common TestHtmlParser

Looks like I'm doing this in a few different places.

* Improve OIDC login test

Complete the OIDC login flow, rather than giving up halfway through.

* Ensure that OIDC login works with multiple OIDC providers

* Fix bugs in handling clientRedirectUrl

 - don't drop duplicate query-params, or params with no value
 - allow utf-8 in query-params
 2021-01-18 14:52:49 +00:00
Patrick Cloke
de1f8de319
Ensure the user ID is serialized in the payload instead of used as an instance name. (#9130) 2021-01-18 11:08:26 +00:00
Erik Johnston
350d9923cd
Make chain cover index bg update go faster (#9124) 
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
 2021-01-15 17:18:37 +00:00
Richard van der Hoff
2de7e263ed
Ensure we store pusher data as text (#9117) 
I don't think there's any need to use canonicaljson here.

Fixes: #4475.
 2021-01-15 16:57:23 +00:00
Richard van der Hoff
9de6b94117
Land support for multiple OIDC providers (#9110) 
This is the final step for supporting multiple OIDC providers concurrently.

First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before:

    oidc_config:
       enabled: true
       issuer: "https://oidc_provider"
       # etc

After:

    oidc_providers:
     - idp_id: prov1
       issuer: "https://oidc_provider"

     - idp_id: prov2
       issuer: "https://another_oidc_provider"

The old format is still grandfathered in.

With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
 2021-01-15 16:55:29 +00:00
Patrick Cloke
3e4cdfe5d9
Add an admin API endpoint to protect media. (#9086) 
Protecting media stops it from being quarantined when
e.g. all media in a room is quarantined. This is useful
for sticker packs and other media that is uploaded by
server administrators, but used by many people.
 2021-01-15 11:18:09 -05:00
Patrick Cloke
74dd906041
Avoid raising the body exceeded error multiple times. (#9108) 
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
 2021-01-15 11:00:13 -05:00
Richard van der Hoff
9ffac2bef1
Remote dependency on distutils (#9125) 
`distutils` is pretty much deprecated these days, and replaced with
`setuptools`. It's also annoying because it's you can't `pip install` it, and
it's hard to figure out which debian package we should depend on to make sure
it's there.

Since we only use it for a tiny function anyway, let's just vendor said
function into our codebase.
 2021-01-15 15:59:20 +00:00
Patrick Cloke
d34c6e1279
Add type hints to media rest resources. (#9093) 2021-01-15 10:57:37 -05:00
Richard van der Hoff
4575ad0b1e
Store an IdP ID in the OIDC session (#9109) 
Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP.

As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
 2021-01-15 13:22:12 +00:00
Richard van der Hoff
14950a45d6
Merge pull request #9091 from matrix-org/rav/error_on_bad_sso 
Give the user a better error when they present bad SSO creds
 2021-01-15 00:27:13 +00:00
Erik Johnston
1a08e0cdab
Fix event chain bg update. (#9118) 
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
 2021-01-14 18:57:32 +00:00
Erik Johnston
d2479c6870
Fix perf of get_cross_signing_keys (#9116) 2021-01-14 17:57:09 +00:00
Erik Johnston
659c415ed4
Fix chain cover background update to work with split out event persisters (#9115) 2021-01-14 17:19:35 +00:00
Erik Johnston
631dd06f2c
Fix get destinations to catch up query. (#9114) 
t was doing a sequential scan on `destination_rooms`, which took
minutes.
 2021-01-14 16:47:21 +00:00
Erik Johnston
7036e24e98
Add background update for add chain cover index (#9029) 2021-01-14 15:18:27 +00:00
Richard van der Hoff
21a296cd5a
Split OidcProvider out of OidcHandler (#9107) 
The idea here is that we will have an instance of OidcProvider for each
configured IdP, with OidcHandler just doing the marshalling of them.

For now it's still hardcoded with a single provider.
 2021-01-14 13:29:17 +00:00
Tim Leung
12702be951
Fix wrong arguments being passed to BlacklistingAgentWrapper (#9098) 
A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper
used by the WellyKnownResolver. This coulld cause exceptions when attempting to
connect to IP addresses that are blacklisted, but in reality this did not have any
observable affect since this code is not used for IP literals.
 2021-01-14 06:59:26 -05:00
Richard van der Hoff
420031906a Move complete_sso_ui_auth into SSOHandler 
since we're hacking on this code anyway, may as well move it out of the
cluttered AuthHandler.
 2021-01-13 20:22:41 +00:00
Richard van der Hoff
5310808d3b Give the user a better error when they present bad SSO creds 
If a user tries to do UI Auth via SSO, but uses the wrong account on the SSO
IdP, try to give them a better error.

Previously, the UIA would claim to be successful, but then the operation in
question would simply fail with "auth fail". Instead, serve up an error page
which explains the failure.
 2021-01-13 20:22:41 +00:00
Richard van der Hoff
d02e4b2825
Merge pull request #9105 from matrix-org/rav/multi_idp/oidc_provider_config 
Enhancements to OIDC configuration handling
 2021-01-13 19:51:46 +00:00
Patrick Cloke
aee8e6a95d
Reduce scope of exception handler. (#9106) 
Removes a bare `except Exception` clause and replaces it with
catching a specific exception around the portion that might throw.
 2021-01-13 13:27:49 -05:00
Richard van der Hoff
dc3c83a933 Add jsonschema verification for the oidc provider config 2021-01-13 17:47:27 +00:00
Patrick Cloke
d1eb1b96e8
Register the /devices endpoint on workers. (#9092) 2021-01-13 12:35:40 -05:00
Richard van der Hoff
7cc9509eca Extract OIDCProviderConfig object 
Collect all the config options which related to an OIDC provider into a single
object.
 2021-01-13 16:40:02 +00:00
Patrick Cloke
98a64b7f7f
Add basic domain validation for DomainSpecificString.is_valid. (#9071) 
This checks that the domain given to `DomainSpecificString.is_valid` (e.g.
`UserID`, `RoomAlias`, etc.) is of a valid form. Previously some validation
was done on the localpart (e.g. the sigil), but not the domain portion.
 2021-01-13 07:05:16 -05:00
Erik Johnston
aa4d8c1f9a Merge branch 'master' into develop 2021-01-13 10:36:55 +00:00
Richard van der Hoff
bc4bf7b384
Preparatory refactors of OidcHandler (#9067) 
Some light refactoring of OidcHandler, in preparation for bigger things:

  * remove inheritance from deprecated BaseHandler
  * add an object to hold the things that go into a session cookie
  * factor out a separate class for manipulating said cookies
 2021-01-13 10:26:12 +00:00
Erik Johnston
3dd6ba135e 1.25.0 2021-01-13 10:19:12 +00:00
Dirk Klimpel
7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932) 
This only applies if the user's data is to be erased.
 2021-01-12 16:30:15 -05:00
Richard van der Hoff
789d9ebad3
UI Auth via SSO: redirect the user to an appropriate SSO. (#9081) 
If we have integrations with multiple identity providers, when the user does a UI Auth, we need to redirect them to the right one.

There are a few steps to this. First of all we actually need to store the userid of the user we are trying to validate in the UIA session, since the /auth/sso/fallback/web request is unauthenticated.

Then, once we get the /auth/sso/fallback/web request, we can fish the user id out of the session, and use it to look up the external id mappings, and hence pick an SSO provider for them.
 2021-01-12 17:38:03 +00:00
Marcus
e385c8b473
Don't apply the IP range blacklist to proxy connections (#9084) 
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
 2021-01-12 12:20:30 -05:00
Patrick Cloke
723b19748a
Handle bad JSON data being returned from the federation API. (#9070) 2021-01-12 11:07:01 -05:00
Richard van der Hoff
0f8945e166
Kill off HomeServer.get_ip_from_request() (#9080) 
Homeserver.get_ip_from_request() used to be a bit more complicated, but now it is totally redundant. Let's get rid of it.
 2021-01-12 12:48:12 +00:00
Richard van der Hoff
2ec8ca5e60
Remove SynapseRequest.get_user_agent (#9069) 
SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests",
which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the 
LoggingContext and write the right entries to the request log).

Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a
SynapseRequest when there is nothing synapse-specific about the Request at all, and any old
twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult.

In short: move get_user_agent out to a utility function.
 2021-01-12 12:34:16 +00:00
Jason Robinson
c177faf5a9 Remove trailing whitespace to appease the linter 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-11 23:55:44 +02:00
Jason Robinson
49c619a9a2 Simplify delete_forward_extremities_for_room_txn SQL 
As per feedback.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-11 23:49:58 +02:00
Jason Robinson
da16d06301 Address pr feedback 
* docs updates
* prettify SQL
* add missing copyright
* cursor_to_dict
* update touched files copyright years

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-11 23:43:58 +02:00
David Teller
b161528fcc
Also support remote users on the joined_rooms admin API. (#8948) 
For remote users, only the rooms which the server knows about are returned.
Local users have all of their joined rooms returned.
 2021-01-11 14:32:17 -05:00
Erik Johnston
c9195744a4
Move more encryption endpoints off master (#9068) 2021-01-11 18:01:27 +00:00
Erik Johnston
1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00
Richard van der Hoff
671138f658
Clean up exception handling in the startup code (#9059) 
Factor out the exception handling in the startup code to a utility function,
and fix the some logging and exit code stuff.
 2021-01-11 15:55:05 +00:00
Erik Johnston
4e04435bda
Remove old tables after schema version bump (#9055) 
These tables are unused, and can be dropped now the schema version has been bumped.
 2021-01-11 13:58:19 +00:00
Erik Johnston
63f4990298
Ensure rejected events get added to some metadata tables (#9016) 
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-01-11 13:57:33 +00:00
Richard van der Hoff
7db2622d30
Remove unused SynapseService (#9058) 2021-01-11 10:24:22 +00:00
Jerin J Titus
c21d8f1c1d
Drop last_used column from access_tokens (#9025) 
* Dropped last_used column from access_tokens

Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
 2021-01-11 10:23:49 +00:00
Jason Robinson
b52fb703f7 Don't try to use f-strings 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-11 09:47:03 +02:00
Jason Robinson
2eb421b606 Merge branch 'develop' into jaywink/admin-forward-extremities 2021-01-09 22:00:04 +02:00
Jason Robinson
90ad4d443a Implement clearing cache after deleting forward extremities 
Also run linter.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-09 21:57:41 +02:00
Erik Johnston
a03d71dc9d
Fix "Starting metrics collection from sentinel context" errors (#9053) 2021-01-08 14:33:53 +00:00
Richard van der Hoff
12f79da587
Merge pull request #9036 from matrix-org/rav/multi_idp/tests 
Add tests for the IdP picker
 2021-01-08 14:24:41 +00:00
Richard van der Hoff
d32870ffa5
Fix validate_config on nested objects (#9054) 2021-01-08 14:23:04 +00:00
Erik Johnston
fa5f5cbc74
Fix error handling during insertion of client IPs (#9051) 
You can't continue using a transaction once an exception has been
raised, so catching and dropping the error here is pointless and just
causes more errors.
 2021-01-08 14:15:20 +00:00
Jason Robinson
85c0999bfb Add Rooms admin forward extremities DELETE endpoint 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-08 00:12:23 +02:00
Jason Robinson
c91045f56c Move unknown room ID error into resolve_room_id 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-07 23:03:54 +02:00
Jason Robinson
b849e46139 Add forward extremities endpoint to rooms admin API 
GET /_synapse/admin/v1/rooms/<identifier>/forward_extremities now gets forward extremities for a room, returning count and the list of extremities.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-07 23:01:59 +02:00
Erik Johnston
b530eaa262
Allow running sendToDevice on workers (#9044) 2021-01-07 20:19:26 +00:00
Erik Johnston
5e99a94502
Support routing edu's to multiple instances (#9042) 
This is in preparation for moving `SendToDeviceServlet` off master
 2021-01-07 18:07:28 +00:00
Erik Johnston
e34df813ce
Ensure that remote users' device list resyncing always happens on master (#9043) 
Currently `DeviceMessageHandler` only ever exists on master, but that is about to change.
 2021-01-07 18:06:52 +00:00
Erik Johnston
63593134a1
Some cleanups to device inbox store. (#9041) 2021-01-07 17:20:44 +00:00
Richard van der Hoff
8a910f97a4 Add some tests for the IDP picker flow 2021-01-07 14:56:42 +00:00
Patrick Cloke
23d701864f
Improve the performance of calculating ignored users in large rooms (#9024) 
This allows for efficiently finding which users ignore a particular
user.

Co-authored-by: Erik Johnston <erik@matrix.org>
 2021-01-07 13:03:38 +00:00
Erik Johnston
eee3c3c52f
Handle updating schema version without any deltas. (#9033) 
This can happen when using a split out state database and we've upgraded
the schema version without there being any changes in the state schema.
 2021-01-07 11:33:36 +00:00
Patrick Cloke
1b4d5d6acf
Empty iterables should count towards cache usage. (#9028) 2021-01-06 12:33:20 -05:00
Patrick Cloke
0312266ee3 Synapse 1.25.0rc1 (2021-01-06) 
==============================
 
 Removal warning
 ---------------
 
 The old [Purge Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/purge_room.md)
 and [Shutdown Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/shutdown_room.md)
 are deprecated and will be removed in a future release. They will be replaced by the
 [Delete Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/rooms.md#delete-room-api).
 
 `POST /_synapse/admin/v1/rooms/<room_id>/delete` replaces `POST /_synapse/admin/v1/purge_room` and
 `POST /_synapse/admin/v1/shutdown_room/<room_id>`.
 
 Features
 --------
 
 - Add an admin API that lets server admins get power in rooms in which local users have power. ([\#8756](https://github.com/matrix-org/synapse/issues/8756))
 - Add optional HTTP authentication to replication endpoints. ([\#8853](https://github.com/matrix-org/synapse/issues/8853))
 - Improve the error messages printed as a result of configuration problems for extension modules. ([\#8874](https://github.com/matrix-org/synapse/issues/8874))
 - Add the number of local devices to Room Details Admin API. Contributed by @dklimpel. ([\#8886](https://github.com/matrix-org/synapse/issues/8886))
 - Add `X-Robots-Tag` header to stop web crawlers from indexing media. Contributed by Aaron Raimist. ([\#8887](https://github.com/matrix-org/synapse/issues/8887))
 - Spam-checkers may now define their methods as `async`. ([\#8890](https://github.com/matrix-org/synapse/issues/8890))
 - Add support for allowing users to pick their own user ID during a single-sign-on login. ([\#8897](https://github.com/matrix-org/synapse/issues/8897), [\#8900](https://github.com/matrix-org/synapse/issues/8900), [\#8911](https://github.com/matrix-org/synapse/issues/8911), [\#8938](https://github.com/matrix-org/synapse/issues/8938), [\#8941](https://github.com/matrix-org/synapse/issues/8941), [\#8942](https://github.com/matrix-org/synapse/issues/8942), [\#8951](https://github.com/matrix-org/synapse/issues/8951))
 - Add an `email.invite_client_location` configuration option to send a web client location to the invite endpoint on the identity server which allows customisation of the email template. ([\#8930](https://github.com/matrix-org/synapse/issues/8930))
 - The search term in the list room and list user Admin APIs is now treated as case-insensitive. ([\#8931](https://github.com/matrix-org/synapse/issues/8931))
 - Apply an IP range blacklist to push and key revocation requests. ([\#8821](https://github.com/matrix-org/synapse/issues/8821), [\#8870](https://github.com/matrix-org/synapse/issues/8870), [\#8954](https://github.com/matrix-org/synapse/issues/8954))
 - Add an option to allow re-use of user-interactive authentication sessions for a period of time. ([\#8970](https://github.com/matrix-org/synapse/issues/8970))
 - Allow running the redact endpoint on workers. ([\#8994](https://github.com/matrix-org/synapse/issues/8994))
 
 Bugfixes
 --------
 
 - Fix bug where we might not correctly calculate the current state for rooms with multiple extremities. ([\#8827](https://github.com/matrix-org/synapse/issues/8827))
 - Fix a long-standing bug in the register admin endpoint (`/_synapse/admin/v1/register`) when the `mac` field was not provided. The endpoint now properly returns a 400 error. Contributed by @edwargix. ([\#8837](https://github.com/matrix-org/synapse/issues/8837))
 - Fix a long-standing bug on Synapse instances supporting Single-Sign-On, where users would be prompted to enter their password to confirm certain actions, even though they have not set a password. ([\#8858](https://github.com/matrix-org/synapse/issues/8858))
 - Fix a longstanding bug where a 500 error would be returned if the `Content-Length` header was not provided to the upload media resource. ([\#8862](https://github.com/matrix-org/synapse/issues/8862))
 - Add additional validation to pusher URLs to be compliant with the specification. ([\#8865](https://github.com/matrix-org/synapse/issues/8865))
 - Fix the error code that is returned when a user tries to register on a homeserver on which new-user registration has been disabled. ([\#8867](https://github.com/matrix-org/synapse/issues/8867))
 - Fix a bug where `PUT /_synapse/admin/v2/users/<user_id>` failed to create a new user when `avatar_url` is specified. Bug introduced in Synapse v1.9.0. ([\#8872](https://github.com/matrix-org/synapse/issues/8872))
 - Fix a 500 error when attempting to preview an empty HTML file. ([\#8883](https://github.com/matrix-org/synapse/issues/8883))
 - Fix occasional deadlock when handling SIGHUP. ([\#8918](https://github.com/matrix-org/synapse/issues/8918))
 - Fix login API to not ratelimit application services that have ratelimiting disabled. ([\#8920](https://github.com/matrix-org/synapse/issues/8920))
 - Fix bug where we ratelimited auto joining of rooms on registration (using `auto_join_rooms` config). ([\#8921](https://github.com/matrix-org/synapse/issues/8921))
 - Fix a bug where deactivated users appeared in the user directory when their profile information was updated. ([\#8933](https://github.com/matrix-org/synapse/issues/8933), [\#8964](https://github.com/matrix-org/synapse/issues/8964))
 - Fix bug introduced in Synapse v1.24.0 which would cause an exception on startup if both `enabled` and `localdb_enabled` were set to `False` in the `password_config` setting of the configuration file. ([\#8937](https://github.com/matrix-org/synapse/issues/8937))
 - Fix a bug where 500 errors would be returned if the `m.room_history_visibility` event had invalid content. ([\#8945](https://github.com/matrix-org/synapse/issues/8945))
 - Fix a bug causing common English words to not be considered for a user directory search. ([\#8959](https://github.com/matrix-org/synapse/issues/8959))
 - Fix bug where application services couldn't register new ghost users if the server had reached its MAU limit. ([\#8962](https://github.com/matrix-org/synapse/issues/8962))
 - Fix a long-standing bug where a `m.image` event without a `url` would cause errors on push. ([\#8965](https://github.com/matrix-org/synapse/issues/8965))
 - Fix a small bug in v2 state resolution algorithm, which could also cause performance issues for rooms with large numbers of power levels. ([\#8971](https://github.com/matrix-org/synapse/issues/8971))
 - Add validation to the `sendToDevice` API to raise a missing parameters error instead of a 500 error. ([\#8975](https://github.com/matrix-org/synapse/issues/8975))
 - Add validation of group IDs to raise a 400 error instead of a 500 eror. ([\#8977](https://github.com/matrix-org/synapse/issues/8977))
 
 Improved Documentation
 ----------------------
 
 - Fix the "Event persist rate" section of the included grafana dashboard by adding missing prometheus rules. ([\#8802](https://github.com/matrix-org/synapse/issues/8802))
 - Combine related media admin API docs. ([\#8839](https://github.com/matrix-org/synapse/issues/8839))
 - Fix an error in the documentation for the SAML username mapping provider. ([\#8873](https://github.com/matrix-org/synapse/issues/8873))
 - Clarify comments around template directories in `sample_config.yaml`. ([\#8891](https://github.com/matrix-org/synapse/issues/8891))
 - Moved instructions for database setup, adjusted heading levels and improved syntax highlighting in [INSTALL.md](../INSTALL.md). Contributed by fossterer. ([\#8987](https://github.com/matrix-org/synapse/issues/8987))
 - Update the example value of `group_creation_prefix` in the sample configuration. ([\#8992](https://github.com/matrix-org/synapse/issues/8992))
 - Link the Synapse developer room to the development section in the docs. ([\#9002](https://github.com/matrix-org/synapse/issues/9002))
 
 Deprecations and Removals
 -------------------------
 
 - Deprecate Shutdown Room and Purge Room Admin APIs. ([\#8829](https://github.com/matrix-org/synapse/issues/8829))
 
 Internal Changes
 ----------------
 
 - Properly store the mapping of external ID to Matrix ID for CAS users. ([\#8856](https://github.com/matrix-org/synapse/issues/8856), [\#8958](https://github.com/matrix-org/synapse/issues/8958))
 - Remove some unnecessary stubbing from unit tests. ([\#8861](https://github.com/matrix-org/synapse/issues/8861))
 - Remove unused `FakeResponse` class from unit tests. ([\#8864](https://github.com/matrix-org/synapse/issues/8864))
 - Pass `room_id` to `get_auth_chain_difference`. ([\#8879](https://github.com/matrix-org/synapse/issues/8879))
 - Add type hints to push module. ([\#8880](https://github.com/matrix-org/synapse/issues/8880), [\#8882](https://github.com/matrix-org/synapse/issues/8882), [\#8901](https://github.com/matrix-org/synapse/issues/8901), [\#8940](https://github.com/matrix-org/synapse/issues/8940), [\#8943](https://github.com/matrix-org/synapse/issues/8943), [\#9020](https://github.com/matrix-org/synapse/issues/9020))
 - Simplify logic for handling user-interactive-auth via single-sign-on servers. ([\#8881](https://github.com/matrix-org/synapse/issues/8881))
 - Skip the SAML tests if the requirements (`pysaml2` and `xmlsec1`) aren't available. ([\#8905](https://github.com/matrix-org/synapse/issues/8905))
 - Fix multiarch docker image builds. ([\#8906](https://github.com/matrix-org/synapse/issues/8906))
 - Don't publish `latest` docker image until all archs are built. ([\#8909](https://github.com/matrix-org/synapse/issues/8909))
 - Various clean-ups to the structured logging and logging context code. ([\#8916](https://github.com/matrix-org/synapse/issues/8916), [\#8935](https://github.com/matrix-org/synapse/issues/8935))
 - Automatically drop stale forward-extremities under some specific conditions. ([\#8929](https://github.com/matrix-org/synapse/issues/8929))
 - Refactor test utilities for injecting HTTP requests. ([\#8946](https://github.com/matrix-org/synapse/issues/8946))
 - Add a maximum size of 50 kilobytes to .well-known lookups. ([\#8950](https://github.com/matrix-org/synapse/issues/8950))
 - Fix bug in `generate_log_config` script which made it write empty files. ([\#8952](https://github.com/matrix-org/synapse/issues/8952))
 - Clean up tox.ini file; disable coverage checking for non-test runs. ([\#8963](https://github.com/matrix-org/synapse/issues/8963))
 - Add type hints to the admin and room list handlers. ([\#8973](https://github.com/matrix-org/synapse/issues/8973))
 - Add type hints to the receipts and user directory handlers. ([\#8976](https://github.com/matrix-org/synapse/issues/8976))
 - Drop the unused `local_invites` table. ([\#8979](https://github.com/matrix-org/synapse/issues/8979))
 - Add type hints to the base storage code. ([\#8980](https://github.com/matrix-org/synapse/issues/8980))
 - Support using PyJWT v2.0.0 in the test suite. ([\#8986](https://github.com/matrix-org/synapse/issues/8986))
 - Fix `tests.federation.transport.RoomDirectoryFederationTests` and ensure it runs in CI. ([\#8998](https://github.com/matrix-org/synapse/issues/8998))
 - Add type hints to the crypto module. ([\#8999](https://github.com/matrix-org/synapse/issues/8999))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl/1rv4ACgkQM/xY9qcR
 MEiIGw/+PytgC2yOpaR7cIQl0K1v+y4X/AmNbshdQch7GtC3vtKxdmo4PJp4KwY4
 Sxiiykq46Xx48w/Hqd3hoC7OqsTyfmNjRKW0InsTarbaXMrSv9umOtScItzMMWq8
 SCmDPQ+cF0SP2GhOMwB/N5JpRS12OpdBzHXtcbwxULilTuO/EKD2/W1zq3wiSP7r
 BxhHHYsx69hufUGC3OGD0I8e26TqjgkLiVm0FeM6cC5TubozO3qrJDlmwTPSpFrL
 WZEBEbnYUVz4ydMKRq4ALiZzPzxxBfgQmITb79CNqim4+fc+M+AqV6mRez2ULrqh
 d4/UEMTaA0yh54i0oqei3gwf2ucns/NDftQ9cYwF1+a9LCcWAyJZY7+/9Hk4VKSU
 KqQUzGdBfxyefsLc2hNGANzS/VbThdejC09IjaxKtHSgaT+bb5LxS8GcbvcG5FxY
 LyD8yeazgXPub+Ixr6dvD51roCuBtpCYvGZtHHYOiX/S1MYbq22Duet2Z1zhy7qB
 Jl8r3L32x5wAG3lR8aLqDVNTrC1ur0TW9wUJ88bDjcwn0PsseQG41DHbQSoXQIlk
 p6/IKjJjfnbKJFTwzfC8RpL29NtyTPqA/d2JDJ7yqUbpEkAnc5GN+pO0B/736aOH
 aTRu8nVdP7GOoP4whZ9C7r5kYjiCR3jPtpZHRYXaPJtkw57dWS8=
 =/g7u
 -----END PGP SIGNATURE-----

Merge tag 'v1.25.0rc1' into develop

Synapse 1.25.0rc1 (2021-01-06)
==============================

Removal warning
---------------

The old [Purge Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/purge_room.md)
and [Shutdown Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/shutdown_room.md)
are deprecated and will be removed in a future release. They will be replaced by the
[Delete Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/rooms.md#delete-room-api).

`POST /_synapse/admin/v1/rooms/<room_id>/delete` replaces `POST /_synapse/admin/v1/purge_room` and
`POST /_synapse/admin/v1/shutdown_room/<room_id>`.

Features
--------

- Add an admin API that lets server admins get power in rooms in which local users have power. ([\#8756](https://github.com/matrix-org/synapse/issues/8756))
- Add optional HTTP authentication to replication endpoints. ([\#8853](https://github.com/matrix-org/synapse/issues/8853))
- Improve the error messages printed as a result of configuration problems for extension modules. ([\#8874](https://github.com/matrix-org/synapse/issues/8874))
- Add the number of local devices to Room Details Admin API. Contributed by @dklimpel. ([\#8886](https://github.com/matrix-org/synapse/issues/8886))
- Add `X-Robots-Tag` header to stop web crawlers from indexing media. Contributed by Aaron Raimist. ([\#8887](https://github.com/matrix-org/synapse/issues/8887))
- Spam-checkers may now define their methods as `async`. ([\#8890](https://github.com/matrix-org/synapse/issues/8890))
- Add support for allowing users to pick their own user ID during a single-sign-on login. ([\#8897](https://github.com/matrix-org/synapse/issues/8897), [\#8900](https://github.com/matrix-org/synapse/issues/8900), [\#8911](https://github.com/matrix-org/synapse/issues/8911), [\#8938](https://github.com/matrix-org/synapse/issues/8938), [\#8941](https://github.com/matrix-org/synapse/issues/8941), [\#8942](https://github.com/matrix-org/synapse/issues/8942), [\#8951](https://github.com/matrix-org/synapse/issues/8951))
- Add an `email.invite_client_location` configuration option to send a web client location to the invite endpoint on the identity server which allows customisation of the email template. ([\#8930](https://github.com/matrix-org/synapse/issues/8930))
- The search term in the list room and list user Admin APIs is now treated as case-insensitive. ([\#8931](https://github.com/matrix-org/synapse/issues/8931))
- Apply an IP range blacklist to push and key revocation requests. ([\#8821](https://github.com/matrix-org/synapse/issues/8821), [\#8870](https://github.com/matrix-org/synapse/issues/8870), [\#8954](https://github.com/matrix-org/synapse/issues/8954))
- Add an option to allow re-use of user-interactive authentication sessions for a period of time. ([\#8970](https://github.com/matrix-org/synapse/issues/8970))
- Allow running the redact endpoint on workers. ([\#8994](https://github.com/matrix-org/synapse/issues/8994))

Bugfixes
--------

- Fix bug where we might not correctly calculate the current state for rooms with multiple extremities. ([\#8827](https://github.com/matrix-org/synapse/issues/8827))
- Fix a long-standing bug in the register admin endpoint (`/_synapse/admin/v1/register`) when the `mac` field was not provided. The endpoint now properly returns a 400 error. Contributed by @edwargix. ([\#8837](https://github.com/matrix-org/synapse/issues/8837))
- Fix a long-standing bug on Synapse instances supporting Single-Sign-On, where users would be prompted to enter their password to confirm certain actions, even though they have not set a password. ([\#8858](https://github.com/matrix-org/synapse/issues/8858))
- Fix a longstanding bug where a 500 error would be returned if the `Content-Length` header was not provided to the upload media resource. ([\#8862](https://github.com/matrix-org/synapse/issues/8862))
- Add additional validation to pusher URLs to be compliant with the specification. ([\#8865](https://github.com/matrix-org/synapse/issues/8865))
- Fix the error code that is returned when a user tries to register on a homeserver on which new-user registration has been disabled. ([\#8867](https://github.com/matrix-org/synapse/issues/8867))
- Fix a bug where `PUT /_synapse/admin/v2/users/<user_id>` failed to create a new user when `avatar_url` is specified. Bug introduced in Synapse v1.9.0. ([\#8872](https://github.com/matrix-org/synapse/issues/8872))
- Fix a 500 error when attempting to preview an empty HTML file. ([\#8883](https://github.com/matrix-org/synapse/issues/8883))
- Fix occasional deadlock when handling SIGHUP. ([\#8918](https://github.com/matrix-org/synapse/issues/8918))
- Fix login API to not ratelimit application services that have ratelimiting disabled. ([\#8920](https://github.com/matrix-org/synapse/issues/8920))
- Fix bug where we ratelimited auto joining of rooms on registration (using `auto_join_rooms` config). ([\#8921](https://github.com/matrix-org/synapse/issues/8921))
- Fix a bug where deactivated users appeared in the user directory when their profile information was updated. ([\#8933](https://github.com/matrix-org/synapse/issues/8933), [\#8964](https://github.com/matrix-org/synapse/issues/8964))
- Fix bug introduced in Synapse v1.24.0 which would cause an exception on startup if both `enabled` and `localdb_enabled` were set to `False` in the `password_config` setting of the configuration file. ([\#8937](https://github.com/matrix-org/synapse/issues/8937))
- Fix a bug where 500 errors would be returned if the `m.room_history_visibility` event had invalid content. ([\#8945](https://github.com/matrix-org/synapse/issues/8945))
- Fix a bug causing common English words to not be considered for a user directory search. ([\#8959](https://github.com/matrix-org/synapse/issues/8959))
- Fix bug where application services couldn't register new ghost users if the server had reached its MAU limit. ([\#8962](https://github.com/matrix-org/synapse/issues/8962))
- Fix a long-standing bug where a `m.image` event without a `url` would cause errors on push. ([\#8965](https://github.com/matrix-org/synapse/issues/8965))
- Fix a small bug in v2 state resolution algorithm, which could also cause performance issues for rooms with large numbers of power levels. ([\#8971](https://github.com/matrix-org/synapse/issues/8971))
- Add validation to the `sendToDevice` API to raise a missing parameters error instead of a 500 error. ([\#8975](https://github.com/matrix-org/synapse/issues/8975))
- Add validation of group IDs to raise a 400 error instead of a 500 eror. ([\#8977](https://github.com/matrix-org/synapse/issues/8977))

Improved Documentation
----------------------

- Fix the "Event persist rate" section of the included grafana dashboard by adding missing prometheus rules. ([\#8802](https://github.com/matrix-org/synapse/issues/8802))
- Combine related media admin API docs. ([\#8839](https://github.com/matrix-org/synapse/issues/8839))
- Fix an error in the documentation for the SAML username mapping provider. ([\#8873](https://github.com/matrix-org/synapse/issues/8873))
- Clarify comments around template directories in `sample_config.yaml`. ([\#8891](https://github.com/matrix-org/synapse/issues/8891))
- Moved instructions for database setup, adjusted heading levels and improved syntax highlighting in [INSTALL.md](../INSTALL.md). Contributed by fossterer. ([\#8987](https://github.com/matrix-org/synapse/issues/8987))
- Update the example value of `group_creation_prefix` in the sample configuration. ([\#8992](https://github.com/matrix-org/synapse/issues/8992))
- Link the Synapse developer room to the development section in the docs. ([\#9002](https://github.com/matrix-org/synapse/issues/9002))

Deprecations and Removals
-------------------------

- Deprecate Shutdown Room and Purge Room Admin APIs. ([\#8829](https://github.com/matrix-org/synapse/issues/8829))

Internal Changes
----------------

- Properly store the mapping of external ID to Matrix ID for CAS users. ([\#8856](https://github.com/matrix-org/synapse/issues/8856), [\#8958](https://github.com/matrix-org/synapse/issues/8958))
- Remove some unnecessary stubbing from unit tests. ([\#8861](https://github.com/matrix-org/synapse/issues/8861))
- Remove unused `FakeResponse` class from unit tests. ([\#8864](https://github.com/matrix-org/synapse/issues/8864))
- Pass `room_id` to `get_auth_chain_difference`. ([\#8879](https://github.com/matrix-org/synapse/issues/8879))
- Add type hints to push module. ([\#8880](https://github.com/matrix-org/synapse/issues/8880), [\#8882](https://github.com/matrix-org/synapse/issues/8882), [\#8901](https://github.com/matrix-org/synapse/issues/8901), [\#8940](https://github.com/matrix-org/synapse/issues/8940), [\#8943](https://github.com/matrix-org/synapse/issues/8943), [\#9020](https://github.com/matrix-org/synapse/issues/9020))
- Simplify logic for handling user-interactive-auth via single-sign-on servers. ([\#8881](https://github.com/matrix-org/synapse/issues/8881))
- Skip the SAML tests if the requirements (`pysaml2` and `xmlsec1`) aren't available. ([\#8905](https://github.com/matrix-org/synapse/issues/8905))
- Fix multiarch docker image builds. ([\#8906](https://github.com/matrix-org/synapse/issues/8906))
- Don't publish `latest` docker image until all archs are built. ([\#8909](https://github.com/matrix-org/synapse/issues/8909))
- Various clean-ups to the structured logging and logging context code. ([\#8916](https://github.com/matrix-org/synapse/issues/8916), [\#8935](https://github.com/matrix-org/synapse/issues/8935))
- Automatically drop stale forward-extremities under some specific conditions. ([\#8929](https://github.com/matrix-org/synapse/issues/8929))
- Refactor test utilities for injecting HTTP requests. ([\#8946](https://github.com/matrix-org/synapse/issues/8946))
- Add a maximum size of 50 kilobytes to .well-known lookups. ([\#8950](https://github.com/matrix-org/synapse/issues/8950))
- Fix bug in `generate_log_config` script which made it write empty files. ([\#8952](https://github.com/matrix-org/synapse/issues/8952))
- Clean up tox.ini file; disable coverage checking for non-test runs. ([\#8963](https://github.com/matrix-org/synapse/issues/8963))
- Add type hints to the admin and room list handlers. ([\#8973](https://github.com/matrix-org/synapse/issues/8973))
- Add type hints to the receipts and user directory handlers. ([\#8976](https://github.com/matrix-org/synapse/issues/8976))
- Drop the unused `local_invites` table. ([\#8979](https://github.com/matrix-org/synapse/issues/8979))
- Add type hints to the base storage code. ([\#8980](https://github.com/matrix-org/synapse/issues/8980))
- Support using PyJWT v2.0.0 in the test suite. ([\#8986](https://github.com/matrix-org/synapse/issues/8986))
- Fix `tests.federation.transport.RoomDirectoryFederationTests` and ensure it runs in CI. ([\#8998](https://github.com/matrix-org/synapse/issues/8998))
- Add type hints to the crypto module. ([\#8999](https://github.com/matrix-org/synapse/issues/8999))
 2021-01-06 07:42:38 -05:00
Patrick Cloke
0248409bfa
Handle a display name / avatar URL not included in a federation request. (#9023) 
These may be omitted if not set, but Synapse assumed they would
be in the response.
 2021-01-06 07:29:03 -05:00
Patrick Cloke
2fe0fb21f6 1.25.0rc1 2021-01-06 07:08:13 -05:00
Patrick Cloke
37eaf9c272
Fix-up assertions about last stream token in push (#9020) 
The last stream token is always known and we do not need to handle none.
 2021-01-05 10:53:15 -05:00
Patrick Cloke
06fefe0bb1
Add type hints to the logging context code. (#8939) 2021-01-05 08:06:55 -05:00
Patrick Cloke
9dde9c9f01
Implement MSC2176: Updated redaction rules (#8984) 
An experimental room version ("org.matrix.msc2176") contains
the new redaction rules for testing.
 2021-01-05 07:41:48 -05:00
Richard van der Hoff
111b673fc1
Add initial support for a "pick your IdP" page (#9017) 
During login, if there are multiple IdPs enabled, offer the user a choice of
IdPs.
 2021-01-05 11:25:28 +00:00
Richard van der Hoff
d2c616a413
Combine the SSO Redirect Servlets (#9015) 
* Implement CasHandler.handle_redirect_request

... to make it match OidcHandler and SamlHandler

* Clean up interface for OidcHandler.handle_redirect_request

Make it accept `client_redirect_url=None`.

* Clean up interface for `SamlHandler.handle_redirect_request`

... bring it into line with CAS and OIDC by making it take a Request parameter,
move the magic for `client_redirect_url` for UIA into the handler, and fix the
return type to be a `str` rather than a `bytes`.

* Define a common protocol for SSO auth provider impls

* Give SsoIdentityProvider an ID and register them

* Combine the SSO Redirect servlets

Now that the SsoHandler knows about the identity providers, we can combine the
various *RedirectServlets into a single implementation which delegates to the
right IdP.

* changelog
 2021-01-04 18:13:49 +00:00