Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-22 00:25:02 -05:00
Code Issues Packages Projects Releases Wiki Activity
18,350 Commits 4 Branches 609 Tags 60 MiB
7950aa8a27
Commit Graph

12924 Commits

Author SHA1 Message Date
Ivan Shapovalov
13c7ab8181
Fixes for PyPy compatibility (#9270) 
* synapse.app.base: only call gc.freeze() on CPython

gc.freeze() is an implementation detail of CPython garbage collector,
and notably does not exist on PyPy.

Rather than playing whack-a-mole and skipping the call when under PyPy,
simply restrict it to CPython because the whole gc module is
implementation-defined.

Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
 2021-01-30 17:22:05 +00:00
Erik Johnston
f2c1560eca
Ratelimit invites by room and target user (#9258) 2021-01-29 16:38:29 +00:00
Richard van der Hoff
0d81a6fa3e Merge branch 'social_login' into develop 2021-01-28 22:08:11 +00:00
Erik Johnston
4b73488e81
Ratelimit 3PID /requestToken API (#9238) 2021-01-28 17:39:21 +00:00
Erik Johnston
54a6afeee3
Cache config options in SSL verification (#9255) 
Reading from the config object is *slow*.
 2021-01-28 17:38:59 +00:00
David Teller
31d072aea0 FIXUP: linter 2021-01-28 16:53:40 +01:00
Patrick Cloke
a78016dadf
Add type hints to E2E handler. (#9232) 
This finishes adding type hints to the `synapse.handlers` module.
 2021-01-28 08:34:19 -05:00
David Teller
93f84e0373 FIXUP: Making get_event_context a bit more paranoid 2021-01-28 12:31:07 +01:00
David Teller
b755f60ce2 FIXUP: Removing awaitable 2021-01-28 12:31:07 +01:00
David Teller
a764869623 FIXUP: Doc 2021-01-28 12:31:07 +01:00
David Teller
b859919acc FIXUP: Now testing that the user is admin! 2021-01-28 12:31:07 +01:00
David Teller
de7f049527 FIXUP: Don't filter events at all for admin/v1/rooms/.../context/... 2021-01-28 12:31:07 +01:00
David Teller
10332c175c New API /_synapse/admin/rooms/{roomId}/context/{eventId} 
Signed-off-by: David Teller <davidt@element.io>
 2021-01-28 12:29:49 +01:00
Richard van der Hoff
a083aea396
Add 'brand' field to MSC2858 response (#9242) 
We've decided to add a 'brand' field to help clients decide how to style the
buttons.

Also, fix up the allowed characters for idp_id, while I'm in the area.
 2021-01-27 21:31:45 +00:00
Richard van der Hoff
869667760f
Support for scraping email addresses from OIDC providers (#9245) 2021-01-27 21:28:59 +00:00
Pankaj Yadav
2e537a0280
Check if a user is in the room before sending a PowerLevel event on their behalf (#9235) 2021-01-27 17:38:08 +00:00
Richard van der Hoff
fbd9de6d1f Synapse 1.26.0 (2021-01-27) 
===========================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes and for general upgrade guidance.
 
 No significant changes since 1.26.0rc2.
 
 Synapse 1.26.0rc2 (2021-01-25)
 ==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 
 Synapse 1.26.0rc1 (2021-01-20)
 ==============================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes and for general upgrade guidance.
 
 Features
 --------
 
 - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
 - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
 - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
 - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
 - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
 - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
 - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
 - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
 - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
 - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
 - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
 - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))
 
 Bugfixes
 --------
 
 - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
 - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
 - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
 - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
 - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
 - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
 - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
 - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
 - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
 - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
 - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
 - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
 - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
 - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))
 
 Improved Documentation
 ----------------------
 
 - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
 - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
 - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
 - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
 - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))
 
 Deprecations and Removals
 -------------------------
 
 - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))
 
 Internal Changes
 ----------------
 
 - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
 - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
 - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
 - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
 - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
 - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
 - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
 - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
 - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
 - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
 - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
 - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
 - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
 - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
 - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
 - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
 - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
 - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
 - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
 - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
 - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
 - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
 - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmARjqYACgkQM/xY9qcR
 MEiknRAAtga+KXEG40xg7w78XXQk6nXIhhi30nYue9/SeHMTCmLmd1+f0yTySrRz
 JMc/xe9VP1th5DThHZeHJf0tLIVz/VFA5gr/fmk44FFEFvc77IhttVHSXbYZLuM8
 komOzLTJy7ueVSpyF9xVfEEz6UIRc098b2KHjuBh3SGlZAOmFuFHUMVBeEuYo80E
 ayVWXPNP+7U5y1qLA9uY/oxMsFjl6bPvjak3YDyGWCr2Y4nOhM24AwccErs0yAtu
 kCRm9JR0yD72uRLyaGyMZ9Atg4o01BD7hu0ilxhbRIdAvI5Cdk0E7rZshoLr33wk
 Ls4XlK9SxJSV4vEu44OJBCRnDjQS9HQnwYk13z+H5HrJcxXRyIeCwRD7aofzNY8O
 kNg/1oasBDiW2iVoLJ4Y0nWLHPcNlWJxUzkkjlF8BmEUQjURNVdh0n3y3/ST+T1S
 scex0J9Z95HNoLfRYID0adrYqWTOl5GLzy9C1AwqlLWY79y6YsB4LDmmfQgoTKCM
 VICSYtEKWph6WXdpd/kJsTgHyG3UFAMDraWVez7TxXpc7lpvOMkld+0oQnHWAXC3
 XRc+63jTs6pFJ74M+ziqOrC/1ORg1DK8OLIm/TskLoYnn6L9Dh8ssXeabLv63H8o
 cEqETUbZ77rZro0Nz2H1OJG09uTFUzFEy5Ecya7g4DqtRT/Fs9Q=
 =I9CI
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0' into social_login

Synapse 1.26.0 (2021-01-27)
===========================

This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.

No significant changes since 1.26.0rc2.

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))

Synapse 1.26.0rc1 (2021-01-20)
==============================

This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.

Features
--------

- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))

Bugfixes
--------

- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))

Improved Documentation
----------------------

- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))

Deprecations and Removals
-------------------------

- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))

Internal Changes
----------------

- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 2021-01-27 17:27:58 +00:00
Richard van der Hoff
7fa1346f93 Merge branch 'social_login' into develop 2021-01-27 17:27:24 +00:00
Patrick Cloke
17b713850f Merge branch 'master' into develop 2021-01-27 11:13:21 -05:00
Patrick Cloke
e54746bdf7
Clean-up the template loading code. (#9200) 
* Enables autoescape by default for HTML files.
* Adds a new read_template method for reading a single template.
* Some logic clean-up.
 2021-01-27 10:59:50 -05:00
Patrick Cloke
73ed289bd2 1.26.0 2021-01-27 10:50:37 -05:00
Richard van der Hoff
a737cc2713
Implement MSC2858 support (#9183) 
Fixes #8928.
 2021-01-27 12:41:24 +00:00
Andrew Morgan
a64c29926e
Pass a dict, instead of None, to modules if a None config is specified in the homeserver config (#9229) 
If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead.

This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](5310808d3b/docs/sample_config.yaml (L1816-L1845)) states) then `None` is not passed to the module. An empty dict is passed instead.

This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.
 2021-01-27 11:49:31 +00:00
Patrick Cloke
1baab20352
Add type hints to various handlers. (#9223) 
With this change all handlers except the e2e_* ones have
type hints enabled.
 2021-01-26 10:50:21 -05:00
Patrick Cloke
26837d5dbe
Do not require the CAS service URL setting (use public_baseurl instead). (#9199) 
The current configuration is handled for backwards compatibility,
but is considered deprecated.
 2021-01-26 10:49:25 -05:00
Erik Johnston
dd8da8c5f6
Precompute joined hosts and store in Redis (#9198) 2021-01-26 13:57:31 +00:00
Patrick Cloke
4937fe3d6b
Try to recover from unknown encodings when previewing media. (#9164) 
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
 2021-01-26 07:32:17 -05:00
Jason Robinson
e5b659e9e1
Merge pull request #9062 from matrix-org/jaywink/admin-forward-extremities 
Add forward extremities endpoint to rooms admin API
 2021-01-26 12:57:38 +02:00
Erik Johnston
a1ff1e967f
Periodically send pings to detect dead Redis connections (#9218) 
This is done by creating a custom `RedisFactory` subclass that
periodically pings all connections in its pool.

We also ensure that the `replyTimeout` param is non-null, so that we
timeout waiting for the reply to those pings (and thus triggering a
reconnect).
 2021-01-26 10:54:54 +00:00
Jason Robinson
4936fc59fc Fix get forward extremities query 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-26 10:21:02 +02:00
Jason Robinson
cee4010f94 Merge branch 'develop' into jaywink/admin-forward-extremities 
# Conflicts:
#	synapse/rest/admin/__init__.py
 2021-01-26 10:15:32 +02:00
Jason Robinson
e20f18a766
Make natural join inner join 
Co-authored-by: Erik Johnston <erik@matrix.org>
 2021-01-26 10:13:35 +02:00
Patrick Cloke
5b857b77f7
Don't error if deleting a non-existent pusher. (#9121) 2021-01-25 14:52:30 -05:00
Patrick Cloke
4a55d267ee
Add an admin API for shadow-banning users. (#9209) 
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.

A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.

Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
 2021-01-25 14:49:39 -05:00
Richard van der Hoff
65fb3b2e25 Synapse 1.26.0rc2 (2021-01-25) 
==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
 MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
 +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
 BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
 d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
 oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
 O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
 VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
 g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
 =Qc7+
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc2' into social_login

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 2021-01-25 19:37:58 +00:00
Jason Robinson
fe18882bb5 Merge remote-tracking branch 'origin/develop' into jaywink/admin-forward-extremities 2021-01-25 15:55:54 +02:00
Patrick Cloke
e448dbbf5b Synapse 1.26.0rc2 (2021-01-25) 
==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
 MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
 +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
 BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
 d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
 oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
 O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
 VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
 g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
 =Qc7+
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc2' into develop

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 2021-01-25 08:51:45 -05:00
Patrick Cloke
a01605c136 1.26.0rc2 2021-01-25 08:25:40 -05:00
Patrick Cloke
6f7417c3db
Handle missing content keys when calculating presentable names. (#9165) 
Treat the content as untrusted and do not assume it is of
the proper form.
 2021-01-25 07:27:16 -05:00
Jason Robinson
8965b6cfec Merge branch 'develop' into jaywink/admin-forward-extremities 2021-01-23 21:41:35 +02:00
Jason Robinson
930ba00971 Add depth and received_ts to forward_extremities admin API response 
Also add a warning on the admin API documentation.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-23 21:34:32 +02:00
Erik Johnston
056327457f
Fix chain cover update to handle events with duplicate auth events (#9210) 2021-01-22 19:44:08 +00:00
Erik Johnston
28f255d5f3
Bump psycopg2 version (#9204) 
As we use `execute_values` with the `fetch` parameter.
 2021-01-22 11:14:49 +00:00
Patrick Cloke
a7882f9887
Return a 404 if no valid thumbnail is found. (#9163) 
If no thumbnail of the requested type exists, return a 404 instead
of erroring. This doesn't quite match the spec (which does not define
what happens if no thumbnail can be found), but is consistent with
what Synapse already does.
 2021-01-21 14:53:58 -05:00
Erik Johnston
758ed5f1bc
Speed up chain cover calculation (#9176) 2021-01-21 17:00:12 +00:00
Erik Johnston
12ec55bfaa
Increase perf of handling concurrent use of StreamIDGenerators. (#9190) 
We have seen a failure mode here where if there are many in flight
unfinished IDs then marking an ID as finished takes a lot of CPU (as
calling deque.remove iterates over the list)
 2021-01-21 16:31:51 +00:00
Erik Johnston
939ef657ce Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 16:05:13 +00:00
Erik Johnston
ccfafac882
Add schema update to fix existing DBs affected by #9193 (#9195) 2021-01-21 16:03:25 +00:00
Erik Johnston
b249f002b8 Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 15:09:30 +00:00
Erik Johnston
2506074ef0
Fix receipts or account data not being sent down sync (#9193) 
Introduced in #9104 

This wasn't picked up by the tests as this is all fine the first time you run Synapse (after upgrading), but then when you restart the wrong value is pulled from `stream_positions`.
 2021-01-21 15:09:09 +00:00
Erik Johnston
7a43482f19
Use execute_batch in more places (#9188) 
* Use execute_batch in more places

* Newsfile
 2021-01-21 14:44:12 +00:00
Dirk Klimpel
c55e62548c
Add tests for List Users Admin API (#9045) 2021-01-21 09:18:46 -05:00
Richard van der Hoff
42a8e81370
Add a check for duplicate IdP ids (#9184) 2021-01-21 13:20:58 +00:00
Richard van der Hoff
b5120f09f1 Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 13:17:07 +00:00
Richard van der Hoff
7447f19702
Prefix idp_id with "oidc-" (#9189) 
... to avoid clashes with other SSO mechanisms
 2021-01-21 12:25:02 +00:00
Erik Johnston
eee6fcf5fa
Use execute_batch instead of executemany in places (#9181) 
`execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.
 2021-01-21 10:22:53 +00:00
Patrick Cloke
f81d02d75b Synapse 1.26.0rc1 (2021-01-20) 
==============================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 verious is not trivial.  Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes  and for general upgrade guidance.
 
 Features
 --------
 
 - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
 - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
 - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
 - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
 - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
 - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
 - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
 - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
 - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
 - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
 - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
 - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))
 
 Bugfixes
 --------
 
 - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
 - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
 - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
 - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
 - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
 - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
 - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
 - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
 - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
 - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
 - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
 - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
 - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
 - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))
 
 Improved Documentation
 ----------------------
 
 - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
 - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
 - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
 - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
 - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))
 
 Deprecations and Removals
 -------------------------
 
 - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))
 
 Internal Changes
 ----------------
 
 - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
 - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
 - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
 - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
 - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
 - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
 - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
 - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
 - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
 - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
 - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
 - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
 - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
 - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
 - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
 - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
 - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
 - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
 - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
 - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
 - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
 - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
 - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAIVFkACgkQM/xY9qcR
 MEhLfQ//Z4xnP5Icx732k2xpdR+JaXX6heNVFm/UuLNd6wq+uXjO/Dgc9IKQ5LXD
 kfG+/OnNGDr/eUk4mQ5s/ccBvaTrOid2IIemJd4lUBy/Q5LvPvfKIp08QFr72WJT
 7Y4Ma54nNSSMUAdESgj/aFAHjATbfHhxPOZ5OdQHGLAuJ/OUwR2ksasVNCuINbgh
 8vrxrzjeYn1Zl3UTBrsdCcat7AFqYjxK/Y0i6JxfSgRwDZjUbW4M2C4OGWZJPUrU
 2YTPHangxd+22+HObE2KDEzaHDdCR/Kj4dhCiJBynZ5iucuHSS+mgf39RtbjqWML
 01cmXXUV+FH7rnCilmF9rYdmHZ6L9BJxtrQDQU4dB106BX9G7hKR6rYHc9Z6tjOh
 3tg0cKWYMXOsTNnvRqiFNh2q0yBKS228HRAwyGosW/6NiYj43ArtYP6knX7dQyBX
 0mYNkfry2dyX6Kj5el0i9MOTHYQxfc8apsBm2M3OTNYukgQKA0NhbAquibaAxEEW
 2qIqbSp4CUlGPvM7+u+ZGTu0hEbVKKNjqMfXhuY8A4/BWiV2mSqKcEirR5cE/LqY
 mq5Ac7vbO4Uh+1xiRw/G9ITi1dqAjYIzhBawlhdUPOh+aINWTUikYYzjUSmm4PbN
 H9BTPLA9iAdVRfuWJ9um2G0DdS8Qpx/aIRh3MScXVly6cGCC6Do=
 =kgqn
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc1' into develop

Synapse 1.26.0rc1 (2021-01-20)
==============================

This release brings a new schema version for Synapse and rolling back to a previous
verious is not trivial.  Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes  and for general upgrade guidance.

Features
--------

- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))

Bugfixes
--------

- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))

Improved Documentation
----------------------

- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))

Deprecations and Removals
-------------------------

- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))

Internal Changes
----------------

- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 2021-01-20 11:27:39 -05:00
Richard van der Hoff
e51b2f3f91
Tighten the restrictions on idp_id (#9177) 2021-01-20 08:55:14 -05:00
Richard van der Hoff
0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Patrick Cloke
620ecf13b0
Various improvements to the federation client. (#9129) 
* Type hints for `FederationClient`.
* Using `async` functions instead of returning `Awaitable` instances.
 2021-01-20 07:59:18 -05:00
Richard van der Hoff
6c0dfd2e8e Merge remote-tracking branch 'origin/develop' into release-v1.26.0 2021-01-20 12:33:05 +00:00
Richard van der Hoff
fa50e4bf4d
Give public_baseurl a default value (#9159) 2021-01-20 12:30:41 +00:00
Patrick Cloke
72822e60be 1.26.0rc1 2021-01-20 07:14:25 -05:00
Patrick Cloke
fa842a9866
Use the account handler in additional places. (#9166) 2021-01-20 10:44:52 +00:00
Patrick Cloke
47d48a5853
Validate the server name for the /publicRooms endpoint. (#9161) 
If a remote server name is provided, ensure it is something reasonable
before making remote connections to it.
 2021-01-19 14:21:59 -05:00
Richard van der Hoff
73b03722f4
Fix error messages from OIDC config parsing (#9153) 
Make sure we report the correct config path for errors in the OIDC configs.
 2021-01-19 14:56:54 +00:00
Erik Johnston
6633a4015a
Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
Erik Johnston
f08ef64926
Enforce all replication HTTP clients calls use kwargs (#9144) 2021-01-18 15:24:04 +00:00
Patrick Cloke
2b467d0b61
Properly raise an exception when the body exceeds the max size. (#9145) 
...instead of just creating the exception object and doing nothing with it.
 2021-01-18 10:21:42 -05:00
Richard van der Hoff
02070c69fa
Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128) 
* Factor out a common TestHtmlParser

Looks like I'm doing this in a few different places.

* Improve OIDC login test

Complete the OIDC login flow, rather than giving up halfway through.

* Ensure that OIDC login works with multiple OIDC providers

* Fix bugs in handling clientRedirectUrl

 - don't drop duplicate query-params, or params with no value
 - allow utf-8 in query-params
 2021-01-18 14:52:49 +00:00
Patrick Cloke
de1f8de319
Ensure the user ID is serialized in the payload instead of used as an instance name. (#9130) 2021-01-18 11:08:26 +00:00
Erik Johnston
350d9923cd
Make chain cover index bg update go faster (#9124) 
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
 2021-01-15 17:18:37 +00:00
Richard van der Hoff
2de7e263ed
Ensure we store pusher data as text (#9117) 
I don't think there's any need to use canonicaljson here.

Fixes: #4475.
 2021-01-15 16:57:23 +00:00
Richard van der Hoff
9de6b94117
Land support for multiple OIDC providers (#9110) 
This is the final step for supporting multiple OIDC providers concurrently.

First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before:

    oidc_config:
       enabled: true
       issuer: "https://oidc_provider"
       # etc

After:

    oidc_providers:
     - idp_id: prov1
       issuer: "https://oidc_provider"

     - idp_id: prov2
       issuer: "https://another_oidc_provider"

The old format is still grandfathered in.

With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
 2021-01-15 16:55:29 +00:00
Patrick Cloke
3e4cdfe5d9
Add an admin API endpoint to protect media. (#9086) 
Protecting media stops it from being quarantined when
e.g. all media in a room is quarantined. This is useful
for sticker packs and other media that is uploaded by
server administrators, but used by many people.
 2021-01-15 11:18:09 -05:00
Patrick Cloke
74dd906041
Avoid raising the body exceeded error multiple times. (#9108) 
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
 2021-01-15 11:00:13 -05:00
Richard van der Hoff
9ffac2bef1
Remote dependency on distutils (#9125) 
`distutils` is pretty much deprecated these days, and replaced with
`setuptools`. It's also annoying because it's you can't `pip install` it, and
it's hard to figure out which debian package we should depend on to make sure
it's there.

Since we only use it for a tiny function anyway, let's just vendor said
function into our codebase.
 2021-01-15 15:59:20 +00:00
Patrick Cloke
d34c6e1279
Add type hints to media rest resources. (#9093) 2021-01-15 10:57:37 -05:00
Richard van der Hoff
4575ad0b1e
Store an IdP ID in the OIDC session (#9109) 
Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP.

As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
 2021-01-15 13:22:12 +00:00
Richard van der Hoff
14950a45d6
Merge pull request #9091 from matrix-org/rav/error_on_bad_sso 
Give the user a better error when they present bad SSO creds
 2021-01-15 00:27:13 +00:00
Erik Johnston
1a08e0cdab
Fix event chain bg update. (#9118) 
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
 2021-01-14 18:57:32 +00:00
Erik Johnston
d2479c6870
Fix perf of get_cross_signing_keys (#9116) 2021-01-14 17:57:09 +00:00
Erik Johnston
659c415ed4
Fix chain cover background update to work with split out event persisters (#9115) 2021-01-14 17:19:35 +00:00
Erik Johnston
631dd06f2c
Fix get destinations to catch up query. (#9114) 
t was doing a sequential scan on `destination_rooms`, which took
minutes.
 2021-01-14 16:47:21 +00:00
Erik Johnston
7036e24e98
Add background update for add chain cover index (#9029) 2021-01-14 15:18:27 +00:00
Richard van der Hoff
21a296cd5a
Split OidcProvider out of OidcHandler (#9107) 
The idea here is that we will have an instance of OidcProvider for each
configured IdP, with OidcHandler just doing the marshalling of them.

For now it's still hardcoded with a single provider.
 2021-01-14 13:29:17 +00:00
Tim Leung
12702be951
Fix wrong arguments being passed to BlacklistingAgentWrapper (#9098) 
A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper
used by the WellyKnownResolver. This coulld cause exceptions when attempting to
connect to IP addresses that are blacklisted, but in reality this did not have any
observable affect since this code is not used for IP literals.
 2021-01-14 06:59:26 -05:00
Richard van der Hoff
420031906a Move complete_sso_ui_auth into SSOHandler 
since we're hacking on this code anyway, may as well move it out of the
cluttered AuthHandler.
 2021-01-13 20:22:41 +00:00
Richard van der Hoff
5310808d3b Give the user a better error when they present bad SSO creds 
If a user tries to do UI Auth via SSO, but uses the wrong account on the SSO
IdP, try to give them a better error.

Previously, the UIA would claim to be successful, but then the operation in
question would simply fail with "auth fail". Instead, serve up an error page
which explains the failure.
 2021-01-13 20:22:41 +00:00
Richard van der Hoff
d02e4b2825
Merge pull request #9105 from matrix-org/rav/multi_idp/oidc_provider_config 
Enhancements to OIDC configuration handling
 2021-01-13 19:51:46 +00:00
Patrick Cloke
aee8e6a95d
Reduce scope of exception handler. (#9106) 
Removes a bare `except Exception` clause and replaces it with
catching a specific exception around the portion that might throw.
 2021-01-13 13:27:49 -05:00
Richard van der Hoff
dc3c83a933 Add jsonschema verification for the oidc provider config 2021-01-13 17:47:27 +00:00
Patrick Cloke
d1eb1b96e8
Register the /devices endpoint on workers. (#9092) 2021-01-13 12:35:40 -05:00
Richard van der Hoff
7cc9509eca Extract OIDCProviderConfig object 
Collect all the config options which related to an OIDC provider into a single
object.
 2021-01-13 16:40:02 +00:00
Patrick Cloke
98a64b7f7f
Add basic domain validation for DomainSpecificString.is_valid. (#9071) 
This checks that the domain given to `DomainSpecificString.is_valid` (e.g.
`UserID`, `RoomAlias`, etc.) is of a valid form. Previously some validation
was done on the localpart (e.g. the sigil), but not the domain portion.
 2021-01-13 07:05:16 -05:00
Erik Johnston
aa4d8c1f9a Merge branch 'master' into develop 2021-01-13 10:36:55 +00:00
Richard van der Hoff
bc4bf7b384
Preparatory refactors of OidcHandler (#9067) 
Some light refactoring of OidcHandler, in preparation for bigger things:

  * remove inheritance from deprecated BaseHandler
  * add an object to hold the things that go into a session cookie
  * factor out a separate class for manipulating said cookies
 2021-01-13 10:26:12 +00:00
Erik Johnston
3dd6ba135e 1.25.0 2021-01-13 10:19:12 +00:00
Dirk Klimpel
7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932) 
This only applies if the user's data is to be erased.
 2021-01-12 16:30:15 -05:00
Richard van der Hoff
789d9ebad3
UI Auth via SSO: redirect the user to an appropriate SSO. (#9081) 
If we have integrations with multiple identity providers, when the user does a UI Auth, we need to redirect them to the right one.

There are a few steps to this. First of all we actually need to store the userid of the user we are trying to validate in the UIA session, since the /auth/sso/fallback/web request is unauthenticated.

Then, once we get the /auth/sso/fallback/web request, we can fish the user id out of the session, and use it to look up the external id mappings, and hence pick an SSO provider for them.
 2021-01-12 17:38:03 +00:00