13879 Commits

Author	SHA1	Message	Date
Richard van der Hoff	785bceef72	Merge branch 'release-v1.37' into develop	2021-06-29 20:25:47 +01:00
Richard van der Hoff	f99e9cc2da	v1.37.1a1	2021-06-29 19:58:25 +01:00
Richard van der Hoff	c0bebd00ef	Merge remote-tracking branch 'origin/erikj/async_federation_base_branch' into release-v1.37	2021-06-29 19:55:55 +01:00
Erik Johnston	c54db67d0e	Handle inbound events from federation asynchronously (#10272) ... Fixes #9490 This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous. Two drawbacks: Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately. We now only process one event per room at a time. This can be fixed up further down the line.	2021-06-29 19:55:22 +01:00
Erik Johnston	85d237eba7	Add a distributed lock (#10269) ... This adds a simple best effort locking mechanism that works cross workers.	2021-06-29 19:15:47 +01:00
Patrick Cloke	f55836929d	Do not recurse into non-spaces in the spaces summary. (#10256) ... Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.	2021-06-29 12:00:04 -04:00
Richard van der Hoff	7647b0337f	Fix populate_stream_ordering2 background job (#10267) ... It was possible for us not to find any rows in a batch, and hence conclude that we had finished. Let's not do that.	2021-06-29 12:43:36 +01:00
Richard van der Hoff	60efc51a2b	Migrate stream_ordering to a bigint (#10264) ... * Move background update names out to a separate class `EventsBackgroundUpdatesStore` gets inherited and we don't really want to further pollute the namespace. * Migrate stream_ordering to a bigint * changelog	2021-06-29 11:25:34 +01:00
Richard van der Hoff	a0ed0f363e	Soft-fail spammy events received over federation (#10263)	2021-06-29 11:08:06 +01:00
Brendan Abolivier	3d370efc6d	Merge branch 'master' into develop	2021-06-29 10:20:38 +01:00
Brendan Abolivier	cdf569e468	1.37.0	2021-06-29 10:15:34 +01:00
Patrick Cloke	0555d7b0dc	Add additional types to the federation transport server. (#10213)	2021-06-28 07:36:41 -04:00
Richard van der Hoff	8165ba48b1	Return errors from send_join etc if the event is rejected (#10243) ... Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.	2021-06-24 16:00:08 +01:00
Richard van der Hoff	6e8fb42be7	Improve validation for send_{join,leave,knock} (#10225) ... The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.	2021-06-24 15:30:49 +01:00
Quentin Gliech	bd4919fb72	MSC2918 Refresh tokens implementation (#9450) ... This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>	2021-06-24 14:33:20 +01:00
Brendan Abolivier	763dba77ef	Synapse 1.37.0rc1 (2021-06-24) ... ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUW7UACgkQHgFcFF8Z Fs1ZfQ//bSyzg9mB7F43JdVBH9cpGups3Ur6nmsqE608/i/OEtomx9kTZsoN/9AO j6WoM1bhJ3j9hoE63w4DIPVhBrojtwmYWeFrRvpRVlwTzZPhfij1nIH9pPrNVwd6 /LE0uA1kW5sDZCTiOhrz84BBBz6knjBH4otl+MDDdrszYixoP3i3SrrZVJAEYyGz Dpw2OudpoOtMdGyzFMr2ewf2mQm5UFEPs9MT0AqhQJMjyM1lJYzrq3Obv1oTdCJq BdLx6kMhK8+MbZPrcqIMXgSMHLYeQYeJOrdTzuCozFo0DSblThpW//aa21+sZtPx dnIE0PpjAsm2Qg1II3EUX+YycYpI5KPhMq0ydEmUD7vTuGgJCKqAV3FBRwMlF2VE 9SOr8Mjd1ip03dHz2vlg38OlrFJ52P0e1RDZwlDtWfeJjZGZ4W6VjMhT7H/Z5USg 2J1FRPTD8E9w8wBOkjVcl76Lj6AY1KU36/ZLoYUbtrt58XYfkD4OzZmAxdJrqbMg na5T9bd9iM8vWditU+vncxU+c7Bkn55GFyDVMQYl8QtXzBvnbv8gQilGwJLkncZJ MreQ3OIBPbLNcHml56nmmY+ab+AtrIxuFl0UcZu+LGW8+lN+UZ9G9q0NFO3HbAWN /LuhCVpO42SLAuLlIhnO3mcHM5jzZodKIa6oXpcMS2jv0nylQWo= =MtOo -----END PGP SIGNATURE----- gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUXG8ACgkQHgFcFF8Z Fs2buxAAg7X4mYKHwlWLn2n2W6bUKY24HAnjtzMrNW76xIHXBJJiJMnhQD3MSKVd uiUdolc+Tt3drJYOJyurP8R3YI0OuEVbSL0Krs8f3l6jNOU0Gy7YQzc6XUHnA42J rb6mU8gMdCusk9RHHr1r+sBefwzmxTk4naCkw2n5i+r4ZfNparJK7H9yXCPoedUN VBZi9f09jaUIm1SiHgK24sW9XfE6ifjdVg+E/r599v2eHJxtCTHUasH9TCAFD8KL g17NtAP95+SQWPGwUXJFZYcVee+pGbylXuwP0RXXmMvyOWVKloI0LE6phGyLQ5cH rMRagIkvVLMdYagaX8Tf47hNLVCXRJpuryxMgY3+5fx0O7lCMXlC3mDfs9zze+Fj KXmJtVy/vpd+zpjQw+HHPnL1dDzsKnDqLFXFCj9Sjv6A/ruckxwWofuh0z83dYtv 1Abp8k7gVTVGZqLGPo8gRoqI2jQhWDvmJd+crb05qp86SzCz1sCma2ujWQtv3c1t 6aQpfmGMhFPp+hS2lS8AcyxK4vq+GudtXUOgzdqCNvIs+eJAV5cq6uufLIy6c2XR FAR7UowoAoDvqTdD7+i5nHEjbudjjN5KgPXlO3a9+gG5jOR3IsOLZ8FVN0ML6qIJ wzl0wM9YIlxyleMlWb0l4CFNDKrGJVhWgOupruHS+D5jfZQsZ3g= =AjRf -----END PGP SIGNATURE----- Merge tag 'v1.37.0rc1' into develop Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))	2021-06-24 11:20:28 +01:00
Brendan Abolivier	c955e37868	Fix wrapping of legacy check_registration_for_spam (#10238) ... Fixes #10234	2021-06-23 17:22:08 +02:00
Andrew Morgan	394673055d	Re-introduce "Leave out optional keys from /sync" change (#10214) ... Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.	2021-06-23 15:57:41 +01:00
Brendan Abolivier	e19e3d452d	Improve the reliability of auto-joining remote rooms (#10237) ... If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do. Fixes #10233 (at least to some extent)	2021-06-23 16:14:52 +02:00
Richard van der Hoff	8beead66ae	Send out invite rejections and knocks over federation (#10223) ... ensure that events sent via `send_leave` and `send_knock` are sent on to the rest of the federation.	2021-06-23 12:54:50 +01:00
Michael[tm] Smith	27c06a6e06	Drop Origin & Accept from Access-Control-Allow-Headers value (#10114) ... * Drop Origin & Accept from Access-Control-Allow-Headers value This change drops the Origin and Accept header names from the value of the Access-Control-Allow-Headers response header sent by Synapse. Per the CORS protocol, it’s not necessary or useful to include those header names. Details: Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name, Origin is a “forbidden header name” set by the browser and that frontend JavaScript code is never allowed to set. So the value of Access-Control-Allow-Headers isn’t relevant to Origin or in general to other headers set by the browser itself — the browser never ever consults the Access-Control-Allow-Headers value to confirm that it’s OK for the request to include an Origin header. And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header, Accept is a “CORS-safelisted request-header”, which means that browsers allow requests to contain the Accept header regardless of whether the Access-Control-Allow-Headers value contains "Accept". So it’s unnecessary for the Access-Control-Allow-Headers to explicitly include Accept. Browsers will not perform a CORS preflight for requests containing an Accept request header. Related: https://github.com/matrix-org/matrix-doc/pull/3225 Signed-off-by: Michael[tm] Smith <mike@w3.org>	2021-06-23 11:25:03 +01:00
Brendan Abolivier	9ec45aca1f	1.37.0rc1	2021-06-23 09:38:27 +01:00
Erik Johnston	33701dc116	Fix schema delta to not take as long on large servers (#10227) ... Introduced in #6739	2021-06-22 12:00:45 +01:00
Brendan Abolivier	34db6bb9f5	Warn users trying to use the deprecated spam checker interface (#10210) ... So admins aren't surprised if things break when we remove this code in a couple of months.	2021-06-22 12:24:10 +02:00
Eric Eastwood	96f6293de5	Add endpoints for backfilling history (MSC2716) (#9247) ... Work on https://github.com/matrix-org/matrix-doc/pull/2716	2021-06-22 10:02:53 +01:00
jkanefendt	756fd513df	Implement config option sso.update_profile_information (#10108) ... Implemented config option sso.update_profile_information to keep user's display name in sync with the SSO displayname. Signed-off-by: Johannes Kanefendt <johannes.kanefendt@krzn.de>	2021-06-21 23:48:57 +01:00
Erik Johnston	a5cd05beee	Fix performance of responding to user key requests over federation (#10221) ... We were repeatedly looking up a config option in a loop (using the unclassed config style), which is expensive enough that it can cause large CPU usage.	2021-06-21 14:38:59 +01:00
Andrew Morgan	182147195b	Check third party rules before persisting knocks over federation (#10212) ... An accidental mis-ordering of operations during #6739 technically allowed an incoming knock event over federation in before checking it against any configured Third Party Access Rules modules. This PR corrects that by performing the TPAR check *before* persisting the event.	2021-06-21 11:57:09 +01:00
Richard van der Hoff	107c06081f	Ensure that errors during startup are written to the logs and the console. (#10191) ... * Defer stdio redirection until we are about to start the reactor * Catch and handle exceptions during startup	2021-06-21 11:41:25 +01:00
Patrick Cloke	0bd968921c	Fix a missing await when in the spaces summary. (#10208) ... This could cause a minor data leak if someone defined a non-restricted join rule with an allow key or used a restricted join rule in an older room version, but this is unlikely. Additionally this starts adding unit tests to the spaces summary handler.	2021-06-18 18:41:33 +01:00
Brendan Abolivier	1b3e398bea	Standardise the module interface (#10062) ... This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.	2021-06-18 12:15:52 +01:00
Richard van der Hoff	91fa9cca99	Expose opentracing trace id in response headers (#10199) ... Fixes: #9480	2021-06-18 11:43:22 +01:00
Brendan Abolivier	08c8469322	Remove support for ACME v1 (#10194) ... Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.	2021-06-17 18:56:48 +01:00
Patrick Cloke	8c97d5863f	Update MSC3083 support per changes in the MSC. (#10189) ... Adds a "type" field and generalize "space" to "room_id".	2021-06-17 12:53:27 -04:00
Richard van der Hoff	fcf3c7032b	Ensure that we do not cache empty sync responses after a timeout (#10158) ... Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.	2021-06-17 16:23:11 +01:00
Richard van der Hoff	9cf6e0eae7	Rip out the DNS lookup limiter (#10190) ... As I've written in various places in the past (#7113, #9865) I'm pretty sure this is doing nothing useful at all.	2021-06-17 16:22:41 +01:00
Marcus	8070b893db	update black to 21.6b0 (#10197) ... Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>	2021-06-17 15:20:06 +01:00
Andrew Morgan	6f1a28de19	Fix incorrect time magnitude on delayed call (#10195) ... Fixes https://github.com/matrix-org/synapse/issues/10030. We were expecting milliseconds where we should have provided a value in seconds. The impact of this bug isn't too bad. The code is intended to count the number of remote servers that the homeserver can see and report that as a metric. This metric is supposed to run initially 1 second after server startup, and every 60s as well. Instead, it ran 1,000 seconds after server startup, and every 60s after startup. This fix allows for the correct metrics to be collected immediately, as well as preventing a random collection 1,000s in the future after startup.	2021-06-17 15:04:26 +01:00
Eric Eastwood	a911dd768b	Add fields to better debug where events are being soft_failed (#10168) ... Follow-up to https://github.com/matrix-org/synapse/pull/10156#discussion_r650292223	2021-06-17 14:59:45 +01:00
Richard van der Hoff	52c60bd0a9	Fix persist_events to stop leaking opentracing contexts (#10193)	2021-06-17 11:21:53 +01:00
Patrick Cloke	18edc9ab06	Improve comments in the structured logging code. (#10188)	2021-06-16 19:18:02 +01:00
Patrick Cloke	76f9c701c3	Always require users to re-authenticate for dangerous operations. (#10184) ... Dangerous actions means deactivating an account, modifying an account password, or adding a 3PID. Other actions (deleting devices, uploading keys) can re-use the same UI auth session if ui_auth.session_timeout is configured.	2021-06-16 11:07:28 -04:00
Erik Johnston	36c426e294	Add debug logging when we enter/exit Measure block (#10183) ... It can be helpful to know when trying to track down slow requests.	2021-06-16 13:29:54 +01:00
Lukas Lihotzki	2c240213f4	Fix requestOpenIdToken response: integer expires_in (#10175) ... `expires_in` must be an integer according to the OpenAPI spec: https://github.com/matrix-org/matrix-doc/blob/master/data/api/client-server/definitions/openid_token.yaml#L32 True division (`/`) returns a float instead (`"expires_in": 3600.0`). Floor division (`//`) returns an integer, so the response is spec compliant. Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>	2021-06-16 13:16:35 +01:00
Richard van der Hoff	9e405034e5	Make opentracing trace into event persistence (#10134) ... * Trace event persistence When we persist a batch of events, set the parent opentracing span to the that from the request, so that we can trace all the way in. * changelog * When we force tracing, set a baggage item ... so that we can check again later. * Link in both directions between persist_events spans	2021-06-16 11:41:15 +01:00
Erik Johnston	d09e24a52d	Merge branch 'master' into develop	2021-06-15 15:52:24 +01:00
Erik Johnston	1c8045f674	1.36.0	2021-06-15 15:42:02 +01:00
Patrick Cloke	4911f7931d	Remove support for unstable MSC1772 prefixes. (#10161) ... The stable prefixes have been supported since v1.34.0. The unstable prefixes are not supported by any known clients.	2021-06-15 08:03:17 -04:00
Patrick Cloke	9e5ab6dd58	Remove the experimental flag for knocking and use stable prefixes / endpoints. (#10167) ... * Room version 7 for knocking. * Stable prefixes and endpoints (both client and federation) for knocking. * Removes the experimental configuration flag.	2021-06-15 07:45:14 -04:00
Michael Kutzner	aac2c49b9b	Fix 'ip_range_whitelist' not working for federation servers (#10115) ... Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers. Signed-off-by: Michael Kutzner 1mikure@gmail.com	2021-06-15 08:53:55 +01:00
Richard van der Hoff	1dfdc87b9b	Refactor EventPersistenceQueue (#10145) ... some cleanup, pulled out of #10134.	2021-06-14 11:59:27 +01:00
Richard van der Hoff	d7808a2dde	Extend ResponseCache to pass a context object into the callback (#10157) ... This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.	2021-06-14 10:26:09 +01:00
Erik Johnston	29966a285d	Synapse 1.36.0rc2 (2021-06-11) ... ============================== Bugfixes -------- - Fix a bug which caused presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149)) - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163)) - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165)) -----BEGIN PGP SIGNATURE----- iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmDDdrcQHGVyaWtAbWF0 cml4Lm9yZwAKCRClQuTtGw+sCTzfB/4qaTqW2mBiwjf52SmOu6HNyd8uQd6nLIAZ mJC218Wakh2tT0W4iVkKwUgpuHFbtcy0rSNTlXtW4kG8XzhpTvT56RH9qls99aD3 SGKqYpOv6NkWZibN6NdVvLDW85ixficDTXco3BljRCIMlORhY0swy+LWLwksdjWj 6kQ+Gi/QAtKP3Pt5epYs0Ix5o1T94DfZOWE//mqBhG5cMDAw/K/G/c8tRfEjclt9 wACBjmt2fw/Lbn9j3b0feNVp+xnFcFNuAK2bSEd8Y3yph1mhjdsIszULnM7IFNsR Q8zg+i7PJKNq8pQjei8j8T/aKscPTPH5XGqOSLlizj15snsiwlkz =C6lV -----END PGP SIGNATURE----- Merge tag 'v1.36.0rc2' into develop Synapse 1.36.0rc2 (2021-06-11) ============================== Bugfixes -------- - Fix a bug which caused presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149)) - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163)) - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165))	2021-06-11 15:46:38 +01:00
Erik Johnston	fb10a73e85	1.36.0rc2	2021-06-11 15:21:34 +01:00
Erik Johnston	cdd985c64f	Only send a presence state to a destination once (#10165) ... It turns out that we were sending the same presence state to a remote potentially multiple times.	2021-06-11 15:21:08 +01:00
Erik Johnston	5e0b4719ea	Fix sending presence over federation when using workers (#10163) ... When using a federation sender we'd send out all local presence updates over federation even when they shouldn't be. Fixes #10153.	2021-06-11 15:20:54 +01:00
Erik Johnston	c955f22e2c	Fix bug when running presence off master (#10149) ... Hopefully fixes #10027.	2021-06-11 15:20:45 +01:00
Erik Johnston	968f8283b4	Only send a presence state to a destination once (#10165) ... It turns out that we were sending the same presence state to a remote potentially multiple times.	2021-06-11 15:19:42 +01:00
Richard van der Hoff	c1b9922498	Support for database schema version ranges (#9933) ... This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.	2021-06-11 14:45:53 +01:00
Erik Johnston	c8dd4db9eb	Fix sending presence over federation when using workers (#10163) ... When using a federation sender we'd send out all local presence updates over federation even when they shouldn't be. Fixes #10153.	2021-06-11 13:08:30 +01:00
Andrew Morgan	a15a046c93	Clean up a broken import in admin_cmd.py (#10154)	2021-06-11 11:34:40 +01:00
Erik Johnston	d26d15ba3d	Fix bug when running presence off master (#10149) ... Hopefully fixes #10027.	2021-06-11 10:27:12 +01:00
Eric Eastwood	b31daac01c	Add metrics to track how often events are soft_failed (#10156) ... Spawned from missing messages we were seeing on `matrix.org` from a federated Gtiter bridged room, https://gitlab.com/gitterHQ/webapp/-/issues/2770. The underlying issue in Synapse is tracked by https://github.com/matrix-org/synapse/issues/10066 where the message and join event race and the message is `soft_failed` before the `join` event reaches the remote federated server. Less soft_failed events = better and usually this should only trigger for events where people are doing bad things and trying to fuzz and fake everything.	2021-06-11 10:12:35 +01:00
Aaron Raimist	e6245e6d48	Mention that you need to configure max upload size in reverse proxy as well (#10122) ... Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-06-10 11:40:24 +01:00
Andrew Morgan	a7a37437bc	Integrate knock rooms with the public rooms directory (#9359) ... This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403. Specifically, it: * Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory * Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room Based on https://github.com/matrix-org/synapse/issues/6739. Complement tests for this change: https://github.com/matrix-org/complement/pull/72	2021-06-09 20:31:31 +01:00
Sorunome	d936371b69	Implement knock feature (#6739) ... This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io	2021-06-09 19:39:51 +01:00
Patrick Cloke	11846dff8c	Limit the number of in-flight /keys/query requests from a single device. (#10144)	2021-06-09 07:05:32 -04:00
Richard van der Hoff	1bf83a191b	Clean up the interface for injecting opentracing over HTTP (#10143) ... * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog	2021-06-09 11:33:00 +01:00
Patrick Cloke	c7f3fb2745	Add type hints to the federation server transport. (#10080)	2021-06-08 11:19:25 -04:00
Andrew Morgan	8df9941cc2	1.36.0rc1	2021-06-08 14:09:00 +01:00
Erik Johnston	1092718cac	Fix logging context when opening new DB connection (#10141) ... Fixes #10140	2021-06-08 13:49:29 +01:00
Patrick Cloke	9e4610cc27	Correct type hints for parse_string(s)_from_args. (#10137)	2021-06-08 08:30:48 -04:00
Erik Johnston	c842c581ed	When joining a remote room limit the number of events we concurrently check signatures/hashes for (#10117) ... If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.	2021-06-08 11:07:46 +01:00
Erik Johnston	a0101fc021	Handle /backfill returning no events (#10133) ... Fixes #10123	2021-06-08 10:37:01 +01:00
Richard van der Hoff	0acb5010ec	More database opentracing (#10136) ... Add a couple of extra logs/spans, to give a bit of a better idea.	2021-06-07 18:01:32 +01:00
Richard van der Hoff	b2557cbf42	opentracing: use a consistent name for background processes (#10135) ... ... otherwise we tend to get a namespace clash between the bg process and the functions that it calls.	2021-06-07 17:57:49 +01:00
14mRh4X0r	8942e23a69	Always update AS last_pos, even on no events (#10107) ... Fixes #1834. `get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether. Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list. Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>	2021-06-07 15:42:05 +01:00
Dirk Klimpel	d558292548	Add missing type hints to the admin API servlets (#10105)	2021-06-07 15:12:34 +01:00
Richard van der Hoff	fa1db8f156	Delete completes to-device messages earlier in /sync (#10124) ... I hope this will improve https://github.com/matrix-org/synapse/issues/9564.	2021-06-07 09:19:06 +01:00
Erik Johnston	a0cd8ae8cb	Don't try and backfill the same room in parallel. (#10116) ... If backfilling is slow then the client may time out and retry, causing Synapse to start a new `/backfill` before the existing backfill has finished, duplicating work.	2021-06-04 10:47:58 +01:00
Erik Johnston	c96ab31dff	Limit number of events in a replication request (#10118) ... Fixes #9956.	2021-06-04 10:35:47 +01:00
Richard van der Hoff	d8be7d493d	Enable Prometheus metrics for the jaeger client library (#10112)	2021-06-04 09:25:33 +01:00
Richard van der Hoff	9eea4646be	Add OpenTracing for database activity. (#10113) ... This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).	2021-06-03 16:31:56 +01:00
Richard van der Hoff	1d143074c5	Improve opentracing annotations for Notifier (#10111) ... The existing tracing reports an error each time there is a timeout, which isn't really representative. Additionally, we log things about the way `wait_for_events` works (eg, the result of the callback) to the *parent* span, which is confusing.	2021-06-03 16:01:30 +01:00
Travis Ralston	5325f0308c	r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224) ... [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. This is progress towards https://github.com/matrix-org/synapse/issues/8334	2021-06-03 13:50:49 +01:00
Patrick Cloke	d7a646abca	Merge branch 'master' into develop	2021-06-03 08:43:38 -04:00
Patrick Cloke	57c01dca29	1.35.1	2021-06-03 08:18:22 -04:00
Patrick Cloke	36a7ff0c86	Do not show invite-only rooms in spaces summary (unless joined/invited). (#10109)	2021-06-03 08:04:01 -04:00
Dirk Klimpel	0284d2a297	Add new admin APIs to remove media by media ID from quarantine. (#10044) ... Related to: #6681, #5956, #10040 Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-06-02 18:50:35 +01:00
Erik Johnston	fc3d2dc269	Rewrite the KeyRing (#10035)	2021-06-02 16:37:59 +01:00
Patrick Cloke	3cf6b34b4e	Do not show invite-only rooms in spaces summary (unless joined/invited). (#10109)	2021-06-02 11:31:41 -04:00
Erik Johnston	4deaebfe00	Make /sync do less state res (#10102)	2021-06-02 15:48:17 +01:00
Andrew Morgan	3ff6fe2851	Merge branch 'master' into develop	2021-06-01 13:47:27 +01:00
Andrew Morgan	a8372ad591	1.35.0	2021-06-01 13:23:55 +01:00
Richard van der Hoff	b4b2fd2ece	add a cache to have_seen_event (#9953) ... Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.	2021-06-01 12:04:47 +01:00
Brad Murray	10e6d2abce	Fix opentracing inject to use the SpanContext, not the Span (#10074) ... Signed-off-by: Brad Murray brad@beeper.com	2021-06-01 08:40:26 +01:00
Erik Johnston	258a9a9e8b	1.35.0rc3	2021-05-28 17:06:05 +01:00
Erik Johnston	6b6c6a02db	Merge remote-tracking branch 'origin/release-v1.35.0' into release-v1.35	2021-05-28 17:03:14 +01:00
Brendan Abolivier	9408b86f5c	Limit the number of events sent over replication when persisting events. (#10082)	2021-05-28 16:29:09 +01:00
Erik Johnston	1641c5c707	Log method and path when dropping request due to size limit (#10091)	2021-05-28 16:29:09 +01:00
Erik Johnston	84cf3e47a0	Allow response of /send_join to be larger. (#10093) ... Fixes #10087.	2021-05-28 16:28:01 +01:00
Richard van der Hoff	ed53bf314f	Set opentracing priority before setting other tags (#10092) ... ... because tags on spans which aren't being sampled get thrown away.	2021-05-28 16:14:08 +01:00
Erik Johnston	3f96dbbda7	Log method and path when dropping request due to size limit (#10091)	2021-05-28 15:57:53 +01:00
Eric Eastwood	ac3e02d089	Add parse_strings_from_args to get prev_events array (#10048) ... Split out from https://github.com/matrix-org/synapse/pull/9247 Strings: - `parse_string` - `parse_string_from_args` - `parse_strings_from_args` For comparison with ints: - `parse_integer` - `parse_integer_from_args` Previous discussions: - https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687 - https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791 Signed-off-by: Eric Eastwood <erice@element.io>	2021-05-28 14:19:06 +01:00
Erik Johnston	5eed6348ce	Move some more endpoints off master (#10084)	2021-05-27 22:45:43 +01:00
Callum Brown	8fb9af570f	Make reason and score optional for report_event (#10077) ... Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414 See #8551 Signed-off-by: Callum Brown <callum@calcuode.com>	2021-05-27 18:42:23 +01:00
Brendan Abolivier	f828a70be3	Limit the number of events sent over replication when persisting events. (#10082)	2021-05-27 17:10:58 +01:00
Erik Johnston	8e132fe64e	Synapse 1.35.0rc2 (2021-05-27) ... ============================== Bugfixes -------- - Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](https://github.com/matrix-org/synapse/issues/10079)) -----BEGIN PGP SIGNATURE----- iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmCvpZMQHGVyaWtAbWF0 cml4Lm9yZwAKCRClQuTtGw+sCdPwCACQIlIWd6eIoXLUc+wDcrd+k5xL376EdYah x7ABswiYSm+9C4xr58gJD3xc6eiD2PCIWdZN0rsQDLIOfSXW6x1lyKD+Ds0HySok MaVpsoxbb9o/Zf9qtXF2bLSArZUQwfoNaA45NLgNzfUIijf1e+bd2wNEgHlRSoMz m10GggOFU0Ds/CCYZpxZw/eXDbLWL7eaHR30vw/jQ1cEsV+S4ucnUmLHFCF7YCyI Np80pnywH5cYKerecldFWenL4YZJswVbx+AW9e3lBzq5jOrRZkmLkaHg10mCR2f6 CV03ie65Ce+7x5UU6v6nHA0DYUTQGIjlJBtyCN3tFglUduQ8Gpu0 =Mfsf -----END PGP SIGNATURE----- Merge tag 'v1.35.0rc2' into develop Synapse 1.35.0rc2 (2021-05-27) ============================== Bugfixes -------- - Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](https://github.com/matrix-org/synapse/issues/10079))	2021-05-27 14:59:46 +01:00
Erik Johnston	b1bc26a909	1.35.0rc2	2021-05-27 14:46:24 +01:00
Erik Johnston	78b5102ae7	Fix up BatchingQueue (#10078) ... Fixes #10068	2021-05-27 14:32:31 +01:00
Patrick Cloke	8e15c92c2f	Pass the origin when calculating the spaces summary over GET. (#10079) ... Fixes a bug due to conflicting PRs which were merged. (One added a new caller to a method, the other added a new parameter to the same method.)	2021-05-27 08:52:28 -04:00
Richard van der Hoff	fe5dad46b0	Remove redundant code to reload tls cert (#10054) ... we don't need to reload the tls cert if we don't have any tls listeners. Follow-up to #9280.	2021-05-27 10:34:24 +01:00
Richard van der Hoff	224f2f949b	Combine LruCache.invalidate and invalidate_many (#9973) ... * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog	2021-05-27 10:33:56 +01:00
Patrick Cloke	f42e4c4eb9	Remove the experimental spaces enabled flag. (#10063) ... In lieu of just always enabling the unstable spaces endpoint and unstable room version.	2021-05-26 14:35:16 -04:00
Dirk Klimpel	65e6c64d83	Add an admin API for unprotecting local media from quarantine (#10040) ... Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-05-26 11:19:47 +01:00
Erik Johnston	557635f69a	1.35.0rc1	2021-05-25 11:00:13 +01:00
Patrick Cloke	7adcb20fc0	Add missing type hints to synapse.util (#9982)	2021-05-24 15:32:01 -04:00
Jerin J Titus	057ce7b754	Remove tls_fingerprints option (#9280) ... Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>	2021-05-24 17:43:30 +01:00
Richard van der Hoff	c0df6bae06	Remove keylen from LruCache. (#9993) ... `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.	2021-05-24 14:02:01 +01:00
Patrick Cloke	316f89e87f	Enable experimental spaces by default. (#10011) ... The previous spaces_enabled flag now defaults to true and is exposed in the sample config.	2021-05-24 08:57:14 -04:00
Eric Eastwood	5f1198a67e	Fix get_state_ids_for_event return type typo to match what the function actually does (#10050) ... It looks like a typo copy/paste from `get_state_for_event` above.	2021-05-24 10:43:33 +01:00
Erik Johnston	3e831f24ff	Don't hammer the database for destination retry timings every ~5mins (#10036)	2021-05-21 17:57:08 +01:00
Michael Telatynski	e8ac9ac8ca	Fix /upload 500'ing when presented a very large image (#10029) ... * Fix /upload 500'ing when presented a very large image Catch DecompressionBombError and re-raise as ThumbnailErrors * Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml to get it to bomb out quicker, to load less into memory in the case of super large images * Add changelog entry for 10029	2021-05-21 18:31:59 +02:00
Patrick Cloke	c5413d0e9e	Remove unused properties from the SpaceSummaryHandler. (#10038)	2021-05-21 12:02:01 -04:00
Marek Matys	6a8643ff3d	Fixed removal of new presence stream states (#10014) ... Fixes: https://github.com/matrix-org/synapse/issues/9962 This is a fix for above problem. I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts. Signed-off-by: Marek Matys <themarcq@gmail.com>	2021-05-21 12:02:06 +01:00
Erik Johnston	7958eadcd1	Add a batching queue implementation. (#10017)	2021-05-21 11:20:51 +01:00
Erik Johnston	1c6a19002c	Add Keyring.verify_events_for_server and reduce memory usage (#10018) ... Also add support for giving a callback to generate the JSON object to verify. This should reduce memory usage, as we no longer have the event in memory in dict form (which has a large memory footprint) for extend periods of time.	2021-05-20 16:25:11 +01:00
Erik Johnston	64887f06fc	Use ijson to parse the response to /send_join, reducing memory usage. (#9958) ... Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.	2021-05-20 16:11:48 +01:00
Patrick Cloke	551d2c3f4b	Allow a user who could join a restricted room to see it in spaces summary. (#9922) ... This finishes up the experimental implementation of MSC3083 by showing the restricted rooms in the spaces summary (from MSC2946).	2021-05-20 11:10:36 -04:00
Patrick Cloke	ac6bfcd52f	Refactor checking restricted join rules (#10007) ... To be more consistent with similar code. The check now automatically raises an AuthError instead of passing back a boolean. It also absorbs some shared logic between callers.	2021-05-18 12:17:04 -04:00
Andrew Morgan	4d6e5a5e99	Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823)	2021-05-18 14:13:45 +01:00
Patrick Cloke	206a7b5f12	Fix the allowed range of valid ordering characters for spaces. (#10002) ... \x7F was meant to be \0x7E (~) this was originally incorrect in MSC1772.	2021-05-17 09:59:17 -04:00
Patrick Cloke	9752849e2b	Clarify comments in the space summary handler. (#9974)	2021-05-17 09:01:19 -04:00
Erik Johnston	653fe2f3cd	Merge branch 'master' into develop	2021-05-17 12:14:58 +01:00
Erik Johnston	afb6dcf806	1.34.0	2021-05-17 11:34:39 +01:00
Brendan Abolivier	41ac128fd3	Split multiplart email sending into a dedicated handler (#9977) ... Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2021-05-17 12:33:38 +02:00
Richard van der Hoff	5090f26b63	Minor @cachedList enhancements (#9975) ... - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.	2021-05-14 11:12:36 +01:00
Dan Callahan	52ed9655ed	Remove unnecessary SystemRandom from SQLBaseStore (#9987) ... It's not obvious that instances of SQLBaseStore each need their own instances of random.SystemRandom(); let's just use random directly. Introduced by 52839886d6 Signed-off-by: Dan Callahan <danc@element.io>	2021-05-14 10:59:10 +01:00
Dan Callahan	ebdef256b3	Remove superfluous call to bool() (#9986) ... Our strtobool already returns a bool, so no need to re-cast here Signed-off-by: Dan Callahan <danc@element.io>	2021-05-14 10:58:57 +01:00
Dan Callahan	bd918d874f	Simplify exception handling in is_ascii. (#9985) ... We can get away with just catching UnicodeError here. ⋮ +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError ⋮ https://docs.python.org/3/library/exceptions.html#exception-hierarchy Signed-off-by: Dan Callahan <danc@element.io>	2021-05-14 10:58:52 +01:00
Dan Callahan	498084228b	Use Python's secrets module instead of random (#9984) ... Functionally identical, but more obviously cryptographically secure. ...Explicit is better than implicit? Avoids needing to know that SystemRandom() implies a CSPRNG, and complies with the big scary red box on the documentation for random: > Warning: > The pseudo-random generators of this module should not be used for > security purposes. For security or cryptographic uses, see the > secrets module. https://docs.python.org/3/library/random.html Signed-off-by: Dan Callahan <danc@element.io>	2021-05-14 10:58:46 +01:00
Richard van der Hoff	c14f99be46	Support enabling opentracing by user (#9978) ... Add a config option which allows enabling opentracing by user id, eg for debugging requests made by a test user.	2021-05-14 10:51:08 +01:00
Brendan Abolivier	47806b0869	1.34.0rc1	2021-05-12 16:59:46 +01:00
Brendan Abolivier	a683028d81	Correctly ratelimit invites when creating a room (#9968) ... * Correctly ratelimit invites when creating a room Also allow ratelimiting for more than one action at a time.	2021-05-12 16:05:28 +02:00
Richard van der Hoff	7562d887e1	Change the format of access tokens away from macaroons (#5588)	2021-05-12 15:04:51 +01:00
Erik Johnston	affaffb0ab	Run cache_joined_hosts_for_event in background (#9951)	2021-05-12 13:17:11 +01:00
Patrick Cloke	27c375f812	Sort child events according to MSC1772 for the spaces summary API. (#9954) ... This should help ensure that equivalent results are achieved between homeservers querying for the summary of a space. This implements modified MSC1772 rules, according to MSC2946. The different is that the origin_server_ts of the m.room.create event is not used as a tie-breaker since this might not be known if the homeserver is not part of the room.	2021-05-11 12:57:39 -04:00
Patrick Cloke	f4833e0c06	Support fetching the spaces summary via GET over federation. (#9947) ... Per changes in MSC2946, the C-S and S-S APIs for spaces summary should use GET requests. Until this is stable, the POST endpoints still exist. This does not switch federation requests to use the GET version yet since it is newly added and already deployed servers might not support it. When switching to the stable endpoint we should switch to GET requests.	2021-05-11 12:21:43 -04:00
Patrick Cloke	28c6841102	Send the m.room.create stripped event with invites (support MSC1772). (#9966) ... MSC1772 specifies the m.room.create event should be sent as part of the invite_state. This was done optionally behind an experimental flag, but is now done by default due to MSC1772 being approved.	2021-05-11 10:58:58 -04:00
Brendan Abolivier	652a6b094d	Merge branch 'master' into develop	2021-05-11 14:15:30 +01:00
Aaron Raimist	dc6366a9bd	Add config option to hide device names over federation (#9945) ... Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers. Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-05-11 14:03:23 +01:00
Brendan Abolivier	86fb71431c	1.33.2	2021-05-11 14:01:32 +01:00
Richard van der Hoff	b378d98c8f	Add debug logging for issue #9533 (#9959) ... Hopefully this will help us track down where to-device messages are getting lost/delayed.	2021-05-11 11:04:03 +01:00
Richard van der Hoff	7967b36efe	Fix m.room_key_request to-device messages (#9961) ... fixes #9960	2021-05-11 11:02:56 +01:00
Richard van der Hoff	03318a766c	Merge pull request from GHSA-x345-32rc-8h85 ... * tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>	2021-05-11 11:47:23 +02:00
Erik Johnston	2b2985b5cf	Improve performance of backfilling in large rooms. (#9935) ... We were pulling the full auth chain for the room out of the DB each time we backfilled, which can be *huge* for large rooms and is totally unnecessary.	2021-05-10 13:29:02 +01:00
Erik Johnston	6c84778549	Always cache 'event_to_prev_state_group' (#9950) ... Fixes regression in send PDU times introduced in #9905.	2021-05-07 14:54:09 +01:00
Erik Johnston	b65ecaff9b	Merge remote-tracking branch 'origin/release-v1.33.2' into develop	2021-05-07 13:27:19 +01:00
Erik Johnston	4df26abf28	Unpin attrs dep after new version has been released (#9946) ... c.f. #9936	2021-05-07 12:57:21 +01:00
Richard van der Hoff	25f43faa70	Reorganise the database schema directories (#9932) ... The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.	2021-05-07 10:22:05 +01:00
Erik Johnston	8771b1337d	Export jemalloc stats to prometheus when used (#9882)	2021-05-06 15:54:07 +01:00
Erik Johnston	eba431c539	Revert "Leave out optional keys from /sync (#9919)" (#9940) ... This reverts commit e9eb3549d3.	2021-05-06 15:06:35 +01:00
Erik Johnston	a8803e2b6e	Merge remote-tracking branch 'origin/master' into develop	2021-05-06 14:36:59 +01:00
Erik Johnston	ac88aca7f7	1.33.1	2021-05-06 14:06:38 +01:00
Erik Johnston	24f07a83e6	Pin attrs to <21.1.0 (#9937) ... Fixes #9936	2021-05-06 14:06:06 +01:00
Patrick Cloke	70f0ffd2fc	Follow-up to #9915 to correct the identifier for room types.	2021-05-05 16:31:16 -04:00
Patrick Cloke	d783880083	Include the time of the create event in Spaces Summary. (#9928) ... This is an update based on changes to MSC2946. The origin_server_ts of the m.room.create event is copied into the creation_ts field for each room returned from the spaces summary.	2021-05-05 13:33:05 -04:00
Erik Johnston	37623e3382	Increase perf of handling presence when joining large rooms. (#9916)	2021-05-05 17:27:05 +01:00
Patrick Cloke	e2a443550e	Support stable MSC1772 spaces identifiers. (#9915) ... Support both the unstable and stable identifiers. A future release will disable the unstable identifiers.	2021-05-05 11:56:51 -04:00
Erik Johnston	ef889c98a6	Optionally track memory usage of each LruCache (#9881) ... This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.	2021-05-05 16:54:36 +01:00
Erik Johnston	1fb9a2d0bf	Limit how often GC happens by time. (#9902) ... Synapse can be quite memory intensive, and unless care is taken to tune the GC thresholds it can end up thrashing, causing noticable performance problems for large servers. We fix this by limiting how often we GC a given generation, regardless of current counts/thresholds. This does not help with the reverse problem where the thresholds are set too high, but that should only happen in situations where they've been manually configured. Adds a `gc_min_seconds_between` config option to override the defaults. Fixes #9890.	2021-05-05 16:53:45 +01:00
Erik Johnston	de8f0a03a3	Don't set the external cache if its been done recently (#9905)	2021-05-05 16:53:22 +01:00
Erik Johnston	d0aee697ac	Use get_current_users_in_room from store and not StateHandler (#9910)	2021-05-05 16:49:34 +01:00
DeepBlueV7.X	e9eb3549d3	Leave out optional keys from /sync (#9919) ... This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.) Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>	2021-05-05 14:37:56 +01:00
Brendan Abolivier	a61b13c0a1	Merge branch 'master' into develop	2021-05-05 14:25:49 +01:00
Brendan Abolivier	0644ac0989	1.33.0	2021-05-05 14:15:54 +01:00
Erik Johnston	e3bc4617fc	Time external cache response time (#9904)	2021-05-04 15:14:22 +01:00
Andrew Morgan	b85821aca2	Add port parameter to the sample config for psycopg2 args (#9911) ... Adds the `port` option with the default value to the sample config file.	2021-05-04 13:28:59 +01:00
Andrew Morgan	4d624f467a	Synapse 1.33.0rc2 (2021-04-29) ... ============================== Bugfixes -------- - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900)) -----BEGIN PGP SIGNATURE----- iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCKtVATHGFuZHJld0Bh bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ok1D/40l93bxGBFtSNEHlqem7lZRLY2On9j AUnzIMqtJTLUgXf67vKIYqtTWkRjTdLIhIuINwhRmB0K23vcRrONnnyRLdhUC4Hl v6qLlPBHzupAJQPRcy3FGIVwsloAZ3JqVhV8AG0gXBRA0Dc34qYXh1DlA8EQ8mW/ l2umQIJI562xLNss7c5tpbCQxi7mq6OmTZ+Nwh45VRDavKEQQzSb9sPfs5w8IGLA ViYosUoXoXEWus/j60E38LSA/v1YcEE+Uyf5+LieV4b2c305xKj8gIbZqjLHR2K5 U9IrUp4QlYpPiebfGn7EmiPnOnlWdR9soIVZMzJ/m29tlyRSL5MBagtWA2fFlXo6 PywW1R/VsnLTdOuWY/t+et11iI4lVa7VSpywEcbMKGgmGKexDnfB10K7jHw0YICq 0pyLgy8JWyGUk/xpH2W1pvak8vyyN/zI/H09bDgGu721Qr/EPiHWF6RADNASpUVW w10SnO1udVc8ajugNwpX2oOQA7i5uiEuTdyFUHYfZ/aSpaCvdxM+rL3v0K955bNc /AGjSbs9ITe+JJQDiAJsWCYc+XLkR4h5KHVaZm2XlusbYb4iobK1DflkWXnQFjsi rlTcUH3sIFot3i9mjBBvXdvmzuI9qyx81/Sh8gRunPYT5TPhwaXKfyMBKE93/kif 4bKoHIXVcJvAqA== =sboq -----END PGP SIGNATURE----- Merge tag 'v1.33.0rc2' into develop Synapse 1.33.0rc2 (2021-04-29) ============================== Bugfixes -------- - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))	2021-04-29 14:35:14 +01:00
Patrick Cloke	bb4b11846f	Add missing type hints to handlers and fix a Spam Checker type hint. (#9896) ... The user_may_create_room_alias method on spam checkers declared the room_alias parameter as a str when in reality it is passed a RoomAlias object.	2021-04-29 07:17:28 -04:00
Andrew Morgan	e9444cc74d	1.33.0rc2	2021-04-29 11:45:37 +01:00
ThibF	0085dc5abc	Delete room endpoint (#9889) ... Support the delete of a room through DELETE request and mark previous request as deprecated through documentation. Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>	2021-04-29 10:31:45 +01:00
Erik Johnston	802560211a	Merge remote-tracking branch 'origin/release-v1.33.0' into develop	2021-04-28 14:43:10 +01:00
Erik Johnston	e4ab8676b4	Fix tight loop handling presence replication. (#9900) ... Only affects workers. Introduced in #9819. Fixes #9899.	2021-04-28 14:42:50 +01:00
Patrick Cloke	10a08ab88a	Use the parent's logging context name for runWithConnection. (#9895) ... This fixes a regression where the logging context for runWithConnection was reported as runWithConnection instead of the connection name, e.g. "POST-XYZ".	2021-04-28 07:44:52 -04:00
Andrew Morgan	fa6679e794	Synapse 1.33.0rc1 (2021-04-28) ... ============================== Features -------- - Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814)) - Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850)) - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832)) Bugfixes -------- - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726)) - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788)) - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802)) - Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833)) - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867)) - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868)) Improved Documentation ---------------------- - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801)) Internal Changes ---------------- - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162)) - Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786)) - Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796)) - Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815)) - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816)) - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817)) - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821)) - Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838)) - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845)) - Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855)) - Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856)) - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858)) - Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871)) - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874)) - Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875)) - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876)) - Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878)) - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879)) - Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887)) -----BEGIN PGP SIGNATURE----- iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCJQmgTHGFuZHJld0Bh bW9yZ2FuLnh5egAKCRCIhIgNLv5f9EKWD/9MZG4ngjECrk3QMBlYqaipGl/l7wcl 8vBOTxiEqIgVJxHLCRFBbtFOItdxB4YkrorIo2fqshkk+lv4CRtTD9jEKHRvS4T6 7p1icRdRTv2K94OkJ8R9jMFlmywZFU87oHPfI2xUYg6hvOKrR+RwPvIjA7c24UZt 6MJqDhgGDlZD7/hQdQof9O4oOJzzIgLJPk6o8E42y6c0bLlPPKgH3sh0vlenDLfE 15thRCOeiP237YYvXSdbr7G3PI66Efhq4BwQowSrgFg+B0BR68l4747iSIeWmLQJ Ow6QLLFPCOhgAPC4amp9PdaaV/9NYiBcvNxlOvyQAVl/+ioEiATNHnzdNtDouCQo nDdSHw0Mt9D3i+rxpu5Pf0gZN9dXRGiczqnq5QKL8+EvT/4FLYeluEeduuy3rE+G o5OTCd3EajxynzjftuopeysNAw6zeDpbulZoTCeCumxBL2+wAod2PsyY5Ei9Gapn iJvExNOJX4OlkFc67jO2CK8o3sUTNEDeIDWCQ0fVKAwIt7T45ebTA/UzDNWg7YzN EyUp+3NZcUZBskgMB5hpcijPJoXNYzWZPews73vMPV7AfQRxDzU5xrM9AdbadBDd Idv6wT9ssDUA+M0aKnafvoSJZ+qE85mi2x0rRsueZNd9uO9/QHIXKIb+4PvVpj3C BJnZf34m568AGQ== =smAk -----END PGP SIGNATURE----- Merge tag 'v1.33.0rc1' into develop Synapse 1.33.0rc1 (2021-04-28) ============================== Features -------- - Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814)) - Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850)) - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832)) Bugfixes -------- - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726)) - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788)) - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802)) - Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833)) - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867)) - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868)) Improved Documentation ---------------------- - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801)) Internal Changes ---------------- - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162)) - Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786)) - Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796)) - Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815)) - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816)) - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817)) - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821)) - Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838)) - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845)) - Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855)) - Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856)) - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858)) - Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871)) - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874)) - Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875)) - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876)) - Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878)) - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879)) - Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))	2021-04-28 12:12:29 +01:00
Erik Johnston	391bfe9a7b	Reduce memory footprint of caches (#9886)	2021-04-28 11:59:28 +01:00
Andrew Morgan	787de3190f	1.33.0rc1	2021-04-28 11:43:33 +01:00
Andrew Morgan	4e0fd35bc9	Revert "Experimental Federation Speedup (#9702)" ... This reverts commit 05e8c70c05.	2021-04-28 11:38:33 +01:00
Erik Johnston	dd2d32dcdb	Add type hints to presence handler (#9885)	2021-04-28 11:07:47 +01:00
Andrew Morgan	fe604a022a	Remove various bits of compatibility code for Python <3.6 (#9879) ... I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.	2021-04-27 13:13:07 +01:00
Patrick Cloke	1350b053da	Pass errors back to the client when trying multiple federation destinations. (#9868) ... This ensures that something like an auth error (403) will be returned to the requester instead of attempting to try more servers, which will likely result in the same error, and then passing back a generic 400 error.	2021-04-27 07:30:34 -04:00
Erik Johnston	0ffa5fb935	Use current state table for presence.get_interested_remotes (#9887) ... This should be a lot quicker than asking the state handler.	2021-04-27 10:09:41 +01:00
Richard van der Hoff	3ff2251754	Improved validation for received requests (#9817) ... * Simplify `start_listening` callpath * Correctly check the size of uploaded files	2021-04-23 19:20:44 +01:00
Andrew Morgan	695b73c861	Allow OIDC cookies to work on non-root public baseurls (#9726) ... Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574. As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.	2021-04-23 18:22:47 +01:00
Richard van der Hoff	59d24c5bef	pass a reactor into SynapseSite (#9874)	2021-04-23 17:06:47 +01:00
Patrick Cloke	e83627926f	Add type hints to auth and auth_blocking. (#9876)	2021-04-23 12:02:16 -04:00
Erik Johnston	a15c003e5b	Make DomainSpecificString an attrs class (#9875)	2021-04-23 15:46:29 +01:00
Erik Johnston	9d25a0ae65	Split presence out of master (#9820)	2021-04-23 12:21:55 +01:00
Patrick Cloke	d924827da1	Check for space membership during a remote join of a restricted room (#9814) ... When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.	2021-04-23 07:05:51 -04:00
Erik Johnston	3853a7edfc	Only store data in caches, not "smart" objects (#9845)	2021-04-23 11:47:07 +01:00
Richard van der Hoff	51a20914a8	Limit the size of HTTP responses read over federation. (#9833)	2021-04-23 11:08:41 +01:00
Erik Johnston	177dae2704	Limit length of accepted email addresses (#9855)	2021-04-22 17:49:11 +01:00
Richard van der Hoff	69018acbd2	Clear the resync bit after resyncing device lists (#9867) ... Fixes #9866.	2021-04-22 16:53:24 +01:00
Richard van der Hoff	294c675033	Remove synapse.types.Collection (#9856) ... This is no longer required, since we have dropped support for Python 3.5.	2021-04-22 16:43:50 +01:00
Andrew Morgan	3186324260	Merge branch 'master' into develop	2021-04-22 11:23:56 +01:00
Andrew Morgan	55159c48e3	1.32.2	2021-04-21 18:45:39 +01:00
Richard van der Hoff	d9bd62f9d1	Make LoggingContext's name optional (#9857) ... Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55	2021-04-21 16:39:34 +01:00
Andrew Morgan	4b2217ace2	Merge branch 'master' into develop	2021-04-21 14:55:06 +01:00
Andrew Morgan	a745531c10	1.32.1	2021-04-21 14:01:12 +01:00
Richard van der Hoff	5d281c10dd	Stop BackgroundProcessLoggingContext making new prometheus timeseries (#9854) ... This undoes part of b076bc276e.	2021-04-21 10:03:31 +01:00
Patrick Cloke	683d6f75af	Rename handler and config modules which end in handler/config. (#9816)	2021-04-20 14:55:20 -04:00
Andrew Morgan	6982db9651	Merge branch 'master' into develop	2021-04-20 14:55:16 +01:00
Andrew Morgan	e031c7e0cc	1.32.0	2021-04-20 14:31:27 +01:00
Patrick Cloke	b076bc276e	Always use the name as the log ID. (#9829) ... As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).	2021-04-20 14:19:00 +01:00
Erik Johnston	de0d088adc	Add presence federation stream (#9819)	2021-04-20 14:11:24 +01:00
Erik Johnston	db70435de7	Fix bug where we sent remote presence states to remote servers (#9850)	2021-04-20 13:37:54 +01:00
Jonathan de Jong	495b214f4f	Fix (final) Bugbear violations (#9838)	2021-04-20 11:50:49 +01:00
Andrew Morgan	71f0623de9	Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) ... This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).	2021-04-19 19:16:34 +01:00
Denis Kasak	e694a598f8	Sanity check identity server passed to bind/unbind. (#9802) ... Signed-off-by: Denis Kasak <dkasak@termina.org.uk>	2021-04-19 17:21:46 +01:00
Erik Johnston	2b7dd21655	Don't send normal presence updates over federation replication stream (#9828)	2021-04-19 10:50:49 +01:00
Andrew Morgan	c571736c6c	User directory: use calculated room membership state instead (#9821) ... Fixes: #9797. Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.	2021-04-16 18:17:18 +01:00
Erik Johnston	601b893352	Small speed up joining large remote rooms (#9825) ... There are a couple of points in `persist_events` where we are doing a query per event in series, which we can replace.	2021-04-16 14:44:55 +01:00
Richard van der Hoff	5a153772c1	remove HomeServer.get_config (#9815) ... Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.	2021-04-14 19:09:08 +01:00
Patrick Cloke	936e69825a	Separate creating an event context from persisting it in the federation handler (#9800) ... This refactoring allows adding logic that uses the event context before persisting it.	2021-04-14 12:35:28 -04:00
Patrick Cloke	e8816c6ace	Revert "Check for space membership during a remote join of a restricted room. (#9763)" ... This reverts commit cc51aaaa7a. The PR was prematurely merged and not yet approved.	2021-04-14 12:33:37 -04:00
Patrick Cloke	cc51aaaa7a	Check for space membership during a remote join of a restricted room. (#9763) ... When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.	2021-04-14 12:32:20 -04:00
Jonathan de Jong	05e8c70c05	Experimental Federation Speedup (#9702) ... This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max). Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>	2021-04-14 17:19:02 +01:00
Erik Johnston	00a6db9676	Move some replication processing out of generic_worker (#9796) ... Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2021-04-14 17:06:06 +01:00
rkfg	c9a2b5d402	More robust handling of the Content-Type header for thumbnail generation (#9788) ... Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>	2021-04-14 16:30:59 +01:00
Jonathan de Jong	4b965c862d	Remove redundant "coding: utf-8" lines (#9786) ... Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	2021-04-14 15:34:27 +01:00
Andrew Morgan	3efd98aa1c	1.32.0rc1	2021-04-13 14:23:43 +01:00
Dan Callahan	1d5f0e3529	Bump black configuration to target py36 (#9781) ... Signed-off-by: Dan Callahan <danc@element.io>	2021-04-13 10:41:34 +01:00
Dirk Klimpel	1fc97ee876	Add an admin API to manage ratelimit for a specific user (#9648)	2021-04-13 10:26:37 +01:00
Will Hunt	e300ef64b1	Require AppserviceRegistrationType (#9548) ... This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.	2021-04-12 15:13:55 +01:00
Richard van der Hoff	f946450184	Fix duplicate logging of exceptions in transaction processing (#9780) ... There's no point logging this twice.	2021-04-09 18:12:15 +01:00
Dirk Klimpel	48a1f4db31	Remove old admin API GET /_synapse/admin/v1/users/<user_id> (#9401) ... Related: #8334 Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25) `GET /_synapse/admin/v1/users/<user_id>` has no - unit tests - documentation API in v2 is available (#5925 - 12/2019, v1.7.0). API is misleading. It expects `user_id` and returns a list of all users. Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-04-09 09:44:40 +01:00
Jonathan de Jong	2ca4e349e9	Bugbear: Add Mutable Parameter fixes (#9682) ... Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>	2021-04-08 22:38:54 +01:00
Richard van der Hoff	9e167d9c53	Merge remote-tracking branch 'origin/develop' into rav/drop_py35	2021-04-08 18:30:38 +01:00
Richard van der Hoff	24c58ebfc9	remove unused param on make_tuple_comparison_clause	2021-04-08 18:29:57 +01:00
Erik Johnston	be0e722fe1	Merge branch 'erikj/fix_stalled_catchup' into develop	2021-04-08 18:05:35 +01:00
Erik Johnston	3a569fb200	Fix sharded federation sender sometimes using 100% CPU. ... We pull all destinations requiring catchup from the DB in batches. However, if all those destinations get filtered out (due to the federation sender being sharded), then the `last_processed` destination doesn't get updated, and we keep requesting the same set repeatedly.	2021-04-08 17:34:07 +01:00
Richard van der Hoff	3ada9b4264	Drop support for sqlite<3.22 as well	2021-04-08 16:42:32 +01:00
Richard van der Hoff	abade34633	Require py36 and Postgres 9.6	2021-04-08 16:42:32 +01:00
Patrick Cloke	452991527a	MSC3083: Check for space membership during a local join of restricted rooms. (#9735) ... When joining a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.	2021-04-08 08:28:32 -04:00
Patrick Cloke	48d44ab142	Record more information into structured logs. (#9654) ... Records additional request information into the structured logs, e.g. the requester, IP address, etc.	2021-04-08 08:01:14 -04:00
Andrew Morgan	0d87c6bd12	Don't report anything from GaugeBucketCollector metrics until data is present (#8926) ... This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.	2021-04-06 16:32:04 +01:00
Andrew Morgan	04819239ba	Add a Synapse Module for configuring presence update routing (#9491) ... At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.	2021-04-06 14:38:30 +01:00
Patrick Cloke	44bb881096	Add type hints to expiring cache. (#9730)	2021-04-06 08:58:18 -04:00
Richard van der Hoff	0ef321ff3b	Remove outdated constraint on remote_media_cache_thumbnails (#9725) ... The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key` constraint is superceded by `remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds `thumbnail_method` to the unique key). PR #7124 made an attempt to remove the old constraint, but got the name wrong, so it didn't work. Here we update the bg update and rerun it. Fixes #8649.	2021-04-06 13:36:05 +01:00
Erik Johnston	5688a74cf3	Merge branch 'master' into develop	2021-04-06 13:29:29 +01:00
Erik Johnston	1d8863c67d	1.31.0	2021-04-06 13:09:56 +01:00
Patrick Cloke	d959d28730	Add type hints to the federation handler and server. (#9743)	2021-04-06 07:21:57 -04:00
Jonathan de Jong	e2b8a90897	Update mypy configuration: no_implicit_optional = True (#9742)	2021-04-05 09:10:18 -04:00
Erik Johnston	33548f37aa	Improve tracing for to device messages (#9686)	2021-04-01 17:08:21 +01:00
Dirk Klimpel	bb0fe02a52	Add order_by to list user admin API (#9691)	2021-04-01 11:28:53 +01:00
Patrick Cloke	35c5ef2d24	Add an experimental room version to support restricted join rules. (#9717) ... Per MSC3083.	2021-03-31 16:39:08 -04:00
Denis Kasak	5ff8eb97c6	Make sample config allowed_local_3pids regex stricter. (#9719) ... The regex should be terminated so that subdomain matches of another domain are not accepted. Just ensuring that someone doesn't shoot themselves in the foot by copying our example. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>	2021-03-31 12:27:20 +00:00
Cristina	670564446c	Deprecate imp (#9718) ... Fixes #9642. Signed-off-by: Cristina Muñoz <hi@xmunoz.com>	2021-03-31 12:04:27 +01:00
Richard van der Hoff	4dabcf026e	Include m.room.create in invite_room_state for Spaces (#9710)	2021-03-30 14:03:17 +01:00
Richard van der Hoff	f02663c4dd	Replace room_invite_state_types with room_prejoin_state (#9700) ... `room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.	2021-03-30 12:12:44 +01:00
Erik Johnston	963f4309fe	Make RateLimiter class check for ratelimit overrides (#9711) ... This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited. We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits. Fixes #9663	2021-03-30 12:06:09 +01:00
Erik Johnston	78e48f61bf	1.31.0rc1	2021-03-30 11:19:21 +01:00
Patrick Cloke	01dd90b0f0	Add type hints to DictionaryCache and TTLCache. (#9442)	2021-03-29 12:15:33 -04:00
Patrick Cloke	da75d2ea1f	Add type hints for the federation sender. (#9681) ... Includes an abstract base class which both the FederationSender and the FederationRemoteSendQueue must implement.	2021-03-29 11:43:20 -04:00
Richard van der Hoff	4bbd535450	Update the OIDC sample config (#9695) ... I've reiterated the advice about using `oidc` to migrate, since I've seen a few people caught by this. I've also removed a couple of the examples as they are duplicating the OIDC documentation, and I think they might be leading people astray.	2021-03-29 15:40:11 +01:00
Jonathan de Jong	fc53a606e4	Fix re.Pattern mypy error on 3.6 (#9703)	2021-03-29 09:40:45 -04:00
Richard van der Hoff	ad8690a26c	Fix the suggested pip incantation for cryptography (#9699) ... If you have the wrong version of `cryptography` installed, synapse suggests: ``` To install run: pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6'' ``` However, the use of ' inside '...' doesn't work, so when you run this, you get an error.	2021-03-29 11:55:33 +01:00
Richard van der Hoff	7c8402ddb8	Suppress CryptographyDeprecationWarning (#9698) ... This warning is somewhat confusing to users, so let's suppress it	2021-03-26 17:33:55 +00:00
Erik Johnston	b5efcb577e	Make it possible to use dmypy (#9692) ... Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.	2021-03-26 16:49:46 +00:00
Erik Johnston	019010964d	Merge branch 'master' into develop	2021-03-26 12:26:58 +00:00
Erik Johnston	c6f8e8086c	1.30.1	2021-03-26 12:03:29 +00:00
Erik Johnston	12d6184713	Explicitly upgrade openssl in docker file and enforce new version of cryptography (#9697)	2021-03-26 12:00:25 +00:00
Andrew Morgan	fae81f2f68	Add a storage method for returning all current presence from all users (#9650) ... Split off from https://github.com/matrix-org/synapse/pull/9491 Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: b588f16e39/synapse/handlers/presence.py (L1131-L1133) Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.	2021-03-25 10:34:23 +00:00
Erik Johnston	c602ba8336	Fixed undefined variable error in catchup (#9664) ... Broke in #9640 Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2021-03-24 16:12:47 +00:00
Jonathan de Jong	4c3827f2c1	Enable addtional flake8-bugbear linting checks. (#9659)	2021-03-24 09:34:30 -04:00
Richard van der Hoff	c73cc2c2ad	Spaces summary: call out to other servers (#9653) ... When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data. Fixes: #9447	2021-03-24 12:45:39 +00:00
Patrick Cloke	af387cf52a	Add type hints to misc. files. (#9676)	2021-03-24 06:49:01 -04:00
Patrick Cloke	7e8dc9934e	Add a type hints for service notices to the HomeServer object. (#9675)	2021-03-24 06:48:46 -04:00
Erik Johnston	e550ab17ad	Increase default join burst ratelimiting (#9674) ... It's legitimate behaviour to try and join a bunch of rooms at once.	2021-03-23 14:52:20 +00:00
Jonathan de Jong	0caf2a338e	Fix federation stall on concurrent access errors (#9639)	2021-03-23 13:52:30 +00:00
Richard van der Hoff	4ecba9bd5c	Federation API for Space summary (#9652) ... Builds on the work done in #9643 to add a federation API for space summaries. There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.	2021-03-23 11:51:12 +00:00
Patrick Cloke	b7748d3c00	Import HomeServer from the proper module. (#9665)	2021-03-23 07:12:48 -04:00
Andrew Morgan	5b268997bd	Allow providing credentials to HTTPS_PROXY (#9657) ... Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints.	2021-03-22 17:20:47 +00:00
Ankit Dobhal	d66f9070cd	Fixed code misc. quality issues (#9649) ... - Merge 'isinstance' calls. - Remove unnecessary dict call outside of comprehension. - Use 'sys.exit()' calls.	2021-03-22 11:18:13 -04:00
Erik Johnston	d600d4506b	Merge branch 'master' into develop	2021-03-22 13:36:36 +00:00
Erik Johnston	e2904f720d	1.30.0	2021-03-22 13:15:55 +00:00
Brendan Abolivier	b6ed4f55ac	Incorporate review	2021-03-19 18:19:50 +01:00
Brendan Abolivier	592d6305fd	Merge branch 'develop' into babolivier/msc3026	2021-03-19 16:12:40 +01:00
Brendan Abolivier	0b56481caa	Fix lint	2021-03-19 16:11:08 +01:00
Richard van der Hoff	066068f034	fix mypy	2021-03-19 12:20:11 +00:00
Patrick Cloke	9b0e3009fa	Fix type-hints from bad merge.	2021-03-18 14:40:56 -04:00
Richard van der Hoff	004234f03a	Initial spaces summary API (#9643) ... This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.	2021-03-18 18:24:16 +00:00
Brendan Abolivier	066c703729	Move support for MSC3026 behind an experimental flag	2021-03-18 18:37:19 +01:00
Dirk Klimpel	8dd2ea65a9	Consistently check whether a password may be set for a user. (#9636)	2021-03-18 12:54:08 -04:00
Erik Johnston	dd71eb0f8a	Make federation catchup send last event from any server. (#9640) ... Currently federation catchup will send the last *local* event that we failed to send to the remote. This can cause issues for large rooms where lots of servers have sent events while the remote server was down, as when it comes back up again it'll be flooded with events from various points in the DAG. Instead, let's make it so that all the servers send the most recent events, even if its not theirs. The remote should deduplicate the events, so there shouldn't be much overhead in doing this. Alternatively, the servers could only send local events if they were also extremities and hope that the other server will send the event over, but that is a bit risky.	2021-03-18 15:52:26 +00:00
Brendan Abolivier	405aeb0b2c	Implement MSC3026: busy presence state	2021-03-18 16:34:47 +01:00
Andrew Morgan	7b06f85c0e	Ensure we use a copy of the event content dict before modifying it in serialize_event (#9585) ... This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)) - will raise a 500 if you try to a edit a reply to a message. `serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised. To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.	2021-03-17 16:51:55 +00:00
Patrick Cloke	cc324d53fe	Fix up types for the typing handler. (#9638) ... By splitting this to two separate methods the callers know what methods they can expect on the handler.	2021-03-17 11:30:21 -04:00
Hubert Chathi	73dbce5523	only save remote cross-signing keys if they're different from the current ones (#9634) ... Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2021-03-17 11:04:57 -04:00
Erik Johnston	ad721fc559	Fix bad naming of storage function (#9637) ... We had two functions named `get_forward_extremities_for_room` and `get_forward_extremeties_for_room` that took different paramters. We rename one of them to avoid confusion.	2021-03-17 13:20:08 +00:00
Richard van der Hoff	567f88f835	Prep work for removing outlier from internal_metadata (#9411) ... * Populate `internal_metadata.outlier` based on `events` table Rather than relying on `outlier` being in the `internal_metadata` column, populate it based on the `events.outlier` column. * Move `outlier` out of InternalMetadata._dict Ultimately, this will allow us to stop writing it to the database. For now, we have to grandfather it back in so as to maintain compatibility with older versions of Synapse.	2021-03-17 12:33:18 +00:00
Patrick Cloke	b449af0379	Add type hints to the room member handler. (#9631)	2021-03-17 07:14:39 -04:00
Jonathan de Jong	27d2820c33	Enable flake8-bugbear, but disable most checks. (#9499) ... * Adds B00 to ignored checks. * Fixes remaining issues.	2021-03-16 14:19:27 -04:00
Hubbe	dd5e5dc1d6	Add SSO attribute requirements for OIDC providers (#9609) ... Allows limiting who can login using OIDC via the claims made from the IdP.	2021-03-16 11:46:07 -04:00
Dirk Klimpel	8000cf1315	Return m.change_password.enabled=false if local database is disabled (#9588) ... Instead of if the user does not have a password hash. This allows a SSO user to add a password to their account, but only if the local password database is configured.	2021-03-16 11:44:25 -04:00
Andrew Morgan	d315e96443	1.30.0rc1	2021-03-16 13:45:46 +00:00
Andrew Morgan	847ecdd8fa	Pass SSO IdP information to spam checker's registration function (#9626) ... Fixes https://github.com/matrix-org/synapse/issues/9572 When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time. This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.	2021-03-16 12:41:41 +00:00
Patrick Cloke	1383508f29	Handle an empty cookie as an invalid macaroon. (#9620) ... * Handle an empty cookie as an invalid macaroon. * Newsfragment	2021-03-16 11:29:35 +00:00
Richard van der Hoff	dd69110d95	Add support for stable MSC2858 API (#9617) ... The stable format uses different brand identifiers, so we need to support two identifiers for each IdP.	2021-03-16 11:21:26 +00:00
Richard van der Hoff	5b5bc188cf	Clean up config settings for stats (#9604) ... ... and complain if people try to turn it off.	2021-03-16 10:57:54 +00:00
Andrew Morgan	1b0eaed21f	Prevent bundling aggregations for state events (#9619) ... There's no need to do aggregation bundling for state events. Doing so can cause performance issues.	2021-03-16 10:27:51 +00:00
Richard van der Hoff	1c8a2541da	Fix Internal Server Error on GET /saml2/authn_response (#9623) ... * Fix Internal Server Error on `GET /saml2/authn_response` Seems to have been introduced in #8765 (Synapse 1.24.0) * Fix newsfile	2021-03-16 10:20:20 +00:00
Patrick Cloke	d29b71aa50	Fix remaining mypy issues due to Twisted upgrade. (#9608)	2021-03-15 11:14:39 -04:00
Erik Johnston	026503fa3b	Don't go into federation catch up mode so easily (#9561) ... Federation catch up mode is very inefficient if the number of events that the remote server has missed is small, since handling gaps can be very expensive, c.f. #9492. Instead of going into catch up mode whenever we see an error, we instead do so only if we've backed off from trying the remote for more than an hour (the assumption being that in such a case it is more than a transient failure).	2021-03-15 14:42:40 +00:00
Richard van der Hoff	af2248f8bf	Optimise missing prev_event handling (#9601) ... Background: When we receive incoming federation traffic, and notice that we are missing prev_events from the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events, we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote server for the state at those prev_events, and then we may need to then ask the remote server for any events in that state which we don't already have, as well as the auth events for those missing state events, so that we can auth them. This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the observation that we are currently loading all of those auth events into memory at the start of the operation, but we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're actually going to need, but it doesn't.) The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about 60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache churn. (NB I've already tripled the size of the cache from its default of 10K). Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different (ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting that we clean it up.	2021-03-15 13:51:02 +00:00
Patrick Cloke	55da8df078	Fix additional type hints from Twisted 21.2.0. (#9591)	2021-03-12 11:37:57 -05:00
Richard van der Hoff	1e67bff833	Reject concurrent transactions (#9597) ... If more transactions arrive from an origin while we're still processing the first one, reject them. Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489	2021-03-12 15:14:55 +00:00
Richard van der Hoff	2b328d7e02	Improve logging when processing incoming transactions (#9596) ... Put the room id in the logcontext, to make it easier to understand what's going on.	2021-03-12 15:08:03 +00:00
Richard van der Hoff	464e5da7b2	Add logging for redis connection setup (#9590)	2021-03-11 18:35:09 +00:00
Patrick Cloke	e55bd0e110	Add tests for blacklisting reactor/agent. (#9563)	2021-03-11 09:15:22 -05:00
Dirk Klimpel	70d1b6abff	Re-Activating account when local passwords are disabled (#9587) ... Fixes: #8393	2021-03-11 13:52:32 +00:00
Richard van der Hoff	a7a3790066	Convert Requester to attrs (#9586) ... ... because namedtuples suck Fix up a couple of other annotations to keep mypy happy.	2021-03-10 18:15:56 +00:00
Richard van der Hoff	1107214a1d	Fix the auth provider on the logins metric (#9573) ... We either need to pass the auth provider over the replication api, or make sure we report the auth provider on the worker that received the request. I've gone with the latter.	2021-03-10 18:15:03 +00:00
Patrick Cloke	2a99cc6524	Use the chain cover index in get_auth_chain_ids. (#9576) ... This uses a simplified version of get_chain_cover_difference to calculate auth chain of events.	2021-03-10 09:57:59 -05:00
Patrick Cloke	918f6ed827	Fix a bug in the background task for purging chain cover. (#9583)	2021-03-10 08:55:52 -05:00
Patrick Cloke	dc51d8ffaf	Add a background task to purge unused chain IDs. (#9542) ... This is a companion change to apply the fix in #9498 / 922788c604 to previously purged rooms.	2021-03-09 11:22:25 -05:00
Richard van der Hoff	eaada74075	JWT OIDC secrets for Sign in with Apple (#9549) ... Apple had to be special. They want a client secret which is generated from an EC key. Fixes #9220. Also fixes #9212 while I'm here.	2021-03-09 15:03:37 +00:00
Erik Johnston	9cd18cc588	Retry 5xx errors in federation client (#9567) ... Fixes #8915	2021-03-09 13:15:12 +00:00
Patrick Cloke	7fdc6cefb3	Fix additional type hints. (#9543) ... Type hint fixes due to Twisted 21.2.0 adding type hints.	2021-03-09 07:41:32 -05:00
Patrick Cloke	075c16b410	Handle image transparency better when thumbnailing. (#9473) ... Properly uses RGBA mode for 1- and 8-bit images with transparency (instead of RBG mode).	2021-03-09 07:37:09 -05:00
Jonathan de Jong	9898470e7d	Add logging to ObservableDeferred callbacks (#9523)	2021-03-09 11:09:31 +00:00
Matthew Hodgson	0764d0c6e5	quick config comment tweak to clarify allow_profile_lookup_over_federation	2021-03-08 21:52:04 +00:00
Jonathan de Jong	d6196efafc	Add ResponseCache tests. (#9458)	2021-03-08 14:00:07 -05:00
Dirk Klimpel	7076eee4b9	Add type hints to purge room and server notice admin API. (#9520)	2021-03-08 10:34:38 -05:00
Erik Johnston	b988b07bb0	Merge branch 'master' into develop	2021-03-08 14:06:35 +00:00
Erik Johnston	15c788e22d	1.29.0	2021-03-08 13:52:13 +00:00
Patrick Cloke	58114f8a17	Create a SynapseReactor type which incorporates the necessary reactor interfaces. (#9528) ... This helps fix some type hints when running with Twisted 21.2.0.	2021-03-08 08:25:43 -05:00
Richard van der Hoff	8a4b3738f3	Replace last_*_pdu_age metrics with timestamps (#9540) ... Following the advice at https://prometheus.io/docs/practices/instrumentation/#timestamps-not-time-since, it's preferable to export unix timestamps, not ages. There doesn't seem to be any particular naming convention for timestamp metrics.	2021-03-04 16:40:18 +00:00
Richard van der Hoff	df425c2c63	Prometheus metrics for logins and registrations (#9511) ... Add prom metrics for number of users successfully registering and logging in, by SSO provider.	2021-03-04 16:39:27 +00:00
Richard van der Hoff	7eb6e39a8f	Record the SSO Auth Provider in the login token (#9510) ... This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)	2021-03-04 14:44:22 +00:00
Erik Johnston	3f49d80dcf	1.29.0rc1	2021-03-04 10:12:53 +00:00
Patrick Cloke	33a02f0f52	Fix additional type hints from Twisted upgrade. (#9518)	2021-03-03 15:47:38 -05:00
Richard van der Hoff	4db07f9aef	Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539) ... Should fix some remaining warnings	2021-03-03 18:49:08 +00:00
Erik Johnston	a4fa044c00	Fix 'rejected_events_metadata' background update (#9537) ... Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.	2021-03-03 16:04:24 +00:00
Patrick Cloke	922788c604	Purge chain cover tables when purging events. (#9498)	2021-03-03 11:04:08 -05:00
Dirk Klimpel	d790d0d314	Add type hints to user admin API. (#9521)	2021-03-03 08:09:39 -05:00
Patrick Cloke	0c330423bc	Bump the mypy and mypy-zope versions. (#9529)	2021-03-03 07:19:19 -05:00
Erik Johnston	16f9f93eb7	Make deleting stale pushers a background update (#9536)	2021-03-03 12:08:16 +00:00
Aaron Raimist	0279e0e086	Prevent presence background jobs from running when presence is disabled (#9530) ... Prevent presence background jobs from running when presence is disabled Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-03-03 10:21:46 +00:00
Patrick Cloke	aee10768d8	Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)" ... This reverts commit f5c93fc993. This is being backed out due to a regression (#9507) and additional review feedback being provided.	2021-03-02 09:43:34 -05:00
Erik Johnston	7f5d753d06	Re-run rejected metadata background update. (#9503) ... It landed in schema version 58 after 59 had been created, causing some servers to not run it. The main effect of was that not all rooms had their chain cover calculated correctly. After the BG updates complete the chain covers will get fixed when a new state event in the affected rooms is received.	2021-03-02 14:31:23 +00:00
Erik Johnston	16108c579d	Fix SQL delta file taking a long time to run (#9516) ... Fixes #9504	2021-03-02 14:05:01 +00:00
Dirk Klimpel	f00c4e7af0	Add type hints to device and event report admin API (#9519)	2021-03-02 09:31:12 +00:00
Patrick Cloke	ad8589d392	Fix a bug when a room alias is given to the admin join endpoint (#9506)	2021-03-01 13:59:01 -05:00
Patrick Cloke	16ec8c3272	(Hopefully) stop leaking file descriptors in media repo. (#9497) ... By consuming the response if the headers imply that the content is too large.	2021-03-01 12:45:00 -05:00
Patrick Cloke	a0bc9d387e	Use the proper Request in type hints. (#9515) ... This also pins the Twisted version in the mypy job for CI until proper type hints are fixed throughout Synapse.	2021-03-01 12:23:46 -05:00
Jonathan de Jong	e12077a78a	Allow bytecode again (#9502) ... In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again. Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`	2021-02-26 18:30:54 +00:00
Tim Leung	ddb240293a	Add support for no_proxy and case insensitive env variables (#9372) ... ### Changes proposed in this PR - Add support for the `no_proxy` and `NO_PROXY` environment variables - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519)) - Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment This does contain behaviour changes for consumers so making sure these are called out: - `no_proxy`/`NO_PROXY` is now respected - lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY` Related to #9306 which also uses `ProxyAgent` Signed-off-by: Timothy Leung tim95@hotmail.co.uk	2021-02-26 17:37:57 +00:00
Richard van der Hoff	15090de850	SSO: redirect to public URL before setting cookies (#9436) ... ... otherwise, we don't get the cookie back.	2021-02-26 14:02:06 +00:00
Erik Johnston	2566dc57ce	Test that we require validated email for email pushers (#9496)	2021-02-25 15:35:14 +00:00
Erik Johnston	1e62d9ee8c	Ensure pushers are deleted for deactivated accounts (#9285)	2021-02-25 13:56:55 +00:00
Erik Johnston	1efdcc3e87	Merge branch 'master' into develop	2021-02-25 10:53:31 +00:00
Erik Johnston	b5c4fe1971	1.28.0	2021-02-25 10:22:07 +00:00
Richard van der Hoff	d8e95e5452	Add support for X-Forwarded-Proto (#9472) ... rewrite XForwardedForRequest to set `isSecure()` based on `X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.	2021-02-24 18:11:33 +00:00
Erik Johnston	7cc571510b	Add SQL delta for deleting stale pushers (#9479)	2021-02-24 17:21:10 +00:00
Jonathan de Jong	f5c93fc993	Fix #8518 (sync requests being cached wrongly on timeout) (#9358) ... This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false". This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>	2021-02-24 13:57:00 +00:00
Erik Johnston	2927921942	Clean up ShardedWorkerHandlingConfig (#9466) ... * Split ShardedWorkerHandlingConfig This is so that we have a type level understanding of when it is safe to call `get_instance(..)` (as opposed to `should_handle(..)`). * Remove special cases in ShardedWorkerHandlingConfig. `ShardedWorkerHandlingConfig` tried to handle the various different ways it was possible to configure federation senders and pushers. This led to special cases that weren't hit during testing. To fix this the handling of the different cases is moved from there and `generic_worker` into the worker config class. This allows us to have the logic in one place and allows the rest of the code to ignore the different cases.	2021-02-24 13:23:18 +00:00
Erik Johnston	0b5c967813	Refactor to ensure we call check_consistency (#9470) ... The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.	2021-02-24 10:13:53 +00:00
Patrick Cloke	7292b7c0eb	Add back the deprecated SAML endpoint. (#9474)	2021-02-23 12:57:37 -05:00
Richard van der Hoff	713145d3de	Add a comment about systemd-python. (#9464) ... This confused me for a while.	2021-02-23 13:42:36 +00:00
Patrick Cloke	65a9eb8994	Include newly added sequences in the port DB script. (#9449) ... And ensure the consistency of `event_auth_chain_id`.	2021-02-23 07:33:24 -05:00
Erik Johnston	66f4949e7f	Fix deleting pushers when using sharded pushers. (#9465)	2021-02-22 21:14:42 +00:00
Richard van der Hoff	1b2d6d55c5	Remove vestiges of uploads_path config (#9462) ... `uploads_path` was a thing that was never used; most of it was removed in #6628 but a few vestiges remained.	2021-02-22 19:54:49 +00:00
Dirk Klimpel	71c9f8de6d	Add an order_by field to list users' media admin API. (#8978)	2021-02-22 14:38:51 -05:00
Andrew Morgan	0a363f9ca4	Remove cache for get_shared_rooms_for_users (#9416) ... This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed. The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately. Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.	2021-02-22 16:52:45 +00:00
Andrew Morgan	e22b71810e	Clean up the user directory sample config section (#9385) ... The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format). This PR cleans that up a bit.	2021-02-22 11:44:31 +00:00
Patrick Cloke	fc8b3d8809	Ratelimit cross-user key sharing requests. (#8957)	2021-02-19 13:20:34 -05:00
Erik Johnston	179c0953ff	Regenerate exact thumbnails if missing (#9438)	2021-02-19 17:09:57 +00:00
Patrick Cloke	a1901abd6b	Add documentation and type hints to parse_duration. (#9432)	2021-02-19 08:32:21 -05:00
Patrick Cloke	c4a55ac4a4	Fix style checking due to updated black.	2021-02-19 08:19:54 -05:00
Patrick Cloke	d9f1dccba9	Synapse 1.28.0rc1 (2021-02-19) ... ============================== Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails). This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. Removal warning --------------- The v1 list accounts API is deprecated and will be removed in a future release. This API was undocumented and misleading. It can be replaced by the [v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts), which has been available since Synapse 1.7.0 (2019-12-13). Please check if you're using any scripts which use the admin API and replace `GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`. Features -------- - New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150)) - Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301)) - Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311)) - Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376)) - Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400)) Bugfixes -------- - Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257)) - Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296)) - Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393)) - Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333)) - Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361)) - Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391)) - Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395)) - Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421)) - Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425)) - The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426)) - Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428)) Updates to the Docker image --------------------------- - Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433)) Improved Documentation ---------------------- - Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281)) - Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291)) - Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308)) - Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317)) - Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322)) - Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390)) - Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404)) - Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407)) - Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434)) Deprecations and Removals ------------------------- - Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429)) Internal Changes ---------------- - Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003)) - Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123)) - Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240)) - Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299)) - Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305)) - Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307)) - Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326)) - Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362)) - Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396)) - Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381)) - Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384)) - Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394)) - Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAvtsUACgkQM/xY9qcR MEgEmA/+O3fCMKdyh4soBirnEZa88m1HL4Jyd+jYl7+IuFChZz0k+RYLdQ376GaC tmT4aGQISdu1wYUwO6zpN5ercvFCJ5OaSJzHinEaHZMnEfWe9SyMNB1oD9h6UYSO 8r7OT2U28pTTAbj/VpSNK8osbx/F9T5o1a7v3PLkB1kJcK9j9oqNrqpdsSW2Cz1B rGlBKRSICKR4bzfk3dR9Wr4Fcyb+gINn0LTtYYqMWxP1EtRC+VI3NEWOk3OD2dzk UztvUn0F7rKFmtcBRTXsxuGiGI9IGrd9tyKc7ilBz24eSA95psperUtHNKGwSUaO kt+bdMfEKM2cMnydMuiCYtdl/M59Rf3ShRE3OZd7TK4S4NoPy6whFsAJf7mBGxfp RcMwO/7mWRX5qjDRG6x5nM+aCqO0Y8h8WNF4gZtiN2XGyxlU0TBQMBwBM66qAoXO zCPt4tMlLsOBCABNkWT3W5Ww5KxNeMp2qEA9UnmjezA5XHtSx/DVN+6D5E4Rxw2v PzS278Sq3YO/JK6TfXsFBkZvDFPmCDdTQfKpoDbT6Hv16CXTz66XdiEg5BPZYIvp bNurfxr22/RiScxiA73cXbRBHq2qAPTxZo1O40R2LnhX0wQPQ5wfUW8oxZOnuhZ3 bWw4PvhWsZDt2SB9EIKf4RQtSxNlar88C6tpHSC89ScmE6PPWnI= =Srwm -----END PGP SIGNATURE----- Merge tag 'v1.28.0rc1' into develop Synapse 1.28.0rc1 (2021-02-19) ============================== Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails). This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. Removal warning --------------- The v1 list accounts API is deprecated and will be removed in a future release. This API was undocumented and misleading. It can be replaced by the [v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts), which has been available since Synapse 1.7.0 (2019-12-13). Please check if you're using any scripts which use the admin API and replace `GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`. Features -------- - New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150)) - Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301)) - Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311)) - Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376)) - Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400)) Bugfixes -------- - Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257)) - Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296)) - Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393)) - Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333)) - Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361)) - Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391)) - Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395)) - Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421)) - Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425)) - The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426)) - Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428)) Updates to the Docker image --------------------------- - Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433)) Improved Documentation ---------------------- - Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281)) - Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291)) - Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308)) - Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317)) - Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322)) - Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390)) - Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404)) - Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407)) - Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434)) Deprecations and Removals ------------------------- - Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429)) Internal Changes ---------------- - Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003)) - Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123)) - Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240)) - Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299)) - Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305)) - Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307)) - Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326)) - Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362)) - Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396)) - Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381)) - Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384)) - Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394)) - Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))	2021-02-19 08:10:19 -05:00
Patrick Cloke	b114a45f5f	Support not providing an IdP icon when choosing a username. (#9440)	2021-02-19 07:48:46 -05:00
Andrew Morgan	8bcfc2eaad	Be smarter about which hosts to send presence to when processing room joins (#9402) ... This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually. --- When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed. It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw. This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence: * If it was a local user join, send that user's latest presence to all servers in the room * If it was a remote user join, send the presence for all local users in the room to that homeserver We deduplicate by inserting all of those pending updates into a dictionary of the form: ``` { server_name1: {presence_update1, ...}, server_name2: {presence_update1, presence_update2, ...} } ``` Only after building this dict do we then start sending out presence updates.	2021-02-19 11:37:29 +00:00
Andrew Morgan	13e9029f44	Add a config option to prioritise local users in user directory search results (#9383) ... This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default. Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical. This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.	2021-02-19 11:02:03 +00:00
Erik Johnston	3d2acc930f	Return a 404 if we don't have the original file	2021-02-19 10:46:18 +00:00
AndrewFerr	9bc74743d5	Add configs to make profile data more private (#9203) ... Add off-by-default configuration settings to: - disable putting an invitee's profile info in invite events - disable profile lookup via federation Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>	2021-02-19 09:50:41 +00:00
Patrick Cloke	1381cd05b0	1.28.0rc1	2021-02-18 12:32:49 -05:00
Erik Johnston	b106080fb4	Regenerate exact thumbnails if missing	2021-02-18 17:05:32 +00:00
Patrick Cloke	9ee3b9775f	Remove deprecated SAML2 callback URL since it does not work. (#9434) ... Updates documentation from #9289 and removes a deprecated endpoint which didn't work as expected.	2021-02-18 11:20:33 -05:00
Rishabh Arya	e17553e185	Parse ui_auth.session_timeout as a duration (instead of treating it as ms) (#9426)	2021-02-18 09:18:14 -05:00
Patrick Cloke	8ec2217103	Reduce the memory usage of previewing media files. (#9421) ... This reduces the memory usage of previewing media files which end up larger than the `max_spider_size` by avoiding buffering content internally in treq. It also checks the `Content-Length` header in additional places instead of streaming the content to check the body length.	2021-02-18 09:01:29 -05:00
Patrick Cloke	43f1c82457	Add back the guard against the user directory stream position not existing. (#9428) ... As the comment says, this guard was there for when the initial user directory update has yet to happen.	2021-02-18 08:44:19 -05:00
Dirk Klimpel	c8d9383cfb	Add the shadow-banning status to the display user admin API. (#9400)	2021-02-17 15:19:23 -05:00
Andrew Morgan	a25661b2eb	Remove dead notify_for_states presence method (#9408)	2021-02-17 17:32:26 +00:00
Andrew Morgan	3e5749b99f	Fix only handling the last presence state for each user (#9425) ... This is a small bug that I noticed while working on #8956. We have a for-loop which attempts to strip all presence changes for each user except for the final one, as we don't really care about older presence: 9e19c6aab4/synapse/handlers/presence.py (L368-L371) `new_states_dict` stores this stripped copy of latest presence state for each user, before it is... put into a new variable `new_state`, which is just overridden by the subsequent for loop. I believe this was instead meant to override `new_states`. Without doing so, it effectively meant: 1. The for loop had no effect. 2. We were still processing old presence state for users.	2021-02-17 17:31:37 +00:00
Patrick Cloke	d2f0ec12d5	Add type hints to groups code. (#9393)	2021-02-17 08:41:47 -05:00
Richard van der Hoff	e1071fd625	Support for form_post in OIDC responses (#9376) ... Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.	2021-02-17 10:15:14 +00:00
Richard van der Hoff	33f64ca7d6	Allow OIDC config to override discovered values (#9384) ... Fixes #9347	2021-02-16 22:33:09 +00:00
Eric Eastwood	0a00b7ff14	Update black, and run auto formatting over the codebase (#9381) ... - Update black version to the latest - Run black auto formatting over the codebase - Run autoformatting according to [`docs/code_style.md `](80d6dc9783/docs/code_style.md) - Update `code_style.md` docs around installing black to use the correct version	2021-02-16 22:32:34 +00:00
Richard van der Hoff	3b754aea27	Clean up caching/locking of OIDC metadata load (#9362) ... Ensure that we lock correctly to prevent multiple concurrent metadata load requests, and generally clean up the way we construct the metadata cache.	2021-02-16 16:27:38 +00:00
Erik Johnston	0ad087273c	Merge branch 'master' into develop	2021-02-16 13:39:30 +00:00
Patrick Cloke	731e08c63a	Handle missing data in power levels events during room upgrade. (#9395)	2021-02-16 08:31:39 -05:00
Erik Johnston	a27c1fd74b	1.27.0	2021-02-16 13:12:02 +00:00
Andrew Morgan	594f2853e0	Remove dead handled_events set in invite_join (#9394) ... This PR removes a set that was created and [initially used](1d2a0040cf (diff-0bc92da3d703202f5b9be2d3f845e375f5b1a6bc6ba61705a8af9be1121f5e42R435-R436)), but is no longer today. May help cut down a bit on the time it takes to accept invites.	2021-02-12 22:15:50 +00:00
Patrick Cloke	7950aa8a27	Fix some typos.	2021-02-12 11:14:12 -05:00
Patrick Cloke	2c9b4a5f16	Synapse 1.27.0rc2 (2021-02-11) ... ============================== Features -------- - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297)) Bugfixes -------- - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302)) - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313)) Improved Documentation ---------------------- - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS /tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN 309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb /QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4 awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU 2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0= =NSyp -----END PGP SIGNATURE----- Merge tag 'v1.27.0rc2' into develop Synapse 1.27.0rc2 (2021-02-11) ============================== Features -------- - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297)) Bugfixes -------- - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302)) - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313)) Improved Documentation ---------------------- - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))	2021-02-11 11:56:03 -05:00
Patrick Cloke	40de534238	1.27.0rc2	2021-02-11 11:22:29 -05:00
Patrick Cloke	e40d88cff3	Backout changes for automatically calculating the public baseurl. (#9313) ... This breaks some people's configurations (if their Client-Server API is not accessed via port 443).	2021-02-11 11:16:54 -05:00
Erik Johnston	6aa87f8ce3	Ensure that we never stop reconnecting to redis (#9391)	2021-02-11 16:06:29 +00:00
Patrick Cloke	6dade80048	Combine the CAS & SAML implementations for required attributes. (#9326)	2021-02-11 10:05:15 -05:00
Eric Eastwood	80d6dc9783	Remove conflicting sqlite tables that are "reserved" (shadow fts4 tables) (#9003) ... Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests. Fix #8996	2021-02-10 20:12:57 +00:00
Brendan Abolivier	29ae04af3b	Remove unneeded type constraints on 3rd party protocol lookup responses	2021-02-09 17:50:25 +01:00
Patrick Cloke	3f58fc848d	Type hints and validation improvements. (#9321) ... * Adds type hints to the groups servlet and stringutils code. * Assert the maximum length of some input values for spec compliance.	2021-02-08 13:59:54 -05:00
Patrick Cloke	0963d39ea6	Handle additional errors when previewing URLs. (#9333) ... * Handle the case of lxml not finding a document tree. * Parse the document encoding from the XML tag.	2021-02-08 12:33:30 -05:00
David Teller	b0b2cac057	Merge pull request #9150 from Yoric/develop-context ... New API /_synapse/admin/rooms/{roomId}/context/{eventId}	2021-02-08 15:53:44 +01:00
Jonathan de Jong	d882fbca38	Update type hints for Cursor to match PEP 249. (#9299)	2021-02-05 15:39:19 -05:00
Erik Johnston	adc96d4236	Merge branch 'erikj/media_spam_checker' into develop	2021-02-04 17:01:59 +00:00
Erik Johnston	7e8083eb48	Add check_media_file_for_spam spam checker hook	2021-02-04 17:01:30 +00:00
Patrick Cloke	792263c97c	Handle empty rooms when generating email notifications. (#9257) ... Fixes some exceptions if the room state isn't quite as expected. If the expected state events aren't found, try to find them in the historical room state. If they still aren't found, fallback to a reasonable, although ugly, value.	2021-02-04 10:18:25 -05:00
Patrick Cloke	2ab6e67ab7	Fix escaping of braces in OIDC sample config. (#9317) ... This fixes the Jinja2 templates for the mapping provider.	2021-02-04 09:06:20 -05:00
Jonathan de Jong	2814028ce5	Add experimental support for PyPy. (#9123) ... * Adds proper dependencies. * Minor fixes in database layer.	2021-02-04 08:29:47 -05:00
Marcus	b0f4119b8b	Add debug logging to DNS SRV requests. (#9305)	2021-02-03 16:47:30 -05:00
Richard van der Hoff	3f534d3fdf	Merge branch 'social_login_hotfixes' into develop	2021-02-03 20:34:27 +00:00
Richard van der Hoff	17f2a512f3	Merge remote-tracking branch 'origin/release-v1.27.0' into social_login_hotfixes	2021-02-03 20:33:32 +00:00
Richard van der Hoff	e288499c60	Social login UI polish (#9301)	2021-02-03 20:31:23 +00:00
Patrick Cloke	afa18f1baa	Clarify documentation about escaping URLs in templates. (#9310)	2021-02-03 14:51:38 -05:00
Richard van der Hoff	ce669863b9	Add debug for OIDC flow (#9307)	2021-02-03 19:45:34 +00:00
Richard van der Hoff	7a0dcea3e5	social login Fix username validation javascript (#9297) ... * fix validation and don't use built-in validation UI Co-authored-by: Bruno Windels <brunow@element.io>	2021-02-03 17:52:55 +00:00
Richard van der Hoff	f20dadb649	Fix formatting for "bad session" error during sso registration flow (#9296)	2021-02-03 16:13:09 +00:00
dykstranet	e4cdecb310	config: Add detail to auto_join_rooms comment (#9291) ... config: Add detail to auto_join_rooms comment Signed-off-by: Gary Dykstra <gary@dykstranet.com>	2021-02-03 15:21:30 +00:00
Tim Gates	e1943d1353	Typo fix in a comment: subequently -> subsequently. (#8988)	2021-02-03 07:24:53 -05:00
Patrick Cloke	4ca054a4ea	Convert blacklisted IPv4 addresses to compatible IPv6 addresses. (#9240) ... Also add a few more IP ranges to the default blacklist.	2021-02-03 07:13:46 -05:00
Erik Johnston	ff55300b91	Honour ratelimit flag for application services for invite ratelimiting (#9302)	2021-02-03 10:17:37 +00:00
Richard van der Hoff	96e460df2e	social login: add noopener to terms link (#9300)	2021-02-02 18:35:28 +00:00
Erik Johnston	2610930721	1.27.0rc1	2021-02-02 13:32:05 +00:00
Travis Ralston	b60bb28bbc	Add an admin API to get the current room state (#9168) ... This could arguably replace the existing admin API for `/members`, however that is out of scope of this change. This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.	2021-02-02 11:16:29 +00:00
Richard van der Hoff	8f75bf1df7	Put SAML callback URI under /_synapse/client. (#9289)	2021-02-02 09:43:50 +00:00
Richard van der Hoff	846b9d3df0	Put OIDC callback URI under /_synapse/client. (#9288)	2021-02-01 22:56:01 +00:00
Richard van der Hoff	8fee6a3ab2	Merge branch 'social_login' into develop	2021-02-01 18:48:11 +00:00
Richard van der Hoff	351845452c	fix broken HTML tag	2021-02-01 18:47:01 +00:00
Richard van der Hoff	5963426b95	Merge branch 'social_login' into develop	2021-02-01 18:46:12 +00:00
Bruno Windels	f30c3a99be	make primary button not wider than viewport	2021-02-01 18:39:17 +00:00
Richard van der Hoff	c543bf87ec	Collect terms consent from the user during SSO registration (#9276)	2021-02-01 18:37:41 +00:00
Richard van der Hoff	e5d70c8a82	Improve styling and wording of SSO UIA templates (#9286) ... fixes #9171	2021-02-01 18:36:04 +00:00
Patrick Cloke	5d38a3c97f	Refactor email summary generation. (#9260) ... * Fixes a case where no summary text was returned. * The use of messages_from_person vs. messages_from_person_and_others was tweaked to depend on whether there was 1 sender or multiple senders, not based on if there was 1 room or multiple rooms.	2021-02-01 13:09:39 -05:00
Richard van der Hoff	419313b06a	Improve styling and wording of SSO error templates (#9287)	2021-02-01 18:01:15 +00:00
Richard van der Hoff	85c56b5a67	Make importing display name and email optional (#9277)	2021-02-01 17:30:42 +00:00
Richard van der Hoff	18ab35284a	Merge branch 'social_login' into develop	2021-02-01 17:28:37 +00:00
Jan Christian Grünhage	43dd93bb26	Add phone home stats for encrypted messages. (#9283) ... Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>	2021-02-01 17:06:22 +00:00
Andrew Morgan	a800603561	Prevent email UIA failures from raising a LoginError (#9265) ... Context, Fixes: https://github.com/matrix-org/synapse/issues/9263 In the past to fix an issue with old Riots re-requesting threepid validation tokens, we raised a `LoginError` during UIA instead of `InteractiveAuthIncompleteError`. This is now breaking the way Tchap logs in - which isn't standard, but also isn't disallowed by the spec. An easy fix is just to remove the 4 year old workaround.	2021-02-01 15:54:39 +00:00
Richard van der Hoff	4167494c90	Replace username picker with a template (#9275) ... There's some prelimiary work here to pull out the construction of a jinja environment to a separate function. I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.	2021-02-01 15:52:50 +00:00
Richard van der Hoff	8aed29dc61	Improve styling and wording of SSO redirect confirm template (#9272)	2021-02-01 15:50:56 +00:00
Richard van der Hoff	9c715a5f19	Fix SSO on workers (#9271) ... Fixes #8966. * Factor out build_synapse_client_resource_tree Start a function which will mount resources common to all workers. * Move sso init into build_synapse_client_resource_tree ... so that we don't have to do it for each worker * Fix SSO-login-via-a-worker Expose the SSO login endpoints on workers, like the documentation says. * Update workers config for new endpoints Add documentation for endpoints recently added (#8942, #9017, #9262) * remove submit_token from workers endpoints list this *doesn't* work on workers (yet). * changelog * Add a comment about the odd path for SAML2Resource	2021-02-01 15:47:59 +00:00
Richard van der Hoff	f78d07bf00	Split out a separate endpoint to complete SSO registration (#9262) ... There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.	2021-02-01 13:15:51 +00:00
Ivan Shapovalov	13c7ab8181	Fixes for PyPy compatibility (#9270) ... * synapse.app.base: only call gc.freeze() on CPython gc.freeze() is an implementation detail of CPython garbage collector, and notably does not exist on PyPy. Rather than playing whack-a-mole and skipping the call when under PyPy, simply restrict it to CPython because the whole gc module is implementation-defined. Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>	2021-01-30 17:22:05 +00:00
Erik Johnston	f2c1560eca	Ratelimit invites by room and target user (#9258)	2021-01-29 16:38:29 +00:00
Richard van der Hoff	0d81a6fa3e	Merge branch 'social_login' into develop	2021-01-28 22:08:11 +00:00
Erik Johnston	4b73488e81	Ratelimit 3PID /requestToken API (#9238)	2021-01-28 17:39:21 +00:00
Erik Johnston	54a6afeee3	Cache config options in SSL verification (#9255) ... Reading from the config object is *slow*.	2021-01-28 17:38:59 +00:00
David Teller	31d072aea0	FIXUP: linter	2021-01-28 16:53:40 +01:00
Patrick Cloke	a78016dadf	Add type hints to E2E handler. (#9232) ... This finishes adding type hints to the `synapse.handlers` module.	2021-01-28 08:34:19 -05:00
David Teller	93f84e0373	FIXUP: Making get_event_context a bit more paranoid	2021-01-28 12:31:07 +01:00
David Teller	b755f60ce2	FIXUP: Removing awaitable	2021-01-28 12:31:07 +01:00
David Teller	a764869623	FIXUP: Doc	2021-01-28 12:31:07 +01:00
David Teller	b859919acc	FIXUP: Now testing that the user is admin!	2021-01-28 12:31:07 +01:00
David Teller	de7f049527	FIXUP: Don't filter events at all for admin/v1/rooms/.../context/...	2021-01-28 12:31:07 +01:00
David Teller	10332c175c	New API /_synapse/admin/rooms/{roomId}/context/{eventId} ... Signed-off-by: David Teller <davidt@element.io>	2021-01-28 12:29:49 +01:00
Richard van der Hoff	a083aea396	Add 'brand' field to MSC2858 response (#9242) ... We've decided to add a 'brand' field to help clients decide how to style the buttons. Also, fix up the allowed characters for idp_id, while I'm in the area.	2021-01-27 21:31:45 +00:00
Richard van der Hoff	869667760f	Support for scraping email addresses from OIDC providers (#9245)	2021-01-27 21:28:59 +00:00
Pankaj Yadav	2e537a0280	Check if a user is in the room before sending a PowerLevel event on their behalf (#9235)	2021-01-27 17:38:08 +00:00
Richard van der Hoff	fbd9de6d1f	Synapse 1.26.0 (2021-01-27) ... =========================== This release brings a new schema version for Synapse and rolling back to a previous version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. No significant changes since 1.26.0rc2. Synapse 1.26.0rc2 (2021-01-25) ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204)) Synapse 1.26.0rc1 (2021-01-20) ============================== This release brings a new schema version for Synapse and rolling back to a previous version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. Features -------- - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177)) - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091)) - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159)) - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024)) - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984)) - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086)) - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932)) - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948)) - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130)) - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068)) - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092)) - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166)) Bugfixes -------- - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023)) - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028)) - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051)) - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053)) - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054)) - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059)) - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070)) - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071)) - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116)) - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117)) - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128)) - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108)) - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145)) - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161)) Improved Documentation ---------------------- - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997)) - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035)) - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040)) - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057)) - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151)) Deprecations and Removals ------------------------- - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039)) Internal Changes ---------------- - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124)) - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939)) - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016)) - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018)) - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025)) - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030)) - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031)) - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033)) - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038)) - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041)) - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055)) - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058)) - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063)) - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069)) - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080)) - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093)) - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098)) - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106)) - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112)) - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125)) - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144)) - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146)) - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmARjqYACgkQM/xY9qcR MEiknRAAtga+KXEG40xg7w78XXQk6nXIhhi30nYue9/SeHMTCmLmd1+f0yTySrRz JMc/xe9VP1th5DThHZeHJf0tLIVz/VFA5gr/fmk44FFEFvc77IhttVHSXbYZLuM8 komOzLTJy7ueVSpyF9xVfEEz6UIRc098b2KHjuBh3SGlZAOmFuFHUMVBeEuYo80E ayVWXPNP+7U5y1qLA9uY/oxMsFjl6bPvjak3YDyGWCr2Y4nOhM24AwccErs0yAtu kCRm9JR0yD72uRLyaGyMZ9Atg4o01BD7hu0ilxhbRIdAvI5Cdk0E7rZshoLr33wk Ls4XlK9SxJSV4vEu44OJBCRnDjQS9HQnwYk13z+H5HrJcxXRyIeCwRD7aofzNY8O kNg/1oasBDiW2iVoLJ4Y0nWLHPcNlWJxUzkkjlF8BmEUQjURNVdh0n3y3/ST+T1S scex0J9Z95HNoLfRYID0adrYqWTOl5GLzy9C1AwqlLWY79y6YsB4LDmmfQgoTKCM VICSYtEKWph6WXdpd/kJsTgHyG3UFAMDraWVez7TxXpc7lpvOMkld+0oQnHWAXC3 XRc+63jTs6pFJ74M+ziqOrC/1ORg1DK8OLIm/TskLoYnn6L9Dh8ssXeabLv63H8o cEqETUbZ77rZro0Nz2H1OJG09uTFUzFEy5Ecya7g4DqtRT/Fs9Q= =I9CI -----END PGP SIGNATURE----- Merge tag 'v1.26.0' into social_login Synapse 1.26.0 (2021-01-27) =========================== This release brings a new schema version for Synapse and rolling back to a previous version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. No significant changes since 1.26.0rc2. Synapse 1.26.0rc2 (2021-01-25) ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204)) Synapse 1.26.0rc1 (2021-01-20) ============================== This release brings a new schema version for Synapse and rolling back to a previous version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. Features -------- - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177)) - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091)) - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159)) - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024)) - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984)) - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086)) - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932)) - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948)) - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130)) - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068)) - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092)) - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166)) Bugfixes -------- - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023)) - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028)) - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051)) - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053)) - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054)) - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059)) - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070)) - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071)) - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116)) - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117)) - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128)) - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108)) - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145)) - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161)) Improved Documentation ---------------------- - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997)) - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035)) - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040)) - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057)) - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151)) Deprecations and Removals ------------------------- - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039)) Internal Changes ---------------- - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124)) - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939)) - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016)) - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018)) - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025)) - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030)) - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031)) - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033)) - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038)) - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041)) - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055)) - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058)) - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063)) - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069)) - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080)) - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093)) - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098)) - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106)) - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112)) - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125)) - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144)) - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146)) - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))	2021-01-27 17:27:58 +00:00
Richard van der Hoff	7fa1346f93	Merge branch 'social_login' into develop	2021-01-27 17:27:24 +00:00
Patrick Cloke	17b713850f	Merge branch 'master' into develop	2021-01-27 11:13:21 -05:00
Patrick Cloke	e54746bdf7	Clean-up the template loading code. (#9200) ... * Enables autoescape by default for HTML files. * Adds a new read_template method for reading a single template. * Some logic clean-up.	2021-01-27 10:59:50 -05:00
Patrick Cloke	73ed289bd2	1.26.0	2021-01-27 10:50:37 -05:00
Richard van der Hoff	a737cc2713	Implement MSC2858 support (#9183) ... Fixes #8928.	2021-01-27 12:41:24 +00:00
Andrew Morgan	a64c29926e	Pass a dict, instead of None, to modules if a None config is specified in the homeserver config (#9229) ... If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead. This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](5310808d3b/docs/sample_config.yaml (L1816-L1845)) states) then `None` is not passed to the module. An empty dict is passed instead. This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.	2021-01-27 11:49:31 +00:00
Patrick Cloke	1baab20352	Add type hints to various handlers. (#9223) ... With this change all handlers except the e2e_* ones have type hints enabled.	2021-01-26 10:50:21 -05:00
Patrick Cloke	26837d5dbe	Do not require the CAS service URL setting (use public_baseurl instead). (#9199) ... The current configuration is handled for backwards compatibility, but is considered deprecated.	2021-01-26 10:49:25 -05:00
Erik Johnston	dd8da8c5f6	Precompute joined hosts and store in Redis (#9198)	2021-01-26 13:57:31 +00:00
Patrick Cloke	4937fe3d6b	Try to recover from unknown encodings when previewing media. (#9164) ... Treat unknown encodings (according to lxml) as UTF-8 when generating a preview for HTML documents. This isn't fully accurate, but will hopefully give a reasonable title and summary.	2021-01-26 07:32:17 -05:00
Jason Robinson	e5b659e9e1	Merge pull request #9062 from matrix-org/jaywink/admin-forward-extremities ... Add forward extremities endpoint to rooms admin API	2021-01-26 12:57:38 +02:00
Erik Johnston	a1ff1e967f	Periodically send pings to detect dead Redis connections (#9218) ... This is done by creating a custom `RedisFactory` subclass that periodically pings all connections in its pool. We also ensure that the `replyTimeout` param is non-null, so that we timeout waiting for the reply to those pings (and thus triggering a reconnect).	2021-01-26 10:54:54 +00:00
Jason Robinson	4936fc59fc	Fix get forward extremities query ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-26 10:21:02 +02:00
Jason Robinson	cee4010f94	Merge branch 'develop' into jaywink/admin-forward-extremities ... # Conflicts: # synapse/rest/admin/__init__.py	2021-01-26 10:15:32 +02:00
Jason Robinson	e20f18a766	Make natural join inner join ... Co-authored-by: Erik Johnston <erik@matrix.org>	2021-01-26 10:13:35 +02:00
Patrick Cloke	5b857b77f7	Don't error if deleting a non-existent pusher. (#9121)	2021-01-25 14:52:30 -05:00
Patrick Cloke	4a55d267ee	Add an admin API for shadow-banning users. (#9209) ... This expands the current shadow-banning feature to be usable via the admin API and adds documentation for it. A shadow-banned users receives successful responses to their client-server API requests, but the events are not propagated into rooms. Shadow-banning a user should be used as a tool of last resort and may lead to confusing or broken behaviour for the client.	2021-01-25 14:49:39 -05:00
Richard van der Hoff	65fb3b2e25	Synapse 1.26.0rc2 (2021-01-25) ... ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM= =Qc7+ -----END PGP SIGNATURE----- Merge tag 'v1.26.0rc2' into social_login Synapse 1.26.0rc2 (2021-01-25) ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))	2021-01-25 19:37:58 +00:00
Jason Robinson	fe18882bb5	Merge remote-tracking branch 'origin/develop' into jaywink/admin-forward-extremities	2021-01-25 15:55:54 +02:00
Patrick Cloke	e448dbbf5b	Synapse 1.26.0rc2 (2021-01-25) ... ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM= =Qc7+ -----END PGP SIGNATURE----- Merge tag 'v1.26.0rc2' into develop Synapse 1.26.0rc2 (2021-01-25) ============================== Bugfixes -------- - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195)) - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210)) Internal Changes ---------------- - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189)) - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))	2021-01-25 08:51:45 -05:00
Patrick Cloke	a01605c136	1.26.0rc2	2021-01-25 08:25:40 -05:00
Patrick Cloke	6f7417c3db	Handle missing content keys when calculating presentable names. (#9165) ... Treat the content as untrusted and do not assume it is of the proper form.	2021-01-25 07:27:16 -05:00
Jason Robinson	8965b6cfec	Merge branch 'develop' into jaywink/admin-forward-extremities	2021-01-23 21:41:35 +02:00
Jason Robinson	930ba00971	Add depth and received_ts to forward_extremities admin API response ... Also add a warning on the admin API documentation. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-23 21:34:32 +02:00
Erik Johnston	056327457f	Fix chain cover update to handle events with duplicate auth events (#9210)	2021-01-22 19:44:08 +00:00
Erik Johnston	28f255d5f3	Bump psycopg2 version (#9204) ... As we use `execute_values` with the `fetch` parameter.	2021-01-22 11:14:49 +00:00
Patrick Cloke	a7882f9887	Return a 404 if no valid thumbnail is found. (#9163) ... If no thumbnail of the requested type exists, return a 404 instead of erroring. This doesn't quite match the spec (which does not define what happens if no thumbnail can be found), but is consistent with what Synapse already does.	2021-01-21 14:53:58 -05:00
Erik Johnston	758ed5f1bc	Speed up chain cover calculation (#9176)	2021-01-21 17:00:12 +00:00
Erik Johnston	12ec55bfaa	Increase perf of handling concurrent use of StreamIDGenerators. (#9190) ... We have seen a failure mode here where if there are many in flight unfinished IDs then marking an ID as finished takes a lot of CPU (as calling deque.remove iterates over the list)	2021-01-21 16:31:51 +00:00
Erik Johnston	939ef657ce	Merge remote-tracking branch 'origin/release-v1.26.0' into develop	2021-01-21 16:05:13 +00:00
Erik Johnston	ccfafac882	Add schema update to fix existing DBs affected by #9193 (#9195)	2021-01-21 16:03:25 +00:00
Erik Johnston	b249f002b8	Merge remote-tracking branch 'origin/release-v1.26.0' into develop	2021-01-21 15:09:30 +00:00
Erik Johnston	2506074ef0	Fix receipts or account data not being sent down sync (#9193) ... Introduced in #9104 This wasn't picked up by the tests as this is all fine the first time you run Synapse (after upgrading), but then when you restart the wrong value is pulled from `stream_positions`.	2021-01-21 15:09:09 +00:00
Erik Johnston	7a43482f19	Use execute_batch in more places (#9188) ... * Use execute_batch in more places * Newsfile	2021-01-21 14:44:12 +00:00
Dirk Klimpel	c55e62548c	Add tests for List Users Admin API (#9045)	2021-01-21 09:18:46 -05:00
Richard van der Hoff	42a8e81370	Add a check for duplicate IdP ids (#9184)	2021-01-21 13:20:58 +00:00
Richard van der Hoff	b5120f09f1	Merge remote-tracking branch 'origin/release-v1.26.0' into develop	2021-01-21 13:17:07 +00:00
Richard van der Hoff	7447f19702	Prefix idp_id with "oidc-" (#9189) ... ... to avoid clashes with other SSO mechanisms	2021-01-21 12:25:02 +00:00
Erik Johnston	eee6fcf5fa	Use execute_batch instead of executemany in places (#9181) ... `execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.	2021-01-21 10:22:53 +00:00
Patrick Cloke	f81d02d75b	Synapse 1.26.0rc1 (2021-01-20) ... ============================== This release brings a new schema version for Synapse and rolling back to a previous verious is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. Features -------- - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177)) - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091)) - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159)) - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024)) - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984)) - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086)) - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932)) - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948)) - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130)) - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068)) - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092)) - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166)) Bugfixes -------- - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023)) - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028)) - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051)) - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053)) - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054)) - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059)) - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070)) - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071)) - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116)) - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117)) - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128)) - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108)) - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145)) - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161)) Improved Documentation ---------------------- - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997)) - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035)) - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040)) - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057)) - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151)) Deprecations and Removals ------------------------- - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039)) Internal Changes ---------------- - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124)) - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939)) - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016)) - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018)) - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025)) - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030)) - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031)) - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033)) - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038)) - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041)) - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055)) - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058)) - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063)) - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069)) - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080)) - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093)) - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098)) - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106)) - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112)) - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125)) - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144)) - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146)) - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAIVFkACgkQM/xY9qcR MEhLfQ//Z4xnP5Icx732k2xpdR+JaXX6heNVFm/UuLNd6wq+uXjO/Dgc9IKQ5LXD kfG+/OnNGDr/eUk4mQ5s/ccBvaTrOid2IIemJd4lUBy/Q5LvPvfKIp08QFr72WJT 7Y4Ma54nNSSMUAdESgj/aFAHjATbfHhxPOZ5OdQHGLAuJ/OUwR2ksasVNCuINbgh 8vrxrzjeYn1Zl3UTBrsdCcat7AFqYjxK/Y0i6JxfSgRwDZjUbW4M2C4OGWZJPUrU 2YTPHangxd+22+HObE2KDEzaHDdCR/Kj4dhCiJBynZ5iucuHSS+mgf39RtbjqWML 01cmXXUV+FH7rnCilmF9rYdmHZ6L9BJxtrQDQU4dB106BX9G7hKR6rYHc9Z6tjOh 3tg0cKWYMXOsTNnvRqiFNh2q0yBKS228HRAwyGosW/6NiYj43ArtYP6knX7dQyBX 0mYNkfry2dyX6Kj5el0i9MOTHYQxfc8apsBm2M3OTNYukgQKA0NhbAquibaAxEEW 2qIqbSp4CUlGPvM7+u+ZGTu0hEbVKKNjqMfXhuY8A4/BWiV2mSqKcEirR5cE/LqY mq5Ac7vbO4Uh+1xiRw/G9ITi1dqAjYIzhBawlhdUPOh+aINWTUikYYzjUSmm4PbN H9BTPLA9iAdVRfuWJ9um2G0DdS8Qpx/aIRh3MScXVly6cGCC6Do= =kgqn -----END PGP SIGNATURE----- Merge tag 'v1.26.0rc1' into develop Synapse 1.26.0rc1 (2021-01-20) ============================== This release brings a new schema version for Synapse and rolling back to a previous verious is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes and for general upgrade guidance. Features -------- - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177)) - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091)) - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159)) - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024)) - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984)) - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086)) - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932)) - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948)) - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130)) - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068)) - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092)) - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166)) Bugfixes -------- - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023)) - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028)) - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051)) - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053)) - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054)) - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059)) - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070)) - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071)) - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116)) - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117)) - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128)) - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108)) - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145)) - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161)) Improved Documentation ---------------------- - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997)) - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035)) - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040)) - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057)) - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151)) Deprecations and Removals ------------------------- - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039)) Internal Changes ---------------- - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124)) - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939)) - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016)) - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018)) - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025)) - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030)) - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031)) - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033)) - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038)) - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041)) - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055)) - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058)) - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063)) - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069)) - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080)) - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093)) - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098)) - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106)) - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112)) - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125)) - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144)) - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146)) - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))	2021-01-20 11:27:39 -05:00
Richard van der Hoff	e51b2f3f91	Tighten the restrictions on idp_id (#9177)	2021-01-20 08:55:14 -05:00
Richard van der Hoff	0cd2938bc8	Support icons for Identity Providers (#9154)	2021-01-20 08:15:14 -05:00
Patrick Cloke	620ecf13b0	Various improvements to the federation client. (#9129) ... * Type hints for `FederationClient`. * Using `async` functions instead of returning `Awaitable` instances.	2021-01-20 07:59:18 -05:00
Richard van der Hoff	6c0dfd2e8e	Merge remote-tracking branch 'origin/develop' into release-v1.26.0	2021-01-20 12:33:05 +00:00
Richard van der Hoff	fa50e4bf4d	Give public_baseurl a default value (#9159)	2021-01-20 12:30:41 +00:00
Patrick Cloke	72822e60be	1.26.0rc1	2021-01-20 07:14:25 -05:00
Patrick Cloke	fa842a9866	Use the account handler in additional places. (#9166)	2021-01-20 10:44:52 +00:00
Patrick Cloke	47d48a5853	Validate the server name for the /publicRooms endpoint. (#9161) ... If a remote server name is provided, ensure it is something reasonable before making remote connections to it.	2021-01-19 14:21:59 -05:00
Richard van der Hoff	73b03722f4	Fix error messages from OIDC config parsing (#9153) ... Make sure we report the correct config path for errors in the OIDC configs.	2021-01-19 14:56:54 +00:00
Erik Johnston	6633a4015a	Allow moving account data and receipts streams off master (#9104)	2021-01-18 15:47:59 +00:00
Erik Johnston	f08ef64926	Enforce all replication HTTP clients calls use kwargs (#9144)	2021-01-18 15:24:04 +00:00
Patrick Cloke	2b467d0b61	Properly raise an exception when the body exceeds the max size. (#9145) ... ...instead of just creating the exception object and doing nothing with it.	2021-01-18 10:21:42 -05:00
Richard van der Hoff	02070c69fa	Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128) ... * Factor out a common TestHtmlParser Looks like I'm doing this in a few different places. * Improve OIDC login test Complete the OIDC login flow, rather than giving up halfway through. * Ensure that OIDC login works with multiple OIDC providers * Fix bugs in handling clientRedirectUrl - don't drop duplicate query-params, or params with no value - allow utf-8 in query-params	2021-01-18 14:52:49 +00:00
Patrick Cloke	de1f8de319	Ensure the user ID is serialized in the payload instead of used as an instance name. (#9130)	2021-01-18 11:08:26 +00:00
Erik Johnston	350d9923cd	Make chain cover index bg update go faster (#9124) ... We do this by allowing a single iteration to process multiple rooms at a time, as there are often a lot of really tiny rooms, which can massively slow things down.	2021-01-15 17:18:37 +00:00
Richard van der Hoff	2de7e263ed	Ensure we store pusher data as text (#9117) ... I don't think there's any need to use canonicaljson here. Fixes: #4475.	2021-01-15 16:57:23 +00:00
Richard van der Hoff	9de6b94117	Land support for multiple OIDC providers (#9110) ... This is the final step for supporting multiple OIDC providers concurrently. First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before: oidc_config: enabled: true issuer: "https://oidc_provider" # etc After: oidc_providers: - idp_id: prov1 issuer: "https://oidc_provider" - idp_id: prov2 issuer: "https://another_oidc_provider" The old format is still grandfathered in. With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.	2021-01-15 16:55:29 +00:00
Patrick Cloke	3e4cdfe5d9	Add an admin API endpoint to protect media. (#9086) ... Protecting media stops it from being quarantined when e.g. all media in a room is quarantined. This is useful for sticker packs and other media that is uploaded by server administrators, but used by many people.	2021-01-15 11:18:09 -05:00
Patrick Cloke	74dd906041	Avoid raising the body exceeded error multiple times. (#9108) ... Previously this code generated unreferenced `Deferred` instances which caused "Unhandled Deferreds" errors to appear in error situations.	2021-01-15 11:00:13 -05:00
Richard van der Hoff	9ffac2bef1	Remote dependency on distutils (#9125) ... `distutils` is pretty much deprecated these days, and replaced with `setuptools`. It's also annoying because it's you can't `pip install` it, and it's hard to figure out which debian package we should depend on to make sure it's there. Since we only use it for a tiny function anyway, let's just vendor said function into our codebase.	2021-01-15 15:59:20 +00:00
Patrick Cloke	d34c6e1279	Add type hints to media rest resources. (#9093)	2021-01-15 10:57:37 -05:00
Richard van der Hoff	4575ad0b1e	Store an IdP ID in the OIDC session (#9109) ... Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP. As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.	2021-01-15 13:22:12 +00:00
Richard van der Hoff	14950a45d6	Merge pull request #9091 from matrix-org/rav/error_on_bad_sso ... Give the user a better error when they present bad SSO creds	2021-01-15 00:27:13 +00:00
Erik Johnston	1a08e0cdab	Fix event chain bg update. (#9118) ... We passed in a graph to `sorted_topologically` which didn't have an entry for each node (as we dropped nodes with no edges).	2021-01-14 18:57:32 +00:00
Erik Johnston	d2479c6870	Fix perf of get_cross_signing_keys (#9116)	2021-01-14 17:57:09 +00:00
Erik Johnston	659c415ed4	Fix chain cover background update to work with split out event persisters (#9115)	2021-01-14 17:19:35 +00:00
Erik Johnston	631dd06f2c	Fix get destinations to catch up query. (#9114) ... t was doing a sequential scan on `destination_rooms`, which took minutes.	2021-01-14 16:47:21 +00:00
Erik Johnston	7036e24e98	Add background update for add chain cover index (#9029)	2021-01-14 15:18:27 +00:00
Richard van der Hoff	21a296cd5a	Split OidcProvider out of OidcHandler (#9107) ... The idea here is that we will have an instance of OidcProvider for each configured IdP, with OidcHandler just doing the marshalling of them. For now it's still hardcoded with a single provider.	2021-01-14 13:29:17 +00:00
Tim Leung	12702be951	Fix wrong arguments being passed to BlacklistingAgentWrapper (#9098) ... A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper used by the WellyKnownResolver. This coulld cause exceptions when attempting to connect to IP addresses that are blacklisted, but in reality this did not have any observable affect since this code is not used for IP literals.	2021-01-14 06:59:26 -05:00
Richard van der Hoff	420031906a	Move complete_sso_ui_auth into SSOHandler ... since we're hacking on this code anyway, may as well move it out of the cluttered AuthHandler.	2021-01-13 20:22:41 +00:00
Richard van der Hoff	5310808d3b	Give the user a better error when they present bad SSO creds ... If a user tries to do UI Auth via SSO, but uses the wrong account on the SSO IdP, try to give them a better error. Previously, the UIA would claim to be successful, but then the operation in question would simply fail with "auth fail". Instead, serve up an error page which explains the failure.	2021-01-13 20:22:41 +00:00
Richard van der Hoff	d02e4b2825	Merge pull request #9105 from matrix-org/rav/multi_idp/oidc_provider_config ... Enhancements to OIDC configuration handling	2021-01-13 19:51:46 +00:00
Patrick Cloke	aee8e6a95d	Reduce scope of exception handler. (#9106) ... Removes a bare `except Exception` clause and replaces it with catching a specific exception around the portion that might throw.	2021-01-13 13:27:49 -05:00
Richard van der Hoff	dc3c83a933	Add jsonschema verification for the oidc provider config	2021-01-13 17:47:27 +00:00
Patrick Cloke	d1eb1b96e8	Register the /devices endpoint on workers. (#9092)	2021-01-13 12:35:40 -05:00
Richard van der Hoff	7cc9509eca	Extract OIDCProviderConfig object ... Collect all the config options which related to an OIDC provider into a single object.	2021-01-13 16:40:02 +00:00
Patrick Cloke	98a64b7f7f	Add basic domain validation for DomainSpecificString.is_valid. (#9071) ... This checks that the domain given to `DomainSpecificString.is_valid` (e.g. `UserID`, `RoomAlias`, etc.) is of a valid form. Previously some validation was done on the localpart (e.g. the sigil), but not the domain portion.	2021-01-13 07:05:16 -05:00
Erik Johnston	aa4d8c1f9a	Merge branch 'master' into develop	2021-01-13 10:36:55 +00:00
Richard van der Hoff	bc4bf7b384	Preparatory refactors of OidcHandler (#9067) ... Some light refactoring of OidcHandler, in preparation for bigger things: * remove inheritance from deprecated BaseHandler * add an object to hold the things that go into a session cookie * factor out a separate class for manipulating said cookies	2021-01-13 10:26:12 +00:00
Erik Johnston	3dd6ba135e	1.25.0	2021-01-13 10:19:12 +00:00
Dirk Klimpel	7a2e9b549d	Remove user's avatar URL and displayname when deactivated. (#8932) ... This only applies if the user's data is to be erased.	2021-01-12 16:30:15 -05:00
Richard van der Hoff	789d9ebad3	UI Auth via SSO: redirect the user to an appropriate SSO. (#9081) ... If we have integrations with multiple identity providers, when the user does a UI Auth, we need to redirect them to the right one. There are a few steps to this. First of all we actually need to store the userid of the user we are trying to validate in the UIA session, since the /auth/sso/fallback/web request is unauthenticated. Then, once we get the /auth/sso/fallback/web request, we can fish the user id out of the session, and use it to look up the external id mappings, and hence pick an SSO provider for them.	2021-01-12 17:38:03 +00:00
Marcus	e385c8b473	Don't apply the IP range blacklist to proxy connections (#9084) ... It is expected that the proxy would be on a private IP address so the configured proxy should be connected to regardless of the IP range blacklist.	2021-01-12 12:20:30 -05:00
Patrick Cloke	723b19748a	Handle bad JSON data being returned from the federation API. (#9070)	2021-01-12 11:07:01 -05:00
Richard van der Hoff	0f8945e166	Kill off HomeServer.get_ip_from_request() (#9080) ... Homeserver.get_ip_from_request() used to be a bit more complicated, but now it is totally redundant. Let's get rid of it.	2021-01-12 12:48:12 +00:00
Richard van der Hoff	2ec8ca5e60	Remove SynapseRequest.get_user_agent (#9069) ... SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests", which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the LoggingContext and write the right entries to the request log). Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a SynapseRequest when there is nothing synapse-specific about the Request at all, and any old twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult. In short: move get_user_agent out to a utility function.	2021-01-12 12:34:16 +00:00
Jason Robinson	c177faf5a9	Remove trailing whitespace to appease the linter ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-11 23:55:44 +02:00
Jason Robinson	49c619a9a2	Simplify delete_forward_extremities_for_room_txn SQL ... As per feedback. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-11 23:49:58 +02:00
Jason Robinson	da16d06301	Address pr feedback ... * docs updates * prettify SQL * add missing copyright * cursor_to_dict * update touched files copyright years Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-11 23:43:58 +02:00
David Teller	b161528fcc	Also support remote users on the joined_rooms admin API. (#8948) ... For remote users, only the rooms which the server knows about are returned. Local users have all of their joined rooms returned.	2021-01-11 14:32:17 -05:00
Erik Johnston	c9195744a4	Move more encryption endpoints off master (#9068)	2021-01-11 18:01:27 +00:00
Erik Johnston	1315a2e8be	Use a chain cover index to efficiently calculate auth chain difference (#8868)	2021-01-11 16:09:22 +00:00
Richard van der Hoff	671138f658	Clean up exception handling in the startup code (#9059) ... Factor out the exception handling in the startup code to a utility function, and fix the some logging and exit code stuff.	2021-01-11 15:55:05 +00:00
Erik Johnston	4e04435bda	Remove old tables after schema version bump (#9055) ... These tables are unused, and can be dropped now the schema version has been bumped.	2021-01-11 13:58:19 +00:00
Erik Johnston	63f4990298	Ensure rejected events get added to some metadata tables (#9016) ... Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2021-01-11 13:57:33 +00:00
Richard van der Hoff	7db2622d30	Remove unused SynapseService (#9058)	2021-01-11 10:24:22 +00:00
Jerin J Titus	c21d8f1c1d	Drop last_used column from access_tokens (#9025) ... * Dropped last_used column from access_tokens Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>	2021-01-11 10:23:49 +00:00
Jason Robinson	b52fb703f7	Don't try to use f-strings ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-11 09:47:03 +02:00
Jason Robinson	2eb421b606	Merge branch 'develop' into jaywink/admin-forward-extremities	2021-01-09 22:00:04 +02:00
Jason Robinson	90ad4d443a	Implement clearing cache after deleting forward extremities ... Also run linter. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-09 21:57:41 +02:00
Erik Johnston	a03d71dc9d	Fix "Starting metrics collection from sentinel context" errors (#9053)	2021-01-08 14:33:53 +00:00
Richard van der Hoff	12f79da587	Merge pull request #9036 from matrix-org/rav/multi_idp/tests ... Add tests for the IdP picker	2021-01-08 14:24:41 +00:00
Richard van der Hoff	d32870ffa5	Fix validate_config on nested objects (#9054)	2021-01-08 14:23:04 +00:00
Erik Johnston	fa5f5cbc74	Fix error handling during insertion of client IPs (#9051) ... You can't continue using a transaction once an exception has been raised, so catching and dropping the error here is pointless and just causes more errors.	2021-01-08 14:15:20 +00:00
Jason Robinson	85c0999bfb	Add Rooms admin forward extremities DELETE endpoint ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-08 00:12:23 +02:00
Jason Robinson	c91045f56c	Move unknown room ID error into resolve_room_id ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-07 23:03:54 +02:00
Jason Robinson	b849e46139	Add forward extremities endpoint to rooms admin API ... GET /_synapse/admin/v1/rooms/<identifier>/forward_extremities now gets forward extremities for a room, returning count and the list of extremities. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-01-07 23:01:59 +02:00
Erik Johnston	b530eaa262	Allow running sendToDevice on workers (#9044)	2021-01-07 20:19:26 +00:00
Erik Johnston	5e99a94502	Support routing edu's to multiple instances (#9042) ... This is in preparation for moving `SendToDeviceServlet` off master	2021-01-07 18:07:28 +00:00
Erik Johnston	e34df813ce	Ensure that remote users' device list resyncing always happens on master (#9043) ... Currently `DeviceMessageHandler` only ever exists on master, but that is about to change.	2021-01-07 18:06:52 +00:00
Erik Johnston	63593134a1	Some cleanups to device inbox store. (#9041)	2021-01-07 17:20:44 +00:00
Richard van der Hoff	8a910f97a4	Add some tests for the IDP picker flow	2021-01-07 14:56:42 +00:00
Patrick Cloke	23d701864f	Improve the performance of calculating ignored users in large rooms (#9024) ... This allows for efficiently finding which users ignore a particular user. Co-authored-by: Erik Johnston <erik@matrix.org>	2021-01-07 13:03:38 +00:00
Erik Johnston	eee3c3c52f	Handle updating schema version without any deltas. (#9033) ... This can happen when using a split out state database and we've upgraded the schema version without there being any changes in the state schema.	2021-01-07 11:33:36 +00:00
Patrick Cloke	1b4d5d6acf	Empty iterables should count towards cache usage. (#9028)	2021-01-06 12:33:20 -05:00
Patrick Cloke	0312266ee3	Synapse 1.25.0rc1 (2021-01-06) ... ============================== Removal warning --------------- The old [Purge Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/purge_room.md) and [Shutdown Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/shutdown_room.md) are deprecated and will be removed in a future release. They will be replaced by the [Delete Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/rooms.md#delete-room-api). `POST /_synapse/admin/v1/rooms/<room_id>/delete` replaces `POST /_synapse/admin/v1/purge_room` and `POST /_synapse/admin/v1/shutdown_room/<room_id>`. Features -------- - Add an admin API that lets server admins get power in rooms in which local users have power. ([\#8756](https://github.com/matrix-org/synapse/issues/8756)) - Add optional HTTP authentication to replication endpoints. ([\#8853](https://github.com/matrix-org/synapse/issues/8853)) - Improve the error messages printed as a result of configuration problems for extension modules. ([\#8874](https://github.com/matrix-org/synapse/issues/8874)) - Add the number of local devices to Room Details Admin API. Contributed by @dklimpel. ([\#8886](https://github.com/matrix-org/synapse/issues/8886)) - Add `X-Robots-Tag` header to stop web crawlers from indexing media. Contributed by Aaron Raimist. ([\#8887](https://github.com/matrix-org/synapse/issues/8887)) - Spam-checkers may now define their methods as `async`. ([\#8890](https://github.com/matrix-org/synapse/issues/8890)) - Add support for allowing users to pick their own user ID during a single-sign-on login. ([\#8897](https://github.com/matrix-org/synapse/issues/8897), [\#8900](https://github.com/matrix-org/synapse/issues/8900), [\#8911](https://github.com/matrix-org/synapse/issues/8911), [\#8938](https://github.com/matrix-org/synapse/issues/8938), [\#8941](https://github.com/matrix-org/synapse/issues/8941), [\#8942](https://github.com/matrix-org/synapse/issues/8942), [\#8951](https://github.com/matrix-org/synapse/issues/8951)) - Add an `email.invite_client_location` configuration option to send a web client location to the invite endpoint on the identity server which allows customisation of the email template. ([\#8930](https://github.com/matrix-org/synapse/issues/8930)) - The search term in the list room and list user Admin APIs is now treated as case-insensitive. ([\#8931](https://github.com/matrix-org/synapse/issues/8931)) - Apply an IP range blacklist to push and key revocation requests. ([\#8821](https://github.com/matrix-org/synapse/issues/8821), [\#8870](https://github.com/matrix-org/synapse/issues/8870), [\#8954](https://github.com/matrix-org/synapse/issues/8954)) - Add an option to allow re-use of user-interactive authentication sessions for a period of time. ([\#8970](https://github.com/matrix-org/synapse/issues/8970)) - Allow running the redact endpoint on workers. ([\#8994](https://github.com/matrix-org/synapse/issues/8994)) Bugfixes -------- - Fix bug where we might not correctly calculate the current state for rooms with multiple extremities. ([\#8827](https://github.com/matrix-org/synapse/issues/8827)) - Fix a long-standing bug in the register admin endpoint (`/_synapse/admin/v1/register`) when the `mac` field was not provided. The endpoint now properly returns a 400 error. Contributed by @edwargix. ([\#8837](https://github.com/matrix-org/synapse/issues/8837)) - Fix a long-standing bug on Synapse instances supporting Single-Sign-On, where users would be prompted to enter their password to confirm certain actions, even though they have not set a password. ([\#8858](https://github.com/matrix-org/synapse/issues/8858)) - Fix a longstanding bug where a 500 error would be returned if the `Content-Length` header was not provided to the upload media resource. ([\#8862](https://github.com/matrix-org/synapse/issues/8862)) - Add additional validation to pusher URLs to be compliant with the specification. ([\#8865](https://github.com/matrix-org/synapse/issues/8865)) - Fix the error code that is returned when a user tries to register on a homeserver on which new-user registration has been disabled. ([\#8867](https://github.com/matrix-org/synapse/issues/8867)) - Fix a bug where `PUT /_synapse/admin/v2/users/<user_id>` failed to create a new user when `avatar_url` is specified. Bug introduced in Synapse v1.9.0. ([\#8872](https://github.com/matrix-org/synapse/issues/8872)) - Fix a 500 error when attempting to preview an empty HTML file. ([\#8883](https://github.com/matrix-org/synapse/issues/8883)) - Fix occasional deadlock when handling SIGHUP. ([\#8918](https://github.com/matrix-org/synapse/issues/8918)) - Fix login API to not ratelimit application services that have ratelimiting disabled. ([\#8920](https://github.com/matrix-org/synapse/issues/8920)) - Fix bug where we ratelimited auto joining of rooms on registration (using `auto_join_rooms` config). ([\#8921](https://github.com/matrix-org/synapse/issues/8921)) - Fix a bug where deactivated users appeared in the user directory when their profile information was updated. ([\#8933](https://github.com/matrix-org/synapse/issues/8933), [\#8964](https://github.com/matrix-org/synapse/issues/8964)) - Fix bug introduced in Synapse v1.24.0 which would cause an exception on startup if both `enabled` and `localdb_enabled` were set to `False` in the `password_config` setting of the configuration file. ([\#8937](https://github.com/matrix-org/synapse/issues/8937)) - Fix a bug where 500 errors would be returned if the `m.room_history_visibility` event had invalid content. ([\#8945](https://github.com/matrix-org/synapse/issues/8945)) - Fix a bug causing common English words to not be considered for a user directory search. ([\#8959](https://github.com/matrix-org/synapse/issues/8959)) - Fix bug where application services couldn't register new ghost users if the server had reached its MAU limit. ([\#8962](https://github.com/matrix-org/synapse/issues/8962)) - Fix a long-standing bug where a `m.image` event without a `url` would cause errors on push. ([\#8965](https://github.com/matrix-org/synapse/issues/8965)) - Fix a small bug in v2 state resolution algorithm, which could also cause performance issues for rooms with large numbers of power levels. ([\#8971](https://github.com/matrix-org/synapse/issues/8971)) - Add validation to the `sendToDevice` API to raise a missing parameters error instead of a 500 error. ([\#8975](https://github.com/matrix-org/synapse/issues/8975)) - Add validation of group IDs to raise a 400 error instead of a 500 eror. ([\#8977](https://github.com/matrix-org/synapse/issues/8977)) Improved Documentation ---------------------- - Fix the "Event persist rate" section of the included grafana dashboard by adding missing prometheus rules. ([\#8802](https://github.com/matrix-org/synapse/issues/8802)) - Combine related media admin API docs. ([\#8839](https://github.com/matrix-org/synapse/issues/8839)) - Fix an error in the documentation for the SAML username mapping provider. ([\#8873](https://github.com/matrix-org/synapse/issues/8873)) - Clarify comments around template directories in `sample_config.yaml`. ([\#8891](https://github.com/matrix-org/synapse/issues/8891)) - Moved instructions for database setup, adjusted heading levels and improved syntax highlighting in [INSTALL.md](../INSTALL.md). Contributed by fossterer. ([\#8987](https://github.com/matrix-org/synapse/issues/8987)) - Update the example value of `group_creation_prefix` in the sample configuration. ([\#8992](https://github.com/matrix-org/synapse/issues/8992)) - Link the Synapse developer room to the development section in the docs. ([\#9002](https://github.com/matrix-org/synapse/issues/9002)) Deprecations and Removals ------------------------- - Deprecate Shutdown Room and Purge Room Admin APIs. ([\#8829](https://github.com/matrix-org/synapse/issues/8829)) Internal Changes ---------------- - Properly store the mapping of external ID to Matrix ID for CAS users. ([\#8856](https://github.com/matrix-org/synapse/issues/8856), [\#8958](https://github.com/matrix-org/synapse/issues/8958)) - Remove some unnecessary stubbing from unit tests. ([\#8861](https://github.com/matrix-org/synapse/issues/8861)) - Remove unused `FakeResponse` class from unit tests. ([\#8864](https://github.com/matrix-org/synapse/issues/8864)) - Pass `room_id` to `get_auth_chain_difference`. ([\#8879](https://github.com/matrix-org/synapse/issues/8879)) - Add type hints to push module. ([\#8880](https://github.com/matrix-org/synapse/issues/8880), [\#8882](https://github.com/matrix-org/synapse/issues/8882), [\#8901](https://github.com/matrix-org/synapse/issues/8901), [\#8940](https://github.com/matrix-org/synapse/issues/8940), [\#8943](https://github.com/matrix-org/synapse/issues/8943), [\#9020](https://github.com/matrix-org/synapse/issues/9020)) - Simplify logic for handling user-interactive-auth via single-sign-on servers. ([\#8881](https://github.com/matrix-org/synapse/issues/8881)) - Skip the SAML tests if the requirements (`pysaml2` and `xmlsec1`) aren't available. ([\#8905](https://github.com/matrix-org/synapse/issues/8905)) - Fix multiarch docker image builds. ([\#8906](https://github.com/matrix-org/synapse/issues/8906)) - Don't publish `latest` docker image until all archs are built. ([\#8909](https://github.com/matrix-org/synapse/issues/8909)) - Various clean-ups to the structured logging and logging context code. ([\#8916](https://github.com/matrix-org/synapse/issues/8916), [\#8935](https://github.com/matrix-org/synapse/issues/8935)) - Automatically drop stale forward-extremities under some specific conditions. ([\#8929](https://github.com/matrix-org/synapse/issues/8929)) - Refactor test utilities for injecting HTTP requests. ([\#8946](https://github.com/matrix-org/synapse/issues/8946)) - Add a maximum size of 50 kilobytes to .well-known lookups. ([\#8950](https://github.com/matrix-org/synapse/issues/8950)) - Fix bug in `generate_log_config` script which made it write empty files. ([\#8952](https://github.com/matrix-org/synapse/issues/8952)) - Clean up tox.ini file; disable coverage checking for non-test runs. ([\#8963](https://github.com/matrix-org/synapse/issues/8963)) - Add type hints to the admin and room list handlers. ([\#8973](https://github.com/matrix-org/synapse/issues/8973)) - Add type hints to the receipts and user directory handlers. ([\#8976](https://github.com/matrix-org/synapse/issues/8976)) - Drop the unused `local_invites` table. ([\#8979](https://github.com/matrix-org/synapse/issues/8979)) - Add type hints to the base storage code. ([\#8980](https://github.com/matrix-org/synapse/issues/8980)) - Support using PyJWT v2.0.0 in the test suite. ([\#8986](https://github.com/matrix-org/synapse/issues/8986)) - Fix `tests.federation.transport.RoomDirectoryFederationTests` and ensure it runs in CI. ([\#8998](https://github.com/matrix-org/synapse/issues/8998)) - Add type hints to the crypto module. ([\#8999](https://github.com/matrix-org/synapse/issues/8999)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl/1rv4ACgkQM/xY9qcR MEiIGw/+PytgC2yOpaR7cIQl0K1v+y4X/AmNbshdQch7GtC3vtKxdmo4PJp4KwY4 Sxiiykq46Xx48w/Hqd3hoC7OqsTyfmNjRKW0InsTarbaXMrSv9umOtScItzMMWq8 SCmDPQ+cF0SP2GhOMwB/N5JpRS12OpdBzHXtcbwxULilTuO/EKD2/W1zq3wiSP7r BxhHHYsx69hufUGC3OGD0I8e26TqjgkLiVm0FeM6cC5TubozO3qrJDlmwTPSpFrL WZEBEbnYUVz4ydMKRq4ALiZzPzxxBfgQmITb79CNqim4+fc+M+AqV6mRez2ULrqh d4/UEMTaA0yh54i0oqei3gwf2ucns/NDftQ9cYwF1+a9LCcWAyJZY7+/9Hk4VKSU KqQUzGdBfxyefsLc2hNGANzS/VbThdejC09IjaxKtHSgaT+bb5LxS8GcbvcG5FxY LyD8yeazgXPub+Ixr6dvD51roCuBtpCYvGZtHHYOiX/S1MYbq22Duet2Z1zhy7qB Jl8r3L32x5wAG3lR8aLqDVNTrC1ur0TW9wUJ88bDjcwn0PsseQG41DHbQSoXQIlk p6/IKjJjfnbKJFTwzfC8RpL29NtyTPqA/d2JDJ7yqUbpEkAnc5GN+pO0B/736aOH aTRu8nVdP7GOoP4whZ9C7r5kYjiCR3jPtpZHRYXaPJtkw57dWS8= =/g7u -----END PGP SIGNATURE----- Merge tag 'v1.25.0rc1' into develop Synapse 1.25.0rc1 (2021-01-06) ============================== Removal warning --------------- The old [Purge Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/purge_room.md) and [Shutdown Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/shutdown_room.md) are deprecated and will be removed in a future release. They will be replaced by the [Delete Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/rooms.md#delete-room-api). `POST /_synapse/admin/v1/rooms/<room_id>/delete` replaces `POST /_synapse/admin/v1/purge_room` and `POST /_synapse/admin/v1/shutdown_room/<room_id>`. Features -------- - Add an admin API that lets server admins get power in rooms in which local users have power. ([\#8756](https://github.com/matrix-org/synapse/issues/8756)) - Add optional HTTP authentication to replication endpoints. ([\#8853](https://github.com/matrix-org/synapse/issues/8853)) - Improve the error messages printed as a result of configuration problems for extension modules. ([\#8874](https://github.com/matrix-org/synapse/issues/8874)) - Add the number of local devices to Room Details Admin API. Contributed by @dklimpel. ([\#8886](https://github.com/matrix-org/synapse/issues/8886)) - Add `X-Robots-Tag` header to stop web crawlers from indexing media. Contributed by Aaron Raimist. ([\#8887](https://github.com/matrix-org/synapse/issues/8887)) - Spam-checkers may now define their methods as `async`. ([\#8890](https://github.com/matrix-org/synapse/issues/8890)) - Add support for allowing users to pick their own user ID during a single-sign-on login. ([\#8897](https://github.com/matrix-org/synapse/issues/8897), [\#8900](https://github.com/matrix-org/synapse/issues/8900), [\#8911](https://github.com/matrix-org/synapse/issues/8911), [\#8938](https://github.com/matrix-org/synapse/issues/8938), [\#8941](https://github.com/matrix-org/synapse/issues/8941), [\#8942](https://github.com/matrix-org/synapse/issues/8942), [\#8951](https://github.com/matrix-org/synapse/issues/8951)) - Add an `email.invite_client_location` configuration option to send a web client location to the invite endpoint on the identity server which allows customisation of the email template. ([\#8930](https://github.com/matrix-org/synapse/issues/8930)) - The search term in the list room and list user Admin APIs is now treated as case-insensitive. ([\#8931](https://github.com/matrix-org/synapse/issues/8931)) - Apply an IP range blacklist to push and key revocation requests. ([\#8821](https://github.com/matrix-org/synapse/issues/8821), [\#8870](https://github.com/matrix-org/synapse/issues/8870), [\#8954](https://github.com/matrix-org/synapse/issues/8954)) - Add an option to allow re-use of user-interactive authentication sessions for a period of time. ([\#8970](https://github.com/matrix-org/synapse/issues/8970)) - Allow running the redact endpoint on workers. ([\#8994](https://github.com/matrix-org/synapse/issues/8994)) Bugfixes -------- - Fix bug where we might not correctly calculate the current state for rooms with multiple extremities. ([\#8827](https://github.com/matrix-org/synapse/issues/8827)) - Fix a long-standing bug in the register admin endpoint (`/_synapse/admin/v1/register`) when the `mac` field was not provided. The endpoint now properly returns a 400 error. Contributed by @edwargix. ([\#8837](https://github.com/matrix-org/synapse/issues/8837)) - Fix a long-standing bug on Synapse instances supporting Single-Sign-On, where users would be prompted to enter their password to confirm certain actions, even though they have not set a password. ([\#8858](https://github.com/matrix-org/synapse/issues/8858)) - Fix a longstanding bug where a 500 error would be returned if the `Content-Length` header was not provided to the upload media resource. ([\#8862](https://github.com/matrix-org/synapse/issues/8862)) - Add additional validation to pusher URLs to be compliant with the specification. ([\#8865](https://github.com/matrix-org/synapse/issues/8865)) - Fix the error code that is returned when a user tries to register on a homeserver on which new-user registration has been disabled. ([\#8867](https://github.com/matrix-org/synapse/issues/8867)) - Fix a bug where `PUT /_synapse/admin/v2/users/<user_id>` failed to create a new user when `avatar_url` is specified. Bug introduced in Synapse v1.9.0. ([\#8872](https://github.com/matrix-org/synapse/issues/8872)) - Fix a 500 error when attempting to preview an empty HTML file. ([\#8883](https://github.com/matrix-org/synapse/issues/8883)) - Fix occasional deadlock when handling SIGHUP. ([\#8918](https://github.com/matrix-org/synapse/issues/8918)) - Fix login API to not ratelimit application services that have ratelimiting disabled. ([\#8920](https://github.com/matrix-org/synapse/issues/8920)) - Fix bug where we ratelimited auto joining of rooms on registration (using `auto_join_rooms` config). ([\#8921](https://github.com/matrix-org/synapse/issues/8921)) - Fix a bug where deactivated users appeared in the user directory when their profile information was updated. ([\#8933](https://github.com/matrix-org/synapse/issues/8933), [\#8964](https://github.com/matrix-org/synapse/issues/8964)) - Fix bug introduced in Synapse v1.24.0 which would cause an exception on startup if both `enabled` and `localdb_enabled` were set to `False` in the `password_config` setting of the configuration file. ([\#8937](https://github.com/matrix-org/synapse/issues/8937)) - Fix a bug where 500 errors would be returned if the `m.room_history_visibility` event had invalid content. ([\#8945](https://github.com/matrix-org/synapse/issues/8945)) - Fix a bug causing common English words to not be considered for a user directory search. ([\#8959](https://github.com/matrix-org/synapse/issues/8959)) - Fix bug where application services couldn't register new ghost users if the server had reached its MAU limit. ([\#8962](https://github.com/matrix-org/synapse/issues/8962)) - Fix a long-standing bug where a `m.image` event without a `url` would cause errors on push. ([\#8965](https://github.com/matrix-org/synapse/issues/8965)) - Fix a small bug in v2 state resolution algorithm, which could also cause performance issues for rooms with large numbers of power levels. ([\#8971](https://github.com/matrix-org/synapse/issues/8971)) - Add validation to the `sendToDevice` API to raise a missing parameters error instead of a 500 error. ([\#8975](https://github.com/matrix-org/synapse/issues/8975)) - Add validation of group IDs to raise a 400 error instead of a 500 eror. ([\#8977](https://github.com/matrix-org/synapse/issues/8977)) Improved Documentation ---------------------- - Fix the "Event persist rate" section of the included grafana dashboard by adding missing prometheus rules. ([\#8802](https://github.com/matrix-org/synapse/issues/8802)) - Combine related media admin API docs. ([\#8839](https://github.com/matrix-org/synapse/issues/8839)) - Fix an error in the documentation for the SAML username mapping provider. ([\#8873](https://github.com/matrix-org/synapse/issues/8873)) - Clarify comments around template directories in `sample_config.yaml`. ([\#8891](https://github.com/matrix-org/synapse/issues/8891)) - Moved instructions for database setup, adjusted heading levels and improved syntax highlighting in [INSTALL.md](../INSTALL.md). Contributed by fossterer. ([\#8987](https://github.com/matrix-org/synapse/issues/8987)) - Update the example value of `group_creation_prefix` in the sample configuration. ([\#8992](https://github.com/matrix-org/synapse/issues/8992)) - Link the Synapse developer room to the development section in the docs. ([\#9002](https://github.com/matrix-org/synapse/issues/9002)) Deprecations and Removals ------------------------- - Deprecate Shutdown Room and Purge Room Admin APIs. ([\#8829](https://github.com/matrix-org/synapse/issues/8829)) Internal Changes ---------------- - Properly store the mapping of external ID to Matrix ID for CAS users. ([\#8856](https://github.com/matrix-org/synapse/issues/8856), [\#8958](https://github.com/matrix-org/synapse/issues/8958)) - Remove some unnecessary stubbing from unit tests. ([\#8861](https://github.com/matrix-org/synapse/issues/8861)) - Remove unused `FakeResponse` class from unit tests. ([\#8864](https://github.com/matrix-org/synapse/issues/8864)) - Pass `room_id` to `get_auth_chain_difference`. ([\#8879](https://github.com/matrix-org/synapse/issues/8879)) - Add type hints to push module. ([\#8880](https://github.com/matrix-org/synapse/issues/8880), [\#8882](https://github.com/matrix-org/synapse/issues/8882), [\#8901](https://github.com/matrix-org/synapse/issues/8901), [\#8940](https://github.com/matrix-org/synapse/issues/8940), [\#8943](https://github.com/matrix-org/synapse/issues/8943), [\#9020](https://github.com/matrix-org/synapse/issues/9020)) - Simplify logic for handling user-interactive-auth via single-sign-on servers. ([\#8881](https://github.com/matrix-org/synapse/issues/8881)) - Skip the SAML tests if the requirements (`pysaml2` and `xmlsec1`) aren't available. ([\#8905](https://github.com/matrix-org/synapse/issues/8905)) - Fix multiarch docker image builds. ([\#8906](https://github.com/matrix-org/synapse/issues/8906)) - Don't publish `latest` docker image until all archs are built. ([\#8909](https://github.com/matrix-org/synapse/issues/8909)) - Various clean-ups to the structured logging and logging context code. ([\#8916](https://github.com/matrix-org/synapse/issues/8916), [\#8935](https://github.com/matrix-org/synapse/issues/8935)) - Automatically drop stale forward-extremities under some specific conditions. ([\#8929](https://github.com/matrix-org/synapse/issues/8929)) - Refactor test utilities for injecting HTTP requests. ([\#8946](https://github.com/matrix-org/synapse/issues/8946)) - Add a maximum size of 50 kilobytes to .well-known lookups. ([\#8950](https://github.com/matrix-org/synapse/issues/8950)) - Fix bug in `generate_log_config` script which made it write empty files. ([\#8952](https://github.com/matrix-org/synapse/issues/8952)) - Clean up tox.ini file; disable coverage checking for non-test runs. ([\#8963](https://github.com/matrix-org/synapse/issues/8963)) - Add type hints to the admin and room list handlers. ([\#8973](https://github.com/matrix-org/synapse/issues/8973)) - Add type hints to the receipts and user directory handlers. ([\#8976](https://github.com/matrix-org/synapse/issues/8976)) - Drop the unused `local_invites` table. ([\#8979](https://github.com/matrix-org/synapse/issues/8979)) - Add type hints to the base storage code. ([\#8980](https://github.com/matrix-org/synapse/issues/8980)) - Support using PyJWT v2.0.0 in the test suite. ([\#8986](https://github.com/matrix-org/synapse/issues/8986)) - Fix `tests.federation.transport.RoomDirectoryFederationTests` and ensure it runs in CI. ([\#8998](https://github.com/matrix-org/synapse/issues/8998)) - Add type hints to the crypto module. ([\#8999](https://github.com/matrix-org/synapse/issues/8999))	2021-01-06 07:42:38 -05:00
Patrick Cloke	0248409bfa	Handle a display name / avatar URL not included in a federation request. (#9023) ... These may be omitted if not set, but Synapse assumed they would be in the response.	2021-01-06 07:29:03 -05:00
Patrick Cloke	2fe0fb21f6	1.25.0rc1	2021-01-06 07:08:13 -05:00
Patrick Cloke	37eaf9c272	Fix-up assertions about last stream token in push (#9020) ... The last stream token is always known and we do not need to handle none.	2021-01-05 10:53:15 -05:00
Patrick Cloke	06fefe0bb1	Add type hints to the logging context code. (#8939)	2021-01-05 08:06:55 -05:00
Patrick Cloke	9dde9c9f01	Implement MSC2176: Updated redaction rules (#8984) ... An experimental room version ("org.matrix.msc2176") contains the new redaction rules for testing.	2021-01-05 07:41:48 -05:00
Richard van der Hoff	111b673fc1	Add initial support for a "pick your IdP" page (#9017) ... During login, if there are multiple IdPs enabled, offer the user a choice of IdPs.	2021-01-05 11:25:28 +00:00
Richard van der Hoff	d2c616a413	Combine the SSO Redirect Servlets (#9015) ... * Implement CasHandler.handle_redirect_request ... to make it match OidcHandler and SamlHandler * Clean up interface for OidcHandler.handle_redirect_request Make it accept `client_redirect_url=None`. * Clean up interface for `SamlHandler.handle_redirect_request` ... bring it into line with CAS and OIDC by making it take a Request parameter, move the magic for `client_redirect_url` for UIA into the handler, and fix the return type to be a `str` rather than a `bytes`. * Define a common protocol for SSO auth provider impls * Give SsoIdentityProvider an ID and register them * Combine the SSO Redirect servlets Now that the SsoHandler knows about the identity providers, we can combine the various *RedirectServlets into a single implementation which delegates to the right IdP. * changelog	2021-01-04 18:13:49 +00:00
Patrick Cloke	31b1905e13	Add type hints to the receipts and user directory handlers. (#8976)	2021-01-04 10:05:12 -05:00
Patrick Cloke	1c9a850562	Add type hints to the crypto module. (#8999)	2021-01-04 10:04:50 -05:00
Patrick Cloke	0eccf53146	Use the SSO handler helpers for CAS registration/login. (#8856)	2021-01-03 16:25:44 +00:00
Patrick Cloke	b7c580e333	Check if group IDs are valid before using them. (#8977)	2020-12-30 08:39:59 -05:00