2564 Commits

Author	SHA1	Message	Date
Patrick Cloke	4054dfa409	Add type hints for event streams. (#10856)	2021-09-21 13:34:26 -04:00
Patrick Cloke	ba7a91aea5	Refactor oEmbed previews (#10814) ... The major change is moving the decision of whether to use oEmbed further up the call-stack. This reverts the _download_url method to being a "dumb" functionwhich takes a single URL and downloads it (as it was before #7920). This also makes more minor refactorings: * Renames internal variables for clarity. * Factors out shared code between the HTML and rich oEmbed previews. * Fixes tests to preview an oEmbed image.	2021-09-21 16:09:57 +00:00
Brendan Abolivier	2843058a8b	Test that state events sent by modules correctly end up in the room's state (#10835) ... Test for #10830 Ideally the test would also make sure the new state event comes down sync, but this is probably good enough.	2021-09-21 17:40:20 +02:00
Hillery Shay	5fca3c8ae6	Allow Synapse Admin API's Room Search to accept non-ASCII characters (#10859) ... * add tests for checking if room search works with non-ascii char * change encoding on parse_string to UTF-8 * lints * properly encode search term * lints * add changelog file * update changelog number * set changelog entry filetype to .bugfix * Revert "set changelog entry filetype to .bugfix" This reverts commit be8e5a314251438ec4ec7dbc59ba32162c93e550. * update changelog message and file type * change parse_string default encoding back to ascii and update room search admin api calll to parse string * refactor tests * Update tests/rest/admin/test_room.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2021-09-21 08:04:35 -07:00
David Robertson	60453315bd	Always add local users to the user directory (#10796) ... It's a simplification, but one that'll help make the user directory logic easier to follow with the other changes upcoming. It's not strictly required for those changes, but this will help simplify the resulting logic that listens for `m.room.member` events and generally make the logic easier to follow. This means the config option `search_all_users` ends up controlling the search query only, and not the data we store. The cost of doing so is an extra row in the `user_directory` and `user_directory_search` tables for each local user which - belongs to no public rooms - belongs to no private rooms of size ≥ 2 I think the cost of this will be marginal (since they'll already have entries in `users` and `profiles` anyway). As a small upside, a homeserver whose directory was built with this change can toggle `search_all_users` without having to rebuild their directory. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2021-09-21 12:02:34 +00:00
Aaron Raimist	6a751ff5e0	Allow sending a membership event to unban a user (#10807) ... * Allow membership event to unban user Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-09-21 10:23:34 +00:00
Patrick Cloke	bfb4b858a9	Create a constant for a small png image in tests. (#10834) ... To avoid duplicating it between a few tests.	2021-09-16 12:01:14 -04:00
reivilibre	8eb7cb2e0d	Make StateFilter frozen so we can hash it (#10816) ... Also enables Mypy for related tests.	2021-09-14 16:35:53 +01:00
Patrick Cloke	01c88a09cd	Use direct references for some configuration variables (#10798) ... Instead of proxying through the magic getter of the RootConfig object. This should be more performant (and is more explicit).	2021-09-13 13:07:12 -04:00
reivilibre	524b8ead77	Add types to synapse.util. (#10601)	2021-09-10 17:03:18 +01:00
Patrick Cloke	63f28e4a0c	Handle room upgrades for spaces (#10774) ... By copying the `room_type` field of the create event and migrating any non-empty `m.space.child` events to the new room that is created.	2021-09-10 07:30:05 -04:00
David Robertson	318162f5de	Easy refactors of the user directory (#10789) ... No functional changes here. This came out as I was working to tackle #5677	2021-09-10 10:54:38 +01:00
Sean	273b6861f2	Remove unstable MSC2858 API, including experimental.msc2858_enabled config option (#10693) ... Signed-off-by: Sean Quah <seanq@element.io>	2021-09-09 17:59:59 +01:00
Richard van der Hoff	abedf7d77f	Get rid of _auth_and_persist_event (#10781) ... This is only called in two places, and the code seems much clearer without it.	2021-09-08 19:03:08 +01:00
Erik Johnston	74f01e11c9	Skip handling of push actions for outlier events (#10780) ... Outlier events don't ever have push actions associated with them, so we can skip some expensive queries during event persistence.	2021-09-08 15:18:35 +01:00
Patrick Cloke	580a15e039	Request JSON for oEmbed requests (and ignore XML only providers). (#10759) ... This adds the format to the request arguments / URL to ensure that JSON data is returned (which is all that Synapse supports). This also adds additional error checking / filtering to the configuration file to ignore XML-only providers.	2021-09-08 07:17:52 -04:00
Brendan Abolivier	287108fb2e	Merge tag 'v1.42.0rc2' into develop ... Synapse 1.42.0rc2 (2021-09-06) ============================== This version of Synapse removes deprecated room-management admin APIs, removes out-of-date email pushers, and improves error handling for fallback templates for user-interactive authentication. For more information on these points, server administrators are encouraged to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420). Features -------- - Support room version 9 from [MSC3375](https://github.com/matrix-org/matrix-doc/pull/3375). ([\#10747](https://github.com/matrix-org/synapse/issues/10747)) Internal Changes ---------------- - Print a warning when using one of the deprecated `template_dir` settings. ([\#10768](https://github.com/matrix-org/synapse/issues/10768))	2021-09-06 16:29:59 +01:00
Azrenbeth	6e895366ea	Add config option to use non-default manhole password and keys (#10643)	2021-09-06 16:08:03 +01:00
David Robertson	5e9b382505	Pull out encrypted_by_default tests from user_directory tests (#10752)	2021-09-06 11:37:54 +01:00
Patrick Cloke	ae3c16318b	Support MSC3375: room version 9. (#10747)	2021-09-03 12:51:15 -04:00
Patrick Cloke	c586d6803a	Ignore rooms with unknown room versions in the spaces summary. (#10727) ... This avoids breaking the entire endpoint if a room with an unsupported room version is encountered.	2021-09-01 17:01:08 +00:00
Patrick Cloke	6258730ebe	Consider the origin_server_ts of the m.space.child event when ordering rooms. (#10730) ... This updates the ordering of the returned events from the spaces summary API to that defined in MSC2946 (which updates MSC1772). Previously a step was skipped causing ordering to be inconsistent with clients.	2021-09-01 12:59:52 -04:00
Eric Eastwood	dc75fb7f05	Populate rooms.creator field for easy lookup (#10697) ... Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values	2021-09-01 16:27:58 +01:00
Olivier Wilkinson (reivilibre)	e059094119	Synapse 1.42.0rc1 (2021-09-01) ... ============================== Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420). Features -------- - Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142)) - Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452)) - Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524)) - Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725)) Bugfixes -------- - Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232)) - Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561)) - Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734)) - Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593)) - Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644)) - Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654)) - Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677)) - Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679)) - Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684)) - Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703)) - Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713)) Improved Documentation ---------------------- - Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192)) - Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595)) - Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639)) - Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708)) - Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711)) Deprecations and Removals ------------------------- - Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830)) Internal Changes ---------------- - Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608)) - Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640)) - Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621)) - Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627)) - Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642)) - Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630)) - Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645)) - Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647)) - Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672)) - Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662)) - Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664)) - Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674)) - Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667)) - Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686)) - Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692)) - Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706)) -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmEvd3cACgkQLS76LzL7 4EcF0hAAvOjDeH2hn0eI4YX03/BTCesJ5pCTbEJeC9Hsui3SbWamyYarH0Ae0gM6 BP9x5/XkCdsZhKT2OJedtZQUQ0cUws5FdJP87NnE1m7KBW9djbTxmkue3JKaHNTT ajvEoLe8UjOFbjvyMc+188uWVgtKQ6KLcetwllTpYY3FJ3+yk9CQP1ydYzwfHcHe hXW3b20WoRdAmH0B27mj1oQzCeXclbj1kCthcBcxH5RytA8bBc1itBtpAP2FFaeS Ml6paoXSs0wZaXXL5T1MhQtcVOOG5Yh7EPsemdyfqkwhIk9QtV3nmGfC7K5iEeq0 DldTgBNE0wLO+ycdgEZ2oCvitCLT4WDoKkVqKPA91wpuvCMeK5lnDdqbFUSN3hfd l8Oz+ydhjsCx7+pBBopnsyGrPIHNaN/vySYGF3lPa3ANKS2J0IVlAaJ59bZqzg2C vzdyMBCnn0fhnXMkADPLnIYqtebvC09JlPBRGo5BKmwmlM8Ea1dLQ+Lcy5vXalcw +GaYw3U805c03EvsWQxWchjwq6IwMeP06YpK/1ruMjlrCWYJp3V7f/4hXjlTdYm5 OEWc/kHOnlLfH2cA4aS0sV5WsnEuftotQIzBXuXWw5hSMdFmH7eAviPg0vXgfeA/ i0FOkOgEYR8O3RMh4Cip8+h44t8SPNjYUFJK9ip3DmNateeUCH0= =XxMc -----END PGP SIGNATURE----- Merge tag 'v1.42.0rc1' into develop Synapse 1.42.0rc1 (2021-09-01) ============================== Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420). Features -------- - Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142)) - Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452)) - Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524)) - Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725)) Bugfixes -------- - Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232)) - Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561)) - Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734)) - Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593)) - Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644)) - Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654)) - Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677)) - Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679)) - Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684)) - Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703)) - Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713)) Improved Documentation ---------------------- - Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192)) - Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595)) - Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639)) - Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708)) - Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711)) Deprecations and Removals ------------------------- - Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830)) Internal Changes ---------------- - Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608)) - Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640)) - Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621)) - Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627)) - Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642)) - Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630)) - Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645)) - Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647)) - Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672)) - Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662)) - Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664)) - Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674)) - Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667)) - Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686)) - Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692)) - Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))	2021-09-01 14:58:14 +01:00
David Robertson	d9069388f3	Correctly include room avatars in email notifications (#10658) ... Judging by the template, this was intended ages ago, but we never actually passed an avatar URL to the template. So let's provide one. Closes #1546. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>	2021-09-01 13:48:41 +01:00
Andrew Morgan	3693ea61f5	Fix iteration in _remove_deleted_email_pushers background job. (#10734)	2021-09-01 09:13:01 +00:00
Patrick Cloke	e2481dbe93	Allow configuration of the oEmbed URLs. (#10714) ... This adds configuration options (under an `oembed` section) to configure which URLs are matched to use oEmbed for URL previews.	2021-08-31 18:37:07 -04:00
Richard van der Hoff	5d9e7e0c71	Merge branch 'master' into develop	2021-08-31 14:09:48 +01:00
Richard van der Hoff	8c26f16c76	Fix up unit tests (#10723) ... These were broken in an incorrect merge of GHSA-jj53-8fmw-f2w2 (cb35df9)	2021-08-31 12:56:22 +01:00
reivilibre	cb35df940a	Merge pull request from GHSA-jj53-8fmw-f2w2	2021-08-31 11:24:09 +01:00
reivilibre	52c7a51cfc	Merge pull request from GHSA-3x4c-pq33-4w3q ... * Add some tests to characterise the problem Some failing. Current states: RoomsMemberListTestCase test_get_member_list ... [OK] test_get_member_list_mixed_memberships ... [OK] test_get_member_list_no_permission ... [OK] test_get_member_list_no_permission_former_member ... [OK] test_get_member_list_no_permission_former_member_with_at_token ... [FAIL] test_get_member_list_no_room ... [OK] test_get_member_list_no_permission_with_at_token ... [FAIL] * Correct the tests * Check user is/was member before divulging room membership * Pull out only the 1 membership event we want. * Update tests/rest/client/v1/test_rooms.py Co-authored-by: Erik Johnston <erik@matrix.org> * Fixup tests (following apply review suggestion) Co-authored-by: Erik Johnston <erik@matrix.org>	2021-08-31 10:09:58 +01:00
Richard van der Hoff	e3abc0a5cc	Merge remote-tracking branch 'origin/release-v1.41' into develop	2021-08-27 16:35:02 +01:00
Richard van der Hoff	8f98260552	Fix incompatibility with Twisted < 21. (#10713) ... Turns out that the functionality added in #10546 to skip TLS was incompatible with older Twisted versions, so we need to be a bit more inventive. Also, add a test to (hopefully) not break this in future. Sadly, testing TLS is really hard.	2021-08-27 16:33:41 +01:00
Dirk Klimpel	e62cdbef1a	Improve ServerNoticeServlet to avoid duplicate requests (#10679) ... Fixes: #9544	2021-08-27 09:16:40 +00:00
Richard van der Hoff	1800aabfc2	Split FederationHandler in half (#10692) ... The idea here is to take anything to do with incoming events and move it out to a separate handler, as a way of making FederationHandler smaller.	2021-08-26 21:41:44 +01:00
Aaron Raimist	40f619eaa5	Validate new m.room.power_levels events (#10232) ... Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-08-26 17:07:58 +01:00
Azrenbeth	ad17fbd20e	Remove pushers when deleting 3pid from account (#10581) ... When a user deletes an email from their account it will now also remove all pushers for that email and that user (even if these pushers were created by a different client)	2021-08-26 13:53:57 +01:00
Sean	7367473f96	Fix error when selecting between thumbnails with the same quality (#10684) ... Fixes #10318	2021-08-25 09:51:08 +00:00
Erik Johnston	cd22fb568a	Merge branch 'master' into develop	2021-08-24 17:13:29 +01:00
Callum Brown	947dbbdfd1	Implement MSC3231: Token authenticated registration (#10142) ... Signed-off-by: Callum Brown <callum@calcuode.com> This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).	2021-08-21 22:14:43 +01:00
David Robertson	ecd823d766	Flatten tests/rest/client/{v1,v2_alpha} too (#10667)	2021-08-20 17:50:44 +01:00
Dirk Klimpel	f499dc38bc	Simplify tests for the device admin rest API. (#10664) ... By replacing duplicated code with parameterized tests and avoiding unnecessary dumping of JSON data.	2021-08-20 15:43:26 +00:00
David Robertson	7862d704fd	Follow-up: format changelog, add licence (#10593) ... Merged before approval; these comments from @clokep on that PR.	2021-08-20 16:33:52 +01:00
David Robertson	ee3b2ac59a	Validate device_keys for C-S /keys/query requests (#10593) ... * Validate device_keys for C-S /keys/query requests Closes #10354 A small, not particularly critical fix. I'm interested in seeing if we can find a more systematic approach though. #8445 is the place for any discussion.	2021-08-20 15:47:03 +01:00
Patrick Cloke	5cda75fede	Set room version 8 as preferred for restricted rooms. (#10571)	2021-08-20 07:17:50 -04:00
Richard van der Hoff	e81d62009e	Split on_receive_pdu in half (#10640) ... Here we split on_receive_pdu into two functions (on_receive_pdu and process_pulled_event), rather than having both cases in the same method. There's a tiny bit of overlap, but not that much.	2021-08-19 17:05:12 +00:00
Patrick Cloke	000aa89be6	Do not include rooms with an unknown room version in a sync response. (#10644) ... A user will still see this room if it is in a local cache, but it will not reappear if clearing the cache and reloading.	2021-08-19 11:12:55 -04:00
Dirk Klimpel	b5fef6054a	Support MSC3283: Expose enable_set_displayname in capabilities (#10452)	2021-08-19 09:40:40 +00:00
Dirk Klimpel	220f901229	Remove not needed database updates in modify user admin API (#10627)	2021-08-19 10:25:05 +01:00
Patrick Cloke	bec01c0758	Convert room member storage tuples to attrs. (#10629) ... Instead of using namedtuples. This helps with asserting type hints and code completion.	2021-08-18 09:22:07 -04:00
Dirk Klimpel	430241a1e9	Remove deprecated Shutdown Room and Purge Room Admin API (#8830)	2021-08-17 21:19:13 +01:00
Azrenbeth	1a9f531c79	Port the PresenceRouter module interface to the new generic interface (#10524) ... Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0	2021-08-17 13:22:45 +00:00
reivilibre	642a42edde	Flatten the synapse.rest.client package (#10600)	2021-08-17 11:57:58 +00:00
Dirk Klimpel	3bcd525b46	Allow to edit external_ids by Edit User admin API (#10598) ... Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-08-17 11:56:11 +01:00
Brendan Abolivier	ae2714c1f3	Allow using several custom template directories (#10587) ... Allow using several directories in read_templates.	2021-08-17 10:23:14 +00:00
Will Hunt	a933c2c7d8	Add an admin API to check if a username is available (#10578) ... This adds a new API GET /_synapse/admin/v1/username_available?username=foo to check if a username is available. It is the counterpart to https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available, except that it works even if registration is disabled.	2021-08-17 10:52:38 +01:00
Michael Telatynski	0ace38b7b3	Experimental support for MSC3266 Room Summary API. (#10394)	2021-08-16 14:49:12 +00:00
Patrick Cloke	7de445161f	Support federation in the new spaces summary API (MSC2946). (#10569)	2021-08-16 08:06:17 -04:00
Šimon Brandner	a3a7514570	Handle string read receipt data (#10606) ... * Handle string read receipt data Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Test that we handle string read receipt data Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add changelog for #10606 Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Add docs Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Ignore malformed RRs Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Only surround hidden = ... Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Remove unnecessary argument Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com> * Update changelog.d/10606.bugfix Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2021-08-16 12:22:38 +01:00
Patrick Cloke	98a3355d9a	Update the pagination parameter name based on MSC2946 review. (#10579)	2021-08-11 15:44:45 -04:00
Dirk Klimpel	915b37e5ef	Admin API to delete media for a specific user (#10558)	2021-08-11 19:29:59 +00:00
Patrick Cloke	3ebb6694f0	Allow requesting the summary of a space which is joinable. (#10580) ... As opposed to only allowing the summary of spaces which the user is already in or has world-readable visibility. This makes the logic consistent with whether a space/room is returned as part of a space and whether a space summary can start at a space.	2021-08-11 15:04:51 -04:00
Patrick Cloke	fab352ac2c	Fix type hints in space summary tests. (#10575) ... And ensure that the file is checked via mypy.	2021-08-11 10:43:40 -04:00
Dirk Klimpel	339c3918e1	support federation queries through http connect proxy (#10475) ... Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu> Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-08-11 15:34:59 +01:00
Patrick Cloke	fe1d0c8618	Add local support for the new spaces summary endpoint (MSC2946) (#10549) ... This adds support for the /hierarchy endpoint, which is an update to MSC2946. Currently this only supports rooms known locally to the homeserver.	2021-08-10 13:08:17 -04:00
Patrick Cloke	691593bf71	Fix an edge-case with invited rooms over federation in the spaces summary. (#10560) ... If a room which the requesting user was invited to was queried over federation it will now properly appear in the spaces summary (instead of being stripped out by the requesting server).	2021-08-10 14:56:54 +00:00
Dirk Klimpel	6b61debf5c	Do not remove status_msg when user going offline (#10550) ... Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-08-09 16:21:04 +00:00
Brendan Abolivier	f8e86b7d2e	Merge branch 'release-v1.40' into develop	2021-08-09 14:44:28 +01:00
Patrick Cloke	0c246dd4a0	Support MSC3289: Room version 8 (#10449) ... This adds support for MSC3289: room version 8. This is room version 7 + MSC3083.	2021-08-09 10:46:39 +02:00
Erik Johnston	60f0534b6e	Fix exceptions in logs when failing to get remote room list (#10541)	2021-08-06 14:05:41 +01:00
Richard van der Hoff	1bebc0b78c	Clean up federation event auth code (#10539) ... * drop old-room hack pretty sure we don't need this any more. * Remove incorrect comment about modifying `context` It doesn't look like the supplied context is ever modified. * Stop `_auth_and_persist_event` modifying its parameters This is only called in three places. Two of them don't pass `auth_events`, and the third doesn't use the dict after passing it in, so this should be non-functional. * Stop `_check_event_auth` modifying its parameters `_check_event_auth` is only called in three places. `on_send_membership_event` doesn't pass an `auth_events`, and `prep` and `_auth_and_persist_event` do not use the map after passing it in. * Stop `_update_auth_events_and_context_for_auth` modifying its parameters Return the updated auth event dict, rather than modifying the parameter. This is only called from `_check_event_auth`. * Improve documentation on `_auth_and_persist_event` Rename `auth_events` parameter to better reflect what it contains. * Improve documentation on `_NewEventInfo` * Improve documentation on `_check_event_auth` rename `auth_events` parameter to better describe what it contains * changelog	2021-08-06 13:54:23 +01:00
Richard van der Hoff	74d7336686	Add a setting to disable TLS for sending email (#10546) ... This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.	2021-08-06 10:13:34 +00:00
Patrick Cloke	3b354faad0	Refactoring before implementing the updated spaces summary. (#10527) ... This should have no user-visible changes, but refactors some pieces of the SpaceSummaryHandler before adding support for the updated MSC2946.	2021-08-05 12:39:17 +00:00
Will Hunt	a8a27b2b8b	Only return an appservice protocol if it has a service providing it. (#10532) ... If there are no services providing a protocol, omit it completely instead of returning an empty dictionary. This fixes a long-standing spec compliance bug.	2021-08-05 08:22:14 -04:00
Erik Johnston	a36d77c563	Merge tag 'v1.40.0rc2' into develop ... Synapse 1.40.0rc2 (2021-08-04) ============================== Bugfixes -------- - Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\#10517](https://github.com/matrix-org/synapse/issues/10517)) - Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\#10531](https://github.com/matrix-org/synapse/issues/10531)) Internal Changes ---------------- - Fix release script to open the correct URL for the release. ([\#10516](https://github.com/matrix-org/synapse/issues/10516))	2021-08-05 11:15:29 +01:00
Brendan Abolivier	e8a3e81402	Don't fail on empty bodies when sending out read receipts (#10531) ... Fixes a bug introduced in rc1 that would cause Synapse to 400 on read receipts requests with empty bodies. Broken in #10413	2021-08-04 16:13:24 +02:00
Erik Johnston	c37dad67ab	Improve event caching code (#10119) ... Ensure we only load an event from the DB once when the same event is requested multiple times at once.	2021-08-04 13:54:51 +01:00
Jason Robinson	c2000ab35b	Add get_userinfo_by_id method to ModuleApi (#9581) ... Makes it easier to fetch user details in for example spam checker modules, without needing to use api._store or figure out database interactions. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2021-08-04 10:40:25 +00:00
reivilibre	fb086edaed	Fix codestyle CI from #10440 (#10511) ... Co-authored-by: Erik Johnston <erik@matrix.org>	2021-08-02 15:50:22 +00:00
Erik Johnston	01d45fe964	Prune inbound federation queues if they get too long (#10390)	2021-08-02 13:37:25 +00:00
Toni Spets	ba5287f5e8	Allow setting transaction limit for db connections (#10440) ... Setting the value will help PostgreSQL free up memory by recycling the connections in the connection pool. Signed-off-by: Toni Spets <toni.spets@iki.fi>	2021-08-02 13:24:43 +00:00
Eric Eastwood	db6e7f15ea	Fix backfilled events being rejected for no state_groups (#10439) ... Reproducible on a federated homeserver when there is a membership auth event as a floating outlier. Then when we try to backfill one of that persons messages, it has missing membership auth to fetch which caused us to mistakenly replace the `context` for the message with that of the floating membership `outlier` event. Since `outliers` have no `state` or `state_group`, the error bubbles up when we continue down the persisting route: `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group` Call stack: ``` backfill _auth_and_persist_event _check_event_auth _update_auth_events_and_context_for_auth ```	2021-07-29 09:46:51 +01:00
Erik Johnston	8c201c97ec	Synapse 1.39.0rc3 (2021-07-28) ... ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465)) -----BEGIN PGP SIGNATURE----- iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmEBUJUQHGVyaWtAbWF0 cml4Lm9yZwAKCRClQuTtGw+sCWi6CACfA3B7rXfQIO893mctSZkNhtAG/e4o310M etvjAtovKmKeFqBvm89FPmFoXvGA2ypoz7Jz2fdkP56DNKIXvihnDc8EeQ3gGawf hQ8GBjICOL1e7R/4qzuOCWYdppBGoGNAbz7qyxAUxZ/XOzsPJ2VBGkgyXWZkyPe/ nJYsuMuMp117HUwTqPqs3oZuTN0MRTy6xgaDfbFbcX9UC/amLmGIWOhlme8iV15r HQAW0X7F2Un5h7eQwogWTDb9gBHKnJ4ApaSJLfZl9E72i0Sv7o9UFcLIdy2DRVSq gxlRBVzVU6ZndCQ4vQXFSjQ20VGNXc2vpKVRE1D6YXxmxVNbBQB4 =lb6H -----END PGP SIGNATURE----- Merge tag 'v1.39.0rc3' into develop Synapse 1.39.0rc3 (2021-07-28) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477)) - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485)) - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486)) Internal Changes ---------------- - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461)) - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))	2021-07-28 15:53:53 +01:00
Patrick Cloke	752fe0cd98	Restricted rooms (MSC3083) should not have their allow key redacted. (#10489)	2021-07-28 07:03:01 -04:00
Šimon Brandner	c3b037795a	Support for MSC2285 (hidden read receipts) (#10413) ... Implementation of matrix-org/matrix-doc#2285	2021-07-28 10:05:11 +02:00
sri-vidyut	8e1febc6a1	Support underscores (in addition to hyphens) for charset detection. (#10410)	2021-07-27 17:29:42 +00:00
Dirk Klimpel	076deade02	allow specifying https:// proxy (#10411)	2021-07-27 17:31:06 +01:00
Andrew Morgan	74d09a43d9	Always communicate device OTK counts to clients (#10485) ... Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2021-07-27 14:36:38 +01:00
Travis Ralston	b3a757eb3b	Support MSC2033: Device ID on whoami (#9918) ... * Fix no-access-token bug in deactivation tests * Support MSC2033: Device ID on whoami * Test for appservices too MSC: https://github.com/matrix-org/matrix-doc/pull/2033 The MSC has passed FCP, which means stable endpoints can be used.	2021-07-27 05:28:20 +00:00
Patrick Cloke	228decfce1	Update the MSC3083 support to verify if joins are from an authorized server. (#10254)	2021-07-26 12:17:00 -04:00
Dirk Klimpel	89c4ca81bb	Add creation_ts to list users admin API (#10448) ... Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-07-22 16:05:16 +02:00
Patrick Cloke	590cc4e888	Add type hints to additional servlet functions (#10437) ... Improves type hints for: * parse_{boolean,integer} * parse_{boolean,integer}_from_args * parse_json_{value,object}_from_request And fixes any incorrect calls that resulted from unknown types.	2021-07-21 18:12:22 +00:00
Andrew Morgan	b181dc402d	Synapse 1.39.0rc1 (2021-07-20) ... ============================== The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. Features -------- - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884)) - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305)) - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427)) - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360)) Bugfixes -------- - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433)) - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250)) - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276)) - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317)) - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343)) - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344)) - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355)) - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359)) - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367)) - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414)) - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404)) Improved Documentation ---------------------- - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971)) - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313)) - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337)) - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368)) - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399)) - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395)) - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396)) Deprecations and Removals ------------------------- - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721)) - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386)) Internal Changes ---------------- - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289)) - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315)) - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316)) - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324)) - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357)) - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348)) - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349)) - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418)) - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383)) - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385)) - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391)) - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393)) - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398)) - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421)) - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430)) -----BEGIN PGP SIGNATURE----- iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmD20AQTHGFuZHJld0Bh bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ah9EACjDbd67+fYT4CIBp1/gwRZm73qAinF HtJQV3b4KjvF+KC3NTxyfuSo1r6UHSwVP7hYMcW4D7bueYaFVklG4Pb0L58pRdrr nKpVmCTuzQ7KAP165Mea2ZJUihTGrghUV1NTtkgbL2UtBn/j0ymUzWC4dDu2I3V9 vSoa0QOwaBb/iE7RCfwkI/FWNdhJis5CRtzZg31OWqs7kf1FU6+iH7XEtyckkV8M U5aryWPlVzt1c9ag7eLf4OeySyVjl9MqKDjrUd6utLM3bK7DkHSZV/L9wA/NDfwA O6jwnSCsp5TuPjpBFvIOId5P7Ank0WFJTHZj2yzXzrP86gVmmzJi5tTbVDVAaOrw Ept1zKZjNUNBrgB5nu5oSOZQXPrWfSQeQgOfNPkS6SN2fPEvV+koIuv2CBgttmmB 9/iFr0bairlU1LllvyO2Q9VdFb1V2dTFW5MKVW5VdzieeeXDBZYisHT9nr62CRkD ne52Qf84IaLJhwbyFaTssBX7KrmMTfasPzSaAQiLe7i6xlzC6ysdHajrdWf/Vn5c z4I22vaBjZ8FqPucfvC/AiAm5at5bXkVu0UTxvfqeHw6ZXan+ApBYwUVHg50Rx96 J0FkZNfDPZlFTOJpf+jiO/kx4my2pMuiTseoi3wBr21u2j07vRoeeLiGHMuICqlO 1h0ekjbYsXrvdQ== =WSa3 -----END PGP SIGNATURE----- Merge tag 'v1.39.0rc1' into develop Synapse 1.39.0rc1 (2021-07-20) ============================== The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. Features -------- - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884)) - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305)) - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427)) - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360)) Bugfixes -------- - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433)) - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250)) - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276)) - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317)) - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343)) - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344)) - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355)) - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359)) - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367)) - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414)) - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404)) Improved Documentation ---------------------- - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971)) - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313)) - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337)) - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368)) - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399)) - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395)) - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396)) Deprecations and Removals ------------------------- - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721)) - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386)) Internal Changes ---------------- - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289)) - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315)) - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316)) - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324)) - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357)) - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348)) - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349)) - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418)) - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383)) - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385)) - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391)) - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393)) - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398)) - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421)) - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))	2021-07-20 16:47:44 +01:00
Erik Johnston	54389d5697	Fix dropping locks on shut down (#10433)	2021-07-20 14:24:25 +01:00
Michael Telatynski	69226c1ab4	MSC3244 room capabilities implementation (#10283)	2021-07-20 12:59:23 +01:00
Brendan Abolivier	a743bf4694	Port the ThirdPartyEventRules module interface to the new generic interface (#10386) ... Port the third-party event rules interface to the generic module interface introduced in v1.37.0	2021-07-20 12:39:46 +02:00
reivilibre	4e340412c0	Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric (#10332) ... Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2021-07-19 16:11:34 +01:00
Brendan Abolivier	36dc15412d	Add a module type for account validity (#9884) ... This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity. To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.	2021-07-16 18:11:53 +02:00
reivilibre	6a60068250	Add tests to characterise the current behaviour of R30 phone-home metrics (#10315) ... Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2021-07-15 13:51:27 +01:00
Erik Johnston	ac5c221208	Stagger send presence to remotes (#10398) ... This is to help with performance, where trying to connect to thousands of hosts at once can consume a lot of CPU (due to TLS etc). Co-authored-by: Brendan Abolivier <babolivier@matrix.org>	2021-07-15 11:52:56 +01:00
Dirk Klimpel	c7603af1d0	Allow providing credentials to http_proxy (#10360)	2021-07-15 10:37:08 +01:00
Patrick Cloke	eb3beb8f12	Add type hints and comments to event auth code. (#10393)	2021-07-14 14:13:40 +01:00
Patrick Cloke	2d16e69b4b	Show all joinable rooms in the spaces summary. (#10298) ... Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.	2021-07-13 08:59:27 -04:00
Jonathan de Jong	93729719b8	Use inline type hints in tests/ (#10350) ... This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)	2021-07-13 11:52:58 +01:00
Jonathan de Jong	89cfc3dd98	[pyupgrade] tests/ (#10347)	2021-07-13 11:43:15 +01:00
Patrick Cloke	19d0401c56	Additional unit tests for spaces summary. (#10305)	2021-07-12 11:21:04 -04:00
Cristina	f6767abc05	Remove functionality associated with unused historical stats tables (#9721) ... Fixes #9602	2021-07-08 16:57:13 +01:00
Dagfinn Ilmari Mannsåker	47e28b4031	Ignore EDUs for rooms we're not in (#10317)	2021-07-06 14:31:13 +01:00
Dirk Klimpel	bcb0962a72	Fix deactivate a user if he does not have a profile (#10252)	2021-07-06 13:08:53 +01:00
Erik Johnston	7a5873277e	Add support for evicting cache entries based on last access time. (#10205)	2021-07-05 16:32:12 +01:00
Patrick Cloke	8d609435c0	Move methods involving event authentication to EventAuthHandler. (#10268) ... Instead of mixing them with user authentication methods.	2021-07-01 14:25:37 -04:00
Dirk Klimpel	6c02cca95f	Add SSO external_ids to Query User Account admin API (#10261) ... Related to #10251	2021-07-01 11:26:24 +02:00
Richard van der Hoff	785bceef72	Merge branch 'release-v1.37' into develop	2021-06-29 20:25:47 +01:00
Erik Johnston	85d237eba7	Add a distributed lock (#10269) ... This adds a simple best effort locking mechanism that works cross workers.	2021-06-29 19:15:47 +01:00
Patrick Cloke	f55836929d	Do not recurse into non-spaces in the spaces summary. (#10256) ... Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.	2021-06-29 12:00:04 -04:00
Richard van der Hoff	8165ba48b1	Return errors from send_join etc if the event is rejected (#10243) ... Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.	2021-06-24 16:00:08 +01:00
Richard van der Hoff	6e8fb42be7	Improve validation for send_{join,leave,knock} (#10225) ... The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.	2021-06-24 15:30:49 +01:00
Quentin Gliech	bd4919fb72	MSC2918 Refresh tokens implementation (#9450) ... This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>	2021-06-24 14:33:20 +01:00
Brendan Abolivier	763dba77ef	Synapse 1.37.0rc1 (2021-06-24) ... ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUW7UACgkQHgFcFF8Z Fs1ZfQ//bSyzg9mB7F43JdVBH9cpGups3Ur6nmsqE608/i/OEtomx9kTZsoN/9AO j6WoM1bhJ3j9hoE63w4DIPVhBrojtwmYWeFrRvpRVlwTzZPhfij1nIH9pPrNVwd6 /LE0uA1kW5sDZCTiOhrz84BBBz6knjBH4otl+MDDdrszYixoP3i3SrrZVJAEYyGz Dpw2OudpoOtMdGyzFMr2ewf2mQm5UFEPs9MT0AqhQJMjyM1lJYzrq3Obv1oTdCJq BdLx6kMhK8+MbZPrcqIMXgSMHLYeQYeJOrdTzuCozFo0DSblThpW//aa21+sZtPx dnIE0PpjAsm2Qg1II3EUX+YycYpI5KPhMq0ydEmUD7vTuGgJCKqAV3FBRwMlF2VE 9SOr8Mjd1ip03dHz2vlg38OlrFJ52P0e1RDZwlDtWfeJjZGZ4W6VjMhT7H/Z5USg 2J1FRPTD8E9w8wBOkjVcl76Lj6AY1KU36/ZLoYUbtrt58XYfkD4OzZmAxdJrqbMg na5T9bd9iM8vWditU+vncxU+c7Bkn55GFyDVMQYl8QtXzBvnbv8gQilGwJLkncZJ MreQ3OIBPbLNcHml56nmmY+ab+AtrIxuFl0UcZu+LGW8+lN+UZ9G9q0NFO3HbAWN /LuhCVpO42SLAuLlIhnO3mcHM5jzZodKIa6oXpcMS2jv0nylQWo= =MtOo -----END PGP SIGNATURE----- gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUXG8ACgkQHgFcFF8Z Fs2buxAAg7X4mYKHwlWLn2n2W6bUKY24HAnjtzMrNW76xIHXBJJiJMnhQD3MSKVd uiUdolc+Tt3drJYOJyurP8R3YI0OuEVbSL0Krs8f3l6jNOU0Gy7YQzc6XUHnA42J rb6mU8gMdCusk9RHHr1r+sBefwzmxTk4naCkw2n5i+r4ZfNparJK7H9yXCPoedUN VBZi9f09jaUIm1SiHgK24sW9XfE6ifjdVg+E/r599v2eHJxtCTHUasH9TCAFD8KL g17NtAP95+SQWPGwUXJFZYcVee+pGbylXuwP0RXXmMvyOWVKloI0LE6phGyLQ5cH rMRagIkvVLMdYagaX8Tf47hNLVCXRJpuryxMgY3+5fx0O7lCMXlC3mDfs9zze+Fj KXmJtVy/vpd+zpjQw+HHPnL1dDzsKnDqLFXFCj9Sjv6A/ruckxwWofuh0z83dYtv 1Abp8k7gVTVGZqLGPo8gRoqI2jQhWDvmJd+crb05qp86SzCz1sCma2ujWQtv3c1t 6aQpfmGMhFPp+hS2lS8AcyxK4vq+GudtXUOgzdqCNvIs+eJAV5cq6uufLIy6c2XR FAR7UowoAoDvqTdD7+i5nHEjbudjjN5KgPXlO3a9+gG5jOR3IsOLZ8FVN0ML6qIJ wzl0wM9YIlxyleMlWb0l4CFNDKrGJVhWgOupruHS+D5jfZQsZ3g= =AjRf -----END PGP SIGNATURE----- Merge tag 'v1.37.0rc1' into develop Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))	2021-06-24 11:20:28 +01:00
Brendan Abolivier	c955e37868	Fix wrapping of legacy check_registration_for_spam (#10238) ... Fixes #10234	2021-06-23 17:22:08 +02:00
Andrew Morgan	394673055d	Re-introduce "Leave out optional keys from /sync" change (#10214) ... Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.	2021-06-23 15:57:41 +01:00
Brendan Abolivier	e19e3d452d	Improve the reliability of auto-joining remote rooms (#10237) ... If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do. Fixes #10233 (at least to some extent)	2021-06-23 16:14:52 +02:00
Eric Eastwood	96f6293de5	Add endpoints for backfilling history (MSC2716) (#9247) ... Work on https://github.com/matrix-org/matrix-doc/pull/2716	2021-06-22 10:02:53 +01:00
Patrick Cloke	0bd968921c	Fix a missing await when in the spaces summary. (#10208) ... This could cause a minor data leak if someone defined a non-restricted join rule with an allow key or used a restricted join rule in an older room version, but this is unlikely. Additionally this starts adding unit tests to the spaces summary handler.	2021-06-18 18:41:33 +01:00
Brendan Abolivier	1b3e398bea	Standardise the module interface (#10062) ... This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.	2021-06-18 12:15:52 +01:00
Brendan Abolivier	08c8469322	Remove support for ACME v1 (#10194) ... Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.	2021-06-17 18:56:48 +01:00
Richard van der Hoff	fcf3c7032b	Ensure that we do not cache empty sync responses after a timeout (#10158) ... Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.	2021-06-17 16:23:11 +01:00
Marcus	8070b893db	update black to 21.6b0 (#10197) ... Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>	2021-06-17 15:20:06 +01:00
Patrick Cloke	9e5ab6dd58	Remove the experimental flag for knocking and use stable prefixes / endpoints. (#10167) ... * Room version 7 for knocking. * Stable prefixes and endpoints (both client and federation) for knocking. * Removes the experimental configuration flag.	2021-06-15 07:45:14 -04:00
Richard van der Hoff	d7808a2dde	Extend ResponseCache to pass a context object into the callback (#10157) ... This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.	2021-06-14 10:26:09 +01:00
Sorunome	d936371b69	Implement knock feature (#6739) ... This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io	2021-06-09 19:39:51 +01:00
Patrick Cloke	11846dff8c	Limit the number of in-flight /keys/query requests from a single device. (#10144)	2021-06-09 07:05:32 -04:00
14mRh4X0r	8942e23a69	Always update AS last_pos, even on no events (#10107) ... Fixes #1834. `get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether. Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list. Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>	2021-06-07 15:42:05 +01:00
Travis Ralston	5325f0308c	r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224) ... [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. This is progress towards https://github.com/matrix-org/synapse/issues/8334	2021-06-03 13:50:49 +01:00
Dirk Klimpel	0284d2a297	Add new admin APIs to remove media by media ID from quarantine. (#10044) ... Related to: #6681, #5956, #10040 Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-06-02 18:50:35 +01:00
Erik Johnston	fc3d2dc269	Rewrite the KeyRing (#10035)	2021-06-02 16:37:59 +01:00
Richard van der Hoff	b4b2fd2ece	add a cache to have_seen_event (#9953) ... Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.	2021-06-01 12:04:47 +01:00
Callum Brown	8fb9af570f	Make reason and score optional for report_event (#10077) ... Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414 See #8551 Signed-off-by: Callum Brown <callum@calcuode.com>	2021-05-27 18:42:23 +01:00
Erik Johnston	78b5102ae7	Fix up BatchingQueue (#10078) ... Fixes #10068	2021-05-27 14:32:31 +01:00
Richard van der Hoff	fe5dad46b0	Remove redundant code to reload tls cert (#10054) ... we don't need to reload the tls cert if we don't have any tls listeners. Follow-up to #9280.	2021-05-27 10:34:24 +01:00
Richard van der Hoff	224f2f949b	Combine LruCache.invalidate and invalidate_many (#9973) ... * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog	2021-05-27 10:33:56 +01:00
Dirk Klimpel	65e6c64d83	Add an admin API for unprotecting local media from quarantine (#10040) ... Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-05-26 11:19:47 +01:00
Patrick Cloke	7adcb20fc0	Add missing type hints to synapse.util (#9982)	2021-05-24 15:32:01 -04:00
Richard van der Hoff	c0df6bae06	Remove keylen from LruCache. (#9993) ... `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.	2021-05-24 14:02:01 +01:00
Erik Johnston	3e831f24ff	Don't hammer the database for destination retry timings every ~5mins (#10036)	2021-05-21 17:57:08 +01:00
Andrew Morgan	21bd230831	Add a test for update_presence (#10033) ... https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale. The bug was fixed in #10014, and this PR just adds a test that failed on the old code, and was used to initially verify the bug. The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.	2021-05-21 17:29:14 +01:00
Erik Johnston	7958eadcd1	Add a batching queue implementation. (#10017)	2021-05-21 11:20:51 +01:00
Andrew Morgan	4d6e5a5e99	Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823)	2021-05-18 14:13:45 +01:00
Richard van der Hoff	5090f26b63	Minor @cachedList enhancements (#9975) ... - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.	2021-05-14 11:12:36 +01:00
Brendan Abolivier	a683028d81	Correctly ratelimit invites when creating a room (#9968) ... * Correctly ratelimit invites when creating a room Also allow ratelimiting for more than one action at a time.	2021-05-12 16:05:28 +02:00