Merge remote-tracking branch 'origin/release-v1.46' into develop

CHANGES.md

@@ -1,3 +1,71 @@
+Synapse 1.46.0rc1 (2021-10-27)
+==============================
+
+The cause of the [performance regression affecting Synapse 1.44](https://github.com/matrix-org/synapse/issues/11049) has been identified and fixed. ([\#11177](https://github.com/matrix-org/synapse/issues/11177))
+
+Features
+--------
+
+- Add support for Ubuntu 21.10 "Impish Indri". ([\#11024](https://github.com/matrix-org/synapse/issues/11024))
+- Port the Password Auth Providers module interface to the new generic interface. ([\#10548](https://github.com/matrix-org/synapse/issues/10548), [\#11180](https://github.com/matrix-org/synapse/issues/11180))
+- Experimental support for the thread relation defined in [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440). ([\#11088](https://github.com/matrix-org/synapse/issues/11088), [\#11181](https://github.com/matrix-org/synapse/issues/11181), [\#11192](https://github.com/matrix-org/synapse/issues/11192))
+- Users admin API can now also modify user type in addition to allowing it to be set on user creation. ([\#11174](https://github.com/matrix-org/synapse/issues/11174))
+
+
+Bugfixes
+--------
+
+- Newly-created public rooms are now only assigned an alias if the room's creation has not been blocked by permission settings. Contributed by @AndrewFerr. ([\#10930](https://github.com/matrix-org/synapse/issues/10930))
+- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#11001](https://github.com/matrix-org/synapse/issues/11001), [\#11009](https://github.com/matrix-org/synapse/issues/11009), [\#11012](https://github.com/matrix-org/synapse/issues/11012))
+- Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room. ([\#11027](https://github.com/matrix-org/synapse/issues/11027))
+- Fix a bug where setting a user's `external_id` via the admin API returns 500 and deletes user's existing external mappings if that external ID is already mapped. ([\#11051](https://github.com/matrix-org/synapse/issues/11051))
+- Fix a long-standing bug where users excluded from the user directory were added into the directory if they belonged to a room which became public or private. ([\#11075](https://github.com/matrix-org/synapse/issues/11075))
+- Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding. ([\#11077](https://github.com/matrix-org/synapse/issues/11077), [\#11089](https://github.com/matrix-org/synapse/issues/11089))
+- Fix broken export-data admin command and add test script checking the command to CI. ([\#11078](https://github.com/matrix-org/synapse/issues/11078))
+- Show an error when timestamp in seconds is provided to the `/purge_media_cache` Admin API. ([\#11101](https://github.com/matrix-org/synapse/issues/11101))
+- Fix local users who left all their rooms being removed from the user directory, even if the `search_all_users` config option was enabled. ([\#11103](https://github.com/matrix-org/synapse/issues/11103))
+- Fix a bug which caused the module API's `get_user_ip_and_agents` function to always fail on workers. `get_user_ip_and_agents` was introduced in 1.44.0 and did not function correctly on worker processes at the time. ([\#11112](https://github.com/matrix-org/synapse/issues/11112))
+- Identity server connection is no longer ignoring `ip_range_whitelist`. ([\#11120](https://github.com/matrix-org/synapse/issues/11120))
+- Fix a bug introduced in Synapse 1.45.0 breaking the configuration file parsing script. ([\#11145](https://github.com/matrix-org/synapse/issues/11145))
+- Fix a performance regression introduced in 1.44.0 which could cause client requests to time out when making large numbers of outbound requests. ([\#11177](https://github.com/matrix-org/synapse/issues/11177), [\#11190](https://github.com/matrix-org/synapse/issues/11190))
+- Resolve and share `state_groups` for all [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical events in batch. ([\#10975](https://github.com/matrix-org/synapse/issues/10975))
+
+
+Improved Documentation
+----------------------
+
+- Fix broken links relating to module API deprecation in the upgrade notes. ([\#11069](https://github.com/matrix-org/synapse/issues/11069))
+- Add more information about what happens when a user is deactivated. ([\#11083](https://github.com/matrix-org/synapse/issues/11083))
+- Clarify the the sample log config can be copied from the documentation without issue. ([\#11092](https://github.com/matrix-org/synapse/issues/11092))
+- Update the admin API documentation with an updated list of the characters allowed in registration tokens. ([\#11093](https://github.com/matrix-org/synapse/issues/11093))
+- Document Synapse's behaviour when dealing with multiple modules registering the same callbacks and/or handlers for the same HTTP endpoints. ([\#11096](https://github.com/matrix-org/synapse/issues/11096))
+- Fix instances of `[example]{.title-ref}` in the upgrade documentation as a result of prior RST to Markdown conversion. ([\#11118](https://github.com/matrix-org/synapse/issues/11118))
+- Document the version of Synapse each module callback was introduced in. ([\#11132](https://github.com/matrix-org/synapse/issues/11132))
+- Document the version of Synapse that introduced each module API method. ([\#11183](https://github.com/matrix-org/synapse/issues/11183))
+
+
+Internal Changes
+----------------
+- Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection. ([\#10984](https://github.com/matrix-org/synapse/issues/10984))
+- Include rejected status when we log events. ([\#11008](https://github.com/matrix-org/synapse/issues/11008))
+- Add some extra logging to the event persistence code. ([\#11014](https://github.com/matrix-org/synapse/issues/11014))
+- Rearrange the internal workings of the incremental user directory updates. ([\#11035](https://github.com/matrix-org/synapse/issues/11035))
+- Fix a long-standing bug where users excluded from the directory could still be added to the `users_who_share_private_rooms` table after a regular user joins a private room. ([\#11143](https://github.com/matrix-org/synapse/issues/11143))
+- Add and improve type hints. ([\#10972](https://github.com/matrix-org/synapse/issues/10972), [\#11055](https://github.com/matrix-org/synapse/issues/11055), [\#11066](https://github.com/matrix-org/synapse/issues/11066), [\#11076](https://github.com/matrix-org/synapse/issues/11076), [\#11095](https://github.com/matrix-org/synapse/issues/11095), [\#11109](https://github.com/matrix-org/synapse/issues/11109), [\#11121](https://github.com/matrix-org/synapse/issues/11121), [\#11146](https://github.com/matrix-org/synapse/issues/11146))
+- Mark the Synapse package as containing type annotations and fix export declarations so that Synapse pluggable modules may be type checked against Synapse. ([\#11054](https://github.com/matrix-org/synapse/issues/11054))
+- Remove dead code from `MediaFilePaths`. ([\#11056](https://github.com/matrix-org/synapse/issues/11056))
+- Be more lenient when parsing oEmbed response versions. ([\#11065](https://github.com/matrix-org/synapse/issues/11065))
+- Create a separate module for the retention configuration. ([\#11070](https://github.com/matrix-org/synapse/issues/11070))
+- Clean up some of the federation event authentication code for clarity. ([\#11115](https://github.com/matrix-org/synapse/issues/11115), [\#11116](https://github.com/matrix-org/synapse/issues/11116), [\#11122](https://github.com/matrix-org/synapse/issues/11122))
+- Add docstrings and comments to the application service ephemeral event sending code. ([\#11138](https://github.com/matrix-org/synapse/issues/11138))
+- Update the `sign_json` script to support inline configuration of the signing key. ([\#11139](https://github.com/matrix-org/synapse/issues/11139))
+- Fix broken link in the docker image README. ([\#11144](https://github.com/matrix-org/synapse/issues/11144))
+- Always dump logs from unit tests during CI runs. ([\#11068](https://github.com/matrix-org/synapse/issues/11068))
+- Add tests for `MediaFilePaths` class. ([\#11057](https://github.com/matrix-org/synapse/issues/11057))
+- Simplify the user admin API tests. ([\#11048](https://github.com/matrix-org/synapse/issues/11048))
+- Add a test for the workaround introduced in [\#11042](https://github.com/matrix-org/synapse/pull/11042) concerning the behaviour of third-party rule modules and `SynapseError`s. ([\#11071](https://github.com/matrix-org/synapse/issues/11071))
+
+
 Synapse 1.45.1 (2021-10-20)
 ===========================
 

changelog.d/10548.feature

@@ -1 +0,0 @@
-Port the Password Auth Providers module interface to the new generic interface.

changelog.d/10930.bugfix

@@ -1 +0,0 @@
-Newly-created public rooms are now only assigned an alias if the room's creation has not been blocked by permission settings. Contributed by @AndrewFerr.

changelog.d/10972.misc

@@ -1 +0,0 @@
-Add type hints to `synapse.storage.databases.main.client_ips`.

changelog.d/10975.feature

@@ -1 +0,0 @@
-Resolve and share `state_groups` for all [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical events in batch.

changelog.d/10984.misc

@@ -1 +0,0 @@
-Fix spurious warnings about losing the logging context on the `ReplicationCommandHandler` when losing the replication connection.

changelog.d/11001.bugfix

@@ -1 +0,0 @@
- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.

changelog.d/11008.misc

@@ -1 +0,0 @@
-Include rejected status when we log events.

changelog.d/11009.bugfix

@@ -1 +0,0 @@
-Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.

changelog.d/11012.bugfix

@@ -1 +0,0 @@
-Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state.

changelog.d/11014.misc

@@ -1 +0,0 @@
-Add some extra logging to the event persistence code.

changelog.d/11024.misc

@@ -1 +0,0 @@
-Add support for Ubuntu 21.10 "Impish Indri".

changelog.d/11027.bugfix

@@ -1 +0,0 @@
-Fix 500 error on `/messages` when the server accumulates more than 5 backwards extremities at a given depth for a room.

changelog.d/11035.misc

@@ -1 +0,0 @@
-Rearrange the internal workings of the incremental user directory updates.

changelog.d/11048.misc

@@ -1 +0,0 @@
-Simplify the user admin API tests.

changelog.d/11051.bugfix

@@ -1 +0,0 @@
-Fix a bug where setting a user's external_id via the admin API returns 500 and deletes users existing external mappings if that external ID is already mapped.

changelog.d/11054.misc

@@ -1 +0,0 @@
-Mark the Synapse package as containing type annotations and fix export declarations so that Synapse pluggable modules may be type checked against Synapse.

changelog.d/11055.misc

@@ -1 +0,0 @@
-Improve type hints for `_wrap_in_base_path` decorator used by `MediaFilePaths`.

changelog.d/11056.misc

@@ -1 +0,0 @@
-Remove dead code from `MediaFilePaths`.

changelog.d/11057.misc

@@ -1 +0,0 @@
-Add tests for `MediaFilePaths` class.

changelog.d/11065.misc

@@ -1 +0,0 @@
-Be more lenient when parsing oEmbed response versions.

changelog.d/11066.misc

@@ -1 +0,0 @@
-Add type hints to `synapse.events`.

changelog.d/11068.misc

@@ -1 +0,0 @@
-Always dump logs from unit tests during CI runs.

changelog.d/11069.doc

@@ -1 +0,0 @@
-Fix broken links relating to module API deprecation in the upgrade notes.

changelog.d/11070.misc

@@ -1 +0,0 @@
-Create a separate module for the retention configuration.

changelog.d/11071.misc

@@ -1 +0,0 @@
-Add a test for the workaround introduced in [\#11042](https://github.com/matrix-org/synapse/pull/11042) concerning the behaviour of third-party rule modules and `SynapseError`s.

changelog.d/11075.bugfix

@@ -1 +0,0 @@
-Fix a long-standing bug where users excluded from the user directory were added into the directory if they belonged to a room which became public or private.

changelog.d/11076.misc

@@ -1 +0,0 @@
-Fix type hints in the relations tests.

changelog.d/11077.bugfix

@@ -1 +0,0 @@
-Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding.

changelog.d/11078.bugfix

@@ -1 +0,0 @@
-Fix broken export-data admin command and add test script checking the command to CI.

changelog.d/11083.doc

@@ -1 +0,0 @@
-Add more information about what happens when a user is deactivated.

changelog.d/11088.feature

@@ -1 +0,0 @@
-Experimental support for the thread relation defined in [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440).

changelog.d/11089.bugfix

@@ -1 +0,0 @@
-Fix a long-standing bug when attempting to preview URLs which are in the `windows-1252` character encoding.

changelog.d/11092.doc

@@ -1 +0,0 @@
-Clarify the the sample log config can be copied from the documentation without issue.

changelog.d/11093.doc

@@ -1 +0,0 @@
-Update the admin API documentation with an updated list of the characters allowed in registration tokens.

changelog.d/11095.misc

@@ -1 +0,0 @@
-Add type hints to most `HomeServer` parameters.

changelog.d/11096.doc

@@ -1 +0,0 @@
-Document Synapse's behaviour when dealing with multiple modules registering the same callbacks and/or handlers for the same HTTP endpoints.

changelog.d/11101.bugfix

@@ -1 +0,0 @@
-Show an error when timestamp in seconds is provided to the `/purge_media_cache` Admin API.

changelog.d/11103.bugfix

@@ -1 +0,0 @@
-Fix local users who left all their rooms being removed from the user directory, even if the "search_all_users" config option was enabled.

changelog.d/11109.misc

@@ -1 +0,0 @@
-Add missing type hints to `synapse.api` module.

changelog.d/11112.bugfix

@@ -1 +0,0 @@
-Fix a bug which caused the module API's `get_user_ip_and_agents` function to always fail on workers. `get_user_ip_and_agents` was introduced in 1.44.0 and did not function correctly on worker processes at the time.

changelog.d/11115.misc

@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.

changelog.d/11116.misc

@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.

changelog.d/11118.doc

@@ -1 +0,0 @@
-Fix instances of `[example]{.title-ref}` in the upgrade documentation as a result of prior RST to Markdown conversion.

changelog.d/11120.bugfix

@@ -1 +0,0 @@
-Identity server connection is no longer ignoring `ip_range_whitelist`.

changelog.d/11121.misc

@@ -1 +0,0 @@
-Add type hints for event fetching.

changelog.d/11122.misc

@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.

changelog.d/11132.doc

@@ -1 +0,0 @@
-Document the version of Synapse each module callback was introduced in.

changelog.d/11138.misc

@@ -1 +0,0 @@
-Add docstrings and comments to the application service ephemeral event sending code.

changelog.d/11139.misc

@@ -1 +0,0 @@
-Update the `sign_json` script to support inline configuration of the signing key.

changelog.d/11143.misc

@@ -1 +0,0 @@
-Fix a long-standing bug where users excluded from the directory could still be added to the `users_who_share_private_rooms` table after a regular user joins a private room.

changelog.d/11144.misc

@@ -1 +0,0 @@
-Fix broken link in the docker image README.

changelog.d/11145.bugfix

@@ -1 +0,0 @@
-Fix a bug introduced in Synapse v1.45.0 breaking the configuration file parsing script.

changelog.d/11146.misc

@@ -1 +0,0 @@
-Add missing type hints to `synapse.crypto`.

changelog.d/11174.feature

@@ -1 +0,0 @@
-Users admin API can now also modify user type in addition to allowing it to be set on user creation.

changelog.d/11177.bugfix

@@ -1 +0,0 @@
-Fix a performance regression introduced in v1.44.0 which could cause client requests to time out when making large numbers of outbound requests.

changelog.d/11180.feature

@@ -1 +0,0 @@
-Port the Password Auth Providers module interface to the new generic interface.

changelog.d/11181.feature

@@ -1 +0,0 @@
-Experimental support for the thread relation defined in [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440).

changelog.d/11183.doc

@@ -1 +0,0 @@
-Document the version of Synapse that introduced each module API method.

debian/changelog

@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.46.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.46.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 26 Oct 2021 14:04:04 +0100
+
 matrix-synapse-py3 (1.45.1) stable; urgency=medium
 
   * New synapse release 1.45.1.

synapse/__init__.py

@@ -47,7 +47,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.45.1"
+__version__ = "1.46.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/app/_base.py

@@ -346,6 +346,7 @@ async def start(hs: "HomeServer"):
     # numbers of DNS requests don't starve out other users of the threadpool.
     resolver_threadpool = ThreadPool(name="gai_resolver")
     resolver_threadpool.start()
+    reactor.addSystemEventTrigger("during", "shutdown", resolver_threadpool.stop)
     reactor.installNameResolver(
         GAIResolver(reactor, getThreadPool=lambda: resolver_threadpool)
     )

synapse/storage/databases/main/events_bg_updates.py

@@ -1108,7 +1108,7 @@ class EventsBackgroundUpdatesStore(SQLBaseStore):
                 """
                 SELECT event_id, json FROM event_json
                 LEFT JOIN event_relations USING (event_id)
-                WHERE event_id > ? AND relates_to_id IS NULL
+                WHERE event_id > ? AND event_relations.event_id IS NULL
                 ORDER BY event_id LIMIT ?
                 """,
                 (last_event_id, batch_size),