Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-12-30 02:16:10 -05:00
Code Issues Packages Projects Releases Wiki Activity
19,528 Commits 4 Branches 615 Tags 63 MiB
b3e9b00fb2
Commit Graph

13724 Commits

Author SHA1 Message Date
David Robertson
b3e9b00fb2
Merge remote-tracking branch 'origin/release-v1.45' into develop 2021-10-13 12:46:30 +01:00
Patrick Cloke
1f9d0b8a7a
Add type hints to synapse.events.*. (#11066) 
Except `synapse/events/__init__.py`, which will be done in a follow-up.
 2021-10-13 07:24:07 -04:00
Azrenbeth
cdd308845b
Port the Password Auth Providers module interface to the new generic interface (#10548) 
Co-authored-by: Azrenbeth <7782548+Azrenbeth@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2021-10-13 11:21:52 +00:00
Patrick Cloke
732bbf6737
Be more lenient when parsing the version for oEmbed responses. (#11065) 2021-10-13 07:00:07 -04:00
David Robertson
b83e822556
Stop user directory from failing if it encounters users not in the users table. (#11053) 
The following scenarios would halt the user directory updater:

- user joins room
- user leaves room
- user present in room which switches from private to public, or vice versa.

for two classes of users:

- appservice senders
- users missing from the user table.

If this happened, the user directory would be stuck, unable to make forward progress.

Exclude both cases from the user directory, so that we ignore them.

Co-authored-by: Eric Eastwood <erice@element.io>
Co-authored-by: reivilibre <oliverw@matrix.org>
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2021-10-13 09:38:22 +00:00
reivilibre
2a2b189130
Mark Module API error imports as re-exported and mark Synapse as containing type annotations (#11054) 2021-10-13 08:42:41 +01:00
Patrick Cloke
9abc5f2a05 Merge remote-tracking branch 'origin/release-v1.45' into develop 2021-10-12 14:21:05 -04:00
Sean Quah
8eaffe013c
Update _wrap_in_base_path type hints to preserve function arguments (#11055) 2021-10-12 18:19:21 +01:00
Patrick Cloke
1db9282dfa
Fix formatting string when oEmbed errors occur. (#11061) 2021-10-12 17:15:42 +00:00
Erik Johnston
77ea03086c Merge remote-tracking branch 'origin/release-v1.45' into develop 2021-10-12 14:41:13 +01:00
Erik Johnston
333d6f4e84
Fix race in MultiWriterIdGenerator (#11045) 
The race allowed the current position to advance too far when stream IDs
are still being persisted.

This happened when it received a new stream ID from a remote write
between a new stream ID being allocated and it being added to the set of
unpersisted stream IDs.

Fixes #9424.
 2021-10-12 14:27:09 +01:00
Sean Quah
36224e056a
Add type hints to synapse.storage.databases.main.client_ips (#10972) 2021-10-12 13:50:34 +01:00
Brendan Abolivier
a18c568516 Synapse 1.45.0rc1 (2021-10-12) 
==============================
 
 **Note:** We are aware of [a performance issue](https://github.com/matrix-org/synapse/issues/11049) introduced in Synapse v1.44.0, as well as [a bug](https://github.com/matrix-org/synapse/issues/11025) with the user directory when using application services. While this release candidate doesn't fix either of those issues, a second release candidate is expected to come out in a few days to address them.
 
 Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450) for more information.
 
 Features
 --------
 
 - Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655))
 - Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822))
 - Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894))
 - Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910))
 - Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954))
 - Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028))
 
 Bugfixes
 --------
 
 - Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922))
 - Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924))
 - Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](https://github.com/matrix-org/synapse/issues/10927))
 - Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947))
 - Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956))
 - Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960))
 - Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](https://github.com/matrix-org/synapse/issues/10962))
 - Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981))
 - Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](https://github.com/matrix-org/synapse/issues/10982))
 - Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](https://github.com/matrix-org/synapse/issues/10995))
 - Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002))
 - Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](https://github.com/matrix-org/synapse/issues/11003))
 - Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](https://github.com/matrix-org/synapse/issues/11042))
 - Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](https://github.com/matrix-org/synapse/issues/10877))
 
 Improved Documentation
 ----------------------
 
 - Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971))
 - Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973))
 - Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990))
 - Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991))
 
 Internal Changes
 ----------------
 
 - Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888))
 - Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892))
 - Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895))
 - Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902))
 - Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903))
 - Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915))
 - Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916), [\#10959](https://github.com/matrix-org/synapse/issues/10959), [\#10985](https://github.com/matrix-org/synapse/issues/10985))
 - Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926), [\#10940](https://github.com/matrix-org/synapse/issues/10940), [\#10986](https://github.com/matrix-org/synapse/issues/10986), [\#10987](https://github.com/matrix-org/synapse/issues/10987), [\#10988](https://github.com/matrix-org/synapse/issues/10988), [\#11010](https://github.com/matrix-org/synapse/issues/11010), [\#11011](https://github.com/matrix-org/synapse/issues/11011))
 - Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934))
 - Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935))
 - Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936))
 - Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939))
 - Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945))
 - Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958))
 - Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961))
 - Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](https://github.com/matrix-org/synapse/issues/10963))
 - Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966))
 - Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974))
 - Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983))
 - Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992))
 - Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993))
 - Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994))
 - Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004))
 - Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005))
 - Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006))
 - Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017))
 - Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019))
 - Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021))
 - Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023))
 - When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034))
 - Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmFlaJYACgkQHgFcFF8Z
 Fs0OBQ//QuIZHqwajqu6mVdcm5/GzYzerMVqcxByOsGX2Xeq0am1AnAG3MFH6TT/
 f+e7T0ZIed7KTQFGKxyNX9nOi1RmQIaypBn66jt+tMzaCqMmlto3s0w2M5C+x4l4
 lf9I0Shk/0wvrklc1GbmM89wB8eLmSz+UsrNK/ksMsevlP0K8GxA7HE3TrzDAth+
 cLJOine4yfW4VlHus00u09E/2uILOE2wQ8V7YSpDMYOZc0bp0QEB5TGaab+4ESLz
 JVLh1j3+dAmYPAAZMyWAp3PGN430NlNEwqomB1TYPdDRJ5ohlkWe22ACF6PD8eq+
 TlnrANlo2CThZkTFQAecYkVPziETQv25BkGRho1IoBOfS8pTIDMoVwrrTc3okGfj
 L6XlB1Rn+WauHXPq7EBJZnk9iw8qD0Pq8arMZeKYmQ1OMhUQmYTwBOj82fFU+jfu
 a4v2rrMSVf3lmP9nAEHozGN0MAGsD0in+uvQuehdqVmKUbYUD0AWtHRZlO0ca4fH
 BZxE2/CCJWaK/2IdH4I4S1l/AF4kdX0K0tcmAiwUKrm5jSC2Ww8BWeEqjwF9Bd+s
 e1tF2pjw/VXjhAMmLIJVLEysb0yrw+QyZHWJ9fQP1KgaCcSwQjMgUBVrM3gcliIS
 +/qjTl1QgkdTVHis6r5rdWBZrVJ/CeLFUz7pATEfoomf/EiEMHQ=
 =VnsK
 -----END PGP SIGNATURE-----

Merge tag 'v1.45.0rc1' into develop

Synapse 1.45.0rc1 (2021-10-12)
==============================

**Note:** We are aware of [a performance issue](https://github.com/matrix-org/synapse/issues/11049) introduced in Synapse v1.44.0, as well as [a bug](https://github.com/matrix-org/synapse/issues/11025) with the user directory when using application services. While this release candidate doesn't fix either of those issues, a second release candidate is expected to come out in a few days to address them.

Media storage providers module that read from Synapse's configuration need changes as of this version, see the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1450) for more information.

Features
--------

- Add [MSC3069](https://github.com/matrix-org/matrix-doc/pull/3069) support to `/account/whoami`. ([\#9655](https://github.com/matrix-org/synapse/issues/9655))
- Support autodiscovery of oEmbed previews. ([\#10822](https://github.com/matrix-org/synapse/issues/10822))
- Add a `user_may_send_3pid_invite` spam checker callback for modules to allow or deny 3PID invites. ([\#10894](https://github.com/matrix-org/synapse/issues/10894))
- Add a spam checker callback to allow or deny room joins. ([\#10910](https://github.com/matrix-org/synapse/issues/10910))
- Include an `update_synapse_database` script in the distribution. Contributed by @Fizzadar at Beeper. ([\#10954](https://github.com/matrix-org/synapse/issues/10954))
- Include exception information in JSON logging output. Contributed by @Fizzadar at Beeper. ([\#11028](https://github.com/matrix-org/synapse/issues/11028))

Bugfixes
--------

- Fix a minor bug in the response to `/_matrix/client/r0/voip/turnServer`. Contributed by @lukaslihotzki. ([\#10922](https://github.com/matrix-org/synapse/issues/10922))
- Fix a bug where empty `yyyy-mm-dd/` directories would be left behind in the media store's `url_cache_thumbnails/` directory. ([\#10924](https://github.com/matrix-org/synapse/issues/10924))
- Fix a bug introduced in Synapse v1.40.0 where the signature checks for room version 8 and 9 could be applied to earlier room versions in some situations. ([\#10927](https://github.com/matrix-org/synapse/issues/10927))
- Fix a long-standing bug wherein deactivated users still count towards the monthly active users limit. ([\#10947](https://github.com/matrix-org/synapse/issues/10947))
- Fix a long-standing bug which meant that events received over federation were sometimes incorrectly accepted into the room state. ([\#10956](https://github.com/matrix-org/synapse/issues/10956))
- Fix a long-standing bug where rebuilding the user directory wouldn't exclude support and deactivated users. ([\#10960](https://github.com/matrix-org/synapse/issues/10960))
- Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint rejecting subsequent batches with unknown batch ID error in existing room versions from the room creator. ([\#10962](https://github.com/matrix-org/synapse/issues/10962))
- Fix a bug that could leak local users' per-room nicknames and avatars when the user directory is rebuilt. ([\#10981](https://github.com/matrix-org/synapse/issues/10981))
- Fix a long-standing bug where the remainder of a batch of user directory changes would be silently dropped if the server left a room early in the batch. ([\#10982](https://github.com/matrix-org/synapse/issues/10982))
- Correct a bugfix introduced in Synapse v1.44.0 that would catch the wrong error if a connection is lost before a response could be written to it. ([\#10995](https://github.com/matrix-org/synapse/issues/10995))
- Fix a long-standing bug where local users' per-room nicknames/avatars were visible to anyone who could see you in the user directory. ([\#11002](https://github.com/matrix-org/synapse/issues/11002))
- Fix a long-standing bug where a user's per-room nickname/avatar would overwrite their profile in the user directory when a room was made public. ([\#11003](https://github.com/matrix-org/synapse/issues/11003))
- Work around a regression, introduced in Synapse v1.39.0, that caused `SynapseError`s raised by the experimental third-party rules module callback `check_event_allowed` to be ignored. ([\#11042](https://github.com/matrix-org/synapse/issues/11042))
- Fix a bug in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) insertion events in rooms that could cause cross-talk/conflicts between batches. ([\#10877](https://github.com/matrix-org/synapse/issues/10877))

Improved Documentation
----------------------

- Change wording ("reference homeserver") in Synapse repository documentation. Contributed by @maxkratz. ([\#10971](https://github.com/matrix-org/synapse/issues/10971))
- Fix a dead URL in development documentation (SAML) and change wording from "Riot" to "Element". Contributed by @maxkratz. ([\#10973](https://github.com/matrix-org/synapse/issues/10973))
- Add additional content to the Welcome and Overview page of the documentation. ([\#10990](https://github.com/matrix-org/synapse/issues/10990))
- Update links to MSCs in documentation. Contributed by @dklimpel. ([\#10991](https://github.com/matrix-org/synapse/issues/10991))

Internal Changes
----------------

- Improve type hinting in `synapse.util`. ([\#10888](https://github.com/matrix-org/synapse/issues/10888))
- Add further type hints to `synapse.storage.util`. ([\#10892](https://github.com/matrix-org/synapse/issues/10892))
- Fix type hints to be compatible with an upcoming change to Twisted. ([\#10895](https://github.com/matrix-org/synapse/issues/10895))
- Update utility code to handle C implementations of frozendict. ([\#10902](https://github.com/matrix-org/synapse/issues/10902))
- Drop old functionality which maintained database compatibility with Synapse versions before v1.31. ([\#10903](https://github.com/matrix-org/synapse/issues/10903))
- Clean-up configuration helper classes for the `ServerConfig` class. ([\#10915](https://github.com/matrix-org/synapse/issues/10915))
- Use direct references to config flags. ([\#10916](https://github.com/matrix-org/synapse/issues/10916), [\#10959](https://github.com/matrix-org/synapse/issues/10959), [\#10985](https://github.com/matrix-org/synapse/issues/10985))
- Clean up some of the federation event authentication code for clarity. ([\#10926](https://github.com/matrix-org/synapse/issues/10926), [\#10940](https://github.com/matrix-org/synapse/issues/10940), [\#10986](https://github.com/matrix-org/synapse/issues/10986), [\#10987](https://github.com/matrix-org/synapse/issues/10987), [\#10988](https://github.com/matrix-org/synapse/issues/10988), [\#11010](https://github.com/matrix-org/synapse/issues/11010), [\#11011](https://github.com/matrix-org/synapse/issues/11011))
- Refactor various parts of the codebase to use `RoomVersion` objects instead of room version identifier strings. ([\#10934](https://github.com/matrix-org/synapse/issues/10934))
- Refactor user directory tests in preparation for upcoming changes. ([\#10935](https://github.com/matrix-org/synapse/issues/10935))
- Include the event id in the logcontext when handling PDUs received over federation. ([\#10936](https://github.com/matrix-org/synapse/issues/10936))
- Fix logged errors in unit tests. ([\#10939](https://github.com/matrix-org/synapse/issues/10939))
- Fix a broken test to ensure that consent configuration works during registration. ([\#10945](https://github.com/matrix-org/synapse/issues/10945))
- Add type hints to filtering classes. ([\#10958](https://github.com/matrix-org/synapse/issues/10958))
- Add type-hint to `HomeserverTestcase.setup_test_homeserver`. ([\#10961](https://github.com/matrix-org/synapse/issues/10961))
- Fix the test utility function `create_room_as` so that `is_public=True` will explicitly set the `visibility` parameter of room creation requests to `public`. Contributed by @AndrewFerr. ([\#10963](https://github.com/matrix-org/synapse/issues/10963))
- Make the release script more robust and transparent. ([\#10966](https://github.com/matrix-org/synapse/issues/10966))
- Refactor [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` mega function into smaller handler functions. ([\#10974](https://github.com/matrix-org/synapse/issues/10974))
- Log stack traces when a missing opentracing span is detected. ([\#10983](https://github.com/matrix-org/synapse/issues/10983))
- Update GHA config to run tests against Python 3.10 and PostgreSQL 14. ([\#10992](https://github.com/matrix-org/synapse/issues/10992))
- Fix a long-standing bug where `ReadWriteLock`s could drop logging contexts on exit. ([\#10993](https://github.com/matrix-org/synapse/issues/10993))
- Add a `CODEOWNERS` file to automatically request reviews from the `@matrix-org/synapse-core` team on new pull requests. ([\#10994](https://github.com/matrix-org/synapse/issues/10994))
- Add further type hints to `synapse.state`. ([\#11004](https://github.com/matrix-org/synapse/issues/11004))
- Remove the deprecated `BaseHandler` object. ([\#11005](https://github.com/matrix-org/synapse/issues/11005))
- Bump mypy version for CI to 0.910, and pull in new type stubs for dependencies. ([\#11006](https://github.com/matrix-org/synapse/issues/11006))
- Fix CI to run the unit tests without optional deps. ([\#11017](https://github.com/matrix-org/synapse/issues/11017))
- Ensure that cache config tests do not share state. ([\#11019](https://github.com/matrix-org/synapse/issues/11019))
- Add additional type hints to `synapse.server_notices`. ([\#11021](https://github.com/matrix-org/synapse/issues/11021))
- Add additional type hints for `synapse.push`. ([\#11023](https://github.com/matrix-org/synapse/issues/11023))
- When installing the optional developer dependencies, also include the dependencies needed for type-checking and unit testing. ([\#11034](https://github.com/matrix-org/synapse/issues/11034))
- Remove unnecessary list comprehension from `synapse_port_db` to satisfy code style requirements. ([\#11043](https://github.com/matrix-org/synapse/issues/11043))
 2021-10-12 12:33:41 +01:00
Sean Quah
b8b905c4ea
Fix inconsistent behavior of get_last_client_by_ip (#10970) 
Make `get_last_client_by_ip` return the same dictionary structure
regardless of whether the data has been persisted to the database.

This change will allow slightly cleaner type hints to be applied later
on.
 2021-10-12 11:24:05 +01:00
Sean Quah
6b18eb4430
Fix opentracing and Prometheus metrics for replication requests (#10996) 
This commit fixes two bugs to do with decorators not instrumenting
`ReplicationEndpoint`'s `send_request` correctly. There are two
decorators on `send_request`: Prometheus' `Gauge.track_inprogress()`
and Synapse's `opentracing.trace`.

`Gauge.track_inprogress()` does not have any support for async
functions when used as a decorator. Since async functions behave like
regular functions that return coroutines, only the creation of the
coroutine was covered by the metric and none of the actual body of
`send_request`.

`Gauge.track_inprogress()` returns a regular, non-async function
wrapping `send_request`, which is the source of the next bug.
The `opentracing.trace` decorator would normally handle async functions
correctly, but since the wrapped `send_request` is a non-async function,
the decorator ends up suffering from the same issue as
`Gauge.track_inprogress()`: the opentracing span only measures the
creation of the coroutine and none of the actual function body.

Using `Gauge.track_inprogress()` as a context manager instead of a
decorator resolves both bugs.
 2021-10-12 11:23:46 +01:00
Brendan Abolivier
8c5255b664 1.45.0rc1 2021-10-12 10:47:15 +01:00
reivilibre
406f7bfa17
Add an approximate difference method to StateFilters (#10825) 2021-10-12 10:44:59 +01:00
David Robertson
e0f11ae4a5
disallow-untyped-defs for synapse.push (#11023) 2021-10-11 17:42:10 +01:00
reivilibre
3828dd819b
Pass through SynapseErrors that are raised from experimental check_event_allowed callback of the module API (#11042) 
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2021-10-11 16:29:02 +01:00
Eric Eastwood
a7d22c36db
Refactor MSC2716 /batch_send endpoint into separate handler functions (#10974) 2021-10-08 18:35:00 -05:00
Patrick Cloke
1b112840d2
Autodiscover oEmbed endpoint from returned HTML (#10822) 
Searches the returned HTML for an oEmbed endpoint using the
autodiscovery mechanism (`<link rel=...>`), and will request it
to generate the preview.
 2021-10-08 14:14:42 -04:00
David Robertson
51a5da74cc
Annotate synapse.storage.util (#10892) 
Also mark `synapse.streams` as having has no untyped defs

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
 2021-10-08 14:25:16 +00:00
David Robertson
797ee7812d
Relax ignore-missing-imports for modules that have stubs now and update mypy (#11006) 
Updating mypy past version 0.9 means that third-party stubs are no-longer distributed with typeshed. See http://mypy-lang.blogspot.com/2021/06/mypy-0900-released.html for details.
We therefore pull in stub packages in setup.py

Additionally, some modules that we were previously ignoring import failures for now have stubs. So let's use them.

The rest of this change consists of fixups to make the newer mypy + stubs pass CI.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-10-08 14:49:41 +01:00
David Robertson
670a8d9a1e
Fix overwriting profile when making room public (#11003) 
This splits apart `handle_new_user` into a function which adds an entry to the `user_directory` and a function which updates the room sharing tables. I plan to continue doing more of this kind of refactoring to clarify the implementation.
 2021-10-08 12:52:48 +01:00
Patrick Cloke
eb9ddc8c2e
Remove the deprecated BaseHandler. (#11005) 
The shared ratelimit function was replaced with a dedicated
RequestRatelimiter class (accessible from the HomeServer
object).

Other properties were copied to each sub-class that inherited
from BaseHandler.
 2021-10-08 07:44:43 -04:00
Sean Quah
49a683d871
Fix long-standing bug where ReadWriteLock could drop logging contexts (#10993) 
Use `PreserveLoggingContext()` to ensure that logging contexts are not
lost when exiting a read/write lock.

When exiting a read/write lock, callbacks on a `Deferred` are triggered
as a signal to any waiting coroutines. Any waiting coroutine that
becomes runnable is likely to follow the Synapse logging context rules
and will restore its own logging context, then either run to completion
or await another `Deferred`, resetting the logging context in the
process.
 2021-10-08 12:27:16 +01:00
Nick Barrett
bb228f3523
Include exception in json logging (#11028) 2021-10-08 13:08:25 +02:00
David Robertson
e79ee48313
disallow-untyped-defs for synapse.server_notices (#11021) 2021-10-07 19:55:15 +01:00
David Robertson
e0bf34dada
Don't alter directory entries for local users when setting a per-room nickname (#11002) 
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-10-07 13:26:11 +01:00
Richard van der Hoff
96fe77c254
Improve the logging in _auth_and_persist_outliers (#11010) 
Include the event ids being peristed
 2021-10-07 11:43:25 +00:00
Richard van der Hoff
86af6b2f0e
Add a comment in _process_received_pdu (#11011) 2021-10-07 12:20:03 +01:00
Brendan Abolivier
52aefd5086
Catch AttributeErrors when calling registerProducer (#10995) 
Looks like the wrong exception type was caught in #10932.
 2021-10-07 10:37:10 +00:00
David Robertson
f563676c09
disallow-untyped-defs for synapse.state (#11004) 
* `disallow-untyped-defs` for `synapse.state`

Much smaller than I was expecting!
 2021-10-06 17:55:25 +00:00
Brendan Abolivier
4e51621064
Add a spamchecker method to allow or deny 3pid invites (#10894) 
This is in the context of creating new module callbacks that modules in https://github.com/matrix-org/synapse-dinsic can use, in an effort to reconcile the spam checker API in synapse-dinsic with the one in mainline.

Note that a module callback already exists for 3pid invites (https://matrix-org.github.io/synapse/develop/modules/third_party_rules_callbacks.html#check_threepid_can_be_invited) but it doesn't check whether the sender of the invite is allowed to send it.
 2021-10-06 17:18:13 +02:00
Patrick Cloke
f4b1a9a527
Require direct references to configuration variables. (#10985) 
This removes the magic allowing accessing configurable
variables directly from the config object. It is now required
that a specific configuration class is used (e.g. `config.foo`
must be replaced with `config.server.foo`).
 2021-10-06 10:47:41 -04:00
Brendan Abolivier
829f2a82b0
Add a spamchecker callback to allow or deny room joins (#10910) 
Co-authored-by: Erik Johnston <erik@matrix.org>
 2021-10-06 14:32:16 +00:00
David Robertson
370bca32e6
Don't drop user dir deltas when server leaves room (#10982) 
Fix a long-standing bug where a batch of user directory changes would be
silently dropped if the server left a room early in the batch.

* Pull out `wait_for_background_update` in tests

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2021-10-06 12:56:45 +00:00
David Robertson
f8d0f72b27
More types for synapse.util, part 1 (#10888) 
The following modules now pass `disallow_untyped_defs`:

* synapse.util.caches.cached_call 
* synapse.util.caches.lrucache
* synapse.util.caches.response_cache 
* synapse.util.caches.stream_change_cache
* synapse.util.caches.ttlcache pass
* synapse.util.daemonize
* synapse.util.patch_inline_callbacks pass `no-untyped-defs`
* synapse.util.versionstring

Additional typing in synapse.util.metrics. Didn't get this to pass `no-untyped-defs`, think I'll need to watch #10847
 2021-10-06 11:20:49 +01:00
David Robertson
4f00432ce1
Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981) 
There are two steps to rebuilding the user directory:

1. a scan over rooms, followed by
2. a scan over local users.

The former reads avatars and display names from the `room_memberships`
table and therefore contains potentially private avatars and
display names. The latter reads from the the `profiles` table which only
contains public data; moreover it will overwrite any private profiles
that the rooms scan may have written to the user directory. This means
that the rebuild could leak private user while the rebuild was in
progress, only to later cover up the leaks once the rebuild had completed.

This change skips over local users when writing user_directory rows
when scanning rooms. Doing so means that it'll take longer for a rebuild
to make local users searchable, which is unfortunate. I think a future
PR can improve this by swapping the order of the two steps above. (And
indeed there's more to do here, e.g. copying from `profiles` without
going via Python.)

Small tidy-ups while I'm here:

* Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`.
* Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`.
* Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2021-10-05 18:35:25 +01:00
Eric Eastwood
392863fbf1
Fix logic flaw preventing tracking of MSC2716 events in existing room versions (#10962) 
We correctly allowed using the MSC2716 batch endpoint for
the room creator in existing room versions but accidentally didn't track
the events because of a logic flaw.

This prevented you from connecting subsequent chunks together because it would
throw the unknown batch ID error.

We only want to process MSC2716 events when:

 - The room version supports MSC2716
 - Any room where the homeserver has the `msc2716_enabled` experimental feature enabled and the event is from the room creator
 2021-10-05 11:51:57 -05:00
Brendan Abolivier
2faac70e63 Merge branch 'master' into develop 2021-10-05 14:13:25 +01:00
Brendan Abolivier
b2c5e79291 1.44.0 2021-10-05 13:45:24 +01:00
Richard van der Hoff
3a5b0cbe7a
Ensure that we reject events which use rejected events for auth (#10956) 
When we consider whether to accept events, we should not accept those which
depend on rejected events for their auth events.

This (together with earlier changes such as
https://github.com/matrix-org/synapse/pull/10771 and
https://github.com/matrix-org/synapse/pull/10896) forms a partial fix to
https://github.com/matrix-org/synapse/issues/9595. There still remain code
paths where we do not check the `auth_events` at all.
 2021-10-05 13:23:29 +01:00
Richard van der Hoff
787af4a106
Host cache_joined_hosts_for_event to caller (#10986) 
`_check_event_auth` is only called in two places, and only one of those sets
`send_on_behalf_of`. Warming the cache isn't really part of auth anyway, so
moving it out makes a lot more sense.
 2021-10-05 13:01:41 +01:00
Richard van der Hoff
d099535deb
_update_auth_events_and_context_for_auth: add some comments (#10987) 
Add some more comments about wtf is going on here.
 2021-10-05 12:50:38 +01:00
Richard van der Hoff
cb88ed912b
_check_event_auth: move event validation earlier (#10988) 
There's little point in doing a fancy state reconciliation dance if the event
itself is invalid.

Likewise, there's no point checking it again in `_check_for_soft_fail`.
 2021-10-05 12:50:07 +01:00
Sean Quah
660c8c1415
Log stack traces when a missing opentracing span is detected (#10983) 
Make it easier to track down where opentracing spans are going missing
by including stack traces in the logs.
 2021-10-05 12:23:25 +01:00
Hillery Shay
eda8c88b84
Add functionality to remove deactivated users from the monthly_active_users table (#10947) 
* add test

* add function to remove user from monthly active table in deactivate code

* add function to remove user from monthly active table

* add changelog entry

* update changelog number

* requested changes

* update docstring on new function

* fix lint error

* Update synapse/storage/databases/main/monthly_active_users.py

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2021-10-04 08:34:42 -07:00
Brendan Abolivier
730b40dd5e Synapse 1.44.0rc3 (2021-10-04) 
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933))
 - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmFbCRUACgkQHgFcFF8Z
 Fs1n2Q/+P6WZmqNPEyPb4zgEhHgBZtAiYsS8i+6TBdu/viricfe3xmiifmKYyblV
 C07a8Qu14OMgmXKYR8AY0HvLRx+zdhJFGuYa3I0+yY1GTlQWlf+OKnQtsbgdmJ6O
 b8HbmGB1qJCJ0rbbWhCQvx/XXlqqMIATc6qJj3HTwOpWxNL8TymOFg6TGtb+rLkN
 /s2NfxWPJqKMKTLVgZUVjkNGBBtJmu/+Ow3PYz7J9hEW69REONed/014wVgiWx+W
 BJOoUj8dHAS5ZhdKWiSKAzl5A9FQyxUPDwkO44wsK5OIu+dVy4eF9HCMi0EP/9nT
 G3VWwr3z/TA55foL8XdrIPdp1SFqJmIEwDLgaibISD1/8MoC15YzAkt4CoKYOsL6
 EQtDDQal1BNSFZPITz7PWSFGOMgN3tKfMQUqCC+eagHvNfSxVH+J1zNg7ve2/24h
 PbRU/tt4mJiPu7M2Ejj0EWNHyI2fp92ARzzAzQ0JlrPJe34Z/hAiqf7w4kjtJ7Ew
 Lm890EAw2azo7RYU9xevkOsU2CEtLEnKsGMW8pJc9eRxUjRKfk/EW39dCQq9Myhu
 uQFeYHALcn4vu9jGhGyoO9fJDKIxpM76h37Cwu7shg84Gp2ZwucSjwW2l2rZKiIS
 YUeruLOavUueUZYTcyXxAqAAsH8z1hbKmnXIbNxFrcu+NOl+o84=
 =HR3N
 -----END PGP SIGNATURE-----

Merge tag 'v1.44.0rc3' into develop

Synapse 1.44.0rc3 (2021-10-04)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933))
- Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
 2021-10-04 15:33:42 +01:00
Brendan Abolivier
2d2c6a41fe 1.44.0rc3 2021-10-04 14:57:40 +01:00