Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
17,872 Commits 5 Branches 723 Tags 124 MiB
fb56dfdccd
Commit Graph

3637 Commits

Author SHA1 Message Date
Dirk Klimpel
4ac3a8c5dc
Fix a bug in the joined_rooms admin API (#8643) 
If the user was not in any rooms then the API returned the same error
as if the user did not exist.
 2020-10-26 12:25:48 -04:00
Erik Johnston
cf9a17a2b3 Synapse 1.22.0rc2 (2020-10-26) 
==============================
 
 Bugfixes
 --------
 
 - Fix bugs where ephemeral events were not sent to appservices. Broke in v1.22.0rc1. ([\#8648](https://github.com/matrix-org/synapse/issues/8648), [\#8656](https://github.com/matrix-org/synapse/issues/8656))
 - Fix `user_daily_visits` table to not have duplicate rows per user/device due to multiple user agents. Broke in v1.22.0rc1. ([\#8654](https://github.com/matrix-org/synapse/issues/8654))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAl+W6QIQHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCR/EB/9Afs/QiqD7uJ7XtAASwlOxcpSrvB6xSMzU
 VIiYLrUSCvHe8SwcPG6nToy69hHmwLBPu9e8gvlQkOY4nJpW9VTGvhFcP5eCgeKd
 vDp+elVD6BPg+6S3RaI/F5hyQ7gn9qZ/WexkVixDkVs5eAL3gNlZcjw9RwaT9ei9
 fEOYlMi1CRSRzKVBPMEFCAvOpJJ8rA/iq4JK7+P0TplkhD+HydG0cYr6OoL7CmaD
 L/au565Ynp6o0hSpNpTFqW6C5W5JRvz/TyG9/N95h9+WqubZzsPPDUzh98B3eBoX
 43Wu9Qoc3mNamf0++HbHGUPI8dRV1gssxL01hs/iLUZz7CxXY+Ir
 =Z3sQ
 -----END PGP SIGNATURE-----

Merge tag 'v1.22.0rc2' into develop

Synapse 1.22.0rc2 (2020-10-26)
==============================

Bugfixes
--------

- Fix bugs where ephemeral events were not sent to appservices. Broke in v1.22.0rc1. ([\#8648](https://github.com/matrix-org/synapse/issues/8648), [\#8656](https://github.com/matrix-org/synapse/issues/8656))
- Fix `user_daily_visits` table to not have duplicate rows per user/device due to multiple user agents. Broke in v1.22.0rc1. ([\#8654](https://github.com/matrix-org/synapse/issues/8654))
 2020-10-26 15:23:13 +00:00
Erik Johnston
7a3adbd7af 1.22.0rc2 2020-10-26 15:11:03 +00:00
Erik Johnston
ff7f0e8a14 Merge branch 'release-v1.22.0' into develop 2020-10-26 15:02:55 +00:00
Will Hunt
9e0f5a0ac4 Fix get|set_type_stream_id_for_appservice store functions (#8648) 2020-10-26 15:00:56 +00:00
Will Hunt
e8dbbcb64c
Fix get|set_type_stream_id_for_appservice store functions (#8648) 2020-10-26 10:51:33 -04:00
Andrew Morgan
73d8209694
Correct the package name in OpenID Connect install instructions (#8634) 
The OpenID Connect install instructions suggested installing `synapse[oidc]`, but our PyPI package is called `matrix-synapse`.
 2020-10-26 14:45:33 +00:00
Dirk Klimpel
913f8a06e4
Add field total to device list in admin API (#8644) 2020-10-26 14:07:51 +00:00
LEdoian
7b13780c54
Check status codes that profile handler returns (#8580) 
Fixes #8520

Signed-off-by: Pavel Turinsky <pavel.turinsky@matfyz.cz>

Co-authored-by: Erik Johnston <erikj@jki.re>
 2020-10-26 13:55:21 +00:00
Patrick Cloke
5eda018561
Properly handle presence events for application services. (#8656) 2020-10-26 09:19:07 -04:00
Erik Johnston
437a99fb99
Fix user_daily_visits to not have duplicate rows for UA. (#8654) 
* Fix user_daily_visits to not have duplicate rows for UA.

Fixes #8641.

* Newsfile

* Fix typo.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2020-10-26 13:16:32 +00:00
Erik Johnston
2b7c180879
Start fewer opentracing spans (#8640) 
#8567 started a span for every background process. This is good as it means all Synapse code that gets run should be in a span (unless in the sentinel logging context), but it means we generate about 15x the number of spans as we did previously.

This PR attempts to reduce that number by a) not starting one for send commands to Redis, and b) deferring starting background processes until after we're sure they're necessary.

I don't really know how much this will help.
 2020-10-26 09:30:19 +00:00
Patrick Cloke
34a5696f93
Fix typos and spelling errors. (#8639) 2020-10-23 12:38:40 -04:00
Erik Johnston
c850dd9a8e
Fix handling of User-Agent headers with bad utf-8. (#8632) 2020-10-23 17:12:59 +01:00
Erik Johnston
db9ef792f0
Fix email notifications for invites without local state. (#8627) 
This can happen if e.g. the room invited into is no longer on the
server (or if all users left the room).
 2020-10-23 10:41:32 +01:00
Andrew Morgan
f28756bb40 Changelog 2020-10-22 18:33:02 +01:00
Erik Johnston
054a6b9538 Synapse 1.22.0rc1 (2020-10-22) 
==============================
 
 Features
 --------
 
 - Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. ([\#7658](https://github.com/matrix-org/synapse/issues/7658))
 - Add ability for `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory. ([\#8292](https://github.com/matrix-org/synapse/issues/8292), [\#8467](https://github.com/matrix-org/synapse/issues/8467))
 - Add support for olm fallback keys ([MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732)). ([\#8312](https://github.com/matrix-org/synapse/issues/8312), [\#8501](https://github.com/matrix-org/synapse/issues/8501))
 - Add support for running background tasks in a separate worker process. ([\#8369](https://github.com/matrix-org/synapse/issues/8369), [\#8458](https://github.com/matrix-org/synapse/issues/8458), [\#8489](https://github.com/matrix-org/synapse/issues/8489), [\#8513](https://github.com/matrix-org/synapse/issues/8513), [\#8544](https://github.com/matrix-org/synapse/issues/8544), [\#8599](https://github.com/matrix-org/synapse/issues/8599))
 - Add support for device dehydration ([MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697)). ([\#8380](https://github.com/matrix-org/synapse/issues/8380))
 - Add support for [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409), which allows sending typing, read receipts, and presence events to appservices. ([\#8437](https://github.com/matrix-org/synapse/issues/8437), [\#8590](https://github.com/matrix-org/synapse/issues/8590))
 - Change default room version to "6", per [MSC2788](https://github.com/matrix-org/matrix-doc/pull/2788). ([\#8461](https://github.com/matrix-org/synapse/issues/8461))
 - Add the ability to send non-membership events into a room via the `ModuleApi`. ([\#8479](https://github.com/matrix-org/synapse/issues/8479))
 - Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. ([\#8502](https://github.com/matrix-org/synapse/issues/8502))
 - Add support for modifying event content in `ThirdPartyRules` modules. ([\#8535](https://github.com/matrix-org/synapse/issues/8535), [\#8564](https://github.com/matrix-org/synapse/issues/8564))
 
 Bugfixes
 --------
 
 - Fix a longstanding bug where invalid ignored users in account data could break clients. ([\#8454](https://github.com/matrix-org/synapse/issues/8454))
 - Fix a bug where backfilling a room with an event that was missing the `redacts` field would break. ([\#8457](https://github.com/matrix-org/synapse/issues/8457))
 - Don't attempt to respond to some requests if the client has already disconnected. ([\#8465](https://github.com/matrix-org/synapse/issues/8465))
 - Fix message duplication if something goes wrong after persisting the event. ([\#8476](https://github.com/matrix-org/synapse/issues/8476))
 - Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. ([\#8486](https://github.com/matrix-org/synapse/issues/8486))
 - Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. ([\#8504](https://github.com/matrix-org/synapse/issues/8504))
 - Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`. ([\#8517](https://github.com/matrix-org/synapse/issues/8517))
 - Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table. ([\#8527](https://github.com/matrix-org/synapse/issues/8527))
 - Fix not sending events over federation when using sharded event writers. ([\#8536](https://github.com/matrix-org/synapse/issues/8536))
 - Fix a long standing bug where email notifications for encrypted messages were blank. ([\#8545](https://github.com/matrix-org/synapse/issues/8545))
 - Fix increase in the number of `There was no active span...` errors logged when using OpenTracing. ([\#8567](https://github.com/matrix-org/synapse/issues/8567))
 - Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed. ([\#8585](https://github.com/matrix-org/synapse/issues/8585))
 - Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. ([\#8606](https://github.com/matrix-org/synapse/issues/8606))
 
 Updates to the Docker image
 ---------------------------
 
 - Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. ([\#7921](https://github.com/matrix-org/synapse/issues/7921))
 - Add support for passing commandline args to the synapse process. Contributed by @samuel-p. ([\#8390](https://github.com/matrix-org/synapse/issues/8390))
 
 Improved Documentation
 ----------------------
 
 - Update the directions for using the manhole with coroutines. ([\#8462](https://github.com/matrix-org/synapse/issues/8462))
 - Improve readme by adding new shield.io badges. ([\#8493](https://github.com/matrix-org/synapse/issues/8493))
 - Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @Maquis196. ([\#8526](https://github.com/matrix-org/synapse/issues/8526))
 - Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration. ([\#8529](https://github.com/matrix-org/synapse/issues/8529))
 
 Deprecations and Removals
 -------------------------
 
 - Drop unused `device_max_stream_id` table. ([\#8589](https://github.com/matrix-org/synapse/issues/8589))
 
 Internal Changes
 ----------------
 
 - Check for unreachable code with mypy. ([\#8432](https://github.com/matrix-org/synapse/issues/8432))
 - Add unit test for event persister sharding. ([\#8433](https://github.com/matrix-org/synapse/issues/8433))
 - Allow events to be sent to clients sooner when using sharded event persisters. ([\#8439](https://github.com/matrix-org/synapse/issues/8439), [\#8488](https://github.com/matrix-org/synapse/issues/8488), [\#8496](https://github.com/matrix-org/synapse/issues/8496), [\#8499](https://github.com/matrix-org/synapse/issues/8499))
 - Configure `public_baseurl` when using demo scripts. ([\#8443](https://github.com/matrix-org/synapse/issues/8443))
 - Add SQL logging on queries that happen during startup. ([\#8448](https://github.com/matrix-org/synapse/issues/8448))
 - Speed up unit tests when using PostgreSQL. ([\#8450](https://github.com/matrix-org/synapse/issues/8450))
 - Remove redundant database loads of stream_ordering for events we already have. ([\#8452](https://github.com/matrix-org/synapse/issues/8452))
 - Reduce inconsistencies between codepaths for membership and non-membership events. ([\#8463](https://github.com/matrix-org/synapse/issues/8463))
 - Combine `SpamCheckerApi` with the more generic `ModuleApi`. ([\#8464](https://github.com/matrix-org/synapse/issues/8464))
 - Additional testing for `ThirdPartyEventRules`. ([\#8468](https://github.com/matrix-org/synapse/issues/8468))
 - Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit. ([\#8472](https://github.com/matrix-org/synapse/issues/8472))
 - Unblacklist some sytests. ([\#8474](https://github.com/matrix-org/synapse/issues/8474))
 - Include the log level in the phone home stats. ([\#8477](https://github.com/matrix-org/synapse/issues/8477))
 - Remove outdated sphinx documentation, scripts and configuration. ([\#8480](https://github.com/matrix-org/synapse/issues/8480))
 - Clarify error message when plugin config parsers raise an error. ([\#8492](https://github.com/matrix-org/synapse/issues/8492))
 - Remove the deprecated `Handlers` object. ([\#8494](https://github.com/matrix-org/synapse/issues/8494))
 - Fix a threadsafety bug in unit tests. ([\#8497](https://github.com/matrix-org/synapse/issues/8497))
 - Add user agent to user_daily_visits table. ([\#8503](https://github.com/matrix-org/synapse/issues/8503))
 - Add type hints to various parts of the code base. ([\#8407](https://github.com/matrix-org/synapse/issues/8407), [\#8505](https://github.com/matrix-org/synapse/issues/8505), [\#8507](https://github.com/matrix-org/synapse/issues/8507), [\#8547](https://github.com/matrix-org/synapse/issues/8547), [\#8562](https://github.com/matrix-org/synapse/issues/8562), [\#8609](https://github.com/matrix-org/synapse/issues/8609))
 - Remove unused code from the test framework. ([\#8514](https://github.com/matrix-org/synapse/issues/8514))
 - Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable. ([\#8515](https://github.com/matrix-org/synapse/issues/8515))
 - Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`. ([\#8537](https://github.com/matrix-org/synapse/issues/8537))
 - Improve database performance by executing more queries without starting transactions. ([\#8542](https://github.com/matrix-org/synapse/issues/8542))
 - Rename `Cache` to `DeferredCache`, to better reflect its purpose. ([\#8548](https://github.com/matrix-org/synapse/issues/8548))
 - Move metric registration code down into `LruCache`. ([\#8561](https://github.com/matrix-org/synapse/issues/8561), [\#8591](https://github.com/matrix-org/synapse/issues/8591))
 - Replace `DeferredCache` with the lighter-weight `LruCache` where possible. ([\#8563](https://github.com/matrix-org/synapse/issues/8563))
 - Add virtualenv-generated folders to `.gitignore`. ([\#8566](https://github.com/matrix-org/synapse/issues/8566))
 - Add `get_immediate` method to `DeferredCache`. ([\#8568](https://github.com/matrix-org/synapse/issues/8568))
 - Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`. ([\#8569](https://github.com/matrix-org/synapse/issues/8569))
 - Fix `synmark` benchmark runner. ([\#8571](https://github.com/matrix-org/synapse/issues/8571))
 - Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s. ([\#8572](https://github.com/matrix-org/synapse/issues/8572))
 - Adjust a protocol-type definition to fit `sqlite3` assertions. ([\#8577](https://github.com/matrix-org/synapse/issues/8577))
 - Support macOS on the `synmark` benchmark runner. ([\#8578](https://github.com/matrix-org/synapse/issues/8578))
 - Update `mypy` static type checker to 0.790. ([\#8583](https://github.com/matrix-org/synapse/issues/8583), [\#8600](https://github.com/matrix-org/synapse/issues/8600))
 - Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8587](https://github.com/matrix-org/synapse/issues/8587))
 - Remove extraneous unittest logging decorators from unit tests. ([\#8592](https://github.com/matrix-org/synapse/issues/8592))
 - Minor optimisations in caching code. ([\#8593](https://github.com/matrix-org/synapse/issues/8593), [\#8594](https://github.com/matrix-org/synapse/issues/8594))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAl+RfFAQHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCZI4CACibRDTo+DFIp787ZMErW0xFc3S8DOA6K4u
 G8J2GCrBT0pfWAi7jqfQO9GySlvS5ixqbWYxud9jG0elXtnVgjIcbToSCYvNFxzH
 TDaQAaOwFVe/7fhZG6TvO2EcAXYXNfoqQKwc8xvTOcBOhObfLAzchV6t7mRTRYE4
 +C3kM8BUJUnSp+OAY0zxRrUIVjX4W280BQ/+UxvPBLIi0IUQz/BufvTtgEhRBb6B
 eK2qjDMTud3/WAzM/TYDoA/LmdAcFCogfkRRHYqDPw0gUIXm1tdHyLWdgHoV+WhX
 6p/kacnKxqonKW5/UrqheYT8J0SMA1AlkSm2niy/K+Tde3Nga+Sw
 =TLb5
 -----END PGP SIGNATURE-----

Merge tag 'v1.22.0rc1' into develop

Synapse 1.22.0rc1 (2020-10-22)
==============================

Features
--------

- Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. ([\#7658](https://github.com/matrix-org/synapse/issues/7658))
- Add ability for `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory. ([\#8292](https://github.com/matrix-org/synapse/issues/8292), [\#8467](https://github.com/matrix-org/synapse/issues/8467))
- Add support for olm fallback keys ([MSC2732](https://github.com/matrix-org/matrix-doc/pull/2732)). ([\#8312](https://github.com/matrix-org/synapse/issues/8312), [\#8501](https://github.com/matrix-org/synapse/issues/8501))
- Add support for running background tasks in a separate worker process. ([\#8369](https://github.com/matrix-org/synapse/issues/8369), [\#8458](https://github.com/matrix-org/synapse/issues/8458), [\#8489](https://github.com/matrix-org/synapse/issues/8489), [\#8513](https://github.com/matrix-org/synapse/issues/8513), [\#8544](https://github.com/matrix-org/synapse/issues/8544), [\#8599](https://github.com/matrix-org/synapse/issues/8599))
- Add support for device dehydration ([MSC2697](https://github.com/matrix-org/matrix-doc/pull/2697)). ([\#8380](https://github.com/matrix-org/synapse/issues/8380))
- Add support for [MSC2409](https://github.com/matrix-org/matrix-doc/pull/2409), which allows sending typing, read receipts, and presence events to appservices. ([\#8437](https://github.com/matrix-org/synapse/issues/8437), [\#8590](https://github.com/matrix-org/synapse/issues/8590))
- Change default room version to "6", per [MSC2788](https://github.com/matrix-org/matrix-doc/pull/2788). ([\#8461](https://github.com/matrix-org/synapse/issues/8461))
- Add the ability to send non-membership events into a room via the `ModuleApi`. ([\#8479](https://github.com/matrix-org/synapse/issues/8479))
- Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. ([\#8502](https://github.com/matrix-org/synapse/issues/8502))
- Add support for modifying event content in `ThirdPartyRules` modules. ([\#8535](https://github.com/matrix-org/synapse/issues/8535), [\#8564](https://github.com/matrix-org/synapse/issues/8564))

Bugfixes
--------

- Fix a longstanding bug where invalid ignored users in account data could break clients. ([\#8454](https://github.com/matrix-org/synapse/issues/8454))
- Fix a bug where backfilling a room with an event that was missing the `redacts` field would break. ([\#8457](https://github.com/matrix-org/synapse/issues/8457))
- Don't attempt to respond to some requests if the client has already disconnected. ([\#8465](https://github.com/matrix-org/synapse/issues/8465))
- Fix message duplication if something goes wrong after persisting the event. ([\#8476](https://github.com/matrix-org/synapse/issues/8476))
- Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. ([\#8486](https://github.com/matrix-org/synapse/issues/8486))
- Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. ([\#8504](https://github.com/matrix-org/synapse/issues/8504))
- Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`. ([\#8517](https://github.com/matrix-org/synapse/issues/8517))
- Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table. ([\#8527](https://github.com/matrix-org/synapse/issues/8527))
- Fix not sending events over federation when using sharded event writers. ([\#8536](https://github.com/matrix-org/synapse/issues/8536))
- Fix a long standing bug where email notifications for encrypted messages were blank. ([\#8545](https://github.com/matrix-org/synapse/issues/8545))
- Fix increase in the number of `There was no active span...` errors logged when using OpenTracing. ([\#8567](https://github.com/matrix-org/synapse/issues/8567))
- Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed. ([\#8585](https://github.com/matrix-org/synapse/issues/8585))
- Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. ([\#8606](https://github.com/matrix-org/synapse/issues/8606))

Updates to the Docker image
---------------------------

- Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. ([\#7921](https://github.com/matrix-org/synapse/issues/7921))
- Add support for passing commandline args to the synapse process. Contributed by @samuel-p. ([\#8390](https://github.com/matrix-org/synapse/issues/8390))

Improved Documentation
----------------------

- Update the directions for using the manhole with coroutines. ([\#8462](https://github.com/matrix-org/synapse/issues/8462))
- Improve readme by adding new shield.io badges. ([\#8493](https://github.com/matrix-org/synapse/issues/8493))
- Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @Maquis196. ([\#8526](https://github.com/matrix-org/synapse/issues/8526))
- Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration. ([\#8529](https://github.com/matrix-org/synapse/issues/8529))

Deprecations and Removals
-------------------------

- Drop unused `device_max_stream_id` table. ([\#8589](https://github.com/matrix-org/synapse/issues/8589))

Internal Changes
----------------

- Check for unreachable code with mypy. ([\#8432](https://github.com/matrix-org/synapse/issues/8432))
- Add unit test for event persister sharding. ([\#8433](https://github.com/matrix-org/synapse/issues/8433))
- Allow events to be sent to clients sooner when using sharded event persisters. ([\#8439](https://github.com/matrix-org/synapse/issues/8439), [\#8488](https://github.com/matrix-org/synapse/issues/8488), [\#8496](https://github.com/matrix-org/synapse/issues/8496), [\#8499](https://github.com/matrix-org/synapse/issues/8499))
- Configure `public_baseurl` when using demo scripts. ([\#8443](https://github.com/matrix-org/synapse/issues/8443))
- Add SQL logging on queries that happen during startup. ([\#8448](https://github.com/matrix-org/synapse/issues/8448))
- Speed up unit tests when using PostgreSQL. ([\#8450](https://github.com/matrix-org/synapse/issues/8450))
- Remove redundant database loads of stream_ordering for events we already have. ([\#8452](https://github.com/matrix-org/synapse/issues/8452))
- Reduce inconsistencies between codepaths for membership and non-membership events. ([\#8463](https://github.com/matrix-org/synapse/issues/8463))
- Combine `SpamCheckerApi` with the more generic `ModuleApi`. ([\#8464](https://github.com/matrix-org/synapse/issues/8464))
- Additional testing for `ThirdPartyEventRules`. ([\#8468](https://github.com/matrix-org/synapse/issues/8468))
- Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit. ([\#8472](https://github.com/matrix-org/synapse/issues/8472))
- Unblacklist some sytests. ([\#8474](https://github.com/matrix-org/synapse/issues/8474))
- Include the log level in the phone home stats. ([\#8477](https://github.com/matrix-org/synapse/issues/8477))
- Remove outdated sphinx documentation, scripts and configuration. ([\#8480](https://github.com/matrix-org/synapse/issues/8480))
- Clarify error message when plugin config parsers raise an error. ([\#8492](https://github.com/matrix-org/synapse/issues/8492))
- Remove the deprecated `Handlers` object. ([\#8494](https://github.com/matrix-org/synapse/issues/8494))
- Fix a threadsafety bug in unit tests. ([\#8497](https://github.com/matrix-org/synapse/issues/8497))
- Add user agent to user_daily_visits table. ([\#8503](https://github.com/matrix-org/synapse/issues/8503))
- Add type hints to various parts of the code base. ([\#8407](https://github.com/matrix-org/synapse/issues/8407), [\#8505](https://github.com/matrix-org/synapse/issues/8505), [\#8507](https://github.com/matrix-org/synapse/issues/8507), [\#8547](https://github.com/matrix-org/synapse/issues/8547), [\#8562](https://github.com/matrix-org/synapse/issues/8562), [\#8609](https://github.com/matrix-org/synapse/issues/8609))
- Remove unused code from the test framework. ([\#8514](https://github.com/matrix-org/synapse/issues/8514))
- Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable. ([\#8515](https://github.com/matrix-org/synapse/issues/8515))
- Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`. ([\#8537](https://github.com/matrix-org/synapse/issues/8537))
- Improve database performance by executing more queries without starting transactions. ([\#8542](https://github.com/matrix-org/synapse/issues/8542))
- Rename `Cache` to `DeferredCache`, to better reflect its purpose. ([\#8548](https://github.com/matrix-org/synapse/issues/8548))
- Move metric registration code down into `LruCache`. ([\#8561](https://github.com/matrix-org/synapse/issues/8561), [\#8591](https://github.com/matrix-org/synapse/issues/8591))
- Replace `DeferredCache` with the lighter-weight `LruCache` where possible. ([\#8563](https://github.com/matrix-org/synapse/issues/8563))
- Add virtualenv-generated folders to `.gitignore`. ([\#8566](https://github.com/matrix-org/synapse/issues/8566))
- Add `get_immediate` method to `DeferredCache`. ([\#8568](https://github.com/matrix-org/synapse/issues/8568))
- Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`. ([\#8569](https://github.com/matrix-org/synapse/issues/8569))
- Fix `synmark` benchmark runner. ([\#8571](https://github.com/matrix-org/synapse/issues/8571))
- Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s. ([\#8572](https://github.com/matrix-org/synapse/issues/8572))
- Adjust a protocol-type definition to fit `sqlite3` assertions. ([\#8577](https://github.com/matrix-org/synapse/issues/8577))
- Support macOS on the `synmark` benchmark runner. ([\#8578](https://github.com/matrix-org/synapse/issues/8578))
- Update `mypy` static type checker to 0.790. ([\#8583](https://github.com/matrix-org/synapse/issues/8583), [\#8600](https://github.com/matrix-org/synapse/issues/8600))
- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8587](https://github.com/matrix-org/synapse/issues/8587))
- Remove extraneous unittest logging decorators from unit tests. ([\#8592](https://github.com/matrix-org/synapse/issues/8592))
- Minor optimisations in caching code. ([\#8593](https://github.com/matrix-org/synapse/issues/8593), [\#8594](https://github.com/matrix-org/synapse/issues/8594))
 2020-10-22 13:37:08 +01:00
Patrick Cloke
514a240aed
Remove unused OPTIONS handlers. (#8621) 
The handling of OPTIONS requests was consolidated in #7534, but the endpoint
specific handlers were not removed.
 2020-10-22 08:35:55 -04:00
Erik Johnston
b19b63e6b4
Don't 500 for invalid group IDs (#8628) 2020-10-22 13:19:06 +01:00
Erik Johnston
ec0e9c4695 1.22.0rc1 2020-10-22 13:08:42 +01:00
Erik Johnston
a9f90fa73a
Type hints for RegistrationStore (#8615) 2020-10-22 11:56:58 +01:00
Erik Johnston
2ac908f377
Don't instansiate Requester directly (#8614) 2020-10-22 10:11:06 +01:00
Richard van der Hoff
b28aaeb3a5
Optimise CacheDescriptor (#8594) 
don't bother constricting a CacheContext unless we need one.
 2020-10-21 22:57:45 +01:00
Richard van der Hoff
2b3af01791 optimise DeferredCache.set 2020-10-21 17:55:53 +01:00
Richard van der Hoff
9146a8a691
Merge pull request #8572 from matrix-org/rav/cache_hacking/2 
Push some deferred wrangling down into DeferredCache
 2020-10-21 17:55:04 +01:00
Richard van der Hoff
1f4269700c Push some deferred wrangling down into DeferredCache 2020-10-21 15:39:25 +01:00
Will Hunt
70259d8c8c
Limit AS transactions to 100 events (#8606) 
* Limit AS transactions to 100 events

* Update changelog.d/8606.feature

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

* Add tests

* Update synapse/appservice/scheduler.py

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
 2020-10-21 15:36:53 +01:00
Patrick Cloke
20a67aa70d
Separate the TCP and terse JSON formatting code. (#8587) 
This should (theoretically) allow for using the TCP code with a different output type
and make it easier to use the JSON code with files / console.
 2020-10-21 06:59:54 -04:00
Patrick Cloke
654cc9470e
Pin mypy-zope for compatibility with mypy. (#8600) 2020-10-21 06:45:01 -04:00
Patrick Cloke
de5cafe980
Add type hints to profile and base handlers. (#8609) 2020-10-21 06:44:31 -04:00
Patrick Cloke
9e0f22874f
Consistently use wrap_as_background_task in more places (#8599) 2020-10-20 11:29:38 -04:00
Jonathan de Jong
84c0e46cce
Update mypy to 0.790, and move dependencies to extras (#8583) 2020-10-20 07:55:21 -04:00
Andrew Morgan
74f29284aa
Remove some extraneous @unittest.INFOs on unit tests (#8592) 2020-10-20 11:49:15 +01:00
Andrew Morgan
a312e890f5
Cast errors generated during synapse_port_db to str (#8585) 
I noticed in https://github.com/matrix-org/synapse/issues/8575 that the `end_error` variable in `synapse_port_db` is set to an `Exception`, even though later we expect it to be a `str`.

This PR simply casts an exception raised to a string. I'm doing this instead of having `end_error` be of type exception as we explicitly set `end_error` to a str here:

d25eb8f370/scripts/synapse_port_db (L542-L547)

This whole file could probably use some heavy refactoring, but until then at least this fix will prevent exception contents from being hidden from us and users.
 2020-10-20 11:47:24 +01:00
Will Hunt
626b8f0846
Move schema file for as_device_stream (#8590) 
* Move schema file

* Add a .

* Add matching changelog entry

* Fix sqlite
 2020-10-20 10:18:55 +01:00
Richard van der Hoff
96e7d3c4a0
Fix 'LruCache' object has no attribute '_on_resize' (#8591) 
We need to make sure we are readu for the `set_cache_factor` callback.
 2020-10-19 21:13:50 +01:00
Vasilis Gerakaris
34c20493b9
Drop unused device_max_stream_id table (#8589) 
Signed-off-by: Vasilis Gerakaris <vasilis.gerakaris@navarino.gr>
 2020-10-19 19:06:54 +01:00
Jonathan de Jong
21bb50ca3f
Fix mypy error: auth handler "checkpw" internal function type mismatch (#8569) 2020-10-19 18:32:24 +01:00
Patrick Cloke
8f27b7fde1
Expose the experimental appservice login flow to clients. (#8504) 2020-10-19 13:03:55 -04:00
Richard van der Hoff
903d11c43a
Add DeferredCache.get_immediate method (#8568) 
* Add `DeferredCache.get_immediate` method

A bunch of things that are currently calling `DeferredCache.get` are only
really interested in the result if it's completed. We can optimise and simplify
this case.

* Remove unused 'default' parameter to DeferredCache.get()

* another get_immediate instance
 2020-10-19 15:00:12 +01:00
Patrick Cloke
c356b4bf42
Include a simple message in email notifications that include encrypted content (#8545) 2020-10-19 09:12:39 -04:00
Patrick Cloke
85c56445fb
Support running synmark on macOS. (#8578) 
By using the "poll" reactor since macOS doesn't support epoll.
 2020-10-19 07:27:46 -04:00
Erik Johnston
1fcdbeb3ab
Start an opentracing span for background processes. (#8567) 
This should reduce the number of `There was no active span` errors we
see.

Fixes #8510.
 2020-10-19 12:26:26 +01:00
Richard van der Hoff
97647b33c2
Replace DeferredCache with LruCache where possible (#8563) 
Most of these uses don't need a full-blown DeferredCache; LruCache is lighter and more appropriate.
 2020-10-19 12:20:29 +01:00
Jonathan de Jong
79c1f973ce
Pre-emptively fix synapse.storage.types.Connection for future mypy release (#8577) 
Fix the Connection protocol according to typeshed's assertions about sqlite3.Connection
 2020-10-17 09:51:38 +01:00
Richard van der Hoff
0afd83584b
Fix synmark (#8571) 
This seems to have been broken since #6513.
 2020-10-16 21:45:31 +01:00
Richard van der Hoff
d6094176d1
Type annotations for LruCache (#8562) 
* type annotations for LruCache

* changelog

* Apply suggestions from code review

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* review comments

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2020-10-16 17:06:50 +01:00
Patrick Cloke
1b70662be9
Clean-up old transaction IDs on the background worker. (#8544) 2020-10-16 12:06:17 -04:00
Jonathan de Jong
c8e9dc4cf4
Add .venv* to .gitignore (#8566) 
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
 2020-10-16 17:03:38 +01:00
Richard van der Hoff
402213bf41 changelog 2020-10-16 15:56:47 +01:00
Richard van der Hoff
3ee17585cd
Make LruCache register its own metrics (#8561) 
rather than have everything that instantiates an LruCache manage metrics
separately, have LruCache do it itself.
 2020-10-16 15:51:57 +01:00
Richard van der Hoff
da0090fdff
Fix modifying events in ThirdPartyRules modules (#8564) 
EventBuilder.build wants auth events these days
 2020-10-16 13:39:46 +01:00
Richard van der Hoff
5649669c3c
Merge pull request #8535 from matrix-org/rav/third_party_events_updates 
Support modifying event content from ThirdPartyRules modules
 2020-10-15 20:55:41 +01:00
Jonathan de Jong
6b5a115c0a
Solidify the HomeServer constructor. (#8515) 
This implements a more standard API for instantiating a homeserver and
moves some of the dependency injection into the test suite.

More concretely this stops using `setattr` on all `kwargs` passed to `HomeServer`.
 2020-10-15 15:29:13 -04:00
Will Hunt
c276bd9969
Send some ephemeral events to appservices (#8437) 
Optionally sends typing, presence, and read receipt information to appservices.
 2020-10-15 12:33:28 -04:00
Andrew Morgan
654e239b25
Add option to scripts-dev/lint.sh to only lint files changed since the last git commit (#8472) 
This PR makes several changes to the `./scripts-dev/lint.sh` script, which lints the codebase with a number of tools:

* Adds usage information, with `-h` flag to show it. Otherwise it will show when providing an unknown flag.
* Adds option `-d` which will check both staged and unstaged files that have changed since the last commit and add them to the list of files to lint.
  - Note that only files without an extension, or with a `.py` extension will be allowed. This prevents editing bash scripts causing the linters to break on non-python files.
* Improves the print-out of which files/directories are being linted.
 2020-10-15 15:45:13 +01:00
Patrick Cloke
74976a8e43 Merge branch 'master' into develop 2020-10-15 10:43:54 -04:00
Patrick Cloke
9991aaa49c 1.21.2 2020-10-15 09:24:10 -04:00
Erik Johnston
20fa83f374 Remove racey assertion in MultiWriterIDGenerator (#8530) 
We asserted that the IDs returned by postgres sequence was greater than
any we had seen, however this is technically racey as we may update the
current positions out of order.

We now assert that the sequences are correct on startup, so the
assertion is no longer really required, so we remove them.
 2020-10-15 09:14:29 -04:00
Richard van der Hoff
8075504a60
Enable mypy for synapse.util.caches (#8547) 
This seemed to entail dragging in a type stub for SortedList.
 2020-10-15 11:44:39 +01:00
Richard van der Hoff
0a08cd1065
Merge pull request #8548 from matrix-org/rav/deferred_cache 
Rename Cache to DeferredCache, and related changes
 2020-10-15 11:42:07 +01:00
Neil Johnson
1f39155071
Include user agent in user daily visits table (#8503) 
Include user agent in user daily visits table.
 2020-10-15 10:36:40 +01:00
Richard van der Hoff
4433d01519
Merge pull request #8537 from matrix-org/rav/simplify_locally_reject_invite 
Simplify `_locally_reject_invite`
 2020-10-15 10:20:19 +01:00
Richard van der Hoff
27cfd712b3 changelog 2020-10-14 23:49:27 +01:00
Erik Johnston
19b15d63e8
Use autocommit mode for single statement DB functions. (#8542) 
Autocommit means that we don't wrap the functions in transactions, and instead get executed directly. Introduced in #8456. This will help:

1. reduce the number of `could not serialize access due to concurrent delete` errors that we see (though there are a few functions that often cause serialization errors that we don't fix here);
2. improve the DB performance, as it no longer needs to deal with the overhead of `REPEATABLE READ` isolation levels; and
3. improve wall clock speed of these functions, as we no longer need to send `BEGIN` and `COMMIT` to the DB.

Some notes about the differences between autocommit mode and our default `REPEATABLE READ` transactions:

1. Currently `autocommit` only applies when using PostgreSQL, and is ignored when using SQLite (due to silliness with [Twisted DB classes](https://twistedmatrix.com/trac/ticket/9998)).
2. Autocommit functions may get retried on error, which means they can get applied *twice* (or more) to the DB (since they are not in a transaction the previous call would not get rolled back). This means that the functions need to be idempotent (or otherwise not care about being called multiple times). Read queries, simple deletes, and updates/upserts that replace rows (rather than generating new values from existing rows) are all idempotent.
3. Autocommit functions no longer get executed in [`REPEATABLE READ`](https://www.postgresql.org/docs/current/transaction-iso.html) isolation level, and so data can change queries, which is fine for single statement queries.
 2020-10-14 15:50:59 +01:00
Erik Johnston
618d405a32
Remove racey assertion in MultiWriterIDGenerator (#8530) 
We asserted that the IDs returned by postgres sequence was greater than
any we had seen, however this is technically racey as we may update the
current positions out of order.

We now assert that the sequences are correct on startup, so the
assertion is no longer really required, so we remove them.
 2020-10-14 15:40:06 +01:00
Christopher May-Townsend
1cf4a68108
Add note to manhole.md about bind_address when using with docker (#8526) 
Signed-off-by: Christopher May-Townsend <chris@maytownsend.co.uk>
 2020-10-14 15:28:59 +01:00
Brendan Abolivier
9e66f3761c
Update documentation on retention policies limits (#8529) 
* Update documentation on retention policies limits

Document the changes from https://github.com/matrix-org/synapse/pull/8104
 2020-10-14 15:00:49 +01:00
Erik Johnston
1264c8ac89
Add basic tests for sync/pagination with vector clock tokens. (#8488) 
These are tests for #8439
 2020-10-14 13:53:20 +01:00
Erik Johnston
921a3f8a59
Fix not sending events over federation when using sharded event persisters (#8536) 
* Fix outbound federaion with multiple event persisters.

We incorrectly notified federation senders that the minimum persisted
stream position had advanced when we got an `RDATA` from an event
persister.

Notifying of federation senders already correctly happens in the
notifier, so we just delete the offending line.

* Change some interfaces to use RoomStreamToken.

By enforcing use of `RoomStreamTokens` we make it less likely that
people pass in random ints that they got from somewhere random.
 2020-10-14 13:27:51 +01:00
Brendan Abolivier
3ee97a2748
Make sure a retention policy is a state event (#8527) 
* Make sure a retention policy is a state event

* Changelog
 2020-10-14 12:00:52 +01:00
Aaron Raimist
ec606ea9e3
Add correct M_BAD_JSON error code to /profile/{userId}/displayname (#8517) 
Fixes #8029
 2020-10-14 11:24:58 +01:00
Richard van der Hoff
d9dc6185d3 changelog 2020-10-13 23:58:48 +01:00
Richard van der Hoff
091e9482af changelog 2020-10-13 23:24:50 +01:00
Patrick Cloke
629a951b49
Move additional tasks to the background worker, part 4 (#8513) 2020-10-13 08:20:32 -04:00
Erik Johnston
b2486f6656
Fix message duplication if something goes wrong after persisting the event (#8476) 
Should fix #3365.
 2020-10-13 12:07:56 +01:00
Christopher May-Townsend
b76f53bb79
Multi arch docker support: add arm/v7 and arm64 to our docker images (#7921) 
Signed-off-by: Christopher May-Townsend (chris@maytownsend.co.uk)
 2020-10-12 22:00:33 +01:00
Erik Johnston
8de3703d21
Make event persisters periodically announce position over replication. (#8499) 
Currently background proccesses stream the events stream use the "minimum persisted position" (i.e. `get_current_token()`) rather than the vector clock style tokens. This is broadly fine as it doesn't matter if the background processes lag a small amount. However, in extreme cases (i.e. SyTests) where we only write to one event persister the background processes will never make progress.

This PR changes it so that the `MultiWriterIDGenerator` keeps the current position of a given instance as up to date as possible (i.e using the latest token it sees if its not in the process of persisting anything), and then periodically announces that over replication. This then allows the "minimum persisted position" to advance, albeit with a small lag.
 2020-10-12 15:51:41 +01:00
Samuel Philipp
6905f5751a
Docker: support passing additional commandline args to synapse (#8390) 2020-10-11 20:51:11 +01:00
Patrick Cloke
d35a451399
Clean-up some broken/unused code in the test framework (#8514) 2020-10-09 14:19:29 -04:00
Richard van der Hoff
9789b1fba5
Fix threadsafety in ThreadedMemoryReactorClock (#8497) 
This could, very occasionally, cause:

```
tests.test_visibility.FilterEventsForServerTestCase.test_large_room
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/src/tests/rest/media/v1/test_media_storage.py", line 86, in test_ensure_media_is_in_local_cache
    self.wait_on_thread(x)
  File "/src/tests/unittest.py", line 296, in wait_on_thread
    self.reactor.advance(0.01)
  File "/src/.tox/py35/lib/python3.5/site-packages/twisted/internet/task.py", line 826, in advance
    self._sortCalls()
  File "/src/.tox/py35/lib/python3.5/site-packages/twisted/internet/task.py", line 787, in _sortCalls
    self.calls.sort(key=lambda a: a.getTime())
builtins.ValueError: list modified during sort

tests.rest.media.v1.test_media_storage.MediaStorageTests.test_ensure_media_is_in_local_cache
```
 2020-10-09 17:22:25 +01:00
Mateusz Przybyłowicz
ca2db5dd0c
Increase default max_upload_size from 10M to 50M (#8502) 
Signed-off-by: Mateusz Przybyłowicz <uamfhq@gmail.com>
 2020-10-09 16:58:23 +01:00
Patrick Cloke
1781bbe319
Add type hints to response cache. (#8507) 2020-10-09 11:35:11 -04:00
Andrew Morgan
66ac4b1e34
Allow modules to create and send events into rooms (#8479) 
This PR allows Synapse modules making use of the `ModuleApi` to create and send non-membership events into a room. This can useful to have modules send messages, or change power levels in a room etc. Note that they must send event through a user that's already in the room.

The non-membership event limitation is currently arbitrary, as it's another chunk of work and not necessary at the moment.
 2020-10-09 13:46:36 +01:00
Erik Johnston
5009ffcaa4
Only send RDATA for instance local events. (#8496) 
When pulling events out of the DB to send over replication we were not
filtering by instance name, and so we were sending events for other
instances.
 2020-10-09 13:10:33 +01:00
Patrick Cloke
fe0f4a3591
Move additional tasks to the background worker, part 3 (#8489) 2020-10-09 07:37:51 -04:00
Patrick Cloke
c9c0ad5e20
Remove the deprecated Handlers object (#8494) 
All handlers now available via get_*_handler() methods on the HomeServer.
 2020-10-09 07:24:34 -04:00
Patrick Cloke
a93f3121f8
Add type hints to some handlers (#8505) 2020-10-09 07:20:51 -04:00
Hubert Chathi
a97cec18bb
Invalidate the cache when an olm fallback key is uploaded (#8501) 2020-10-08 13:24:46 -04:00
Erik Johnston
7859c4d079 Synapse 1.21.0rc3 (2020-10-08) 
==============================
 
 Bugfixes
 --------
 
 - Fix duplication of events on high traffic servers, caused by PostgreSQL `could not serialize access due to concurrent update` errors. ([\#8456](https://github.com/matrix-org/synapse/issues/8456))
 
 Internal Changes
 ----------------
 
 - Add Groovy Gorilla to the list of distributions we build `.deb`s for. ([\#8475](https://github.com/matrix-org/synapse/issues/8475))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAl9+7JIQHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCUc1B/9Bz8uAEJWX9qahuWZGrsvj0ukF/9O6YWrU
 ZZGHnyaAXg4MF0lR6DVCV2NhW/l3RWO01G/LgGT//0vac+EFxWDbjuaHP9pRVBln
 02TQpn12ceZtW/n+X+8wmQzfSn1zNWRUycV43MyGu49z3IAG7KVjjVDQPkY0mEHC
 h6THy30SYIOKDCdXUUL9i+sV1JLzKPX0NlzO3ONKe2LbfLiBTvdfDRlZGq6rJu5H
 GAsIPwgadhanhXBPhG4WrealG31TkcPbZCwFMv9aFzltU841jps+QN6OKvbGFVer
 FboD2df0ZJqKNJOJVVlOMs7I5uSYbrg+n1zolZfrbo0lk/xEyIAT
 =F5cw
 -----END PGP SIGNATURE-----

Merge tag 'v1.21.0rc3' into develop

Synapse 1.21.0rc3 (2020-10-08)
==============================

Bugfixes
--------

- Fix duplication of events on high traffic servers, caused by PostgreSQL `could not serialize access due to concurrent update` errors. ([\#8456](https://github.com/matrix-org/synapse/issues/8456))

Internal Changes
----------------

- Add Groovy Gorilla to the list of distributions we build `.deb`s for. ([\#8475](https://github.com/matrix-org/synapse/issues/8475))
 2020-10-08 11:43:21 +01:00
Erik Johnston
31fe46e0a3 1.21.0rc3 2020-10-08 11:19:22 +01:00
Mateusz Przybyłowicz
719474cae0
Add useful shields to readme (#8493) 
Added shields directing to synapse-dev room, showing license, latest version on PyPi and supported Python versions.
I've moved substitution definitions to the bottom to improve readability.

Signed-off-by: Mateusz Przybyłowicz <uamfhq@gmail.com>
 2020-10-08 11:16:56 +01:00
Richard van der Hoff
b28bfd905d
Clarify error message when plugin config parsers raise an error (#8492) 
This turns:

    Failed to parse config for 'myplugin': Exception('error message')

into:

    Failed to parse config for 'myplugin': error message.
 2020-10-08 11:10:15 +01:00
Erik Johnston
1baa895310 Merge remote-tracking branch 'origin/release-v1.21.0' into develop 2020-10-07 17:20:24 +01:00
Erik Johnston
fa8934b175 Reduce serialization errors in MultiWriterIdGen (#8456) 
We call `_update_stream_positions_table_txn` a lot, which is an UPSERT
that can conflict in `REPEATABLE READ` isolation level. Instead of doing
a transaction consisting of a single query we may as well run it outside
of a transaction.
 2020-10-07 17:08:58 +01:00
Patrick Cloke
e4f72ddc44
Move additional tasks to the background worker (#8458) 2020-10-07 11:27:56 -04:00
Patrick Cloke
8dbf62fada
Include the configured log level in phone home stats. (#8477) 
By reporting the log level of the synapse logger as a string.
 2020-10-07 11:13:38 -04:00
Erik Johnston
ae5b2a72c0
Reduce serialization errors in MultiWriterIdGen (#8456) 
We call `_update_stream_positions_table_txn` a lot, which is an UPSERT
that can conflict in `REPEATABLE READ` isolation level. Instead of doing
a transaction consisting of a single query we may as well run it outside
of a transaction.
 2020-10-07 15:15:57 +01:00
Erik Johnston
52a50e8686
Use vector clocks for room stream tokens. (#8439) 
Currently when using multiple event persisters we (in the worst case) don't tell clients about events until all event persisters have persisted new events after the original event. This is a suboptimal, especially if one of the event persisters goes down.

To handle this, we encode the position of each event persister in the room tokens so that we can send events to clients immediately. To reduce the size of the token we do two things:

1. We create a unique immutable persistent mapping between instance names and a generated small integer ID, which we can encode in the tokens instead of the instance name; and
2. We encode the "persisted upto position" of the room token and then only explicitly include instances that have positions strictly greater than that.

The new tokens look something like: `m3478~1.3488~2.3489`, where the first number is the min position, and the subsequent `-` separated pairs are the instance ID to positions map. (We use `.` and `~` as separators as they're URL safe and not already used by `StreamToken`).
 2020-10-07 15:15:33 +01:00
Patrick Cloke
b460a088c6
Add typing information to the device handler. (#8407) 2020-10-07 08:58:21 -04:00