Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/new_push_rules

This commit is contained in:
Brendan Abolivier 2020-07-30 19:00:29 +01:00
commit 69158e554f
230 changed files with 5343 additions and 4040 deletions

View File

@ -1,3 +1,138 @@
Synapse 1.18.0 (2020-07-30)
===========================
Deprecation Warnings
--------------------
### Docker Tags with `-py3` Suffix
From 10th August 2020, we will no longer publish Docker images with the `-py3` tag suffix. The images tagged with the `-py3` suffix have been identical to the non-suffixed tags since release 0.99.0, and the suffix is obsolete.
On 10th August, we will remove the `latest-py3` tag. Existing per-release tags (such as `v1.18.0-py3`) will not be removed, but no new `-py3` tags will be added.
Scripts relying on the `-py3` suffix will need to be updated.
### TCP-based Replication
When setting up worker processes, we now recommend the use of a Redis server for replication. The old direct TCP connection method is deprecated and will be removed in a future release. See [docs/workers.md](https://github.com/matrix-org/synapse/blob/release-v1.18.0/docs/workers.md) for more details.
Improved Documentation
----------------------
- Update worker docs with latest enhancements. ([\#7969](https://github.com/matrix-org/synapse/issues/7969))
Synapse 1.18.0rc2 (2020-07-28)
==============================
Bugfixes
--------
- Fix an `AssertionError` exception introduced in v1.18.0rc1. ([\#7876](https://github.com/matrix-org/synapse/issues/7876))
- Fix experimental support for moving typing off master when worker is restarted, which is broken in v1.18.0rc1. ([\#7967](https://github.com/matrix-org/synapse/issues/7967))
Internal Changes
----------------
- Further optimise queueing of inbound replication commands. ([\#7876](https://github.com/matrix-org/synapse/issues/7876))
Synapse 1.18.0rc1 (2020-07-27)
==============================
Features
--------
- Include room states on invite events that are sent to application services. Contributed by @Sorunome. ([\#6455](https://github.com/matrix-org/synapse/issues/6455))
- Add delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel. ([\#7613](https://github.com/matrix-org/synapse/issues/7613), [\#7953](https://github.com/matrix-org/synapse/issues/7953))
- Add experimental support for running multiple federation sender processes. ([\#7798](https://github.com/matrix-org/synapse/issues/7798))
- Add the option to validate the `iss` and `aud` claims for JWT logins. ([\#7827](https://github.com/matrix-org/synapse/issues/7827))
- Add support for handling registration requests across multiple client reader workers. ([\#7830](https://github.com/matrix-org/synapse/issues/7830))
- Add an admin API to list the users in a room. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#7842](https://github.com/matrix-org/synapse/issues/7842))
- Allow email subjects to be customised through Synapse's configuration. ([\#7846](https://github.com/matrix-org/synapse/issues/7846))
- Add the ability to re-activate an account from the admin API. ([\#7847](https://github.com/matrix-org/synapse/issues/7847), [\#7908](https://github.com/matrix-org/synapse/issues/7908))
- Add experimental support for running multiple pusher workers. ([\#7855](https://github.com/matrix-org/synapse/issues/7855))
- Add experimental support for moving typing off master. ([\#7869](https://github.com/matrix-org/synapse/issues/7869), [\#7959](https://github.com/matrix-org/synapse/issues/7959))
- Report CPU metrics to prometheus for time spent processing replication commands. ([\#7879](https://github.com/matrix-org/synapse/issues/7879))
- Support oEmbed for media previews. ([\#7920](https://github.com/matrix-org/synapse/issues/7920))
- Abort federation requests where the client disconnects before the ratelimiter expires. ([\#7930](https://github.com/matrix-org/synapse/issues/7930))
- Cache responses to `/_matrix/federation/v1/state_ids` to reduce duplicated work. ([\#7931](https://github.com/matrix-org/synapse/issues/7931))
Bugfixes
--------
- Fix detection of out of sync remote device lists when receiving events from remote users. ([\#7815](https://github.com/matrix-org/synapse/issues/7815))
- Fix bug where Synapse fails to process an incoming event over federation if the server is missing too much of the event's auth chain. ([\#7817](https://github.com/matrix-org/synapse/issues/7817))
- Fix a bug causing Synapse to misinterpret the value `off` for `encryption_enabled_by_default_for_room_type` in its configuration file(s) if that value isn't surrounded by quotes. This bug was introduced in v1.16.0. ([\#7822](https://github.com/matrix-org/synapse/issues/7822))
- Fix bug where we did not always pass in `app_name` or `server_name` to email templates, including e.g. for registration emails. ([\#7829](https://github.com/matrix-org/synapse/issues/7829))
- Errors which occur while using the non-standard JWT login now return the proper error: `403 Forbidden` with an error code of `M_FORBIDDEN`. ([\#7844](https://github.com/matrix-org/synapse/issues/7844))
- Fix "AttributeError: 'str' object has no attribute 'get'" error message when applying per-room message retention policies. The bug was introduced in Synapse 1.7.0. ([\#7850](https://github.com/matrix-org/synapse/issues/7850))
- Fix a bug introduced in Synapse 1.10.0 which could cause a "no create event in auth events" error during room creation. ([\#7854](https://github.com/matrix-org/synapse/issues/7854))
- Fix a bug which allowed empty rooms to be rejoined over federation. ([\#7859](https://github.com/matrix-org/synapse/issues/7859))
- Fix 'Unable to find a suitable guest user ID' error when using multiple client_reader workers. ([\#7866](https://github.com/matrix-org/synapse/issues/7866))
- Fix a long standing bug where the tracing of async functions with opentracing was broken. ([\#7872](https://github.com/matrix-org/synapse/issues/7872), [\#7961](https://github.com/matrix-org/synapse/issues/7961))
- Fix "TypeError in `synapse.notifier`" exceptions. ([\#7880](https://github.com/matrix-org/synapse/issues/7880))
- Fix deprecation warning due to invalid escape sequences. ([\#7895](https://github.com/matrix-org/synapse/issues/7895))
Updates to the Docker image
---------------------------
- Base docker image on Debian Buster rather than Alpine Linux. Contributed by @maquis196. ([\#7839](https://github.com/matrix-org/synapse/issues/7839))
Improved Documentation
----------------------
- Provide instructions on using `register_new_matrix_user` via docker. ([\#7885](https://github.com/matrix-org/synapse/issues/7885))
- Change the sample config postgres user section to use `synapse_user` instead of `synapse` to align with the documentation. ([\#7889](https://github.com/matrix-org/synapse/issues/7889))
- Reorder database paragraphs to promote postgres over sqlite. ([\#7933](https://github.com/matrix-org/synapse/issues/7933))
- Update the dates of ACME v1's end of life in [`ACME.md`](https://github.com/matrix-org/synapse/blob/master/docs/ACME.md). ([\#7934](https://github.com/matrix-org/synapse/issues/7934))
Deprecations and Removals
-------------------------
- Remove unused `synapse_replication_tcp_resource_invalidate_cache` prometheus metric. ([\#7878](https://github.com/matrix-org/synapse/issues/7878))
- Remove Ubuntu Eoan from the list of `.deb` packages that we build as it is now end-of-life. Contributed by @gary-kim. ([\#7888](https://github.com/matrix-org/synapse/issues/7888))
Internal Changes
----------------
- Switch parts of the codebase from `simplejson` to the standard library `json`. ([\#7802](https://github.com/matrix-org/synapse/issues/7802))
- Add type hints to the http server code and remove an unused parameter. ([\#7813](https://github.com/matrix-org/synapse/issues/7813))
- Add type hints to synapse.api.errors module. ([\#7820](https://github.com/matrix-org/synapse/issues/7820))
- Ensure that calls to `json.dumps` are compatible with the standard library json. ([\#7836](https://github.com/matrix-org/synapse/issues/7836))
- Remove redundant `retry_on_integrity_error` wrapper for event persistence code. ([\#7848](https://github.com/matrix-org/synapse/issues/7848))
- Consistently use `db_to_json` to convert from database values to JSON objects. ([\#7849](https://github.com/matrix-org/synapse/issues/7849))
- Convert various parts of the codebase to async/await. ([\#7851](https://github.com/matrix-org/synapse/issues/7851), [\#7860](https://github.com/matrix-org/synapse/issues/7860), [\#7868](https://github.com/matrix-org/synapse/issues/7868), [\#7871](https://github.com/matrix-org/synapse/issues/7871), [\#7873](https://github.com/matrix-org/synapse/issues/7873), [\#7874](https://github.com/matrix-org/synapse/issues/7874), [\#7884](https://github.com/matrix-org/synapse/issues/7884), [\#7912](https://github.com/matrix-org/synapse/issues/7912), [\#7935](https://github.com/matrix-org/synapse/issues/7935), [\#7939](https://github.com/matrix-org/synapse/issues/7939), [\#7942](https://github.com/matrix-org/synapse/issues/7942), [\#7944](https://github.com/matrix-org/synapse/issues/7944))
- Add support for handling registration requests across multiple client reader workers. ([\#7853](https://github.com/matrix-org/synapse/issues/7853))
- Small performance improvement in typing processing. ([\#7856](https://github.com/matrix-org/synapse/issues/7856))
- The default value of `filter_timeline_limit` was changed from -1 (no limit) to 100. ([\#7858](https://github.com/matrix-org/synapse/issues/7858))
- Optimise queueing of inbound replication commands. ([\#7861](https://github.com/matrix-org/synapse/issues/7861))
- Add some type annotations to `HomeServer` and `BaseHandler`. ([\#7870](https://github.com/matrix-org/synapse/issues/7870))
- Clean up `PreserveLoggingContext`. ([\#7877](https://github.com/matrix-org/synapse/issues/7877))
- Change "unknown room version" logging from 'error' to 'warning'. ([\#7881](https://github.com/matrix-org/synapse/issues/7881))
- Stop using `device_max_stream_id` table and just use `device_inbox.stream_id`. ([\#7882](https://github.com/matrix-org/synapse/issues/7882))
- Return an empty body for OPTIONS requests. ([\#7886](https://github.com/matrix-org/synapse/issues/7886))
- Fix typo in generated config file. Contributed by @ThiefMaster. ([\#7890](https://github.com/matrix-org/synapse/issues/7890))
- Import ABC from `collections.abc` for Python 3.10 compatibility. ([\#7892](https://github.com/matrix-org/synapse/issues/7892))
- Remove unused functions `time_function`, `trace_function`, `get_previous_frames`
and `get_previous_frame` from `synapse.logging.utils` module. ([\#7897](https://github.com/matrix-org/synapse/issues/7897))
- Lint the `contrib/` directory in CI and linting scripts, add `synctl` to the linting script for consistency with CI. ([\#7914](https://github.com/matrix-org/synapse/issues/7914))
- Use Element CSS and logo in notification emails when app name is Element. ([\#7919](https://github.com/matrix-org/synapse/issues/7919))
- Optimisation to /sync handling: skip serializing the response if the client has already disconnected. ([\#7927](https://github.com/matrix-org/synapse/issues/7927))
- When a client disconnects, don't log it as 'Error processing request'. ([\#7928](https://github.com/matrix-org/synapse/issues/7928))
- Add debugging to `/sync` response generation (disabled by default). ([\#7929](https://github.com/matrix-org/synapse/issues/7929))
- Update comments that refer to Deferreds for async functions. ([\#7945](https://github.com/matrix-org/synapse/issues/7945))
- Simplify error handling in federation handler. ([\#7950](https://github.com/matrix-org/synapse/issues/7950))
Synapse 1.17.0 (2020-07-13) Synapse 1.17.0 (2020-07-13)
=========================== ===========================

View File

@ -1,10 +1,12 @@
- [Choosing your server name](#choosing-your-server-name) - [Choosing your server name](#choosing-your-server-name)
- [Picking a database engine](#picking-a-database-engine)
- [Installing Synapse](#installing-synapse) - [Installing Synapse](#installing-synapse)
- [Installing from source](#installing-from-source) - [Installing from source](#installing-from-source)
- [Platform-Specific Instructions](#platform-specific-instructions) - [Platform-Specific Instructions](#platform-specific-instructions)
- [Prebuilt packages](#prebuilt-packages) - [Prebuilt packages](#prebuilt-packages)
- [Setting up Synapse](#setting-up-synapse) - [Setting up Synapse](#setting-up-synapse)
- [TLS certificates](#tls-certificates) - [TLS certificates](#tls-certificates)
- [Client Well-Known URI](#client-well-known-uri)
- [Email](#email) - [Email](#email)
- [Registering a user](#registering-a-user) - [Registering a user](#registering-a-user)
- [Setting up a TURN server](#setting-up-a-turn-server) - [Setting up a TURN server](#setting-up-a-turn-server)
@ -27,6 +29,25 @@ that your email address is probably `user@example.com` rather than
`user@email.example.com`) - but doing so may require more advanced setup: see `user@email.example.com`) - but doing so may require more advanced setup: see
[Setting up Federation](docs/federate.md). [Setting up Federation](docs/federate.md).
# Picking a database engine
Synapse offers two database engines:
* [PostgreSQL](https://www.postgresql.org)
* [SQLite](https://sqlite.org/)
Almost all installations should opt to use PostgreSQL. Advantages include:
* significant performance improvements due to the superior threading and
caching model, smarter query optimiser
* allowing the DB to be run on separate hardware
For information on how to install and use PostgreSQL, please see
[docs/postgres.md](docs/postgres.md)
By default Synapse uses SQLite and in doing so trades performance for convenience.
SQLite is only recommended in Synapse for testing purposes or for servers with
light workloads.
# Installing Synapse # Installing Synapse
## Installing from source ## Installing from source
@ -234,9 +255,9 @@ for a number of platforms.
There is an offical synapse image available at There is an offical synapse image available at
https://hub.docker.com/r/matrixdotorg/synapse which can be used with https://hub.docker.com/r/matrixdotorg/synapse which can be used with
the docker-compose file available at [contrib/docker](contrib/docker). Further information on the docker-compose file available at [contrib/docker](contrib/docker). Further
this including configuration options is available in the README on information on this including configuration options is available in the README
hub.docker.com. on hub.docker.com.
Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a
Dockerfile to automate a synapse server in a single Docker image, at Dockerfile to automate a synapse server in a single Docker image, at
@ -244,7 +265,8 @@ https://hub.docker.com/r/avhost/docker-matrix/tags/
Slavi Pantaleev has created an Ansible playbook, Slavi Pantaleev has created an Ansible playbook,
which installs the offical Docker image of Matrix Synapse which installs the offical Docker image of Matrix Synapse
along with many other Matrix-related services (Postgres database, riot-web, coturn, mxisd, SSL support, etc.). along with many other Matrix-related services (Postgres database, Element, coturn,
ma1sd, SSL support, etc.).
For more details, see For more details, see
https://github.com/spantaleev/matrix-docker-ansible-deploy https://github.com/spantaleev/matrix-docker-ansible-deploy
@ -277,22 +299,27 @@ The fingerprint of the repository signing key (as shown by `gpg
/usr/share/keyrings/matrix-org-archive-keyring.gpg`) is /usr/share/keyrings/matrix-org-archive-keyring.gpg`) is
`AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`. `AAF9AE843A7584B5A3E4CD2BCF45A512DE2DA058`.
#### Downstream Debian/Ubuntu packages #### Downstream Debian packages
For `buster` and `sid`, Synapse is available in the Debian repositories and We do not recommend using the packages from the default Debian `buster`
it should be possible to install it with simply: repository at this time, as they are old and suffer from known security
vulnerabilities. You can install the latest version of Synapse from
[our repository](#matrixorg-packages) or from `buster-backports`. Please
see the [Debian documentation](https://backports.debian.org/Instructions/)
for information on how to use backports.
If you are using Debian `sid` or testing, Synapse is available in the default
repositories and it should be possible to install it simply with:
``` ```
sudo apt install matrix-synapse sudo apt install matrix-synapse
``` ```
There is also a version of `matrix-synapse` in `stretch-backports`. Please see #### Downstream Ubuntu packages
the [Debian documentation on
backports](https://backports.debian.org/Instructions/) for information on how
to use them.
We do not recommend using the packages in downstream Ubuntu at this time, as We do not recommend using the packages in the default Ubuntu repository
they are old and suffer from known security vulnerabilities. at this time, as they are old and suffer from known security vulnerabilities.
The latest version of Synapse can be installed from [our repository](#matrixorg-packages).
### Fedora ### Fedora
@ -419,6 +446,60 @@ so, you will need to edit `homeserver.yaml`, as follows:
For a more detailed guide to configuring your server for federation, see For a more detailed guide to configuring your server for federation, see
[federate.md](docs/federate.md). [federate.md](docs/federate.md).
## Client Well-Known URI
Setting up the client Well-Known URI is optional but if you set it up, it will
allow users to enter their full username (e.g. `@user:<server_name>`) into clients
which support well-known lookup to automatically configure the homeserver and
identity server URLs. This is useful so that users don't have to memorize or think
about the actual homeserver URL you are using.
The URL `https://<server_name>/.well-known/matrix/client` should return JSON in
the following format.
```
{
"m.homeserver": {
"base_url": "https://<matrix.example.com>"
}
}
```
It can optionally contain identity server information as well.
```
{
"m.homeserver": {
"base_url": "https://<matrix.example.com>"
},
"m.identity_server": {
"base_url": "https://<identity.example.com>"
}
}
```
To work in browser based clients, the file must be served with the appropriate
Cross-Origin Resource Sharing (CORS) headers. A recommended value would be
`Access-Control-Allow-Origin: *` which would allow all browser based clients to
view it.
In nginx this would be something like:
```
location /.well-known/matrix/client {
return 200 '{"m.homeserver": {"base_url": "https://<matrix.example.com>"}}';
add_header Content-Type application/json;
add_header Access-Control-Allow-Origin *;
}
```
You should also ensure the `public_baseurl` option in `homeserver.yaml` is set
correctly. `public_baseurl` should be set to the URL that clients will use to
connect to your server. This is the same URL you put for the `m.homeserver`
`base_url` above.
```
public_baseurl: "https://<matrix.example.com>"
```
## Email ## Email
@ -437,7 +518,7 @@ email will be disabled.
## Registering a user ## Registering a user
The easiest way to create a new user is to do so from a client like [Riot](https://riot.im). The easiest way to create a new user is to do so from a client like [Element](https://element.io/).
Alternatively you can do so from the command line if you have installed via pip. Alternatively you can do so from the command line if you have installed via pip.

View File

@ -45,7 +45,7 @@ which handle:
- Eventually-consistent cryptographically secure synchronisation of room - Eventually-consistent cryptographically secure synchronisation of room
state across a global open network of federated servers and services state across a global open network of federated servers and services
- Sending and receiving extensible messages in a room with (optional) - Sending and receiving extensible messages in a room with (optional)
end-to-end encryption[1] end-to-end encryption
- Inviting, joining, leaving, kicking, banning room members - Inviting, joining, leaving, kicking, banning room members
- Managing user accounts (registration, login, logout) - Managing user accounts (registration, login, logout)
- Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers, - Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers,
@ -82,9 +82,6 @@ at the `Matrix spec <https://matrix.org/docs/spec>`_, and experiment with the
Thanks for using Matrix! Thanks for using Matrix!
[1] End-to-end encryption is currently in beta: `blog post <https://matrix.org/blog/2016/11/21/matrixs-olm-end-to-end-encryption-security-assessment-released-and-implemented-cross-platform-on-riot-at-last>`_.
Support Support
======= =======
@ -115,12 +112,11 @@ Unless you are running a test instance of Synapse on your local machine, in
general, you will need to enable TLS support before you can successfully general, you will need to enable TLS support before you can successfully
connect from a client: see `<INSTALL.md#tls-certificates>`_. connect from a client: see `<INSTALL.md#tls-certificates>`_.
An easy way to get started is to login or register via Riot at An easy way to get started is to login or register via Element at
https://riot.im/app/#/login or https://riot.im/app/#/register respectively. https://app.element.io/#/login or https://app.element.io/#/register respectively.
You will need to change the server you are logging into from ``matrix.org`` You will need to change the server you are logging into from ``matrix.org``
and instead specify a Homeserver URL of ``https://<server_name>:8448`` and instead specify a Homeserver URL of ``https://<server_name>:8448``
(or just ``https://<server_name>`` if you are using a reverse proxy). (or just ``https://<server_name>`` if you are using a reverse proxy).
(Leave the identity server as the default - see `Identity servers`_.)
If you prefer to use another client, refer to our If you prefer to use another client, refer to our
`client breakdown <https://matrix.org/docs/projects/clients-matrix>`_. `client breakdown <https://matrix.org/docs/projects/clients-matrix>`_.
@ -137,7 +133,7 @@ it, specify ``enable_registration: true`` in ``homeserver.yaml``. (It is then
recommended to also set up CAPTCHA - see `<docs/CAPTCHA_SETUP.md>`_.) recommended to also set up CAPTCHA - see `<docs/CAPTCHA_SETUP.md>`_.)
Once ``enable_registration`` is set to ``true``, it is possible to register a Once ``enable_registration`` is set to ``true``, it is possible to register a
user via `riot.im <https://riot.im/app/#/register>`_ or other Matrix clients. user via a Matrix client.
Your new user name will be formed partly from the ``server_name``, and partly Your new user name will be formed partly from the ``server_name``, and partly
from a localpart you specify when you create the account. Your name will take from a localpart you specify when you create the account. Your name will take
@ -183,30 +179,6 @@ versions of synapse.
.. _UPGRADE.rst: UPGRADE.rst .. _UPGRADE.rst: UPGRADE.rst
Using PostgreSQL
================
Synapse offers two database engines:
* `SQLite <https://sqlite.org/>`_
* `PostgreSQL <https://www.postgresql.org>`_
By default Synapse uses SQLite in and doing so trades performance for convenience.
SQLite is only recommended in Synapse for testing purposes or for servers with
light workloads.
Almost all installations should opt to use PostgreSQL. Advantages include:
* significant performance improvements due to the superior threading and
caching model, smarter query optimiser
* allowing the DB to be run on separate hardware
* allowing basic active/backup high-availability with a "hot spare" synapse
pointing at the same DB master, as well as enabling DB replication in
synapse itself.
For information on how to install and use PostgreSQL, please see
`docs/postgres.md <docs/postgres.md>`_.
.. _reverse-proxy: .. _reverse-proxy:
Using a reverse proxy with Synapse Using a reverse proxy with Synapse
@ -255,10 +227,9 @@ email address.
Password reset Password reset
============== ==============
If a user has registered an email address to their account using an identity Users can reset their password through their client. Alternatively, a server admin
server, they can request a password-reset token via clients such as Riot. can reset a users password using the `admin API <docs/admin_api/user_admin_api.rst#reset-password>`_
or by directly editing the database as shown below.
A manual password reset can be done via direct database access as follows.
First calculate the hash of the new password:: First calculate the hash of the new password::

View File

@ -75,6 +75,24 @@ for example:
wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb wget https://packages.matrix.org/debian/pool/main/m/matrix-synapse-py3/matrix-synapse-py3_1.3.0+stretch1_amd64.deb
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
Upgrading to v1.18.0
====================
Docker `-py3` suffix will be removed in future versions
-------------------------------------------------------
From 10th August 2020, we will no longer publish Docker images with the `-py3` tag suffix. The images tagged with the `-py3` suffix have been identical to the non-suffixed tags since release 0.99.0, and the suffix is obsolete.
On 10th August, we will remove the `latest-py3` tag. Existing per-release tags (such as `v1.18.0-py3`) will not be removed, but no new `-py3` tags will be added.
Scripts relying on the `-py3` suffix will need to be updated.
Redis replication is now recommended in lieu of TCP replication
---------------------------------------------------------------
When setting up worker processes, we now recommend the use of a Redis server for replication. **The old direct TCP connection method is deprecated and will be removed in a future release.**
See `docs/workers.md <docs/workers.md>`_ for more details.
Upgrading to v1.14.0 Upgrading to v1.14.0
==================== ====================

View File

@ -1 +0,0 @@
Include room states on invite events that are sent to application services. Contributed by @Sorunome.

View File

@ -1 +0,0 @@
Add delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel.

1
changelog.d/7736.feature Normal file
View File

@ -0,0 +1 @@
Add unread messages count to sync responses, as specified in [MSC2654](https://github.com/matrix-org/matrix-doc/pull/2654).

View File

@ -1 +0,0 @@
Add experimental support for running multiple federation sender processes.

View File

@ -1 +0,0 @@
Switch from simplejson to the standard library json.

View File

@ -1 +0,0 @@
Add type hints to the http server code and remove an unused parameter.

View File

@ -1 +0,0 @@
Fix detection of out of sync remote device lists when receiving events from remote users.

View File

@ -1 +0,0 @@
Fix bug where Synapse fails to process an incoming event over federation if the server is missing too much of the event's auth chain.

View File

@ -1 +0,0 @@
Add type hints to synapse.api.errors module.

View File

@ -1 +0,0 @@
Fix a bug causing Synapse to misinterpret the value `off` for `encryption_enabled_by_default_for_room_type` in its configuration file(s) if that value isn't surrounded by quotes. This bug was introduced in v1.16.0.

View File

@ -1 +0,0 @@
Add the option to validate the `iss` and `aud` claims for JWT logins.

View File

@ -1 +0,0 @@
Fix bug where we did not always pass in `app_name` or `server_name` to email templates, including e.g. for registration emails.

View File

@ -1 +0,0 @@
Add support for handling registration requests across multiple client reader workers.

View File

@ -1 +0,0 @@
Ensure that calls to `json.dumps` are compatible with the standard library json.

View File

@ -1 +0,0 @@
Base docker image on Debian Buster rather than Alpine Linux. Contributed by @maquis196.

View File

@ -1 +0,0 @@
Add an admin API to list the users in a room. Contributed by Awesome Technologies Innovationslabor GmbH.

View File

@ -1 +0,0 @@
Errors which occur while using the non-standard JWT login now return the proper error: `403 Forbidden` with an error code of `M_FORBIDDEN`.

View File

@ -1 +0,0 @@
Allow email subjects to be customised through Synapse's configuration.

View File

@ -1 +0,0 @@
Add the ability to re-activate an account from the admin API.

View File

@ -1 +0,0 @@
Remove redundant `retry_on_integrity_error` wrapper for event persistence code.

View File

@ -1 +0,0 @@
Consistently use `db_to_json` to convert from database values to JSON objects.

View File

@ -1 +0,0 @@
Fix "AttributeError: 'str' object has no attribute 'get'" error message when applying per-room message retention policies. The bug was introduced in Synapse 1.7.0.

View File

@ -1 +0,0 @@
Convert E2E keys and room keys handlers to async/await.

View File

@ -1 +0,0 @@
Add support for handling registration requests across multiple client reader workers.

View File

@ -1 +0,0 @@
Fix a bug introduced in Synapse 1.10.0 which could cause a "no create event in auth events" error during room creation.

View File

@ -1 +0,0 @@
Add experimental support for running multiple pusher workers.

View File

@ -1 +0,0 @@
Small performance improvement in typing processing.

View File

@ -1 +0,0 @@
The default value of `filter_timeline_limit` was changed from -1 (no limit) to 100.

View File

@ -1 +0,0 @@
Fix a bug which allowed empty rooms to be rejoined over federation.

View File

@ -1 +0,0 @@
Convert _base, profile, and _receipts handlers to async/await.

View File

@ -1 +0,0 @@
Optimise queueing of inbound replication commands.

View File

@ -1 +0,0 @@
Fix 'Unable to find a suitable guest user ID' error when using multiple client_reader workers.

View File

@ -1 +0,0 @@
Convert synapse.app and federation client to async/await.

View File

@ -1 +0,0 @@
Add experimental support for moving typing off master.

View File

@ -1 +0,0 @@
Add some type annotations to `HomeServer` and `BaseHandler`.

View File

@ -1 +0,0 @@
Convert device handler to async/await.

View File

@ -1 +0,0 @@
Fix a long standing bug where the tracing of async functions with opentracing was broken.

View File

@ -1 +0,0 @@
Clean up `PreserveLoggingContext`.

View File

@ -1 +0,0 @@
Remove unused `synapse_replication_tcp_resource_invalidate_cache` prometheus metric.

View File

@ -1 +0,0 @@
Report CPU metrics to prometheus for time spent processing replication commands.

View File

@ -1 +0,0 @@
Fix "TypeError in `synapse.notifier`" exceptions.

View File

@ -1 +0,0 @@
Change "unknown room version" logging from 'error' to 'warning'.

View File

@ -1 +0,0 @@
Stop using `device_max_stream_id` table and just use `device_inbox.stream_id`.

View File

@ -1 +0,0 @@
Provide instructions on using `register_new_matrix_user` via docker.

View File

@ -1 +0,0 @@
Remove Ubuntu Eoan from the list of `.deb` packages that we build as it is now end-of-life. Contributed by @gary-kim.

View File

@ -1 +0,0 @@
Change the sample config postgres user section to use `synapse_user` instead of `synapse` to align with the documentation.

View File

@ -1 +0,0 @@
Fix typo in generated config file. Contributed by @ThiefMaster.

View File

@ -1 +0,0 @@
Import ABC from `collections.abc` for Python 3.10 compatibility.

View File

@ -1 +0,0 @@
Fix deprecation warning due to invalid escape sequences.

View File

@ -1,2 +0,0 @@
Remove unused functions `time_function`, `trace_function`, `get_previous_frames`
and `get_previous_frame` from `synapse.logging.utils` module.

1
changelog.d/7899.doc Normal file
View File

@ -0,0 +1 @@
Document how to set up a Client Well-Known file and fix several pieces of outdated documentation.

1
changelog.d/7902.feature Normal file
View File

@ -0,0 +1 @@
Add option to allow server admins to join rooms which fail complexity checks. Contributed by @lugino-emeritus.

View File

@ -1 +0,0 @@
Convert `RoomListHandler` to async/await.

View File

@ -1 +0,0 @@
Lint the `contrib/` directory in CI and linting scripts, add `synctl` to the linting script for consistency with CI.

View File

@ -1 +0,0 @@
Use Element CSS and logo in notification emails when app name is Element.

View File

@ -1 +0,0 @@
Optimisation to /sync handling: skip serializing the response if the client has already disconnected.

View File

@ -1 +0,0 @@
Add debugging to `/sync` response generation (disabled by default).

1
changelog.d/7936.misc Normal file
View File

@ -0,0 +1 @@
Switch to the JSON implementation from the standard library and bump the minimum version of the canonicaljson library to 1.2.0.

1
changelog.d/7947.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7948.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7949.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7951.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7952.misc Normal file
View File

@ -0,0 +1 @@
Move some database-related log lines from the default logger to the database/transaction loggers.

1
changelog.d/7963.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7964.feature Normal file
View File

@ -0,0 +1 @@
Add an option to purge room or not with delete room admin endpoint (`POST /_synapse/admin/v1/rooms/<room_id>/delete`). Contributed by @dklimpel.

1
changelog.d/7965.misc Normal file
View File

@ -0,0 +1 @@
Add a script to detect source code files using non-unix line terminators.

1
changelog.d/7970.misc Normal file
View File

@ -0,0 +1 @@
Add a script to detect source code files using non-unix line terminators.

1
changelog.d/7971.misc Normal file
View File

@ -0,0 +1 @@
Log the SAML session ID during creation.

1
changelog.d/7973.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7975.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7976.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7978.bugfix Normal file
View File

@ -0,0 +1 @@
Fix a long standing bug: 'Duplicate key value violates unique constraint "event_relations_id"' when message retention is configured.

1
changelog.d/7979.misc Normal file
View File

@ -0,0 +1 @@
Switch to the JSON implementation from the standard library and bump the minimum version of the canonicaljson library to 1.2.0.

1
changelog.d/7980.bugfix Normal file
View File

@ -0,0 +1 @@
Fix "no create event in auth events" when trying to reject invitation after inviter leaves. Bug introduced in Synapse v1.10.0.

1
changelog.d/7981.misc Normal file
View File

@ -0,0 +1 @@
Convert various parts of the codebase to async/await.

1
changelog.d/7990.doc Normal file
View File

@ -0,0 +1 @@
Improve workers docs.

1
changelog.d/7992.doc Normal file
View File

@ -0,0 +1 @@
Fix typo in `docs/workers.md`.

View File

@ -609,7 +609,8 @@ class SynapseCmd(cmd.Cmd):
@defer.inlineCallbacks @defer.inlineCallbacks
def _do_event_stream(self, timeout): def _do_event_stream(self, timeout):
res = yield self.http_client.get_json( res = yield defer.ensureDeferred(
self.http_client.get_json(
self._url() + "/events", self._url() + "/events",
{ {
"access_token": self._tok(), "access_token": self._tok(),
@ -617,6 +618,7 @@ class SynapseCmd(cmd.Cmd):
"from": self.event_stream_token, "from": self.event_stream_token,
}, },
) )
)
print(json.dumps(res, indent=4)) print(json.dumps(res, indent=4))
if "chunk" in res: if "chunk" in res:

16
debian/changelog vendored
View File

@ -1,3 +1,19 @@
matrix-synapse-py3 (1.xx.0) stable; urgency=medium
[ Synapse Packaging team ]
* New synapse release 1.xx.0.
[ Aaron Raimist ]
* Fix outdated documentation for SYNAPSE_CACHE_FACTOR
-- Synapse Packaging team <packages@matrix.org> XXXXX
matrix-synapse-py3 (1.18.0) stable; urgency=medium
* New synapse release 1.18.0.
-- Synapse Packaging team <packages@matrix.org> Thu, 30 Jul 2020 10:55:53 +0100
matrix-synapse-py3 (1.17.0) stable; urgency=medium matrix-synapse-py3 (1.17.0) stable; urgency=medium
* New synapse release 1.17.0. * New synapse release 1.17.0.

View File

@ -1,2 +1,2 @@
# Specify environment variables used when running Synapse # Specify environment variables used when running Synapse
# SYNAPSE_CACHE_FACTOR=1 (default) # SYNAPSE_CACHE_FACTOR=0.5 (default)

27
debian/synctl.ronn vendored
View File

@ -46,19 +46,20 @@ Configuration file may be generated as follows:
## ENVIRONMENT ## ENVIRONMENT
* `SYNAPSE_CACHE_FACTOR`: * `SYNAPSE_CACHE_FACTOR`:
Synapse's architecture is quite RAM hungry currently - a lot of Synapse's architecture is quite RAM hungry currently - we deliberately
recent room data and metadata is deliberately cached in RAM in cache a lot of recent room data and metadata in RAM in order to speed up
order to speed up common requests. This will be improved in common requests. We'll improve this in the future, but for now the easiest
future, but for now the easiest way to either reduce the RAM usage way to either reduce the RAM usage (at the risk of slowing things down)
(at the risk of slowing things down) is to set the is to set the almost-undocumented ``SYNAPSE_CACHE_FACTOR`` environment
SYNAPSE_CACHE_FACTOR environment variable. Roughly speaking, a variable. The default is 0.5, which can be decreased to reduce RAM usage
SYNAPSE_CACHE_FACTOR of 1.0 will max out at around 3-4GB of in memory constrained enviroments, or increased if performance starts to
resident memory - this is what we currently run the matrix.org degrade.
on. The default setting is currently 0.1, which is probably around
a ~700MB footprint. You can dial it down further to 0.02 if However, degraded performance due to a low cache factor, common on
desired, which targets roughly ~512MB. Conversely you can dial it machines with slow disks, often leads to explosions in memory use due
up if you need performance for lots of users and have a box with a backlogged requests. In this case, reducing the cache factor will make
lot of RAM. things worse. Instead, try increasing it drastically. 2.0 is a good
starting value.
## COPYRIGHT ## COPYRIGHT

View File

@ -10,5 +10,16 @@
# homeserver.yaml. Instead, if you are starting from scratch, please generate # homeserver.yaml. Instead, if you are starting from scratch, please generate
# a fresh config using Synapse by following the instructions in INSTALL.md. # a fresh config using Synapse by following the instructions in INSTALL.md.
# Configuration options that take a time period can be set using a number
# followed by a letter. Letters have the following meanings:
# s = second
# m = minute
# h = hour
# d = day
# w = week
# y = year
# For example, setting redaction_retention_period: 5m would remove redacted
# messages from the database after 5 minutes, rather than 5 months.
################################################################################ ################################################################################

View File

@ -12,13 +12,14 @@ introduced support for automatically provisioning certificates through
In [March 2019](https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430), In [March 2019](https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430),
Let's Encrypt announced that they were deprecating version 1 of the ACME Let's Encrypt announced that they were deprecating version 1 of the ACME
protocol, with the plan to disable the use of it for new accounts in protocol, with the plan to disable the use of it for new accounts in
November 2019, and for existing accounts in June 2020. November 2019, for new domains in June 2020, and for existing accounts and
domains in June 2021.
Synapse doesn't currently support version 2 of the ACME protocol, which Synapse doesn't currently support version 2 of the ACME protocol, which
means that: means that:
* for existing installs, Synapse's built-in ACME support will continue * for existing installs, Synapse's built-in ACME support will continue
to work until June 2020. to work until June 2021.
* for new installs, this feature will not work at all. * for new installs, this feature will not work at all.
Either way, it is recommended to move from Synapse's ACME support Either way, it is recommended to move from Synapse's ACME support

View File

@ -369,7 +369,9 @@ to the new room will have power level `-10` by default, and thus be unable to sp
If `block` is `True` it prevents new joins to the old room. If `block` is `True` it prevents new joins to the old room.
This API will remove all trace of the old room from your database after removing This API will remove all trace of the old room from your database after removing
all local users. all local users. If `purge` is `true` (the default), all traces of the old room will
be removed from your database after removing all local users. If you do not want
this to happen, set `purge` to `false`.
Depending on the amount of history being purged a call to the API may take Depending on the amount of history being purged a call to the API may take
several minutes or longer. several minutes or longer.
@ -388,7 +390,8 @@ with a body of:
"new_room_user_id": "@someuser:example.com", "new_room_user_id": "@someuser:example.com",
"room_name": "Content Violation Notification", "room_name": "Content Violation Notification",
"message": "Bad Room has been shutdown due to content violations on this server. Please review our Terms of Service.", "message": "Bad Room has been shutdown due to content violations on this server. Please review our Terms of Service.",
"block": true "block": true,
"purge": true
} }
``` ```
@ -430,8 +433,10 @@ The following JSON body parameters are available:
`new_room_user_id` in the new room. Ideally this will clearly convey why the `new_room_user_id` in the new room. Ideally this will clearly convey why the
original room was shut down. Defaults to `Sharing illegal content on this server original room was shut down. Defaults to `Sharing illegal content on this server
is not permitted and rooms in violation will be blocked.` is not permitted and rooms in violation will be blocked.`
* `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing future attempts to * `block` - Optional. If set to `true`, this room will be added to a blocking list, preventing
join the room. Defaults to `false`. future attempts to join the room. Defaults to `false`.
* `purge` - Optional. If set to `true`, it will remove all traces of the room from your database.
Defaults to `true`.
The JSON body must not be empty. The body must be at least `{}`. The JSON body must not be empty. The body must be at least `{}`.

View File

@ -27,7 +27,7 @@
different thread to Synapse. This can make it more resilient to different thread to Synapse. This can make it more resilient to
heavy load meaning metrics cannot be retrieved, and can be exposed heavy load meaning metrics cannot be retrieved, and can be exposed
to just internal networks easier. The served metrics are available to just internal networks easier. The served metrics are available
over HTTP only, and will be available at `/`. over HTTP only, and will be available at `/_synapse/metrics`.
Add a new listener to homeserver.yaml: Add a new listener to homeserver.yaml:

View File

@ -19,102 +19,103 @@ password auth provider module implementations:
Password auth provider classes must provide the following methods: Password auth provider classes must provide the following methods:
*class* `SomeProvider.parse_config`(*config*) * `parse_config(config)`
This method is passed the `config` object for this module from the
homeserver configuration file.
> This method is passed the `config` object for this module from the It should perform any appropriate sanity checks on the provided
> homeserver configuration file. configuration, and return an object which is then passed into
>
> It should perform any appropriate sanity checks on the provided
> configuration, and return an object which is then passed into
> `__init__`.
*class* `SomeProvider`(*config*, *account_handler*) This method should have the `@staticmethod` decoration.
> The constructor is passed the config object returned by * `__init__(self, config, account_handler)`
> `parse_config`, and a `synapse.module_api.ModuleApi` object which
> allows the password provider to check if accounts exist and/or create The constructor is passed the config object returned by
> new ones. `parse_config`, and a `synapse.module_api.ModuleApi` object which
allows the password provider to check if accounts exist and/or create
new ones.
## Optional methods ## Optional methods
Password auth provider classes may optionally provide the following Password auth provider classes may optionally provide the following methods:
methods.
*class* `SomeProvider.get_db_schema_files`() * `get_db_schema_files(self)`
> This method, if implemented, should return an Iterable of This method, if implemented, should return an Iterable of
> `(name, stream)` pairs of database schema files. Each file is applied `(name, stream)` pairs of database schema files. Each file is applied
> in turn at initialisation, and a record is then made in the database in turn at initialisation, and a record is then made in the database
> so that it is not re-applied on the next start. so that it is not re-applied on the next start.
`someprovider.get_supported_login_types`() * `get_supported_login_types(self)`
> This method, if implemented, should return a `dict` mapping from a This method, if implemented, should return a `dict` mapping from a
> login type identifier (such as `m.login.password`) to an iterable login type identifier (such as `m.login.password`) to an iterable
> giving the fields which must be provided by the user in the submission giving the fields which must be provided by the user in the submission
> to the `/login` api. These fields are passed in the `login_dict` to [the `/login` API](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-login).
> dictionary to `check_auth`. These fields are passed in the `login_dict` dictionary to `check_auth`.
>
> For example, if a password auth provider wants to implement a custom
> login type of `com.example.custom_login`, where the client is expected
> to pass the fields `secret1` and `secret2`, the provider should
> implement this method and return the following dict:
>
> {"com.example.custom_login": ("secret1", "secret2")}
`someprovider.check_auth`(*username*, *login_type*, *login_dict*) For example, if a password auth provider wants to implement a custom
login type of `com.example.custom_login`, where the client is expected
to pass the fields `secret1` and `secret2`, the provider should
implement this method and return the following dict:
> This method is the one that does the real work. If implemented, it ```python
> will be called for each login attempt where the login type matches one {"com.example.custom_login": ("secret1", "secret2")}
> of the keys returned by `get_supported_login_types`. ```
>
> It is passed the (possibly UNqualified) `user` provided by the client,
> the login type, and a dictionary of login secrets passed by the
> client.
>
> The method should return a Twisted `Deferred` object, which resolves
> to the canonical `@localpart:domain` user id if authentication is
> successful, and `None` if not.
>
> Alternatively, the `Deferred` can resolve to a `(str, func)` tuple, in
> which case the second field is a callback which will be called with
> the result from the `/login` call (including `access_token`,
> `device_id`, etc.)
`someprovider.check_3pid_auth`(*medium*, *address*, *password*) * `check_auth(self, username, login_type, login_dict)`
> This method, if implemented, is called when a user attempts to This method does the real work. If implemented, it
> register or log in with a third party identifier, such as email. It is will be called for each login attempt where the login type matches one
> passed the medium (ex. "email"), an address (ex. of the keys returned by `get_supported_login_types`.
> "<jdoe@example.com>") and the user's password.
>
> The method should return a Twisted `Deferred` object, which resolves
> to a `str` containing the user's (canonical) User ID if
> authentication was successful, and `None` if not.
>
> As with `check_auth`, the `Deferred` may alternatively resolve to a
> `(user_id, callback)` tuple.
`someprovider.check_password`(*user_id*, *password*) It is passed the (possibly unqualified) `user` field provided by the client,
the login type, and a dictionary of login secrets passed by the
client.
> This method provides a simpler interface than The method should return an `Awaitable` object, which resolves
> `get_supported_login_types` and `check_auth` for password auth to the canonical `@localpart:domain` user ID if authentication is
> providers that just want to provide a mechanism for validating successful, and `None` if not.
> `m.login.password` logins.
>
> Iif implemented, it will be called to check logins with an
> `m.login.password` login type. It is passed a qualified
> `@localpart:domain` user id, and the password provided by the user.
>
> The method should return a Twisted `Deferred` object, which resolves
> to `True` if authentication is successful, and `False` if not.
`someprovider.on_logged_out`(*user_id*, *device_id*, *access_token*) Alternatively, the `Awaitable` can resolve to a `(str, func)` tuple, in
which case the second field is a callback which will be called with
the result from the `/login` call (including `access_token`,
`device_id`, etc.)
> This method, if implemented, is called when a user logs out. It is * `check_3pid_auth(self, medium, address, password)`
> passed the qualified user ID, the ID of the deactivated device (if
> any: access tokens are occasionally created without an associated This method, if implemented, is called when a user attempts to
> device ID), and the (now deactivated) access token. register or log in with a third party identifier, such as email. It is
> passed the medium (ex. "email"), an address (ex.
> It may return a Twisted `Deferred` object; the logout request will "<jdoe@example.com>") and the user's password.
> wait for the deferred to complete but the result is ignored.
The method should return an `Awaitable` object, which resolves
to a `str` containing the user's (canonical) User id if
authentication was successful, and `None` if not.
As with `check_auth`, the `Awaitable` may alternatively resolve to a
`(user_id, callback)` tuple.
* `check_password(self, user_id, password)`
This method provides a simpler interface than
`get_supported_login_types` and `check_auth` for password auth
providers that just want to provide a mechanism for validating
`m.login.password` logins.
If implemented, it will be called to check logins with an
`m.login.password` login type. It is passed a qualified
`@localpart:domain` user id, and the password provided by the user.
The method should return an `Awaitable` object, which resolves
to `True` if authentication is successful, and `False` if not.
* `on_logged_out(self, user_id, device_id, access_token)`
This method, if implemented, is called when a user logs out. It is
passed the qualified user ID, the ID of the deactivated device (if
any: access tokens are occasionally created without an associated
device ID), and the (now deactivated) access token.
It may return an `Awaitable` object; the logout request will
wait for the `Awaitable` to complete, but the result is ignored.

View File

@ -188,6 +188,9 @@ to do step 2.
It is safe to at any time kill the port script and restart it. It is safe to at any time kill the port script and restart it.
Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres.
### Using the port script ### Using the port script
Firstly, shut down the currently running synapse server and copy its Firstly, shut down the currently running synapse server and copy its

View File

@ -10,6 +10,17 @@
# homeserver.yaml. Instead, if you are starting from scratch, please generate # homeserver.yaml. Instead, if you are starting from scratch, please generate
# a fresh config using Synapse by following the instructions in INSTALL.md. # a fresh config using Synapse by following the instructions in INSTALL.md.
# Configuration options that take a time period can be set using a number
# followed by a letter. Letters have the following meanings:
# s = second
# m = minute
# h = hour
# d = day
# w = week
# y = year
# For example, setting redaction_retention_period: 5m would remove redacted
# messages from the database after 5 minutes, rather than 5 months.
################################################################################ ################################################################################
# Configuration file for Synapse. # Configuration file for Synapse.
@ -314,6 +325,10 @@ limit_remote_rooms:
# #
#complexity_error: "This room is too complex." #complexity_error: "This room is too complex."
# allow server admins to join complex rooms. Default is false.
#
#admins_can_join: true
# Whether to require a user to be in the room to add an alias to it. # Whether to require a user to be in the room to add an alias to it.
# Defaults to 'true'. # Defaults to 'true'.
# #
@ -1145,24 +1160,6 @@ account_validity:
# #
#default_identity_server: https://matrix.org #default_identity_server: https://matrix.org
# The list of identity servers trusted to verify third party
# identifiers by this server.
#
# Also defines the ID server which will be called when an account is
# deactivated (one will be picked arbitrarily).
#
# Note: This option is deprecated. Since v0.99.4, Synapse has tracked which identity
# server a 3PID has been bound to. For 3PIDs bound before then, Synapse runs a
# background migration script, informing itself that the identity server all of its
# 3PIDs have been bound to is likely one of the below.
#
# As of Synapse v1.4.0, all other functionality of this option has been deprecated, and
# it is now solely used for the purposes of the background migration script, and can be
# removed once it has run.
#trusted_third_party_id_servers:
# - matrix.org
# - vector.im
# Handle threepid (email/phone etc) registration and password resets through a set of # Handle threepid (email/phone etc) registration and password resets through a set of
# *trusted* identity servers. Note that this allows the configured identity server to # *trusted* identity servers. Note that this allows the configured identity server to
# reset passwords for accounts! # reset passwords for accounts!
@ -2398,3 +2395,57 @@ opentracing:
# #
# logging: # logging:
# false # false
## Workers ##
# Disables sending of outbound federation transactions on the main process.
# Uncomment if using a federation sender worker.
#
#send_federation: false
# It is possible to run multiple federation sender workers, in which case the
# work is balanced across them.
#
# This configuration must be shared between all federation sender workers, and if
# changed all federation sender workers must be stopped at the same time and then
# started, to ensure that all instances are running with the same config (otherwise
# events may be dropped).
#
#federation_sender_instances:
# - federation_sender1
# When using workers this should be a map from `worker_name` to the
# HTTP replication listener of the worker, if configured.
#
#instance_map:
# worker1:
# host: localhost
# port: 8034
# Experimental: When using workers you can define which workers should
# handle event persistence and typing notifications. Any worker
# specified here must also be in the `instance_map`.
#
#stream_writers:
# events: worker1
# typing: worker1
# Configuration for Redis when using workers. This *must* be enabled when
# using workers (unless using old style direct TCP configuration).
#
redis:
# Uncomment the below to enable Redis support.
#
#enabled: true
# Optional host and port to use to connect to redis. Defaults to
# localhost and 6379
#
#host: localhost
#port: 6379
# Optional password if configured on the Redis instance
#
#password: <secret_password>

32
docs/synctl_workers.md Normal file
View File

@ -0,0 +1,32 @@
### Using synctl with workers
If you want to use `synctl` to manage your synapse processes, you will need to
create an an additional configuration file for the main synapse process. That
configuration should look like this:
```yaml
worker_app: synapse.app.homeserver
```
Additionally, each worker app must be configured with the name of a "pid file",
to which it will write its process ID when it starts. For example, for a
synchrotron, you might write:
```yaml
worker_pid_file: /home/matrix/synapse/worker1.pid
```
Finally, to actually run your worker-based synapse, you must pass synctl the `-a`
commandline option to tell it to operate on all the worker configurations found
in the given directory, e.g.:
synctl -a $CONFIG/workers start
Currently one should always restart all workers when restarting or upgrading
synapse, unless you explicitly know it's safe not to. For instance, restarting
synapse without restarting all the synchrotrons may result in broken typing
notifications.
To manipulate a specific worker, you pass the -w option to synctl:
synctl -w $CONFIG/workers/worker1.yaml restart

View File

@ -1,10 +1,10 @@
# Scaling synapse via workers # Scaling synapse via workers
For small instances it recommended to run Synapse in monolith mode (the For small instances it recommended to run Synapse in the default monolith mode.
default). For larger instances where performance is a concern it can be helpful For larger instances where performance is a concern it can be helpful to split
to split out functionality into multiple separate python processes. These out functionality into multiple separate python processes. These processes are
processes are called 'workers', and are (eventually) intended to scale called 'workers', and are (eventually) intended to scale horizontally
horizontally independently. independently.
Synapse's worker support is under active development and subject to change as Synapse's worker support is under active development and subject to change as
we attempt to rapidly scale ever larger Synapse instances. However we are we attempt to rapidly scale ever larger Synapse instances. However we are
@ -16,69 +16,115 @@ workers only work with PostgreSQL-based Synapse deployments. SQLite should only
be used for demo purposes and any admin considering workers should already be be used for demo purposes and any admin considering workers should already be
running PostgreSQL. running PostgreSQL.
## Master/worker communication ## Main process/worker communication
The workers communicate with the master process via a Synapse-specific protocol The processes communicate with each other via a Synapse-specific protocol called
called 'replication' (analogous to MySQL- or Postgres-style database 'replication' (analogous to MySQL- or Postgres-style database replication) which
replication) which feeds a stream of relevant data from the master to the feeds streams of newly written data between processes so they can be kept in
workers so they can be kept in sync with the master process and database state. sync with the database state.
Additionally, workers may make HTTP requests to the master, to send information When configured to do so, Synapse uses a
in the other direction. Typically this is used for operations which need to [Redis pub/sub channel](https://redis.io/topics/pubsub) to send the replication
wait for a reply - such as sending an event. stream between all configured Synapse processes. Additionally, processes may
make HTTP requests to each other, primarily for operations which need to wait
for a reply ─ such as sending an event.
## Configuration Redis support was added in v1.13.0 with it becoming the recommended method in
v1.18.0. It replaced the old direct TCP connections (which is deprecated as of
v1.18.0) to the main process. With Redis, rather than all the workers connecting
to the main process, all the workers and the main process connect to Redis,
which relays replication commands between processes. This can give a significant
cpu saving on the main process and will be a prerequisite for upcoming
performance improvements.
See the [Architectural diagram](#architectural-diagram) section at the end for
a visualisation of what this looks like.
## Setting up workers
A Redis server is required to manage the communication between the processes.
The Redis server should be installed following the normal procedure for your
distribution (e.g. `apt install redis-server` on Debian). It is safe to use an
existing Redis deployment if you have one.
Once installed, check that Redis is running and accessible from the host running
Synapse, for example by executing `echo PING | nc -q1 localhost 6379` and seeing
a response of `+PONG`.
The appropriate dependencies must also be installed for Synapse. If using a
virtualenv, these can be installed with:
```sh
pip install matrix-synapse[redis]
```
Note that these dependencies are included when synapse is installed with `pip
install matrix-synapse[all]`. They are also included in the debian packages from
`matrix.org` and in the docker images at
https://hub.docker.com/r/matrixdotorg/synapse/.
To make effective use of the workers, you will need to configure an HTTP To make effective use of the workers, you will need to configure an HTTP
reverse-proxy such as nginx or haproxy, which will direct incoming requests to reverse-proxy such as nginx or haproxy, which will direct incoming requests to
the correct worker, or to the main synapse instance. Note that this includes the correct worker, or to the main synapse instance. See
requests made to the federation port. See [reverse_proxy.md](reverse_proxy.md) [reverse_proxy.md](reverse_proxy.md) for information on setting up a reverse
for information on setting up a reverse proxy. proxy.
To enable workers, you need to add *two* replication listeners to the To enable workers you should create a configuration file for each worker
main Synapse configuration file (`homeserver.yaml`). For example: process. Each worker configuration file inherits the configuration of the shared
homeserver configuration file. You can then override configuration specific to
that worker, e.g. the HTTP listener that it provides (if any); logging
configuration; etc. You should minimise the number of overrides though to
maintain a usable config.
### Shared Configuration
Next you need to add both a HTTP replication listener, used for HTTP requests
between processes, and redis config to the shared Synapse configuration file
(`homeserver.yaml`). For example:
```yaml ```yaml
# extend the existing `listeners` section. This defines the ports that the
# main process will listen on.
listeners: listeners:
# The TCP replication port
- port: 9092
bind_address: '127.0.0.1'
type: replication
# The HTTP replication port # The HTTP replication port
- port: 9093 - port: 9093
bind_address: '127.0.0.1' bind_address: '127.0.0.1'
type: http type: http
resources: resources:
- names: [replication] - names: [replication]
redis:
enabled: true
``` ```
Under **no circumstances** should these replication API listeners be exposed to See the sample config for the full documentation of each option.
the public internet; they have no authentication and are unencrypted.
You should then create a set of configs for the various worker processes. Each Under **no circumstances** should the replication listener be exposed to the
worker configuration file inherits the configuration of the main homeserver public internet; it has no authentication and is unencrypted.
configuration file. You can then override configuration specific to that
worker, e.g. the HTTP listener that it provides (if any); logging
configuration; etc. You should minimise the number of overrides though to ### Worker Configuration
maintain a usable config.
In the config file for each worker, you must specify the type of worker In the config file for each worker, you must specify the type of worker
application (`worker_app`). The currently available worker applications are application (`worker_app`), and you should specify a unqiue name for the worker
listed below. You must also specify the replication endpoints that it should (`worker_name`). The currently available worker applications are listed below.
talk to on the main synapse process. `worker_replication_host` should specify You must also specify the HTTP replication endpoint that it should talk to on
the host of the main synapse, `worker_replication_port` should point to the TCP the main synapse process. `worker_replication_host` should specify the host of
replication listener port and `worker_replication_http_port` should point to the main synapse and `worker_replication_http_port` should point to the HTTP
the HTTP replication port. replication port. If the worker will handle HTTP requests then the
`worker_listeners` option should be set with a `http` listener, in the same way
as the `listeners` option in the shared config.
For example: For example:
```yaml ```yaml
worker_app: synapse.app.synchrotron worker_app: synapse.app.generic_worker
worker_name: worker1
# The replication listener on the synapse to talk to. # The replication listener on the main synapse process.
worker_replication_host: 127.0.0.1 worker_replication_host: 127.0.0.1
worker_replication_port: 9092
worker_replication_http_port: 9093 worker_replication_http_port: 9093
worker_listeners: worker_listeners:
@ -87,13 +133,14 @@ worker_listeners:
resources: resources:
- names: - names:
- client - client
- federation
worker_log_config: /home/matrix/synapse/config/synchrotron_log_config.yaml worker_log_config: /home/matrix/synapse/config/worker1_log_config.yaml
``` ```
...is a full configuration for a synchrotron worker instance, which will expose a ...is a full configuration for a generic worker instance, which will expose a
plain HTTP `/sync` endpoint on port 8083 separately from the `/sync` endpoint provided plain HTTP endpoint on port 8083 separately serving various endpoints, e.g.
by the main synapse. `/sync`, which are listed below.
Obviously you should configure your reverse-proxy to route the relevant Obviously you should configure your reverse-proxy to route the relevant
endpoints to the worker (`localhost:8083` in the above example). endpoints to the worker (`localhost:8083` in the above example).
@ -102,127 +149,24 @@ Finally, you need to start your worker processes. This can be done with either
`synctl` or your distribution's preferred service manager such as `systemd`. We `synctl` or your distribution's preferred service manager such as `systemd`. We
recommend the use of `systemd` where available: for information on setting up recommend the use of `systemd` where available: for information on setting up
`systemd` to start synapse workers, see `systemd` to start synapse workers, see
[systemd-with-workers](systemd-with-workers). To use `synctl`, see below. [systemd-with-workers](systemd-with-workers). To use `synctl`, see
[synctl_workers.md](synctl_workers.md).
### **Experimental** support for replication over redis
As of Synapse v1.13.0, it is possible to configure Synapse to send replication
via a [Redis pub/sub channel](https://redis.io/topics/pubsub). This is an
alternative to direct TCP connections to the master: rather than all the
workers connecting to the master, all the workers and the master connect to
Redis, which relays replication commands between processes. This can give a
significant cpu saving on the master and will be a prerequisite for upcoming
performance improvements.
Note that this support is currently experimental; you may experience lost
messages and similar problems! It is strongly recommended that admins setting
up workers for the first time use direct TCP replication as above.
To configure Synapse to use Redis:
1. Install Redis following the normal procedure for your distribution - for
example, on Debian, `apt install redis-server`. (It is safe to use an
existing Redis deployment if you have one: we use a pub/sub stream named
according to the `server_name` of your synapse server.)
2. Check Redis is running and accessible: you should be able to `echo PING | nc -q1
localhost 6379` and get a response of `+PONG`.
3. Install the python prerequisites. If you installed synapse into a
virtualenv, this can be done with:
```sh
pip install matrix-synapse[redis]
```
The debian packages from matrix.org already include the required
dependencies.
4. Add config to the shared configuration (`homeserver.yaml`):
```yaml
redis:
enabled: true
```
Optional parameters which can go alongside `enabled` are `host`, `port`,
`password`. Normally none of these are required.
5. Restart master and all workers.
Once redis replication is in use, `worker_replication_port` is redundant and
can be removed from the worker configuration files. Similarly, the
configuration for the `listener` for the TCP replication port can be removed
from the main configuration file. Note that the HTTP replication port is
still required.
### Using synctl
If you want to use `synctl` to manage your synapse processes, you will need to
create an an additional configuration file for the master synapse process. That
configuration should look like this:
```yaml
worker_app: synapse.app.homeserver
```
Additionally, each worker app must be configured with the name of a "pid file",
to which it will write its process ID when it starts. For example, for a
synchrotron, you might write:
```yaml
worker_pid_file: /home/matrix/synapse/synchrotron.pid
```
Finally, to actually run your worker-based synapse, you must pass synctl the `-a`
commandline option to tell it to operate on all the worker configurations found
in the given directory, e.g.:
synctl -a $CONFIG/workers start
Currently one should always restart all workers when restarting or upgrading
synapse, unless you explicitly know it's safe not to. For instance, restarting
synapse without restarting all the synchrotrons may result in broken typing
notifications.
To manipulate a specific worker, you pass the -w option to synctl:
synctl -w $CONFIG/workers/synchrotron.yaml restart
## Available worker applications ## Available worker applications
### `synapse.app.pusher` ### `synapse.app.generic_worker`
Handles sending push notifications to sygnal and email. Doesn't handle any This worker can handle API requests matching the following regular
REST endpoints itself, but you should set `start_pushers: False` in the expressions:
shared configuration file to stop the main synapse sending these notifications.
Note this worker cannot be load-balanced: only one instance should be active.
### `synapse.app.synchrotron`
The synchrotron handles `sync` requests from clients. In particular, it can
handle REST endpoints matching the following regular expressions:
# Sync requests
^/_matrix/client/(v2_alpha|r0)/sync$ ^/_matrix/client/(v2_alpha|r0)/sync$
^/_matrix/client/(api/v1|v2_alpha|r0)/events$ ^/_matrix/client/(api/v1|v2_alpha|r0)/events$
^/_matrix/client/(api/v1|r0)/initialSync$ ^/_matrix/client/(api/v1|r0)/initialSync$
^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$ ^/_matrix/client/(api/v1|r0)/rooms/[^/]+/initialSync$
The above endpoints should all be routed to the synchrotron worker by the # Federation requests
reverse-proxy configuration.
It is possible to run multiple instances of the synchrotron to scale
horizontally. In this case the reverse-proxy should be configured to
load-balance across the instances, though it will be more efficient if all
requests from a particular user are routed to a single instance. Extracting
a userid from the access token is currently left as an exercise for the reader.
### `synapse.app.appservice`
Handles sending output traffic to Application Services. Doesn't handle any
REST endpoints itself, but you should set `notify_appservices: False` in the
shared configuration file to stop the main synapse sending these notifications.
Note this worker cannot be load-balanced: only one instance should be active.
### `synapse.app.federation_reader`
Handles a subset of federation endpoints. In particular, it can handle REST
endpoints matching the following regular expressions:
^/_matrix/federation/v1/event/ ^/_matrix/federation/v1/event/
^/_matrix/federation/v1/state/ ^/_matrix/federation/v1/state/
^/_matrix/federation/v1/state_ids/ ^/_matrix/federation/v1/state_ids/
@ -242,40 +186,145 @@ endpoints matching the following regular expressions:
^/_matrix/federation/v1/event_auth/ ^/_matrix/federation/v1/event_auth/
^/_matrix/federation/v1/exchange_third_party_invite/ ^/_matrix/federation/v1/exchange_third_party_invite/
^/_matrix/federation/v1/user/devices/ ^/_matrix/federation/v1/user/devices/
^/_matrix/federation/v1/send/
^/_matrix/federation/v1/get_groups_publicised$ ^/_matrix/federation/v1/get_groups_publicised$
^/_matrix/key/v2/query ^/_matrix/key/v2/query
# Inbound federation transaction request
^/_matrix/federation/v1/send/
# Client API requests
^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$
^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
^/_matrix/client/(api/v1|r0|unstable)/keys/query$
^/_matrix/client/(api/v1|r0|unstable)/keys/changes$
^/_matrix/client/versions$
^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$
^/_matrix/client/(api/v1|r0|unstable)/joined_groups$
^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$
^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/
# Registration/login requests
^/_matrix/client/(api/v1|r0|unstable)/login$
^/_matrix/client/(r0|unstable)/register$
^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$
# Event sending requests
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
^/_matrix/client/(api/v1|r0|unstable)/join/
^/_matrix/client/(api/v1|r0|unstable)/profile/
Additionally, the following REST endpoints can be handled for GET requests: Additionally, the following REST endpoints can be handled for GET requests:
^/_matrix/federation/v1/groups/ ^/_matrix/federation/v1/groups/
The above endpoints should all be routed to the federation_reader worker by the Pagination requests can also be handled, but all requests for a given
reverse-proxy configuration. room must be routed to the same instance. Additionally, care must be taken to
ensure that the purge history admin API is not used while pagination requests
for the room are in flight:
The `^/_matrix/federation/v1/send/` endpoint must only be handled by a single ^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$
instance.
Note that `federation` must be added to the listener resources in the worker config: Note that a HTTP listener with `client` and `federation` resources must be
configured in the `worker_listeners` option in the worker config.
#### Load balancing
It is possible to run multiple instances of this worker app, with incoming requests
being load-balanced between them by the reverse-proxy. However, different endpoints
have different characteristics and so admins
may wish to run multiple groups of workers handling different endpoints so that
load balancing can be done in different ways.
For `/sync` and `/initialSync` requests it will be more efficient if all
requests from a particular user are routed to a single instance. Extracting a
user ID from the access token or `Authorization` header is currently left as an
exercise for the reader. Admins may additionally wish to separate out `/sync`
requests that have a `since` query parameter from those that don't (and
`/initialSync`), as requests that don't are known as "initial sync" that happens
when a user logs in on a new device and can be *very* resource intensive, so
isolating these requests will stop them from interfering with other users ongoing
syncs.
Federation and client requests can be balanced via simple round robin.
The inbound federation transaction request `^/_matrix/federation/v1/send/`
should be balanced by source IP so that transactions from the same remote server
go to the same process.
Registration/login requests can be handled separately purely to help ensure that
unexpected load doesn't affect new logins and sign ups.
Finally, event sending requests can be balanced by the room ID in the URI (or
the full URI, or even just round robin), the room ID is the path component after
`/rooms/`. If there is a large bridge connected that is sending or may send lots
of events, then a dedicated set of workers can be provisioned to limit the
effects of bursts of events from that bridge on events sent by normal users.
#### Stream writers
Additionally, there is *experimental* support for moving writing of specific
streams (such as events) off of the main process to a particular worker. (This
is only supported with Redis-based replication.)
Currently support streams are `events` and `typing`.
To enable this, the worker must have a HTTP replication listener configured,
have a `worker_name` and be listed in the `instance_map` config. For example to
move event persistence off to a dedicated worker, the shared configuration would
include:
```yaml ```yaml
worker_app: synapse.app.federation_reader instance_map:
... event_persister1:
worker_listeners: host: localhost
- type: http port: 8034
port: <port>
resources: stream_writers:
- names: events: event_persister1
- federation
``` ```
### `synapse.app.pusher`
Handles sending push notifications to sygnal and email. Doesn't handle any
REST endpoints itself, but you should set `start_pushers: False` in the
shared configuration file to stop the main synapse sending push notifications.
Note this worker cannot be load-balanced: only one instance should be active.
### `synapse.app.appservice`
Handles sending output traffic to Application Services. Doesn't handle any
REST endpoints itself, but you should set `notify_appservices: False` in the
shared configuration file to stop the main synapse sending appservice notifications.
Note this worker cannot be load-balanced: only one instance should be active.
### `synapse.app.federation_sender` ### `synapse.app.federation_sender`
Handles sending federation traffic to other servers. Doesn't handle any Handles sending federation traffic to other servers. Doesn't handle any
REST endpoints itself, but you should set `send_federation: False` in the REST endpoints itself, but you should set `send_federation: False` in the
shared configuration file to stop the main synapse sending this traffic. shared configuration file to stop the main synapse sending this traffic.
Note this worker cannot be load-balanced: only one instance should be active. If running multiple federation senders then you must list each
instance in the `federation_sender_instances` option by their `worker_name`.
All instances must be stopped and started when adding or removing instances.
For example:
```yaml
federation_sender_instances:
- federation_sender1
- federation_sender2
```
### `synapse.app.media_repository` ### `synapse.app.media_repository`
@ -314,46 +363,6 @@ and you must configure a single instance to run the background tasks, e.g.:
media_instance_running_background_jobs: "media-repository-1" media_instance_running_background_jobs: "media-repository-1"
``` ```
### `synapse.app.client_reader`
Handles client API endpoints. It can handle REST endpoints matching the
following regular expressions:
^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$
^/_matrix/client/(api/v1|r0|unstable)/login$
^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
^/_matrix/client/(api/v1|r0|unstable)/keys/query$
^/_matrix/client/(api/v1|r0|unstable)/keys/changes$
^/_matrix/client/versions$
^/_matrix/client/(api/v1|r0|unstable)/voip/turnServer$
^/_matrix/client/(api/v1|r0|unstable)/joined_groups$
^/_matrix/client/(api/v1|r0|unstable)/publicised_groups$
^/_matrix/client/(api/v1|r0|unstable)/publicised_groups/
Additionally, the following REST endpoints can be handled for GET requests:
^/_matrix/client/(api/v1|r0|unstable)/pushrules/.*$
^/_matrix/client/(api/v1|r0|unstable)/groups/.*$
^/_matrix/client/(api/v1|r0|unstable)/user/[^/]*/account_data/
^/_matrix/client/(api/v1|r0|unstable)/user/[^/]*/rooms/[^/]*/account_data/
Additionally, the following REST endpoints can be handled, but all requests must
be routed to the same instance:
^/_matrix/client/(r0|unstable)/register$
^/_matrix/client/(r0|unstable)/auth/.*/fallback/web$
Pagination requests can also be handled, but all requests with the same path
room must be routed to the same instance. Additionally, care must be taken to
ensure that the purge history admin API is not used while pagination requests
for the room are in flight:
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/messages$
### `synapse.app.user_dir` ### `synapse.app.user_dir`
Handles searches in the user directory. It can handle REST endpoints matching Handles searches in the user directory. It can handle REST endpoints matching
@ -388,15 +397,48 @@ file. For example:
worker_main_http_uri: http://127.0.0.1:8008 worker_main_http_uri: http://127.0.0.1:8008
### `synapse.app.event_creator` ### Historical apps
Handles some event creation. It can handle REST endpoints matching: *Note:* Historically there used to be more apps, however they have been
amalgamated into a single `synapse.app.generic_worker` app. The remaining apps
are ones that do specific processing unrelated to requests, e.g. the `pusher`
that handles sending out push notifications for new events. The intention is for
all these to be folded into the `generic_worker` app and to use config to define
which processes handle the various proccessing such as push notifications.
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/send
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state/
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
^/_matrix/client/(api/v1|r0|unstable)/join/
^/_matrix/client/(api/v1|r0|unstable)/profile/
It will create events locally and then send them on to the main synapse ## Architectural diagram
instance to be persisted and handled.
The following shows an example setup using Redis and a reverse proxy:
```
Clients & Federation
|
v
+-----------+
| |
| Reverse |
| Proxy |
| |
+-----------+
| | |
| | | HTTP requests
+-------------------+ | +-----------+
| +---+ |
| | |
v v v
+--------------+ +--------------+ +--------------+ +--------------+
| Main | | Generic | | Generic | | Event |
| Process | | Worker 1 | | Worker 2 | | Persister |
+--------------+ +--------------+ +--------------+ +--------------+
^ ^ | ^ | | ^ | ^ ^
| | | | | | | | | |
| | | | | HTTP | | | | |
| +----------+<--|---|---------+ | | | |
| | +-------------|-->+----------+ |
| | | |
| | | |
v v v v
====================================================================
Redis pub/sub channel
```

View File

@ -0,0 +1,34 @@
#!/bin/bash
#
# Copyright 2020 The Matrix.org Foundation C.I.C.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This script checks that line terminators in all repository files (excluding
# those in the .git directory) feature unix line terminators.
#
# Usage:
#
# ./check_line_terminators.sh
#
# The script will emit exit code 1 if any files that do not use unix line
# terminators are found, 0 otherwise.
# cd to the root of the repository
cd `dirname $0`/..
# Find and print files with non-unix line terminators
if find . -path './.git/*' -prune -o -type f -print0 | xargs -0 grep -I -l $'\r$'; then
echo -e '\e[31mERROR: found files with CRLF line endings. See above.\e[39m'
exit 1
fi

View File

@ -69,7 +69,7 @@ logger = logging.getLogger("synapse_port_db")
BOOLEAN_COLUMNS = { BOOLEAN_COLUMNS = {
"events": ["processed", "outlier", "contains_url"], "events": ["processed", "outlier", "contains_url", "count_as_unread"],
"rooms": ["is_public"], "rooms": ["is_public"],
"event_edges": ["is_state"], "event_edges": ["is_state"],
"presence_list": ["accepted"], "presence_list": ["accepted"],

View File

@ -17,6 +17,7 @@
""" This is a reference implementation of a Matrix homeserver. """ This is a reference implementation of a Matrix homeserver.
""" """
import json
import os import os
import sys import sys
@ -25,6 +26,9 @@ if sys.version_info < (3, 5):
print("Synapse requires Python 3.5 or above.") print("Synapse requires Python 3.5 or above.")
sys.exit(1) sys.exit(1)
# Twisted and canonicaljson will fail to import when this file is executed to
# get the __version__ during a fresh install. That's OK and subsequent calls to
# actually start Synapse will import these libraries fine.
try: try:
from twisted.internet import protocol from twisted.internet import protocol
from twisted.internet.protocol import Factory from twisted.internet.protocol import Factory
@ -36,7 +40,15 @@ try:
except ImportError: except ImportError:
pass pass
__version__ = "1.17.0" # Use the standard library json implementation instead of simplejson.
try:
from canonicaljson import set_json_library
set_json_library(json)
except ImportError:
pass
__version__ = "1.18.0"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)): if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when # We import here so that we don't have to install a bunch of deps when

View File

@ -82,7 +82,7 @@ class Auth(object):
@defer.inlineCallbacks @defer.inlineCallbacks
def check_from_context(self, room_version: str, event, context, do_sig_check=True): def check_from_context(self, room_version: str, event, context, do_sig_check=True):
prev_state_ids = yield context.get_prev_state_ids() prev_state_ids = yield defer.ensureDeferred(context.get_prev_state_ids())
auth_events_ids = yield self.compute_auth_events( auth_events_ids = yield self.compute_auth_events(
event, prev_state_ids, for_verification=True event, prev_state_ids, for_verification=True
) )
@ -127,9 +127,11 @@ class Auth(object):
if current_state: if current_state:
member = current_state.get((EventTypes.Member, user_id), None) member = current_state.get((EventTypes.Member, user_id), None)
else: else:
member = yield self.state.get_current_state( member = yield defer.ensureDeferred(
self.state.get_current_state(
room_id=room_id, event_type=EventTypes.Member, state_key=user_id room_id=room_id, event_type=EventTypes.Member, state_key=user_id
) )
)
membership = member.membership if member else None membership = member.membership if member else None
if membership == Membership.JOIN: if membership == Membership.JOIN:
@ -665,9 +667,11 @@ class Auth(object):
) )
return member_event.membership, member_event.event_id return member_event.membership, member_event.event_id
except AuthError: except AuthError:
visibility = yield self.state.get_current_state( visibility = yield defer.ensureDeferred(
self.state.get_current_state(
room_id, EventTypes.RoomHistoryVisibility, "" room_id, EventTypes.RoomHistoryVisibility, ""
) )
)
if ( if (
visibility visibility
and visibility.content["history_visibility"] == "world_readable" and visibility.content["history_visibility"] == "world_readable"

View File

@ -87,7 +87,6 @@ from synapse.replication.tcp.streams import (
ReceiptsStream, ReceiptsStream,
TagAccountDataStream, TagAccountDataStream,
ToDeviceStream, ToDeviceStream,
TypingStream,
) )
from synapse.rest.admin import register_servlets_for_media_repo from synapse.rest.admin import register_servlets_for_media_repo
from synapse.rest.client.v1 import events from synapse.rest.client.v1 import events
@ -629,7 +628,7 @@ class GenericWorkerServer(HomeServer):
self.get_tcp_replication().start_replication(self) self.get_tcp_replication().start_replication(self)
def remove_pusher(self, app_id, push_key, user_id): async def remove_pusher(self, app_id, push_key, user_id):
self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id) self.get_tcp_replication().send_remove_pusher(app_id, push_key, user_id)
def build_replication_data_handler(self): def build_replication_data_handler(self):
@ -644,7 +643,6 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
super(GenericWorkerReplicationHandler, self).__init__(hs) super(GenericWorkerReplicationHandler, self).__init__(hs)
self.store = hs.get_datastore() self.store = hs.get_datastore()
self.typing_handler = hs.get_typing_handler()
self.presence_handler = hs.get_presence_handler() # type: GenericWorkerPresence self.presence_handler = hs.get_presence_handler() # type: GenericWorkerPresence
self.notifier = hs.get_notifier() self.notifier = hs.get_notifier()
@ -681,11 +679,6 @@ class GenericWorkerReplicationHandler(ReplicationDataHandler):
await self.pusher_pool.on_new_receipts( await self.pusher_pool.on_new_receipts(
token, token, {row.room_id for row in rows} token, token, {row.room_id for row in rows}
) )
elif stream_name == TypingStream.NAME:
self.typing_handler.process_replication_rows(token, rows)
self.notifier.on_new_event(
"typing_key", token, rooms=[row.room_id for row in rows]
)
elif stream_name == ToDeviceStream.NAME: elif stream_name == ToDeviceStream.NAME:
entities = [row.entity for row in rows if row.entity.startswith("@")] entities = [row.entity for row in rows if row.entity.startswith("@")]
if entities: if entities:
@ -947,7 +940,7 @@ def start(config_options):
config.server.update_user_directory = False config.server.update_user_directory = False
if config.worker_app == "synapse.app.federation_sender": if config.worker_app == "synapse.app.federation_sender":
if config.federation.send_federation: if config.worker.send_federation:
sys.stderr.write( sys.stderr.write(
"\nThe send_federation must be disabled in the main synapse process" "\nThe send_federation must be disabled in the main synapse process"
"\nbefore they can be run in a separate worker." "\nbefore they can be run in a separate worker."
@ -957,10 +950,10 @@ def start(config_options):
sys.exit(1) sys.exit(1)
# Force the pushers to start since they will be disabled in the main config # Force the pushers to start since they will be disabled in the main config
config.federation.send_federation = True config.worker.send_federation = True
else: else:
# For other worker types we force this to off. # For other worker types we force this to off.
config.federation.send_federation = False config.worker.send_federation = False
synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts synapse.events.USE_FROZEN_DICTS = config.use_frozen_dicts

View File

@ -15,11 +15,9 @@
import logging import logging
import re import re
from twisted.internet import defer
from synapse.api.constants import EventTypes from synapse.api.constants import EventTypes
from synapse.types import GroupID, get_domain_from_id from synapse.types import GroupID, get_domain_from_id
from synapse.util.caches.descriptors import cachedInlineCallbacks from synapse.util.caches.descriptors import cached
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@ -43,7 +41,7 @@ class AppServiceTransaction(object):
Args: Args:
as_api(ApplicationServiceApi): The API to use to send. as_api(ApplicationServiceApi): The API to use to send.
Returns: Returns:
A Deferred which resolves to True if the transaction was sent. An Awaitable which resolves to True if the transaction was sent.
""" """
return as_api.push_bulk( return as_api.push_bulk(
service=self.service, events=self.events, txn_id=self.id service=self.service, events=self.events, txn_id=self.id
@ -172,8 +170,7 @@ class ApplicationService(object):
return regex_obj["exclusive"] return regex_obj["exclusive"]
return False return False
@defer.inlineCallbacks async def _matches_user(self, event, store):
def _matches_user(self, event, store):
if not event: if not event:
return False return False
@ -188,12 +185,12 @@ class ApplicationService(object):
if not store: if not store:
return False return False
does_match = yield self._matches_user_in_member_list(event.room_id, store) does_match = await self._matches_user_in_member_list(event.room_id, store)
return does_match return does_match
@cachedInlineCallbacks(num_args=1, cache_context=True) @cached(num_args=1, cache_context=True)
def _matches_user_in_member_list(self, room_id, store, cache_context): async def _matches_user_in_member_list(self, room_id, store, cache_context):
member_list = yield store.get_users_in_room( member_list = await store.get_users_in_room(
room_id, on_invalidate=cache_context.invalidate room_id, on_invalidate=cache_context.invalidate
) )
@ -208,35 +205,33 @@ class ApplicationService(object):
return self.is_interested_in_room(event.room_id) return self.is_interested_in_room(event.room_id)
return False return False
@defer.inlineCallbacks async def _matches_aliases(self, event, store):
def _matches_aliases(self, event, store):
if not store or not event: if not store or not event:
return False return False
alias_list = yield store.get_aliases_for_room(event.room_id) alias_list = await store.get_aliases_for_room(event.room_id)
for alias in alias_list: for alias in alias_list:
if self.is_interested_in_alias(alias): if self.is_interested_in_alias(alias):
return True return True
return False return False
@defer.inlineCallbacks async def is_interested(self, event, store=None) -> bool:
def is_interested(self, event, store=None):
"""Check if this service is interested in this event. """Check if this service is interested in this event.
Args: Args:
event(Event): The event to check. event(Event): The event to check.
store(DataStore) store(DataStore)
Returns: Returns:
bool: True if this service would like to know about this event. True if this service would like to know about this event.
""" """
# Do cheap checks first # Do cheap checks first
if self._matches_room_id(event): if self._matches_room_id(event):
return True return True
if (yield self._matches_aliases(event, store)): if await self._matches_aliases(event, store):
return True return True
if (yield self._matches_user(event, store)): if await self._matches_user(event, store):
return True return True
return False return False

Some files were not shown because too many files have changed in this diff Show More