2019-09-17 07:55:29 -04:00
# Scaling synapse via workers
2016-08-19 13:55:57 -04:00
2022-05-25 05:14:03 -04:00
For small instances it is recommended to run Synapse in the default monolith mode.
2020-07-30 12:30:11 -04:00
For larger instances where performance is a concern it can be helpful to split
out functionality into multiple separate python processes. These processes are
called 'workers', and are (eventually) intended to scale horizontally
independently.
2016-08-19 13:55:57 -04:00
2020-05-11 08:21:15 -04:00
Synapse's worker support is under active development and subject to change as
we attempt to rapidly scale ever larger Synapse instances. However we are
documenting it here to help admins needing a highly scalable Synapse instance
similar to the one running `matrix.org` .
2017-11-21 08:22:43 -05:00
2020-05-11 08:21:15 -04:00
All processes continue to share the same database instance, and as such,
workers only work with PostgreSQL-based Synapse deployments. SQLite should only
be used for demo purposes and any admin considering workers should already be
running PostgreSQL.
2016-08-19 13:55:57 -04:00
2021-06-16 08:15:52 -04:00
See also [Matrix.org blog post ](https://matrix.org/blog/2020/11/03/how-we-fixed-synapses-scalability )
2021-01-17 19:27:27 -05:00
for a higher level overview.
2020-07-29 18:22:13 -04:00
## Main process/worker communication
2020-05-11 08:21:15 -04:00
2020-07-29 18:22:13 -04:00
The processes communicate with each other via a Synapse-specific protocol called
'replication' (analogous to MySQL- or Postgres-style database replication) which
feeds streams of newly written data between processes so they can be kept in
sync with the database state.
2020-05-11 08:21:15 -04:00
2020-08-06 05:35:59 -04:00
When configured to do so, Synapse uses a
2022-04-04 20:40:31 -04:00
[Redis pub/sub channel ](https://redis.io/docs/manual/pubsub/ ) to send the replication
2020-07-30 12:30:11 -04:00
stream between all configured Synapse processes. Additionally, processes may
make HTTP requests to each other, primarily for operations which need to wait
for a reply ─ such as sending an event.
2016-08-19 13:55:57 -04:00
2022-09-06 03:50:02 -04:00
All the workers and the main process connect to Redis, which relays replication
commands between processes.
2020-07-29 18:22:13 -04:00
2021-01-27 09:06:27 -05:00
If Redis support is enabled Synapse will use it as a shared cache, as well as a
pub/sub mechanism.
2020-07-30 12:30:11 -04:00
See the [Architectural diagram ](#architectural-diagram ) section at the end for
a visualisation of what this looks like.
2020-07-29 18:22:13 -04:00
## Setting up workers
A Redis server is required to manage the communication between the processes.
2020-07-30 12:30:11 -04:00
The Redis server should be installed following the normal procedure for your
distribution (e.g. `apt install redis-server` on Debian). It is safe to use an
existing Redis deployment if you have one.
2020-07-29 18:22:13 -04:00
Once installed, check that Redis is running and accessible from the host running
Synapse, for example by executing `echo PING | nc -q1 localhost 6379` and seeing
a response of `+PONG` .
The appropriate dependencies must also be installed for Synapse. If using a
virtualenv, these can be installed with:
```sh
2021-01-18 11:12:20 -05:00
pip install "matrix-synapse[redis]"
2020-07-29 18:22:13 -04:00
```
Note that these dependencies are included when synapse is installed with `pip
install matrix-synapse[all]`. They are also included in the debian packages from
2023-12-13 10:18:58 -05:00
`packages.matrix.org` and in the docker images at
2023-11-20 14:22:46 -05:00
https://hub.docker.com/r/ectorim/synapse/.
2017-11-21 08:22:43 -05:00
To make effective use of the workers, you will need to configure an HTTP
reverse-proxy such as nginx or haproxy, which will direct incoming requests to
2020-08-06 05:35:59 -04:00
the correct worker, or to the main synapse instance. See
2021-07-15 07:47:55 -04:00
[the reverse proxy documentation ](reverse_proxy.md ) for information on setting up a reverse
2020-07-30 12:30:11 -04:00
proxy.
2017-11-21 08:22:43 -05:00
2020-08-06 05:35:59 -04:00
When using workers, each worker process has its own configuration file which
contains settings specific to that worker, such as the HTTP listener that it
provides (if any), logging configuration, etc.
2020-07-29 18:22:13 -04:00
2020-08-06 05:35:59 -04:00
Normally, the worker processes are configured to read from a shared
configuration file as well as the worker-specific configuration files. This
makes it easier to keep common configuration settings synchronised across all
the processes.
2020-07-30 12:30:11 -04:00
2020-08-06 05:35:59 -04:00
The main process is somewhat special in this respect: it does not normally
need its own configuration file and can take all of its configuration from the
shared configuration file.
### Shared configuration
2023-05-11 06:30:56 -04:00
Normally, only a few changes are needed to make an existing configuration
file suitable for use with workers:
* First, you need to enable an
2022-10-27 09:39:47 -04:00
["HTTP replication listener" ](usage/configuration/config_documentation.md#listeners )
2023-05-11 06:30:56 -04:00
for the main process
* Secondly, you need to enable
[redis-based replication ](usage/configuration/config_documentation.md#redis )
2023-12-13 10:18:58 -05:00
* You will need to add an [`instance_map` ](usage/configuration/config_documentation.md#instance_map )
2023-07-11 14:08:06 -04:00
with the `main` process defined, as well as the relevant connection information from
it's HTTP `replication` listener (defined in step 1 above).
* Note that the `host` defined is the address the worker needs to look for the `main`
process at, not necessarily the same address that is bound to.
* If you are using Unix sockets for the `replication` resource, make sure to
use a `path` to the socket file instead of a `port` .
2023-05-11 06:30:56 -04:00
* Optionally, a [shared secret ](usage/configuration/config_documentation.md#worker_replication_secret )
2022-10-27 09:39:47 -04:00
can be used to authenticate HTTP traffic between workers. For example:
2020-07-30 12:30:11 -04:00
2020-05-11 08:21:15 -04:00
```yaml
2020-07-29 18:22:13 -04:00
# extend the existing `listeners` section. This defines the ports that the
# main process will listen on.
2020-05-11 08:21:15 -04:00
listeners:
# The HTTP replication port
- port: 9093
bind_address: '127.0.0.1'
type: http
resources:
- names: [replication]
2020-07-29 18:22:13 -04:00
2020-12-04 10:56:28 -05:00
# Add a random shared secret to authenticate traffic.
worker_replication_secret: ""
2020-07-29 18:22:13 -04:00
redis:
enabled: true
2023-05-11 06:30:56 -04:00
instance_map:
main:
host: 'localhost'
port: 9093
2020-05-11 08:21:15 -04:00
```
2018-02-12 12:18:07 -05:00
2022-10-27 09:39:47 -04:00
See the [configuration manual ](usage/configuration/config_documentation.md )
for the full documentation of each option.
2016-08-19 13:55:57 -04:00
2020-07-29 18:22:13 -04:00
Under **no circumstances** should the replication listener be exposed to the
2022-08-30 08:21:19 -04:00
public internet; replication traffic is:
* always unencrypted
2022-10-27 09:39:47 -04:00
* unauthenticated, unless [`worker_replication_secret` ](usage/configuration/config_documentation.md#worker_replication_secret )
is configured
2016-08-19 13:55:57 -04:00
2020-07-30 12:30:11 -04:00
2020-08-06 05:35:59 -04:00
### Worker configuration
2020-07-30 12:30:11 -04:00
2022-08-30 13:01:51 -04:00
In the config file for each worker, you must specify:
2022-10-27 09:39:47 -04:00
* The type of worker ([`worker_app`](usage/configuration/config_documentation.md#worker_app)).
The currently available worker applications are listed [below ](#available-worker-applications ).
* A unique name for the worker ([`worker_name`](usage/configuration/config_documentation.md#worker_name)).
* If handling HTTP requests, a [`worker_listeners` ](usage/configuration/config_documentation.md#worker_listeners ) option
with an `http` listener.
2022-11-17 11:11:08 -05:00
* **Synapse 1.72 and older:** if handling the `^/_matrix/client/v3/keys/upload` endpoint, the HTTP URI for
the main process (`worker_main_http_uri`). This config option is no longer required and is ignored when running Synapse 1.73 and newer.
2016-08-19 13:55:57 -04:00
2020-05-11 08:21:15 -04:00
For example:
2016-08-19 13:55:57 -04:00
2020-05-11 08:21:15 -04:00
```yaml
2022-04-25 08:17:03 -04:00
{{#include systemd-with-workers/workers/generic_worker.yaml}}
2020-05-11 08:21:15 -04:00
```
2016-08-19 13:55:57 -04:00
2020-07-29 18:22:13 -04:00
...is a full configuration for a generic worker instance, which will expose a
plain HTTP endpoint on port 8083 separately serving various endpoints, e.g.
`/sync` , which are listed below.
2016-08-19 13:55:57 -04:00
2017-11-21 08:22:43 -05:00
Obviously you should configure your reverse-proxy to route the relevant
2019-09-17 07:55:29 -04:00
endpoints to the worker (`localhost:8083` in the above example).
2016-08-19 13:55:57 -04:00
2020-08-06 05:35:59 -04:00
### Running Synapse with workers
2020-04-08 06:59:26 -04:00
Finally, you need to start your worker processes. This can be done with either
`synctl` or your distribution's preferred service manager such as `systemd` . We
recommend the use of `systemd` where available: for information on setting up
`systemd` to start synapse workers, see
2023-01-05 13:18:00 -05:00
[Systemd with Workers ](systemd-with-workers/ ). To use `synctl` , see
2021-07-15 07:47:55 -04:00
[Using synctl with Workers ](synctl_workers.md ).
2020-04-08 06:59:26 -04:00
2023-02-16 13:20:02 -05:00
## Start Synapse with Poetry
2016-08-19 13:55:57 -04:00
2023-02-16 13:20:02 -05:00
The following applies to Synapse installations that have been installed from source using `poetry` .
You can start the main Synapse process with Poetry by running the following command:
```console
2023-07-11 18:13:54 -04:00
poetry run synapse_homeserver --config-file [your homeserver.yaml]
2023-02-16 13:20:02 -05:00
```
For worker setups, you can run the following command
```console
2023-07-11 18:13:54 -04:00
poetry run synapse_worker --config-file [your homeserver.yaml] --config-file [your worker.yaml]
2023-02-16 13:20:02 -05:00
```
2019-09-17 07:55:29 -04:00
## Available worker applications
2017-11-21 08:22:43 -05:00
2020-07-29 18:22:13 -04:00
### `synapse.app.generic_worker`
2017-11-21 08:22:43 -05:00
2022-02-23 17:49:04 -05:00
This worker can handle API requests matching the following regular expressions.
These endpoints can be routed to any worker. If a worker is set up to handle a
stream then, for maximum efficiency, additional endpoints should be routed to that
worker: refer to the [stream writers ](#stream-writers ) section below for further
information.
2017-11-21 08:22:43 -05:00
2020-07-29 18:22:13 -04:00
# Sync requests
2022-03-18 12:51:27 -04:00
^/_matrix/client/(r0|v3)/sync$
^/_matrix/client/(api/v1|r0|v3)/events$
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3)/initialSync$
^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$
2017-11-21 08:22:43 -05:00
2020-07-29 18:22:13 -04:00
# Federation requests
2017-11-21 08:22:43 -05:00
^/_matrix/federation/v1/event/
^/_matrix/federation/v1/state/
^/_matrix/federation/v1/state_ids/
^/_matrix/federation/v1/backfill/
^/_matrix/federation/v1/get_missing_events/
^/_matrix/federation/v1/publicRooms
2018-08-09 05:33:55 -04:00
^/_matrix/federation/v1/query/
^/_matrix/federation/v1/make_join/
^/_matrix/federation/v1/make_leave/
2022-03-18 12:51:27 -04:00
^/_matrix/federation/(v1|v2)/send_join/
^/_matrix/federation/(v1|v2)/send_leave/
2024-05-17 05:54:11 -04:00
^/_matrix/federation/v1/make_knock/
^/_matrix/federation/v1/send_knock/
2022-03-18 12:51:27 -04:00
^/_matrix/federation/(v1|v2)/invite/
2018-08-09 05:33:55 -04:00
^/_matrix/federation/v1/event_auth/
2022-11-28 16:54:18 -05:00
^/_matrix/federation/v1/timestamp_to_event/
2018-08-09 05:33:55 -04:00
^/_matrix/federation/v1/exchange_third_party_invite/
2020-02-07 10:45:39 -05:00
^/_matrix/federation/v1/user/devices/
2019-02-27 08:43:53 -05:00
^/_matrix/key/v2/query
2022-05-26 07:10:28 -04:00
^/_matrix/federation/v1/hierarchy/
2017-11-21 08:22:43 -05:00
2020-07-29 18:22:13 -04:00
# Inbound federation transaction request
^/_matrix/federation/v1/send/
# Client API requests
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/createRoom$
^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$
2022-05-26 07:10:28 -04:00
^/_matrix/client/v1/rooms/.*/hierarchy$
2022-10-12 06:46:13 -04:00
^/_matrix/client/(v1|unstable)/rooms/.*/relations/
2022-10-14 07:16:50 -04:00
^/_matrix/client/v1/rooms/.*/threads$
2024-04-26 04:46:42 -04:00
^/_matrix/client/unstable/im.nheko.summary/summary/.*$
2022-02-23 17:49:04 -05:00
^/_matrix/client/(r0|v3|unstable)/account/3pid$
2022-05-25 05:41:41 -04:00
^/_matrix/client/(r0|v3|unstable)/account/whoami$
2022-02-23 17:49:04 -05:00
^/_matrix/client/(r0|v3|unstable)/devices$
2020-07-29 18:22:13 -04:00
^/_matrix/client/versions$
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/
^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$
2022-11-28 16:54:18 -05:00
^/_matrix/client/v1/rooms/.*/timestamp_to_event$
2023-03-08 07:51:34 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable/.*)/rooms/.*/aliases
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/search$
2023-02-28 11:37:19 -05:00
^/_matrix/client/(r0|v3|unstable)/user/.*/filter(/|$)
2023-04-14 05:24:06 -04:00
^/_matrix/client/(api/v1|r0|v3|unstable)/directory/room/.*$
2023-04-14 12:26:07 -04:00
^/_matrix/client/(r0|v3|unstable)/capabilities$
2023-09-07 05:26:07 -04:00
^/_matrix/client/(r0|v3|unstable)/notifications$
2020-07-29 18:22:13 -04:00
2022-02-23 17:49:04 -05:00
# Encryption requests
^/_matrix/client/(r0|v3|unstable)/keys/query$
^/_matrix/client/(r0|v3|unstable)/keys/changes$
^/_matrix/client/(r0|v3|unstable)/keys/claim$
^/_matrix/client/(r0|v3|unstable)/room_keys/
2022-08-30 13:01:51 -04:00
^/_matrix/client/(r0|v3|unstable)/keys/upload/
2022-02-23 17:49:04 -05:00
2020-07-29 18:22:13 -04:00
# Registration/login requests
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/login$
^/_matrix/client/(r0|v3|unstable)/register$
2023-03-17 09:50:31 -04:00
^/_matrix/client/(r0|v3|unstable)/register/available$
2022-02-04 07:15:13 -05:00
^/_matrix/client/v1/register/m.login.registration_token/validity$
2023-03-27 07:37:17 -04:00
^/_matrix/client/(r0|v3|unstable)/password_policy$
2020-07-29 18:22:13 -04:00
# Event sending requests
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$
^/_matrix/client/(api/v1|r0|v3|unstable)/join/
2023-03-02 12:59:53 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/knock/
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/profile/
2020-07-29 18:22:13 -04:00
2022-02-23 17:49:04 -05:00
# Account data requests
^/_matrix/client/(r0|v3|unstable)/.*/tags
^/_matrix/client/(r0|v3|unstable)/.*/account_data
# Receipts requests
^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt
^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers
# Presence requests
^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
2022-05-19 07:03:12 -04:00
# User directory search requests
^/_matrix/client/(r0|v3|unstable)/user_directory/search$
2020-07-29 18:22:13 -04:00
2020-02-07 06:14:19 -05:00
Additionally, the following REST endpoints can be handled for GET requests:
2022-03-18 12:51:27 -04:00
^/_matrix/client/(api/v1|r0|v3|unstable)/pushrules/
2020-02-07 06:14:19 -05:00
2020-07-29 18:22:13 -04:00
Pagination requests can also be handled, but all requests for a given
room must be routed to the same instance. Additionally, care must be taken to
ensure that the purge history admin API is not used while pagination requests
for the room are in flight:
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/messages$
2020-07-29 18:22:13 -04:00
2020-09-30 13:02:43 -04:00
Additionally, the following endpoints should be included if Synapse is configured
to use SSO (you only need to include the ones for whichever SSO provider you're
using):
2021-02-01 10:47:59 -05:00
# for all SSO providers
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/login/sso/redirect
2021-02-01 10:47:59 -05:00
^/_synapse/client/pick_idp$
2021-02-18 06:56:25 -05:00
^/_synapse/client/pick_username
2021-02-01 13:37:41 -05:00
^/_synapse/client/new_user_consent$
2021-02-01 10:47:59 -05:00
^/_synapse/client/sso_register$
2020-09-30 13:02:43 -04:00
# OpenID Connect requests.
2021-02-01 17:56:01 -05:00
^/_synapse/client/oidc/callback$
2020-09-30 13:02:43 -04:00
# SAML requests.
2021-02-02 04:43:50 -05:00
^/_synapse/client/saml2/authn_response$
2020-09-30 13:02:43 -04:00
# CAS requests.
2021-11-17 10:30:24 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/login/cas/ticket$
2020-09-30 13:02:43 -04:00
2021-02-01 10:47:59 -05:00
Ensure that all SSO logins go to a single process.
2020-11-06 09:33:07 -05:00
For multiple workers not handling the SSO endpoints properly, see
2021-03-24 07:43:04 -04:00
[#7530 ](https://github.com/matrix-org/synapse/issues/7530 ) and
2021-02-17 12:14:23 -05:00
[#9427 ](https://github.com/matrix-org/synapse/issues/9427 ).
2020-07-29 18:22:13 -04:00
2022-09-30 08:41:32 -04:00
Note that a [HTTP listener ](usage/configuration/config_documentation.md#listeners )
2022-10-27 09:39:47 -04:00
with `client` and `federation` `resources` must be configured in the
[`worker_listeners` ](usage/configuration/config_documentation.md#worker_listeners )
2022-09-30 08:41:32 -04:00
option in the worker config.
2021-02-01 10:47:59 -05:00
2020-07-29 18:22:13 -04:00
#### Load balancing
It is possible to run multiple instances of this worker app, with incoming requests
being load-balanced between them by the reverse-proxy. However, different endpoints
have different characteristics and so admins
may wish to run multiple groups of workers handling different endpoints so that
load balancing can be done in different ways.
For `/sync` and `/initialSync` requests it will be more efficient if all
2022-11-07 08:51:53 -05:00
requests from a particular user are routed to a single instance. This can
2023-04-27 06:25:44 -04:00
be done in reverse proxy by extracting username part from the users access token.
2022-11-07 08:51:53 -05:00
Admins may additionally wish to separate out `/sync`
2020-07-29 18:22:13 -04:00
requests that have a `since` query parameter from those that don't (and
`/initialSync` ), as requests that don't are known as "initial sync" that happens
when a user logs in on a new device and can be *very* resource intensive, so
isolating these requests will stop them from interfering with other users ongoing
syncs.
2023-04-27 06:25:44 -04:00
Example `nginx` configuration snippet that handles the cases above. This is just an
example and probably requires some changes according to your particular setup:
```nginx
# Choose sync worker based on the existence of "since" query parameter
map $arg_since $sync {
default synapse_sync;
'' synapse_initial_sync;
}
# Extract username from access token passed as URL parameter
map $arg_access_token $accesstoken_from_urlparam {
# Defaults to just passing back the whole accesstoken
default $arg_access_token;
# Try to extract username part from accesstoken URL parameter
"~syt_(?< username > .*?)_.*" $username;
}
# Extract username from access token passed as authorization header
map $http_authorization $mxid_localpart {
# Defaults to just passing back the whole accesstoken
default $http_authorization;
# Try to extract username part from accesstoken header
"~Bearer syt_(?< username > .*?)_.*" $username;
# if no authorization-header exist, try mapper for URL parameter "access_token"
"" $accesstoken_from_urlparam;
}
upstream synapse_initial_sync {
# Use the username mapper result for hash key
hash $mxid_localpart consistent;
server 127.0.0.1:8016;
server 127.0.0.1:8036;
}
upstream synapse_sync {
# Use the username mapper result for hash key
hash $mxid_localpart consistent;
server 127.0.0.1:8013;
server 127.0.0.1:8037;
server 127.0.0.1:8038;
server 127.0.0.1:8039;
}
# Sync initial/normal
location ~ ^/_matrix/client/(r0|v3)/sync$ {
proxy_pass http://$sync;
}
# Normal sync
location ~ ^/_matrix/client/(api/v1|r0|v3)/events$ {
proxy_pass http://synapse_sync;
}
# Initial_sync
location ~ ^/_matrix/client/(api/v1|r0|v3)/initialSync$ {
proxy_pass http://synapse_initial_sync;
}
location ~ ^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$ {
proxy_pass http://synapse_initial_sync;
}
```
2020-07-29 18:22:13 -04:00
Federation and client requests can be balanced via simple round robin.
2017-11-21 08:22:43 -05:00
2020-07-29 18:22:13 -04:00
The inbound federation transaction request `^/_matrix/federation/v1/send/`
should be balanced by source IP so that transactions from the same remote server
go to the same process.
2018-08-09 05:33:55 -04:00
2020-07-29 18:22:13 -04:00
Registration/login requests can be handled separately purely to help ensure that
unexpected load doesn't affect new logins and sign ups.
Finally, event sending requests can be balanced by the room ID in the URI (or
the full URI, or even just round robin), the room ID is the path component after
`/rooms/` . If there is a large bridge connected that is sending or may send lots
of events, then a dedicated set of workers can be provisioned to limit the
effects of bursts of events from that bridge on events sent by normal users.
#### Stream writers
2022-04-14 10:25:22 -04:00
Additionally, the writing of specific streams (such as events) can be moved off
of the main process to a particular worker.
2020-07-29 18:22:13 -04:00
2023-05-11 06:30:56 -04:00
To enable this, the worker must have:
* An [HTTP `replication` listener ](usage/configuration/config_documentation.md#listeners ) configured,
* Have a [`worker_name` ](usage/configuration/config_documentation.md#worker_name )
2022-10-27 09:39:47 -04:00
and be listed in the [`instance_map` ](usage/configuration/config_documentation.md#instance_map )
2023-12-13 10:18:58 -05:00
config.
2023-05-11 06:30:56 -04:00
* Have the main process declared on the [`instance_map` ](usage/configuration/config_documentation.md#instance_map ) as well.
Note: The same worker can handle multiple streams, but unless otherwise documented,
2022-10-27 09:39:47 -04:00
each stream can only have a single writer.
2022-03-10 13:15:19 -05:00
For example, to move event persistence off to a dedicated worker, the shared
configuration would include:
2020-01-27 03:20:48 -05:00
```yaml
2020-07-29 18:22:13 -04:00
instance_map:
2023-05-11 06:30:56 -04:00
main:
host: localhost
port: 8030
2020-07-29 18:22:13 -04:00
event_persister1:
host: localhost
port: 8034
2020-07-30 11:28:36 -04:00
stream_writers:
2020-07-29 18:22:13 -04:00
events: event_persister1
2020-01-27 03:20:48 -05:00
```
2020-01-24 04:01:57 -05:00
2022-04-25 08:17:03 -04:00
An example for a stream writer instance:
```yaml
{{#include systemd-with-workers/workers/event_persister.yaml}}
```
2022-02-23 17:49:04 -05:00
Some of the streams have associated endpoints which, for maximum efficiency, should
be routed to the workers handling that stream. See below for the currently supported
streams and the endpoints associated with them:
##### The `events` stream
2022-10-27 09:39:47 -04:00
The `events` stream experimentally supports having multiple writer workers, where load
is sharded between them by room ID. Each writer is called an _event persister_ . They are
responsible for
- receiving new events,
- linking them to those already in the room [DAG ](development/room-dag-concepts.md ),
- persisting them to the DB, and finally
- updating the events stream.
2022-11-16 17:16:25 -05:00
Because load is sharded in this way, you *must* restart all worker instances when
2022-10-27 09:39:47 -04:00
adding or removing event persisters.
An `event_persister` should not be mistaken for an `event_creator` .
An `event_creator` listens for requests from clients to create new events and does
so. It will then pass those events over HTTP replication to any configured event
persisters (or the main process if none are configured).
Note that `event_creator` s and `event_persister` s are implemented using the same
2023-01-05 13:18:00 -05:00
[`synapse.app.generic_worker` ](#synapseappgeneric_worker ).
2022-10-27 09:39:47 -04:00
An example [`stream_writers` ](usage/configuration/config_documentation.md#stream_writers )
2020-11-03 05:27:11 -05:00
configuration with multiple writers:
```yaml
stream_writers:
events:
- event_persister1
- event_persister2
```
2022-02-23 17:49:04 -05:00
##### The `typing` stream
2022-03-10 13:15:19 -05:00
The following endpoints should be routed directly to the worker configured as
the stream writer for the `typing` stream:
2022-02-23 17:49:04 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing
##### The `to_device` stream
2022-03-10 13:15:19 -05:00
The following endpoints should be routed directly to the worker configured as
the stream writer for the `to_device` stream:
2022-02-23 17:49:04 -05:00
2022-03-18 12:51:27 -04:00
^/_matrix/client/(r0|v3|unstable)/sendToDevice/
2022-02-23 17:49:04 -05:00
##### The `account_data` stream
2022-03-10 13:15:19 -05:00
The following endpoints should be routed directly to the worker configured as
the stream writer for the `account_data` stream:
2022-02-23 17:49:04 -05:00
2022-03-18 12:51:27 -04:00
^/_matrix/client/(r0|v3|unstable)/.*/tags
^/_matrix/client/(r0|v3|unstable)/.*/account_data
2022-02-23 17:49:04 -05:00
##### The `receipts` stream
2022-03-10 13:15:19 -05:00
The following endpoints should be routed directly to the worker configured as
the stream writer for the `receipts` stream:
2022-02-23 17:49:04 -05:00
2022-03-18 12:51:27 -04:00
^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt
^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers
2022-02-23 17:49:04 -05:00
##### The `presence` stream
2022-03-10 13:15:19 -05:00
The following endpoints should be routed directly to the worker configured as
the stream writer for the `presence` stream:
2022-02-23 17:49:04 -05:00
^/_matrix/client/(api/v1|r0|v3|unstable)/presence/
2024-03-28 12:29:23 -04:00
##### The `push_rules` stream
2024-03-28 11:44:07 -04:00
The following endpoints should be routed directly to the worker configured as
2024-05-09 06:50:05 -04:00
the stream writer for the `push_rules` stream:
2024-03-28 11:44:07 -04:00
2024-03-28 12:29:23 -04:00
^/_matrix/client/(api/v1|r0|v3|unstable)/pushrules/
2024-03-28 11:44:07 -04:00
2023-07-18 04:49:21 -04:00
#### Restrict outbound federation traffic to a specific set of workers
The
[`outbound_federation_restricted_to` ](usage/configuration/config_documentation.md#outbound_federation_restricted_to )
configuration is useful to make sure outbound federation traffic only goes through a
specified subset of workers. This allows you to set more strict access controls (like a
firewall) for all workers and only allow the `federation_sender` 's to contact the
outside world.
```yaml
instance_map:
main:
host: localhost
port: 8030
federation_sender1:
host: localhost
port: 8034
outbound_federation_restricted_to:
- federation_sender1
worker_replication_secret: "secret_secret"
```
2020-10-02 08:23:15 -04:00
#### Background tasks
2022-04-14 10:25:22 -04:00
There is also support for moving background tasks to a separate
2020-10-02 08:23:15 -04:00
worker. Background tasks are run periodically or started via replication. Exactly
which tasks are configured to run depends on your Synapse configuration (e.g. if
2022-09-30 08:41:32 -04:00
stats is enabled). This worker doesn't handle any REST endpoints itself.
2020-10-02 08:23:15 -04:00
2022-10-27 09:39:47 -04:00
To enable this, the worker must have a unique
[`worker_name` ](usage/configuration/config_documentation.md#worker_name )
and can be configured to run background tasks. For example, to move background tasks
to a dedicated worker, the shared configuration would include:
2020-10-02 08:23:15 -04:00
```yaml
run_background_tasks_on: background_worker
```
2022-10-27 09:39:47 -04:00
You might also wish to investigate the
[`update_user_directory_from_worker` ](#updating-the-user-directory ) and
[`media_instance_running_background_jobs` ](#synapseappmedia_repository ) settings.
2020-07-29 18:22:13 -04:00
2022-04-25 08:17:03 -04:00
An example for a dedicated background worker instance:
```yaml
{{#include systemd-with-workers/workers/background_worker.yaml}}
```
2022-05-10 06:08:45 -04:00
#### Updating the User Directory
You can designate one generic worker to update the user directory.
2023-01-09 13:35:19 -05:00
Specify its name in the [shared configuration ](usage/configuration/config_documentation.md#update_user_directory_from_worker )
as follows:
2022-05-10 06:08:45 -04:00
```yaml
update_user_directory_from_worker: worker_name
```
This work cannot be load-balanced; please ensure the main process is restarted
after setting this option in the shared configuration!
2022-05-19 07:03:12 -04:00
User directory updates allow REST endpoints matching the following regular
expressions to work:
^/_matrix/client/(r0|v3|unstable)/user_directory/search$
The above endpoints can be routed to any worker, though you may choose to route
it to the chosen user directory worker.
2022-05-10 06:08:45 -04:00
This style of configuration supersedes the legacy `synapse.app.user_dir`
worker application type.
2022-05-06 06:43:53 -04:00
#### Notifying Application Services
2022-05-10 06:08:45 -04:00
You can designate one generic worker to send output traffic to Application Services.
2022-09-30 08:41:32 -04:00
Doesn't handle any REST endpoints itself, but you should specify its name in the
2023-01-09 13:35:19 -05:00
[shared configuration ](usage/configuration/config_documentation.md#notify_appservices_from_worker )
as follows:
2022-05-06 06:43:53 -04:00
```yaml
notify_appservices_from_worker: worker_name
```
This work cannot be load-balanced; please ensure the main process is restarted
after setting this option in the shared configuration!
This style of configuration supersedes the legacy `synapse.app.appservice`
worker application type.
2023-01-09 13:35:19 -05:00
#### Push Notifications
2024-04-26 14:10:45 -04:00
You can designate generic workers to send push notifications to
2023-01-09 13:35:19 -05:00
a [push gateway ](https://spec.matrix.org/v1.5/push-gateway-api/ ) such as
2024-02-06 04:26:55 -05:00
[sygnal ](https://github.com/matrix-org/sygnal ) and email.
2023-01-09 13:35:19 -05:00
This will stop the main process sending push notifications.
The workers responsible for sending push notifications can be defined using the
[`pusher_instances` ](usage/configuration/config_documentation.md#pusher_instances )
option. For example:
```yaml
pusher_instances:
- pusher_worker1
- pusher_worker2
```
Multiple workers can be added to this map, in which case the work is balanced
across them. Ensure the main process and all pusher workers are restarted after changing
this option.
These workers don't need to accept incoming HTTP requests to send push notifications,
so no additional reverse proxy configuration is required for pusher workers.
This style of configuration supersedes the legacy `synapse.app.pusher`
worker application type.
2022-05-06 06:43:53 -04:00
2020-07-29 18:22:13 -04:00
### `synapse.app.pusher`
2022-12-02 06:38:01 -05:00
It is likely this option will be deprecated in the future and is not recommended for new
2023-01-09 13:35:19 -05:00
installations. Instead, [use `synapse.app.generic_worker` with the `pusher_instances` ](#push-notifications ).
2022-12-02 06:38:01 -05:00
2020-07-29 18:22:13 -04:00
Handles sending push notifications to sygnal and email. Doesn't handle any
2022-10-27 09:39:47 -04:00
REST endpoints itself, but you should set
[`start_pushers: false` ](usage/configuration/config_documentation.md#start_pushers ) in the
2020-07-29 18:22:13 -04:00
shared configuration file to stop the main synapse sending push notifications.
2022-10-27 09:39:47 -04:00
To run multiple instances at once the
[`pusher_instances` ](usage/configuration/config_documentation.md#pusher_instances )
option should list all pusher instances by their
[`worker_name` ](usage/configuration/config_documentation.md#worker_name ), e.g.:
2021-02-16 08:27:49 -05:00
```yaml
2022-10-27 09:39:47 -04:00
start_pushers: false
2021-02-16 08:27:49 -05:00
pusher_instances:
- pusher_worker1
- pusher_worker2
```
2022-10-07 09:25:28 -04:00
An example for a pusher instance:
```yaml
{{#include systemd-with-workers/workers/pusher_worker.yaml}}
```
2020-07-29 18:22:13 -04:00
### `synapse.app.appservice`
2022-05-10 06:08:45 -04:00
**Deprecated as of Synapse v1.59.** [Use `synapse.app.generic_worker` with the
2022-05-06 06:43:53 -04:00
`notify_appservices_from_worker` option instead.](#notifying-application-services)
2020-07-29 18:22:13 -04:00
Handles sending output traffic to Application Services. Doesn't handle any
REST endpoints itself, but you should set `notify_appservices: False` in the
shared configuration file to stop the main synapse sending appservice notifications.
Note this worker cannot be load-balanced: only one instance should be active.
2019-09-17 07:55:29 -04:00
### `synapse.app.federation_sender`
2017-11-21 08:22:43 -05:00
2022-12-02 06:38:01 -05:00
It is likely this option will be deprecated in the future and not recommended for
2023-01-09 13:35:19 -05:00
new installations. Instead, [use `synapse.app.generic_worker` with the `federation_sender_instances` ](usage/configuration/config_documentation.md#federation_sender_instances ).
2022-12-02 06:38:01 -05:00
2017-11-21 08:22:43 -05:00
Handles sending federation traffic to other servers. Doesn't handle any
2022-10-27 09:39:47 -04:00
REST endpoints itself, but you should set
[`send_federation: false` ](usage/configuration/config_documentation.md#send_federation )
in the shared configuration file to stop the main synapse sending this traffic.
2017-11-21 08:22:43 -05:00
2020-07-29 18:22:13 -04:00
If running multiple federation senders then you must list each
2022-10-27 09:39:47 -04:00
instance in the
[`federation_sender_instances` ](usage/configuration/config_documentation.md#federation_sender_instances )
option by their
[`worker_name` ](usage/configuration/config_documentation.md#worker_name ).
2020-07-29 18:22:13 -04:00
All instances must be stopped and started when adding or removing instances.
For example:
```yaml
2022-10-27 09:39:47 -04:00
send_federation: false
2020-07-29 18:22:13 -04:00
federation_sender_instances:
- federation_sender1
- federation_sender2
```
2017-11-21 08:22:43 -05:00
2022-10-07 09:25:28 -04:00
An example for a federation sender instance:
```yaml
{{#include systemd-with-workers/workers/federation_sender.yaml}}
```
2019-09-17 07:55:29 -04:00
### `synapse.app.media_repository`
2017-11-21 08:22:43 -05:00
2019-09-17 07:55:29 -04:00
Handles the media repository. It can handle all endpoints starting with:
2017-11-21 08:22:43 -05:00
/_matrix/media/
2024-07-02 09:07:04 -04:00
/_matrix/client/v1/media/
2024-07-09 11:41:51 -04:00
/_matrix/federation/v1/media/
2017-11-21 08:22:43 -05:00
2019-12-11 11:23:38 -05:00
... and the following regular expressions matching media-specific administration APIs:
2019-08-13 07:49:28 -04:00
^/_synapse/admin/v1/purge_media_cache$
2020-01-13 13:10:43 -05:00
^/_synapse/admin/v1/room/.*/media.*$
^/_synapse/admin/v1/user/.*/media.*$
^/_synapse/admin/v1/media/.*$
2019-08-13 07:49:28 -04:00
^/_synapse/admin/v1/quarantine_media/.*$
2021-08-18 08:25:12 -04:00
^/_synapse/admin/v1/users/.*/media$
2019-08-13 07:49:28 -04:00
2022-10-27 09:39:47 -04:00
You should also set
[`enable_media_repo: False` ](usage/configuration/config_documentation.md#enable_media_repo )
in the shared configuration
2017-11-21 08:29:39 -05:00
file to stop the main synapse running background jobs related to managing the
2021-08-18 08:25:12 -04:00
media repository. Note that doing so will prevent the main process from being
able to handle the above endpoints.
2017-11-21 08:29:39 -05:00
2022-09-30 08:41:32 -04:00
In the `media_repository` worker configuration file, configure the
[HTTP listener ](usage/configuration/config_documentation.md#listeners ) to
2019-12-11 11:23:38 -05:00
expose the `media` resource. For example:
```yaml
2022-09-30 08:41:32 -04:00
{{#include systemd-with-workers/workers/media_worker.yaml}}
2019-12-11 11:23:38 -05:00
```
2020-06-17 09:13:30 -04:00
Note that if running multiple media repositories they must be on the same server
2023-01-09 13:35:19 -05:00
and you must specify a single instance to run the background tasks in the
[shared configuration ](usage/configuration/config_documentation.md#media_instance_running_background_jobs ),
e.g.:
2020-06-17 09:13:30 -04:00
```yaml
2021-11-01 07:35:55 -04:00
media_instance_running_background_jobs: "media-repository-1"
2020-06-17 09:13:30 -04:00
```
2017-11-21 08:22:43 -05:00
2020-11-06 09:33:07 -05:00
Note that if a reverse proxy is used , then `/_matrix/media/` must be routed for both inbound client and federation requests (if they are handled separately).
2019-09-17 07:55:29 -04:00
### `synapse.app.user_dir`
2017-11-21 08:22:43 -05:00
2022-05-10 06:08:45 -04:00
**Deprecated as of Synapse v1.59.** [Use `synapse.app.generic_worker` with the
`update_user_directory_from_worker` option instead.](#updating-the-user-directory)
2017-11-21 08:22:43 -05:00
Handles searches in the user directory. It can handle REST endpoints matching
2019-09-17 07:55:29 -04:00
the following regular expressions:
2017-11-21 08:22:43 -05:00
2022-03-18 12:51:27 -04:00
^/_matrix/client/(r0|v3|unstable)/user_directory/search$
2017-11-21 08:22:43 -05:00
2022-03-23 08:47:07 -04:00
When using this worker you must also set `update_user_directory: false` in the
2020-02-18 10:27:45 -05:00
shared configuration file to stop the main synapse running background
2020-01-24 04:01:57 -05:00
jobs related to updating the user directory.
2022-03-23 08:47:07 -04:00
Above endpoint is not *required* to be routed to this worker. By default,
`update_user_directory` is set to `true` , which means the main process
will handle updates. All workers configured with `client` can handle the above
endpoint as long as either this worker or the main process are configured to
handle it, and are online.
If `update_user_directory` is set to `false` , and this worker is not running,
the above endpoint may give outdated results.
2020-07-29 18:22:13 -04:00
### Historical apps
2018-02-06 12:23:13 -05:00
2022-08-30 13:01:51 -04:00
The following used to be separate worker application types, but are now
equivalent to `synapse.app.generic_worker` :
* `synapse.app.client_reader`
* `synapse.app.event_creator`
* `synapse.app.federation_reader`
2022-12-02 06:38:01 -05:00
* `synapse.app.federation_sender`
2022-08-30 13:01:51 -04:00
* `synapse.app.frontend_proxy`
2022-12-02 06:38:01 -05:00
* `synapse.app.pusher`
2022-08-30 13:01:51 -04:00
* `synapse.app.synchrotron`
2018-02-06 12:23:13 -05:00
2020-08-06 05:35:59 -04:00
## Migration from old config
2022-09-06 03:50:02 -04:00
A main change that has occurred is the merging of worker apps into
`synapse.app.generic_worker` . This change is backwards compatible and so no
changes to the config are required.
2020-08-06 05:35:59 -04:00
To migrate apps to use `synapse.app.generic_worker` simply update the
`worker_app` option in the worker configs, and where worker are started (e.g.
in systemd service files, but not required for synctl).
2020-07-29 18:22:13 -04:00
## Architectural diagram
The following shows an example setup using Redis and a reverse proxy:
```
Clients & Federation
|
v
+-----------+
| |
| Reverse |
| Proxy |
| |
+-----------+
| | |
| | | HTTP requests
+-------------------+ | +-----------+
| +---+ |
| | |
v v v
+--------------+ +--------------+ +--------------+ +--------------+
| Main | | Generic | | Generic | | Event |
| Process | | Worker 1 | | Worker 2 | | Persister |
+--------------+ +--------------+ +--------------+ +--------------+
2022-04-25 08:17:03 -04:00
^ ^ | ^ | | ^ | | ^ ^
| | | | | | | | | | |
| | | | | HTTP | | | | | |
| +----------+< -- | --- | --------- + < -- | --- | --------- + | |
| | +-------------|-->+-------------+ |
| | | |
| | | |
v v v v
======================================================================
2020-07-29 18:22:13 -04:00
Redis pub/sub channel
```