From 0d25724419c43313f9b8345b07aaebe3eeeeee7e Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Mon, 21 May 2018 14:45:33 +0100 Subject: [PATCH 1/8] Refactor docker locations and README. This addresses #3224 --- README.rst | 2 +- contrib/docker/README.md | 110 +--------------- Dockerfile => docker/Dockerfile | 2 +- docker/README.md | 122 ++++++++++++++++++ .../docker => docker}/conf/homeserver.yaml | 0 {contrib/docker => docker}/conf/log.config | 0 {contrib/docker => docker}/start.py | 0 7 files changed, 126 insertions(+), 110 deletions(-) rename Dockerfile => docker/Dockerfile (90%) create mode 100644 docker/README.md rename {contrib/docker => docker}/conf/homeserver.yaml (100%) rename {contrib/docker => docker}/conf/log.config (100%) rename {contrib/docker => docker}/start.py (100%) diff --git a/README.rst b/README.rst index 5fdfad345..90bd49c9a 100644 --- a/README.rst +++ b/README.rst @@ -157,7 +157,7 @@ if you prefer. In case of problems, please see the _`Troubleshooting` section below. -There is an offical synapse image available at https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with the docker-compose file available at `contrib/docker`. Further information on this including configuration options is available in `contrib/docker/README.md`. +There is an offical synapse image available at https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with the docker-compose file available at `contrib/docker`. Further information on this including configuration options is available in the README on hub.docker.com. Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a Dockerfile to automate a synapse server in a single Docker image, at https://hub.docker.com/r/avhost/docker-matrix/tags/ diff --git a/contrib/docker/README.md b/contrib/docker/README.md index 562cdaac2..10b15b040 100644 --- a/contrib/docker/README.md +++ b/contrib/docker/README.md @@ -1,23 +1,5 @@ # Synapse Docker -The `matrixdotorg/synapse` Docker image will run Synapse as a single process. It does not provide a -database server or a TURN server, you should run these separately. - -If you run a Postgres server, you should simply include it in the same Compose -project or set the proper environment variables and the image will automatically -use that server. - -## Build - -Build the docker image with the `docker-compose build` command. - -You may have a local Python wheel cache available, in which case copy the relevant packages in the ``cache/`` directory at the root of the project. - -## Run - -This image is designed to run either with an automatically generated configuration -file or with a custom configuration that requires manual edition. - ### Automated configuration It is recommended that you use Docker Compose to run your containers, including @@ -54,94 +36,6 @@ Then, customize your configuration and run the server: docker-compose up -d ``` -### Without Compose +### More information -If you do not wish to use Compose, you may still run this image using plain -Docker commands. Note that the following is just a guideline and you may need -to add parameters to the docker run command to account for the network situation -with your postgres database. - -``` -docker run \ - -d \ - --name synapse \ - -v ${DATA_PATH}:/data \ - -e SYNAPSE_SERVER_NAME=my.matrix.host \ - -e SYNAPSE_REPORT_STATS=yes \ - docker.io/matrixdotorg/synapse:latest -``` - -## Volumes - -The image expects a single volume, located at ``/data``, that will hold: - -* temporary files during uploads; -* uploaded media and thumbnails; -* the SQLite database if you do not configure postgres; -* the appservices configuration. - -You are free to use separate volumes depending on storage endpoints at your -disposal. For instance, ``/data/media`` coud be stored on a large but low -performance hdd storage while other files could be stored on high performance -endpoints. - -In order to setup an application service, simply create an ``appservices`` -directory in the data volume and write the application service Yaml -configuration file there. Multiple application services are supported. - -## Environment - -Unless you specify a custom path for the configuration file, a very generic -file will be generated, based on the following environment settings. -These are a good starting point for setting up your own deployment. - -Global settings: - -* ``UID``, the user id Synapse will run as [default 991] -* ``GID``, the group id Synapse will run as [default 991] -* ``SYNAPSE_CONFIG_PATH``, path to a custom config file - -If ``SYNAPSE_CONFIG_PATH`` is set, you should generate a configuration file -then customize it manually. No other environment variable is required. - -Otherwise, a dynamic configuration file will be used. The following environment -variables are available for configuration: - -* ``SYNAPSE_SERVER_NAME`` (mandatory), the current server public hostname. -* ``SYNAPSE_REPORT_STATS``, (mandatory, ``yes`` or ``no``), enable anonymous - statistics reporting back to the Matrix project which helps us to get funding. -* ``SYNAPSE_NO_TLS``, set this variable to disable TLS in Synapse (use this if - you run your own TLS-capable reverse proxy). -* ``SYNAPSE_ENABLE_REGISTRATION``, set this variable to enable registration on - the Synapse instance. -* ``SYNAPSE_ALLOW_GUEST``, set this variable to allow guest joining this server. -* ``SYNAPSE_EVENT_CACHE_SIZE``, the event cache size [default `10K`]. -* ``SYNAPSE_CACHE_FACTOR``, the cache factor [default `0.5`]. -* ``SYNAPSE_RECAPTCHA_PUBLIC_KEY``, set this variable to the recaptcha public - key in order to enable recaptcha upon registration. -* ``SYNAPSE_RECAPTCHA_PRIVATE_KEY``, set this variable to the recaptcha private - key in order to enable recaptcha upon registration. -* ``SYNAPSE_TURN_URIS``, set this variable to the coma-separated list of TURN - uris to enable TURN for this homeserver. -* ``SYNAPSE_TURN_SECRET``, set this to the TURN shared secret if required. - -Shared secrets, that will be initialized to random values if not set: - -* ``SYNAPSE_REGISTRATION_SHARED_SECRET``, secret for registrering users if - registration is disable. -* ``SYNAPSE_MACAROON_SECRET_KEY`` secret for signing access tokens - to the server. - -Database specific values (will use SQLite if not set): - -* `POSTGRES_DB` - The database name for the synapse postgres database. [default: `synapse`] -* `POSTGRES_HOST` - The host of the postgres database if you wish to use postgresql instead of sqlite3. [default: `db` which is useful when using a container on the same docker network in a compose file where the postgres service is called `db`] -* `POSTGRES_PASSWORD` - The password for the synapse postgres database. **If this is set then postgres will be used instead of sqlite3.** [default: none] **NOTE**: You are highly encouraged to use postgresql! Please use the compose file to make it easier to deploy. -* `POSTGRES_USER` - The user for the synapse postgres database. [default: `matrix`] - -Mail server specific values (will not send emails if not set): - -* ``SYNAPSE_SMTP_HOST``, hostname to the mail server. -* ``SYNAPSE_SMTP_PORT``, TCP port for accessing the mail server [default ``25``]. -* ``SYNAPSE_SMTP_USER``, username for authenticating against the mail server if any. -* ``SYNAPSE_SMTP_PASSWORD``, password for authenticating against the mail server if any. +For more information on required environment variables and mounts, see the main docker documentation at `docker/README.md` diff --git a/Dockerfile b/docker/Dockerfile similarity index 90% rename from Dockerfile rename to docker/Dockerfile index 0242be5f6..26fb3a6bf 100644 --- a/Dockerfile +++ b/docker/Dockerfile @@ -22,7 +22,7 @@ RUN cd /synapse \ setuptools \ && mkdir -p /synapse/cache \ && pip install -f /synapse/cache --upgrade --process-dependency-links . \ - && mv /synapse/contrib/docker/start.py /synapse/contrib/docker/conf / \ + && mv /synapse/docker/start.py /synapse/docker/conf / \ && rm -rf \ setup.cfg \ setup.py \ diff --git a/docker/README.md b/docker/README.md new file mode 100644 index 000000000..f60ea4923 --- /dev/null +++ b/docker/README.md @@ -0,0 +1,122 @@ +# Synapse Docker + +This Docker image will run Synapse as a single process. It does not provide a database +server or a TURN server, you should run these separately. + +## Run + +We do not currently offer a `latest` image, as this has somewhat undefined semantics. +We instead release only tagged versions so upgrading between releases is entirely +within your control. + +### Using docker-compose (easier) + +This image is designed to run either with an automatically generated configuration +file or with a custom configuration that requires manual edition. + +An easy way to make use of this image is via docker-compose, see the (https://github.com/matrix-org/synapse/tree/develop/contrib/docker)[contrib] section of the synapse project for examples. + +### Without Compose (harder) + +If you do not wish to use Compose, you may still run this image using plain +Docker commands. Note that the following is just a guideline and you may need +to add parameters to the docker run command to account for the network situation +with your postgres database. + +``` +docker run \ + -d \ + --name synapse \ + -v ${DATA_PATH}:/data \ + -e SYNAPSE_SERVER_NAME=my.matrix.host \ + -e SYNAPSE_REPORT_STATS=yes \ + docker.io/matrixdotorg/synapse:latest +``` + +## Volumes + +The image expects a single volume, located at ``/data``, that will hold: + +* temporary files during uploads; +* uploaded media and thumbnails; +* the SQLite database if you do not configure postgres; +* the appservices configuration. + +You are free to use separate volumes depending on storage endpoints at your +disposal. For instance, ``/data/media`` coud be stored on a large but low +performance hdd storage while other files could be stored on high performance +endpoints. + +In order to setup an application service, simply create an ``appservices`` +directory in the data volume and write the application service Yaml +configuration file there. Multiple application services are supported. + +## Environment + +Unless you specify a custom path for the configuration file, a very generic +file will be generated, based on the following environment settings. +These are a good starting point for setting up your own deployment. + +Global settings: + +* ``UID``, the user id Synapse will run as [default 991] +* ``GID``, the group id Synapse will run as [default 991] +* ``SYNAPSE_CONFIG_PATH``, path to a custom config file + +If ``SYNAPSE_CONFIG_PATH`` is set, you should generate a configuration file +then customize it manually. No other environment variable is required. + +Otherwise, a dynamic configuration file will be used. The following environment +variables are available for configuration: + +* ``SYNAPSE_SERVER_NAME`` (mandatory), the current server public hostname. +* ``SYNAPSE_REPORT_STATS``, (mandatory, ``yes`` or ``no``), enable anonymous + statistics reporting back to the Matrix project which helps us to get funding. +* ``SYNAPSE_NO_TLS``, set this variable to disable TLS in Synapse (use this if + you run your own TLS-capable reverse proxy). +* ``SYNAPSE_ENABLE_REGISTRATION``, set this variable to enable registration on + the Synapse instance. +* ``SYNAPSE_ALLOW_GUEST``, set this variable to allow guest joining this server. +* ``SYNAPSE_EVENT_CACHE_SIZE``, the event cache size [default `10K`]. +* ``SYNAPSE_CACHE_FACTOR``, the cache factor [default `0.5`]. +* ``SYNAPSE_RECAPTCHA_PUBLIC_KEY``, set this variable to the recaptcha public + key in order to enable recaptcha upon registration. +* ``SYNAPSE_RECAPTCHA_PRIVATE_KEY``, set this variable to the recaptcha private + key in order to enable recaptcha upon registration. +* ``SYNAPSE_TURN_URIS``, set this variable to the coma-separated list of TURN + uris to enable TURN for this homeserver. +* ``SYNAPSE_TURN_SECRET``, set this to the TURN shared secret if required. + +Shared secrets, that will be initialized to random values if not set: + +* ``SYNAPSE_REGISTRATION_SHARED_SECRET``, secret for registrering users if + registration is disable. +* ``SYNAPSE_MACAROON_SECRET_KEY`` secret for signing access tokens + to the server. + +Database specific values (will use SQLite if not set): + +* `POSTGRES_DB` - The database name for the synapse postgres database. [default: `synapse`] +* `POSTGRES_HOST` - The host of the postgres database if you wish to use postgresql instead of sqlite3. [default: `db` which is useful when using a container on the same docker network in a compose file where the postgres service is called `db`] +* `POSTGRES_PASSWORD` - The password for the synapse postgres database. **If this is set then postgres will be used instead of sqlite3.** [default: none] **NOTE**: You are highly encouraged to use postgresql! Please use the compose file to make it easier to deploy. +* `POSTGRES_USER` - The user for the synapse postgres database. [default: `matrix`] + +Mail server specific values (will not send emails if not set): + +* ``SYNAPSE_SMTP_HOST``, hostname to the mail server. +* ``SYNAPSE_SMTP_PORT``, TCP port for accessing the mail server [default ``25``]. +* ``SYNAPSE_SMTP_USER``, username for authenticating against the mail server if any. +* ``SYNAPSE_SMTP_PASSWORD``, password for authenticating against the mail server if any. + +## Build + +Build the docker image with the `docker build` command from the root of the synapse repository. + +``` +docker build -t docker.io/matrixdotorg/synapse . -f docker/Dockerfile +``` + +The `-t` option sets the image tag. Official images are tagged `matrixdotorg/synapse:` where `` is the same as the release tag in the synapse git repository. + +You may have a local Python wheel cache available, in which case copy the relevant +packages in the ``cache/`` directory at the root of the project. diff --git a/contrib/docker/conf/homeserver.yaml b/docker/conf/homeserver.yaml similarity index 100% rename from contrib/docker/conf/homeserver.yaml rename to docker/conf/homeserver.yaml diff --git a/contrib/docker/conf/log.config b/docker/conf/log.config similarity index 100% rename from contrib/docker/conf/log.config rename to docker/conf/log.config diff --git a/contrib/docker/start.py b/docker/start.py similarity index 100% rename from contrib/docker/start.py rename to docker/start.py From 26a37f3d4dcd2257005d9afe1d90af470198e111 Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Fri, 8 Jun 2018 13:35:07 +0100 Subject: [PATCH 2/8] Do not include docker files in python build --- MANIFEST.in | 1 + 1 file changed, 1 insertion(+) diff --git a/MANIFEST.in b/MANIFEST.in index 7076b608d..1ff98d95d 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -35,3 +35,4 @@ recursive-include changelog.d * prune .github prune demo/etc +prune docker From 1758f4e1c793cd5f219469c9d5c340eea071b6c8 Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Fri, 8 Jun 2018 13:37:39 +0100 Subject: [PATCH 3/8] Address SPAG issues --- docker/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docker/README.md b/docker/README.md index f60ea4923..8303a7fec 100644 --- a/docker/README.md +++ b/docker/README.md @@ -12,9 +12,11 @@ within your control. ### Using docker-compose (easier) This image is designed to run either with an automatically generated configuration -file or with a custom configuration that requires manual edition. +file or with a custom configuration that requires manual editing. -An easy way to make use of this image is via docker-compose, see the (https://github.com/matrix-org/synapse/tree/develop/contrib/docker)[contrib] section of the synapse project for examples. +An easy way to make use of this image is via docker-compose. See the +(https://github.com/matrix-org/synapse/tree/develop/contrib/docker)[contrib] +section of the synapse project for examples. ### Without Compose (harder) From 489949879ede04f104852f11993c734e9109cd16 Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Thu, 2 Aug 2018 18:49:50 +0100 Subject: [PATCH 4/8] Add news entry --- changelog.d/3644.misc | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog.d/3644.misc diff --git a/changelog.d/3644.misc b/changelog.d/3644.misc new file mode 100644 index 000000000..2347fc850 --- /dev/null +++ b/changelog.d/3644.misc @@ -0,0 +1 @@ +Refactor location of docker build script. From feacd13932b848546f75a8f646546c3215d35316 Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Fri, 3 Aug 2018 09:10:41 +0100 Subject: [PATCH 5/8] Update README.rst wrap at 80ish --- README.rst | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/README.rst b/README.rst index 90bd49c9a..1d602adc7 100644 --- a/README.rst +++ b/README.rst @@ -157,12 +157,19 @@ if you prefer. In case of problems, please see the _`Troubleshooting` section below. -There is an offical synapse image available at https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with the docker-compose file available at `contrib/docker`. Further information on this including configuration options is available in the README on hub.docker.com. +There is an offical synapse image available at +https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with +the docker-compose file available at `contrib/docker`. Further information on +this including configuration options is available in the README on +hub.docker.com. -Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a Dockerfile to automate a synapse server in a single Docker image, at https://hub.docker.com/r/avhost/docker-matrix/tags/ +Alternatively, Andreas Peters (previously Silvio Fricke) has contributed a +Dockerfile to automate a synapse server in a single Docker image, at +https://hub.docker.com/r/avhost/docker-matrix/tags/ Also, Martin Giess has created an auto-deployment process with vagrant/ansible, -tested with VirtualBox/AWS/DigitalOcean - see https://github.com/EMnify/matrix-synapse-auto-deploy +tested with VirtualBox/AWS/DigitalOcean - see +https://github.com/EMnify/matrix-synapse-auto-deploy for details. Configuring synapse From 637b11b9edd7de93aeabb3688ab2dda6c58015a3 Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Fri, 3 Aug 2018 09:13:54 +0100 Subject: [PATCH 6/8] Update README.rst Link to contrib/docker --- README.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.rst b/README.rst index 1d602adc7..4c5971d04 100644 --- a/README.rst +++ b/README.rst @@ -159,7 +159,7 @@ In case of problems, please see the _`Troubleshooting` section below. There is an offical synapse image available at https://hub.docker.com/r/matrixdotorg/synapse/tags/ which can be used with -the docker-compose file available at `contrib/docker`. Further information on +the docker-compose file available at `contrib/docker `_. Further information on this including configuration options is available in the README on hub.docker.com. From c3f596180f19299aee6735cb9c91e3f61d073ccc Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Fri, 3 Aug 2018 09:15:19 +0100 Subject: [PATCH 7/8] Update README.md Link to docker/README.md --- contrib/docker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/docker/README.md b/contrib/docker/README.md index 10b15b040..05254e519 100644 --- a/contrib/docker/README.md +++ b/contrib/docker/README.md @@ -38,4 +38,4 @@ docker-compose up -d ### More information -For more information on required environment variables and mounts, see the main docker documentation at `docker/README.md` +For more information on required environment variables and mounts, see the main docker documentation at [/docker/README.md](../../docker/README.md) From 42960aa0475edc59f83d6a47c423d9e5f12e9417 Mon Sep 17 00:00:00 2001 From: Michael Kaye <1917473+michaelkaye@users.noreply.github.com> Date: Fri, 3 Aug 2018 09:16:01 +0100 Subject: [PATCH 8/8] Update README.md Link to contrib/docker --- docker/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/README.md b/docker/README.md index 8303a7fec..038c78f7c 100644 --- a/docker/README.md +++ b/docker/README.md @@ -15,7 +15,7 @@ This image is designed to run either with an automatically generated configurati file or with a custom configuration that requires manual editing. An easy way to make use of this image is via docker-compose. See the -(https://github.com/matrix-org/synapse/tree/develop/contrib/docker)[contrib] +[contrib/docker](../contrib/docker) section of the synapse project for examples. ### Without Compose (harder)