Commit Graph

265 Commits

Author SHA1 Message Date
Erik Johnston
c95b04bb0e
Change default sync_response_cache_duration (#13042) 2022-06-15 16:55:20 +00:00
reivilibre
212be2edc1
Use updated update_user_directory_from_worker options in Complement tests. (#13069) 2022-06-15 15:54:32 +01:00
reivilibre
538044ac01
Collapse Docker build commands in Complement CI runs to make the logs easier to read. (#13058) 2022-06-15 14:42:27 +00:00
reivilibre
67f51c84f8
Merge the Complement testing Docker images into a single, multi-purpose image. (#12881)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-08 09:57:05 +00:00
Nick Mills-Barrett
1f9013ce60
Add the batch_send endpoint to generic workers (#12868) 2022-05-25 09:51:07 +00:00
Erik Johnston
f5b1c09909
Pin poetry.core in Docker images (#12853) 2022-05-24 11:35:08 +01:00
reivilibre
67aae05ece
Support registering Application Services when running with workers under Complement. (#12826)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-05-23 14:11:06 +01:00
reivilibre
438925c422
Fix Complement TestCanRegisterAdmin with workers, by adding Complement's shared registration secret. (#12819) 2022-05-23 12:20:30 +01:00
reivilibre
4fef76ca34
Remove Caddy from the Synapse workers image used in Complement. (#12818) 2022-05-23 10:29:24 +01:00
Till
e8ae472d3b
Update configs used by Complement to allow more invites (#12731) 2022-05-13 16:45:47 +01:00
Richard van der Hoff
34e84fee68
Tweaks to workers-under-complement (#12637)
* Bump the HS startup timeout
* Log prefixes for more processes
* Bump the overall timeout
2022-05-09 22:41:06 +01:00
reivilibre
c2d50e9f6c
Add the notify_appservices_from_worker configuration option (superseding notify_appservices) to allow a generic worker to be designated as the worker to send traffic to Application Services. (#12452) 2022-05-06 11:43:53 +01:00
reivilibre
d743b25c8f
Use supervisord to supervise Postgres and Caddy in the Complement image. (#12480)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-27 14:39:41 +01:00
Jason Robinson
706456de1f
Mark Dockerfile as requiring BuildKit (#12541)
Co-authored-by: David Robertson <davidr@element.io>
2022-04-26 15:31:52 +01:00
David Robertson
8bac3e0435
disallow-untyped-defs in docker and stubs directories (#12528) 2022-04-25 12:32:35 +00:00
Richard van der Hoff
f5668f0b4a
Await un-partial-stating after a partial-state join (#12399)
When we join a room via the faster-joins mechanism, we end up with "partial
state" at some points on the event DAG. Many parts of the codebase need to
wait for the full state to load. So, we implement a mechanism to keep track of
which events have partial state, and wait for them to be fully-populated.
2022-04-21 07:42:03 +01:00
David Robertson
e5a76ec00b
Dump setuptools; correct pyproject version number (#12478) 2022-04-20 17:33:20 +01:00
Richard van der Hoff
aaaff98202
Dockerfile-workers: reduce the amount we install (#12464)
This is an attempt to reduce the rebuild time. In short, we reduce the amount
of stuff that the dockerfile installs, so as to give a faster startup.
2022-04-14 15:36:49 +01:00
Richard van der Hoff
960b4fb409
complement-synapse-workers: factor out separate entry point script (#12467)
... with a bit more verbosity.
2022-04-14 14:56:10 +01:00
Richard van der Hoff
8af8a9bce5
Dockerfile-workers: give the master its own log config (#12466)
When we run a worker-mode synapse under docker, everything gets logged to stdout. Currently, output from the workers is tacked with a worker name, for example:

```
2022-04-13 15:27:56,810 - worker:frontend_proxy1 - synapse.util.caches.lrucache - 154 - INFO - LruCache._expire_old_entries-0 - Dropped 0 items from caches
```

- note `worker:frontend_proxy1`. No such tag is applied to log lines from the master, which makes somewhat confusing reading.

To fix this, we generate a dedicated log config file for the master in the same way that we do for the workers, and use that.
2022-04-13 20:50:08 +01:00
David Robertson
5f8173dd80
Workaround pip bug installing latest treq and not-latest twisted from hashes (#12439)
The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0.

When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true

> ```
> #15 9.204 Collecting Twisted[tls]>=18.7.0
> #15 9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
> #15 9.205     Twisted[tls]>=18.7.0 from 38622ff95b/Twisted-22.4.0-py3-none-any.whl (sha256)=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724))
> #15 ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1
> ```

The underlying pip issue is https://github.com/pypa/pip/issues/9644. A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so.

(At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311, but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).
2022-04-12 10:16:01 +01:00
Richard van der Hoff
214f3b7d21
Enable certificate checking during complement tests (#12435) 2022-04-11 16:35:41 +01:00
Richard van der Hoff
3cdf5a1386
Fix up healthcheck generation for workers docker image (#12405)
This wasn't quite generating the right thing.
2022-04-11 13:38:58 +00:00
Richard van der Hoff
5f72ea1bde
Move complement setup stuff into the Synapse repo (#12404)
Fixes matrix-org/complement#330 (or it will, once we remove the old files).

It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
2022-04-11 11:39:28 +01:00
Jorge Florian
78e4d96a4d
Add missing type definitions for scripts in docker folder (#12280)
Signed-off-by: Jorge Florian <jafn28@gmail.com>
2022-04-08 10:10:58 +00:00
David Robertson
3a7e97c7ad
Poetry: use locked environment in Docker images (#12385) 2022-04-07 11:43:31 +00:00
Richard van der Hoff
34a8370d7b
README-testing.md: fix minor error 2022-03-31 13:08:03 +01:00
David Robertson
15cdcf8f30
Remove dockerfile-pgtests (#12336) 2022-03-31 11:30:07 +01:00
David Robertson
4aeb00ca20
Move synctl into synapse._scripts and expose as an entrypoint (#12140) 2022-03-04 11:58:49 +00:00
David Robertson
f3f0ab10fe
Move scripts directory inside synapse, exposing as setuptools entry_points (#12118)
* Two scripts are basically entry_points already
* Move and rename scripts/* to synapse/_scripts/*.py
* Delete sync_room_to_group.pl
* Expose entry points in setup.py
* Update linter script and config
* Fixup scripts & docs mentioning scripts that moved

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2022-03-02 13:00:16 +00:00
Andrew Morgan
f26e390a40
Use Python 3.9 in Synapse dockerfiles by default (#12112) 2022-03-01 13:55:18 +00:00
Richard van der Hoff
5598556b77
Docker: remove VOLUME directive (#11997)
The driver for this is to stop Complement complaining about it, but as far as I can tell it was pointless and needed to go away anyway.

I'm a bit unclear about what exactly VOLUME does, but I think what it means is that, if you don't override it with an explicit -v argument, then docker run will create a temporary volume, and copy things into it. The temporary volume is then deleted when the container finishes.

That only sounds useful if your image has something to copy into it (otherwise you may as well just use the default root filesystem), and our image notably doesn't copy anything into /data.

So... this wasn't doing anything, except annoying Complement?
2022-02-15 13:59:15 +00:00
Shay
9006ee36d1
Drop support for and remove references to EOL Python 3.6 (#11683)
* remove reference in comments to python3.6

* upgrade tox python env in script

* bump python version in example for completeness

* upgrade python version requirement in setup doc

* upgrade necessary python version in __init__.py

* upgrade python version in setup.py

* newsfragment

* drops refs to bionic and replace with focal

* bump refs to postgres 9.6 to 10

* fix hanging ci

* try installing tzdata first

* revert change made in b979f336

* ignore new random mypy error while debugging other error

* fix lint error for temporary workaround

* revert change to install list

* try passing env var

* export debian frontend var?

* move line and add comment

* bump pillow dependency

* bump lxml depenency

* install libjpeg-dev for pillow

* bump automat version to one compatible with py3.8

* add libwebp for pillow

* bump twisted trunk python version

* change suffix of newsfragment

* remove redundant python 3.7 checks

* lint
2022-01-21 14:23:26 -08:00
Richard van der Hoff
d41c4654db
Use buildkit's cache feature to speed up docker builds (#11691)
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
2022-01-12 10:37:57 +00:00
Callum Macdonald
7013e06e2f
Improve Docker docs for use with Postgres (#11640) 2022-01-05 10:50:28 +00:00
Shay
13c974ed35
Drop Bionic from Debian builds (#11633)
* update Trove classifiers to remove py36

* stop building bionic

* update dh-virtualenv

* newsfragment

* fix newsfragment

* update version refs

* another try at correct tag

* Update changelog
2022-01-03 11:17:16 -08:00
Michael Kaye
e2c300e7e4
Create healthcheck script for synapse-workers container (#11429)
The intent is to iterate through all the worker ports and only
report healthy when all are healthy, starting with the main process.
2021-11-26 14:05:20 +00:00
Shay
7cebaf9644
Remove code invalidated by deprecated config flag 'trust_identity_servers_for_password_resets' (#11395)
* remove background update code related to deprecated config flag

* changelog entry

* update changelog

* Delete 11394.removal

Duplicate, wrong number

* add no-op background update and change newfragment so it will be consolidated with associated work

* remove unused code

* Remove code associated with deprecated flag from legacy docker dynamic config file

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-23 06:46:40 -08:00
Aaron R
d993c3bb1e
Add support for /_matrix/media/v3 APIs (#11371)
* Add support for `/_matrix/media/v3` APIs

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Update `workers.md` to use v3 client and media APIs

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-11-17 15:30:24 +00:00
Richard van der Hoff
0b99d4c8d2
Docker: avoid changing userid unnecessarily (#11209)
* Docker image: avoid changing user during `generate`

The intention was always that the config files get written as the initial user
(normally root) - only the data directory needs to be writable by Synapse. This
got changed in https://github.com/matrix-org/synapse/pull/5970, but that seems
to have been a mistake.

* Avoid changing user if no explicit UID is given

* changelog
2021-11-01 13:55:30 +00:00
Dan Callahan
d7141e0b8b
Fix Shellcheck SC2006: Use $(...) notation
Use $(...) notation instead of legacy backticked `...`.

https://github.com/koalaman/shellcheck/wiki/SC2006

Signed-off-by: Dan Callahan <danc@element.io>
2021-10-22 23:08:55 +01:00
Dan Callahan
13f084eb58
Fix Shellcheck SC2086: Quote to prevent splitting
Double quote to prevent globbing and word splitting.

https://github.com/koalaman/shellcheck/wiki/SC2086

Signed-off-by: Dan Callahan <danc@element.io>
2021-10-22 23:08:54 +01:00
Dan Callahan
6c736fa472
Fix Shellcheck SC2154: variable possibly undefined
var is referenced but not assigned.

https://github.com/koalaman/shellcheck/wiki/SC2154

Signed-off-by: Dan Callahan <danc@element.io>
2021-10-22 23:08:54 +01:00
Richard van der Hoff
09eff1b3db
fix relative link in docker readme (#11144)
relative links don't work when it's on dockerhub.
2021-10-21 13:50:43 +01:00
Erik Johnston
a8bbf08576
Fix debian package builds. (#10931)
This was due to dh-virtualenv builds being broken due to Shpinx removing
deprecated APIs.
2021-09-28 12:13:51 +01:00
David Robertson
ea01d4c2de
Update postgresql testing script (#10906)
- Use sytest:bionic. Sytest:latest is two years old (do we want
  CI to push out latest at all?) and comes with Python 3.5, which we
  explictly no longer support. The script now runs under PostgreSQL 10
  as a result.
- Advertise script in the docs
- Move pg testing script to scripts-dev directory
- Write to host as the script's exector, not root

A few changes to make it speedier to re-run the tests:

- Create blank DB in the container, not the script, so we don't have to
  `initdb` each time
- Use a named volume to persist the tox environment, so we don't have to
  fetch and install a bunch of packages from PyPI each time

Co-authored-by: reivilibre <olivier@librepush.net>
2021-09-24 14:27:09 +00:00
kegsay
01df612e1e
Fix frontend_proxy jinja script in docker workers (#10783) 2021-09-08 17:24:53 +01:00
reivilibre
5f7b1e1f27
Make PeriodicallyFlushingMemoryHandler the default logging handler. (#10518) 2021-08-17 13:13:11 +01:00
Richard van der Hoff
2afdb5c984
Fix deb build script to set prerelease flag correctly (#10500) 2021-08-01 10:47:36 +01:00
Richard van der Hoff
8ae0bdca75
Drop xenial-support hacks (#10429) 2021-07-21 21:25:28 +01:00
Richard van der Hoff
c82eb02d64
Set section for prerelease debs (#10391)
This is part of fixing #6116: we want to put RC debs into a different place than release debs, so reprepro has to be able to tell them apart.
2021-07-14 14:41:40 +01:00
Dirk Klimpel
974261cd81
Fix broken links in INSTALL.md (#10331)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-08 15:46:13 +01:00
Brendan Abolivier
08c8469322
Remove support for ACME v1 (#10194)
Fixes #9778

ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
2021-06-17 18:56:48 +01:00
Chris Castle
543e423fce
Fix broken link to README at root of repo (#10132)
Signed-off-by: Chris Castle chris@crc.io
2021-06-07 16:31:39 +01:00
Patrick Cloke
7d90d6ce9b
Run complement with Synapse workers manually. (#10039)
Adds an option to complement.sh to run Synapse in worker
mode (instead of the default monolith mode).
2021-05-24 15:32:45 -04:00
Sergio Miguéns
22a8838f62
Fix docker image to not log at /homeserver.log (#10045)
Fixes #9970

Signed-off-by: Sergio Miguéns Iglesias lonyelon@lony.xyz
2021-05-24 20:23:54 +01:00
Christopher May-Townsend
d5305000f1
Docker healthcheck timings - add startup delay and changed interval (#9913)
* Add healthcheck startup delay by 5secs and reduced interval check to 15s
to reduce waiting time for docker aware edge routers bringing an
instance online
2021-05-05 17:33:04 +02:00
Andrew Morgan
7e460ec2a5
Add a dockerfile for running a set of Synapse worker processes (#9162)
This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with:

* A Synapse main process.
* Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime.
* A redis for worker communication.
* A nginx for routing traffic.
* A supervisord to start all worker processes and monitor them if any go down.

Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite.

`configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI.

Huge thanks to @MatMaul for helping get this all working 🎉 This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62.

Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
2021-04-14 13:54:49 +01:00
Andrew Morgan
f16c6cf59a
Add note to docker docs explaining platform support (#9801)
Context is in https://github.com/matrix-org/synapse/issues/9764#issuecomment-818615894.

I struggled to find a more official link for this. The problem occurs when using WSL1 instead of WSL2, which some Windows platforms (at least Server 2019) still don't have. Docker have updated their documentation to paint a much happier picture now given WSL2's support.

The last sentence here can probably be removed once WSL1 is no longer around... though that will likely not be for a very long time.
2021-04-14 12:06:19 +01:00
Johannes Wienke
cb657eb2f8
Put opencontainers labels to the final image (#9765)
They don't make any sense on the intermediate builder image. The final
images needs them to be of use for anyone.

Signed-off-by: Johannes Wienke <languitar@semipol.de>
2021-04-08 13:49:14 +01:00
Richard van der Hoff
f02663c4dd
Replace room_invite_state_types with room_prejoin_state (#9700)
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
2021-03-30 12:12:44 +01:00
Eric Eastwood
0a778c135f
Make pip install faster in Docker build for Complement testing (#9610)
Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing.

If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘

As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`.

Fix https://github.com/matrix-org/synapse/issues/9364
2021-03-26 18:42:58 +00:00
Erik Johnston
019010964d Merge branch 'master' into develop 2021-03-26 12:26:58 +00:00
Erik Johnston
12d6184713
Explicitly upgrade openssl in docker file and enforce new version of cryptography (#9697) 2021-03-26 12:00:25 +00:00
Quentin Gliech
d4c4798a25
Use interpreter from $PATH instead of absolute paths in various scripts using /usr/bin/env (#9689)
On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work.

`/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters.

Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-03-25 16:53:54 +00:00
Johannes Wienke
4612302399
Include opencontainers labels in Docker image (#9612)
Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys

Signed-off-by: Johannes Wienke <languitar@semipol.de>
2021-03-22 15:31:00 +00:00
Mathieu Velten
ccf1dc51d7
Install jemalloc in docker image (#8553)
Co-authored-by: Will Hunt <willh@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-03-16 11:32:18 +00:00
Richard van der Hoff
1b2d6d55c5
Remove vestiges of uploads_path config (#9462)
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
2021-02-22 19:54:49 +00:00
Dan Callahan
b8b172466f
Add rustc to Docker image build environment (#9405)
This is needed to build the cryptography library, since it does not
provide wheels for ARMv7.

Fixes #9403

Signed-off-by: Dan Callahan <danc@element.io>
2021-02-15 15:30:16 +00:00
Dan Callahan
e19396d622
Fix Debian builds on Xenial (#9254)
Adds note about updating dh-virtualenv once we drop support for Xenial.

We can't update now, because it needs debhelper 12, while Xenial only
backports 10.

Signed-off-by: Dan Callahan <danc@element.io>
2021-01-29 14:56:04 +00:00
Andrew Morgan
94549771f7
Replace 'perspectives' config block with 'trusted_key_servers' in docker homeserver.yaml template (#9157) 2021-01-19 16:19:32 +00:00
Dan Callahan
fa6deb298b
Fix failures in Debian packaging (#9079)
Debian package builds were failing for two reasons:

 1. Python versions prior to 3.7 throw exceptions when attempting to print
    Unicode characters under a "C" locale. (#9076)

 2. We depended on `dh-systemd` which no longer exists in Debian Bullseye, but
    is necessary in Ubuntu Xenial. (#9073)

Setting `LANG="C.UTF-8"` in the build environment fixes the first issue.
See also: https://bugs.python.org/issue19846

The second issue is a bit trickier. The dh-systemd package was merged into
debhelper version 9.20160709 and a transitional package left in its wake.

The transitional dh-systemd package was removed in Debian Bullseye.

However, Ubuntu Xenial ships an older debhelper, and still needs dh-systemd.

Thus, builds were failing on Bullseye since we depended on a package which had
ceased existing, but we couldn't remove it from the debian/control file and our
build scripts because we still needed it for Ubuntu Xenial.

We can fix the debian/control issue by listing dh-systemd as an alternative to
the newer versions of debhelper. Since dh-systemd declares that it depends on
debhelper, Ubuntu Xenial will select its older dh-systemd which will in turn
pull in its older debhelper, resulting in no change from the status quo. All
other supported releases will satisfy the debhelper dependency constraint and
skip the dh-systemd alternative.

Build scripts were fixed by unconditionally attempting to install dh-systemd on
all releases and suppressing failures.

Once we drop support for Ubuntu Xenial, we can revert most of this commit and
rely on the version constraint on debhelper in debian/control.

Fixes #9076
Fixes #9073

Signed-off-by: Dan Callahan <danc@element.io>
2021-01-12 14:15:04 +00:00
Erik Johnston
57068eae75 Add 'xmlsec1' to dependency list 2020-12-09 13:48:16 +00:00
Jordan Bancino
295c209cdd
Remove version pin prometheus_client dependency (#8875)
This removes the version pin of the `prometheus_client` dependency, in direct response to #8831. If merged, this will close #8831 

As far as I can tell, no other changes are needed, but as I'm no synapse expert, I'm relying heavily on CI and maintainer reviews for this. My very primitive test of synapse with prometheus_client v0.9.0 on my home server didn't bring up any issues, so we'll see what happens.

Signed-off-by: Jordan Bancino
2020-12-04 13:01:06 +00:00
Andrew Morgan
c087f68053
Cap the version of prometheus_client to <v0.9.0 in the dockerfile (#8767)
Short-term fix for https://github.com/matrix-org/synapse/issues/8766.
2020-11-17 16:01:33 +00:00
Dan Callahan
ca39e67f3d
Use Python 3.8 in Docker images by default (#8698)
This bumps us closer to current Python without going all the way to 3.9.

Fixes #8674

Signed-off-by: Dan Callahan <danc@element.io>
2020-11-02 16:33:06 +00:00
Samuel Philipp
6905f5751a
Docker: support passing additional commandline args to synapse (#8390) 2020-10-11 20:51:11 +01:00
Mateusz Przybyłowicz
ca2db5dd0c
Increase default max_upload_size from 10M to 50M (#8502)
Signed-off-by: Mateusz Przybyłowicz <uamfhq@gmail.com>
2020-10-09 16:58:23 +01:00
Christopher May-Townsend
ed18f32e1b
Add required Debian dependencies to allow docker builds on the arm platform (#8144)
Signed-off-by: Christopher May-Townsend <chris@maytownsend.co.uk>
2020-08-26 15:03:20 +01:00
Christopher May-Townsend
64e8a4697a
Add healthcheck for default localhost 8008 port on /health endpoint. (#8147) 2020-08-24 18:15:18 +01:00
Erik Johnston
0304ad0c3d Move setting of Filter into code.
We do this to prevent foot guns. The default config uses a MemoryFilter,
but users are free to change to logging to files directly. If they do
then they have to ensure to set the `filters: [context]` on the right
handler, otherwise records get written with the wrong context.

Instead we move the logic to happen when we generate a record, which is
when we *log* rather than *handle*.

(It's possible to add filters to loggers in the config, however they
don't apply to descendant loggers and so they have to be manually set on
*every* logger used in the code base)
2020-08-11 18:10:46 +01:00
Michael Kaye
1ec2961b3b
Add help for creating a user via docker (#7885) 2020-07-17 13:25:48 -04:00
Christopher May-Townsend
a5545cf86d
Switch to Debian:Slim from Alpine for the docker image (#7839)
As mentioned in #7397, switching to a debian base should help with multi-arch work to save time on compiling. This is unashamedly based on #6373, but without the extra functionality. Switch python version back to generic 3.7 to always pull the latest. Essentially, keeping this as small as possible. The image is bigger though unfortunately.
2020-07-17 17:40:53 +01:00
Juho Vanhanen
d378c3da78
Add libwebp dependency to Dockerfile (#7791)
* Add libwebp dependency to Dockerfile

Signed-off-by: Juho Vanhanen <juho@vanhanen.io>
2020-07-06 13:37:39 +01:00
Richard van der Hoff
2a8ed93bd4
Switch back to upstream dh-virtualenv (#7621)
Upstream have merged our changes
(https://github.com/spotify/dh-virtualenv/pull/300), so let's switch back to it
instead of using our fork.
2020-06-03 12:21:58 +01:00
Richard van der Hoff
8c75da916c
Refresh apt cache when building dh_virtualenv docker image (#7555)
When we tried to build debs for 1.13.0, the build failed because docker used a
base docker image which had a stale apt cache.

Fixes: #7540
2020-05-22 10:17:47 +01:00
Patrick Cloke
02919bf4d8 Merge branch 'master' into develop 2020-05-19 09:56:15 -04:00
Richard van der Hoff
1fc8914f76
update dh-virtualenv (#7526) 2020-05-19 13:48:41 +01:00
Cédric Laubacher
a251e0f4ba
Update runtime docker image to Alpine v3.11 2020-05-03 16:07:24 +02:00
Richard van der Hoff
3be2abd0a9
Kill off deprecated "config-on-the-fly" docker mode (#6918)
Lots of people seem to get confused by this mode, and it's been deprecated
since Synapse 1.1.0. It's time for it to go.
2020-02-18 11:41:53 +00:00
Richard van der Hoff
3dd2b5f5e3
bump the version of Alpine Linux used in the docker images (#6897) 2020-02-12 12:02:53 +00:00
Andrew Morgan
265c0bd2fe Add working build command for docker image (#6390)
* Add working build command for docker image

* Add changelog
2019-11-23 06:54:05 +11:00
Andrew Morgan
234f55f3c4
Docker: Change permissions for data dir before attempting to write to it (#6389) 2019-11-20 13:32:31 +00:00
Richard van der Hoff
eb9a0d9e48 Merge remote-tracking branch 'origin/master' into develop 2019-10-31 11:17:05 +00:00
Anton Lazarev
213d7eb227 Clarify environment variable usage when running in Docker (#6181) 2019-10-30 07:30:04 +00:00
Tobia De Koninck
29207b4488 Fix broken URL in docker/README.md (#6264)
Signed-off-by: Tobia De Koninck <LEDfan@users.noreply.github.com>
2019-10-28 15:39:57 +00:00
Richard van der Hoff
560c122267
Fix logging config for the docker image (#6197)
Turns out that loggers that are instantiated before the config is loaded get
turned off.

Also bring the logging config that is generated by --generate-config into line.

Fixes #6194.
2019-10-18 13:34:33 +02:00
Michael Kaye
2def5ea0da Docker: support SYNAPSE_WORKER envvar (#6058)
* Allow passing SYNAPSE_WORKER envvar

* changelog.d

* Document SYNAPSE_WORKER.

Attempting to imply that you don't need to change this default
unless you're in worker mode.

Also aware that there's a bigger problem of attempting to document
a complete working configuration of workers using docker, as we
currently only document to use `synctl` for worker mode, and synctl
doesn't work that way in docker.
2019-09-19 22:29:47 +01:00
Michael Kaye
894c1a5759
Docker packaging should not su-exec or chmod if already running as UID/GID (#5970)
Adjust su-exec to only be used if needed.

If UID == getuid() and GID == getgid() then we do not need to su-exec, and chmod will not work.
2019-09-03 16:36:01 +01:00
Victor Goff
1196ee32b3 Typographical corrections in docker/README (#5921) 2019-08-28 09:34:49 +01:00
Erik Johnston
3b476f5767 Fix debian packages for sid being called buster. (#5775)
* Fix debian packages for sid being called buster.

I don't know why the sid images return buster as its codename in
`lsb_release` but it does, so lets just grab the codename from the
distro we pass into dockerfile

* Newsfile
2019-07-30 00:33:32 +10:00
Richard van der Hoff
1def298119
Improve Depends specs in debian package. (#5675)
This is basically a contrived way of adding a `Recommends` on `libpq5`, to fix #5653.

The way this is supposed to happen in debhelper is to run
`dh_shlibdeps`, which in turn runs `dpkg-shlibdeps`, which spits things out
into `debian/<package>.substvars` whence they can later be included by
`control`.

Previously, we had disabled `dh_shlibdeps`, mostly because `dpkg-shlibdeps`
gets confused about PIL's interdependent objects, but that's not really the
right thing to do and there is another way to work around that.

Since we don't always use postgres, we don't necessarily want a hard Depends on
libpq5, so I've actually ended up adding an explicit invocation of
`dpkg-shlibdeps` for `psycopg2`.

I've also updated the build-depends list for the package, which was missing a
couple of entries.
2019-07-17 17:47:07 +01:00
Slavi Pantaleev
59f15309ca Add missing space in default logging file format generated by the Docker image (#5620)
This adds a missing space, without which log lines appear uglier.

Signed-off-by: Slavi Pantaleev <slavi@devture.com>
2019-07-12 11:43:42 +01:00
Slavi Pantaleev
f369164761 Upgrade Alpine Linux used in the Docker image (3.8 -> 3.10) (#5619)
Alpine Linux 3.8 is still supported, but it seems like
it's quite outdated now.

While Python should be the same on both, all other libraries, etc.,
are much newer in Alpine 3.9 and 3.10.

Signed-off-by: Slavi Pantaleev <slavi@devture.com>
2019-07-12 11:38:25 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Andrew Morgan
c7b48bd42d Remove SMTP_* env var functionality from docker conf (#5596)
Removes any `SMTP_*` docker container environment variables from having any effect on the default config.

Fixes https://github.com/matrix-org/synapse/issues/5430
2019-07-03 07:14:48 +01:00
Amir Zarrinkafsh
de8077a164 Add ability to set timezone for Docker container (#5383)
Signed-off-by: Amir Zarrinkafsh <nightah@me.com>
2019-07-02 10:31:06 +01:00
Richard van der Hoff
555b6fa0d5
Docker image: Add a migrate_config mode (#5567)
... to help people escape env var hell
2019-06-27 13:52:40 +01:00
Richard van der Hoff
1ddc7b39c9
Docker image: open the non-TLS port by default. (#5568)
There's not much point in binding to localhost when it's in a docker container.
2019-06-27 13:50:10 +01:00
Richard van der Hoff
2f7ebc2a55
Deprecate the env var way of running the docker image (#5566)
This is mostly a documentation change, but also adds a default value for
SYNAPSE_CONFIG_PATH, so that running from the generated config is the default,
and will Just Work provided your config is in the right place.
2019-06-27 13:49:48 +01:00
Richard van der Hoff
b4db70e167
Merge pull request #5565 from matrix-org/rav/docker/fix_log_config
Docker: generate our own log config
2019-06-27 11:19:37 +01:00
Richard van der Hoff
b1b8a24b63
Merge pull request #5563 from matrix-org/rav/docker/data_dir
Docker image: add support for SYNAPSE_DATA_DIR parameter
2019-06-27 11:17:44 +01:00
Richard van der Hoff
28e30c6581 Docker: generate our own log config
When running under docker, we want to use docker's own logging stuff rather
than losing the logs somewhere on the container's filesystem, so let's use log
configs that spit logs out to stdout instead.
2019-06-26 15:48:38 +01:00
Richard van der Hoff
6347dc1bed Add support for SYNAPSE_CONFIG_DIR 2019-06-26 15:48:38 +01:00
Richard van der Hoff
7e433beb65 Docker image: add support for SYNAPSE_DATA_DIR parameter
Fixes #4830.
2019-06-26 15:38:08 +01:00
Richard van der Hoff
c58a6e6108 document supported env vars for docker 'generate' option 2019-06-26 15:38:08 +01:00
Richard van der Hoff
a5fba9c27c Docker: only run --generate-keys when generating config on-the-fly.
We don't want to generate any missing configs when running from a precanned
config.

(There's a strong argument that we don't want to do this at all, since
generating a new signing key on each invocation sounds disasterous, but I don't
fancy unpicking that for now.)
2019-06-26 15:31:19 +01:00
Richard van der Hoff
a1732bbff9 improve logging for generate_config_from_template 2019-06-26 15:31:19 +01:00
Richard van der Hoff
5375c3a9b8 isort 2019-06-25 15:30:19 +01:00
Richard van der Hoff
3f24e4dce7 Add a main() function 2019-06-25 15:30:19 +01:00
Richard van der Hoff
b1fddb7f69 Factor out a run_generate_config function 2019-06-25 15:30:19 +01:00
Richard van der Hoff
a52e1a3b6c Factor out "generate_config_from_template"
... and inline generate_secrets
2019-06-25 15:30:19 +01:00
Richard van der Hoff
fe2d876e2a Increase default log level for docker image to INFO. (#5547)
Fixes #3370.
2019-06-25 14:38:38 +10:00
Richard van der Hoff
f817fc9ad5 Update docker image to use Python 3.7. (#5546)
Python 3.7 is apparently faster than 3.6, and should be mature enough.
2019-06-25 14:20:53 +10:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Richard van der Hoff
8e7ef3a023 Include xmlsec in the docker image
Fixes #5467.
2019-06-18 22:35:19 +01:00
Amber Brown
eba7caf09f
Remove Postgres 9.4 support (#5448) 2019-06-18 00:59:00 +10:00
Neil Johnson
4f68188d0b
Change to absolute path for contrib/docker
because this file is reproduced on dockerhub and relative paths don't work
2019-06-13 16:42:36 +01:00
Steffen
2dfbeea66f Update README.md (#5222)
Add missing backslash
2019-05-22 12:53:16 +01:00
Richard van der Hoff
836d3adcce Merge branch 'master' into develop 2019-05-03 19:25:01 +01:00
Richard van der Hoff
ecc0967315 Debian: we now need libpq-dev.
psycopg 2.8 is now out, which means that the C library gets built from source,
so we now need libpq-dev when building.

Turns out the need for this package is already documented in
docs/postgres.rst.
2019-05-03 18:34:34 +01:00
*=0=1=4=*
4a9a118a94 Fix handling of SYNAPSE_NO_TLS in docker image (#5005) 2019-04-25 14:47:22 +01:00
Silke Hofstra
a137f4eac0 Add systemd-python to optional dependencies (#4339)
Using systemd-python allows for logging to the systemd journal,
as is documented in: `synapse/contrib/systemd/log_config.yaml`.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
2019-04-16 20:41:17 +10:00
*=0=1=4=*
40810b81d2 Correct default POSTGRES_USER in Docker README (#4987)
Correct default POSTGRES_USER in Docker README
2019-04-04 10:38:16 +01:00
Jurrie Overgoor
e0f219789d Add -p argument for docker run command example
Signed-off-by: Jurrie Overgoor <1213142+Jurrie@users.noreply.github.com>
2019-03-29 10:25:41 +01:00
Jurrie Overgoor
aee4ea8ba8 Fix typo in TLS filenames
Signed-off-by: Jurrie Overgoor <1213142+Jurrie@users.noreply.github.com>
2019-03-29 10:25:41 +01:00
Richard van der Hoff
5580616235 Document the generate option for the docker image. 2019-03-07 07:35:42 +00:00
Richard van der Hoff
649fe1c2be Fix debian build dockerfile
Make sure it refreshes the apt cache before trying to install stuff
2019-02-14 17:29:40 +00:00
Richard van der Hoff
91f8cd3307 Remove redundant entries from docker config
* no_tls is now redundant (#4613)
* we don't need a dummy cert any more (#4618)
2019-02-11 22:16:44 +00:00
Richard van der Hoff
40b35fb875
Enable ACME support in the docker image (#4566)
Also:

* Fix wrapping in docker readme
* Clean up some docs on the docker image
* a workaround for #4554
2019-02-05 13:42:21 +00:00
Richard van der Hoff
3ef71a6ea0
Docker: only copy what we need to the build image (#4562)
There are two reasons this is a good thing:

 * first, it means that you don't end up with stuff kicking around your working
   copy ending up in the build image by mistake (which can upset the pip
   install process)

 * second: it means that the docker image cache is more effective, and we can
   reuse docker images when iterating on the docker stuff.
2019-02-05 11:44:40 +00:00
Amber Brown
23b0813599
Require ECDH key exchange & remove dh_params (#4429)
* remove dh_params and set better cipher string
2019-01-22 21:58:50 +11:00
Richard van der Hoff
b5b868d41e Rewrite build_debian_packages
Rewrite this in python so that it can be run in parallel.
2019-01-12 12:40:58 +00:00
Richard van der Hoff
95fca1c7e9 fix docker build to install optional deps 2019-01-09 16:37:51 +00:00
Richard van der Hoff
7134832c01
Install the optional dependencies into the debian package (#4325)
since #4298, the optional dependencies are no longer installed with a simple
`pip install .`, which meant that they were not being included in the debian
package.

The easy fix to that is dh_virtualenv --extras, but that needs dh_virtualenv
1.1...
2019-01-02 07:17:39 +00:00
Richard van der Hoff
bdc44b99a2 buster is a thing 2018-12-20 16:11:21 +00:00
Richard van der Hoff
1a6d5bfa08 Debian packaging via dh_virtualenv (#4285) 2018-12-20 11:33:29 +00:00
Richard van der Hoff
75937e9033 Remove obsolete settings from docker homeserver.yaml
These aren't used, because we have a `log_config` setting.
2018-12-04 12:31:00 +00:00
Neil Johnson
78ba0e7ab8 Remove riot.im from the list of trusted Identity Servers in the default configuration (#4207) 2018-11-20 12:29:25 +01:00
Cédric Laudrel
379376e5e6 Make Docker image listening on ipv6 as well as ipv4
Signed-off-by: Cédric Laudrel <dek@iono.me>
2018-10-25 20:03:47 +02:00