Commit Graph

155 Commits

Author SHA1 Message Date
Richard van der Hoff
34e84fee68
Tweaks to workers-under-complement ()
* Bump the HS startup timeout
* Log prefixes for more processes
* Bump the overall timeout
2022-05-09 22:41:06 +01:00
reivilibre
c2d50e9f6c
Add the notify_appservices_from_worker configuration option (superseding notify_appservices) to allow a generic worker to be designated as the worker to send traffic to Application Services. () 2022-05-06 11:43:53 +01:00
reivilibre
d743b25c8f
Use supervisord to supervise Postgres and Caddy in the Complement image. ()
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-27 14:39:41 +01:00
Jason Robinson
706456de1f
Mark Dockerfile as requiring BuildKit ()
Co-authored-by: David Robertson <davidr@element.io>
2022-04-26 15:31:52 +01:00
David Robertson
8bac3e0435
disallow-untyped-defs in docker and stubs directories () 2022-04-25 12:32:35 +00:00
Richard van der Hoff
f5668f0b4a
Await un-partial-stating after a partial-state join ()
When we join a room via the faster-joins mechanism, we end up with "partial
state" at some points on the event DAG. Many parts of the codebase need to
wait for the full state to load. So, we implement a mechanism to keep track of
which events have partial state, and wait for them to be fully-populated.
2022-04-21 07:42:03 +01:00
David Robertson
e5a76ec00b
Dump setuptools; correct pyproject version number () 2022-04-20 17:33:20 +01:00
Richard van der Hoff
aaaff98202
Dockerfile-workers: reduce the amount we install ()
This is an attempt to reduce the rebuild time. In short, we reduce the amount
of stuff that the dockerfile installs, so as to give a faster startup.
2022-04-14 15:36:49 +01:00
Richard van der Hoff
960b4fb409
complement-synapse-workers: factor out separate entry point script ()
... with a bit more verbosity.
2022-04-14 14:56:10 +01:00
Richard van der Hoff
8af8a9bce5
Dockerfile-workers: give the master its own log config ()
When we run a worker-mode synapse under docker, everything gets logged to stdout. Currently, output from the workers is tacked with a worker name, for example:

```
2022-04-13 15:27:56,810 - worker:frontend_proxy1 - synapse.util.caches.lrucache - 154 - INFO - LruCache._expire_old_entries-0 - Dropped 0 items from caches
```

- note `worker:frontend_proxy1`. No such tag is applied to log lines from the master, which makes somewhat confusing reading.

To fix this, we generate a dedicated log config file for the master in the same way that we do for the workers, and use that.
2022-04-13 20:50:08 +01:00
David Robertson
5f8173dd80
Workaround pip bug installing latest treq and not-latest twisted from hashes ()
The requirements file generated by `poetry export` isn't correctly processed by `pip install -r requirements.txt`. It contains twisted and treq, both pinned to 22.2.0.

When `pip` installs treq, it notices that `Twisted[tls]` is required. It then tries to acquire the latest twisted release, only to fail (because this hash isn't listed in the requirements file).From e.g. https://github.com/matrix-org/synapse/runs/5977154990?check_suite_focus=true

> ```
>  9.204 Collecting Twisted[tls]>=18.7.0
>  9.205 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
>  9.205     Twisted[tls]>=18.7.0 from 38622ff95b/Twisted-22.4.0-py3-none-any.whl (sha256)=f9f7a91f94932477a9fc3b169d57f54f96c6e74a23d78d9ce54039a7f48928a2 (from treq==22.2.0->-r /synapse/requirements.txt (line 724))
>  ERROR: executor failed running [/bin/sh -c pip install --prefix="/install" --no-warn-script-location -r /synapse/requirements.txt]: exit code: 1
> ```

The underlying pip issue is https://github.com/pypa/pip/issues/9644. A comment notes that one can avoid this behaviour with by `pip install`ing with the `--no-deps` flag. Let us do so.

(At first glance, the problem looks like https://github.com/python-poetry/poetry/issues/5311, but that was a bug in `poetry install`; this is `poetry export`, whose behaviour is fine AFAICS).
2022-04-12 10:16:01 +01:00
Richard van der Hoff
214f3b7d21
Enable certificate checking during complement tests () 2022-04-11 16:35:41 +01:00
Richard van der Hoff
3cdf5a1386
Fix up healthcheck generation for workers docker image ()
This wasn't quite generating the right thing.
2022-04-11 13:38:58 +00:00
Richard van der Hoff
5f72ea1bde
Move complement setup stuff into the Synapse repo ()
Fixes  (or it will, once we remove the old files).

It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
2022-04-11 11:39:28 +01:00
Jorge Florian
78e4d96a4d
Add missing type definitions for scripts in docker folder ()
Signed-off-by: Jorge Florian <jafn28@gmail.com>
2022-04-08 10:10:58 +00:00
David Robertson
3a7e97c7ad
Poetry: use locked environment in Docker images () 2022-04-07 11:43:31 +00:00
Richard van der Hoff
34a8370d7b
README-testing.md: fix minor error 2022-03-31 13:08:03 +01:00
David Robertson
15cdcf8f30
Remove dockerfile-pgtests () 2022-03-31 11:30:07 +01:00
David Robertson
4aeb00ca20
Move synctl into synapse._scripts and expose as an entrypoint () 2022-03-04 11:58:49 +00:00
David Robertson
f3f0ab10fe
Move scripts directory inside synapse, exposing as setuptools entry_points ()
* Two scripts are basically entry_points already
* Move and rename scripts/* to synapse/_scripts/*.py
* Delete sync_room_to_group.pl
* Expose entry points in setup.py
* Update linter script and config
* Fixup scripts & docs mentioning scripts that moved

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2022-03-02 13:00:16 +00:00
Andrew Morgan
f26e390a40
Use Python 3.9 in Synapse dockerfiles by default () 2022-03-01 13:55:18 +00:00
Richard van der Hoff
5598556b77
Docker: remove VOLUME directive ()
The driver for this is to stop Complement complaining about it, but as far as I can tell it was pointless and needed to go away anyway.

I'm a bit unclear about what exactly VOLUME does, but I think what it means is that, if you don't override it with an explicit -v argument, then docker run will create a temporary volume, and copy things into it. The temporary volume is then deleted when the container finishes.

That only sounds useful if your image has something to copy into it (otherwise you may as well just use the default root filesystem), and our image notably doesn't copy anything into /data.

So... this wasn't doing anything, except annoying Complement?
2022-02-15 13:59:15 +00:00
Shay
9006ee36d1
Drop support for and remove references to EOL Python 3.6 ()
* remove reference in comments to python3.6

* upgrade tox python env in script

* bump python version in example for completeness

* upgrade python version requirement in setup doc

* upgrade necessary python version in __init__.py

* upgrade python version in setup.py

* newsfragment

* drops refs to bionic and replace with focal

* bump refs to postgres 9.6 to 10

* fix hanging ci

* try installing tzdata first

* revert change made in b979f336

* ignore new random mypy error while debugging other error

* fix lint error for temporary workaround

* revert change to install list

* try passing env var

* export debian frontend var?

* move line and add comment

* bump pillow dependency

* bump lxml depenency

* install libjpeg-dev for pillow

* bump automat version to one compatible with py3.8

* add libwebp for pillow

* bump twisted trunk python version

* change suffix of newsfragment

* remove redundant python 3.7 checks

* lint
2022-01-21 14:23:26 -08:00
Richard van der Hoff
d41c4654db
Use buildkit's cache feature to speed up docker builds ()
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
2022-01-12 10:37:57 +00:00
Callum Macdonald
7013e06e2f
Improve Docker docs for use with Postgres () 2022-01-05 10:50:28 +00:00
Shay
13c974ed35
Drop Bionic from Debian builds ()
* update Trove classifiers to remove py36

* stop building bionic

* update dh-virtualenv

* newsfragment

* fix newsfragment

* update version refs

* another try at correct tag

* Update changelog
2022-01-03 11:17:16 -08:00
Michael Kaye
e2c300e7e4
Create healthcheck script for synapse-workers container ()
The intent is to iterate through all the worker ports and only
report healthy when all are healthy, starting with the main process.
2021-11-26 14:05:20 +00:00
Shay
7cebaf9644
Remove code invalidated by deprecated config flag 'trust_identity_servers_for_password_resets' ()
* remove background update code related to deprecated config flag

* changelog entry

* update changelog

* Delete 11394.removal

Duplicate, wrong number

* add no-op background update and change newfragment so it will be consolidated with associated work

* remove unused code

* Remove code associated with deprecated flag from legacy docker dynamic config file

Co-authored-by: reivilibre <oliverw@matrix.org>
2021-11-23 06:46:40 -08:00
Aaron R
d993c3bb1e
Add support for /_matrix/media/v3 APIs ()
* Add support for `/_matrix/media/v3` APIs

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Update `workers.md` to use v3 client and media APIs

Signed-off-by: Aaron Raimist <aaron@raim.ist>

* Add changelog

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-11-17 15:30:24 +00:00
Richard van der Hoff
0b99d4c8d2
Docker: avoid changing userid unnecessarily ()
* Docker image: avoid changing user during `generate`

The intention was always that the config files get written as the initial user
(normally root) - only the data directory needs to be writable by Synapse. This
got changed in https://github.com/matrix-org/synapse/pull/5970, but that seems
to have been a mistake.

* Avoid changing user if no explicit UID is given

* changelog
2021-11-01 13:55:30 +00:00
Dan Callahan
d7141e0b8b
Fix Shellcheck SC2006: Use $(...) notation
Use $(...) notation instead of legacy backticked `...`.

https://github.com/koalaman/shellcheck/wiki/SC2006

Signed-off-by: Dan Callahan <danc@element.io>
2021-10-22 23:08:55 +01:00
Dan Callahan
13f084eb58
Fix Shellcheck SC2086: Quote to prevent splitting
Double quote to prevent globbing and word splitting.

https://github.com/koalaman/shellcheck/wiki/SC2086

Signed-off-by: Dan Callahan <danc@element.io>
2021-10-22 23:08:54 +01:00
Dan Callahan
6c736fa472
Fix Shellcheck SC2154: variable possibly undefined
var is referenced but not assigned.

https://github.com/koalaman/shellcheck/wiki/SC2154

Signed-off-by: Dan Callahan <danc@element.io>
2021-10-22 23:08:54 +01:00
Richard van der Hoff
09eff1b3db
fix relative link in docker readme ()
relative links don't work when it's on dockerhub.
2021-10-21 13:50:43 +01:00
Erik Johnston
a8bbf08576
Fix debian package builds. ()
This was due to dh-virtualenv builds being broken due to Shpinx removing
deprecated APIs.
2021-09-28 12:13:51 +01:00
David Robertson
ea01d4c2de
Update postgresql testing script ()
- Use sytest:bionic. Sytest:latest is two years old (do we want
  CI to push out latest at all?) and comes with Python 3.5, which we
  explictly no longer support. The script now runs under PostgreSQL 10
  as a result.
- Advertise script in the docs
- Move pg testing script to scripts-dev directory
- Write to host as the script's exector, not root

A few changes to make it speedier to re-run the tests:

- Create blank DB in the container, not the script, so we don't have to
  `initdb` each time
- Use a named volume to persist the tox environment, so we don't have to
  fetch and install a bunch of packages from PyPI each time

Co-authored-by: reivilibre <olivier@librepush.net>
2021-09-24 14:27:09 +00:00
kegsay
01df612e1e
Fix frontend_proxy jinja script in docker workers () 2021-09-08 17:24:53 +01:00
reivilibre
5f7b1e1f27
Make PeriodicallyFlushingMemoryHandler the default logging handler. () 2021-08-17 13:13:11 +01:00
Richard van der Hoff
2afdb5c984
Fix deb build script to set prerelease flag correctly () 2021-08-01 10:47:36 +01:00
Richard van der Hoff
8ae0bdca75
Drop xenial-support hacks () 2021-07-21 21:25:28 +01:00
Richard van der Hoff
c82eb02d64
Set section for prerelease debs ()
This is part of fixing : we want to put RC debs into a different place than release debs, so reprepro has to be able to tell them apart.
2021-07-14 14:41:40 +01:00
Dirk Klimpel
974261cd81
Fix broken links in INSTALL.md ()
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-08 15:46:13 +01:00
Brendan Abolivier
08c8469322
Remove support for ACME v1 ()
Fixes 

ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
2021-06-17 18:56:48 +01:00
Chris Castle
543e423fce
Fix broken link to README at root of repo ()
Signed-off-by: Chris Castle chris@crc.io
2021-06-07 16:31:39 +01:00
Patrick Cloke
7d90d6ce9b
Run complement with Synapse workers manually. ()
Adds an option to complement.sh to run Synapse in worker
mode (instead of the default monolith mode).
2021-05-24 15:32:45 -04:00
Sergio Miguéns
22a8838f62
Fix docker image to not log at /homeserver.log ()
Fixes 

Signed-off-by: Sergio Miguéns Iglesias lonyelon@lony.xyz
2021-05-24 20:23:54 +01:00
Christopher May-Townsend
d5305000f1
Docker healthcheck timings - add startup delay and changed interval ()
* Add healthcheck startup delay by 5secs and reduced interval check to 15s
to reduce waiting time for docker aware edge routers bringing an
instance online
2021-05-05 17:33:04 +02:00
Andrew Morgan
7e460ec2a5
Add a dockerfile for running a set of Synapse worker processes ()
This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with:

* A Synapse main process.
* Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime.
* A redis for worker communication.
* A nginx for routing traffic.
* A supervisord to start all worker processes and monitor them if any go down.

Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite.

`configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI.

Huge thanks to @MatMaul for helping get this all working 🎉 This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62.

Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
2021-04-14 13:54:49 +01:00
Andrew Morgan
f16c6cf59a
Add note to docker docs explaining platform support ()
Context is in https://github.com/matrix-org/synapse/issues/9764#issuecomment-818615894.

I struggled to find a more official link for this. The problem occurs when using WSL1 instead of WSL2, which some Windows platforms (at least Server 2019) still don't have. Docker have updated their documentation to paint a much happier picture now given WSL2's support.

The last sentence here can probably be removed once WSL1 is no longer around... though that will likely not be for a very long time.
2021-04-14 12:06:19 +01:00
Johannes Wienke
cb657eb2f8
Put opencontainers labels to the final image ()
They don't make any sense on the intermediate builder image. The final
images needs them to be of use for anyone.

Signed-off-by: Johannes Wienke <languitar@semipol.de>
2021-04-08 13:49:14 +01:00