Commit Graph

916 Commits

Author SHA1 Message Date
realtyem
58214dbb9b
Allow enabling the asyncio reactor in complement ()
Signed-off-by: Jason Little realtyem@gmail.com
2023-02-01 23:42:45 +00:00
Dirk Klimpel
bf82b56bab
Add more user information to export-data command. ()
* The user's profile information.
* The user's devices.
* The user's connections / IP address information.
2023-02-01 15:45:19 +00:00
reivilibre
22cc93afe3
Enable Faster Remote Room Joins against worker-mode Synapse. ()
* Enable Complement tests for Faster Remote Room Joins on worker-mode

* (dangerous) Add an override to allow Complement to use FRRJ under workers

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Fix race where we didn't send out replication notification

* MORE HACKS

* Fix get_un_partial_stated_rooms_token to take instance_name

* Fix bad merge

* Remove warning

* Correctly advance un_partial_stated_room_stream

* Fix merge

* Add another notify_replication

* Fixups

* Create a separate ReplicationNotifier

* Fix test

* Fix portdb

* Create a separate ReplicationNotifier

* Fix test

* Fix portdb

* Fix presence test

* Newsfile

* Apply suggestions from code review

* Update changelog.d/14752.misc

Co-authored-by: Erik Johnston <erik@matrix.org>

* lint

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>
2023-01-22 21:10:11 +00:00
Dirk Klimpel
73f097888e
Add listener health ()
Fixes: 
2023-01-11 12:00:38 +00:00
Patrick Cloke
f685318c2a
Use ClientRestResource on both the main process and workers. ()
Add logic to ClientRestResource to decide whether to mount servlets
or not based on whether the current process is a worker.

This is clearer to see what a worker runs than the completely separate /
copy & pasted list of servlets being mounted for workers.
2022-12-02 13:10:05 -05:00
reivilibre
9af2be192a
Remove legacy Prometheus metrics names. They were deprecated in Synapse v1.69.0 and disabled by default in Synapse v1.71.0. () 2022-11-24 09:09:17 +00:00
realtyem
c15e9a0edb
Remove need for worker_main_http_uri setting to use /keys/upload. () 2022-11-16 22:16:25 +00:00
David Robertson
258b5285b6
Fix typechecking errors introduced in ()
* Fix typechecking errors introduced in 

* Changelog

* Correct annotations

so that context_factory works if you don't use TLS
2022-11-15 16:36:43 +00:00
Tuomas Ojamies
b5ab2c428a
Support using SSL on worker endpoints. ()
* Fix missing SSL support in worker endpoints.

* Add changelog

* SSL for Replication endpoint

* Remove unit test change

* Refactor listener creation to reduce duplicated code

* Fix the logger message

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Update synapse/app/_base.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Add config documentation for new TLS option

Co-authored-by: Tuomas Ojamies <tojamies@palantir.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-11-15 12:55:00 +00:00
Nick Mills-Barrett
3a4f80f8c6
Merge/remove Slaved* stores into WorkerStores () 2022-11-11 10:51:49 +00:00
reivilibre
b922b54b61
Fix type annotation causing import time error in the Complement forking launcher. ()
Co-authored-by: David Robertson <davidr@element.io>
2022-11-01 10:30:43 +00:00
David Robertson
730b13dbc9
Improve RawHeaders type hints () 2022-10-28 16:04:02 +00:00
Richard van der Hoff
d24346f530
Fix logging error on SIGHUP () 2022-10-21 16:03:44 +01:00
Patrick Cloke
755bfeee3a
Use servlets for /key/ endpoints. ()
To fix the response for unknown endpoints under that prefix.

See MSC3743.
2022-10-20 11:32:47 -04:00
Patrick Cloke
c604d2c218
Mark /relations endpoint as usable on workers. ()
Co-authored-by: Eric Eastwood <erice@element.io>
2022-10-12 06:46:13 -04:00
David Robertson
285d72556b
Update mypy and mypy-zope, attempt 3 ()
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-30 17:36:28 +01:00
David Robertson
8e52cb0bce
Revert "Update mypy and mypy-zope ()"
This reverts commit 6d543d6d9f.
2022-09-30 16:37:48 +01:00
David Robertson
6d543d6d9f
Update mypy and mypy-zope ()
* Update mypy and mypy-zope

* Unignore assigning to LogRecord attributes

Presumably https://github.com/python/typeshed/pull/8064 makes this ok

Cherry-picked from 

* Remove unused ignores due to mypy ParamSpec fixes

https://github.com/python/mypy/pull/12668

Cherry-picked from 

* Remove additional unused ignores

* Fix new mypy complaints related to `assertGreater`

Presumably due to https://github.com/python/typeshed/pull/8077

* Changelog

* Reword changelog

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-09-30 16:34:47 +01:00
Erik Johnston
e8318a4333
Handle the case of remote users leaving a partial join room for device lists () 2022-09-27 13:01:08 +01:00
Richard van der Hoff
d6b85a2a7d
Complement image: propagate SIGTERM to all workers ()
This should mean that logs from worker processes are flushed before shutdown.

When a test completes, Complement stops the docker container, which means that
synapse will receive a SIGTERM. Currently, the `complement_fork_starter` exits
immediately (without notifying the worker processes), which means that the
workers never get a chance to flush their logs before the whole container is
vaped. We can fix this by propagating the SIGTERM to the children.
2022-09-26 23:07:02 +01:00
reivilibre
526f84bc2e
Fix Prometheus recording rules to not use legacy metric names. () 2022-09-08 15:01:42 +01:00
Patrick Cloke
32fc3b7ba4
Remove configuration options for direct TCP replication. ()
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
2022-09-06 07:50:02 +00:00
Brendan Abolivier
898fef2789
Share some metrics between the Prometheus exporter and the phone home stats () 2022-09-05 10:26:43 +00:00
reivilibre
be4250c7a8
Add experimental configuration option to allow disabling legacy Prometheus metric names. ()
Co-authored-by: David Robertson <davidr@element.io>
2022-08-24 11:35:54 +00:00
David Robertson
956e015413
Drop support for delegating email validation, round 2 () 2022-08-23 11:40:00 +00:00
Ayush Anand
22ea51faf9
Add support for compression to federation responses ()
Closes .

Signed-off-by: Ayush Anand <iamayushanand@gmail.com>
2022-08-18 15:14:47 +01:00
Sean Quah
f4ab6a4a96
Refuse to start when faster_joins is enabled on a worker deployment ()
Synapse does not currently support faster room joins on deployments with
workers.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-08-16 16:21:55 +01:00
David Robertson
19e5d44886
Revert "Update locked versions of mypy and mypy-zope ()"
This reverts commit f383b9b3ec. Other PRs
were seeing mypy failures that looked to be related to mypy-zope.
Confusingly, we didn't see this on .

Revert this for now and investigate later.
2022-08-15 14:51:05 +01:00
David Robertson
f383b9b3ec
Update locked versions of mypy and mypy-zope () 2022-08-15 11:32:30 +01:00
3nprob
98fb610cc0
Revert "Drop support for delegating email validation ()" ()
Reverts commit fa71bb18b5, and tweaks documentation.

Signed-off-by: 3nprob <git@3n.anonaddy.com>
2022-07-29 10:29:23 +00:00
Nick Mills-Barrett
86e366a46e
Remove old empty/redundant slaved stores. () 2022-07-21 17:56:45 +00:00
Nick Mills-Barrett
190f49d8ab
Use cache store remove base slaved ()
This comes from two identical definitions in each of the base stores, and means the base slaved store is now empty and can be removed.
2022-07-21 11:51:30 +01:00
Richard van der Hoff
fa71bb18b5
Drop support for delegating email validation ()
* Drop support for delegating email validation

Delegating email validation to an IS is insecure (since it allows the owner of
the IS to do a password reset on your HS), and has long been deprecated. It
will now cause a config error at startup.

* Update unit test which checks for email verification

Give it an `email` config instead of a threepid delegate

* Remove unused method `requestEmailToken`

* Simplify config handling for email verification

Rather than an enum and a boolean, all we need here is a single bool, which
says whether we are or are not doing email verification.

* update docs

* changelog

* upgrade.md: fix typo

* update version number

this will be in 1.64, not 1.63

* update version number

this one too
2022-07-12 19:18:53 +01:00
Erik Johnston
e610128c50
Add a filter_event_for_clients_with_state function () 2022-07-11 14:14:09 +01:00
reivilibre
9667bad55d
Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments. ()
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-30 11:58:12 +00:00
Brendan Abolivier
417f4cf40b
Don't use keyword arguments when initialising modules () 2022-06-15 15:36:16 +02:00
reivilibre
5b645ae2ad
Refactor entry points so that they all have a main function. () 2022-06-14 17:41:06 +00:00
David Robertson
f30bcbd84a
Fix Synapse git info missing in version strings () 2022-06-07 15:24:11 +01:00
Patrick Cloke
cf05258f76
Remove groups replication code. ()
The replication logic for groups is no longer used, so the message
passing infrastructure can be removed.
2022-05-31 13:04:08 -04:00
Patrick Cloke
a8db8c6eba
Remove user-visible groups/communities code ()
Makes it so that groups/communities no longer exist from a user-POV. E.g. we remove:

* All API endpoints (including Client-Server, Server-Server, and admin).
* Documented configuration options (and the experimental flag, which is now unused).
* Special handling during room upgrades.
* The `groups` section of the `/sync` response.
2022-05-25 07:53:40 -04:00
Nick Mills-Barrett
1f9013ce60
Add the batch_send endpoint to generic workers () 2022-05-25 09:51:07 +00:00
Nick Mills-Barrett
2e5f88b5e6
Add the /account/whoami endpoint to generic workers () 2022-05-25 10:41:41 +01:00
David Robertson
d38d242411
Reload cache factors from disk on SIGHUP () 2022-05-11 13:43:22 +00:00
reivilibre
699192fc1a
Add the update_user_directory_from_worker configuration option (superseding update_user_directory) to allow a generic worker to be designated as the worker to update the user directory. ()
Co-authored-by: Shay <hillerys@element.io>
2022-05-10 11:08:45 +01:00
David Robertson
fa0eab9c8e
Use ParamSpec in a few places () 2022-05-09 10:27:39 +00:00
reivilibre
c2d50e9f6c
Add the notify_appservices_from_worker configuration option (superseding notify_appservices) to allow a generic worker to be designated as the worker to send traffic to Application Services. () 2022-05-06 11:43:53 +01:00
David Robertson
6463244375
Remove unused # type: ignores ()
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. ()

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
Patrick Cloke
4586119f0b
Add missing type hints to config classes. () 2022-04-11 12:07:23 -04:00
reivilibre
f871222880
Move update_client_ip background job from the main process to the background worker. () 2022-04-01 13:08:55 +01:00
Shay
3c41d87b67
Add restrictions by default to open registration in Synapse () 2022-03-25 10:11:01 -07:00