forked-synapse

mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00

Author	SHA1	Message	Date
Dirk Klimpel	89c4ca81bb	Add `creation_ts` to list users admin API (#10448 ) Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-07-22 16:05:16 +02:00
Erik Johnston	38b346a504	Replace `or_ignore` in `simple_insert` with `simple_upsert` (#10442 ) Now that we have `simple_upsert` that should be used in preference to trying to insert and looking for an exception. The main benefit is that we ERROR message don't get written to postgres logs. We also have tidy up the return value on `simple_upsert`, rather than having a tri-state of inserted/not-inserted/unknown.	2021-07-22 12:39:50 +01:00
Richard van der Hoff	d8324b8238	Fix a handful of type annotations. (#10446 ) * switch from `types.CoroutineType` to `typing.Coroutine` these should be identical semantically, and since `defer.ensureDeferred` is defined to take a `typing.Coroutine`, will keep mypy happy * Fix some annotations on inlineCallbacks functions * changelog	2021-07-22 12:00:16 +01:00
Eric Eastwood	d518b05a86	Move dev/ docs to development/ (#10453 )	2021-07-22 12:58:24 +02:00
Richard van der Hoff	5e2df47f72	Cancel redundant GHA workflows (#10451 )	2021-07-22 11:35:06 +01:00
Richard van der Hoff	f1347bcfdc	Fix the tests-done Github Actions job (#10444 )	2021-07-22 11:10:30 +01:00
Richard van der Hoff	8ae0bdca75	Drop xenial-support hacks (#10429 )	2021-07-21 21:25:28 +01:00
Patrick Cloke	590cc4e888	Add type hints to additional servlet functions (#10437 ) Improves type hints for: * parse_{boolean,integer} * parse_{boolean,integer}_from_args * parse_json_{value,object}_from_request And fixes any incorrect calls that resulted from unknown types.	2021-07-21 18:12:22 +00:00
Patrick Cloke	5b68816de9	Fix the hierarchy of OpenID providers in the docs. (#10445 )	2021-07-21 13:48:06 -04:00
Patrick Cloke	d15e72e511	Update the notification email subject when invited to a space. (#10426 )	2021-07-21 17:29:54 +00:00
Richard van der Hoff	b2629e7016	Merge remote-tracking branch 'origin/release-v1.39' into develop	2021-07-21 16:12:23 +01:00
Patrick Cloke	5db118626b	Add a return type to parse_string. (#10438 ) And set the required attribute in a few places which will error if a parameter is not provided.	2021-07-21 09:47:56 -04:00
Richard van der Hoff	c6509991f3	Move the docker image build to Github Actions (#10416 ) it's flaky on circleCI, and having to manage multiple CI providers is painful.	2021-07-21 12:33:35 +01:00
Eric Eastwood	2d89c66b88	Switch to `chunk` events so we can auth via power_levels (MSC2716) (#10432 ) Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room.	2021-07-21 10:29:57 +00:00
Andrew Morgan	b181dc402d	Synapse 1.39.0rc1 (2021-07-20) ============================== The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. Features -------- - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884)) - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305)) - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427)) - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360)) Bugfixes -------- - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433)) - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250)) - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276)) - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317)) - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343)) - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344)) - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355)) - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359)) - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367)) - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414)) - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404)) Improved Documentation ---------------------- - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971)) - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313)) - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337)) - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368)) - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399)) - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395)) - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396)) Deprecations and Removals ------------------------- - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721)) - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386)) Internal Changes ---------------- - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289)) - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315)) - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316)) - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324)) - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357)) - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348)) - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349)) - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418)) - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383)) - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385)) - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391)) - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393)) - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398)) - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421)) - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430)) -----BEGIN PGP SIGNATURE----- iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmD20AQTHGFuZHJld0Bh bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ah9EACjDbd67+fYT4CIBp1/gwRZm73qAinF HtJQV3b4KjvF+KC3NTxyfuSo1r6UHSwVP7hYMcW4D7bueYaFVklG4Pb0L58pRdrr nKpVmCTuzQ7KAP165Mea2ZJUihTGrghUV1NTtkgbL2UtBn/j0ymUzWC4dDu2I3V9 vSoa0QOwaBb/iE7RCfwkI/FWNdhJis5CRtzZg31OWqs7kf1FU6+iH7XEtyckkV8M U5aryWPlVzt1c9ag7eLf4OeySyVjl9MqKDjrUd6utLM3bK7DkHSZV/L9wA/NDfwA O6jwnSCsp5TuPjpBFvIOId5P7Ank0WFJTHZj2yzXzrP86gVmmzJi5tTbVDVAaOrw Ept1zKZjNUNBrgB5nu5oSOZQXPrWfSQeQgOfNPkS6SN2fPEvV+koIuv2CBgttmmB 9/iFr0bairlU1LllvyO2Q9VdFb1V2dTFW5MKVW5VdzieeeXDBZYisHT9nr62CRkD ne52Qf84IaLJhwbyFaTssBX7KrmMTfasPzSaAQiLe7i6xlzC6ysdHajrdWf/Vn5c z4I22vaBjZ8FqPucfvC/AiAm5at5bXkVu0UTxvfqeHw6ZXan+ApBYwUVHg50Rx96 J0FkZNfDPZlFTOJpf+jiO/kx4my2pMuiTseoi3wBr21u2j07vRoeeLiGHMuICqlO 1h0ekjbYsXrvdQ== =WSa3 -----END PGP SIGNATURE----- Merge tag 'v1.39.0rc1' into develop Synapse 1.39.0rc1 (2021-07-20) ============================== The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. Features -------- - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884)) - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305)) - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427)) - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360)) Bugfixes -------- - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433)) - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250)) - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276)) - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317)) - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343)) - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344)) - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355)) - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359)) - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367)) - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414)) - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404)) Improved Documentation ---------------------- - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971)) - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313)) - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337)) - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368)) - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399)) - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395)) - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396)) Deprecations and Removals ------------------------- - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721)) - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386)) Internal Changes ---------------- - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289)) - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315)) - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316)) - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324)) - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357)) - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348)) - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349)) - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418)) - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383)) - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385)) - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391)) - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393)) - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398)) - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421)) - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))	2021-07-20 16:47:44 +01:00
Andrew Morgan	f2501f1972	Incorporate changelog of #10433	2021-07-20 14:27:46 +01:00
Erik Johnston	54389d5697	Fix dropping locks on shut down (#10433 )	2021-07-20 14:24:25 +01:00
Michael Telatynski	69226c1ab4	MSC3244 room capabilities implementation (#10283 )	2021-07-20 12:59:23 +01:00
Andrew Morgan	12623cf38c	1.39.0rc1	2021-07-20 12:31:51 +01:00
Richard van der Hoff	97c8ae90f7	Add a github actions job recording success of other jobs. (#10430 )	2021-07-20 11:41:19 +01:00
Brendan Abolivier	a743bf4694	Port the ThirdPartyEventRules module interface to the new generic interface (#10386 ) Port the third-party event rules interface to the generic module interface introduced in v1.37.0	2021-07-20 12:39:46 +02:00
Erik Johnston	f3ac9c6750	Fix exception when failing to get remote room list (#10414 )	2021-07-20 11:35:23 +01:00
reivilibre	eebfd024e9	Factorise `get_datastore` calls in phone_stats_home. (#10427 ) Follow-up to #10332.	2021-07-19 19:31:17 +01:00
reivilibre	4e340412c0	Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric (#10332 ) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2021-07-19 16:11:34 +01:00
Jonathan de Jong	95e47b2e78	[pyupgrade] `synapse/` (#10348 ) This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744	2021-07-19 15:28:05 +01:00
Eric Eastwood	7387d6f624	Remove unused `events_by_room` (#10421 ) It looks like it was first used and introduced in `5130d80d79 (diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8R305)` but the But the usage was removed in `4c6a31cd6e (diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8)`	2021-07-19 10:16:46 +01:00
Jonathan de Jong	323452944e	One last inline type hint (for the whole repo) (#10418 )	2021-07-16 20:12:56 +01:00
Jonathan de Jong	98aec1cc9d	Use inline type hints in `handlers/` and `rest/`. (#10382 )	2021-07-16 18:22:36 +01:00
Brendan Abolivier	36dc15412d	Add a module type for account validity (#9884 ) This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity. To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.	2021-07-16 18:11:53 +02:00
Patrick Cloke	d427f64724	Do not include signatures/hashes in make_{join,leave,knock} responses. (#10404 ) These signatures would end up invalid since the joining/leaving/knocking server would modify the response before calling send_{join,leave,knock}.	2021-07-16 10:36:38 -04:00
Jonathan de Jong	bdfde6dca1	Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381 )	2021-07-15 12:46:54 -04:00
Erik Johnston	3acf85c85f	Reduce likelihood of Postgres table scanning `state_groups_state`. (#10359 ) The postgres statistics collector sometimes massively underestimates the number of distinct state groups are in the `state_groups_state`, which can cause postgres to use table scans for queries for multiple state groups. We fix this by manually setting `n_distinct` on the column.	2021-07-15 16:02:12 +01:00
Brendan Abolivier	9f497024aa	Merge branch 'master' into develop	2021-07-15 14:54:45 +01:00
Brendan Abolivier	3fffb71254	Make deprecation notice of the spam checker doc more obvious (#10395 )	2021-07-15 15:54:22 +02:00
reivilibre	6a60068250	Add tests to characterise the current behaviour of R30 phone-home metrics (#10315 ) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2021-07-15 13:51:27 +01:00
Luke Walsh	23a90a6a5c	Updating install prerequisites for newer macOS & ARM Macs. (#9971 )	2021-07-15 13:18:58 +01:00
Andrew Morgan	c141455049	Docs: Use something other than the document name to describe a page (#10399 ) Our documentation has a history of using a document's name as a way to link to it, such as "See [workers.md]() for details". This makes sense when you're traversing a directory of files, but less sense when the files are abstracted away - as they are on the documentation website. This PR changes the links to various documentation pages to something that fits better into the surrounding sentence, as you would when making any hyperlink on the web.	2021-07-15 12:47:55 +01:00
Erik Johnston	ac5c221208	Stagger send presence to remotes (#10398 ) This is to help with performance, where trying to connect to thousands of hosts at once can consume a lot of CPU (due to TLS etc). Co-authored-by: Brendan Abolivier <babolivier@matrix.org>	2021-07-15 11:52:56 +01:00
Brendan Abolivier	5ecad4e7a5	Update the logcontext doc (#10353 ) By referring to awaitables instead of deferreds.	2021-07-15 11:38:05 +01:00
Jonathan de Jong	bf72d10dbf	Use inline type hints in various other places (in `synapse/`) (#10380 )	2021-07-15 11:02:43 +01:00
Dirk Klimpel	c7603af1d0	Allow providing credentials to `http_proxy` (#10360 )	2021-07-15 10:37:08 +01:00
Erik Johnston	7695ca0618	Fix a number of logged errors caused by remote servers being down. (#10400 )	2021-07-15 10:35:46 +01:00
Moritz Dietz	0ae95b3847	doc: Add delegation example to the caddy reverse proxy section (#10368 )	2021-07-14 18:50:30 +02:00
Richard van der Hoff	28ffff73c1	Instructions on installing RC debs (#10396 )	2021-07-14 17:12:01 +01:00
Richard van der Hoff	c82eb02d64	Set section for prerelease debs (#10391 ) This is part of fixing #6116: we want to put RC debs into a different place than release debs, so reprepro has to be able to tell them apart.	2021-07-14 14:41:40 +01:00
Richard van der Hoff	07e0992a76	Make GHA config more efficient (#10383 ) A few things here: * Build the debs for single distro for each PR, so that we can see if it breaks. Do the same for develop. Building all the debs ties up the GHA workers for ages. * Stop building the debs for release branches. Again, it takes ages, and I don't think anyone is actually going to stop and look at them. We'll know they are working when we make an RC. * Change the configs so that if we manually cancel a workflow, it actually does something.	2021-07-14 14:41:23 +01:00
Patrick Cloke	eb3beb8f12	Add type hints and comments to event auth code. (#10393 )	2021-07-14 14:13:40 +01:00
Eric Eastwood	0d5b08ac7a	Fix messages from multiple senders in historical chunk (MSC2716) (#10276 ) Fix messages from multiple senders in historical chunk. This also means that an app service does not need to define `?user_id` when using this endpoint. Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716	2021-07-13 14:12:33 -05:00
Patrick Cloke	30b56f6925	Add type hints to get_domain_from_id and get_localpart_from_id. (#10385 )	2021-07-13 12:08:47 -04:00
Patrick Cloke	2d16e69b4b	Show all joinable rooms in the spaces summary. (#10298 ) Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.	2021-07-13 08:59:27 -04:00
Richard van der Hoff	f7bfa694ae	1.38.0rc3	2021-07-13 11:57:55 +01:00
Dirk Klimpel	e938f69697	Fix some links in `docs` and `contrib` (#10370 )	2021-07-13 11:55:48 +01:00
Dirk Klimpel	d9b3637e44	Bugfix `make_room_admin` fails for users that have left a private room (#10367 ) Fixes: #10338	2021-07-13 11:53:45 +01:00
Jonathan de Jong	93729719b8	Use inline type hints in `tests/` (#10350 ) This PR is tantamount to running: python3.8 -m com2ann -v 6 tests/ (com2ann requires python 3.8 to run)	2021-07-13 11:52:58 +01:00
Richard van der Hoff	2d8b60e0f2	Github Actions workflow to attach release artifacts to release (#10379 )	2021-07-13 11:50:14 +01:00
Jonathan de Jong	89cfc3dd98	[pyupgrade] `tests/` (#10347 )	2021-07-13 11:43:15 +01:00
Erik Johnston	879d8c1ee1	Fix federation inbound age metric. (#10355 ) We should be reporting the age rather than absolute timestamp.	2021-07-13 11:33:15 +01:00
Richard van der Hoff	5f2848f379	build debs in GHA (#10247 ) GHA workflow to build the debs	2021-07-12 19:03:14 +01:00
Richard van der Hoff	c2c364f27f	Replace `room_depth.min_depth` with a BIGINT (#10289 ) while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth with a BIGINT.	2021-07-12 17:22:54 +01:00
Patrick Cloke	19d0401c56	Additional unit tests for spaces summary. (#10305 )	2021-07-12 11:21:04 -04:00
Erik Johnston	8eddbde0e2	Unblacklist fixed tests (#10357 )	2021-07-09 17:51:15 +01:00
Richard van der Hoff	0f7ed3fc08	Re-enable room v6 sytest (#10345 ) ... now that it has been fixed in https://github.com/matrix-org/sytest/pull/1061.	2021-07-09 17:13:11 +01:00
Erik Johnston	ac036e26c6	Revert "Newsfile" This reverts commit `944428d116`.	2021-07-09 14:52:00 +01:00
Erik Johnston	944428d116	Newsfile	2021-07-09 14:51:37 +01:00
reivilibre	ca9dface8c	Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. (#10344 ) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2021-07-09 14:12:47 +01:00
Richard van der Hoff	751372fa61	Switch `application_services_txns.txn_id` to BIGINT (#10349 )	2021-07-09 13:01:11 +01:00
Erik Johnston	251cfc4e09	Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331)) -----BEGIN PGP SIGNATURE----- iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmDoH+4QHGVyaWtAbWF0 cml4Lm9yZwAKCRClQuTtGw+sCXxYCACneuRvkdvYqiH+PhPe8tXqhhJIifH1LecY FlJqp4OJPR2VFzio1btsgpRPQyLBLHZkJ9pgWsXAETbYOO+hSeOc4nIHsyqlSJhe v01sCUE4sle3DBrw15fG4XpercsiM3TFMyR9pV9laq9nIn8j+CY5K6W5t12/mYGy asHS0IKilCMhJlFwgE3eBr6P6fywi0JoIrr8EpfIs4eC2qDFpUlsrAQSkbE1JvdP O4BGZJKVysg3a6WYSWdJytqLYe942k8qUF4B4h4VmQi0xbuKSsTLiK/cFC8ohRMv E+O5O/KgwqwE/XOcukbsjlHxuiiFZTq6154PwLxXUpNnsMNn2/ph =6iBw -----END PGP SIGNATURE----- Merge tag 'v1.38.0rc2' into develop Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))	2021-07-09 11:26:17 +01:00
Andreas Rammhold	e3e73e181b	Upsert redactions in case they already exists (#10343 ) * Upsert redactions in case they already exists Occasionally, in combination with retention, redactions aren't deleted from the database whenever they are due for deletion. The server will eventually try to backfill the deleted events and trip over the already existing redaction events. Switching to an UPSERT for those events allows us to recover from there situations. The retention code still needs fixing but that is outside of my current comfort zone on this code base. This is related to #8707 where the error was discussed already. Signed-off-by: Andreas Rammhold <andreas@rammhold.de> * Also purge redactions when purging events Previously redacints where left behind leading to backfilling issues when the server stumbled across the already existing yet to be backfilled redactions. This issues has been discussed in #8707. Signed-off-by: Andreas Rammhold <andreas@rammhold.de>	2021-07-09 11:03:02 +01:00
Erik Johnston	717a07b73f	1.38.0rc2	2021-07-09 10:59:28 +01:00
Erik Johnston	1579fdd54a	Ensure we always drop the federation inbound lock (#10336 )	2021-07-09 10:16:54 +01:00
Eric Eastwood	d26094e92c	Add base starting insertion event when no chunk ID is provided (MSC2716) (#10250 ) * Add base starting insertion point when no chunk ID is provided This is so we can have the marker event point to this initial insertion event and be able to traverse the events in the first chunk.	2021-07-08 20:25:59 -05:00
Brendan Abolivier	33ae301fee	Fix formatting in the logcontext doc (#10337 )	2021-07-08 18:16:30 +02:00
Cristina	f6767abc05	Remove functionality associated with unused historical stats tables (#9721 ) Fixes #9602	2021-07-08 16:57:13 +01:00
Dirk Klimpel	974261cd81	Fix broken links in INSTALL.md (#10331 ) Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-07-08 15:46:13 +01:00
reivilibre	aa78064869	Minor changes to `user_daily_visits` (#10324 ) * Use fake time in tests in _get_start_of_day. * Change the inequality of last_seen in user_daily_visits Co-authored-by: Erik Johnston <erik@matrix.org>	2021-07-08 14:27:12 +01:00
Brendan Abolivier	225be77787	Rebuild event auth when rebuilding an event after a call to a `ThirdPartyEventRules` module (#10316 ) Because modules might send extra state events when processing an event (e.g. matrix-org/synapse-dinsic#100), and in some cases these extra events might get dropped if we don't recalculate the initial event's auth.	2021-07-08 13:00:05 +02:00
Richard van der Hoff	189652b2fe	Fix a broken link in the admin api docs (#10322 ) * Fix a broken link in the admin api docs * Rename 10321.doc to 10321.docs * Rename 10321.docs to 10322.doc	2021-07-07 12:54:57 +01:00
Richard van der Hoff	240b3ce253	Merge remote-tracking branch 'origin/release-v1.38' into develop merge @dklimpel's fixes to the sample config	2021-07-07 12:37:10 +01:00
Dirk Klimpel	56fd5fa8e1	Update links to documentation in sample config (#10287 ) Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-07-07 12:35:45 +01:00
Dirk Klimpel	2d044667cf	Simplify structure of room admin API docs (#10313 )	2021-07-07 12:18:36 +01:00
Brendan Abolivier	bc60f999e8	Synapse 1.38.0rc1 (2021-07-06) ============================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. Features -------- - Implement refresh tokens as specified by [MSC2918](https://github.com/matrix-org/matrix-doc/pull/2918). ([\#9450](https://github.com/matrix-org/synapse/issues/9450)) - Add support for evicting cache entries based on last access time. ([\#10205](https://github.com/matrix-org/synapse/issues/10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](https://github.com/matrix-org/synapse/issues/10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](https://github.com/matrix-org/synapse/issues/10225), [\#10243](https://github.com/matrix-org/synapse/issues/10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\#10261](https://github.com/matrix-org/synapse/issues/10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](https://github.com/matrix-org/synapse/issues/10263)) - Add metrics for new inbound federation staging area. ([\#10284](https://github.com/matrix-org/synapse/issues/10284)) - Add script to print information about recently registered users. ([\#10290](https://github.com/matrix-org/synapse/issues/10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](https://github.com/matrix-org/synapse/issues/10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](https://github.com/matrix-org/synapse/issues/10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](https://github.com/matrix-org/synapse/issues/10264), [\#10267](https://github.com/matrix-org/synapse/issues/10267), [\#10282](https://github.com/matrix-org/synapse/issues/10282), [\#10286](https://github.com/matrix-org/synapse/issues/10286), [\#10291](https://github.com/matrix-org/synapse/issues/10291), [\#10314](https://github.com/matrix-org/synapse/issues/10314), [\#10326](https://github.com/matrix-org/synapse/issues/10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](https://github.com/matrix-org/synapse/issues/10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](https://github.com/matrix-org/synapse/issues/10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](https://github.com/matrix-org/synapse/issues/10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](https://github.com/matrix-org/synapse/issues/10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](https://github.com/matrix-org/synapse/issues/10258)) - Fix homeserver config option name in presence router documentation. ([\#10288](https://github.com/matrix-org/synapse/issues/10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\#10302](https://github.com/matrix-org/synapse/issues/10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](https://github.com/matrix-org/synapse/issues/10114)) - Add type hints to the federation servlets. ([\#10213](https://github.com/matrix-org/synapse/issues/10213)) - Improve the reliability of auto-joining remote rooms. ([\#10237](https://github.com/matrix-org/synapse/issues/10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](https://github.com/matrix-org/synapse/issues/10239)) - Fix type hints for computing auth events. ([\#10253](https://github.com/matrix-org/synapse/issues/10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](https://github.com/matrix-org/synapse/issues/10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](https://github.com/matrix-org/synapse/issues/10268)) - Re-enable a SyTest after it has been fixed. ([\#10292](https://github.com/matrix-org/synapse/issues/10292)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDlhGgACgkQHgFcFF8Z Fs006Q/9GEO7DfsDqxidkRmQJ1rVb+yXSvNe1RwTE1hBqmhUg7OvOdQQ5Y48mAyu xoK3M9L3HwaEFe+aJyOUzvqOdnMH31JE/yALuX0C9P5gbamFYdXXXZcJjSo3fWoY 4N8wbNkB/xNPsMI5gtCt7ApGnKm2uWc9tVah/9pNSB/Dg3LIRRuDet38TdjgucB/ gIDCO3FY86SGXdGlSk2bnQb2PKAQcgOIG15tZpju2j2QU6opo5UXir57NGdXNZPo nhkssNJ6Cz2PIvSa6Rz4Arw2lHWALmK8sE/1xI/gvtdScHZkQGUNouRiz+cZbGRr cPEs3IJgzJZzsQ9bSHehRfPJUiYuXB0ij0a9+AeuBmdajH+RRZ6Z/6w6fgHGwIM+ 2T/5oXb3TUh80RnagyvR08Dy+8bAZykZtkuONkBl9LuKhFyHqrkkXfI5bAxXl8qU NT7tq2t0HSOwfxVGg357GNsGd8AbmJE2NR9WkDLSqOnQmIL2zq+6ucvUSdmoePfX c4aGgFmPIeY76Ahm8tpQ4EXLYbZk6l0PYTbQIJa2SypeQ6P4+/LVFERASWl0gRTA I5zy44CZ6IaQ1SEsTQ1oh10wibwfP/nA64U/1JVk0BPoLBr7JoKwMmYKQMpFiyBT XpStxK3lPhbAJXYvEENJx9vJyQd77GOz31fxzFowfZIGPMoN35U= =eCg1 -----END PGP SIGNATURE----- Merge tag 'v1.38.0rc1' into develop Synapse 1.38.0rc1 (2021-07-06) ============================== This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information. Features -------- - Implement refresh tokens as specified by [MSC2918](https://github.com/matrix-org/matrix-doc/pull/2918). ([\#9450](https://github.com/matrix-org/synapse/issues/9450)) - Add support for evicting cache entries based on last access time. ([\#10205](https://github.com/matrix-org/synapse/issues/10205)) - Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](https://github.com/matrix-org/synapse/issues/10214)) - Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](https://github.com/matrix-org/synapse/issues/10225), [\#10243](https://github.com/matrix-org/synapse/issues/10243)) - Add SSO `external_ids` to the Query User Account admin API. ([\#10261](https://github.com/matrix-org/synapse/issues/10261)) - Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](https://github.com/matrix-org/synapse/issues/10263)) - Add metrics for new inbound federation staging area. ([\#10284](https://github.com/matrix-org/synapse/issues/10284)) - Add script to print information about recently registered users. ([\#10290](https://github.com/matrix-org/synapse/issues/10290)) Bugfixes -------- - Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](https://github.com/matrix-org/synapse/issues/10223)) - Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](https://github.com/matrix-org/synapse/issues/10252)) - Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](https://github.com/matrix-org/synapse/issues/10264), [\#10267](https://github.com/matrix-org/synapse/issues/10267), [\#10282](https://github.com/matrix-org/synapse/issues/10282), [\#10286](https://github.com/matrix-org/synapse/issues/10286), [\#10291](https://github.com/matrix-org/synapse/issues/10291), [\#10314](https://github.com/matrix-org/synapse/issues/10314), [\#10326](https://github.com/matrix-org/synapse/issues/10326)) - Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](https://github.com/matrix-org/synapse/issues/10279)) - Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](https://github.com/matrix-org/synapse/issues/10303)) Improved Documentation ---------------------- - Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](https://github.com/matrix-org/synapse/issues/10166)) - Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](https://github.com/matrix-org/synapse/issues/10242)) - Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](https://github.com/matrix-org/synapse/issues/10258)) - Fix homeserver config option name in presence router documentation. ([\#10288](https://github.com/matrix-org/synapse/issues/10288)) - Fix link pointing at the wrong section in the modules documentation page. ([\#10302](https://github.com/matrix-org/synapse/issues/10302)) Internal Changes ---------------- - Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](https://github.com/matrix-org/synapse/issues/10114)) - Add type hints to the federation servlets. ([\#10213](https://github.com/matrix-org/synapse/issues/10213)) - Improve the reliability of auto-joining remote rooms. ([\#10237](https://github.com/matrix-org/synapse/issues/10237)) - Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](https://github.com/matrix-org/synapse/issues/10239)) - Fix type hints for computing auth events. ([\#10253](https://github.com/matrix-org/synapse/issues/10253)) - Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](https://github.com/matrix-org/synapse/issues/10256)) - Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](https://github.com/matrix-org/synapse/issues/10268)) - Re-enable a SyTest after it has been fixed. ([\#10292](https://github.com/matrix-org/synapse/issues/10292))	2021-07-07 11:41:16 +01:00
Brendan Abolivier	24796f80ba	Merge latest fix into the changelog	2021-07-07 11:21:58 +01:00
Richard van der Hoff	4358f51bb6	Merge branch 'release-v1.38' into develop	2021-07-07 11:09:49 +01:00
Brendan Abolivier	9ad8455895	ANALYZE new stream ordering column (#10326 ) Fixes #10325	2021-07-07 11:56:17 +02:00
Dagfinn Ilmari Mannsåker	47e28b4031	Ignore EDUs for rooms we're not in (#10317 )	2021-07-06 14:31:13 +01:00
Brendan Abolivier	37da9db082	1.38.0rc1	2021-07-06 13:54:23 +01:00
Dirk Klimpel	bcb0962a72	Fix deactivate a user if he does not have a profile (#10252 )	2021-07-06 13:08:53 +01:00
Erik Johnston	6655ea5587	Add script for getting info about recently registered users (#10290 )	2021-07-06 13:03:16 +01:00
Erik Johnston	c65067d673	Handle old staged inbound events (#10303 ) We might have events in the staging area if the service was restarted while there were unhandled events in the staging area. Fixes #10295	2021-07-06 13:02:37 +01:00
Richard van der Hoff	d7a94a7dcc	Add upgrade notes about disk space for events migration (#10314 )	2021-07-06 11:00:05 +01:00
Erik Johnston	7a5873277e	Add support for evicting cache entries based on last access time. (#10205 )	2021-07-05 16:32:12 +01:00
Brendan Abolivier	10671da05b	Fix bad link in modules documentation (#10302 ) Fix link in modules doc to point at instructions on registering a callback instead of ones on registering a web resource.	2021-07-02 13:20:43 +02:00
Patrick Cloke	8d609435c0	Move methods involving event authentication to EventAuthHandler. (#10268 ) Instead of mixing them with user authentication methods.	2021-07-01 14:25:37 -04:00
Richard van der Hoff	0aab50c772	fix ordering of bg update (#10291 ) this was a typo introduced in #10282. We don't want to end up doing the `replace_stream_ordering_column` update after anything that comes up in migration 60/03.	2021-07-01 18:45:55 +01:00
Erik Johnston	e72c287418	Reenable 'Backfilled events whose prev_events...' sytest (#10292 ) Now that we've fixed it.	2021-07-01 12:21:58 +01:00
Dirk Klimpel	6c02cca95f	Add SSO `external_ids` to Query User Account admin API (#10261 ) Related to #10251	2021-07-01 11:26:24 +02:00
Erik Johnston	76addadd7c	Add some metrics to staging area (#10284 )	2021-07-01 10:18:25 +01:00
Andrew Morgan	04c8f308f4	Fix the homeserver config example in presence router docs (#10288 ) The presence router docs include some sample homeserver config. At some point we changed the name of the [config option](`859dc05b36/docs/sample_config.yaml (L104-L113)`), but forgot to update the docs. I've also added `presence.enabled: true` to the example, as that's the new way to enable presence (the `presence_enabled` option has been deprecated).	2021-06-30 23:43:58 +01:00
Richard van der Hoff	b6dbf89fae	Change more stream_ordering columns to BIGINT (#10286 )	2021-06-30 17:27:20 +01:00
Richard van der Hoff	859dc05b36	Rebuild other indexes using `stream_ordering` (#10282 ) We need to rebuild all of the indexes that use the current `stream_ordering` column.	2021-06-30 15:01:24 +01:00
Patrick Cloke	aaf7d1acb8	Correct type hints for synapse.event_auth. (#10253 )	2021-06-30 07:08:42 -04:00
Erik Johnston	329ef5c715	Fix the inbound PDU metric (#10279 ) This broke in #10272	2021-06-30 12:07:16 +01:00
Richard van der Hoff	bc5589a1bb	Merge branch 'release-v1.37' into develop	2021-06-29 21:48:24 +01:00
Richard van der Hoff	d561367c18	1.37.1rc1	2021-06-29 21:39:30 +01:00
Richard van der Hoff	785bceef72	Merge branch 'release-v1.37' into develop	2021-06-29 20:25:47 +01:00
Richard van der Hoff	ba9b744bb2	Update newsfiles	2021-06-29 20:02:39 +01:00
Erik Johnston	c54db67d0e	Handle inbound events from federation asynchronously (#10272 ) Fixes #9490 This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous. Two drawbacks: Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately. We now only process one event per room at a time. This can be fixed up further down the line.	2021-06-29 19:55:22 +01:00
Erik Johnston	85d237eba7	Add a distributed lock (#10269 ) This adds a simple best effort locking mechanism that works cross workers.	2021-06-29 19:15:47 +01:00
Patrick Cloke	f55836929d	Do not recurse into non-spaces in the spaces summary. (#10256 ) Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.	2021-06-29 12:00:04 -04:00
Richard van der Hoff	7647b0337f	Fix `populate_stream_ordering2` background job (#10267 ) It was possible for us not to find any rows in a batch, and hence conclude that we had finished. Let's not do that.	2021-06-29 12:43:36 +01:00
Richard van der Hoff	60efc51a2b	Migrate stream_ordering to a bigint (#10264 ) * Move background update names out to a separate class `EventsBackgroundUpdatesStore` gets inherited and we don't really want to further pollute the namespace. * Migrate stream_ordering to a bigint * changelog	2021-06-29 11:25:34 +01:00
Richard van der Hoff	a0ed0f363e	Soft-fail spammy events received over federation (#10263 )	2021-06-29 11:08:06 +01:00
Patrick Cloke	0555d7b0dc	Add additional types to the federation transport server. (#10213 )	2021-06-28 07:36:41 -04:00
Felix Kronlage-Dammers	717f73c411	Adjust the URL in the README.rst file to point to LiberaChat instead of freenode (#10258 )	2021-06-28 11:07:25 +02:00
Andrew Morgan	f0e02f5df2	Create an index.html file when generating a docs build (#10242 ) Currently when a new build of the docs is created, an `index.html` file does not exist. Typically this would be generated from a`docs/README.md` file - which we have - however we're currently using [docs/README.md](`394673055d/docs/README.md`) to explain the docs and point to the website. It is not part of the content of the website. So we end up not having an `index.html` file, which will result in a 404 page if one tries to navigate to `https://matrix-org.github.io/synapse/<docs_version>/index.html`. This isn't a really problem for the default version of the documentation (currently `develop`), as [navigating to the top-level root](https://matrix-org.github.io/synapse/) of the website (without specifying a version) will [redirect](`a77e6925f2/index.html (L2)`) you to the Welcome and Overview page of the `develop` docs version. However, ideally once we add a GUI for switching between versions, we'll want to send the user to `matrix-org.github.io/synapse/<version>/index.html`, which currently isn't generated. This PR modifies the CI that builds the docs to simply copy the rendered [Welcome & Overview page](https://matrix-org.github.io/synapse/develop/welcome_and_overview.html) to `index.html`.	2021-06-24 18:00:56 +01:00
Richard van der Hoff	8165ba48b1	Return errors from `send_join` etc if the event is rejected (#10243 ) Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.	2021-06-24 16:00:08 +01:00
Richard van der Hoff	6e8fb42be7	Improve validation for `send_{join,leave,knock}` (#10225 ) The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.	2021-06-24 15:30:49 +01:00
Quentin Gliech	bd4919fb72	MSC2918 Refresh tokens implementation (#9450 ) This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>	2021-06-24 14:33:20 +01:00
Brendan Abolivier	763dba77ef	Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199)) -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUW7UACgkQHgFcFF8Z Fs1ZfQ//bSyzg9mB7F43JdVBH9cpGups3Ur6nmsqE608/i/OEtomx9kTZsoN/9AO j6WoM1bhJ3j9hoE63w4DIPVhBrojtwmYWeFrRvpRVlwTzZPhfij1nIH9pPrNVwd6 /LE0uA1kW5sDZCTiOhrz84BBBz6knjBH4otl+MDDdrszYixoP3i3SrrZVJAEYyGz Dpw2OudpoOtMdGyzFMr2ewf2mQm5UFEPs9MT0AqhQJMjyM1lJYzrq3Obv1oTdCJq BdLx6kMhK8+MbZPrcqIMXgSMHLYeQYeJOrdTzuCozFo0DSblThpW//aa21+sZtPx dnIE0PpjAsm2Qg1II3EUX+YycYpI5KPhMq0ydEmUD7vTuGgJCKqAV3FBRwMlF2VE 9SOr8Mjd1ip03dHz2vlg38OlrFJ52P0e1RDZwlDtWfeJjZGZ4W6VjMhT7H/Z5USg 2J1FRPTD8E9w8wBOkjVcl76Lj6AY1KU36/ZLoYUbtrt58XYfkD4OzZmAxdJrqbMg na5T9bd9iM8vWditU+vncxU+c7Bkn55GFyDVMQYl8QtXzBvnbv8gQilGwJLkncZJ MreQ3OIBPbLNcHml56nmmY+ab+AtrIxuFl0UcZu+LGW8+lN+UZ9G9q0NFO3HbAWN /LuhCVpO42SLAuLlIhnO3mcHM5jzZodKIa6oXpcMS2jv0nylQWo= =MtOo -----END PGP SIGNATURE----- gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEdVkXOgzrGzds0jtrHgFcFF8ZFs0FAmDUXG8ACgkQHgFcFF8Z Fs2buxAAg7X4mYKHwlWLn2n2W6bUKY24HAnjtzMrNW76xIHXBJJiJMnhQD3MSKVd uiUdolc+Tt3drJYOJyurP8R3YI0OuEVbSL0Krs8f3l6jNOU0Gy7YQzc6XUHnA42J rb6mU8gMdCusk9RHHr1r+sBefwzmxTk4naCkw2n5i+r4ZfNparJK7H9yXCPoedUN VBZi9f09jaUIm1SiHgK24sW9XfE6ifjdVg+E/r599v2eHJxtCTHUasH9TCAFD8KL g17NtAP95+SQWPGwUXJFZYcVee+pGbylXuwP0RXXmMvyOWVKloI0LE6phGyLQ5cH rMRagIkvVLMdYagaX8Tf47hNLVCXRJpuryxMgY3+5fx0O7lCMXlC3mDfs9zze+Fj KXmJtVy/vpd+zpjQw+HHPnL1dDzsKnDqLFXFCj9Sjv6A/ruckxwWofuh0z83dYtv 1Abp8k7gVTVGZqLGPo8gRoqI2jQhWDvmJd+crb05qp86SzCz1sCma2ujWQtv3c1t 6aQpfmGMhFPp+hS2lS8AcyxK4vq+GudtXUOgzdqCNvIs+eJAV5cq6uufLIy6c2XR FAR7UowoAoDvqTdD7+i5nHEjbudjjN5KgPXlO3a9+gG5jOR3IsOLZ8FVN0ML6qIJ wzl0wM9YIlxyleMlWb0l4CFNDKrGJVhWgOupruHS+D5jfZQsZ3g= =AjRf -----END PGP SIGNATURE----- Merge tag 'v1.37.0rc1' into develop Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))	2021-06-24 11:20:28 +01:00
Brendan Abolivier	acac4535c5	Tweak changelog	2021-06-24 10:58:08 +01:00
Andrew Morgan	7f25d73859	Convert UPGRADE.rst to markdown (#10166 ) This PR: * Converts UPGRADE.rst to markdown and moves the contents into the `docs/` directory. * Updates the contents of UPGRADE.rst to point to the website instead. * Updates links around the codebase that point to UPGRADE.rst. `pandoc` + some manual editing was used to convert from RST to md.	2021-06-23 16:57:57 +01:00
Brendan Abolivier	d731ed70d9	Fixes to the release script (#10239 ) * rename major/minor into the right semver terminology minor/patch (since this was something that got me very confused the first couple of times I've used the script) * name the release branch based on the new version, not the previous one	2021-06-23 17:55:26 +02:00
Brendan Abolivier	c955e37868	Fix wrapping of legacy check_registration_for_spam (#10238 ) Fixes #10234	2021-06-23 17:22:08 +02:00
Andrew Morgan	394673055d	Re-introduce "Leave out optional keys from /sync" change (#10214 ) Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.	2021-06-23 15:57:41 +01:00
Brendan Abolivier	e19e3d452d	Improve the reliability of auto-joining remote rooms (#10237 ) If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do. Fixes #10233 (at least to some extent)	2021-06-23 16:14:52 +02:00
Richard van der Hoff	8beead66ae	Send out invite rejections and knocks over federation (#10223 ) ensure that events sent via `send_leave` and `send_knock` are sent on to the rest of the federation.	2021-06-23 12:54:50 +01:00
Michael[tm] Smith	27c06a6e06	Drop Origin & Accept from Access-Control-Allow-Headers value (#10114 ) * Drop Origin & Accept from Access-Control-Allow-Headers value This change drops the Origin and Accept header names from the value of the Access-Control-Allow-Headers response header sent by Synapse. Per the CORS protocol, it’s not necessary or useful to include those header names. Details: Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name, Origin is a “forbidden header name” set by the browser and that frontend JavaScript code is never allowed to set. So the value of Access-Control-Allow-Headers isn’t relevant to Origin or in general to other headers set by the browser itself — the browser never ever consults the Access-Control-Allow-Headers value to confirm that it’s OK for the request to include an Origin header. And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header, Accept is a “CORS-safelisted request-header”, which means that browsers allow requests to contain the Accept header regardless of whether the Access-Control-Allow-Headers value contains "Accept". So it’s unnecessary for the Access-Control-Allow-Headers to explicitly include Accept. Browsers will not perform a CORS preflight for requests containing an Accept request header. Related: https://github.com/matrix-org/matrix-doc/pull/3225 Signed-off-by: Michael[tm] Smith <mike@w3.org>	2021-06-23 11:25:03 +01:00
Brendan Abolivier	9ec45aca1f	1.37.0rc1	2021-06-23 09:38:27 +01:00
Erik Johnston	33701dc116	Fix schema delta to not take as long on large servers (#10227 ) Introduced in #6739	2021-06-22 12:00:45 +01:00
Brendan Abolivier	34db6bb9f5	Warn users trying to use the deprecated spam checker interface (#10210 ) So admins aren't surprised if things break when we remove this code in a couple of months.	2021-06-22 12:24:10 +02:00
Eric Eastwood	96f6293de5	Add endpoints for backfilling history (MSC2716) (#9247 ) Work on https://github.com/matrix-org/matrix-doc/pull/2716	2021-06-22 10:02:53 +01:00
jkanefendt	756fd513df	Implement config option `sso.update_profile_information` (#10108 ) Implemented config option sso.update_profile_information to keep user's display name in sync with the SSO displayname. Signed-off-by: Johannes Kanefendt <johannes.kanefendt@krzn.de>	2021-06-21 23:48:57 +01:00
Erik Johnston	a5cd05beee	Fix performance of responding to user key requests over federation (#10221 ) We were repeatedly looking up a config option in a loop (using the unclassed config style), which is expensive enough that it can cause large CPU usage.	2021-06-21 14:38:59 +01:00
Andrew Morgan	182147195b	Check third party rules before persisting knocks over federation (#10212 ) An accidental mis-ordering of operations during #6739 technically allowed an incoming knock event over federation in before checking it against any configured Third Party Access Rules modules. This PR corrects that by performing the TPAR check before persisting the event.	2021-06-21 11:57:09 +01:00
Richard van der Hoff	107c06081f	Ensure that errors during startup are written to the logs and the console. (#10191 ) * Defer stdio redirection until we are about to start the reactor * Catch and handle exceptions during startup	2021-06-21 11:41:25 +01:00
Andrew Morgan	7c536d0fef	Deploy a documentation version for each new Synapse release (#10198 ) This PR will run a new "Deploy release-specific documentation" job whenever a push to a branch name matching `release-v*` occurs. Doing so will create/add to a folder named `vX.Y` on the `gh-pages` branch. Doing so will allow us to build up `major.minor` releases of the docs as we release Synapse. This is especially useful for having a mechanism for keeping around documentation of old/removed features (for those running older versions of Synapse), without needing to clutter the latest copy of the docs. After a [discussion](https://matrix.to/#/!XaqDhxuTIlvldquJaV:matrix.org/$rKmkBmQle8OwTlGcoyu0BkcWXdnHW3_oap8BMgclwIY?via=matrix.org&via=vector.modular.im&via=envs.net) in #synapse-dev, we wanted to use tags to trigger the documentation deployments, which I agreed with. However, I soon realised that the bash-foo required to turn a tag of `v1.2.3rc1` into `1.2` was a lot more complex than the branch's `release-v1.2`. So, I've gone with the latter for simplicity. In the future we'll have some UI on the website to switch between versions, but for now you can simply just change 'develop' to 'v1.2' in the URL.	2021-06-18 19:26:25 +01:00
Patrick Cloke	0bd968921c	Fix a missing await when in the spaces summary. (#10208 ) This could cause a minor data leak if someone defined a non-restricted join rule with an allow key or used a restricted join rule in an older room version, but this is unlikely. Additionally this starts adding unit tests to the spaces summary handler.	2021-06-18 18:41:33 +01:00
Brendan Abolivier	e9f2ad8603	Describe callbacks signatures as async in new modules doc (#10206 )	2021-06-18 16:55:53 +02:00
Brendan Abolivier	1b3e398bea	Standardise the module interface (#10062 ) This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.	2021-06-18 12:15:52 +01:00
Richard van der Hoff	91fa9cca99	Expose opentracing trace id in response headers (#10199 ) Fixes: #9480	2021-06-18 11:43:22 +01:00
Brendan Abolivier	08c8469322	Remove support for ACME v1 (#10194 ) Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.	2021-06-17 18:56:48 +01:00
Patrick Cloke	8c97d5863f	Update MSC3083 support per changes in the MSC. (#10189 ) Adds a "type" field and generalize "space" to "room_id".	2021-06-17 12:53:27 -04:00
Richard van der Hoff	fcf3c7032b	Ensure that we do not cache empty sync responses after a timeout (#10158 ) Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.	2021-06-17 16:23:11 +01:00
Richard van der Hoff	9cf6e0eae7	Rip out the DNS lookup limiter (#10190 ) As I've written in various places in the past (#7113, #9865) I'm pretty sure this is doing nothing useful at all.	2021-06-17 16:22:41 +01:00
Marcus	8070b893db	update black to 21.6b0 (#10197 ) Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>	2021-06-17 15:20:06 +01:00
Andrew Morgan	6f1a28de19	Fix incorrect time magnitude on delayed call (#10195 ) Fixes https://github.com/matrix-org/synapse/issues/10030. We were expecting milliseconds where we should have provided a value in seconds. The impact of this bug isn't too bad. The code is intended to count the number of remote servers that the homeserver can see and report that as a metric. This metric is supposed to run initially 1 second after server startup, and every 60s as well. Instead, it ran 1,000 seconds after server startup, and every 60s after startup. This fix allows for the correct metrics to be collected immediately, as well as preventing a random collection 1,000s in the future after startup.	2021-06-17 15:04:26 +01:00
Eric Eastwood	a911dd768b	Add fields to better debug where events are being soft_failed (#10168 ) Follow-up to https://github.com/matrix-org/synapse/pull/10156#discussion_r650292223	2021-06-17 14:59:45 +01:00
Richard van der Hoff	52c60bd0a9	Fix persist_events to stop leaking opentracing contexts (#10193 )	2021-06-17 11:21:53 +01:00
Patrick Cloke	18edc9ab06	Improve comments in the structured logging code. (#10188 )	2021-06-16 19:18:02 +01:00
Patrick Cloke	76f9c701c3	Always require users to re-authenticate for dangerous operations. (#10184 ) Dangerous actions means deactivating an account, modifying an account password, or adding a 3PID. Other actions (deleting devices, uploading keys) can re-use the same UI auth session if ui_auth.session_timeout is configured.	2021-06-16 11:07:28 -04:00
Michael Kaye	b8b282aa32	A guide to the request log lines format. (#8436 ) This doc is short but a useful guide to what the request log lines mean. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org> Co-authored-by: Daniele Sluijters <daenney@users.noreply.github.com>	2021-06-16 13:31:55 +01:00
Erik Johnston	36c426e294	Add debug logging when we enter/exit Measure block (#10183 ) It can be helpful to know when trying to track down slow requests.	2021-06-16 13:29:54 +01:00
Lukas Lihotzki	2c240213f4	Fix requestOpenIdToken response: integer expires_in (#10175 ) `expires_in` must be an integer according to the OpenAPI spec: https://github.com/matrix-org/matrix-doc/blob/master/data/api/client-server/definitions/openid_token.yaml#L32 True division (`/`) returns a float instead (`"expires_in": 3600.0`). Floor division (`//`) returns an integer, so the response is spec compliant. Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>	2021-06-16 13:16:35 +01:00
Dirk Klimpel	0adc2882c1	Fix broken links in documentation (#10180 ) * Fix broken links in documentation * newsfile	2021-06-16 13:15:52 +01:00
Richard van der Hoff	9e405034e5	Make opentracing trace into event persistence (#10134 ) * Trace event persistence When we persist a batch of events, set the parent opentracing span to the that from the request, so that we can trace all the way in. * changelog * When we force tracing, set a baggage item ... so that we can check again later. * Link in both directions between persist_events spans	2021-06-16 11:41:15 +01:00
Patrick Cloke	4911f7931d	Remove support for unstable MSC1772 prefixes. (#10161 ) The stable prefixes have been supported since v1.34.0. The unstable prefixes are not supported by any known clients.	2021-06-15 08:03:17 -04:00
Patrick Cloke	9e5ab6dd58	Remove the experimental flag for knocking and use stable prefixes / endpoints. (#10167 ) * Room version 7 for knocking. * Stable prefixes and endpoints (both client and federation) for knocking. * Removes the experimental configuration flag.	2021-06-15 07:45:14 -04:00
Michael Kutzner	aac2c49b9b	Fix 'ip_range_whitelist' not working for federation servers (#10115 ) Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers. Signed-off-by: Michael Kutzner 1mikure@gmail.com	2021-06-15 08:53:55 +01:00
Richard van der Hoff	1dfdc87b9b	Refactor `EventPersistenceQueue` (#10145 ) some cleanup, pulled out of #10134.	2021-06-14 11:59:27 +01:00
Richard van der Hoff	d7808a2dde	Extend `ResponseCache` to pass a context object into the callback (#10157 ) This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.	2021-06-14 10:26:09 +01:00
Richard van der Hoff	13577aa55e	Notes on boolean columns in database schemas (#10164 )	2021-06-11 17:13:56 +01:00
Erik Johnston	29966a285d	Synapse 1.36.0rc2 (2021-06-11) ============================== Bugfixes -------- - Fix a bug which caused presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149)) - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163)) - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165)) -----BEGIN PGP SIGNATURE----- iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmDDdrcQHGVyaWtAbWF0 cml4Lm9yZwAKCRClQuTtGw+sCTzfB/4qaTqW2mBiwjf52SmOu6HNyd8uQd6nLIAZ mJC218Wakh2tT0W4iVkKwUgpuHFbtcy0rSNTlXtW4kG8XzhpTvT56RH9qls99aD3 SGKqYpOv6NkWZibN6NdVvLDW85ixficDTXco3BljRCIMlORhY0swy+LWLwksdjWj 6kQ+Gi/QAtKP3Pt5epYs0Ix5o1T94DfZOWE//mqBhG5cMDAw/K/G/c8tRfEjclt9 wACBjmt2fw/Lbn9j3b0feNVp+xnFcFNuAK2bSEd8Y3yph1mhjdsIszULnM7IFNsR Q8zg+i7PJKNq8pQjei8j8T/aKscPTPH5XGqOSLlizj15snsiwlkz =C6lV -----END PGP SIGNATURE----- Merge tag 'v1.36.0rc2' into develop Synapse 1.36.0rc2 (2021-06-11) ============================== Bugfixes -------- - Fix a bug which caused presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149)) - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163)) - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165))	2021-06-11 15:46:38 +01:00
Erik Johnston	fb10a73e85	1.36.0rc2	2021-06-11 15:21:34 +01:00
Erik Johnston	cdd985c64f	Only send a presence state to a destination once (#10165 ) It turns out that we were sending the same presence state to a remote potentially multiple times.	2021-06-11 15:21:08 +01:00
Erik Johnston	5e0b4719ea	Fix sending presence over federation when using workers (#10163 ) When using a federation sender we'd send out all local presence updates over federation even when they shouldn't be. Fixes #10153.	2021-06-11 15:20:54 +01:00
Erik Johnston	c955f22e2c	Fix bug when running presence off master (#10149 ) Hopefully fixes #10027.	2021-06-11 15:20:45 +01:00
Erik Johnston	968f8283b4	Only send a presence state to a destination once (#10165 ) It turns out that we were sending the same presence state to a remote potentially multiple times.	2021-06-11 15:19:42 +01:00
Richard van der Hoff	c1b9922498	Support for database schema version ranges (#9933 ) This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.	2021-06-11 14:45:53 +01:00
Patrick Cloke	a14884fbb0	Use the matching complement branch when running tests in CI. (#10160 ) This implements similar behavior to sytest where a matching branch is used, if one exists. This is useful when needing to modify both application code and tests at the same time. The following rules are used to find a matching complement branch: 1. Search for the branch name of the pull request. (E.g. feature/foo.) 2. Search for the base branch of the pull request. (E.g. develop or release-vX.Y.) 3. Search for the reference branch of the commit. (E.g. master or release-vX.Y.) 4. Fallback to 'master', the default complement branch name.	2021-06-11 08:17:17 -04:00
Erik Johnston	c8dd4db9eb	Fix sending presence over federation when using workers (#10163 ) When using a federation sender we'd send out all local presence updates over federation even when they shouldn't be. Fixes #10153.	2021-06-11 13:08:30 +01:00
Andrew Morgan	a15a046c93	Clean up a broken import in admin_cmd.py (#10154 )	2021-06-11 11:34:40 +01:00
Erik Johnston	d26d15ba3d	Fix bug when running presence off master (#10149 ) Hopefully fixes #10027.	2021-06-11 10:27:12 +01:00
Eric Eastwood	b31daac01c	Add metrics to track how often events are `soft_failed` (#10156 ) Spawned from missing messages we were seeing on `matrix.org` from a federated Gtiter bridged room, https://gitlab.com/gitterHQ/webapp/-/issues/2770. The underlying issue in Synapse is tracked by https://github.com/matrix-org/synapse/issues/10066 where the message and join event race and the message is `soft_failed` before the `join` event reaches the remote federated server. Less soft_failed events = better and usually this should only trigger for events where people are doing bad things and trying to fuzz and fake everything.	2021-06-11 10:12:35 +01:00
Eric Eastwood	e21c347332	Document how to see logger output when running the twisted tests (#10148 )	2021-06-11 09:57:34 +01:00
Aaron Raimist	e6245e6d48	Mention that you need to configure max upload size in reverse proxy as well (#10122 ) Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-06-10 11:40:24 +01:00
Andrew Morgan	aec2cf1c98	Update Complement run with Synapse-supported MSC-related build tags (#10155 ) This PR updates the build tags that we perform Complement runs with to match our [buildkite pipeline](`618b3e90bc/synapse/pipeline.yml (L570)`), as well as adding `msc2403` (as it will be required once #9359 is merged). Build tags are what we use to determine which tests to run in Complement (really it determines which test files are compiled into the final binary). I haven't put in a comment about updating the buildkite side here, as we've decided to migrate fully to GitHub Actions anyhow.	2021-06-09 20:59:40 +01:00
Andrew Morgan	a7a37437bc	Integrate knock rooms with the public rooms directory (#9359 ) This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403. Specifically, it: * Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory * Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room Based on https://github.com/matrix-org/synapse/issues/6739. Complement tests for this change: https://github.com/matrix-org/complement/pull/72	2021-06-09 20:31:31 +01:00
Sorunome	d936371b69	Implement knock feature (#6739 ) This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io	2021-06-09 19:39:51 +01:00
Patrick Cloke	11846dff8c	Limit the number of in-flight /keys/query requests from a single device. (#10144 )	2021-06-09 07:05:32 -04:00
Richard van der Hoff	1bf83a191b	Clean up the interface for injecting opentracing over HTTP (#10143 ) * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog	2021-06-09 11:33:00 +01:00
Patrick Cloke	c7f3fb2745	Add type hints to the federation server transport. (#10080 )	2021-06-08 11:19:25 -04:00
Andrew Morgan	8df9941cc2	1.36.0rc1	2021-06-08 14:09:00 +01:00
Erik Johnston	1092718cac	Fix logging context when opening new DB connection (#10141 ) Fixes #10140	2021-06-08 13:49:29 +01:00
Patrick Cloke	9e4610cc27	Correct type hints for parse_string(s)_from_args. (#10137 )	2021-06-08 08:30:48 -04:00
Dan Callahan	7dc14730d9	Name release branches just after major.minor (#10013 ) With the prior format, 1.33.0 / 1.33.1 / 1.33.2 got separate branches: release-v1.33.0 release-v1.33.1 release-v1.33.2 Under the new model, all three would share a common branch: release-v1.33 As before, RCs and actual releases exist as tags on these branches. This better reflects our support model, e.g., that the "1.33" series had a formal release followed by two patches / updates. Signed-off-by: Dan Callahan <danc@element.io>	2021-06-08 11:44:50 +01:00
Erik Johnston	c842c581ed	When joining a remote room limit the number of events we concurrently check signatures/hashes for (#10117 ) If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.	2021-06-08 11:07:46 +01:00
Erik Johnston	a0101fc021	Handle /backfill returning no events (#10133 ) Fixes #10123	2021-06-08 10:37:01 +01:00
Richard van der Hoff	0acb5010ec	More database opentracing (#10136 ) Add a couple of extra logs/spans, to give a bit of a better idea.	2021-06-07 18:01:32 +01:00
Richard van der Hoff	b2557cbf42	opentracing: use a consistent name for background processes (#10135 ) ... otherwise we tend to get a namespace clash between the bg process and the functions that it calls.	2021-06-07 17:57:49 +01:00
Rohan Sharma	beb251e3ee	Make link in docs use HTTPS (#10130 ) Fixes #10121 Signed-off-by: Rohan Sharma <rhnsharma5113@gmail.com>	2021-06-07 16:35:02 +01:00
Chris Castle	543e423fce	Fix broken link to README at root of repo (#10132 ) Signed-off-by: Chris Castle chris@crc.io	2021-06-07 16:31:39 +01:00
14mRh4X0r	8942e23a69	Always update AS last_pos, even on no events (#10107 ) Fixes #1834. `get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether. Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list. Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>	2021-06-07 15:42:05 +01:00
Dirk Klimpel	d558292548	Add missing type hints to the admin API servlets (#10105 )	2021-06-07 15:12:34 +01:00
Richard van der Hoff	fa1db8f156	Delete completes to-device messages earlier in /sync (#10124 ) I hope this will improve https://github.com/matrix-org/synapse/issues/9564.	2021-06-07 09:19:06 +01:00
Erik Johnston	a0cd8ae8cb	Don't try and backfill the same room in parallel. (#10116 ) If backfilling is slow then the client may time out and retry, causing Synapse to start a new `/backfill` before the existing backfill has finished, duplicating work.	2021-06-04 10:47:58 +01:00
Erik Johnston	c96ab31dff	Limit number of events in a replication request (#10118 ) Fixes #9956.	2021-06-04 10:35:47 +01:00
Richard van der Hoff	d8be7d493d	Enable Prometheus metrics for the jaeger client library (#10112 )	2021-06-04 09:25:33 +01:00
Andrew Morgan	fd9856e4a9	Compile and render Synapse's docs into a browsable, mobile-friendly and searchable website (#10086 )	2021-06-03 17:20:40 +01:00
Richard van der Hoff	9eea4646be	Add OpenTracing for database activity. (#10113 ) This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).	2021-06-03 16:31:56 +01:00
Richard van der Hoff	1d143074c5	Improve opentracing annotations for Notifier (#10111 ) The existing tracing reports an error each time there is a timeout, which isn't really representative. Additionally, we log things about the way `wait_for_events` works (eg, the result of the callback) to the parent span, which is confusing.	2021-06-03 16:01:30 +01:00
Andrew Morgan	73636cab69	Convert admin api docs to markdown (#10089 ) So that they render nicely in mdbook (see #10086), and so that we no longer have a mix of structured text languages in our documentation (excluding files outside of `docs/`).	2021-06-03 14:06:03 +01:00
Travis Ralston	5325f0308c	r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224 ) [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. This is progress towards https://github.com/matrix-org/synapse/issues/8334	2021-06-03 13:50:49 +01:00
Patrick Cloke	d7a646abca	Merge branch 'master' into develop	2021-06-03 08:43:38 -04:00
Patrick Cloke	57c01dca29	1.35.1	2021-06-03 08:18:22 -04:00
Patrick Cloke	36a7ff0c86	Do not show invite-only rooms in spaces summary (unless joined/invited). (#10109 )	2021-06-03 08:04:01 -04:00
Dirk Klimpel	0284d2a297	Add new admin APIs to remove media by media ID from quarantine. (#10044 ) Related to: #6681, #5956, #10040 Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-06-02 18:50:35 +01:00
Richard van der Hoff	bf6fd9f4fd	github actions: summarize Sytest results in an easy-to-read format (#10094 ) ... using the script from matrix-org/sytest#1052	2021-06-02 17:10:37 +01:00
Erik Johnston	fc3d2dc269	Rewrite the KeyRing (#10035 )	2021-06-02 16:37:59 +01:00
Patrick Cloke	3cf6b34b4e	Do not show invite-only rooms in spaces summary (unless joined/invited). (#10109 )	2021-06-02 11:31:41 -04:00
Erik Johnston	4deaebfe00	Make /sync do less state res (#10102 )	2021-06-02 15:48:17 +01:00
Andrew Morgan	a8372ad591	1.35.0	2021-06-01 13:23:55 +01:00
Erik Johnston	408ecf8ece	Announce deprecation of experimental `msc2858_enabled` option. (#10101 ) c.f. https://github.com/matrix-org/synapse/pull/9617 and https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md Fixes #9627.	2021-06-01 13:19:50 +01:00
Richard van der Hoff	b4b2fd2ece	add a cache to have_seen_event (#9953 ) Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.	2021-06-01 12:04:47 +01:00
Brad Murray	10e6d2abce	Fix opentracing inject to use the SpanContext, not the Span (#10074 ) Signed-off-by: Brad Murray brad@beeper.com	2021-06-01 08:40:26 +01:00
Erik Johnston	258a9a9e8b	1.35.0rc3	2021-05-28 17:06:05 +01:00
Erik Johnston	6b6c6a02db	Merge remote-tracking branch 'origin/release-v1.35.0' into release-v1.35	2021-05-28 17:03:14 +01:00
Brendan Abolivier	9408b86f5c	Limit the number of events sent over replication when persisting events. (#10082 )	2021-05-28 16:29:09 +01:00
Erik Johnston	1641c5c707	Log method and path when dropping request due to size limit (#10091 )	2021-05-28 16:29:09 +01:00
Erik Johnston	84cf3e47a0	Allow response of `/send_join` to be larger. (#10093 ) Fixes #10087.	2021-05-28 16:28:01 +01:00
Richard van der Hoff	ed53bf314f	Set opentracing priority before setting other tags (#10092 ) ... because tags on spans which aren't being sampled get thrown away.	2021-05-28 16:14:08 +01:00
Erik Johnston	3f96dbbda7	Log method and path when dropping request due to size limit (#10091 )	2021-05-28 15:57:53 +01:00
Eric Eastwood	ac3e02d089	Add `parse_strings_from_args` to get `prev_events` array (#10048 ) Split out from https://github.com/matrix-org/synapse/pull/9247 Strings: - `parse_string` - `parse_string_from_args` - `parse_strings_from_args` For comparison with ints: - `parse_integer` - `parse_integer_from_args` Previous discussions: - https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687 - https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791 Signed-off-by: Eric Eastwood <erice@element.io>	2021-05-28 14:19:06 +01:00
Erik Johnston	5eed6348ce	Move some more endpoints off master (#10084 )	2021-05-27 22:45:43 +01:00
Callum Brown	8fb9af570f	Make reason and score optional for report_event (#10077 ) Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414 See #8551 Signed-off-by: Callum Brown <callum@calcuode.com>	2021-05-27 18:42:23 +01:00
Brendan Abolivier	f828a70be3	Limit the number of events sent over replication when persisting events. (#10082 )	2021-05-27 17:10:58 +01:00
Erik Johnston	b1bc26a909	1.35.0rc2	2021-05-27 14:46:24 +01:00
Erik Johnston	78b5102ae7	Fix up `BatchingQueue` (#10078 ) Fixes #10068	2021-05-27 14:32:31 +01:00
Patrick Cloke	8e15c92c2f	Pass the origin when calculating the spaces summary over GET. (#10079 ) Fixes a bug due to conflicting PRs which were merged. (One added a new caller to a method, the other added a new parameter to the same method.)	2021-05-27 08:52:28 -04:00
Denis Kasak	d9f44fd0b9	Clarify security note regarding the domain Synapse is hosted on. (#9221 )	2021-05-27 11:41:16 +00:00
Richard van der Hoff	dcbfec919b	Improve the error message printed by synctl when synapse fails to start. (#10059 )	2021-05-27 10:35:06 +01:00
Richard van der Hoff	5447a76332	Remove redundant, unmaintained `convert_server_keys` script. (#10055 )	2021-05-27 10:34:55 +01:00
Richard van der Hoff	fe5dad46b0	Remove redundant code to reload tls cert (#10054 ) we don't need to reload the tls cert if we don't have any tls listeners. Follow-up to #9280.	2021-05-27 10:34:24 +01:00
Richard van der Hoff	224f2f949b	Combine `LruCache.invalidate` and `invalidate_many` (#9973 ) * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog	2021-05-27 10:33:56 +01:00
Patrick Cloke	f42e4c4eb9	Remove the experimental spaces enabled flag. (#10063 ) In lieu of just always enabling the unstable spaces endpoint and unstable room version.	2021-05-26 14:35:16 -04:00
Dan Callahan	49df2c28e3	Fix GitHub Actions lint for newsfragments (#10069 ) * Fix GitHub Actions lint for newsfragments Signed-off-by: Dan Callahan <danc@element.io>	2021-05-26 14:14:43 +01:00
Aaron Raimist	f95e7a03fa	Tweak wording of database recommendation in INSTALL.md (#10057 ) * Tweak wording of database recommendation in INSTALL.md Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-05-26 13:29:02 +01:00
Dan Callahan	913a761a53	Tell CircleCI to build Docker images from `main` (#9906 ) The `only` field takes a string or list of strings per the Circle docs: https://circleci.com/docs/2.0/configuration-reference/#branches Signed-off-by: Dan Callahan <danc@element.io>	2021-05-26 13:16:06 +01:00
Dirk Klimpel	65e6c64d83	Add an admin API for unprotecting local media from quarantine (#10040 ) Signed-off-by: Dirk Klimpel dirk@klimpel.org	2021-05-26 11:19:47 +01:00
Aaron Raimist	3e1beb75e6	Update CAPTCHA documentation to mention turning off verify origin feature (#10046 ) * Update CAPTCHA documentation to mention turning off verify origin Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-05-26 10:55:30 +01:00
Erik Johnston	557635f69a	1.35.0rc1	2021-05-25 11:00:13 +01:00
Patrick Cloke	7d90d6ce9b	Run complement with Synapse workers manually. (#10039 ) Adds an option to complement.sh to run Synapse in worker mode (instead of the default monolith mode).	2021-05-24 15:32:45 -04:00
Patrick Cloke	7adcb20fc0	Add missing type hints to synapse.util (#9982 )	2021-05-24 15:32:01 -04:00
Sergio Miguéns	22a8838f62	Fix docker image to not log at `/homeserver.log` (#10045 ) Fixes #9970 Signed-off-by: Sergio Miguéns Iglesias lonyelon@lony.xyz	2021-05-24 20:23:54 +01:00
Jerin J Titus	057ce7b754	Remove tls_fingerprints option (#9280 ) Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>	2021-05-24 17:43:30 +01:00
Richard van der Hoff	82eacb0e07	Fix --no-daemonize for synctl with workers (#9995 )	2021-05-24 14:03:30 +01:00
Richard van der Hoff	daca7b2794	Fix off-by-one-error in synapse_port_db (#9991 ) fixes #9979	2021-05-24 14:03:00 +01:00
Richard van der Hoff	c0df6bae06	Remove `keylen` from `LruCache`. (#9993 ) `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.	2021-05-24 14:02:01 +01:00
Patrick Cloke	316f89e87f	Enable experimental spaces by default. (#10011 ) The previous spaces_enabled flag now defaults to true and is exposed in the sample config.	2021-05-24 08:57:14 -04:00
Dirk Klimpel	387c297489	Add missing entry to the table of contents of room admin API (#10043 )	2021-05-24 07:37:30 -04:00
Eric Eastwood	5f1198a67e	Fix `get_state_ids_for_event` return type typo to match what the function actually does (#10050 ) It looks like a typo copy/paste from `get_state_for_event` above.	2021-05-24 10:43:33 +01:00
Erik Johnston	3e831f24ff	Don't hammer the database for destination retry timings every ~5mins (#10036 )	2021-05-21 17:57:08 +01:00
Michael Telatynski	e8ac9ac8ca	Fix /upload 500'ing when presented a very large image (#10029 ) * Fix /upload 500'ing when presented a very large image Catch DecompressionBombError and re-raise as ThumbnailErrors * Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml to get it to bomb out quicker, to load less into memory in the case of super large images * Add changelog entry for 10029	2021-05-21 18:31:59 +02:00
Andrew Morgan	21bd230831	Add a test for update_presence (#10033 ) https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale. The bug was fixed in #10014, and this PR just adds a test that failed on the old code, and was used to initially verify the bug. The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.	2021-05-21 17:29:14 +01:00
Patrick Cloke	c5413d0e9e	Remove unused properties from the SpaceSummaryHandler. (#10038 )	2021-05-21 12:02:01 -04:00
Marek Matys	6a8643ff3d	Fixed removal of new presence stream states (#10014 ) Fixes: https://github.com/matrix-org/synapse/issues/9962 This is a fix for above problem. I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts. Signed-off-by: Marek Matys <themarcq@gmail.com>	2021-05-21 12:02:06 +01:00
Erik Johnston	7958eadcd1	Add a batching queue implementation. (#10017 )	2021-05-21 11:20:51 +01:00
Erik Johnston	1c6a19002c	Add `Keyring.verify_events_for_server` and reduce memory usage (#10018 ) Also add support for giving a callback to generate the JSON object to verify. This should reduce memory usage, as we no longer have the event in memory in dict form (which has a large memory footprint) for extend periods of time.	2021-05-20 16:25:11 +01:00
Erik Johnston	64887f06fc	Use ijson to parse the response to `/send_join`, reducing memory usage. (#9958 ) Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and then parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.	2021-05-20 16:11:48 +01:00
Patrick Cloke	551d2c3f4b	Allow a user who could join a restricted room to see it in spaces summary. (#9922 ) This finishes up the experimental implementation of MSC3083 by showing the restricted rooms in the spaces summary (from MSC2946).	2021-05-20 11:10:36 -04:00
Andrew Morgan	d983ced596	Merge branch 'master' of github.com:matrix-org/synapse into develop	2021-05-20 14:39:46 +01:00
Javier Junquera Sánchez	141b073c7b	Update user_directory.md (#10016 ) Signed-off-by: Javier Junquera Sánchez <javier@junquera.io>	2021-05-20 14:24:19 +01:00
Erik Johnston	9c76d0561b	Update the contrib grafana dashboard (#10001 )	2021-05-19 11:47:16 +01:00
Savyasachee Jha	5bba1b4905	Hardened systemd unit files (#9803 ) Signed-off-by: Savyasachee Jha savya.jha@hawkradius.com	2021-05-19 11:44:16 +01:00
Patrick Cloke	ac6bfcd52f	Refactor checking restricted join rules (#10007 ) To be more consistent with similar code. The check now automatically raises an AuthError instead of passing back a boolean. It also absorbs some shared logic between callers.	2021-05-18 12:17:04 -04:00
Andrew Morgan	4d6e5a5e99	Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823 )	2021-05-18 14:13:45 +01:00
Patrick Cloke	206a7b5f12	Fix the allowed range of valid ordering characters for spaces. (#10002 ) \x7F was meant to be \0x7E (~) this was originally incorrect in MSC1772.	2021-05-17 09:59:17 -04:00
Patrick Cloke	9752849e2b	Clarify comments in the space summary handler. (#9974 )	2021-05-17 09:01:19 -04:00
Brendan Abolivier	41ac128fd3	Split multiplart email sending into a dedicated handler (#9977 ) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2021-05-17 12:33:38 +02:00
Richard van der Hoff	6660912226	Update postgres docs (#9989 )	2021-05-14 13:14:48 +01:00
Richard van der Hoff	6482075c95	Run `black` on the scripts (#9981 ) Turns out these scripts weren't getting linted.	2021-05-14 11:46:35 +01:00
Richard van der Hoff	5090f26b63	Minor `@cachedList` enhancements (#9975 ) - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.	2021-05-14 11:12:36 +01:00
Dan Callahan	52ed9655ed	Remove unnecessary SystemRandom from SQLBaseStore (#9987 ) It's not obvious that instances of SQLBaseStore each need their own instances of random.SystemRandom(); let's just use random directly. Introduced by `52839886d6` Signed-off-by: Dan Callahan <danc@element.io>	2021-05-14 10:59:10 +01:00
Dan Callahan	ebdef256b3	Remove superfluous call to bool() (#9986 ) Our strtobool already returns a bool, so no need to re-cast here Signed-off-by: Dan Callahan <danc@element.io>	2021-05-14 10:58:57 +01:00
Dan Callahan	bd918d874f	Simplify exception handling in is_ascii. (#9985 ) We can get away with just catching UnicodeError here. ⋮ +-- ValueError \| +-- UnicodeError \| +-- UnicodeDecodeError \| +-- UnicodeEncodeError \| +-- UnicodeTranslateError ⋮ https://docs.python.org/3/library/exceptions.html#exception-hierarchy Signed-off-by: Dan Callahan <danc@element.io>	2021-05-14 10:58:52 +01:00
Dan Callahan	498084228b	Use Python's secrets module instead of random (#9984 ) Functionally identical, but more obviously cryptographically secure. ...Explicit is better than implicit? Avoids needing to know that SystemRandom() implies a CSPRNG, and complies with the big scary red box on the documentation for random: > Warning: > The pseudo-random generators of this module should not be used for > security purposes. For security or cryptographic uses, see the > secrets module. https://docs.python.org/3/library/random.html Signed-off-by: Dan Callahan <danc@element.io>	2021-05-14 10:58:46 +01:00
Richard van der Hoff	c14f99be46	Support enabling opentracing by user (#9978 ) Add a config option which allows enabling opentracing by user id, eg for debugging requests made by a test user.	2021-05-14 10:51:08 +01:00
Andrew Morgan	976216959b	Update minimum supported version in postgres.md (#9988 )	2021-05-14 09:21:00 +01:00
Patrick Cloke	d19bccdbec	Update SSO mapping providers documentation about unique IDs. (#9980 )	2021-05-13 14:37:20 -04:00
Brendan Abolivier	47806b0869	1.34.0rc1	2021-05-12 16:59:46 +01:00
Brendan Abolivier	a683028d81	Correctly ratelimit invites when creating a room (#9968 ) * Correctly ratelimit invites when creating a room Also allow ratelimiting for more than one action at a time.	2021-05-12 16:05:28 +02:00
Richard van der Hoff	7562d887e1	Change the format of access tokens away from macaroons (#5588 )	2021-05-12 15:04:51 +01:00
Erik Johnston	affaffb0ab	Run cache_joined_hosts_for_event in background (#9951 )	2021-05-12 13:17:11 +01:00
Richard van der Hoff	63fb220e5f	Tests for to-device messages (#9965 )	2021-05-11 18:01:11 +01:00
Patrick Cloke	27c375f812	Sort child events according to MSC1772 for the spaces summary API. (#9954 ) This should help ensure that equivalent results are achieved between homeservers querying for the summary of a space. This implements modified MSC1772 rules, according to MSC2946. The different is that the origin_server_ts of the m.room.create event is not used as a tie-breaker since this might not be known if the homeserver is not part of the room.	2021-05-11 12:57:39 -04:00
Patrick Cloke	f4833e0c06	Support fetching the spaces summary via GET over federation. (#9947 ) Per changes in MSC2946, the C-S and S-S APIs for spaces summary should use GET requests. Until this is stable, the POST endpoints still exist. This does not switch federation requests to use the GET version yet since it is newly added and already deployed servers might not support it. When switching to the stable endpoint we should switch to GET requests.	2021-05-11 12:21:43 -04:00
Patrick Cloke	28c6841102	Send the `m.room.create` stripped event with invites (support MSC1772). (#9966 ) MSC1772 specifies the m.room.create event should be sent as part of the invite_state. This was done optionally behind an experimental flag, but is now done by default due to MSC1772 being approved.	2021-05-11 10:58:58 -04:00
Brendan Abolivier	652a6b094d	Merge branch 'master' into develop	2021-05-11 14:15:30 +01:00
Aaron Raimist	dc6366a9bd	Add config option to hide device names over federation (#9945 ) Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers. Signed-off-by: Aaron Raimist <aaron@raim.ist>	2021-05-11 14:03:23 +01:00
Brendan Abolivier	86fb71431c	1.33.2	2021-05-11 14:01:32 +01:00
Richard van der Hoff	b378d98c8f	Add debug logging for issue #9533 (#9959 ) Hopefully this will help us track down where to-device messages are getting lost/delayed.	2021-05-11 11:04:03 +01:00
Richard van der Hoff	7967b36efe	Fix `m.room_key_request` to-device messages (#9961 ) fixes #9960	2021-05-11 11:02:56 +01:00
Erik Johnston	2b2985b5cf	Improve performance of backfilling in large rooms. (#9935 ) We were pulling the full auth chain for the room out of the DB each time we backfilled, which can be huge for large rooms and is totally unnecessary.	2021-05-10 13:29:02 +01:00
Richard van der Hoff	51065c44bb	Fix port_db on empty db (#9930 ) ... and test it.	2021-05-10 13:02:55 +01:00
Erik Johnston	6c84778549	Always cache 'event_to_prev_state_group' (#9950 ) Fixes regression in send PDU times introduced in #9905.	2021-05-07 14:54:09 +01:00
Richard van der Hoff	765473567c	Fix make_full_schema to create the db with the right options and user (#9931 )	2021-05-07 14:01:57 +01:00
Erik Johnston	b65ecaff9b	Merge remote-tracking branch 'origin/release-v1.33.2' into develop	2021-05-07 13:27:19 +01:00
Erik Johnston	4df26abf28	Unpin attrs dep after new version has been released (#9946 ) c.f. #9936	2021-05-07 12:57:21 +01:00
Richard van der Hoff	25f43faa70	Reorganise the database schema directories (#9932 ) The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.	2021-05-07 10:22:05 +01:00
Erik Johnston	8771b1337d	Export jemalloc stats to prometheus when used (#9882 )	2021-05-06 15:54:07 +01:00
Erik Johnston	eba431c539	Revert "Leave out optional keys from /sync (#9919 )" (#9940 ) This reverts commit `e9eb3549d3`.	2021-05-06 15:06:35 +01:00
Erik Johnston	ac88aca7f7	1.33.1	2021-05-06 14:06:38 +01:00
Erik Johnston	24f07a83e6	Pin attrs to <21.1.0 (#9937 ) Fixes #9936	2021-05-06 14:06:06 +01:00
Patrick Cloke	d783880083	Include the time of the create event in Spaces Summary. (#9928 ) This is an update based on changes to MSC2946. The origin_server_ts of the m.room.create event is copied into the creation_ts field for each room returned from the spaces summary.	2021-05-05 13:33:05 -04:00
Erik Johnston	37623e3382	Increase perf of handling presence when joining large rooms. (#9916 )	2021-05-05 17:27:05 +01:00
Patrick Cloke	e2a443550e	Support stable MSC1772 spaces identifiers. (#9915 ) Support both the unstable and stable identifiers. A future release will disable the unstable identifiers.	2021-05-05 11:56:51 -04:00
Erik Johnston	ef889c98a6	Optionally track memory usage of each LruCache (#9881 ) This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.	2021-05-05 16:54:36 +01:00
Erik Johnston	1fb9a2d0bf	Limit how often GC happens by time. (#9902 ) Synapse can be quite memory intensive, and unless care is taken to tune the GC thresholds it can end up thrashing, causing noticable performance problems for large servers. We fix this by limiting how often we GC a given generation, regardless of current counts/thresholds. This does not help with the reverse problem where the thresholds are set too high, but that should only happen in situations where they've been manually configured. Adds a `gc_min_seconds_between` config option to override the defaults. Fixes #9890.	2021-05-05 16:53:45 +01:00
Erik Johnston	de8f0a03a3	Don't set the external cache if its been done recently (#9905 )	2021-05-05 16:53:22 +01:00
Erik Johnston	d0aee697ac	Use get_current_users_in_room from store and not StateHandler (#9910 )	2021-05-05 16:49:34 +01:00
Christopher May-Townsend	d5305000f1	Docker healthcheck timings - add startup delay and changed interval (#9913 ) * Add healthcheck startup delay by 5secs and reduced interval check to 15s to reduce waiting time for docker aware edge routers bringing an instance online	2021-05-05 17:33:04 +02:00
DeepBlueV7.X	e9eb3549d3	Leave out optional keys from /sync (#9919 ) This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.) Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>	2021-05-05 14:37:56 +01:00
Brendan Abolivier	0644ac0989	1.33.0	2021-05-05 14:15:54 +01:00
Erik Johnston	e3bc4617fc	Time external cache response time (#9904 )	2021-05-04 15:14:22 +01:00
Andrew Morgan	b85821aca2	Add port parameter to the sample config for psycopg2 args (#9911 ) Adds the `port` option with the default value to the sample config file.	2021-05-04 13:28:59 +01:00
Dan Callahan	56c4b47df3	Build Debian packages for Ubuntu 21.04 Hirsute (#9909 ) Signed-off-by: Dan Callahan <danc@element.io>	2021-04-30 15:36:05 +01:00
Andrew Morgan	4d624f467a	Synapse 1.33.0rc2 (2021-04-29) ============================== Bugfixes -------- - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900)) -----BEGIN PGP SIGNATURE----- iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCKtVATHGFuZHJld0Bh bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ok1D/40l93bxGBFtSNEHlqem7lZRLY2On9j AUnzIMqtJTLUgXf67vKIYqtTWkRjTdLIhIuINwhRmB0K23vcRrONnnyRLdhUC4Hl v6qLlPBHzupAJQPRcy3FGIVwsloAZ3JqVhV8AG0gXBRA0Dc34qYXh1DlA8EQ8mW/ l2umQIJI562xLNss7c5tpbCQxi7mq6OmTZ+Nwh45VRDavKEQQzSb9sPfs5w8IGLA ViYosUoXoXEWus/j60E38LSA/v1YcEE+Uyf5+LieV4b2c305xKj8gIbZqjLHR2K5 U9IrUp4QlYpPiebfGn7EmiPnOnlWdR9soIVZMzJ/m29tlyRSL5MBagtWA2fFlXo6 PywW1R/VsnLTdOuWY/t+et11iI4lVa7VSpywEcbMKGgmGKexDnfB10K7jHw0YICq 0pyLgy8JWyGUk/xpH2W1pvak8vyyN/zI/H09bDgGu721Qr/EPiHWF6RADNASpUVW w10SnO1udVc8ajugNwpX2oOQA7i5uiEuTdyFUHYfZ/aSpaCvdxM+rL3v0K955bNc /AGjSbs9ITe+JJQDiAJsWCYc+XLkR4h5KHVaZm2XlusbYb4iobK1DflkWXnQFjsi rlTcUH3sIFot3i9mjBBvXdvmzuI9qyx81/Sh8gRunPYT5TPhwaXKfyMBKE93/kif 4bKoHIXVcJvAqA== =sboq -----END PGP SIGNATURE----- Merge tag 'v1.33.0rc2' into develop Synapse 1.33.0rc2 (2021-04-29) ============================== Bugfixes -------- - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900))	2021-04-29 14:35:14 +01:00
Patrick Cloke	bb4b11846f	Add missing type hints to handlers and fix a Spam Checker type hint. (#9896 ) The user_may_create_room_alias method on spam checkers declared the room_alias parameter as a str when in reality it is passed a RoomAlias object.	2021-04-29 07:17:28 -04:00
Andrew Morgan	e9444cc74d	1.33.0rc2	2021-04-29 11:45:37 +01:00
ThibF	0085dc5abc	Delete room endpoint (#9889 ) Support the delete of a room through DELETE request and mark previous request as deprecated through documentation. Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>	2021-04-29 10:31:45 +01:00
Erik Johnston	802560211a	Merge remote-tracking branch 'origin/release-v1.33.0' into develop	2021-04-28 14:43:10 +01:00
Erik Johnston	e4ab8676b4	Fix tight loop handling presence replication. (#9900 ) Only affects workers. Introduced in #9819. Fixes #9899.	2021-04-28 14:42:50 +01:00
Patrick Cloke	10a08ab88a	Use the parent's logging context name for runWithConnection. (#9895 ) This fixes a regression where the logging context for runWithConnection was reported as runWithConnection instead of the connection name, e.g. "POST-XYZ".	2021-04-28 07:44:52 -04:00
Andrew Morgan	fa6679e794	Synapse 1.33.0rc1 (2021-04-28) ============================== Features -------- - Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814)) - Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850)) - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832)) Bugfixes -------- - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726)) - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788)) - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802)) - Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833)) - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867)) - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868)) Improved Documentation ---------------------- - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801)) Internal Changes ---------------- - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162)) - Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786)) - Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796)) - Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815)) - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816)) - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817)) - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821)) - Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838)) - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845)) - Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855)) - Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856)) - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858)) - Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871)) - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874)) - Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875)) - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876)) - Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878)) - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879)) - Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887)) -----BEGIN PGP SIGNATURE----- iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCJQmgTHGFuZHJld0Bh bW9yZ2FuLnh5egAKCRCIhIgNLv5f9EKWD/9MZG4ngjECrk3QMBlYqaipGl/l7wcl 8vBOTxiEqIgVJxHLCRFBbtFOItdxB4YkrorIo2fqshkk+lv4CRtTD9jEKHRvS4T6 7p1icRdRTv2K94OkJ8R9jMFlmywZFU87oHPfI2xUYg6hvOKrR+RwPvIjA7c24UZt 6MJqDhgGDlZD7/hQdQof9O4oOJzzIgLJPk6o8E42y6c0bLlPPKgH3sh0vlenDLfE 15thRCOeiP237YYvXSdbr7G3PI66Efhq4BwQowSrgFg+B0BR68l4747iSIeWmLQJ Ow6QLLFPCOhgAPC4amp9PdaaV/9NYiBcvNxlOvyQAVl/+ioEiATNHnzdNtDouCQo nDdSHw0Mt9D3i+rxpu5Pf0gZN9dXRGiczqnq5QKL8+EvT/4FLYeluEeduuy3rE+G o5OTCd3EajxynzjftuopeysNAw6zeDpbulZoTCeCumxBL2+wAod2PsyY5Ei9Gapn iJvExNOJX4OlkFc67jO2CK8o3sUTNEDeIDWCQ0fVKAwIt7T45ebTA/UzDNWg7YzN EyUp+3NZcUZBskgMB5hpcijPJoXNYzWZPews73vMPV7AfQRxDzU5xrM9AdbadBDd Idv6wT9ssDUA+M0aKnafvoSJZ+qE85mi2x0rRsueZNd9uO9/QHIXKIb+4PvVpj3C BJnZf34m568AGQ== =smAk -----END PGP SIGNATURE----- Merge tag 'v1.33.0rc1' into develop Synapse 1.33.0rc1 (2021-04-28) ============================== Features -------- - Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814)) - Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850)) - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832)) Bugfixes -------- - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726)) - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788)) - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802)) - Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833)) - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867)) - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868)) Improved Documentation ---------------------- - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801)) Internal Changes ---------------- - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162)) - Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786)) - Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796)) - Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815)) - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816)) - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817)) - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821)) - Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838)) - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845)) - Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855)) - Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856)) - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858)) - Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871)) - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874)) - Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875)) - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876)) - Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878)) - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879)) - Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))	2021-04-28 12:12:29 +01:00
Erik Johnston	391bfe9a7b	Reduce memory footprint of caches (#9886 )	2021-04-28 11:59:28 +01:00
Andrew Morgan	787de3190f	1.33.0rc1	2021-04-28 11:43:33 +01:00
Andrew Morgan	4e0fd35bc9	Revert "Experimental Federation Speedup (#9702 )" This reverts commit `05e8c70c05`.	2021-04-28 11:38:33 +01:00
Erik Johnston	dd2d32dcdb	Add type hints to presence handler (#9885 )	2021-04-28 11:07:47 +01:00
Andrew Morgan	fe604a022a	Remove various bits of compatibility code for Python <3.6 (#9879 ) I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.	2021-04-27 13:13:07 +01:00
Patrick Cloke	1350b053da	Pass errors back to the client when trying multiple federation destinations. (#9868 ) This ensures that something like an auth error (403) will be returned to the requester instead of attempting to try more servers, which will likely result in the same error, and then passing back a generic 400 error.	2021-04-27 07:30:34 -04:00
Erik Johnston	0ffa5fb935	Use current state table for `presence.get_interested_remotes` (#9887 ) This should be a lot quicker than asking the state handler.	2021-04-27 10:09:41 +01:00
Richard van der Hoff	3ff2251754	Improved validation for received requests (#9817 ) * Simplify `start_listening` callpath * Correctly check the size of uploaded files	2021-04-23 19:20:44 +01:00
Richard van der Hoff	84936e2264	Kill off `_PushHTTPChannel`. (#9878 ) First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation. Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.	2021-04-23 18:40:57 +01:00
Andrew Morgan	695b73c861	Allow OIDC cookies to work on non-root public baseurls (#9726 ) Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574. As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.	2021-04-23 18:22:47 +01:00
Richard van der Hoff	59d24c5bef	pass a reactor into SynapseSite (#9874 )	2021-04-23 17:06:47 +01:00
Patrick Cloke	e83627926f	Add type hints to auth and auth_blocking. (#9876 )	2021-04-23 12:02:16 -04:00
Erik Johnston	a15c003e5b	Make DomainSpecificString an attrs class (#9875 )	2021-04-23 15:46:29 +01:00
Andrew Morgan	ceaa76970f	Remove room and user invite ratelimits in default unit test config (#9871 )	2021-04-23 13:37:48 +01:00
Erik Johnston	9d25a0ae65	Split presence out of master (#9820 )	2021-04-23 12:21:55 +01:00
Patrick Cloke	d924827da1	Check for space membership during a remote join of a restricted room (#9814 ) When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.	2021-04-23 07:05:51 -04:00
Erik Johnston	3853a7edfc	Only store data in caches, not "smart" objects (#9845 )	2021-04-23 11:47:07 +01:00
Richard van der Hoff	51a20914a8	Limit the size of HTTP responses read over federation. (#9833 )	2021-04-23 11:08:41 +01:00
manuroe	c1ddbbde4f	Handle all new rate limits in demo scripts (#9858 )	2021-04-22 17:49:42 +01:00
Erik Johnston	177dae2704	Limit length of accepted email addresses (#9855 )	2021-04-22 17:49:11 +01:00
Richard van der Hoff	69018acbd2	Clear the resync bit after resyncing device lists (#9867 ) Fixes #9866.	2021-04-22 16:53:24 +01:00
Richard van der Hoff	294c675033	Remove `synapse.types.Collection` (#9856 ) This is no longer required, since we have dropped support for Python 3.5.	2021-04-22 16:43:50 +01:00
Patrick Cloke	683d6f75af	Rename handler and config modules which end in handler/config. (#9816 )	2021-04-20 14:55:20 -04:00
Erik Johnston	de0d088adc	Add presence federation stream (#9819 )	2021-04-20 14:11:24 +01:00
Erik Johnston	db70435de7	Fix bug where we sent remote presence states to remote servers (#9850 )	2021-04-20 13:37:54 +01:00
Jonathan de Jong	495b214f4f	Fix (final) Bugbear violations (#9838 )	2021-04-20 11:50:49 +01:00
Andrew Morgan	71f0623de9	Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74 " from synapse-dinsic (#9832 ) This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).	2021-04-19 19:16:34 +01:00

... 5 6 7 8 9 ...

4860 Commits