Commit Graph

17607 Commits

Author SHA1 Message Date
Erik Johnston
b1433bf231
Don't table scan events on worker startup (#8419)
* Fix table scan of events on worker startup.

This happened because we assumed "new" writers had an initial stream
position of 0, so the replication code tried to fetch all events written
by the instance between 0 and the current position.

Instead, set the initial position of new writers to the current
persisted up to position, on the assumption that new writers won't have
written anything before that point.

* Consider old writers coming back as "new".

Otherwise we'd try and fetch entries between the old stale token and the
current position, even though it won't have written any rows.

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-09-29 16:42:19 +01:00
Richard van der Hoff
2649d545a5
Mypy fixes for synapse.handlers.federation (#8422)
For some reason, an apparently unrelated PR upset mypy about this module. Here are a number of little fixes.
2020-09-29 15:57:36 +01:00
Andrew Morgan
12f0d18611
Add support for running Complement against the local checkout (#8317)
This PR adds a script that:

* Builds the local Synapse checkout using our existing `docker/Dockerfile` image.
* Downloads [Complement](https://github.com/matrix-org/complement/)'s source code.
* Builds the [Synapse.Dockerfile](https://github.com/matrix-org/complement/blob/master/dockerfiles/Synapse.Dockerfile) using the above dockerfile as a base.
* Builds and runs Complement against it.

This set up differs slightly from [that of the dendrite repo](https://github.com/matrix-org/dendrite/blob/master/build/scripts/complement.sh) (`complement.sh`, `Complement.Dockerfile`), which instead stores a separate, but slightly modified, dockerfile in Dendrite's repo rather than running the one stored in Complement's repo. That synapse equivalent to that dockerfile (`Synapse.Dockerfile`) in Complement's repo is just based on top of `matrixdotorg/synapse:latest`, which we opt to build here locally.

Thus copying over the files from Complement's repo wouldn't change any functionality, and would result in two instances of the same files. So just using the dockerfile in Complement's repo was decided upon instead.
2020-09-29 13:47:47 +01:00
Will Hunt
8676d8ab2e
Filter out appservices from mau count (#8404)
This is an attempt to fix #8403.
2020-09-29 13:11:02 +01:00
Andrew Morgan
1c6b8752b8
Only assert valid next_link params when provided (#8417)
Broken in https://github.com/matrix-org/synapse/pull/8275 and has yet to be put in a release. Fixes https://github.com/matrix-org/synapse/issues/8418.

`next_link` is an optional parameter. However, we were checking whether the `next_link` param was valid, even if it wasn't provided. In that case, `next_link` was `None`, which would clearly not be a valid URL.

This would prevent password reset and other operations if `next_link` was not provided, and the `next_link_domain_whitelist` config option was set.
2020-09-29 12:36:44 +01:00
Richard van der Hoff
866c84da8d
Add metrics to track success/otherwise of replication requests (#8406)
One hope is that this might provide some insights into #3365.
2020-09-29 11:06:11 +01:00
Richard van der Hoff
1c262431f9
Fix handling of connection timeouts in outgoing http requests (#8400)
* Remove `on_timeout_cancel` from `timeout_deferred`

The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a
timeout (in particular if the canceller raised an exception), so it was
unreliable. It was also only used in one place, and to be honest it's easier to
do what it does a different way.

* Fix handling of connection timeouts in outgoing http requests

Turns out that if we get a timeout during connection, then a different
exception is raised, which wasn't always handled correctly.

To fix it, catch the exception in SimpleHttpClient and turn it into a
RequestTimedOutError (which is already a documented exception).

Also add a description to RequestTimedOutError so that we can see which stage
it failed at.

* Fix incorrect handling of timeouts reading federation responses

This was trapping the wrong sort of TimeoutError, so was never being hit.

The effect was relatively minor, but we should fix this so that it does the
expected thing.

* Fix inconsistent handling of `timeout` param between methods

`get_json`, `put_json` and `delete_json` were applying a different timeout to
the response body to `post_json`; bring them in line and test.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2020-09-29 10:29:21 +01:00
Erik Johnston
bd380d942f
Add checks for postgres sequence consistency (#8402) 2020-09-28 18:00:30 +01:00
Richard van der Hoff
5e3ca12b15
Create a mechanism for marking tests "logcontext clean" (#8399) 2020-09-28 17:58:33 +01:00
Dagfinn Ilmari Mannsåker
bd715e1278
Add ui_auth_sessions_ips table to synapse_port_db ignore list (#8410)
This table was created in #8034 (1.20.0).  It references
`ui_auth_sessions`, which is ignored, so this one should be too.

Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2020-09-28 15:35:02 +01:00
Richard van der Hoff
450ec48445
A pair of tiny cleanups in the federation request code. (#8401) 2020-09-28 13:15:00 +01:00
Matthew Hodgson
4b3a1faa08 typo 2020-09-28 00:23:35 +01:00
Patrick Cloke
31acc5c309
Escape the error description on the sso_error template. (#8405) 2020-09-25 11:05:54 -04:00
Richard van der Hoff
fec6f9ac17
Fix occasional "Re-starting finished log context" from keyring (#8398)
* Fix test_verify_json_objects_for_server_awaits_previous_requests

It turns out that this wasn't really testing what it thought it was testing
(in particular, `check_context` was turning failures into success, which was
making the tests pass even though it wasn't clear they should have been.

It was also somewhat overcomplex - we can test what it was trying to test
without mocking out perspectives servers.

* Fix warnings about finished logcontexts in the keyring

We need to make sure that we finish the key fetching magic before we run the
verifying code, to ensure that we don't mess up our logcontexts.
2020-09-25 12:29:54 +01:00
Tdxdxoz
abd04b6af0
Allow existing users to login via OpenID Connect. (#8345)
Co-authored-by: Benjamin Koch <bbbsnowball@gmail.com>

This adds configuration flags that will match a user to pre-existing users
when logging in via OpenID Connect. This is useful when switching to
an existing SSO system.
2020-09-25 07:01:45 -04:00
Erik Johnston
3e87d79e1c
Fix schema delta for servers that have not backfilled (#8396)
Fixes #8395.
2020-09-25 09:58:32 +01:00
Andrew Morgan
c77c4a2fcd Merge branch 'master' into develop 2020-09-24 17:00:33 +01:00
Erik Johnston
f112cfe5bb
Fix MultiWriteIdGenerator's handling of restarts. (#8374)
On startup `MultiWriteIdGenerator` fetches the maximum stream ID for
each instance from the table and uses that as its initial "current
position" for each writer. This is problematic as a) it involves either
a scan of events table or an index (neither of which is ideal), and b)
if rows are being persisted out of order elsewhere while the process
restarts then using the maximum stream ID is not correct. This could
theoretically lead to race conditions where e.g. events that are
persisted out of order are not sent down sync streams.

We fix this by creating a new table that tracks the current positions of
each writer to the stream, and update it each time we finish persisting
a new entry. This is a relatively small overhead when persisting events.
However for the cache invalidation stream this is a much bigger relative
overhead, so instead we note that for invalidation we don't actually
care about reliability over restarts (as there's no caches to
invalidate) and simply don't bother reading and writing to the new table
in that particular case.
2020-09-24 16:53:51 +01:00
Andrew Morgan
ab903e7337 s/URLs/variables in changelog 2020-09-24 16:35:31 +01:00
Andrew Morgan
271086ebda s/accidentally/incorrectly in changelog 2020-09-24 16:33:49 +01:00
Andrew Morgan
5ce5a9f144 Update changelog wording 2020-09-24 16:26:57 +01:00
Andrew Morgan
920dd1083e 1.20.1 2020-09-24 16:25:33 +01:00
Patrick Cloke
f3e5c2e702 Mark the shadow_banned column as boolean in synapse_port_db. (#8386) 2020-09-24 16:24:24 +01:00
Andrew Morgan
3f4a2a7064
Hotfix: disable autoescape by default when rendering Jinja2 templates (#8394)
#8037 changed the default `autoescape` option when rendering Jinja2 templates from `False` to `True`. This caused some bugs, noticeably around redirect URLs being escaped in SAML2 auth confirmation templates, causing those URLs to break for users.

This change returns the previous behaviour as it stood. We may want to look at each template individually and see whether autoescaping is a good idea at some point, but for now lets just fix the breakage.
2020-09-24 16:24:08 +01:00
Richard van der Hoff
11c9e17738
Add type annotations to SimpleHttpClient (#8372) 2020-09-24 15:47:20 +01:00
Erik Johnston
6fdf577593
Add new sequences to port DB script (#8387) 2020-09-24 13:43:49 +01:00
Erik Johnston
ac11fcbbb8
Add EventStreamPosition type (#8388)
The idea is to remove some of the places we pass around `int`, where it can represent one of two things:

1. the position of an event in the stream; or
2. a token that partitions the stream, used as part of the stream tokens.

The valid operations are then:

1. did a position happen before or after a token;
2. get all events that happened before or after a token; and
3. get all events between two tokens.

(Note that we don't want to allow other operations as we want to change the tokens to be vector clocks rather than simple ints)
2020-09-24 13:24:17 +01:00
Patrick Cloke
13099ae431
Mark the shadow_banned column as boolean in synapse_port_db. (#8386) 2020-09-24 08:13:55 -04:00
Richard van der Hoff
2983049a77
Factor out _send_dummy_event_for_room (#8370)
this makes it possible to use from the manhole, and seems cleaner anyway.
2020-09-23 18:18:43 +01:00
Richard van der Hoff
91c60f3042
Improve logging of state resolution (#8371)
I'd like to get a better insight into what we are doing with respect to state
res. The list of state groups we are resolving across should be short (if it
isn't, that's a massive problem in itself), so it should be fine to log it in
ite entiretly.

I've done some grepping and found approximately zero cases in which the
"shortcut" code delivered the result, so I've ripped that out too.
2020-09-23 16:42:44 +01:00
Richard van der Hoff
302dc89f6a
Fix bug which caused failure on join with malformed membership events (#8385) 2020-09-23 16:42:14 +01:00
Erik Johnston
cbabb312e0
Use async with for ID gens (#8383)
This will allow us to hit the DB after we've finished using the generated stream ID.
2020-09-23 16:11:18 +01:00
Mathieu Velten
916bb9d0d1
Don't push if an user account has expired (#8353) 2020-09-23 16:06:28 +01:00
Andrew Morgan
4bb203ea4f
Fix missing null character check on guest_access room state (#8373)
When updating the `room_stats_state` table, we try to check for null bytes slipping in to the content for state events. It turns out we had added `guest_access` as a field to room_stats_state without including it in the null byte check.

Lo and behold, a null byte in a `m.room.guest_access` event then breaks `room_stats_state` updates.

This PR adds the check for `guest_access`.
2020-09-23 15:58:20 +01:00
Patrick Cloke
bbde4038df
Do not check lint/test dependencies at runtime. (#8377)
moves non-runtime dependencies out of synapse.python_dependencies (test and lint)
2020-09-23 11:45:37 +01:00
Julian Fietkau
a4e63e5a47
Add note to reverse_proxy.md about disabling Apache's mod_security2 (#8375)
This change adds a note and a few lines of configuration settings for Apache users to disable ModSecurity for Synapse's virtual hosts. With ModSecurity enabled and running with its default settings, Matrix clients are unable to send chat messages through the Synapse installation. With this change, ModSecurity can be disabled only for the Synapse virtual hosts.
2020-09-23 11:14:08 +01:00
Andrew Morgan
48336eeb85 Changelog 2020-09-22 19:39:57 +01:00
Andrew Morgan
4325be1a52 Fix missing null character check on guest_access room state
When updating room_stats_state, we try to check for null bytes slipping
in to the
content for state events. It turns out we had added guest_access as a
field to
room_stats_state without including it in the null byte check.

Lo and behold, a null byte in a m.room.guest_access event then breaks
room_stats_state
updates.

This PR adds the check for guest_access. A further PR will improve this
function so that this hopefully does not happen again in future.
2020-09-22 19:39:29 +01:00
Dirk Klimpel
8998217540
Fixed a bug with reactivating users with the admin API (#8362)
Fixes: #8359 

Trying to reactivate a user with the admin API (`PUT /_synapse/admin/v2/users/<user_name>`) causes an internal server error.

Seems to be a regression in #8033.
2020-09-22 18:19:01 +01:00
Dirk Klimpel
4da01f9c61
Admin API for reported events (#8217)
Add an admin API to read entries of table `event_reports`. API: `GET /_synapse/admin/v1/event_reports`
2020-09-22 18:15:04 +01:00
Andrew Morgan
b29a9bdaa9 Merge branch 'master' into develop 2020-09-22 16:00:30 +01:00
Andrew Morgan
d191dbdaa6 Fix wording of deprecation notice in changelog 2020-09-22 15:42:53 +01:00
Andrew Morgan
012736ff07 Deprecation warning for synapse admin api being accessible under /_matrix 2020-09-22 15:30:44 +01:00
Andrew Morgan
55bb5fda33 1.20.0 2020-09-22 15:18:31 +01:00
Dionysis Grigoropoulos
37ca5924bd
Create function to check for long names in devices (#8364)
* Create a new function to verify that the length of a device name is
under a certain threshold.
* Refactor old code and tests to use said function.
* Verify device name length during registration of device
* Add a test for the above

Signed-off-by: Dionysis Grigoropoulos <dgrig@erethon.com>
2020-09-22 11:42:55 +01:00
Richard van der Hoff
4f3096d866 Add a comment re #1691 2020-09-21 12:34:06 +01:00
Patrick Cloke
babc027543
Fix a bad merge from release-v1.20.0. (#8354) 2020-09-18 12:54:04 -04:00
Patrick Cloke
00db7786de Synapse 1.20.0rc5 (2020-09-18)
==============================
 
 In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.
 
 Features
 --------
 
 - Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343))
 
 Bugfixes
 --------
 
 - Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342))
 - Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349))
 
 Internal Changes
 ----------------
 
 - Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl9kzk8ACgkQM/xY9qcR
 MEim6A//aERkhyLGRlGpLd37lCyFQCeffTMH1rTvu04iIBQBaUZ6g7CYWOpK43zT
 U8kt379+5OShjdAXs/X4XP+ucdHVbrwsRSP3hBS/fFLiDT0fJgP8uiSf5QqO6NnT
 OqDyXYjcXvj/c6tMKglVtsdh8u4hFwNZjGPMGG68IzJu14uEhnD100cL9jSB9bLB
 ongWpsQzzdGBpJPSFRjv9dCUSeRbzyUdl1t0uqzrNqyN9s/JnzFTn7ZYo6y3lnSS
 dHGVMMo/12M2PkbBHnbJVvDY5Q/R7ZxyXlpz0gvSNOQIw8FqYFnuB0Niy5dQhXSR
 Sy5h4qbczLxqbql1x+lmzeQm4ZMORsW/Tl4C3z6yK6OYaOCJHIf9en4DplTSTqp1
 t+85JxWR2wH10d99YHBpaYKmkVovpwgchrO4YWrtXljUFAhhavzf+YiAdOHYT52s
 RDsDLsvjMbxEHsz4cHfycmshYhjzjb340wkoDXuQpj0zrO99d+Zd83xdK8pS0UQn
 OaljLRAd/5iBjTSyZPSrB1U5141OzlM3QZVJzaYAnP12yhR9eaX2twSCk+lPYOWd
 nhLJjNnj1B1XSGArthuE5NLyEiCPz6KyN2RhO0EOx5YjZN9TwH7LS9upyNFe1nN1
 GIhO5gz+jWLuBZE3xzRNjJyCx/I/LolpCwGMvKDu6638rpsbrPs=
 =tT5/
 -----END PGP SIGNATURE-----

Merge tag 'v1.20.0rc5' into develop

Synapse 1.20.0rc5 (2020-09-18)
==============================

In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.

Features
--------

- Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343))

Bugfixes
--------

- Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342))
- Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349))

Internal Changes
----------------

- Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
2020-09-18 11:17:58 -04:00
Patrick Cloke
c7e060bfee Add a note about including the changes from 1.19.3. 2020-09-18 11:10:59 -04:00
Patrick Cloke
c4e8b18c72 Tweak wording in the changelog. 2020-09-18 10:57:29 -04:00