Commit Graph

16729 Commits

Author SHA1 Message Date
Richard van der Hoff
ce428a1abe Fix EventsStream raising assertions when it falls behind
Figuring out how to correctly limit updates from this stream without dropping
entries is far more complicated than just counting the number of rows being
returned. We need to consider each query separately and, if any one query hits
the limit, truncate the results from the others.

I think this also fixes some potentially long-standing bugs where events or
state changes could get missed if we hit the limit on either query.
2020-04-24 13:59:21 +01:00
Richard van der Hoff
9cbdfb3a2f Make it clear that the limit for an update_function is a target 2020-04-23 15:45:12 +01:00
Richard van der Hoff
23b28266ac Remove 'limit' param from get_repl_stream_updates API
there doesn't seem to be much point in passing this limit all around, since
both sides agree it's meant to be 100.
2020-04-23 15:44:35 +01:00
Brendan Abolivier
2e3b9a0fcb
Revert "Revert "Merge pull request #7315 from matrix-org/babolivier/request_token""
This reverts commit 1adf6a5587.
2020-04-23 11:23:53 +02:00
Brendan Abolivier
fb825759e3 Merge branch 'master' into develop 2020-04-23 11:23:33 +02:00
Brendan Abolivier
1adf6a5587
Revert "Merge pull request #7315 from matrix-org/babolivier/request_token"
This reverts commit 6f4319368b, reversing
changes made to 0d775fcc2d.
2020-04-23 11:23:10 +02:00
Brendan Abolivier
6f4319368b
Merge pull request #7315 from matrix-org/babolivier/request_token
Config option to inhibit 3PID errors on /requestToken
2020-04-23 10:38:57 +02:00
Richard van der Hoff
71a1abb8a1
Stop the master relaying USER_SYNC for other workers (#7318)
Long story short: if we're handling presence on the current worker, we shouldn't be sending USER_SYNC commands over replication.

In an attempt to figure out what is going on here, I ended up refactoring some bits of the presencehandler code, so the first 4 commits here are non-functional refactors to move this code slightly closer to sanity. (There's still plenty to do here :/). Suggest reviewing individual commits.

Fixes (I hope) #7257.
2020-04-22 22:39:04 +01:00
Brendan Abolivier
69ad7cc13b
Config option to inhibit 3PID errors on /requestToken
Adds a request_token_inhibit_errors configuration flag (disabled by
default) which, if enabled, change the behaviour of all /requestToken
endpoints so that they return a 200 and a fake sid if the 3PID was/was
not found associated with an account (depending on the endpoint),
instead of an error.

Co-Authored-By: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2020-04-22 23:38:42 +02:00
Erik Johnston
841c581c40
Fix replication metrics when using redis (#7325) 2020-04-22 16:26:19 +01:00
Richard van der Hoff
f16beaa969 Synapse 1.12.4rc1 (2020-04-22)
==============================
 
 Features
 --------
 
 - Always send users their own device updates. ([\#7160](https://github.com/matrix-org/synapse/issues/7160))
 - Add support for handling GET requests for `account_data` on a worker. ([\#7311](https://github.com/matrix-org/synapse/issues/7311))
 
 Bugfixes
 --------
 
 - Fix a bug that prevented cross-signing with users on worker-mode synapses. ([\#7255](https://github.com/matrix-org/synapse/issues/7255))
 - Do not treat display names as globs in push rules. ([\#7271](https://github.com/matrix-org/synapse/issues/7271))
 - Fix a bug with cross-signing devices belonging to remote users who did not share a room with any user on the local homeserver. ([\#7289](https://github.com/matrix-org/synapse/issues/7289))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEv27Axt/F4vrTL/8QOSor00I9eP8FAl6gS+IACgkQOSor00I9
 eP8CTwf+KLehHQMcrz0yoHN+/O86mpNv3Uz83BJ0qDIgezVbtqT7+ZwnK2Jb8dwM
 zTcGRq5a6V+JLdfSTnXfrPzvAPt0WLaEENSAEGwZ6unqqxcG2PgATkRb0XtY8Rzh
 Feu8WgJTnORbiiM3sBLI8toZMxal3X3ZcftPEAan1SaoyqLPQ2s14edDwgx64S5e
 rBGzPDNSxZU1McNBq7Q11QTg3zgBUF+TYwyhnLbC5X1RdDwQpKhNZhwhYBS1yOD6
 TsPjtlixC+HXb21Rmob7XY8VFxTBSBdUdtVQKAT4h3+l9bNFQLpBfDSVc7xkPiLt
 /Gj+PL+YgY9xQDvHO2fVw8m4ololZw==
 =p53e
 -----END PGP SIGNATURE-----

Merge tag 'v1.12.4rc1' into develop

Synapse 1.12.4rc1 (2020-04-22)
==============================

Features
--------

- Always send users their own device updates. ([\#7160](https://github.com/matrix-org/synapse/issues/7160))
- Add support for handling GET requests for `account_data` on a worker. ([\#7311](https://github.com/matrix-org/synapse/issues/7311))

Bugfixes
--------

- Fix a bug that prevented cross-signing with users on worker-mode synapses. ([\#7255](https://github.com/matrix-org/synapse/issues/7255))
- Do not treat display names as globs in push rules. ([\#7271](https://github.com/matrix-org/synapse/issues/7271))
- Fix a bug with cross-signing devices belonging to remote users who did not share a room with any user on the local homeserver. ([\#7289](https://github.com/matrix-org/synapse/issues/7289))
2020-04-22 14:54:08 +01:00
Richard van der Hoff
ba0aac5e44 formatting for the changelog 2020-04-22 14:50:51 +01:00
Richard van der Hoff
82d8b1dd1f
Another go at fixing one-word commands (#7326)
I messed this up last time I tried (#7239 / e13c6c7).
2020-04-22 14:34:31 +01:00
Richard van der Hoff
dc8003f921 1.12.4rc1 2020-04-22 14:29:39 +01:00
Richard van der Hoff
83af1079d6 1.12.4rc1 2020-04-22 14:28:23 +01:00
Richard van der Hoff
7c7618c7e3 fix changelog name 2020-04-22 14:01:11 +01:00
Richard van der Hoff
13683a3a22
Extend StreamChangeCache to support multiple entities per stream ID (#7303)
First some background: StreamChangeCache is used to keep track of what "entities" have 
changed since a given stream ID. So for example, we might use it to keep track of when the last
to-device message for a given user was received [1], and hence whether we need to pull any to-device messages from the database on a sync [2].

Now, it turns out that StreamChangeCache didn't support more than one thing being changed at
a given stream_id (this was part of the problem with #7206). However, it's entirely valid to send
to-device messages to more than one user at a time.

As it turns out, this did in fact work, because *some* methods of StreamChangeCache coped
ok with having multiple things changing on the same stream ID, and it seems we never actually
use the methods which don't work on the stream change caches where we allow multiple
changes at the same stream ID. But that feels horribly fragile, hence: let's update
StreamChangeCache to properly support this, and add some typing and some more tests while
we're at it.

[1]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L301
[2]: https://github.com/matrix-org/synapse/blob/release-v1.12.3/synapse/storage/data_stores/main/deviceinbox.py#L47-L51
2020-04-22 13:45:40 +01:00
Dirk Klimpel
6b6685db9f
Extend room admin api with additional attributes (#7225) 2020-04-22 13:38:41 +01:00
Richard van der Hoff
2aa5bf13c8 Merge branch 'release-v1.12.4' into develop 2020-04-22 13:09:23 +01:00
Erik Johnston
51f7eaf908
Add ability to run replication protocol over redis. (#7040)
This is configured via the `redis` config options.
2020-04-22 13:07:41 +01:00
Patrick Cloke
51f358e2fe Do not treat display names as globs for push rules. (#7271) 2020-04-22 13:07:12 +01:00
Michael Kaye
5308239d5d
Reduce logging verbosity of URL cache cleanup. (#7295) 2020-04-22 07:45:16 -04:00
Andrew Morgan
f89ad3b6df
Query missing cross-signing keys on local sig upload (#7289) 2020-04-22 12:29:36 +01:00
Andrew Morgan
ff5604e7f1
import urllib.parse when using urllib.parse.quote (#7319) 2020-04-22 12:18:51 +01:00
Michael Kaye
336989a57f
Reduce federation logging on success (#7321)
Splitting based on the response code means we can avoid double logging here and identical information from line 164 while still logging at info if we don't get a good response and need to retry.
2020-04-22 11:18:18 +01:00
Richard van der Hoff
556566f0b8 Fix changelog file
I updated the PR and forgot to update the changelog.
2020-04-21 13:20:16 +01:00
Richard van der Hoff
974c0d726a
Support GET account_data requests on a worker (#7311) 2020-04-21 10:46:30 +01:00
Richard van der Hoff
461f01ad43 Merge branch 'release-v1.12.4' into develop 2020-04-20 17:55:39 +01:00
Richard van der Hoff
d41c8f6d4d Revert "Query missing cross-signing keys on local sig upload"
This was incorrectly merged to the release branch before it was ready.

This reverts commit 72fe2affb6.
2020-04-20 17:54:35 +01:00
Richard van der Hoff
a46ff43319 Merge branch 'release-v1.12.4' into develop 2020-04-20 17:42:27 +01:00
David Baker
40f79f58bf Always send the user updates to their own device list (#7160) 2020-04-20 17:20:38 +01:00
Lars Franke
13917232d5
Fix indention in generated config file (#7300)
Also adjust sample_config.yaml

Signed-off-by: Lars Franke <frcl@mailbox.org>
2020-04-20 16:51:27 +01:00
Patrick Cloke
f5ea8b48bd
Reject unknown UI auth sessions (instead of silently generating a new one) (#7268) 2020-04-20 08:54:42 -04:00
Richard van der Hoff
0f8f02bc39
On catchup, process each row with its own stream id (#7286)
Other parts of the code (such as the StreamChangeCache) assume that there will
not be multiple changes with the same stream id.

This code was introduced in #7024, and I hope this fixes #7206.
2020-04-20 11:43:29 +01:00
Patrick Cloke
054c231e58
Use a template for the SSO success page to allow for customization. (#7279) 2020-04-17 13:34:55 -04:00
James
701788a227
Added explicit Python build tools to snap requirements (#7213)
Signed-off-by: James Hebden <james@ec0.io>
2020-04-17 17:28:00 +01:00
Andrew Morgan
72fe2affb6 Query missing cross-signing keys on local sig upload
Add changelog

Save retrieved keys to the db

lint

Fix and de-brittle remote result dict processing

Use query_user_devices instead, assume only master, self_signing key types

Make changelog more useful

Remove very specific exception handling

Wrap get_verify_key_from_cross_signing_key in a try/except

Note that _get_e2e_cross_signing_verify_key can raise a SynapseError

lint

Add comment explaining why this is useful

Only fetch master and self_signing key types

Fix log statements, docstrings

Remove extraneous items from remote query try/except

lint

Factor key retrieval out into a separate function

Send device updates, modeled after SigningKeyEduUpdater._handle_signing_key_updates

Update method docstring
2020-04-17 15:47:49 +01:00
Richard van der Hoff
67ff7b8ba0
Improve type checking in replication.tcp.Stream (#7291)
The general idea here is to get rid of the type: ignore annotations on all of the current_token and update_function assignments, which would have caught #7290.

After a bit of experimentation, it seems like the least-awful way to do this is to pass the offending functions in as parameters to the Stream constructor. Unfortunately that means that the concrete implementations no longer have the same constructor signature as Stream itself, which means that it gets hard to correctly annotate STREAMS_MAP.

I've also introduced a couple of new types, to take out some duplication.
2020-04-17 14:49:55 +01:00
nataraj-hates-MS-for-stealing-github
0d775fcc2d
Improve example TURN configuration in documentation (#7284) 2020-04-17 08:04:23 -04:00
Tristan Lins
c07fca9e2f
Clarify the comments for media_storage_providers options (#7272) 2020-04-17 07:09:33 -04:00
Patrick Cloke
01294e6b3a
Do not treat display names as globs for push rules. (#7271) 2020-04-16 10:52:55 -04:00
Richard van der Hoff
d7d42387f5
Fix 'generator object is not subscriptable' error (#7290)
Some of the query functions return generators rather than lists, so we can't
index into the result. Happily we already have a copy of the results.

(think this was introduced in #7024)
2020-04-16 14:37:06 +01:00
Patrick Cloke
eed7c5b89e
Convert auth handler to async/await (#7261) 2020-04-15 12:40:18 -04:00
Patrick Cloke
17a2433b0d
Add notes to the changelog about an additional SSO template. (#7259) 2020-04-15 10:36:33 -04:00
Andrew Morgan
a48138784e
Allow specifying the value of Accept-Language header for URL previews (#7265) 2020-04-15 13:35:29 +01:00
Richard van der Hoff
6fb63d6426
Add some warnings and notes about slow upgrade (#7275)
Fixes #7144.
2020-04-15 12:40:43 +01:00
Richard van der Hoff
f2049a8d21
Fix a potentially-huge sql query (#7274)
We could end up looking up tens of thousands of events, which could cause large
amounts of data to be logged to the postgres log.
2020-04-15 10:16:35 +01:00
Zay11Zay
f1097e7720
Fix the parameters of a test fixture (#7243) 2020-04-14 15:37:28 -04:00
Andrew Morgan
ac6a84818f Only register devices edu handler on the master process (#7255) 2020-04-14 11:36:24 +01:00
Andrew Morgan
fef82f4e22
Only register devices edu handler on the master process (#7255) 2020-04-14 10:09:58 +01:00