Commit Graph

18766 Commits

Author SHA1 Message Date
Erik Johnston
3e831f24ff
Don't hammer the database for destination retry timings every ~5mins (#10036) 2021-05-21 17:57:08 +01:00
Michael Telatynski
e8ac9ac8ca
Fix /upload 500'ing when presented a very large image (#10029)
* Fix /upload 500'ing when presented a very large image

Catch DecompressionBombError and re-raise as ThumbnailErrors

* Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml

to get it to bomb out quicker, to load less into memory
in the case of super large images

* Add changelog entry for 10029
2021-05-21 18:31:59 +02:00
Andrew Morgan
21bd230831
Add a test for update_presence (#10033)
https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale.

The bug was fixed in #10014, and this PR just adds a test that failed on the old code, and was used to initially verify the bug.

The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.
2021-05-21 17:29:14 +01:00
Patrick Cloke
c5413d0e9e
Remove unused properties from the SpaceSummaryHandler. (#10038) 2021-05-21 12:02:01 -04:00
Marek Matys
6a8643ff3d
Fixed removal of new presence stream states (#10014)
Fixes: https://github.com/matrix-org/synapse/issues/9962

This is a fix for above problem.

I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts.

Signed-off-by: Marek Matys <themarcq@gmail.com>
2021-05-21 12:02:06 +01:00
Erik Johnston
7958eadcd1
Add a batching queue implementation. (#10017) 2021-05-21 11:20:51 +01:00
Erik Johnston
1c6a19002c
Add Keyring.verify_events_for_server and reduce memory usage (#10018)
Also add support for giving a callback to generate the JSON object to
verify. This should reduce memory usage, as we no longer have the event
in memory in dict form (which has a large memory footprint) for extend
periods of time.
2021-05-20 16:25:11 +01:00
Erik Johnston
64887f06fc
Use ijson to parse the response to /send_join, reducing memory usage. (#9958)
Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
2021-05-20 16:11:48 +01:00
Patrick Cloke
551d2c3f4b
Allow a user who could join a restricted room to see it in spaces summary. (#9922)
This finishes up the experimental implementation of MSC3083 by showing
the restricted rooms in the spaces summary (from MSC2946).
2021-05-20 11:10:36 -04:00
Andrew Morgan
d983ced596 Merge branch 'master' of github.com:matrix-org/synapse into develop 2021-05-20 14:39:46 +01:00
Javier Junquera Sánchez
141b073c7b
Update user_directory.md (#10016)
Signed-off-by: Javier Junquera Sánchez <javier@junquera.io>
2021-05-20 14:24:19 +01:00
Erik Johnston
9c76d0561b
Update the contrib grafana dashboard (#10001) 2021-05-19 11:47:16 +01:00
Savyasachee Jha
5bba1b4905
Hardened systemd unit files (#9803)
Signed-off-by: Savyasachee Jha savya.jha@hawkradius.com
2021-05-19 11:44:16 +01:00
Patrick Cloke
ac6bfcd52f
Refactor checking restricted join rules (#10007)
To be more consistent with similar code. The check now automatically
raises an AuthError instead of passing back a boolean. It also absorbs
some shared logic between callers.
2021-05-18 12:17:04 -04:00
Andrew Morgan
4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823) 2021-05-18 14:13:45 +01:00
Patrick Cloke
206a7b5f12
Fix the allowed range of valid ordering characters for spaces. (#10002)
\x7F was meant to be \0x7E (~) this was originally incorrect
in MSC1772.
2021-05-17 09:59:17 -04:00
Patrick Cloke
9752849e2b
Clarify comments in the space summary handler. (#9974) 2021-05-17 09:01:19 -04:00
Erik Johnston
653fe2f3cd Merge branch 'master' into develop 2021-05-17 12:14:58 +01:00
Erik Johnston
13b0673b5a Changelog 2021-05-17 12:00:28 +01:00
Erik Johnston
8dde0bf8b3 Update UPGRADE.rst 2021-05-17 11:50:08 +01:00
Erik Johnston
afb6dcf806 1.34.0 2021-05-17 11:34:39 +01:00
Brendan Abolivier
41ac128fd3
Split multiplart email sending into a dedicated handler (#9977)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-05-17 12:33:38 +02:00
Richard van der Hoff
6660912226
Update postgres docs (#9989) 2021-05-14 13:14:48 +01:00
Richard van der Hoff
6482075c95
Run black on the scripts (#9981)
Turns out these scripts weren't getting linted.
2021-05-14 11:46:35 +01:00
Richard van der Hoff
5090f26b63
Minor @cachedList enhancements (#9975)
- use a tuple rather than a list for the iterable that is passed into the
  wrapped function, for performance

- test that we can pass an iterable and that keys are correctly deduped.
2021-05-14 11:12:36 +01:00
Dan Callahan
52ed9655ed
Remove unnecessary SystemRandom from SQLBaseStore (#9987)
It's not obvious that instances of SQLBaseStore each need their own
instances of random.SystemRandom(); let's just use random directly.

Introduced by 52839886d6

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:59:10 +01:00
Dan Callahan
ebdef256b3
Remove superfluous call to bool() (#9986)
Our strtobool already returns a bool, so no need to re-cast here

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:57 +01:00
Dan Callahan
bd918d874f
Simplify exception handling in is_ascii. (#9985)
We can get away with just catching UnicodeError here.

    ⋮
    +-- ValueError
    |    +-- UnicodeError
    |         +-- UnicodeDecodeError
    |         +-- UnicodeEncodeError
    |         +-- UnicodeTranslateError
    ⋮

https://docs.python.org/3/library/exceptions.html#exception-hierarchy

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:52 +01:00
Dan Callahan
498084228b
Use Python's secrets module instead of random (#9984)
Functionally identical, but more obviously cryptographically secure.
...Explicit is better than implicit?

Avoids needing to know that SystemRandom() implies a CSPRNG, and
complies with the big scary red box on the documentation for random:

> Warning:
>   The pseudo-random generators of this module should not be used for
>   security purposes. For security or cryptographic uses, see the
>   secrets module.

https://docs.python.org/3/library/random.html

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:46 +01:00
Richard van der Hoff
c14f99be46
Support enabling opentracing by user (#9978)
Add a config option which allows enabling opentracing by user id, eg for
debugging requests made by a test user.
2021-05-14 10:51:08 +01:00
Andrew Morgan
976216959b
Update minimum supported version in postgres.md (#9988) 2021-05-14 09:21:00 +01:00
Patrick Cloke
d19bccdbec
Update SSO mapping providers documentation about unique IDs. (#9980) 2021-05-13 14:37:20 -04:00
Brendan Abolivier
451f25172a
Incorporate changes from review 2021-05-12 17:10:42 +01:00
Brendan Abolivier
91143bb24e
Refer and link to the upgrade notes rather than to the file name 2021-05-12 17:04:00 +01:00
Brendan Abolivier
47806b0869 1.34.0rc1 2021-05-12 16:59:46 +01:00
Brendan Abolivier
a683028d81
Correctly ratelimit invites when creating a room (#9968)
* Correctly ratelimit invites when creating a room

Also allow ratelimiting for more than one action at a time.
2021-05-12 16:05:28 +02:00
Richard van der Hoff
7562d887e1
Change the format of access tokens away from macaroons (#5588) 2021-05-12 15:04:51 +01:00
Erik Johnston
affaffb0ab
Run cache_joined_hosts_for_event in background (#9951) 2021-05-12 13:17:11 +01:00
Richard van der Hoff
63fb220e5f
Tests for to-device messages (#9965) 2021-05-11 18:01:11 +01:00
Patrick Cloke
27c375f812
Sort child events according to MSC1772 for the spaces summary API. (#9954)
This should help ensure that equivalent results are achieved between
homeservers querying for the summary of a space.

This implements modified MSC1772 rules, according to MSC2946.
The different is that the origin_server_ts of the m.room.create event
is not used as a tie-breaker since this might not be known if the
homeserver is not part of the room.
2021-05-11 12:57:39 -04:00
Patrick Cloke
f4833e0c06
Support fetching the spaces summary via GET over federation. (#9947)
Per changes in MSC2946, the C-S and S-S APIs for spaces summary
should use GET requests.

Until this is stable, the POST endpoints still exist.

This does not switch federation requests to use the GET version yet
since it is newly added and already deployed servers might not support
it. When switching to the stable endpoint we should switch to GET
requests.
2021-05-11 12:21:43 -04:00
Patrick Cloke
28c6841102
Send the m.room.create stripped event with invites (support MSC1772). (#9966)
MSC1772 specifies the m.room.create event should be sent as part
of the invite_state. This was done optionally behind an experimental
flag, but is now done by default due to MSC1772 being approved.
2021-05-11 10:58:58 -04:00
Brendan Abolivier
652a6b094d Merge branch 'master' into develop 2021-05-11 14:15:30 +01:00
Brendan Abolivier
d1473f7362
Use link to advisory rather than to the CVE repo 2021-05-11 14:09:46 +01:00
Aaron Raimist
dc6366a9bd
Add config option to hide device names over federation (#9945)
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-05-11 14:03:23 +01:00
Brendan Abolivier
86fb71431c
1.33.2 2021-05-11 14:01:32 +01:00
Richard van der Hoff
b378d98c8f
Add debug logging for issue #9533 (#9959)
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
2021-05-11 11:04:03 +01:00
Richard van der Hoff
7967b36efe
Fix m.room_key_request to-device messages (#9961)
fixes #9960
2021-05-11 11:02:56 +01:00
Richard van der Hoff
03318a766c
Merge pull request from GHSA-x345-32rc-8h85
* tests for push rule pattern matching

* tests for acl pattern matching

* factor out common `re.escape`

* Factor out common re.compile

* Factor out common anchoring code

* add word_boundary support to `glob_to_regex`

* Use `glob_to_regex` in push rule evaluator

NB that this drops support for character classes. I don't think anyone ever
used them.

* Improve efficiency of globs with multiple wildcards

The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.

* add assertion on regex pattern

* Fix mypy

* Simplify glob_to_regex

* Inline the glob_to_regex helper function

Signed-off-by: Dan Callahan <danc@element.io>

* Moar comments

Signed-off-by: Dan Callahan <danc@element.io>

Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
Erik Johnston
2b2985b5cf
Improve performance of backfilling in large rooms. (#9935)
We were pulling the full auth chain for the room out of the DB each time
we backfilled, which can be *huge* for large rooms and is totally
unnecessary.
2021-05-10 13:29:02 +01:00