Commit Graph

13218 Commits

Author SHA1 Message Date
Brendan Abolivier
9408b86f5c Limit the number of events sent over replication when persisting events. (#10082) 2021-05-28 16:29:09 +01:00
Erik Johnston
1641c5c707 Log method and path when dropping request due to size limit (#10091) 2021-05-28 16:29:09 +01:00
Erik Johnston
84cf3e47a0
Allow response of /send_join to be larger. (#10093)
Fixes #10087.
2021-05-28 16:28:01 +01:00
Patrick Cloke
8e15c92c2f
Pass the origin when calculating the spaces summary over GET. (#10079)
Fixes a bug due to conflicting PRs which were merged. (One added a new caller to
a method, the other added a new parameter to the same method.)
2021-05-27 08:52:28 -04:00
Erik Johnston
557635f69a 1.35.0rc1 2021-05-25 11:00:13 +01:00
Patrick Cloke
7adcb20fc0
Add missing type hints to synapse.util (#9982) 2021-05-24 15:32:01 -04:00
Jerin J Titus
057ce7b754
Remove tls_fingerprints option (#9280)
Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
2021-05-24 17:43:30 +01:00
Richard van der Hoff
c0df6bae06
Remove keylen from LruCache. (#9993)
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it.

The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`).

Commits should be independently reviewable.
2021-05-24 14:02:01 +01:00
Patrick Cloke
316f89e87f
Enable experimental spaces by default. (#10011)
The previous spaces_enabled flag now defaults to true and
is exposed in the sample config.
2021-05-24 08:57:14 -04:00
Eric Eastwood
5f1198a67e
Fix get_state_ids_for_event return type typo to match what the function actually does (#10050)
It looks like a typo copy/paste from `get_state_for_event` above.
2021-05-24 10:43:33 +01:00
Erik Johnston
3e831f24ff
Don't hammer the database for destination retry timings every ~5mins (#10036) 2021-05-21 17:57:08 +01:00
Michael Telatynski
e8ac9ac8ca
Fix /upload 500'ing when presented a very large image (#10029)
* Fix /upload 500'ing when presented a very large image

Catch DecompressionBombError and re-raise as ThumbnailErrors

* Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml

to get it to bomb out quicker, to load less into memory
in the case of super large images

* Add changelog entry for 10029
2021-05-21 18:31:59 +02:00
Patrick Cloke
c5413d0e9e
Remove unused properties from the SpaceSummaryHandler. (#10038) 2021-05-21 12:02:01 -04:00
Marek Matys
6a8643ff3d
Fixed removal of new presence stream states (#10014)
Fixes: https://github.com/matrix-org/synapse/issues/9962

This is a fix for above problem.

I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts.

Signed-off-by: Marek Matys <themarcq@gmail.com>
2021-05-21 12:02:06 +01:00
Erik Johnston
7958eadcd1
Add a batching queue implementation. (#10017) 2021-05-21 11:20:51 +01:00
Erik Johnston
1c6a19002c
Add Keyring.verify_events_for_server and reduce memory usage (#10018)
Also add support for giving a callback to generate the JSON object to
verify. This should reduce memory usage, as we no longer have the event
in memory in dict form (which has a large memory footprint) for extend
periods of time.
2021-05-20 16:25:11 +01:00
Erik Johnston
64887f06fc
Use ijson to parse the response to /send_join, reducing memory usage. (#9958)
Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
2021-05-20 16:11:48 +01:00
Patrick Cloke
551d2c3f4b
Allow a user who could join a restricted room to see it in spaces summary. (#9922)
This finishes up the experimental implementation of MSC3083 by showing
the restricted rooms in the spaces summary (from MSC2946).
2021-05-20 11:10:36 -04:00
Patrick Cloke
ac6bfcd52f
Refactor checking restricted join rules (#10007)
To be more consistent with similar code. The check now automatically
raises an AuthError instead of passing back a boolean. It also absorbs
some shared logic between callers.
2021-05-18 12:17:04 -04:00
Andrew Morgan
4d6e5a5e99
Use a database table to hold the users that should have full presence sent to them, instead of something in-memory (#9823) 2021-05-18 14:13:45 +01:00
Patrick Cloke
206a7b5f12
Fix the allowed range of valid ordering characters for spaces. (#10002)
\x7F was meant to be \0x7E (~) this was originally incorrect
in MSC1772.
2021-05-17 09:59:17 -04:00
Patrick Cloke
9752849e2b
Clarify comments in the space summary handler. (#9974) 2021-05-17 09:01:19 -04:00
Erik Johnston
653fe2f3cd Merge branch 'master' into develop 2021-05-17 12:14:58 +01:00
Erik Johnston
afb6dcf806 1.34.0 2021-05-17 11:34:39 +01:00
Brendan Abolivier
41ac128fd3
Split multiplart email sending into a dedicated handler (#9977)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-05-17 12:33:38 +02:00
Richard van der Hoff
5090f26b63
Minor @cachedList enhancements (#9975)
- use a tuple rather than a list for the iterable that is passed into the
  wrapped function, for performance

- test that we can pass an iterable and that keys are correctly deduped.
2021-05-14 11:12:36 +01:00
Dan Callahan
52ed9655ed
Remove unnecessary SystemRandom from SQLBaseStore (#9987)
It's not obvious that instances of SQLBaseStore each need their own
instances of random.SystemRandom(); let's just use random directly.

Introduced by 52839886d6

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:59:10 +01:00
Dan Callahan
ebdef256b3
Remove superfluous call to bool() (#9986)
Our strtobool already returns a bool, so no need to re-cast here

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:57 +01:00
Dan Callahan
bd918d874f
Simplify exception handling in is_ascii. (#9985)
We can get away with just catching UnicodeError here.

    ⋮
    +-- ValueError
    |    +-- UnicodeError
    |         +-- UnicodeDecodeError
    |         +-- UnicodeEncodeError
    |         +-- UnicodeTranslateError
    ⋮

https://docs.python.org/3/library/exceptions.html#exception-hierarchy

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:52 +01:00
Dan Callahan
498084228b
Use Python's secrets module instead of random (#9984)
Functionally identical, but more obviously cryptographically secure.
...Explicit is better than implicit?

Avoids needing to know that SystemRandom() implies a CSPRNG, and
complies with the big scary red box on the documentation for random:

> Warning:
>   The pseudo-random generators of this module should not be used for
>   security purposes. For security or cryptographic uses, see the
>   secrets module.

https://docs.python.org/3/library/random.html

Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:46 +01:00
Richard van der Hoff
c14f99be46
Support enabling opentracing by user (#9978)
Add a config option which allows enabling opentracing by user id, eg for
debugging requests made by a test user.
2021-05-14 10:51:08 +01:00
Brendan Abolivier
47806b0869 1.34.0rc1 2021-05-12 16:59:46 +01:00
Brendan Abolivier
a683028d81
Correctly ratelimit invites when creating a room (#9968)
* Correctly ratelimit invites when creating a room

Also allow ratelimiting for more than one action at a time.
2021-05-12 16:05:28 +02:00
Richard van der Hoff
7562d887e1
Change the format of access tokens away from macaroons (#5588) 2021-05-12 15:04:51 +01:00
Erik Johnston
affaffb0ab
Run cache_joined_hosts_for_event in background (#9951) 2021-05-12 13:17:11 +01:00
Patrick Cloke
27c375f812
Sort child events according to MSC1772 for the spaces summary API. (#9954)
This should help ensure that equivalent results are achieved between
homeservers querying for the summary of a space.

This implements modified MSC1772 rules, according to MSC2946.
The different is that the origin_server_ts of the m.room.create event
is not used as a tie-breaker since this might not be known if the
homeserver is not part of the room.
2021-05-11 12:57:39 -04:00
Patrick Cloke
f4833e0c06
Support fetching the spaces summary via GET over federation. (#9947)
Per changes in MSC2946, the C-S and S-S APIs for spaces summary
should use GET requests.

Until this is stable, the POST endpoints still exist.

This does not switch federation requests to use the GET version yet
since it is newly added and already deployed servers might not support
it. When switching to the stable endpoint we should switch to GET
requests.
2021-05-11 12:21:43 -04:00
Patrick Cloke
28c6841102
Send the m.room.create stripped event with invites (support MSC1772). (#9966)
MSC1772 specifies the m.room.create event should be sent as part
of the invite_state. This was done optionally behind an experimental
flag, but is now done by default due to MSC1772 being approved.
2021-05-11 10:58:58 -04:00
Brendan Abolivier
652a6b094d Merge branch 'master' into develop 2021-05-11 14:15:30 +01:00
Aaron Raimist
dc6366a9bd
Add config option to hide device names over federation (#9945)
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-05-11 14:03:23 +01:00
Brendan Abolivier
86fb71431c
1.33.2 2021-05-11 14:01:32 +01:00
Richard van der Hoff
b378d98c8f
Add debug logging for issue #9533 (#9959)
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
2021-05-11 11:04:03 +01:00
Richard van der Hoff
7967b36efe
Fix m.room_key_request to-device messages (#9961)
fixes #9960
2021-05-11 11:02:56 +01:00
Richard van der Hoff
03318a766c
Merge pull request from GHSA-x345-32rc-8h85
* tests for push rule pattern matching

* tests for acl pattern matching

* factor out common `re.escape`

* Factor out common re.compile

* Factor out common anchoring code

* add word_boundary support to `glob_to_regex`

* Use `glob_to_regex` in push rule evaluator

NB that this drops support for character classes. I don't think anyone ever
used them.

* Improve efficiency of globs with multiple wildcards

The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.

* add assertion on regex pattern

* Fix mypy

* Simplify glob_to_regex

* Inline the glob_to_regex helper function

Signed-off-by: Dan Callahan <danc@element.io>

* Moar comments

Signed-off-by: Dan Callahan <danc@element.io>

Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
Erik Johnston
2b2985b5cf
Improve performance of backfilling in large rooms. (#9935)
We were pulling the full auth chain for the room out of the DB each time
we backfilled, which can be *huge* for large rooms and is totally
unnecessary.
2021-05-10 13:29:02 +01:00
Erik Johnston
6c84778549
Always cache 'event_to_prev_state_group' (#9950)
Fixes regression in send PDU times introduced in #9905.
2021-05-07 14:54:09 +01:00
Erik Johnston
b65ecaff9b Merge remote-tracking branch 'origin/release-v1.33.2' into develop 2021-05-07 13:27:19 +01:00
Erik Johnston
4df26abf28
Unpin attrs dep after new version has been released (#9946)
c.f. #9936
2021-05-07 12:57:21 +01:00
Richard van der Hoff
25f43faa70
Reorganise the database schema directories (#9932)
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.

It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
2021-05-07 10:22:05 +01:00
Erik Johnston
8771b1337d
Export jemalloc stats to prometheus when used (#9882) 2021-05-06 15:54:07 +01:00