Dan Callahan
ebdef256b3
Remove superfluous call to bool() ( #9986 )
...
Our strtobool already returns a bool, so no need to re-cast here
Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:57 +01:00
Dan Callahan
bd918d874f
Simplify exception handling in is_ascii. ( #9985 )
...
We can get away with just catching UnicodeError here.
⋮
+-- ValueError
| +-- UnicodeError
| +-- UnicodeDecodeError
| +-- UnicodeEncodeError
| +-- UnicodeTranslateError
⋮
https://docs.python.org/3/library/exceptions.html#exception-hierarchy
Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:52 +01:00
Dan Callahan
498084228b
Use Python's secrets module instead of random ( #9984 )
...
Functionally identical, but more obviously cryptographically secure.
...Explicit is better than implicit?
Avoids needing to know that SystemRandom() implies a CSPRNG, and
complies with the big scary red box on the documentation for random:
> Warning:
> The pseudo-random generators of this module should not be used for
> security purposes. For security or cryptographic uses, see the
> secrets module.
https://docs.python.org/3/library/random.html
Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14 10:58:46 +01:00
Richard van der Hoff
c14f99be46
Support enabling opentracing by user ( #9978 )
...
Add a config option which allows enabling opentracing by user id, eg for
debugging requests made by a test user.
2021-05-14 10:51:08 +01:00
Brendan Abolivier
47806b0869
1.34.0rc1
2021-05-12 16:59:46 +01:00
Brendan Abolivier
a683028d81
Correctly ratelimit invites when creating a room ( #9968 )
...
* Correctly ratelimit invites when creating a room
Also allow ratelimiting for more than one action at a time.
2021-05-12 16:05:28 +02:00
Richard van der Hoff
7562d887e1
Change the format of access tokens away from macaroons ( #5588 )
2021-05-12 15:04:51 +01:00
Erik Johnston
affaffb0ab
Run cache_joined_hosts_for_event in background ( #9951 )
2021-05-12 13:17:11 +01:00
Patrick Cloke
27c375f812
Sort child events according to MSC1772 for the spaces summary API. ( #9954 )
...
This should help ensure that equivalent results are achieved between
homeservers querying for the summary of a space.
This implements modified MSC1772 rules, according to MSC2946.
The different is that the origin_server_ts of the m.room.create event
is not used as a tie-breaker since this might not be known if the
homeserver is not part of the room.
2021-05-11 12:57:39 -04:00
Patrick Cloke
f4833e0c06
Support fetching the spaces summary via GET over federation. ( #9947 )
...
Per changes in MSC2946, the C-S and S-S APIs for spaces summary
should use GET requests.
Until this is stable, the POST endpoints still exist.
This does not switch federation requests to use the GET version yet
since it is newly added and already deployed servers might not support
it. When switching to the stable endpoint we should switch to GET
requests.
2021-05-11 12:21:43 -04:00
Patrick Cloke
28c6841102
Send the m.room.create stripped event with invites (support MSC1772). ( #9966 )
...
MSC1772 specifies the m.room.create event should be sent as part
of the invite_state. This was done optionally behind an experimental
flag, but is now done by default due to MSC1772 being approved.
2021-05-11 10:58:58 -04:00
Brendan Abolivier
652a6b094d
Merge branch 'master' into develop
2021-05-11 14:15:30 +01:00
Aaron Raimist
dc6366a9bd
Add config option to hide device names over federation ( #9945 )
...
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers.
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-05-11 14:03:23 +01:00
Brendan Abolivier
86fb71431c
1.33.2
2021-05-11 14:01:32 +01:00
Richard van der Hoff
b378d98c8f
Add debug logging for issue #9533 ( #9959 )
...
Hopefully this will help us track down where to-device messages are getting
lost/delayed.
2021-05-11 11:04:03 +01:00
Richard van der Hoff
7967b36efe
Fix m.room_key_request to-device messages ( #9961 )
...
fixes #9960
2021-05-11 11:02:56 +01:00
Richard van der Hoff
03318a766c
Merge pull request from GHSA-x345-32rc-8h85
...
* tests for push rule pattern matching
* tests for acl pattern matching
* factor out common `re.escape`
* Factor out common re.compile
* Factor out common anchoring code
* add word_boundary support to `glob_to_regex`
* Use `glob_to_regex` in push rule evaluator
NB that this drops support for character classes. I don't think anyone ever
used them.
* Improve efficiency of globs with multiple wildcards
The idea here is that we compress multiple `*` globs into a single `.*`. We
also need to consider `?`, since `*?*` is as hard to implement efficiently as
`**`.
* add assertion on regex pattern
* Fix mypy
* Simplify glob_to_regex
* Inline the glob_to_regex helper function
Signed-off-by: Dan Callahan <danc@element.io>
* Moar comments
Signed-off-by: Dan Callahan <danc@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
2021-05-11 11:47:23 +02:00
Erik Johnston
2b2985b5cf
Improve performance of backfilling in large rooms. ( #9935 )
...
We were pulling the full auth chain for the room out of the DB each time
we backfilled, which can be *huge* for large rooms and is totally
unnecessary.
2021-05-10 13:29:02 +01:00
Erik Johnston
6c84778549
Always cache 'event_to_prev_state_group' ( #9950 )
...
Fixes regression in send PDU times introduced in #9905 .
2021-05-07 14:54:09 +01:00
Erik Johnston
b65ecaff9b
Merge remote-tracking branch 'origin/release-v1.33.2' into develop
2021-05-07 13:27:19 +01:00
Erik Johnston
4df26abf28
Unpin attrs dep after new version has been released ( #9946 )
...
c.f. #9936
2021-05-07 12:57:21 +01:00
Richard van der Hoff
25f43faa70
Reorganise the database schema directories ( #9932 )
...
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate.
It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
2021-05-07 10:22:05 +01:00
Erik Johnston
8771b1337d
Export jemalloc stats to prometheus when used ( #9882 )
2021-05-06 15:54:07 +01:00
Erik Johnston
eba431c539
Revert "Leave out optional keys from /sync ( #9919 )" ( #9940 )
...
This reverts commit e9eb3549d3
2021-05-06 15:06:35 +01:00
Erik Johnston
a8803e2b6e
Merge remote-tracking branch 'origin/master' into develop
2021-05-06 14:36:59 +01:00
Erik Johnston
ac88aca7f7
1.33.1
2021-05-06 14:06:38 +01:00
Erik Johnston
24f07a83e6
Pin attrs to <21.1.0 ( #9937 )
...
Fixes #9936
2021-05-06 14:06:06 +01:00
Patrick Cloke
70f0ffd2fc
Follow-up to #9915 to correct the identifier for room types.
2021-05-05 16:31:16 -04:00
Patrick Cloke
d783880083
Include the time of the create event in Spaces Summary. ( #9928 )
...
This is an update based on changes to MSC2946. The origin_server_ts
of the m.room.create event is copied into the creation_ts field for each
room returned from the spaces summary.
2021-05-05 13:33:05 -04:00
Erik Johnston
37623e3382
Increase perf of handling presence when joining large rooms. ( #9916 )
2021-05-05 17:27:05 +01:00
Patrick Cloke
e2a443550e
Support stable MSC1772 spaces identifiers. ( #9915 )
...
Support both the unstable and stable identifiers. A future release
will disable the unstable identifiers.
2021-05-05 11:56:51 -04:00
Erik Johnston
ef889c98a6
Optionally track memory usage of each LruCache ( #9881 )
...
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
2021-05-05 16:54:36 +01:00
Erik Johnston
1fb9a2d0bf
Limit how often GC happens by time. ( #9902 )
...
Synapse can be quite memory intensive, and unless care is taken to tune
the GC thresholds it can end up thrashing, causing noticable performance
problems for large servers. We fix this by limiting how often we GC a
given generation, regardless of current counts/thresholds.
This does not help with the reverse problem where the thresholds are set
too high, but that should only happen in situations where they've been
manually configured.
Adds a `gc_min_seconds_between` config option to override the defaults.
Fixes #9890 .
2021-05-05 16:53:45 +01:00
Erik Johnston
de8f0a03a3
Don't set the external cache if its been done recently ( #9905 )
2021-05-05 16:53:22 +01:00
Erik Johnston
d0aee697ac
Use get_current_users_in_room from store and not StateHandler ( #9910 )
2021-05-05 16:49:34 +01:00
DeepBlueV7.X
e9eb3549d3
Leave out optional keys from /sync ( #9919 )
...
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2021-05-05 14:37:56 +01:00
Brendan Abolivier
a61b13c0a1
Merge branch 'master' into develop
2021-05-05 14:25:49 +01:00
Brendan Abolivier
0644ac0989
1.33.0
2021-05-05 14:15:54 +01:00
Erik Johnston
e3bc4617fc
Time external cache response time ( #9904 )
2021-05-04 15:14:22 +01:00
Andrew Morgan
b85821aca2
Add port parameter to the sample config for psycopg2 args ( #9911 )
...
Adds the `port` option with the default value to the sample config file.
2021-05-04 13:28:59 +01:00
Andrew Morgan
4d624f467a
Synapse 1.33.0rc2 (2021-04-29)
...
==============================
Bugfixes
--------
- Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900 ))
-----BEGIN PGP SIGNATURE-----
iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCKtVATHGFuZHJld0Bh
bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ok1D/40l93bxGBFtSNEHlqem7lZRLY2On9j
AUnzIMqtJTLUgXf67vKIYqtTWkRjTdLIhIuINwhRmB0K23vcRrONnnyRLdhUC4Hl
v6qLlPBHzupAJQPRcy3FGIVwsloAZ3JqVhV8AG0gXBRA0Dc34qYXh1DlA8EQ8mW/
l2umQIJI562xLNss7c5tpbCQxi7mq6OmTZ+Nwh45VRDavKEQQzSb9sPfs5w8IGLA
ViYosUoXoXEWus/j60E38LSA/v1YcEE+Uyf5+LieV4b2c305xKj8gIbZqjLHR2K5
U9IrUp4QlYpPiebfGn7EmiPnOnlWdR9soIVZMzJ/m29tlyRSL5MBagtWA2fFlXo6
PywW1R/VsnLTdOuWY/t+et11iI4lVa7VSpywEcbMKGgmGKexDnfB10K7jHw0YICq
0pyLgy8JWyGUk/xpH2W1pvak8vyyN/zI/H09bDgGu721Qr/EPiHWF6RADNASpUVW
w10SnO1udVc8ajugNwpX2oOQA7i5uiEuTdyFUHYfZ/aSpaCvdxM+rL3v0K955bNc
/AGjSbs9ITe+JJQDiAJsWCYc+XLkR4h5KHVaZm2XlusbYb4iobK1DflkWXnQFjsi
rlTcUH3sIFot3i9mjBBvXdvmzuI9qyx81/Sh8gRunPYT5TPhwaXKfyMBKE93/kif
4bKoHIXVcJvAqA==
=sboq
-----END PGP SIGNATURE-----
Merge tag 'v1.33.0rc2' into develop
Synapse 1.33.0rc2 (2021-04-29)
==============================
Bugfixes
--------
- Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](https://github.com/matrix-org/synapse/issues/9900 ))
2021-04-29 14:35:14 +01:00
Patrick Cloke
bb4b11846f
Add missing type hints to handlers and fix a Spam Checker type hint. ( #9896 )
...
The user_may_create_room_alias method on spam checkers
declared the room_alias parameter as a str when in reality it is
passed a RoomAlias object.
2021-04-29 07:17:28 -04:00
Andrew Morgan
e9444cc74d
1.33.0rc2
2021-04-29 11:45:37 +01:00
ThibF
0085dc5abc
Delete room endpoint ( #9889 )
...
Support the delete of a room through DELETE request and mark
previous request as deprecated through documentation.
Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
2021-04-29 10:31:45 +01:00
Erik Johnston
802560211a
Merge remote-tracking branch 'origin/release-v1.33.0' into develop
2021-04-28 14:43:10 +01:00
Erik Johnston
e4ab8676b4
Fix tight loop handling presence replication. ( #9900 )
...
Only affects workers. Introduced in #9819 .
Fixes #9899 .
2021-04-28 14:42:50 +01:00
Patrick Cloke
10a08ab88a
Use the parent's logging context name for runWithConnection. ( #9895 )
...
This fixes a regression where the logging context for runWithConnection
was reported as runWithConnection instead of the connection name,
e.g. "POST-XYZ".
2021-04-28 07:44:52 -04:00
Andrew Morgan
fa6679e794
Synapse 1.33.0rc1 (2021-04-28)
...
==============================
Features
--------
- Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083 ): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800 ), [\#9814](https://github.com/matrix-org/synapse/issues/9814 ))
- Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819 ), [\#9820](https://github.com/matrix-org/synapse/issues/9820 ), [\#9828](https://github.com/matrix-org/synapse/issues/9828 ), [\#9850](https://github.com/matrix-org/synapse/issues/9850 ))
- Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832 ))
Bugfixes
--------
- Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726 ))
- Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788 ))
- Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802 ))
- Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833 ))
- Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867 ))
- Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868 ))
Improved Documentation
----------------------
- Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801 ))
Internal Changes
----------------
- Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162 ))
- Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786 ))
- Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796 ))
- Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815 ))
- Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816 ))
- Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817 ))
- Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821 ))
- Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825 ))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838 ))
- Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845 ))
- Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855 ))
- Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856 ))
- Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858 ))
- Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871 ))
- Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874 ))
- Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875 ))
- Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876 ))
- Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878 ))
- Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879 ))
- Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887 ))
-----BEGIN PGP SIGNATURE-----
iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmCJQmgTHGFuZHJld0Bh
bW9yZ2FuLnh5egAKCRCIhIgNLv5f9EKWD/9MZG4ngjECrk3QMBlYqaipGl/l7wcl
8vBOTxiEqIgVJxHLCRFBbtFOItdxB4YkrorIo2fqshkk+lv4CRtTD9jEKHRvS4T6
7p1icRdRTv2K94OkJ8R9jMFlmywZFU87oHPfI2xUYg6hvOKrR+RwPvIjA7c24UZt
6MJqDhgGDlZD7/hQdQof9O4oOJzzIgLJPk6o8E42y6c0bLlPPKgH3sh0vlenDLfE
15thRCOeiP237YYvXSdbr7G3PI66Efhq4BwQowSrgFg+B0BR68l4747iSIeWmLQJ
Ow6QLLFPCOhgAPC4amp9PdaaV/9NYiBcvNxlOvyQAVl/+ioEiATNHnzdNtDouCQo
nDdSHw0Mt9D3i+rxpu5Pf0gZN9dXRGiczqnq5QKL8+EvT/4FLYeluEeduuy3rE+G
o5OTCd3EajxynzjftuopeysNAw6zeDpbulZoTCeCumxBL2+wAod2PsyY5Ei9Gapn
iJvExNOJX4OlkFc67jO2CK8o3sUTNEDeIDWCQ0fVKAwIt7T45ebTA/UzDNWg7YzN
EyUp+3NZcUZBskgMB5hpcijPJoXNYzWZPews73vMPV7AfQRxDzU5xrM9AdbadBDd
Idv6wT9ssDUA+M0aKnafvoSJZ+qE85mi2x0rRsueZNd9uO9/QHIXKIb+4PvVpj3C
BJnZf34m568AGQ==
=smAk
-----END PGP SIGNATURE-----
Merge tag 'v1.33.0rc1' into develop
Synapse 1.33.0rc1 (2021-04-28)
==============================
Features
--------
- Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083 ): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800 ), [\#9814](https://github.com/matrix-org/synapse/issues/9814 ))
- Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819 ), [\#9820](https://github.com/matrix-org/synapse/issues/9820 ), [\#9828](https://github.com/matrix-org/synapse/issues/9828 ), [\#9850](https://github.com/matrix-org/synapse/issues/9850 ))
- Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832 ))
Bugfixes
--------
- Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726 ))
- Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788 ))
- Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802 ))
- Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833 ))
- Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867 ))
- Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868 ))
Improved Documentation
----------------------
- Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801 ))
Internal Changes
----------------
- Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162 ))
- Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786 ))
- Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796 ))
- Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815 ))
- Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816 ))
- Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817 ))
- Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821 ))
- Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825 ))
- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838 ))
- Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845 ))
- Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855 ))
- Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856 ))
- Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858 ))
- Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871 ))
- Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874 ))
- Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875 ))
- Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876 ))
- Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878 ))
- Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879 ))
- Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887 ))
2021-04-28 12:12:29 +01:00
Erik Johnston
391bfe9a7b
Reduce memory footprint of caches ( #9886 )
2021-04-28 11:59:28 +01:00
Andrew Morgan
787de3190f
1.33.0rc1
2021-04-28 11:43:33 +01:00
Andrew Morgan
4e0fd35bc9
Revert "Experimental Federation Speedup ( #9702 )"
...
This reverts commit 05e8c70c05
2021-04-28 11:38:33 +01:00
Erik Johnston
dd2d32dcdb
Add type hints to presence handler ( #9885 )
2021-04-28 11:07:47 +01:00
Andrew Morgan
fe604a022a
Remove various bits of compatibility code for Python <3.6 ( #9879 )
...
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
2021-04-27 13:13:07 +01:00
Patrick Cloke
1350b053da
Pass errors back to the client when trying multiple federation destinations. ( #9868 )
...
This ensures that something like an auth error (403) will be
returned to the requester instead of attempting to try more
servers, which will likely result in the same error, and then
passing back a generic 400 error.
2021-04-27 07:30:34 -04:00
Erik Johnston
0ffa5fb935
Use current state table for presence.get_interested_remotes ( #9887 )
...
This should be a lot quicker than asking the state handler.
2021-04-27 10:09:41 +01:00
Richard van der Hoff
3ff2251754
Improved validation for received requests ( #9817 )
...
* Simplify `start_listening` callpath
* Correctly check the size of uploaded files
2021-04-23 19:20:44 +01:00
Andrew Morgan
695b73c861
Allow OIDC cookies to work on non-root public baseurls ( #9726 )
...
Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574 .
As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.
2021-04-23 18:22:47 +01:00
Richard van der Hoff
59d24c5bef
pass a reactor into SynapseSite ( #9874 )
2021-04-23 17:06:47 +01:00
Patrick Cloke
e83627926f
Add type hints to auth and auth_blocking. ( #9876 )
2021-04-23 12:02:16 -04:00
Erik Johnston
a15c003e5b
Make DomainSpecificString an attrs class ( #9875 )
2021-04-23 15:46:29 +01:00
Erik Johnston
9d25a0ae65
Split presence out of master ( #9820 )
2021-04-23 12:21:55 +01:00
Patrick Cloke
d924827da1
Check for space membership during a remote join of a restricted room ( #9814 )
...
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join rules.
This only applies to an experimental room version, as defined in MSC3083.
2021-04-23 07:05:51 -04:00
Erik Johnston
3853a7edfc
Only store data in caches, not "smart" objects ( #9845 )
2021-04-23 11:47:07 +01:00
Richard van der Hoff
51a20914a8
Limit the size of HTTP responses read over federation. ( #9833 )
2021-04-23 11:08:41 +01:00
Erik Johnston
177dae2704
Limit length of accepted email addresses ( #9855 )
2021-04-22 17:49:11 +01:00
Richard van der Hoff
69018acbd2
Clear the resync bit after resyncing device lists ( #9867 )
...
Fixes #9866 .
2021-04-22 16:53:24 +01:00
Richard van der Hoff
294c675033
Remove synapse.types.Collection ( #9856 )
...
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22 16:43:50 +01:00
Andrew Morgan
3186324260
Merge branch 'master' into develop
2021-04-22 11:23:56 +01:00
Andrew Morgan
55159c48e3
1.32.2
2021-04-21 18:45:39 +01:00
Richard van der Hoff
d9bd62f9d1
Make LoggingContext's name optional ( #9857 )
...
Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55
2021-04-21 16:39:34 +01:00
Andrew Morgan
4b2217ace2
Merge branch 'master' into develop
2021-04-21 14:55:06 +01:00
Andrew Morgan
a745531c10
1.32.1
2021-04-21 14:01:12 +01:00
Richard van der Hoff
5d281c10dd
Stop BackgroundProcessLoggingContext making new prometheus timeseries ( #9854 )
...
This undoes part of b076bc276e
2021-04-21 10:03:31 +01:00
Patrick Cloke
683d6f75af
Rename handler and config modules which end in handler/config. ( #9816 )
2021-04-20 14:55:20 -04:00
Andrew Morgan
6982db9651
Merge branch 'master' into develop
2021-04-20 14:55:16 +01:00
Andrew Morgan
e031c7e0cc
1.32.0
2021-04-20 14:31:27 +01:00
Patrick Cloke
b076bc276e
Always use the name as the log ID. ( #9829 )
...
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used.
This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20 14:19:00 +01:00
Erik Johnston
de0d088adc
Add presence federation stream ( #9819 )
2021-04-20 14:11:24 +01:00
Erik Johnston
db70435de7
Fix bug where we sent remote presence states to remote servers ( #9850 )
2021-04-20 13:37:54 +01:00
Jonathan de Jong
495b214f4f
Fix (final) Bugbear violations ( #9838 )
2021-04-20 11:50:49 +01:00
Andrew Morgan
71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74 " from synapse-dinsic ( #9832 )
...
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
Denis Kasak
e694a598f8
Sanity check identity server passed to bind/unbind. ( #9802 )
...
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-04-19 17:21:46 +01:00
Erik Johnston
2b7dd21655
Don't send normal presence updates over federation replication stream ( #9828 )
2021-04-19 10:50:49 +01:00
Andrew Morgan
c571736c6c
User directory: use calculated room membership state instead ( #9821 )
...
Fixes : #9797 .
Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
2021-04-16 18:17:18 +01:00
Erik Johnston
601b893352
Small speed up joining large remote rooms ( #9825 )
...
There are a couple of points in `persist_events` where we are doing a
query per event in series, which we can replace.
2021-04-16 14:44:55 +01:00
Richard van der Hoff
5a153772c1
remove HomeServer.get_config ( #9815 )
...
Every single time I want to access the config object, I have to remember
whether or not we use `get_config`. Let's just get rid of it.
2021-04-14 19:09:08 +01:00
Patrick Cloke
936e69825a
Separate creating an event context from persisting it in the federation handler ( #9800 )
...
This refactoring allows adding logic that uses the event context
before persisting it.
2021-04-14 12:35:28 -04:00
Patrick Cloke
e8816c6ace
Revert "Check for space membership during a remote join of a restricted room. ( #9763 )"
...
This reverts commit cc51aaaa7a
2021-04-14 12:33:37 -04:00
Patrick Cloke
cc51aaaa7a
Check for space membership during a remote join of a restricted room. ( #9763 )
...
When receiving a /send_join request for a room with join rules set to 'restricted',
check if the user is a member of the spaces defined in the 'allow' key of the join
rules.
This only applies to an experimental room version, as defined in MSC3083.
2021-04-14 12:32:20 -04:00
Jonathan de Jong
05e8c70c05
Experimental Federation Speedup ( #9702 )
...
This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max).
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-14 17:19:02 +01:00
Erik Johnston
00a6db9676
Move some replication processing out of generic_worker ( #9796 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-04-14 17:06:06 +01:00
rkfg
c9a2b5d402
More robust handling of the Content-Type header for thumbnail generation ( #9788 )
...
Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
2021-04-14 16:30:59 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ( #9786 )
...
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Andrew Morgan
3efd98aa1c
1.32.0rc1
2021-04-13 14:23:43 +01:00
Dan Callahan
1d5f0e3529
Bump black configuration to target py36 ( #9781 )
...
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13 10:41:34 +01:00
Dirk Klimpel
1fc97ee876
Add an admin API to manage ratelimit for a specific user ( #9648 )
2021-04-13 10:26:37 +01:00
Will Hunt
e300ef64b1
Require AppserviceRegistrationType ( #9548 )
...
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
2021-04-12 15:13:55 +01:00
Richard van der Hoff
f946450184
Fix duplicate logging of exceptions in transaction processing ( #9780 )
...
There's no point logging this twice.
2021-04-09 18:12:15 +01:00
Dirk Klimpel
48a1f4db31
Remove old admin API GET /_synapse/admin/v1/users/<user_id> ( #9401 )
...
Related: #8334
Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25)
`GET /_synapse/admin/v1/users/<user_id>` has no
- unit tests
- documentation
API in v2 is available (#5925 - 12/2019, v1.7.0).
API is misleading. It expects `user_id` and returns a list of all users.
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-04-09 09:44:40 +01:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes ( #9682 )
...
Part of #9366
Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08 22:38:54 +01:00
Richard van der Hoff
9e167d9c53
Merge remote-tracking branch 'origin/develop' into rav/drop_py35
2021-04-08 18:30:38 +01:00
Richard van der Hoff
24c58ebfc9
remove unused param on make_tuple_comparison_clause
2021-04-08 18:29:57 +01:00
Erik Johnston
be0e722fe1
Merge branch 'erikj/fix_stalled_catchup' into develop
2021-04-08 18:05:35 +01:00
Erik Johnston
3a569fb200
Fix sharded federation sender sometimes using 100% CPU.
...
We pull all destinations requiring catchup from the DB in batches.
However, if all those destinations get filtered out (due to the
federation sender being sharded), then the `last_processed` destination
doesn't get updated, and we keep requesting the same set repeatedly.
2021-04-08 17:34:07 +01:00
Richard van der Hoff
3ada9b4264
Drop support for sqlite<3.22 as well
2021-04-08 16:42:32 +01:00
Richard van der Hoff
abade34633
Require py36 and Postgres 9.6
2021-04-08 16:42:32 +01:00
Patrick Cloke
452991527a
MSC3083: Check for space membership during a local join of restricted rooms. ( #9735 )
...
When joining a room with join rules set to 'restricted', check if the
user is a member of the spaces defined in the 'allow' key of the join rules.
This only applies to an experimental room version, as defined in MSC3083.
2021-04-08 08:28:32 -04:00
Patrick Cloke
48d44ab142
Record more information into structured logs. ( #9654 )
...
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
2021-04-08 08:01:14 -04:00
Andrew Morgan
0d87c6bd12
Don't report anything from GaugeBucketCollector metrics until data is present ( #8926 )
...
This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.
2021-04-06 16:32:04 +01:00
Andrew Morgan
04819239ba
Add a Synapse Module for configuring presence update routing ( #9491 )
...
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.
This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.
A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync.
The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:
* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06 14:38:30 +01:00
Patrick Cloke
44bb881096
Add type hints to expiring cache. ( #9730 )
2021-04-06 08:58:18 -04:00
Richard van der Hoff
0ef321ff3b
Remove outdated constraint on remote_media_cache_thumbnails ( #9725 )
...
The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key`
constraint is superceded by
`remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds
`thumbnail_method` to the unique key).
PR #7124 made an attempt to remove the old constraint, but got the name wrong,
so it didn't work. Here we update the bg update and rerun it.
Fixes #8649 .
2021-04-06 13:36:05 +01:00
Erik Johnston
5688a74cf3
Merge branch 'master' into develop
2021-04-06 13:29:29 +01:00
Erik Johnston
1d8863c67d
1.31.0
2021-04-06 13:09:56 +01:00
Patrick Cloke
d959d28730
Add type hints to the federation handler and server. ( #9743 )
2021-04-06 07:21:57 -04:00
Jonathan de Jong
e2b8a90897
Update mypy configuration: no_implicit_optional = True ( #9742 )
2021-04-05 09:10:18 -04:00
Erik Johnston
33548f37aa
Improve tracing for to device messages ( #9686 )
2021-04-01 17:08:21 +01:00
Dirk Klimpel
bb0fe02a52
Add order_by to list user admin API ( #9691 )
2021-04-01 11:28:53 +01:00
Patrick Cloke
35c5ef2d24
Add an experimental room version to support restricted join rules. ( #9717 )
...
Per MSC3083.
2021-03-31 16:39:08 -04:00
Denis Kasak
5ff8eb97c6
Make sample config allowed_local_3pids regex stricter. ( #9719 )
...
The regex should be terminated so that subdomain matches of another
domain are not accepted. Just ensuring that someone doesn't shoot
themselves in the foot by copying our example.
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-03-31 12:27:20 +00:00
Cristina
670564446c
Deprecate imp ( #9718 )
...
Fixes #9642 .
Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
2021-03-31 12:04:27 +01:00
Richard van der Hoff
4dabcf026e
Include m.room.create in invite_room_state for Spaces ( #9710 )
2021-03-30 14:03:17 +01:00
Richard van der Hoff
f02663c4dd
Replace room_invite_state_types with room_prejoin_state ( #9700 )
...
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
2021-03-30 12:12:44 +01:00
Erik Johnston
963f4309fe
Make RateLimiter class check for ratelimit overrides ( #9711 )
...
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited.
We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits.
Fixes #9663
2021-03-30 12:06:09 +01:00
Erik Johnston
78e48f61bf
1.31.0rc1
2021-03-30 11:19:21 +01:00
Patrick Cloke
01dd90b0f0
Add type hints to DictionaryCache and TTLCache. ( #9442 )
2021-03-29 12:15:33 -04:00
Patrick Cloke
da75d2ea1f
Add type hints for the federation sender. ( #9681 )
...
Includes an abstract base class which both the FederationSender
and the FederationRemoteSendQueue must implement.
2021-03-29 11:43:20 -04:00
Richard van der Hoff
4bbd535450
Update the OIDC sample config ( #9695 )
...
I've reiterated the advice about using `oidc` to migrate, since I've seen a few
people caught by this.
I've also removed a couple of the examples as they are duplicating the OIDC
documentation, and I think they might be leading people astray.
2021-03-29 15:40:11 +01:00
Jonathan de Jong
fc53a606e4
Fix re.Pattern mypy error on 3.6 ( #9703 )
2021-03-29 09:40:45 -04:00
Richard van der Hoff
ad8690a26c
Fix the suggested pip incantation for cryptography ( #9699 )
...
If you have the wrong version of `cryptography` installed, synapse suggests:
```
To install run:
pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6''
```
However, the use of ' inside '...' doesn't work, so when you run this, you get
an error.
2021-03-29 11:55:33 +01:00
Richard van der Hoff
7c8402ddb8
Suppress CryptographyDeprecationWarning ( #9698 )
...
This warning is somewhat confusing to users, so let's suppress it
2021-03-26 17:33:55 +00:00
Erik Johnston
b5efcb577e
Make it possible to use dmypy ( #9692 )
...
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.
`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
2021-03-26 16:49:46 +00:00
Erik Johnston
019010964d
Merge branch 'master' into develop
2021-03-26 12:26:58 +00:00
Erik Johnston
c6f8e8086c
1.30.1
2021-03-26 12:03:29 +00:00
Erik Johnston
12d6184713
Explicitly upgrade openssl in docker file and enforce new version of cryptography ( #9697 )
2021-03-26 12:00:25 +00:00
Andrew Morgan
fae81f2f68
Add a storage method for returning all current presence from all users ( #9650 )
...
Split off from https://github.com/matrix-org/synapse/pull/9491
Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: b588f16e39/synapse/handlers/presence.py (L1131-L1133)
2021-03-25 10:34:23 +00:00
Erik Johnston
c602ba8336
Fixed undefined variable error in catchup ( #9664 )
...
Broke in #9640
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-24 16:12:47 +00:00
Jonathan de Jong
4c3827f2c1
Enable addtional flake8-bugbear linting checks. ( #9659 )
2021-03-24 09:34:30 -04:00
Richard van der Hoff
c73cc2c2ad
Spaces summary: call out to other servers ( #9653 )
...
When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data.
Fixes : #9447
2021-03-24 12:45:39 +00:00
Patrick Cloke
af387cf52a
Add type hints to misc. files. ( #9676 )
2021-03-24 06:49:01 -04:00
Patrick Cloke
7e8dc9934e
Add a type hints for service notices to the HomeServer object. ( #9675 )
2021-03-24 06:48:46 -04:00
Erik Johnston
e550ab17ad
Increase default join burst ratelimiting ( #9674 )
...
It's legitimate behaviour to try and join a bunch of rooms at once.
2021-03-23 14:52:20 +00:00
Jonathan de Jong
0caf2a338e
Fix federation stall on concurrent access errors ( #9639 )
2021-03-23 13:52:30 +00:00
Richard van der Hoff
4ecba9bd5c
Federation API for Space summary ( #9652 )
...
Builds on the work done in #9643 to add a federation API for space summaries.
There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
2021-03-23 11:51:12 +00:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. ( #9665 )
2021-03-23 07:12:48 -04:00
Andrew Morgan
5b268997bd
Allow providing credentials to HTTPS_PROXY ( #9657 )
...
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70
This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization )) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.
Also added some type hints.
2021-03-22 17:20:47 +00:00
Ankit Dobhal
d66f9070cd
Fixed code misc. quality issues ( #9649 )
...
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
2021-03-22 11:18:13 -04:00
Erik Johnston
d600d4506b
Merge branch 'master' into develop
2021-03-22 13:36:36 +00:00
Erik Johnston
e2904f720d
1.30.0
2021-03-22 13:15:55 +00:00
Brendan Abolivier
b6ed4f55ac
Incorporate review
2021-03-19 18:19:50 +01:00
Brendan Abolivier
592d6305fd
Merge branch 'develop' into babolivier/msc3026
2021-03-19 16:12:40 +01:00
Brendan Abolivier
0b56481caa
Fix lint
2021-03-19 16:11:08 +01:00
Richard van der Hoff
066068f034
fix mypy
2021-03-19 12:20:11 +00:00
Patrick Cloke
9b0e3009fa
Fix type-hints from bad merge.
2021-03-18 14:40:56 -04:00
Richard van der Hoff
004234f03a
Initial spaces summary API ( #9643 )
...
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
2021-03-18 18:24:16 +00:00
Brendan Abolivier
066c703729
Move support for MSC3026 behind an experimental flag
2021-03-18 18:37:19 +01:00
Dirk Klimpel
8dd2ea65a9
Consistently check whether a password may be set for a user. ( #9636 )
2021-03-18 12:54:08 -04:00
Erik Johnston
dd71eb0f8a
Make federation catchup send last event from any server. ( #9640 )
...
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.
Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
2021-03-18 15:52:26 +00:00
Brendan Abolivier
405aeb0b2c
Implement MSC3026: busy presence state
2021-03-18 16:34:47 +01:00
Andrew Morgan
7b06f85c0e
Ensure we use a copy of the event content dict before modifying it in serialize_event ( #9585 )
...
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)
2021-03-17 16:51:55 +00:00
Patrick Cloke
cc324d53fe
Fix up types for the typing handler. ( #9638 )
...
By splitting this to two separate methods the callers know
what methods they can expect on the handler.
2021-03-17 11:30:21 -04:00
Hubert Chathi
73dbce5523
only save remote cross-signing keys if they're different from the current ones ( #9634 )
...
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-17 11:04:57 -04:00
Erik Johnston
ad721fc559
Fix bad naming of storage function ( #9637 )
...
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
2021-03-17 13:20:08 +00:00
Richard van der Hoff
567f88f835
Prep work for removing outlier from internal_metadata ( #9411 )
...
* Populate `internal_metadata.outlier` based on `events` table
Rather than relying on `outlier` being in the `internal_metadata` column,
populate it based on the `events.outlier` column.
* Move `outlier` out of InternalMetadata._dict
Ultimately, this will allow us to stop writing it to the database. For now, we
have to grandfather it back in so as to maintain compatibility with older
versions of Synapse.
2021-03-17 12:33:18 +00:00
Patrick Cloke
b449af0379
Add type hints to the room member handler. ( #9631 )
2021-03-17 07:14:39 -04:00
Jonathan de Jong
27d2820c33
Enable flake8-bugbear, but disable most checks. ( #9499 )
...
* Adds B00 to ignored checks.
* Fixes remaining issues.
2021-03-16 14:19:27 -04:00
Hubbe
dd5e5dc1d6
Add SSO attribute requirements for OIDC providers ( #9609 )
...
Allows limiting who can login using OIDC via the claims
made from the IdP.
2021-03-16 11:46:07 -04:00
Dirk Klimpel
8000cf1315
Return m.change_password.enabled=false if local database is disabled ( #9588 )
...
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
2021-03-16 11:44:25 -04:00
Andrew Morgan
d315e96443
1.30.0rc1
2021-03-16 13:45:46 +00:00
Andrew Morgan
847ecdd8fa
Pass SSO IdP information to spam checker's registration function ( #9626 )
...
Fixes https://github.com/matrix-org/synapse/issues/9572
When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.
This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
2021-03-16 12:41:41 +00:00
Patrick Cloke
1383508f29
Handle an empty cookie as an invalid macaroon. ( #9620 )
...
* Handle an empty cookie as an invalid macaroon.
* Newsfragment
2021-03-16 11:29:35 +00:00
Richard van der Hoff
dd69110d95
Add support for stable MSC2858 API ( #9617 )
...
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
2021-03-16 11:21:26 +00:00
Richard van der Hoff
5b5bc188cf
Clean up config settings for stats ( #9604 )
...
... and complain if people try to turn it off.
2021-03-16 10:57:54 +00:00
Andrew Morgan
1b0eaed21f
Prevent bundling aggregations for state events ( #9619 )
...
There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
2021-03-16 10:27:51 +00:00
Richard van der Hoff
1c8a2541da
Fix Internal Server Error on GET /saml2/authn_response ( #9623 )
...
* Fix Internal Server Error on `GET /saml2/authn_response`
Seems to have been introduced in #8765 (Synapse 1.24.0)
* Fix newsfile
2021-03-16 10:20:20 +00:00
Patrick Cloke
d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. ( #9608 )
2021-03-15 11:14:39 -04:00
Erik Johnston
026503fa3b
Don't go into federation catch up mode so easily ( #9561 )
...
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492 .
Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
2021-03-15 14:42:40 +00:00
Richard van der Hoff
af2248f8bf
Optimise missing prev_event handling ( #9601 )
...
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from
the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events,
we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote
server for the state at those prev_events, and then we may need to then ask the remote server for any events
in that state which we don't already have, as well as the auth events for those missing state events, so that we
can auth them.
This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state
events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the
observation that we are currently loading all of those auth events into memory at the start of the operation, but
we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and
leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're
actually going to need, but it doesn't.)
The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about
60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number
of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache
churn. (NB I've already tripled the size of the cache from its default of 10K).
Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into
a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be
returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different
(ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting
that we clean it up.
2021-03-15 13:51:02 +00:00
Patrick Cloke
55da8df078
Fix additional type hints from Twisted 21.2.0. ( #9591 )
2021-03-12 11:37:57 -05:00
Richard van der Hoff
1e67bff833
Reject concurrent transactions ( #9597 )
...
If more transactions arrive from an origin while we're still processing the
first one, reject them.
Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489
2021-03-12 15:14:55 +00:00
Richard van der Hoff
2b328d7e02
Improve logging when processing incoming transactions ( #9596 )
...
Put the room id in the logcontext, to make it easier to understand what's going on.
2021-03-12 15:08:03 +00:00
Richard van der Hoff
464e5da7b2
Add logging for redis connection setup ( #9590 )
2021-03-11 18:35:09 +00:00
Patrick Cloke
e55bd0e110
Add tests for blacklisting reactor/agent. ( #9563 )
2021-03-11 09:15:22 -05:00
Dirk Klimpel
70d1b6abff
Re-Activating account when local passwords are disabled ( #9587 )
...
Fixes : #8393
2021-03-11 13:52:32 +00:00
Richard van der Hoff
a7a3790066
Convert Requester to attrs ( #9586 )
...
... because namedtuples suck
Fix up a couple of other annotations to keep mypy happy.
2021-03-10 18:15:56 +00:00
Richard van der Hoff
1107214a1d
Fix the auth provider on the logins metric ( #9573 )
...
We either need to pass the auth provider over the replication api, or make sure
we report the auth provider on the worker that received the request. I've gone
with the latter.
2021-03-10 18:15:03 +00:00
Patrick Cloke
2a99cc6524
Use the chain cover index in get_auth_chain_ids. ( #9576 )
...
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
2021-03-10 09:57:59 -05:00
Patrick Cloke
918f6ed827
Fix a bug in the background task for purging chain cover. ( #9583 )
2021-03-10 08:55:52 -05:00
Patrick Cloke
dc51d8ffaf
Add a background task to purge unused chain IDs. ( #9542 )
...
This is a companion change to apply the fix in #9498 /
922788c604
2021-03-09 11:22:25 -05:00
Richard van der Hoff
eaada74075
JWT OIDC secrets for Sign in with Apple ( #9549 )
...
Apple had to be special. They want a client secret which is generated from an EC key.
Fixes #9220 . Also fixes #9212 while I'm here.
2021-03-09 15:03:37 +00:00
Erik Johnston
9cd18cc588
Retry 5xx errors in federation client ( #9567 )
...
Fixes #8915
2021-03-09 13:15:12 +00:00
Patrick Cloke
7fdc6cefb3
Fix additional type hints. ( #9543 )
...
Type hint fixes due to Twisted 21.2.0 adding type hints.
2021-03-09 07:41:32 -05:00
Patrick Cloke
075c16b410
Handle image transparency better when thumbnailing. ( #9473 )
...
Properly uses RGBA mode for 1- and 8-bit images with transparency
(instead of RBG mode).
2021-03-09 07:37:09 -05:00
Jonathan de Jong
9898470e7d
Add logging to ObservableDeferred callbacks ( #9523 )
2021-03-09 11:09:31 +00:00
Matthew Hodgson
0764d0c6e5
quick config comment tweak to clarify allow_profile_lookup_over_federation
2021-03-08 21:52:04 +00:00
Jonathan de Jong
d6196efafc
Add ResponseCache tests. ( #9458 )
2021-03-08 14:00:07 -05:00
Dirk Klimpel
7076eee4b9
Add type hints to purge room and server notice admin API. ( #9520 )
2021-03-08 10:34:38 -05:00
Erik Johnston
b988b07bb0
Merge branch 'master' into develop
2021-03-08 14:06:35 +00:00
Erik Johnston
15c788e22d
1.29.0
2021-03-08 13:52:13 +00:00
Patrick Cloke
58114f8a17
Create a SynapseReactor type which incorporates the necessary reactor interfaces. ( #9528 )
...
This helps fix some type hints when running with Twisted 21.2.0.
2021-03-08 08:25:43 -05:00
Richard van der Hoff
8a4b3738f3
Replace last_*_pdu_age metrics with timestamps ( #9540 )
...
Following the advice at
https://prometheus.io/docs/practices/instrumentation/#timestamps-not-time-since ,
it's preferable to export unix timestamps, not ages.
There doesn't seem to be any particular naming convention for timestamp
metrics.
2021-03-04 16:40:18 +00:00
Richard van der Hoff
df425c2c63
Prometheus metrics for logins and registrations ( #9511 )
...
Add prom metrics for number of users successfully registering and logging in, by SSO provider.
2021-03-04 16:39:27 +00:00
Richard van der Hoff
7eb6e39a8f
Record the SSO Auth Provider in the login token ( #9510 )
...
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
2021-03-04 14:44:22 +00:00
Erik Johnston
3f49d80dcf
1.29.0rc1
2021-03-04 10:12:53 +00:00
Patrick Cloke
33a02f0f52
Fix additional type hints from Twisted upgrade. ( #9518 )
2021-03-03 15:47:38 -05:00
Richard van der Hoff
4db07f9aef
Set X-Forwarded-Proto header when frontend-proxy proxies a request ( #9539 )
...
Should fix some remaining warnings
2021-03-03 18:49:08 +00:00
Erik Johnston
a4fa044c00
Fix 'rejected_events_metadata' background update ( #9537 )
...
Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.
2021-03-03 16:04:24 +00:00
Patrick Cloke
922788c604
Purge chain cover tables when purging events. ( #9498 )
2021-03-03 11:04:08 -05:00
Dirk Klimpel
d790d0d314
Add type hints to user admin API. ( #9521 )
2021-03-03 08:09:39 -05:00
Patrick Cloke
0c330423bc
Bump the mypy and mypy-zope versions. ( #9529 )
2021-03-03 07:19:19 -05:00
Erik Johnston
16f9f93eb7
Make deleting stale pushers a background update ( #9536 )
2021-03-03 12:08:16 +00:00
Aaron Raimist
0279e0e086
Prevent presence background jobs from running when presence is disabled ( #9530 )
...
Prevent presence background jobs from running when presence is disabled
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-03-03 10:21:46 +00:00
Patrick Cloke
aee10768d8
Revert "Fix #8518 (sync requests being cached wrongly on timeout) ( #9358 )"
...
This reverts commit f5c93fc993#9507 ) and additional
review feedback being provided.
2021-03-02 09:43:34 -05:00
Erik Johnston
7f5d753d06
Re-run rejected metadata background update. ( #9503 )
...
It landed in schema version 58 after 59 had been created, causing some
servers to not run it. The main effect of was that not all rooms had
their chain cover calculated correctly. After the BG updates complete
the chain covers will get fixed when a new state event in the affected
rooms is received.
2021-03-02 14:31:23 +00:00
Erik Johnston
16108c579d
Fix SQL delta file taking a long time to run ( #9516 )
...
Fixes #9504
2021-03-02 14:05:01 +00:00
Dirk Klimpel
f00c4e7af0
Add type hints to device and event report admin API ( #9519 )
2021-03-02 09:31:12 +00:00
Patrick Cloke
ad8589d392
Fix a bug when a room alias is given to the admin join endpoint ( #9506 )
2021-03-01 13:59:01 -05:00
Patrick Cloke
16ec8c3272
(Hopefully) stop leaking file descriptors in media repo. ( #9497 )
...
By consuming the response if the headers imply that the
content is too large.
2021-03-01 12:45:00 -05:00
Patrick Cloke
a0bc9d387e
Use the proper Request in type hints. ( #9515 )
...
This also pins the Twisted version in the mypy job for CI until
proper type hints are fixed throughout Synapse.
2021-03-01 12:23:46 -05:00
Jonathan de Jong
e12077a78a
Allow bytecode again ( #9502 )
...
In #75 , bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again.
Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-02-26 18:30:54 +00:00
Tim Leung
ddb240293a
Add support for no_proxy and case insensitive env variables ( #9372 )
...
### Changes proposed in this PR
- Add support for the `no_proxy` and `NO_PROXY` environment variables
- Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519)bdb941be42/Lib/urllib/request.py (L2488)#9306 which also uses `ProxyAgent`
Signed-off-by: Timothy Leung tim95@hotmail.co.uk
2021-02-26 17:37:57 +00:00
Richard van der Hoff
15090de850
SSO: redirect to public URL before setting cookies ( #9436 )
...
... otherwise, we don't get the cookie back.
2021-02-26 14:02:06 +00:00
Erik Johnston
2566dc57ce
Test that we require validated email for email pushers ( #9496 )
2021-02-25 15:35:14 +00:00
Erik Johnston
1e62d9ee8c
Ensure pushers are deleted for deactivated accounts ( #9285 )
2021-02-25 13:56:55 +00:00
Erik Johnston
1efdcc3e87
Merge branch 'master' into develop
2021-02-25 10:53:31 +00:00
Erik Johnston
b5c4fe1971
1.28.0
2021-02-25 10:22:07 +00:00
Richard van der Hoff
d8e95e5452
Add support for X-Forwarded-Proto ( #9472 )
...
rewrite XForwardedForRequest to set `isSecure()` based on
`X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
2021-02-24 18:11:33 +00:00
Erik Johnston
7cc571510b
Add SQL delta for deleting stale pushers ( #9479 )
2021-02-24 17:21:10 +00:00
Jonathan de Jong
f5c93fc993
Fix #8518 (sync requests being cached wrongly on timeout) ( #9358 )
...
This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false".
This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values.
Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-02-24 13:57:00 +00:00
Erik Johnston
2927921942
Clean up ShardedWorkerHandlingConfig ( #9466 )
...
* Split ShardedWorkerHandlingConfig
This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).
* Remove special cases in ShardedWorkerHandlingConfig.
`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.
To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
2021-02-24 13:23:18 +00:00
Erik Johnston
0b5c967813
Refactor to ensure we call check_consistency ( #9470 )
...
The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
2021-02-24 10:13:53 +00:00
Patrick Cloke
7292b7c0eb
Add back the deprecated SAML endpoint. ( #9474 )
2021-02-23 12:57:37 -05:00
Richard van der Hoff
713145d3de
Add a comment about systemd-python. ( #9464 )
...
This confused me for a while.
2021-02-23 13:42:36 +00:00
Patrick Cloke
65a9eb8994
Include newly added sequences in the port DB script. ( #9449 )
...
And ensure the consistency of `event_auth_chain_id`.
2021-02-23 07:33:24 -05:00
Erik Johnston
66f4949e7f
Fix deleting pushers when using sharded pushers. ( #9465 )
2021-02-22 21:14:42 +00:00
Richard van der Hoff
1b2d6d55c5
Remove vestiges of uploads_path config ( #9462 )
...
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
2021-02-22 19:54:49 +00:00
Dirk Klimpel
71c9f8de6d
Add an order_by field to list users' media admin API. ( #8978 )
2021-02-22 14:38:51 -05:00
Andrew Morgan
0a363f9ca4
Remove cache for get_shared_rooms_for_users ( #9416 )
...
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666 )). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.
The cache was added in https://github.com/matrix-org/synapse/pull/7785 , but we forgot to ensure it was invalidated appropriately.
Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
2021-02-22 16:52:45 +00:00
Andrew Morgan
e22b71810e
Clean up the user directory sample config section ( #9385 )
...
The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format ).
This PR cleans that up a bit.
2021-02-22 11:44:31 +00:00
Patrick Cloke
fc8b3d8809
Ratelimit cross-user key sharing requests. ( #8957 )
2021-02-19 13:20:34 -05:00
Erik Johnston
179c0953ff
Regenerate exact thumbnails if missing ( #9438 )
2021-02-19 17:09:57 +00:00
Patrick Cloke
a1901abd6b
Add documentation and type hints to parse_duration. ( #9432 )
2021-02-19 08:32:21 -05:00
Patrick Cloke
c4a55ac4a4
Fix style checking due to updated black.
2021-02-19 08:19:54 -05:00
Patrick Cloke
d9f1dccba9
Synapse 1.28.0rc1 (2021-02-19)
...
==============================
Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).
This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.
Removal warning
---------------
The v1 list accounts API is deprecated and will be removed in a future release.
This API was undocumented and misleading. It can be replaced by the
[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts ),
which has been available since Synapse 1.7.0 (2019-12-13).
Please check if you're using any scripts which use the admin API and replace
`GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.
Features
--------
- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150 ))
- Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300 ), [\#9301](https://github.com/matrix-org/synapse/issues/9301 ))
- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311 ))
- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376 ))
- Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400 ))
Bugfixes
--------
- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257 ))
- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296 ))
- Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321 ), [\#9393](https://github.com/matrix-org/synapse/issues/9393 ))
- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333 ))
- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361 ))
- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391 ))
- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395 ))
- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421 ))
- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425 ))
- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426 ))
- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428 ))
Updates to the Docker image
---------------------------
- Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433 ))
Improved Documentation
----------------------
- Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281 ))
- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291 ))
- Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308 ))
- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317 ))
- Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322 ))
- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390 ))
- Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404 ))
- Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407 ))
- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434 ))
Deprecations and Removals
-------------------------
- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429 ))
Internal Changes
----------------
- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003 ))
- Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123 ))
- Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240 ))
- Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299 ))
- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305 ))
- Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307 ))
- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326 ))
- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362 ))
- Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377 ), [\#9396](https://github.com/matrix-org/synapse/issues/9396 ))
- Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381 ))
- Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384 ))
- Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394 ))
- Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAvtsUACgkQM/xY9qcR
MEgEmA/+O3fCMKdyh4soBirnEZa88m1HL4Jyd+jYl7+IuFChZz0k+RYLdQ376GaC
tmT4aGQISdu1wYUwO6zpN5ercvFCJ5OaSJzHinEaHZMnEfWe9SyMNB1oD9h6UYSO
8r7OT2U28pTTAbj/VpSNK8osbx/F9T5o1a7v3PLkB1kJcK9j9oqNrqpdsSW2Cz1B
rGlBKRSICKR4bzfk3dR9Wr4Fcyb+gINn0LTtYYqMWxP1EtRC+VI3NEWOk3OD2dzk
UztvUn0F7rKFmtcBRTXsxuGiGI9IGrd9tyKc7ilBz24eSA95psperUtHNKGwSUaO
kt+bdMfEKM2cMnydMuiCYtdl/M59Rf3ShRE3OZd7TK4S4NoPy6whFsAJf7mBGxfp
RcMwO/7mWRX5qjDRG6x5nM+aCqO0Y8h8WNF4gZtiN2XGyxlU0TBQMBwBM66qAoXO
zCPt4tMlLsOBCABNkWT3W5Ww5KxNeMp2qEA9UnmjezA5XHtSx/DVN+6D5E4Rxw2v
PzS278Sq3YO/JK6TfXsFBkZvDFPmCDdTQfKpoDbT6Hv16CXTz66XdiEg5BPZYIvp
bNurfxr22/RiScxiA73cXbRBHq2qAPTxZo1O40R2LnhX0wQPQ5wfUW8oxZOnuhZ3
bWw4PvhWsZDt2SB9EIKf4RQtSxNlar88C6tpHSC89ScmE6PPWnI=
=Srwm
-----END PGP SIGNATURE-----
Merge tag 'v1.28.0rc1' into develop
Synapse 1.28.0rc1 (2021-02-19)
==============================
Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).
This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.
Removal warning
---------------
The v1 list accounts API is deprecated and will be removed in a future release.
This API was undocumented and misleading. It can be replaced by the
[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts ),
which has been available since Synapse 1.7.0 (2019-12-13).
Please check if you're using any scripts which use the admin API and replace
`GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.
Features
--------
- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150 ))
- Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300 ), [\#9301](https://github.com/matrix-org/synapse/issues/9301 ))
- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311 ))
- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376 ))
- Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400 ))
Bugfixes
--------
- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257 ))
- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296 ))
- Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321 ), [\#9393](https://github.com/matrix-org/synapse/issues/9393 ))
- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333 ))
- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361 ))
- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391 ))
- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395 ))
- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421 ))
- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425 ))
- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426 ))
- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428 ))
Updates to the Docker image
---------------------------
- Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433 ))
Improved Documentation
----------------------
- Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281 ))
- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291 ))
- Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308 ))
- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317 ))
- Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322 ))
- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390 ))
- Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404 ))
- Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407 ))
- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434 ))
Deprecations and Removals
-------------------------
- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429 ))
Internal Changes
----------------
- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003 ))
- Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123 ))
- Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240 ))
- Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299 ))
- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305 ))
- Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307 ))
- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326 ))
- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362 ))
- Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377 ), [\#9396](https://github.com/matrix-org/synapse/issues/9396 ))
- Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381 ))
- Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384 ))
- Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394 ))
- Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408 ))
2021-02-19 08:10:19 -05:00
Patrick Cloke
b114a45f5f
Support not providing an IdP icon when choosing a username. ( #9440 )
2021-02-19 07:48:46 -05:00
Andrew Morgan
8bcfc2eaad
Be smarter about which hosts to send presence to when processing room joins ( #9402 )
...
This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually.
---
When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed.
It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw.
This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence:
* If it was a local user join, send that user's latest presence to all servers in the room
* If it was a remote user join, send the presence for all local users in the room to that homeserver
We deduplicate by inserting all of those pending updates into a dictionary of the form:
```
{
server_name1: {presence_update1, ...},
server_name2: {presence_update1, presence_update2, ...}
}
```
Only after building this dict do we then start sending out presence updates.
2021-02-19 11:37:29 +00:00
Andrew Morgan
13e9029f44
Add a config option to prioritise local users in user directory search results ( #9383 )
...
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.
Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.
This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
2021-02-19 11:02:03 +00:00
Erik Johnston
3d2acc930f
Return a 404 if we don't have the original file
2021-02-19 10:46:18 +00:00
AndrewFerr
9bc74743d5
Add configs to make profile data more private ( #9203 )
...
Add off-by-default configuration settings to:
- disable putting an invitee's profile info in invite events
- disable profile lookup via federation
Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
2021-02-19 09:50:41 +00:00
Patrick Cloke
1381cd05b0
1.28.0rc1
2021-02-18 12:32:49 -05:00
Erik Johnston
b106080fb4
Regenerate exact thumbnails if missing
2021-02-18 17:05:32 +00:00
Patrick Cloke
9ee3b9775f
Remove deprecated SAML2 callback URL since it does not work. ( #9434 )
...
Updates documentation from #9289 and removes a deprecated
endpoint which didn't work as expected.
2021-02-18 11:20:33 -05:00
Rishabh Arya
e17553e185
Parse ui_auth.session_timeout as a duration (instead of treating it as ms) ( #9426 )
2021-02-18 09:18:14 -05:00
Patrick Cloke
8ec2217103
Reduce the memory usage of previewing media files. ( #9421 )
...
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.
It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
2021-02-18 09:01:29 -05:00
Patrick Cloke
43f1c82457
Add back the guard against the user directory stream position not existing. ( #9428 )
...
As the comment says, this guard was there for when the
initial user directory update has yet to happen.
2021-02-18 08:44:19 -05:00
Dirk Klimpel
c8d9383cfb
Add the shadow-banning status to the display user admin API. ( #9400 )
2021-02-17 15:19:23 -05:00
Andrew Morgan
a25661b2eb
Remove dead notify_for_states presence method ( #9408 )
2021-02-17 17:32:26 +00:00
Andrew Morgan
3e5749b99f
Fix only handling the last presence state for each user ( #9425 )
...
This is a small bug that I noticed while working on #8956 .
We have a for-loop which attempts to strip all presence changes for each user except for the final one, as we don't really care about older presence:
9e19c6aab4/synapse/handlers/presence.py (L368-L371)
2021-02-17 17:31:37 +00:00
Patrick Cloke
d2f0ec12d5
Add type hints to groups code. ( #9393 )
2021-02-17 08:41:47 -05:00
Richard van der Hoff
e1071fd625
Support for form_post in OIDC responses ( #9376 )
...
Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
2021-02-17 10:15:14 +00:00
Richard van der Hoff
33f64ca7d6
Allow OIDC config to override discovered values ( #9384 )
...
Fixes #9347
2021-02-16 22:33:09 +00:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase ( #9381 )
...
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md
2021-02-16 22:32:34 +00:00
Richard van der Hoff
3b754aea27
Clean up caching/locking of OIDC metadata load ( #9362 )
...
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
2021-02-16 16:27:38 +00:00
Erik Johnston
0ad087273c
Merge branch 'master' into develop
2021-02-16 13:39:30 +00:00
Patrick Cloke
731e08c63a
Handle missing data in power levels events during room upgrade. ( #9395 )
2021-02-16 08:31:39 -05:00
Erik Johnston
a27c1fd74b
1.27.0
2021-02-16 13:12:02 +00:00
Andrew Morgan
594f2853e0
Remove dead handled_events set in invite_join ( #9394 )
...
This PR removes a set that was created and [initially used](1d2a0040cf (diff-0bc92da3d703202f5b9be2d3f845e375f5b1a6bc6ba61705a8af9be1121f5e42R435-R436)
2021-02-12 22:15:50 +00:00
Patrick Cloke
7950aa8a27
Fix some typos.
2021-02-12 11:14:12 -05:00
Patrick Cloke
2c9b4a5f16
Synapse 1.27.0rc2 (2021-02-11)
...
==============================
Features
--------
- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297 ))
Bugfixes
--------
- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302 ))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313 ))
Improved Documentation
----------------------
- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR
MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS
/tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN
309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL
GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb
/QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP
xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn
ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B
iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4
awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU
2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H
lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0=
=NSyp
-----END PGP SIGNATURE-----
Merge tag 'v1.27.0rc2' into develop
Synapse 1.27.0rc2 (2021-02-11)
==============================
Features
--------
- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297 ))
Bugfixes
--------
- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302 ))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313 ))
Improved Documentation
----------------------
- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310 ))
2021-02-11 11:56:03 -05:00
Patrick Cloke
40de534238
1.27.0rc2
2021-02-11 11:22:29 -05:00
Patrick Cloke
e40d88cff3
Backout changes for automatically calculating the public baseurl. ( #9313 )
...
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
2021-02-11 11:16:54 -05:00
Erik Johnston
6aa87f8ce3
Ensure that we never stop reconnecting to redis ( #9391 )
2021-02-11 16:06:29 +00:00
Patrick Cloke
6dade80048
Combine the CAS & SAML implementations for required attributes. ( #9326 )
2021-02-11 10:05:15 -05:00
Eric Eastwood
80d6dc9783
Remove conflicting sqlite tables that are "reserved" (shadow fts4 tables) ( #9003 )
...
Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests.
Fix #8996
2021-02-10 20:12:57 +00:00
Brendan Abolivier
29ae04af3b
Remove unneeded type constraints on 3rd party protocol lookup responses
2021-02-09 17:50:25 +01:00
Patrick Cloke
3f58fc848d
Type hints and validation improvements. ( #9321 )
...
* Adds type hints to the groups servlet and stringutils code.
* Assert the maximum length of some input values for spec compliance.
2021-02-08 13:59:54 -05:00
Patrick Cloke
0963d39ea6
Handle additional errors when previewing URLs. ( #9333 )
...
* Handle the case of lxml not finding a document tree.
* Parse the document encoding from the XML tag.
2021-02-08 12:33:30 -05:00
David Teller
b0b2cac057
Merge pull request #9150 from Yoric/develop-context
...
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
2021-02-08 15:53:44 +01:00
Jonathan de Jong
d882fbca38
Update type hints for Cursor to match PEP 249. ( #9299 )
2021-02-05 15:39:19 -05:00
Erik Johnston
adc96d4236
Merge branch 'erikj/media_spam_checker' into develop
2021-02-04 17:01:59 +00:00
Erik Johnston
7e8083eb48
Add check_media_file_for_spam spam checker hook
2021-02-04 17:01:30 +00:00
Patrick Cloke
792263c97c
Handle empty rooms when generating email notifications. ( #9257 )
...
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
2021-02-04 10:18:25 -05:00
Patrick Cloke
2ab6e67ab7
Fix escaping of braces in OIDC sample config. ( #9317 )
...
This fixes the Jinja2 templates for the mapping provider.
2021-02-04 09:06:20 -05:00
Jonathan de Jong
2814028ce5
Add experimental support for PyPy. ( #9123 )
...
* Adds proper dependencies.
* Minor fixes in database layer.
2021-02-04 08:29:47 -05:00
Marcus
b0f4119b8b
Add debug logging to DNS SRV requests. ( #9305 )
2021-02-03 16:47:30 -05:00
Richard van der Hoff
3f534d3fdf
Merge branch 'social_login_hotfixes' into develop
2021-02-03 20:34:27 +00:00
Richard van der Hoff
17f2a512f3
Merge remote-tracking branch 'origin/release-v1.27.0' into social_login_hotfixes
2021-02-03 20:33:32 +00:00
Richard van der Hoff
e288499c60
Social login UI polish ( #9301 )
2021-02-03 20:31:23 +00:00
Patrick Cloke
afa18f1baa
Clarify documentation about escaping URLs in templates. ( #9310 )
2021-02-03 14:51:38 -05:00
Richard van der Hoff
ce669863b9
Add debug for OIDC flow ( #9307 )
2021-02-03 19:45:34 +00:00
Richard van der Hoff
7a0dcea3e5
social login Fix username validation javascript ( #9297 )
...
* fix validation and don't use built-in validation UI
Co-authored-by: Bruno Windels <brunow@element.io>
2021-02-03 17:52:55 +00:00
Richard van der Hoff
f20dadb649
Fix formatting for "bad session" error during sso registration flow ( #9296 )
2021-02-03 16:13:09 +00:00
dykstranet
e4cdecb310
config: Add detail to auto_join_rooms comment ( #9291 )
...
config: Add detail to auto_join_rooms comment
Signed-off-by: Gary Dykstra <gary@dykstranet.com>
2021-02-03 15:21:30 +00:00
Tim Gates
e1943d1353
Typo fix in a comment: subequently -> subsequently. ( #8988 )
2021-02-03 07:24:53 -05:00
Patrick Cloke
4ca054a4ea
Convert blacklisted IPv4 addresses to compatible IPv6 addresses. ( #9240 )
...
Also add a few more IP ranges to the default blacklist.
2021-02-03 07:13:46 -05:00
Erik Johnston
ff55300b91
Honour ratelimit flag for application services for invite ratelimiting ( #9302 )
2021-02-03 10:17:37 +00:00
Richard van der Hoff
96e460df2e
social login: add noopener to terms link ( #9300 )
2021-02-02 18:35:28 +00:00
Erik Johnston
2610930721
1.27.0rc1
2021-02-02 13:32:05 +00:00
Travis Ralston
b60bb28bbc
Add an admin API to get the current room state ( #9168 )
...
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.
This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
2021-02-02 11:16:29 +00:00
Richard van der Hoff
8f75bf1df7
Put SAML callback URI under /_synapse/client. ( #9289 )
2021-02-02 09:43:50 +00:00
Richard van der Hoff
846b9d3df0
Put OIDC callback URI under /_synapse/client. ( #9288 )
2021-02-01 22:56:01 +00:00
Richard van der Hoff
8fee6a3ab2
Merge branch 'social_login' into develop
2021-02-01 18:48:11 +00:00
Richard van der Hoff
351845452c
fix broken HTML tag
2021-02-01 18:47:01 +00:00
Richard van der Hoff
5963426b95
Merge branch 'social_login' into develop
2021-02-01 18:46:12 +00:00
Bruno Windels
f30c3a99be
make primary button not wider than viewport
2021-02-01 18:39:17 +00:00
Richard van der Hoff
c543bf87ec
Collect terms consent from the user during SSO registration ( #9276 )
2021-02-01 18:37:41 +00:00
Richard van der Hoff
e5d70c8a82
Improve styling and wording of SSO UIA templates ( #9286 )
...
fixes #9171
2021-02-01 18:36:04 +00:00
Patrick Cloke
5d38a3c97f
Refactor email summary generation. ( #9260 )
...
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
was tweaked to depend on whether there was 1 sender or multiple senders,
not based on if there was 1 room or multiple rooms.
2021-02-01 13:09:39 -05:00
Richard van der Hoff
419313b06a
Improve styling and wording of SSO error templates ( #9287 )
2021-02-01 18:01:15 +00:00
Richard van der Hoff
85c56b5a67
Make importing display name and email optional ( #9277 )
2021-02-01 17:30:42 +00:00
Richard van der Hoff
18ab35284a
Merge branch 'social_login' into develop
2021-02-01 17:28:37 +00:00
Jan Christian Grünhage
43dd93bb26
Add phone home stats for encrypted messages. ( #9283 )
...
Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
2021-02-01 17:06:22 +00:00
Andrew Morgan
a800603561
Prevent email UIA failures from raising a LoginError ( #9265 )
...
Context, Fixes: https://github.com/matrix-org/synapse/issues/9263
In the past to fix an issue with old Riots re-requesting threepid validation tokens, we raised a `LoginError` during UIA instead of `InteractiveAuthIncompleteError`. This is now breaking the way Tchap logs in - which isn't standard, but also isn't disallowed by the spec.
An easy fix is just to remove the 4 year old workaround.
2021-02-01 15:54:39 +00:00
Richard van der Hoff
4167494c90
Replace username picker with a template ( #9275 )
...
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.
I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
2021-02-01 15:52:50 +00:00
Richard van der Hoff
8aed29dc61
Improve styling and wording of SSO redirect confirm template ( #9272 )
2021-02-01 15:50:56 +00:00
Richard van der Hoff
9c715a5f19
Fix SSO on workers ( #9271 )
...
Fixes #8966 .
* Factor out build_synapse_client_resource_tree
Start a function which will mount resources common to all workers.
* Move sso init into build_synapse_client_resource_tree
... so that we don't have to do it for each worker
* Fix SSO-login-via-a-worker
Expose the SSO login endpoints on workers, like the documentation says.
* Update workers config for new endpoints
Add documentation for endpoints recently added (#8942 , #9017 , #9262 )
* remove submit_token from workers endpoints list
this *doesn't* work on workers (yet).
* changelog
* Add a comment about the odd path for SAML2Resource
2021-02-01 15:47:59 +00:00
Richard van der Hoff
f78d07bf00
Split out a separate endpoint to complete SSO registration ( #9262 )
...
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
2021-02-01 13:15:51 +00:00
Ivan Shapovalov
13c7ab8181
Fixes for PyPy compatibility ( #9270 )
...
* synapse.app.base: only call gc.freeze() on CPython
gc.freeze() is an implementation detail of CPython garbage collector,
and notably does not exist on PyPy.
Rather than playing whack-a-mole and skipping the call when under PyPy,
simply restrict it to CPython because the whole gc module is
implementation-defined.
Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
2021-01-30 17:22:05 +00:00
Erik Johnston
f2c1560eca
Ratelimit invites by room and target user ( #9258 )
2021-01-29 16:38:29 +00:00
Richard van der Hoff
0d81a6fa3e
Merge branch 'social_login' into develop
2021-01-28 22:08:11 +00:00
Erik Johnston
4b73488e81
Ratelimit 3PID /requestToken API ( #9238 )
2021-01-28 17:39:21 +00:00
Erik Johnston
54a6afeee3
Cache config options in SSL verification ( #9255 )
...
Reading from the config object is *slow*.
2021-01-28 17:38:59 +00:00
David Teller
31d072aea0
FIXUP: linter
2021-01-28 16:53:40 +01:00
Patrick Cloke
a78016dadf
Add type hints to E2E handler. ( #9232 )
...
This finishes adding type hints to the `synapse.handlers` module.
2021-01-28 08:34:19 -05:00
David Teller
93f84e0373
FIXUP: Making get_event_context a bit more paranoid
2021-01-28 12:31:07 +01:00
David Teller
b755f60ce2
FIXUP: Removing awaitable
2021-01-28 12:31:07 +01:00
David Teller
a764869623
FIXUP: Doc
2021-01-28 12:31:07 +01:00
David Teller
b859919acc
FIXUP: Now testing that the user is admin!
2021-01-28 12:31:07 +01:00
David Teller
de7f049527
FIXUP: Don't filter events at all for admin/v1/rooms/.../context/...
2021-01-28 12:31:07 +01:00
David Teller
10332c175c
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
...
Signed-off-by: David Teller <davidt@element.io>
2021-01-28 12:29:49 +01:00
Richard van der Hoff
a083aea396
Add 'brand' field to MSC2858 response ( #9242 )
...
We've decided to add a 'brand' field to help clients decide how to style the
buttons.
Also, fix up the allowed characters for idp_id, while I'm in the area.
2021-01-27 21:31:45 +00:00
Richard van der Hoff
869667760f
Support for scraping email addresses from OIDC providers ( #9245 )
2021-01-27 21:28:59 +00:00
Pankaj Yadav
2e537a0280
Check if a user is in the room before sending a PowerLevel event on their behalf ( #9235 )
2021-01-27 17:38:08 +00:00
Richard van der Hoff
fbd9de6d1f
Synapse 1.26.0 (2021-01-27)
...
===========================
This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.
No significant changes since 1.26.0rc2.
Synapse 1.26.0rc2 (2021-01-25)
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193 ), [\#9195](https://github.com/matrix-org/synapse/issues/9195 ))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210 ))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189 ))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204 ))
Synapse 1.26.0rc1 (2021-01-20)
==============================
This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.
Features
--------
- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015 ), [\#9017](https://github.com/matrix-org/synapse/issues/9017 ), [\#9036](https://github.com/matrix-org/synapse/issues/9036 ), [\#9067](https://github.com/matrix-org/synapse/issues/9067 ), [\#9081](https://github.com/matrix-org/synapse/issues/9081 ), [\#9082](https://github.com/matrix-org/synapse/issues/9082 ), [\#9105](https://github.com/matrix-org/synapse/issues/9105 ), [\#9107](https://github.com/matrix-org/synapse/issues/9107 ), [\#9109](https://github.com/matrix-org/synapse/issues/9109 ), [\#9110](https://github.com/matrix-org/synapse/issues/9110 ), [\#9127](https://github.com/matrix-org/synapse/issues/9127 ), [\#9153](https://github.com/matrix-org/synapse/issues/9153 ), [\#9154](https://github.com/matrix-org/synapse/issues/9154 ), [\#9177](https://github.com/matrix-org/synapse/issues/9177 ))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091 ))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159 ))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024 ))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176 ) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984 ))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086 ))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932 ))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948 ))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042 ), [\#9043](https://github.com/matrix-org/synapse/issues/9043 ), [\#9044](https://github.com/matrix-org/synapse/issues/9044 ), [\#9130](https://github.com/matrix-org/synapse/issues/9130 ))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068 ))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092 ))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104 ), [\#9166](https://github.com/matrix-org/synapse/issues/9166 ))
Bugfixes
--------
- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023 ))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028 ))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051 ))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053 ))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054 ))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059 ))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070 ))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071 ))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114 ), [\#9116](https://github.com/matrix-org/synapse/issues/9116 ))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117 ))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128 ))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108 ))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145 ))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161 ))
Improved Documentation
----------------------
- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997 ))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035 ))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040 ))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057 ))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151 ))
Deprecations and Removals
-------------------------
- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039 ))
Internal Changes
----------------
- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868 ), [\#9029](https://github.com/matrix-org/synapse/issues/9029 ), [\#9115](https://github.com/matrix-org/synapse/issues/9115 ), [\#9118](https://github.com/matrix-org/synapse/issues/9118 ), [\#9124](https://github.com/matrix-org/synapse/issues/9124 ))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939 ))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016 ))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018 ))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025 ))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030 ))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031 ))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033 ))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038 ))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041 ))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055 ))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058 ))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063 ))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069 ))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080 ))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093 ))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098 ))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106 ))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112 ))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125 ))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144 ))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146 ))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmARjqYACgkQM/xY9qcR
MEiknRAAtga+KXEG40xg7w78XXQk6nXIhhi30nYue9/SeHMTCmLmd1+f0yTySrRz
JMc/xe9VP1th5DThHZeHJf0tLIVz/VFA5gr/fmk44FFEFvc77IhttVHSXbYZLuM8
komOzLTJy7ueVSpyF9xVfEEz6UIRc098b2KHjuBh3SGlZAOmFuFHUMVBeEuYo80E
ayVWXPNP+7U5y1qLA9uY/oxMsFjl6bPvjak3YDyGWCr2Y4nOhM24AwccErs0yAtu
kCRm9JR0yD72uRLyaGyMZ9Atg4o01BD7hu0ilxhbRIdAvI5Cdk0E7rZshoLr33wk
Ls4XlK9SxJSV4vEu44OJBCRnDjQS9HQnwYk13z+H5HrJcxXRyIeCwRD7aofzNY8O
kNg/1oasBDiW2iVoLJ4Y0nWLHPcNlWJxUzkkjlF8BmEUQjURNVdh0n3y3/ST+T1S
scex0J9Z95HNoLfRYID0adrYqWTOl5GLzy9C1AwqlLWY79y6YsB4LDmmfQgoTKCM
VICSYtEKWph6WXdpd/kJsTgHyG3UFAMDraWVez7TxXpc7lpvOMkld+0oQnHWAXC3
XRc+63jTs6pFJ74M+ziqOrC/1ORg1DK8OLIm/TskLoYnn6L9Dh8ssXeabLv63H8o
cEqETUbZ77rZro0Nz2H1OJG09uTFUzFEy5Ecya7g4DqtRT/Fs9Q=
=I9CI
-----END PGP SIGNATURE-----
Merge tag 'v1.26.0' into social_login
Synapse 1.26.0 (2021-01-27)
===========================
This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.
No significant changes since 1.26.0rc2.
Synapse 1.26.0rc2 (2021-01-25)
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193 ), [\#9195](https://github.com/matrix-org/synapse/issues/9195 ))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210 ))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189 ))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204 ))
Synapse 1.26.0rc1 (2021-01-20)
==============================
This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.
Features
--------
- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015 ), [\#9017](https://github.com/matrix-org/synapse/issues/9017 ), [\#9036](https://github.com/matrix-org/synapse/issues/9036 ), [\#9067](https://github.com/matrix-org/synapse/issues/9067 ), [\#9081](https://github.com/matrix-org/synapse/issues/9081 ), [\#9082](https://github.com/matrix-org/synapse/issues/9082 ), [\#9105](https://github.com/matrix-org/synapse/issues/9105 ), [\#9107](https://github.com/matrix-org/synapse/issues/9107 ), [\#9109](https://github.com/matrix-org/synapse/issues/9109 ), [\#9110](https://github.com/matrix-org/synapse/issues/9110 ), [\#9127](https://github.com/matrix-org/synapse/issues/9127 ), [\#9153](https://github.com/matrix-org/synapse/issues/9153 ), [\#9154](https://github.com/matrix-org/synapse/issues/9154 ), [\#9177](https://github.com/matrix-org/synapse/issues/9177 ))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091 ))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159 ))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024 ))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176 ) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984 ))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086 ))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932 ))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948 ))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042 ), [\#9043](https://github.com/matrix-org/synapse/issues/9043 ), [\#9044](https://github.com/matrix-org/synapse/issues/9044 ), [\#9130](https://github.com/matrix-org/synapse/issues/9130 ))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068 ))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092 ))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104 ), [\#9166](https://github.com/matrix-org/synapse/issues/9166 ))
Bugfixes
--------
- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023 ))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028 ))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051 ))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053 ))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054 ))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059 ))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070 ))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071 ))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114 ), [\#9116](https://github.com/matrix-org/synapse/issues/9116 ))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117 ))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128 ))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108 ))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145 ))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161 ))
Improved Documentation
----------------------
- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997 ))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035 ))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040 ))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057 ))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151 ))
Deprecations and Removals
-------------------------
- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039 ))
Internal Changes
----------------
- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868 ), [\#9029](https://github.com/matrix-org/synapse/issues/9029 ), [\#9115](https://github.com/matrix-org/synapse/issues/9115 ), [\#9118](https://github.com/matrix-org/synapse/issues/9118 ), [\#9124](https://github.com/matrix-org/synapse/issues/9124 ))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939 ))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016 ))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018 ))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025 ))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030 ))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031 ))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033 ))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038 ))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041 ))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055 ))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058 ))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063 ))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069 ))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080 ))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093 ))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098 ))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106 ))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112 ))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125 ))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144 ))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146 ))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157 ))
2021-01-27 17:27:58 +00:00
Richard van der Hoff
7fa1346f93
Merge branch 'social_login' into develop
2021-01-27 17:27:24 +00:00
Patrick Cloke
17b713850f
Merge branch 'master' into develop
2021-01-27 11:13:21 -05:00
Patrick Cloke
e54746bdf7
Clean-up the template loading code. ( #9200 )
...
* Enables autoescape by default for HTML files.
* Adds a new read_template method for reading a single template.
* Some logic clean-up.
2021-01-27 10:59:50 -05:00
Patrick Cloke
73ed289bd2
1.26.0
2021-01-27 10:50:37 -05:00
Richard van der Hoff
a737cc2713
Implement MSC2858 support ( #9183 )
...
Fixes #8928 .
2021-01-27 12:41:24 +00:00
Andrew Morgan
a64c29926e
Pass a dict, instead of None, to modules if a None config is specified in the homeserver config ( #9229 )
...
If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead.
This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](5310808d3b/docs/sample_config.yaml (L1816-L1845)
2021-01-27 11:49:31 +00:00
Patrick Cloke
1baab20352
Add type hints to various handlers. ( #9223 )
...
With this change all handlers except the e2e_* ones have
type hints enabled.
2021-01-26 10:50:21 -05:00
Patrick Cloke
26837d5dbe
Do not require the CAS service URL setting (use public_baseurl instead). ( #9199 )
...
The current configuration is handled for backwards compatibility,
but is considered deprecated.
2021-01-26 10:49:25 -05:00
Erik Johnston
dd8da8c5f6
Precompute joined hosts and store in Redis ( #9198 )
2021-01-26 13:57:31 +00:00
Patrick Cloke
4937fe3d6b
Try to recover from unknown encodings when previewing media. ( #9164 )
...
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
2021-01-26 07:32:17 -05:00
Jason Robinson
e5b659e9e1
Merge pull request #9062 from matrix-org/jaywink/admin-forward-extremities
...
Add forward extremities endpoint to rooms admin API
2021-01-26 12:57:38 +02:00
Erik Johnston
a1ff1e967f
Periodically send pings to detect dead Redis connections ( #9218 )
...
This is done by creating a custom `RedisFactory` subclass that
periodically pings all connections in its pool.
We also ensure that the `replyTimeout` param is non-null, so that we
timeout waiting for the reply to those pings (and thus triggering a
reconnect).
2021-01-26 10:54:54 +00:00
Jason Robinson
4936fc59fc
Fix get forward extremities query
...
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-01-26 10:21:02 +02:00
Jason Robinson
cee4010f94
Merge branch 'develop' into jaywink/admin-forward-extremities
...
# Conflicts:
# synapse/rest/admin/__init__.py
2021-01-26 10:15:32 +02:00
Jason Robinson
e20f18a766
Make natural join inner join
...
Co-authored-by: Erik Johnston <erik@matrix.org>
2021-01-26 10:13:35 +02:00
Patrick Cloke
5b857b77f7
Don't error if deleting a non-existent pusher. ( #9121 )
2021-01-25 14:52:30 -05:00
Patrick Cloke
4a55d267ee
Add an admin API for shadow-banning users. ( #9209 )
...
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.
A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.
Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
2021-01-25 14:49:39 -05:00
Richard van der Hoff
65fb3b2e25
Synapse 1.26.0rc2 (2021-01-25)
...
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193 ), [\#9195](https://github.com/matrix-org/synapse/issues/9195 ))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210 ))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189 ))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
+uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
/U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
=Qc7+
-----END PGP SIGNATURE-----
Merge tag 'v1.26.0rc2' into social_login
Synapse 1.26.0rc2 (2021-01-25)
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193 ), [\#9195](https://github.com/matrix-org/synapse/issues/9195 ))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210 ))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189 ))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204 ))
2021-01-25 19:37:58 +00:00
Jason Robinson
fe18882bb5
Merge remote-tracking branch 'origin/develop' into jaywink/admin-forward-extremities
2021-01-25 15:55:54 +02:00
Patrick Cloke
e448dbbf5b
Synapse 1.26.0rc2 (2021-01-25)
...
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193 ), [\#9195](https://github.com/matrix-org/synapse/issues/9195 ))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210 ))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189 ))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
+uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
/U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
=Qc7+
-----END PGP SIGNATURE-----
Merge tag 'v1.26.0rc2' into develop
Synapse 1.26.0rc2 (2021-01-25)
==============================
Bugfixes
--------
- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193 ), [\#9195](https://github.com/matrix-org/synapse/issues/9195 ))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210 ))
Internal Changes
----------------
- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189 ))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204 ))
2021-01-25 08:51:45 -05:00
Patrick Cloke
a01605c136
1.26.0rc2
2021-01-25 08:25:40 -05:00
Patrick Cloke
6f7417c3db
Handle missing content keys when calculating presentable names. ( #9165 )
...
Treat the content as untrusted and do not assume it is of
the proper form.
2021-01-25 07:27:16 -05:00
Jason Robinson
8965b6cfec
Merge branch 'develop' into jaywink/admin-forward-extremities
2021-01-23 21:41:35 +02:00
Jason Robinson
930ba00971
Add depth and received_ts to forward_extremities admin API response
...
Also add a warning on the admin API documentation.
Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-01-23 21:34:32 +02:00
Erik Johnston
056327457f
Fix chain cover update to handle events with duplicate auth events ( #9210 )
2021-01-22 19:44:08 +00:00
Erik Johnston
28f255d5f3
Bump psycopg2 version ( #9204 )
...
As we use `execute_values` with the `fetch` parameter.
2021-01-22 11:14:49 +00:00
Patrick Cloke
a7882f9887
Return a 404 if no valid thumbnail is found. ( #9163 )
...
If no thumbnail of the requested type exists, return a 404 instead
of erroring. This doesn't quite match the spec (which does not define
what happens if no thumbnail can be found), but is consistent with
what Synapse already does.
2021-01-21 14:53:58 -05:00
Erik Johnston
758ed5f1bc
Speed up chain cover calculation ( #9176 )
2021-01-21 17:00:12 +00:00
Erik Johnston
12ec55bfaa
Increase perf of handling concurrent use of StreamIDGenerators. ( #9190 )
...
We have seen a failure mode here where if there are many in flight
unfinished IDs then marking an ID as finished takes a lot of CPU (as
calling deque.remove iterates over the list)
2021-01-21 16:31:51 +00:00
Erik Johnston
939ef657ce
Merge remote-tracking branch 'origin/release-v1.26.0' into develop
2021-01-21 16:05:13 +00:00
Erik Johnston
ccfafac882
Add schema update to fix existing DBs affected by #9193 ( #9195 )
2021-01-21 16:03:25 +00:00
Erik Johnston
b249f002b8
Merge remote-tracking branch 'origin/release-v1.26.0' into develop
2021-01-21 15:09:30 +00:00
Erik Johnston
2506074ef0
Fix receipts or account data not being sent down sync ( #9193 )
...
Introduced in #9104
This wasn't picked up by the tests as this is all fine the first time you run Synapse (after upgrading), but then when you restart the wrong value is pulled from `stream_positions`.
2021-01-21 15:09:09 +00:00
Erik Johnston
7a43482f19
Use execute_batch in more places ( #9188 )
...
* Use execute_batch in more places
* Newsfile
2021-01-21 14:44:12 +00:00
Dirk Klimpel
c55e62548c
Add tests for List Users Admin API ( #9045 )
2021-01-21 09:18:46 -05:00
Richard van der Hoff
42a8e81370
Add a check for duplicate IdP ids ( #9184 )
2021-01-21 13:20:58 +00:00
Richard van der Hoff
b5120f09f1
Merge remote-tracking branch 'origin/release-v1.26.0' into develop
2021-01-21 13:17:07 +00:00
Richard van der Hoff
7447f19702
Prefix idp_id with "oidc-" ( #9189 )
...
... to avoid clashes with other SSO mechanisms
2021-01-21 12:25:02 +00:00
Erik Johnston
eee6fcf5fa
Use execute_batch instead of executemany in places ( #9181 )
...
`execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.
2021-01-21 10:22:53 +00:00
Patrick Cloke
f81d02d75b
Synapse 1.26.0rc1 (2021-01-20)
...
==============================
This release brings a new schema version for Synapse and rolling back to a previous
verious is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.
Features
--------
- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015 ), [\#9017](https://github.com/matrix-org/synapse/issues/9017 ), [\#9036](https://github.com/matrix-org/synapse/issues/9036 ), [\#9067](https://github.com/matrix-org/synapse/issues/9067 ), [\#9081](https://github.com/matrix-org/synapse/issues/9081 ), [\#9082](https://github.com/matrix-org/synapse/issues/9082 ), [\#9105](https://github.com/matrix-org/synapse/issues/9105 ), [\#9107](https://github.com/matrix-org/synapse/issues/9107 ), [\#9109](https://github.com/matrix-org/synapse/issues/9109 ), [\#9110](https://github.com/matrix-org/synapse/issues/9110 ), [\#9127](https://github.com/matrix-org/synapse/issues/9127 ), [\#9153](https://github.com/matrix-org/synapse/issues/9153 ), [\#9154](https://github.com/matrix-org/synapse/issues/9154 ), [\#9177](https://github.com/matrix-org/synapse/issues/9177 ))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091 ))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159 ))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024 ))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176 ) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984 ))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086 ))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932 ))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948 ))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042 ), [\#9043](https://github.com/matrix-org/synapse/issues/9043 ), [\#9044](https://github.com/matrix-org/synapse/issues/9044 ), [\#9130](https://github.com/matrix-org/synapse/issues/9130 ))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068 ))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092 ))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104 ), [\#9166](https://github.com/matrix-org/synapse/issues/9166 ))
Bugfixes
--------
- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023 ))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028 ))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051 ))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053 ))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054 ))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059 ))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070 ))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071 ))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114 ), [\#9116](https://github.com/matrix-org/synapse/issues/9116 ))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117 ))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128 ))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108 ))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145 ))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161 ))
Improved Documentation
----------------------
- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997 ))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035 ))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040 ))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057 ))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151 ))
Deprecations and Removals
-------------------------
- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039 ))
Internal Changes
----------------
- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868 ), [\#9029](https://github.com/matrix-org/synapse/issues/9029 ), [\#9115](https://github.com/matrix-org/synapse/issues/9115 ), [\#9118](https://github.com/matrix-org/synapse/issues/9118 ), [\#9124](https://github.com/matrix-org/synapse/issues/9124 ))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939 ))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016 ))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018 ))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025 ))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030 ))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031 ))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033 ))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038 ))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041 ))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055 ))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058 ))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063 ))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069 ))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080 ))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093 ))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098 ))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106 ))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112 ))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125 ))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144 ))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146 ))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157 ))
-----BEGIN PGP SIGNATURE-----
iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAIVFkACgkQM/xY9qcR
MEhLfQ//Z4xnP5Icx732k2xpdR+JaXX6heNVFm/UuLNd6wq+uXjO/Dgc9IKQ5LXD
kfG+/OnNGDr/eUk4mQ5s/ccBvaTrOid2IIemJd4lUBy/Q5LvPvfKIp08QFr72WJT
7Y4Ma54nNSSMUAdESgj/aFAHjATbfHhxPOZ5OdQHGLAuJ/OUwR2ksasVNCuINbgh
8vrxrzjeYn1Zl3UTBrsdCcat7AFqYjxK/Y0i6JxfSgRwDZjUbW4M2C4OGWZJPUrU
2YTPHangxd+22+HObE2KDEzaHDdCR/Kj4dhCiJBynZ5iucuHSS+mgf39RtbjqWML
01cmXXUV+FH7rnCilmF9rYdmHZ6L9BJxtrQDQU4dB106BX9G7hKR6rYHc9Z6tjOh
3tg0cKWYMXOsTNnvRqiFNh2q0yBKS228HRAwyGosW/6NiYj43ArtYP6knX7dQyBX
0mYNkfry2dyX6Kj5el0i9MOTHYQxfc8apsBm2M3OTNYukgQKA0NhbAquibaAxEEW
2qIqbSp4CUlGPvM7+u+ZGTu0hEbVKKNjqMfXhuY8A4/BWiV2mSqKcEirR5cE/LqY
mq5Ac7vbO4Uh+1xiRw/G9ITi1dqAjYIzhBawlhdUPOh+aINWTUikYYzjUSmm4PbN
H9BTPLA9iAdVRfuWJ9um2G0DdS8Qpx/aIRh3MScXVly6cGCC6Do=
=kgqn
-----END PGP SIGNATURE-----
Merge tag 'v1.26.0rc1' into develop
Synapse 1.26.0rc1 (2021-01-20)
==============================
This release brings a new schema version for Synapse and rolling back to a previous
verious is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.
Features
--------
- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015 ), [\#9017](https://github.com/matrix-org/synapse/issues/9017 ), [\#9036](https://github.com/matrix-org/synapse/issues/9036 ), [\#9067](https://github.com/matrix-org/synapse/issues/9067 ), [\#9081](https://github.com/matrix-org/synapse/issues/9081 ), [\#9082](https://github.com/matrix-org/synapse/issues/9082 ), [\#9105](https://github.com/matrix-org/synapse/issues/9105 ), [\#9107](https://github.com/matrix-org/synapse/issues/9107 ), [\#9109](https://github.com/matrix-org/synapse/issues/9109 ), [\#9110](https://github.com/matrix-org/synapse/issues/9110 ), [\#9127](https://github.com/matrix-org/synapse/issues/9127 ), [\#9153](https://github.com/matrix-org/synapse/issues/9153 ), [\#9154](https://github.com/matrix-org/synapse/issues/9154 ), [\#9177](https://github.com/matrix-org/synapse/issues/9177 ))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091 ))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159 ))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024 ))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176 ) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984 ))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086 ))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932 ))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948 ))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042 ), [\#9043](https://github.com/matrix-org/synapse/issues/9043 ), [\#9044](https://github.com/matrix-org/synapse/issues/9044 ), [\#9130](https://github.com/matrix-org/synapse/issues/9130 ))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068 ))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092 ))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104 ), [\#9166](https://github.com/matrix-org/synapse/issues/9166 ))
Bugfixes
--------
- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023 ))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028 ))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051 ))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053 ))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054 ))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059 ))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070 ))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071 ))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114 ), [\#9116](https://github.com/matrix-org/synapse/issues/9116 ))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117 ))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128 ))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108 ))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145 ))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161 ))
Improved Documentation
----------------------
- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997 ))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035 ))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040 ))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057 ))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151 ))
Deprecations and Removals
-------------------------
- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039 ))
Internal Changes
----------------
- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868 ), [\#9029](https://github.com/matrix-org/synapse/issues/9029 ), [\#9115](https://github.com/matrix-org/synapse/issues/9115 ), [\#9118](https://github.com/matrix-org/synapse/issues/9118 ), [\#9124](https://github.com/matrix-org/synapse/issues/9124 ))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939 ))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016 ))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018 ))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025 ))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030 ))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031 ))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033 ))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038 ))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041 ))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055 ))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058 ))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063 ))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069 ))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080 ))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093 ))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098 ))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106 ))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112 ))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125 ))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144 ))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146 ))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157 ))
2021-01-20 11:27:39 -05:00
Richard van der Hoff
e51b2f3f91
Tighten the restrictions on idp_id ( #9177 )
2021-01-20 08:55:14 -05:00
Richard van der Hoff
0cd2938bc8
Support icons for Identity Providers ( #9154 )
2021-01-20 08:15:14 -05:00
Patrick Cloke
620ecf13b0
Various improvements to the federation client. ( #9129 )
...
* Type hints for `FederationClient`.
* Using `async` functions instead of returning `Awaitable` instances.
2021-01-20 07:59:18 -05:00
Richard van der Hoff
6c0dfd2e8e
Merge remote-tracking branch 'origin/develop' into release-v1.26.0
2021-01-20 12:33:05 +00:00
Richard van der Hoff
fa50e4bf4d
Give public_baseurl a default value ( #9159 )
2021-01-20 12:30:41 +00:00
Patrick Cloke
72822e60be
1.26.0rc1
2021-01-20 07:14:25 -05:00
Patrick Cloke
fa842a9866
Use the account handler in additional places. ( #9166 )
2021-01-20 10:44:52 +00:00
Patrick Cloke
47d48a5853
Validate the server name for the /publicRooms endpoint. ( #9161 )
...
If a remote server name is provided, ensure it is something reasonable
before making remote connections to it.
2021-01-19 14:21:59 -05:00
Richard van der Hoff
73b03722f4
Fix error messages from OIDC config parsing ( #9153 )
...
Make sure we report the correct config path for errors in the OIDC configs.
2021-01-19 14:56:54 +00:00
Erik Johnston
6633a4015a
Allow moving account data and receipts streams off master ( #9104 )
2021-01-18 15:47:59 +00:00
Erik Johnston
f08ef64926
Enforce all replication HTTP clients calls use kwargs ( #9144 )
2021-01-18 15:24:04 +00:00
Patrick Cloke
2b467d0b61
Properly raise an exception when the body exceeds the max size. ( #9145 )
...
...instead of just creating the exception object and doing nothing with it.
2021-01-18 10:21:42 -05:00
Richard van der Hoff
02070c69fa
Fix bugs in handling clientRedirectUrl, and improve OIDC tests ( #9127 , #9128 )
...
* Factor out a common TestHtmlParser
Looks like I'm doing this in a few different places.
* Improve OIDC login test
Complete the OIDC login flow, rather than giving up halfway through.
* Ensure that OIDC login works with multiple OIDC providers
* Fix bugs in handling clientRedirectUrl
- don't drop duplicate query-params, or params with no value
- allow utf-8 in query-params
2021-01-18 14:52:49 +00:00
Patrick Cloke
de1f8de319
Ensure the user ID is serialized in the payload instead of used as an instance name. ( #9130 )
2021-01-18 11:08:26 +00:00
Erik Johnston
350d9923cd
Make chain cover index bg update go faster ( #9124 )
...
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
2021-01-15 17:18:37 +00:00
Richard van der Hoff
2de7e263ed
Ensure we store pusher data as text ( #9117 )
...
I don't think there's any need to use canonicaljson here.
Fixes : #4475 .
2021-01-15 16:57:23 +00:00
Richard van der Hoff
9de6b94117
Land support for multiple OIDC providers ( #9110 )
...
This is the final step for supporting multiple OIDC providers concurrently.
First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before:
oidc_config:
enabled: true
issuer: "https://oidc_provider "
# etc
After:
oidc_providers:
- idp_id: prov1
issuer: "https://oidc_provider "
- idp_id: prov2
issuer: "https://another_oidc_provider "
The old format is still grandfathered in.
With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
2021-01-15 16:55:29 +00:00
Patrick Cloke
3e4cdfe5d9
Add an admin API endpoint to protect media. ( #9086 )
...
Protecting media stops it from being quarantined when
e.g. all media in a room is quarantined. This is useful
for sticker packs and other media that is uploaded by
server administrators, but used by many people.
2021-01-15 11:18:09 -05:00
Patrick Cloke
74dd906041
Avoid raising the body exceeded error multiple times. ( #9108 )
...
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
2021-01-15 11:00:13 -05:00
Richard van der Hoff
9ffac2bef1
Remote dependency on distutils ( #9125 )
...
`distutils` is pretty much deprecated these days, and replaced with
`setuptools`. It's also annoying because it's you can't `pip install` it, and
it's hard to figure out which debian package we should depend on to make sure
it's there.
Since we only use it for a tiny function anyway, let's just vendor said
function into our codebase.
2021-01-15 15:59:20 +00:00
Patrick Cloke
d34c6e1279
Add type hints to media rest resources. ( #9093 )
2021-01-15 10:57:37 -05:00
Richard van der Hoff
4575ad0b1e
Store an IdP ID in the OIDC session ( #9109 )
...
Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP.
As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
2021-01-15 13:22:12 +00:00
Richard van der Hoff
14950a45d6
Merge pull request #9091 from matrix-org/rav/error_on_bad_sso
...
Give the user a better error when they present bad SSO creds
2021-01-15 00:27:13 +00:00
Erik Johnston
1a08e0cdab
Fix event chain bg update. ( #9118 )
...
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
2021-01-14 18:57:32 +00:00
Erik Johnston
d2479c6870
Fix perf of get_cross_signing_keys ( #9116 )
2021-01-14 17:57:09 +00:00
Erik Johnston
659c415ed4
Fix chain cover background update to work with split out event persisters ( #9115 )
2021-01-14 17:19:35 +00:00
