Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-13 17:44:38 -05:00
Code Issues Packages Projects Releases Wiki Activity
19,142 Commits 4 Branches 609 Tags 60 MiB
f5a368bb48
Commit Graph

13450 Commits

Author SHA1 Message Date
rkfg
c9a2b5d402
More robust handling of the Content-Type header for thumbnail generation (#9788) 
Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
 2021-04-14 16:30:59 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786) 
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
 2021-04-14 15:34:27 +01:00
Andrew Morgan
3efd98aa1c 1.32.0rc1 2021-04-13 14:23:43 +01:00
Dan Callahan
1d5f0e3529
Bump black configuration to target py36 (#9781) 
Signed-off-by: Dan Callahan <danc@element.io>
 2021-04-13 10:41:34 +01:00
Dirk Klimpel
1fc97ee876
Add an admin API to manage ratelimit for a specific user (#9648) 2021-04-13 10:26:37 +01:00
Will Hunt
e300ef64b1
Require AppserviceRegistrationType (#9548) 
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
 2021-04-12 15:13:55 +01:00
Richard van der Hoff
f946450184
Fix duplicate logging of exceptions in transaction processing (#9780) 
There's no point logging this twice.
 2021-04-09 18:12:15 +01:00
Dirk Klimpel
48a1f4db31
Remove old admin API GET /_synapse/admin/v1/users/<user_id> (#9401) 
Related: #8334
Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25)

`GET /_synapse/admin/v1/users/<user_id>` has no
- unit tests
- documentation

API in v2 is available (#5925 - 12/2019, v1.7.0).
API is misleading. It expects `user_id` and returns a list of all users.

Signed-off-by: Dirk Klimpel dirk@klimpel.org
 2021-04-09 09:44:40 +01:00
Jonathan de Jong
2ca4e349e9
Bugbear: Add Mutable Parameter fixes (#9682) 
Part of #9366

Adds in fixes for B006 and B008, both relating to mutable parameter lint errors.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
 2021-04-08 22:38:54 +01:00
Richard van der Hoff
9e167d9c53 Merge remote-tracking branch 'origin/develop' into rav/drop_py35 2021-04-08 18:30:38 +01:00
Richard van der Hoff
24c58ebfc9 remove unused param on make_tuple_comparison_clause 2021-04-08 18:29:57 +01:00
Erik Johnston
be0e722fe1 Merge branch 'erikj/fix_stalled_catchup' into develop 2021-04-08 18:05:35 +01:00
Erik Johnston
3a569fb200 Fix sharded federation sender sometimes using 100% CPU. 
We pull all destinations requiring catchup from the DB in batches.
However, if all those destinations get filtered out (due to the
federation sender being sharded), then the `last_processed` destination
doesn't get updated, and we keep requesting the same set repeatedly.
 2021-04-08 17:34:07 +01:00
Richard van der Hoff
3ada9b4264 Drop support for sqlite<3.22 as well 2021-04-08 16:42:32 +01:00
Richard van der Hoff
abade34633 Require py36 and Postgres 9.6 2021-04-08 16:42:32 +01:00
Patrick Cloke
452991527a
MSC3083: Check for space membership during a local join of restricted rooms. (#9735) 
When joining a room with join rules set to 'restricted', check if the
user is a member of the spaces defined in the 'allow' key of the join rules.

This only applies to an experimental room version, as defined in MSC3083.
 2021-04-08 08:28:32 -04:00
Patrick Cloke
48d44ab142
Record more information into structured logs. (#9654) 
Records additional request information into the structured logs,
e.g. the requester, IP address, etc.
 2021-04-08 08:01:14 -04:00
Andrew Morgan
0d87c6bd12
Don't report anything from GaugeBucketCollector metrics until data is present (#8926) 
This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.
 2021-04-06 16:32:04 +01:00
Andrew Morgan
04819239ba
Add a Synapse Module for configuring presence update routing (#9491) 
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though.

This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around.

A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. 

The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being:

* Sending state for a specific set or all known users to a defined set of local and remote users.
* The ability to trigger an initial sync for specific users, so they receive all current state.
 2021-04-06 14:38:30 +01:00
Patrick Cloke
44bb881096
Add type hints to expiring cache. (#9730) 2021-04-06 08:58:18 -04:00
Richard van der Hoff
0ef321ff3b
Remove outdated constraint on remote_media_cache_thumbnails (#9725) 
The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key`
constraint is superceded by
`remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds
`thumbnail_method` to the unique key).

PR #7124 made an attempt to remove the old constraint, but got the name wrong,
so it didn't work. Here we update the bg update and rerun it.

Fixes #8649.
 2021-04-06 13:36:05 +01:00
Erik Johnston
5688a74cf3 Merge branch 'master' into develop 2021-04-06 13:29:29 +01:00
Erik Johnston
1d8863c67d 1.31.0 2021-04-06 13:09:56 +01:00
Patrick Cloke
d959d28730
Add type hints to the federation handler and server. (#9743) 2021-04-06 07:21:57 -04:00
Jonathan de Jong
e2b8a90897
Update mypy configuration: no_implicit_optional = True (#9742) 2021-04-05 09:10:18 -04:00
Erik Johnston
33548f37aa
Improve tracing for to device messages (#9686) 2021-04-01 17:08:21 +01:00
Dirk Klimpel
bb0fe02a52
Add order_by to list user admin API (#9691) 2021-04-01 11:28:53 +01:00
Patrick Cloke
35c5ef2d24
Add an experimental room version to support restricted join rules. (#9717) 
Per MSC3083.
 2021-03-31 16:39:08 -04:00
Denis Kasak
5ff8eb97c6
Make sample config allowed_local_3pids regex stricter. (#9719) 
The regex should be terminated so that subdomain matches of another
domain are not accepted. Just ensuring that someone doesn't shoot
themselves in the foot by copying our example.

Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
 2021-03-31 12:27:20 +00:00
Cristina
670564446c
Deprecate imp (#9718) 
Fixes #9642.

Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
 2021-03-31 12:04:27 +01:00
Richard van der Hoff
4dabcf026e
Include m.room.create in invite_room_state for Spaces (#9710) 2021-03-30 14:03:17 +01:00
Richard van der Hoff
f02663c4dd
Replace room_invite_state_types with room_prejoin_state (#9700) 
`room_invite_state_types` was inconvenient as a configuration setting, because
anyone that ever set it would not receive any new types that were added to the
defaults. Here, we deprecate the old setting, and replace it with a couple of
new settings under `room_prejoin_state`.
 2021-03-30 12:12:44 +01:00
Erik Johnston
963f4309fe
Make RateLimiter class check for ratelimit overrides (#9711) 
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited.

We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits.

Fixes #9663
 2021-03-30 12:06:09 +01:00
Erik Johnston
78e48f61bf 1.31.0rc1 2021-03-30 11:19:21 +01:00
Patrick Cloke
01dd90b0f0
Add type hints to DictionaryCache and TTLCache. (#9442) 2021-03-29 12:15:33 -04:00
Patrick Cloke
da75d2ea1f
Add type hints for the federation sender. (#9681) 
Includes an abstract base class which both the FederationSender
and the FederationRemoteSendQueue must implement.
 2021-03-29 11:43:20 -04:00
Richard van der Hoff
4bbd535450
Update the OIDC sample config (#9695) 
I've reiterated the advice about using `oidc` to migrate, since I've seen a few
people caught by this.

I've also removed a couple of the examples as they are duplicating the OIDC
documentation, and I think they might be leading people astray.
 2021-03-29 15:40:11 +01:00
Jonathan de Jong
fc53a606e4
Fix re.Pattern mypy error on 3.6 (#9703) 2021-03-29 09:40:45 -04:00
Richard van der Hoff
ad8690a26c
Fix the suggested pip incantation for cryptography (#9699) 
If you have the wrong version of `cryptography` installed, synapse suggests:

```
To install run:
    pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6''
```

However, the use of ' inside '...' doesn't work, so when you run this, you get
an error.
 2021-03-29 11:55:33 +01:00
Richard van der Hoff
7c8402ddb8
Suppress CryptographyDeprecationWarning (#9698) 
This warning is somewhat confusing to users, so let's suppress it
 2021-03-26 17:33:55 +00:00
Erik Johnston
b5efcb577e
Make it possible to use dmypy (#9692) 
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
 2021-03-26 16:49:46 +00:00
Erik Johnston
019010964d Merge branch 'master' into develop 2021-03-26 12:26:58 +00:00
Erik Johnston
c6f8e8086c 1.30.1 2021-03-26 12:03:29 +00:00
Erik Johnston
12d6184713
Explicitly upgrade openssl in docker file and enforce new version of cryptography (#9697) 2021-03-26 12:00:25 +00:00
Andrew Morgan
fae81f2f68
Add a storage method for returning all current presence from all users (#9650) 
Split off from https://github.com/matrix-org/synapse/pull/9491

Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: b588f16e39/synapse/handlers/presence.py (L1131-L1133)

Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
 2021-03-25 10:34:23 +00:00
Erik Johnston
c602ba8336
Fixed undefined variable error in catchup (#9664) 
Broke in #9640

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-03-24 16:12:47 +00:00
Jonathan de Jong
4c3827f2c1
Enable addtional flake8-bugbear linting checks. (#9659) 2021-03-24 09:34:30 -04:00
Richard van der Hoff
c73cc2c2ad
Spaces summary: call out to other servers (#9653) 
When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data.

Fixes: #9447
 2021-03-24 12:45:39 +00:00
Patrick Cloke
af387cf52a
Add type hints to misc. files. (#9676) 2021-03-24 06:49:01 -04:00
Patrick Cloke
7e8dc9934e
Add a type hints for service notices to the HomeServer object. (#9675) 2021-03-24 06:48:46 -04:00
Erik Johnston
e550ab17ad
Increase default join burst ratelimiting (#9674) 
It's legitimate behaviour to try and join a bunch of rooms at once.
 2021-03-23 14:52:20 +00:00
Jonathan de Jong
0caf2a338e
Fix federation stall on concurrent access errors (#9639) 2021-03-23 13:52:30 +00:00
Richard van der Hoff
4ecba9bd5c
Federation API for Space summary (#9652) 
Builds on the work done in #9643 to add a federation API for space summaries.

There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
 2021-03-23 11:51:12 +00:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. (#9665) 2021-03-23 07:12:48 -04:00
Andrew Morgan
5b268997bd
Allow providing credentials to HTTPS_PROXY (#9657) 
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70

This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server.

Also added some type hints.
 2021-03-22 17:20:47 +00:00
Ankit Dobhal
d66f9070cd
Fixed code misc. quality issues (#9649) 
- Merge 'isinstance' calls.
- Remove unnecessary dict call outside of comprehension.
- Use 'sys.exit()' calls.
 2021-03-22 11:18:13 -04:00
Erik Johnston
d600d4506b Merge branch 'master' into develop 2021-03-22 13:36:36 +00:00
Erik Johnston
e2904f720d 1.30.0 2021-03-22 13:15:55 +00:00
Brendan Abolivier
b6ed4f55ac
Incorporate review 2021-03-19 18:19:50 +01:00
Brendan Abolivier
592d6305fd
Merge branch 'develop' into babolivier/msc3026 2021-03-19 16:12:40 +01:00
Brendan Abolivier
0b56481caa
Fix lint 2021-03-19 16:11:08 +01:00
Richard van der Hoff
066068f034 fix mypy 2021-03-19 12:20:11 +00:00
Patrick Cloke
9b0e3009fa Fix type-hints from bad merge. 2021-03-18 14:40:56 -04:00
Richard van der Hoff
004234f03a
Initial spaces summary API (#9643) 
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
 2021-03-18 18:24:16 +00:00
Brendan Abolivier
066c703729
Move support for MSC3026 behind an experimental flag 2021-03-18 18:37:19 +01:00
Dirk Klimpel
8dd2ea65a9
Consistently check whether a password may be set for a user. (#9636) 2021-03-18 12:54:08 -04:00
Erik Johnston
dd71eb0f8a
Make federation catchup send last event from any server. (#9640) 
Currently federation catchup will send the last *local* event that we
failed to send to the remote. This can cause issues for large rooms
where lots of servers have sent events while the remote server was down,
as when it comes back up again it'll be flooded with events from various
points in the DAG.

Instead, let's make it so that all the servers send the most recent
events, even if its not theirs. The remote should deduplicate the
events, so there shouldn't be much overhead in doing this.
Alternatively, the servers could only send local events if they were
also extremities and hope that the other server will send the event
over, but that is a bit risky.
 2021-03-18 15:52:26 +00:00
Brendan Abolivier
405aeb0b2c
Implement MSC3026: busy presence state 2021-03-18 16:34:47 +01:00
Andrew Morgan
7b06f85c0e
Ensure we use a copy of the event content dict before modifying it in serialize_event (#9585) 
This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](17cd48fe51/synapse/events/third_party_rules.py (L73-L76)) - will raise a 500 if you try to a edit a reply to a message.

`serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised.

To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
 2021-03-17 16:51:55 +00:00
Patrick Cloke
cc324d53fe
Fix up types for the typing handler. (#9638) 
By splitting this to two separate methods the callers know
what methods they can expect on the handler.
 2021-03-17 11:30:21 -04:00
Hubert Chathi
73dbce5523
only save remote cross-signing keys if they're different from the current ones (#9634) 
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-03-17 11:04:57 -04:00
Erik Johnston
ad721fc559
Fix bad naming of storage function (#9637) 
We had two functions named `get_forward_extremities_for_room` and
`get_forward_extremeties_for_room` that took different paramters. We
rename one of them to avoid confusion.
 2021-03-17 13:20:08 +00:00
Richard van der Hoff
567f88f835
Prep work for removing outlier from internal_metadata (#9411) 
* Populate `internal_metadata.outlier` based on `events` table

Rather than relying on `outlier` being in the `internal_metadata` column,
populate it based on the `events.outlier` column.

* Move `outlier` out of InternalMetadata._dict

Ultimately, this will allow us to stop writing it to the database. For now, we
have to grandfather it back in so as to maintain compatibility with older
versions of Synapse.
 2021-03-17 12:33:18 +00:00
Patrick Cloke
b449af0379
Add type hints to the room member handler. (#9631) 2021-03-17 07:14:39 -04:00
Jonathan de Jong
27d2820c33
Enable flake8-bugbear, but disable most checks. (#9499) 
* Adds B00 to ignored checks.
* Fixes remaining issues.
 2021-03-16 14:19:27 -04:00
Hubbe
dd5e5dc1d6
Add SSO attribute requirements for OIDC providers (#9609) 
Allows limiting who can login using OIDC via the claims
made from the IdP.
 2021-03-16 11:46:07 -04:00
Dirk Klimpel
8000cf1315
Return m.change_password.enabled=false if local database is disabled (#9588) 
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
 2021-03-16 11:44:25 -04:00
Andrew Morgan
d315e96443 1.30.0rc1 2021-03-16 13:45:46 +00:00
Andrew Morgan
847ecdd8fa
Pass SSO IdP information to spam checker's registration function (#9626) 
Fixes https://github.com/matrix-org/synapse/issues/9572

When a SSO user logs in for the first time, we create a local Matrix user for them. This goes through the register_user flow, which ends up triggering the spam checker. Spam checker modules don't currently have any way to differentiate between a user trying to sign up initially, versus an SSO user (whom has presumably already been approved elsewhere) trying to log in for the first time.

This PR passes `auth_provider_id` as an argument to the `check_registration_for_spam` function. This argument will contain an ID of an SSO provider (`"saml"`, `"cas"`, etc.) if one was used, else `None`.
 2021-03-16 12:41:41 +00:00
Patrick Cloke
1383508f29
Handle an empty cookie as an invalid macaroon. (#9620) 
* Handle an empty cookie as an invalid macaroon.

* Newsfragment
 2021-03-16 11:29:35 +00:00
Richard van der Hoff
dd69110d95
Add support for stable MSC2858 API (#9617) 
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
 2021-03-16 11:21:26 +00:00
Richard van der Hoff
5b5bc188cf
Clean up config settings for stats (#9604) 
... and complain if people try to turn it off.
 2021-03-16 10:57:54 +00:00
Andrew Morgan
1b0eaed21f
Prevent bundling aggregations for state events (#9619) 
There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
 2021-03-16 10:27:51 +00:00
Richard van der Hoff
1c8a2541da
Fix Internal Server Error on GET /saml2/authn_response (#9623) 
* Fix Internal Server Error on `GET /saml2/authn_response`

Seems to have been introduced in #8765 (Synapse 1.24.0)

* Fix newsfile
 2021-03-16 10:20:20 +00:00
Patrick Cloke
d29b71aa50
Fix remaining mypy issues due to Twisted upgrade. (#9608) 2021-03-15 11:14:39 -04:00
Erik Johnston
026503fa3b
Don't go into federation catch up mode so easily (#9561) 
Federation catch up mode is very inefficient if the number of events
that the remote server has missed is small, since handling gaps can be
very expensive, c.f. #9492.

Instead of going into catch up mode whenever we see an error, we instead
do so only if we've backed off from trying the remote for more than an
hour (the assumption being that in such a case it is more than a
transient failure).
 2021-03-15 14:42:40 +00:00
Richard van der Hoff
af2248f8bf
Optimise missing prev_event handling (#9601) 
Background: When we receive incoming federation traffic, and notice that we are missing prev_events from 
the incoming traffic, first we do a `/get_missing_events` request, and then if we still have missing prev_events,
we set up new backwards-extremities. To do that, we need to make a `/state_ids` request to ask the remote
server for the state at those prev_events, and then we may need to then ask the remote server for any events
in that state which we don't already have, as well as the auth events for those missing state events, so that we
can auth them.

This PR attempts to optimise the processing of that state request. The `state_ids` API returns a list of the state
events, as well as a list of all the auth events for *all* of those state events. The optimisation comes from the
observation that we are currently loading all of those auth events into memory at the start of the operation, but
we almost certainly aren't going to need *all* of the auth events. Rather, we can check that we have them, and
leave the actual load into memory for later. (Ideally the federation API would tell us which auth events we're
actually going to need, but it doesn't.)

The effect of this is to reduce the number of events that I need to load for an event in Matrix HQ from about
60000 to about 22000, which means it can stay in my in-memory cache, whereas previously the sheer number
of events meant that all 60K events had to be loaded from db for each request, due to the amount of cache
churn. (NB I've already tripled the size of the cache from its default of 10K).

Unfortunately I've ended up basically C&Ping `_get_state_for_room` and `_get_events_from_store_or_dest` into
a new method, because `_get_state_for_room` is also called during backfill, which expects the auth events to be
returned, so the same tricks don't work. That said, I don't really know why that codepath is completely different
(ultimately we're doing the same thing in setting up a new backwards extremity) so I've left a TODO suggesting
that we clean it up.
 2021-03-15 13:51:02 +00:00
Patrick Cloke
55da8df078
Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
Richard van der Hoff
1e67bff833
Reject concurrent transactions (#9597) 
If more transactions arrive from an origin while we're still processing the
first one, reject them.

Hopefully a quick fix to https://github.com/matrix-org/synapse/issues/9489
 2021-03-12 15:14:55 +00:00
Richard van der Hoff
2b328d7e02
Improve logging when processing incoming transactions (#9596) 
Put the room id in the logcontext, to make it easier to understand what's going on.
 2021-03-12 15:08:03 +00:00
Richard van der Hoff
464e5da7b2
Add logging for redis connection setup (#9590) 2021-03-11 18:35:09 +00:00
Patrick Cloke
e55bd0e110
Add tests for blacklisting reactor/agent. (#9563) 2021-03-11 09:15:22 -05:00
Dirk Klimpel
70d1b6abff
Re-Activating account when local passwords are disabled (#9587) 
Fixes: #8393
 2021-03-11 13:52:32 +00:00
Richard van der Hoff
a7a3790066
Convert Requester to attrs (#9586) 
... because namedtuples suck

Fix up a couple of other annotations to keep mypy happy.
 2021-03-10 18:15:56 +00:00
Richard van der Hoff
1107214a1d
Fix the auth provider on the logins metric (#9573) 
We either need to pass the auth provider over the replication api, or make sure
we report the auth provider on the worker that received the request. I've gone
with the latter.
 2021-03-10 18:15:03 +00:00
Patrick Cloke
2a99cc6524
Use the chain cover index in get_auth_chain_ids. (#9576) 
This uses a simplified version of get_chain_cover_difference to calculate
auth chain of events.
 2021-03-10 09:57:59 -05:00
Patrick Cloke
918f6ed827
Fix a bug in the background task for purging chain cover. (#9583) 2021-03-10 08:55:52 -05:00
Patrick Cloke
dc51d8ffaf
Add a background task to purge unused chain IDs. (#9542) 
This is a companion change to apply the fix in #9498 /
922788c604 to previously
purged rooms.
 2021-03-09 11:22:25 -05:00
Richard van der Hoff
eaada74075
JWT OIDC secrets for Sign in with Apple (#9549) 
Apple had to be special. They want a client secret which is generated from an EC key.

Fixes #9220. Also fixes #9212 while I'm here.
 2021-03-09 15:03:37 +00:00
Erik Johnston
9cd18cc588
Retry 5xx errors in federation client (#9567) 
Fixes #8915
 2021-03-09 13:15:12 +00:00
Patrick Cloke
7fdc6cefb3
Fix additional type hints. (#9543) 
Type hint fixes due to Twisted 21.2.0 adding type hints.
 2021-03-09 07:41:32 -05:00
Patrick Cloke
075c16b410
Handle image transparency better when thumbnailing. (#9473) 
Properly uses RGBA mode for 1- and 8-bit images with transparency
(instead of RBG mode).
 2021-03-09 07:37:09 -05:00
Jonathan de Jong
9898470e7d
Add logging to ObservableDeferred callbacks (#9523) 2021-03-09 11:09:31 +00:00
Matthew Hodgson
0764d0c6e5 quick config comment tweak to clarify allow_profile_lookup_over_federation 2021-03-08 21:52:04 +00:00
Jonathan de Jong
d6196efafc
Add ResponseCache tests. (#9458) 2021-03-08 14:00:07 -05:00
Dirk Klimpel
7076eee4b9
Add type hints to purge room and server notice admin API. (#9520) 2021-03-08 10:34:38 -05:00
Erik Johnston
b988b07bb0 Merge branch 'master' into develop 2021-03-08 14:06:35 +00:00
Erik Johnston
15c788e22d 1.29.0 2021-03-08 13:52:13 +00:00
Patrick Cloke
58114f8a17
Create a SynapseReactor type which incorporates the necessary reactor interfaces. (#9528) 
This helps fix some type hints when running with Twisted 21.2.0.
 2021-03-08 08:25:43 -05:00
Richard van der Hoff
8a4b3738f3
Replace last_*_pdu_age metrics with timestamps (#9540) 
Following the advice at
https://prometheus.io/docs/practices/instrumentation/#timestamps-not-time-since,
it's preferable to export unix timestamps, not ages.

There doesn't seem to be any particular naming convention for timestamp
metrics.
 2021-03-04 16:40:18 +00:00
Richard van der Hoff
df425c2c63
Prometheus metrics for logins and registrations (#9511) 
Add prom metrics for number of users successfully registering and logging in, by SSO provider.
 2021-03-04 16:39:27 +00:00
Richard van der Hoff
7eb6e39a8f
Record the SSO Auth Provider in the login token (#9510) 
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
 2021-03-04 14:44:22 +00:00
Erik Johnston
3f49d80dcf 1.29.0rc1 2021-03-04 10:12:53 +00:00
Patrick Cloke
33a02f0f52
Fix additional type hints from Twisted upgrade. (#9518) 2021-03-03 15:47:38 -05:00
Richard van der Hoff
4db07f9aef
Set X-Forwarded-Proto header when frontend-proxy proxies a request (#9539) 
Should fix some remaining warnings
 2021-03-03 18:49:08 +00:00
Erik Johnston
a4fa044c00
Fix 'rejected_events_metadata' background update (#9537) 
Turns out matrix.org has an event that has duplicate auth events (which really isn't supposed to happen, but here we are). This caused the background update to fail due to `UniqueViolation`.
 2021-03-03 16:04:24 +00:00
Patrick Cloke
922788c604
Purge chain cover tables when purging events. (#9498) 2021-03-03 11:04:08 -05:00
Dirk Klimpel
d790d0d314
Add type hints to user admin API. (#9521) 2021-03-03 08:09:39 -05:00
Patrick Cloke
0c330423bc
Bump the mypy and mypy-zope versions. (#9529) 2021-03-03 07:19:19 -05:00
Erik Johnston
16f9f93eb7
Make deleting stale pushers a background update (#9536) 2021-03-03 12:08:16 +00:00
Aaron Raimist
0279e0e086
Prevent presence background jobs from running when presence is disabled (#9530) 
Prevent presence background jobs from running when presence is disabled

Signed-off-by: Aaron Raimist <aaron@raim.ist>
 2021-03-03 10:21:46 +00:00
Patrick Cloke
aee10768d8 Revert "Fix #8518 (sync requests being cached wrongly on timeout) (#9358)" 
This reverts commit f5c93fc993.

This is being backed out due to a regression (#9507) and additional
review feedback being provided.
 2021-03-02 09:43:34 -05:00
Erik Johnston
7f5d753d06
Re-run rejected metadata background update. (#9503) 
It landed in schema version 58 after 59 had been created, causing some
servers to not run it. The main effect of was that not all rooms had
their chain cover calculated correctly. After the BG updates complete
the chain covers will get fixed when a new state event in the affected
rooms is received.
 2021-03-02 14:31:23 +00:00
Erik Johnston
16108c579d
Fix SQL delta file taking a long time to run (#9516) 
Fixes #9504
 2021-03-02 14:05:01 +00:00
Dirk Klimpel
f00c4e7af0
Add type hints to device and event report admin API (#9519) 2021-03-02 09:31:12 +00:00
Patrick Cloke
ad8589d392
Fix a bug when a room alias is given to the admin join endpoint (#9506) 2021-03-01 13:59:01 -05:00
Patrick Cloke
16ec8c3272
(Hopefully) stop leaking file descriptors in media repo. (#9497) 
By consuming the response if the headers imply that the
content is too large.
 2021-03-01 12:45:00 -05:00
Patrick Cloke
a0bc9d387e
Use the proper Request in type hints. (#9515) 
This also pins the Twisted version in the mypy job for CI until
proper type hints are fixed throughout Synapse.
 2021-03-01 12:23:46 -05:00
Jonathan de Jong
e12077a78a
Allow bytecode again (#9502) 
In #75, bytecode was disabled (from a bit of FUD back in `python<2.4` days, according to dev chat), I think it's safe enough to enable it again.

Added in `__pycache__/` and `.pyc`/`.pyd` to `.gitignore`, to extra-insure compiled files don't get committed.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
 2021-02-26 18:30:54 +00:00
Tim Leung
ddb240293a
Add support for no_proxy and case insensitive env variables (#9372) 
### Changes proposed in this PR

- Add support for the `no_proxy` and `NO_PROXY` environment variables
  - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519))
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment

This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`

Related to #9306 which also uses `ProxyAgent`

Signed-off-by: Timothy Leung tim95@hotmail.co.uk
 2021-02-26 17:37:57 +00:00
Richard van der Hoff
15090de850
SSO: redirect to public URL before setting cookies (#9436) 
... otherwise, we don't get the cookie back.
 2021-02-26 14:02:06 +00:00
Erik Johnston
2566dc57ce
Test that we require validated email for email pushers (#9496) 2021-02-25 15:35:14 +00:00
Erik Johnston
1e62d9ee8c
Ensure pushers are deleted for deactivated accounts (#9285) 2021-02-25 13:56:55 +00:00
Erik Johnston
1efdcc3e87 Merge branch 'master' into develop 2021-02-25 10:53:31 +00:00
Erik Johnston
b5c4fe1971 1.28.0 2021-02-25 10:22:07 +00:00
Richard van der Hoff
d8e95e5452
Add support for X-Forwarded-Proto (#9472) 
rewrite XForwardedForRequest to set `isSecure()` based on
`X-Forwarded-Proto`. Also implement `getClientAddress()` while we're here.
 2021-02-24 18:11:33 +00:00
Erik Johnston
7cc571510b
Add SQL delta for deleting stale pushers (#9479) 2021-02-24 17:21:10 +00:00
Jonathan de Jong
f5c93fc993
Fix #8518 (sync requests being cached wrongly on timeout) (#9358) 
This fixes #8518 by adding a conditional check on `SyncResult` in a function when `prev_stream_token == current_stream_token`, as a sanity check. In `CachedResponse.set.<remove>()`, the result is immediately popped from the cache if the conditional function returns "false".

This prevents the caching of a timed-out `SyncResult` (that has `next_key` as the stream key that produced that `SyncResult`). The cache is prevented from returning a `SyncResult` that makes the client request the same stream key over and over again, effectively making it stuck in a loop of requesting and getting a response immediately for as long as the cache keeps those values.

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
 2021-02-24 13:57:00 +00:00
Erik Johnston
2927921942
Clean up ShardedWorkerHandlingConfig (#9466) 
* Split ShardedWorkerHandlingConfig

This is so that we have a type level understanding of when it is safe to
call `get_instance(..)` (as opposed to `should_handle(..)`).

* Remove special cases in ShardedWorkerHandlingConfig.

`ShardedWorkerHandlingConfig` tried to handle the various different ways
it was possible to configure federation senders and pushers. This led to
special cases that weren't hit during testing.

To fix this the handling of the different cases is moved from there and
`generic_worker` into the worker config class. This allows us to have
the logic in one place and allows the rest of the code to ignore the
different cases.
 2021-02-24 13:23:18 +00:00
Erik Johnston
0b5c967813
Refactor to ensure we call check_consistency (#9470) 
The idea here is to stop people forgetting to call `check_consistency`. Folks can still just pass in `None` to the new args in `build_sequence_generator`, but hopefully they won't.
 2021-02-24 10:13:53 +00:00
Patrick Cloke
7292b7c0eb
Add back the deprecated SAML endpoint. (#9474) 2021-02-23 12:57:37 -05:00
Richard van der Hoff
713145d3de
Add a comment about systemd-python. (#9464) 
This confused me for a while.
 2021-02-23 13:42:36 +00:00
Patrick Cloke
65a9eb8994
Include newly added sequences in the port DB script. (#9449) 
And ensure the consistency of `event_auth_chain_id`.
 2021-02-23 07:33:24 -05:00
Erik Johnston
66f4949e7f
Fix deleting pushers when using sharded pushers. (#9465) 2021-02-22 21:14:42 +00:00
Richard van der Hoff
1b2d6d55c5
Remove vestiges of uploads_path config (#9462) 
`uploads_path` was a thing that was never used; most of it was removed in #6628
but a few vestiges remained.
 2021-02-22 19:54:49 +00:00
Dirk Klimpel
71c9f8de6d
Add an order_by field to list users' media admin API. (#8978) 2021-02-22 14:38:51 -05:00
Andrew Morgan
0a363f9ca4
Remove cache for get_shared_rooms_for_users (#9416) 
This PR remove the cache for the `get_shared_rooms_for_users` storage method (the db method driving the experimental "what rooms do I share with this user?" feature: [MSC2666](https://github.com/matrix-org/matrix-doc/pull/2666)). Currently subsequent requests to the endpoint will return the same result, even if your shared rooms with that user have changed.

The cache was added in https://github.com/matrix-org/synapse/pull/7785, but we forgot to ensure it was invalidated appropriately.

Upon attempting to invalidate it, I found that the cache had to be entirely invalidated whenever a user (remote or local) joined or left a room. This didn't make for a very useful cache, especially for a function that may or may not be called very often. Thus, I've opted to remove it instead of invalidating it.
 2021-02-22 16:52:45 +00:00
Andrew Morgan
e22b71810e
Clean up the user directory sample config section (#9385) 
The user directory sample config section was a little messy, and didn't adhere to our [recommended config format guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).

This PR cleans that up a bit.
 2021-02-22 11:44:31 +00:00
Patrick Cloke
fc8b3d8809
Ratelimit cross-user key sharing requests. (#8957) 2021-02-19 13:20:34 -05:00
Erik Johnston
179c0953ff
Regenerate exact thumbnails if missing (#9438) 2021-02-19 17:09:57 +00:00
Patrick Cloke
a1901abd6b
Add documentation and type hints to parse_duration. (#9432) 2021-02-19 08:32:21 -05:00
Patrick Cloke
c4a55ac4a4 Fix style checking due to updated black. 2021-02-19 08:19:54 -05:00
Patrick Cloke
d9f1dccba9 Synapse 1.28.0rc1 (2021-02-19) 
==============================
 
 Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).
 
 This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.
 
 Removal warning
 ---------------
 
 The v1 list accounts API is deprecated and will be removed in a future release.
 This API was undocumented and misleading. It can be replaced by the
 [v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
 which has been available since Synapse 1.7.0 (2019-12-13).
 
 Please check if you're using any scripts which use the admin API and replace
 `GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.
 
 Features
 --------
 
 - New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150))
 - Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301))
 - Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311))
 - Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376))
 - Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400))
 
 Bugfixes
 --------
 
 - Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257))
 - Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296))
 - Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393))
 - Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333))
 - Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361))
 - Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391))
 - Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395))
 - Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421))
 - Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425))
 - The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426))
 - Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428))
 
 Updates to the Docker image
 ---------------------------
 
 - Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433))
 
 Improved Documentation
 ----------------------
 
 - Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281))
 - Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291))
 - Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308))
 - Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317))
 - Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322))
 - Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390))
 - Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404))
 - Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407))
 - Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434))
 
 Deprecations and Removals
 -------------------------
 
 - Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429))
 
 Internal Changes
 ----------------
 
 - Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003))
 - Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123))
 - Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240))
 - Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299))
 - Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305))
 - Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307))
 - Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326))
 - Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362))
 - Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396))
 - Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381))
 - Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384))
 - Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394))
 - Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAvtsUACgkQM/xY9qcR
 MEgEmA/+O3fCMKdyh4soBirnEZa88m1HL4Jyd+jYl7+IuFChZz0k+RYLdQ376GaC
 tmT4aGQISdu1wYUwO6zpN5ercvFCJ5OaSJzHinEaHZMnEfWe9SyMNB1oD9h6UYSO
 8r7OT2U28pTTAbj/VpSNK8osbx/F9T5o1a7v3PLkB1kJcK9j9oqNrqpdsSW2Cz1B
 rGlBKRSICKR4bzfk3dR9Wr4Fcyb+gINn0LTtYYqMWxP1EtRC+VI3NEWOk3OD2dzk
 UztvUn0F7rKFmtcBRTXsxuGiGI9IGrd9tyKc7ilBz24eSA95psperUtHNKGwSUaO
 kt+bdMfEKM2cMnydMuiCYtdl/M59Rf3ShRE3OZd7TK4S4NoPy6whFsAJf7mBGxfp
 RcMwO/7mWRX5qjDRG6x5nM+aCqO0Y8h8WNF4gZtiN2XGyxlU0TBQMBwBM66qAoXO
 zCPt4tMlLsOBCABNkWT3W5Ww5KxNeMp2qEA9UnmjezA5XHtSx/DVN+6D5E4Rxw2v
 PzS278Sq3YO/JK6TfXsFBkZvDFPmCDdTQfKpoDbT6Hv16CXTz66XdiEg5BPZYIvp
 bNurfxr22/RiScxiA73cXbRBHq2qAPTxZo1O40R2LnhX0wQPQ5wfUW8oxZOnuhZ3
 bWw4PvhWsZDt2SB9EIKf4RQtSxNlar88C6tpHSC89ScmE6PPWnI=
 =Srwm
 -----END PGP SIGNATURE-----

Merge tag 'v1.28.0rc1' into develop

Synapse 1.28.0rc1 (2021-02-19)
==============================

Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails).

This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes.

Removal warning
---------------

The v1 list accounts API is deprecated and will be removed in a future release.
This API was undocumented and misleading. It can be replaced by the
[v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts),
which has been available since Synapse 1.7.0 (2019-12-13).

Please check if you're using any scripts which use the admin API and replace
`GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`.

Features
--------

- New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](https://github.com/matrix-org/synapse/issues/9150))
- Further improvements to the user experience of registration via single sign-on. ([\#9300](https://github.com/matrix-org/synapse/issues/9300), [\#9301](https://github.com/matrix-org/synapse/issues/9301))
- Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](https://github.com/matrix-org/synapse/issues/9311))
- Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](https://github.com/matrix-org/synapse/issues/9376))
- Add the shadow-banning status to the admin API for user info. ([\#9400](https://github.com/matrix-org/synapse/issues/9400))

Bugfixes
--------

- Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](https://github.com/matrix-org/synapse/issues/9257))
- Fix bug in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](https://github.com/matrix-org/synapse/issues/9296))
- Assert a maximum length for some parameters for spec compliance. ([\#9321](https://github.com/matrix-org/synapse/issues/9321), [\#9393](https://github.com/matrix-org/synapse/issues/9393))
- Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](https://github.com/matrix-org/synapse/issues/9333))
- Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](https://github.com/matrix-org/synapse/issues/9361))
- Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](https://github.com/matrix-org/synapse/issues/9391))
- Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](https://github.com/matrix-org/synapse/issues/9395))
- Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](https://github.com/matrix-org/synapse/issues/9421))
- Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](https://github.com/matrix-org/synapse/issues/9425))
- The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](https://github.com/matrix-org/synapse/issues/9426))
- Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](https://github.com/matrix-org/synapse/issues/9428))

Updates to the Docker image
---------------------------

- Drop support for ARMv7 in Docker images. ([\#9433](https://github.com/matrix-org/synapse/issues/9433))

Improved Documentation
----------------------

- Reorganize CHANGELOG.md. ([\#9281](https://github.com/matrix-org/synapse/issues/9281))
- Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](https://github.com/matrix-org/synapse/issues/9291))
- Correct name of Synapse's service file in TURN howto. ([\#9308](https://github.com/matrix-org/synapse/issues/9308))
- Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](https://github.com/matrix-org/synapse/issues/9317))
- Update installation instructions on Fedora. ([\#9322](https://github.com/matrix-org/synapse/issues/9322))
- Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](https://github.com/matrix-org/synapse/issues/9390))
- Update docs for using Gitea as OpenID provider. ([\#9404](https://github.com/matrix-org/synapse/issues/9404))
- Document that pusher instances are shardable. ([\#9407](https://github.com/matrix-org/synapse/issues/9407))
- Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](https://github.com/matrix-org/synapse/issues/9434))

Deprecations and Removals
-------------------------

- Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](https://github.com/matrix-org/synapse/issues/9429))

Internal Changes
----------------

- Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](https://github.com/matrix-org/synapse/issues/9003))
- Add experimental support for running Synapse with PyPy. ([\#9123](https://github.com/matrix-org/synapse/issues/9123))
- Deny access to additional IP addresses by default. ([\#9240](https://github.com/matrix-org/synapse/issues/9240))
- Update the `Cursor` type hints to better match PEP 249. ([\#9299](https://github.com/matrix-org/synapse/issues/9299))
- Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](https://github.com/matrix-org/synapse/issues/9305))
- Improve logging for OIDC login flow. ([\#9307](https://github.com/matrix-org/synapse/issues/9307))
- Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](https://github.com/matrix-org/synapse/issues/9326))
- Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](https://github.com/matrix-org/synapse/issues/9362))
- Convert tests to use `HomeserverTestCase`. ([\#9377](https://github.com/matrix-org/synapse/issues/9377), [\#9396](https://github.com/matrix-org/synapse/issues/9396))
- Update the version of black used to 20.8b1. ([\#9381](https://github.com/matrix-org/synapse/issues/9381))
- Allow OIDC config to override discovered values. ([\#9384](https://github.com/matrix-org/synapse/issues/9384))
- Remove some dead code from the acceptance of room invites path. ([\#9394](https://github.com/matrix-org/synapse/issues/9394))
- Clean up an unused method in the presence handler code. ([\#9408](https://github.com/matrix-org/synapse/issues/9408))
 2021-02-19 08:10:19 -05:00
Patrick Cloke
b114a45f5f
Support not providing an IdP icon when choosing a username. (#9440) 2021-02-19 07:48:46 -05:00
Andrew Morgan
8bcfc2eaad
Be smarter about which hosts to send presence to when processing room joins (#9402) 
This PR attempts to eliminate unnecessary presence sending work when your local server joins a room, or when a remote server joins a room your server is participating in by processing state deltas in chunks rather than individually.

---

When your server joins a room for the first time, it requests the historical state as well. This chunk of new state is passed to the presence handler which, after filtering that state down to only membership joins, will send presence updates to homeservers for each join processed.

It turns out that we were being a bit naive and processing each event individually, and sending out presence updates for every one of those joins. Even if many different joins were users on the same server (hello IRC bridges), we'd send presence to that same homeserver for every remote user join we saw.

This PR attempts to deduplicate all of that by processing the entire batch of state deltas at once, instead of only doing each join individually. We process the joins and note down which servers need which presence:

* If it was a local user join, send that user's latest presence to all servers in the room
* If it was a remote user join, send the presence for all local users in the room to that homeserver

We deduplicate by inserting all of those pending updates into a dictionary of the form:

```
{
  server_name1: {presence_update1, ...},
  server_name2: {presence_update1, presence_update2, ...}
}
```

Only after building this dict do we then start sending out presence updates.
 2021-02-19 11:37:29 +00:00
Andrew Morgan
13e9029f44
Add a config option to prioritise local users in user directory search results (#9383) 
This PR adds a homeserver config option, `user_directory.prefer_local_users`, that when enabled will show local users higher in user directory search results than remote users. This option is off by default.

Note that turning this on doesn't necessarily mean that remote users will always be put below local users, but they should be assuming all other ranking factors (search query match, profile information present etc) are identical.

This is useful for, say, University networks that are openly federating, but want to prioritise local students and staff in the user directory over other random users.
 2021-02-19 11:02:03 +00:00
Erik Johnston
3d2acc930f Return a 404 if we don't have the original file 2021-02-19 10:46:18 +00:00
AndrewFerr
9bc74743d5
Add configs to make profile data more private (#9203) 
Add off-by-default configuration settings to:
- disable putting an invitee's profile info in invite events
- disable profile lookup via federation

Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
 2021-02-19 09:50:41 +00:00
Patrick Cloke
1381cd05b0 1.28.0rc1 2021-02-18 12:32:49 -05:00
Erik Johnston
b106080fb4 Regenerate exact thumbnails if missing 2021-02-18 17:05:32 +00:00
Patrick Cloke
9ee3b9775f
Remove deprecated SAML2 callback URL since it does not work. (#9434) 
Updates documentation from #9289 and removes a deprecated
endpoint which didn't work as expected.
 2021-02-18 11:20:33 -05:00
Rishabh Arya
e17553e185
Parse ui_auth.session_timeout as a duration (instead of treating it as ms) (#9426) 2021-02-18 09:18:14 -05:00
Patrick Cloke
8ec2217103
Reduce the memory usage of previewing media files. (#9421) 
This reduces the memory usage of previewing media files which
end up larger than the `max_spider_size` by avoiding buffering
content internally in treq.

It also checks the `Content-Length` header in additional places
instead of streaming the content to check the body length.
 2021-02-18 09:01:29 -05:00
Patrick Cloke
43f1c82457
Add back the guard against the user directory stream position not existing. (#9428) 
As the comment says, this guard was there for when the
initial user directory update has yet to happen.
 2021-02-18 08:44:19 -05:00
Dirk Klimpel
c8d9383cfb
Add the shadow-banning status to the display user admin API. (#9400) 2021-02-17 15:19:23 -05:00
Andrew Morgan
a25661b2eb
Remove dead notify_for_states presence method (#9408) 2021-02-17 17:32:26 +00:00
Andrew Morgan
3e5749b99f
Fix only handling the last presence state for each user (#9425) 
This is a small bug that I noticed while working on #8956.

We have a for-loop which attempts to strip all presence changes for each user except for the final one, as we don't really care about older presence:

9e19c6aab4/synapse/handlers/presence.py (L368-L371)

`new_states_dict` stores this stripped copy of latest presence state for each user, before it is... put into a new variable `new_state`, which is just overridden by the subsequent for loop.

I believe this was instead meant to override `new_states`. Without doing so, it effectively meant:

1. The for loop had no effect.
2. We were still processing old presence state for users.
 2021-02-17 17:31:37 +00:00
Patrick Cloke
d2f0ec12d5
Add type hints to groups code. (#9393) 2021-02-17 08:41:47 -05:00
Richard van der Hoff
e1071fd625
Support for form_post in OIDC responses (#9376) 
Apple want to POST the OIDC auth response back to us rather than using query-params; add the necessary support to make that work.
 2021-02-17 10:15:14 +00:00
Richard van der Hoff
33f64ca7d6
Allow OIDC config to override discovered values (#9384) 
Fixes #9347
 2021-02-16 22:33:09 +00:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381) 
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
 2021-02-16 22:32:34 +00:00
Richard van der Hoff
3b754aea27
Clean up caching/locking of OIDC metadata load (#9362) 
Ensure that we lock correctly to prevent multiple concurrent metadata load
requests, and generally clean up the way we construct the metadata cache.
 2021-02-16 16:27:38 +00:00
Erik Johnston
0ad087273c Merge branch 'master' into develop 2021-02-16 13:39:30 +00:00
Patrick Cloke
731e08c63a
Handle missing data in power levels events during room upgrade. (#9395) 2021-02-16 08:31:39 -05:00
Erik Johnston
a27c1fd74b 1.27.0 2021-02-16 13:12:02 +00:00
Andrew Morgan
594f2853e0
Remove dead handled_events set in invite_join (#9394) 
This PR removes a set that was created and [initially used](1d2a0040cf (diff-0bc92da3d703202f5b9be2d3f845e375f5b1a6bc6ba61705a8af9be1121f5e42R435-R436)), but is no longer today.

May help cut down a bit on the time it takes to accept invites.
 2021-02-12 22:15:50 +00:00
Patrick Cloke
7950aa8a27 Fix some typos. 2021-02-12 11:14:12 -05:00
Patrick Cloke
2c9b4a5f16 Synapse 1.27.0rc2 (2021-02-11) 
==============================
 
 Features
 --------
 
 - Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))
 
 Bugfixes
 --------
 
 - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
 - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))
 
 Improved Documentation
 ----------------------
 
 - Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAlX4AACgkQM/xY9qcR
 MEjxhg/+JzOGDmgxy4Vm9oU84htkkJsflU755ykxhwbIjVy2j+07pMgtWMmebXIS
 /tWjinwRB9OgKIK+j5RWdYzz/GRTkLCbafTLhhNOWsnaCO7nnAfZFozLOMd9g2qN
 309cS8efZkxGCISGSRm1QQKjQDx6HHx7nZVAwpqb778Q9TMrBYRAQeCk9yws7FfL
 GsP5YrBI42n84YhvflWA8J8QRYkeJJaggVWzE0XvwdI6raY1BmxMKcDvyl9iKpGb
 /QXsdketD3eJxdOlU92O5ZxRXmL00bdyAFdw2+J3Y+wjoGBnC5njKSFG3j8Z2UcP
 xIdB6w/zyGoPLKd4s7tkDI3axE7qrwFetA4NytannWGEHZ9q5tgOd5PA1kTeKYvn
 ao2os4pKMjWQOHiWLskqZVXgmoW5Kb2zgyZU/vhFAz8ImuidFULOjpEytAWqtt1B
 iuL/fRlM3z0BpwQNd3NBsblYTpZri8gdTp7ULJYtaKpT2MeG0sg3swJAptUjw9L4
 awbQuBFZ8o/E/0xwiGieSxbR6b6Nz5WiCyLaeMz6b3Y4YdomttOAZ6pL6E9y8ygU
 2wKYleSmg+tbaBKKZR5CSCwXOduSYPhWw7oqsfEfvT0NCQLuItfGNFZqxRzsf73H
 lCV0wwxNZBv8kRyMs6KivMvSI6SKinAjXS8AjzHE402ozdwOCv0=
 =NSyp
 -----END PGP SIGNATURE-----

Merge tag 'v1.27.0rc2' into develop

Synapse 1.27.0rc2 (2021-02-11)
==============================

Features
--------

- Further improvements to the user experience of registration via single sign-on. ([\#9297](https://github.com/matrix-org/synapse/issues/9297))

Bugfixes
--------

- Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](https://github.com/matrix-org/synapse/issues/9302))
- Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](https://github.com/matrix-org/synapse/issues/9313))

Improved Documentation
----------------------

- Clarify the sample configuration for changes made to the template loading code. ([\#9310](https://github.com/matrix-org/synapse/issues/9310))
 2021-02-11 11:56:03 -05:00
Patrick Cloke
40de534238 1.27.0rc2 2021-02-11 11:22:29 -05:00
Patrick Cloke
e40d88cff3
Backout changes for automatically calculating the public baseurl. (#9313) 
This breaks some people's configurations (if their Client-Server API
is not accessed via port 443).
 2021-02-11 11:16:54 -05:00
Erik Johnston
6aa87f8ce3
Ensure that we never stop reconnecting to redis (#9391) 2021-02-11 16:06:29 +00:00
Patrick Cloke
6dade80048
Combine the CAS & SAML implementations for required attributes. (#9326) 2021-02-11 10:05:15 -05:00
Eric Eastwood
80d6dc9783
Remove conflicting sqlite tables that are "reserved" (shadow fts4 tables) (#9003) 
Remove conflicting sqlite tables that throw sqlite3.OperationalError: object name reserved for internal use: event_search_content when running the twisted unit tests.

Fix #8996
 2021-02-10 20:12:57 +00:00
Brendan Abolivier
29ae04af3b
Remove unneeded type constraints on 3rd party protocol lookup responses 2021-02-09 17:50:25 +01:00
Patrick Cloke
3f58fc848d
Type hints and validation improvements. (#9321) 
* Adds type hints to the groups servlet and stringutils code.
* Assert the maximum length of some input values for spec compliance.
 2021-02-08 13:59:54 -05:00
Patrick Cloke
0963d39ea6
Handle additional errors when previewing URLs. (#9333) 
* Handle the case of lxml not finding a document tree.
* Parse the document encoding from the XML tag.
 2021-02-08 12:33:30 -05:00
David Teller
b0b2cac057
Merge pull request #9150 from Yoric/develop-context 
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
 2021-02-08 15:53:44 +01:00
Jonathan de Jong
d882fbca38
Update type hints for Cursor to match PEP 249. (#9299) 2021-02-05 15:39:19 -05:00
Erik Johnston
adc96d4236 Merge branch 'erikj/media_spam_checker' into develop 2021-02-04 17:01:59 +00:00
Erik Johnston
7e8083eb48 Add check_media_file_for_spam spam checker hook 2021-02-04 17:01:30 +00:00
Patrick Cloke
792263c97c
Handle empty rooms when generating email notifications. (#9257) 
Fixes some exceptions if the room state isn't quite as expected.
If the expected state events aren't found, try to find them in the
historical room state. If they still aren't found, fallback to a reasonable,
although ugly, value.
 2021-02-04 10:18:25 -05:00
Patrick Cloke
2ab6e67ab7
Fix escaping of braces in OIDC sample config. (#9317) 
This fixes the Jinja2 templates for the mapping provider.
 2021-02-04 09:06:20 -05:00
Jonathan de Jong
2814028ce5
Add experimental support for PyPy. (#9123) 
* Adds proper dependencies.
* Minor fixes in database layer.
 2021-02-04 08:29:47 -05:00
Marcus
b0f4119b8b
Add debug logging to DNS SRV requests. (#9305) 2021-02-03 16:47:30 -05:00
Richard van der Hoff
3f534d3fdf Merge branch 'social_login_hotfixes' into develop 2021-02-03 20:34:27 +00:00
Richard van der Hoff
17f2a512f3 Merge remote-tracking branch 'origin/release-v1.27.0' into social_login_hotfixes 2021-02-03 20:33:32 +00:00
Richard van der Hoff
e288499c60
Social login UI polish (#9301) 2021-02-03 20:31:23 +00:00
Patrick Cloke
afa18f1baa
Clarify documentation about escaping URLs in templates. (#9310) 2021-02-03 14:51:38 -05:00
Richard van der Hoff
ce669863b9
Add debug for OIDC flow (#9307) 2021-02-03 19:45:34 +00:00
Richard van der Hoff
7a0dcea3e5
social login Fix username validation javascript (#9297) 
* fix validation and don't use built-in validation UI

Co-authored-by: Bruno Windels <brunow@element.io>
 2021-02-03 17:52:55 +00:00
Richard van der Hoff
f20dadb649
Fix formatting for "bad session" error during sso registration flow (#9296) 2021-02-03 16:13:09 +00:00
dykstranet
e4cdecb310
config: Add detail to auto_join_rooms comment (#9291) 
config: Add detail to auto_join_rooms comment

Signed-off-by: Gary Dykstra <gary@dykstranet.com>
 2021-02-03 15:21:30 +00:00
Tim Gates
e1943d1353
Typo fix in a comment: subequently -> subsequently. (#8988) 2021-02-03 07:24:53 -05:00
Patrick Cloke
4ca054a4ea
Convert blacklisted IPv4 addresses to compatible IPv6 addresses. (#9240) 
Also add a few more IP ranges to the default blacklist.
 2021-02-03 07:13:46 -05:00
Erik Johnston
ff55300b91
Honour ratelimit flag for application services for invite ratelimiting (#9302) 2021-02-03 10:17:37 +00:00
Richard van der Hoff
96e460df2e
social login: add noopener to terms link (#9300) 2021-02-02 18:35:28 +00:00
Erik Johnston
2610930721 1.27.0rc1 2021-02-02 13:32:05 +00:00
Travis Ralston
b60bb28bbc
Add an admin API to get the current room state (#9168) 
This could arguably replace the existing admin API for `/members`, however that is out of scope of this change.

This sort of endpoint is ideal for moderation use cases as well as other applications, such as needing to retrieve various bits of information about a room to perform a task (like syncing power levels between two places). This endpoint exposes nothing more than an admin would be able to access with a `select *` query on their database.
 2021-02-02 11:16:29 +00:00
Richard van der Hoff
8f75bf1df7
Put SAML callback URI under /_synapse/client. (#9289) 2021-02-02 09:43:50 +00:00
Richard van der Hoff
846b9d3df0
Put OIDC callback URI under /_synapse/client. (#9288) 2021-02-01 22:56:01 +00:00
Richard van der Hoff
8fee6a3ab2 Merge branch 'social_login' into develop 2021-02-01 18:48:11 +00:00
Richard van der Hoff
351845452c fix broken HTML tag 2021-02-01 18:47:01 +00:00
Richard van der Hoff
5963426b95 Merge branch 'social_login' into develop 2021-02-01 18:46:12 +00:00
Bruno Windels
f30c3a99be make primary button not wider than viewport 2021-02-01 18:39:17 +00:00
Richard van der Hoff
c543bf87ec
Collect terms consent from the user during SSO registration (#9276) 2021-02-01 18:37:41 +00:00
Richard van der Hoff
e5d70c8a82
Improve styling and wording of SSO UIA templates (#9286) 
fixes #9171
 2021-02-01 18:36:04 +00:00
Patrick Cloke
5d38a3c97f
Refactor email summary generation. (#9260) 
* Fixes a case where no summary text was returned.
* The use of messages_from_person vs. messages_from_person_and_others
  was tweaked to depend on whether there was 1 sender or multiple senders,
  not based on if there was 1 room or multiple rooms.
 2021-02-01 13:09:39 -05:00
Richard van der Hoff
419313b06a
Improve styling and wording of SSO error templates (#9287) 2021-02-01 18:01:15 +00:00
Richard van der Hoff
85c56b5a67
Make importing display name and email optional (#9277) 2021-02-01 17:30:42 +00:00
Richard van der Hoff
18ab35284a Merge branch 'social_login' into develop 2021-02-01 17:28:37 +00:00
Jan Christian Grünhage
43dd93bb26
Add phone home stats for encrypted messages. (#9283) 
Signed-off-by: Jan Christian Grünhage <jan.christian@gruenhage.xyz>
 2021-02-01 17:06:22 +00:00
Andrew Morgan
a800603561
Prevent email UIA failures from raising a LoginError (#9265) 
Context, Fixes: https://github.com/matrix-org/synapse/issues/9263

In the past to fix an issue with old Riots re-requesting threepid validation tokens, we raised a `LoginError` during UIA instead of `InteractiveAuthIncompleteError`. This is now breaking the way Tchap logs in - which isn't standard, but also isn't disallowed by the spec.

An easy fix is just to remove the 4 year old workaround.
 2021-02-01 15:54:39 +00:00
Richard van der Hoff
4167494c90
Replace username picker with a template (#9275) 
There's some prelimiary work here to pull out the construction of a jinja environment to a separate function.

I wanted to load the template at display time rather than load time, so that it's easy to update on the fly. Honestly, I think we should do this with all our templates: the risk of ending up with malformed templates is far outweighed by the improved turnaround time for an admin trying to update them.
 2021-02-01 15:52:50 +00:00
Richard van der Hoff
8aed29dc61
Improve styling and wording of SSO redirect confirm template (#9272) 2021-02-01 15:50:56 +00:00
Richard van der Hoff
9c715a5f19
Fix SSO on workers (#9271) 
Fixes #8966.

* Factor out build_synapse_client_resource_tree

Start a function which will mount resources common to all workers.

* Move sso init into build_synapse_client_resource_tree

... so that we don't have to do it for each worker

* Fix SSO-login-via-a-worker

Expose the SSO login endpoints on workers, like the documentation says.

* Update workers config for new endpoints

Add documentation for endpoints recently added (#8942, #9017, #9262)

* remove submit_token from workers endpoints list

this *doesn't* work on workers (yet).

* changelog

* Add a comment about the odd path for SAML2Resource
 2021-02-01 15:47:59 +00:00
Richard van der Hoff
f78d07bf00
Split out a separate endpoint to complete SSO registration (#9262) 
There are going to be a couple of paths to get to the final step of SSO reg, and I want the URL in the browser to consistent. So, let's move the final step onto a separate path, which we redirect to.
 2021-02-01 13:15:51 +00:00
Ivan Shapovalov
13c7ab8181
Fixes for PyPy compatibility (#9270) 
* synapse.app.base: only call gc.freeze() on CPython

gc.freeze() is an implementation detail of CPython garbage collector,
and notably does not exist on PyPy.

Rather than playing whack-a-mole and skipping the call when under PyPy,
simply restrict it to CPython because the whole gc module is
implementation-defined.

Signed-off-by: Ivan Shapovalov <intelfx@intelfx.name>
 2021-01-30 17:22:05 +00:00
Erik Johnston
f2c1560eca
Ratelimit invites by room and target user (#9258) 2021-01-29 16:38:29 +00:00
Richard van der Hoff
0d81a6fa3e Merge branch 'social_login' into develop 2021-01-28 22:08:11 +00:00
Erik Johnston
4b73488e81
Ratelimit 3PID /requestToken API (#9238) 2021-01-28 17:39:21 +00:00
Erik Johnston
54a6afeee3
Cache config options in SSL verification (#9255) 
Reading from the config object is *slow*.
 2021-01-28 17:38:59 +00:00
David Teller
31d072aea0 FIXUP: linter 2021-01-28 16:53:40 +01:00
Patrick Cloke
a78016dadf
Add type hints to E2E handler. (#9232) 
This finishes adding type hints to the `synapse.handlers` module.
 2021-01-28 08:34:19 -05:00
David Teller
93f84e0373 FIXUP: Making get_event_context a bit more paranoid 2021-01-28 12:31:07 +01:00
David Teller
b755f60ce2 FIXUP: Removing awaitable 2021-01-28 12:31:07 +01:00
David Teller
a764869623 FIXUP: Doc 2021-01-28 12:31:07 +01:00
David Teller
b859919acc FIXUP: Now testing that the user is admin! 2021-01-28 12:31:07 +01:00
David Teller
de7f049527 FIXUP: Don't filter events at all for admin/v1/rooms/.../context/... 2021-01-28 12:31:07 +01:00
David Teller
10332c175c New API /_synapse/admin/rooms/{roomId}/context/{eventId} 
Signed-off-by: David Teller <davidt@element.io>
 2021-01-28 12:29:49 +01:00
Richard van der Hoff
a083aea396
Add 'brand' field to MSC2858 response (#9242) 
We've decided to add a 'brand' field to help clients decide how to style the
buttons.

Also, fix up the allowed characters for idp_id, while I'm in the area.
 2021-01-27 21:31:45 +00:00
Richard van der Hoff
869667760f
Support for scraping email addresses from OIDC providers (#9245) 2021-01-27 21:28:59 +00:00
Pankaj Yadav
2e537a0280
Check if a user is in the room before sending a PowerLevel event on their behalf (#9235) 2021-01-27 17:38:08 +00:00
Richard van der Hoff
fbd9de6d1f Synapse 1.26.0 (2021-01-27) 
===========================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes and for general upgrade guidance.
 
 No significant changes since 1.26.0rc2.
 
 Synapse 1.26.0rc2 (2021-01-25)
 ==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 
 Synapse 1.26.0rc1 (2021-01-20)
 ==============================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes and for general upgrade guidance.
 
 Features
 --------
 
 - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
 - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
 - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
 - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
 - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
 - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
 - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
 - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
 - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
 - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
 - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
 - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))
 
 Bugfixes
 --------
 
 - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
 - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
 - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
 - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
 - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
 - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
 - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
 - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
 - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
 - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
 - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
 - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
 - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
 - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))
 
 Improved Documentation
 ----------------------
 
 - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
 - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
 - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
 - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
 - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))
 
 Deprecations and Removals
 -------------------------
 
 - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))
 
 Internal Changes
 ----------------
 
 - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
 - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
 - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
 - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
 - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
 - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
 - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
 - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
 - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
 - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
 - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
 - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
 - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
 - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
 - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
 - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
 - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
 - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
 - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
 - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
 - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
 - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
 - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmARjqYACgkQM/xY9qcR
 MEiknRAAtga+KXEG40xg7w78XXQk6nXIhhi30nYue9/SeHMTCmLmd1+f0yTySrRz
 JMc/xe9VP1th5DThHZeHJf0tLIVz/VFA5gr/fmk44FFEFvc77IhttVHSXbYZLuM8
 komOzLTJy7ueVSpyF9xVfEEz6UIRc098b2KHjuBh3SGlZAOmFuFHUMVBeEuYo80E
 ayVWXPNP+7U5y1qLA9uY/oxMsFjl6bPvjak3YDyGWCr2Y4nOhM24AwccErs0yAtu
 kCRm9JR0yD72uRLyaGyMZ9Atg4o01BD7hu0ilxhbRIdAvI5Cdk0E7rZshoLr33wk
 Ls4XlK9SxJSV4vEu44OJBCRnDjQS9HQnwYk13z+H5HrJcxXRyIeCwRD7aofzNY8O
 kNg/1oasBDiW2iVoLJ4Y0nWLHPcNlWJxUzkkjlF8BmEUQjURNVdh0n3y3/ST+T1S
 scex0J9Z95HNoLfRYID0adrYqWTOl5GLzy9C1AwqlLWY79y6YsB4LDmmfQgoTKCM
 VICSYtEKWph6WXdpd/kJsTgHyG3UFAMDraWVez7TxXpc7lpvOMkld+0oQnHWAXC3
 XRc+63jTs6pFJ74M+ziqOrC/1ORg1DK8OLIm/TskLoYnn6L9Dh8ssXeabLv63H8o
 cEqETUbZ77rZro0Nz2H1OJG09uTFUzFEy5Ecya7g4DqtRT/Fs9Q=
 =I9CI
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0' into social_login

Synapse 1.26.0 (2021-01-27)
===========================

This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.

No significant changes since 1.26.0rc2.

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))

Synapse 1.26.0rc1 (2021-01-20)
==============================

This release brings a new schema version for Synapse and rolling back to a previous
version is not trivial. Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes and for general upgrade guidance.

Features
--------

- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))

Bugfixes
--------

- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))

Improved Documentation
----------------------

- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))

Deprecations and Removals
-------------------------

- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))

Internal Changes
----------------

- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 2021-01-27 17:27:58 +00:00
Richard van der Hoff
7fa1346f93 Merge branch 'social_login' into develop 2021-01-27 17:27:24 +00:00
Patrick Cloke
17b713850f Merge branch 'master' into develop 2021-01-27 11:13:21 -05:00
Patrick Cloke
e54746bdf7
Clean-up the template loading code. (#9200) 
* Enables autoescape by default for HTML files.
* Adds a new read_template method for reading a single template.
* Some logic clean-up.
 2021-01-27 10:59:50 -05:00
Patrick Cloke
73ed289bd2 1.26.0 2021-01-27 10:50:37 -05:00
Richard van der Hoff
a737cc2713
Implement MSC2858 support (#9183) 
Fixes #8928.
 2021-01-27 12:41:24 +00:00
Andrew Morgan
a64c29926e
Pass a dict, instead of None, to modules if a None config is specified in the homeserver config (#9229) 
If a Synapse module's config block were empty in YAML, thus being translated to a `Nonetype` in Python, then some modules could fail as that None ends up getting passed to their `parse_config` method. Modules are expected to accept a `dict` instead.

This PR ensures that if the user does end up specifying an empty config block (such as what [the default oidc config in the sample config](5310808d3b/docs/sample_config.yaml (L1816-L1845)) states) then `None` is not passed to the module. An empty dict is passed instead.

This code assumes that no existing modules are relying on receiving a `None` config block, but I'd really hope that they aren't.
 2021-01-27 11:49:31 +00:00
Patrick Cloke
1baab20352
Add type hints to various handlers. (#9223) 
With this change all handlers except the e2e_* ones have
type hints enabled.
 2021-01-26 10:50:21 -05:00
Patrick Cloke
26837d5dbe
Do not require the CAS service URL setting (use public_baseurl instead). (#9199) 
The current configuration is handled for backwards compatibility,
but is considered deprecated.
 2021-01-26 10:49:25 -05:00
Erik Johnston
dd8da8c5f6
Precompute joined hosts and store in Redis (#9198) 2021-01-26 13:57:31 +00:00
Patrick Cloke
4937fe3d6b
Try to recover from unknown encodings when previewing media. (#9164) 
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
 2021-01-26 07:32:17 -05:00
Jason Robinson
e5b659e9e1
Merge pull request #9062 from matrix-org/jaywink/admin-forward-extremities 
Add forward extremities endpoint to rooms admin API
 2021-01-26 12:57:38 +02:00
Erik Johnston
a1ff1e967f
Periodically send pings to detect dead Redis connections (#9218) 
This is done by creating a custom `RedisFactory` subclass that
periodically pings all connections in its pool.

We also ensure that the `replyTimeout` param is non-null, so that we
timeout waiting for the reply to those pings (and thus triggering a
reconnect).
 2021-01-26 10:54:54 +00:00
Jason Robinson
4936fc59fc Fix get forward extremities query 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-26 10:21:02 +02:00
Jason Robinson
cee4010f94 Merge branch 'develop' into jaywink/admin-forward-extremities 
# Conflicts:
#	synapse/rest/admin/__init__.py
 2021-01-26 10:15:32 +02:00
Jason Robinson
e20f18a766
Make natural join inner join 
Co-authored-by: Erik Johnston <erik@matrix.org>
 2021-01-26 10:13:35 +02:00
Patrick Cloke
5b857b77f7
Don't error if deleting a non-existent pusher. (#9121) 2021-01-25 14:52:30 -05:00
Patrick Cloke
4a55d267ee
Add an admin API for shadow-banning users. (#9209) 
This expands the current shadow-banning feature to be usable via
the admin API and adds documentation for it.

A shadow-banned users receives successful responses to their
client-server API requests, but the events are not propagated into rooms.

Shadow-banning a user should be used as a tool of last resort and may lead
to confusing or broken behaviour for the client.
 2021-01-25 14:49:39 -05:00
Richard van der Hoff
65fb3b2e25 Synapse 1.26.0rc2 (2021-01-25) 
==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
 MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
 +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
 BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
 d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
 oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
 O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
 VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
 g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
 =Qc7+
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc2' into social_login

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 2021-01-25 19:37:58 +00:00
Jason Robinson
fe18882bb5 Merge remote-tracking branch 'origin/develop' into jaywink/admin-forward-extremities 2021-01-25 15:55:54 +02:00
Patrick Cloke
e448dbbf5b Synapse 1.26.0rc2 (2021-01-25) 
==============================
 
 Bugfixes
 --------
 
 - Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
 - Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))
 
 Internal Changes
 ----------------
 
 - Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
 - Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAOy7IACgkQM/xY9qcR
 MEj63w//WlHcArwcpJG4LdiNaKHBUQm00DFmtV27Tl7bixa7LlClUP4qhvE1PP1n
 +uFWQUUAMUCUC31ySF3X5MEny7svD0J3r2BzbUzP8Vo1d0bHXvpKDgMrh8GbIJsF
 BEe+uQkRII15Zlkg+Oa7sk7ZI3oyg8y+SQ6yodzc8fL1cRw9bCIUDvFjFcKR3JEw
 2dHwzLq4MdYNnPME+mzhNKj30XsQ78VlbSImhGOoKdD/iBQ32E+RGpPNdVx6WDTb
 09C8pFA7qvB8d1nrnH5yGLaBzbU6mxc6jaG9xfadnhMJzG7RDVIJya+1JLm0KF2C
 d8HJWZMIFn6IdHADr7xoQF2km6QN1JTedCSzYzpfbAHwq5bOCiRqBjNnU6xC4giw
 oYqsV7xRTqRd0psh+/nN8Gz2XiZzkeMbC31kAzjDofPIZFTcte3gR1NhpxWIKoQJ
 O2gZb2wV5mq8DgJEP4Xjfe/PXiMhFIpb3fbkLdX5tZonPhs95yQHAqZFeZXUaV51
 /U3AWQrvxvkM7TpdcdDi+kqzMTCi6imXBuAzXKvY+nmcAd0nvmDg6WXAOSK3DL9O
 VDZJKm5urzBgbv8R0eKz7cWdW9YXqIIc0mcS/LCu/KYOJnK5YfhJc3grfJx9Dv/S
 g2T7T+xXUIj2ok9U5M56ACW/bdATNs62ihHx1uiereHQtP/GjtM=
 =Qc7+
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc2' into develop

Synapse 1.26.0rc2 (2021-01-25)
==============================

Bugfixes
--------

- Fix receipts and account data not being sent down sync. Introduced in v1.26.0rc1. ([\#9193](https://github.com/matrix-org/synapse/issues/9193), [\#9195](https://github.com/matrix-org/synapse/issues/9195))
- Fix chain cover update to handle events with duplicate auth events. Introduced in v1.26.0rc1. ([\#9210](https://github.com/matrix-org/synapse/issues/9210))

Internal Changes
----------------

- Add an `oidc-` prefix to any `idp_id`s which are given in the `oidc_providers` configuration. ([\#9189](https://github.com/matrix-org/synapse/issues/9189))
- Bump minimum `psycopg2` version to v2.8. ([\#9204](https://github.com/matrix-org/synapse/issues/9204))
 2021-01-25 08:51:45 -05:00
Patrick Cloke
a01605c136 1.26.0rc2 2021-01-25 08:25:40 -05:00
Patrick Cloke
6f7417c3db
Handle missing content keys when calculating presentable names. (#9165) 
Treat the content as untrusted and do not assume it is of
the proper form.
 2021-01-25 07:27:16 -05:00
Jason Robinson
8965b6cfec Merge branch 'develop' into jaywink/admin-forward-extremities 2021-01-23 21:41:35 +02:00
Jason Robinson
930ba00971 Add depth and received_ts to forward_extremities admin API response 
Also add a warning on the admin API documentation.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-23 21:34:32 +02:00
Erik Johnston
056327457f
Fix chain cover update to handle events with duplicate auth events (#9210) 2021-01-22 19:44:08 +00:00
Erik Johnston
28f255d5f3
Bump psycopg2 version (#9204) 
As we use `execute_values` with the `fetch` parameter.
 2021-01-22 11:14:49 +00:00
Patrick Cloke
a7882f9887
Return a 404 if no valid thumbnail is found. (#9163) 
If no thumbnail of the requested type exists, return a 404 instead
of erroring. This doesn't quite match the spec (which does not define
what happens if no thumbnail can be found), but is consistent with
what Synapse already does.
 2021-01-21 14:53:58 -05:00
Erik Johnston
758ed5f1bc
Speed up chain cover calculation (#9176) 2021-01-21 17:00:12 +00:00
Erik Johnston
12ec55bfaa
Increase perf of handling concurrent use of StreamIDGenerators. (#9190) 
We have seen a failure mode here where if there are many in flight
unfinished IDs then marking an ID as finished takes a lot of CPU (as
calling deque.remove iterates over the list)
 2021-01-21 16:31:51 +00:00
Erik Johnston
939ef657ce Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 16:05:13 +00:00
Erik Johnston
ccfafac882
Add schema update to fix existing DBs affected by #9193 (#9195) 2021-01-21 16:03:25 +00:00
Erik Johnston
b249f002b8 Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 15:09:30 +00:00
Erik Johnston
2506074ef0
Fix receipts or account data not being sent down sync (#9193) 
Introduced in #9104 

This wasn't picked up by the tests as this is all fine the first time you run Synapse (after upgrading), but then when you restart the wrong value is pulled from `stream_positions`.
 2021-01-21 15:09:09 +00:00
Erik Johnston
7a43482f19
Use execute_batch in more places (#9188) 
* Use execute_batch in more places

* Newsfile
 2021-01-21 14:44:12 +00:00
Dirk Klimpel
c55e62548c
Add tests for List Users Admin API (#9045) 2021-01-21 09:18:46 -05:00
Richard van der Hoff
42a8e81370
Add a check for duplicate IdP ids (#9184) 2021-01-21 13:20:58 +00:00
Richard van der Hoff
b5120f09f1 Merge remote-tracking branch 'origin/release-v1.26.0' into develop 2021-01-21 13:17:07 +00:00
Richard van der Hoff
7447f19702
Prefix idp_id with "oidc-" (#9189) 
... to avoid clashes with other SSO mechanisms
 2021-01-21 12:25:02 +00:00
Erik Johnston
eee6fcf5fa
Use execute_batch instead of executemany in places (#9181) 
`execute_batch` does fewer round trips in postgres than `executemany`, but does not give a correct `txn.rowcount` result after.
 2021-01-21 10:22:53 +00:00
Patrick Cloke
f81d02d75b Synapse 1.26.0rc1 (2021-01-20) 
==============================
 
 This release brings a new schema version for Synapse and rolling back to a previous
 verious is not trivial.  Please review [UPGRADE.rst](UPGRADE.rst) for more details
 on these changes  and for general upgrade guidance.
 
 Features
 --------
 
 - Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
 - During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
 - Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
 - Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
 - Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
 - Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
 - Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
 - Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
 - Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
 - Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
 - Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
 - Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))
 
 Bugfixes
 --------
 
 - Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
 - Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
 - Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
 - Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
 - Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
 - Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
 - Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
 - Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
 - Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
 - Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
 - Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
 - Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
 - Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
 - Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))
 
 Improved Documentation
 ----------------------
 
 - Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
 - Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
 - Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
 - Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
 - Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))
 
 Deprecations and Removals
 -------------------------
 
 - Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))
 
 Internal Changes
 ----------------
 
 - Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
 - Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
 - Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
 - Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
 - Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
 - Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
 - Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
 - Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
 - Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
 - Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
 - Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
 - Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
 - Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
 - Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
 - Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
 - Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
 - Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
 - Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
 - Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
 - Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
 - Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
 - Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
 - Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAmAIVFkACgkQM/xY9qcR
 MEhLfQ//Z4xnP5Icx732k2xpdR+JaXX6heNVFm/UuLNd6wq+uXjO/Dgc9IKQ5LXD
 kfG+/OnNGDr/eUk4mQ5s/ccBvaTrOid2IIemJd4lUBy/Q5LvPvfKIp08QFr72WJT
 7Y4Ma54nNSSMUAdESgj/aFAHjATbfHhxPOZ5OdQHGLAuJ/OUwR2ksasVNCuINbgh
 8vrxrzjeYn1Zl3UTBrsdCcat7AFqYjxK/Y0i6JxfSgRwDZjUbW4M2C4OGWZJPUrU
 2YTPHangxd+22+HObE2KDEzaHDdCR/Kj4dhCiJBynZ5iucuHSS+mgf39RtbjqWML
 01cmXXUV+FH7rnCilmF9rYdmHZ6L9BJxtrQDQU4dB106BX9G7hKR6rYHc9Z6tjOh
 3tg0cKWYMXOsTNnvRqiFNh2q0yBKS228HRAwyGosW/6NiYj43ArtYP6knX7dQyBX
 0mYNkfry2dyX6Kj5el0i9MOTHYQxfc8apsBm2M3OTNYukgQKA0NhbAquibaAxEEW
 2qIqbSp4CUlGPvM7+u+ZGTu0hEbVKKNjqMfXhuY8A4/BWiV2mSqKcEirR5cE/LqY
 mq5Ac7vbO4Uh+1xiRw/G9ITi1dqAjYIzhBawlhdUPOh+aINWTUikYYzjUSmm4PbN
 H9BTPLA9iAdVRfuWJ9um2G0DdS8Qpx/aIRh3MScXVly6cGCC6Do=
 =kgqn
 -----END PGP SIGNATURE-----

Merge tag 'v1.26.0rc1' into develop

Synapse 1.26.0rc1 (2021-01-20)
==============================

This release brings a new schema version for Synapse and rolling back to a previous
verious is not trivial.  Please review [UPGRADE.rst](UPGRADE.rst) for more details
on these changes  and for general upgrade guidance.

Features
--------

- Add support for multiple SSO Identity Providers. ([\#9015](https://github.com/matrix-org/synapse/issues/9015), [\#9017](https://github.com/matrix-org/synapse/issues/9017), [\#9036](https://github.com/matrix-org/synapse/issues/9036), [\#9067](https://github.com/matrix-org/synapse/issues/9067), [\#9081](https://github.com/matrix-org/synapse/issues/9081), [\#9082](https://github.com/matrix-org/synapse/issues/9082), [\#9105](https://github.com/matrix-org/synapse/issues/9105), [\#9107](https://github.com/matrix-org/synapse/issues/9107), [\#9109](https://github.com/matrix-org/synapse/issues/9109), [\#9110](https://github.com/matrix-org/synapse/issues/9110), [\#9127](https://github.com/matrix-org/synapse/issues/9127), [\#9153](https://github.com/matrix-org/synapse/issues/9153), [\#9154](https://github.com/matrix-org/synapse/issues/9154), [\#9177](https://github.com/matrix-org/synapse/issues/9177))
- During user-interactive authentication via single-sign-on, give a better error if the user uses the wrong account on the SSO IdP. ([\#9091](https://github.com/matrix-org/synapse/issues/9091))
- Give the `public_baseurl` a default value, if it is not explicitly set in the configuration file. ([\#9159](https://github.com/matrix-org/synapse/issues/9159))
- Improve performance when calculating ignored users in large rooms. ([\#9024](https://github.com/matrix-org/synapse/issues/9024))
- Implement [MSC2176](https://github.com/matrix-org/matrix-doc/pull/2176) in an experimental room version. ([\#8984](https://github.com/matrix-org/synapse/issues/8984))
- Add an admin API for protecting local media from quarantine. ([\#9086](https://github.com/matrix-org/synapse/issues/9086))
- Remove a user's avatar URL and display name when deactivated with the Admin API. ([\#8932](https://github.com/matrix-org/synapse/issues/8932))
- Update `/_synapse/admin/v1/users/<user_id>/joined_rooms` to work for both local and remote users. ([\#8948](https://github.com/matrix-org/synapse/issues/8948))
- Add experimental support for handling to-device messages on worker processes. ([\#9042](https://github.com/matrix-org/synapse/issues/9042), [\#9043](https://github.com/matrix-org/synapse/issues/9043), [\#9044](https://github.com/matrix-org/synapse/issues/9044), [\#9130](https://github.com/matrix-org/synapse/issues/9130))
- Add experimental support for handling `/keys/claim` and `/room_keys` APIs on worker processes. ([\#9068](https://github.com/matrix-org/synapse/issues/9068))
- Add experimental support for handling `/devices` API on worker processes. ([\#9092](https://github.com/matrix-org/synapse/issues/9092))
- Add experimental support for moving off receipts and account data persistence off master. ([\#9104](https://github.com/matrix-org/synapse/issues/9104), [\#9166](https://github.com/matrix-org/synapse/issues/9166))

Bugfixes
--------

- Fix a long-standing issue where an internal server error would occur when requesting a profile over federation that did not include a display name / avatar URL. ([\#9023](https://github.com/matrix-org/synapse/issues/9023))
- Fix a long-standing bug where some caches could grow larger than configured. ([\#9028](https://github.com/matrix-org/synapse/issues/9028))
- Fix error handling during insertion of client IPs into the database. ([\#9051](https://github.com/matrix-org/synapse/issues/9051))
- Fix bug where we didn't correctly record CPU time spent in `on_new_event` block. ([\#9053](https://github.com/matrix-org/synapse/issues/9053))
- Fix a minor bug which could cause confusing error messages from invalid configurations. ([\#9054](https://github.com/matrix-org/synapse/issues/9054))
- Fix incorrect exit code when there is an error at startup. ([\#9059](https://github.com/matrix-org/synapse/issues/9059))
- Fix `JSONDecodeError` spamming the logs when sending transactions to remote servers. ([\#9070](https://github.com/matrix-org/synapse/issues/9070))
- Fix "Failed to send request" errors when a client provides an invalid room alias. ([\#9071](https://github.com/matrix-org/synapse/issues/9071))
- Fix bugs in federation catchup logic that caused outbound federation to be delayed for large servers after start up. Introduced in v1.8.0 and v1.21.0. ([\#9114](https://github.com/matrix-org/synapse/issues/9114), [\#9116](https://github.com/matrix-org/synapse/issues/9116))
- Fix corruption of `pushers` data when a postgres bouncer is used. ([\#9117](https://github.com/matrix-org/synapse/issues/9117))
- Fix minor bugs in handling the `clientRedirectUrl` parameter for SSO login. ([\#9128](https://github.com/matrix-org/synapse/issues/9128))
- Fix "Unhandled error in Deferred: BodyExceededMaxSize" errors when .well-known files that are too large. ([\#9108](https://github.com/matrix-org/synapse/issues/9108))
- Fix "UnboundLocalError: local variable 'length' referenced before assignment" errors when the response body exceeds the expected size. This bug was introduced in v1.25.0. ([\#9145](https://github.com/matrix-org/synapse/issues/9145))
- Fix a long-standing bug "ValueError: invalid literal for int() with base 10" when `/publicRooms` is requested with an invalid `server` parameter. ([\#9161](https://github.com/matrix-org/synapse/issues/9161))

Improved Documentation
----------------------

- Add some extra docs for getting Synapse running on macOS. ([\#8997](https://github.com/matrix-org/synapse/issues/8997))
- Correct a typo in the `systemd-with-workers` documentation. ([\#9035](https://github.com/matrix-org/synapse/issues/9035))
- Correct a typo in `INSTALL.md`. ([\#9040](https://github.com/matrix-org/synapse/issues/9040))
- Add missing `user_mapping_provider` configuration to the Keycloak OIDC example. Contributed by @chris-ruecker. ([\#9057](https://github.com/matrix-org/synapse/issues/9057))
- Quote `pip install` packages when extras are used to avoid shells interpreting bracket characters. ([\#9151](https://github.com/matrix-org/synapse/issues/9151))

Deprecations and Removals
-------------------------

- Remove broken and unmaintained `demo/webserver.py` script. ([\#9039](https://github.com/matrix-org/synapse/issues/9039))

Internal Changes
----------------

- Improve efficiency of large state resolutions. ([\#8868](https://github.com/matrix-org/synapse/issues/8868), [\#9029](https://github.com/matrix-org/synapse/issues/9029), [\#9115](https://github.com/matrix-org/synapse/issues/9115), [\#9118](https://github.com/matrix-org/synapse/issues/9118), [\#9124](https://github.com/matrix-org/synapse/issues/9124))
- Various clean-ups to the structured logging and logging context code. ([\#8939](https://github.com/matrix-org/synapse/issues/8939))
- Ensure rejected events get added to some metadata tables. ([\#9016](https://github.com/matrix-org/synapse/issues/9016))
- Ignore date-rotated homeserver logs saved to disk. ([\#9018](https://github.com/matrix-org/synapse/issues/9018))
- Remove an unused column from `access_tokens` table. ([\#9025](https://github.com/matrix-org/synapse/issues/9025))
- Add a `-noextras` factor to `tox.ini`, to support running the tests with no optional dependencies. ([\#9030](https://github.com/matrix-org/synapse/issues/9030))
- Fix running unit tests when optional dependencies are not installed. ([\#9031](https://github.com/matrix-org/synapse/issues/9031))
- Allow bumping schema version when using split out state database. ([\#9033](https://github.com/matrix-org/synapse/issues/9033))
- Configure the linters to run on a consistent set of files. ([\#9038](https://github.com/matrix-org/synapse/issues/9038))
- Various cleanups to device inbox store. ([\#9041](https://github.com/matrix-org/synapse/issues/9041))
- Drop unused database tables. ([\#9055](https://github.com/matrix-org/synapse/issues/9055))
- Remove unused `SynapseService` class. ([\#9058](https://github.com/matrix-org/synapse/issues/9058))
- Remove unnecessary declarations in the tests for the admin API. ([\#9063](https://github.com/matrix-org/synapse/issues/9063))
- Remove `SynapseRequest.get_user_agent`. ([\#9069](https://github.com/matrix-org/synapse/issues/9069))
- Remove redundant `Homeserver.get_ip_from_request` method. ([\#9080](https://github.com/matrix-org/synapse/issues/9080))
- Add type hints to media repository. ([\#9093](https://github.com/matrix-org/synapse/issues/9093))
- Fix the wrong arguments being passed to `BlacklistingAgentWrapper` from `MatrixFederationAgent`. Contributed by Timothy Leung. ([\#9098](https://github.com/matrix-org/synapse/issues/9098))
- Reduce the scope of caught exceptions in `BlacklistingAgentWrapper`. ([\#9106](https://github.com/matrix-org/synapse/issues/9106))
- Improve `UsernamePickerTestCase`. ([\#9112](https://github.com/matrix-org/synapse/issues/9112))
- Remove dependency on `distutils`. ([\#9125](https://github.com/matrix-org/synapse/issues/9125))
- Enforce that replication HTTP clients are called with keyword arguments only. ([\#9144](https://github.com/matrix-org/synapse/issues/9144))
- Fix the Python 3.5 / old dependencies build in CI. ([\#9146](https://github.com/matrix-org/synapse/issues/9146))
- Replace the old `perspectives` option in the Synapse docker config file template with `trusted_key_servers`. ([\#9157](https://github.com/matrix-org/synapse/issues/9157))
 2021-01-20 11:27:39 -05:00
Richard van der Hoff
e51b2f3f91
Tighten the restrictions on idp_id (#9177) 2021-01-20 08:55:14 -05:00
Richard van der Hoff
0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Patrick Cloke
620ecf13b0
Various improvements to the federation client. (#9129) 
* Type hints for `FederationClient`.
* Using `async` functions instead of returning `Awaitable` instances.
 2021-01-20 07:59:18 -05:00
Richard van der Hoff
6c0dfd2e8e Merge remote-tracking branch 'origin/develop' into release-v1.26.0 2021-01-20 12:33:05 +00:00
Richard van der Hoff
fa50e4bf4d
Give public_baseurl a default value (#9159) 2021-01-20 12:30:41 +00:00
Patrick Cloke
72822e60be 1.26.0rc1 2021-01-20 07:14:25 -05:00
Patrick Cloke
fa842a9866
Use the account handler in additional places. (#9166) 2021-01-20 10:44:52 +00:00
Patrick Cloke
47d48a5853
Validate the server name for the /publicRooms endpoint. (#9161) 
If a remote server name is provided, ensure it is something reasonable
before making remote connections to it.
 2021-01-19 14:21:59 -05:00
Richard van der Hoff
73b03722f4
Fix error messages from OIDC config parsing (#9153) 
Make sure we report the correct config path for errors in the OIDC configs.
 2021-01-19 14:56:54 +00:00
Erik Johnston
6633a4015a
Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
Erik Johnston
f08ef64926
Enforce all replication HTTP clients calls use kwargs (#9144) 2021-01-18 15:24:04 +00:00
Patrick Cloke
2b467d0b61
Properly raise an exception when the body exceeds the max size. (#9145) 
...instead of just creating the exception object and doing nothing with it.
 2021-01-18 10:21:42 -05:00
Richard van der Hoff
02070c69fa
Fix bugs in handling clientRedirectUrl, and improve OIDC tests (#9127, #9128) 
* Factor out a common TestHtmlParser

Looks like I'm doing this in a few different places.

* Improve OIDC login test

Complete the OIDC login flow, rather than giving up halfway through.

* Ensure that OIDC login works with multiple OIDC providers

* Fix bugs in handling clientRedirectUrl

 - don't drop duplicate query-params, or params with no value
 - allow utf-8 in query-params
 2021-01-18 14:52:49 +00:00
Patrick Cloke
de1f8de319
Ensure the user ID is serialized in the payload instead of used as an instance name. (#9130) 2021-01-18 11:08:26 +00:00
Erik Johnston
350d9923cd
Make chain cover index bg update go faster (#9124) 
We do this by allowing a single iteration to process multiple rooms at a
time, as there are often a lot of really tiny rooms, which can massively
slow things down.
 2021-01-15 17:18:37 +00:00
Richard van der Hoff
2de7e263ed
Ensure we store pusher data as text (#9117) 
I don't think there's any need to use canonicaljson here.

Fixes: #4475.
 2021-01-15 16:57:23 +00:00
Richard van der Hoff
9de6b94117
Land support for multiple OIDC providers (#9110) 
This is the final step for supporting multiple OIDC providers concurrently.

First of all, we reorganise the config so that you can specify a list of OIDC providers, instead of a single one. Before:

    oidc_config:
       enabled: true
       issuer: "https://oidc_provider"
       # etc

After:

    oidc_providers:
     - idp_id: prov1
       issuer: "https://oidc_provider"

     - idp_id: prov2
       issuer: "https://another_oidc_provider"

The old format is still grandfathered in.

With that done, it's then simply a matter of having OidcHandler instantiate a new OidcProvider for each configured provider.
 2021-01-15 16:55:29 +00:00
Patrick Cloke
3e4cdfe5d9
Add an admin API endpoint to protect media. (#9086) 
Protecting media stops it from being quarantined when
e.g. all media in a room is quarantined. This is useful
for sticker packs and other media that is uploaded by
server administrators, but used by many people.
 2021-01-15 11:18:09 -05:00
Patrick Cloke
74dd906041
Avoid raising the body exceeded error multiple times. (#9108) 
Previously this code generated unreferenced `Deferred` instances
which caused "Unhandled Deferreds" errors to appear in error
situations.
 2021-01-15 11:00:13 -05:00
Richard van der Hoff
9ffac2bef1
Remote dependency on distutils (#9125) 
`distutils` is pretty much deprecated these days, and replaced with
`setuptools`. It's also annoying because it's you can't `pip install` it, and
it's hard to figure out which debian package we should depend on to make sure
it's there.

Since we only use it for a tiny function anyway, let's just vendor said
function into our codebase.
 2021-01-15 15:59:20 +00:00
Patrick Cloke
d34c6e1279
Add type hints to media rest resources. (#9093) 2021-01-15 10:57:37 -05:00
Richard van der Hoff
4575ad0b1e
Store an IdP ID in the OIDC session (#9109) 
Again in preparation for handling more than one OIDC provider, add a new caveat to the macaroon used as an OIDC session cookie, which remembers which OIDC provider we are talking to. In future, when we get a callback, we'll need it to make sure we talk to the right IdP.

As part of this, I'm adding an idp_id and idp_name field to the OIDC configuration object. They aren't yet documented, and we'll just use the old values by default.
 2021-01-15 13:22:12 +00:00
Richard van der Hoff
14950a45d6
Merge pull request #9091 from matrix-org/rav/error_on_bad_sso 
Give the user a better error when they present bad SSO creds
 2021-01-15 00:27:13 +00:00
Erik Johnston
1a08e0cdab
Fix event chain bg update. (#9118) 
We passed in a graph to `sorted_topologically` which didn't have an
entry for each node (as we dropped nodes with no edges).
 2021-01-14 18:57:32 +00:00
Erik Johnston
d2479c6870
Fix perf of get_cross_signing_keys (#9116) 2021-01-14 17:57:09 +00:00
Erik Johnston
659c415ed4
Fix chain cover background update to work with split out event persisters (#9115) 2021-01-14 17:19:35 +00:00
Erik Johnston
631dd06f2c
Fix get destinations to catch up query. (#9114) 
t was doing a sequential scan on `destination_rooms`, which took
minutes.
 2021-01-14 16:47:21 +00:00
Erik Johnston
7036e24e98
Add background update for add chain cover index (#9029) 2021-01-14 15:18:27 +00:00
Richard van der Hoff
21a296cd5a
Split OidcProvider out of OidcHandler (#9107) 
The idea here is that we will have an instance of OidcProvider for each
configured IdP, with OidcHandler just doing the marshalling of them.

For now it's still hardcoded with a single provider.
 2021-01-14 13:29:17 +00:00
Tim Leung
12702be951
Fix wrong arguments being passed to BlacklistingAgentWrapper (#9098) 
A reactor was being passed instead of a whitelist for the BlacklistingAgentWrapper
used by the WellyKnownResolver. This coulld cause exceptions when attempting to
connect to IP addresses that are blacklisted, but in reality this did not have any
observable affect since this code is not used for IP literals.
 2021-01-14 06:59:26 -05:00
Richard van der Hoff
420031906a Move complete_sso_ui_auth into SSOHandler 
since we're hacking on this code anyway, may as well move it out of the
cluttered AuthHandler.
 2021-01-13 20:22:41 +00:00
Richard van der Hoff
5310808d3b Give the user a better error when they present bad SSO creds 
If a user tries to do UI Auth via SSO, but uses the wrong account on the SSO
IdP, try to give them a better error.

Previously, the UIA would claim to be successful, but then the operation in
question would simply fail with "auth fail". Instead, serve up an error page
which explains the failure.
 2021-01-13 20:22:41 +00:00
Richard van der Hoff
d02e4b2825
Merge pull request #9105 from matrix-org/rav/multi_idp/oidc_provider_config 
Enhancements to OIDC configuration handling
 2021-01-13 19:51:46 +00:00
Patrick Cloke
aee8e6a95d
Reduce scope of exception handler. (#9106) 
Removes a bare `except Exception` clause and replaces it with
catching a specific exception around the portion that might throw.
 2021-01-13 13:27:49 -05:00
Richard van der Hoff
dc3c83a933 Add jsonschema verification for the oidc provider config 2021-01-13 17:47:27 +00:00
Patrick Cloke
d1eb1b96e8
Register the /devices endpoint on workers. (#9092) 2021-01-13 12:35:40 -05:00
Richard van der Hoff
7cc9509eca Extract OIDCProviderConfig object 
Collect all the config options which related to an OIDC provider into a single
object.
 2021-01-13 16:40:02 +00:00
Patrick Cloke
98a64b7f7f
Add basic domain validation for DomainSpecificString.is_valid. (#9071) 
This checks that the domain given to `DomainSpecificString.is_valid` (e.g.
`UserID`, `RoomAlias`, etc.) is of a valid form. Previously some validation
was done on the localpart (e.g. the sigil), but not the domain portion.
 2021-01-13 07:05:16 -05:00
Erik Johnston
aa4d8c1f9a Merge branch 'master' into develop 2021-01-13 10:36:55 +00:00
Richard van der Hoff
bc4bf7b384
Preparatory refactors of OidcHandler (#9067) 
Some light refactoring of OidcHandler, in preparation for bigger things:

  * remove inheritance from deprecated BaseHandler
  * add an object to hold the things that go into a session cookie
  * factor out a separate class for manipulating said cookies
 2021-01-13 10:26:12 +00:00
Erik Johnston
3dd6ba135e 1.25.0 2021-01-13 10:19:12 +00:00
Dirk Klimpel
7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932) 
This only applies if the user's data is to be erased.
 2021-01-12 16:30:15 -05:00
Richard van der Hoff
789d9ebad3
UI Auth via SSO: redirect the user to an appropriate SSO. (#9081) 
If we have integrations with multiple identity providers, when the user does a UI Auth, we need to redirect them to the right one.

There are a few steps to this. First of all we actually need to store the userid of the user we are trying to validate in the UIA session, since the /auth/sso/fallback/web request is unauthenticated.

Then, once we get the /auth/sso/fallback/web request, we can fish the user id out of the session, and use it to look up the external id mappings, and hence pick an SSO provider for them.
 2021-01-12 17:38:03 +00:00
Marcus
e385c8b473
Don't apply the IP range blacklist to proxy connections (#9084) 
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
 2021-01-12 12:20:30 -05:00
Patrick Cloke
723b19748a
Handle bad JSON data being returned from the federation API. (#9070) 2021-01-12 11:07:01 -05:00
Richard van der Hoff
0f8945e166
Kill off HomeServer.get_ip_from_request() (#9080) 
Homeserver.get_ip_from_request() used to be a bit more complicated, but now it is totally redundant. Let's get rid of it.
 2021-01-12 12:48:12 +00:00
Richard van der Hoff
2ec8ca5e60
Remove SynapseRequest.get_user_agent (#9069) 
SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests",
which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the 
LoggingContext and write the right entries to the request log).

Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a
SynapseRequest when there is nothing synapse-specific about the Request at all, and any old
twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult.

In short: move get_user_agent out to a utility function.
 2021-01-12 12:34:16 +00:00
Jason Robinson
c177faf5a9 Remove trailing whitespace to appease the linter 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-11 23:55:44 +02:00
Jason Robinson
49c619a9a2 Simplify delete_forward_extremities_for_room_txn SQL 
As per feedback.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-11 23:49:58 +02:00
Jason Robinson
da16d06301 Address pr feedback 
* docs updates
* prettify SQL
* add missing copyright
* cursor_to_dict
* update touched files copyright years

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-11 23:43:58 +02:00
David Teller
b161528fcc
Also support remote users on the joined_rooms admin API. (#8948) 
For remote users, only the rooms which the server knows about are returned.
Local users have all of their joined rooms returned.
 2021-01-11 14:32:17 -05:00
Erik Johnston
c9195744a4
Move more encryption endpoints off master (#9068) 2021-01-11 18:01:27 +00:00
Erik Johnston
1315a2e8be
Use a chain cover index to efficiently calculate auth chain difference (#8868) 2021-01-11 16:09:22 +00:00
Richard van der Hoff
671138f658
Clean up exception handling in the startup code (#9059) 
Factor out the exception handling in the startup code to a utility function,
and fix the some logging and exit code stuff.
 2021-01-11 15:55:05 +00:00
Erik Johnston
4e04435bda
Remove old tables after schema version bump (#9055) 
These tables are unused, and can be dropped now the schema version has been bumped.
 2021-01-11 13:58:19 +00:00
Erik Johnston
63f4990298
Ensure rejected events get added to some metadata tables (#9016) 
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
 2021-01-11 13:57:33 +00:00
Richard van der Hoff
7db2622d30
Remove unused SynapseService (#9058) 2021-01-11 10:24:22 +00:00
Jerin J Titus
c21d8f1c1d
Drop last_used column from access_tokens (#9025) 
* Dropped last_used column from access_tokens

Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
 2021-01-11 10:23:49 +00:00
Jason Robinson
b52fb703f7 Don't try to use f-strings 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-11 09:47:03 +02:00
Jason Robinson
2eb421b606 Merge branch 'develop' into jaywink/admin-forward-extremities 2021-01-09 22:00:04 +02:00
Jason Robinson
90ad4d443a Implement clearing cache after deleting forward extremities 
Also run linter.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-09 21:57:41 +02:00
Erik Johnston
a03d71dc9d
Fix "Starting metrics collection from sentinel context" errors (#9053) 2021-01-08 14:33:53 +00:00
Richard van der Hoff
12f79da587
Merge pull request #9036 from matrix-org/rav/multi_idp/tests 
Add tests for the IdP picker
 2021-01-08 14:24:41 +00:00
Richard van der Hoff
d32870ffa5
Fix validate_config on nested objects (#9054) 2021-01-08 14:23:04 +00:00
Erik Johnston
fa5f5cbc74
Fix error handling during insertion of client IPs (#9051) 
You can't continue using a transaction once an exception has been
raised, so catching and dropping the error here is pointless and just
causes more errors.
 2021-01-08 14:15:20 +00:00
Jason Robinson
85c0999bfb Add Rooms admin forward extremities DELETE endpoint 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-08 00:12:23 +02:00
Jason Robinson
c91045f56c Move unknown room ID error into resolve_room_id 
Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-07 23:03:54 +02:00
Jason Robinson
b849e46139 Add forward extremities endpoint to rooms admin API 
GET /_synapse/admin/v1/rooms/<identifier>/forward_extremities now gets forward extremities for a room, returning count and the list of extremities.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
 2021-01-07 23:01:59 +02:00
Erik Johnston
b530eaa262
Allow running sendToDevice on workers (#9044) 2021-01-07 20:19:26 +00:00
Erik Johnston
5e99a94502
Support routing edu's to multiple instances (#9042) 
This is in preparation for moving `SendToDeviceServlet` off master
 2021-01-07 18:07:28 +00:00
Erik Johnston
e34df813ce
Ensure that remote users' device list resyncing always happens on master (#9043) 
Currently `DeviceMessageHandler` only ever exists on master, but that is about to change.
 2021-01-07 18:06:52 +00:00
Erik Johnston
63593134a1
Some cleanups to device inbox store. (#9041) 2021-01-07 17:20:44 +00:00
Richard van der Hoff
8a910f97a4 Add some tests for the IDP picker flow 2021-01-07 14:56:42 +00:00
Patrick Cloke
23d701864f
Improve the performance of calculating ignored users in large rooms (#9024) 
This allows for efficiently finding which users ignore a particular
user.

Co-authored-by: Erik Johnston <erik@matrix.org>
 2021-01-07 13:03:38 +00:00
Erik Johnston
eee3c3c52f
Handle updating schema version without any deltas. (#9033) 
This can happen when using a split out state database and we've upgraded
the schema version without there being any changes in the state schema.
 2021-01-07 11:33:36 +00:00
Patrick Cloke
1b4d5d6acf
Empty iterables should count towards cache usage. (#9028) 2021-01-06 12:33:20 -05:00
Patrick Cloke
0312266ee3 Synapse 1.25.0rc1 (2021-01-06) 
==============================
 
 Removal warning
 ---------------
 
 The old [Purge Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/purge_room.md)
 and [Shutdown Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/shutdown_room.md)
 are deprecated and will be removed in a future release. They will be replaced by the
 [Delete Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/rooms.md#delete-room-api).
 
 `POST /_synapse/admin/v1/rooms/<room_id>/delete` replaces `POST /_synapse/admin/v1/purge_room` and
 `POST /_synapse/admin/v1/shutdown_room/<room_id>`.
 
 Features
 --------
 
 - Add an admin API that lets server admins get power in rooms in which local users have power. ([\#8756](https://github.com/matrix-org/synapse/issues/8756))
 - Add optional HTTP authentication to replication endpoints. ([\#8853](https://github.com/matrix-org/synapse/issues/8853))
 - Improve the error messages printed as a result of configuration problems for extension modules. ([\#8874](https://github.com/matrix-org/synapse/issues/8874))
 - Add the number of local devices to Room Details Admin API. Contributed by @dklimpel. ([\#8886](https://github.com/matrix-org/synapse/issues/8886))
 - Add `X-Robots-Tag` header to stop web crawlers from indexing media. Contributed by Aaron Raimist. ([\#8887](https://github.com/matrix-org/synapse/issues/8887))
 - Spam-checkers may now define their methods as `async`. ([\#8890](https://github.com/matrix-org/synapse/issues/8890))
 - Add support for allowing users to pick their own user ID during a single-sign-on login. ([\#8897](https://github.com/matrix-org/synapse/issues/8897), [\#8900](https://github.com/matrix-org/synapse/issues/8900), [\#8911](https://github.com/matrix-org/synapse/issues/8911), [\#8938](https://github.com/matrix-org/synapse/issues/8938), [\#8941](https://github.com/matrix-org/synapse/issues/8941), [\#8942](https://github.com/matrix-org/synapse/issues/8942), [\#8951](https://github.com/matrix-org/synapse/issues/8951))
 - Add an `email.invite_client_location` configuration option to send a web client location to the invite endpoint on the identity server which allows customisation of the email template. ([\#8930](https://github.com/matrix-org/synapse/issues/8930))
 - The search term in the list room and list user Admin APIs is now treated as case-insensitive. ([\#8931](https://github.com/matrix-org/synapse/issues/8931))
 - Apply an IP range blacklist to push and key revocation requests. ([\#8821](https://github.com/matrix-org/synapse/issues/8821), [\#8870](https://github.com/matrix-org/synapse/issues/8870), [\#8954](https://github.com/matrix-org/synapse/issues/8954))
 - Add an option to allow re-use of user-interactive authentication sessions for a period of time. ([\#8970](https://github.com/matrix-org/synapse/issues/8970))
 - Allow running the redact endpoint on workers. ([\#8994](https://github.com/matrix-org/synapse/issues/8994))
 
 Bugfixes
 --------
 
 - Fix bug where we might not correctly calculate the current state for rooms with multiple extremities. ([\#8827](https://github.com/matrix-org/synapse/issues/8827))
 - Fix a long-standing bug in the register admin endpoint (`/_synapse/admin/v1/register`) when the `mac` field was not provided. The endpoint now properly returns a 400 error. Contributed by @edwargix. ([\#8837](https://github.com/matrix-org/synapse/issues/8837))
 - Fix a long-standing bug on Synapse instances supporting Single-Sign-On, where users would be prompted to enter their password to confirm certain actions, even though they have not set a password. ([\#8858](https://github.com/matrix-org/synapse/issues/8858))
 - Fix a longstanding bug where a 500 error would be returned if the `Content-Length` header was not provided to the upload media resource. ([\#8862](https://github.com/matrix-org/synapse/issues/8862))
 - Add additional validation to pusher URLs to be compliant with the specification. ([\#8865](https://github.com/matrix-org/synapse/issues/8865))
 - Fix the error code that is returned when a user tries to register on a homeserver on which new-user registration has been disabled. ([\#8867](https://github.com/matrix-org/synapse/issues/8867))
 - Fix a bug where `PUT /_synapse/admin/v2/users/<user_id>` failed to create a new user when `avatar_url` is specified. Bug introduced in Synapse v1.9.0. ([\#8872](https://github.com/matrix-org/synapse/issues/8872))
 - Fix a 500 error when attempting to preview an empty HTML file. ([\#8883](https://github.com/matrix-org/synapse/issues/8883))
 - Fix occasional deadlock when handling SIGHUP. ([\#8918](https://github.com/matrix-org/synapse/issues/8918))
 - Fix login API to not ratelimit application services that have ratelimiting disabled. ([\#8920](https://github.com/matrix-org/synapse/issues/8920))
 - Fix bug where we ratelimited auto joining of rooms on registration (using `auto_join_rooms` config). ([\#8921](https://github.com/matrix-org/synapse/issues/8921))
 - Fix a bug where deactivated users appeared in the user directory when their profile information was updated. ([\#8933](https://github.com/matrix-org/synapse/issues/8933), [\#8964](https://github.com/matrix-org/synapse/issues/8964))
 - Fix bug introduced in Synapse v1.24.0 which would cause an exception on startup if both `enabled` and `localdb_enabled` were set to `False` in the `password_config` setting of the configuration file. ([\#8937](https://github.com/matrix-org/synapse/issues/8937))
 - Fix a bug where 500 errors would be returned if the `m.room_history_visibility` event had invalid content. ([\#8945](https://github.com/matrix-org/synapse/issues/8945))
 - Fix a bug causing common English words to not be considered for a user directory search. ([\#8959](https://github.com/matrix-org/synapse/issues/8959))
 - Fix bug where application services couldn't register new ghost users if the server had reached its MAU limit. ([\#8962](https://github.com/matrix-org/synapse/issues/8962))
 - Fix a long-standing bug where a `m.image` event without a `url` would cause errors on push. ([\#8965](https://github.com/matrix-org/synapse/issues/8965))
 - Fix a small bug in v2 state resolution algorithm, which could also cause performance issues for rooms with large numbers of power levels. ([\#8971](https://github.com/matrix-org/synapse/issues/8971))
 - Add validation to the `sendToDevice` API to raise a missing parameters error instead of a 500 error. ([\#8975](https://github.com/matrix-org/synapse/issues/8975))
 - Add validation of group IDs to raise a 400 error instead of a 500 eror. ([\#8977](https://github.com/matrix-org/synapse/issues/8977))
 
 Improved Documentation
 ----------------------
 
 - Fix the "Event persist rate" section of the included grafana dashboard by adding missing prometheus rules. ([\#8802](https://github.com/matrix-org/synapse/issues/8802))
 - Combine related media admin API docs. ([\#8839](https://github.com/matrix-org/synapse/issues/8839))
 - Fix an error in the documentation for the SAML username mapping provider. ([\#8873](https://github.com/matrix-org/synapse/issues/8873))
 - Clarify comments around template directories in `sample_config.yaml`. ([\#8891](https://github.com/matrix-org/synapse/issues/8891))
 - Moved instructions for database setup, adjusted heading levels and improved syntax highlighting in [INSTALL.md](../INSTALL.md). Contributed by fossterer. ([\#8987](https://github.com/matrix-org/synapse/issues/8987))
 - Update the example value of `group_creation_prefix` in the sample configuration. ([\#8992](https://github.com/matrix-org/synapse/issues/8992))
 - Link the Synapse developer room to the development section in the docs. ([\#9002](https://github.com/matrix-org/synapse/issues/9002))
 
 Deprecations and Removals
 -------------------------
 
 - Deprecate Shutdown Room and Purge Room Admin APIs. ([\#8829](https://github.com/matrix-org/synapse/issues/8829))
 
 Internal Changes
 ----------------
 
 - Properly store the mapping of external ID to Matrix ID for CAS users. ([\#8856](https://github.com/matrix-org/synapse/issues/8856), [\#8958](https://github.com/matrix-org/synapse/issues/8958))
 - Remove some unnecessary stubbing from unit tests. ([\#8861](https://github.com/matrix-org/synapse/issues/8861))
 - Remove unused `FakeResponse` class from unit tests. ([\#8864](https://github.com/matrix-org/synapse/issues/8864))
 - Pass `room_id` to `get_auth_chain_difference`. ([\#8879](https://github.com/matrix-org/synapse/issues/8879))
 - Add type hints to push module. ([\#8880](https://github.com/matrix-org/synapse/issues/8880), [\#8882](https://github.com/matrix-org/synapse/issues/8882), [\#8901](https://github.com/matrix-org/synapse/issues/8901), [\#8940](https://github.com/matrix-org/synapse/issues/8940), [\#8943](https://github.com/matrix-org/synapse/issues/8943), [\#9020](https://github.com/matrix-org/synapse/issues/9020))
 - Simplify logic for handling user-interactive-auth via single-sign-on servers. ([\#8881](https://github.com/matrix-org/synapse/issues/8881))
 - Skip the SAML tests if the requirements (`pysaml2` and `xmlsec1`) aren't available. ([\#8905](https://github.com/matrix-org/synapse/issues/8905))
 - Fix multiarch docker image builds. ([\#8906](https://github.com/matrix-org/synapse/issues/8906))
 - Don't publish `latest` docker image until all archs are built. ([\#8909](https://github.com/matrix-org/synapse/issues/8909))
 - Various clean-ups to the structured logging and logging context code. ([\#8916](https://github.com/matrix-org/synapse/issues/8916), [\#8935](https://github.com/matrix-org/synapse/issues/8935))
 - Automatically drop stale forward-extremities under some specific conditions. ([\#8929](https://github.com/matrix-org/synapse/issues/8929))
 - Refactor test utilities for injecting HTTP requests. ([\#8946](https://github.com/matrix-org/synapse/issues/8946))
 - Add a maximum size of 50 kilobytes to .well-known lookups. ([\#8950](https://github.com/matrix-org/synapse/issues/8950))
 - Fix bug in `generate_log_config` script which made it write empty files. ([\#8952](https://github.com/matrix-org/synapse/issues/8952))
 - Clean up tox.ini file; disable coverage checking for non-test runs. ([\#8963](https://github.com/matrix-org/synapse/issues/8963))
 - Add type hints to the admin and room list handlers. ([\#8973](https://github.com/matrix-org/synapse/issues/8973))
 - Add type hints to the receipts and user directory handlers. ([\#8976](https://github.com/matrix-org/synapse/issues/8976))
 - Drop the unused `local_invites` table. ([\#8979](https://github.com/matrix-org/synapse/issues/8979))
 - Add type hints to the base storage code. ([\#8980](https://github.com/matrix-org/synapse/issues/8980))
 - Support using PyJWT v2.0.0 in the test suite. ([\#8986](https://github.com/matrix-org/synapse/issues/8986))
 - Fix `tests.federation.transport.RoomDirectoryFederationTests` and ensure it runs in CI. ([\#8998](https://github.com/matrix-org/synapse/issues/8998))
 - Add type hints to the crypto module. ([\#8999](https://github.com/matrix-org/synapse/issues/8999))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl/1rv4ACgkQM/xY9qcR
 MEiIGw/+PytgC2yOpaR7cIQl0K1v+y4X/AmNbshdQch7GtC3vtKxdmo4PJp4KwY4
 Sxiiykq46Xx48w/Hqd3hoC7OqsTyfmNjRKW0InsTarbaXMrSv9umOtScItzMMWq8
 SCmDPQ+cF0SP2GhOMwB/N5JpRS12OpdBzHXtcbwxULilTuO/EKD2/W1zq3wiSP7r
 BxhHHYsx69hufUGC3OGD0I8e26TqjgkLiVm0FeM6cC5TubozO3qrJDlmwTPSpFrL
 WZEBEbnYUVz4ydMKRq4ALiZzPzxxBfgQmITb79CNqim4+fc+M+AqV6mRez2ULrqh
 d4/UEMTaA0yh54i0oqei3gwf2ucns/NDftQ9cYwF1+a9LCcWAyJZY7+/9Hk4VKSU
 KqQUzGdBfxyefsLc2hNGANzS/VbThdejC09IjaxKtHSgaT+bb5LxS8GcbvcG5FxY
 LyD8yeazgXPub+Ixr6dvD51roCuBtpCYvGZtHHYOiX/S1MYbq22Duet2Z1zhy7qB
 Jl8r3L32x5wAG3lR8aLqDVNTrC1ur0TW9wUJ88bDjcwn0PsseQG41DHbQSoXQIlk
 p6/IKjJjfnbKJFTwzfC8RpL29NtyTPqA/d2JDJ7yqUbpEkAnc5GN+pO0B/736aOH
 aTRu8nVdP7GOoP4whZ9C7r5kYjiCR3jPtpZHRYXaPJtkw57dWS8=
 =/g7u
 -----END PGP SIGNATURE-----

Merge tag 'v1.25.0rc1' into develop

Synapse 1.25.0rc1 (2021-01-06)
==============================

Removal warning
---------------

The old [Purge Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/purge_room.md)
and [Shutdown Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/shutdown_room.md)
are deprecated and will be removed in a future release. They will be replaced by the
[Delete Room API](https://github.com/matrix-org/synapse/tree/master/docs/admin_api/rooms.md#delete-room-api).

`POST /_synapse/admin/v1/rooms/<room_id>/delete` replaces `POST /_synapse/admin/v1/purge_room` and
`POST /_synapse/admin/v1/shutdown_room/<room_id>`.

Features
--------

- Add an admin API that lets server admins get power in rooms in which local users have power. ([\#8756](https://github.com/matrix-org/synapse/issues/8756))
- Add optional HTTP authentication to replication endpoints. ([\#8853](https://github.com/matrix-org/synapse/issues/8853))
- Improve the error messages printed as a result of configuration problems for extension modules. ([\#8874](https://github.com/matrix-org/synapse/issues/8874))
- Add the number of local devices to Room Details Admin API. Contributed by @dklimpel. ([\#8886](https://github.com/matrix-org/synapse/issues/8886))
- Add `X-Robots-Tag` header to stop web crawlers from indexing media. Contributed by Aaron Raimist. ([\#8887](https://github.com/matrix-org/synapse/issues/8887))
- Spam-checkers may now define their methods as `async`. ([\#8890](https://github.com/matrix-org/synapse/issues/8890))
- Add support for allowing users to pick their own user ID during a single-sign-on login. ([\#8897](https://github.com/matrix-org/synapse/issues/8897), [\#8900](https://github.com/matrix-org/synapse/issues/8900), [\#8911](https://github.com/matrix-org/synapse/issues/8911), [\#8938](https://github.com/matrix-org/synapse/issues/8938), [\#8941](https://github.com/matrix-org/synapse/issues/8941), [\#8942](https://github.com/matrix-org/synapse/issues/8942), [\#8951](https://github.com/matrix-org/synapse/issues/8951))
- Add an `email.invite_client_location` configuration option to send a web client location to the invite endpoint on the identity server which allows customisation of the email template. ([\#8930](https://github.com/matrix-org/synapse/issues/8930))
- The search term in the list room and list user Admin APIs is now treated as case-insensitive. ([\#8931](https://github.com/matrix-org/synapse/issues/8931))
- Apply an IP range blacklist to push and key revocation requests. ([\#8821](https://github.com/matrix-org/synapse/issues/8821), [\#8870](https://github.com/matrix-org/synapse/issues/8870), [\#8954](https://github.com/matrix-org/synapse/issues/8954))
- Add an option to allow re-use of user-interactive authentication sessions for a period of time. ([\#8970](https://github.com/matrix-org/synapse/issues/8970))
- Allow running the redact endpoint on workers. ([\#8994](https://github.com/matrix-org/synapse/issues/8994))

Bugfixes
--------

- Fix bug where we might not correctly calculate the current state for rooms with multiple extremities. ([\#8827](https://github.com/matrix-org/synapse/issues/8827))
- Fix a long-standing bug in the register admin endpoint (`/_synapse/admin/v1/register`) when the `mac` field was not provided. The endpoint now properly returns a 400 error. Contributed by @edwargix. ([\#8837](https://github.com/matrix-org/synapse/issues/8837))
- Fix a long-standing bug on Synapse instances supporting Single-Sign-On, where users would be prompted to enter their password to confirm certain actions, even though they have not set a password. ([\#8858](https://github.com/matrix-org/synapse/issues/8858))
- Fix a longstanding bug where a 500 error would be returned if the `Content-Length` header was not provided to the upload media resource. ([\#8862](https://github.com/matrix-org/synapse/issues/8862))
- Add additional validation to pusher URLs to be compliant with the specification. ([\#8865](https://github.com/matrix-org/synapse/issues/8865))
- Fix the error code that is returned when a user tries to register on a homeserver on which new-user registration has been disabled. ([\#8867](https://github.com/matrix-org/synapse/issues/8867))
- Fix a bug where `PUT /_synapse/admin/v2/users/<user_id>` failed to create a new user when `avatar_url` is specified. Bug introduced in Synapse v1.9.0. ([\#8872](https://github.com/matrix-org/synapse/issues/8872))
- Fix a 500 error when attempting to preview an empty HTML file. ([\#8883](https://github.com/matrix-org/synapse/issues/8883))
- Fix occasional deadlock when handling SIGHUP. ([\#8918](https://github.com/matrix-org/synapse/issues/8918))
- Fix login API to not ratelimit application services that have ratelimiting disabled. ([\#8920](https://github.com/matrix-org/synapse/issues/8920))
- Fix bug where we ratelimited auto joining of rooms on registration (using `auto_join_rooms` config). ([\#8921](https://github.com/matrix-org/synapse/issues/8921))
- Fix a bug where deactivated users appeared in the user directory when their profile information was updated. ([\#8933](https://github.com/matrix-org/synapse/issues/8933), [\#8964](https://github.com/matrix-org/synapse/issues/8964))
- Fix bug introduced in Synapse v1.24.0 which would cause an exception on startup if both `enabled` and `localdb_enabled` were set to `False` in the `password_config` setting of the configuration file. ([\#8937](https://github.com/matrix-org/synapse/issues/8937))
- Fix a bug where 500 errors would be returned if the `m.room_history_visibility` event had invalid content. ([\#8945](https://github.com/matrix-org/synapse/issues/8945))
- Fix a bug causing common English words to not be considered for a user directory search. ([\#8959](https://github.com/matrix-org/synapse/issues/8959))
- Fix bug where application services couldn't register new ghost users if the server had reached its MAU limit. ([\#8962](https://github.com/matrix-org/synapse/issues/8962))
- Fix a long-standing bug where a `m.image` event without a `url` would cause errors on push. ([\#8965](https://github.com/matrix-org/synapse/issues/8965))
- Fix a small bug in v2 state resolution algorithm, which could also cause performance issues for rooms with large numbers of power levels. ([\#8971](https://github.com/matrix-org/synapse/issues/8971))
- Add validation to the `sendToDevice` API to raise a missing parameters error instead of a 500 error. ([\#8975](https://github.com/matrix-org/synapse/issues/8975))
- Add validation of group IDs to raise a 400 error instead of a 500 eror. ([\#8977](https://github.com/matrix-org/synapse/issues/8977))

Improved Documentation
----------------------

- Fix the "Event persist rate" section of the included grafana dashboard by adding missing prometheus rules. ([\#8802](https://github.com/matrix-org/synapse/issues/8802))
- Combine related media admin API docs. ([\#8839](https://github.com/matrix-org/synapse/issues/8839))
- Fix an error in the documentation for the SAML username mapping provider. ([\#8873](https://github.com/matrix-org/synapse/issues/8873))
- Clarify comments around template directories in `sample_config.yaml`. ([\#8891](https://github.com/matrix-org/synapse/issues/8891))
- Moved instructions for database setup, adjusted heading levels and improved syntax highlighting in [INSTALL.md](../INSTALL.md). Contributed by fossterer. ([\#8987](https://github.com/matrix-org/synapse/issues/8987))
- Update the example value of `group_creation_prefix` in the sample configuration. ([\#8992](https://github.com/matrix-org/synapse/issues/8992))
- Link the Synapse developer room to the development section in the docs. ([\#9002](https://github.com/matrix-org/synapse/issues/9002))

Deprecations and Removals
-------------------------

- Deprecate Shutdown Room and Purge Room Admin APIs. ([\#8829](https://github.com/matrix-org/synapse/issues/8829))

Internal Changes
----------------

- Properly store the mapping of external ID to Matrix ID for CAS users. ([\#8856](https://github.com/matrix-org/synapse/issues/8856), [\#8958](https://github.com/matrix-org/synapse/issues/8958))
- Remove some unnecessary stubbing from unit tests. ([\#8861](https://github.com/matrix-org/synapse/issues/8861))
- Remove unused `FakeResponse` class from unit tests. ([\#8864](https://github.com/matrix-org/synapse/issues/8864))
- Pass `room_id` to `get_auth_chain_difference`. ([\#8879](https://github.com/matrix-org/synapse/issues/8879))
- Add type hints to push module. ([\#8880](https://github.com/matrix-org/synapse/issues/8880), [\#8882](https://github.com/matrix-org/synapse/issues/8882), [\#8901](https://github.com/matrix-org/synapse/issues/8901), [\#8940](https://github.com/matrix-org/synapse/issues/8940), [\#8943](https://github.com/matrix-org/synapse/issues/8943), [\#9020](https://github.com/matrix-org/synapse/issues/9020))
- Simplify logic for handling user-interactive-auth via single-sign-on servers. ([\#8881](https://github.com/matrix-org/synapse/issues/8881))
- Skip the SAML tests if the requirements (`pysaml2` and `xmlsec1`) aren't available. ([\#8905](https://github.com/matrix-org/synapse/issues/8905))
- Fix multiarch docker image builds. ([\#8906](https://github.com/matrix-org/synapse/issues/8906))
- Don't publish `latest` docker image until all archs are built. ([\#8909](https://github.com/matrix-org/synapse/issues/8909))
- Various clean-ups to the structured logging and logging context code. ([\#8916](https://github.com/matrix-org/synapse/issues/8916), [\#8935](https://github.com/matrix-org/synapse/issues/8935))
- Automatically drop stale forward-extremities under some specific conditions. ([\#8929](https://github.com/matrix-org/synapse/issues/8929))
- Refactor test utilities for injecting HTTP requests. ([\#8946](https://github.com/matrix-org/synapse/issues/8946))
- Add a maximum size of 50 kilobytes to .well-known lookups. ([\#8950](https://github.com/matrix-org/synapse/issues/8950))
- Fix bug in `generate_log_config` script which made it write empty files. ([\#8952](https://github.com/matrix-org/synapse/issues/8952))
- Clean up tox.ini file; disable coverage checking for non-test runs. ([\#8963](https://github.com/matrix-org/synapse/issues/8963))
- Add type hints to the admin and room list handlers. ([\#8973](https://github.com/matrix-org/synapse/issues/8973))
- Add type hints to the receipts and user directory handlers. ([\#8976](https://github.com/matrix-org/synapse/issues/8976))
- Drop the unused `local_invites` table. ([\#8979](https://github.com/matrix-org/synapse/issues/8979))
- Add type hints to the base storage code. ([\#8980](https://github.com/matrix-org/synapse/issues/8980))
- Support using PyJWT v2.0.0 in the test suite. ([\#8986](https://github.com/matrix-org/synapse/issues/8986))
- Fix `tests.federation.transport.RoomDirectoryFederationTests` and ensure it runs in CI. ([\#8998](https://github.com/matrix-org/synapse/issues/8998))
- Add type hints to the crypto module. ([\#8999](https://github.com/matrix-org/synapse/issues/8999))
 2021-01-06 07:42:38 -05:00
Patrick Cloke
0248409bfa
Handle a display name / avatar URL not included in a federation request. (#9023) 
These may be omitted if not set, but Synapse assumed they would
be in the response.
 2021-01-06 07:29:03 -05:00
Patrick Cloke
2fe0fb21f6 1.25.0rc1 2021-01-06 07:08:13 -05:00
Patrick Cloke
37eaf9c272
Fix-up assertions about last stream token in push (#9020) 
The last stream token is always known and we do not need to handle none.
 2021-01-05 10:53:15 -05:00
Patrick Cloke
06fefe0bb1
Add type hints to the logging context code. (#8939) 2021-01-05 08:06:55 -05:00
Patrick Cloke
9dde9c9f01
Implement MSC2176: Updated redaction rules (#8984) 
An experimental room version ("org.matrix.msc2176") contains
the new redaction rules for testing.
 2021-01-05 07:41:48 -05:00
Richard van der Hoff
111b673fc1
Add initial support for a "pick your IdP" page (#9017) 
During login, if there are multiple IdPs enabled, offer the user a choice of
IdPs.
 2021-01-05 11:25:28 +00:00
Richard van der Hoff
d2c616a413
Combine the SSO Redirect Servlets (#9015) 
* Implement CasHandler.handle_redirect_request

... to make it match OidcHandler and SamlHandler

* Clean up interface for OidcHandler.handle_redirect_request

Make it accept `client_redirect_url=None`.

* Clean up interface for `SamlHandler.handle_redirect_request`

... bring it into line with CAS and OIDC by making it take a Request parameter,
move the magic for `client_redirect_url` for UIA into the handler, and fix the
return type to be a `str` rather than a `bytes`.

* Define a common protocol for SSO auth provider impls

* Give SsoIdentityProvider an ID and register them

* Combine the SSO Redirect servlets

Now that the SsoHandler knows about the identity providers, we can combine the
various *RedirectServlets into a single implementation which delegates to the
right IdP.

* changelog
 2021-01-04 18:13:49 +00:00
Patrick Cloke
31b1905e13
Add type hints to the receipts and user directory handlers. (#8976) 2021-01-04 10:05:12 -05:00
Patrick Cloke
1c9a850562
Add type hints to the crypto module. (#8999) 2021-01-04 10:04:50 -05:00
Patrick Cloke
0eccf53146
Use the SSO handler helpers for CAS registration/login. (#8856) 2021-01-03 16:25:44 +00:00
Patrick Cloke
b7c580e333
Check if group IDs are valid before using them. (#8977) 2020-12-30 08:39:59 -05:00
Patrick Cloke
637282bb50
Add additional type hints to the storage module. (#8980) 2020-12-30 08:09:53 -05:00
Patrick Cloke
9999eb2d02
Add type hints to admin and room list handlers. (#8973) 2020-12-29 17:42:10 -05:00
Patrick Cloke
14a7371375
Validate input parameters for the sendToDevice API. (#8975) 
This makes the "messages" key in the content required. This is currently
optional in the spec, but that seems to be an error.
 2020-12-29 12:47:45 -05:00
Jerin J Titus
cfcf5541b4
Update the value of group_creation_prefix in sample config. (#8992) 
Removes the trailing slash with causes issues with matrix.to/Element.
 2020-12-29 09:30:48 -05:00
Patrick Cloke
68bb26da69
Allow redacting events on workers (#8994) 
Adds the redacts endpoint to workers that have the client listener.
 2020-12-29 07:40:12 -05:00
Patrick Cloke
d0c3c24eb2
Drop the unused local_invites table. (#8979) 
This table has been unused since Synapse v1.17.0.
 2020-12-29 07:26:29 -05:00
Patrick Cloke
4218473f9e
Refactor the CAS handler in prep for using the abstracted SSO code. (#8958) 
This makes the CAS handler look more like the SAML/OIDC handlers:

* Render errors to users instead of throwing JSON errors.
* Internal reorganization.
 2020-12-18 13:09:45 -05:00
Patrick Cloke
56e00ca85e
Send the location of the web client to the IS when inviting via 3PIDs. (#8930) 
Adds a new setting `email.invite_client_location` which, if defined, is
passed to the identity server during invites.
 2020-12-18 11:01:57 -05:00
Erik Johnston
d781a81e69
Allow server admin to get admin bit in rooms where local user is an admin (#8756) 
This adds an admin API that allows a server admin to get power in a room if a local user has power in a room. Will also invite the user if they're not in the room and its a private room. Can specify another user (rather than the admin user) to be granted power.

Co-authored-by: Matthew Hodgson <matthew@matrix.org>
 2020-12-18 15:37:19 +00:00
Erik Johnston
5e7d75daa2
Fix mainline ordering in state res v2 (#8971) 
This had two effects 1) it'd give the wrong answer and b) would iterate
*all* power levels in the auth chain of each event. The latter of which
can be *very* expensive for certain types of IRC bridge rooms that have
large numbers of power level changes.
 2020-12-18 15:00:34 +00:00
Richard van der Hoff
28877fade9
Implement a username picker for synapse (#8942) 
The final part (for now) of my work to implement a username picker in synapse itself. The idea is that we allow
`UsernameMappingProvider`s to return `localpart=None`, in which case, rather than redirecting the browser
back to the client, we redirect to a username-picker resource, which allows the user to enter a username.
We *then* complete the SSO flow (including doing the client permission checks).

The static resources for the username picker itself (in 
https://github.com/matrix-org/synapse/tree/rav/username_picker/synapse/res/username_picker)
are essentially lifted wholesale from
https://github.com/matrix-org/matrix-synapse-saml-mozilla/tree/master/matrix_synapse_saml_mozilla/res. 
As the comment says, we might want to think about making them customisable, but that can be a follow-up. 

Fixes #8876.
 2020-12-18 14:19:46 +00:00
Patrick Cloke
5d4c330ed9
Allow re-using a UI auth validation for a period of time (#8970) 2020-12-18 07:33:57 -05:00
Patrick Cloke
4136255d3c
Ensure that a URL exists in the content during push. (#8965) 
This fixes an KeyError exception, after this PR the content
is just considered unknown.
 2020-12-18 07:26:15 -05:00
Erik Johnston
a7a913918c Merge remote-tracking branch 'origin/erikj/as_mau_block' into develop 2020-12-18 09:51:56 +00:00
Erik Johnston
70586aa63e
Try and drop stale extremities. (#8929) 
If we see stale extremities while persisting events, and notice that
they don't change the result of state resolution, we drop them.
 2020-12-18 09:49:18 +00:00
Brendan Abolivier
f2783fc201
Use the simple dictionary in full text search for the user directory (#8959) 
* Use the simple dictionary in fts for the user directory

* Clarify naming
 2020-12-17 14:42:30 +01:00
Erik Johnston
4c33796b20 Correctly handle AS registerations and add test 2020-12-17 12:55:21 +00:00
Dirk Klimpel
c07022303e
Fix a bug that deactivated users appear in the directory (#8933) 
Fixes a bug that deactivated users appear in the directory when their profile information was updated.

To change profile information of deactivated users is neccesary for example you will remove displayname or avatar.
But they should not appear in directory. They are deactivated.



Co-authored-by: Erik Johnston <erikj@jki.re>
 2020-12-17 12:05:39 +00:00
Erik Johnston
7932d4e9f7 Don't MAU limit AS ghost users 2020-12-17 12:04:14 +00:00
Dirk Klimpel
06006058d7
Make search statement in List Room and User Admin API case-insensitive (#8931) 2020-12-17 10:43:37 +00:00
Patrick Cloke
ff5c4da128
Add a maximum size for well-known lookups. (#8950) 2020-12-16 17:25:24 -05:00
Richard van der Hoff
e1b8e37f93
Push login completion down into SsoHandler (#8941) 
This is another part of my work towards fixing #8876. It moves some of the logic currently in the SAML and OIDC handlers - in particular the call to `AuthHandler.complete_sso_login` down into the `SsoHandler`.
 2020-12-16 20:01:53 +00:00
Patrick Cloke
44b7d4c6d6
Fix the sample config location for the ip_range_whitelist setting. (#8954) 
Move it from the federation section to the server section to match
ip_range_blacklist.
 2020-12-16 14:40:47 -05:00
Patrick Cloke
bd30cfe86a
Convert internal pusher dicts to attrs classes. (#8940) 
This improves type hinting and should use less memory.
 2020-12-16 11:25:30 -05:00
Patrick Cloke
be2db93b3c
Do not assume that the contents dictionary includes history_visibility. (#8945) 2020-12-16 08:46:37 -05:00
Richard van der Hoff
01333681bc
Preparatory refactoring of the SamlHandlerTestCase (#8938) 
* move simple_async_mock to test_utils

... so that it can be re-used

* Remove references to `SamlHandler._map_saml_response_to_user` from tests

This method is going away, so we can no longer use it as a test point. Instead,
factor out a higher-level method which takes a SAML object, and verify correct
behaviour by mocking out `AuthHandler.complete_sso_login`.

* changelog
 2020-12-15 20:56:10 +00:00
Patrick Cloke
b3a4b53587
Fix handling of stream tokens for push. (#8943) 
Removes faulty assertions and fixes the logic to ensure the max
stream token is always set.
 2020-12-15 10:41:34 -05:00
Richard van der Hoff
6d02eb22df
Fix startup failure with localdb_enabled: False (#8937) 2020-12-14 20:42:03 +00:00
Patrick Cloke
1619802228
Various clean-ups to the logging context code (#8935) 2020-12-14 14:19:47 -05:00