Commit Graph

1487 Commits

Author SHA1 Message Date
Patrick Cloke
c01bc5f43d
Add remaining type hints to synapse.events. (#11098) 2021-11-02 09:55:52 -04:00
Aaron R
3ae1464efd
Support Client-Server API r0.6.1 (#11097)
Fixes #11064

Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-11-01 13:28:39 +00:00
Patrick Cloke
ba00e20234
Add a thread relation type per MSC3440. (#11088)
Adds experimental support for MSC3440's `io.element.thread` relation
type (and the aggregation for it).
2021-10-21 14:39:16 -04:00
Eric Eastwood
35d6b914eb
Resolve and share state_groups for all historical events in batch (MSC2716) (#10975)
Resolve and share `state_groups` for all historical events in batch.  This also helps for showing the appropriate avatar/displayname in Element and will work whenever `/messages` has one of the historical messages as the first message in the batch.

This does have the flaw where if you just insert a single historical event somewhere, it probably won't resolve the state correctly from `/messages` or `/context` since it will grab a non historical event above or below with resolved state which never included the historical state back then. For the same reasions, this also does not work in Element between the transition from actual messages to historical messages. In the Gitter case, this isn't really a problem since all of the historical messages are in one big lump at the beginning of the room.

For a future iteration, might be good to look at `/messages` and `/context` to additionally add the `state` for any historical messages in that batch.

---

How are the `state_groups` shared? To illustrate the `state_group` sharing, see this example:


**Before** (new `state_group` for every event 😬, very inefficient):
```
# Tests from https://github.com/matrix-org/complement/pull/206
$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events
create_new_client_event m.room.member event=$_JXfwUDIWS6xKGG4SmZXjSFrizhARM7QblhATVWWUcA state_group=None
create_new_client_event org.matrix.msc2716.insertion event=$1ZBfmBKEjg94d-vGYymKrVYeghwBOuGJ3wubU1-I9y0 state_group=9
create_new_client_event org.matrix.msc2716.insertion event=$Mq2JvRetTyclPuozRI682SAjYp3GqRuPc8_cH5-ezPY state_group=10
create_new_client_event m.room.message event=$MfmY4rBQkxrIp8jVwVMTJ4PKnxSigpG9E2cn7S0AtTo state_group=11
create_new_client_event m.room.message event=$uYOv6V8wiF7xHwOMt-60d1AoOIbqLgrDLz6ZIQDdWUI state_group=12
create_new_client_event m.room.message event=$PAbkJRMxb0bX4A6av463faiAhxkE3FEObM1xB4D0UG4 state_group=13
create_new_client_event org.matrix.msc2716.batch event=$Oy_S7AWN7rJQe_MYwGPEy6RtbYklrI-tAhmfiLrCaKI state_group=14
```

**After** (all events in batch sharing `state_group=10`) (the base insertion event has `state_group=8` which matches the `prev_event` we're inserting next to):

```
# Tests from https://github.com/matrix-org/complement/pull/206
$ COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh TestBackfillingHistory/parallel/should_resolve_member_state_events_for_historical_events
create_new_client_event m.room.member event=$PWomJ8PwENYEYuVNoG30gqtybuQQSZ55eldBUSs0i0U state_group=None
create_new_client_event org.matrix.msc2716.insertion event=$e_mCU7Eah9ABF6nQU7lu4E1RxIWccNF05AKaTT5m3lw state_group=9
create_new_client_event org.matrix.msc2716.insertion event=$ui7A3_GdXIcJq0C8GpyrF8X7B3DTjMd_WGCjogax7xU state_group=10
create_new_client_event m.room.message event=$EnTIM5rEGVezQJiYl62uFBl6kJ7B-sMxWqe2D_4FX1I state_group=10
create_new_client_event m.room.message event=$LGx5jGONnBPuNhAuZqHeEoXChd9ryVkuTZatGisOPjk state_group=10
create_new_client_event m.room.message event=$wW0zwoN50lbLu1KoKbybVMxLbKUj7GV_olozIc5i3M0 state_group=10
create_new_client_event org.matrix.msc2716.batch event=$5ZB6dtzqFBCEuMRgpkU201Qhx3WtXZGTz_YgldL6JrQ state_group=10
```
2021-10-13 17:44:00 -05:00
Patrick Cloke
1f9d0b8a7a
Add type hints to synapse.events.*. (#11066)
Except `synapse/events/__init__.py`, which will be done in a follow-up.
2021-10-13 07:24:07 -04:00
Eric Eastwood
a7d22c36db
Refactor MSC2716 /batch_send endpoint into separate handler functions (#10974) 2021-10-08 18:35:00 -05:00
Patrick Cloke
f4b1a9a527
Require direct references to configuration variables. (#10985)
This removes the magic allowing accessing configurable
variables directly from the config object. It is now required
that a specific configuration class is used (e.g. `config.foo`
must be replaced with `config.server.foo`).
2021-10-06 10:47:41 -04:00
Patrick Cloke
a0f48ee89d
Use direct references for configuration variables (part 7). (#10959) 2021-10-04 07:18:54 -04:00
Lukas Lihotzki
145cb6d08e
Fix getTurnServer response: return an integer ttl (#10922)
`ttl` must be an integer according to the OpenAPI spec:
https://github.com/matrix-org/matrix-doc/blob/old_master/data/api/client-server/voip.yaml#L70

True division (`/`) returns a float instead (`"ttl": 7200.0`).
Floor division (`//`) returns an integer, so the response is spec compliant.

Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>
2021-09-30 08:04:55 -04:00
Patrick Cloke
94b620a5ed
Use direct references for configuration variables (part 6). (#10916) 2021-09-29 06:44:15 -04:00
Travis Ralston
8cef1ab2ac
Implement MSC3069: Guest support on whoami (#9655) 2021-09-29 11:32:45 +01:00
Eric Eastwood
9fd057b8c5
Ensure (room_id, next_batch_id) is unique to avoid cross-talk/conflicts between batches (MSC2716) (#10877)
Part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)

Part of https://github.com/matrix-org/synapse/issues/10737
2021-09-28 21:23:16 -05:00
Patrick Cloke
bb7fdd821b
Use direct references for configuration variables (part 5). (#10897) 2021-09-24 07:25:21 -04:00
Patrick Cloke
47854c71e9
Use direct references for configuration variables (part 4). (#10893) 2021-09-23 12:03:01 -04:00
Patrick Cloke
e584534403
Use direct references for some configuration variables (part 3) (#10885)
This avoids the overhead of searching through the various
configuration classes by directly referencing the class that
the attributes are in.

It also improves type hints since mypy can now resolve the
types of the configuration variables.
2021-09-23 07:13:34 -04:00
Andrew Morgan
aa2c027792
Remove unnecessary parentheses around tuples returned from methods (#10889) 2021-09-23 11:59:07 +01:00
Eric Eastwood
51e2db3598
Rename MSC2716 things from chunk to batch to match /batch_send endpoint (#10838)
See https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684574497

Dropping support for older MSC2716 room versions so we don't have to worry about
supporting both chunk and batch events.
2021-09-21 15:06:28 -05:00
Eric Eastwood
ee557b5375
Rename /batch_send query parameter from ?prev_event to more obvious usage with ?prev_event_id (MSC2716) (#10839)
As mentioned in https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r705872887
and https://github.com/matrix-org/synapse/issues/10737
2021-09-21 14:10:01 +01:00
Eric Eastwood
145c006ef7
Verify ?chunk_id actually corresponds to an insertion event that exists (MSC2716) (#10776) 2021-09-15 09:34:30 +01:00
Eric Eastwood
1c555527b3
Split out /batch_send meta events to their own fields (MSC2716) (#10777) 2021-09-15 09:30:58 +01:00
Patrick Cloke
01c88a09cd
Use direct references for some configuration variables (#10798)
Instead of proxying through the magic getter of the RootConfig
object. This should be more performant (and is more explicit).
2021-09-13 13:07:12 -04:00
reivilibre
524b8ead77
Add types to synapse.util. (#10601) 2021-09-10 17:03:18 +01:00
Sean
273b6861f2
Remove unstable MSC2858 API, including experimental.msc2858_enabled config option (#10693)
Signed-off-by: Sean Quah <seanq@element.io>
2021-09-09 17:59:59 +01:00
Patrick Cloke
ecbfa4fe4f
Additional type hints for client REST servlets (part 5) (#10736)
Additionally this enforce type hints on all function signatures inside
of the synapse.rest.client package.
2021-09-03 09:22:22 -04:00
Patrick Cloke
d1f1b46c2c
Additional type hints for client REST servlets (part 4) (#10728) 2021-09-01 11:59:32 -04:00
Patrick Cloke
287918e2d4
Additional type hints for the client REST servlets (part 3). (#10707) 2021-08-31 17:22:29 +00:00
Patrick Cloke
029b7ad7b9
Remove unused compare_digest function. (#10706) 2021-08-27 07:08:02 -04:00
Patrick Cloke
1aa0dad021
Additional type hints for REST servlets (part 2). (#10674)
Applies the changes from #10665 to additional modules.
2021-08-26 11:53:52 +00:00
Richard van der Hoff
15db8b7c7f
Correctly initialise the synapse_user_logins metric. (#10677)
Fix a bug where the prometheus metrics for SSO logins wouldn't be initialised
until the first user logged in with a given auth provider.
2021-08-24 09:17:51 +00:00
Patrick Cloke
bd7d398b05
Additional type hints for the sync REST servlet. (#10666) 2021-08-23 08:14:42 -04:00
Patrick Cloke
2af6d31b78
Addtional type hints for the REST servlets. (#10665) 2021-08-23 08:14:17 -04:00
Callum Brown
947dbbdfd1
Implement MSC3231: Token authenticated registration (#10142)
Signed-off-by: Callum Brown <callum@calcuode.com>

This is part of my GSoC project implementing [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231).
2021-08-21 22:14:43 +01:00
David Robertson
ee3b2ac59a
Validate device_keys for C-S /keys/query requests (#10593)
* Validate device_keys for C-S /keys/query requests

Closes #10354

A small, not particularly critical fix. I'm interested in seeing if we
can find a more systematic approach though. #8445 is the place for any discussion.
2021-08-20 15:47:03 +01:00
Dirk Klimpel
b5fef6054a
Support MSC3283: Expose enable_set_displayname in capabilities (#10452) 2021-08-19 09:40:40 +00:00
Erik Johnston
78a70a2e0b Merge branch 'release-v1.41' into develop 2021-08-18 17:02:47 +01:00
Callum Brown
6e613a10d0
Display an error page during failure of fallback UIA. (#10561) 2021-08-18 08:13:35 -04:00
Andrew Morgan
703e3a9e85
Allow /createRoom to be run on workers (#10564)
Fixes https://github.com/matrix-org/synapse/issues/7867
2021-08-17 14:33:16 +01:00
reivilibre
642a42edde
Flatten the synapse.rest.client package (#10600) 2021-08-17 11:57:58 +00:00
Andrew Morgan
b62eba7705
Always list fallback key types in /sync (#10623) 2021-08-17 12:32:25 +01:00
Patrick Cloke
5af83efe8d
Validate the max_rooms_per_space parameter to ensure it is non-negative. (#10611) 2021-08-16 12:01:30 -04:00
Michael Telatynski
0ace38b7b3
Experimental support for MSC3266 Room Summary API. (#10394) 2021-08-16 14:49:12 +00:00
Eric Eastwood
c8d54be44c
Move /batch_send to /v2_alpha directory (MSC2716) (#10576)
* Move /batch_send to /v2_alpha directory

As pointed out by @erikjohnston,
https://github.com/matrix-org/synapse/pull/10552#discussion_r685836624
2021-08-13 14:37:24 -05:00
Eric Eastwood
8c654b7309
Only return state events that the AS passed in via state_events_at_start (MSC2716) (#10552)
* Only return state events that the AS passed in via state_events_at_start

As discovered by @Half-Shot in
https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684158448

Part of MSC2716

* Add changelog

* Fix changelog extension
2021-08-10 18:10:40 -05:00
Patrick Cloke
fe1d0c8618
Add local support for the new spaces summary endpoint (MSC2946) (#10549)
This adds support for the /hierarchy endpoint, which is an
update to MSC2946. Currently this only supports rooms known
locally to the homeserver.
2021-08-10 13:08:17 -04:00
Erik Johnston
60f0534b6e
Fix exceptions in logs when failing to get remote room list (#10541) 2021-08-06 14:05:41 +01:00
Eric Eastwood
f5a368bb48
Mark all MSC2716 events as historical (#10537)
* Mark all MSC2716 events as historical
2021-08-05 20:35:53 -05:00
Brendan Abolivier
e8a3e81402
Don't fail on empty bodies when sending out read receipts (#10531)
Fixes a bug introduced in rc1 that would cause Synapse to 400 on read receipts requests with empty bodies.

Broken in #10413
2021-08-04 16:13:24 +02:00
Eric Eastwood
d0b294ad97
Make historical events discoverable from backfill for servers without any scrollback history (MSC2716) (#10245)
* Make historical messages available to federated servers

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

Follow-up to https://github.com/matrix-org/synapse/pull/9247

* Debug message not available on federation

* Add base starting insertion point when no chunk ID is provided

* Fix messages from multiple senders in historical chunk

Follow-up to https://github.com/matrix-org/synapse/pull/9247

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.

* Remove debug lines

* Messing with selecting insertion event extremeties

* Move db schema change to new version

* Add more better comments

* Make a fake requester with just what we need

See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080

* Store insertion events in table

* Make base insertion event float off on its own

See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889

Conflicts:
	synapse/rest/client/v1/room.py

* Validate that the app service can actually control the given user

See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455

Conflicts:
	synapse/rest/client/v1/room.py

* Add some better comments on what we're trying to check for

* Continue debugging

* Share validation logic

* Add inserted historical messages to /backfill response

* Remove debug sql queries

* Some marker event implemntation trials

* Clean up PR

* Rename insertion_event_id to just event_id

* Add some better sql comments

* More accurate description

* Add changelog

* Make it clear what MSC the change is part of

* Add more detail on which insertion event came through

* Address review and improve sql queries

* Only use event_id as unique constraint

* Fix test case where insertion event is already in the normal DAG

* Remove debug changes

* Switch to chunk events so we can auth via power_levels

Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.

So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.

* Switch to chunk events for federation

* Add unstable room version to support new historical PL

* Fix federated events being rejected for no state_groups

Add fix from https://github.com/matrix-org/synapse/pull/10439
until it merges.

* Only connect base insertion event to prev_event_ids

Per discussion with @erikjohnston,
https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org

* Make it possible to get the room_version with txn

* Allow but ignore historical events in unsupported room version

See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489

We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either.

Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` ->  `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`.

* Move to unique index syntax

See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509

* High-level document how the insertion->chunk lookup works

* Remove create_event fallback for room_versions

See https://github.com/matrix-org/synapse/pull/10245/files#r677641879

* Use updated method name
2021-07-28 10:46:37 -05:00
Šimon Brandner
c3b037795a
Support for MSC2285 (hidden read receipts) (#10413)
Implementation of matrix-org/matrix-doc#2285
2021-07-28 10:05:11 +02:00
Travis Ralston
b3a757eb3b
Support MSC2033: Device ID on whoami (#9918)
* Fix no-access-token bug in deactivation tests
* Support MSC2033: Device ID on whoami
* Test for appservices too

MSC: https://github.com/matrix-org/matrix-doc/pull/2033

The MSC has passed FCP, which means stable endpoints can be used.
2021-07-27 05:28:20 +00:00
Erik Johnston
683deee9a4 Merge branch 'master' into develop 2021-07-23 09:03:19 +01:00
David Baker
7da24b975d
Always send device_one_time_keys_count (#10457)
As per comment

Fixes https://github.com/matrix-org/synapse/issues/10456
See also https://github.com/vector-im/element-android/issues/3725
2021-07-22 15:29:27 +01:00
Patrick Cloke
590cc4e888
Add type hints to additional servlet functions (#10437)
Improves type hints for:

* parse_{boolean,integer}
* parse_{boolean,integer}_from_args
* parse_json_{value,object}_from_request

And fixes any incorrect calls that resulted from unknown types.
2021-07-21 18:12:22 +00:00
Patrick Cloke
5db118626b
Add a return type to parse_string. (#10438)
And set the required attribute in a few places which will error if
a parameter is not provided.
2021-07-21 09:47:56 -04:00
Eric Eastwood
2d89c66b88
Switch to chunk events so we can auth via power_levels (MSC2716) (#10432)
Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.

So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.
2021-07-21 10:29:57 +00:00
Michael Telatynski
69226c1ab4
MSC3244 room capabilities implementation (#10283) 2021-07-20 12:59:23 +01:00
Jonathan de Jong
95e47b2e78
[pyupgrade] synapse/ (#10348)
This PR is tantamount to running 
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```

Part of #9744
2021-07-19 15:28:05 +01:00
Jonathan de Jong
98aec1cc9d
Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
Brendan Abolivier
36dc15412d
Add a module type for account validity (#9884)
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity.

To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
2021-07-16 18:11:53 +02:00
Eric Eastwood
0d5b08ac7a
Fix messages from multiple senders in historical chunk (MSC2716) (#10276)
Fix messages from multiple senders in historical chunk. This also means that an app service does not need to define `?user_id` when using this endpoint.

Follow-up to https://github.com/matrix-org/synapse/pull/9247

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
2021-07-13 14:12:33 -05:00
Eric Eastwood
d26094e92c
Add base starting insertion event when no chunk ID is provided (MSC2716) (#10250)
* Add base starting insertion point when no chunk ID is provided

This is so we can have the marker event point to this initial
insertion event and be able to traverse the events in the first chunk.
2021-07-08 20:25:59 -05:00
Quentin Gliech
bd4919fb72
MSC2918 Refresh tokens implementation (#9450)
This implements refresh tokens, as defined by MSC2918

This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235

The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one.

Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-06-24 14:33:20 +01:00
Andrew Morgan
394673055d
Re-introduce "Leave out optional keys from /sync" change (#10214)
Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.
2021-06-23 15:57:41 +01:00
Eric Eastwood
96f6293de5
Add endpoints for backfilling history (MSC2716) (#9247)
Work on https://github.com/matrix-org/matrix-doc/pull/2716
2021-06-22 10:02:53 +01:00
Patrick Cloke
76f9c701c3
Always require users to re-authenticate for dangerous operations. (#10184)
Dangerous actions means deactivating an account, modifying an account
password, or adding a 3PID.

Other actions (deleting devices, uploading keys) can re-use the same UI
auth session if ui_auth.session_timeout is configured.
2021-06-16 11:07:28 -04:00
Lukas Lihotzki
2c240213f4
Fix requestOpenIdToken response: integer expires_in (#10175)
`expires_in` must be an integer according to the OpenAPI spec:
https://github.com/matrix-org/matrix-doc/blob/master/data/api/client-server/definitions/openid_token.yaml#L32

True division (`/`) returns a float instead (`"expires_in": 3600.0`).
Floor division (`//`) returns an integer, so the response is spec compliant.

Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>
2021-06-16 13:16:35 +01:00
Patrick Cloke
9e5ab6dd58
Remove the experimental flag for knocking and use stable prefixes / endpoints. (#10167)
* Room version 7 for knocking.
* Stable prefixes and endpoints (both client and federation) for knocking.
* Removes the experimental configuration flag.
2021-06-15 07:45:14 -04:00
Sorunome
d936371b69
Implement knock feature (#6739)
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
Patrick Cloke
11846dff8c
Limit the number of in-flight /keys/query requests from a single device. (#10144) 2021-06-09 07:05:32 -04:00
Patrick Cloke
9e4610cc27
Correct type hints for parse_string(s)_from_args. (#10137) 2021-06-08 08:30:48 -04:00
Travis Ralston
5325f0308c
r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224)
[MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. 

This is progress towards https://github.com/matrix-org/synapse/issues/8334
2021-06-03 13:50:49 +01:00
Erik Johnston
5eed6348ce
Move some more endpoints off master (#10084) 2021-05-27 22:45:43 +01:00
Callum Brown
8fb9af570f
Make reason and score optional for report_event (#10077)
Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414
See #8551 

Signed-off-by: Callum Brown <callum@calcuode.com>
2021-05-27 18:42:23 +01:00
Patrick Cloke
f42e4c4eb9
Remove the experimental spaces enabled flag. (#10063)
In lieu of just always enabling the unstable spaces endpoint and
unstable room version.
2021-05-26 14:35:16 -04:00
Patrick Cloke
f4833e0c06
Support fetching the spaces summary via GET over federation. (#9947)
Per changes in MSC2946, the C-S and S-S APIs for spaces summary
should use GET requests.

Until this is stable, the POST endpoints still exist.

This does not switch federation requests to use the GET version yet
since it is newly added and already deployed servers might not support
it. When switching to the stable endpoint we should switch to GET
requests.
2021-05-11 12:21:43 -04:00
Erik Johnston
eba431c539
Revert "Leave out optional keys from /sync (#9919)" (#9940)
This reverts commit e9eb3549d3.
2021-05-06 15:06:35 +01:00
DeepBlueV7.X
e9eb3549d3
Leave out optional keys from /sync (#9919)
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.)

Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2021-05-05 14:37:56 +01:00
Erik Johnston
9d25a0ae65
Split presence out of master (#9820) 2021-04-23 12:21:55 +01:00
Erik Johnston
177dae2704
Limit length of accepted email addresses (#9855) 2021-04-22 17:49:11 +01:00
Patrick Cloke
683d6f75af
Rename handler and config modules which end in handler/config. (#9816) 2021-04-20 14:55:20 -04:00
Andrew Morgan
71f0623de9
Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832)
This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19 19:16:34 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Will Hunt
e300ef64b1
Require AppserviceRegistrationType (#9548)
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
2021-04-12 15:13:55 +01:00
Erik Johnston
963f4309fe
Make RateLimiter class check for ratelimit overrides (#9711)
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited.

We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits.

Fixes #9663
2021-03-30 12:06:09 +01:00
Erik Johnston
b5efcb577e
Make it possible to use dmypy (#9692)
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
2021-03-26 16:49:46 +00:00
Patrick Cloke
7e8dc9934e
Add a type hints for service notices to the HomeServer object. (#9675) 2021-03-24 06:48:46 -04:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. (#9665) 2021-03-23 07:12:48 -04:00
Brendan Abolivier
b6ed4f55ac
Incorporate review 2021-03-19 18:19:50 +01:00
Brendan Abolivier
592d6305fd
Merge branch 'develop' into babolivier/msc3026 2021-03-19 16:12:40 +01:00
Richard van der Hoff
066068f034 fix mypy 2021-03-19 12:20:11 +00:00
Patrick Cloke
9b0e3009fa Fix type-hints from bad merge. 2021-03-18 14:40:56 -04:00
Richard van der Hoff
004234f03a
Initial spaces summary API (#9643)
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
2021-03-18 18:24:16 +00:00
Brendan Abolivier
405aeb0b2c
Implement MSC3026: busy presence state 2021-03-18 16:34:47 +01:00
Patrick Cloke
cc324d53fe
Fix up types for the typing handler. (#9638)
By splitting this to two separate methods the callers know
what methods they can expect on the handler.
2021-03-17 11:30:21 -04:00
Dirk Klimpel
8000cf1315
Return m.change_password.enabled=false if local database is disabled (#9588)
Instead of if the user does not have a password hash. This allows a SSO
user to add a password to their account, but only if the local password
database is configured.
2021-03-16 11:44:25 -04:00
Richard van der Hoff
dd69110d95
Add support for stable MSC2858 API (#9617)
The stable format uses different brand identifiers, so we need to support two
identifiers for each IdP.
2021-03-16 11:21:26 +00:00
Andrew Morgan
1b0eaed21f
Prevent bundling aggregations for state events (#9619)
There's no need to do aggregation bundling for state events. Doing so can cause performance issues.
2021-03-16 10:27:51 +00:00
Patrick Cloke
55da8df078
Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
Richard van der Hoff
df425c2c63
Prometheus metrics for logins and registrations (#9511)
Add prom metrics for number of users successfully registering and logging in, by SSO provider.
2021-03-04 16:39:27 +00:00
Richard van der Hoff
7eb6e39a8f
Record the SSO Auth Provider in the login token (#9510)
This great big stack of commits is a a whole load of hoop-jumping to make it easier to store additional values in login tokens, and then to actually store the SSO Identity Provider in the login token. (Making use of that data will follow in a subsequent PR.)
2021-03-04 14:44:22 +00:00
Patrick Cloke
a0bc9d387e
Use the proper Request in type hints. (#9515)
This also pins the Twisted version in the mypy job for CI until
proper type hints are fixed throughout Synapse.
2021-03-01 12:23:46 -05:00
Richard van der Hoff
15090de850
SSO: redirect to public URL before setting cookies (#9436)
... otherwise, we don't get the cookie back.
2021-02-26 14:02:06 +00:00
Patrick Cloke
fc8b3d8809
Ratelimit cross-user key sharing requests. (#8957) 2021-02-19 13:20:34 -05:00
Patrick Cloke
d2f0ec12d5
Add type hints to groups code. (#9393) 2021-02-17 08:41:47 -05:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke
7950aa8a27 Fix some typos. 2021-02-12 11:14:12 -05:00
Patrick Cloke
3f58fc848d
Type hints and validation improvements. (#9321)
* Adds type hints to the groups servlet and stringutils code.
* Assert the maximum length of some input values for spec compliance.
2021-02-08 13:59:54 -05:00
David Teller
b0b2cac057
Merge pull request #9150 from Yoric/develop-context
New API /_synapse/admin/rooms/{roomId}/context/{eventId}
2021-02-08 15:53:44 +01:00
Richard van der Hoff
0d81a6fa3e Merge branch 'social_login' into develop 2021-01-28 22:08:11 +00:00
Erik Johnston
4b73488e81
Ratelimit 3PID /requestToken API (#9238) 2021-01-28 17:39:21 +00:00
David Teller
93f84e0373 FIXUP: Making get_event_context a bit more paranoid 2021-01-28 12:31:07 +01:00
Richard van der Hoff
a083aea396
Add 'brand' field to MSC2858 response (#9242)
We've decided to add a 'brand' field to help clients decide how to style the
buttons.

Also, fix up the allowed characters for idp_id, while I'm in the area.
2021-01-27 21:31:45 +00:00
Richard van der Hoff
a737cc2713
Implement MSC2858 support (#9183)
Fixes #8928.
2021-01-27 12:41:24 +00:00
Richard van der Hoff
0cd2938bc8
Support icons for Identity Providers (#9154) 2021-01-20 08:15:14 -05:00
Patrick Cloke
47d48a5853
Validate the server name for the /publicRooms endpoint. (#9161)
If a remote server name is provided, ensure it is something reasonable
before making remote connections to it.
2021-01-19 14:21:59 -05:00
Erik Johnston
6633a4015a
Allow moving account data and receipts streams off master (#9104) 2021-01-18 15:47:59 +00:00
Dirk Klimpel
7a2e9b549d
Remove user's avatar URL and displayname when deactivated. (#8932)
This only applies if the user's data is to be erased.
2021-01-12 16:30:15 -05:00
Richard van der Hoff
789d9ebad3
UI Auth via SSO: redirect the user to an appropriate SSO. (#9081)
If we have integrations with multiple identity providers, when the user does a UI Auth, we need to redirect them to the right one.

There are a few steps to this. First of all we actually need to store the userid of the user we are trying to validate in the UIA session, since the /auth/sso/fallback/web request is unauthenticated.

Then, once we get the /auth/sso/fallback/web request, we can fish the user id out of the session, and use it to look up the external id mappings, and hence pick an SSO provider for them.
2021-01-12 17:38:03 +00:00
Richard van der Hoff
0f8945e166
Kill off HomeServer.get_ip_from_request() (#9080)
Homeserver.get_ip_from_request() used to be a bit more complicated, but now it is totally redundant. Let's get rid of it.
2021-01-12 12:48:12 +00:00
Richard van der Hoff
8a910f97a4 Add some tests for the IDP picker flow 2021-01-07 14:56:42 +00:00
Richard van der Hoff
d2c616a413
Combine the SSO Redirect Servlets (#9015)
* Implement CasHandler.handle_redirect_request

... to make it match OidcHandler and SamlHandler

* Clean up interface for OidcHandler.handle_redirect_request

Make it accept `client_redirect_url=None`.

* Clean up interface for `SamlHandler.handle_redirect_request`

... bring it into line with CAS and OIDC by making it take a Request parameter,
move the magic for `client_redirect_url` for UIA into the handler, and fix the
return type to be a `str` rather than a `bytes`.

* Define a common protocol for SSO auth provider impls

* Give SsoIdentityProvider an ID and register them

* Combine the SSO Redirect servlets

Now that the SsoHandler knows about the identity providers, we can combine the
various *RedirectServlets into a single implementation which delegates to the
right IdP.

* changelog
2021-01-04 18:13:49 +00:00
Patrick Cloke
b7c580e333
Check if group IDs are valid before using them. (#8977) 2020-12-30 08:39:59 -05:00
Patrick Cloke
14a7371375
Validate input parameters for the sendToDevice API. (#8975)
This makes the "messages" key in the content required. This is currently
optional in the spec, but that seems to be an error.
2020-12-29 12:47:45 -05:00
Patrick Cloke
68bb26da69
Allow redacting events on workers (#8994)
Adds the redacts endpoint to workers that have the client listener.
2020-12-29 07:40:12 -05:00
Patrick Cloke
5d4c330ed9
Allow re-using a UI auth validation for a period of time (#8970) 2020-12-18 07:33:57 -05:00
Erik Johnston
a7a913918c Merge remote-tracking branch 'origin/erikj/as_mau_block' into develop 2020-12-18 09:51:56 +00:00
Erik Johnston
4c33796b20 Correctly handle AS registerations and add test 2020-12-17 12:55:21 +00:00
Patrick Cloke
bd30cfe86a
Convert internal pusher dicts to attrs classes. (#8940)
This improves type hinting and should use less memory.
2020-12-16 11:25:30 -05:00
Erik Johnston
a8eceb01e5
Honour AS ratelimit settings for /login requests (#8920)
Fixes #8846.
2020-12-11 16:33:31 +00:00
Richard van der Hoff
cf3b8156be
Fix errorcode for disabled registration (#8867)
The spec says we should return `M_FORBIDDEN` when someone tries to register and
registration is disabled.
2020-12-03 15:41:19 +00:00
Richard van der Hoff
4d9496559d
Support "identifier" dicts in UIA (#8848)
The spec requires synapse to support `identifier` dicts for `m.login.password`
user-interactive auth, which it did not (instead, it required an undocumented
`user` parameter.)

To fix this properly, we need to pull the code that interprets `identifier`
into `AuthHandler.validate_login` so that it can be called from the UIA code.

Fixes #5665.
2020-12-01 17:42:26 +00:00
Richard van der Hoff
59e18a1333
Simplify appservice login code (#8847)
we don't need to support legacy login dictionaries here.
2020-11-30 19:20:56 +00:00
Jonathan de Jong
ca60822b34
Simplify the way the HomeServer object caches its internal attributes. (#8565)
Changes `@cache_in_self` to use underscore-prefixed attributes.
2020-11-30 13:28:44 -05:00
Patrick Cloke
473dfec1e5
Use TYPE_CHECKING instead of magic MYPY variable. (#8770) 2020-11-17 09:09:40 -05:00
Erik Johnston
f737368a26
Add admin API for logging in as a user (#8617) 2020-11-17 10:51:25 +00:00
Erik Johnston
427ede619f
Add metrics for tracking 3PID /requestToken requests. (#8712)
The main use case is to see how many requests are being made, and how
many are second/third/etc attempts. If there are large number of retries
then that likely indicates a delivery problem.
2020-11-13 12:03:51 +00:00
Patrick Cloke
514a240aed
Remove unused OPTIONS handlers. (#8621)
The handling of OPTIONS requests was consolidated in #7534, but the endpoint
specific handlers were not removed.
2020-10-22 08:35:55 -04:00
Patrick Cloke
8f27b7fde1
Expose the experimental appservice login flow to clients. (#8504) 2020-10-19 13:03:55 -04:00
Aaron Raimist
ec606ea9e3
Add correct M_BAD_JSON error code to /profile/{userId}/displayname (#8517)
Fixes #8029
2020-10-14 11:24:58 +01:00
Patrick Cloke
c9c0ad5e20
Remove the deprecated Handlers object (#8494)
All handlers now available via get_*_handler() methods on the HomeServer.
2020-10-09 07:24:34 -04:00
Hubert Chathi
4cb44a1585
Add support for MSC2697: Dehydrated devices (#8380)
This allows a user to store an offline device on the server and
then restore it at a subsequent login.
2020-10-07 08:00:17 -04:00
Hubert Chathi
3cd78bbe9e
Add support for MSC2732: olm fallback keys (#8312) 2020-10-06 13:26:29 -04:00
Patrick Cloke
34ff8da83b
Convert additional templates to Jinja (#8444)
This converts a few more of our inline HTML templates to Jinja. This is somewhat part of #7280 and should make it a bit easier to customize these in the future.
2020-10-02 11:15:53 +01:00
Patrick Cloke
61aaf36a1c
Do not expose the experimental appservice login flow to clients. (#8440) 2020-10-01 13:38:20 -04:00
Erik Johnston
7941372ec8
Make token serializing/deserializing async (#8427)
The idea is that in future tokens will encode a mapping of instance to position. However, we don't want to include the full instance name in the string representation, so instead we'll have a mapping between instance name and an immutable integer ID in the DB that we can use instead. We'll then do the lookup when we serialize/deserialize the token (we could alternatively pass around an `Instance` type that includes both the name and ID, but that turns out to be a lot more invasive).
2020-09-30 20:29:19 +01:00
Patrick Cloke
8b40843392
Allow additional SSO properties to be passed to the client (#8413) 2020-09-30 13:02:43 -04:00
Andrew Morgan
f43c66d23b Merge branch 'develop' of github.com:matrix-org/synapse into anoa/info-mainline-no-check-password-reset 2020-09-29 14:21:41 +01:00
Andrew Morgan
1c6b8752b8
Only assert valid next_link params when provided (#8417)
Broken in https://github.com/matrix-org/synapse/pull/8275 and has yet to be put in a release. Fixes https://github.com/matrix-org/synapse/issues/8418.

`next_link` is an optional parameter. However, we were checking whether the `next_link` param was valid, even if it wasn't provided. In that case, `next_link` was `None`, which would clearly not be a valid URL.

This would prevent password reset and other operations if `next_link` was not provided, and the `next_link_domain_whitelist` config option was set.
2020-09-29 12:36:44 +01:00
Andrew Morgan
d4605d1f16 Don't check whether a 3pid is allowed to register during password reset
This endpoint should only deal with emails that have already been approved, and
are attached with user's account. There's no need to re-check them here.
2020-09-28 18:46:59 +01:00
Patrick Cloke
00db7786de Synapse 1.20.0rc5 (2020-09-18)
==============================
 
 In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.
 
 Features
 --------
 
 - Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343))
 
 Bugfixes
 --------
 
 - Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342))
 - Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349))
 
 Internal Changes
 ----------------
 
 - Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEEF3tZXk38tRDFVnUIM/xY9qcRMEgFAl9kzk8ACgkQM/xY9qcR
 MEim6A//aERkhyLGRlGpLd37lCyFQCeffTMH1rTvu04iIBQBaUZ6g7CYWOpK43zT
 U8kt379+5OShjdAXs/X4XP+ucdHVbrwsRSP3hBS/fFLiDT0fJgP8uiSf5QqO6NnT
 OqDyXYjcXvj/c6tMKglVtsdh8u4hFwNZjGPMGG68IzJu14uEhnD100cL9jSB9bLB
 ongWpsQzzdGBpJPSFRjv9dCUSeRbzyUdl1t0uqzrNqyN9s/JnzFTn7ZYo6y3lnSS
 dHGVMMo/12M2PkbBHnbJVvDY5Q/R7ZxyXlpz0gvSNOQIw8FqYFnuB0Niy5dQhXSR
 Sy5h4qbczLxqbql1x+lmzeQm4ZMORsW/Tl4C3z6yK6OYaOCJHIf9en4DplTSTqp1
 t+85JxWR2wH10d99YHBpaYKmkVovpwgchrO4YWrtXljUFAhhavzf+YiAdOHYT52s
 RDsDLsvjMbxEHsz4cHfycmshYhjzjb340wkoDXuQpj0zrO99d+Zd83xdK8pS0UQn
 OaljLRAd/5iBjTSyZPSrB1U5141OzlM3QZVJzaYAnP12yhR9eaX2twSCk+lPYOWd
 nhLJjNnj1B1XSGArthuE5NLyEiCPz6KyN2RhO0EOx5YjZN9TwH7LS9upyNFe1nN1
 GIhO5gz+jWLuBZE3xzRNjJyCx/I/LolpCwGMvKDu6638rpsbrPs=
 =tT5/
 -----END PGP SIGNATURE-----

Merge tag 'v1.20.0rc5' into develop

Synapse 1.20.0rc5 (2020-09-18)
==============================

In addition to the below, Synapse 1.20.0rc5 also includes the bug fix that was included in 1.19.3.

Features
--------

- Add flags to the `/versions` endpoint for whether new rooms default to using E2EE. ([\#8343](https://github.com/matrix-org/synapse/issues/8343))

Bugfixes
--------

- Fix rate limiting of federation `/send` requests. ([\#8342](https://github.com/matrix-org/synapse/issues/8342))
- Fix a longstanding bug where back pagination over federation could get stuck if it failed to handle a received event. ([\#8349](https://github.com/matrix-org/synapse/issues/8349))

Internal Changes
----------------

- Blacklist [MSC2753](https://github.com/matrix-org/matrix-doc/pull/2753) SyTests until it is implemented. ([\#8285](https://github.com/matrix-org/synapse/issues/8285))
2020-09-18 11:17:58 -04:00