Commit Graph

13510 Commits

Author SHA1 Message Date
Eric Eastwood
7afb615839
When redacting, keep event fields around that maintain the historical event structure intact (MSC2716) (#10538)
* Keep event fields that maintain the historical event structure intact

Fix https://github.com/matrix-org/synapse/issues/10521

* Add changelog

* Bump room version

* Better changelog text

* Fix up room version after develop merge
2021-08-09 20:23:31 -05:00
Dirk Klimpel
6b61debf5c
Do not remove status_msg when user going offline (#10550)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-09 16:21:04 +00:00
Brendan Abolivier
f8e86b7d2e
Merge branch 'release-v1.40' into develop 2021-08-09 14:44:28 +01:00
Brendan Abolivier
ad35b7739e 1.40.0rc3 2021-08-09 13:41:29 +01:00
Patrick Cloke
0c246dd4a0
Support MSC3289: Room version 8 (#10449)
This adds support for MSC3289: room version 8. This is room version 7 + MSC3083.
2021-08-09 10:46:39 +02:00
Patrick Cloke
1de26b3467
Convert Transaction and Edu object to attrs (#10542)
Instead of wrapping the JSON into an object, this creates concrete
instances for Transaction and Edu. This allows for improved type
hints and simplified code.
2021-08-06 09:39:59 -04:00
Erik Johnston
60f0534b6e
Fix exceptions in logs when failing to get remote room list (#10541) 2021-08-06 14:05:41 +01:00
Richard van der Hoff
1bebc0b78c
Clean up federation event auth code (#10539)
* drop old-room hack

pretty sure we don't need this any more.

* Remove incorrect comment about modifying `context`

It doesn't look like the supplied context is ever modified.

* Stop `_auth_and_persist_event` modifying its parameters

This is only called in three places. Two of them don't pass `auth_events`, and
the third doesn't use the dict after passing it in, so this should be non-functional.

* Stop `_check_event_auth` modifying its parameters

`_check_event_auth` is only called in three places. `on_send_membership_event`
doesn't pass an `auth_events`, and `prep` and `_auth_and_persist_event` do not
use the map after passing it in.

* Stop `_update_auth_events_and_context_for_auth` modifying its parameters

Return the updated auth event dict, rather than modifying the parameter.

This is only called from `_check_event_auth`.

* Improve documentation on `_auth_and_persist_event`

Rename `auth_events` parameter to better reflect what it contains.

* Improve documentation on `_NewEventInfo`

* Improve documentation on `_check_event_auth`

rename `auth_events` parameter to better describe what it contains

* changelog
2021-08-06 13:54:23 +01:00
Patrick Cloke
f4ade972ad
Update the API response for spaces summary over federation. (#10530)
This adds 'allowed_room_ids' (in addition to 'allowed_spaces', for backwards
compatibility) to the federation response of the spaces summary.

A future PR will remove the 'allowed_spaces' flag.
2021-08-06 07:40:29 -04:00
Richard van der Hoff
74d7336686
Add a setting to disable TLS for sending email (#10546)
This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.
2021-08-06 10:13:34 +00:00
Eric Eastwood
f5a368bb48
Mark all MSC2716 events as historical (#10537)
* Mark all MSC2716 events as historical
2021-08-05 20:35:53 -05:00
Patrick Cloke
3b354faad0
Refactoring before implementing the updated spaces summary. (#10527)
This should have no user-visible changes, but refactors some pieces of
the SpaceSummaryHandler before adding support for the updated
MSC2946.
2021-08-05 12:39:17 +00:00
Will Hunt
a8a27b2b8b
Only return an appservice protocol if it has a service providing it. (#10532)
If there are no services providing a protocol, omit it completely
instead of returning an empty dictionary.

This fixes a long-standing spec compliance bug.
2021-08-05 08:22:14 -04:00
Erik Johnston
a36d77c563 Merge tag 'v1.40.0rc2' into develop
Synapse 1.40.0rc2 (2021-08-04)
==============================

Bugfixes
--------

- Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\#10517](https://github.com/matrix-org/synapse/issues/10517))
- Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\#10531](https://github.com/matrix-org/synapse/issues/10531))

Internal Changes
----------------

- Fix release script to open the correct URL for the release. ([\#10516](https://github.com/matrix-org/synapse/issues/10516))
2021-08-05 11:15:29 +01:00
Michael Telatynski
9db24cc50d
Send unstable-prefixed room_type in store-invite IS API requests (#10435)
The room type is per MSC3288 to allow the identity-server to
change invitation wording based on whether the invitation is to
a room or a space.

The prefixed key will be replaced once MSC3288 is accepted
into the spec.
2021-08-04 13:39:57 -04:00
Eric Eastwood
684d19a11c
Add support for MSC2716 marker events (#10498)
* Make historical messages available to federated servers

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

Follow-up to https://github.com/matrix-org/synapse/pull/9247

* Debug message not available on federation

* Add base starting insertion point when no chunk ID is provided

* Fix messages from multiple senders in historical chunk

Follow-up to https://github.com/matrix-org/synapse/pull/9247

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.

* Remove debug lines

* Messing with selecting insertion event extremeties

* Move db schema change to new version

* Add more better comments

* Make a fake requester with just what we need

See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080

* Store insertion events in table

* Make base insertion event float off on its own

See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889

Conflicts:
	synapse/rest/client/v1/room.py

* Validate that the app service can actually control the given user

See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455

Conflicts:
	synapse/rest/client/v1/room.py

* Add some better comments on what we're trying to check for

* Continue debugging

* Share validation logic

* Add inserted historical messages to /backfill response

* Remove debug sql queries

* Some marker event implemntation trials

* Clean up PR

* Rename insertion_event_id to just event_id

* Add some better sql comments

* More accurate description

* Add changelog

* Make it clear what MSC the change is part of

* Add more detail on which insertion event came through

* Address review and improve sql queries

* Only use event_id as unique constraint

* Fix test case where insertion event is already in the normal DAG

* Remove debug changes

* Add support for MSC2716 marker events

* Process markers when we receive it over federation

* WIP: make hs2 backfill historical messages after marker event

* hs2 to better ask for insertion event extremity

But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group`
error

* Add insertion_event_extremities table

* Switch to chunk events so we can auth via power_levels

Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.

So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.

* Switch to chunk events for federation

* Add unstable room version to support new historical PL

* Messy: Fix undefined state_group for federated historical events

```
2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill
    await self.backfill(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill
    await self._auth_and_persist_event(dest, event, context, backfilled=True)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event
    await self._run_push_actions_and_persist_event(event, context, backfilled)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event
    await self.persist_events_and_notify(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify
    events, max_stream_token = await self.storage.persistence.persist_events(
  File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events
    ret_vals = await yieldable_gather_results(enqueue, partitioned.items())
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop
    ret = await self._per_item_callback(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch
    await self.persist_events_store._persist_events_and_state_updates(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise
    raise exception.with_traceback(traceback)
  File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn
    self._store_event_state_mappings_txn(txn, events_and_contexts)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn
    self.db_pool.simple_insert_many_txn(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn
    txn.execute_batch(sql, vals)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch
    self.executemany(sql, args)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany
    self._do_execute(self.txn.executemany, sql, *args)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute
    return func(sql, *args)
sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group
```

* Revert "Messy: Fix undefined state_group for federated historical events"

This reverts commit 187ab28611546321e02770944c86f30ee2bc742a.

* Fix federated events being rejected for no state_groups

Add fix from https://github.com/matrix-org/synapse/pull/10439
until it merges.

* Adapting to experimental room version

* Some log cleanup

* Add better comments around extremity fetching code and why

* Rename to be more accurate to what the function returns

* Add changelog

* Ignore rejected events

* Use simplified upsert

* Add Erik's explanation of extra event checks

See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332

* Clarify that the depth is not directly correlated to the backwards extremity that we return

See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404

* lock only matters for sqlite

See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061

* Move new SQL changes to its own delta file

* Clean up upsert docstring

* Bump database schema version (62)
2021-08-04 12:07:57 -05:00
Brendan Abolivier
02c2f631ae 1.40.0rc2 2021-08-04 17:09:27 +01:00
Brendan Abolivier
e8a3e81402
Don't fail on empty bodies when sending out read receipts (#10531)
Fixes a bug introduced in rc1 that would cause Synapse to 400 on read receipts requests with empty bodies.

Broken in #10413
2021-08-04 16:13:24 +02:00
Erik Johnston
c37dad67ab
Improve event caching code (#10119)
Ensure we only load an event from the DB once when the same event is requested multiple times at once.
2021-08-04 13:54:51 +01:00
Erik Johnston
11540be55e
Fix could not serialize access errors for claim_e2e_one_time_keys (#10504) 2021-08-04 13:09:04 +01:00
Jason Robinson
c2000ab35b
Add get_userinfo_by_id method to ModuleApi (#9581)
Makes it easier to fetch user details in for example spam checker modules, without needing to use api._store or figure out database interactions.

Signed-off-by: Jason Robinson <jasonr@matrix.org>
2021-08-04 10:40:25 +00:00
Kento Okamoto
72935b7c50
Add warnings to ip_range_blacklist usage with proxies (#10129)
Per issue #9812 using `url_preview_ip_range_blacklist` with a proxy via `HTTPS_PROXY` or `HTTP_PROXY` environment variables has some inconsistent bahavior than mentioned. This PR changes the following:

- Changes the Sample Config file to include a note mentioning that `url_preview_ip_range_blacklist` and `ip_range_blacklist` is ignored when using a proxy
- Changes some logic in synapse/config/repository.py to send a warning when both `*ip_range_blacklist` configs and a proxy environment variable are set and but no longer throws an error.

Signed-off-by: Kento Okamoto <kentokamoto@protonmail.com>
2021-08-03 18:13:34 +00:00
Richard van der Hoff
4b10880da3
Make sync response cache time configurable. (#10513) 2021-08-03 14:45:04 +01:00
Dagfinn Ilmari Mannsåker
dc46f12725
Include room ID in ignored EDU log messages (#10507)
Signed-off-by: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
2021-08-03 13:35:49 +00:00
reivilibre
903db99ed5
Fix PeriodicallyFlushingMemoryHandler inhibiting application shutdown (#10517) 2021-08-03 14:28:30 +01:00
Erik Johnston
c8566191fc 1.40.0rc1 2021-08-03 11:32:10 +01:00
reivilibre
f4ac934afe
Revert use of PeriodicallyFlushingMemoryHandler by default (#10515) 2021-08-03 11:30:39 +01:00
Erik Johnston
01d45fe964
Prune inbound federation queues if they get too long (#10390) 2021-08-02 13:37:25 +00:00
Toni Spets
ba5287f5e8
Allow setting transaction limit for db connections (#10440)
Setting the value will help PostgreSQL free up memory by recycling
the connections in the connection pool.

Signed-off-by: Toni Spets <toni.spets@iki.fi>
2021-08-02 13:24:43 +00:00
reivilibre
c167e09fe5
Fix explicit assignment of PL 0 from being misinterpreted in rare circumstances (#10499) 2021-07-30 12:34:21 +01:00
V02460
b7f7ca24b1
Remove shebang line from module files (#10415)
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
2021-07-29 21:34:14 +01:00
Erik Johnston
65f520697d Merge remote-tracking branch 'origin/master' into develop 2021-07-29 16:29:17 +01:00
Patrick Cloke
3a541a7daa
Improve failover logic for MSC3083 restricted rooms. (#10447)
If the federation client receives an M_UNABLE_TO_AUTHORISE_JOIN or
M_UNABLE_TO_GRANT_JOIN response it will attempt another server
before giving up completely.
2021-07-29 11:50:14 +00:00
Erik Johnston
5522a103a9 1.39.0 2021-07-29 09:59:07 +01:00
Richard van der Hoff
858363d0b7
Generics for ObservableDeferred (#10491)
Now that `Deferred` is a generic class, let's update `ObeservableDeferred` to
follow suit.
2021-07-28 19:55:50 +00:00
Eric Eastwood
d0b294ad97
Make historical events discoverable from backfill for servers without any scrollback history (MSC2716) (#10245)
* Make historical messages available to federated servers

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

Follow-up to https://github.com/matrix-org/synapse/pull/9247

* Debug message not available on federation

* Add base starting insertion point when no chunk ID is provided

* Fix messages from multiple senders in historical chunk

Follow-up to https://github.com/matrix-org/synapse/pull/9247

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.

* Remove debug lines

* Messing with selecting insertion event extremeties

* Move db schema change to new version

* Add more better comments

* Make a fake requester with just what we need

See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080

* Store insertion events in table

* Make base insertion event float off on its own

See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889

Conflicts:
	synapse/rest/client/v1/room.py

* Validate that the app service can actually control the given user

See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455

Conflicts:
	synapse/rest/client/v1/room.py

* Add some better comments on what we're trying to check for

* Continue debugging

* Share validation logic

* Add inserted historical messages to /backfill response

* Remove debug sql queries

* Some marker event implemntation trials

* Clean up PR

* Rename insertion_event_id to just event_id

* Add some better sql comments

* More accurate description

* Add changelog

* Make it clear what MSC the change is part of

* Add more detail on which insertion event came through

* Address review and improve sql queries

* Only use event_id as unique constraint

* Fix test case where insertion event is already in the normal DAG

* Remove debug changes

* Switch to chunk events so we can auth via power_levels

Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.

So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.

* Switch to chunk events for federation

* Add unstable room version to support new historical PL

* Fix federated events being rejected for no state_groups

Add fix from https://github.com/matrix-org/synapse/pull/10439
until it merges.

* Only connect base insertion event to prev_event_ids

Per discussion with @erikjohnston,
https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$12bTUiObDFdHLAYtT7E-BvYRp3k_xv8w0dUQHibasJk?via=jki.re&via=matrix.org

* Make it possible to get the room_version with txn

* Allow but ignore historical events in unsupported room version

See https://github.com/matrix-org/synapse/pull/10245#discussion_r675592489

We can't reject historical events on unsupported room versions because homeservers without knowledge of MSC2716 or the new room version don't reject historical events either.

Since we can't rely on the auth check here to stop historical events on unsupported room versions, I've added some additional checks in the processing/persisting code (`synapse/storage/databases/main/events.py` ->  `_handle_insertion_event` and `_handle_chunk_event`). I've had to do some refactoring so there is method to fetch the room version by `txn`.

* Move to unique index syntax

See https://github.com/matrix-org/synapse/pull/10245#discussion_r675638509

* High-level document how the insertion->chunk lookup works

* Remove create_event fallback for room_versions

See https://github.com/matrix-org/synapse/pull/10245/files#r677641879

* Use updated method name
2021-07-28 10:46:37 -05:00
Erik Johnston
8c201c97ec Synapse 1.39.0rc3 (2021-07-28)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477))
 - Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485))
 - Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486))
 
 Internal Changes
 ----------------
 
 - Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461))
 - Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmEBUJUQHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCWi6CACfA3B7rXfQIO893mctSZkNhtAG/e4o310M
 etvjAtovKmKeFqBvm89FPmFoXvGA2ypoz7Jz2fdkP56DNKIXvihnDc8EeQ3gGawf
 hQ8GBjICOL1e7R/4qzuOCWYdppBGoGNAbz7qyxAUxZ/XOzsPJ2VBGkgyXWZkyPe/
 nJYsuMuMp117HUwTqPqs3oZuTN0MRTy6xgaDfbFbcX9UC/amLmGIWOhlme8iV15r
 HQAW0X7F2Un5h7eQwogWTDb9gBHKnJ4ApaSJLfZl9E72i0Sv7o9UFcLIdy2DRVSq
 gxlRBVzVU6ZndCQ4vQXFSjQ20VGNXc2vpKVRE1D6YXxmxVNbBQB4
 =lb6H
 -----END PGP SIGNATURE-----

Merge tag 'v1.39.0rc3' into develop

Synapse 1.39.0rc3 (2021-07-28)
==============================

Bugfixes
--------

- Fix a bug introduced in Synapse 1.38 which caused an exception at startup when SAML authentication was enabled. ([\#10477](https://github.com/matrix-org/synapse/issues/10477))
- Fix a long-standing bug where Synapse would not inform clients that a device had exhausted its one-time-key pool, potentially causing problems decrypting events. ([\#10485](https://github.com/matrix-org/synapse/issues/10485))
- Fix reporting old R30 stats as R30v2 stats. Introduced in v1.39.0rc1. ([\#10486](https://github.com/matrix-org/synapse/issues/10486))

Internal Changes
----------------

- Fix an error which prevented the Github Actions workflow to build the docker images from running. ([\#10461](https://github.com/matrix-org/synapse/issues/10461))
- Fix release script to correctly version debian changelog when doing RCs. ([\#10465](https://github.com/matrix-org/synapse/issues/10465))
2021-07-28 15:53:53 +01:00
Erik Johnston
5146e19880 1.39.0rc3 2021-07-28 13:31:18 +01:00
Richard van der Hoff
d9cb658c78
Fix up type hints for Twisted 21.7 (#10490)
Mostly this involves decorating a few Deferred declarations with extra type hints. We wrap the types in quotes to avoid runtime errors when running against older versions of Twisted that don't have generics on Deferred.
2021-07-28 12:04:11 +00:00
Richard van der Hoff
9643dfde6a
improve typing annotations in CachedCall (#10450)
tighten up some of the typing in CachedCall, which is going to be needed when
Twisted 21.7 brings better typing on Deferred.
2021-07-28 12:25:12 +01:00
Patrick Cloke
752fe0cd98
Restricted rooms (MSC3083) should not have their allow key redacted. (#10489) 2021-07-28 07:03:01 -04:00
Šimon Brandner
c3b037795a
Support for MSC2285 (hidden read receipts) (#10413)
Implementation of matrix-org/matrix-doc#2285
2021-07-28 10:05:11 +02:00
sri-vidyut
8e1febc6a1
Support underscores (in addition to hyphens) for charset detection. (#10410) 2021-07-27 17:29:42 +00:00
Erik Johnston
5b22d5ee03
Fix oldest_pdu_in_federation_staging (#10455)
If the staging area was empty we'd report an age of 51 years, which is
not true or helpful.
2021-07-27 18:01:04 +01:00
Dirk Klimpel
076deade02
allow specifying https:// proxy (#10411) 2021-07-27 17:31:06 +01:00
Jason Robinson
31c6b30dd4
Fix import of the default SAML mapping provider. (#10477)
Fix a circular import, which was causing exceptions on boot if SAML
was configured.
2021-07-27 11:34:15 -04:00
reivilibre
10dcfae46f
Fix typo that causes R30v2 to actually be old R30 (#10486)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-27 15:25:39 +01:00
Andrew Morgan
74d09a43d9
Always communicate device OTK counts to clients (#10485)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-07-27 14:36:38 +01:00
reivilibre
e16eab29d6
Add a PeriodicallyFlushingMemoryHandler to prevent logging silence (#10407)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-27 14:32:05 +01:00
Denis Kasak
2476d5373c
Mitigate media repo XSSs on IE11. (#10468)
IE11 doesn't support Content-Security-Policy but it has support for
a non-standard X-Content-Security-Policy header, which only supports the
sandbox directive. This prevents script execution, so it at least offers
some protection against media repo-based attacks.

Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-07-27 13:45:10 +02:00
Travis Ralston
b3a757eb3b
Support MSC2033: Device ID on whoami (#9918)
* Fix no-access-token bug in deactivation tests
* Support MSC2033: Device ID on whoami
* Test for appservices too

MSC: https://github.com/matrix-org/matrix-doc/pull/2033

The MSC has passed FCP, which means stable endpoints can be used.
2021-07-27 05:28:20 +00:00
Patrick Cloke
b7186c6e8d
Add type hints to state handler. (#10482) 2021-07-26 12:49:53 -04:00
Patrick Cloke
228decfce1
Update the MSC3083 support to verify if joins are from an authorized server. (#10254) 2021-07-26 12:17:00 -04:00
Patrick Cloke
4fb92d93ea
Add type hints to synapse.federation.transport.client. (#10408) 2021-07-26 11:53:09 -04:00
Erik Johnston
c39a417de0 Synapse 1.39.0rc2 (2021-07-22)
==============================
 
 Bugfixes
 --------
 
 - Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457))
 
 Internal Changes
 ----------------
 
 - Move docker image build to Github Actions. ([\#10416](https://github.com/matrix-org/synapse/issues/10416))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmD5hF0QHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCRR8B/4oto6w4GVN/HVl9hsYCv/ZR/JvUb/yefVs
 mqUz6CbSB6X+jkkz5tORovz6lagQZUXET7xFnNuw/R63I1E4g5uAgfIpIbD4m79U
 LS+aq0YzbnLSzZ+p8ISWURT+QkSqmpAtmsqT3S3TYNYSvAYKtjlbYB8SI/m+um/Q
 7cgU6WCW2uMFKtBYu+anPzv3z0Bip4Xf8xsrLWQ/Rlue/xyW/KOuCu1TeoSU18bY
 v6MOYwMChDG4dbxHmDs981b6G8TN9L+CysQbkaj31MftxCNJSILgWRYc2BgZIoQk
 SMVcd+v+fcaI4zTC3AKTnqqBoKEZWpxqS438a8E1vkO92sXcxk9u
 =7Jr7
 -----END PGP SIGNATURE-----

Merge tag 'v1.39.0rc2' into develop

Synapse 1.39.0rc2 (2021-07-22)
==============================

Bugfixes
--------

- Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457))

Internal Changes
----------------

- Move docker image build to Github Actions. ([\#10416](https://github.com/matrix-org/synapse/issues/10416))
2021-07-23 09:04:41 +01:00
Erik Johnston
683deee9a4 Merge branch 'master' into develop 2021-07-23 09:03:19 +01:00
Erik Johnston
f76f8c1567 1.39.0rc2 2021-07-22 15:43:26 +01:00
Erik Johnston
4565063e36 Merge commit '7da24b975dfb10c277cf963dfddb88f55b1ca598' into release-v1.39 2021-07-22 15:42:45 +01:00
Erik Johnston
283bb5c94e 1.38.1 2021-07-22 15:37:10 +01:00
David Baker
7da24b975d
Always send device_one_time_keys_count (#10457)
As per comment

Fixes https://github.com/matrix-org/synapse/issues/10456
See also https://github.com/vector-im/element-android/issues/3725
2021-07-22 15:29:27 +01:00
Dirk Klimpel
89c4ca81bb
Add creation_ts to list users admin API (#10448)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-07-22 16:05:16 +02:00
Erik Johnston
38b346a504
Replace or_ignore in simple_insert with simple_upsert (#10442)
Now that we have `simple_upsert` that should be used in preference to
trying to insert and looking for an exception. The main benefit is that
we ERROR message don't get written to postgres logs.

We also have tidy up the return value on `simple_upsert`, rather than
having a tri-state of inserted/not-inserted/unknown.
2021-07-22 12:39:50 +01:00
Richard van der Hoff
d8324b8238
Fix a handful of type annotations. (#10446)
* switch from `types.CoroutineType` to `typing.Coroutine`

these should be identical semantically, and since `defer.ensureDeferred` is
defined to take a `typing.Coroutine`, will keep mypy happy

* Fix some annotations on inlineCallbacks functions

* changelog
2021-07-22 12:00:16 +01:00
Patrick Cloke
590cc4e888
Add type hints to additional servlet functions (#10437)
Improves type hints for:

* parse_{boolean,integer}
* parse_{boolean,integer}_from_args
* parse_json_{value,object}_from_request

And fixes any incorrect calls that resulted from unknown types.
2021-07-21 18:12:22 +00:00
Patrick Cloke
d15e72e511
Update the notification email subject when invited to a space. (#10426) 2021-07-21 17:29:54 +00:00
Patrick Cloke
5db118626b
Add a return type to parse_string. (#10438)
And set the required attribute in a few places which will error if
a parameter is not provided.
2021-07-21 09:47:56 -04:00
Eric Eastwood
2d89c66b88
Switch to chunk events so we can auth via power_levels (MSC2716) (#10432)
Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.

So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.
2021-07-21 10:29:57 +00:00
Andrew Morgan
b181dc402d Synapse 1.39.0rc1 (2021-07-20)
==============================
 
 The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information.
 
 Features
 --------
 
 - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884))
 - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305))
 - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427))
 - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360))
 
 Bugfixes
 --------
 
 - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433))
 - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250))
 - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276))
 - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317))
 - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343))
 - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344))
 - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355))
 - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359))
 - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367))
 - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414))
 - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404))
 
 Improved Documentation
 ----------------------
 
 - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971))
 - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313))
 - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337))
 - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368))
 - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399))
 - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395))
 - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396))
 
 Deprecations and Removals
 -------------------------
 
 - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721))
 - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386))
 
 Internal Changes
 ----------------
 
 - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289))
 - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315))
 - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316))
 - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324))
 - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357))
 - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348))
 - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349))
 - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418))
 - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383))
 - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385))
 - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391))
 - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393))
 - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398))
 - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421))
 - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCAAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAmD20AQTHGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9Ah9EACjDbd67+fYT4CIBp1/gwRZm73qAinF
 HtJQV3b4KjvF+KC3NTxyfuSo1r6UHSwVP7hYMcW4D7bueYaFVklG4Pb0L58pRdrr
 nKpVmCTuzQ7KAP165Mea2ZJUihTGrghUV1NTtkgbL2UtBn/j0ymUzWC4dDu2I3V9
 vSoa0QOwaBb/iE7RCfwkI/FWNdhJis5CRtzZg31OWqs7kf1FU6+iH7XEtyckkV8M
 U5aryWPlVzt1c9ag7eLf4OeySyVjl9MqKDjrUd6utLM3bK7DkHSZV/L9wA/NDfwA
 O6jwnSCsp5TuPjpBFvIOId5P7Ank0WFJTHZj2yzXzrP86gVmmzJi5tTbVDVAaOrw
 Ept1zKZjNUNBrgB5nu5oSOZQXPrWfSQeQgOfNPkS6SN2fPEvV+koIuv2CBgttmmB
 9/iFr0bairlU1LllvyO2Q9VdFb1V2dTFW5MKVW5VdzieeeXDBZYisHT9nr62CRkD
 ne52Qf84IaLJhwbyFaTssBX7KrmMTfasPzSaAQiLe7i6xlzC6ysdHajrdWf/Vn5c
 z4I22vaBjZ8FqPucfvC/AiAm5at5bXkVu0UTxvfqeHw6ZXan+ApBYwUVHg50Rx96
 J0FkZNfDPZlFTOJpf+jiO/kx4my2pMuiTseoi3wBr21u2j07vRoeeLiGHMuICqlO
 1h0ekjbYsXrvdQ==
 =WSa3
 -----END PGP SIGNATURE-----

Merge tag 'v1.39.0rc1' into develop

Synapse 1.39.0rc1 (2021-07-20)
==============================

The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information.

Features
--------

- Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884))
- The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305))
- Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427))
- Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360))

Bugfixes
--------

- Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433))
- Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250))
- Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276))
- Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317))
- Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343))
- Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344))
- Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355))
- Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359))
- Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367))
- Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414))
- Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404))

Improved Documentation
----------------------

- Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971))
- Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313))
- Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337))
- Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368))
- Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399))
- Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395))
- Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396))

Deprecations and Removals
-------------------------

- Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721))
- The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386))

Internal Changes
----------------

- Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289))
- Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315))
- Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316))
- Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324))
- Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357))
- Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348))
- Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349))
- Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418))
- Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383))
- Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385))
- When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391))
- Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393))
- Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398))
- Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421))
- Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))
2021-07-20 16:47:44 +01:00
Erik Johnston
54389d5697
Fix dropping locks on shut down (#10433) 2021-07-20 14:24:25 +01:00
Michael Telatynski
69226c1ab4
MSC3244 room capabilities implementation (#10283) 2021-07-20 12:59:23 +01:00
Andrew Morgan
12623cf38c 1.39.0rc1 2021-07-20 12:31:51 +01:00
Erik Johnston
794371b1bf Revert "Fix dropping locks on shut down"
This reverts commit 83f1ccfcab.
2021-07-20 12:28:40 +01:00
Erik Johnston
83f1ccfcab Fix dropping locks on shut down 2021-07-20 12:28:00 +01:00
Brendan Abolivier
a743bf4694
Port the ThirdPartyEventRules module interface to the new generic interface (#10386)
Port the third-party event rules interface to the generic module interface introduced in v1.37.0
2021-07-20 12:39:46 +02:00
Erik Johnston
f3ac9c6750
Fix exception when failing to get remote room list (#10414) 2021-07-20 11:35:23 +01:00
reivilibre
eebfd024e9
Factorise get_datastore calls in phone_stats_home. (#10427)
Follow-up to #10332.
2021-07-19 19:31:17 +01:00
reivilibre
4e340412c0
Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric (#10332)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-19 16:11:34 +01:00
Jonathan de Jong
95e47b2e78
[pyupgrade] synapse/ (#10348)
This PR is tantamount to running 
```
pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"`
```

Part of #9744
2021-07-19 15:28:05 +01:00
Eric Eastwood
7387d6f624
Remove unused events_by_room (#10421)
It looks like it was first used and introduced in 5130d80d79 (diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8R305) but the 

But the usage was removed in 4c6a31cd6e (diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8)
2021-07-19 10:16:46 +01:00
Jonathan de Jong
323452944e
One last inline type hint (for the whole repo) (#10418) 2021-07-16 20:12:56 +01:00
Jonathan de Jong
98aec1cc9d
Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
Brendan Abolivier
36dc15412d
Add a module type for account validity (#9884)
This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity.

To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
2021-07-16 18:11:53 +02:00
Patrick Cloke
d427f64724
Do not include signatures/hashes in make_{join,leave,knock} responses. (#10404)
These signatures would end up invalid since the joining/leaving/knocking
server would modify the response before calling send_{join,leave,knock}.
2021-07-16 10:36:38 -04:00
Jonathan de Jong
bdfde6dca1
Use inline type hints in http/federation/, storage/ and util/ (#10381) 2021-07-15 12:46:54 -04:00
Erik Johnston
3acf85c85f
Reduce likelihood of Postgres table scanning state_groups_state. (#10359)
The postgres statistics collector sometimes massively underestimates the
number of distinct state groups are in the `state_groups_state`, which
can cause postgres to use table scans for queries for multiple state
groups.

We fix this by manually setting `n_distinct` on the column.
2021-07-15 16:02:12 +01:00
Erik Johnston
ac5c221208
Stagger send presence to remotes (#10398)
This is to help with performance, where trying to connect to thousands
of hosts at once can consume a lot of CPU (due to TLS etc).

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-07-15 11:52:56 +01:00
Jonathan de Jong
bf72d10dbf
Use inline type hints in various other places (in synapse/) (#10380) 2021-07-15 11:02:43 +01:00
Dirk Klimpel
c7603af1d0
Allow providing credentials to http_proxy (#10360) 2021-07-15 10:37:08 +01:00
Erik Johnston
7695ca0618
Fix a number of logged errors caused by remote servers being down. (#10400) 2021-07-15 10:35:46 +01:00
Patrick Cloke
eb3beb8f12
Add type hints and comments to event auth code. (#10393) 2021-07-14 14:13:40 +01:00
Eric Eastwood
0d5b08ac7a
Fix messages from multiple senders in historical chunk (MSC2716) (#10276)
Fix messages from multiple senders in historical chunk. This also means that an app service does not need to define `?user_id` when using this endpoint.

Follow-up to https://github.com/matrix-org/synapse/pull/9247

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
2021-07-13 14:12:33 -05:00
Patrick Cloke
30b56f6925
Add type hints to get_domain_from_id and get_localpart_from_id. (#10385) 2021-07-13 12:08:47 -04:00
Patrick Cloke
2d16e69b4b
Show all joinable rooms in the spaces summary. (#10298)
Previously only world-readable rooms were shown. This means that
rooms which are public, knockable, or invite-only with a pending invitation,
are included in a space summary. It also applies the same logic to
the experimental room version from MSC3083 -- if a user has access
to the proper allowed rooms then it is shown in the spaces summary.

This change is made per MSC3173 allowing stripped state of a room to
be shown to any potential room joiner.
2021-07-13 08:59:27 -04:00
Richard van der Hoff
475fcb0f20 Merge branch 'master' into develop 2021-07-13 13:30:16 +01:00
Richard van der Hoff
c647c2a9ac 1.38.0 2021-07-13 13:19:06 +01:00
Richard van der Hoff
f7bfa694ae 1.38.0rc3 2021-07-13 11:57:55 +01:00
Dirk Klimpel
d9b3637e44
Bugfix make_room_admin fails for users that have left a private room (#10367)
Fixes: #10338
2021-07-13 11:53:45 +01:00
Erik Johnston
879d8c1ee1
Fix federation inbound age metric. (#10355)
We should be reporting the age rather than absolute timestamp.
2021-07-13 11:33:15 +01:00
Richard van der Hoff
c2c364f27f
Replace room_depth.min_depth with a BIGINT (#10289)
while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth
with a BIGINT.
2021-07-12 17:22:54 +01:00
reivilibre
ca9dface8c
Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. (#10344)
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2021-07-09 14:12:47 +01:00