Commit Graph

13494 Commits

Author SHA1 Message Date
Patrick Cloke
d9856d9150
Fix weakref_slot parameter for room member storage attrs. (#10642)
Follow-up to #10629 which set it to true, not false.
2021-08-18 15:00:37 +00:00
Patrick Cloke
bec01c0758
Convert room member storage tuples to attrs. (#10629)
Instead of using namedtuples. This helps with asserting type hints
and code completion.
2021-08-18 09:22:07 -04:00
Callum Brown
6e613a10d0
Display an error page during failure of fallback UIA. (#10561) 2021-08-18 08:13:35 -04:00
Richard van der Hoff
964f29cb6f
Refactor on_receive_pdu code (#10615)
* drop room pdu linearizer sooner

No point holding onto it while we recheck the db

* move out `missing_prevs` calculation

we're going to need `missing_prevs` whatever we do, so we may as well calculate
it eagerly and just update it if it gets outdated.

* Add another `if missing_prevs` condition

this should be a no-op, since all the code inside the block already checks `if
missing_prevs`

* reorder if conditions

This shouldn't change the logic at all.

* Push down `min_depth` read

No point reading it from the database unless we're going to use it.

* Collect the sent_to_us_directly code together

Move the remaining `sent_to_us_directly` code inside the `if
sent_to_us_directly` block.

* Properly separate the `not sent_to_us_directly` branch

Since the only way this second block is now reachable is if we
*didn't* go into the `sent_to_us_directly` branch, we can replace it with a
simple `else`.

* changelog
2021-08-18 12:36:22 +01:00
Patrick Cloke
6a5f8fbcda
Use auto-attribs for attrs classes for sync. (#10630) 2021-08-18 07:27:32 -04:00
Dirk Klimpel
430241a1e9
Remove deprecated Shutdown Room and Purge Room Admin API (#8830) 2021-08-17 21:19:13 +01:00
Azrenbeth
1a9f531c79
Port the PresenceRouter module interface to the new generic interface (#10524)
Port the PresenceRouter module interface to the new generic interface introduced in v1.37.0
2021-08-17 13:22:45 +00:00
Richard van der Hoff
272b89d547
Stop setting the outlier flag for things that aren't (#10614)
Marking things as outliers to inhibit pushes is a sledgehammer to crack a
nut. Move the test further down the stack so that we just inhibit the thing we
want.
2021-08-17 13:13:42 +01:00
reivilibre
642a42edde
Flatten the synapse.rest.client package (#10600) 2021-08-17 11:57:58 +00:00
Andrew Morgan
b62eba7705
Always list fallback key types in /sync (#10623) 2021-08-17 12:32:25 +01:00
Dirk Klimpel
3bcd525b46
Allow to edit external_ids by Edit User admin API (#10598)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-17 11:56:11 +01:00
Richard van der Hoff
58f0d97275
update links to schema doc (#10620) 2021-08-17 10:45:35 +00:00
Brendan Abolivier
ae2714c1f3
Allow using several custom template directories (#10587)
Allow using several directories in read_templates.
2021-08-17 10:23:14 +00:00
Will Hunt
a933c2c7d8
Add an admin API to check if a username is available (#10578)
This adds a new API GET /_synapse/admin/v1/username_available?username=foo to check if a username is available. It is the counterpart to https://matrix.org/docs/spec/client_server/r0.6.0#get-matrix-client-r0-register-available, except that it works even if registration is disabled.
2021-08-17 10:52:38 +01:00
reivilibre
19e51b14d2
Manhole: wrap coroutines in defer.ensureDeferred automatically (#10602) 2021-08-16 18:11:48 +01:00
Patrick Cloke
5af83efe8d
Validate the max_rooms_per_space parameter to ensure it is non-negative. (#10611) 2021-08-16 12:01:30 -04:00
Michael Telatynski
0ace38b7b3
Experimental support for MSC3266 Room Summary API. (#10394) 2021-08-16 14:49:12 +00:00
Patrick Cloke
87b62f8bb2
Split synapse.federation.transport.server into multiple files. (#10590) 2021-08-16 10:14:31 -04:00
Richard van der Hoff
2d9ca4ca77
Clean up some logging in the federation event handler (#10591)
* Include outlier status in `str(event)`

In places where we log event objects, knowing whether or not you're dealing
with an outlier is super useful.

* Remove duplicated logging in get_missing_events

When we process events received from get_missing_events, we log them twice
(once in `_get_missing_events_for_pdu`, and once in `on_receive_pdu`). Reduce
the duplication by removing the logging in `on_receive_pdu`, and ensuring the
call sites do sensible logging.

* log in `on_receive_pdu` when we already have the event

* Log which prev_events we are missing

* changelog
2021-08-16 13:19:02 +01:00
Patrick Cloke
7de445161f
Support federation in the new spaces summary API (MSC2946). (#10569) 2021-08-16 08:06:17 -04:00
Šimon Brandner
a3a7514570
Handle string read receipt data (#10606)
* Handle string read receipt data

Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>

* Test that we handle string read receipt data

Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>

* Add changelog for #10606

Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>

* Add docs

Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>

* Ignore malformed RRs

Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>

* Only surround hidden = ...

Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>

* Remove unnecessary argument

Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>

* Update changelog.d/10606.bugfix

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-08-16 12:22:38 +01:00
Eric Eastwood
c8d54be44c
Move /batch_send to /v2_alpha directory (MSC2716) (#10576)
* Move /batch_send to /v2_alpha directory

As pointed out by @erikjohnston,
https://github.com/matrix-org/synapse/pull/10552#discussion_r685836624
2021-08-13 14:37:24 -05:00
Patrick Cloke
c12b5577f2
Fix a harmless exception when the staged events queue is empty. (#10592) 2021-08-13 11:49:06 +00:00
Patrick Cloke
98a3355d9a
Update the pagination parameter name based on MSC2946 review. (#10579) 2021-08-11 15:44:45 -04:00
Dirk Klimpel
915b37e5ef
Admin API to delete media for a specific user (#10558) 2021-08-11 19:29:59 +00:00
Patrick Cloke
3ebb6694f0
Allow requesting the summary of a space which is joinable. (#10580)
As opposed to only allowing the summary of spaces which the user is
already in or has world-readable visibility.

This makes the logic consistent with whether a space/room is returned
as part of a space and whether a space summary can start at a space.
2021-08-11 15:04:51 -04:00
Patrick Cloke
5acd8b5a96
Expire old spaces summary pagination sessions. (#10574) 2021-08-11 18:52:09 +00:00
Patrick Cloke
2ae2a04616
Clarify error message when joining a restricted room. (#10572) 2021-08-11 14:31:39 -04:00
Dirk Klimpel
339c3918e1
support federation queries through http connect proxy (#10475)
Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-11 15:34:59 +01:00
Eric Eastwood
8c654b7309
Only return state events that the AS passed in via state_events_at_start (MSC2716) (#10552)
* Only return state events that the AS passed in via state_events_at_start

As discovered by @Half-Shot in
https://github.com/matrix-org/matrix-doc/pull/2716#discussion_r684158448

Part of MSC2716

* Add changelog

* Fix changelog extension
2021-08-10 18:10:40 -05:00
Patrick Cloke
fe1d0c8618
Add local support for the new spaces summary endpoint (MSC2946) (#10549)
This adds support for the /hierarchy endpoint, which is an
update to MSC2946. Currently this only supports rooms known
locally to the homeserver.
2021-08-10 13:08:17 -04:00
Patrick Cloke
691593bf71
Fix an edge-case with invited rooms over federation in the spaces summary. (#10560)
If a room which the requesting user was invited to was queried over
federation it will now properly appear in the spaces summary (instead
of being stripped out by the requesting server).
2021-08-10 14:56:54 +00:00
Brendan Abolivier
b5de77cf86 Merge branch 'master' into develop 2021-08-10 14:23:57 +01:00
Brendan Abolivier
9f7c038272 1.40.0 2021-08-10 13:50:58 +01:00
Eric Eastwood
7afb615839
When redacting, keep event fields around that maintain the historical event structure intact (MSC2716) (#10538)
* Keep event fields that maintain the historical event structure intact

Fix https://github.com/matrix-org/synapse/issues/10521

* Add changelog

* Bump room version

* Better changelog text

* Fix up room version after develop merge
2021-08-09 20:23:31 -05:00
Dirk Klimpel
6b61debf5c
Do not remove status_msg when user going offline (#10550)
Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-08-09 16:21:04 +00:00
Brendan Abolivier
f8e86b7d2e
Merge branch 'release-v1.40' into develop 2021-08-09 14:44:28 +01:00
Brendan Abolivier
ad35b7739e 1.40.0rc3 2021-08-09 13:41:29 +01:00
Patrick Cloke
0c246dd4a0
Support MSC3289: Room version 8 (#10449)
This adds support for MSC3289: room version 8. This is room version 7 + MSC3083.
2021-08-09 10:46:39 +02:00
Patrick Cloke
1de26b3467
Convert Transaction and Edu object to attrs (#10542)
Instead of wrapping the JSON into an object, this creates concrete
instances for Transaction and Edu. This allows for improved type
hints and simplified code.
2021-08-06 09:39:59 -04:00
Erik Johnston
60f0534b6e
Fix exceptions in logs when failing to get remote room list (#10541) 2021-08-06 14:05:41 +01:00
Richard van der Hoff
1bebc0b78c
Clean up federation event auth code (#10539)
* drop old-room hack

pretty sure we don't need this any more.

* Remove incorrect comment about modifying `context`

It doesn't look like the supplied context is ever modified.

* Stop `_auth_and_persist_event` modifying its parameters

This is only called in three places. Two of them don't pass `auth_events`, and
the third doesn't use the dict after passing it in, so this should be non-functional.

* Stop `_check_event_auth` modifying its parameters

`_check_event_auth` is only called in three places. `on_send_membership_event`
doesn't pass an `auth_events`, and `prep` and `_auth_and_persist_event` do not
use the map after passing it in.

* Stop `_update_auth_events_and_context_for_auth` modifying its parameters

Return the updated auth event dict, rather than modifying the parameter.

This is only called from `_check_event_auth`.

* Improve documentation on `_auth_and_persist_event`

Rename `auth_events` parameter to better reflect what it contains.

* Improve documentation on `_NewEventInfo`

* Improve documentation on `_check_event_auth`

rename `auth_events` parameter to better describe what it contains

* changelog
2021-08-06 13:54:23 +01:00
Patrick Cloke
f4ade972ad
Update the API response for spaces summary over federation. (#10530)
This adds 'allowed_room_ids' (in addition to 'allowed_spaces', for backwards
compatibility) to the federation response of the spaces summary.

A future PR will remove the 'allowed_spaces' flag.
2021-08-06 07:40:29 -04:00
Richard van der Hoff
74d7336686
Add a setting to disable TLS for sending email (#10546)
This is mostly useful in case the server offers TLS, but doesn't present a valid certificate.
2021-08-06 10:13:34 +00:00
Eric Eastwood
f5a368bb48
Mark all MSC2716 events as historical (#10537)
* Mark all MSC2716 events as historical
2021-08-05 20:35:53 -05:00
Patrick Cloke
3b354faad0
Refactoring before implementing the updated spaces summary. (#10527)
This should have no user-visible changes, but refactors some pieces of
the SpaceSummaryHandler before adding support for the updated
MSC2946.
2021-08-05 12:39:17 +00:00
Will Hunt
a8a27b2b8b
Only return an appservice protocol if it has a service providing it. (#10532)
If there are no services providing a protocol, omit it completely
instead of returning an empty dictionary.

This fixes a long-standing spec compliance bug.
2021-08-05 08:22:14 -04:00
Erik Johnston
a36d77c563 Merge tag 'v1.40.0rc2' into develop
Synapse 1.40.0rc2 (2021-08-04)
==============================

Bugfixes
--------

- Fix the `PeriodicallyFlushingMemoryHandler` inhibiting application shutdown because of its background thread. ([\#10517](https://github.com/matrix-org/synapse/issues/10517))
- Fix a bug introduced in Synapse v1.40.0rc1 that could cause Synapse to respond with an error when clients would update read receipts. ([\#10531](https://github.com/matrix-org/synapse/issues/10531))

Internal Changes
----------------

- Fix release script to open the correct URL for the release. ([\#10516](https://github.com/matrix-org/synapse/issues/10516))
2021-08-05 11:15:29 +01:00
Michael Telatynski
9db24cc50d
Send unstable-prefixed room_type in store-invite IS API requests (#10435)
The room type is per MSC3288 to allow the identity-server to
change invitation wording based on whether the invitation is to
a room or a space.

The prefixed key will be replaced once MSC3288 is accepted
into the spec.
2021-08-04 13:39:57 -04:00
Eric Eastwood
684d19a11c
Add support for MSC2716 marker events (#10498)
* Make historical messages available to federated servers

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

Follow-up to https://github.com/matrix-org/synapse/pull/9247

* Debug message not available on federation

* Add base starting insertion point when no chunk ID is provided

* Fix messages from multiple senders in historical chunk

Follow-up to https://github.com/matrix-org/synapse/pull/9247

Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716

---

Previously, Synapse would throw a 403,
`Cannot force another user to join.`,
because we were trying to use `?user_id` from a single virtual user
which did not match with messages from other users in the chunk.

* Remove debug lines

* Messing with selecting insertion event extremeties

* Move db schema change to new version

* Add more better comments

* Make a fake requester with just what we need

See https://github.com/matrix-org/synapse/pull/10276#discussion_r660999080

* Store insertion events in table

* Make base insertion event float off on its own

See https://github.com/matrix-org/synapse/pull/10250#issuecomment-875711889

Conflicts:
	synapse/rest/client/v1/room.py

* Validate that the app service can actually control the given user

See https://github.com/matrix-org/synapse/pull/10276#issuecomment-876316455

Conflicts:
	synapse/rest/client/v1/room.py

* Add some better comments on what we're trying to check for

* Continue debugging

* Share validation logic

* Add inserted historical messages to /backfill response

* Remove debug sql queries

* Some marker event implemntation trials

* Clean up PR

* Rename insertion_event_id to just event_id

* Add some better sql comments

* More accurate description

* Add changelog

* Make it clear what MSC the change is part of

* Add more detail on which insertion event came through

* Address review and improve sql queries

* Only use event_id as unique constraint

* Fix test case where insertion event is already in the normal DAG

* Remove debug changes

* Add support for MSC2716 marker events

* Process markers when we receive it over federation

* WIP: make hs2 backfill historical messages after marker event

* hs2 to better ask for insertion event extremity

But running into the `sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group`
error

* Add insertion_event_extremities table

* Switch to chunk events so we can auth via power_levels

Previously, we were using `content.chunk_id` to connect one
chunk to another. But these events can be from any `sender`
and we can't tell who should be able to send historical events.
We know we only want the application service to do it but these
events have the sender of a real historical message, not the
application service user ID as the sender. Other federated homeservers
also have no indicator which senders are an application service on
the originating homeserver.

So we want to auth all of the MSC2716 events via power_levels
and have them be sent by the application service with proper
PL levels in the room.

* Switch to chunk events for federation

* Add unstable room version to support new historical PL

* Messy: Fix undefined state_group for federated historical events

```
2021-07-13 02:27:57,810 - synapse.handlers.federation - 1248 - ERROR - GET-4 - Failed to backfill from hs1 because NOT NULL constraint failed: event_to_state_groups.state_group
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1216, in try_backfill
    await self.backfill(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 1035, in backfill
    await self._auth_and_persist_event(dest, event, context, backfilled=True)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2222, in _auth_and_persist_event
    await self._run_push_actions_and_persist_event(event, context, backfilled)
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 2244, in _run_push_actions_and_persist_event
    await self.persist_events_and_notify(
  File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation.py", line 3290, in persist_events_and_notify
    events, max_stream_token = await self.storage.persistence.persist_events(
  File "/usr/local/lib/python3.8/site-packages/synapse/logging/opentracing.py", line 774, in _trace_inner
    return await func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 320, in persist_events
    ret_vals = await yieldable_gather_results(enqueue, partitioned.items())
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 237, in handle_queue_loop
    ret = await self._per_item_callback(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/persist_events.py", line 577, in _persist_event_batch
    await self.persist_events_store._persist_events_and_state_updates(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 176, in _persist_events_and_state_updates
    await self.db_pool.runInteraction(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 681, in runInteraction
    result = await self.runWithConnection(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 770, in runWithConnection
    return await make_deferred_yieldable(
  File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 238, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.8/site-packages/twisted/python/threadpool.py", line 254, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/context.py", line 83, in callWithContext
    return func(*args, **kw)
  File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/twisted/python/compat.py", line 403, in reraise
    raise exception.with_traceback(traceback)
  File "/usr/local/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 765, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 549, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/logging/utils.py", line 69, in wrapped
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 385, in _persist_events_txn
    self._store_event_state_mappings_txn(txn, events_and_contexts)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/databases/main/events.py", line 2065, in _store_event_state_mappings_txn
    self.db_pool.simple_insert_many_txn(
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 923, in simple_insert_many_txn
    txn.execute_batch(sql, vals)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 280, in execute_batch
    self.executemany(sql, args)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 300, in executemany
    self._do_execute(self.txn.executemany, sql, *args)
  File "/usr/local/lib/python3.8/site-packages/synapse/storage/database.py", line 330, in _do_execute
    return func(sql, *args)
sqlite3.IntegrityError: NOT NULL constraint failed: event_to_state_groups.state_group
```

* Revert "Messy: Fix undefined state_group for federated historical events"

This reverts commit 187ab28611546321e02770944c86f30ee2bc742a.

* Fix federated events being rejected for no state_groups

Add fix from https://github.com/matrix-org/synapse/pull/10439
until it merges.

* Adapting to experimental room version

* Some log cleanup

* Add better comments around extremity fetching code and why

* Rename to be more accurate to what the function returns

* Add changelog

* Ignore rejected events

* Use simplified upsert

* Add Erik's explanation of extra event checks

See https://github.com/matrix-org/synapse/pull/10498#discussion_r680880332

* Clarify that the depth is not directly correlated to the backwards extremity that we return

See https://github.com/matrix-org/synapse/pull/10498#discussion_r681725404

* lock only matters for sqlite

See https://github.com/matrix-org/synapse/pull/10498#discussion_r681728061

* Move new SQL changes to its own delta file

* Clean up upsert docstring

* Bump database schema version (62)
2021-08-04 12:07:57 -05:00