Commit Graph

15421 Commits

Author SHA1 Message Date
Patrick Cloke
28bceef84e
Check appservices for devices during a /user/devices query. (#15539)
MSC3984 proxies /keys/query requests to appservices, but servers will
can also requests devices / keys from the /user/devices endpoint.

The formats are close enough that we can "proxy" that /user/devices to
appservices (by calling /keys/query) and then change the format of the
returned data before returning it over federation.
2023-05-05 15:18:47 -04:00
Patrick Cloke
36df9c5e36
Implement MSC4009 to widen the allowed Matrix ID grammar (#15536)
Behind a configuration flag this adds + to the list of allowed
characters in Matrix IDs. The main feature this enables is
using full E.164 phone numbers as Matrix IDs.
2023-05-05 12:13:50 -04:00
Zdziszek
a0f53afd62
Handle DNSNotImplementedError in SRV resolver (#15523)
Signed-off-by: Zdzichu <zdzichu.rks@protonmail.com>
2023-05-05 15:54:32 +01:00
Andrew Morgan
7c95b65873
Clean up and clarify "Create or modify Account" Admin API documentation (#15544) 2023-05-05 15:51:46 +01:00
Sean Quah
e46d5f3586
Factor out an is_mine_server_name method (#15542)
Add an `is_mine_server_name` method, similar to `is_mine_id`.

Ideally we would use this consistently, instead of sometimes comparing
against `hs.hostname` and other times reaching into
`hs.config.server.server_name`.

Also fix a bug in the tests where `hs.hostname` would sometimes differ
from `hs.config.server.server_name`.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-05 15:06:22 +01:00
Andrew Morgan
2e59e97ebd
Move ThirdPartyEventRules into module_api/callbacks (#15535) 2023-05-04 14:18:22 +00:00
Patrick Cloke
ded8f3d349
Update the base rules to remove the dont_notify action. (MSC3987) (#15534)
A dont_notify action is a no-op (and coalesce is undefined). These are
both considered no-ops by the spec, per MSC3987 and the predefined
push rules were updated to remove dont_notify from the list of actions.
2023-05-04 11:54:13 +00:00
Sandro
5f8822854d
Use oEmbed for YouTube Shorts (#15025)
It seems that YouTube Short previews do not work in some
regions, but the oEmbed information for those areas is still
valid.

This causes YouTube Shorts to always use (only) the oEmbed
endpoint which is a minor regression for regions where the URL
preview was already working -- some of the additional video
metadata is lost. It is not likely that clients are using this today
and it is more beneficial to have a limited preview working everywhere
than unused metadata in the Open Graph response.
2023-05-03 12:54:42 -04:00
Sean Quah
8aee823393 Merge branch 'release-v1.83' into develop 2023-05-03 15:23:16 +01:00
Erik Johnston
28ac1a1a91
Speed up deleting of old rows in event_push_actions (#15531)
Enforce that we use index scans (rather than seq scans), which we also do for state queries. The reason to enforce this is that we can't correctly get PostgreSQL to understand the distribution of `stream_ordering` depends on `highlight`, and so it always defaults (on matrix.org) to sequential scans.
2023-05-03 13:42:43 +00:00
Erik Johnston
fc3a878220
Speed up rebuilding of the user directory for local users (#15529)
The idea here is to batch up the work.
2023-05-03 13:41:37 +00:00
Sean Quah
3b837d856c
Revert "Reduce the size of the HTTP connection pool for non-pushers" (#15530)
#15514 introduced a regression where Synapse would encounter
`PartialDownloadError`s when fetching OpenID metadata for certain
providers on startup. Due to #8088, this prevents Synapse from starting
entirely.

Revert the change while we decide what to do about the regression.
2023-05-03 13:09:20 +01:00
Patrick Cloke
a7b3e9ce65
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15437)
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.

For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.

For SQLite we simply rebuild the table & copy the data.
2023-05-03 07:49:03 -04:00
Sean Quah
04e79e6a18
Add config option to forget rooms automatically when users leave them (#15224)
This is largely based off the stats and user directory updater code.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-03 12:27:33 +01:00
Shay
0e8aa2a1b2
Remove references to supporting per-user flag for msc2654 (#15522) 2023-05-02 14:21:36 -07:00
Erik Johnston
4de271a7fc
Allow adding random delay to push (#15516)
This is to discourage timing based profiling on the push gateways.
2023-05-02 16:45:44 +00:00
Patrick Cloke
6aca4e7cb8
Reduce the size of the HTTP connection pool for non-pushers. (#15514)
Pushers tend to make many connections to the same HTTP host
(e.g. a new event comes in, causes events to be pushed, and then
the homeserver connects to the same host many times). Due to this
the per-host HTTP connection pool size was increased, but this does
not make sense for other SimpleHttpClients.

Add a parameter for the connection pool and override it for pushers
(making a separate SimpleHttpClient for pushers with the increased
configuration).

This returns the HTTP connection pool settings to the default Twisted
ones for non-pusher HTTP clients.
2023-05-02 09:29:40 -04:00
Patrick Cloke
07b1c70d6b
Initial implementation of MSC3981: recursive relations API (#15315)
Adds an optional keyword argument to the /relations API which
will recurse a limited number of event relationships.

This will cause the API to return not just the events related to the
parent event, but also events related to those related to the parent
event, etc.

This is disabled by default behind an experimental configuration
flag and is currently implemented using prefixed parameters.
2023-05-02 07:59:55 -04:00
Shay
89f6fb0d5a
Add an admin API endpoint to support per-user feature flags (#15344) 2023-04-28 11:33:45 -07:00
Patrick Cloke
57aeeb308b
Add support for claiming multiple OTKs at once. (#15468)
MSC3983 provides a way to request multiple OTKs at once from appservices,
this extends this concept to the Client-Server API.

Note that this will likely be spit out into a separate MSC, but is currently part of
MSC3983.
2023-04-27 12:57:46 -04:00
Patrick Cloke
6efa674004
Add type hints to schema deltas (#15497)
Cleans-up the schema delta files:

* Removes no-op functions.
* Adds missing type hints to function parameters.
* Fixes any issues with type hints.

This also renames one (very old) schema delta to avoid a conflict
that mypy complains about.
2023-04-27 12:44:53 +00:00
Patrick Cloke
a346b43837
Check databases/__init__ and main/cache with mypy. (#15496) 2023-04-27 07:59:14 -04:00
mcalinghee
486c059479
Disable push rule evaluation for rooms excluded from sync (#15361)
* no push for excluded room from sync

* add changelog
Signed-off-by: Maghen Calinghee <maghen.calinghee@beta.gouv.fr>

* correct changelog
2023-04-27 11:32:02 +01:00
Shay
301b4156d5
Add column full_user_id to tables profiles and user_filters. (#15458) 2023-04-26 16:03:26 -07:00
Mathieu Velten
247e6a8a78
Add a module API to send an HTTP push notification (#15387)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-04-26 21:10:51 +02:00
Erik Johnston
9900f7c231
Add admin endpoint to query room sizes (#15482) 2023-04-26 16:00:11 +00:00
Patrick Cloke
8e9739449d
Add unstable /keys/claim endpoint which always returns fallback keys. (#15462)
It can be useful to always return the fallback key when attempting to
claim keys. This adds an unstable endpoint for `/keys/claim` which
always returns fallback keys in addition to one-time-keys.

The fallback key(s) are not marked as "used" unless there are no
corresponding OTKs.

This is currently defined in MSC3983 (although likely to be split out
to a separate MSC). The endpoint shape may change or be requested
differently (i.e. a keyword parameter on the current endpoint), but the
core logic should be reasonable.
2023-04-25 13:30:41 -04:00
Nick Mills-Barrett
c55293c230
Re re introduce membership tables event stream ordering (#15356) 2023-04-25 09:44:29 +01:00
Quentin Gliech
8b3a502996
Experimental support for MSC3970: per-device transaction IDs (#15318) 2023-04-25 09:37:09 +01:00
Patrick Cloke
ea5c3ede4f
Finish type hints for federation client HTTP code. (#15465) 2023-04-24 13:12:06 -04:00
Alok Kumar Singh
197fbb123b
Remove legacy code of single user device resync api (#15418)
* Removed single-user resync usage and updated it to use multi-user counterpart

Signed-off-by: Alok Kumar Singh alokaks601@gmail.com
2023-04-21 12:06:39 +01:00
Patrick Cloke
5e024a0645
Modify StoreKeyFetcher to read from server_keys_json. (#15417)
Before this change:

* `PerspectivesKeyFetcher` and `ServerKeyFetcher` write to `server_keys_json`.
* `PerspectivesKeyFetcher` also writes to `server_signature_keys`.
* `StoreKeyFetcher` reads from `server_signature_keys`.

After this change:

* `PerspectivesKeyFetcher` and `ServerKeyFetcher` write to `server_keys_json`.
* `PerspectivesKeyFetcher` also writes to `server_signature_keys`.
* `StoreKeyFetcher` reads from `server_keys_json`.

This results in `StoreKeyFetcher` now using the results from `ServerKeyFetcher`
in addition to those from `PerspectivesKeyFetcher`, i.e. keys which are directly
fetched from a server will now be pulled from the database instead of refetched.

An additional minor change is included to avoid creating a `PerspectivesKeyFetcher`
(and checking it) if no `trusted_key_servers` are configured.

The overall impact of this should be better usage of cached results:

* If a server has no trusted key servers configured then it should reduce how often keys
  are fetched.
* if a server's trusted key server does not have a requested server's keys cached then it
  should reduce how often keys are directly fetched.
2023-04-20 12:30:32 -04:00
Andrew Morgan
aec639e3e3
Move Spam Checker callbacks to a dedicated file (#15453) 2023-04-18 00:57:40 +00:00
Jason Little
e12d788bb7
Switch InstanceLocationConfig to a pydantic BaseModel (#15431)
* Switch InstanceLocationConfig to a pydantic BaseModel, apply Strict* types and add a few helper methods(that will make more sense in follow up work).

Co-authored-by: David Robertson <davidr@element.io>
2023-04-17 23:53:43 +00:00
Jason Little
c9326140dc
Refactor SimpleHttpClient to pull out reusable methods (#15427)
Pulls out some methods to `BaseHttpClient` to eventually be
reused in other contexts.
2023-04-14 20:46:04 +00:00
David Robertson
8a47d6e3a6
More precise type for LoggingTransaction.execute (#15432)
* More precise type for LoggingTransaction.execute
* Add an annotation for stream_ordering_month_ago

This would have spotted the error that was fixed in "Add comma missing from #15382. (#15429)"
2023-04-14 18:04:49 +00:00
Dirk Klimpel
24b61f32ff
Disable directory listing for StaticResource (#15438) 2023-04-14 13:49:47 -04:00
Dirk Klimpel
e4a25d022c
Load /capabilities endpoint on workers (#15436) 2023-04-14 12:26:07 -04:00
Erik Johnston
b5192355f6
User directory background update speedup (#15435)
c.f. #15264

The two changes are:
1. Add indexes so that the select / deletes don't do sequential scans
2. Don't repeatedly call `SELECT count(*)` each iteration, as that's slow
2023-04-14 16:10:32 +01:00
Mathieu Velten
dabbb94faf
Delete pushers after calling on_logged_out module hook on device delete (#15410) 2023-04-14 14:12:37 +02:00
Dirk Klimpel
4af0aec54d
Load /directory/room/{roomAlias} endpoint on workers (#15333)
* Enable `directory`

* move to worker store

* newsfile

* disable `ClientDirectoryListServer` and `ClientAppserviceDirectoryListServer` for workers
2023-04-14 10:24:06 +01:00
Patrick Cloke
d751f65e71
Remove registration fallback code. (#15405)
The registration fallback is broken and unspecced. This removes it
since there is no plan to spec it.

Note that this does not modify the login fallback code.
2023-04-13 11:36:29 -04:00
reivilibre
edae20f926
Improve robustness when handling a perspective key response by deduplicating received server keys. (#15423)
* Change `store_server_verify_keys` to take a `Mapping[(str, str), FKR]`

This is because we already can't handle duplicate keys — leads to cardinality violation

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-04-13 15:35:03 +01:00
reivilibre
38272be037
Add comma missing from #15382. (#15429)
* Add missing comma

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-04-13 15:06:25 +01:00
Patrick Cloke
2503126d52
Implement MSC2174: move redacts to a content property. (#15395)
This moves `redacts` from being a top-level property to
a `content` property in a new room version.

MSC2176 (which was previously implemented) states to not
`redact` this property.
2023-04-13 13:47:07 +00:00
Dirk Klimpel
c9723a1c1f
Only load the SSO redirect servlet if SSO is enabled. (#15421) 2023-04-13 13:08:00 +00:00
Dirk Klimpel
be36600327
Disable loading RefreshTokenServlet on workers (#15428) 2023-04-13 13:28:55 +02:00
Will Hunt
253e86a72e
Throw if the appservice config list is the wrong type (#15425)
* raise a ConfigError on an invalid app_service_config_files

* changelog

* Move config check to read_config

* Add test

* Ensure list also contains strings
2023-04-12 11:28:46 +00:00
Patrick Cloke
d07d255830
Implement MSC2175: remove the creator field from create events. (#15394) 2023-04-06 16:26:28 -04:00
Erik Johnston
485b9fdefb
Don't keep old stream_ordering_to_exterm around (#15382) 2023-04-06 16:42:39 +00:00