Commit Graph

16018 Commits

Author SHA1 Message Date
dependabot[bot]
871f51c270
Bump lxml-stubs from 0.4.0 to 0.5.1 (#16885) 2024-02-06 09:29:17 +00:00
Erik Johnston
adf15c4f6b
Run ANALYZE after fiddling with stats (#16849)
Introduced in #16833

Fixes #16844
2024-01-24 13:57:12 +00:00
Erik Johnston
c925b45567
Speed up e2e device keys queries for bot accounts (#16841)
This helps with bot accounts with lots of non-e2e devices.

The change is basically to change the order of the join for the case of
using `INNER JOIN`
2024-01-23 11:37:16 +00:00
Erik Johnston
23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Erik Johnston
14c725f73b
Preparatory work for tweaking performance of auth chain lookups (#16833) 2024-01-23 11:26:27 +00:00
Shay
a68b48a5dd
Allow room creation but not publishing to continue if room publication rules are violated when creating a new room. (#16811)
Prior to this PR, if a request to create a public (public as in
published to the rooms directory) room violated the room list
publication rules set in the
[config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules),
the request to create the room was denied and the room was not created.

This PR changes the behavior such that when a request to create a room
published to the directory violates room list publication rules, the
room is still created but the room is not published to the directory.
2024-01-22 13:59:45 +00:00
Mo Balaa
b99f6db039
Handle wildcard type filters properly (#14984) 2024-01-22 10:46:30 +00:00
Hanadi
42e1aaea68
feat: add msc4028 to versions api (#16787)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-16 14:36:08 +00:00
Erik Johnston
c43f751013
Optimize query for fetching to-device messages in /sync (#16805)
The current query supports passing in a list of users, which generates a
query using `user_id = ANY(..)`. This is generates a less efficient
query plan that is notably slower than a simple `user_id = ?` condition.

Note: The new function is mostly a copy and paste and then a
simplification of the existing function.
2024-01-11 13:37:57 +00:00
Erik Johnston
b11f7b5122
Improve DB performance of calculating badge counts for push. (#16756)
The crux of the change is to try and make the queries simpler and pull
out fewer rows. Before, there were quite a few joins against subqueries,
which caused postgres to pull out more rows than necessary.

Instead, let's simplify the query and do some of the filtering out in
Python instead, letting Postgres do better optimizations now that it
doesn't have to deal with joins against subqueries.

Review note: this is a complete rewrite of the function, so not sure how
useful the diff is.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-11 11:52:13 +00:00
Erik Johnston
a986f86c82
Correctly handle OIDC config with no client_secret set (#16806)
In previous versions of authlib using `client_secret_basic` without a
`client_secret` would result in an invalid auth header. Since authlib
1.3 it throws an exception.

The configuration may be accepted in by very lax servers, so we don't
want to deny it outright. Instead, let's default the
`client_auth_method` to `none`, which does the right thing. If the
config specifies `client_auth_method` and no `client_secret` then that
is going to be bogus and we should reject it
2024-01-10 17:16:49 +00:00
Erik Johnston
cbe8a80d10
Faster load recents for sync (#16783)
This hopefully reduces the amount of state we need to keep in memory
2024-01-10 15:11:59 +00:00
Erik Johnston
0a96fa52a2
Pull less state out if we fail to backfill (#16788)
Sometimes we fail to fetch events during backfill due to missing state,
and we often end up querying the same bad events periodically (as people
backpaginate). In such cases its likely we will continue to fail to get
the state, and therefore we should try *before* loading the state that
we have from the DB (as otherwise it's wasted DB and memory).

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-01-10 14:42:13 +00:00
Erik Johnston
578c5c736e
Reduce amount of state pulled out when querying federation hierachy (#16785)
There are two changes here:

1. Only pull out the required state when handling the request.
2. Change the get filtered state return type to check that we're only
querying state that was requested

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-01-10 14:31:35 +00:00
Erik Johnston
4c67f0391b
Split up deleting devices into batches (#16766)
Otherwise for users with large numbers of devices this can cause a lot
of woe.
2024-01-10 13:55:16 +00:00
Erik Johnston
c3f2f0f063
Faster partial join to room with complex auth graph (#7)
Instead of persisting outliers in a bunch of batches, let's just do them
all at once.

This is fine because all `_auth_and_persist_outliers_inner` is doing is
checking the auth rules for each event, which requires the events to be
topologically sorted by the auth graph.
2024-01-10 12:29:42 +00:00
reivilibre
a83a337c4d
Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. (#16759)
The idea here being that the directory server shouldn't advertise rooms
to a requesting server is the requesting server would not be allowed to
join or participate in the room.

<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Pass `from_federation_origin` down into room list retrieval code 

</li>
<li>

Don't cache /publicRooms response for inbound federated requests 

</li>
<li>

fixup! Don't cache /publicRooms response for inbound federated requests 

</li>
<li>

Cap the number of /publicRooms entries to 100 

</li>
<li>

Simplify code now that you can't request unlimited rooms 

</li>
<li>

Filter out rooms from federated requests that don't have the correct ACL

</li>
<li>

Request a handful more when filtering ACLs so that we can try to avoid
shortchanging the requester

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-08 17:24:20 +00:00
Erik Johnston
5d3850b038
Port EventInternalMetadata class to Rust (#16782)
There are a couple of things we need to be careful of here:

1. The current python code does no validation when loading from the DB,
so we need to be careful to ignore such errors (at least on jki.re there
are some old events with internal metadata fields of the wrong type).
2. We want to be memory efficient, as we often have many hundreds of
thousands of events in the cache at a time.

---------

Co-authored-by: Quentin Gliech <quenting@element.io>
2024-01-08 14:06:48 +00:00
Erik Johnston
81b1c56288
Fix linting (#16780)
Introduced in #16762
2024-01-05 13:29:00 +00:00
Erik Johnston
7469fa7585
Simplify internal metadata class. (#16762)
We remove these fields as they're just duplicating data the event
already stores, and (for reasons 🤫) I'd like to simplify
the class to only store simple types.

I'm not entirely convinced that we shouldn't instead add helper methods
to the event class to generate stream tokens, but I don't really think
that's where they belong either
2024-01-05 13:03:20 +00:00
David Baker
25b3ba5328
Add recursion_depth to /relations if recursing (#16775)
This is an extra response parameter just added to MSC3981. In the
current impl, the recursion depth is always 3, so this just returns a
static 3 if the recurse parameter is supplied.
2024-01-04 09:57:21 +00:00
Adam Jędrzejewski
c7d0d02be7
Search non ASCII display names using Admin API (#16767)
Closes #16370

Signed-off-by: Adam Jedrzejewski <adamjedrzejewski@icloud.com>
2024-01-04 09:36:57 +00:00
FadhlanR
eabedd9520
Fix email verification redirection (#16761)
Previously, the response status of `HTMLResource` was hardcoded as
`200`. However, for proper redirection after the user verifies their
email, we require the status to be `302`. This PR addresses that issue
by using `code` as response status.
2024-01-02 16:25:26 +00:00
Dirk Klimpel
14ed84ac33
Enable user without password (#16770)
Closes:
- https://github.com/matrix-org/synapse/issues/10397
- #10397 

An administrator should know whether he wants to set a password or not.
There are many uses cases where a blank password is required.

- Use of only some users with SSO.
- Use of bots with password, users with SSO
2024-01-02 11:52:51 +00:00
Erik Johnston
8a50312099
Move the rust stubs inline for better IDE integration (#16757)
At least for vscode this allows click through / type checking / syntax
highlighting.
2023-12-21 13:31:36 +00:00
Erik Johnston
0455c40085 Update book location 2023-12-13 16:15:22 +00:00
Erik Johnston
23a919d267 Fix links 2023-12-13 15:15:34 +00:00
Patrick Cloke
3c964e513e Log the new license during start. 2023-12-13 15:14:35 +00:00
Erik Johnston
eaad9bb156 Merge remote-tracking branch 'gitlab/clokep/license-license' into new_develop 2023-12-13 15:11:56 +00:00
Erik Johnston
230decd5b8 Revert changes to README 2023-12-13 15:02:08 +00:00
Erik Johnston
025951bc3b Merge remote-tracking branch 'origin/clokep/morg-readme' into develop 2023-12-13 14:46:11 +00:00
Erik Johnston
70c020b532 Update text 2023-12-12 20:32:48 +00:00
Patrick Cloke
e1f8440c89 Update the README pointing to the Element fork. 2023-12-12 20:28:30 +00:00
Zeeshan Rafiq
e108cde669
Sentry Alert configuration based on production and development environment (#16738) 2023-12-12 16:04:41 +00:00
Mathieu Velten
e108c31fc0
Add avatar and topic settings for server notice room (#16679) 2023-12-12 15:22:19 +00:00
Mathieu Velten
9f6c644825
Add config to change the delay before sending a notification email (#16696) 2023-12-12 10:28:56 +00:00
elara-leitstellentechnik
10ada2ff6d
Write signing keys with file mode 0640 (#16740)
Co-authored-by: Fabian Klemp <fabian.klemp@frequentis.com>
2023-12-08 16:25:57 +00:00
David Robertson
fcf3c493c2
Expose OIDC discovery information under the CSAPI (#16726)
Co-authored-by: Quentin Gliech <quenting@element.io>
2023-12-06 13:48:09 +00:00
David Robertson
44377f5ac0
Revert postgres logical replication deltaas
This reverts two commits:

    0bb8e418a4
    "Fix postgres schema after dropping old tables (#16730)"

and

    51e4e35653
    "Add a Postgres `REPLICA IDENTITY` to tables that do not have an implicit one. This should allow use of Postgres logical replication. (take  2, now with no added deadlocks!) (#16658)"

and also amends the changelog.
2023-12-05 16:10:48 +00:00
David Robertson
0a00c99823
Fix upgrading a room without events field in power levels (#16725) 2023-12-05 12:06:21 +00:00
Will Hunt
ea783550bb
Set response values to zero if None for /_synapse/admin/v1/federation/destinations (#16729) 2023-12-05 11:40:27 +00:00
David Robertson
0bb8e418a4
Fix postgres schema after dropping old tables (#16730) 2023-12-05 11:08:40 +00:00
reivilibre
51e4e35653
Add a Postgres REPLICA IDENTITY to tables that do not have an implicit one. This should allow use of Postgres logical replication. (take 2, now with no added deadlocks!) (#16658)
* Add `ALTER TABLE ... REPLICA IDENTITY ...` for individual tables

We can't combine them into one file as it makes it likely to hit a deadlock

if Synapse is running, as it only takes one other transaction to access two

tables in a different order to the schema delta.

* Add notes

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Re-introduce REPLICA IDENTITY test

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-12-04 14:57:28 +00:00
Mathieu Velten
9e7f80037d
Server notices: add an autojoin setting for the notices room (#16699)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-12-04 12:31:42 +00:00
Travis Ralston
d6e194b2bc
Implement MSC4069: Inhibit profile propagation (#16636)
MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/4069
2023-12-04 11:36:12 +00:00
Andrew Yasinishyn
63d96bfc61
ModuleAPI SSO auth callbacks (#15207)
Signed-off-by: Andrii Yasynyshyn yasinishyn.a.n@gmail.com
2023-12-01 14:31:50 +00:00
Patrick Cloke
579c6be5f6
Drop unused tables & unneeded access token ID for events. (#16522) 2023-12-01 10:12:00 +00:00
Patrick Cloke
dcf949cd87
Declare support for Matrix v1.7, v1.8, and v1.9. (#16707) 2023-11-29 15:02:09 -05:00
Patrick Cloke
d6c3b7584f
Request & follow redirects for /media/v3/download (#16701)
Implement MSC3860 to follow redirects for federated media downloads.

Note that the Client-Server API doesn't support this (yet) since the media
repository in Synapse doesn't have a way of supporting redirects.
2023-11-29 19:03:42 +00:00
Erik Johnston
a14678492e
Reduce DB load when forget on leave setting is disabled (#16668)
* Reduce DB load when forget on leave setting is disabled

* Newsfile
2023-11-29 18:21:30 +00:00
Erik Johnston
df366966b4
Speed up pruning of user_ips table (#16667)
Silly query planner
2023-11-29 11:54:42 +00:00
Mathieu Velten
b0ed14d815
Ignore encryption_enabled_by_default_for_room_type for notices room (#16677) 2023-11-28 13:15:26 +00:00
Patrick Cloke
d199b84006
Remove old full schema dumps. (#16697)
These are not useful and make it difficult to search for
table definitions, etc.
2023-11-28 07:28:07 -05:00
David Robertson
c3627d0f99
Correctly read to-device stream pos on SQLite (#16682) 2023-11-24 13:42:38 +00:00
David Robertson
32a59a6495
Keep track of user_ips and monthly_active_users when delegating auth (#16672)
* Describe `insert_client_ip`
* Pull out client_ips and MAU tracking to BaseAuth
* Define HAS_AUTHLIB once in tests

sick of copypasting

* Track ips and token usage when delegating auth
* Test that we track MAU and user_ips
* Don't track `__oidc_admin`
2023-11-23 12:35:37 +00:00
Charles Wright
1a5f9bb651
Enable refreshable tokens on the admin registration endpoint (#16642)
Signed-off-by: Charles Wright <cvwright@futo.org>
2023-11-22 15:01:09 +00:00
Mathieu Velten
c432d8f18f
Admin API for server notice: consistently bypass rate limits (#16670)
* Admin API for server notice: disable rate limit for all calls

* Add changelog

* Update changelog.d/16670.bugfix
2023-11-22 13:47:29 +00:00
Jason Little
460743da16
Filter out auth chain queries that don't exist (#16552) 2023-11-22 10:59:16 +00:00
Patrick Cloke
8e1e62c9e0 Update license headers 2023-11-21 15:29:58 -05:00
Erik Johnston
6088303efb
Speed up how quickly we launch new tasks (#16660)
Now that we're reducing concurrency (#16656), this is more important.
2023-11-17 16:36:02 +00:00
Erik Johnston
9c02ef21e0
Speed up purge room by adding index (#16657)
What it says on the tin
2023-11-17 14:15:44 +00:00
Erik Johnston
6fec2d035f
Also discard 'caches' and 'backfill' stream POSITIONS (#16655)
Follow on from #16640
2023-11-17 14:14:29 +00:00
Patrick Cloke
bdb0cbc5ca Merge branch 'master' into develop 2023-11-17 08:43:47 -05:00
Erik Johnston
700c8a0de5
Reduce task concurrency (#16656) 2023-11-17 13:14:26 +00:00
Erik Johnston
4d6b800385
Revert "Fix test not detecting tables with missing primary keys and missing replica identities, then add more replica identities. (#16647)" (#16652)
This reverts commit 830988ae72.
2023-11-16 16:57:26 +00:00
Erik Johnston
ef5329a9f9
Revert "Add a Postgres REPLICA IDENTITY to tables that do not have an implicit one. This should allow use of Postgres logical replication. (#16456)" (#16651)
This reverts commit 69afe3f7a0.
2023-11-16 16:48:48 +00:00
Erik Johnston
3e8531d3ba
Speed up deleting device messages (#16643)
Keeping track of a lower bound of stream ID where we've deleted everything below makes the queries much faster. Otherwise, every time we scan for rows to delete we'd re-scan across all the rows that have previously deleted (until the next table VACUUM).
2023-11-16 15:19:35 +00:00
Erik Johnston
1b238e8837
Speed up persisting large number of outliers (#16649)
Recalculating the roots tuple every iteration could be very expensive, so instead let's do a topological sort.
2023-11-16 14:25:35 +00:00
Erik Johnston
fef08cbee8
Fix sending out of order POSITION over replication (#16639)
If a worker reconnects to Redis we send out the current positions of all our streams. However, if we're also trying to send out a backlog of RDATA at the same time then we can end up sending a `POSITION` with the current token *before* we've sent all the RDATA before the current token.

This doesn't cause actual bugs as the receiving servers see the POSITION, fetch the relevant rows from the DB, and then ignore the old RDATA as they come in. However, this is inefficient so it'd be better if we didn't  send out-of-order positions
2023-11-16 13:05:09 +00:00
Erik Johnston
898655fd12
More efficiently handle no-op POSITION (#16640)
We may receive `POSITION` commands where we already know that worker has
advanced past that position, so there is no point in handling it.
2023-11-16 12:32:17 +00:00
reivilibre
830988ae72
Fix test not detecting tables with missing primary keys and missing replica identities, then add more replica identities. (#16647)
* Fix the CI query that did not detect all cases of missing primary keys

* Add more missing REPLICA IDENTITY entries

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-11-16 12:26:27 +00:00
David Robertson
43d1aa75e8
Add an Admin API to temporarily grant the ability to update an existing cross-signing key without UIA (#16634) 2023-11-15 17:28:10 +00:00
Sumner Evans
999bd77d3a
Asynchronous Uploads (#15503)
Support asynchronous uploads as defined in MSC2246.
2023-11-15 09:19:24 -05:00
Patrick Cloke
f2f2c7c1f0
Use full GitHub links instead of bare issue numbers. (#16637) 2023-11-15 08:02:11 -05:00
Nick Mills-Barrett
0e36a57b60
Remove whole table locks on push rule add/delete (#16051)
The statements are already executed within a transaction thus a table
level lock is unnecessary.
2023-11-13 16:57:44 +00:00
reivilibre
69afe3f7a0
Add a Postgres REPLICA IDENTITY to tables that do not have an implicit one. This should allow use of Postgres logical replication. (#16456)
* Add Postgres replica identities to tables that don't have an implicit one

Fixes #16224

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Move the delta to version 83 as we missed the boat for 82

* Add a test that all tables have a REPLICA IDENTITY

* Extend the test to include when indices are deleted

* isort

* black

* Fully qualify `oid` as it is a 'hidden attribute' in Postgres 11

* Update tests/storage/test_database.py

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Add missed tables

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-11-13 16:03:22 +00:00
Patrick Cloke
2c6a7dfcbf
Use attempt_to_set_autocommit everywhere. (#16615)
To avoid asserting the type of the database connection.
2023-11-09 16:19:42 -05:00
reivilibre
dc7f068d9c
Fix a long-standing bug where Synapse would not unbind third-party identifiers for Application Service users when deactivated and would not emit a compliant response. (#16617)
* Don't skip unbinding 3PIDs and returning success status when deactivating AS user

Fixes #16608

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-11-09 20:18:25 +00:00
Patrick Cloke
9f514dd0fb
Use _invalidate_cache_and_stream_bulk in more places. (#16616)
This takes advantage of the new bulk method in more places to
invalidate caches for many keys at once (and then to stream that
over replication).
2023-11-09 14:40:30 -05:00
Patrick Cloke
ab3f1b3b53
Convert simple_select_one_txn and simple_select_one to return tuples. (#16612) 2023-11-09 11:13:31 -05:00
Patrick Cloke
ff716b483b
Return attrs for more media repo APIs. (#16611) 2023-11-09 11:00:30 -05:00
David Robertson
91587d4cf9
Bulk-invalidate e2e cached queries after claiming keys (#16613)
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-11-09 15:57:09 +00:00
Patrick Cloke
455ef04187
Avoid updating the same rows multiple times with simple_update_many_txn. (#16609)
simple_update_many_txn had a bug in it which would cause each
update to be applied twice.
2023-11-07 14:02:09 -05:00
Patrick Cloke
9738b1c497
Avoid executing no-op queries. (#16583)
If simple_{insert,upsert,update}_many_txn is called without any data
to modify then return instead of executing the query.

This matches the behavior of simple_{select,delete}_many_txn.
2023-11-07 14:00:25 -05:00
Patrick Cloke
ec9ff389f4
More tests for the simple_* methods. (#16596)
Expand tests for the simple_* database methods, additionally
test against both PostgreSQL and SQLite variants.
2023-11-07 09:34:23 -05:00
Patrick Cloke
7e5d3b06fa
Collect information for PushRuleEvaluator in parallel. (#16590)
Fetch information needed for push rule evaluation in parallel.
Ideally this would use query pipelining, but this is not
available in psycopg2.

Due to the database thread pool this may result in little
to no parallelization.
2023-11-06 15:41:57 -05:00
Patrick Cloke
cc4fe68adf
Support reactor timing metric on more reactors. (#16532)
Previously only Twisted's EPollReactor was compatible with the
reactor timing metric, notably not working when asyncio was used.

After this change, the following configurations support the reactor
timing metric:

* poll, epoll, or select reactors
* asyncio reactor with a poll, epoll, select, /dev/poll, or kqueue event loop.
2023-11-06 08:31:22 -05:00
Patrick Cloke
92828a7f95
Simplify event persistence code (#16584)
The event persistence code used to handle multiple rooms
at a time, but was simplified to only ever be called with a
single room at a time (different rooms are now handled in
parallel). The code is still generic to multiple rooms causing
a lot of work that is unnecessary (e.g. unnecessary loops, and
partitioning data by room).

This strips out the ability to handle multiple rooms at once, greatly
simplifying the code.
2023-11-03 07:30:31 -04:00
Patrick Cloke
bf69b57422
Fix "'int' object is not iterable" error in set_device_id_for_pushers background update (#16594)
A regression from removing the cursor_to_dict call, adds back
the wrapping into a tuple.
2023-11-02 14:00:18 +00:00
Patrick Cloke
0afbef30cf
Use simple_select_many_txn in event persistance code. (#16585)
Just to standardize on the normal helpers, it might also have
a slight perf improvement on PostgreSQL which will now use
`ANY (?)` instead of `IN (?, ?, ...)`.
2023-11-02 09:41:00 -04:00
dependabot[bot]
c812f43bd7
Bump twisted from 23.8.0 to 23.10.0 (#16588) 2023-11-01 10:23:13 +00:00
Patrick Cloke
ed1b879576
Do not call getfullargspec on every call. (#16589)
getfullargspec is relatively expensive and the results will
not change between calls, so precalculate it outside the
wrapper.
2023-10-31 20:16:17 +00:00
Patrick Cloke
cfb6d38c47
Remove remaining usage of cursor_to_dict. (#16564) 2023-10-31 13:13:28 -04:00
Patrick Cloke
70b503f144 Fix import ordering issue introduced in 7a3a55ac98. 2023-10-31 10:32:35 -04:00
Patrick Cloke
7a3a55ac98
Merge pull request from GHSA-mp92-3jfm-3575 2023-10-31 13:58:30 +00:00
David Robertson
de981ae567
Claim local one-time-keys in bulk (#16565)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-10-30 21:25:21 +00:00
Erik Johnston
408c13801a
Add fast path for replication events stream fetch (#16580)
We can bail early if the from token is greater than or equal to the
current token.
2023-10-30 14:47:57 +00:00
David Robertson
fdce83ee60
Claim fallback keys in bulk (#16570) 2023-10-30 14:34:37 +00:00
Erik Johnston
8c63e93286
Fix HTTP repl response to use minimum token (#16578) 2023-10-30 12:27:14 +00:00
David Robertson
747416e94c
Portdb: don't copy a table that gets rebuilt (#16563) 2023-10-27 20:14:02 +01:00
Patrick Cloke
2bf9341406
Ensure local invited & knocking users leave before purge. (#16559)
This is mostly useful for federated rooms where some users
would get stuck in the invite or knock state when the room
was purged from their homeserver.
2023-10-27 12:50:50 -04:00
Erik Johnston
5413cefe32
Reduce amount of caches POSITIONS we send (#16561)
Follow on from / actually correctly does #16557
2023-10-27 16:07:11 +01:00
Erik Johnston
89dbbd68e1
Reduce spurious replication catchup (#16555) 2023-10-27 13:27:20 +00:00
Erik Johnston
928e964857
Fix cross-worker ratelimiting (#16558)
c.f. #16481
2023-10-27 12:52:40 +01:00
Erik Johnston
0680d76659
Reduce replication traffic due to reflected cache stream POSITION (#16557) 2023-10-27 12:51:08 +01:00
Erik Johnston
c02406ac71
Add new module API for adding custom fields to events unsigned section (#16549) 2023-10-27 09:04:08 +00:00
Patrick Cloke
679c691f6f
Remove more usages of cursor_to_dict. (#16551)
Mostly to improve type safety.
2023-10-26 15:12:28 -04:00
Patrick Cloke
85e5f2dc25
Add a new module API to update user presence state. (#16544)
This adds a module API which allows a module to update a user's
presence state/status message. This is useful for controlling presence
from an external system.

To fully control presence from the module the presence.enabled config
parameter gains a new state of "untracked" which disables internal tracking
of presence changes via user actions, etc. Only updates from the module will
be persisted and sent down sync properly).
2023-10-26 15:11:24 -04:00
Patrick Cloke
9407d5ba78
Convert simple_select_list and simple_select_list_txn to return lists of tuples (#16505)
This should use fewer allocations and improves type hints.
2023-10-26 13:01:36 -04:00
Erik Johnston
ba47fea528
Allow multiple workers to write to receipts stream. (#16432)
Fixes #16417
2023-10-25 16:16:19 +01:00
Richard Brežák
95076f77c1
Fix http/s proxy authentication with long username/passwords (#16504) 2023-10-24 13:45:21 +00:00
Jason Little
ffbe9b7666
Remove duplicate call to wake a remote destination when using federation sending worker (#16515) 2023-10-24 08:09:59 -04:00
Patrick Cloke
3ab861ab9e
Fix type hint errors from Twisted trunk (#16526) 2023-10-23 14:28:05 -04:00
Erik Johnston
8f35f8148e
Fix bug where a new writer advances their token too quickly (#16473)
* Fix bug where a new writer advances their token too quickly

When starting a new writer (for e.g. persisting events), the
`MultiWriterIdGenerator` doesn't have a minimum token for it as there
are no rows matching that new writer in the DB.

This results in the the first stream ID it acquired being announced as
persisted *before* it actually finishes persisting, if another writer
gets and persists a subsequent stream ID. This is due to the logic of
setting the minimum persisted position to the minimum known position of
across all writers, and the new writer starts off not being considered.

* Fix sending out POSITIONs when our token advances without update

Broke in #14820

* For replication HTTP requests, only wait for minimal position
2023-10-23 16:57:30 +01:00
Erik Johnston
3bc23cc45c
Fix bug that could cause a /sync to tightloop with sqlite after restart (#16540)
This could happen if the last rows in the account data stream were inserted into `account_data`. After a restart the max account ID would be calculated without looking at the `account_data` table, and so have an old ID.
2023-10-23 13:39:25 +00:00
Denis Kasak
3a0aa6fe76
Force TLS certificate verification in registration script. (#16530)
If using the script remotely, there's no particularly convincing reason
to disable certificate verification, as this makes the connection
interceptible.

If on the other hand, the script is used locally (the most common use
case), you can simply target the HTTP listener and avoid TLS altogether.
This is what the script already attempts to do if passed a homeserver
configuration YAML file.
2023-10-23 07:38:51 -04:00
Patrick Cloke
12ca87f5ea
Remove the last reference to event_txn_id. (#16521)
This table was no longer used, except for a background process
which purged old entries in it.
2023-10-23 07:37:45 -04:00
Erik Johnston
e9069c9f91
Mark sync as limited if there is a gap in the timeline (#16485)
This splits thinsg into two queries, but most of the time we won't have
new event backwards extremities so this shouldn't actually add an extra
RTT for the majority of cases.

Note this removes the check for events with no prev events, but that was
part of MSC2716 work that has since been removed.
2023-10-19 15:04:18 +01:00
Patrick Cloke
49c9745b45
Avoid sending massive replication updates when purging a room. (#16510) 2023-10-18 12:26:01 -04:00
Mathieu Velten
bcff01b406
Improve performance of delete device messages query (#16492) 2023-10-18 16:42:01 +01:00
Patrick Cloke
6ad1f9eac2
Convert DeviceLastConnectionInfo to attrs. (#16507)
To improve type safety & memory usage.
2023-10-17 12:47:42 +00:00
Patrick Cloke
77dfc1f939
Fix a bug where servers could be marked as up when they were failing (#16506)
After this change a server will only be reported as back online
if they were previously having requests fail.
2023-10-17 07:32:40 -04:00
Patrick Cloke
e3e0ae4ab1
Convert state delta processing from a dict to attrs. (#16469)
For improved type checking & memory usage.
2023-10-16 07:35:22 -04:00
Mathieu Velten
eee6474bce
Remove useless async job to delete device messages on sync (#16491) 2023-10-16 11:06:27 +01:00
Richard van der Hoff
109882230c
Clean up logging on event persister endpoints (#16488) 2023-10-14 17:57:27 +01:00
Patrick Cloke
4cc729d480
Revert "Drop unused tables & unneeded access token ID for events. (#16268)" (#16465)
This reverts commit cabd577460.

There are additional usages of these tables
which need to be removed first.
2023-10-12 08:56:10 -04:00
Patrick Cloke
cc865fffc0
Convert user_get_threepids response to attrs. (#16468)
This improves type annotations by not having a dictionary of Any values.
2023-10-11 20:08:11 -04:00
Patrick Cloke
a4904dcb04
Convert simple_select_many_batch, simple_select_many_txn to tuples. (#16444) 2023-10-11 13:24:56 -04:00
Patrick Cloke
d6b7d49a61
Handle content types with parameters. (#16440) 2023-10-11 07:50:34 -04:00
Patrick Cloke
f1e43018b7
Inline simple_search_list/simple_search_list_txn. (#16434)
This only has a single use and is over abstracted. Inline it so that
we can improve type hints.
2023-10-10 12:16:36 -04:00
David Robertson
28fd28e92e
Add DB indices to speed up purging rooms (#16457) 2023-10-10 10:33:39 +01:00
reivilibre
8902b3031d
Disable statement timeout whilst purging rooms (#16455)
* Disable statement timeout whilst purging rooms

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Note the introduction version

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-10-09 15:41:17 +01:00
dependabot[bot]
0a67743d9e
Bump ruff from 0.0.290 to 0.0.292 (#16449)
* Bump ruff from 0.0.290 to 0.0.292

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.0.290 to 0.0.292.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/BREAKING_CHANGES.md)
- [Commits](https://github.com/astral-sh/ruff/compare/v0.0.290...v0.0.292)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix up lint

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2023-10-09 10:46:17 +01:00
Christoph
32fd9bc673
Fix possible AttributeError when account-api is called over unix socket (#16404)
Fixes #16396
2023-10-09 10:16:07 +01:00
David Robertson
1f10c20806
Apply join rate limiter outside the lineariser (#16441) 2023-10-06 17:31:52 +00:00
Patrick Cloke
06bbf1029c
Convert simple_select_list_paginate_txn to return tuples. (#16433) 2023-10-06 11:41:57 -04:00
Patrick Cloke
7615e2bf48
Return ThumbnailInfo in more places (#16438)
Improves type hints by using concrete types instead of
dictionaries.
2023-10-06 10:12:43 -04:00
Patrick Cloke
cabd577460
Drop unused tables & unneeded access token ID for events. (#16268)
Drop the event_txn_id table and the tables related to MSC2716,
which is no longer supported in Synapse.
2023-10-06 08:29:33 -04:00
Patrick Cloke
fc31b495b3
Stop sending incorrect knock_state_events. (#16403)
Synapse was incorrectly implemented with a knock_state_events
property on some APIs (instead of knock_room_state). This was
correct in Synapse 1.70.0, but *both* fields were sent to also be
compatible with Synapse versions expecting the wrong field.

Enough time has passed that only the correct field needs to be
included/handled.
2023-10-06 07:27:35 -04:00
Patrick Cloke
ae5b997cfa
Fix comments related to replication. (#16428) 2023-10-06 07:25:44 -04:00
Patrick Cloke
26b960b08b
Register media servlets via regex. (#16419)
This converts the media servlet URLs in the same way as
(most) of the rest of Synapse. This will give more flexibility
in the versions each endpoint exists under.
2023-10-06 07:22:55 -04:00
Patrick Cloke
3555790b27
Remove unused method. (#16435) 2023-10-05 17:42:44 -04:00
Patrick Cloke
fa907025f4
Remove manys calls to cursor_to_dict (#16431)
This avoids calling cursor_to_dict and then immediately
unpacking the values in the dict for other users. By not
creating the intermediate dictionary we can avoid allocating
the dictionary and strings for the keys, which should generally
be more performant.

Additionally this improves type hints by avoid Dict[str, Any]
dictionaries coming out of the database layer.
2023-10-05 11:07:38 -04:00
Patrick Cloke
4e302b30b6
Add __slots__ to replication commands. (#16429)
To slightly reduce the amount of memory each command takes.
2023-10-05 07:38:55 -04:00
Erik Johnston
009b47badf
Factor out MultiWriter token from RoomStreamToken (#16427) 2023-10-05 10:46:28 +01:00
Erik Johnston
80ec81dcc5
Some refactors around receipts stream (#16426) 2023-10-04 16:28:40 +01:00
David Robertson
1026776380
mypy plugin to check @cached return types (#14911)
Co-authored-by: David Robertson <davidr@element.io>
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>

Assert that the return type of callables wrapped in @cached
and @cachedList are cachable (aka immutable).
2023-10-02 14:22:36 +00:00
MomentQYC
5725712d47
Remove Python version from /_synapse/admin/v1/server_version (#16380)
There's no reason to expose the full Python version over what is
frequently a public API.
2023-10-02 09:07:53 -04:00
Erik Johnston
20fb08ec80
Downgrade repl stream time out error to warning (#16401)
This is because if a worker reaches ~100% CPU then everything starts
lagging and we hit the log line a lot. When at error we invoke sentry
and that has a lot of overhead, which then puts even more pressure on
the worker.
2023-09-29 11:52:48 +00:00
Will Hunt
79eb6c0cdc
Support rendering some media downloads as inline (#15988)
Use an `inline` Content-Disposition header when the media is
"safe" to display inline (some known text, image, video, audio
formats).
2023-09-29 07:19:38 -04:00
Patrick Cloke
cdb89dcefe
Improve state types. (#16395) 2023-09-28 07:01:46 -04:00
Patrick Cloke
f84da3c32e
Add a cache around server ACL checking (#16360)
* Pre-compiles the server ACLs onto an object per room and
  invalidates them when new events come in.
* Converts the server ACL checking into Rust.
2023-09-26 11:57:50 -04:00
Patrick Cloke
17800a0e97
Implement MSC4028: push all encrypted events. (#16361)
This unstable push rule is implemented behind an experimental
configuration flag.
2023-09-26 11:52:19 -04:00
Erik Johnston
47ffc7e548
Reduce calls to send_presence_to_destinations (#16385) 2023-09-26 11:49:44 +01:00
Patrick Cloke
2763c49eca
Improve comments in StateGroupBackgroundUpdateStore. (#16383) 2023-09-25 18:50:47 +00:00
dependabot[bot]
b225acf3e6
Bump types-psycopg2 from 2.9.21.11 to 2.9.21.14 (#16381)
* Bump types-psycopg2 from 2.9.21.11 to 2.9.21.14

Bumps [types-psycopg2](https://github.com/python/typeshed) from 2.9.21.11 to 2.9.21.14.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-psycopg2
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Relax the annotation of Cursor.description

See
https://github.com/matrix-org/synapse/pull/16343#issuecomment-1726083384
for rationale.

* Changelog

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Robertson <davidr@element.io>
2023-09-25 17:48:42 +01:00
Maxwell G
12611bfcdd
Add support for pydantic v2 via pydantic.v1 compat module (#16332)
While maintaining support with pydantic v1.
2023-09-25 15:19:08 +00:00
Patrick Cloke
c9a0e1673a
Add missing license header. (#16359) 2023-09-21 09:47:29 -04:00
Patrick Cloke
7ec0a141b4
Convert more cached return values to immutable types (#16356) 2023-09-20 07:48:55 -04:00
Patrick Cloke
d7c89c5908
Return immutable objects for cachedList decorators (#16350) 2023-09-19 15:26:44 -04:00
dependabot[bot]
eee2b6642d
Bump ruff from 0.0.286 to 0.0.290 (#16342)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-09-18 11:30:43 -04:00
Hanadi
eef2b9e344
Filter locked users in the admin API (#16328)
Co-authored-by: Hanadi Tamimi <hanadi.tamimi@sdui.de>
2023-09-18 15:37:51 +01:00
Patrick Cloke
c1e244c8f7
Make cached account data/tags/admin types immutable (#16325) 2023-09-18 09:55:04 -04:00
Patrick Cloke
85bfd4735e
Return an immutable value from get_latest_event_ids_in_room. (#16326) 2023-09-18 09:29:05 -04:00
Patrick Cloke
63d28a88c1
Additional validation of receipts (#16327)
Reject invalid receipts with a reasonable error message &
expands tests for receipts.
2023-09-18 13:02:12 +00:00
José Joaquín Atria
6946209e67
Set email charset as utf-8 rather than utf8 (#16329) 2023-09-18 07:32:01 -04:00
Mathieu Velten
dd44ee00b6
Add automatic purge after all users forget a room (#15488)
Also add restore of purge/shutdown rooms after a synapse restart.

Co-authored-by:  Eric Eastwood <erice@matrix.org>
Co-authored-by: Erik Johnston <erikj@matrix.org>
2023-09-15 15:37:44 +02:00
Jason Little
2a0f86f88f
Convert _insert_graph_receipts_txn to simple_upsert (#16299) 2023-09-15 09:16:45 +01:00
Erik Johnston
edec0b93ca
Only use literal strings for process names (#16315) 2023-09-15 09:10:24 +01:00
Erik Johnston
329597022e
Some minor performance fixes for task schedular (#16313) 2023-09-14 16:20:47 +01:00
Erik Johnston
e9e2904eb2
Speed up deleting to-device messages task (#16318) 2023-09-14 14:56:07 +01:00
Erik Johnston
954921736b
Refactor get_user_by_id (#16316) 2023-09-14 12:46:30 +01:00
Erik Johnston
032cf84f52
Remove a reference cycle in background process (#16314) 2023-09-13 16:17:06 +01:00
Hanadi
7afb5e0410
Fix using dehydrated devices (MSC2697) & refresh tokens (#16288)
Refresh tokens were not correctly moved to the rehydrated
device (similar to how the access token is currently handled).
This resulted in invalid refresh tokens after rehydration.
2023-09-13 08:33:39 -04:00
Patrick Cloke
d38d0dffc9
Use StrCollection in additional places. (#16301) 2023-09-13 07:57:19 -04:00
Erik Johnston
e9addf6a01
Don't schedule an async task on every sync (#16312) 2023-09-13 11:59:44 +01:00
Erik Johnston
be3c7b08a3
Fix deleting device inbox when using background worker (#16311)
Introduced in #16240

The action for the task was only defined on the "master" handler, rather than the base worker one.
2023-09-13 11:54:16 +01:00
Erik Johnston
ab13fb08bf
Improve logging of replication (#16309) 2023-09-13 09:51:50 +00:00
Patrick Cloke
ba48c563c9
Bump mypy from 1.4.1 to 1.5.1. (#16300) 2023-09-12 07:16:09 -04:00
Patrick Cloke
16ef6f1e3c
Stop purging tables which are slated for removal. (#16273) 2023-09-12 07:12:31 -04:00
Erik Johnston
2b35626b6b
Refactor storing of server keys (#16261) 2023-09-12 11:08:04 +01:00
Patrick Cloke
9400dc0535
Add the List-Unsubscribe header for notification emails. (#16274)
Adds both the List-Unsubscribe (RFC2369) and List-Unsubscribe-Post (RFC8058)
headers to push notification emails, which together should:

* Show an "Unsubscribe" link in the MUA UI when viewing Synapse notification emails.
* Enable "one-click" unsubscribe (the user never leaves their MUA, which automatically
  makes a POST request to the specified endpoint).
2023-09-11 09:49:48 -04:00
Erik Johnston
151e4bbc45
Filter out down hosts when retrying fetching device lists (#16298) 2023-09-11 13:11:02 +01:00
David Robertson
edd83f23b7
Improve type hints for attrs classes (#16276) 2023-09-08 19:29:38 +01:00
Patrick Cloke
aa483cb4c9
Update ruff config (#16283)
Enable additional checks & clean-up unneeded configuration.
2023-09-08 11:24:36 -04:00
David Robertson
c1c6c95d72
Log values at DEBUG level with execute_values (#16281) 2023-09-08 14:50:13 +01:00
Erik Johnston
f43d994624
Fix bug with new task scheduler using lots of CPU. (#16278)
Using the new `TaskScheduler` meant that we'ed create lots of new
metrics (due to adding task ID to the desc of background process),
resulting in requests for metrics taking an increasing amount of CPU.
2023-09-08 14:43:01 +01:00
Patrick Cloke
69b74d9330
Avoid temporary storage of sensitive information. (#16272)
During the UI auth process, avoid storing sensitive information
into the database.
2023-09-08 08:57:56 -04:00
Erik Johnston
d23c394669
Reduce CPU overhead of change password endpoint (#16264) 2023-09-08 13:06:00 +01:00
Erik Johnston
1cd410a783
Recheck if remote device is cached before requesting it (#16252)
This fixes a bug where we could get stuck re-requesting the device over
replication again and again.
2023-09-07 12:45:43 +00:00
Erik Johnston
8940d1b28e
Add /notifications endpoint to workers (#16265) 2023-09-07 09:26:07 +00:00
Marcel
13e9cad537
Send the opentracing span information to appservices (#16227) 2023-09-06 15:19:17 -04:00
Aurélien Grimpard
fe69e7f617
Handle "registration_enabled" parameter for CAS (#16262)
Similar to OIDC, CAS providers can now disable registration such
that only existing users are able to login via SSO.
2023-09-06 14:32:24 -04:00
Patrick Cloke
32fb264120 Merge remote-tracking branch 'origin/release-v1.92' into develop 2023-09-06 13:08:22 -04:00
Patrick Cloke
55c20da4a3 Merge remote-tracking branch 'origin/release-v1.91' into release-v1.92 2023-09-06 11:25:28 -04:00
Quentin Gliech
1940d990a3
Revert MSC3861 introspection cache, admin impersonation and account lock (#16258) 2023-09-06 15:19:51 +01:00
reivilibre
e937e2111a
Add the ability to use G (GiB) and T (TiB) suffixes in configuration options that refer to numbers of bytes. (#16219)
* Add more suffixes to `parse_size`

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-09-06 14:01:10 +01:00
reivilibre
698f6fa250
Allow modules to delete rooms. (#15997)
* Allow user_id to be optional for room deletion

* Add module API method to delete a room

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Don't worry about the case block=True && requester_user_id is None

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-09-06 11:50:07 +01:00
Mathieu Velten
4f1840a88a
Delete device messages asynchronously and in staged batches (#16240) 2023-09-06 09:30:53 +02:00
Will Hunt
1e571cd664
Fix appservices being unable to handle to_device messages for multiple users (#16251) 2023-09-05 15:46:57 -04:00