Commit Graph

5027 Commits

Author SHA1 Message Date
Patrick Cloke
9f514dd0fb
Use _invalidate_cache_and_stream_bulk in more places. (#16616)
This takes advantage of the new bulk method in more places to
invalidate caches for many keys at once (and then to stream that
over replication).
2023-11-09 14:40:30 -05:00
Patrick Cloke
ab3f1b3b53
Convert simple_select_one_txn and simple_select_one to return tuples. (#16612) 2023-11-09 11:13:31 -05:00
Patrick Cloke
ff716b483b
Return attrs for more media repo APIs. (#16611) 2023-11-09 11:00:30 -05:00
David Robertson
91587d4cf9
Bulk-invalidate e2e cached queries after claiming keys (#16613)
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-11-09 15:57:09 +00:00
Patrick Cloke
455ef04187
Avoid updating the same rows multiple times with simple_update_many_txn. (#16609)
simple_update_many_txn had a bug in it which would cause each
update to be applied twice.
2023-11-07 14:02:09 -05:00
Patrick Cloke
9738b1c497
Avoid executing no-op queries. (#16583)
If simple_{insert,upsert,update}_many_txn is called without any data
to modify then return instead of executing the query.

This matches the behavior of simple_{select,delete}_many_txn.
2023-11-07 14:00:25 -05:00
Patrick Cloke
ec9ff389f4
More tests for the simple_* methods. (#16596)
Expand tests for the simple_* database methods, additionally
test against both PostgreSQL and SQLite variants.
2023-11-07 09:34:23 -05:00
Patrick Cloke
7e5d3b06fa
Collect information for PushRuleEvaluator in parallel. (#16590)
Fetch information needed for push rule evaluation in parallel.
Ideally this would use query pipelining, but this is not
available in psycopg2.

Due to the database thread pool this may result in little
to no parallelization.
2023-11-06 15:41:57 -05:00
Patrick Cloke
92828a7f95
Simplify event persistence code (#16584)
The event persistence code used to handle multiple rooms
at a time, but was simplified to only ever be called with a
single room at a time (different rooms are now handled in
parallel). The code is still generic to multiple rooms causing
a lot of work that is unnecessary (e.g. unnecessary loops, and
partitioning data by room).

This strips out the ability to handle multiple rooms at once, greatly
simplifying the code.
2023-11-03 07:30:31 -04:00
Patrick Cloke
0afbef30cf
Use simple_select_many_txn in event persistance code. (#16585)
Just to standardize on the normal helpers, it might also have
a slight perf improvement on PostgreSQL which will now use
`ANY (?)` instead of `IN (?, ?, ...)`.
2023-11-02 09:41:00 -04:00
Patrick Cloke
cfb6d38c47
Remove remaining usage of cursor_to_dict. (#16564) 2023-10-31 13:13:28 -04:00
David Robertson
de981ae567
Claim local one-time-keys in bulk (#16565)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2023-10-30 21:25:21 +00:00
David Robertson
fdce83ee60
Claim fallback keys in bulk (#16570) 2023-10-30 14:34:37 +00:00
Patrick Cloke
2bf9341406
Ensure local invited & knocking users leave before purge. (#16559)
This is mostly useful for federated rooms where some users
would get stuck in the invite or knock state when the room
was purged from their homeserver.
2023-10-27 12:50:50 -04:00
Patrick Cloke
679c691f6f
Remove more usages of cursor_to_dict. (#16551)
Mostly to improve type safety.
2023-10-26 15:12:28 -04:00
Patrick Cloke
9407d5ba78
Convert simple_select_list and simple_select_list_txn to return lists of tuples (#16505)
This should use fewer allocations and improves type hints.
2023-10-26 13:01:36 -04:00
Erik Johnston
ba47fea528
Allow multiple workers to write to receipts stream. (#16432)
Fixes #16417
2023-10-25 16:16:19 +01:00
Erik Johnston
8f35f8148e
Fix bug where a new writer advances their token too quickly (#16473)
* Fix bug where a new writer advances their token too quickly

When starting a new writer (for e.g. persisting events), the
`MultiWriterIdGenerator` doesn't have a minimum token for it as there
are no rows matching that new writer in the DB.

This results in the the first stream ID it acquired being announced as
persisted *before* it actually finishes persisting, if another writer
gets and persists a subsequent stream ID. This is due to the logic of
setting the minimum persisted position to the minimum known position of
across all writers, and the new writer starts off not being considered.

* Fix sending out POSITIONs when our token advances without update

Broke in #14820

* For replication HTTP requests, only wait for minimal position
2023-10-23 16:57:30 +01:00
Erik Johnston
3bc23cc45c
Fix bug that could cause a /sync to tightloop with sqlite after restart (#16540)
This could happen if the last rows in the account data stream were inserted into `account_data`. After a restart the max account ID would be calculated without looking at the `account_data` table, and so have an old ID.
2023-10-23 13:39:25 +00:00
Patrick Cloke
12ca87f5ea
Remove the last reference to event_txn_id. (#16521)
This table was no longer used, except for a background process
which purged old entries in it.
2023-10-23 07:37:45 -04:00
Erik Johnston
e9069c9f91
Mark sync as limited if there is a gap in the timeline (#16485)
This splits thinsg into two queries, but most of the time we won't have
new event backwards extremities so this shouldn't actually add an extra
RTT for the majority of cases.

Note this removes the check for events with no prev events, but that was
part of MSC2716 work that has since been removed.
2023-10-19 15:04:18 +01:00
Patrick Cloke
49c9745b45
Avoid sending massive replication updates when purging a room. (#16510) 2023-10-18 12:26:01 -04:00
Mathieu Velten
bcff01b406
Improve performance of delete device messages query (#16492) 2023-10-18 16:42:01 +01:00
Patrick Cloke
6ad1f9eac2
Convert DeviceLastConnectionInfo to attrs. (#16507)
To improve type safety & memory usage.
2023-10-17 12:47:42 +00:00
Patrick Cloke
e3e0ae4ab1
Convert state delta processing from a dict to attrs. (#16469)
For improved type checking & memory usage.
2023-10-16 07:35:22 -04:00
Mathieu Velten
eee6474bce
Remove useless async job to delete device messages on sync (#16491) 2023-10-16 11:06:27 +01:00
Patrick Cloke
4cc729d480
Revert "Drop unused tables & unneeded access token ID for events. (#16268)" (#16465)
This reverts commit cabd577460.

There are additional usages of these tables
which need to be removed first.
2023-10-12 08:56:10 -04:00
Patrick Cloke
cc865fffc0
Convert user_get_threepids response to attrs. (#16468)
This improves type annotations by not having a dictionary of Any values.
2023-10-11 20:08:11 -04:00
Patrick Cloke
a4904dcb04
Convert simple_select_many_batch, simple_select_many_txn to tuples. (#16444) 2023-10-11 13:24:56 -04:00
Patrick Cloke
f1e43018b7
Inline simple_search_list/simple_search_list_txn. (#16434)
This only has a single use and is over abstracted. Inline it so that
we can improve type hints.
2023-10-10 12:16:36 -04:00
David Robertson
28fd28e92e
Add DB indices to speed up purging rooms (#16457) 2023-10-10 10:33:39 +01:00
reivilibre
8902b3031d
Disable statement timeout whilst purging rooms (#16455)
* Disable statement timeout whilst purging rooms

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Note the introduction version

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-10-09 15:41:17 +01:00
Patrick Cloke
06bbf1029c
Convert simple_select_list_paginate_txn to return tuples. (#16433) 2023-10-06 11:41:57 -04:00
Patrick Cloke
7615e2bf48
Return ThumbnailInfo in more places (#16438)
Improves type hints by using concrete types instead of
dictionaries.
2023-10-06 10:12:43 -04:00
Patrick Cloke
cabd577460
Drop unused tables & unneeded access token ID for events. (#16268)
Drop the event_txn_id table and the tables related to MSC2716,
which is no longer supported in Synapse.
2023-10-06 08:29:33 -04:00
Patrick Cloke
3555790b27
Remove unused method. (#16435) 2023-10-05 17:42:44 -04:00
Patrick Cloke
fa907025f4
Remove manys calls to cursor_to_dict (#16431)
This avoids calling cursor_to_dict and then immediately
unpacking the values in the dict for other users. By not
creating the intermediate dictionary we can avoid allocating
the dictionary and strings for the keys, which should generally
be more performant.

Additionally this improves type hints by avoid Dict[str, Any]
dictionaries coming out of the database layer.
2023-10-05 11:07:38 -04:00
Erik Johnston
009b47badf
Factor out MultiWriter token from RoomStreamToken (#16427) 2023-10-05 10:46:28 +01:00
Erik Johnston
80ec81dcc5
Some refactors around receipts stream (#16426) 2023-10-04 16:28:40 +01:00
David Robertson
1026776380
mypy plugin to check @cached return types (#14911)
Co-authored-by: David Robertson <davidr@element.io>
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
Co-authored-by: Erik Johnston <erik@matrix.org>

Assert that the return type of callables wrapped in @cached
and @cachedList are cachable (aka immutable).
2023-10-02 14:22:36 +00:00
Patrick Cloke
f84da3c32e
Add a cache around server ACL checking (#16360)
* Pre-compiles the server ACLs onto an object per room and
  invalidates them when new events come in.
* Converts the server ACL checking into Rust.
2023-09-26 11:57:50 -04:00
Patrick Cloke
17800a0e97
Implement MSC4028: push all encrypted events. (#16361)
This unstable push rule is implemented behind an experimental
configuration flag.
2023-09-26 11:52:19 -04:00
Patrick Cloke
2763c49eca
Improve comments in StateGroupBackgroundUpdateStore. (#16383) 2023-09-25 18:50:47 +00:00
dependabot[bot]
b225acf3e6
Bump types-psycopg2 from 2.9.21.11 to 2.9.21.14 (#16381)
* Bump types-psycopg2 from 2.9.21.11 to 2.9.21.14

Bumps [types-psycopg2](https://github.com/python/typeshed) from 2.9.21.11 to 2.9.21.14.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-psycopg2
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Relax the annotation of Cursor.description

See
https://github.com/matrix-org/synapse/pull/16343#issuecomment-1726083384
for rationale.

* Changelog

* Changelog

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: David Robertson <davidr@element.io>
2023-09-25 17:48:42 +01:00
Maxwell G
12611bfcdd
Add support for pydantic v2 via pydantic.v1 compat module (#16332)
While maintaining support with pydantic v1.
2023-09-25 15:19:08 +00:00
Patrick Cloke
7ec0a141b4
Convert more cached return values to immutable types (#16356) 2023-09-20 07:48:55 -04:00
Patrick Cloke
d7c89c5908
Return immutable objects for cachedList decorators (#16350) 2023-09-19 15:26:44 -04:00
dependabot[bot]
eee2b6642d
Bump ruff from 0.0.286 to 0.0.290 (#16342)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Patrick Cloke <patrickc@matrix.org>
2023-09-18 11:30:43 -04:00
Hanadi
eef2b9e344
Filter locked users in the admin API (#16328)
Co-authored-by: Hanadi Tamimi <hanadi.tamimi@sdui.de>
2023-09-18 15:37:51 +01:00
Patrick Cloke
c1e244c8f7
Make cached account data/tags/admin types immutable (#16325) 2023-09-18 09:55:04 -04:00