Commit Graph

1169 Commits

Author SHA1 Message Date
Erik Johnston
5624c8b961
In sync wait for worker to catch up since token (#17215)
Otherwise things will get confused.

An alternative would be to make sure that for lagging stream we don't
return anything (and make sure the returned next_batch token doesn't go
backwards). But that is a faff.
2024-05-30 14:03:49 +01:00
Erik Johnston
d16910ca02
Replaces all usages of StreamIdGenerator with MultiWriterIdGenerator (#17229)
Replaces all usages of `StreamIdGenerator` with `MultiWriterIdGenerator`, which is safer.
2024-05-30 11:07:32 +00:00
Erik Johnston
225f378ffa
Clean out invalid destinations from outbox (#17242)
We started ensuring we only insert valid destinations:
https://github.com/element-hq/synapse/pull/17240
2024-05-30 11:25:24 +01:00
Erik Johnston
466f344547
Move towards using MultiWriterIdGenerator everywhere (#17226)
There is a problem with `StreamIdGenerator` where it can go backwards
over restarts when a stream ID is requested but then not inserted into
the DB. This is problematic if we want to land #17215, and is generally
a potential cause for all sorts of nastiness.

Instead of trying to fix `StreamIdGenerator`, we may as well move to
`MultiWriterIdGenerator` that does not suffer from this problem (the
latest positions are stored in `stream_positions` table). This involves
adding SQLite support to the class.

This only changes id generators that were already using
`MultiWriterIdGenerator` under postgres, a separate PR will move the
rest of the uses of `StreamIdGenerator` over.
2024-05-29 12:19:10 +00:00
Erik Johnston
726006cdf2
Don't invalidate all get_relations_for_event on history purge (#17083)
This is a tree cache already, so may as well move the room ID to the
front and use that
2024-05-29 12:57:10 +01:00
Erik Johnston
b71d277438
Reduce work of calculating outbound device pokes (#17211) 2024-05-22 13:55:18 +01:00
Erik Johnston
b5facbac0f
Improve perf of sync device lists (#17216)
Re-introduces #17191, and includes #17197 and #17214

The basic idea is to stop calling `get_rooms_for_user` everywhere, and
instead use the table `device_lists_changes_in_room`.

Commits reviewable one-by-one.
2024-05-21 16:48:20 +01:00
Erik Johnston
fd12003441
Revert "Improve perf of sync device lists" (#17207)
Reverts element-hq/synapse#17191
2024-05-16 16:07:54 +01:00
Erik Johnston
0b91ccce47
Improve perf of sync device lists (#17191)
It's almost always more efficient to query the rooms that have device
list changes, rather than looking at the list of all users whose devices
have changed and then look for shared rooms.
2024-05-14 14:39:04 +01:00
Erik Johnston
59ac541310
Actually fix public rooms (#17184)
See #17177.

I'm an idiot and moved them to the wrong store 🤦
2024-05-13 13:11:07 +01:00
Erik Johnston
a2e6f43f11
Fix bug with creating public rooms on workers (#17177)
If room publication is disabled then creating public rooms on workers
would not work.

Introduced in #16811.
2024-05-13 12:12:26 +01:00
Shay
37558d5e4c
Add support for MSC3823 - Account Suspension (#17051) 2024-05-01 17:45:17 +01:00
Andrew Morgan
89fc579329
Fix filtering of rooms when supplying the destination query parameter to /_synapse/admin/v1/federation/destinations/<destination>/rooms (#17077) 2024-04-26 10:52:24 +01:00
Melvyn Laïly
59710437e4
Return the search terms as search highlights for SQLite instead of nothing (#17000)
Fixes https://github.com/element-hq/synapse/issues/16999 and
https://github.com/element-hq/element-android/pull/8729 by returning the
search terms as search highlights.
2024-04-26 09:43:52 +01:00
Till
47773232b0
Redact membership events if the user requested erasure upon deactivating (#17076)
Fixes #15355 by redacting all membership events before leaving rooms.
2024-04-25 14:25:31 +01:00
Andrew Morgan
646cb6ff24
Add type annotation to visited_chains (#17125)
This should fix CI on `develop`. Broke in
0fe9e1f7da,
presumably due to a `mypy` dependency upgrade.
2024-04-25 12:25:26 +00:00
Erik Johnston
0fe9e1f7da Merge branch 'master' into develop 2024-04-23 17:06:52 +01:00
Erik Johnston
55b0aa847a Fix GHSA-3h7q-rfh9-xm4v
Weakness in auth chain indexing allows DoS from remote room members
through disk fill and high CPU usage.

A remote Matrix user with malicious intent, sharing a room with Synapse
instances before 1.104.1, can dispatch specially crafted events to
exploit a weakness in how the auth chain cover index is calculated. This
can induce high CPU consumption and accumulate excessive data in the
database of such instances, resulting in a denial of service.

Servers in private federations, or those that do not federate, are not
affected.
2024-04-23 15:25:49 +01:00
Erik Johnston
803f05f60c
Fix remote receipts for events we don't have (#17096)
Introduced in #17032
2024-04-17 16:08:40 +01:00
Nick Mills-Barrett
fe4719a268
Use receipts event_stream_ordering instead of joins (#17032)
Resurrecting https://github.com/matrix-org/synapse/pull/13918.

This should reduce IOPs incurred by joining to the events table to
lookup stream ordering, which happens in many receipt handling code
paths. Like the previous PR I believe sufficient time has passed between
the original migration in DB schema 72 and now to merge this as-is. It's
highly unlikely that both the migration is still ongoing AND (active)
users still have any receipts prior to that date.

In the unlikely event there is a receipt without a populated
`event_stream_ordering` synapse will behave just as it does now when
receipts exist for events that don't (yet): for push action calculation
the receipts are just ignored.

I've removed the validation on event IDs as this is already covered
here:

59ceabcb97/synapse/handlers/receipts.py (L189-L192)
2024-04-12 09:28:44 +01:00
Erik Johnston
d40878451c
Add forgotten schema delta (#17054)
This should have been in #17045. Whoops.
2024-04-09 13:03:41 +01:00
Erik Johnston
5360baeb64
Pull out fewer receipts from DB when doing push (#17049)
Before we were pulling out *all* read receipts for a user for every
event we pushed. Instead let's only pull out the relevant receipts.

This also pulled out the event rows for each receipt, causing load on
the events table.
2024-04-05 12:46:34 +01:00
Erik Johnston
31122b71bc
Add missing index to access_tokens table (#17045)
This was causing sequential scans when using refresh tokens.
2024-04-04 11:05:40 +01:00
Erik Johnston
ec174d0470
Refactor chain fetching (#17044)
Since these queries are duplicated in two places.
2024-04-02 15:33:56 +01:00
Erik Johnston
fd48fc4585
Fixups to new push stream (#17038)
Follow on from #17037
2024-03-28 16:29:23 +00:00
Erik Johnston
ea6bfae0fc
Add support for moving /push_rules off of main process (#17037) 2024-03-28 15:44:07 +00:00
Richard van der Hoff
b5322b4daf
Ensure that pending to-device events are sent over federation at startup (#16925)
Fixes https://github.com/element-hq/synapse/issues/16680, as well as a
related bug, where servers which we had *never* successfully sent an
event to would not be retried.

In order to fix the case of pending to-device messages, we hook into the
existing `wake_destinations_needing_catchup` process, by extending it to
look for destinations that have pending to-device messages. The
federation transmission loop then attempts to send the pending to-device
messages as normal.
2024-03-22 13:24:11 +00:00
SpiritCroc
9ad49e7ecf
Do not refuse to set read_marker if previous event_id is in wrong room (#16990) 2024-03-21 18:43:07 +00:00
Hanadi
f7a3ebe44d
Fix reject knocks on deactivating account (#17010) 2024-03-21 18:05:54 +00:00
dependabot[bot]
e161103b46
Bump mypy from 1.5.1 to 1.8.0 (#16901) 2024-03-13 17:05:57 +00:00
dependabot[bot]
1e68b56a62
Bump black from 23.10.1 to 24.2.0 (#16936) 2024-03-13 16:46:44 +00:00
Alexander Fechler
48f59d3806
deactivated flag refactored to filter deactivated users. (#16874)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-03-11 16:08:04 +00:00
Andrew Morgan
8a05304222
Revert "Improve DB performance of calculating badge counts for push. (#16756)" (#16979) 2024-03-05 12:27:27 +00:00
kegsay
c51a2240d1
bugfix: always prefer unthreaded receipt when >1 exist (MSC4102) (#16927)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-02-20 14:12:06 +00:00
Erik Johnston
7b4d7429f8
Don't invalidate the entire event cache when we purge history (#16905)
We do this by adding support to the LRU cache for "extra indices" based
on the cached value. This allows us to efficiently map from room ID to
the cached events and only invalidate those.
2024-02-13 13:24:11 +00:00
Erik Johnston
02a147039c
Increase batching when fetching auth chains (#16893)
This basically reverts a change that was in
https://github.com/element-hq/synapse/pull/16833, where we reduced the
batching.

The smaller batching can cause performance issues on busy servers and
databases.
2024-02-09 10:51:00 +00:00
Erik Johnston
c925b45567
Speed up e2e device keys queries for bot accounts (#16841)
This helps with bot accounts with lots of non-e2e devices.

The change is basically to change the order of the join for the case of
using `INNER JOIN`
2024-01-23 11:37:16 +00:00
Erik Johnston
23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
Erik Johnston
14c725f73b
Preparatory work for tweaking performance of auth chain lookups (#16833) 2024-01-23 11:26:27 +00:00
Mo Balaa
b99f6db039
Handle wildcard type filters properly (#14984) 2024-01-22 10:46:30 +00:00
Erik Johnston
c43f751013
Optimize query for fetching to-device messages in /sync (#16805)
The current query supports passing in a list of users, which generates a
query using `user_id = ANY(..)`. This is generates a less efficient
query plan that is notably slower than a simple `user_id = ?` condition.

Note: The new function is mostly a copy and paste and then a
simplification of the existing function.
2024-01-11 13:37:57 +00:00
Erik Johnston
b11f7b5122
Improve DB performance of calculating badge counts for push. (#16756)
The crux of the change is to try and make the queries simpler and pull
out fewer rows. Before, there were quite a few joins against subqueries,
which caused postgres to pull out more rows than necessary.

Instead, let's simplify the query and do some of the filtering out in
Python instead, letting Postgres do better optimizations now that it
doesn't have to deal with joins against subqueries.

Review note: this is a complete rewrite of the function, so not sure how
useful the diff is.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-01-11 11:52:13 +00:00
Erik Johnston
cbe8a80d10
Faster load recents for sync (#16783)
This hopefully reduces the amount of state we need to keep in memory
2024-01-10 15:11:59 +00:00
Erik Johnston
578c5c736e
Reduce amount of state pulled out when querying federation hierachy (#16785)
There are two changes here:

1. Only pull out the required state when handling the request.
2. Change the get filtered state return type to check that we're only
querying state that was requested

---------

Co-authored-by: reivilibre <oliverw@matrix.org>
2024-01-10 14:31:35 +00:00
Erik Johnston
4c67f0391b
Split up deleting devices into batches (#16766)
Otherwise for users with large numbers of devices this can cause a lot
of woe.
2024-01-10 13:55:16 +00:00
Erik Johnston
5d3850b038
Port EventInternalMetadata class to Rust (#16782)
There are a couple of things we need to be careful of here:

1. The current python code does no validation when loading from the DB,
so we need to be careful to ignore such errors (at least on jki.re there
are some old events with internal metadata fields of the wrong type).
2. We want to be memory efficient, as we often have many hundreds of
thousands of events in the cache at a time.

---------

Co-authored-by: Quentin Gliech <quenting@element.io>
2024-01-08 14:06:48 +00:00
Erik Johnston
7469fa7585
Simplify internal metadata class. (#16762)
We remove these fields as they're just duplicating data the event
already stores, and (for reasons 🤫) I'd like to simplify
the class to only store simple types.

I'm not entirely convinced that we shouldn't instead add helper methods
to the event class to generate stream tokens, but I don't really think
that's where they belong either
2024-01-05 13:03:20 +00:00
Erik Johnston
eaad9bb156 Merge remote-tracking branch 'gitlab/clokep/license-license' into new_develop 2023-12-13 15:11:56 +00:00
Erik Johnston
df366966b4
Speed up pruning of user_ips table (#16667)
Silly query planner
2023-11-29 11:54:42 +00:00
David Robertson
c3627d0f99
Correctly read to-device stream pos on SQLite (#16682) 2023-11-24 13:42:38 +00:00