Commit Graph

135 Commits

Author SHA1 Message Date
Mathieu Velten
dac97642e4
Implements admin API to lock an user (MSC3939) (#15870) 2023-08-10 09:10:55 +00:00
Mathieu Velten
8ebfd577e2
Bump DB version to 79 since synapse v1.88 was already there (#15998) 2023-07-26 14:51:44 +02:00
Erik Johnston
19796e20aa
Fix bad merge of #15933 (#15958)
This was because we reverted the bump of the schema version, so we were not applying the new deltas.
2023-07-19 12:17:08 +00:00
Erik Johnston
40a3583ba1
Fix race in triggers for read/write locks. (#15933) 2023-07-19 12:06:38 +01:00
Shay
cb6e2c6cc7
Fix background schema updates failing over a large upgrade gap (#15887) 2023-07-18 16:59:27 -07:00
Shay
e625c3dca0
Revert "Stop writing to column user_id of tables profiles and user_filters. (#15953)
* Revert "Stop writing to column `user_id` of tables `profiles` and `user_filters` (#15787)"

This reverts commit f25b0f8808.

* newsfragement
2023-07-18 11:44:09 +01:00
Erik Johnston
e55a9b3e41
Fix downgrading to previous version of Synapse (#15907)
We do this by marking the constraint as deferrable.
2023-07-10 16:24:42 +01:00
Shay
f25b0f8808
Stop writing to column user_id of tables profiles and user_filters (#15787) 2023-07-07 09:23:27 -07:00
Erik Johnston
39d131b016
Add basic read/write lock (#15782) 2023-07-05 17:25:00 +01:00
Erik Johnston
95a96b21eb
Add foreign key constraint to event_forward_extremities. (#15751) 2023-07-05 09:43:19 +00:00
Shay
78cfa55dad
Fix sqlite user_filters upgrade (#15817) 2023-06-27 09:41:42 +01:00
Erik Johnston
8934c11935 Merge branch 'master' into develop 2023-06-07 14:45:19 +01:00
Erik Johnston
f7c6553ebc
Fix schema delta error in 1.85 (#15739)
Some users seem to have multiple rows per user / room with a null thread
ID, which we need to handle.
2023-06-07 13:02:42 +01:00
Erik Johnston
a701c089fa
Fix schema delta error in 1.85 (#15738)
There appears to be a race where you can end up with entries in
`event_push_summary` with both a `NULL` and `main` thread ID.

Fixes #15736

Introduced in #15597
2023-06-07 10:50:32 +01:00
Shay
d0c4257f14
N + 3: Read from column full_user_id rather than user_id of tables profiles and user_filters (#15649) 2023-06-02 17:24:13 -07:00
reivilibre
11e15d79b8
Fix a performance issue introduced in Synapse v1.83.0 which meant that purging rooms was very slow and database-intensive. (#15693)
* Add indices required to efficiently validate new foreign key constraints on stream_ordering

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-05-31 14:59:56 +01:00
Patrick Cloke
2ad91ec628
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15597)
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.

For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.

Each table is updated as a separate schema delta to avoid
deadlocks between them.

For SQLite we simply rebuild the table & copy the data.
2023-05-26 13:16:08 -04:00
Patrick Cloke
375b0a8a11
Update code to refer to "workers". (#15606)
A bunch of comments and variables are out of date and use
obsolete terms.
2023-05-16 15:56:38 -04:00
Shay
9f6ff6a0eb
Add not null constraint to column full_user_id of tables profiles and user_filters (#15537) 2023-05-16 10:57:39 -07:00
Erik Johnston
808105bd31
Revert "Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15437)" (#15580)
This reverts commit a7b3e9ce65.
2023-05-12 11:38:16 +01:00
Patrick Cloke
a7b3e9ce65
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15437)
Updates the database schema to require a thread_id (by adding a
constraint that the column is non-null) for event_push_actions,
event_push_actions_staging, and event_push_actions_summary.

For PostgreSQL we add the constraint as NOT VALID, then
VALIDATE the constraint a background job to avoid locking
the table during an upgrade.

For SQLite we simply rebuild the table & copy the data.
2023-05-03 07:49:03 -04:00
Sean Quah
04e79e6a18
Add config option to forget rooms automatically when users leave them (#15224)
This is largely based off the stats and user directory updater code.

Signed-off-by: Sean Quah <seanq@matrix.org>
2023-05-03 12:27:33 +01:00
Shay
89f6fb0d5a
Add an admin API endpoint to support per-user feature flags (#15344) 2023-04-28 11:33:45 -07:00
Patrick Cloke
6efa674004
Add type hints to schema deltas (#15497)
Cleans-up the schema delta files:

* Removes no-op functions.
* Adds missing type hints to function parameters.
* Fixes any issues with type hints.

This also renames one (very old) schema delta to avoid a conflict
that mypy complains about.
2023-04-27 12:44:53 +00:00
Shay
301b4156d5
Add column full_user_id to tables profiles and user_filters. (#15458) 2023-04-26 16:03:26 -07:00
Nick Mills-Barrett
c55293c230
Re re introduce membership tables event stream ordering (#15356) 2023-04-25 09:44:29 +01:00
Quentin Gliech
8b3a502996
Experimental support for MSC3970: per-device transaction IDs (#15318) 2023-04-25 09:37:09 +01:00
Shay
6b23d74ad1
Delete server-side backup keys when deactivating an account. (#15181) 2023-04-04 20:16:08 +00:00
Erik Johnston
79d2e2e79c
Speed up membership queries for users with forgotten rooms (#15385) 2023-04-04 14:11:34 +01:00
Olivier Wilkinson (reivilibre)
72d2ceaa9a Revert "Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15350)"
This reverts commit 2a234b788e.

See #15359 for context.
2023-03-31 12:10:10 +01:00
Patrick Cloke
2a234b788e
Set thread_id column to non-null for event_push_{actions,actions_staging,summary} (#15350)
Clean-up from adding the thread_id column, which was initially
null but backfilled with values. It is desirable to require it to now
be non-null.

In addition to altering this column to be non-null, we clean up
obsolete background jobs, indexes, and just-in-time updating
code.
2023-03-30 15:11:31 -04:00
Erik Johnston
5350b5d04d
Revert "Reintroduce membership tables event stream ordering (#15128)" (#15347)
This reverts commit e6af49fbea.
2023-03-29 13:24:28 +01:00
Quentin Gliech
5b70f240cf
Make cleaning up pushers depend on the device_id instead of the token_id (#15280)
This makes it so that we rely on the `device_id` to delete pushers on logout,
instead of relying on the `access_token_id`. This ensures we're not removing
pushers on token refresh, and prepares for a world without access token IDs
(also known as the OIDC).

This actually runs the `set_device_id_for_pushers` background update, which
was forgotten in #13831.

Note that for backwards compatibility it still deletes pushers based on the
`access_token` until the background update finishes.
2023-03-24 11:09:39 -04:00
Nick Mills-Barrett
e6af49fbea
Reintroduce membership tables event stream ordering (#15128)
* Add `event_stream_ordering` column to membership state tables

Specifically this adds the column to `current_state_events`,
`local_current_membership` and `room_memberships`. Each of these tables
is regularly joined with the `events` table to get the stream ordering
and denormalising this into each table will yield significant query
performance improvements once used.

* Make denormalised `event_stream_ordering` columns foreign keys
* Add comment in schema file explaining new denormalised columns
* Add triggers to enforce consistency of `event_stream_ordering` columns
* Re-order purge room tables to account for foreign keys
* Bump schema version to 75

Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2023-03-24 11:44:01 +00:00
reivilibre
f54f877f27
Preparatory work to fix the user directory assuming that any remote membership state events represent a profile change. [rei:userdirpriv] (#14755)
* Remove special-case method for new memberships only, use more generic method

* Only collect profiles from state events in public rooms

* Add a table to track stale remote user profiles

* Add store methods to set and delete rows in this new table

* Mark remote profiles as stale when a member state event comes in to a private room

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

* Simplify by removing Optionality of `event_id`

* Replace names and avatars with None if they're set to dodgy things

I think this makes more sense anyway.

* Move schema delta to 74 (I missed the boat?)

* Turns out these can be None after all

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-03-16 09:55:19 +00:00
reivilibre
63d87c08c8
Add schema comments about the destinations and destination_rooms tables. (#15247) 2023-03-15 09:25:58 +00:00
David Robertson
9cd7610f86
Revert "Add event_stream_ordering column to membership state tables (#14979)"
This reverts commit 5fdc12f482.
2023-02-07 15:26:55 +00:00
Nick Mills-Barrett
5fdc12f482
Add event_stream_ordering column to membership state tables (#14979)
This adds an `event_stream_ordering` column to `current_state_events`,
`local_current_membership` and `room_memberships`. Each of these tables
is regularly joined with the `events` table to get the stream ordering
and denormalising this into each table will yield significant query
performance improvements once used. Includes a background job to
populate these values from the `events` table.

Same idea as https://github.com/matrix-org/synapse/pull/13703.

Signed off by Nick @ Beeper (@fizzadar).
2023-02-07 00:10:54 +00:00
Dirk Klimpel
8d5325ec0c
Drop unused table presence (#14825) 2023-01-13 14:17:03 +00:00
reivilibre
d6bda5addd
Add index to improve performance of the /timestamp_to_event endpoint used for jumping to a specific date in the timeline of a room. (#14799) 2023-01-11 12:29:13 +00:00
reivilibre
fb60cb16fe
Faster remote room joins: stream the un-partial-stating of events over replication. [rei:frrj/streams/unpsr] (#14545) 2022-12-14 14:47:11 +00:00
Patrick Cloke
24a97b3e71
Delete event_push_summary_unique_index again. (#14669)
if a Synapse deployment upgraded (from < 1.62.0 to >= 1.70.0) then it
is possible for schema deltas to run before background updates causing
drift in the database schema due to:

1. A delta registered a background update to create an index.
2. A delta dropped the above index if it exists (but it yet exist won't since
  the background job hasn't run).
3. The code assumed the index was dropped.

To fix this we:

1. Cancel the background update which could create the index.
2. Drop the index again.
3. Drop a related index which is dropped by the background update.
2022-12-14 09:25:33 -05:00
Patrick Cloke
c369e95691
Rebuild the user directory and stats tables. (#14643)
Due to the various fixes to the StreamChangeCache it is not
safe to trust the information in the user directory or room/user
stats tables. Rebuild them as background jobs.

In particular see da77720752 (#14639),
and 6a8310f3df (#14435).

Maybe also be related to fac8a38525
(#14592).
2022-12-08 11:40:20 -05:00
reivilibre
501f62d1a6
Faster remote room joins: stream the un-partial-stating of rooms over replication. [rei:frrj/streams/unpsr] (#14473) 2022-12-05 13:07:55 +00:00
Erik Johnston
f38d7d79c8
Add another index to device_lists_changes_in_room (#14534)
This helps avoid reading unnecessarily large amounts of data from the
table when querying with a set of room IDs.
2022-11-23 14:09:00 +00:00
Sean Quah
9cae44f49e
Track unconverted device list outbound pokes using a position instead (#14516)
When a local device list change is added to
`device_lists_changes_in_room`, the `converted_to_destinations` flag is
set to `FALSE` and the `_handle_new_device_update_async` background
process is started. This background process looks for unconverted rows
in `device_lists_changes_in_room`, copies them to
`device_lists_outbound_pokes` and updates the flag.

To update the `converted_to_destinations` flag, the database performs a
`DELETE` and `INSERT` internally, which fragments the table. To avoid
this, track unconverted rows using a `(stream ID, room ID)` position
instead of the flag.

From now on, the `converted_to_destinations` column indicates rows that
need converting to outbound pokes, but does not indicate whether the
conversion has already taken place.

Closes #14037.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-22 16:46:52 +00:00
Sean Quah
b2c2b03079
Fix PostgreSQL sometimes using table scans for event_search (#14409)
PostgreSQL may underestimate the number of distinct `room_id`s in
`event_search`, which can cause it to use table scans for queries for
multiple rooms.

Fix this by setting `n_distinct` on the column.

Resolves #14402.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-11-10 19:02:27 +00:00
Quentin Gliech
8756d5c87e
Save login tokens in database (#13844)
* Save login tokens in database

Signed-off-by: Quentin Gliech <quenting@element.io>

* Add upgrade notes

* Track login token reuse in a Prometheus metric

Signed-off-by: Quentin Gliech <quenting@element.io>
2022-10-26 11:45:41 +01:00
James Salter
d902181de9
Unified search query syntax using the full-text search capabilities of the underlying DB. (#11635)
Support a unified search query syntax which leverages more of the full-text
search of each database supported by Synapse.

Supports, with the same syntax across Postgresql 11+ and Sqlite:

- quoted "search terms"
- `AND`, `OR`, `-` (negation) operators
- Matching words based on their stem, e.g. searches for "dog" matches
  documents containing "dogs". 

This is achieved by 

- If on postgresql 11+, pass the user input to `websearch_to_tsquery`
- If on sqlite, manually parse the query and transform it into the sqlite-specific
  query syntax.

Note that postgresql 10, which is close to end-of-life, falls back to using
`phraseto_tsquery`, which only supports a subset of the features.

Multiple terms separated by a space are implicitly ANDed.

Note that:

1. There is no escaping of full-text syntax that might be supported by the database;
  e.g. `NOT`, `NEAR`, `*` in sqlite. This runs the risk that people might discover this
  as accidental functionality and depend on something we don't guarantee.
2. English text is assumed for stemming. To support other languages, either the target
  language needs to be known at the time of indexing the message (via room metadata,
  or otherwise), or a separate index for each language supported could be created.

Sqlite docs: https://www.sqlite.org/fts3.html#full_text_index_queries
Postgres docs: https://www.postgresql.org/docs/11/textsearch-controls.html
2022-10-25 14:05:22 -04:00
Patrick Cloke
dbf18f514e
Update the thread_id right before use (in case the bg update hasn't finished) (#14222)
This avoids running a forced-update of a null thread_id rows.

An index is added (in the background) to hopefully make this
easier in the future.
2022-10-18 14:55:41 +00:00