Commit Graph

76 Commits

Author SHA1 Message Date
Sean Quah
373c485d8c
Handle half-created indices in receipts index background update ()
When Synapse is terminated while running the background update to create
the `receipts_graph` or `receipts_linearized` indexes, the indexes may
be successfully created (or marked as invalid on postgres) while the
background update remains unfinished. When Synapse next starts up, the
background update will fail because the index already exists, or exists
but is invalid on postgres.

Use the existing code to create indices in background updates, since it
handles these edge cases.

Signed-off-by: Sean Quah <seanq@matrix.org>
2022-12-09 23:02:11 +00:00
David Robertson
fff9b955fa
Generate separate snapshots for logical databases ()
* Generate separate snapshots for sqlite, postgres and common
* Cleanup postgres dbs in the TRAP
* Say which logical DB we're applying updates to
* Run background updates on the state DB
* Add new option for accepting a SCHEMA_NUMBER
2022-09-20 14:14:12 +01:00
Patrick Cloke
b2b0c85279
Support providing an index predicate for upserts. ()
This is useful to upsert against a table which has a unique
partial index while avoiding conflicts.
2022-09-15 18:28:48 +00:00
Patrick Cloke
f2d12ccabe
Use partial indices on SQLIte. ()
Partial indices have been supported since SQLite 3.8, but Synapse
now requires >= 3.27, so we can enable support for them.

This requires rebuilding previous indices which were partial on
PostgreSQL, but not on SQLite.
2022-09-14 12:01:42 -04:00
Patrick Cloke
53b77b203a
Replace noop background updates with DELETE. ()
Removes the `register_noop_background_update` and deletes the background
updates directly in a delta file.
2022-06-13 14:06:27 -04:00
reivilibre
66a5f6c400
Add a unique index to state_group_edges to prevent duplicates being accidentally introduced and the consequential impact to performance. () 2022-05-19 14:16:49 +01:00
reivilibre
635f0d916b
Do not keep going if there are 5 back-to-back background update failures. () 2022-05-18 13:57:59 +00:00
David Robertson
aec69d2481
Another batch of type annotations () 2022-05-13 12:35:31 +01:00
Shay
ef3619e61d
Add config settings for background update parameters () 2022-03-11 10:46:45 -08:00
Shay
26211fec24
Fix a bug in background updates wherein background updates are never run using the default batch size () 2022-03-07 09:44:33 -08:00
Erik Johnston
d08ef6f155
Make background updates controllable via a plugin ()
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-11-29 17:57:06 +01:00
Brendan Abolivier
7f9841bdec
Lower minumum batch size to 1 for background updates ()
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-24 19:21:44 +00:00
Dirk Klimpel
ea20937084
Add an admin API to run background jobs. ()
Instead of having admins poke into the database directly.

Can currently run jobs to populate stats and to populate
the user directory.
2021-11-19 19:39:46 +00:00
Erik Johnston
4ee71b9637
Add some background update admin APIs ()
Fixes 
2021-11-08 16:08:02 +00:00
Jonathan de Jong
bdfde6dca1
Use inline type hints in http/federation/, storage/ and util/ () 2021-07-15 12:46:54 -04:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ()
Part of 

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. () 2021-03-23 07:12:48 -04:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase ()
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke
637282bb50
Add additional type hints to the storage module. () 2020-12-30 08:09:53 -05:00
Patrick Cloke
c619253db8
Stop sub-classing object () 2020-09-04 06:54:56 -04:00
Patrick Cloke
37db6252b7
Convert additional databases to async/await part 3 () 2020-09-01 11:04:17 -04:00
Patrick Cloke
30426c7063
Convert additional database methods to async (select list, search, insert_many, delete_*) () 2020-08-27 07:41:01 -04:00
Patrick Cloke
dbc630a628
Use the JSON encoder without whitespace in more places. () 2020-08-20 10:32:33 -04:00
Patrick Cloke
894dae74fe
Convert misc database code to async () 2020-08-14 07:24:26 -04:00
Erik Johnston
a7bdf98d01
Rename database classes to make some sense () 2020-08-05 21:38:57 +01:00
Patrick Cloke
f460da6031
Consistently use db_to_json to convert from database values to JSON objects. () 2020-07-16 11:32:19 -04:00
Richard van der Hoff
fcc2de7a0c Update docstring per review comments 2020-04-03 10:51:32 +01:00
Richard van der Hoff
af47264b78
review comment 2020-04-02 12:04:55 +01:00
Richard van der Hoff
7b608cf468 Only run one background update at a time 2020-03-31 17:43:58 +01:00
Richard van der Hoff
b4c2234232 Make do_next_background_update return a bool
returning a None or an int that we don't use is confusing.
2020-03-31 17:43:58 +01:00
Richard van der Hoff
26d17b9bdc Make has_completed_background_updates async
(Almost) everywhere that uses it is happy with an awaitable.
2020-03-31 17:43:58 +01:00
Richard van der Hoff
cfe8c8ab8e Remove unused start_background_update
This was only used in a unit test, so let's just inline it in the test.
2020-03-31 17:24:06 +01:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 ()
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Richard van der Hoff
85db7f73be
Add a background update to clear tombstoned rooms from the directory ()
* Add a background update to clear tombstoned rooms from the directory

* use the ABC metaclass
2020-01-07 14:18:43 +00:00
Richard van der Hoff
9824a39d80
Async/await for background updates ()
so that bg update routines can be async
2020-01-07 14:12:42 +00:00
Erik Johnston
d537be1ebd Pass Database into the data store 2019-12-06 15:49:44 +00:00
Erik Johnston
4a33a6dd19 Move background update handling out of store 2019-12-05 11:11:26 +00:00
Erik Johnston
756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston
ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Brendan Abolivier
14504ad573
Add CI for synapse_port_db ()
This adds:

* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)

as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:

1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step `2` is done via a new script located at `scripts-dev/update_database`.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
Erik Johnston
4fb3c129aa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2 2019-09-25 17:53:13 +01:00
Erik Johnston
39b50ad42a Review comments 2019-09-25 17:22:33 +01:00
Erik Johnston
2135c198d1 Add has_completed_background_update
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
Brendan Abolivier
f02f14e09a
Fix logging 2019-09-24 14:39:07 +01:00
Amber Brown
4806651744
Replace returnValue with return () 2019-07-23 23:00:55 +10:00
Amber Brown
32e7c9e7f2
Run Black. () 2019-06-20 19:32:02 +10:00
Amber Brown
7efd1d87c2 Run black on the rest of the storage module () 2019-04-03 10:07:29 +01:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task () 2019-03-19 04:50:24 +11:00
Andrew Morgan
25d64a846a Fix typos 2019-01-22 11:00:04 +00:00
Richard van der Hoff
667fba68f3 Run things as background processes
This fixes , and ensures that we get useful logs and metrics for lots of
things that happen in the background.

(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00