David Robertson
fff9b955fa
Generate separate snapshots for logical databases ( #13792 )
...
* Generate separate snapshots for sqlite, postgres and common
* Cleanup postgres dbs in the TRAP
* Say which logical DB we're applying updates to
* Run background updates on the state DB
* Add new option for accepting a SCHEMA_NUMBER
2022-09-20 14:14:12 +01:00
Patrick Cloke
b2b0c85279
Support providing an index predicate for upserts. ( #13822 )
...
This is useful to upsert against a table which has a unique
partial index while avoiding conflicts.
2022-09-15 18:28:48 +00:00
Patrick Cloke
f2d12ccabe
Use partial indices on SQLIte. ( #13802 )
...
Partial indices have been supported since SQLite 3.8, but Synapse
now requires >= 3.27, so we can enable support for them.
This requires rebuilding previous indices which were partial on
PostgreSQL, but not on SQLite.
2022-09-14 12:01:42 -04:00
Patrick Cloke
53b77b203a
Replace noop background updates with DELETE. ( #12954 )
...
Removes the `register_noop_background_update` and deletes the background
updates directly in a delta file.
2022-06-13 14:06:27 -04:00
reivilibre
66a5f6c400
Add a unique index to state_group_edges to prevent duplicates being accidentally introduced and the consequential impact to performance. ( #12687 )
2022-05-19 14:16:49 +01:00
reivilibre
635f0d916b
Do not keep going if there are 5 back-to-back background update failures. ( #12781 )
2022-05-18 13:57:59 +00:00
David Robertson
aec69d2481
Another batch of type annotations ( #12726 )
2022-05-13 12:35:31 +01:00
Shay
ef3619e61d
Add config settings for background update parameters ( #11980 )
2022-03-11 10:46:45 -08:00
Shay
26211fec24
Fix a bug in background updates wherein background updates are never run using the default batch size ( #12157 )
2022-03-07 09:44:33 -08:00
Erik Johnston
d08ef6f155
Make background updates controllable via a plugin ( #11306 )
...
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2021-11-29 17:57:06 +01:00
Brendan Abolivier
7f9841bdec
Lower minumum batch size to 1 for background updates ( #11422 )
...
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-11-24 19:21:44 +00:00
Dirk Klimpel
ea20937084
Add an admin API to run background jobs. ( #11352 )
...
Instead of having admins poke into the database directly.
Can currently run jobs to populate stats and to populate
the user directory.
2021-11-19 19:39:46 +00:00
Erik Johnston
4ee71b9637
Add some background update admin APIs ( #11263 )
...
Fixes #11259
2021-11-08 16:08:02 +00:00
Jonathan de Jong
bdfde6dca1
Use inline type hints in http/federation/, storage/ and util/ ( #10381 )
2021-07-15 12:46:54 -04:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines ( #9786 )
...
Part of #9744
Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.
`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. ( #9665 )
2021-03-23 07:12:48 -04:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase ( #9381 )
...
- Update black version to the latest
- Run black auto formatting over the codebase
- Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md
2021-02-16 22:32:34 +00:00
Patrick Cloke
637282bb50
Add additional type hints to the storage module. ( #8980 )
2020-12-30 08:09:53 -05:00
Patrick Cloke
c619253db8
Stop sub-classing object ( #8249 )
2020-09-04 06:54:56 -04:00
Patrick Cloke
37db6252b7
Convert additional databases to async/await part 3 ( #8201 )
2020-09-01 11:04:17 -04:00
Patrick Cloke
30426c7063
Convert additional database methods to async (select list, search, insert_many, delete_*) ( #8168 )
2020-08-27 07:41:01 -04:00
Patrick Cloke
dbc630a628
Use the JSON encoder without whitespace in more places. ( #8124 )
2020-08-20 10:32:33 -04:00
Patrick Cloke
894dae74fe
Convert misc database code to async ( #8087 )
2020-08-14 07:24:26 -04:00
Erik Johnston
a7bdf98d01
Rename database classes to make some sense ( #8033 )
2020-08-05 21:38:57 +01:00
Patrick Cloke
f460da6031
Consistently use db_to_json to convert from database values to JSON objects. ( #7849 )
2020-07-16 11:32:19 -04:00
Richard van der Hoff
fcc2de7a0c
Update docstring per review comments
2020-04-03 10:51:32 +01:00
Richard van der Hoff
af47264b78
review comment
2020-04-02 12:04:55 +01:00
Richard van der Hoff
7b608cf468
Only run one background update at a time
2020-03-31 17:43:58 +01:00
Richard van der Hoff
b4c2234232
Make do_next_background_update return a bool
...
returning a None or an int that we don't use is confusing.
2020-03-31 17:43:58 +01:00
Richard van der Hoff
26d17b9bdc
Make has_completed_background_updates async
...
(Almost) everywhere that uses it is happy with an awaitable.
2020-03-31 17:43:58 +01:00
Richard van der Hoff
cfe8c8ab8e
Remove unused start_background_update
...
This was only used in a unit test, so let's just inline it in the test.
2020-03-31 17:24:06 +01:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 ( #6957 )
...
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Richard van der Hoff
85db7f73be
Add a background update to clear tombstoned rooms from the directory ( #6648 )
...
* Add a background update to clear tombstoned rooms from the directory
* use the ABC metaclass
2020-01-07 14:18:43 +00:00
Richard van der Hoff
9824a39d80
Async/await for background updates ( #6647 )
...
so that bg update routines can be async
2020-01-07 14:12:42 +00:00
Erik Johnston
d537be1ebd
Pass Database into the data store
2019-12-06 15:49:44 +00:00
Erik Johnston
4a33a6dd19
Move background update handling out of store
2019-12-05 11:11:26 +00:00
Erik Johnston
756d4942f5
Move DB pool and helper functions into dedicated Database class
2019-12-05 10:46:37 +00:00
Erik Johnston
ee86abb2d6
Remove underscore from SQLBaseStore functions
2019-12-04 16:23:43 +00:00
Brendan Abolivier
14504ad573
Add CI for synapse_port_db ( #6140 )
...
This adds:
* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)
as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:
1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)
Step `2` is done via a new script located at `scripts-dev/update_database`.
The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
Erik Johnston
4fb3c129aa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2
2019-09-25 17:53:13 +01:00
Erik Johnston
39b50ad42a
Review comments
2019-09-25 17:22:33 +01:00
Erik Johnston
2135c198d1
Add has_completed_background_update
...
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
Brendan Abolivier
f02f14e09a
Fix logging
2019-09-24 14:39:07 +01:00
Amber Brown
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
Amber Brown
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
Amber Brown
7efd1d87c2
Run black on the rest of the storage module ( #4996 )
2019-04-03 10:07:29 +01:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task ( #4864 )
2019-03-19 04:50:24 +11:00
Andrew Morgan
25d64a846a
Fix typos
2019-01-22 11:00:04 +00:00
Richard van der Hoff
667fba68f3
Run things as background processes
...
This fixes #3518 , and ensures that we get useful logs and metrics for lots of
things that happen in the background.
(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
Amber Brown
49af402019
run isort
2018-07-09 16:09:20 +10:00
