Commit Graph

56 Commits

Author SHA1 Message Date
Patrick Cloke
37db6252b7
Convert additional databases to async/await part 3 (#8201) 2020-09-01 11:04:17 -04:00
Patrick Cloke
30426c7063
Convert additional database methods to async (select list, search, insert_many, delete_*) (#8168) 2020-08-27 07:41:01 -04:00
Patrick Cloke
dbc630a628
Use the JSON encoder without whitespace in more places. (#8124) 2020-08-20 10:32:33 -04:00
Patrick Cloke
894dae74fe
Convert misc database code to async (#8087) 2020-08-14 07:24:26 -04:00
Erik Johnston
a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00
Patrick Cloke
f460da6031
Consistently use db_to_json to convert from database values to JSON objects. (#7849) 2020-07-16 11:32:19 -04:00
Richard van der Hoff
fcc2de7a0c Update docstring per review comments 2020-04-03 10:51:32 +01:00
Richard van der Hoff
af47264b78
review comment 2020-04-02 12:04:55 +01:00
Richard van der Hoff
7b608cf468 Only run one background update at a time 2020-03-31 17:43:58 +01:00
Richard van der Hoff
b4c2234232 Make do_next_background_update return a bool
returning a None or an int that we don't use is confusing.
2020-03-31 17:43:58 +01:00
Richard van der Hoff
26d17b9bdc Make has_completed_background_updates async
(Almost) everywhere that uses it is happy with an awaitable.
2020-03-31 17:43:58 +01:00
Richard van der Hoff
cfe8c8ab8e Remove unused start_background_update
This was only used in a unit test, so let's just inline it in the test.
2020-03-31 17:24:06 +01:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957)
Ensure good comprehension hygiene using flake8-comprehensions.
2020-02-21 07:15:07 -05:00
Richard van der Hoff
85db7f73be
Add a background update to clear tombstoned rooms from the directory (#6648)
* Add a background update to clear tombstoned rooms from the directory

* use the ABC metaclass
2020-01-07 14:18:43 +00:00
Richard van der Hoff
9824a39d80
Async/await for background updates (#6647)
so that bg update routines can be async
2020-01-07 14:12:42 +00:00
Erik Johnston
d537be1ebd Pass Database into the data store 2019-12-06 15:49:44 +00:00
Erik Johnston
4a33a6dd19 Move background update handling out of store 2019-12-05 11:11:26 +00:00
Erik Johnston
756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston
ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Brendan Abolivier
14504ad573
Add CI for synapse_port_db (#6140)
This adds:

* a test sqlite database
* a configuration file for the sqlite database
* a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`)

as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that:

1. installs Synapse
2. updates the test sqlite database to the latest schema and runs background updates on it
3. creates an empty postgresql database
4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage)

Step `2` is done via a new script located at `scripts-dev/update_database`.

The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.
2019-10-28 17:45:32 +00:00
Erik Johnston
4fb3c129aa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2 2019-09-25 17:53:13 +01:00
Erik Johnston
39b50ad42a Review comments 2019-09-25 17:22:33 +01:00
Erik Johnston
2135c198d1 Add has_completed_background_update
This allows checking if a specific background update has completed.
2019-09-24 15:53:17 +01:00
Brendan Abolivier
f02f14e09a
Fix logging 2019-09-24 14:39:07 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown
7efd1d87c2 Run black on the rest of the storage module (#4996) 2019-04-03 10:07:29 +01:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task (#4864) 2019-03-19 04:50:24 +11:00
Andrew Morgan
25d64a846a Fix typos 2019-01-22 11:00:04 +00:00
Richard van der Hoff
667fba68f3 Run things as background processes
This fixes #3518, and ensures that we get useful logs and metrics for lots of
things that happen in the background.

(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
2018-07-18 20:55:05 +01:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy (#3462) 2018-06-28 14:49:57 +01:00
Amber Brown
77ac14b960
Pass around the reactor explicitly (#3385) 2018-06-22 09:37:10 +01:00
Erik Johnston
fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Erik Johnston
926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Richard van der Hoff
6b02fc80d1 Reinstate event_search_postgres_gist handler
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
Richard van der Hoff
2908f955d1 Check database in has_completed_background_updates
so that the right thing happens on workers.
2017-11-22 18:02:15 +00:00
Richard van der Hoff
7098b65cb8 Fix error on sqlite 3.7
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.

To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.

Fixes https://github.com/matrix-org/synapse/issues/2572.
2017-11-21 11:14:17 +00:00
Richard van der Hoff
6cfee09be9 Make __init__ consitstent across Store heirarchy
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Richard van der Hoff
eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff
34194aaff7 Don't create event_search index on sqlite
... because the table is virtual
2017-05-11 12:46:55 +01:00
Richard van der Hoff
baafb85ba4 Add an index to event_search
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
Richard van der Hoff
c84770b877 Fix bgupdate error if index already exists (#2167)
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135.

Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
2017-04-27 15:27:48 +01:00
Richard van der Hoff
994d7ae7c5 Remove broken use of clock.call_later
background_updates was using `call_later` in a way that leaked the logcontext
into the reactor.

We could have rewritten it to do it properly, but given that we weren't using
the fancier facilities provided by `call_later`, we might as well just use
`async.sleep`, which does the logcontext stuff properly.
2017-03-18 00:01:37 +00:00
Erik Johnston
c94de0ab60 Add WHERE clause support to index creation 2016-09-12 16:55:01 +01:00
Erik Johnston
5beda10bbd Reindex state_groups_state after pruning 2016-09-08 16:18:01 +01:00
Mark Haines
955ef1f06c fix: defer.returnValue takes one argument 2016-07-25 16:04:45 +01:00
Richard van der Hoff
2ee4c9ee02 background updates: fix assert again 2016-07-25 16:01:46 +01:00
Richard van der Hoff
9dbd903f41 background updates: Fix assertion to do something 2016-07-25 14:05:23 +01:00
Richard van der Hoff
465117d7ca Fix background_update tests
A bit of a cleanup for background_updates, and make sure that the real
background updates have run before we start the unit tests, so that they don't
interfere with the tests.
2016-07-25 12:10:42 +01:00