58 Commits

Author	SHA1	Message	Date
Patrick Cloke	637282bb50	Add additional type hints to the storage module. (#8980)	2020-12-30 08:09:53 -05:00
Patrick Cloke	c619253db8	Stop sub-classing object (#8249)	2020-09-04 06:54:56 -04:00
Patrick Cloke	37db6252b7	Convert additional databases to async/await part 3 (#8201)	2020-09-01 11:04:17 -04:00
Patrick Cloke	30426c7063	Convert additional database methods to async (select list, search, insert_many, delete_*) (#8168)	2020-08-27 07:41:01 -04:00
Patrick Cloke	dbc630a628	Use the JSON encoder without whitespace in more places. (#8124)	2020-08-20 10:32:33 -04:00
Patrick Cloke	894dae74fe	Convert misc database code to async (#8087)	2020-08-14 07:24:26 -04:00
Erik Johnston	a7bdf98d01	Rename database classes to make some sense (#8033)	2020-08-05 21:38:57 +01:00
Patrick Cloke	f460da6031	Consistently use db_to_json to convert from database values to JSON objects. (#7849)	2020-07-16 11:32:19 -04:00
Richard van der Hoff	fcc2de7a0c	Update docstring per review comments	2020-04-03 10:51:32 +01:00
Richard van der Hoff	af47264b78	review comment	2020-04-02 12:04:55 +01:00
Richard van der Hoff	7b608cf468	Only run one background update at a time	2020-03-31 17:43:58 +01:00
Richard van der Hoff	b4c2234232	Make do_next_background_update return a bool ... returning a None or an int that we don't use is confusing.	2020-03-31 17:43:58 +01:00
Richard van der Hoff	26d17b9bdc	Make has_completed_background_updates async ... (Almost) everywhere that uses it is happy with an awaitable.	2020-03-31 17:43:58 +01:00
Richard van der Hoff	cfe8c8ab8e	Remove unused start_background_update ... This was only used in a unit test, so let's just inline it in the test.	2020-03-31 17:24:06 +01:00
Patrick Cloke	509e381afa	Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957) ... Ensure good comprehension hygiene using flake8-comprehensions.	2020-02-21 07:15:07 -05:00
Richard van der Hoff	85db7f73be	Add a background update to clear tombstoned rooms from the directory (#6648) ... * Add a background update to clear tombstoned rooms from the directory * use the ABC metaclass	2020-01-07 14:18:43 +00:00
Richard van der Hoff	9824a39d80	Async/await for background updates (#6647) ... so that bg update routines can be async	2020-01-07 14:12:42 +00:00
Erik Johnston	d537be1ebd	Pass Database into the data store	2019-12-06 15:49:44 +00:00
Erik Johnston	4a33a6dd19	Move background update handling out of store	2019-12-05 11:11:26 +00:00
Erik Johnston	756d4942f5	Move DB pool and helper functions into dedicated Database class	2019-12-05 10:46:37 +00:00
Erik Johnston	ee86abb2d6	Remove underscore from SQLBaseStore functions	2019-12-04 16:23:43 +00:00
Brendan Abolivier	14504ad573	Add CI for synapse_port_db (#6140) ... This adds: * a test sqlite database * a configuration file for the sqlite database * a configuration file for a postgresql database (using the credentials in `.buildkite/docker-compose.pyXX.pgXX.yaml`) as well as a new script named `.buildkite/scripts/test_synapse_port_db.sh` that: 1. installs Synapse 2. updates the test sqlite database to the latest schema and runs background updates on it 3. creates an empty postgresql database 4. run the `synapse_port_db` script to migrate the test sqlite database to the empty postgresql database (with coverage) Step `2` is done via a new script located at `scripts-dev/update_database`. The test sqlite database is extracted from a SyTest run, so that it can be considered as an actual homeserver's database with actual data in it.	2019-10-28 17:45:32 +00:00
Erik Johnston	4fb3c129aa	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cleanup_user_ips_2	2019-09-25 17:53:13 +01:00
Erik Johnston	39b50ad42a	Review comments	2019-09-25 17:22:33 +01:00
Erik Johnston	2135c198d1	Add has_completed_background_update ... This allows checking if a specific background update has completed.	2019-09-24 15:53:17 +01:00
Brendan Abolivier	f02f14e09a	Fix logging	2019-09-24 14:39:07 +01:00
Amber Brown	4806651744	Replace returnValue with return (#5736)	2019-07-23 23:00:55 +10:00
Amber Brown	32e7c9e7f2	Run Black. (#5482)	2019-06-20 19:32:02 +10:00
Amber Brown	7efd1d87c2	Run black on the rest of the storage module (#4996)	2019-04-03 10:07:29 +01:00
Amber Brown	282c97327f	Migrate the user directory initial population to a background task (#4864)	2019-03-19 04:50:24 +11:00
Andrew Morgan	25d64a846a	Fix typos	2019-01-22 11:00:04 +00:00
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	2018-07-18 20:55:05 +01:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Erik Johnston	fa72803490	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-19 11:41:01 +00:00
Erik Johnston	926ba76e23	Replace ujson with simplejson	2018-03-15 23:43:31 +00:00
Richard van der Hoff	6b02fc80d1	Reinstate event_search_postgres_gist handler ... People may have queued updates for this, so we can't just delete it.	2018-02-02 14:32:51 +00:00
Richard van der Hoff	2908f955d1	Check database in has_completed_background_updates ... so that the right thing happens on workers.	2017-11-22 18:02:15 +00:00
Richard van der Hoff	7098b65cb8	Fix error on sqlite 3.7 ... Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572.	2017-11-21 11:14:17 +00:00
Richard van der Hoff	6cfee09be9	Make __init__ consitstent across Store heirarchy ... Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)	2017-11-13 10:46:07 +00:00
Richard van der Hoff	eaaabc6c4f	replace 'except:' with 'except Exception:' ... what could possibly go wrong	2017-10-23 15:52:32 +01:00
Richard van der Hoff	34194aaff7	Don't create event_search index on sqlite ... ... because the table is virtual	2017-05-11 12:46:55 +01:00
Richard van der Hoff	baafb85ba4	Add an index to event_search ... - to make the purge API quicker	2017-05-11 12:05:22 +01:00
Richard van der Hoff	c84770b877	Fix bgupdate error if index already exists (#2167) ... When creating a new table index in the background, guard against it existing already. Fixes https://github.com/matrix-org/synapse/issues/2135. Also, make sure we restore the autocommit flag when we're done, otherwise we get more failures from other operations later on. Fixes https://github.com/matrix-org/synapse/issues/1890 (hopefully).	2017-04-27 15:27:48 +01:00
Richard van der Hoff	994d7ae7c5	Remove broken use of clock.call_later ... background_updates was using `call_later` in a way that leaked the logcontext into the reactor. We could have rewritten it to do it properly, but given that we weren't using the fancier facilities provided by `call_later`, we might as well just use `async.sleep`, which does the logcontext stuff properly.	2017-03-18 00:01:37 +00:00
Erik Johnston	c94de0ab60	Add WHERE clause support to index creation	2016-09-12 16:55:01 +01:00
Erik Johnston	5beda10bbd	Reindex state_groups_state after pruning	2016-09-08 16:18:01 +01:00
Mark Haines	955ef1f06c	fix: defer.returnValue takes one argument	2016-07-25 16:04:45 +01:00
Richard van der Hoff	2ee4c9ee02	background updates: fix assert again	2016-07-25 16:01:46 +01:00