3523 Commits

Author	SHA1	Message	Date
Richard van der Hoff	350f654e7b	Add unique indexes to a couple of tables ... The indexes on device_lists_remote_extremeties can be unique, and they therefore should, to ensure that the db remains consistent.	2018-11-02 10:36:13 +00:00
Richard van der Hoff	50e328d1e7	Remove redundant database locks for device list updates ... We can rely on the application-level per-user linearizer.	2018-11-02 10:36:13 +00:00
Erik Johnston	0794504bce	Merge pull request #4006 from matrix-org/erikj/purge_state_groups ... Delete unreferenced state groups during purge	2018-10-30 16:58:22 +00:00
David Baker	e0934acdbb	Cast to int here too	2018-10-30 11:12:23 +00:00
David Baker	12941f5f8b	Cast bacjup version to int when querying	2018-10-30 11:01:07 +00:00
David Baker	2f0f911c52	Convert version back to a string	2018-10-30 10:35:18 +00:00
David Baker	64fa557f80	Try & make it work on postgres	2018-10-30 09:51:04 +00:00
David Baker	563f9b61b1	Make e2e backup versions numeric in the DB ... We were doing max(version) which does not do what we wanted on a column of type TEXT.	2018-10-29 21:01:22 +00:00
Erik Johnston	88e5ffe6fe	Deduplicate device updates sent over replication ... We currently send several kHz of device list updates over replication occisonally, which often causes the replications streams to lag and then get dropped. A lot of those updates will actually be duplicates, since we don't send e.g. device_ids across replication, so let's deduplicate it when we pull them out of the database.	2018-10-29 17:34:34 +00:00
Erik Johnston	ad88460e0d	Move _find_unreferenced_groups	2018-10-29 14:24:19 +00:00
Erik Johnston	664b192a3b	Fix set operations thinko	2018-10-29 14:21:43 +00:00
Erik Johnston	f4f223aa44	Don't make temporary list	2018-10-29 14:01:49 +00:00
Erik Johnston	b2399f6281	Make SQL a bit cleaner	2018-10-29 14:01:11 +00:00
Erik Johnston	03e634dad4	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups	2018-10-26 16:22:45 +01:00
Richard van der Hoff	4cda300058	preserve room visibility	2018-10-25 19:10:24 +01:00
Erik Johnston	cb53ce9d64	Refactor state group lookup to reduce DB hits (#4011) ... Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.	2018-10-25 17:49:55 +01:00
Neil Johnson	95ad128851	Merge pull request #4081 from matrix-org/neilj/fix_mau_init ... fix race condiftion in calling initialise_reserved_users	2018-10-25 16:33:40 +01:00
Neil Johnson	fcbd488e9a	add new line	2018-10-25 16:13:43 +01:00
Neil Johnson	f8fe98812b	improve comments	2018-10-25 14:58:59 +01:00
Neil Johnson	f7f487e14c	Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin	2018-10-25 14:40:06 +01:00
Neil Johnson	ea69a84bbb	fix style inconsistencies	2018-10-24 17:18:08 +01:00
Neil Johnson	07126e43a4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init	2018-10-24 16:25:39 +01:00
Richard van der Hoff	7328039117	Merge pull request #4082 from matrix-org/rav/fix_pep8 ... Fix a number of flake8 errors	2018-10-24 11:23:35 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase ... Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	ef771cc4c2	Fix a number of flake8 errors ... Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.	2018-10-24 10:39:03 +01:00
Neil Johnson	329d18b39c	remove white space	2018-10-23 15:27:20 +01:00
Neil Johnson	6105c6101f	fix race condiftion in calling initialise_reserved_users	2018-10-23 15:24:58 +01:00
Amber Brown	e1728dfcbe	Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068)	2018-10-20 11:16:55 +11:00
Erik Johnston	67f7b9cb50	pep8	2018-10-19 16:06:59 +01:00
Erik Johnston	056f099126	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups	2018-10-19 15:48:59 +01:00
Erik Johnston	47a9da28ca	Batch process handling state groups	2018-10-19 15:48:15 +01:00
Richard van der Hoff	fc0f13dd03	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-16 21:10:04 +01:00
Erik Johnston	4a28d3d36f	Update event_auth table for rejected events	2018-10-16 16:28:42 +01:00
Erik Johnston	e238013c44	Add v2 state res algorithm. ... We hook this up to the vdh test room version.	2018-10-16 16:16:13 +01:00
David Baker	03c11032c3	Merge pull request #4019 from matrix-org/dbkr/e2e_backups ... E2E backups	2018-10-15 10:19:43 +01:00
Erik Johnston	67a1e315cc	Fix up comments	2018-10-12 13:49:48 +01:00
David Baker	306361b31b	Misc PR feedback bits	2018-10-12 11:48:56 +01:00
Richard van der Hoff	8ddd0f273c	Comments on get_all_new_events_stream ... just some docstrings to clarify the behaviour here	2018-10-12 09:55:41 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
David Baker	497444f1fd	Don't reuse backup versions ... Since we don't actually delete the keys, just mark the versions as deleted in the db rather than actually deleting them, then we won't reuse versions. Fixes https://github.com/vector-im/riot-web/issues/7448	2018-10-05 15:08:36 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Erik Johnston	4917ff5523	Add state_group index to event_to_state_groups ... This is needed to efficiently check for unreferenced state groups during purge.	2018-10-04 16:03:08 +01:00
Erik Johnston	17d585753f	Delete unreferened state groups during purge	2018-10-04 16:03:06 +01:00
Richard van der Hoff	9693625e55	actually exclude outliers	2018-10-03 10:19:41 +01:00
Richard van der Hoff	3e39783d5d	remove debugging	2018-10-02 23:44:14 +01:00
Richard van der Hoff	ae61ade891	Fix bug in forward_extremity update logic ... An event does not stop being a forward_extremity just because an outlier or rejected event refers to it.	2018-10-02 23:36:08 +01:00
Erik Johnston	7258d081a5	Fix bug when invalidating destination retry timings	2018-10-02 15:47:57 +01:00
Matthew Hodgson	5b68f29f48	fix thinkos	2018-09-29 02:14:40 +01:00
Richard van der Hoff	9c8cec5dab	Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache	2018-09-28 10:51:09 +01:00
Richard van der Hoff	51d33d5178	Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts ... fix #3854 MAU transaction errors	2018-09-27 12:31:37 +01:00
Richard van der Hoff	ae6ad4cf41	docstrings and unittests for storage.state (#3958) ... I spent ages trying to figure out how I was going mad...	2018-09-27 11:22:25 +01:00
Neil Johnson	28781b65e7	fix #3854	2018-09-26 16:16:41 +01:00
Erik Johnston	fdd1a62e8d	Add a five minute cache to get_destination_retry_timings ... Hopefully helps with #3931	2018-09-21 14:56:12 +01:00
Amber Brown	1f3f5fcf52	Fix client IPs being broken on Python 3 (#3908)	2018-09-20 20:14:34 +10:00
Erik Johnston	bbab6ebfd9	Fix up changelog and remove spurious comment	2018-09-19 14:45:14 +01:00
Erik Johnston	392a54128c	pep8	2018-09-19 14:37:49 +01:00
Erik Johnston	b9158ac2bf	Remove get_destination_retry_timings cache ... Currently we rely on the master to invalidate this cache promptly. However, after having moved most federation endpoints off of master this no longer happens, causing outbound fedeariont to get blackholed. Fixes #3798	2018-09-19 14:22:57 +01:00
Richard van der Hoff	85a43f4167	Return a 404 when deleting unknown room alias ... As per https://github.com/matrix-org/matrix-doc/issues/1675 Fixes https://github.com/matrix-org/synapse/issues/2782	2018-09-17 13:19:00 +01:00
David Baker	bc74925c5b	WIP e2e key backups ... Continues from uhoreg's branch This just fixed the errcode on /room_keys/version if no backup and updates the schema delta to be on the latest so it gets run	2018-09-13 17:02:59 +01:00
Erik Johnston	ed5331a627	comment	2018-09-13 16:10:56 +01:00
Erik Johnston	9cbd0094f0	pep8	2018-09-13 15:15:35 +01:00
Erik Johnston	9dbe38ea7d	Create indices after insertion	2018-09-13 15:05:52 +01:00
Erik Johnston	c857f5ef9b	Make purge history slightly faster ... Don't pull out events that are outliers and won't be deleted, as nothing should happen to them.	2018-09-13 12:48:10 +01:00
Neil Johnson	f30a303590	Merge pull request #3846 from matrix-org/neilj/expose-registered-users ... expose number of real reserved users	2018-09-12 17:14:04 +01:00
Neil Johnson	8decd6233d	improve naming	2018-09-12 16:22:15 +01:00
Neil Johnson	0ddf486724	expose number of real reserved users	2018-09-12 11:58:52 +01:00
Amber Brown	33716c4aea	Merge pull request #3826 from matrix-org/rav/logging_for_keyring ... add some logging for the keyring queue	2018-09-12 20:43:47 +10:00
Matthew Hodgson	b041115415	Speed up lazy loading (#3827) ... * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)	2018-09-12 00:50:39 +01:00
Richard van der Hoff	cd7ef43872	clearer logging when things fail, too	2018-09-06 23:56:47 +01:00
Neil Johnson	84a750e0c3	ensure guests never enter mau list	2018-09-06 17:22:53 +01:00
Hubert Chathi	3801b8aa03	try to make flake8 and isort happy	2018-09-06 11:35:19 -04:00
Neil Johnson	61b05727fa	guest users should not be part of mau total	2018-09-05 22:30:36 +01:00
Neil Johnson	0b01281e77	move threepid checker to config, add missing yields	2018-08-31 17:11:11 +01:00
Neil Johnson	09f3cf1a7e	ensure post registration auth checks do not fail erroneously	2018-08-31 15:42:51 +01:00
Amber Brown	14e4d4f4bf	Port storage/ to Python 3 (#3725)	2018-08-31 00:19:58 +10:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	cd77270a66	Implement trail users	2018-08-23 19:17:19 +01:00
Erik Johnston	7a0da69eee	Add missing yield	2018-08-23 10:28:12 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles ... Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	3bf8bab8f9	Merge pull request #3673 from matrix-org/erikj/refactor_state_handler ... Refactor state module to support multiple room versions	2018-08-22 10:04:55 +01:00
Matthew Hodgson	bb81e78ec6	Split the state_group_cache in two (#3726) ... Splits the state_group_cache in two. One half contains normal state events; the other contains member events. The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache. This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching. Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow. Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.	2018-08-22 00:56:37 +02:00
Hubert Chathi	42a394caa2	allow session_data to be any JSON instead of just a string	2018-08-21 14:51:34 -04:00
Hubert Chathi	8550a7e9c2	allow auth_data to be any JSON instead of a string	2018-08-21 10:38:00 -04:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	38f708a2bb	Remote profile cache should remain in master worker	2018-08-17 11:37:42 +01:00
Neil Johnson	bcfeb44afe	call reap on start up and fix under reaping bug	2018-08-16 22:55:32 +01:00
Matthew Hodgson	3f543dc021	initial cut at a room summary API (#3574)	2018-08-16 09:46:50 +01:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568)	2018-08-15 16:35:22 +01:00
Erik Johnston	ef184caf30	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-15 14:25:46 +01:00
Neil Johnson	9ecbaf8ba8	adding missing yield	2018-08-14 16:55:28 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block ... Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Amber Brown	591bf87c6a	Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres	2018-08-14 20:56:23 +10:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	2018-08-13 16:47:46 +10:00
Matthew Hodgson	f0cede5556	missing import	2018-08-12 19:14:31 -04:00
Matthew Hodgson	66a4ca1d28	404 nicely if you try to interact with a missing current version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	edc427a351	flake8	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	15d513f16f	fix idiocies and so make tests pass	2018-08-12 19:14:31 -04:00
Matthew Hodgson	b5eee511c7	don't needlessly return user_id	2018-08-12 19:14:31 -04:00
Matthew Hodgson	982edca380	fix flakes	2018-08-12 19:14:31 -04:00
Matthew Hodgson	234611f347	fix typos	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f0791b7bd	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8d14598e90	add storage docstring; remove unused set_e2e_room_keys	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cac0253799	rename room_key_version table correctly, and fix opt args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	0abb205b47	blindly incorporate PR review - needs testing & fixing	2018-08-12 19:14:31 -04:00
Matthew Hodgson	69e51c7ba4	make /room_keys/version work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8ae64b270f	implement /room_keys/version too (untested)	2018-08-12 19:14:31 -04:00
Matthew Hodgson	6b8c07abc2	make it work and fix pep8	2018-08-12 19:13:09 -04:00
Matthew Hodgson	0bc4627a73	interim WIP checkin; doesn't build yet	2018-08-12 18:23:10 -04:00
Matthew Hodgson	53ace904b2	total WIP skeleton for /room_keys API	2018-08-12 18:23:10 -04:00
Neil Johnson	31fa743567	fix sqlite/postgres incompatibility in reap_monthly_active_users	2018-08-11 22:38:34 +01:00
Amber Brown	b37c472419	Rename async to async_helpers because async is a keyword on Python 3.7 (#3678)	2018-08-10 23:50:21 +10:00
Neil Johnson	885ea9c602	rename _user_last_seen_monthly_active	2018-08-09 18:02:12 +01:00
Erik Johnston	ce6db0e547	Choose state algorithm based on room version	2018-08-09 14:58:47 +01:00
Erik Johnston	484a0ebdfc	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-09 10:16:29 +01:00
Erik Johnston	5785b93711	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-09 10:16:16 +01:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions ... Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	be59910b93	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users	2018-08-08 13:45:21 +01:00
Neil Johnson	990fe9fc23	Merge pull request #3633 from matrix-org/neilj/mau_tracker ... API for monthly_active_users table	2018-08-08 12:44:15 +00:00
Neil Johnson	312ae74746	typos	2018-08-08 13:33:16 +01:00
Neil Johnson	ef3589063a	prevent total number of reserved users being too large	2018-08-07 17:59:27 +01:00
Neil Johnson	e8eba2b4e3	implement reserved users for mau limits	2018-08-07 17:49:43 +01:00
Erik Johnston	cd9765805e	Allow ratelimiting on workers	2018-08-07 10:50:28 +01:00
Erik Johnston	495cb100d1	Allow profile changes to happen on workers	2018-08-07 10:50:26 +01:00
Richard van der Hoff	ca9bc1f4fe	Fix occasional glitches in the synapse_event_persisted_position metric ... Every so often this metric glitched to a negative number. I'm assuming it was due to backfilled events.	2018-08-07 10:00:03 +01:00
Neil Johnson	a74b25faaa	WIP building out mau reserved users	2018-08-06 23:25:25 +01:00
Neil Johnson	e54794f5b6	Fix postgres compatibility bug	2018-08-06 21:55:54 +01:00
Neil Johnson	16d78be315	make use of _simple_select_one_onecol, improved comments	2018-08-06 17:51:15 +01:00
Erik Johnston	96a9a29645	Pull in necessary stores in federation_reader	2018-08-06 15:23:57 +01:00
Erik Johnston	62ace05c45	Move necessary storage functions to worker classes	2018-08-06 15:23:38 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Neil Johnson	e10830e976	wip commit - tests failing	2018-08-03 17:55:50 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning ... This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Neil Johnson	950807d93a	fix caching and tests	2018-08-03 13:49:53 +01:00
Neil Johnson	897c51d274	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-03 13:40:47 +01:00
Neil Johnson	5e2f7b8084	typo	2018-08-02 22:47:48 +01:00
Neil Johnson	4ecb4bdac9	wip attempt at caching	2018-08-02 22:41:05 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Neil Johnson	9180061b49	remove unused count_monthly_users	2018-08-02 15:55:29 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups ... Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	0a65450d04	Validation for events/rooms in fed requests ... When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)	2018-08-02 13:48:40 +01:00
Neil Johnson	00f99f74b1	insertion into monthly_active_users	2018-08-02 13:47:19 +01:00
Neil Johnson	4a6725d9d1	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-02 11:04:18 +01:00
Neil Johnson	165e067033	Revert "change monthly_active_users table to be a single column" ... This reverts commit ec716a35b2.	2018-08-02 10:59:58 +01:00
Erik Johnston	40c1c59cf4	Merge pull request #3621 from matrix-org/erikj/split_fed_store ... Split out DB writes in federation handler	2018-08-02 10:41:42 +01:00
Neil Johnson	08281fe6b7	self.db_conn unused	2018-08-01 23:26:24 +01:00
Neil Johnson	c21d82bab3	normalise reaping query	2018-08-01 23:24:38 +01:00
Neil Johnson	ec716a35b2	change monthly_active_users table to be a single column	2018-08-01 17:54:37 +01:00
Neil Johnson	d766f26de9	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-01 17:49:41 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits ... Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Neil Johnson	7ff44d9215	improve clarity	2018-08-01 16:17:00 +01:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	2018-08-02 00:54:06 +10:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Erik Johnston	a6d7b74915	update docs	2018-08-01 13:39:14 +01:00
Neil Johnson	4e5ac901dd	clean up	2018-08-01 12:03:57 +01:00
Neil Johnson	f9f5559971	fix comment	2018-08-01 12:03:42 +01:00
Neil Johnson	6023cdd227	remove errant print	2018-08-01 10:27:17 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Neil Johnson	6ef983ce5c	api into monthly_active_users table	2018-07-31 16:36:24 +01:00
Richard van der Hoff	5de936caa1	Merge pull request #3612 from matrix-org/rav/store_heirarchy ... Make EventStore inherit from EventFederationStore	2018-07-31 13:44:04 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Richard van der Hoff	0bc9b9e397	reinstate explicit include of EventsWorkerStore	2018-07-31 13:11:04 +01:00
Neil Johnson	fef7e58ac6	actually close conn	2018-07-30 22:29:44 +01:00
Neil Johnson	9b13817e06	factor out metrics from __init__ to app/homeserver	2018-07-30 22:07:07 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Erik Johnston	143f1a2532	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store	2018-07-30 09:56:18 +01:00
Matthew Hodgson	e9b2d047f6	make /context lazyload & filter aware (#3567) ... make /context lazyload & filter aware.	2018-07-27 15:12:50 +01:00
Richard van der Hoff	65c8dee900	Merge pull request #3614 from matrix-org/rav/stop_populating_event_content ... Stop populating events.content	2018-07-26 22:54:08 +01:00
Richard van der Hoff	9e68b1bd2d	Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id ... Remove some redundant joins on event_edges.room_id	2018-07-26 22:31:01 +01:00
Richard van der Hoff	ef9d51b081	Merge pull request #3610 from matrix-org/rav/fix_looping_calls ... Fix some looping_call calls which were broken in #3604	2018-07-26 14:55:57 +01:00
Richard van der Hoff	51d7df1915	Create the column nullable ... There's no real point in ever making the column non-nullable, and doing so breaks the sytests.	2018-07-26 14:54:04 +01:00
Richard van der Hoff	5c1d301fd9	Stop populating events.content ... This field is no longer read from, so we should stop populating it. Once we're happy that this doesn't break everything, and a rollback is unlikely, we can think about dropping the column.	2018-07-26 14:43:02 +01:00
Richard van der Hoff	bd4b25f4d0	Remove some redundant joins on event_edges.room_id ... We've long passed the point where it's possible to have the same event_id in different tables, so these join conditions are redundant: we can just join on event_id. event_edges is of non-trivial size, and the room_id column is wasteful, so let's stop reading from it. In future, we can stop writing to it, and then drop it.	2018-07-26 13:19:08 +01:00
Richard van der Hoff	1b4d73fa52	comment on event_edges	2018-07-26 12:53:51 +01:00
Richard van der Hoff	21e878ebb6	Make EventStore inherit from EventFederationStore ... (since it uses methods therein) Turns out that we had a bunch of things which were incorrectly importing EventWorkerStore from events.py rather than events_worker.py, which broke once I removed the import into events.py.	2018-07-26 12:48:51 +01:00
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	2018-07-26 11:48:08 +01:00
Matthew Hodgson	bc7944e6d2	switch missing_types to be a bool	2018-07-25 23:36:31 +01:00
Matthew Hodgson	2565804030	Merge branch 'develop' into matthew/filter_members	2018-07-25 17:27:49 +01:00
Matthew Hodgson	7d9fb88617	incorporate more review.	2018-07-25 16:33:50 +01:00
Erik Johnston	78a691d005	Split out DB writes in federation handler ... This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.	2018-07-25 16:22:56 +01:00
Erik Johnston	3849f7f69f	Merge pull request #3603 from matrix-org/erikj/handle_outliers ... Correctly handle outliers during persist events	2018-07-25 13:24:04 +01:00
Richard van der Hoff	cee1ae1b72	Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more ... Fix another logcontext leak in _persist_events	2018-07-25 11:56:00 +01:00
Richard van der Hoff	32b30e15f2	Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error ... Fix occasional 'tuple index out of range' error	2018-07-25 11:55:45 +01:00
Erik Johnston	7780a7b47c	Actually fix it by adding continue	2018-07-25 11:13:20 +01:00
Richard van der Hoff	1be94440d3	Fix occasional 'tuple index out of range' error ... This fixes a bug in _delete_existing_rows_txn which was introduced in #3435 (though it's been on matrix-org-hotfixes for *years*). This code is only called when there is some sort of conflict the first time we try to persist an event, so it only happens rarely. Still, the exceptions are annoying.	2018-07-25 11:05:58 +01:00
Richard van der Hoff	07defd5fe6	Fix another logcontext leak in _persist_events ... We need to run the errback in the sentinel context to avoid losing our own context. Also: add logging to runInteraction to help identify where "Starting db connection from sentinel context" warnings are coming from	2018-07-25 10:53:23 +01:00
Erik Johnston	a297ff2b16	Fix typo in conditional	2018-07-25 09:48:01 +01:00
Richard van der Hoff	371da42ae4	Wrap a number of things that run in the background ... This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.	2018-07-25 09:41:12 +01:00
Erik Johnston	ec56121b0d	Correctly handle outliers during persist events ... We incorrectly asserted that all contexts must have a non None state group without consider outliers. This would usually be fine as the assertion would never be hit, as there is a shortcut during persistence if the forward extremities don't change. However, if the outlier is being persisted with non-outlier events, the function would be called and the assertion would be hit. Fixes #3601	2018-07-25 09:35:02 +01:00
Matthew Hodgson	cb5c37a57c	handle the edge case for _get_some_state_from_cache where types is []	2018-07-24 20:34:45 +01:00
Erik Johnston	3188973857	Pull out did_forget to worker store	2018-07-24 16:49:14 +01:00
Erik Johnston	60a1d147a7	Merge pull request #3595 from matrix-org/erikj/use_deltas ... Use deltas to calculate current state deltas	2018-07-24 15:41:18 +01:00
Erik Johnston	709c309b0e	Expand on docstring comment about return value	2018-07-24 15:12:50 +01:00
Erik Johnston	8f65ab98d2	Remove unnecessary iteritems	2018-07-24 15:08:01 +01:00
Erik Johnston	ed0dd68731	Fixup comment (and indent)	2018-07-24 14:31:38 +01:00
Erik Johnston	811ac73a42	Don't fetch state from the database unless needed	2018-07-24 14:18:23 +01:00
Erik Johnston	a79410e7b8	Have _get_new_state_after_events return delta ... If we have a delta from the existing to new current state, then we can reuse that rather than manually working it out by fetching both lots of state.	2018-07-24 14:14:17 +01:00
Richard van der Hoff	93b0722c50	Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room ... Remove redundant checks on room forgottenness	2018-07-24 14:11:11 +01:00
Matthew Hodgson	454f59b7ad	Merge branch 'develop' into matthew/filter_members	2018-07-24 14:03:37 +01:00
Erik Johnston	223341205e	Don't require to_delete to have event_ids	2018-07-24 14:02:40 +01:00
Matthew Hodgson	e22700c3dd	consider non-filter_type types as wildcards, thus missing from the state-group-cache	2018-07-24 13:59:07 +01:00
Matthew Hodgson	d19fba3655	Merge branch 'develop' into matthew/filter_members	2018-07-24 12:39:54 +01:00
Matthew Hodgson	cd241d6bda	incorporate more review	2018-07-24 12:39:40 +01:00
Richard van der Hoff	30bfed5aa5	Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room	2018-07-24 11:46:09 +01:00
Erik Johnston	97acd385a3	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta	2018-07-24 11:32:13 +01:00
Erik Johnston	0fa73e4a63	Remove unnecessary if	2018-07-24 11:19:23 +01:00
Erik Johnston	ff5426f6b8	Speed up _calculate_state_delta	2018-07-24 10:55:11 +01:00
Richard van der Hoff	a678145010	Merge branch 'develop' into rav/logcontext_fixes	2018-07-24 10:43:30 +01:00
Richard van der Hoff	8dff6e0322	Logcontext fixes ... Fix some random logcontext leaks.	2018-07-24 00:37:17 +01:00
Richard van der Hoff	69fb5dbdab	fix idiocy	2018-07-24 00:04:44 +01:00
Richard van der Hoff	1938cffaea	Add some measure blocks to persist_events ... ... to help us figure out where 40% of CPU is going	2018-07-23 23:48:19 +01:00
Matthew Hodgson	efcdacad7d	handle case where types is [] on postgres correctly	2018-07-23 22:41:05 +01:00
Matthew Hodgson	adfe29ec0b	Merge branch 'develop' into matthew/filter_members	2018-07-23 19:21:37 +01:00
Matthew Hodgson	254fb430d1	incorporate review	2018-07-23 19:21:20 +01:00
Erik Johnston	8b9f164fff	Comments	2018-07-23 17:43:01 +01:00
Erik Johnston	50c60e5fad	Only get cached state from context in persist_event ... We don't want to bother pulling out the current state from the DB since until we know we have to. Checking the context for state is just an optimisation.	2018-07-23 17:21:40 +01:00
Richard van der Hoff	4f5cc8e4e7	Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room	2018-07-23 17:15:12 +01:00
Richard van der Hoff	dae6dc1e77	Remove redundant checks on room forgottenness ... Fixes #3550	2018-07-23 17:13:34 +01:00
Erik Johnston	e42510ba63	Use new getters	2018-07-23 13:17:22 +01:00
Matthew Hodgson	9b34f3ea3a	Merge branch 'develop' into matthew/sync_deleted_devices	2018-07-23 10:03:28 +01:00
Matthew Hodgson	c1bf2b587e	add trailing comma	2018-07-23 09:56:23 +01:00
Matthew Hodgson	2f558300cc	fix thinkos; unbreak tests	2018-07-19 19:22:27 +01:00
Matthew Hodgson	bcaec2915a	incorporate review	2018-07-19 19:03:50 +01:00
Matthew Hodgson	924eb34d94	add a filtered_types param to limit filtering to specific types	2018-07-19 18:32:02 +01:00
Matthew Hodgson	be3adfc331	merge develop pydoc for _get_state_for_groups	2018-07-19 11:26:04 +01:00
Matthew Hodgson	9e40834f74	yes, we do need to invalidate the device_id_exists_cache when deleting a remote device	2018-07-19 11:15:10 +01:00
Richard van der Hoff	f1a15ea206	revert 00bc979 ... ... we've fixed the things that caused the warnings, so we should reinstate the warning.	2018-07-19 11:14:20 +01:00
Richard van der Hoff	1ffb7bec20	Merge remote-tracking branch 'origin/release-v0.33.0' into develop	2018-07-19 11:12:33 +01:00
Matthew Hodgson	c0685f67c0	spell out that include_deleted_devices requires include_all_devices	2018-07-19 10:59:02 +01:00
Richard van der Hoff	00bc979137	Disable logcontext warning ... Temporary workaround to #3518 while we release 0.33.0.	2018-07-19 10:51:15 +01:00
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	2018-07-18 20:55:05 +01:00
Matthew Hodgson	12ec58301f	shift to using an explicit deleted flag on m.device_list_update EDUs ... and generally make it work.	2018-07-12 11:39:43 +01:00
Erik Johnston	0456e05977	Merge pull request #3505 from matrix-org/erikj/receipts_cahce ... Use stream cache in get_linearized_receipts_for_room	2018-07-12 09:46:29 +01:00
Erik Johnston	aff1dfdf3d	Update return value docstring	2018-07-12 09:45:37 +01:00
Matthew Hodgson	5797f5542b	WIP to announce deleted devices over federation ... Previously we queued up the poke correctly when the device was deleted, but then the actual EDU wouldn't get sent, as the device was no longer known. Instead, we now send EDUs for deleted devices too if there's a poke for them.	2018-07-12 01:32:39 +01:00
Erik Johnston	6ccefef07a	Use 'is not None' and add comments	2018-07-10 18:12:39 +01:00
Matthew Hodgson	ea752bdd99	s/becuase/because/g	2018-07-10 17:58:18 +01:00
Erik Johnston	05f5dabc10	Use stream cache in get_linearized_receipts_for_room ... This avoids us from uncessarily hitting the database when there has been no change for the room	2018-07-10 17:22:42 +01:00
Richard van der Hoff	c3c29aa196	Attempt to include db threads in cpu usage stats (#3496) ... Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.	2018-07-10 16:12:36 +01:00
Richard van der Hoff	e31e5dee38	Add CPU metrics for _fetch_event_list ... add a Measure block on _fetch_event_list, in the hope that we can better measure CPU usage here.	2018-07-09 18:15:54 +01:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Erik Johnston	3905c693c5	Invalidate cache on correct thread	2018-07-02 11:36:44 +01:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
David Baker	028490afd4	Fix error on deleting users pending deactivation ... Use simple_delete instead of simple_delete_one as commented	2018-06-26 10:52:52 +01:00
Erik Johnston	244484bf3c	Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" ... This reverts commit 1d009013b3.	2018-06-25 13:42:55 +01:00
Richard van der Hoff	1d009013b3	Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" ... This reverts commit ce0d911156, reversing changes made to b4a5d767a9.	2018-06-22 16:35:10 +01:00
Mark Haines	9850f66abe	Deleting from event_push_actions needs to use an index	2018-06-22 15:54:48 +01:00
Richard van der Hoff	200e11c5bf	Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable ... Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:18:51 +01:00
Erik Johnston	f8272813a9	Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:12:26 +01:00
Richard van der Hoff	1d7ad11747	Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation ... Make push actions rotation configurable	2018-06-22 15:07:31 +01:00
Erik Johnston	ce0d911156	Merge pull request #3431 from matrix-org/rav/erasure_visibility ... Support hiding events from deleted users	2018-06-22 15:06:44 +01:00
Erik Johnston	f79abda87f	Merge pull request #3427 from matrix-org/erikj/remove_filters ... remove dead filter_events_for_clients	2018-06-22 14:45:42 +01:00
Erik Johnston	75dc3ddeab	Make push actions rotation configurable	2018-06-22 14:44:37 +01:00
Richard van der Hoff	43e02c409d	Disable partial state group caching for wildcard lookups ... When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.	2018-06-22 11:52:07 +01:00
Richard van der Hoff	240f192523	Merge pull request #3382 from matrix-org/rav/optimise_state_groups ... Optimise state_group_cache update	2018-06-22 11:20:20 +01:00
Richard van der Hoff	70e6501913	Merge pull request #3419 from matrix-org/rav/events_per_request ... Log number of events fetched from DB	2018-06-22 11:17:56 +01:00
Richard van der Hoff	0495fe0035	Indirect evt_count updates via method call ... so that we can stub it for the sentinel and not have a billion failing UTs	2018-06-22 10:42:28 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	b088aafcae	Log number of events fetched from DB ... When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]	2018-06-21 06:15:03 +01:00
Richard van der Hoff	3ff8a619f5	UserErasureStore ... to store which users have been erased	2018-06-12 09:52:22 +01:00
Richard van der Hoff	bd348f0af6	remove dead filter_events_for_clients ... This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.	2018-06-12 09:51:31 +01:00
Richard van der Hoff	eb32b2ca20	Optimise state_group_cache update ... (1) matrix-org-hotfixes has removed the intern calls; let's do the same here. (2) remove redundant iteritems() so we can used an optimised db update.	2018-06-11 22:56:11 +01:00
David Baker	187a546bff	Merge pull request #3276 from matrix-org/dbkr/unbind ... Remove email addresses / phone numbers from ID servers when they're removed from synapse	2018-06-11 16:02:00 +01:00
Matthew Hodgson	c96d882a02	Merge branch 'develop' into matthew/filter_members	2018-06-10 12:26:14 +03:00
Richard van der Hoff	d3a8c9c55e	Fix sql error in _get_state_groups_from_groups ... If this was called with a `(type, None)` entry in types (which is supposed to return all state of type `type`), it would explode with a sql error.	2018-06-06 14:19:01 +01:00
Richard van der Hoff	f29b41fde9	Merge pull request #3324 from matrix-org/rav/remove_dead_method ... Remove was_forgotten_at	2018-06-04 18:18:08 +01:00
Erik Johnston	042eedfa2b	Add hacky cache factor override system	2018-06-04 15:39:28 +01:00
Matthew Hodgson	28f09fcdd5	Merge branch 'develop' into matthew/filter_members	2018-06-04 00:09:17 +03:00
Richard van der Hoff	c1f4118bb6	Remove was_forgotten_at ... This is unused. IT MUST DIE!!!1 ̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢ ̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *	2018-06-01 18:21:49 +01:00
Neil Johnson	71477f3317	Merge pull request #3264 from matrix-org/neil/sign-up-stats ... daily user type phone home stats	2018-06-01 13:42:01 +00:00
Richard van der Hoff	9f797a24a4	Handle RRs which arrive before their events	2018-06-01 14:01:43 +01:00
Richard van der Hoff	857e6fd8b6	Ignore depth when updating read-receipts ... Order read receipts by stream ordering instead of depth	2018-06-01 12:18:11 +01:00
Neil Johnson	4986b084f8	remove unnecessary INSERT	2018-06-01 10:50:40 +01:00
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	2018-05-31 19:03:47 +10:00
Amber Brown	872cf43516	Merge pull request #3303 from NotAFile/py3-memoryview ... use memoryview in py3	2018-05-30 12:51:42 +10:00
Amber Brown	debff7ae09	Merge pull request #3281 from NotAFile/py3-six-isinstance ... remaining isintance fixes	2018-05-30 12:44:46 +10:00
Richard van der Hoff	4a9cbdbc15	Exempt AS-registered users from doing gdpr	2018-05-29 19:54:32 +01:00
Neil Johnson	ab0ef31dc7	create users index on creation_ts	2018-05-29 17:51:08 +01:00
Neil Johnson	558f3d376a	create index in background	2018-05-29 17:47:55 +01:00
Neil Johnson	c379acd4fd	bump version	2018-05-29 17:47:28 +01:00
Adrian Tschira	4b9d0cde97	add remaining memoryview changes	2018-05-29 17:42:43 +02:00
Adrian Tschira	1afafb3497	use memoryview in py3 ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-29 17:14:34 +02:00
Matthew Hodgson	a6c8f7c875	add pydoc	2018-05-29 01:09:55 +01:00
Matthew Hodgson	7a6df013cc	merge develop	2018-05-29 00:25:22 +01:00
Amber Brown	a2eb5db4a0	update metrics to be in seconds	2018-05-28 19:10:27 +10:00
Amber Brown	754826a830	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-28 18:57:23 +10:00
Amber Brown	cc1349c06a	Merge pull request #3279 from NotAFile/py3-more-iteritems ... more six iteritems	2018-05-24 14:23:13 -05:00
Adrian Tschira	dd068ca979	remaining isintance fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:55:08 +02:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Adrian Tschira	6c16a4ec1b	more iteritems	2018-05-24 20:19:06 +02:00
Amber Brown	7ea07c7305	Merge pull request #3278 from NotAFile/py3-storage-base ... Py3 storage/_base.py	2018-05-24 13:08:09 -05:00
Amber Brown	1f69693347	Merge pull request #3244 from NotAFile/py3-six-4 ... replace some iteritems with six	2018-05-24 13:04:07 -05:00
Amber Brown	36501068d8	Merge pull request #3247 from NotAFile/py3-misc ... Misc Python3 fixes	2018-05-24 12:58:37 -05:00
Adrian Tschira	095292304f	Py3 storage/_base.py ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 18:24:12 +02:00
David Baker	77a23e2e05	Merge remote-tracking branch 'origin/develop' into dbkr/unbind	2018-05-24 16:20:53 +01:00
David Baker	2c7866d664	Hit the 3pid unbind endpoint on deactivation	2018-05-23 14:38:56 +01:00
Amber Brown	53cc2cde1f	cleanup	2018-05-22 17:32:57 -05:00
Erik Johnston	a17e901f4d	Remove unused string formatting param	2018-05-22 18:24:32 +01:00
Erik Johnston	5494c1d71e	Don't support limitless pagination ... The pagination storage function supported not specifiying a limit on the number of events returned. This was triggered when using the search or context API with a limit of zero, which the storage function took to mean not being limited.	2018-05-22 18:15:21 +01:00
Neil Johnson	d8cb7225d2	daily user type phone home stats	2018-05-22 18:09:09 +01:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Amber Brown	fcc525b0b7	rest of the changes	2018-05-21 19:48:57 -05:00
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:59:26 +02:00
Adrian Tschira	dcc235b47d	use stand-in value if maxint is not available ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:35:44 +02:00
Neil Johnson	644aac5f73	Tighter filtering for user_daily_visits	2018-05-18 17:10:35 +01:00
Neil Johnson	ef466b3a13	fix psql compatability bug	2018-05-18 15:51:21 +01:00
Richard van der Hoff	ed3125b0a1	Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred ... Fix error in handling receipts	2018-05-18 11:23:11 +01:00
Erik Johnston	fa30ac38cc	Merge pull request #3221 from matrix-org/erikj/purge_token ... Make purge_history operate on tokens	2018-05-18 10:35:23 +01:00
Richard van der Hoff	8b1c856d81	Fix error in handling receipts ... Fixes an error which has been happening ever since #2158 (v0.21.0-rc1): > TypeError: argument of type 'ObservableDeferred' is not iterable fixes #3234	2018-05-18 09:15:35 +01:00
Erik Johnston	f7906203f6	Merge pull request #3212 from matrix-org/erikj/epa_stream ... Use stream rather depth ordering for push actions	2018-05-17 12:01:21 +01:00
Erik Johnston	680530cc7f	Clarify comment	2018-05-16 11:47:29 +01:00
Erik Johnston	43e6e82c4d	Comments	2018-05-16 11:13:31 +01:00
Neil Johnson	dc8930ea9e	Merge pull request #3163 from matrix-org/cohort_analytics ... user visit data	2018-05-16 10:09:24 +00:00
Erik Johnston	c945af8799	Move and rename variable	2018-05-16 10:52:06 +01:00
Neil Johnson	be11a02c4f	remove empty line	2018-05-16 10:45:40 +01:00
Neil Johnson	31c2502ca8	style and further contraining query	2018-05-16 09:46:43 +01:00
Richard van der Hoff	8030a825c8	Merge pull request #3213 from matrix-org/rav/consent_handler ... ConsentResource to gather policy consent from users	2018-05-16 07:19:18 +01:00
Neil Johnson	05ac15ae82	Limit query load of generate_user_daily_visits ... The aim is to keep track of when it was last called and only query from that point in time	2018-05-15 17:01:33 +01:00
Erik Johnston	5f27ed75ad	Make purge_history operate on tokens ... As we're soon going to change how topological_ordering works	2018-05-15 16:23:50 +01:00
Erik Johnston	37dbee6490	Use events_to_purge table rather than token	2018-05-15 16:23:47 +01:00
Richard van der Hoff	47815edcfa	ConsentResource to gather policy consent from users ... Hopefully there are enough comments and docs in this that it makes sense on its own.	2018-05-15 15:11:59 +01:00
Neil Johnson	f077e97914	instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day	2018-05-14 13:50:58 +01:00
David Baker	8cbbfd16fb	Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate ... Part user from rooms on account deactivate	2018-05-14 11:31:48 +01:00
Neil Johnson	977765bde2	Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics	2018-05-14 09:31:42 +01:00
Erik Johnston	6406b70aeb	Use stream rather depth ordering for push actions ... This simplifies things as it is, but will also allow us to change the way we traverse topologically without having to update the way push actions work.	2018-05-11 15:30:11 +01:00
David Baker	217bc53c98	Many docstrings	2018-05-10 12:20:40 +01:00
Erik Johnston	bfe1f73855	Merge pull request #3199 from matrix-org/erikj/pagination_sync ... Refactor sync APIs to reuse pagination API	2018-05-09 16:16:56 +01:00
Erik Johnston	5adb75bcba	Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination ... Refactor get_recent_events_for_room return type	2018-05-09 16:07:14 +01:00
Erik Johnston	7ce98804ff	Fix up comment	2018-05-09 15:42:39 +01:00
Erik Johnston	cddf91c8b9	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change	2018-05-09 15:32:07 +01:00
Erik Johnston	9896dab8f6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination	2018-05-09 15:31:33 +01:00
Erik Johnston	75552d2148	Update comments	2018-05-09 15:15:38 +01:00
David Baker	46df23f581	Add the schema file	2018-05-09 15:07:54 +01:00
David Baker	7e8726b8fb	Part deactivated users in the background ... One room at a time so we don't take out the whole server with leave events, and restart at server restart.	2018-05-09 14:54:28 +01:00
Erik Johnston	c0e08dc45b	Remove unused code path from member change DB func ... The function is never called without a from_key, so we can remove all the handling for that scenario.	2018-05-09 14:31:32 +01:00
Erik Johnston	e2accd7f1d	Refactor sync APIs to reuse pagination API ... The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.	2018-05-09 13:43:39 +01:00
Erik Johnston	e5ab9cd24b	Don't unnecessarily require token to be stream token ... This allows calling the `get_recent_event_ids_for_room` function in more situations.	2018-05-09 11:58:35 +01:00
Erik Johnston	c4af4c24ca	Refactor get_recent_events_for_room return type ... There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs	2018-05-09 11:55:34 +01:00
Erik Johnston	05e0a2462c	Refactor pagination DB API to return concrete type ... This makes it easier to document what is being returned by the storage functions and what some functions expect as arguments.	2018-05-09 11:34:24 +01:00
Erik Johnston	7dd13415db	Remove unused from_token param	2018-05-09 10:58:16 +01:00
Erik Johnston	27cf170558	Refactor recent events func to use pagination func ... This also removes a cache that is unlikely to ever get hit.	2018-05-09 10:55:55 +01:00
Erik Johnston	23ec51c94c	Fix up comments and make function private	2018-05-09 09:55:19 +01:00
Erik Johnston	696f532453	Reuse existing pagination code for context API	2018-05-08 16:20:19 +01:00
Erik Johnston	3e6d306e94	Parse tokens before calling DB function	2018-05-08 16:18:58 +01:00
Erik Johnston	274b8c6025	Only fetch required fields from database	2018-05-08 16:15:25 +01:00
Erik Johnston	06c0d0ed08	Split paginate_room_events storage function	2018-05-08 16:14:26 +01:00
Erik Johnston	0a3b51c420	Merge pull request #3141 from matrix-org/erikj/fixup_state ... Refactor event storage to prepare for changes in state calculations	2018-05-03 10:39:20 +01:00
Erik Johnston	31c7c29d43	Fix up grammar	2018-05-03 10:38:58 +01:00
Neil Johnson	dd1a832419	remove user agent from data model, will just join on user_ips	2018-05-01 12:13:49 +01:00
Neil Johnson	d0857702e8	add inidexes based on usage	2018-05-01 12:12:57 +01:00
Matthew Hodgson	99a54bf2af	Merge pull request #3129 from matrix-org/matthew/fix_group_dups ... remove duplicates from groups tables	2018-04-30 11:47:25 +01:00
Richard van der Hoff	db75c86e84	Merge branch 'develop' into py3-xrange-1	2018-04-30 01:02:25 +01:00
Matthew Hodgson	adaf3ec87f	fix missing import	2018-04-28 22:39:15 +01:00
Matthew Hodgson	006e18b6bb	pep8	2018-04-28 22:32:24 +01:00
Matthew Hodgson	42c89c8215	make it work with sqlite	2018-04-28 22:27:30 +01:00
Adrian Tschira	d82b6ea9e6	Move more xrange to six ... plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:57:00 +02:00
Richard van der Hoff	fc149b4eeb	Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background	2018-04-27 14:31:23 +01:00
Richard van der Hoff	2a13af23bc	Use run_in_background in preference to preserve_fn ... While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.	2018-04-27 12:55:51 +01:00
Richard van der Hoff	9c3da24561	Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions ... Improve exception handling for background processes	2018-04-27 11:47:49 +01:00
Erik Johnston	6e10eed28e	Refactor event storage to not require state ... This is in preparation for using contexts that may or may not have the current_state_ids set. This will allow us to avoid unnecessarily pulling out state for an event on the master process when using workers. We also add a check to see if the state groups of the old extremities are the same as the new ones.	2018-04-27 11:38:02 +01:00
Richard van der Hoff	605defb9e4	Add missing consumeErrors ... In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.	2018-04-27 11:16:28 +01:00
Richard van der Hoff	9255a6cb17	Improve exception handling for background processes ... There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.	2018-04-27 11:07:40 +01:00
Erik Johnston	7ec8e798b4	Fix media admin APIs	2018-04-26 11:31:22 +01:00
Neil Johnson	fb6015d0a6	pep8	2018-04-25 17:56:11 +01:00
Neil Johnson	617bf40924	Generate user daily stats	2018-04-25 17:37:29 +01:00
Erik Johnston	22881b3d69	Also fix reindexing of search	2018-04-25 15:32:04 +01:00
Erik Johnston	ba3166743c	Fix quarantine media admin API	2018-04-25 15:11:18 +01:00
Matthew Hodgson	e3a373f002	remove duplicates from groups tables ... and rename inconsistently named indexes. Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!	2018-04-25 14:58:43 +01:00
Richard van der Hoff	c09a6daf09	Merge pull request #3110 from NotAFile/py3-six-queue ... Replace Queue with six.moves.queue	2018-04-20 15:35:00 +01:00
Richard van der Hoff	bc381d5798	Merge pull request #3117 from matrix-org/rav/refactor_have_events ... Refactor store.have_events	2018-04-20 10:26:12 +01:00
Richard van der Hoff	b1dfbc3c40	Refactor store.have_events ... It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.	2018-04-20 10:25:56 +01:00
Richard van der Hoff	639480e14a	Avoid creating events with huge numbers of prev_events ... In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.	2018-04-16 18:41:37 +01:00
Adrian Tschira	878995e660	Replace Queue with six.moves.queue ... and a six.range change which I missed the last time Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-16 00:46:21 +02:00
Erik Johnston	415aeefd89	Format docstring	2018-04-12 12:07:09 +01:00
Erik Johnston	23a7f9d7f4	Doc we raise on unknown event	2018-04-12 11:20:51 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	92e34615c5	Track where event stream processing have gotten up to	2018-04-11 12:13:40 +01:00
Richard van der Hoff	233699c42e	Merge pull request #2760 from Valodim/pypy ... Synapse on PyPy	2018-04-11 11:20:01 +01:00
Neil Johnson	d54cfbb7a8	fix typo	2018-04-10 17:38:16 +01:00
Richard van der Hoff	d1e56cfcd1	Fix pep8 error on psycopg2cffi hack	2018-04-10 11:35:29 +02:00
Vincent Breitmoser	89de934981	Use psycopg2cffi module instead of psycopg2 if running on pypy ... The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.	2018-04-10 11:29:52 +02:00
Neil Johnson	41e0611895	remove errant print	2018-04-09 18:44:20 +01:00
Neil Johnson	61b439c904	Fix msec to sec, again	2018-04-09 18:43:48 +01:00
Neil Johnson	87770300d5	Fix msec to sec	2018-04-09 18:38:59 +01:00
Neil Johnson	64bc2162ef	Fix psycopg2 interpolation	2018-04-09 17:50:36 +01:00
Luke Barnard	cd087a265d	Don't use redundant inlineCallbacks	2018-04-06 15:43:27 +01:00
Luke Barnard	87c864b698	join_rule -> join_policy	2018-04-06 15:43:27 +01:00
Luke Barnard	ae85c7804e	is_joinable -> join_rule	2018-04-06 15:43:27 +01:00
David Baker	edb45aae38	pep8	2018-04-06 15:43:27 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Neil Johnson	68b0ee4e8d	Merge pull request #3041 from matrix-org/r30_stats ... R30 stats	2018-04-05 15:37:37 +01:00
Richard van der Hoff	22284a6f65	Merge pull request #3060 from matrix-org/rav/kill_event_content ... Remove uses of events.content	2018-04-05 15:02:17 +01:00
Luke Barnard	917380e89d	NON NULL -> NOT NULL	2018-04-05 14:32:12 +01:00
Luke Barnard	700e5e7198	Use DEFAULT join_policy of "invite" in db	2018-04-05 14:01:17 +01:00
Luke Barnard	b214a04ffc	Document set_group_join_policy	2018-04-05 13:29:16 +01:00
Neil Johnson	0e5f479fc0	Review comments ... Use iteritems over item to loop over dict formatting	2018-04-05 12:16:46 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	2fe3f848b9	Remove uses of events.content	2018-03-29 23:17:12 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Neil Johnson	b4e37c6f50	pep8	2018-03-29 17:27:39 +01:00
Neil Johnson	9ee44a372d	Remove need for sqlite specific query	2018-03-29 16:45:34 +01:00
Neil Johnson	dc7c020b33	fix pep8 errors	2018-03-28 17:25:15 +01:00
Neil Johnson	241e4e8687	remove twisted deferral cruft	2018-03-28 16:25:53 +01:00
David Baker	929b34963d	OK, smallint it is then	2018-03-28 14:53:55 +01:00
David Baker	a838444a70	Grr. Copy the definition from is_admin	2018-03-28 14:50:30 +01:00
Neil Johnson	4262aba17b	bump schema version	2018-03-28 14:40:03 +01:00
Neil Johnson	86932be2cb	Support multi client R30 for psql	2018-03-28 14:36:53 +01:00
David Baker	a164270833	Make column definition that works on both dbs	2018-03-28 14:23:00 +01:00
David Baker	352e1ff9ed	Add schema delta file	2018-03-28 14:07:57 +01:00
David Baker	79452edeee	Add joinability for groups ... Adds API to set the 'joinable' flag, and corresponding flag in the table.	2018-03-28 14:03:37 +01:00
Neil Johnson	788e69098c	Add user_ips last seen index	2018-03-28 12:03:13 +01:00
Neil Johnson	0f890f477e	No need to cast in count_daily_users	2018-03-28 11:49:57 +01:00
Neil Johnson	545001b9e4	Fix search_user_dir multiple sqlite versions do different things	2018-03-28 11:19:45 +01:00
Neil Johnson	a32d2548d9	query and call for r30 stats	2018-03-28 10:39:13 +01:00
Neil Johnson	9187e0762f	count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home	2018-03-28 10:02:32 +01:00
Erik Johnston	f879127aaa	Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id ... Linearize calls to _generate_user_id	2018-03-28 10:00:31 +01:00
Erik Johnston	e6d87c93f3	Merge pull request #3030 from matrix-org/erikj/no_ujson ... Remove last usage of ujson	2018-03-28 10:00:06 +01:00
Erik Johnston	800cfd5774	Comment	2018-03-27 13:30:39 +01:00
Erik Johnston	152c2ac19e	Fix indent	2018-03-27 13:13:46 +01:00
Erik Johnston	e70287cff3	Comment	2018-03-27 13:13:38 +01:00
Erik Johnston	3f49e131d9	Add counter metrics for calculating state delta ... This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event.	2018-03-27 10:57:35 +01:00
Erik Johnston	fecb45e0c3	Remove last usage of ujson	2018-03-26 13:32:29 +01:00
Erik Johnston	8d6dc106d1	Don't use _cursor_to_dict in find_next_generated_user_id_localpart	2018-03-26 12:02:44 +01:00
Erik Johnston	fa72803490	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-19 11:41:01 +00:00
Erik Johnston	bf49d2dca8	Replace some ujson with simplejson to make it work	2018-03-16 00:55:44 +00:00
Erik Johnston	a8ce159be4	Replace some ujson with simplejson to make it work	2018-03-16 00:27:09 +00:00
Erik Johnston	5b631ff41a	Remove wrong comment	2018-03-16 00:07:08 +00:00
Erik Johnston	926ba76e23	Replace ujson with simplejson	2018-03-15 23:43:31 +00:00
Matthew Hodgson	056a6df546	Merge branch 'develop' into matthew/filter_members	2018-03-14 15:38:05 +00:00
Erik Johnston	ebd0127999	Merge pull request #2988 from matrix-org/erikj/split_profile_store ... Split up ProfileStore	2018-03-14 09:41:06 +00:00
Erik Johnston	f51565e023	Merge pull request #2993 from matrix-org/erikj/is_blocked ... Add is_blocked to worker store	2018-03-14 09:39:18 +00:00
Matthew Hodgson	9f77001e27	pep8	2018-03-14 00:07:47 +00:00
Matthew Hodgson	d144ed6ffb	fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990) ... Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.	2018-03-13 22:36:04 +00:00
Matthew Hodgson	ccca02846d	make it work	2018-03-13 22:31:41 +00:00
Matthew Hodgson	f0f9a0605b	remove comment now #2969 is fixed	2018-03-13 22:12:15 +00:00
Matthew Hodgson	12350e3f9a	merge proper fix to bug 2969	2018-03-13 22:11:58 +00:00
Matthew Hodgson	afbf4d3dcc	typoe	2018-03-13 19:48:04 +00:00
Matthew Hodgson	865377a70d	disable optimisation for searching for state groups ... when type filter includes wildcards on state_key	2018-03-13 19:46:04 +00:00
Erik Johnston	a08726fc42	Add is_blocked to worker store	2018-03-13 18:28:44 +00:00
Matthew Hodgson	b2aba9e430	build where_clause sanely	2018-03-13 18:13:44 +00:00
Matthew Hodgson	52f7e23c72	PR feedbackz	2018-03-13 18:07:55 +00:00
Matthew Hodgson	1b1c137771	fix bug #2926	2018-03-13 17:52:52 +00:00
Erik Johnston	3518d0ea8f	Split up ProfileStore	2018-03-13 17:36:50 +00:00
Richard van der Hoff	889a2a853a	Add Measure block for persist_events ... This seems like a useful thing to measure.	2018-03-13 10:01:42 +00:00
Matthew Hodgson	fdedcd1f4d	correctly handle None state_keys ... and fix include_other_types thinko	2018-03-12 01:39:06 +00:00
Matthew Hodgson	97c0496cfa	fix sqlite where clause	2018-03-12 00:27:06 +00:00
Matthew Hodgson	8713365265	typos	2018-03-11 20:10:25 +00:00
Matthew Hodgson	9b334b3f97	WIP experiment in lazyloading room members	2018-03-11 20:01:41 +00:00
Erik Johnston	8cb44da4aa	Fix race in sync when joining room ... The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.	2018-03-07 11:55:31 +00:00
Richard van der Hoff	b2932107bb	Merge pull request #2946 from matrix-org/rav/timestamp_to_purge ... Implement purge_history by timestamp	2018-03-06 11:20:23 +00:00
Erik Johnston	7aed50a038	Merge pull request #2948 from matrix-org/erikj/kill_as_sync ... Remove ability for AS users to call /events and /sync	2018-03-06 11:10:09 +00:00
Erik Johnston	b6c4b851f1	Merge pull request #2947 from matrix-org/erikj/split_directory_store ... Split Directory store	2018-03-05 18:17:32 +00:00
Erik Johnston	d4ffe61d4f	Remove ability for AS users to call /events and /sync ... This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144	2018-03-05 15:44:46 +00:00
Erik Johnston	69ce365b79	Fix cache invalidation on deletion	2018-03-05 15:29:03 +00:00
Erik Johnston	2e223163ff	Split Directory store	2018-03-05 15:11:30 +00:00
Richard van der Hoff	f8bfcd7e0d	Provide a means to pass a timestamp to purge_history	2018-03-05 14:37:23 +00:00
Richard van der Hoff	d032785aa7	Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts ... Test and fix find_first_stream_ordering_after_ts	2018-03-05 12:26:14 +00:00
Richard van der Hoff	c818fcab11	Test and fix find_first_stream_ordering_after_ts ... It seemed to suffer from a bunch of off-by-one errors.	2018-03-05 12:04:02 +00:00
Richard van der Hoff	06a14876e5	Add find_first_stream_ordering_after_ts ... Expose this as a public function which can be called outside a txn	2018-03-05 11:53:39 +00:00
Erik Johnston	42174946f8	Merge pull request #2934 from matrix-org/erikj/cache_fix ... Fix bug with delayed cache invalidation stream	2018-03-05 11:33:17 +00:00
Erik Johnston	efb79820b4	Fix bug with delayed cache invalidation stream ... We poked the notifier before updated the current token for the cache invalidation stream. This mean that sometimes the update wouldn't be sent until the next time a cache was invalidated.	2018-03-02 14:45:15 +00:00
Erik Johnston	fafa3e7114	Split registration store	2018-03-02 13:48:27 +00:00
Erik Johnston	1773df0632	Merge pull request #2925 from matrix-org/erikj/split_sig_fed ... Split out SignatureStore and EventFederationStore	2018-03-01 17:32:58 +00:00
Erik Johnston	9e08a93a7b	Merge pull request #2927 from matrix-org/erikj/read_marker_caches ... Improve caching for read_marker API	2018-03-01 17:12:34 +00:00
Erik Johnston	4b44f05f19	Fewer lies are better	2018-03-01 17:08:17 +00:00
Erik Johnston	a83c514d1f	Improve caching for read_marker API ... We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.	2018-03-01 17:08:17 +00:00
Erik Johnston	33bebb63f3	Add some caches to help read marker API	2018-03-01 17:08:17 +00:00
Erik Johnston	2ad4d5b5bb	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed	2018-03-01 16:59:39 +00:00
Erik Johnston	7c371834cc	Stub out broken function only used for cache	2018-03-01 16:44:13 +00:00
Erik Johnston	64346be26d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store	2018-03-01 16:26:42 +00:00
Erik Johnston	22518e2833	Merge pull request #2923 from matrix-org/erikj/stream_ago_worker ... Calculate stream_ordering_month_ago correctly on workers	2018-03-01 16:23:54 +00:00
Erik Johnston	884b26ae41	Remove unused variables	2018-03-01 16:23:48 +00:00
Erik Johnston	1b2af11650	Document abstract class and method better	2018-03-01 16:20:57 +00:00
Erik Johnston	872ff95ed4	Default stream_ordering_*_ago to None	2018-03-01 16:00:05 +00:00
Erik Johnston	22004b524e	Fix comment typo	2018-03-01 15:59:40 +00:00
Erik Johnston	f793bc3877	Split out stream store	2018-03-01 15:13:08 +00:00
Erik Johnston	6411f725be	Calculate stream_ordering_month_ago correctly on workers	2018-03-01 14:20:53 +00:00
Erik Johnston	a9a2d66cdd	Split out SignatureStore and EventFederationStore	2018-03-01 14:17:53 +00:00
Erik Johnston	0c8ba5dd1c	Split up RoomStore	2018-03-01 14:01:19 +00:00
Erik Johnston	493e25d554	Move storage functions for push calculations ... This will allow push actions for an event to be calculated on workers.	2018-02-27 13:58:16 +00:00
Erik Johnston	3594dbc6dc	Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation ... Fix missing invalidations for receipt storage	2018-02-27 11:34:26 +00:00
Erik Johnston	2311189ee4	Merge pull request #2903 from matrix-org/erikj/split_roommember_store ... Split out RoomMemberStore	2018-02-27 11:32:10 +00:00
Erik Johnston	c57607874c	Merge pull request #2901 from matrix-org/erikj/split_as_stores ... Split AS stores	2018-02-27 10:07:07 +00:00
Erik Johnston	8956f0147a	Add comment	2018-02-27 10:06:51 +00:00
Erik Johnston	e5b4a208ce	Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions ... Batch inserts into event_push_actions_staging	2018-02-26 14:45:40 +00:00
Erik Johnston	45b5fe9122	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push	2018-02-26 13:49:24 +00:00
Erik Johnston	d62ce972f8	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store	2018-02-23 11:46:24 +00:00
Erik Johnston	6ae9a3d2a6	Update copyright	2018-02-23 11:44:49 +00:00
Erik Johnston	a90c60912f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions	2018-02-23 11:26:31 +00:00
Erik Johnston	50e8657867	Merge pull request #2902 from matrix-org/erikj/split_events_store ... Split out get_events and co into a worker store	2018-02-23 11:23:52 +00:00
Erik Johnston	1cf9e071dd	Merge pull request #2899 from matrix-org/erikj/split_pushers ... Split PusherStore	2018-02-23 11:23:35 +00:00
Erik Johnston	d0957753bf	Merge pull request #2898 from matrix-org/erikj/split_push_rules_store ... Split PushRulesStore	2018-02-23 11:23:23 +00:00
Erik Johnston	70349872c2	Update copyright	2018-02-23 11:14:35 +00:00
Erik Johnston	eba93b05bf	Split EventsWorkerStore into separate file	2018-02-23 11:01:21 +00:00
Erik Johnston	bf8a36e080	Update copyright	2018-02-23 10:52:10 +00:00
Erik Johnston	5d0f665848	Remove redundant clock	2018-02-23 10:49:58 +00:00
Erik Johnston	3bd760628b	_event_persist_queue shouldn't be in worker store	2018-02-23 10:49:18 +00:00
Erik Johnston	eb9b5eec81	Update copyright	2018-02-23 10:42:39 +00:00
Erik Johnston	c2ecfcc3a4	Update copyright	2018-02-23 10:41:34 +00:00
Erik Johnston	7e6cf89dc2	Update copyright	2018-02-23 10:39:19 +00:00
Erik Johnston	26d37f7a63	Update copyright	2018-02-23 10:33:55 +00:00
Erik Johnston	bb73f55fc6	Use absolute imports	2018-02-23 10:31:16 +00:00
Erik Johnston	faeb369f15	Fix missing invalidations for receipt storage	2018-02-21 15:19:54 +00:00
Erik Johnston	3dec9c66b3	Split out RoomMemberStore	2018-02-21 12:07:26 +00:00
Erik Johnston	46244b2759	Split AS stores	2018-02-21 11:49:34 +00:00
Erik Johnston	27b094f382	Split out get_events and co into a worker store	2018-02-21 11:41:48 +00:00
Erik Johnston	c96d547f4d	Actually use new param	2018-02-21 11:03:42 +00:00
Erik Johnston	d15d237b0d	Split out EventPushActionWorkerStore	2018-02-21 11:01:13 +00:00
Erik Johnston	6f72765371	Split PusherStore	2018-02-21 10:54:21 +00:00
Erik Johnston	cbaad969f9	Split PushRulesStore	2018-02-21 10:43:31 +00:00
Erik Johnston	ca9b9d9703	Split AccountDataStore and TagStore	2018-02-21 10:15:04 +00:00
Erik Johnston	8fbb4d0d19	Raise exception in abstract method	2018-02-20 17:59:23 +00:00
Erik Johnston	95e4cffd85	Fix comment	2018-02-20 17:58:40 +00:00
Erik Johnston	e316bbb4c0	Use abstract base class to access stream IDs	2018-02-20 17:43:57 +00:00
Erik Johnston	f5ac4dc2d4	Split ReceiptsStore	2018-02-20 16:28:28 +00:00
Erik Johnston	24087bffa9	Ensure all push actions are deleted from staging	2018-02-20 12:34:31 +00:00
Erik Johnston	ad0ccf15ea	Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts	2018-02-20 12:34:28 +00:00
Erik Johnston	6ff8c87484	Batch inserts into event_push_actions_staging	2018-02-20 11:33:07 +00:00
Richard van der Hoff	d1a3325f99	(Really) fix tablescan of event_push_actions on purge ... commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/	2018-02-16 14:02:31 +00:00
Erik Johnston	6af025d3c4	Fix typo of double is_highlight	2018-02-16 11:35:31 +00:00
Erik Johnston	012e8e142a	Comments	2018-02-16 11:35:01 +00:00
Erik Johnston	b96278d6fe	Ensure that we delete staging push actions on errors	2018-02-15 15:47:06 +00:00
Erik Johnston	4810f7effd	Remove context.push_actions	2018-02-15 15:47:06 +00:00
Erik Johnston	c714c61853	Update event_push_actions table from staging table	2018-02-15 15:47:06 +00:00
Erik Johnston	acac21248c	Store push actions in staging area	2018-02-15 15:47:04 +00:00
Erik Johnston	106906a65e	Don't serialize current state over replication	2018-02-15 13:53:18 +00:00
Erik Johnston	fd1601c596	Fix state group storage bug in workers ... We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.	2018-02-15 11:04:32 +00:00
Richard van der Hoff	b8d821aa68	Merge pull request #2867 from matrix-org/rav/rework_purge ... purge_history cleanups	2018-02-15 09:49:07 +00:00
Richard van der Hoff	d28ec43e15	Merge pull request #2769 from matrix-org/matthew/hit_the_gin ... switch back from GIST to GIN indexes	2018-02-14 16:59:03 +00:00
Richard van der Hoff	39bf47319f	purge_history: fix sqlite syntax error ... apparently sqlite insists on indexes being named	2018-02-14 16:42:19 +00:00
Richard van der Hoff	ac27f6a35e	purge_history: handle sqlite asshattery ... apparently creating a temporary table commits the transaction. because that's a useful thing.	2018-02-14 16:41:12 +00:00
Richard van der Hoff	5978dccff0	remove overzealous exception handling	2018-02-14 15:54:09 +00:00
Richard van der Hoff	278d21b5e4	purge_history: fix index use ... event_push_actions doesn't have an index on event_id, so we need to specify room_id.	2018-02-14 15:44:51 +00:00
Richard van der Hoff	5fcbf1e07c	Rework event purge to use a temporary table ... ... which should speed things up by reducing the amount of data being shuffled across the connection	2018-02-14 11:02:22 +00:00
Erik Johnston	c0c9327fe0	Merge pull request #2854 from matrix-org/erikj/event_create_worker ... Create a worker for event creation	2018-02-13 18:07:10 +00:00
Richard van der Hoff	d627174da2	Fix log message in purge_history ... (we don't just remove remote events)	2018-02-13 16:51:21 +00:00
Richard van der Hoff	ddb6a79b68	Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin	2018-02-13 16:45:36 +00:00
Richard van der Hoff	0b27ae8dc3	move search reindex to schema 47 ... We're up to schema v47 on develop now, so this will have to go in there to have an effect. This might cause an error if somebody has already run it in the v46 guise, and runs it again in the v47 guise, because it will cause a duplicate entry in the bbackground_updates table. On the other hand, the entry is removed once it is complete, and it is unlikely that anyone other than matrix.org has run it on v46. The update itself is harmless to re-run because it deliberately copes with the index already existing.	2018-02-13 16:44:46 +00:00
Richard van der Hoff	4a6d551704	GIN reindex: Fix syntax errors, improve exception handling	2018-02-13 16:44:46 +00:00
Richard van der Hoff	bfdf7b9237	Merge pull request #2864 from matrix-org/rav/persist_event_caching ... Use StateResolutionHandler to resolve state in persist_events	2018-02-13 14:45:57 +00:00
Richard van der Hoff	630caf8a70	style nit	2018-02-13 14:29:22 +00:00
Richard van der Hoff	a9b712e9dc	Merge branch 'develop' into matthew/gin_work_mem	2018-02-13 12:16:01 +00:00
Richard van der Hoff	10b34dbb9a	Merge pull request #2858 from matrix-org/rav/purge_updates ... delete_local_events for purge_room_history	2018-02-09 14:11:00 +00:00
Richard van der Hoff	39a6b35496	purge: move room_depth update to end ... ... to avoid locking the table for too long	2018-02-09 13:07:41 +00:00
Richard van der Hoff	74fcbf741b	delete_local_events for purge_history ... Add a flag which makes the purger delete local events	2018-02-09 13:07:41 +00:00
Richard van der Hoff	e571aef06d	purge: Move cache invalidation to more appropriate place ... it was a bit of a non-sequitur there	2018-02-09 13:07:41 +00:00
Richard van der Hoff	61ffaa8137	bump purge logging to info ... this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.	2018-02-09 13:07:41 +00:00
Richard van der Hoff	671540dccf	rename delete_old_state -> purge_history ... (beacause it deletes more than state)	2018-02-09 13:07:41 +00:00
Erik Johnston	24dd73028a	Add replication http endpoint for event sending	2018-02-07 10:32:32 +00:00
Erik Johnston	3d33eef6fc	Store state groups separately from events (#2784) ... * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints	2018-02-06 14:31:24 +00:00
Richard van der Hoff	b31bf0bb51	Merge pull request #2849 from matrix-org/rav/clean_up_state_delta ... Remove redundant return value from _calculate_state_delta	2018-02-05 17:42:20 +01:00
Richard van der Hoff	ebfe64e3d6	Use StateResolutionHandler to resolve state in persist events ... ... and thus benefit (hopefully) from its cache.	2018-02-05 16:23:26 +00:00
Richard van der Hoff	225dc3b4cb	Flatten _get_new_state_after_events ... rejig the if statements to simplify the logic and reduce indentation	2018-02-05 16:23:25 +00:00
Richard van der Hoff	9fcbbe8e7d	Check that events being persisted have state_group	2018-02-05 16:23:25 +00:00
Richard van der Hoff	447aed42d2	Add event_map param to resolve_state_groups	2018-02-05 16:23:25 +00:00
Richard van der Hoff	ee6fb4cf85	Remove redundant return value from _calculate_state_delta ... we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.	2018-02-05 16:23:20 +00:00
Richard van der Hoff	3c7b480ba3	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-05 16:12:14 +00:00
Richard van der Hoff	c46e75d3d8	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-05 15:43:22 +00:00
Richard van der Hoff	80b8a28100	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-04 00:23:06 +00:00
Richard van der Hoff	bd25f9cf36	Clean up work_mem handling ... Add some comments and improve exception handling when twiddling work_mem for the search update	2018-02-03 23:05:41 +00:00
Richard van der Hoff	4eeae7ad65	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-03 22:59:45 +00:00
Richard van der Hoff	bb9f0f3cdb	Merge branch 'develop' into matthew/gin_work_mem	2018-02-03 22:40:28 +00:00
Richard van der Hoff	6b02fc80d1	Reinstate event_search_postgres_gist handler ... People may have queued updates for this, so we can't just delete it.	2018-02-02 14:32:51 +00:00
Richard van der Hoff	78d6ddba86	Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta ... factor _get_new_state_after_events out of _calculate_state_delta	2018-02-02 15:27:15 +01:00
Travis Ralston	6e87b34f7b	Merge branch 'develop' into travis/admin-list-media	2018-02-01 18:05:47 -07:00
Richard van der Hoff	77c0629ebc	Merge pull request #2837 from matrix-org/rav/fix_quarantine_media ... Fix sql error in quarantine_media	2018-02-01 11:45:58 +01:00
Travis Ralston	e16e45b1b4	pep8 ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 15:30:38 -07:00
Richard van der Hoff	e1e4ec9f9d	factor _get_new_state_after_events out of _calculate_state_delta ... This reduces the scope of a bunch of variables	2018-01-31 21:32:09 +00:00
Travis Ralston	63c4383927	Documentation and naming ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 08:07:52 -07:00
Richard van der Hoff	773f0eed1e	Fix sql error in quarantine_media	2018-01-30 15:02:51 +00:00
Richard van der Hoff	b387ee17b6	Improve exception handling in persist_event ... 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs	2018-01-29 17:05:33 +00:00
Richard van der Hoff	46022025ea	Fix SQL for user search ... fix some syntax errors for user search when search_all_users is enabled fixes #2801, hopefully	2018-01-27 17:21:57 +00:00
Travis Ralston	a94d9b6b82	Appease the linter ... These are ids anyways, not mxc uris. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:49:46 -07:00
Travis Ralston	5552ed9a7f	Add an admin route to get all the media in a room ... This is intended to be used by administrators to monitor the media that is passing through their server, if they wish. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:37:53 -07:00
Richard van der Hoff	36da256cc6	Merge pull request #2805 from matrix-org/rav/log_state_res ... Log room when doing state resolution	2018-01-17 18:05:04 +00:00
Richard van der Hoff	1224612a79	Log room when doing state resolution ... Mostly because it helps figure out what is prompting the resolution	2018-01-17 17:11:59 +00:00
Erik Johnston	a87006f9c7	Merge pull request #2783 from matrix-org/erikj/media_last_accessed ... Keep track of last access time for local media	2018-01-17 16:39:02 +00:00
Matthew Hodgson	06db5c4b76	Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql ... fix SQL when searching all users	2018-01-17 16:27:54 +00:00
Matthew Hodgson	2d9ab533f9	fix SQL when searching all users	2018-01-17 15:58:52 +00:00
Richard van der Hoff	390093d45e	Split resolve_events into two functions ... ... so that the return type doesn't depend on the arg types	2018-01-17 15:44:31 +00:00
Erik Johnston	2fb3a28c98	Remove lost comment	2018-01-17 14:59:44 +00:00
Erik Johnston	05f98a2224	Keep track of last access time for local media	2018-01-17 10:24:43 +00:00
Richard van der Hoff	3d12d97415	Track DB scheduling delay per-request ... For each request, track the amount of time spent waiting for a db connection. This entails adding it to the LoggingContext and we may as well add metrics for it while we are passing.	2018-01-16 17:23:32 +00:00
Richard van der Hoff	8615f19d20	rework runInteraction in terms of runConnection ... ... so that we can share the code	2018-01-16 17:08:29 +00:00
Richard van der Hoff	64ddec1bc0	Fix a logcontext leak in persist_events ... ObserveableDeferred expects its callbacks to be called without any logcontexts, whereas it turns out we were calling them with the logcontext of the request which initiated the persistence loop. It seems wrong that we are attributing work done in the persistence loop to the request that happened to initiate it, so let's solve this by dropping the logcontext for it. (I'm not sure this actually causes any real problems other than messages in the debug log, but let's clean it up anyway)	2018-01-16 11:47:36 +00:00
hera	174eacc8ba	oops	2018-01-09 18:14:32 +00:00
Matthew Hodgson	a66f489678	fix GIST->GIN switch	2018-01-09 16:55:51 +00:00
Matthew Hodgson	e79db0a673	switch back from GIST to GIN indexes	2018-01-09 16:37:48 +00:00
Matthew Hodgson	e365ad329f	oops, tweak work_mem when actually storing	2018-01-09 16:30:30 +00:00
Matthew Hodgson	19f9227643	avoid 80s GIN inserts by tweaking work_mem ... see https://github.com/matrix-org/synapse/issues/2753 for details	2018-01-09 16:25:04 +00:00
Erik Johnston	b30cd5b107	Remove dead code related to default thumbnails	2018-01-09 14:38:33 +00:00
Matthew Hodgson	cdc2cb5d11	fix StoreError syntax	2017-12-05 11:09:31 +00:00
Matthew Hodgson	74e0cc74ce	fix pep8 and tests	2017-12-04 15:11:38 +00:00
Matthew Hodgson	1bd40ca73e	switch to a simpler 'search_all_users' button as per review feedback	2017-12-04 14:58:39 +00:00
Matthew Hodgson	f397153dfc	Merge branch 'develop' into matthew/search-all-local-users	2017-11-30 01:51:38 +00:00
Matthew Hodgson	f61e107f63	remove null constraint on user_dir.room_id	2017-11-30 01:43:50 +00:00
Matthew Hodgson	4b1fceb913	fix alternation operator for FTS4 - how did this ever work!?	2017-11-30 01:34:03 +00:00
Matthew Hodgson	a4bb133b68	fix thinkos galore	2017-11-30 01:17:15 +00:00
Matthew Hodgson	3241c7aac3	untested WIP but might actually work	2017-11-29 18:27:05 +00:00
Matthew Hodgson	47d99a20d5	Add user_directory_include_pattern config param to expand search results to additional users ... Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive. It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users... ...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too. Committing it here as a WIP	2017-11-29 16:46:45 +00:00
Richard van der Hoff	2c6d63922a	Remove pushers when deleting access tokens ... Whenever an access token is invalidated, we should remove the associated pushers.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	5a4da5bf78	Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error ... Fix error on sqlite 3.7	2017-11-27 12:25:48 +00:00
Richard van der Hoff	63ccaa5873	Avoid retrying forever on IntegrityError	2017-11-27 12:00:07 +00:00
Richard van der Hoff	6b48b3e277	fix sql fails	2017-11-22 18:06:24 +00:00
Richard van der Hoff	2908f955d1	Check database in has_completed_background_updates ... so that the right thing happens on workers.	2017-11-22 18:02:15 +00:00
Richard van der Hoff	0b4aa2dc21	Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert ... Avoid locking account_data tables for upserts	2017-11-21 13:39:14 +00:00
Richard van der Hoff	7098b65cb8	Fix error on sqlite 3.7 ... Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572.	2017-11-21 11:14:17 +00:00
Richard van der Hoff	c46139a17e	Avoid locking account_data tables for upserts	2017-11-16 18:08:01 +00:00
Richard van der Hoff	06e5bcfc83	Avoid locking for upsert on pushers tables ... * replace the upsert into deleted_pushers with an insert * no need to lock for upsert on pusher_throttle	2017-11-16 17:52:23 +00:00
Richard van der Hoff	77a1227870	Fix broken ref to IntegrityError	2017-11-16 16:03:38 +00:00
Richard van der Hoff	7ab2b69e18	Avoid locking pushers table on upsert ... Now that _simple_upsert will retry on IntegrityError, we don't need to lock the table.	2017-11-16 15:32:01 +00:00
Richard van der Hoff	10aaa1bc15	_simple_upsert: retry on IntegrityError ... wrap the call to _simple_upsert_txn in a loop so that we retry on an integrityerror: this means we can avoid locking the table provided there is an unique index.	2017-11-16 15:30:15 +00:00
Richard van der Hoff	cdc9e50a5d	Cleanup in _simple_upsert_txn ... Bail out early to reduce indentation	2017-11-16 15:29:10 +00:00
Richard van der Hoff	f959c01600	Merge pull request #2661 from matrix-org/rav/statereadstore ... Pull out bits of StateStore to a mixin	2017-11-15 17:23:01 +00:00
Richard van der Hoff	1b83c09c03	Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs ... Remove preserve_context_over_{fn, deferred}	2017-11-15 11:13:53 +00:00
Richard van der Hoff	35a4b63240	Pull out bits of StateStore to a mixin ... ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655.	2017-11-14 11:43:58 +00:00
Richard van der Hoff	4dd1bfa8c1	Revert "Revert "move _state_group_cache to statestore"" ... We're going to fix this properly on this branch, so that the _state_group_cache can end up in StateGroupReadStore. This reverts commit ab335edb02.	2017-11-14 11:43:58 +00:00
Richard van der Hoff	7e6fa29cb5	Remove preserve_context_over_{fn, deferred} ... Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.	2017-11-14 11:22:42 +00:00
Richard van der Hoff	6cfee09be9	Make __init__ consitstent across Store heirarchy ... Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)	2017-11-13 10:46:07 +00:00
Erik Johnston	ab335edb02	Revert "move _state_group_cache to statestore" ... This reverts commit f5cf3638e9.	2017-11-13 10:05:33 +00:00
Erik Johnston	bfbf1e1f1a	Up cache size of get_global_account_data_by_type_for_user	2017-11-13 09:52:11 +00:00
Erik Johnston	2dce6b15c3	Fix typo	2017-11-09 15:56:16 +00:00
Richard van der Hoff	618c7b816a	Merge pull request #2656 from matrix-org/rav/fix_deactivate ... Fix 'NoneType' not iterable in /deactivate	2017-11-09 15:20:35 +00:00
Richard van der Hoff	889102315e	Fix 'NoneType' not iterable in /deactivate ... make sure we actually return a value from user_delete_access_tokens	2017-11-09 15:15:33 +00:00
Erik Johnston	e8814410ef	Have an explicit API to update room config	2017-11-08 16:13:27 +00:00
Erik Johnston	94ff2cda73	Revert "Modify group room association API to allow modification of is_public"	2017-11-08 15:43:34 +00:00
Erik Johnston	167eb01d83	Merge pull request #2637 from spantaleev/avoid-noop-media-deletes ... Avoid no-op media deletes	2017-11-08 11:53:27 +00:00
Richard van der Hoff	02a9a93bde	Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res ... Fix bug in state group storage	2017-11-08 09:22:13 +00:00
Matthew Hodgson	228ccf1fe3	Merge pull request #2643 from matrix-org/matthew/user_dir_typos ... Fix various embarrassing typos around user_directory and add some doc.	2017-11-07 17:31:11 +00:00
Richard van der Hoff	f5cf3638e9	move _state_group_cache to statestore ... this is internal to statestore, so let's keep it there.	2017-11-07 16:43:00 +00:00
Luke Barnard	38b265cb51	Remember to pick is_admin out of the db	2017-11-07 11:24:04 +00:00
Matthew Hodgson	631fa4a1b7	create new indexes before dropping old ones to keep safetynet in place	2017-11-07 10:41:55 +00:00
Matthew Hodgson	4ad883398f	s/users_in_pubic_room/users_in_public_rooms/g	2017-11-04 19:39:40 +00:00
Matthew Hodgson	d802e8ca6a	s/users_in_pubic_room/users_in_public_rooms/g	2017-11-04 19:38:13 +00:00
Matthew Hodgson	a100700630	fix copyright....	2017-11-04 19:35:49 +00:00
Matthew Hodgson	b6b075fd49	s/popualte/populate/	2017-11-04 19:35:33 +00:00
Slavi Pantaleev	805196fbeb	Avoid no-op media deletes ... If there are no media entries to delete, avoid creating transactions, prepared statements and unnecessary log entries. Signed-off-by: Slavi Pantaleev <slavi@devture.com>	2017-11-04 09:50:15 +02:00
Richard van der Hoff	bc8a5c0330	Notify auth providers on logout ... Provide a hook by which auth providers can be notified of logouts.	2017-11-01 16:51:51 +00:00
Matthew Hodgson	275826f234	Merge pull request #2617 from matrix-org/matthew/auto-displayname ... automatically set default displayname on register	2017-11-01 16:21:16 +00:00
David Baker	fbbacb284e	Merge pull request #2613 from matrix-org/rav/kill_refresh_tokens ... Remove the last vestiges of refresh_tokens	2017-11-01 15:57:35 +00:00
Matthew Hodgson	9f7a555b4e	switch to setting default displayname in the storage layer ... to avoid clobbering guest user displaynames on registration	2017-11-01 15:51:30 +00:00
Luke Barnard	99354b430e	Merge pull request #2612 from matrix-org/luke/groups-room-relationship-is-public ... Modify group room association API to allow modification of is_public	2017-11-01 11:08:36 +00:00
Luke Barnard	318a249c8b	Leave is_public as required argument of update_room_group_association	2017-11-01 09:36:01 +00:00
Richard van der Hoff	356bcafc44	Remove the last vestiges of refresh_tokens	2017-10-31 20:35:58 +00:00
Luke Barnard	20fe347906	Modify group room association API to allow modification of is_public ... also includes renamings to make things more consistent.	2017-10-31 17:04:28 +00:00
Richard van der Hoff	1650eb5847	DB schema interface for password auth providers ... Provide an interface by which password auth providers can register db schema files to be run at startup	2017-10-31 14:01:53 +00:00
Erik Johnston	4ab8abbc2b	Merge branch 'erikj/attestation_local_fix' of github.com:matrix-org/synapse into develop	2017-10-27 15:07:08 +01:00
Luke Barnard	5451cc7792	Request is_public from database	2017-10-27 11:27:43 +01:00
Luke Barnard	6362298fa5	Create groups with is_public = True	2017-10-27 11:04:20 +01:00
Erik Johnston	82d8c1bacb	Fixup	2017-10-27 10:30:21 +01:00
Erik Johnston	195abfe7a5	Remove incorrect attestations	2017-10-27 09:58:13 +01:00
Luke Barnard	007cd48af6	Recreate groups table instead of adding column ... Adding a column with non-constant default not possible in sqlite3	2017-10-26 17:55:22 +01:00
Luke Barnard	713e60b9b6	Awful hack to get default true	2017-10-26 17:38:14 +01:00
Luke Barnard	cfa4e658e0	Bump schema version to 46	2017-10-26 17:23:49 +01:00
Luke Barnard	9b2feef9eb	Add is_public to groups table to allow for private groups ... Prevent group API access to non-members for private groups Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.	2017-10-26 16:51:32 +01:00
Richard van der Hoff	eaaabc6c4f	replace 'except:' with 'except Exception:' ... what could possibly go wrong	2017-10-23 15:52:32 +01:00
Richard van der Hoff	2e9f5ea31a	Fix logcontext handling for persist_events ... * don't use preserve_context_over_deferred, which is known broken. * remove a redundant preserve_fn. * add/improve some comments	2017-10-17 10:59:30 +01:00
Luke Barnard	2c5972f87f	Implement GET /groups/$groupId/invited_users	2017-10-16 15:31:11 +01:00
Erik Johnston	818b08d0e4	peeeeeeeeep8888888888888888888888888888	2017-10-11 15:54:00 +01:00
Erik Johnston	ea18996f54	Fix group stream replication ... The stream update functions expect the storage function to return a list of tuples.	2017-10-11 15:44:39 +01:00
Erik Johnston	c2c47550f9	Fix schema delta versions	2017-10-11 13:23:15 +01:00
Erik Johnston	535cc49f27	Merge pull request #2466 from matrix-org/erikj/groups_merged ... Initial Group Implementation	2017-10-11 13:20:07 +01:00
Erik Johnston	e8496efe84	Fix up comment	2017-10-09 15:17:34 +01:00
Erik Johnston	11d62f43c9	Invalidate cache	2017-10-03 14:12:28 +01:00
Erik Johnston	e4ab96021e	Update comments	2017-10-03 14:10:41 +01:00
Erik Johnston	30848c0fcd	Ignore incoming events for rooms that we have left ... When synapse receives an event for a room its not in over federation, it double checks with the remote server to see if it is in fact in the room. This is done so that if the server has forgotten about the room (usually as a result of the database being dropped) it can recover from it. However, in the presence of state resets in large rooms, this can cause a lot of work for servers that have legitimately left. As a hacky solution that supports both cases we drop incoming events for rooms that we have explicitly left. This means that we no longer support the case of servers having forgotten that they've rejoined a room, but that is sufficiently rare that we're not going to support it for now.	2017-10-03 11:18:21 +01:00
David Baker	27955056e0	Merge branch 'develop' into erikj/groups_merged	2017-10-02 16:20:41 +01:00
Erik Johnston	768f00dedb	Up the limits on number of url cache entries to delete at one time	2017-09-28 14:27:27 +01:00
Erik Johnston	4dc07e93a8	Add old indices	2017-09-28 14:10:33 +01:00
Erik Johnston	93247a424a	Only pull out local media that were for url cache	2017-09-28 13:48:14 +01:00
Erik Johnston	7a44c01d89	Fix typo	2017-09-28 12:46:04 +01:00
Erik Johnston	ae79764fe5	Change expires column to expires_ts	2017-09-28 12:37:53 +01:00
Erik Johnston	77f1d24de3	More brackets	2017-09-28 12:23:15 +01:00
Erik Johnston	9ccb4226ba	Delete expired url cache data	2017-09-28 12:18:06 +01:00
Erik Johnston	17b8e2bd02	Add remove room API	2017-09-26 15:52:41 +01:00
Erik Johnston	a8e2a3df32	Add unique index to group_rooms table	2017-09-26 15:39:21 +01:00
Erik Johnston	e1dec2f1a7	Remove user from group summary when the leave the group	2017-09-21 16:09:57 +01:00
Erik Johnston	197d82dc07	Correctly return next token	2017-09-20 11:12:11 +01:00
Erik Johnston	069ae2df12	Fix initial sync	2017-09-20 10:52:12 +01:00
Richard van der Hoff	e76d1135dd	Invalidate signing key cache when we gat an update ... This might make the cache slightly more efficient.	2017-09-20 01:32:42 +01:00
Erik Johnston	4a9b1cf253	Add user profiles to summary from group server	2017-08-25 16:23:58 +01:00
Erik Johnston	27ebc5c8f2	Add remote profile cache	2017-08-25 11:25:47 +01:00
Erik Johnston	97c544f91f	Add _simple_update	2017-08-25 11:11:37 +01:00
Erik Johnston	2800983f3e	Merge pull request #2410 from matrix-org/erikj/groups_publicise ... Add ability to publicise group membership	2017-08-21 16:51:56 +01:00
Erik Johnston	8b50fe5330	Use BOOLEAN rather than TEXT type	2017-08-21 16:37:29 +01:00
Erik Johnston	175a01f56c	Groups: Fix mising json.load in initial sync	2017-08-21 14:45:56 +01:00
Erik Johnston	ef8e578677	Add bulk group publicised lookup API	2017-08-09 13:36:22 +01:00
Erik Johnston	b880ff190a	Allow update group publicity	2017-08-08 14:19:41 +01:00
Erik Johnston	05e21285aa	Store whether the user wants to publicise their membership of a group	2017-08-08 13:01:46 +01:00
Erik Johnston	966a70f1fa	Update comment	2017-07-24 17:49:39 +01:00
Erik Johnston	629cdfb124	Use join rather than joined, etc.	2017-07-24 14:54:05 +01:00
Erik Johnston	ed666d3969	Fix all the typos	2017-07-24 14:05:09 +01:00
Erik Johnston	b76ef6ccb8	Include users membership in group in summary API	2017-07-24 13:55:39 +01:00
Erik Johnston	851aeae7c7	Check users/rooms are in group before adding to summary	2017-07-24 13:40:56 +01:00
Erik Johnston	96917d5552	Merge pull request #2378 from matrix-org/erikj/group_sync_support ... Add groups to sync stream	2017-07-21 11:05:39 +01:00
Erik Johnston	0401604222	Merge pull request #2377 from matrix-org/erikj/group_profile_update ... Add update group profile API	2017-07-20 17:53:39 +01:00
Erik Johnston	2cc998fed8	Fix replication. And notify	2017-07-20 17:13:18 +01:00
Erik Johnston	c544188ee3	Add groups to sync stream	2017-07-20 16:36:42 +01:00
Erik Johnston	8209b5f033	Fix a storage desc	2017-07-20 16:22:22 +01:00
Erik Johnston	57826d645b	Fix typo	2017-07-20 13:15:22 +01:00
Erik Johnston	6f443a74cf	Add update group profile API	2017-07-20 09:46:33 +01:00
Erik Johnston	94ecd871a0	Fix typos	2017-07-18 16:38:54 +01:00
Erik Johnston	3e703eb04e	Comment	2017-07-18 10:17:25 +01:00
Erik Johnston	508460f240	Remove sync stuff	2017-07-18 09:55:46 +01:00
Erik Johnston	4540730111	Remove unused tables	2017-07-18 09:38:15 +01:00
Erik Johnston	e96ee95a7e	Remove sync stuff	2017-07-18 09:38:08 +01:00
Erik Johnston	2f9eafdd36	Add local group server support	2017-07-17 12:03:49 +01:00
Erik Johnston	85fda57208	Add DEFAULT_ROLE_ID	2017-07-14 14:03:54 +01:00
Erik Johnston	4b203bdba5	Correctly increment orders	2017-07-14 14:02:31 +01:00
Erik Johnston	3b0470dba5	Remove unused functions	2017-07-13 13:54:01 +01:00
Erik Johnston	8575e3160f	Comments	2017-07-13 13:52:41 +01:00
Erik Johnston	26451a09eb	Comments	2017-07-12 14:47:18 +01:00
Erik Johnston	a62406aaa5	Add group summary APIs	2017-07-12 11:36:15 +01:00
Erik Johnston	6d586dc05c	Comment	2017-07-12 09:58:37 +01:00
Erik Johnston	410b4e14a1	Move comment	2017-07-11 15:44:18 +01:00
Erik Johnston	e52c391cd4	Rename column to attestation_json	2017-07-11 14:25:46 +01:00
Erik Johnston	0aac30d53b	Comments	2017-07-11 14:23:50 +01:00
Erik Johnston	83936293eb	Comments	2017-07-11 11:42:25 +01:00
Erik Johnston	b8ca494ee9	Initial group server implementation	2017-07-10 15:44:15 +01:00
Erik Johnston	f502b0dea1	Perf: Don't filter events for push ... We know the users are joined and we can explicitly check for if they are ignoring the user, so lets do that.	2017-07-07 14:04:40 +01:00
Erik Johnston	ed9a7f5436	Merge pull request #2309 from matrix-org/erikj/user_ip_repl ... Fix up user_ip replication commands	2017-07-06 14:33:14 +01:00
Erik Johnston	b5e8d529e6	Define CACHE_SIZE_FACTOR once	2017-07-04 09:56:44 +01:00
Erik Johnston	6ff14ddd2e	Make into list	2017-06-29 15:47:37 +01:00
Erik Johnston	5946aa0877	Prefill forward extrems and event to state groups	2017-06-29 15:38:48 +01:00
Erik Johnston	8c23221666	Fix up	2017-06-27 15:53:45 +01:00
Erik Johnston	a0a561ae85	Fix up client ips to read from pending data	2017-06-27 14:46:12 +01:00
Erik Johnston	ed3d0170d9	Batch upsert user ips	2017-06-27 13:37:04 +01:00
Erik Johnston	8abdd7b553	Fix up indices for users_who_share_rooms	2017-06-26 14:01:30 +01:00
Erik Johnston	7fe8ed1787	Store URL cache preview downloads seperately ... This makes it easier to clear old media out at a later date	2017-06-23 11:14:11 +01:00
Erik Johnston	e0004aa28a	Add desc	2017-06-22 10:03:48 +01:00
Erik Johnston	b668112320	Merge pull request #2296 from matrix-org/erikj/dont_appserver_shar ... Don't work out users who share room with appservice users	2017-06-21 14:50:24 +01:00
Erik Johnston	dae9a00a28	Initialise exclusive_user_regex	2017-06-21 14:19:33 +01:00
Krombel	4202fba82a	Merge branch 'develop' into avoid_duplicate_filters	2017-06-21 14:48:21 +02:00
Krombel	812c030e87	replaced json.dumps with encode_canonical_json	2017-06-21 14:48:12 +02:00
Erik Johnston	385dcb7c60	Handle thumbnail urls	2017-06-19 17:48:28 +01:00
Erik Johnston	b8b936a6ea	Add API to quarantine media	2017-06-19 17:39:21 +01:00
Erik Johnston	d7fe6b356c	Add shutdown room API	2017-06-19 12:37:27 +01:00
Erik Johnston	dfeca6cf40	Merge pull request #2286 from matrix-org/erikj/split_out_user_dir ... Split out user directory to a separate process	2017-06-16 13:01:19 +01:00
Erik Johnston	6aa5bc8635	Initial worker impl	2017-06-16 11:47:11 +01:00
Erik Johnston	d8f47d2efa	Merge pull request #2280 from matrix-org/erikj/share_room_user_dir ... Include users who you share a room with in user directory	2017-06-16 11:06:00 +01:00
Erik Johnston	052c5d19d5	Merge pull request #2281 from matrix-org/erikj/phone_home_stats ... Fix phone home stats	2017-06-15 12:46:23 +01:00
Erik Johnston	a9d6fa8b2b	Include users who share room with requester in user directory	2017-06-15 10:17:21 +01:00
Erik Johnston	4564b05483	Implement updating users who share rooms on the fly	2017-06-15 10:17:17 +01:00
Erik Johnston	72613bc379	Implement initial population of users who share rooms table	2017-06-15 09:59:04 +01:00