3016 Commits

Author	SHA1	Message	Date
Richard van der Hoff	ebfe64e3d6	Use StateResolutionHandler to resolve state in persist events ... ... and thus benefit (hopefully) from its cache.	2018-02-05 16:23:26 +00:00
Richard van der Hoff	225dc3b4cb	Flatten _get_new_state_after_events ... rejig the if statements to simplify the logic and reduce indentation	2018-02-05 16:23:25 +00:00
Richard van der Hoff	9fcbbe8e7d	Check that events being persisted have state_group	2018-02-05 16:23:25 +00:00
Richard van der Hoff	447aed42d2	Add event_map param to resolve_state_groups	2018-02-05 16:23:25 +00:00
Richard van der Hoff	ee6fb4cf85	Remove redundant return value from _calculate_state_delta ... we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.	2018-02-05 16:23:20 +00:00
Richard van der Hoff	3c7b480ba3	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-05 16:12:14 +00:00
Richard van der Hoff	c46e75d3d8	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-05 15:43:22 +00:00
Richard van der Hoff	80b8a28100	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-04 00:23:06 +00:00
Richard van der Hoff	bd25f9cf36	Clean up work_mem handling ... Add some comments and improve exception handling when twiddling work_mem for the search update	2018-02-03 23:05:41 +00:00
Richard van der Hoff	4eeae7ad65	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-03 22:59:45 +00:00
Richard van der Hoff	bb9f0f3cdb	Merge branch 'develop' into matthew/gin_work_mem	2018-02-03 22:40:28 +00:00
Richard van der Hoff	6b02fc80d1	Reinstate event_search_postgres_gist handler ... People may have queued updates for this, so we can't just delete it.	2018-02-02 14:32:51 +00:00
Richard van der Hoff	78d6ddba86	Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta ... factor _get_new_state_after_events out of _calculate_state_delta	2018-02-02 15:27:15 +01:00
Travis Ralston	6e87b34f7b	Merge branch 'develop' into travis/admin-list-media	2018-02-01 18:05:47 -07:00
Richard van der Hoff	77c0629ebc	Merge pull request #2837 from matrix-org/rav/fix_quarantine_media ... Fix sql error in quarantine_media	2018-02-01 11:45:58 +01:00
Travis Ralston	e16e45b1b4	pep8 ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 15:30:38 -07:00
Richard van der Hoff	e1e4ec9f9d	factor _get_new_state_after_events out of _calculate_state_delta ... This reduces the scope of a bunch of variables	2018-01-31 21:32:09 +00:00
Travis Ralston	63c4383927	Documentation and naming ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 08:07:52 -07:00
Richard van der Hoff	773f0eed1e	Fix sql error in quarantine_media	2018-01-30 15:02:51 +00:00
Richard van der Hoff	b387ee17b6	Improve exception handling in persist_event ... 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs	2018-01-29 17:05:33 +00:00
Richard van der Hoff	46022025ea	Fix SQL for user search ... fix some syntax errors for user search when search_all_users is enabled fixes #2801, hopefully	2018-01-27 17:21:57 +00:00
Travis Ralston	a94d9b6b82	Appease the linter ... These are ids anyways, not mxc uris. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:49:46 -07:00
Travis Ralston	5552ed9a7f	Add an admin route to get all the media in a room ... This is intended to be used by administrators to monitor the media that is passing through their server, if they wish. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:37:53 -07:00
Richard van der Hoff	36da256cc6	Merge pull request #2805 from matrix-org/rav/log_state_res ... Log room when doing state resolution	2018-01-17 18:05:04 +00:00
Richard van der Hoff	1224612a79	Log room when doing state resolution ... Mostly because it helps figure out what is prompting the resolution	2018-01-17 17:11:59 +00:00
Erik Johnston	a87006f9c7	Merge pull request #2783 from matrix-org/erikj/media_last_accessed ... Keep track of last access time for local media	2018-01-17 16:39:02 +00:00
Matthew Hodgson	06db5c4b76	Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql ... fix SQL when searching all users	2018-01-17 16:27:54 +00:00
Matthew Hodgson	2d9ab533f9	fix SQL when searching all users	2018-01-17 15:58:52 +00:00
Richard van der Hoff	390093d45e	Split resolve_events into two functions ... ... so that the return type doesn't depend on the arg types	2018-01-17 15:44:31 +00:00
Erik Johnston	2fb3a28c98	Remove lost comment	2018-01-17 14:59:44 +00:00
Erik Johnston	05f98a2224	Keep track of last access time for local media	2018-01-17 10:24:43 +00:00
Richard van der Hoff	3d12d97415	Track DB scheduling delay per-request ... For each request, track the amount of time spent waiting for a db connection. This entails adding it to the LoggingContext and we may as well add metrics for it while we are passing.	2018-01-16 17:23:32 +00:00
Richard van der Hoff	8615f19d20	rework runInteraction in terms of runConnection ... ... so that we can share the code	2018-01-16 17:08:29 +00:00
Richard van der Hoff	64ddec1bc0	Fix a logcontext leak in persist_events ... ObserveableDeferred expects its callbacks to be called without any logcontexts, whereas it turns out we were calling them with the logcontext of the request which initiated the persistence loop. It seems wrong that we are attributing work done in the persistence loop to the request that happened to initiate it, so let's solve this by dropping the logcontext for it. (I'm not sure this actually causes any real problems other than messages in the debug log, but let's clean it up anyway)	2018-01-16 11:47:36 +00:00
hera	174eacc8ba	oops	2018-01-09 18:14:32 +00:00
Matthew Hodgson	a66f489678	fix GIST->GIN switch	2018-01-09 16:55:51 +00:00
Matthew Hodgson	e79db0a673	switch back from GIST to GIN indexes	2018-01-09 16:37:48 +00:00
Matthew Hodgson	e365ad329f	oops, tweak work_mem when actually storing	2018-01-09 16:30:30 +00:00
Matthew Hodgson	19f9227643	avoid 80s GIN inserts by tweaking work_mem ... see https://github.com/matrix-org/synapse/issues/2753 for details	2018-01-09 16:25:04 +00:00
Erik Johnston	b30cd5b107	Remove dead code related to default thumbnails	2018-01-09 14:38:33 +00:00
Matthew Hodgson	cdc2cb5d11	fix StoreError syntax	2017-12-05 11:09:31 +00:00
Matthew Hodgson	74e0cc74ce	fix pep8 and tests	2017-12-04 15:11:38 +00:00
Matthew Hodgson	1bd40ca73e	switch to a simpler 'search_all_users' button as per review feedback	2017-12-04 14:58:39 +00:00
Matthew Hodgson	f397153dfc	Merge branch 'develop' into matthew/search-all-local-users	2017-11-30 01:51:38 +00:00
Matthew Hodgson	f61e107f63	remove null constraint on user_dir.room_id	2017-11-30 01:43:50 +00:00
Matthew Hodgson	4b1fceb913	fix alternation operator for FTS4 - how did this ever work!?	2017-11-30 01:34:03 +00:00
Matthew Hodgson	a4bb133b68	fix thinkos galore	2017-11-30 01:17:15 +00:00
Matthew Hodgson	3241c7aac3	untested WIP but might actually work	2017-11-29 18:27:05 +00:00
Matthew Hodgson	47d99a20d5	Add user_directory_include_pattern config param to expand search results to additional users ... Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive. It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users... ...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too. Committing it here as a WIP	2017-11-29 16:46:45 +00:00
Richard van der Hoff	2c6d63922a	Remove pushers when deleting access tokens ... Whenever an access token is invalidated, we should remove the associated pushers.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	5a4da5bf78	Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error ... Fix error on sqlite 3.7	2017-11-27 12:25:48 +00:00
Richard van der Hoff	63ccaa5873	Avoid retrying forever on IntegrityError	2017-11-27 12:00:07 +00:00
Richard van der Hoff	6b48b3e277	fix sql fails	2017-11-22 18:06:24 +00:00
Richard van der Hoff	2908f955d1	Check database in has_completed_background_updates ... so that the right thing happens on workers.	2017-11-22 18:02:15 +00:00
Richard van der Hoff	0b4aa2dc21	Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert ... Avoid locking account_data tables for upserts	2017-11-21 13:39:14 +00:00
Richard van der Hoff	7098b65cb8	Fix error on sqlite 3.7 ... Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572.	2017-11-21 11:14:17 +00:00
Richard van der Hoff	c46139a17e	Avoid locking account_data tables for upserts	2017-11-16 18:08:01 +00:00
Richard van der Hoff	06e5bcfc83	Avoid locking for upsert on pushers tables ... * replace the upsert into deleted_pushers with an insert * no need to lock for upsert on pusher_throttle	2017-11-16 17:52:23 +00:00
Richard van der Hoff	77a1227870	Fix broken ref to IntegrityError	2017-11-16 16:03:38 +00:00
Richard van der Hoff	7ab2b69e18	Avoid locking pushers table on upsert ... Now that _simple_upsert will retry on IntegrityError, we don't need to lock the table.	2017-11-16 15:32:01 +00:00
Richard van der Hoff	10aaa1bc15	_simple_upsert: retry on IntegrityError ... wrap the call to _simple_upsert_txn in a loop so that we retry on an integrityerror: this means we can avoid locking the table provided there is an unique index.	2017-11-16 15:30:15 +00:00
Richard van der Hoff	cdc9e50a5d	Cleanup in _simple_upsert_txn ... Bail out early to reduce indentation	2017-11-16 15:29:10 +00:00
Richard van der Hoff	f959c01600	Merge pull request #2661 from matrix-org/rav/statereadstore ... Pull out bits of StateStore to a mixin	2017-11-15 17:23:01 +00:00
Richard van der Hoff	1b83c09c03	Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs ... Remove preserve_context_over_{fn, deferred}	2017-11-15 11:13:53 +00:00
Richard van der Hoff	35a4b63240	Pull out bits of StateStore to a mixin ... ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655.	2017-11-14 11:43:58 +00:00
Richard van der Hoff	4dd1bfa8c1	Revert "Revert "move _state_group_cache to statestore"" ... We're going to fix this properly on this branch, so that the _state_group_cache can end up in StateGroupReadStore. This reverts commit ab335edb02.	2017-11-14 11:43:58 +00:00
Richard van der Hoff	7e6fa29cb5	Remove preserve_context_over_{fn, deferred} ... Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.	2017-11-14 11:22:42 +00:00
Richard van der Hoff	6cfee09be9	Make __init__ consitstent across Store heirarchy ... Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)	2017-11-13 10:46:07 +00:00
Erik Johnston	ab335edb02	Revert "move _state_group_cache to statestore" ... This reverts commit f5cf3638e9.	2017-11-13 10:05:33 +00:00
Erik Johnston	bfbf1e1f1a	Up cache size of get_global_account_data_by_type_for_user	2017-11-13 09:52:11 +00:00
Erik Johnston	2dce6b15c3	Fix typo	2017-11-09 15:56:16 +00:00
Richard van der Hoff	618c7b816a	Merge pull request #2656 from matrix-org/rav/fix_deactivate ... Fix 'NoneType' not iterable in /deactivate	2017-11-09 15:20:35 +00:00
Richard van der Hoff	889102315e	Fix 'NoneType' not iterable in /deactivate ... make sure we actually return a value from user_delete_access_tokens	2017-11-09 15:15:33 +00:00
Erik Johnston	e8814410ef	Have an explicit API to update room config	2017-11-08 16:13:27 +00:00
Erik Johnston	94ff2cda73	Revert "Modify group room association API to allow modification of is_public"	2017-11-08 15:43:34 +00:00
Erik Johnston	167eb01d83	Merge pull request #2637 from spantaleev/avoid-noop-media-deletes ... Avoid no-op media deletes	2017-11-08 11:53:27 +00:00
Richard van der Hoff	02a9a93bde	Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res ... Fix bug in state group storage	2017-11-08 09:22:13 +00:00
Matthew Hodgson	228ccf1fe3	Merge pull request #2643 from matrix-org/matthew/user_dir_typos ... Fix various embarrassing typos around user_directory and add some doc.	2017-11-07 17:31:11 +00:00
Richard van der Hoff	f5cf3638e9	move _state_group_cache to statestore ... this is internal to statestore, so let's keep it there.	2017-11-07 16:43:00 +00:00
Luke Barnard	38b265cb51	Remember to pick is_admin out of the db	2017-11-07 11:24:04 +00:00
Matthew Hodgson	631fa4a1b7	create new indexes before dropping old ones to keep safetynet in place	2017-11-07 10:41:55 +00:00
Matthew Hodgson	4ad883398f	s/users_in_pubic_room/users_in_public_rooms/g	2017-11-04 19:39:40 +00:00
Matthew Hodgson	d802e8ca6a	s/users_in_pubic_room/users_in_public_rooms/g	2017-11-04 19:38:13 +00:00
Matthew Hodgson	a100700630	fix copyright....	2017-11-04 19:35:49 +00:00
Matthew Hodgson	b6b075fd49	s/popualte/populate/	2017-11-04 19:35:33 +00:00
Slavi Pantaleev	805196fbeb	Avoid no-op media deletes ... If there are no media entries to delete, avoid creating transactions, prepared statements and unnecessary log entries. Signed-off-by: Slavi Pantaleev <slavi@devture.com>	2017-11-04 09:50:15 +02:00
Richard van der Hoff	bc8a5c0330	Notify auth providers on logout ... Provide a hook by which auth providers can be notified of logouts.	2017-11-01 16:51:51 +00:00
Matthew Hodgson	275826f234	Merge pull request #2617 from matrix-org/matthew/auto-displayname ... automatically set default displayname on register	2017-11-01 16:21:16 +00:00
David Baker	fbbacb284e	Merge pull request #2613 from matrix-org/rav/kill_refresh_tokens ... Remove the last vestiges of refresh_tokens	2017-11-01 15:57:35 +00:00
Matthew Hodgson	9f7a555b4e	switch to setting default displayname in the storage layer ... to avoid clobbering guest user displaynames on registration	2017-11-01 15:51:30 +00:00
Luke Barnard	99354b430e	Merge pull request #2612 from matrix-org/luke/groups-room-relationship-is-public ... Modify group room association API to allow modification of is_public	2017-11-01 11:08:36 +00:00
Luke Barnard	318a249c8b	Leave is_public as required argument of update_room_group_association	2017-11-01 09:36:01 +00:00
Richard van der Hoff	356bcafc44	Remove the last vestiges of refresh_tokens	2017-10-31 20:35:58 +00:00
Luke Barnard	20fe347906	Modify group room association API to allow modification of is_public ... also includes renamings to make things more consistent.	2017-10-31 17:04:28 +00:00
Richard van der Hoff	1650eb5847	DB schema interface for password auth providers ... Provide an interface by which password auth providers can register db schema files to be run at startup	2017-10-31 14:01:53 +00:00
Erik Johnston	4ab8abbc2b	Merge branch 'erikj/attestation_local_fix' of github.com:matrix-org/synapse into develop	2017-10-27 15:07:08 +01:00
Luke Barnard	5451cc7792	Request is_public from database	2017-10-27 11:27:43 +01:00
Luke Barnard	6362298fa5	Create groups with is_public = True	2017-10-27 11:04:20 +01:00
Erik Johnston	82d8c1bacb	Fixup	2017-10-27 10:30:21 +01:00
Erik Johnston	195abfe7a5	Remove incorrect attestations	2017-10-27 09:58:13 +01:00
Luke Barnard	007cd48af6	Recreate groups table instead of adding column ... Adding a column with non-constant default not possible in sqlite3	2017-10-26 17:55:22 +01:00
Luke Barnard	713e60b9b6	Awful hack to get default true	2017-10-26 17:38:14 +01:00
Luke Barnard	cfa4e658e0	Bump schema version to 46	2017-10-26 17:23:49 +01:00
Luke Barnard	9b2feef9eb	Add is_public to groups table to allow for private groups ... Prevent group API access to non-members for private groups Also make all the group code paths consistent with `requester_user_id` always being the User ID of the requesting user.	2017-10-26 16:51:32 +01:00
Richard van der Hoff	eaaabc6c4f	replace 'except:' with 'except Exception:' ... what could possibly go wrong	2017-10-23 15:52:32 +01:00
Richard van der Hoff	2e9f5ea31a	Fix logcontext handling for persist_events ... * don't use preserve_context_over_deferred, which is known broken. * remove a redundant preserve_fn. * add/improve some comments	2017-10-17 10:59:30 +01:00
Luke Barnard	2c5972f87f	Implement GET /groups/$groupId/invited_users	2017-10-16 15:31:11 +01:00
Erik Johnston	818b08d0e4	peeeeeeeeep8888888888888888888888888888	2017-10-11 15:54:00 +01:00
Erik Johnston	ea18996f54	Fix group stream replication ... The stream update functions expect the storage function to return a list of tuples.	2017-10-11 15:44:39 +01:00
Erik Johnston	c2c47550f9	Fix schema delta versions	2017-10-11 13:23:15 +01:00
Erik Johnston	535cc49f27	Merge pull request #2466 from matrix-org/erikj/groups_merged ... Initial Group Implementation	2017-10-11 13:20:07 +01:00
Erik Johnston	e8496efe84	Fix up comment	2017-10-09 15:17:34 +01:00
Erik Johnston	11d62f43c9	Invalidate cache	2017-10-03 14:12:28 +01:00
Erik Johnston	e4ab96021e	Update comments	2017-10-03 14:10:41 +01:00
Erik Johnston	30848c0fcd	Ignore incoming events for rooms that we have left ... When synapse receives an event for a room its not in over federation, it double checks with the remote server to see if it is in fact in the room. This is done so that if the server has forgotten about the room (usually as a result of the database being dropped) it can recover from it. However, in the presence of state resets in large rooms, this can cause a lot of work for servers that have legitimately left. As a hacky solution that supports both cases we drop incoming events for rooms that we have explicitly left. This means that we no longer support the case of servers having forgotten that they've rejoined a room, but that is sufficiently rare that we're not going to support it for now.	2017-10-03 11:18:21 +01:00
David Baker	27955056e0	Merge branch 'develop' into erikj/groups_merged	2017-10-02 16:20:41 +01:00
Erik Johnston	768f00dedb	Up the limits on number of url cache entries to delete at one time	2017-09-28 14:27:27 +01:00
Erik Johnston	4dc07e93a8	Add old indices	2017-09-28 14:10:33 +01:00
Erik Johnston	93247a424a	Only pull out local media that were for url cache	2017-09-28 13:48:14 +01:00
Erik Johnston	7a44c01d89	Fix typo	2017-09-28 12:46:04 +01:00
Erik Johnston	ae79764fe5	Change expires column to expires_ts	2017-09-28 12:37:53 +01:00
Erik Johnston	77f1d24de3	More brackets	2017-09-28 12:23:15 +01:00
Erik Johnston	9ccb4226ba	Delete expired url cache data	2017-09-28 12:18:06 +01:00
Erik Johnston	17b8e2bd02	Add remove room API	2017-09-26 15:52:41 +01:00
Erik Johnston	a8e2a3df32	Add unique index to group_rooms table	2017-09-26 15:39:21 +01:00
Erik Johnston	e1dec2f1a7	Remove user from group summary when the leave the group	2017-09-21 16:09:57 +01:00
Erik Johnston	197d82dc07	Correctly return next token	2017-09-20 11:12:11 +01:00
Erik Johnston	069ae2df12	Fix initial sync	2017-09-20 10:52:12 +01:00
Richard van der Hoff	e76d1135dd	Invalidate signing key cache when we gat an update ... This might make the cache slightly more efficient.	2017-09-20 01:32:42 +01:00
Erik Johnston	4a9b1cf253	Add user profiles to summary from group server	2017-08-25 16:23:58 +01:00
Erik Johnston	27ebc5c8f2	Add remote profile cache	2017-08-25 11:25:47 +01:00
Erik Johnston	97c544f91f	Add _simple_update	2017-08-25 11:11:37 +01:00
Erik Johnston	2800983f3e	Merge pull request #2410 from matrix-org/erikj/groups_publicise ... Add ability to publicise group membership	2017-08-21 16:51:56 +01:00
Erik Johnston	8b50fe5330	Use BOOLEAN rather than TEXT type	2017-08-21 16:37:29 +01:00
Erik Johnston	175a01f56c	Groups: Fix mising json.load in initial sync	2017-08-21 14:45:56 +01:00
Erik Johnston	ef8e578677	Add bulk group publicised lookup API	2017-08-09 13:36:22 +01:00
Erik Johnston	b880ff190a	Allow update group publicity	2017-08-08 14:19:41 +01:00
Erik Johnston	05e21285aa	Store whether the user wants to publicise their membership of a group	2017-08-08 13:01:46 +01:00
Erik Johnston	966a70f1fa	Update comment	2017-07-24 17:49:39 +01:00
Erik Johnston	629cdfb124	Use join rather than joined, etc.	2017-07-24 14:54:05 +01:00
Erik Johnston	ed666d3969	Fix all the typos	2017-07-24 14:05:09 +01:00
Erik Johnston	b76ef6ccb8	Include users membership in group in summary API	2017-07-24 13:55:39 +01:00
Erik Johnston	851aeae7c7	Check users/rooms are in group before adding to summary	2017-07-24 13:40:56 +01:00
Erik Johnston	96917d5552	Merge pull request #2378 from matrix-org/erikj/group_sync_support ... Add groups to sync stream	2017-07-21 11:05:39 +01:00
Erik Johnston	0401604222	Merge pull request #2377 from matrix-org/erikj/group_profile_update ... Add update group profile API	2017-07-20 17:53:39 +01:00
Erik Johnston	2cc998fed8	Fix replication. And notify	2017-07-20 17:13:18 +01:00
Erik Johnston	c544188ee3	Add groups to sync stream	2017-07-20 16:36:42 +01:00
Erik Johnston	8209b5f033	Fix a storage desc	2017-07-20 16:22:22 +01:00
Erik Johnston	57826d645b	Fix typo	2017-07-20 13:15:22 +01:00
Erik Johnston	6f443a74cf	Add update group profile API	2017-07-20 09:46:33 +01:00
Erik Johnston	94ecd871a0	Fix typos	2017-07-18 16:38:54 +01:00
Erik Johnston	3e703eb04e	Comment	2017-07-18 10:17:25 +01:00
Erik Johnston	508460f240	Remove sync stuff	2017-07-18 09:55:46 +01:00
Erik Johnston	4540730111	Remove unused tables	2017-07-18 09:38:15 +01:00
Erik Johnston	e96ee95a7e	Remove sync stuff	2017-07-18 09:38:08 +01:00
Erik Johnston	2f9eafdd36	Add local group server support	2017-07-17 12:03:49 +01:00
Erik Johnston	85fda57208	Add DEFAULT_ROLE_ID	2017-07-14 14:03:54 +01:00
Erik Johnston	4b203bdba5	Correctly increment orders	2017-07-14 14:02:31 +01:00
Erik Johnston	3b0470dba5	Remove unused functions	2017-07-13 13:54:01 +01:00
Erik Johnston	8575e3160f	Comments	2017-07-13 13:52:41 +01:00
Erik Johnston	26451a09eb	Comments	2017-07-12 14:47:18 +01:00
Erik Johnston	a62406aaa5	Add group summary APIs	2017-07-12 11:36:15 +01:00
Erik Johnston	6d586dc05c	Comment	2017-07-12 09:58:37 +01:00
Erik Johnston	410b4e14a1	Move comment	2017-07-11 15:44:18 +01:00
Erik Johnston	e52c391cd4	Rename column to attestation_json	2017-07-11 14:25:46 +01:00
Erik Johnston	0aac30d53b	Comments	2017-07-11 14:23:50 +01:00
Erik Johnston	83936293eb	Comments	2017-07-11 11:42:25 +01:00
Erik Johnston	b8ca494ee9	Initial group server implementation	2017-07-10 15:44:15 +01:00
Erik Johnston	f502b0dea1	Perf: Don't filter events for push ... We know the users are joined and we can explicitly check for if they are ignoring the user, so lets do that.	2017-07-07 14:04:40 +01:00
Erik Johnston	ed9a7f5436	Merge pull request #2309 from matrix-org/erikj/user_ip_repl ... Fix up user_ip replication commands	2017-07-06 14:33:14 +01:00
Erik Johnston	b5e8d529e6	Define CACHE_SIZE_FACTOR once	2017-07-04 09:56:44 +01:00
Erik Johnston	6ff14ddd2e	Make into list	2017-06-29 15:47:37 +01:00
Erik Johnston	5946aa0877	Prefill forward extrems and event to state groups	2017-06-29 15:38:48 +01:00
Erik Johnston	8c23221666	Fix up	2017-06-27 15:53:45 +01:00
Erik Johnston	a0a561ae85	Fix up client ips to read from pending data	2017-06-27 14:46:12 +01:00
Erik Johnston	ed3d0170d9	Batch upsert user ips	2017-06-27 13:37:04 +01:00
Erik Johnston	8abdd7b553	Fix up indices for users_who_share_rooms	2017-06-26 14:01:30 +01:00
Erik Johnston	7fe8ed1787	Store URL cache preview downloads seperately ... This makes it easier to clear old media out at a later date	2017-06-23 11:14:11 +01:00
Erik Johnston	e0004aa28a	Add desc	2017-06-22 10:03:48 +01:00
Erik Johnston	b668112320	Merge pull request #2296 from matrix-org/erikj/dont_appserver_shar ... Don't work out users who share room with appservice users	2017-06-21 14:50:24 +01:00
Erik Johnston	dae9a00a28	Initialise exclusive_user_regex	2017-06-21 14:19:33 +01:00
Krombel	4202fba82a	Merge branch 'develop' into avoid_duplicate_filters	2017-06-21 14:48:21 +02:00
Krombel	812c030e87	replaced json.dumps with encode_canonical_json	2017-06-21 14:48:12 +02:00
Erik Johnston	385dcb7c60	Handle thumbnail urls	2017-06-19 17:48:28 +01:00
Erik Johnston	b8b936a6ea	Add API to quarantine media	2017-06-19 17:39:21 +01:00
Erik Johnston	d7fe6b356c	Add shutdown room API	2017-06-19 12:37:27 +01:00
Erik Johnston	dfeca6cf40	Merge pull request #2286 from matrix-org/erikj/split_out_user_dir ... Split out user directory to a separate process	2017-06-16 13:01:19 +01:00
Erik Johnston	6aa5bc8635	Initial worker impl	2017-06-16 11:47:11 +01:00
Erik Johnston	d8f47d2efa	Merge pull request #2280 from matrix-org/erikj/share_room_user_dir ... Include users who you share a room with in user directory	2017-06-16 11:06:00 +01:00
Erik Johnston	052c5d19d5	Merge pull request #2281 from matrix-org/erikj/phone_home_stats ... Fix phone home stats	2017-06-15 12:46:23 +01:00
Erik Johnston	a9d6fa8b2b	Include users who share room with requester in user directory	2017-06-15 10:17:21 +01:00
Erik Johnston	4564b05483	Implement updating users who share rooms on the fly	2017-06-15 10:17:17 +01:00
Erik Johnston	72613bc379	Implement initial population of users who share rooms table	2017-06-15 09:59:04 +01:00
Erik Johnston	ebcd55d641	Add DB schema for tracking users who share rooms	2017-06-15 09:45:48 +01:00
Erik Johnston	4b461a6931	Add some more stats	2017-06-15 09:39:39 +01:00
Erik Johnston	617304b2cf	Fix phone home stats	2017-06-14 19:47:15 +01:00
Erik Johnston	d9fd937e39	Fix user directory insertion due to missing room_id	2017-06-13 11:50:24 +01:00
Erik Johnston	505e7e8b9d	Fix up sql	2017-06-13 11:19:18 +01:00
Erik Johnston	6fd7e6db3d	Fix user dir to not assume existence of user	2017-06-13 11:11:26 +01:00
Erik Johnston	fdca6e36ee	Merge pull request #2274 from matrix-org/erikj/cache_is_host_joined ... Add cache for is_host_joined	2017-06-13 10:58:53 +01:00
Erik Johnston	b58e24cc3c	Tweak the ranking of PG user dir search	2017-06-13 10:16:31 +01:00
Erik Johnston	d53fe399eb	Add cache for is_host_joined	2017-06-13 09:56:18 +01:00
Erik Johnston	a837765e8c	Merge pull request #2266 from matrix-org/erikj/host_in_room ... Change is_host_joined to use current_state table	2017-06-12 09:49:51 +01:00
Erik Johnston	8060974344	Fix replication	2017-06-09 16:40:52 +01:00
Erik Johnston	b0d975e216	Comments	2017-06-09 16:25:42 +01:00
Erik Johnston	e54d7d536e	Cache state deltas	2017-06-09 16:24:00 +01:00
Erik Johnston	935e588799	Tweak SQL	2017-06-09 13:01:23 +01:00
Erik Johnston	0185b75381	Change is_host_joined to use current_state table ... This bypasses a bug where using the state groups to figure out if a host is in a room sometimes errors if the servers isn't in the room. (For example when the server rejected an invite to a remote room)	2017-06-09 10:52:26 +01:00
Erik Johnston	ea11ee09f3	Ensure we don't use unpersisted state group as prev group	2017-06-08 11:59:57 +01:00
Erik Johnston	c62c480dc6	Merge pull request #2259 from matrix-org/erikj/fix_state_woes ... Fix bug where state_group tables got corrupted	2017-06-07 17:51:25 +01:00
Erik Johnston	197bd126f0	Fix bug where state_group tables got corrupted ... This is due to the fact that we prefilled caches using txn.call_after, which always gets called including on error. We fix this by making txn.call_after only fire when a transaction completes successfully, which is what we want most of the time anyway.	2017-06-07 17:39:36 +01:00
Erik Johnston	a053ff3979	Merge pull request #2248 from matrix-org/erikj/state_fixup ... Faster cache for get_joined_hosts	2017-06-07 14:01:06 +01:00
Erik Johnston	64ed74c01e	When pruning, delete from device_lists_outbound_last_success	2017-06-07 11:20:47 +01:00
Erik Johnston	1a81a1898e	Keep pruning background task	2017-06-07 11:16:56 +01:00
Erik Johnston	6ba21bf2b8	Comments	2017-06-07 11:08:36 +01:00
Erik Johnston	09e4bc0501	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_fixup	2017-06-07 11:05:23 +01:00
Erik Johnston	6e2a7ee1bc	Remove spurious log lines	2017-06-07 11:05:17 +01:00
Erik Johnston	65f0513a33	Split up device_lists_outbound_pokes table for faster updates.	2017-06-07 11:02:38 +01:00
Erik Johnston	6f83c4537c	Increase size of IP cache	2017-06-07 10:18:44 +01:00
Erik Johnston	1a01af079e	Handle profile updates in user directory	2017-06-01 15:39:51 +01:00
Erik Johnston	4d039aa2ca	Fix sqlite	2017-06-01 14:58:48 +01:00
Erik Johnston	21e255a8f1	Split the table in two	2017-06-01 14:50:46 +01:00
Erik Johnston	d5477c7afd	Tweak search query	2017-06-01 13:28:01 +01:00
Erik Johnston	02a6108235	Tweak search query	2017-06-01 13:16:40 +01:00
Erik Johnston	59dbb47065	Remove spurious inlineCallbacks	2017-06-01 11:41:29 +01:00
Erik Johnston	0fe6f3c521	Bug fixes and logging ... - Check if room is public when a user joins before adding to user dir - Fix typo of field name "content.join_rules" -> "content.join_rule"	2017-06-01 11:09:49 +01:00
Erik Johnston	036362ede6	Order by if they have profile info	2017-06-01 09:41:08 +01:00
Erik Johnston	a757dd4863	Use prefix matching	2017-06-01 09:40:37 +01:00
Erik Johnston	f5cc22bdc6	Comment on why arbitrary comments	2017-05-31 17:30:26 +01:00
Erik Johnston	5dd1b2c525	Use unique indices	2017-05-31 17:29:12 +01:00
Erik Johnston	f1378aef91	Convert to int	2017-05-31 17:03:08 +01:00
Erik Johnston	f091061711	Fix tests	2017-05-31 16:34:40 +01:00
Erik Johnston	4abcff0177	Fix typo	2017-05-31 16:22:36 +01:00
Erik Johnston	63c58c2a3f	Limit number of things we fetch out of the db	2017-05-31 16:17:58 +01:00
Erik Johnston	304880d185	Add stream change cache	2017-05-31 15:46:36 +01:00
Erik Johnston	5d79d728f5	Split out directory and search tables	2017-05-31 15:23:49 +01:00
Erik Johnston	dc51af3d03	Pull max id from correct table	2017-05-31 15:13:49 +01:00
Erik Johnston	350622a107	Handle the server leaving a public room	2017-05-31 15:11:36 +01:00
Erik Johnston	63fda37e20	Add comments	2017-05-31 15:00:29 +01:00
Erik Johnston	293ef29655	Weight differently	2017-05-31 14:29:32 +01:00
Erik Johnston	3b5f22ca40	Add search	2017-05-31 14:00:01 +01:00
Erik Johnston	b5db4ed5f6	Update room column when room becomes unpublic	2017-05-31 13:40:28 +01:00
Erik Johnston	eeb2f9e546	Add user_directory to database	2017-05-31 11:51:01 +01:00
Erik Johnston	dd48f7204c	Add comment	2017-05-30 15:01:22 +01:00
Erik Johnston	04095f7581	Add clobbered event_id	2017-05-30 14:53:01 +01:00
Erik Johnston	a584a81b3e	Add current_state_delta_stream table	2017-05-30 14:44:09 +01:00
Erik Johnston	619e8ecd0c	Handle None state group correctly	2017-05-26 10:46:03 +01:00
Erik Johnston	dfbda5e025	Faster cache for get_joined_hosts	2017-05-25 17:24:44 +01:00
Erik Johnston	11f139a647	Merge pull request #2247 from matrix-org/erikj/auth_event ... Only store event_auth for state events	2017-05-24 16:46:34 +01:00
Erik Johnston	6e614e9e10	Add background task to clear out old event_auth	2017-05-24 15:23:34 +01:00
Erik Johnston	c049472b8a	Only store event_auth for state events	2017-05-24 15:23:31 +01:00
Erik Johnston	8cf9f0a3e7	Remove redundant invalidation	2017-05-23 09:46:59 +01:00
Erik Johnston	e6618ece2d	Missed an invalidation	2017-05-23 09:36:52 +01:00
Erik Johnston	74bf4ee7bf	Stream count_e2e_one_time_keys cache invalidation	2017-05-22 16:19:22 +01:00
Erik Johnston	58ebb96cce	Fix invalidation of get_users_with_read_receipts_in_room	2017-05-19 14:38:50 +01:00
Erik Johnston	107ac7ac96	Increase size of push rule caches	2017-05-18 17:17:53 +01:00
Erik Johnston	760625acba	Make get_if_app_services_interested_in_user faster	2017-05-18 16:34:44 +01:00
Erik Johnston	d9e3a4b5db	Merge pull request #2230 from matrix-org/erikj/speed_up_get_state ... Make get_state_groups_from_groups faster.	2017-05-17 17:23:04 +01:00
Erik Johnston	ace23463c5	Merge pull request #2216 from slipeer/app_services_interested_in_user ... Fix users claimed non-exclusively by an app service don't get notific…	2017-05-17 16:28:50 +01:00
Erik Johnston	bbfe4e996c	Make get_state_groups_from_groups faster. ... Most of the time was spent copying a dict to filter out sentinel values that indicated that keys did not exist in the dict. The sentinel values were added to ensure that we cached the non-existence of keys. By updating DictionaryCache to keep track of which keys were known to not exist itself we can remove a dictionary copy.	2017-05-17 15:12:15 +01:00
Erik Johnston	85e8092cca	Comment	2017-05-17 10:03:09 +01:00
Erik Johnston	ad53fc3cf4	Short circuit when we have delta ids	2017-05-17 09:57:34 +01:00
Erik Johnston	13f540ef1b	Speed up get_joined_hosts	2017-05-16 16:05:22 +01:00
Erik Johnston	b8492b6c2f	Merge pull request #2224 from matrix-org/erikj/prefill_state ... Prefill state caches	2017-05-16 15:50:11 +01:00
Erik Johnston	331570ea6f	Remove spurious merge artifacts	2017-05-16 15:33:07 +01:00
Krombel	55af207321	Merge branch 'develop' into avoid_duplicate_filters	2017-05-16 15:29:59 +02:00
Erik Johnston	608b5a6317	Take a copy before prefilling, as it may be a frozendict	2017-05-16 12:55:29 +01:00
Krombel	64953c8ed2	avoid access-error if no filter_id matches	2017-05-15 18:36:37 +02:00
Erik Johnston	e4435b014e	Update comment	2017-05-15 15:11:30 +01:00
Erik Johnston	871605f4e2	Comments	2017-05-15 15:11:30 +01:00
Erik Johnston	e0d2f6d5b0	Add more granular event send metrics	2017-05-15 15:11:30 +01:00
Erik Johnston	bfbc907cec	Prefill state caches	2017-05-15 15:11:13 +01:00
Krombel	eb7cbf27bc	insert whitespace to fix travis build	2017-05-12 12:09:42 +02:00
Krombel	6b95e35e96	add check to only add a new filter if the same filter does not exist previously ... Signed-off-by: Matthias Kesler <krombel@krombel.de>	2017-05-11 16:05:30 +02:00
Richard van der Hoff	ff3d810ea8	Add a comment to old delta	2017-05-11 12:48:50 +01:00
Richard van der Hoff	34194aaff7	Don't create event_search index on sqlite ... ... because the table is virtual	2017-05-11 12:46:55 +01:00
Richard van der Hoff	114f290947	Add more logging for purging ... Log the number of events we will be deleting at info.	2017-05-11 12:08:47 +01:00
Richard van der Hoff	baafb85ba4	Add an index to event_search ... - to make the purge API quicker	2017-05-11 12:05:22 +01:00
Richard van der Hoff	dc026bb16f	Tidy purge code and add some comments ... Try to make this clearer with more comments and some variable renames	2017-05-11 10:56:12 +01:00
Slipeer	328378f9cb	Fix users claimed non-exclusively by an app service don't get notifications #2211	2017-05-11 11:42:08 +03:00
Richard van der Hoff	8e345ce465	Don't de-delta state groups we're about to delete	2017-05-10 18:44:22 +01:00
Richard van der Hoff	b64d312421	add some logging to purge_history	2017-05-10 18:44:22 +01:00
Erik Johnston	a3648f84b2	Merge pull request #2208 from matrix-org/erikj/ratelimit_overrid ... Add per user ratelimiting overrides	2017-05-10 15:54:48 +01:00
Erik Johnston	b990b2fce5	Add per user ratelimiting overrides	2017-05-10 11:05:43 +01:00
Richard van der Hoff	a7e9d8762d	Allow clients to upload one-time-keys with new sigs ... When a client retries a key upload, don't give an error if the signature has changed (but the key is the same). Fixes https://github.com/vector-im/riot-android/issues/1208, hopefully.	2017-05-09 19:04:56 +01:00
Erik Johnston	40dcf0d856	Merge pull request #2203 from matrix-org/erikj/event_cache_hit_ratio ... Don't update event cache hit ratio from get_joined_users	2017-05-08 16:52:39 +01:00
Erik Johnston	d3c3026496	Merge pull request #2201 from matrix-org/erikj/store_device_cache ... Cache check to see if device exists	2017-05-08 16:23:04 +01:00
Erik Johnston	093f7e47cc	Expand docstring a bit	2017-05-08 16:14:46 +01:00
Erik Johnston	6a12998a83	Add missing yields	2017-05-08 16:10:51 +01:00
Erik Johnston	ffad4fe35b	Don't update event cache hit ratio from get_joined_users ... Otherwise the hit ration of plain get_events gets completely skewed by calls to get_joined_users* functions.	2017-05-08 16:06:17 +01:00
Erik Johnston	94e6ad71f5	Invalidate cache on device deletion	2017-05-08 15:55:59 +01:00
Erik Johnston	8571f864d2	Cache one time key counts	2017-05-08 15:34:27 +01:00
Erik Johnston	fc6d4974a6	Comment	2017-05-08 15:33:57 +01:00
Erik Johnston	738ccf61c0	Cache check to see if device exists	2017-05-08 15:32:18 +01:00
Erik Johnston	dcabef952c	Increase client_ip cache size	2017-05-08 15:09:19 +01:00
Erik Johnston	e0f20e9425	Revert "Remove unused import" ... This reverts commit ab37bef83b.	2017-05-08 13:07:43 +01:00
Erik Johnston	fe7c1b969c	Revert "We don't care about forgotten rooms" ... This reverts commit ad8b316939.	2017-05-08 13:07:43 +01:00
Erik Johnston	78f306a6f7	Revert "Speed up filtering of a single event in push" ... This reverts commit 421fdf7460.	2017-05-08 13:07:41 +01:00
Erik Johnston	66f371e8b8	Merge pull request #2176 from matrix-org/erikj/faster_get_joined ... Make get_joined_users faster	2017-05-05 10:59:55 +01:00
Erik Johnston	e3aa8a7aa8	Merge pull request #2185 from matrix-org/erikj/smaller_caches ... Optimise caches for single key	2017-05-05 10:19:05 +01:00
Erik Johnston	cf589f2c1e	Fixes	2017-05-05 10:17:56 +01:00
Erik Johnston	8af4569583	Merge pull request #2174 from matrix-org/erikj/current_cache_hosts ... Add cache for get_current_hosts_in_room	2017-05-05 10:15:24 +01:00
Erik Johnston	587f07543f	Revert "Prefill state caches"	2017-05-04 15:07:27 +01:00
Erik Johnston	aa93cb9f44	Add comment	2017-05-04 14:59:28 +01:00
Erik Johnston	537dbadea0	Intern host strings	2017-05-04 14:55:28 +01:00
Erik Johnston	07a07588a0	Make caches bigger	2017-05-04 14:52:28 +01:00
Erik Johnston	dfaa58f72d	Fix comment and num args	2017-05-04 14:50:24 +01:00
Erik Johnston	5d8290429c	Reduce size of get_users_in_room	2017-05-04 13:43:19 +01:00
Erik Johnston	7ebf518c02	Make get_joined_users faster	2017-05-03 15:55:54 +01:00
Erik Johnston	2c2dcf81d0	Update comment	2017-05-03 10:00:29 +01:00
Erik Johnston	1827057acc	Comments	2017-05-03 09:56:05 +01:00
Erik Johnston	8346e6e696	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state	2017-05-03 09:46:40 +01:00
Erik Johnston	3e5a62ecd8	Add more granular event send metrics	2017-05-02 14:23:26 +01:00
Erik Johnston	a2c89a225c	Prefill state caches	2017-05-02 10:40:31 +01:00
Erik Johnston	7166854f41	Add cache for get_current_hosts_in_room	2017-05-02 10:36:35 +01:00
Erik Johnston	3033261891	Merge pull request #2080 from matrix-org/erikj/filter_speed ... Speed up filtering of a single event in push	2017-04-28 14:17:13 +01:00
Erik Johnston	ab37bef83b	Remove unused import	2017-04-28 09:57:23 +01:00
Erik Johnston	ad8b316939	We don't care about forgotten rooms	2017-04-28 09:52:36 +01:00
Erik Johnston	421fdf7460	Speed up filtering of a single event in push	2017-04-28 09:52:36 +01:00
Richard van der Hoff	c84770b877	Fix bgupdate error if index already exists (#2167) ... When creating a new table index in the background, guard against it existing already. Fixes https://github.com/matrix-org/synapse/issues/2135. Also, make sure we restore the autocommit flag when we're done, otherwise we get more failures from other operations later on. Fixes https://github.com/matrix-org/synapse/issues/1890 (hopefully).	2017-04-27 15:27:48 +01:00
Erik Johnston	f053a1409e	Make state caches cache in ascii	2017-04-25 17:22:55 +01:00
Erik Johnston	f144365281	Comment	2017-04-25 15:18:26 +01:00
Erik Johnston	d9aa645f86	Reduce size of joined_user cache ... The _get_joined_users_from_context cache stores a mapping from user_id to avatar_url and display_name. Instead of storing those in a dict, store them in a namedtuple as that uses much less memory. We also try converting the string to ascii to further reduce the size.	2017-04-25 14:38:51 +01:00
Erik Johnston	22f3d3ae76	Reduce _get_state_group_for_event cache size	2017-04-25 11:43:03 +01:00
Erik Johnston	119cb9bbcf	Reduce cache size by not storing deferreds ... Currently the cache descriptors store deferreds rather than raw values, this is a simple way of triggering only one database hit and sharing the result if two callers attempt to get the same value. However, there are a few caches that simply store a mapping from string to string (or int). These caches can have a large number of entries, under the assumption that each entry is small. However, the size of a deferred (specifically the size of ObservableDeferred) is signigicantly larger than that of the raw value, 2kb vs 32b. This PR therefore changes the cache descriptors to store the raw values rather than the deferreds. As a side effect cached storage function now either return a deferred or the actual value, as the cached list decriptor already does. This is fine as we always end up just yield'ing on the returned value eventually, which handles that case correctly.	2017-04-25 10:23:11 +01:00
Erik Johnston	e4f3431116	Remove unused cache	2017-04-24 13:27:38 +01:00
Luke Barnard	78f0ddbfad	Merge pull request #2120 from matrix-org/luke/read-markers ... Implement Read Marker API	2017-04-13 14:21:31 +01:00
Luke Barnard	6a70647d45	Correct logic in is_event_after	2017-04-13 13:46:17 +01:00
Luke Barnard	b9557064bf	Simplify is_event_after logic	2017-04-12 14:36:20 +01:00
Luke Barnard	7f94709066	travis flake8..	2017-04-11 18:35:45 +01:00
Luke Barnard	73880268ef	Refactor event ordering check to events store	2017-04-11 17:34:09 +01:00
Erik Johnston	85657eedf8	Bail on where clause instead	2017-04-11 16:24:31 +01:00
Erik Johnston	b48045a8f5	Don't bother with outer check for now	2017-04-11 16:23:24 +01:00
Erik Johnston	34840cdcef	Fix getting latest device IP for user with no devices	2017-04-11 09:56:54 +01:00
Erik Johnston	2a3e822f44	Comment	2017-04-07 13:47:04 +01:00
Erik Johnston	d72667fcce	Speed up get_current_state_ids ... Using _simple_select_list is fairly expensive for functions that return a lot of rows and/or get called a lot. (This is because it carefully constructs a list of dicts). get_current_state_ids gets called a lot on startup and e.g. when the IRC bridge decided to send tonnes of joins/leaves (as it invalidates the cache). We therefore replace it with a custon txn function that builds up the final result dict without building up and intermediate representation.	2017-04-07 10:10:49 +01:00
Erik Johnston	9f26d3b75b	Deduplicate new deviceinbox rows for replication	2017-04-04 16:21:21 +01:00
Erik Johnston	bfcf016714	Fix up docs	2017-03-31 11:19:24 +01:00
Erik Johnston	24d35ab47b	Add new storage functions for new replication ... The new replication protocol will keep all the streams separate, rather than muxing multiple streams into one.	2017-03-30 11:48:35 +01:00
Erik Johnston	30348c924c	Use txn.fetchall() so we can reuse txn	2017-03-30 10:30:05 +01:00
Erik Johnston	3ce8d59176	Increase cache size for _get_state_group_for_event	2017-03-29 14:31:46 +01:00
Erik Johnston	e4df0e189d	Decrank last commit	2017-03-29 11:02:35 +01:00
Erik Johnston	4ad613f6be	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/e2e_one_time_upsert	2017-03-29 10:57:19 +01:00
Erik Johnston	ac6bc55512	Correctly look up key	2017-03-29 10:56:26 +01:00
Erik Johnston	d28db583da	Merge pull request #2063 from matrix-org/erikj/device_list_batch ... Batch sending of device list pokes	2017-03-28 11:35:41 +01:00
Erik Johnston	58a35366be	The algorithm is part of the key id	2017-03-28 11:34:37 +01:00
Erik Johnston	bac9bf1b12	Typo	2017-03-27 18:02:17 +01:00
Erik Johnston	d82c42837f	Short circuit if all new events have same state group	2017-03-27 18:00:47 +01:00
Erik Johnston	2a28b79e04	Batch sending of device list pokes	2017-03-24 14:44:49 +00:00
Erik Johnston	281553afe6	Merge pull request #2062 from matrix-org/erikj/presence_replication ... Use presence replication stream to invalidate cache	2017-03-24 13:57:45 +00:00
Erik Johnston	987f4945b4	Actually call invalidate	2017-03-24 13:28:20 +00:00
Erik Johnston	09f79aaad0	Use presence replication stream to invalidate cache ... Instead of using the cache invalidation replication stream to invalidate the _get_presence_cache, we can instead rely on the presence replication stream. This reduces the amount of replication traffic considerably.	2017-03-24 13:21:08 +00:00
Erik Johnston	7fc1f1e2b6	Cache hosts in room	2017-03-24 11:46:24 +00:00
Erik Johnston	d58b1ffe94	Replace some calls to cursor_to_dict ... cursor_to_dict can be surprisinglh expensive for large result sets, so lets only call it when we need to.	2017-03-24 11:07:02 +00:00
Erik Johnston	e71940aa64	Use iter(items|values)	2017-03-24 10:57:02 +00:00
Erik Johnston	00957d1aa4	User Cursor.__iter__ instead of fetchall ... This prevents unnecessary construction of lists	2017-03-23 17:53:49 +00:00
Erik Johnston	6ebe2d23b1	Raise a more helpful exception	2017-03-23 13:48:30 +00:00
Erik Johnston	e64655c25d	Don't user upsert to persist new one time keys ... Instead we no-op duplicate one time key uploads, an error if the key_id already exists but encodes a different key.	2017-03-23 13:17:00 +00:00
Richard van der Hoff	64778693be	fix up some key verif docstrings	2017-03-21 13:27:50 +00:00
Erik Johnston	37a187bfab	Merge pull request #2033 from matrix-org/erikj/repl_speed ... Don't send the full event json over replication	2017-03-21 13:11:15 +00:00
Richard van der Hoff	eddce9d74a	Merge pull request #2027 from matrix-org/rav/logcontext_leaks ... A few fixes to logcontext things	2017-03-20 11:53:36 +00:00
Richard van der Hoff	994d7ae7c5	Remove broken use of clock.call_later ... background_updates was using `call_later` in a way that leaked the logcontext into the reactor. We could have rewritten it to do it properly, but given that we weren't using the fancier facilities provided by `call_later`, we might as well just use `async.sleep`, which does the logcontext stuff properly.	2017-03-18 00:01:37 +00:00
Erik Johnston	61f471f779	Don't send the full event json over replication	2017-03-17 15:50:01 +00:00
Richard van der Hoff	0c01f829ae	Avoid resetting state on rejected events ... When we get a rejected event, give it the same state_group as its prev_event, rather than no state_group at all. This should fix https://github.com/matrix-org/synapse/issues/1935.	2017-03-17 15:06:08 +00:00
Richard van der Hoff	5068fb16a5	Refactoring and cleanups ... A few non-functional changes: * A bunch of docstrings to document types * Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more readable. * Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid mind-bending conditional. * Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid mind-bending conditional.	2017-03-17 15:06:07 +00:00
Erik Johnston	da146657c9	Comments	2017-03-16 13:04:07 +00:00
Erik Johnston	6957bfdca6	Don't recreate so many sets	2017-03-16 11:54:26 +00:00
Richard van der Hoff	29ed09e80a	Fix assertion to stop transaction queue getting wedged ... ... and update some docstrings to correctly reflect the types being used. get_new_device_msgs_for_remote can return a long under some circumstances, which was being stored in last_device_list_stream_id_by_dest, and was then upsetting things on the next loop.	2017-03-15 12:16:55 +00:00
Erik Johnston	bb256ac96f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed	2017-03-14 11:35:05 +00:00
Erik Johnston	cc7a294e2e	Fix current_state_events table to not lie ... If we try and persist two state events that have the same ancestor we calculate the wrong current state when persisting those events.	2017-03-14 10:57:43 +00:00
Luke Barnard	bbeeb97f75	Implement _simple_delete_many_txn, use it to delete devices ... (But this doesn't implement the same for deleting access tokens or e2e keys. Also respond to code review.	2017-03-13 17:53:23 +00:00
Erik Johnston	79926e016e	Assume rooms likely haven't changed	2017-03-13 09:50:10 +00:00
Erik Johnston	8ffbe43ba1	Get current state by using current_state_events table	2017-03-10 17:39:35 +00:00
Erik Johnston	64d62e41b8	Noop repated delete device inbox calls from /sync	2017-03-10 10:36:43 +00:00
Richard van der Hoff	6ad71cc29d	Remove spurious SQL logging (#1972) ... looks like the upsert function was accidentally sending sql logging to the general logger. We already log the sql in `txn.execute`.	2017-03-08 18:00:44 +00:00
Erik Johnston	ac5491f563	Select distinct devices from DB ... Otherwise we might pull out tonnes of duplicate user_ids and this can make synapse sad.	2017-03-06 11:10:14 +00:00
Erik Johnston	da52d3af31	Fix up	2017-03-03 15:29:13 +00:00
Erik Johnston	d766343668	Add index to device_lists_stream	2017-03-01 15:56:30 +00:00
Erik Johnston	3365117151	Clobber old device list stream entries	2017-03-01 10:21:30 +00:00
Erik Johnston	d638a7484b	Merge pull request #1959 from matrix-org/erikj/intern_once ... Intern table column names once	2017-02-28 15:15:16 +00:00
Erik Johnston	b84907bdbb	Intern table column names once	2017-02-28 14:38:16 +00:00
Erik Johnston	e4919b9329	Add stream_id index to device_lists_outbound_pokes ... As this is used for replication streaming	2017-02-28 11:19:06 +00:00
Erik Johnston	8a12b6f1eb	Fix up txn name	2017-02-28 10:15:50 +00:00
Erik Johnston	a41dce8f8a	Remove needless check	2017-02-27 18:54:43 +00:00
Erik Johnston	c0d6045776	It should be all	2017-02-27 18:45:24 +00:00
Erik Johnston	49f4bc4709	Don't fetch current state in common case ... Currently we fetch the list of current state events whenever we send something in a room. This is overkill for the common case of persisting a simple chain of non-state events, so lets handle that case specially.	2017-02-27 18:33:41 +00:00
Erik Johnston	fc5d876dba	Merge pull request #1954 from matrix-org/erikj/cache_device2 ... Cache get_user_devices_from_cache	2017-02-27 16:36:40 +00:00
Erik Johnston	f58dbb02a6	Cache get_user_devices_from_cache	2017-02-27 16:22:12 +00:00
Erik Johnston	1a4f8022e6	Strip newlines from SQL queries	2017-02-23 11:15:31 +00:00
Erik Johnston	b2d20e94fa	Remove lock from rotate notifs	2017-02-22 14:24:02 +00:00
Erik Johnston	b7442c3e2b	Store looping call	2017-02-21 13:59:25 +00:00
Erik Johnston	699be7d1be	Fix up notif rotation	2017-02-18 14:42:39 +00:00
David Baker	4aa29508af	Use TEXT rather than VARCHAR ... While we're changing anyway	2017-02-17 10:51:49 +00:00
David Baker	b4017539d4	Make the pushers lang field column longer ... To accommodate things like zh-Hans-CN Fixes https://github.com/vector-im/riot-ios/issues/1031	2017-02-17 10:42:57 +00:00
Erik Johnston	138e030cfe	Comment	2017-02-16 15:03:36 +00:00
Erik Johnston	502ae6c663	Comment	2017-02-16 14:47:11 +00:00
Erik Johnston	e6acf0c399	Store the default push actions in a more efficient manner	2017-02-16 14:40:24 +00:00
Erik Johnston	04eca2589d	Merge pull request #1916 from matrix-org/erikj/push_actions_delete ... Aggregate event push actions	2017-02-16 15:28:58 +01:00
Richard van der Hoff	7dcbcca68c	Merge pull request #1921 from matrix-org/rav/fix_key_changes ... Fix bugs in the /keys/changes api	2017-02-15 11:25:16 +00:00
Richard van der Hoff	fc2f29c1d0	Fix bugs in the /keys/changes api ... * `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were passing a `RoomStreamToken`, which meant that it returned the *current* extremities, rather than those corresponding to the `from_token`. However: * `get_state_ids_for_events` required a second ('types') parameter; this meant that a `TypeError` was thrown and we ended up acting as though there was *no* prev state. * `get_state_ids_for_events` actually returns a map from event_id to state dictionary - just looking up the state keys in it again meant that we acted as though there was no prev state. We now check if each member's state has changed since *any* of the extremities. Also add/fix some comments.	2017-02-14 13:59:50 +00:00
Erik Johnston	ce3c8df6df	Less aggressive timers	2017-02-14 13:41:24 +00:00
Erik Johnston	095b45c165	Aggregate event push actions	2017-02-14 13:39:41 +00:00
Erik Johnston	795f8e3fe7	Merge pull request #1873 from matrix-org/erikj/delete_push_actions ... Be more agressive about purging old room event_push_actions	2017-02-14 14:29:04 +01:00
Erik Johnston	9e617cd4c2	Cache get_presence storage	2017-02-13 13:50:03 +00:00
Erik Johnston	52cd019a54	Make None check explicit	2017-02-08 16:04:29 +00:00
Erik Johnston	f20cd34858	Merge pull request #1892 from matrix-org/erikj/rejection_fwd_extrem ... Ignore new rejected events when working out forward extremities.	2017-02-08 16:59:06 +01:00
Erik Johnston	7723b4caa4	Ignore new rejected events when working out forward extremeties.	2017-02-08 14:48:06 +00:00
Erik Johnston	af6da6db2d	Merge pull request #1784 from morteza-araby/user-admin ... Administration functionalities	2017-02-06 16:21:10 +01:00
Erik Johnston	38258a0976	Bump cache sizes for common membership queries	2017-02-02 18:45:55 +00:00
Erik Johnston	832e9c52ca	Comment	2017-02-02 13:09:56 +00:00
Erik Johnston	54a79c1d37	Make presence.get_new_events a bit faster ... We do this by caching the set of users a user shares rooms with.	2017-02-02 13:07:18 +00:00
Morteza Araby	2849d3f29d	admin,storage: added more administrator functionalities ... administrators can now: - Set displayname of users - Update user avatars - Search for users by user_id - Browse all users in a paginated API - Reset user passwords - Deactivate users Helpers for doing paginated queries has also been added to storage Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>	2017-02-02 14:02:26 +01:00
Erik Johnston	fac3c03087	Be more agressive about purging old room event_push_actions	2017-02-01 18:27:24 +00:00
Erik Johnston	fbfe44bb4d	Doc args	2017-02-01 17:52:57 +00:00
Erik Johnston	d61a04583e	Comment	2017-02-01 17:35:23 +00:00
Erik Johnston	7e919bdbd0	Include newly joined users in /keys/changes API	2017-02-01 17:33:16 +00:00
Erik Johnston	ebfaff84c9	Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events ... Add a small cache get_all_new_events	2017-02-01 13:22:02 +00:00
Erik Johnston	62f6b86ba7	Merge pull request #1868 from matrix-org/erikj/replication_cache ... Only invalidate membership caches based on the cache stream	2017-02-01 13:12:30 +00:00
Erik Johnston	f6124311fd	Add m.room.member type to query	2017-02-01 11:59:17 +00:00
Erik Johnston	88a4d54883	Merge pull request #1867 from matrix-org/erikj/member_index ... Add an index to make membership queries faster	2017-02-01 11:44:27 +00:00
Erik Johnston	368c88c487	Add a small cache get_all_new_events	2017-02-01 10:50:44 +00:00
Erik Johnston	5deaf9e30b	Up get_latest_event_ids_in_room cache	2017-02-01 10:39:41 +00:00
Erik Johnston	458b6f4733	Only invalidate membership caches based on the cache stream ... Before we completely invalidated get_users_in_room whenever we updated any current_state_events table. This was way too aggressive.	2017-01-31 16:09:03 +00:00
Erik Johnston	21b7375778	Add an index to make membership queries faster	2017-01-31 15:15:57 +00:00
Erik Johnston	ac001dabdc	Merge pull request #1864 from matrix-org/erikj/device_list_fixes ... Fix clearing out old device list outbound pokes	2017-01-31 13:35:35 +00:00
Erik Johnston	bfb3d255b1	Merge pull request #1862 from matrix-org/erikj/presence_update ... Use DB cache of joined users for presence	2017-01-31 13:23:24 +00:00
Erik Johnston	ab55794b6f	Fix deletion of old sent devices correctly	2017-01-31 13:22:41 +00:00
Erik Johnston	d3169e8d28	Only fetch with row ts and count > 1	2017-01-31 11:20:03 +00:00
Erik Johnston	05b9f48ee5	Fix clearing out old device list outbound pokes	2017-01-31 10:08:55 +00:00
Erik Johnston	4b3403ca9b	Stream cache invalidations for room membership storage functions	2017-01-30 17:28:22 +00:00
Erik Johnston	1c13c9f6b6	Don't have such a large cache	2017-01-30 17:12:14 +00:00
Erik Johnston	c7a26b7c32	Fix unit tests	2017-01-30 17:11:24 +00:00
Erik Johnston	c2c9a78db9	Noop device key changes if they're the same	2017-01-30 16:55:04 +00:00
Erik Johnston	e75a779d9e	Fix query	2017-01-30 16:38:20 +00:00
Erik Johnston	828db669ec	Use get_users_in_room and declare it iterable	2017-01-30 16:37:22 +00:00
Erik Johnston	3670025e64	Rename func	2017-01-30 14:11:31 +00:00
Erik Johnston	d360c97ae1	Clear out old destination pokes.	2017-01-30 10:14:37 +00:00
Erik Johnston	76100203ab	Always use the latest stream_id, sent or unsent	2017-01-30 10:14:25 +00:00
Erik Johnston	d1e1fd6210	Add ts column to device_lists_outbound_pokes	2017-01-27 15:23:48 +00:00
Erik Johnston	252b503fc8	Hook device list updates to replication	2017-01-27 14:31:35 +00:00
Erik Johnston	84a35f32c7	Comment	2017-01-27 10:35:12 +00:00
Erik Johnston	c517a19c2d	Comment	2017-01-27 10:33:26 +00:00
Erik Johnston	738a2867c8	SQL param ordering	2017-01-27 10:31:29 +00:00
Erik Johnston	755adff0e4	User if rather than for	2017-01-27 10:31:06 +00:00
Erik Johnston	888c59c955	Better name	2017-01-27 10:29:47 +00:00
Erik Johnston	f25a4a4692	Remove unused param	2017-01-27 10:27:39 +00:00
Erik Johnston	31aca5589c	Fix on sqlite: use left rather than outer join	2017-01-26 16:55:50 +00:00
Erik Johnston	76d40f4904	Handle users leaving rooms	2017-01-26 16:39:33 +00:00
Erik Johnston	fbfad76c03	Add comments	2017-01-26 16:33:21 +00:00
Erik Johnston	c974116f19	Implement device key caching over federation	2017-01-26 16:07:24 +00:00
Erik Johnston	51e9fe36e4	Fix up sending of m.device_list_update edus	2017-01-25 16:55:21 +00:00
Erik Johnston	2367c5568c	Add basic implementation of local device list changes	2017-01-25 14:27:27 +00:00
Erik Johnston	fdf2a31a51	Typo	2017-01-23 16:14:14 +00:00
Erik Johnston	c77b24c092	Refactor to calculate state delta outside transaction	2017-01-23 14:51:33 +00:00
Erik Johnston	5d2134d485	Comments	2017-01-20 17:13:24 +00:00
Erik Johnston	a55fa2047f	Insert delta of current_state_events to be more efficient	2017-01-20 17:10:18 +00:00
Erik Johnston	d0897dead5	Spelling	2017-01-20 15:05:11 +00:00
Erik Johnston	567aa35b67	Update all call sites after rename	2017-01-20 14:40:31 +00:00
Erik Johnston	f2f40e64a9	Comments	2017-01-20 14:38:13 +00:00
Erik Johnston	4c6a31cd6e	Calculate the forward extremeties once	2017-01-20 14:28:53 +00:00
Erik Johnston	09eb08f910	Derive current_state_events from state groups	2017-01-20 11:52:51 +00:00
Erik Johnston	4ec1cf49e2	Lower loading events log to DEBUG	2017-01-17 17:28:32 +00:00
Erik Johnston	d906206049	Increase state_group_cache_size	2017-01-17 11:31:08 +00:00
Erik Johnston	897f8752da	Up cache max entries for state	2017-01-16 15:08:17 +00:00
Erik Johnston	01521299c7	Increase cache size limit	2017-01-16 11:56:51 +00:00
Erik Johnston	2fae34bd2c	Optionally measure size of cache by sum of length of values	2017-01-13 17:46:17 +00:00
Erik Johnston	f0325a9ccc	Merge pull request #1793 from matrix-org/erikj/change_device_inbox_index ... Change device_inbox stream index to include user	2017-01-13 15:14:51 +00:00
Erik Johnston	c050f493dd	Add comment	2017-01-13 15:14:41 +00:00
Erik Johnston	ab655dca33	Explicitly close the cursor	2017-01-10 15:15:25 +00:00
Erik Johnston	5a32e9273e	Don't disable autocommit	2017-01-10 15:11:27 +00:00
Erik Johnston	caddadfc5a	Change device_inbox stream index to include user ... This makes fetching the nost recently changed users much tricker, and brings it in line with e.g. presence_stream indices.	2017-01-10 15:04:57 +00:00
Erik Johnston	dd52d4de4c	Limit number of entries to prefill from cache ... Some tables, like device_inbox, take a long time to query at startup for the stream change cache prefills. This is likely because they are slower growing streams and so are more fragmented on disk. For now, lets pull fewer entries out to make startup quicker. In future, we should add a better index to make it even faster.	2017-01-10 14:34:50 +00:00
Matthew Hodgson	bd85434cb3	Merge branch 'release-v0.18.6' into develop	2017-01-05 13:58:19 +00:00
Matthew Hodgson	8cfc0165e9	fix annoying typos	2017-01-05 13:39:43 +00:00
Matthew Hodgson	b31ed22738	Merge branch 'release-v0.18.6' into develop	2017-01-05 13:03:02 +00:00
Mark Haines	dd3df11c55	More logging for the linearizer and for get_events	2017-01-05 12:32:47 +00:00
David Baker	0c88ab1844	Add /account/3pid/delete endpoint ... Also fix a typo in a comment	2016-12-20 18:27:30 +00:00
Richard van der Hoff	edc6a1e4f9	Add some logging for syncing to_device events ... Attempt to track down the loss of to_device events (https://github.com/vector-im/riot-web/issues/2711 etc).	2016-12-15 18:16:10 +00:00
Erik Johnston	c3208e45c9	Fixup membership query	2016-12-14 10:46:58 +00:00
Erik Johnston	8588d0eb3d	Merge pull request #1697 from matrix-org/erikj/fix_bg_member ... Fix background update that prematurely stopped	2016-12-12 17:20:17 +00:00
Erik Johnston	1574b839e0	Merge pull request #1676 from matrix-org/erikj/room_list ... Add new API appservice specific public room list	2016-12-12 17:00:10 +00:00
Erik Johnston	7ec2bf9b77	Fix background update that prematurely stopped	2016-12-12 16:54:58 +00:00
Erik Johnston	09cbcb78d3	Add cache to get_public_room_ids_at_stream_id	2016-12-12 14:41:51 +00:00
Erik Johnston	631376e2ac	Notify replication. Use correct network_id	2016-12-12 14:28:15 +00:00
Erik Johnston	d45c984653	Docstring	2016-12-12 11:00:27 +00:00
Erik Johnston	1697f6a323	Merge pull request #1680 from matrix-org/erikj/joined_rooms ... Add new room membership APIs	2016-12-09 11:35:48 +00:00
Erik Johnston	52d12ca782	Add /room/<room_id>/joined_members API ... This returns the currently joined members in the room with their display names and avatar urls. This is more efficient than /members for large rooms where you don't need the full events.	2016-12-08 13:32:07 +00:00
Erik Johnston	c45d8e9ba2	Add profile data to the room_membership table for joins	2016-12-08 13:08:41 +00:00
Richard van der Hoff	bd07a35c29	Fix result of insert_receipt ... This should fix the absence of notifications when new receipts arrive.	2016-12-08 12:11:34 +00:00
Erik Johnston	2687af82d4	Comments	2016-12-07 09:58:33 +00:00
Erik Johnston	f32fb65552	Add new API appservice specific public room list	2016-12-06 16:12:27 +00:00
Richard van der Hoff	aa09d6b8f0	Rip out more refresh_token code ... We might as well treat all refresh_tokens as invalid. Just return a 403 from /tokenrefresh, so that we don't have a load of dead, untestable code hanging around. Still TODO: removing the table from the schema.	2016-11-30 17:40:18 +00:00
Erik Johnston	46ca345b06	Don't send old events as federation	2016-11-24 12:29:02 +00:00
Erik Johnston	1985860c6e	Comment	2016-11-23 15:59:59 +00:00
Erik Johnston	2ac516850b	More efficient notif count queries	2016-11-23 15:57:04 +00:00
Erik Johnston	302fbd218d	Merge pull request #1635 from matrix-org/erikj/split_out_fed_txn ... Split out federation transaction sending to a worker	2016-11-23 15:39:12 +00:00
Erik Johnston	b69f76c106	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_fed_txn	2016-11-23 11:31:53 +00:00
Erik Johnston	0163466d72	Ignore AS users when fetching push rules ... By ignoring AS users early on when fetching push rules for a room we can avoid needlessly hitting the DB and filling up the caches.	2016-11-23 11:01:01 +00:00
Erik Johnston	54fed21c04	Fix tests and flake8	2016-11-22 18:18:31 +00:00
Erik Johnston	90565d015e	Invalidate retry cache in both directions	2016-11-22 17:45:44 +00:00
Kegan Dougal	a2a6c1c22f	Fail with a coherent error message if /sync?filter= is invalid	2016-11-21 13:15:25 +00:00
Erik Johnston	524d61bf7e	Fix tests	2016-11-21 11:53:02 +00:00
Erik Johnston	7c9cdb2245	Store federation stream positions in the database	2016-11-21 11:33:08 +00:00
Erik Johnston	f8ee66250a	Handle sending events and device messages over federation	2016-11-17 15:48:04 +00:00
Erik Johnston	ed787cf09e	Hook up the send queue and create a federation sender worker	2016-11-16 17:34:44 +00:00
Erik Johnston	c7546b3cdb	Merge pull request #1617 from matrix-org/erikj/intern_state_dict ... Correctly intern keys in state cache	2016-11-15 16:45:55 +00:00
Erik Johnston	f9d156d270	New Flake8 fixes	2016-11-15 11:22:29 +00:00
Erik Johnston	587d8ac60f	Correctly intern keys in state cache	2016-11-08 11:53:25 +00:00
Erik Johnston	63772443e6	Comment	2016-11-04 10:53:42 +00:00
Erik Johnston	a3f6576084	Remove unused but buggy function	2016-11-04 10:48:20 +00:00
Erik Johnston	64c6566980	Remove spurious comment	2016-11-03 15:04:32 +00:00
Erik Johnston	8fd4d9129f	Replace postgres GIN with GIST ... This is because GIN can be slow to write too, especially when the table gets large.	2016-11-03 15:00:03 +00:00
Erik Johnston	760469c812	Continue to clean up received_transactions	2016-11-01 11:42:08 +00:00
Erik Johnston	f09d2b692f	Removed unused stuff	2016-10-31 17:10:56 +00:00
Erik Johnston	4c3eb14d68	Increase batching of sent transaction inserts ... This should further reduce the number of individual inserts, transactions and updates that are required for keeping sent_transactions up to date.	2016-10-31 16:07:45 +00:00
Erik Johnston	a9111786f9	Use most recently added binding, not most recently seen user.	2016-10-27 14:32:45 +01:00
Erik Johnston	1fc1bc2a51	Fix user_threepids schema delta ... The delta `37/user_threepids.sql` aimed to update all the email addresses to be lower case, however duplicate emails may exist in the table already. This commit adds a step where the delta moves the duplicate emails to a new `medium` `email_old`. Only the most recently used account keeps the binding intact. We move rather than delete so that we retain some record of which emails were associated with which account.	2016-10-27 14:14:44 +01:00
Erik Johnston	d04e2ff3a4	Fix incredubly slow back pagination query ... If a client didn't specify a from token when paginating backwards synapse would attempt to query the (global) maximum topological token. This a) doesn't make much sense since they're room specific and b) there are no indices that lets postgres do this efficiently.	2016-10-24 13:35:51 +01:00
Erik Johnston	78c083f159	Merge pull request #1164 from pik/error-codes ... Clarify Error codes for GET /filter/	2016-10-19 14:26:17 +01:00
David Baker	0108ed8ae6	Latest delta is now 37	2016-10-19 11:40:35 +01:00
David Baker	a7f48320b1	Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive	2016-10-19 11:28:56 +01:00
David Baker	df2a616c7b	Convert emails to lowercase when storing ... And db migration sql to convert existing addresses.	2016-10-19 11:13:55 +01:00
pik	e8b1d2a452	Refactor test_filter to use real DataStore ... * add tests for filter api errors	2016-10-18 12:17:38 -05:00
Erik Johnston	6942d68247	Bump schema version	2016-10-17 11:17:45 +01:00
Erik Johnston	b59994b454	Remove TODO	2016-10-17 11:17:02 +01:00
Erik Johnston	816988baaa	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth	2016-10-17 11:10:37 +01:00
Erik Johnston	2869a29fd7	Drop some unused indices	2016-10-17 11:08:19 +01:00
David Baker	29c5922021	Revert part of 6207399 ... older sqlite doesn't support indexes on expressions, lets just store things lowercase in the db	2016-10-14 16:20:24 +01:00
David Baker	bcb1245a2d	Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive	2016-10-14 15:10:38 +01:00
David Baker	62073992c5	Make password reset email field case insensitive	2016-10-14 13:56:53 +01:00
Erik Johnston	3061dac53e	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_noop	2016-10-11 14:08:29 +01:00
Patrik Oldsberg	9bfc617791	storage/appservice: make appservice methods only relying on the cache synchronous	2016-10-06 15:24:59 +02:00
Erik Johnston	5875a65253	Merge pull request #1145 from matrix-org/erikj/fix_reindex ... Fix background reindex of origin_server_ts	2016-09-29 13:53:48 +01:00
Erik Johnston	9040c9ffa1	Fix background reindex of origin_server_ts ... The storage function `_get_events_txn` was removed everywhere except from this background reindex. The function was removed due to it being (almost) completely unused while also being large and complex. Therefore, instead of resurrecting `_get_events_txn` we manually reimplement the bits that are needed directly.	2016-09-27 11:23:49 +01:00
Erik Johnston	4974147aa3	Remove duplication	2016-09-27 09:27:54 +01:00
Erik Johnston	13122e5e24	Remove unused variable	2016-09-27 09:21:51 +01:00
Erik Johnston	cf3e1cc200	Fix perf of fetching state in SQLite	2016-09-26 17:16:24 +01:00
Erik Johnston	748d8fdc7b	Reduce DB hits for replication ... Some streams will occaisonally advance their positions without actually having any new rows to send over federation. Currently this means that the token will not advance on the workers, leading to them repeatedly sending a slightly out of date token. This in turns requires the master to hit the DB to check if there are any new rows, rather than hitting the no op logic where we check if the given token matches the current token. This commit changes the API to always return an entry if the position for a stream has changed, allowing workers to advance their tokens correctly.	2016-09-23 16:49:21 +01:00
Erik Johnston	8009d84364	Match against event_id, rather than room_id	2016-09-21 16:46:59 +01:00
Erik Johnston	dc692556d6	Remove spurious AS clause	2016-09-21 16:28:47 +01:00
Erik Johnston	dc78db8c56	Update correct table	2016-09-21 15:52:44 +01:00
Erik Johnston	4f78108d8c	Readd entries to public_room_list_stream that were deleted	2016-09-21 15:24:22 +01:00
Erik Johnston	0b78d8adf2	Fix _delete_old_forward_extrem_cache query	2016-09-21 15:20:56 +01:00
Erik Johnston	a68807d426	Comment	2016-09-16 11:36:20 +01:00
Erik Johnston	e58a9d781c	Filter remote rooms lists locally	2016-09-16 10:19:32 +01:00
Erik Johnston	de4f798f01	Handling expiring stream extrems correctly.	2016-09-15 17:34:59 +01:00
Erik Johnston	955f34d23e	Change get_pos_of_last_change to return upper bound	2016-09-15 15:12:07 +01:00
Erik Johnston	cb3edec6af	Use stream_change cache to make get_forward_extremeties_for_room cache more effective	2016-09-15 14:28:13 +01:00
Erik Johnston	55e6fc917c	Add cache to get_forward_extremeties_for_room	2016-09-15 14:04:28 +01:00
Erik Johnston	e457034e99	Merge pull request #1121 from matrix-org/erikj/public_room_paginate ... Add pagination support to publicRooms	2016-09-15 13:27:09 +01:00
Erik Johnston	211786ecd6	Stream public room changes down replication	2016-09-15 11:47:23 +01:00
Erik Johnston	4fb65a1091	Base public room list off of public_rooms stream	2016-09-15 11:27:04 +01:00
Erik Johnston	c566f0ee17	Calculate the public room list from a stream_ordering	2016-09-14 17:42:47 +01:00
Erik Johnston	baffe96d95	Add a room visibility stream	2016-09-14 17:29:19 +01:00
Erik Johnston	ed992ae6ba	Add a DB index to figure out past state at a stream ordering in a room	2016-09-14 16:20:27 +01:00
Erik Johnston	00f51493f5	Fix reindex	2016-09-14 10:18:30 +01:00
Erik Johnston	d5ae1f1291	Ensure we don't mutate state cache entries	2016-09-14 10:03:48 +01:00
Erik Johnston	43253c10b8	Remove redundant event_auth index	2016-09-13 11:47:48 +01:00
Erik Johnston	b17af156c7	Remove where clause	2016-09-12 17:05:54 +01:00
Erik Johnston	0b32bb20bb	Index contains_url for file search queries	2016-09-12 16:57:05 +01:00
Erik Johnston	c94de0ab60	Add WHERE clause support to index creation	2016-09-12 16:55:01 +01:00
Erik Johnston	7c679b1118	Merge pull request #1106 from matrix-org/erikj/state_reindex_concurrent ... Create new index concurrently	2016-09-12 14:38:40 +01:00
Erik Johnston	d080b3425c	Merge pull request #1105 from matrix-org/erikj/make_notif_highlight_query_fast ... Optimise /notifications query	2016-09-12 14:34:12 +01:00
Erik Johnston	03a98aff3c	Create new index concurrently	2016-09-12 14:27:01 +01:00
Erik Johnston	fa20c9ce94	Change the index to be stream_ordering, highlight	2016-09-12 14:04:08 +01:00
Erik Johnston	5ef5435529	Remove unused import	2016-09-12 13:32:58 +01:00
Mark Haines	aa7b890cfe	Merge pull request #1104 from matrix-org/markjh/direct_to_device_federation_sync ... Fix direct to device messages recieved over federation to notify sync	2016-09-12 13:25:23 +01:00
Erik Johnston	7cd6edb947	Use register_background_index_update	2016-09-12 12:54:48 +01:00
Erik Johnston	0294c14ec4	Add back in query change	2016-09-12 12:43:56 +01:00
Erik Johnston	7fe42cf949	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast	2016-09-12 12:37:09 +01:00
Erik Johnston	15ca0c6a4d	Make reindex happen in bg	2016-09-12 12:36:36 +01:00
Mark Haines	a232e06100	Fix direct to device messages recieved over federation to notify sync	2016-09-12 12:30:46 +01:00
Mark Haines	31f85f9db9	Add comments to existing schema deltas that used "CREATE INDEX" directly	2016-09-12 11:00:26 +01:00
Erik Johnston	54417999b6	Revert "Add index to event_push_actions"	2016-09-12 10:39:55 +01:00
Erik Johnston	897d57bc58	Change state fetch query for postgres to be faster ... It turns out that postgres doesn't like doing a list of OR's and is about 1000x slower, so we just issue a query for each specific type seperately.	2016-09-12 10:05:07 +01:00
David Baker	b91e2833b3	Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast	2016-09-09 19:11:34 +01:00
David Baker	f2acc3dcf9	Add index to event_push_actions ... and remove room_id caluse so it uses it Mostly from @negativemjark	2016-09-09 18:54:54 +01:00
Mark Haines	647c724573	Use the previous MAX value if any to set the stream_id	2016-09-09 11:52:44 +01:00
Mark Haines	6a6cbfcf1e	Track the max_stream_device_id in a separate table, since we delete from the inbox table	2016-09-09 11:48:23 +01:00
Mark Haines	303b6f29f0	Merge pull request #1087 from matrix-org/markjh/reapply_delta ... Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…	2016-09-09 11:24:27 +01:00
Erik Johnston	1fe7ca1362	Merge branch 'release-v0.17.3' of github.com:matrix-org/synapse into develop	2016-09-09 11:15:40 +01:00
Mark Haines	0877157353	Just move the schema and add some DROPs	2016-09-09 11:04:47 +01:00
Erik Johnston	2ffec928e2	Reduce batch size to be under SQL limit	2016-09-09 11:03:31 +01:00
Matthew Hodgson	b8f84f99ff	Merge pull request #1081 from matrix-org/dbkr/notifications_only_highlight ... Implement `only=highlight` on `/notifications`	2016-09-09 00:09:51 +01:00
Mark Haines	43b77c5d97	Only catch databas errors	2016-09-08 17:44:21 +01:00
Mark Haines	7d5b142547	Add a stub run_upgrade	2016-09-08 17:39:11 +01:00
David Baker	c3276aef25	Merge pull request #1080 from matrix-org/dbkr/fix_notifications_api_with_from ... Fix /notifications API when used with `from` param	2016-09-08 17:35:35 +01:00
Mark Haines	fa722a699c	Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop	2016-09-08 17:35:16 +01:00
Erik Johnston	5c688739d6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reindex_state_groups	2016-09-08 16:52:09 +01:00
Erik Johnston	ebb46497ba	Add delta file	2016-09-08 16:38:54 +01:00
Erik Johnston	5beda10bbd	Reindex state_groups_state after pruning	2016-09-08 16:18:01 +01:00
Mark Haines	a1c8f268e5	Support wildcard device_ids for direct to device messages	2016-09-08 15:13:05 +01:00
Mark Haines	fa9d36e050	Merge branch 'develop' into markjh/direct_to_device_federation	2016-09-08 13:43:43 +01:00
David Baker	4ef222ab61	Implement only=highlight on /notifications	2016-09-08 13:43:35 +01:00
Erik Johnston	61cd9af09b	Log delta files we're applying	2016-09-08 13:40:46 +01:00
David Baker	c5b49eb7ca	Fix /notifications API when used with from param	2016-09-08 09:40:10 +01:00
Erik Johnston	b568ca309c	Temporarily disable sequential scans for state fetching	2016-09-08 09:38:54 +01:00
Mark Haines	3c320c006c	Merge pull request #1077 from matrix-org/markjh/device_logging ... Log the types and values when failing to store devices	2016-09-07 18:24:24 +01:00
Mark Haines	85b51fdd6b	Log the types and values when failing to store devices	2016-09-07 17:19:18 +01:00
Mark Haines	2a0159b8ae	Fix the stream change cache to work over replication	2016-09-07 15:58:00 +01:00
Mark Haines	31a07d2335	Add stream change caches for device messages	2016-09-07 15:27:07 +01:00
Erik Johnston	513188aa56	Comment	2016-09-07 14:53:23 +01:00
Erik Johnston	fadb01551a	Add appopriate framing clause	2016-09-07 14:39:01 +01:00
Erik Johnston	d25c20ccbe	Use windowing function to make use of index	2016-09-07 14:22:22 +01:00
Mark Haines	7d893beebe	Comment the add_messages storage functions	2016-09-07 12:03:37 +01:00
Mark Haines	d4a35ada28	Send device messages over federation	2016-09-06 18:16:20 +01:00
Mark Haines	e020834e4f	Add storage methods for federated device messages	2016-09-06 15:12:13 +01:00
Mark Haines	2ad72da931	Add tables for federated device messages ... Adds tables for storing the messages that need to be sent to a remote device and for deduplicating messages received.	2016-09-06 15:10:40 +01:00
Mark Haines	f4164edb70	Move _add_messages_to_device_inbox_txn into a separate method	2016-09-06 11:26:37 +01:00
Erik Johnston	0595413c0f	Scale the batch size so that we're not bitten by the minimum	2016-09-05 15:49:57 +01:00
Erik Johnston	a7032abb2e	Correctly handle reindexing state groups that already have an edge	2016-09-05 15:07:23 +01:00
Erik Johnston	9e6d88f4e2	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage	2016-09-05 15:01:33 +01:00
Erik Johnston	70332a12dd	Take value in a better way	2016-09-05 14:57:14 +01:00
Erik Johnston	373654c635	Comment about sqlite and WITH RECURSIVE	2016-09-05 14:50:36 +01:00
Erik Johnston	485d999c8a	Correctly delete old state groups in purge history API	2016-09-05 14:49:08 +01:00
Erik Johnston	bb36b93f71	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage	2016-09-05 11:53:11 +01:00
Erik Johnston	caed150363	Remove unused imports	2016-09-05 10:52:01 +01:00
Erik Johnston	80a6a445fa	Only fetch local pushers	2016-09-05 10:43:32 +01:00
Erik Johnston	628e65721b	Add comments	2016-09-05 10:41:27 +01:00
Erik Johnston	a99e933550	Add upgrade script that will slowly prune state_groups_state entries	2016-09-05 10:05:36 +01:00
Erik Johnston	3847fa38c4	Make bulk_get_push_rules_for_room use get_joined_users_from_context cache	2016-09-05 10:02:38 +01:00
Mark Haines	81b94c5750	Merge pull request #1066 from matrix-org/markjh/direct_to_device_lowerbound ... Only return new device messages in /sync	2016-09-02 16:18:34 +01:00
Mark Haines	65fa37ac5e	Only return new device messages in /sync	2016-09-02 15:50:37 +01:00
Erik Johnston	3baf641a48	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage	2016-09-02 14:54:07 +01:00
Erik Johnston	cce957e254	Bump max_entries on get_destination_retry_timings	2016-09-02 14:08:33 +01:00
Erik Johnston	657847e4c6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage	2016-09-02 11:04:48 +01:00
Erik Johnston	598317927c	Limit the length of state chains	2016-09-02 10:41:38 +01:00
Erik Johnston	9e25443db8	Move to storing state_groups_state as deltas	2016-09-01 14:31:26 +01:00
Erik Johnston	44982606ee	Merge pull request #1060 from matrix-org/erikj/state_ids ... Assign state groups in state handler.	2016-09-01 14:20:42 +01:00
Erik Johnston	0cfd6c3161	Use state_groups table to test existence	2016-08-31 16:25:57 +01:00
Erik Johnston	5405351b14	Lower get_linearized_receipts_for_room cache size	2016-08-31 16:19:44 +01:00
Erik Johnston	1671913287	Merge pull request #1059 from matrix-org/erikj/sent_transaction_delete ... Clean up old sent transactions	2016-08-31 15:07:33 +01:00
Erik Johnston	c10cb581c6	Correctly handle the difference between prev and current state	2016-08-31 14:26:22 +01:00
Erik Johnston	ef0cc648cf	Clean up old sent transactions	2016-08-31 11:12:02 +01:00
Mark Haines	a662252758	Return the current stream position from add_messages_to_device_inbox	2016-08-31 10:42:52 +01:00
Mark Haines	1aa3e1d287	Add a replication stream for direct to device messages	2016-08-31 10:38:58 +01:00
Erik Johnston	1bb8ec296d	Generate state group ids in state layer	2016-08-31 10:09:46 +01:00
Erik Johnston	5dc2a702cf	Make _state_groups_id_gen a normal IdGenerator	2016-08-30 16:55:11 +01:00
Erik Johnston	55fc17cf4b	Merge pull request #1049 from matrix-org/erikj/presence_users_in_room ... Use state handler instead of get_users_in_room/get_joined_hosts	2016-08-30 10:50:37 +01:00
Erik Johnston	bed10f9880	Use state handler instead of get_users_in_room/get_joined_hosts	2016-08-26 14:54:30 +01:00
Mark Haines	4bbef62124	Merge remote-tracking branch 'origin/develop' into markjh/direct_to_device	2016-08-26 14:35:31 +01:00
Erik Johnston	4daa397a00	Add is_host_joined to slave storage	2016-08-26 13:10:56 +01:00
Erik Johnston	1ccdc1e93a	Cache check_host_in_room	2016-08-26 10:59:40 +01:00
Erik Johnston	778fa85f47	Make sync not pull out full state	2016-08-25 18:59:44 +01:00
Mark Haines	ab34fdecb7	Merge branch 'develop' into markjh/direct_to_device	2016-08-25 18:34:46 +01:00
Mark Haines	641efb6a39	Fix the deduplication of incoming direct-to-device messages	2016-08-25 18:14:02 +01:00
Erik Johnston	721414d98a	Add desc	2016-08-25 17:49:05 +01:00
Mark Haines	e993925279	Add store-and-forward direct-to-device messaging	2016-08-25 17:35:37 +01:00
Erik Johnston	a3dc1e9cbe	Replace context.current_state with context.current_state_ids	2016-08-25 17:32:22 +01:00
Erik Johnston	17f4f14df7	Pull out event ids rather than full events for state	2016-08-25 13:42:44 +01:00
Erik Johnston	9219139351	Preserve some logcontexts	2016-08-24 11:58:40 +01:00
Erik Johnston	122c7a43c9	Merge pull request #1038 from matrix-org/erikj/receved_txn_purge ... Delete old received_transactions rows	2016-08-23 11:02:40 +01:00
Erik Johnston	d143f211c8	Merge pull request #1028 from matrix-org/dbkr/notifications_api ... Add the Notifications API	2016-08-22 18:23:24 +01:00
Erik Johnston	aad8a1a825	Delete old received_transactions	2016-08-22 16:29:46 +01:00
Erik Johnston	e65bc7d315	Merge pull request #1031 from matrix-org/erikj/measure_notifier ... Add more Measure blocks	2016-08-22 12:13:07 +01:00
Matthew Hodgson	6e80c03d45	Merge branch 'develop' into dbkr/notifications_api	2016-08-20 00:16:18 +01:00
Erik Johnston	37adde32dc	Move defer.returnValue out of Measure	2016-08-19 18:23:44 +01:00
Erik Johnston	47dd8f02a1	Measure _get_event_from_row	2016-08-19 18:23:44 +01:00
Erik Johnston	e6784daf07	Merge pull request #1030 from matrix-org/erikj/cache_contexts ... Add concept of cache contexts	2016-08-19 16:29:58 +01:00
Erik Johnston	c0d7d9d642	Rename to on_invalidate	2016-08-19 15:13:58 +01:00
Erik Johnston	dc76a3e909	Make cache_context an explicit option	2016-08-19 15:02:38 +01:00
Erik Johnston	f164fd9220	Move _bulk_get_push_rules_for_room to storage layer	2016-08-19 14:29:20 +01:00
Erik Johnston	ba214a5e32	Remove lru option	2016-08-19 14:17:11 +01:00
Erik Johnston	b770435389	Make get_new_events_for_appservice use indices	2016-08-19 10:28:42 +01:00
David Baker	1e4217c90c	Explicit join	2016-08-18 17:53:44 +01:00
David Baker	602c84cd9c	Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api	2016-08-18 17:15:26 +01:00
Erik Johnston	07229bbdae	Add appservice worker	2016-08-18 14:59:55 +01:00
Erik Johnston	9da84a9a1e	Make AppserviceHandler stream events from database ... This is for two reasons: 1. Suppresses duplicates correctly, as the notifier doesn't do any duplicate suppression. 2. Makes it easier to connect the AppserviceHandler to the replication stream.	2016-08-18 11:54:41 +01:00
Erik Johnston	732cf72b86	Fix push_display_name_rename schema update	2016-08-17 18:12:21 +01:00
David Baker	0d7cef0943	Merge pull request #1011 from matrix-org/dbkr/contains_display_name_override ... Move display name rule	2016-08-17 16:15:45 +01:00
Erik Johnston	949629291c	Do it in storage function	2016-08-16 17:05:34 +01:00
David Baker	ad42322257	Add migration script ... To port existing rule actions & enable entries to the new name	2016-08-16 16:56:30 +01:00
Erik Johnston	25c2332071	Merge pull request #1010 from matrix-org/erikj/refactor_deletions ... Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.	2016-08-16 11:37:53 +01:00
Erik Johnston	a2427981b7	Use cached get_user_by_access_token in slaves	2016-08-16 11:24:32 +01:00
Erik Johnston	6cbd1b495e	Merge branch 'fix_integrity_retry' of https://github.com/Ralith/synapse into Ralith-fix_integrity_retry	2016-08-16 10:50:24 +01:00
Erik Johnston	dc3a00f24f	Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.	2016-08-15 17:04:39 +01:00
Erik Johnston	89e786bd85	Doc get_next() context manager usage	2016-08-15 13:45:26 +01:00
Erik Johnston	d9664344ec	Rename table. Add docs.	2016-08-15 11:45:57 +01:00
Erik Johnston	64e7e11853	Implement cache replication stream	2016-08-15 11:16:45 +01:00
Erik Johnston	4d70d1f80e	Add some invalidations to a cache_stream	2016-08-15 11:15:17 +01:00
Erik Johnston	99bbd90b0d	Always run txn.after_callbacks	2016-08-15 09:45:44 +01:00
Benjamin Saunders	8a57cc3123	Add missing database corruption recovery case ... Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>	2016-08-14 11:50:22 -07:00
David Baker	b4ecf0b886	Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api	2016-08-11 14:09:13 +01:00
Erik Johnston	a8b946decb	Raise 404 when couldn't find event	2016-08-05 15:31:02 +01:00
Erik Johnston	f0fa66f495	Delete more tables	2016-08-05 10:40:08 +01:00
Benjamin Saunders	a2b7102eea	Tweak integrity error recovery to work as intended	2016-08-04 20:38:08 -07:00
Erik Johnston	8ad8490cff	Fix typo	2016-08-04 15:21:29 +01:00
Erik Johnston	59fa91fe88	Retry event persistence on IntegrityError ... Due to a bug in the porting script some backfilled events were not correctly persisted, causing irrecoverable IntegrityErrors on future attempts to persist those events. This commit adds a retry mechanism invoked upon IntegrityError, where when retried the tables are purged for all references to the events being persisted.	2016-08-04 15:02:15 +01:00
Richard van der Hoff	98385888b8	PEP8	2016-08-03 15:42:08 +01:00
Richard van der Hoff	68264d7404	Include device name in /keys/query response ... Add an 'unsigned' section which includes the device display name.	2016-08-03 15:42:08 +01:00
Richard van der Hoff	530bc862dc	Merge branch 'rav/null_default_device_displayname' into develop	2016-08-03 14:30:32 +01:00
Erik Johnston	80ad710217	Remove other bit of deduplication	2016-08-03 13:25:59 +01:00
Richard van der Hoff	4fec5e57be	Default device_display_name to null ... It turns out that it's more useful to return a null device display name (and let clients decide how to handle it: eg, falling back to device_id) than using a constant string like "unknown device".	2016-08-03 11:53:00 +01:00
Erik Johnston	a8a32d2714	Ensure we only persist an event once at a time	2016-08-03 11:23:39 +01:00
Erik Johnston	5aa024e501	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader	2016-07-29 11:24:56 +01:00
Mark Haines	8dad08a950	Fix SQL to supply arguments in the same order	2016-07-29 09:57:13 +01:00
Mark Haines	0a7d3cd00f	Create separate methods for getting messages to push ... for the email and http pushers rather than trying to make a single method that will work with their conflicting requirements. The http pusher needs to get the messages in ascending stream order, and doesn't want to miss a message. The email pusher needs to get the messages in descending timestamp order, and doesn't mind if it misses messages.	2016-07-28 20:24:24 +01:00
Kegan Dougal	370135ad0b	Comment get_unread_push_actions_for_user_in_range function	2016-07-28 16:47:37 +01:00
Erik Johnston	6ede23ff1b	Add more key storage funcs into slave store	2016-07-28 15:41:26 +01:00
Richard van der Hoff	40e539683c	Merge pull request #956 from matrix-org/rav/check_device_id_on_key_upload ... Make the device id on e2e key upload optional	2016-07-27 18:16:57 +01:00
Richard van der Hoff	26cb0efa88	SQL syntax fix	2016-07-27 12:30:22 +01:00
Richard van der Hoff	d47115ff8b	Delete e2e keys on device delete	2016-07-27 12:24:52 +01:00
Richard van der Hoff	2e3d90d67c	Make the device id on e2e key upload optional ... We should now be able to get our device_id from the access_token, so the device_id on the upload request is optional. Where it is supplied, we should check that it matches. For active access_tokens without an associated device_id, we ought to register the device in the devices table. Also update the table on upgrade so that all of the existing e2e keys are associated with real devices.	2016-07-26 23:38:12 +01:00
Mark Haines	87ffd21b29	Fix a couple of bugs in the transaction and keyring code	2016-07-26 19:19:08 +01:00
Richard van der Hoff	33d7776473	Fix typo	2016-07-26 13:32:15 +01:00
Richard van der Hoff	9ad8d9b17c	Merge branch 'develop' into rav/delete_refreshtoken_on_delete_device	2016-07-26 13:29:46 +01:00
Mark Haines	9c4cf83259	Merge pull request #948 from matrix-org/markjh/auth_fixes ... Don't add rejections to the state_group, persist all rejections	2016-07-26 13:22:57 +01:00
Richard van der Hoff	8e02494166	Delete refresh tokens when deleting devices	2016-07-26 11:10:37 +01:00
Mark Haines	a6f06ce3e2	Fix how push_actions are redacted.	2016-07-26 11:05:39 +01:00
Mark Haines	efeb6176c1	Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically	2016-07-26 10:49:52 +01:00
Richard van der Hoff	012b4c1913	Implement updating devices ... You can update the displayname of devices now.	2016-07-26 07:35:48 +01:00
Richard van der Hoff	436bffd15f	Implement deleting devices	2016-07-26 07:35:48 +01:00
Mark Haines	1b3c3e6d68	Only update the events and event_json tables for rejected events	2016-07-25 18:44:30 +01:00
Mark Haines	8f7f4cb92b	Don't add the events to forward extremities if the event is rejected	2016-07-25 17:13:37 +01:00
Mark Haines	2623cec874	Don't add rejections to the state_group, persist all rejections	2016-07-25 16:12:16 +01:00
Mark Haines	955ef1f06c	fix: defer.returnValue takes one argument	2016-07-25 16:04:45 +01:00
Richard van der Hoff	2ee4c9ee02	background updates: fix assert again	2016-07-25 16:01:46 +01:00
Richard van der Hoff	9dbd903f41	background updates: Fix assertion to do something	2016-07-25 14:05:23 +01:00
Richard van der Hoff	465117d7ca	Fix background_update tests ... A bit of a cleanup for background_updates, and make sure that the real background updates have run before we start the unit tests, so that they don't interfere with the tests.	2016-07-25 12:10:42 +01:00
Richard van der Hoff	363786845b	PEP8	2016-07-22 13:21:07 +01:00
Richard van der Hoff	ec5717caf5	Create index on user_ips in the background ... user_ips is kinda big, so really we want to add the index in the background once we're running. Replace the schema delta with one which will do that. I've done this in a way that's reasonably easy to reuse as there a few other indexes I need, and I don't suppose they will be the last.	2016-07-22 13:16:39 +01:00
Erik Johnston	aede7248ab	Split out a FederationReader process	2016-07-21 17:37:44 +01:00
Richard van der Hoff	c445f5fec7	storage/client_ips: remove some dead code	2016-07-21 11:58:47 +01:00
Richard van der Hoff	7314bf4682	Merge branch 'develop' into rav/get_devices_api ... (pick up PR #938 in the hope of fixing the UTs)	2016-07-20 17:40:00 +01:00
Richard van der Hoff	bc8f265f0a	GET /devices endpoint ... implement a GET /devices endpoint which lists all of the user's devices. It also returns the last IP where we saw that device, so there is some dancing to fish that out of the user_ips table.	2016-07-20 16:42:32 +01:00
Richard van der Hoff	ec041b335e	Record device_id in client_ips ... Record the device_id when we add a client ip; it's somewhat redundant as we could get it via the access_token, but it will make querying rather easier.	2016-07-20 16:41:03 +01:00
Richard van der Hoff	053e83dafb	More doc-comments ... Fix some more comments on some things	2016-07-20 16:40:28 +01:00
David Baker	e967bc86e7	Merge pull request #932 from matrix-org/rav/register_refactor ... Further registration refactoring	2016-07-20 11:03:33 +01:00
Erik Johnston	1e2a7f18a1	Merge pull request #922 from matrix-org/erikj/file_api2 ... Feature: Add filter to /messages. Add 'contains_url' to filter.	2016-07-20 10:40:48 +01:00
Erik Johnston	f91faf09b3	Comment	2016-07-20 10:18:09 +01:00
Richard van der Hoff	40cbffb2d2	Further registration refactoring ... * `RegistrationHandler.appservice_register` no longer issues an access token: instead it is left for the caller to do it. (There are two of these, one in `synapse/rest/client/v1/register.py`, which now simply calls `AuthHandler.issue_access_token`, and the other in `synapse/rest/client/v2_alpha/register.py`, which is covered below). * In `synapse/rest/client/v2_alpha/register.py`, move the generation of access_tokens into `_create_registration_details`. This means that the normal flow no longer needs to call `AuthHandler.issue_access_token`; the shared-secret flow can tell `RegistrationHandler.register` not to generate a token; and the appservice flow continues to work despite the above change.	2016-07-19 18:46:19 +01:00
Richard van der Hoff	022b9176fe	schema fix ... device_id should be text, not bigint.	2016-07-19 11:44:05 +01:00
Richard van der Hoff	f863a52cea	Add device_id support to /login ... Add a 'devices' table to the storage, as well as a 'device_id' column to refresh_tokens. Allow the client to pass a device_id, and initial_device_display_name, to /login. If login is successful, then register the device in the devices table if it wasn't known already. If no device_id was supplied, make one up. Associate the device_id with the access token and refresh token, so that we can get at it again later. Ensure that the device_id is copied from the refresh token to the access_token when the token is refreshed.	2016-07-18 16:39:44 +01:00
Erik Johnston	f52565de50	Fix /purge_history bug ... This was caused by trying to insert duplicate backward extremeties	2016-07-15 14:23:15 +01:00
Erik Johnston	978fa53cc2	Pull out min stream_ordering from ex_outlier_stream	2016-07-15 10:22:30 +01:00
Erik Johnston	eec9609e96	event_backwards_extremeties may not be empty	2016-07-15 10:22:09 +01:00
Erik Johnston	d554ca5e1d	Add support for filters in paginate_room_events	2016-07-14 15:59:04 +01:00
Erik Johnston	b64aa6d687	Add sender and contains_url field to events table	2016-07-14 15:35:43 +01:00
Mark Haines	dfde67a6fe	Add a comment explaining allow_none	2016-07-08 15:57:06 +01:00
Mark Haines	10c843fcfb	Ensure that the guest user is in the database when upgrading accounts	2016-07-08 15:15:55 +01:00
Erik Johnston	58930da52b	Merge branch 'master' of github.com:matrix-org/synapse into develop	2016-07-08 14:11:37 +01:00
Erik Johnston	b92e7955be	Comment	2016-07-07 11:42:15 +01:00
Erik Johnston	c98e1479bd	Return 400 rather than 500	2016-07-07 11:41:07 +01:00
Erik Johnston	67f2c901ea	Add rest servlet. Fix SQL.	2016-07-06 15:56:59 +01:00
Erik Johnston	a17e7caeb7	Merge branch 'erikj/shared_secret' into erikj/test2	2016-07-06 14:46:31 +01:00
Erik Johnston	be3548f7e1	Remove spurious txn	2016-07-05 17:46:51 +01:00
Erik Johnston	4adf93e0f7	Fix for postgres	2016-07-05 17:34:25 +01:00
Erik Johnston	651faee698	Add an admin option to shared secret registration	2016-07-05 17:30:22 +01:00
Erik Johnston	8f8798bc0d	Add ReadWriteLock for pagination and history prune	2016-07-05 15:30:25 +01:00
Mark Haines	dd2ccee27d	Fix typo	2016-07-05 14:06:07 +01:00
Mark Haines	b6b0132ac7	Make get_events_around more efficient on sqlite3	2016-07-05 13:55:18 +01:00
Mark Haines	d44d11d864	Use true/false for boolean parameter inclusive to avoid potential for sqli, and possibly make the code clearer	2016-07-05 10:39:13 +01:00
Erik Johnston	2d21d43c34	Add purge_history API	2016-07-05 10:28:51 +01:00
Mark Haines	0fb76c71ac	Use different SQL for postgres and sqlite3 for when using multicolumn indexes	2016-07-04 19:44:55 +01:00
Erik Johnston	a67bf0b074	Add storage function to purge history for a room	2016-07-04 16:02:50 +01:00
Mark Haines	f18d7546c6	Use a query that postgresql optimises better for get_events_around	2016-07-04 15:48:25 +01:00
Erik Johnston	f328d95cef	Feature: Add deactivate account admin API ... Allows server admins to "deactivate" accounts, which: - Revokes all access tokens - Removes all threepids - Removes password The API is a POST to `/admin/deactivate/<user_id>`	2016-06-30 15:40:58 +01:00
Mark Haines	6783534a0f	Merge pull request #886 from matrix-org/markjh/async_commit ... Optionally make committing to postgres asynchronous.	2016-06-29 15:21:58 +01:00
Erik Johnston	a70688445d	Implement purge_media_cache admin API	2016-06-29 14:57:59 +01:00
Erik Johnston	314b146b2e	Track approximate last access time for remote media	2016-06-29 11:41:20 +01:00
David Baker	aa3a4944d5	more pep8	2016-06-24 11:45:23 +01:00
David Baker	870c45913e	Use similar naming we use in email notifs for push ... Fixes https://github.com/vector-im/vector-web/issues/1654	2016-06-24 11:41:11 +01:00
David Baker	b5fb7458d5	Actually we need to order these properly ... otherwise we'll end up returning the wrong 20	2016-06-22 18:07:14 +01:00
David Baker	f73fdb04a6	Style	2016-06-22 17:51:40 +01:00
David Baker	3a4120e49a	Put most recent 20 messages in notif ... Fixes https://github.com/vector-im/vector-web/issues/1648	2016-06-22 17:47:18 +01:00
Mark Haines	d5fb561709	Optionally make committing to postgres asynchronous. ... Useful when running tests when you don't care whether the server will lose data that it claims that it has committed.	2016-06-20 17:53:38 +01:00
Mark Haines	41e4b2efea	Add the create_profile method back since the tests use it	2016-06-17 19:20:47 +01:00
Mark Haines	9f1800fba8	Remove registered_users from the distributor. ... The only place that was observed was to set the profile. I've made it so that the profile is set within store.register in the same transaction that creates the user. This required some slight changes to the registration code for upgrading guest users, since it previously relied on the distributor swallowing errors if the profile already existed.	2016-06-17 19:14:16 +01:00
Erik Johnston	1a815fb04f	Don't hit DB for noop replications queries	2016-06-08 11:33:30 +01:00
Mark Haines	0b2158719c	Remove dead code. ... Loading push rules now happens in the datastore, so we can remove the methods that loaded them outside the datastore. The ``waiting_for_join_list`` in federation handler is populated by anything, so can be removed. The ``_get_members_events_txn`` method isn't called from anywhere so can be removed.	2016-06-07 15:07:11 +01:00
Erik Johnston	84379062f9	Fix AS retries, but with correct ordering	2016-06-07 10:24:50 +01:00
Erik Johnston	310197bab5	Fix AS retries	2016-06-07 09:34:50 +01:00
Erik Johnston	e4134c5e13	Merge pull request #841 from matrix-org/erikj/event_counter ... Add metric counter for number of persisted events	2016-06-06 14:17:40 +01:00
Erik Johnston	7aa778fba9	Add metric counter for number of persisted events	2016-06-06 11:58:09 +01:00
Erik Johnston	70aee0717c	Add events to cache when we persist them	2016-06-06 11:34:53 +01:00
Erik Johnston	cffe46408f	Don't rely on options when inserting event into cache	2016-06-03 18:25:21 +01:00
Erik Johnston	10ea3f46ba	Change the way we cache events	2016-06-03 17:57:50 +01:00
Erik Johnston	05e01f21d7	Remove event fetching from DB threads	2016-06-03 17:22:13 +01:00
Mark Haines	eef541a291	Move insert_client_ip to a separate class	2016-06-03 14:42:35 +01:00
Erik Johnston	d06f2a229e	Merge pull request #828 from matrix-org/erikj/joined_hosts_for_room ... Make get_joined_hosts_for_room use get_users_in_room	2016-06-03 11:50:30 +01:00
Erik Johnston	722472b48c	Merge pull request #825 from matrix-org/erikj/cache_push_rules ... Load push rules in storage layer so that they get cached	2016-06-03 11:44:32 +01:00
Mark Haines	b09f348530	Merge pull request #824 from matrix-org/markjh/slaved_presence_store ... Add a slaved store for presence	2016-06-03 11:26:33 +01:00
Erik Johnston	ccb56fc24b	Make get_joined_hosts_for_room use get_users_in_room	2016-06-03 11:20:23 +01:00
Erik Johnston	6a0afa582a	Load push rules in storage layer, so that they get cached	2016-06-03 11:10:00 +01:00
Mark Haines	3ae915b27e	Add a slaved store for presence	2016-06-03 11:05:53 +01:00
Erik Johnston	9c26b390a2	Only get local users	2016-06-03 11:04:31 +01:00
Erik Johnston	065e739d6e	Merge pull request #811 from matrix-org/erikj/state_users_in_room ... Use state to calculate get_users_in_room	2016-06-03 10:58:27 +01:00
Erik Johnston	696d7c5937	Merge pull request #809 from matrix-org/erikj/cache_receipts_in_room ... Add get_users_with_read_receipts_in_room cache	2016-06-03 10:58:24 +01:00
Mark Haines	80f34d7b57	Fix setting the _clock in SQLBaseStore	2016-06-02 15:23:56 +01:00
Erik Johnston	a04a2d043c	Merge pull request #807 from matrix-org/erikj/push_rules_cache ... Ensure we always return boolean in push rules	2016-06-01 18:07:48 +01:00
Erik Johnston	0f06b496d1	Merge pull request #806 from matrix-org/erikj/hash_cache ... Cache get_event_reference_hashes	2016-06-01 18:07:42 +01:00
Erik Johnston	c8285564a3	Use state to calculate get_users_in_room	2016-06-01 15:25:25 +01:00
David Baker	d60eed0710	Limit number of notifications in an email notification	2016-06-01 11:45:43 +01:00
Erik Johnston	43db0d9f6a	Add get_users_with_read_receipts_in_room cache	2016-06-01 10:54:32 +01:00
Erik Johnston	dea9f20f8c	Force boolean	2016-05-31 17:24:30 +01:00
Mark Haines	c8c5bf950a	Fix synapse/storage/schema/delta/30/as_users.py	2016-05-31 17:10:40 +01:00
Erik Johnston	c9ca285d33	Merge pull request #805 from matrix-org/erikj/push_rules_cache ... Fix GET /push_rules	2016-05-31 16:42:21 +01:00
Erik Johnston	1d4ee854e2	Fix typo	2016-05-31 15:45:53 +01:00
Erik Johnston	cca0093fa9	Change fix	2016-05-31 15:44:08 +01:00
Erik Johnston	aefd2d1cbc	Cache get_event_reference_hashes	2016-05-31 15:32:32 +01:00