4070 Commits

Author	SHA1	Message	Date
Brendan Abolivier	323d685bf7	Typo	2019-09-23 17:23:49 +01:00
Brendan Abolivier	2858d10671	Fix the return value in the users_set_deactivated_flag background job	2019-09-23 17:22:01 +01:00
Erik Johnston	51d28272e2	Query devices table for last seen info. ... This is a) simpler than querying user_ips directly and b) means we can purge older entries from user_ips without losing the required info. The storage functions now no longer return the access_token, since it was unused.	2019-09-23 16:59:45 +01:00
Erik Johnston	ed80231ade	Add BG update to populate devices last seen info	2019-09-23 16:59:45 +01:00
Erik Johnston	2ade05dca3	Add last seen info to devices table. ... This allows us to purge old user_ips entries without having to preserve the latest last seen info for active devices.	2019-09-23 16:59:35 +01:00
Andrew Morgan	30af161af2	Implement MSC2290 (#6043) ... Implements MSC2290. This PR adds two new endpoints, /unstable/account/3pid/add and /unstable/account/3pid/bind. Depending on the progress of that MSC the unstable prefix may go away. This PR also removes the blacklist on some 3PID tests which occurs in #6042, as the corresponding Sytest PR changes them to use the new endpoints. Finally, it also modifies the account deactivation code such that it doesn't just try to deactivate 3PIDs that were bound to the user's account, but any 3PIDs that were bound through the homeserver on that user's account.	2019-09-23 16:50:27 +01:00
Andrew Morgan	df3401a71d	Allow HS to send emails when adding an email to the HS (#6042)	2019-09-20 15:21:30 +01:00
Richard van der Hoff	b65327ff66	Merge branch 'develop' into rav/saml_mapping_work	2019-09-19 18:13:31 +01:00
Richard van der Hoff	bcd9132869	Undo the deletion of some tables (#6047) ... This is a partial revert of #5893. The problem is that if we drop these tables in the same release as removing the code that writes to them, it prevents users users from being able to roll back to a previous release. So let's leave the tables in place for now, and remember to drop them in a subsequent release. (Note that these tables haven't been *read* for *years*, so any missing rows resulting from a temporary upgrade to vNext won't cause a problem.)	2019-09-19 15:06:27 +01:00
Matthew Hodgson	2292dc35fc	Add experimental "dont_push" push action to suppress push for notifications ... This is a potential solution to https://github.com/vector-im/riot-web/issues/3374 and https://github.com/vector-im/riot-web/issues/5953 as raised by Mozilla at https://github.com/vector-im/riot-web/issues/10868. This lets you define a push rule action which increases the badge count (unread notification) count on a given room, but doesn't actually send a push for that notification via email or HTTP. We might want to define this as the default behaviour for group chats in future to solve https://github.com/vector-im/riot-web/issues/3268 at last. This is implemented as a string action rather than a tweak because: * Other pushers don't care about the tweak, given they won't ever get pushed * The DB can store the tweak more efficiently using the existing `notify` table. * It avoids breaking the default_notif/highlight_action optimisations. Clients which generate their own notifs (e.g. desktop notifs from Riot/Web would need to be aware of the new push action) to uphold it. An alternative way to do this would be to maintain a `msg_count` alongside `highlight_count` and `notification_count` in `unread_notifications` in sync responses. However, doing this by counting the rows in `events` since the `stream_position` of the user's last read receipt turns out to be painfully slow (~200ms), perhaps due to the size of the events table. So instead, we use the highly optimised existing event_push_actions (and event_push_actions_staging) table to maintain the counts - using the code paths which already exist for tracking unread notification counts efficiently. These queries are typically ~3ms or so. The biggest issues I see here are: * We're slightly repurposing the `notif` field on `event_push_actions` to track whether a given action actually sent a `push` or not. This doesn't seem unreasonable, but it's slightly naughty given that previously the field explicitly tracked whether `notify` was true for the action (and as a result, it was uselessly always set to 1 in the DB). * We're going to put more load on the `event_push_actions` table for all the random group chats which people had previously muted. In practice i don't think there are many of these though. * There isn't an MSC for this yet (although this comment could become one).	2019-09-19 00:54:05 +01:00
Richard van der Hoff	1e19ce00bf	Add 'failure_ts' column to 'destinations' table (#6016) ... Track the time that a server started failing at, for general analysis purposes.	2019-09-17 11:41:54 +01:00
Richard van der Hoff	a8ac40445c	Record mappings from saml users in an external table ... We want to assign unique mxids to saml users based on an incrementing suffix. For that to work, we need to record the allocated mxid in a separate table.	2019-09-13 16:01:46 +01:00
Richard van der Hoff	7902bf1e1d	Clean up some code in the retry logic (#6017) ... * remove some unused code * make things which were constants into constants for efficiency and clarity	2019-09-11 15:14:56 +01:00
Jason Robinson	63f9317b8e	Merge pull request #6004 from matrix-org/jaywink/autojoin-create-real-users ... Only count real users when checking for auto-creation of auto-join room	2019-09-09 17:37:52 +03:00
Erik Johnston	470dc621ae	Merge pull request #5934 from matrix-org/erikj/censor_redactions ... Censor redactions in DB after a month	2019-09-09 15:29:39 +01:00
Jason Robinson	aaed6b39e1	Fix code style, again ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 17:10:02 +03:00
Erik Johnston	580f3df9b2	Fix comments	2019-09-09 15:08:24 +01:00
Jason Robinson	e89fea4f04	Simplify count_real_users SQL to only count user_type is null rows ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 16:43:32 +03:00
Jason Robinson	8c03cd0e5f	Simplify is_real_user_txn check to trust user_type is null if real user ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 16:40:40 +03:00
Erik Johnston	e7184a4370	Use better names in SQL	2019-09-09 13:33:38 +01:00
Erik Johnston	916c697228	Fixup comment	2019-09-09 13:31:00 +01:00
Erik Johnston	fffe17b77d	Don't start looping call unless enabled	2019-09-09 13:24:24 +01:00
Erik Johnston	80e14a8546	Handle setting retention period to 0	2019-09-09 13:23:41 +01:00
Jason Robinson	62fac9d969	Auto-fix a few code style issues ... Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 14:59:35 +03:00
Jason Robinson	be618e0551	Only count real users when checking for auto-creation of auto-join room ... Previously if the first registered user was a "support" or "bot" user, when the first real user registers, the auto-join rooms were not created. Fix to exclude non-real (ie users with a special user type) users when counting how many users there are to determine whether we should auto-create a room. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-09-09 14:48:08 +03:00
Hubert Chathi	8e86f5b65c	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-09-07 13:20:34 -04:00
Hubert Chathi	0d61d1d735	Merge branch 'develop' into cross-signing_sig_upload	2019-09-07 13:14:45 -04:00
Amber Brown	55d5b3af88	Servers-known-about statistic (#5981)	2019-09-07 01:45:51 +10:00
Erik Johnston	a2a695b7ec	Merge pull request #5998 from matrix-org/erikj/fixup_federate_flag ... Correctly handle non-bool m.federate flag	2019-09-06 15:32:43 +01:00
Erik Johnston	142c9325c2	Correctly handle non-bool m.federate flag	2019-09-06 14:21:06 +01:00
Andrew Morgan	6ddda8152e	Move delete_threepid_session into RegistrationWorkerStore	2019-09-06 13:23:10 +01:00
Andrew Morgan	e059c5e648	Move get_threepid_validation_session into RegistrationWorkerStore	2019-09-06 13:10:11 +01:00
Erik Johnston	591d82f06b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/censor_redactions	2019-09-05 17:27:46 +01:00
Erik Johnston	3ff0422d2d	Make redaction retention period configurable	2019-09-05 17:16:45 +01:00
Jorik Schellekens	5ade977d08	Opentracing context cannot be none	2019-09-05 15:06:13 +01:00
Hubert Chathi	7d6c70fc7a	make black happy	2019-09-04 22:04:12 -04:00
Hubert Chathi	ac4746ac4b	allow uploading signatures of master key signed by devices	2019-09-04 22:03:41 -04:00
Hubert Chathi	4bb4544784	implement device signature uploading/fetching	2019-09-04 20:02:56 -04:00
Hubert Chathi	faf72a4c40	Merge branch 'develop' into cross-signing_keys	2019-09-04 19:12:29 -04:00
Erik Johnston	6e834e94fc	Fix and refactor room and user stats (#5971) ... Previously the stats were not being correctly populated.	2019-09-04 13:04:27 +01:00
Andrew Morgan	2a44782666	Remove double return statements (#5962) ... Remove all the "double return" statements which were a result of us removing all the instances of ``` defer.returnValue(...) return ``` statements when we switched to python3 fully.	2019-09-03 11:42:45 +01:00
Jorik Schellekens	a90d16dabc	Opentrace device lists (#5853) ... Trace device list changes.	2019-09-03 10:21:30 +01:00
Andrew Morgan	4548d1f87e	Remove unnecessary parentheses around return statements (#5931) ... Python will return a tuple whether there are parentheses around the returned values or not. I'm just sick of my editor complaining about this all over the place :)	2019-08-30 16:28:26 +01:00
Erik Johnston	a4bf72c30c	Censor redactions in DB after a month	2019-08-30 11:29:17 +01:00
Hubert Chathi	e3d3fbf63f	Merge branch 'uhoreg/e2e_cross-signing_merged' into cross-signing_keys	2019-08-28 17:36:46 -07:00
Hubert Chathi	72d296a7f3	Merge branch 'develop' into uhoreg/e2e_cross-signing_merged	2019-08-28 17:33:44 -07:00
Hubert Chathi	3b0b22cb05	use stream ID generator instead of timestamp	2019-08-28 17:17:21 -07:00
Will Hunt	c8fa620d7a	Merge pull request #5902 from matrix-org/hs/exempt-support-users-from-consent ... Exempt support users from consent	2019-08-28 16:31:40 +01:00
Richard van der Hoff	49ef8ec399	Fix a cache-invalidation bug for worker-based deployments (#5920) ... Some of the caches on worker processes were not being correctly invalidated when a room's state was changed in a way that did not affect the membership list of the room. We need to make sure we send out cache invalidations even when no memberships are changing.	2019-08-28 10:18:16 +01:00
reivilibre	322ccac33f	Allow schema deltas to be engine-specific (#5911) ... * Allow schema deltas to be engine-specific Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net> * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net> * Code style (Black) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-27 11:53:21 +01:00
reivilibre	1a7e6eb633	Add Admin API capability to set adminship of a user (#5878) ... Admin API: Set adminship of a user	2019-08-27 10:14:00 +01:00
Olivier Wilkinson (reivilibre)	d1e0b91083	Code style (Black) ... Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-27 09:39:11 +01:00
Olivier Wilkinson (reivilibre)	aefa76f5cd	Allow schema deltas to be engine-specific ... Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>	2019-08-27 09:14:00 +01:00
Half-Shot	ae38e0569f	Ignore consent for support users	2019-08-23 09:15:10 +01:00
Jorik Schellekens	8767b63a82	Propagate opentracing contexts through EDUs (#5852) ... Propagate opentracing contexts through EDUs Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-08-22 18:21:10 +01:00
Jorik Schellekens	9a6f2be572	Opentrace e2e keys (#5855) ... Add opentracing tags and logs for e2e keys	2019-08-22 11:28:12 +01:00
Richard van der Hoff	c9f11d09fc	Add missing index on users_in_public_rooms. (#5894)	2019-08-22 10:43:13 +01:00
Richard van der Hoff	119aa31b10	Servlet to purge old rooms (#5845)	2019-08-22 10:42:59 +01:00
Hubert Chathi	7c3abc6572	apply PR review suggestions	2019-08-21 13:19:35 -07:00
Richard van der Hoff	4dab867288	Drop some unused tables. (#5893) ... These tables are never used, so we may as well drop them.	2019-08-21 13:16:28 +01:00
Hubert Chathi	8a5f6ed130	Merge pull request #5857 from matrix-org/uhoreg/fix_e2e_room_keys_index ... add the version field to the index for e2e_room_keys	2019-08-16 17:45:50 -07:00
Hubert Chathi	c058aeb88d	update set_e2e_room_key to agree with fixed index	2019-08-14 18:02:58 -07:00
Hubert Chathi	b7f7cc7ace	add the version field to the index for e2e_room_keys	2019-08-14 17:14:40 -07:00
Erik Johnston	af187805b3	Merge pull request #5809 from matrix-org/erikj/handle_pusher_stop ... Handle pusher being deleted during processing.	2019-08-13 14:08:29 +01:00
Erik Johnston	fb3469f53a	Clarify docstring ... Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-08-13 10:17:23 +01:00
Erik Johnston	2546f32b90	Merge pull request #5826 from matrix-org/erikj/reduce_event_pauses ... Don't unnecessarily block notifying of new events.	2019-08-13 09:36:25 +01:00
Hubert Chathi	f63ba7a795	Cross-signing [1/4] -- hidden devices (#5759) ... * allow devices to be marked as "hidden" This is a prerequisite for cross-signing, as it allows us to create other things that live within the device namespace, so they can be used for signatures.	2019-08-12 15:14:37 -07:00
Richard van der Hoff	fb86217553	Merge pull request #5788 from matrix-org/rav/metaredactions ... Fix handling of redactions of redactions	2019-08-12 12:25:19 +01:00
Erik Johnston	bf4db42920	Don't unnecessarily block notifying of new events. ... When persisting events we calculate new stream orderings up front. Before we notify about an event all events with lower stream orderings must have finished being persisted. This PR moves the assignment of stream ordering till *after* calculated the new current state and split the batch of events into separate chunks for persistence. This means that if it takes a long time to calculate new current state then it will not block events in other rooms being notified about. This should help reduce some global pauses in the events stream which can last for tens of seconds (if not longer), caused by some particularly expensive state resolutions.	2019-08-06 13:32:02 +01:00
Hubert Chathi	fac1cdc562	make changes from PR review	2019-08-01 21:51:19 -04:00
Hubert Chathi	336c546d6a	Merge branch 'cross-signing_hidden' into cross-signing_keys	2019-08-01 16:31:40 -04:00
Andrew Morgan	5d018d23f0	Have ClientReaderSlavedStore inherit RegistrationStore (#5806) ... Fixes #5803	2019-08-01 13:54:56 +01:00
Erik Johnston	d02e41dcb2	Handle pusher being deleted during processing. ... Instead of throwing a StoreError lets break out of processing loop and mark the pusher as stopped.	2019-08-01 13:44:12 +01:00
Hubert Chathi	d78a4fe156	don't need to return the hidden column any more	2019-08-01 02:16:09 -04:00
Hubert Chathi	73b26f827c	really fix queries to work with Postgres ... (by going back to not using SQL directly)	2019-07-31 18:37:05 -04:00
Hubert Chathi	430ea08186	PostgreSQL, Y U no like?	2019-07-31 15:38:11 -04:00
Hubert Chathi	185188be03	remove extra SQL query param	2019-07-31 15:18:15 -04:00
Hubert Chathi	c1f0a5636c	Merge branch 'develop' into cross-signing_hidden	2019-07-30 23:33:06 -04:00
Hubert Chathi	bc95890822	Merge branch 'cross-signing_hidden' into cross-signing_keys	2019-07-30 23:21:26 -04:00
Hubert Chathi	781ade836b	apply changes from PR review	2019-07-30 23:09:50 -04:00
Erik Johnston	123c04daa7	Don't recreate column	2019-07-30 16:42:48 +01:00
Erik Johnston	15056ca208	Fix current_state_events membership background update. ... Turns out not all rooms are in `rooms`, so lets fetch the room list from `current_state_events`. We move the delta file to force it to be run again.	2019-07-30 14:51:41 +01:00
Richard van der Hoff	4e97eb89e5	Handle loops in redaction events	2019-07-30 00:00:34 +01:00
Richard van der Hoff	448bcfd0f9	recursively fetch redactions	2019-07-30 00:00:34 +01:00
Richard van der Hoff	e6a6c4fbab	split _get_events_from_db out of _enqueue_events	2019-07-29 23:15:15 +01:00
Richard van der Hoff	c9964ba600	Return dicts from _fetch_event_list	2019-07-29 23:15:15 +01:00
Erik Johnston	7c8c3b8437	Merge pull request #5774 from matrix-org/erikj/fix_rejected_membership ... Fix room summary when rejected events are in state	2019-07-29 17:15:15 +01:00
Erik Johnston	3e013b7c8e	Merge pull request #5752 from matrix-org/erikj/forgotten_user ... Remove some more joins on room_memberships	2019-07-29 17:15:01 +01:00
Erik Johnston	df3a5db629	Expand comment	2019-07-29 16:40:25 +01:00
Erik Johnston	105e7f6ed3	Remove lost comment	2019-07-29 16:09:48 +01:00
Erik Johnston	45df38e61b	Fix current_state bg update to work on old SQLite	2019-07-29 13:04:10 +01:00
Richard van der Hoff	d74595e2ca	Merge branch 'master' into develop	2019-07-26 12:39:33 +01:00
Erik Johnston	14c24c9037	Fix room summary when rejected events are in state ... Annoyingly, `current_state_events` table can include rejected events, in which case the membership column will be null. To work around this lets just always filter out null membership for now.	2019-07-26 10:11:36 +01:00
Richard van der Hoff	0f2ecb961e	Fix DoS when there is a cycle in redaction events ... Make sure that synapse doesn't explode when a redaction redacts itself, or there is a larger cycle.	2019-07-26 06:36:48 +00:00
Richard van der Hoff	26d742fed6	Merge pull request #5767 from matrix-org/rav/redactions/cross_room_id ... log when a redaction attempts to redact an event in a different room	2019-07-25 18:49:56 +01:00
Richard van der Hoff	b1605cdd23	log when a redaction attempts to redact an event in a different room	2019-07-25 18:26:20 +01:00
Hubert Chathi	c659b9f94f	allow uploading keys for cross-signing	2019-07-25 11:08:24 -04:00
Hubert Chathi	d1c7c2a98a	allow devices to be marked as "hidden" ... This is a prerequisite for cross-signing, as it allows us to create other things that live within the device namespace, so they can be used for signatures.	2019-07-24 23:21:52 -04:00
Richard van der Hoff	f30a71a67b	Stop trying to fetch events with event_id=None. (#5753) ... `None` is not a valid event id, so queuing up a database fetch for it seems like a silly thing to do. I considered making `get_event` return `None` if `event_id is None`, but then its interaction with `allow_none` seemed uninituitive, and strong typing ftw.	2019-07-24 13:16:18 +01:00
Erik Johnston	0c4a99607e	Remove join when calculating room summaries.	2019-07-24 11:49:15 +01:00
Erik Johnston	62921fb53e	Remove join on room_memberships when fetching rooms for user.	2019-07-24 11:45:58 +01:00
Erik Johnston	32768e96d4	Add function to get all forgotten rooms for user ... This will allow us to efficiently filter out rooms that have been forgotten in other queries without having to join against the `room_memberships` table.	2019-07-24 11:44:23 +01:00
Amber Brown	4806651744	Replace returnValue with return (#5736)	2019-07-23 23:00:55 +10:00
Erik Johnston	b2a629ef49	Speed up current state background update. ... Turns out that storing huge JSON arrays in the progress JSON isn't something that postgres particularly likes.	2019-07-23 10:01:30 +01:00
Jason Robinson	66f5ff72fd	Add user_type to returned fields in admin API user list endpoints ... Mostly user type will be empty (normal user) but there is also the "support" user type. Signed-off-by: Jason Robinson <jasonr@matrix.org>	2019-07-22 15:29:18 +03:00
Erik Johnston	7704873cb8	Merge pull request #5720 from matrix-org/erikj/transactions_upsert ... Use upsert when updating destination retry interval	2019-07-19 16:51:16 +01:00
Erik Johnston	d7bd9651bc	Merge pull request #5713 from matrix-org/erikj/use_cache_for_filtered_state ... Delegate to cached version when using get_filtered_current_state_ids	2019-07-19 16:30:49 +01:00
Erik Johnston	5c07c97c09	Merge pull request #5706 from matrix-org/erikj/add_memberships_to_current_state ... Add membership column to current_state_events table	2019-07-19 16:30:33 +01:00
Erik Johnston	2410335507	Use upsert when updating destination retry interval	2019-07-19 13:40:24 +01:00
Erik Johnston	bd2e1a2aa8	LoggingTransaction accepts None for callback lists. ... Its a bit disingenuousto give LoggingTransaction lists to append callbacks to if we're not going to run the callbacks.	2019-07-19 13:36:04 +01:00
Erik Johnston	ebc5ed1296	Update comment for new column	2019-07-19 13:29:02 +01:00
Erik Johnston	10523241d8	Delegate to cached version when using get_filtered_current_state_ids ... In the case where it gets called with `StateFilter.all()`	2019-07-18 15:17:39 +01:00
Erik Johnston	8e1ada9e6f	Use the current_state_events.membership column	2019-07-18 14:16:01 +01:00
Erik Johnston	059d8c1a4e	Track if current_state_events.membership is up to date	2019-07-18 14:16:01 +01:00
Erik Johnston	c618a5d348	Add background update for current_state_events.membership column	2019-07-18 14:16:01 +01:00
Erik Johnston	6de09e07a6	Add membership column to current_state_events table. ... It turns out that doing a join is surprisingly expensive for the DB to do when room_membership table is larger than the disk cache.	2019-07-18 14:15:57 +01:00
Richard van der Hoff	9c70a02a9c	Ignore redactions of m.room.create events (#5701)	2019-07-17 19:08:02 +01:00
Richard van der Hoff	2091c91fde	More refactoring in get_events_as_list (#5707) ... We can now use `_get_events_from_cache_or_db` rather than going right back to the database, which means that (a) we can benefit from caching, and (b) it opens the way forward to more extensive checks on the original event. We now always require the original event to exist before we will serve up a redaction.	2019-07-17 17:34:13 +01:00
Richard van der Hoff	375162b3c3	Fix redaction authentication (#5700) ... Ensures that redactions are correctly authenticated for recent room versions. There are a few things going on here: * `_fetch_event_rows` is updated to return a dict rather than a list of rows. * Rather than returning multiple copies of an event which was redacted multiple times, it returns the redactions as a list within the dict. * It also returns the actual rejection reason, rather than merely the fact that it was rejected, so that we don't have to query the table again in `_get_event_from_row`. * The redaction handling is factored out of `_get_event_from_row`, and now checks if any of the redactions are valid.	2019-07-17 16:52:02 +01:00
Richard van der Hoff	65c5592b8e	Refactor get_events_as_list (#5699) ... A couple of changes here: * get rid of a redundant `allow_rejected` condition - we should already have filtered out any rejected events before we get to that point in the code, and the redundancy is confusing. Instead, let's stick in an assertion just to make double-sure we aren't leaking rejected events by mistake. * factor out a `_get_events_from_cache_or_db` method, which is going to be important for a forthcoming fix to redactions.	2019-07-17 16:49:19 +01:00
Erik Johnston	d86321300a	Merge pull request #5589 from matrix-org/erikj/admin_exfiltrate_data ... Add basic function to get all data for a user out of synapse	2019-07-15 10:04:02 +01:00
Richard van der Hoff	5f158ec039	Implement access token expiry (#5660) ... Record how long an access token is valid for, and raise a soft-logout once it expires.	2019-07-12 17:26:02 +01:00
Erik Johnston	db0a50bc40	Fixup docstrings	2019-07-12 16:59:59 +01:00
Andrew Morgan	24aa0e0a5b	fix typo: backgroud -> background	2019-07-12 15:29:40 +01:00
Andrew Morgan	78a1cd36b5	small typo fix (#5655)	2019-07-11 13:33:23 +01:00
Richard van der Hoff	953dbb7980	Remove access-token support from RegistrationStore.register (#5642) ... The 'token' param is no longer used anywhere except the tests, so let's kill that off too.	2019-07-10 16:26:49 +01:00
Andrew Morgan	d88421ab03	Include the original event in /relations (#5626) ... When asking for the relations of an event, include the original event in the response. This will mostly be used for efficiently showing edit history, but could be useful in other circumstances.	2019-07-09 13:43:08 +01:00
Richard van der Hoff	80cc82a445	Remove support for invite_3pid_guest. (#5625) ... This has never been documented, and I'm not sure it's ever been used outside sytest. It's quite a lot of poorly-maintained code, so I'd like to get rid of it. For now I haven't removed the database table; I suggest we leave that for a future clearout.	2019-07-05 16:47:58 +01:00
Erik Johnston	7f0d8e4288	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/admin_exfiltrate_data	2019-07-05 14:08:21 +01:00
Erik Johnston	9ccea16d45	Assume key existence. Update docstrings	2019-07-05 14:07:56 +01:00
Richard van der Hoff	a6a776f3d8	remove dead transaction persist code (#5622) ... this hasn't done anything for years	2019-07-05 12:59:42 +01:00
Amber Brown	463b072b12	Move logging utilities out of the side drawer of util/ and into logging/ (#5606)	2019-07-04 00:07:04 +10:00
Richard van der Hoff	91753cae59	Fix a number of "Starting txn from sentinel context" warnings (#5605) ... Fixes #5602, #5603	2019-07-03 09:31:27 +01:00
Erik Johnston	8ee69f299c	Add basic function to get all data for a user out of synapse	2019-07-02 12:09:04 +01:00
Erik Johnston	729f5a4fb6	Review comments	2019-06-27 16:06:23 +01:00
Erik Johnston	f335e77d53	Use batch_iter and correct docstring	2019-06-26 19:39:46 +01:00
Erik Johnston	806a06daf2	Rename get_users_whose_devices_changed	2019-06-26 19:39:19 +01:00
Erik Johnston	a2f6d31a63	Refactor get_user_ids_changed to pull less from DB ... When a client asks for users whose devices have changed since a token we used to pull *all* users from the database since the token, which could easily be thousands of rows for old tokens. This PR changes this to only check for changes for users the client is actually interested in. Fixes #5553	2019-06-26 12:03:44 +01:00
Richard van der Hoff	e59a8cd2e5	Merge pull request #5499 from matrix-org/rav/cleanup_metrics ... Cleanups and sanity-checking in cpu and db metrics	2019-06-24 17:12:54 +01:00
Richard van der Hoff	1793de6c6d	black	2019-06-24 11:16:13 +01:00
Richard van der Hoff	5097aee740	Merge branch 'develop' into rav/cleanup_metrics	2019-06-24 10:00:13 +01:00
Richard van der Hoff	c753c098dd	Merge pull request #5498 from matrix-org/rav/fix_clock_reversal ... Use monotonic clock where possible for metrics	2019-06-24 09:55:12 +01:00
Richard van der Hoff	0e8b35f7b0	Fix "Unexpected entry in 'full_schemas'" log warning (#5509) ... There is a README.txt which always sets off this warning, which is a bit alarming when you first start synapse. I don't think we need to warn about this.	2019-06-21 15:11:57 +01:00
Erik Johnston	f8bd30af2a	Black	2019-06-20 13:10:06 +01:00
Erik Johnston	45f28a9d2f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/histogram_extremities	2019-06-20 11:59:14 +01:00
Amber Brown	32e7c9e7f2	Run Black. (#5482)	2019-06-20 19:32:02 +10:00
Richard van der Hoff	f682af052d	Simplify PerformanceCounters.update interface ... we already have the duration for the update, so may as well use it rather than passing extra params around and recalculating it.	2019-06-19 21:18:38 +01:00
Richard van der Hoff	68128d5626	Remove unused _get_event_counters ... This has been redundant since cdb3757942.	2019-06-19 21:14:25 +01:00
Richard van der Hoff	15bf32e062	Use monotonic clock where possible for metrics ... Fixes intermittent errors observed on Apple hardware which were caused by time.clock() appearing to go backwards when called from different threads. Also fixes a bug where database activity times were logged as 1/1000 of their correct ratio due to confusion between milliseconds and seconds.	2019-06-19 21:09:43 +01:00
Erik Johnston	e0be8d7016	Merge pull request #5480 from matrix-org/erikj/extremities_dummy_events ... Add experimental option to reduce extremities.	2019-06-19 13:19:18 +01:00
Erik Johnston	65787b0f7c	Add descriptions and remove redundant set(..)	2019-06-19 11:49:39 +01:00
Brendan Abolivier	092e36457a	Fix typo in deactivation background job	2019-06-19 11:09:41 +01:00
Erik Johnston	b42f90470f	Add experimental option to reduce extremities. ... Adds new config option `cleanup_extremities_with_dummy_events` which periodically sends dummy events to rooms with more than 10 extremities. THIS IS REALLY EXPERIMENTAL.	2019-06-18 15:02:18 +01:00
Erik Johnston	499d4a32cd	Add metrics for len of new extremities persisted. ... Of new events being persisted add metrics for total size of forward extremities and number of unchanged, "stale" extremities.	2019-06-17 17:02:48 +01:00
Amber Brown	eba7caf09f	Remove Postgres 9.4 support (#5448)	2019-06-18 00:59:00 +10:00
Erik Johnston	9ca4ae7131	Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitive ... Prometheus histograms are cumalative	2019-06-14 18:21:42 +01:00
Brendan Abolivier	e0b77b004d	Fix background job for deactivated flag	2019-06-14 16:00:45 +01:00
Brendan Abolivier	6d56a694f4	Don't send renewal emails to deactivated users	2019-06-14 15:05:56 +01:00
Erik Johnston	3ed595e327	Prometheus histograms are cumalative	2019-06-14 14:07:32 +01:00
Brendan Abolivier	d0530382ee	Track deactivated accounts in the database (#5378)	2019-06-14 13:18:24 +01:00
Amber Brown	6312d6cc7c	Expose statistics on extrems to prometheus (#5384)	2019-06-13 22:40:52 +10:00
Erik Johnston	5bc9484537	Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into develop	2019-06-11 10:37:43 +01:00
Neil Johnson	94dac0f3e5	add monthly active users to phonehome stats (#5252) ... * add monthly active users to phonehome stats	2019-06-10 23:33:59 +01:00
Erik Johnston	49e01e5710	Fix defaults on checking threepids	2019-06-10 23:09:31 +01:00
Erik Johnston	c413540fb9	Fix bug sending federation transactions with lots of EDUs ... If we try and send a transaction with lots of EDUs and we run out of space, we call get_new_device_msgs_for_remote with a limit of 0, which then failed.	2019-06-10 16:22:45 +01:00
Erik Johnston	43badd2cd4	Fix key verification when key stored with null valid_until_ms ... Some keys are stored in the synapse database with a null valid_until_ms which caused an exception to be thrown when using that key. We fix this by treating nulls as zeroes, i.e. they keys will match verification requests with a minimum_valid_until_ms of zero (i.e. don't validate ts) but will not match requests with a non-zero minimum_valid_until_ms. Fixes #5391.	2019-06-10 15:37:10 +01:00
Andrew Morgan	2d1d7b7e6f	Prevent multiple device list updates from breaking a batch send (#5156) ... fixes #5153	2019-06-06 23:54:00 +01:00
Erik Johnston	1b8cb64393	Merge pull request #5320 from matrix-org/hawkowl/full-schema-v1 ... Make a full SQL schema	2019-06-06 18:35:38 +01:00
Amber H. Brown	ed872db8df	fix maybe	2019-06-07 02:53:47 +10:00
Andrew Morgan	3719680ee4	Add ability to perform password reset via email without trusting the identity server (#5377) ... Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option. This PR is a culmination of 3 smaller PRs which have each been separately reviewed: * #5308 * #5345 * #5368	2019-06-06 17:34:07 +01:00
Amber H. Brown	837e32ef55	just user dir?	2019-06-07 01:49:25 +10:00
Amber H. Brown	3b6645d3bf	remove background updates that arent needed	2019-06-07 01:20:58 +10:00
Erik Johnston	89d3d7b2c0	Merge pull request #5221 from matrix-org/erikj/fix_worker_sytest ... Fix get_max_topological_token to never return None	2019-06-06 13:54:46 +01:00
Erik Johnston	7f08a3523a	Better words	2019-06-06 11:09:38 +01:00
Amber Brown	6362e3af14	add more comments	2019-06-06 04:20:35 +10:00
Erik Johnston	75538813fc	Fix background updates to handle redactions/rejections (#5352) ... * Fix background updates to handle redactions/rejections In background updates based on current state delta stream we need to handle that we may not have all the events (or at least that `get_events` may raise an exception).	2019-06-06 00:45:46 +10:00
Erik Johnston	0a56966f7d	Fix	2019-06-03 17:42:52 +01:00
Erik Johnston	fa4b54aca5	Ignore room state with null bytes in for room stats	2019-06-03 17:06:54 +01:00
Amber Brown	2198b7ce2a	add stuff in bg updates	2019-06-04 01:06:00 +10:00
Amber Brown	4e75c5e02a	WHY IS THIS CALLED A SLIGHTLY DIFFERENT THING	2019-06-03 22:42:12 +10:00
Amber Brown	be452fc9ac	more fix	2019-06-03 22:24:23 +10:00
Amber Brown	7f81b967ca	fix schemas	2019-06-03 22:23:40 +10:00
Amber Brown	dc72b90cd6	full schema	2019-06-03 22:03:28 +10:00
Amber Brown	37057d5d60	prepare	2019-06-03 22:02:47 +10:00
Erik Johnston	58cce39f3a	Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta ... Allow configuring a range for the account validity startup job	2019-05-31 12:11:56 +01:00
Brendan Abolivier	4d794dae21	Move delta from +10% to -10%	2019-05-31 11:09:39 +01:00
Erik Johnston	31d44ec4bd	Merge pull request #5294 from matrix-org/erikj/speed_up_room_stats ... Speed up room stats background update	2019-05-31 10:48:51 +01:00
Erik Johnston	5037326d66	Add indices. Remove room_ids accidentally added ... We have to do this by re-inserting a background update and recreating tables, as the tables only get created during a background update and will later be deleted. We also make sure that we remove any entries that should have been removed but weren't due to a race that has been fixed in a previous commit.	2019-05-31 10:26:56 +01:00
Brendan Abolivier	847b9dcd1c	Make max_delta equal to period * 10%	2019-05-31 09:54:46 +01:00
Erik Johnston	5ac75fc9a2	Join against events to use its room_id index	2019-05-30 15:26:55 +01:00
Erik Johnston	e2c46ed851	Move deletion from table inside txn	2019-05-30 15:26:38 +01:00
Erik Johnston	04710cc2d7	Fetch membership counts all at once	2019-05-30 15:25:41 +01:00
Erik Johnston	54d50fbfdf	Get events all at once	2019-05-30 15:15:13 +01:00
Erik Johnston	6cdfb0207e	Add index to temp table	2019-05-30 15:02:31 +01:00
Erik Johnston	e9e5d3392d	Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities ... Add DB bg update to cleanup extremities.	2019-05-30 14:28:26 +01:00
Erik Johnston	cb967e2346	Update synapse/storage/events_bg_updates.py ... Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-05-30 14:06:42 +01:00
Erik Johnston	468bd090ff	Rename constant	2019-05-30 11:24:42 +01:00
Erik Johnston	5c1ece0ffc	Move event background updates to a separate file	2019-05-30 11:22:59 +01:00
Erik Johnston	640fcbb07f	Fixup comments and logging	2019-05-30 10:55:55 +01:00
Amber Brown	46c8f7a517	Implement the SHHS complexity API (#5216)	2019-05-30 01:47:16 +10:00
Erik Johnston	7e8e683754	Log actual number of entries deleted	2019-05-29 15:11:28 +01:00
Erik Johnston	d79c9994f4	Add DB bg update to cleanup extremities. ... Due to #5269 we may have extremities in our DB that we shouldn't have, so lets add a cleanup task such to remove those.	2019-05-29 15:11:26 +01:00
Erik Johnston	58c8ed5b0d	Correctly filter out extremities with soft failed prevs (#5274) ... When we receive a soft failed event we, correctly, *do not* update the forward extremity table with the event. However, if we later receive an event that references the soft failed event we then need to remove the soft failed events prev events from the forward extremities table, otherwise we just build up forward extremities. Fixes #5269	2019-05-29 11:56:24 +01:00
Brendan Abolivier	52839886d6	Allow configuring a range for the account validity startup job ... When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal. In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.	2019-05-28 16:52:45 +01:00
Brendan Abolivier	a97d4e218a	Merge pull request #5268 from matrix-org/babolivier/account_validity_fix_schema ... Fix schema update for account validity	2019-05-28 10:30:07 +01:00
Brendan Abolivier	ba17de7fbc	Fix schema update for account validity	2019-05-28 10:11:38 +01:00
Travis Ralston	bc4b2ecf70	Fix logging for room stats background update	2019-05-25 12:02:48 -06:00
Richard van der Hoff	b75537beaf	Store key validity time in the storage layer ... This is a first step to checking that the key is valid at the required moment. The idea here is that, rather than passing VerifyKey objects in and out of the storage layer, we instead pass FetchKeyResult objects, which simply wrap the VerifyKey and add a valid_until_ts field.	2019-05-23 11:52:22 +01:00
Richard van der Hoff	2e052110ee	Rewrite store_server_verify_key to store several keys at once (#5234) ... Storing server keys hammered the database a bit. This replaces the implementation which stored a single key, with one which can do many updates at once.	2019-05-23 11:45:39 +01:00
Richard van der Hoff	85d1e03b9d	Simplifications and comments in do_auth (#5227) ... I was staring at this function trying to figure out wtf it was actually doing. This is (hopefully) a non-functional refactor which makes it a bit clearer.	2019-05-23 11:17:42 +01:00
Amber Brown	4a30e4acb4	Room Statistics (#4338)	2019-05-21 11:36:50 -05:00
Erik Johnston	f4c80d70f8	Merge pull request #5203 from matrix-org/erikj/aggregate_by_sender ... Only count aggregations from distinct senders	2019-05-21 17:10:48 +01:00
Erik Johnston	9526aa96a6	Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions ... Block attempts to annotate the same event twice	2019-05-21 17:08:14 +01:00
Richard van der Hoff	c4aef549ad	Exclude soft-failed events from fwd-extremity candidates. (#5146) ... When considering the candidates to be forward-extremities, we must exclude soft failures. Hopefully fixes #5090.	2019-05-21 16:10:54 +01:00
Erik Johnston	7b0e804a4a	Fix get_max_topological_token to never return None	2019-05-21 15:21:43 +01:00
Brendan Abolivier	6a5a70edf0	Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date ... Add startup background job for account validity	2019-05-21 14:55:15 +01:00
Brendan Abolivier	5ceee46c6b	Do the select and insert in a single transaction	2019-05-21 13:38:51 +01:00
Erik Johnston	c7ec06e8a6	Block attempts to annotate the same event twice	2019-05-20 17:39:05 +01:00
Erik Johnston	1dff859d6a	Rename relation types to match MSC	2019-05-20 14:31:19 +01:00
Erik Johnston	57ba3451b6	Merge pull request #5209 from matrix-org/erikj/reactions_base ... Land basic reaction and edit support.	2019-05-20 14:06:40 +01:00
Erik Johnston	935af0da38	Correctly update aggregation counts after redaction	2019-05-20 12:09:27 +01:00
Brendan Abolivier	ad5b4074e1	Add startup background job for account validity ... If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.	2019-05-17 19:37:31 +01:00
Erik Johnston	b63cc325a9	Only count aggregations from distinct senders ... As a user isn't allowed to send a single emoji more than once.	2019-05-17 18:03:10 +01:00
Erik Johnston	8dd9cca8ea	Spelling and clarifications	2019-05-17 16:40:51 +01:00
Erik Johnston	5dbff34509	Fixup bsaed on review comments	2019-05-17 15:48:04 +01:00
Erik Johnston	d46aab3fa8	Add basic editing support	2019-05-16 16:54:45 +01:00
Erik Johnston	895179a4dc	Update docstring	2019-05-16 16:41:05 +01:00
Erik Johnston	7a7eba8302	Move parsing of tokens out of storage layer	2019-05-16 14:26:23 +01:00
Erik Johnston	2c662ddde4	Indirect tuple conversion	2019-05-16 14:21:39 +01:00
Erik Johnston	b5c62c6b26	Fix relations in worker mode	2019-05-16 10:38:13 +01:00
Erik Johnston	33453419b0	Add cache to relations	2019-05-16 10:02:14 +01:00
Erik Johnston	a0603523d2	Add aggregations API	2019-05-16 09:37:20 +01:00
Erik Johnston	5be34fc3e3	Actually check for None rather falsey	2019-05-15 17:30:23 +01:00
Erik Johnston	b50641e357	Add simple pagination API	2019-05-15 13:36:51 +01:00
Erik Johnston	efe3c7977a	Add simple send_relation API and track in DB	2019-05-15 13:36:51 +01:00
Erik Johnston	54d77107c1	Make generating SQL bounds for pagination generic ... This will allow us to reuse the same structure when we paginate e.g. relations	2019-05-15 11:30:05 +01:00
Erik Johnston	52ddc6c0ed	Update docstring with correct type ... Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-05-15 09:52:15 +01:00
Erik Johnston	4fb44fb5b9	Expose DataStore._get_events as get_events_as_list ... This is in preparation for reaction work which requires it.	2019-05-14 13:37:44 +01:00
Quentin Dufour	11ea16777f	Limit the number of EDUs in transactions to 100 as expected by receiver (#5138) ... Fixes #3951.	2019-05-09 11:01:41 +01:00
Richard van der Hoff	bd0d45ca69	Fix infinite loop in presence handler ... Fixes #5102	2019-04-26 11:14:49 +01:00
Erik Johnston	ca90336a69	Merge branch 'develop' of github.com:matrix-org/synapse into babolivier/account_expiration	2019-04-17 19:44:40 +01:00
Brendan Abolivier	eaf41a943b	Add management endpoints for account validity	2019-04-17 19:34:45 +01:00
Brendan Abolivier	91934025b9	Merge pull request #5047 from matrix-org/babolivier/account_expiration ... Send out emails with links to extend an account's validity period	2019-04-17 14:57:39 +01:00
Brendan Abolivier	20f0617e87	Send out emails with links to extend an account's validity period	2019-04-17 14:42:20 +01:00
Erik Johnston	e446921def	Merge pull request #5033 from matrix-org/erikj/fix_schema_delta ... Fix schema upgrade when dropping tables	2019-04-10 10:22:35 +01:00
Brendan Abolivier	bfc8fdf1fc	Merge pull request #5027 from matrix-org/babolivier/account_expiration ... Add time-based account expiration	2019-04-09 17:02:41 +01:00
Brendan Abolivier	747aa9f8ca	Add account expiration feature	2019-04-09 16:46:04 +01:00
Erik Johnston	50d2a3059d	Fix schema upgrade when dropping tables ... We need to drop tables in the correct order due to foreign table constraints (on `application_services`), otherwise the DROP TABLE command will fail. Introduced in #4992.	2019-04-09 14:39:18 +01:00
Richard van der Hoff	18b69be00f	Rewrite Datastore.get_server_verify_keys ... Rewrite this so that it doesn't hammer the database.	2019-04-09 00:00:10 +01:00
Richard van der Hoff	3352baac4b	Remove unused server_tls_certificates functions (#5028) ... These have been unused since #4120, and with the demise of perspectives, it is unlikely that they will ever be used again.	2019-04-08 21:50:18 +01:00
Neil Johnson	2d951686a7	drop tables listed in #1830 (#4992) ... Tables dropped: * application_services, * application_services_regex, * transaction_id_to_pdu, * stats_reporting * current_state_resets * event_content_hashes * event_destinations * event_edge_hashes * event_signatures * feedback * room_hosts * state_forward_extremities	2019-04-08 15:37:26 +01:00
Richard van der Hoff	7fc1e17f4c	Merge pull request #5001 from matrix-org/rav/keyring_cleanups ... Cleanups in the Keyring	2019-04-08 12:47:09 +01:00
Richard van der Hoff	b43d9a920b	Fix docstring on get_server_keys_json	2019-04-04 18:54:03 +01:00
Amber Brown	a33a5abc4c	Clean up the database pagination code (#5007) ... * rewrite & simplify * changelog * cleanup potential sql injection	2019-04-05 00:21:16 +11:00
Erik Johnston	4a2e13631d	Add functions to delete a group	2019-04-03 16:29:52 +01:00
Andrew Morgan	4a4d5c4fd6	Fix grammar and document get_current_users_in_room (#4998)	2019-04-03 14:32:20 +01:00
Neil Johnson	e8419554ff	Remove presence lists (#4989) ... Remove presence list support as per MSC 1819	2019-04-03 11:11:15 +01:00
Erik Johnston	8f549c1177	Merge pull request #4982 from matrix-org/erikj/msc1915 ... Implement MSC1915 - 3PID unbind APIs	2019-04-03 11:07:09 +01:00
Amber Brown	7efd1d87c2	Run black on the rest of the storage module (#4996)	2019-04-03 10:07:29 +01:00
Erik Johnston	5bec8d660d	Make starting pushers faster during start up ... We start all pushers on start up and immediately start a background process to fetch push to send. This makes start up incredibly painful when dealing with many pushers. Instead, let's do a quick fast DB check to see if there *may* be push to send and only start the background processes for those pushers. We also stagger starting up and doing those checks so that we don't try and handle all pushers at once.	2019-04-02 16:59:13 +01:00
Richard van der Hoff	297bf2547e	Fix sync bug when accepting invites (#4956) ... Hopefully this time we really will fix #4422. We need to make sure that the cache on `get_rooms_for_user_with_stream_ordering` is invalidated *before* the SyncHandler is notified for the new events, and we can now do so reliably via the `events` stream.	2019-04-02 12:42:39 +01:00
Erik Johnston	c75e2017f1	Fixup docstrings	2019-04-02 11:20:06 +01:00
Erik Johnston	3715c124b3	Grandfather in existing user threepids ... We assume, as we did before, that users bound their threepid to one of the trusted identity servers. So we simply fill the new table with all threepids in `user_threepids` joined with the trusted identity servers.	2019-04-01 15:25:19 +01:00
Erik Johnston	1666c0696a	Track IS used to bind 3PIDs ... This will then be used to know which IS to default to when unbinding the threepid.	2019-04-01 15:23:01 +01:00
Richard van der Hoff	54a87a7b08	Collect room-version variations into one place (#4969) ... Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.	2019-04-01 10:24:38 +01:00
Richard van der Hoff	902cdc63b6	Merge pull request #4955 from matrix-org/rav/merge_state_into_events ... Combine the CurrentStateDeltaStream into the EventStream	2019-03-28 18:32:13 +00:00
Amber Brown	2e060774ad	Run black on some storage modules that the stats branch touches (#4959)	2019-03-29 00:37:16 +11:00
Richard van der Hoff	4b91c313a9	Combine the CurrentStateDeltaStream into the EventStream	2019-03-27 22:07:05 +00:00
Amber Brown	903f04c21f	Use the state event amount for userdir import batching, not room count (#4944)	2019-03-27 02:49:28 +11:00
Richard van der Hoff	9bde730ef8	Fix bug where read-receipts lost their timestamps (#4927) ... Make sure that they are sent correctly over the replication stream. Fixes: #4898	2019-03-25 16:38:05 +00:00
Amber Brown	ac396a0d32	Refactor out state delta handling into its own class (#4917)	2019-03-25 20:37:08 +11:00
Erik Johnston	3ecec5ede2	Fix upsert	2019-03-21 10:21:15 +00:00
Erik Johnston	8d8834d3e7	comment block_room	2019-03-20 17:49:56 +00:00
Erik Johnston	67d618e111	Allow blocking a room multiple times	2019-03-20 16:50:05 +00:00
Amber Brown	4d53017432	Batching in the user directory import (#4900)	2019-03-21 03:06:36 +11:00
Erik Johnston	e9eeca1314	Fix user directory background update (#4887)	2019-03-19 22:13:53 +11:00
Amber Brown	282c97327f	Migrate the user directory initial population to a background task (#4864)	2019-03-19 04:50:24 +11:00
Erik Johnston	9073cfc8bd	Merge pull request #4846 from matrix-org/hawkowl/userdir-search ... Improve searching in the userdir	2019-03-14 13:54:15 +00:00
Andrew Morgan	d42c81d724	Transfer local user's push rules on room upgrade (#4838) ... Transfer push rules (notifications) on room upgrade	2019-03-12 14:42:53 +00:00
Amber Brown	c0332d095f	fixup	2019-03-13 01:30:54 +11:00
Amber Brown	797b6a63fc	fixup	2019-03-13 01:17:51 +11:00
Amber Brown	81d9d1bee6	fixup	2019-03-12 22:28:48 +11:00
Amber Brown	6f5890b2fa	fixup	2019-03-12 22:27:56 +11:00
Amber Brown	10480c4348	fixup	2019-03-12 21:47:14 +11:00
Amber Brown	8b618041ef	fixup	2019-03-12 18:06:28 +11:00
Amber Brown	c980c7e31f	use the old method	2019-03-12 17:51:14 +11:00
Amber Brown	78a6b950b3	fix	2019-03-12 00:50:28 +11:00
Amber Brown	1b77bd69fb	pep8	2019-03-12 00:39:12 +11:00
Amber Brown	5ba8ceab4c	fixes	2019-03-12 00:35:31 +11:00
Amber Brown	30a8deeb68	Merge remote-tracking branch 'origin/develop' into hawkowl/userdir-search	2019-03-11 21:12:30 +11:00
Amber Brown	4abc988c6a	initial	2019-03-11 21:11:36 +11:00
Erik Johnston	fe6c12e6cd	Add comment to schema	2019-03-08 16:38:23 +00:00
Erik Johnston	5536ddba75	Make prev_state field optional ... The `prev_state` field on events is not specced and so synapse shouldn't explode if an event is missing the field. Fixes #4787	2019-03-08 15:05:32 +00:00
Erik Johnston	8c4896668f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/soft_fail_impl	2019-03-08 11:44:20 +00:00
Amber Brown	f6135d06cf	Rewrite userdir to be faster (#4537)	2019-03-07 01:22:53 -08:00
Erik Johnston	6d13bdec91	Add docstrings from matrix-org-hotfixes	2019-03-06 17:24:26 +00:00
Erik Johnston	a9de04be72	Implement soft fail	2019-03-06 16:22:16 +00:00
Erik Johnston	b050a10871	Merge pull request #4699 from matrix-org/erikj/stop_fed_not_in_room ... Stop backpaginating when events not visible	2019-03-05 09:32:33 +00:00
Erik Johnston	c3c542bb4a	Merge pull request #4796 from matrix-org/erikj/factor_out_e2e_keys ... Allow /keys/{changes,query} API to run on worker	2019-03-05 09:06:25 +00:00
Richard van der Hoff	2db49ea476	Add some debug about processing read receipts. ... I'm hoping to establish which rooms are having lots of RRs sent for them, and how old the events are when they are sent.	2019-03-04 18:19:40 +00:00
Erik Johnston	a84b8d56c2	Fixup slave stores	2019-03-04 18:04:57 +00:00
Erik Johnston	8b63fe4c26	s/get_forward_events/get_successor_events/	2019-03-04 11:56:03 +00:00
Erik Johnston	fbc047f2a5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/stop_fed_not_in_room	2019-03-04 11:54:58 +00:00
Erik Johnston	54f9ce11a7	Move /account/3pid to client_reader	2019-02-27 14:26:08 +00:00
Erik Johnston	6bb1c028f1	Limit cache invalidation replication line length (#4748)	2019-02-27 10:28:37 +00:00
Richard van der Hoff	f191be822b	Add database version to phonehome stats. (#4753)	2019-02-27 10:21:49 +00:00
Erik Johnston	80467bbac3	Fix state cache invalidation on workers	2019-02-22 14:38:14 +00:00
Erik Johnston	71b625d808	Stop backpaginating when events not visible	2019-02-20 18:14:12 +00:00
Amber Brown	a06614bd2a	UPSERT many functionality (#4644)	2019-02-20 23:03:30 +11:00
Erik Johnston	c003450057	Merge pull request #4671 from matrix-org/erikj/state_cache_invalidation ... Batch cache invalidation over replication	2019-02-19 13:14:30 +00:00
Erik Johnston	62175a20e5	Docs	2019-02-19 11:38:40 +00:00
Erik Johnston	1bb35e3a83	Use itertools	2019-02-19 11:34:40 +00:00
Erik Johnston	bc8fa1509d	Documentation	2019-02-19 11:24:59 +00:00
Erik Johnston	a9b5ea6fc1	Batch cache invalidation over replication ... Currently whenever the current state changes in a room invalidate a lot of caches, which cause *a lot* of traffic over replication. Instead, lets batch up all those invalidations and send a single poke down the replication streams. Hopefully this will reduce load on the master process by substantially reducing traffic.	2019-02-18 17:53:31 +00:00
Erik Johnston	4cc4400b4d	Split /login into client_reader	2019-02-18 17:19:01 +00:00
Erik Johnston	eb2b8523ae	Split out registration to worker ... This allows registration to be handled by a worker, though the actual write to the database still happens on master. Note: due to the in-memory session map all registration requests must be handled by the same worker.	2019-02-18 12:12:57 +00:00
Erik Johnston	eaf4d11af9	Add configurable room list publishing rules ... This allows specifying who and what is allowed to be published onto the public room list	2019-02-14 16:02:23 +00:00
Amber Brown	bb4fd8f927	Run black on user directory code (#4635)	2019-02-13 23:05:32 +11:00
Erik Johnston	cf82338930	Merge pull request #4627 from matrix-org/erikj/user_ips_analyze ... Analyze user_ips before running deduplication	2019-02-12 13:05:09 +00:00
Erik Johnston	495ea92350	Fix pep8	2019-02-12 12:40:42 +00:00
Erik Johnston	483ba85c7a	Analyze user_ips before running deduplication ... Due to the table locks taken out by the naive upsert, the table statistics may be out of date. During deduplication it is important that the correct index is used as otherwise a full table scan may be incorrectly used, which can end up thrashing the database badly.	2019-02-12 11:55:27 +00:00
Erik Johnston	362d80b770	Reduce user_ips bloat during dedupe background update ... The background update to remove duplicate rows naively deleted and reinserted the duplicates. For large tables with a large number of duplicates this causes a lot of bloat (with postgres), as the inserted rows are appended to the table, since deleted rows will not be overwritten until a VACUUM has happened. This should hopefully also help ensure that the query in the last batch uses the correct index, as inserting a large number of new rows without analyzing will upset the query planner.	2019-02-12 11:39:34 +00:00
Erik Johnston	719e073f00	Merge pull request #4580 from matrix-org/uhoreg/e2e_backup_add_updating ... add updating of backup versions	2019-02-11 13:45:49 +00:00
Richard van der Hoff	56710c7df5	Fix 'no unique or exclusion constraint' error (#4591) ... Add more tables to the list of tables which need a background update to complete before we can upsert into them, which fixes a race against the background updates.	2019-02-08 18:30:46 +00:00
Hubert Chathi	8248637173	add new endpoint to update backup versions	2019-02-06 17:57:10 -05:00
Andrew Morgan	627ecd358e	Filter user directory state query to a subset of state events (#4462) ... * Filter user directory state query to a subset of state events * Add changelog	2019-02-05 12:16:28 +00:00
Andrew Morgan	c45fd0dda0	Merge pull request #4530 from matrix-org/anoa/room_upgrade_federatable ... Copy over non-federatable trait on room upgrade	2019-01-31 20:28:37 +00:00
Andrew Morgan	d239f67c25	Raise an exception instead of returning None	2019-01-31 18:34:15 +00:00
Andrew Morgan	3ed3cb4339	New function for getting room's create event	2019-01-31 18:21:39 +00:00
Richard van der Hoff	e707e7b38d	Fix infinite loop when an event is redacted in a v3 room (#4535)	2019-01-31 15:34:17 +00:00
Amber Brown	3f189c902e	Fix flake8 (#4519)	2019-01-30 10:53:17 +00:00
Erik Johnston	a696c48133	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/redactions_eiah	2019-01-29 22:00:33 +00:00
Erik Johnston	c21b7cbc09	Update synapse/storage/events_worker.py	2019-01-29 21:53:48 +00:00
Erik Johnston	4db252c073	Check redaction state when event is pulled out of the database	2019-01-29 21:48:36 +00:00
Erik Johnston	6d23ec2111	Fix typo	2019-01-29 21:45:53 +00:00
Erik Johnston	b5d510ad64	Remove unused arg	2019-01-29 21:45:28 +00:00
Richard van der Hoff	82165eeb05	Update synapse/storage/events_worker.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2019-01-29 21:14:39 +00:00
Erik Johnston	7709d2bd16	Implement rechecking of redactions	2019-01-29 11:56:20 +00:00
Erik Johnston	be47cfa9c9	Refactor event building into EventBuilder ... This is so that everything is done in one place, making it easier to change the event format based on room version	2019-01-29 11:13:00 +00:00
Amber Brown	94fb63e44f	Fix typo in upserts code (#4505) ... * fix obvious problem :| * changelog	2019-01-29 10:04:23 +00:00
Amber Brown	7072fe3084	Fix UPSERTs on SQLite 3.24+ (#4477)	2019-01-28 15:43:32 +00:00
Andrew Morgan	4026d555fa	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-28 14:08:24 +00:00
Erik Johnston	b6dce9b9fd	Merge pull request #4470 from matrix-org/erikj/require_format_version ... Require event format version to parse or create events	2019-01-25 15:59:36 +00:00
Erik Johnston	431e485914	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sqlite_native_upsert	2019-01-25 14:11:17 +00:00
Andrew Morgan	0b3fd1401f	Don't require sqlite3 when using postgres (#4466)	2019-01-25 22:25:02 +11:00
Andrew Morgan	821b65aeb5	Merge branch 'develop' of github.com:matrix-org/synapse into anoa/dm_room_upgrade	2019-01-25 11:09:53 +00:00
Andrew Morgan	b1b6dba2d2	Merge pull request #4415 from matrix-org/anoa/full_search_upgraded_rooms ... Ability to search entire room history after upgrading room	2019-01-25 10:49:58 +00:00
Erik Johnston	1953067136	Disable native upserts for sqlite, as they don't work	2019-01-25 10:46:49 +00:00
Erik Johnston	678a92cb56	Replace missed usages of FrozenEvent	2019-01-25 10:32:30 +00:00
Erik Johnston	be6a7e47fa	Revert "Require event format version to parse or create events"	2019-01-25 10:23:51 +00:00
Erik Johnston	efb8ed1d45	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version	2019-01-24 18:52:34 +00:00
Erik Johnston	80bcca659e	Merge pull request #4405 from matrix-org/erikj/fixup_rejecting_invites ... Store rejected remote invite events as outliers	2019-01-24 18:33:27 +00:00
Erik Johnston	664b7a2920	Merge pull request #4437 from matrix-org/erikj/event_format_version_v2 ... Add support for persisting event format versions	2019-01-24 18:32:30 +00:00
Erik Johnston	b8082a5445	Use term 'out of band membership' instead	2019-01-24 17:33:19 +00:00
Andrew Morgan	a383289b0d	Merge branch 'anoa/full_search_upgraded_rooms' of github.com:matrix-org/synapse into anoa/full_search_upgraded_rooms	2019-01-24 17:23:51 +00:00
Andrew Morgan	e1781b043b	Remove redundant create event None check	2019-01-24 17:23:39 +00:00
Richard van der Hoff	03c85335d1	Apply suggestions from code review ... Co-Authored-By: anoadragon453 <1342360+anoadragon453@users.noreply.github.com>	2019-01-24 17:22:09 +00:00
Amber Brown	0e27501ee5	Fix UPSERT check (#4459)	2019-01-24 22:57:41 +11:00
Erik Johnston	e8c9f15397	Replace missed usages of FrozenEvent	2019-01-24 11:14:07 +00:00
Amber Brown	58f6c48183	Use native UPSERTs where possible (#4306)	2019-01-24 21:31:54 +11:00
Erik Johnston	17898a5ba6	Merge branch 'erikj/fixup_rejecting_invites' of github.com:matrix-org/synapse into erikj/require_format_version	2019-01-23 20:13:07 +00:00
Erik Johnston	7c288c2250	Clarify the invite flows	2019-01-23 20:07:47 +00:00
Erik Johnston	886e5acc76	Store rejected remote invite events as outliers ... Currently they're stored as non-outliers even though the server isn't in the room, which can be problematic in places where the code assumes it has the state for all non outlier events. In particular, there is an edge case where persisting the leave event triggers a state resolution, which requires looking up the room version from state. Since the server doesn't have the state, this causes an exception to be thrown.	2019-01-23 20:04:57 +00:00
Erik Johnston	2a8edbaf74	Merge branch 'erikj/get_pdu_versions' into erikj/require_format_version	2019-01-23 17:27:49 +00:00
Erik Johnston	be1065af59	isort	2019-01-23 11:48:16 +00:00
Erik Johnston	c5a296b10c	Add support for persisting event format versions ... Currently we only have the one event format version defined, but this adds the necessary infrastructure to persist and fetch the format versions alongside the events. We specify the format version rather than the room version as: 1. We don't necessarily know the room version, existing events may be either v1 or v2. 2. We'd need to be careful to prevent/handle correctly if different events in the same room reported to be of different versions, which sounds annoying.	2019-01-23 11:30:01 +00:00
Erik Johnston	90743c9d89	Fixup removal of duplicate user_ips rows (#4432) ... * Remove unnecessary ORDER BY clause * Add logging * Newsfile	2019-01-23 19:45:18 +11:00
Erik Johnston	7f503f83b9	Refactor to rewrite the SQL instead	2019-01-22 16:31:05 +00:00
Erik Johnston	1c9704f8ab	Don't shadow params	2019-01-22 16:20:33 +00:00
Erik Johnston	2557531f0f	Fix bug when removing duplicate rows from user_ips ... This was caused by accidentally overwritting a `last_seen` variable in a for loop, causing the wrong value to be written to the progress table. The result of which was that we didn't scan sections of the table when searching for duplicates, and so some duplicates did not get deleted.	2019-01-22 13:33:46 +00:00
Andrew Morgan	277e50462d	Do not return in a deferred function	2019-01-22 12:40:26 +00:00
Andrew Morgan	c433f61091	Ensure new filter is actually created	2019-01-22 12:06:36 +00:00
Andrew Morgan	c9bfb058d8	Fix a bug with single-room search searching all rooms ... * Create a new method for getting predecessor rooms * Remove formatting change	2019-01-22 12:00:41 +00:00
Andrew Morgan	25d64a846a	Fix typos	2019-01-22 11:00:04 +00:00
Erik Johnston	25dd56ace3	Fix race when persisting create event (#4404) ... * Fix race when persisting create event When persisting a chunk of DAG it is sometimes requried to do a state resolution, which requires knowledge of the room version. If this happens while we're persisting the create event then we need to use that event rather than attempting to look it up in the database.	2019-01-18 23:17:04 +11:00
Andrew Morgan	df3a661e4a	Search for messages across predecessor rooms ... Signed-off-by: Andrew Morgan <andrew@amorgan.xyz>	2019-01-18 11:19:20 +00:00
Amber Brown	a35c66a00b	Remove duplicates in the user_ips table and add an index (#4370)	2019-01-12 06:21:50 +11:00
Amber Brown	7960c26fda	Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369)	2019-01-09 22:26:25 +11:00
Neil Johnson	84b6fae1f5	Ensure synchrotrons can access is_support_user in the storage layer	2019-01-02 10:19:59 +00:00
Will Hunt	0708f437cc	Log roomid along with Unknown room (#4297)	2018-12-24 10:49:42 +01:00
Neil Johnson	d2f7c4e6b1	create support user (#4141) ... Allow for the creation of a support user. A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.	2018-12-14 18:20:59 +00:00
Amber Brown	e93a0ebf50	Settings Fix deleting e2e room keys on xenial (#4295)	2018-12-14 05:10:31 +11:00
Richard van der Hoff	30da50a5b8	Initialise user displayname from SAML2 data (#4272) ... When we register a new user from SAML2 data, initialise their displayname correctly.	2018-12-07 14:44:46 +01:00
Richard van der Hoff	9a3e24a13d	drop undocumented dependency on dateutil (#4266) ... It turns out we were relying on dateutil being pulled in transitively by pysaml2. There's no need for that bloat.	2018-12-06 04:52:42 +11:00
Travis Ralston	158ffb92f1	Add an option to disable search for homeservers which may not be interested in it (#4230) ... This is useful for homeservers not intended for users, such as bot-only homeservers or ones that only process IoT data.	2018-12-04 12:01:02 +01:00
Ben Parsons	b5ac0ffa0a	add more detail to logging regarding "More than one row matched" error (#4234)	2018-12-04 11:57:39 +01:00
Richard van der Hoff	ecc23188f4	Fix UnicodeDecodeError when postgres is not configured in english (#4253) ... This is a bit of a half-assed effort at fixing https://github.com/matrix-org/synapse/issues/4252. Fundamentally the right answer is to drop support for Python 2.	2018-12-04 11:55:52 +01:00
Aaron Raimist	704c5298f0	Drop sent_transactions ... Signed-off-by: Aaron Raimist <aaron@raim.ist>	2018-12-01 23:07:35 -06:00
Neil Johnson	f9b136a886	Neilj/fix mau initial reserved users (#4211) ... * fix transaction wrapping bug that caused get_user_id_by_threepid_txn to fail * towncrier * white space	2018-11-28 20:33:41 +11:00
Richard van der Hoff	6c18cc4b50	Ignore __pycache__ directories in schema delta dir ... Now that we use py3, compiled python ends up in __pycache__ rather than *.pyc.	2018-11-20 22:52:34 +00:00
Travis Ralston	835779f7fb	Add option to track MAU stats (but not limit people) (#3830)	2018-11-15 18:08:27 +00:00
David Baker	bca3b91c2d	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backup_versions_are_numbers	2018-11-09 18:35:02 +00:00
David Baker	d44dea0223	pep8	2018-11-09 14:38:31 +00:00
David Baker	4f93abd62d	add docs	2018-11-09 13:25:38 +00:00
Erik Johnston	5ebed18692	Lets convert bytes to unicode instead	2018-11-08 12:33:13 +00:00
Erik Johnston	b1a22b24ab	Fix noop checks when updating device keys ... Clients often reupload their device keys (for some reason) so its important for the server to check for no-ops before sending out device list update notifications. The check is broken in python 3 due to the fact comparing bytes and unicode always fails, and that we write bytes to the DB but get unicode when we read.	2018-11-08 12:18:38 +00:00
Erik Johnston	bc80b3f454	Add helpers for getting prev and auth events (#4139) ... * Add helpers for getting prev and auth events This is in preparation for allowing the event format to change between room versions.	2018-11-06 00:35:15 +11:00
Richard van der Hoff	350f654e7b	Add unique indexes to a couple of tables ... The indexes on device_lists_remote_extremeties can be unique, and they therefore should, to ensure that the db remains consistent.	2018-11-02 10:36:13 +00:00
Richard van der Hoff	50e328d1e7	Remove redundant database locks for device list updates ... We can rely on the application-level per-user linearizer.	2018-11-02 10:36:13 +00:00
Erik Johnston	0794504bce	Merge pull request #4006 from matrix-org/erikj/purge_state_groups ... Delete unreferenced state groups during purge	2018-10-30 16:58:22 +00:00
David Baker	e0934acdbb	Cast to int here too	2018-10-30 11:12:23 +00:00
David Baker	12941f5f8b	Cast bacjup version to int when querying	2018-10-30 11:01:07 +00:00
David Baker	2f0f911c52	Convert version back to a string	2018-10-30 10:35:18 +00:00
David Baker	64fa557f80	Try & make it work on postgres	2018-10-30 09:51:04 +00:00
David Baker	563f9b61b1	Make e2e backup versions numeric in the DB ... We were doing max(version) which does not do what we wanted on a column of type TEXT.	2018-10-29 21:01:22 +00:00
Erik Johnston	88e5ffe6fe	Deduplicate device updates sent over replication ... We currently send several kHz of device list updates over replication occisonally, which often causes the replications streams to lag and then get dropped. A lot of those updates will actually be duplicates, since we don't send e.g. device_ids across replication, so let's deduplicate it when we pull them out of the database.	2018-10-29 17:34:34 +00:00
Erik Johnston	ad88460e0d	Move _find_unreferenced_groups	2018-10-29 14:24:19 +00:00
Erik Johnston	664b192a3b	Fix set operations thinko	2018-10-29 14:21:43 +00:00
Erik Johnston	f4f223aa44	Don't make temporary list	2018-10-29 14:01:49 +00:00
Erik Johnston	b2399f6281	Make SQL a bit cleaner	2018-10-29 14:01:11 +00:00
Erik Johnston	03e634dad4	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups	2018-10-26 16:22:45 +01:00
Richard van der Hoff	4cda300058	preserve room visibility	2018-10-25 19:10:24 +01:00
Erik Johnston	cb53ce9d64	Refactor state group lookup to reduce DB hits (#4011) ... Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.	2018-10-25 17:49:55 +01:00
Neil Johnson	95ad128851	Merge pull request #4081 from matrix-org/neilj/fix_mau_init ... fix race condiftion in calling initialise_reserved_users	2018-10-25 16:33:40 +01:00
Neil Johnson	fcbd488e9a	add new line	2018-10-25 16:13:43 +01:00
Neil Johnson	f8fe98812b	improve comments	2018-10-25 14:58:59 +01:00
Neil Johnson	f7f487e14c	Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin	2018-10-25 14:40:06 +01:00
Neil Johnson	ea69a84bbb	fix style inconsistencies	2018-10-24 17:18:08 +01:00
Neil Johnson	07126e43a4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_mau_init	2018-10-24 16:25:39 +01:00
Richard van der Hoff	7328039117	Merge pull request #4082 from matrix-org/rav/fix_pep8 ... Fix a number of flake8 errors	2018-10-24 11:23:35 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase ... Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	ef771cc4c2	Fix a number of flake8 errors ... Broadly three things here: * disable W504 which seems a bit whacko * remove a bunch of `as e` expressions from exception handlers that don't use them * use `r""` for strings which include backslashes Also, we don't use pep8 any more, so we can get rid of the duplicate config there.	2018-10-24 10:39:03 +01:00
Neil Johnson	329d18b39c	remove white space	2018-10-23 15:27:20 +01:00
Neil Johnson	6105c6101f	fix race condiftion in calling initialise_reserved_users	2018-10-23 15:24:58 +01:00
Amber Brown	e1728dfcbe	Make scripts/ and scripts-dev/ pass pyflakes (and the rest of the codebase on py3) (#4068)	2018-10-20 11:16:55 +11:00
Erik Johnston	67f7b9cb50	pep8	2018-10-19 16:06:59 +01:00
Erik Johnston	056f099126	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups	2018-10-19 15:48:59 +01:00
Erik Johnston	47a9da28ca	Batch process handling state groups	2018-10-19 15:48:15 +01:00
Richard van der Hoff	fc0f13dd03	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-16 21:10:04 +01:00
Erik Johnston	4a28d3d36f	Update event_auth table for rejected events	2018-10-16 16:28:42 +01:00
Erik Johnston	e238013c44	Add v2 state res algorithm. ... We hook this up to the vdh test room version.	2018-10-16 16:16:13 +01:00
David Baker	03c11032c3	Merge pull request #4019 from matrix-org/dbkr/e2e_backups ... E2E backups	2018-10-15 10:19:43 +01:00
Erik Johnston	67a1e315cc	Fix up comments	2018-10-12 13:49:48 +01:00
David Baker	306361b31b	Misc PR feedback bits	2018-10-12 11:48:56 +01:00
Richard van der Hoff	8ddd0f273c	Comments on get_all_new_events_stream ... just some docstrings to clarify the behaviour here	2018-10-12 09:55:41 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
David Baker	497444f1fd	Don't reuse backup versions ... Since we don't actually delete the keys, just mark the versions as deleted in the db rather than actually deleting them, then we won't reuse versions. Fixes https://github.com/vector-im/riot-web/issues/7448	2018-10-05 15:08:36 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Erik Johnston	4917ff5523	Add state_group index to event_to_state_groups ... This is needed to efficiently check for unreferenced state groups during purge.	2018-10-04 16:03:08 +01:00
Erik Johnston	17d585753f	Delete unreferened state groups during purge	2018-10-04 16:03:06 +01:00
Richard van der Hoff	9693625e55	actually exclude outliers	2018-10-03 10:19:41 +01:00
Richard van der Hoff	3e39783d5d	remove debugging	2018-10-02 23:44:14 +01:00
Richard van der Hoff	ae61ade891	Fix bug in forward_extremity update logic ... An event does not stop being a forward_extremity just because an outlier or rejected event refers to it.	2018-10-02 23:36:08 +01:00
Erik Johnston	7258d081a5	Fix bug when invalidating destination retry timings	2018-10-02 15:47:57 +01:00
Matthew Hodgson	5b68f29f48	fix thinkos	2018-09-29 02:14:40 +01:00
Richard van der Hoff	9c8cec5dab	Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache	2018-09-28 10:51:09 +01:00
Richard van der Hoff	51d33d5178	Merge pull request #3961 from matrix-org/neilj/lock_mau_upserts ... fix #3854 MAU transaction errors	2018-09-27 12:31:37 +01:00
Richard van der Hoff	ae6ad4cf41	docstrings and unittests for storage.state (#3958) ... I spent ages trying to figure out how I was going mad...	2018-09-27 11:22:25 +01:00
Neil Johnson	28781b65e7	fix #3854	2018-09-26 16:16:41 +01:00
Erik Johnston	fdd1a62e8d	Add a five minute cache to get_destination_retry_timings ... Hopefully helps with #3931	2018-09-21 14:56:12 +01:00
Amber Brown	1f3f5fcf52	Fix client IPs being broken on Python 3 (#3908)	2018-09-20 20:14:34 +10:00
Erik Johnston	bbab6ebfd9	Fix up changelog and remove spurious comment	2018-09-19 14:45:14 +01:00
Erik Johnston	392a54128c	pep8	2018-09-19 14:37:49 +01:00
Erik Johnston	b9158ac2bf	Remove get_destination_retry_timings cache ... Currently we rely on the master to invalidate this cache promptly. However, after having moved most federation endpoints off of master this no longer happens, causing outbound fedeariont to get blackholed. Fixes #3798	2018-09-19 14:22:57 +01:00
Richard van der Hoff	85a43f4167	Return a 404 when deleting unknown room alias ... As per https://github.com/matrix-org/matrix-doc/issues/1675 Fixes https://github.com/matrix-org/synapse/issues/2782	2018-09-17 13:19:00 +01:00
David Baker	bc74925c5b	WIP e2e key backups ... Continues from uhoreg's branch This just fixed the errcode on /room_keys/version if no backup and updates the schema delta to be on the latest so it gets run	2018-09-13 17:02:59 +01:00
Erik Johnston	ed5331a627	comment	2018-09-13 16:10:56 +01:00
Erik Johnston	9cbd0094f0	pep8	2018-09-13 15:15:35 +01:00
Erik Johnston	9dbe38ea7d	Create indices after insertion	2018-09-13 15:05:52 +01:00
Erik Johnston	c857f5ef9b	Make purge history slightly faster ... Don't pull out events that are outliers and won't be deleted, as nothing should happen to them.	2018-09-13 12:48:10 +01:00
Neil Johnson	f30a303590	Merge pull request #3846 from matrix-org/neilj/expose-registered-users ... expose number of real reserved users	2018-09-12 17:14:04 +01:00
Neil Johnson	8decd6233d	improve naming	2018-09-12 16:22:15 +01:00
Neil Johnson	0ddf486724	expose number of real reserved users	2018-09-12 11:58:52 +01:00
Amber Brown	33716c4aea	Merge pull request #3826 from matrix-org/rav/logging_for_keyring ... add some logging for the keyring queue	2018-09-12 20:43:47 +10:00
Matthew Hodgson	b041115415	Speed up lazy loading (#3827) ... * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)	2018-09-12 00:50:39 +01:00
Richard van der Hoff	cd7ef43872	clearer logging when things fail, too	2018-09-06 23:56:47 +01:00
Neil Johnson	84a750e0c3	ensure guests never enter mau list	2018-09-06 17:22:53 +01:00
Hubert Chathi	3801b8aa03	try to make flake8 and isort happy	2018-09-06 11:35:19 -04:00
Neil Johnson	61b05727fa	guest users should not be part of mau total	2018-09-05 22:30:36 +01:00
Neil Johnson	0b01281e77	move threepid checker to config, add missing yields	2018-08-31 17:11:11 +01:00
Neil Johnson	09f3cf1a7e	ensure post registration auth checks do not fail erroneously	2018-08-31 15:42:51 +01:00
Amber Brown	14e4d4f4bf	Port storage/ to Python 3 (#3725)	2018-08-31 00:19:58 +10:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	cd77270a66	Implement trail users	2018-08-23 19:17:19 +01:00
Erik Johnston	7a0da69eee	Add missing yield	2018-08-23 10:28:12 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles ... Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	3bf8bab8f9	Merge pull request #3673 from matrix-org/erikj/refactor_state_handler ... Refactor state module to support multiple room versions	2018-08-22 10:04:55 +01:00
Matthew Hodgson	bb81e78ec6	Split the state_group_cache in two (#3726) ... Splits the state_group_cache in two. One half contains normal state events; the other contains member events. The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache. This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching. Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow. Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.	2018-08-22 00:56:37 +02:00
Hubert Chathi	42a394caa2	allow session_data to be any JSON instead of just a string	2018-08-21 14:51:34 -04:00
Hubert Chathi	8550a7e9c2	allow auth_data to be any JSON instead of a string	2018-08-21 10:38:00 -04:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	38f708a2bb	Remote profile cache should remain in master worker	2018-08-17 11:37:42 +01:00
Neil Johnson	bcfeb44afe	call reap on start up and fix under reaping bug	2018-08-16 22:55:32 +01:00
Matthew Hodgson	3f543dc021	initial cut at a room summary API (#3574)	2018-08-16 09:46:50 +01:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568)	2018-08-15 16:35:22 +01:00
Erik Johnston	ef184caf30	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-15 14:25:46 +01:00
Neil Johnson	9ecbaf8ba8	adding missing yield	2018-08-14 16:55:28 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block ... Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Amber Brown	591bf87c6a	Merge remote-tracking branch 'origin/develop' into neilj/fix_reap_users_in_postgres	2018-08-14 20:56:23 +10:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	2018-08-13 16:47:46 +10:00
Matthew Hodgson	f0cede5556	missing import	2018-08-12 19:14:31 -04:00
Matthew Hodgson	66a4ca1d28	404 nicely if you try to interact with a missing current version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	edc427a351	flake8	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	15d513f16f	fix idiocies and so make tests pass	2018-08-12 19:14:31 -04:00
Matthew Hodgson	b5eee511c7	don't needlessly return user_id	2018-08-12 19:14:31 -04:00
Matthew Hodgson	982edca380	fix flakes	2018-08-12 19:14:31 -04:00
Matthew Hodgson	234611f347	fix typos	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f0791b7bd	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8d14598e90	add storage docstring; remove unused set_e2e_room_keys	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cac0253799	rename room_key_version table correctly, and fix opt args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	0abb205b47	blindly incorporate PR review - needs testing & fixing	2018-08-12 19:14:31 -04:00
Matthew Hodgson	69e51c7ba4	make /room_keys/version work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8ae64b270f	implement /room_keys/version too (untested)	2018-08-12 19:14:31 -04:00
Matthew Hodgson	6b8c07abc2	make it work and fix pep8	2018-08-12 19:13:09 -04:00
Matthew Hodgson	0bc4627a73	interim WIP checkin; doesn't build yet	2018-08-12 18:23:10 -04:00
Matthew Hodgson	53ace904b2	total WIP skeleton for /room_keys API	2018-08-12 18:23:10 -04:00
Neil Johnson	31fa743567	fix sqlite/postgres incompatibility in reap_monthly_active_users	2018-08-11 22:38:34 +01:00
Amber Brown	b37c472419	Rename async to async_helpers because async is a keyword on Python 3.7 (#3678)	2018-08-10 23:50:21 +10:00
Neil Johnson	885ea9c602	rename _user_last_seen_monthly_active	2018-08-09 18:02:12 +01:00
Erik Johnston	ce6db0e547	Choose state algorithm based on room version	2018-08-09 14:58:47 +01:00
Erik Johnston	484a0ebdfc	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-09 10:16:29 +01:00
Erik Johnston	5785b93711	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-09 10:16:16 +01:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions ... Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	be59910b93	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/reserved_users	2018-08-08 13:45:21 +01:00
Neil Johnson	990fe9fc23	Merge pull request #3633 from matrix-org/neilj/mau_tracker ... API for monthly_active_users table	2018-08-08 12:44:15 +00:00
Neil Johnson	312ae74746	typos	2018-08-08 13:33:16 +01:00
Neil Johnson	ef3589063a	prevent total number of reserved users being too large	2018-08-07 17:59:27 +01:00
Neil Johnson	e8eba2b4e3	implement reserved users for mau limits	2018-08-07 17:49:43 +01:00
Erik Johnston	cd9765805e	Allow ratelimiting on workers	2018-08-07 10:50:28 +01:00
Erik Johnston	495cb100d1	Allow profile changes to happen on workers	2018-08-07 10:50:26 +01:00
Richard van der Hoff	ca9bc1f4fe	Fix occasional glitches in the synapse_event_persisted_position metric ... Every so often this metric glitched to a negative number. I'm assuming it was due to backfilled events.	2018-08-07 10:00:03 +01:00
Neil Johnson	a74b25faaa	WIP building out mau reserved users	2018-08-06 23:25:25 +01:00
Neil Johnson	e54794f5b6	Fix postgres compatibility bug	2018-08-06 21:55:54 +01:00
Neil Johnson	16d78be315	make use of _simple_select_one_onecol, improved comments	2018-08-06 17:51:15 +01:00
Erik Johnston	96a9a29645	Pull in necessary stores in federation_reader	2018-08-06 15:23:57 +01:00
Erik Johnston	62ace05c45	Move necessary storage functions to worker classes	2018-08-06 15:23:38 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Neil Johnson	e10830e976	wip commit - tests failing	2018-08-03 17:55:50 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning ... This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Neil Johnson	950807d93a	fix caching and tests	2018-08-03 13:49:53 +01:00
Neil Johnson	897c51d274	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-03 13:40:47 +01:00
Neil Johnson	5e2f7b8084	typo	2018-08-02 22:47:48 +01:00
Neil Johnson	4ecb4bdac9	wip attempt at caching	2018-08-02 22:41:05 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Neil Johnson	9180061b49	remove unused count_monthly_users	2018-08-02 15:55:29 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups ... Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	0a65450d04	Validation for events/rooms in fed requests ... When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)	2018-08-02 13:48:40 +01:00
Neil Johnson	00f99f74b1	insertion into monthly_active_users	2018-08-02 13:47:19 +01:00
Neil Johnson	4a6725d9d1	Merge branch 'neilj/mau_tracker' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-02 11:04:18 +01:00
Neil Johnson	165e067033	Revert "change monthly_active_users table to be a single column" ... This reverts commit ec716a35b2.	2018-08-02 10:59:58 +01:00
Erik Johnston	40c1c59cf4	Merge pull request #3621 from matrix-org/erikj/split_fed_store ... Split out DB writes in federation handler	2018-08-02 10:41:42 +01:00
Neil Johnson	08281fe6b7	self.db_conn unused	2018-08-01 23:26:24 +01:00
Neil Johnson	c21d82bab3	normalise reaping query	2018-08-01 23:24:38 +01:00
Neil Johnson	ec716a35b2	change monthly_active_users table to be a single column	2018-08-01 17:54:37 +01:00
Neil Johnson	d766f26de9	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-01 17:49:41 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits ... Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Neil Johnson	7ff44d9215	improve clarity	2018-08-01 16:17:00 +01:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	2018-08-02 00:54:06 +10:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Erik Johnston	a6d7b74915	update docs	2018-08-01 13:39:14 +01:00
Neil Johnson	4e5ac901dd	clean up	2018-08-01 12:03:57 +01:00
Neil Johnson	f9f5559971	fix comment	2018-08-01 12:03:42 +01:00
Neil Johnson	6023cdd227	remove errant print	2018-08-01 10:27:17 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Neil Johnson	6ef983ce5c	api into monthly_active_users table	2018-07-31 16:36:24 +01:00
Richard van der Hoff	5de936caa1	Merge pull request #3612 from matrix-org/rav/store_heirarchy ... Make EventStore inherit from EventFederationStore	2018-07-31 13:44:04 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Richard van der Hoff	0bc9b9e397	reinstate explicit include of EventsWorkerStore	2018-07-31 13:11:04 +01:00
Neil Johnson	fef7e58ac6	actually close conn	2018-07-30 22:29:44 +01:00
Neil Johnson	9b13817e06	factor out metrics from __init__ to app/homeserver	2018-07-30 22:07:07 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Erik Johnston	143f1a2532	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store	2018-07-30 09:56:18 +01:00
Matthew Hodgson	e9b2d047f6	make /context lazyload & filter aware (#3567) ... make /context lazyload & filter aware.	2018-07-27 15:12:50 +01:00
Richard van der Hoff	65c8dee900	Merge pull request #3614 from matrix-org/rav/stop_populating_event_content ... Stop populating events.content	2018-07-26 22:54:08 +01:00
Richard van der Hoff	9e68b1bd2d	Merge pull request #3613 from matrix-org/rav/stop_using_event_edges_room_id ... Remove some redundant joins on event_edges.room_id	2018-07-26 22:31:01 +01:00
Richard van der Hoff	ef9d51b081	Merge pull request #3610 from matrix-org/rav/fix_looping_calls ... Fix some looping_call calls which were broken in #3604	2018-07-26 14:55:57 +01:00
Richard van der Hoff	51d7df1915	Create the column nullable ... There's no real point in ever making the column non-nullable, and doing so breaks the sytests.	2018-07-26 14:54:04 +01:00
Richard van der Hoff	5c1d301fd9	Stop populating events.content ... This field is no longer read from, so we should stop populating it. Once we're happy that this doesn't break everything, and a rollback is unlikely, we can think about dropping the column.	2018-07-26 14:43:02 +01:00
Richard van der Hoff	bd4b25f4d0	Remove some redundant joins on event_edges.room_id ... We've long passed the point where it's possible to have the same event_id in different tables, so these join conditions are redundant: we can just join on event_id. event_edges is of non-trivial size, and the room_id column is wasteful, so let's stop reading from it. In future, we can stop writing to it, and then drop it.	2018-07-26 13:19:08 +01:00
Richard van der Hoff	1b4d73fa52	comment on event_edges	2018-07-26 12:53:51 +01:00
Richard van der Hoff	21e878ebb6	Make EventStore inherit from EventFederationStore ... (since it uses methods therein) Turns out that we had a bunch of things which were incorrectly importing EventWorkerStore from events.py rather than events_worker.py, which broke once I removed the import into events.py.	2018-07-26 12:48:51 +01:00
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	2018-07-26 11:48:08 +01:00
Matthew Hodgson	bc7944e6d2	switch missing_types to be a bool	2018-07-25 23:36:31 +01:00
Matthew Hodgson	2565804030	Merge branch 'develop' into matthew/filter_members	2018-07-25 17:27:49 +01:00
Matthew Hodgson	7d9fb88617	incorporate more review.	2018-07-25 16:33:50 +01:00
Erik Johnston	78a691d005	Split out DB writes in federation handler ... This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.	2018-07-25 16:22:56 +01:00
Erik Johnston	3849f7f69f	Merge pull request #3603 from matrix-org/erikj/handle_outliers ... Correctly handle outliers during persist events	2018-07-25 13:24:04 +01:00
Richard van der Hoff	cee1ae1b72	Merge pull request #3606 from matrix-org/rav/logcontext_fixes_once_more ... Fix another logcontext leak in _persist_events	2018-07-25 11:56:00 +01:00
Richard van der Hoff	32b30e15f2	Merge pull request #3607 from matrix-org/rav/fix_persist_events_integrity_error ... Fix occasional 'tuple index out of range' error	2018-07-25 11:55:45 +01:00
Erik Johnston	7780a7b47c	Actually fix it by adding continue	2018-07-25 11:13:20 +01:00
Richard van der Hoff	1be94440d3	Fix occasional 'tuple index out of range' error ... This fixes a bug in _delete_existing_rows_txn which was introduced in #3435 (though it's been on matrix-org-hotfixes for *years*). This code is only called when there is some sort of conflict the first time we try to persist an event, so it only happens rarely. Still, the exceptions are annoying.	2018-07-25 11:05:58 +01:00
Richard van der Hoff	07defd5fe6	Fix another logcontext leak in _persist_events ... We need to run the errback in the sentinel context to avoid losing our own context. Also: add logging to runInteraction to help identify where "Starting db connection from sentinel context" warnings are coming from	2018-07-25 10:53:23 +01:00
Erik Johnston	a297ff2b16	Fix typo in conditional	2018-07-25 09:48:01 +01:00
Richard van der Hoff	371da42ae4	Wrap a number of things that run in the background ... This will reduce the number of "Starting db connection from sentinel context" warnings, and will help with our metrics.	2018-07-25 09:41:12 +01:00
Erik Johnston	ec56121b0d	Correctly handle outliers during persist events ... We incorrectly asserted that all contexts must have a non None state group without consider outliers. This would usually be fine as the assertion would never be hit, as there is a shortcut during persistence if the forward extremities don't change. However, if the outlier is being persisted with non-outlier events, the function would be called and the assertion would be hit. Fixes #3601	2018-07-25 09:35:02 +01:00
Matthew Hodgson	cb5c37a57c	handle the edge case for _get_some_state_from_cache where types is []	2018-07-24 20:34:45 +01:00
Erik Johnston	3188973857	Pull out did_forget to worker store	2018-07-24 16:49:14 +01:00
Erik Johnston	60a1d147a7	Merge pull request #3595 from matrix-org/erikj/use_deltas ... Use deltas to calculate current state deltas	2018-07-24 15:41:18 +01:00
Erik Johnston	709c309b0e	Expand on docstring comment about return value	2018-07-24 15:12:50 +01:00
Erik Johnston	8f65ab98d2	Remove unnecessary iteritems	2018-07-24 15:08:01 +01:00
Erik Johnston	ed0dd68731	Fixup comment (and indent)	2018-07-24 14:31:38 +01:00
Erik Johnston	811ac73a42	Don't fetch state from the database unless needed	2018-07-24 14:18:23 +01:00
Erik Johnston	a79410e7b8	Have _get_new_state_after_events return delta ... If we have a delta from the existing to new current state, then we can reuse that rather than manually working it out by fetching both lots of state.	2018-07-24 14:14:17 +01:00
Richard van der Hoff	93b0722c50	Merge pull request #3583 from matrix-org/rav/remove_who_forgot_in_room ... Remove redundant checks on room forgottenness	2018-07-24 14:11:11 +01:00
Matthew Hodgson	454f59b7ad	Merge branch 'develop' into matthew/filter_members	2018-07-24 14:03:37 +01:00
Erik Johnston	223341205e	Don't require to_delete to have event_ids	2018-07-24 14:02:40 +01:00
Matthew Hodgson	e22700c3dd	consider non-filter_type types as wildcards, thus missing from the state-group-cache	2018-07-24 13:59:07 +01:00
Matthew Hodgson	d19fba3655	Merge branch 'develop' into matthew/filter_members	2018-07-24 12:39:54 +01:00
Matthew Hodgson	cd241d6bda	incorporate more review	2018-07-24 12:39:40 +01:00
Richard van der Hoff	30bfed5aa5	Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room	2018-07-24 11:46:09 +01:00
Erik Johnston	97acd385a3	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta	2018-07-24 11:32:13 +01:00
Erik Johnston	0fa73e4a63	Remove unnecessary if	2018-07-24 11:19:23 +01:00
Erik Johnston	ff5426f6b8	Speed up _calculate_state_delta	2018-07-24 10:55:11 +01:00
Richard van der Hoff	a678145010	Merge branch 'develop' into rav/logcontext_fixes	2018-07-24 10:43:30 +01:00
Richard van der Hoff	8dff6e0322	Logcontext fixes ... Fix some random logcontext leaks.	2018-07-24 00:37:17 +01:00
Richard van der Hoff	69fb5dbdab	fix idiocy	2018-07-24 00:04:44 +01:00
Richard van der Hoff	1938cffaea	Add some measure blocks to persist_events ... ... to help us figure out where 40% of CPU is going	2018-07-23 23:48:19 +01:00
Matthew Hodgson	efcdacad7d	handle case where types is [] on postgres correctly	2018-07-23 22:41:05 +01:00
Matthew Hodgson	adfe29ec0b	Merge branch 'develop' into matthew/filter_members	2018-07-23 19:21:37 +01:00
Matthew Hodgson	254fb430d1	incorporate review	2018-07-23 19:21:20 +01:00
Erik Johnston	8b9f164fff	Comments	2018-07-23 17:43:01 +01:00
Erik Johnston	50c60e5fad	Only get cached state from context in persist_event ... We don't want to bother pulling out the current state from the DB since until we know we have to. Checking the context for state is just an optimisation.	2018-07-23 17:21:40 +01:00
Richard van der Hoff	4f5cc8e4e7	Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room	2018-07-23 17:15:12 +01:00
Richard van der Hoff	dae6dc1e77	Remove redundant checks on room forgottenness ... Fixes #3550	2018-07-23 17:13:34 +01:00
Erik Johnston	e42510ba63	Use new getters	2018-07-23 13:17:22 +01:00
Matthew Hodgson	9b34f3ea3a	Merge branch 'develop' into matthew/sync_deleted_devices	2018-07-23 10:03:28 +01:00
Matthew Hodgson	c1bf2b587e	add trailing comma	2018-07-23 09:56:23 +01:00
Matthew Hodgson	2f558300cc	fix thinkos; unbreak tests	2018-07-19 19:22:27 +01:00
Matthew Hodgson	bcaec2915a	incorporate review	2018-07-19 19:03:50 +01:00
Matthew Hodgson	924eb34d94	add a filtered_types param to limit filtering to specific types	2018-07-19 18:32:02 +01:00
Matthew Hodgson	be3adfc331	merge develop pydoc for _get_state_for_groups	2018-07-19 11:26:04 +01:00
Matthew Hodgson	9e40834f74	yes, we do need to invalidate the device_id_exists_cache when deleting a remote device	2018-07-19 11:15:10 +01:00
Richard van der Hoff	f1a15ea206	revert 00bc979 ... ... we've fixed the things that caused the warnings, so we should reinstate the warning.	2018-07-19 11:14:20 +01:00
Richard van der Hoff	1ffb7bec20	Merge remote-tracking branch 'origin/release-v0.33.0' into develop	2018-07-19 11:12:33 +01:00
Matthew Hodgson	c0685f67c0	spell out that include_deleted_devices requires include_all_devices	2018-07-19 10:59:02 +01:00
Richard van der Hoff	00bc979137	Disable logcontext warning ... Temporary workaround to #3518 while we release 0.33.0.	2018-07-19 10:51:15 +01:00
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	2018-07-18 20:55:05 +01:00
Matthew Hodgson	12ec58301f	shift to using an explicit deleted flag on m.device_list_update EDUs ... and generally make it work.	2018-07-12 11:39:43 +01:00
Erik Johnston	0456e05977	Merge pull request #3505 from matrix-org/erikj/receipts_cahce ... Use stream cache in get_linearized_receipts_for_room	2018-07-12 09:46:29 +01:00
Erik Johnston	aff1dfdf3d	Update return value docstring	2018-07-12 09:45:37 +01:00
Matthew Hodgson	5797f5542b	WIP to announce deleted devices over federation ... Previously we queued up the poke correctly when the device was deleted, but then the actual EDU wouldn't get sent, as the device was no longer known. Instead, we now send EDUs for deleted devices too if there's a poke for them.	2018-07-12 01:32:39 +01:00
Erik Johnston	6ccefef07a	Use 'is not None' and add comments	2018-07-10 18:12:39 +01:00
Matthew Hodgson	ea752bdd99	s/becuase/because/g	2018-07-10 17:58:18 +01:00
Erik Johnston	05f5dabc10	Use stream cache in get_linearized_receipts_for_room ... This avoids us from uncessarily hitting the database when there has been no change for the room	2018-07-10 17:22:42 +01:00
Richard van der Hoff	c3c29aa196	Attempt to include db threads in cpu usage stats (#3496) ... Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.	2018-07-10 16:12:36 +01:00
Richard van der Hoff	e31e5dee38	Add CPU metrics for _fetch_event_list ... add a Measure block on _fetch_event_list, in the hope that we can better measure CPU usage here.	2018-07-09 18:15:54 +01:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Erik Johnston	3905c693c5	Invalidate cache on correct thread	2018-07-02 11:36:44 +01:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
David Baker	028490afd4	Fix error on deleting users pending deactivation ... Use simple_delete instead of simple_delete_one as commented	2018-06-26 10:52:52 +01:00
Erik Johnston	244484bf3c	Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" ... This reverts commit 1d009013b3.	2018-06-25 13:42:55 +01:00
Richard van der Hoff	1d009013b3	Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" ... This reverts commit ce0d911156, reversing changes made to b4a5d767a9.	2018-06-22 16:35:10 +01:00
Mark Haines	9850f66abe	Deleting from event_push_actions needs to use an index	2018-06-22 15:54:48 +01:00
Richard van der Hoff	200e11c5bf	Merge pull request #3432 from matrix-org/rav/joined_hosts_cache_non_iterable ... Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:18:51 +01:00
Erik Johnston	f8272813a9	Make _get_joined_hosts_cache cache non-iterable	2018-06-22 15:12:26 +01:00
Richard van der Hoff	1d7ad11747	Merge pull request #3430 from matrix-org/rav/configurable_push_action_rotation ... Make push actions rotation configurable	2018-06-22 15:07:31 +01:00
Erik Johnston	ce0d911156	Merge pull request #3431 from matrix-org/rav/erasure_visibility ... Support hiding events from deleted users	2018-06-22 15:06:44 +01:00
Erik Johnston	f79abda87f	Merge pull request #3427 from matrix-org/erikj/remove_filters ... remove dead filter_events_for_clients	2018-06-22 14:45:42 +01:00
Erik Johnston	75dc3ddeab	Make push actions rotation configurable	2018-06-22 14:44:37 +01:00
Richard van der Hoff	43e02c409d	Disable partial state group caching for wildcard lookups ... When _get_state_for_groups is given a wildcard filter, just do a complete lookup. Hopefully this will give us the best of both worlds by not filling up the ram if we only need one or two keys, but also making the cache still work for the federation reader usecase.	2018-06-22 11:52:07 +01:00
Richard van der Hoff	240f192523	Merge pull request #3382 from matrix-org/rav/optimise_state_groups ... Optimise state_group_cache update	2018-06-22 11:20:20 +01:00
Richard van der Hoff	70e6501913	Merge pull request #3419 from matrix-org/rav/events_per_request ... Log number of events fetched from DB	2018-06-22 11:17:56 +01:00
Richard van der Hoff	0495fe0035	Indirect evt_count updates via method call ... so that we can stub it for the sentinel and not have a billion failing UTs	2018-06-22 10:42:28 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	b088aafcae	Log number of events fetched from DB ... When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]	2018-06-21 06:15:03 +01:00
Richard van der Hoff	3ff8a619f5	UserErasureStore ... to store which users have been erased	2018-06-12 09:52:22 +01:00
Richard van der Hoff	bd348f0af6	remove dead filter_events_for_clients ... This is only used by filter_events_for_client, so we can simplify the whole thing by just doing one user at a time, and removing a dead storage function to boot.	2018-06-12 09:51:31 +01:00
Richard van der Hoff	eb32b2ca20	Optimise state_group_cache update ... (1) matrix-org-hotfixes has removed the intern calls; let's do the same here. (2) remove redundant iteritems() so we can used an optimised db update.	2018-06-11 22:56:11 +01:00
David Baker	187a546bff	Merge pull request #3276 from matrix-org/dbkr/unbind ... Remove email addresses / phone numbers from ID servers when they're removed from synapse	2018-06-11 16:02:00 +01:00
Matthew Hodgson	c96d882a02	Merge branch 'develop' into matthew/filter_members	2018-06-10 12:26:14 +03:00
Richard van der Hoff	d3a8c9c55e	Fix sql error in _get_state_groups_from_groups ... If this was called with a `(type, None)` entry in types (which is supposed to return all state of type `type`), it would explode with a sql error.	2018-06-06 14:19:01 +01:00
Richard van der Hoff	f29b41fde9	Merge pull request #3324 from matrix-org/rav/remove_dead_method ... Remove was_forgotten_at	2018-06-04 18:18:08 +01:00
Erik Johnston	042eedfa2b	Add hacky cache factor override system	2018-06-04 15:39:28 +01:00
Matthew Hodgson	28f09fcdd5	Merge branch 'develop' into matthew/filter_members	2018-06-04 00:09:17 +03:00
Richard van der Hoff	c1f4118bb6	Remove was_forgotten_at ... This is unused. IT MUST DIE!!!1 ̧̪͈̱̹̳͖͙H̵̰̤̰͕̖e̛ ͚͉̗̼̞w̶̩̥͉̮h̩̺̪̩͘ͅọ͎͉̟ ̜̩͔̦̘ͅW̪̫̩̣̲͔̳a͏͔̳͖i͖͜t͓̤̠͓͙s̘̰̩̥̙̝ͅ ̲̠̬̥Be̡̙̫̦h̰̩i̛̫͙͔̭̤̗̲n̳͞d̸ ͎̻͘T̛͇̝̲̹̠̗ͅh̫̦̝ͅe̩̫͟ ͓͖̼W͕̳͎͚̙̥ą̙l̘͚̺͔͞ͅl̳͍̙̤̤̮̳.̢ ̟̺̜̙͉Z̤̲̙̙͎̥̝A͎̣͔̙͘L̥̻̗̳̻̳̳͢G͉̖̯͓̞̩̦O̹̹̺!̙͈͎̞̬ *	2018-06-01 18:21:49 +01:00
Neil Johnson	71477f3317	Merge pull request #3264 from matrix-org/neil/sign-up-stats ... daily user type phone home stats	2018-06-01 13:42:01 +00:00
Richard van der Hoff	9f797a24a4	Handle RRs which arrive before their events	2018-06-01 14:01:43 +01:00
Richard van der Hoff	857e6fd8b6	Ignore depth when updating read-receipts ... Order read receipts by stream ordering instead of depth	2018-06-01 12:18:11 +01:00
Neil Johnson	4986b084f8	remove unnecessary INSERT	2018-06-01 10:50:40 +01:00
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	2018-05-31 19:03:47 +10:00
Amber Brown	872cf43516	Merge pull request #3303 from NotAFile/py3-memoryview ... use memoryview in py3	2018-05-30 12:51:42 +10:00
Amber Brown	debff7ae09	Merge pull request #3281 from NotAFile/py3-six-isinstance ... remaining isintance fixes	2018-05-30 12:44:46 +10:00
Richard van der Hoff	4a9cbdbc15	Exempt AS-registered users from doing gdpr	2018-05-29 19:54:32 +01:00
Neil Johnson	ab0ef31dc7	create users index on creation_ts	2018-05-29 17:51:08 +01:00
Neil Johnson	558f3d376a	create index in background	2018-05-29 17:47:55 +01:00
Neil Johnson	c379acd4fd	bump version	2018-05-29 17:47:28 +01:00
Adrian Tschira	4b9d0cde97	add remaining memoryview changes	2018-05-29 17:42:43 +02:00
Adrian Tschira	1afafb3497	use memoryview in py3 ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-29 17:14:34 +02:00
Matthew Hodgson	a6c8f7c875	add pydoc	2018-05-29 01:09:55 +01:00
Matthew Hodgson	7a6df013cc	merge develop	2018-05-29 00:25:22 +01:00
Amber Brown	a2eb5db4a0	update metrics to be in seconds	2018-05-28 19:10:27 +10:00
Amber Brown	754826a830	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-28 18:57:23 +10:00
Amber Brown	cc1349c06a	Merge pull request #3279 from NotAFile/py3-more-iteritems ... more six iteritems	2018-05-24 14:23:13 -05:00
Adrian Tschira	dd068ca979	remaining isintance fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:55:08 +02:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Adrian Tschira	6c16a4ec1b	more iteritems	2018-05-24 20:19:06 +02:00
Amber Brown	7ea07c7305	Merge pull request #3278 from NotAFile/py3-storage-base ... Py3 storage/_base.py	2018-05-24 13:08:09 -05:00
Amber Brown	1f69693347	Merge pull request #3244 from NotAFile/py3-six-4 ... replace some iteritems with six	2018-05-24 13:04:07 -05:00
Amber Brown	36501068d8	Merge pull request #3247 from NotAFile/py3-misc ... Misc Python3 fixes	2018-05-24 12:58:37 -05:00
Adrian Tschira	095292304f	Py3 storage/_base.py ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 18:24:12 +02:00
David Baker	77a23e2e05	Merge remote-tracking branch 'origin/develop' into dbkr/unbind	2018-05-24 16:20:53 +01:00
David Baker	2c7866d664	Hit the 3pid unbind endpoint on deactivation	2018-05-23 14:38:56 +01:00
Amber Brown	53cc2cde1f	cleanup	2018-05-22 17:32:57 -05:00
Erik Johnston	a17e901f4d	Remove unused string formatting param	2018-05-22 18:24:32 +01:00
Erik Johnston	5494c1d71e	Don't support limitless pagination ... The pagination storage function supported not specifiying a limit on the number of events returned. This was triggered when using the search or context API with a limit of zero, which the storage function took to mean not being limited.	2018-05-22 18:15:21 +01:00
Neil Johnson	d8cb7225d2	daily user type phone home stats	2018-05-22 18:09:09 +01:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Amber Brown	fcc525b0b7	rest of the changes	2018-05-21 19:48:57 -05:00
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:59:26 +02:00
Adrian Tschira	dcc235b47d	use stand-in value if maxint is not available ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:35:44 +02:00
Neil Johnson	644aac5f73	Tighter filtering for user_daily_visits	2018-05-18 17:10:35 +01:00
Neil Johnson	ef466b3a13	fix psql compatability bug	2018-05-18 15:51:21 +01:00
Richard van der Hoff	ed3125b0a1	Merge pull request #3235 from matrix-org/rav/fix_receipts_deferred ... Fix error in handling receipts	2018-05-18 11:23:11 +01:00
Erik Johnston	fa30ac38cc	Merge pull request #3221 from matrix-org/erikj/purge_token ... Make purge_history operate on tokens	2018-05-18 10:35:23 +01:00
Richard van der Hoff	8b1c856d81	Fix error in handling receipts ... Fixes an error which has been happening ever since #2158 (v0.21.0-rc1): > TypeError: argument of type 'ObservableDeferred' is not iterable fixes #3234	2018-05-18 09:15:35 +01:00
Erik Johnston	f7906203f6	Merge pull request #3212 from matrix-org/erikj/epa_stream ... Use stream rather depth ordering for push actions	2018-05-17 12:01:21 +01:00
Erik Johnston	680530cc7f	Clarify comment	2018-05-16 11:47:29 +01:00
Erik Johnston	43e6e82c4d	Comments	2018-05-16 11:13:31 +01:00
Neil Johnson	dc8930ea9e	Merge pull request #3163 from matrix-org/cohort_analytics ... user visit data	2018-05-16 10:09:24 +00:00
Erik Johnston	c945af8799	Move and rename variable	2018-05-16 10:52:06 +01:00
Neil Johnson	be11a02c4f	remove empty line	2018-05-16 10:45:40 +01:00
Neil Johnson	31c2502ca8	style and further contraining query	2018-05-16 09:46:43 +01:00
Richard van der Hoff	8030a825c8	Merge pull request #3213 from matrix-org/rav/consent_handler ... ConsentResource to gather policy consent from users	2018-05-16 07:19:18 +01:00
Neil Johnson	05ac15ae82	Limit query load of generate_user_daily_visits ... The aim is to keep track of when it was last called and only query from that point in time	2018-05-15 17:01:33 +01:00
Erik Johnston	5f27ed75ad	Make purge_history operate on tokens ... As we're soon going to change how topological_ordering works	2018-05-15 16:23:50 +01:00
Erik Johnston	37dbee6490	Use events_to_purge table rather than token	2018-05-15 16:23:47 +01:00
Richard van der Hoff	47815edcfa	ConsentResource to gather policy consent from users ... Hopefully there are enough comments and docs in this that it makes sense on its own.	2018-05-15 15:11:59 +01:00
Neil Johnson	f077e97914	instead of inserting user daily visit data at the end of the day, instead insert incrementally through the day	2018-05-14 13:50:58 +01:00
David Baker	8cbbfd16fb	Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate ... Part user from rooms on account deactivate	2018-05-14 11:31:48 +01:00
Neil Johnson	977765bde2	Merge branch 'develop' of https://github.com/matrix-org/synapse into cohort_analytics	2018-05-14 09:31:42 +01:00
Erik Johnston	6406b70aeb	Use stream rather depth ordering for push actions ... This simplifies things as it is, but will also allow us to change the way we traverse topologically without having to update the way push actions work.	2018-05-11 15:30:11 +01:00
David Baker	217bc53c98	Many docstrings	2018-05-10 12:20:40 +01:00
Erik Johnston	bfe1f73855	Merge pull request #3199 from matrix-org/erikj/pagination_sync ... Refactor sync APIs to reuse pagination API	2018-05-09 16:16:56 +01:00
Erik Johnston	5adb75bcba	Merge pull request #3198 from matrix-org/erikj/fixup_return_pagination ... Refactor get_recent_events_for_room return type	2018-05-09 16:07:14 +01:00
Erik Johnston	7ce98804ff	Fix up comment	2018-05-09 15:42:39 +01:00
Erik Johnston	cddf91c8b9	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_membership_change	2018-05-09 15:32:07 +01:00
Erik Johnston	9896dab8f6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fixup_return_pagination	2018-05-09 15:31:33 +01:00
Erik Johnston	75552d2148	Update comments	2018-05-09 15:15:38 +01:00
David Baker	46df23f581	Add the schema file	2018-05-09 15:07:54 +01:00
David Baker	7e8726b8fb	Part deactivated users in the background ... One room at a time so we don't take out the whole server with leave events, and restart at server restart.	2018-05-09 14:54:28 +01:00
Erik Johnston	c0e08dc45b	Remove unused code path from member change DB func ... The function is never called without a from_key, so we can remove all the handling for that scenario.	2018-05-09 14:31:32 +01:00
Erik Johnston	e2accd7f1d	Refactor sync APIs to reuse pagination API ... The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.	2018-05-09 13:43:39 +01:00
Erik Johnston	e5ab9cd24b	Don't unnecessarily require token to be stream token ... This allows calling the `get_recent_event_ids_for_room` function in more situations.	2018-05-09 11:58:35 +01:00
Erik Johnston	c4af4c24ca	Refactor get_recent_events_for_room return type ... There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs	2018-05-09 11:55:34 +01:00
Erik Johnston	05e0a2462c	Refactor pagination DB API to return concrete type ... This makes it easier to document what is being returned by the storage functions and what some functions expect as arguments.	2018-05-09 11:34:24 +01:00
Erik Johnston	7dd13415db	Remove unused from_token param	2018-05-09 10:58:16 +01:00
Erik Johnston	27cf170558	Refactor recent events func to use pagination func ... This also removes a cache that is unlikely to ever get hit.	2018-05-09 10:55:55 +01:00
Erik Johnston	23ec51c94c	Fix up comments and make function private	2018-05-09 09:55:19 +01:00
Erik Johnston	696f532453	Reuse existing pagination code for context API	2018-05-08 16:20:19 +01:00
Erik Johnston	3e6d306e94	Parse tokens before calling DB function	2018-05-08 16:18:58 +01:00
Erik Johnston	274b8c6025	Only fetch required fields from database	2018-05-08 16:15:25 +01:00
Erik Johnston	06c0d0ed08	Split paginate_room_events storage function	2018-05-08 16:14:26 +01:00
Erik Johnston	0a3b51c420	Merge pull request #3141 from matrix-org/erikj/fixup_state ... Refactor event storage to prepare for changes in state calculations	2018-05-03 10:39:20 +01:00
Erik Johnston	31c7c29d43	Fix up grammar	2018-05-03 10:38:58 +01:00
Neil Johnson	dd1a832419	remove user agent from data model, will just join on user_ips	2018-05-01 12:13:49 +01:00
Neil Johnson	d0857702e8	add inidexes based on usage	2018-05-01 12:12:57 +01:00
Matthew Hodgson	99a54bf2af	Merge pull request #3129 from matrix-org/matthew/fix_group_dups ... remove duplicates from groups tables	2018-04-30 11:47:25 +01:00
Richard van der Hoff	db75c86e84	Merge branch 'develop' into py3-xrange-1	2018-04-30 01:02:25 +01:00
Matthew Hodgson	adaf3ec87f	fix missing import	2018-04-28 22:39:15 +01:00
Matthew Hodgson	006e18b6bb	pep8	2018-04-28 22:32:24 +01:00
Matthew Hodgson	42c89c8215	make it work with sqlite	2018-04-28 22:27:30 +01:00
Adrian Tschira	d82b6ea9e6	Move more xrange to six ... plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:57:00 +02:00
Richard van der Hoff	fc149b4eeb	Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background	2018-04-27 14:31:23 +01:00
Richard van der Hoff	2a13af23bc	Use run_in_background in preference to preserve_fn ... While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.	2018-04-27 12:55:51 +01:00
Richard van der Hoff	9c3da24561	Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions ... Improve exception handling for background processes	2018-04-27 11:47:49 +01:00
Erik Johnston	6e10eed28e	Refactor event storage to not require state ... This is in preparation for using contexts that may or may not have the current_state_ids set. This will allow us to avoid unnecessarily pulling out state for an event on the master process when using workers. We also add a check to see if the state groups of the old extremities are the same as the new ones.	2018-04-27 11:38:02 +01:00
Richard van der Hoff	605defb9e4	Add missing consumeErrors ... In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.	2018-04-27 11:16:28 +01:00
Richard van der Hoff	9255a6cb17	Improve exception handling for background processes ... There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.	2018-04-27 11:07:40 +01:00
Erik Johnston	7ec8e798b4	Fix media admin APIs	2018-04-26 11:31:22 +01:00
Neil Johnson	fb6015d0a6	pep8	2018-04-25 17:56:11 +01:00
Neil Johnson	617bf40924	Generate user daily stats	2018-04-25 17:37:29 +01:00
Erik Johnston	22881b3d69	Also fix reindexing of search	2018-04-25 15:32:04 +01:00
Erik Johnston	ba3166743c	Fix quarantine media admin API	2018-04-25 15:11:18 +01:00
Matthew Hodgson	e3a373f002	remove duplicates from groups tables ... and rename inconsistently named indexes. Based on https://github.com/matrix-org/synapse/pull/3128 - thanks @vurpo\!	2018-04-25 14:58:43 +01:00
Richard van der Hoff	c09a6daf09	Merge pull request #3110 from NotAFile/py3-six-queue ... Replace Queue with six.moves.queue	2018-04-20 15:35:00 +01:00
Richard van der Hoff	bc381d5798	Merge pull request #3117 from matrix-org/rav/refactor_have_events ... Refactor store.have_events	2018-04-20 10:26:12 +01:00
Richard van der Hoff	b1dfbc3c40	Refactor store.have_events ... It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.	2018-04-20 10:25:56 +01:00
Richard van der Hoff	639480e14a	Avoid creating events with huge numbers of prev_events ... In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.	2018-04-16 18:41:37 +01:00
Adrian Tschira	878995e660	Replace Queue with six.moves.queue ... and a six.range change which I missed the last time Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-16 00:46:21 +02:00
Erik Johnston	415aeefd89	Format docstring	2018-04-12 12:07:09 +01:00
Erik Johnston	23a7f9d7f4	Doc we raise on unknown event	2018-04-12 11:20:51 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	92e34615c5	Track where event stream processing have gotten up to	2018-04-11 12:13:40 +01:00
Richard van der Hoff	233699c42e	Merge pull request #2760 from Valodim/pypy ... Synapse on PyPy	2018-04-11 11:20:01 +01:00
Neil Johnson	d54cfbb7a8	fix typo	2018-04-10 17:38:16 +01:00
Richard van der Hoff	d1e56cfcd1	Fix pep8 error on psycopg2cffi hack	2018-04-10 11:35:29 +02:00
Vincent Breitmoser	89de934981	Use psycopg2cffi module instead of psycopg2 if running on pypy ... The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.	2018-04-10 11:29:52 +02:00
Neil Johnson	41e0611895	remove errant print	2018-04-09 18:44:20 +01:00
Neil Johnson	61b439c904	Fix msec to sec, again	2018-04-09 18:43:48 +01:00
Neil Johnson	87770300d5	Fix msec to sec	2018-04-09 18:38:59 +01:00
Neil Johnson	64bc2162ef	Fix psycopg2 interpolation	2018-04-09 17:50:36 +01:00
Luke Barnard	cd087a265d	Don't use redundant inlineCallbacks	2018-04-06 15:43:27 +01:00
Luke Barnard	87c864b698	join_rule -> join_policy	2018-04-06 15:43:27 +01:00
Luke Barnard	ae85c7804e	is_joinable -> join_rule	2018-04-06 15:43:27 +01:00
David Baker	edb45aae38	pep8	2018-04-06 15:43:27 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Neil Johnson	68b0ee4e8d	Merge pull request #3041 from matrix-org/r30_stats ... R30 stats	2018-04-05 15:37:37 +01:00
Richard van der Hoff	22284a6f65	Merge pull request #3060 from matrix-org/rav/kill_event_content ... Remove uses of events.content	2018-04-05 15:02:17 +01:00
Luke Barnard	917380e89d	NON NULL -> NOT NULL	2018-04-05 14:32:12 +01:00
Luke Barnard	700e5e7198	Use DEFAULT join_policy of "invite" in db	2018-04-05 14:01:17 +01:00
Luke Barnard	b214a04ffc	Document set_group_join_policy	2018-04-05 13:29:16 +01:00
Neil Johnson	0e5f479fc0	Review comments ... Use iteritems over item to loop over dict formatting	2018-04-05 12:16:46 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	2fe3f848b9	Remove uses of events.content	2018-03-29 23:17:12 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Neil Johnson	b4e37c6f50	pep8	2018-03-29 17:27:39 +01:00
Neil Johnson	9ee44a372d	Remove need for sqlite specific query	2018-03-29 16:45:34 +01:00
Neil Johnson	dc7c020b33	fix pep8 errors	2018-03-28 17:25:15 +01:00
Neil Johnson	241e4e8687	remove twisted deferral cruft	2018-03-28 16:25:53 +01:00
David Baker	929b34963d	OK, smallint it is then	2018-03-28 14:53:55 +01:00
David Baker	a838444a70	Grr. Copy the definition from is_admin	2018-03-28 14:50:30 +01:00
Neil Johnson	4262aba17b	bump schema version	2018-03-28 14:40:03 +01:00
Neil Johnson	86932be2cb	Support multi client R30 for psql	2018-03-28 14:36:53 +01:00
David Baker	a164270833	Make column definition that works on both dbs	2018-03-28 14:23:00 +01:00
David Baker	352e1ff9ed	Add schema delta file	2018-03-28 14:07:57 +01:00
David Baker	79452edeee	Add joinability for groups ... Adds API to set the 'joinable' flag, and corresponding flag in the table.	2018-03-28 14:03:37 +01:00
Neil Johnson	788e69098c	Add user_ips last seen index	2018-03-28 12:03:13 +01:00
Neil Johnson	0f890f477e	No need to cast in count_daily_users	2018-03-28 11:49:57 +01:00
Neil Johnson	545001b9e4	Fix search_user_dir multiple sqlite versions do different things	2018-03-28 11:19:45 +01:00
Neil Johnson	a32d2548d9	query and call for r30 stats	2018-03-28 10:39:13 +01:00
Neil Johnson	9187e0762f	count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home	2018-03-28 10:02:32 +01:00
Erik Johnston	f879127aaa	Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id ... Linearize calls to _generate_user_id	2018-03-28 10:00:31 +01:00
Erik Johnston	e6d87c93f3	Merge pull request #3030 from matrix-org/erikj/no_ujson ... Remove last usage of ujson	2018-03-28 10:00:06 +01:00
Erik Johnston	800cfd5774	Comment	2018-03-27 13:30:39 +01:00
Erik Johnston	152c2ac19e	Fix indent	2018-03-27 13:13:46 +01:00
Erik Johnston	e70287cff3	Comment	2018-03-27 13:13:38 +01:00
Erik Johnston	3f49e131d9	Add counter metrics for calculating state delta ... This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event.	2018-03-27 10:57:35 +01:00
Erik Johnston	fecb45e0c3	Remove last usage of ujson	2018-03-26 13:32:29 +01:00
Erik Johnston	8d6dc106d1	Don't use _cursor_to_dict in find_next_generated_user_id_localpart	2018-03-26 12:02:44 +01:00
Erik Johnston	fa72803490	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-19 11:41:01 +00:00
Erik Johnston	bf49d2dca8	Replace some ujson with simplejson to make it work	2018-03-16 00:55:44 +00:00
Erik Johnston	a8ce159be4	Replace some ujson with simplejson to make it work	2018-03-16 00:27:09 +00:00
Erik Johnston	5b631ff41a	Remove wrong comment	2018-03-16 00:07:08 +00:00
Erik Johnston	926ba76e23	Replace ujson with simplejson	2018-03-15 23:43:31 +00:00
Matthew Hodgson	056a6df546	Merge branch 'develop' into matthew/filter_members	2018-03-14 15:38:05 +00:00
Erik Johnston	ebd0127999	Merge pull request #2988 from matrix-org/erikj/split_profile_store ... Split up ProfileStore	2018-03-14 09:41:06 +00:00
Erik Johnston	f51565e023	Merge pull request #2993 from matrix-org/erikj/is_blocked ... Add is_blocked to worker store	2018-03-14 09:39:18 +00:00
Matthew Hodgson	9f77001e27	pep8	2018-03-14 00:07:47 +00:00
Matthew Hodgson	d144ed6ffb	fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990) ... Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.	2018-03-13 22:36:04 +00:00
Matthew Hodgson	ccca02846d	make it work	2018-03-13 22:31:41 +00:00
Matthew Hodgson	f0f9a0605b	remove comment now #2969 is fixed	2018-03-13 22:12:15 +00:00
Matthew Hodgson	12350e3f9a	merge proper fix to bug 2969	2018-03-13 22:11:58 +00:00
Matthew Hodgson	afbf4d3dcc	typoe	2018-03-13 19:48:04 +00:00
Matthew Hodgson	865377a70d	disable optimisation for searching for state groups ... when type filter includes wildcards on state_key	2018-03-13 19:46:04 +00:00
Erik Johnston	a08726fc42	Add is_blocked to worker store	2018-03-13 18:28:44 +00:00
Matthew Hodgson	b2aba9e430	build where_clause sanely	2018-03-13 18:13:44 +00:00
Matthew Hodgson	52f7e23c72	PR feedbackz	2018-03-13 18:07:55 +00:00
Matthew Hodgson	1b1c137771	fix bug #2926	2018-03-13 17:52:52 +00:00
Erik Johnston	3518d0ea8f	Split up ProfileStore	2018-03-13 17:36:50 +00:00
Richard van der Hoff	889a2a853a	Add Measure block for persist_events ... This seems like a useful thing to measure.	2018-03-13 10:01:42 +00:00
Matthew Hodgson	fdedcd1f4d	correctly handle None state_keys ... and fix include_other_types thinko	2018-03-12 01:39:06 +00:00
Matthew Hodgson	97c0496cfa	fix sqlite where clause	2018-03-12 00:27:06 +00:00
Matthew Hodgson	8713365265	typos	2018-03-11 20:10:25 +00:00
Matthew Hodgson	9b334b3f97	WIP experiment in lazyloading room members	2018-03-11 20:01:41 +00:00
Erik Johnston	8cb44da4aa	Fix race in sync when joining room ... The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.	2018-03-07 11:55:31 +00:00
Richard van der Hoff	b2932107bb	Merge pull request #2946 from matrix-org/rav/timestamp_to_purge ... Implement purge_history by timestamp	2018-03-06 11:20:23 +00:00
Erik Johnston	7aed50a038	Merge pull request #2948 from matrix-org/erikj/kill_as_sync ... Remove ability for AS users to call /events and /sync	2018-03-06 11:10:09 +00:00
Erik Johnston	b6c4b851f1	Merge pull request #2947 from matrix-org/erikj/split_directory_store ... Split Directory store	2018-03-05 18:17:32 +00:00
Erik Johnston	d4ffe61d4f	Remove ability for AS users to call /events and /sync ... This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144	2018-03-05 15:44:46 +00:00
Erik Johnston	69ce365b79	Fix cache invalidation on deletion	2018-03-05 15:29:03 +00:00
Erik Johnston	2e223163ff	Split Directory store	2018-03-05 15:11:30 +00:00
Richard van der Hoff	f8bfcd7e0d	Provide a means to pass a timestamp to purge_history	2018-03-05 14:37:23 +00:00
Richard van der Hoff	d032785aa7	Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts ... Test and fix find_first_stream_ordering_after_ts	2018-03-05 12:26:14 +00:00
Richard van der Hoff	c818fcab11	Test and fix find_first_stream_ordering_after_ts ... It seemed to suffer from a bunch of off-by-one errors.	2018-03-05 12:04:02 +00:00
Richard van der Hoff	06a14876e5	Add find_first_stream_ordering_after_ts ... Expose this as a public function which can be called outside a txn	2018-03-05 11:53:39 +00:00
Erik Johnston	42174946f8	Merge pull request #2934 from matrix-org/erikj/cache_fix ... Fix bug with delayed cache invalidation stream	2018-03-05 11:33:17 +00:00
Erik Johnston	efb79820b4	Fix bug with delayed cache invalidation stream ... We poked the notifier before updated the current token for the cache invalidation stream. This mean that sometimes the update wouldn't be sent until the next time a cache was invalidated.	2018-03-02 14:45:15 +00:00
Erik Johnston	fafa3e7114	Split registration store	2018-03-02 13:48:27 +00:00
Erik Johnston	1773df0632	Merge pull request #2925 from matrix-org/erikj/split_sig_fed ... Split out SignatureStore and EventFederationStore	2018-03-01 17:32:58 +00:00
Erik Johnston	9e08a93a7b	Merge pull request #2927 from matrix-org/erikj/read_marker_caches ... Improve caching for read_marker API	2018-03-01 17:12:34 +00:00
Erik Johnston	4b44f05f19	Fewer lies are better	2018-03-01 17:08:17 +00:00
Erik Johnston	a83c514d1f	Improve caching for read_marker API ... We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.	2018-03-01 17:08:17 +00:00
Erik Johnston	33bebb63f3	Add some caches to help read marker API	2018-03-01 17:08:17 +00:00
Erik Johnston	2ad4d5b5bb	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed	2018-03-01 16:59:39 +00:00
Erik Johnston	7c371834cc	Stub out broken function only used for cache	2018-03-01 16:44:13 +00:00
Erik Johnston	64346be26d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store	2018-03-01 16:26:42 +00:00
Erik Johnston	22518e2833	Merge pull request #2923 from matrix-org/erikj/stream_ago_worker ... Calculate stream_ordering_month_ago correctly on workers	2018-03-01 16:23:54 +00:00
Erik Johnston	884b26ae41	Remove unused variables	2018-03-01 16:23:48 +00:00
Erik Johnston	1b2af11650	Document abstract class and method better	2018-03-01 16:20:57 +00:00
Erik Johnston	872ff95ed4	Default stream_ordering_*_ago to None	2018-03-01 16:00:05 +00:00
Erik Johnston	22004b524e	Fix comment typo	2018-03-01 15:59:40 +00:00
Erik Johnston	f793bc3877	Split out stream store	2018-03-01 15:13:08 +00:00
Erik Johnston	6411f725be	Calculate stream_ordering_month_ago correctly on workers	2018-03-01 14:20:53 +00:00
Erik Johnston	a9a2d66cdd	Split out SignatureStore and EventFederationStore	2018-03-01 14:17:53 +00:00
Erik Johnston	0c8ba5dd1c	Split up RoomStore	2018-03-01 14:01:19 +00:00
Erik Johnston	493e25d554	Move storage functions for push calculations ... This will allow push actions for an event to be calculated on workers.	2018-02-27 13:58:16 +00:00
Erik Johnston	3594dbc6dc	Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation ... Fix missing invalidations for receipt storage	2018-02-27 11:34:26 +00:00
Erik Johnston	2311189ee4	Merge pull request #2903 from matrix-org/erikj/split_roommember_store ... Split out RoomMemberStore	2018-02-27 11:32:10 +00:00
Erik Johnston	c57607874c	Merge pull request #2901 from matrix-org/erikj/split_as_stores ... Split AS stores	2018-02-27 10:07:07 +00:00
Erik Johnston	8956f0147a	Add comment	2018-02-27 10:06:51 +00:00
Erik Johnston	e5b4a208ce	Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions ... Batch inserts into event_push_actions_staging	2018-02-26 14:45:40 +00:00
Erik Johnston	45b5fe9122	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push	2018-02-26 13:49:24 +00:00
Erik Johnston	d62ce972f8	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store	2018-02-23 11:46:24 +00:00
Erik Johnston	6ae9a3d2a6	Update copyright	2018-02-23 11:44:49 +00:00
Erik Johnston	a90c60912f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions	2018-02-23 11:26:31 +00:00
Erik Johnston	50e8657867	Merge pull request #2902 from matrix-org/erikj/split_events_store ... Split out get_events and co into a worker store	2018-02-23 11:23:52 +00:00
Erik Johnston	1cf9e071dd	Merge pull request #2899 from matrix-org/erikj/split_pushers ... Split PusherStore	2018-02-23 11:23:35 +00:00
Erik Johnston	d0957753bf	Merge pull request #2898 from matrix-org/erikj/split_push_rules_store ... Split PushRulesStore	2018-02-23 11:23:23 +00:00
Erik Johnston	70349872c2	Update copyright	2018-02-23 11:14:35 +00:00
Erik Johnston	eba93b05bf	Split EventsWorkerStore into separate file	2018-02-23 11:01:21 +00:00
Erik Johnston	bf8a36e080	Update copyright	2018-02-23 10:52:10 +00:00
Erik Johnston	5d0f665848	Remove redundant clock	2018-02-23 10:49:58 +00:00
Erik Johnston	3bd760628b	_event_persist_queue shouldn't be in worker store	2018-02-23 10:49:18 +00:00