2906 Commits

Author	SHA1	Message	Date
Neil Johnson	9532caf6ef	remove trailing whiter space	2018-10-24 16:08:25 +01:00
Richard van der Hoff	94a49e0636	fix tuple ... Co-Authored-By: neilisfragile <neil@matrix.org>	2018-10-24 14:39:23 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase ... Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	e0b9d5f0af	Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts ... Clean up the way logcontexts and threads work in the pushers	2018-10-24 09:53:57 +01:00
Erik Johnston	dacbeb2e03	Comment	2018-10-24 09:47:49 +01:00
Erik Johnston	810715f79a	Rename resolve_events_with_factory	2018-10-24 09:44:22 +01:00
Erik Johnston	cb23aa4c42	Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation ... Clean up room alias creation	2018-10-24 09:40:27 +01:00
Neil Johnson	a67d8ace9b	remove errant exception and style	2018-10-23 17:44:39 +01:00
Travis Ralston	43c3f0b02f	Merge pull request #3969 from turt2live/travis/fix-federated-group-requests ... Handle HttpResponseException more safely for federated groups	2018-10-23 10:41:04 -06:00
Travis Ralston	3e704822be	Comments help	2018-10-23 10:25:31 -06:00
Richard van der Hoff	b3f6dddad2	Give some more things logcontexts (#4077)	2018-10-23 14:29:17 +01:00
Richard van der Hoff	5c445114d3	Correctly account for cpu usage by background threads (#4074) ... Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064	2018-10-23 13:12:32 +01:00
Richard van der Hoff	e7a16c6210	Remove redundant run_as_background_process() from pusherpool ... `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.	2018-10-22 16:12:11 +01:00
Erik Johnston	084046456e	Add config option to control alias creation	2018-10-19 10:22:45 +01:00
Erik Johnston	74e7617083	Clean up room alias creation	2018-10-19 10:11:56 +01:00
Travis Ralston	49a044aa5f	Merge branch 'develop' into travis/login-terms	2018-10-18 09:57:58 -06:00
Will Hunt	d6a7797dd1	Fix roomlist since tokens on Python 3 (#4046) ... Thanks @Half-Shot !!!	2018-10-17 23:04:55 +11:00
Neil Johnson	c6584f4b5f	clean up config error logic and imports	2018-10-17 11:36:41 +01:00
Richard van der Hoff	fc0f13dd03	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-16 21:10:04 +01:00
Erik Johnston	15133477ee	Fix up use of resolve_events_with_factory	2018-10-16 16:28:42 +01:00
Travis Ralston	a8ed93a4b5	pep8	2018-10-15 16:10:29 -06:00
Travis Ralston	442734ff9e	Ensure the terms params are actually provided	2018-10-15 14:56:13 -06:00
Travis Ralston	762a0982aa	Python is hard	2018-10-15 14:46:09 -06:00
Travis Ralston	f293d124b6	Merge branch 'develop' into travis/login-terms	2018-10-15 14:44:32 -06:00
David Baker	03c11032c3	Merge pull request #4019 from matrix-org/dbkr/e2e_backups ... E2E backups	2018-10-15 10:19:43 +01:00
Neil Johnson	1ccafb0c5e	no need to join room if creator	2018-10-13 21:14:21 +01:00
Travis Ralston	dd99db846d	Update login terms structure for the proposed language support	2018-10-12 18:03:27 -06:00
Travis Ralston	7ede650956	Merge branch 'develop' into travis/login-terms	2018-10-12 16:24:07 -06:00
Travis Ralston	164f8e4843	isort	2018-10-12 15:11:59 -06:00
Travis Ralston	7bb651de6a	More sane handling of group errors and pep8	2018-10-12 14:53:30 -06:00
Travis Ralston	e3586f7c06	Merge branch 'develop' into travis/fix-federated-group-requests	2018-10-12 14:49:58 -06:00
Neil Johnson	a2bfb778c8	improve auto room join logic, comments and tests	2018-10-12 18:17:36 +01:00
David Baker	8c0ff0287a	Linting soothes the savage PEP8 monster	2018-10-12 13:47:43 +01:00
David Baker	83e72bb2f0	PR feedback pt. 1	2018-10-12 11:26:18 +01:00
Erik Johnston	7e561b5c1a	Add description to counter metric	2018-10-10 11:41:15 +01:00
Erik Johnston	3cbe8331e6	Track number of non-empty sync responses instead	2018-10-10 11:23:17 +01:00
Erik Johnston	395276b405	Append _total to metric and fix up spelling	2018-10-10 09:24:39 +01:00
Erik Johnston	bdc27d6716	Add metric to count lazy member sync requests	2018-10-09 14:17:52 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
David Baker	f4a4dbcad1	Apparently this blank line is Very Important	2018-10-09 09:47:04 +01:00
David Baker	0c905ee015	be python3 compatible	2018-10-09 09:39:13 +01:00
Erik Johnston	6982320572	Remove unnecessary extra function call layer	2018-10-08 14:06:19 +01:00
Erik Johnston	8a1817f0d2	Use errback pattern and catch async failures	2018-10-08 13:29:47 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Neil Johnson	2dadc092b8	move logic into register, fix room alias localpart bug, tests	2018-10-04 17:00:27 +01:00
Travis Ralston	537d0b7b36	Use a flag rather than a new route for the public policy ... This also means that the template now has optional parameters, which will need to be documented somehow.	2018-10-03 17:50:11 -06:00
Travis Ralston	149c4f1765	Supply params for terms auth stage ... As per https://github.com/matrix-org/matrix-doc/pull/1692	2018-10-03 15:57:42 -06:00
Travis Ralston	fd99787162	Incorporate Dave's work for GDPR login flows ... As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117	2018-10-03 15:57:42 -06:00
Erik Johnston	495a9d06bb	Fix exception handling in fetching remote profiles	2018-10-03 11:34:30 +01:00
Richard van der Hoff	8c41b0ca66	Merge pull request #3989 from matrix-org/rav/better_stacktraces ... Avoid reraise, to improve stacktraces	2018-10-02 10:47:38 +01:00
Erik Johnston	bc29946809	Merge pull request #3986 from matrix-org/erikj/fix_sync_with_redacted_state ... Fix lazy loaded sync with rejected state events	2018-10-02 11:38:35 +02:00
Richard van der Hoff	8174c6725b	Avoid reraise, to improve stacktraces	2018-10-01 18:50:34 +01:00
Richard van der Hoff	b5b93f45d5	Merge pull request #3968 from matrix-org/rav/fix_federation_errors ... Fix exceptions when handling incoming transactions	2018-10-01 15:54:24 +01:00
Erik Johnston	82f922b4af	Fix lazy loaded sync with rejected state events ... In particular, we assume that the name and canonical alias events in the state have not been rejected. In practice this may not be the case (though we should probably think about fixing that) so lets ensure that we gracefully handle that case, rather than 404'ing the sync request like we do now.	2018-10-01 14:19:40 +01:00
Matthew Hodgson	23b6a0537f	emit room aliases event	2018-09-29 02:19:37 +01:00
Matthew Hodgson	5b68f29f48	fix thinkos	2018-09-29 02:14:40 +01:00
Matthew Hodgson	8f646f2d04	fix UTs	2018-09-28 15:37:28 +01:00
Richard van der Hoff	3deaad2fb4	Merge pull request #3964 from matrix-org/rav/remove_localhost_checks ... remove spurious federation checks on localhost	2018-09-28 13:35:47 +01:00
Richard van der Hoff	965154d60a	Fix complete fail to do the right thing	2018-09-28 12:45:54 +01:00
Matthew Hodgson	07340cdaca	untested stab at autocreating autojoin rooms	2018-09-28 01:42:53 +01:00
Richard van der Hoff	f094f715cf	Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors	2018-09-27 15:18:21 +01:00
Richard van der Hoff	36c62a67c4	Merge pull request #3794 from matrix-org/erikj/faster_typing ... Improve performance of getting typing updates for replication	2018-09-27 15:14:26 +01:00
Richard van der Hoff	e1e3e77bfd	Merge pull request #3967 from matrix-org/rav/federation_handler_cleanups ... Clarifications in FederationHandler	2018-09-27 15:06:59 +01:00
Amber Brown	a512e637ac	Merge pull request #3970 from schnuffle/develop-py3 ... Replaced all occurences of e.message with str(e)	2018-09-27 23:39:45 +10:00
Schnuffle	dc5db01ff2	Replaced all occurences of e.message with str(e) ... Signed-off-by: Schnuffle <schnuffle@github.com>	2018-09-27 13:38:50 +02:00
Richard van der Hoff	333bee27f5	Include event when resolving state for missing prevs ... If we have a forward extremity for a room as `E`, and you receive `A`, `B`, s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need to do state res between `X` and `E`. When that happens, we need to make sure we include `X` in the state that goes into the state res alg. Fixes #3934.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	bd61c82bdf	Include state from remote servers in pdu handling ... If we've fetched state events from remote servers in order to resolve the state for a new event, we need to actually pass those events into resolve_events_with_factory (so that it can do the state res) and then persist the ones we need - otherwise other bits of the codebase get confused about why we have state groups pointing to non-existent events.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	a215b698c4	Fix "unhashable type: 'list'" exception in federation handling ... get_state_groups returns a map from state_group_id to a list of FrozenEvents, so was very much the wrong thing to be putting as one of the entries in the list passed to resolve_events_with_factory (which expects maps from (event_type, state_key) to event id). We actually want get_state_groups_ids().values() rather than get_state_groups(). This fixes the main problem in #3923, but there are other problems with this bit of code which get discovered once you do so.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	28223841e0	more comments	2018-09-27 11:31:51 +01:00
Richard van der Hoff	e3c159863d	Clarifications in FederationHandler ... * add some comments on things that look a bit bogus * rename this `state` variable to avoid confusion with the `state` used elsewhere in this function. (There was no actual conflict, but it was a confusing bit of spaghetti.)	2018-09-27 11:31:51 +01:00
Richard van der Hoff	92abd3d6d6	Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 ... Logging improvements	2018-09-27 11:27:46 +01:00
Richard van der Hoff	4a15a3e4d5	Include eventid in log lines when processing incoming federation transactions (#3959) ... when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.	2018-09-27 11:25:34 +01:00
Travis Ralston	82fa31799c	Remove debugging statement	2018-09-26 14:01:02 -06:00
Travis Ralston	b4c3bc1734	Handle HttpResponseException more safely for federated groups	2018-09-26 13:48:04 -06:00
Richard van der Hoff	e70b4ce069	Logging improvements ... Some logging tweaks to help with debugging incoming federation transactions	2018-09-26 17:36:14 +01:00
Amber Brown	c2185f14d7	Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu ... Comments and interface cleanup for on_receive_pdu	2018-09-26 21:41:26 +10:00
Matthew Hodgson	787d22ed6c	Only lazy load self-members on initial sync ... Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.	2018-09-25 00:49:26 +01:00
Matthew Hodgson	a2ddaa90f2	Always LL ourselves if we're in a room to simplify clients (#3916) ... Should fix https://github.com/vector-im/riot-web/issues/7209	2018-09-20 21:21:54 +01:00
Richard van der Hoff	703de4ec13	Comments and interface cleanup for on_receive_pdu ... Add some informative comments about what's going on here. Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of `get_missing` and use `sent_to_us_directly` consistently.	2018-09-20 13:06:55 +01:00
Richard van der Hoff	642199570c	Improve the logging when handling a federation transaction (#3904) ... Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.	2018-09-19 17:28:18 +01:00
Erik Johnston	ce846bb620	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing	2018-09-19 15:08:36 +01:00
Amber Brown	f773ecbd61	Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout ... Bump timeout on get_missing_events request	2018-09-19 17:57:48 +10:00
Richard van der Hoff	a219ce8726	Use directory server for room joins (#3899) ... When we do a join, always try the server we used for the alias lookup first. Fixes #2418	2018-09-18 18:27:37 +01:00
Richard van der Hoff	550007cb0e	Bump timeout on get_missing_events request	2018-09-18 15:02:51 +01:00
Richard van der Hoff	286d6930b7	Merge pull request #3879 from matrix-org/matthew/fix-autojoin ... don't ratelimit autojoins	2018-09-18 13:07:01 +01:00
Richard van der Hoff	f75b9961c6	Reinstate missing null check	2018-09-17 16:52:02 +01:00
Richard van der Hoff	85a43f4167	Return a 404 when deleting unknown room alias ... As per https://github.com/matrix-org/matrix-doc/issues/1675 Fixes https://github.com/matrix-org/synapse/issues/2782	2018-09-17 13:19:00 +01:00
Matthew Hodgson	d42d79e3c3	don't ratelimit autojoins	2018-09-15 22:27:41 +01:00
Matthew Hodgson	024be6cf18	don't filter membership events based on history visibility (#3874) ... don't filter membership events based on history visibility as we will already have filtered the messages in the timeline, and state events are always visible. and because @erikjohnston said so.	2018-09-14 18:12:52 +01:00
Matthew Hodgson	2ac1abbc7e	show heroes if a room has a 'deleted' name/canonical_alias (#3851)	2018-09-12 17:11:05 +01:00
Matthew Hodgson	0403cf0783	argh pep8	2018-09-12 16:54:28 +01:00
Matthew Hodgson	0e200e366d	correctly log gappy sync metrics	2018-09-12 16:47:20 +01:00
Matthew Hodgson	11bfc2af1c	fix logline	2018-09-12 16:45:42 +01:00
Matthew Hodgson	b041115415	Speed up lazy loading (#3827) ... * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)	2018-09-12 00:50:39 +01:00
Erik Johnston	f60c9e2a01	Don't send empty tags list down sync	2018-09-06 17:01:41 +01:00
Erik Johnston	7baf66ef5d	Send existing room tags down sync on join ... When a user joined a room any existing tags were not sent down the sync stream. Ordinarily this isn't a problem because the user needs to be in the room to have set tags in it, however synapse will sometimes add tags for a user to a room, e.g. for server notices, which need to come down sync.	2018-09-06 16:46:51 +01:00
Hubert Chathi	3801b8aa03	try to make flake8 and isort happy	2018-09-06 11:35:19 -04:00
Amber Brown	ee7c8bd2b5	Merge pull request #3795 from matrix-org/erikj/faster_sync_state ... User iter* during sync state calculations	2018-09-07 00:24:43 +10:00
Amber Brown	2608ebc04c	Port handlers/ to Python 3 (#3803)	2018-09-07 00:22:23 +10:00
Erik Johnston	7419764351	User iter* during sync state calculations	2018-09-05 16:19:50 +01:00
Erik Johnston	5f02017aea	Improve performance of getting typing updates for replication ... Fetching the list of all new typing notifications involved iterating over all rooms and comparing their serial. Lets move to using a stream change cache, like we do for other streams.	2018-09-05 10:20:40 +01:00
Erik Johnston	567363e497	Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func ... Remove unnecessary resolve_events_with_state_map	2018-09-03 16:19:41 +01:00
Neil Johnson	ea068d6f3c	fix bug where preserved threepid user comes to sign up and server is mau blocked	2018-08-31 10:49:14 +01:00
Matthew Hodgson	b1580f50fe	don't return non-LL-member state in incremental sync state blocks (#3760) ... don't return non-LL-member state in incremental sync state blocks	2018-08-28 23:25:58 +01:00
Hubert Chathi	16a31c6fce	update to newer Synapse APIs	2018-08-24 22:51:25 -04:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	5c261107c9	Remove unnecessary resolve_events_with_state_map ... We only ever used the synchronous resolve_events_with_state_map in one place, which is trivial to replace with the async version.	2018-08-22 15:41:15 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles ... Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	8432e2ebd7	Rename WorkerProfileHandler to BaseProfileHandler	2018-08-22 10:13:40 +01:00
Erik Johnston	a81f140880	Add assert to ensure handler is only run on master	2018-08-22 10:11:21 +01:00
Erik Johnston	47b25ba5f3	Remove redundant vars	2018-08-22 10:09:05 +01:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Erik Johnston	8dee601054	Remove redundant room_version checks	2018-08-20 14:48:53 +01:00
Erik Johnston	e21c368b8b	Revert spurious change	2018-08-20 13:54:51 +01:00
Richard van der Hoff	c144252a8c	Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates ... Fix logcontexts for running pushers	2018-08-17 16:21:49 +01:00
Amber Brown	c334ca67bb	Integrate presence from hotfixes (#3694)	2018-08-18 01:08:45 +10:00
Erik Johnston	ab822a2d1f	Add some fixmes	2018-08-17 15:31:50 +01:00
Erik Johnston	91cdb6de08	Call UserDirectoryHandler methods directly ... Turns out that the user directory handling is fairly racey as a bunch of stuff assumes that the processing happens on master, which it doesn't when there is a synapse.app.user_dir worker. So lets just call the function directly until we actually get round to fixing it, since it doesn't make the situation any worse.	2018-08-17 15:26:13 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	ca87ad1def	Split ProfileHandler into master and worker	2018-08-17 14:15:14 +01:00
Richard van der Hoff	66f7dc8c87	Fix logcontexts for running pushers ... First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.	2018-08-17 00:32:39 +01:00
Neil Johnson	372bf073c1	block event creation and room creation on hitting resource limits	2018-08-16 21:25:16 +01:00
Matthew Hodgson	762a758fea	lazyload aware /messages (#3589)	2018-08-16 14:22:47 +01:00
Matthew Hodgson	3f543dc021	initial cut at a room summary API (#3574)	2018-08-16 09:46:50 +01:00
Neil Johnson	81d727efa9	Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus ... Fix Mau off by one errors	2018-08-15 16:19:41 +00:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568)	2018-08-15 16:35:22 +01:00
Neil Johnson	86a00e05e1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-15 16:27:08 +01:00
Erik Johnston	dc56c47dc0	Merge pull request #3653 from matrix-org/erikj/split_federation ... Move more federation APIs to workers	2018-08-15 14:59:02 +01:00
Neil Johnson	4601129c44	Merge pull request #3687 from matrix-org/neilj/admin_email ... support admin_email config and pass through into blocking errors,	2018-08-15 13:52:25 +00:00
Erik Johnston	ef184caf30	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-15 14:25:46 +01:00
Erik Johnston	488ffe6fdb	Use federation handler function rather than duplicate ... This involves renaming _persist_events to be a public function.	2018-08-15 14:17:18 +01:00
Erik Johnston	fef2e65d12	Merge pull request #3667 from matrix-org/erikj/fixup_unbind ... Don't fail requests to unbind 3pids for non supporting ID servers	2018-08-15 10:32:12 +01:00
Neil Johnson	19b433e3f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-14 17:44:46 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block ... Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Neil Johnson	8f9a7eb58d	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-14 15:11:54 +01:00
Neil Johnson	c74c71128d	remove blank line	2018-08-14 15:06:24 +01:00
Neil Johnson	ed4bc3d2fc	fix off by 1s on mau	2018-08-14 15:04:48 +01:00
Neil Johnson	bd92c8eaa7	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:56:45 +01:00
Neil Johnson	99ebaed8e6	Update register.py ... remove comments	2018-08-14 14:55:55 +01:00
Neil Johnson	9b5bf3d858	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:51:38 +01:00
Neil Johnson	e25d87d97b	Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:32:18 +01:00
Neil Johnson	f4b49152e2	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 21:09:47 +01:00
Neil Johnson	ce7de9ae6b	Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases" ... This reverts commit 0d43f991a1.	2018-08-13 18:06:18 +01:00
Neil Johnson	0d43f991a1	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 18:00:23 +01:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	2018-08-13 16:47:46 +10:00
Matthew Hodgson	72788cf9c1	support DELETE /version with no args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	5e42c45c96	switch get_current_version_info back to being get_version_info	2018-08-12 19:14:31 -04:00
Matthew Hodgson	982edca380	fix flakes	2018-08-12 19:14:31 -04:00
Matthew Hodgson	234611f347	fix typos	2018-08-12 19:14:31 -04:00
Matthew Hodgson	14b3da63a3	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f500cb39e	more docstring for the e2e_room_keys rest	2018-08-12 19:14:31 -04:00
Matthew Hodgson	ca0b052307	fix factoring out of _should_replace_room_key	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cac0253799	rename room_key_version table correctly, and fix opt args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	0abb205b47	blindly incorporate PR review - needs testing & fixing	2018-08-12 19:14:31 -04:00
Matthew Hodgson	69e51c7ba4	make /room_keys/version work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8ae64b270f	implement /room_keys/version too (untested)	2018-08-12 19:14:31 -04:00
Matthew Hodgson	6b8c07abc2	make it work and fix pep8	2018-08-12 19:13:09 -04:00
Matthew Hodgson	0bc4627a73	interim WIP checkin; doesn't build yet	2018-08-12 18:23:10 -04:00
Matthew Hodgson	53ace904b2	total WIP skeleton for /room_keys API	2018-08-12 18:23:10 -04:00
Amber Brown	b37c472419	Rename async to async_helpers because async is a keyword on Python 3.7 (#3678)	2018-08-10 23:50:21 +10:00
Neil Johnson	09cf130898	only block on sync where user is not part of the mau cohort	2018-08-09 17:39:12 +01:00
Erik Johnston	ce6db0e547	Choose state algorithm based on room version	2018-08-09 14:58:47 +01:00
Neil Johnson	69ce057ea6	block sync if auth checks fail	2018-08-09 12:26:27 +01:00
Neil Johnson	a5ef110749	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:40:37 +01:00
Erik Johnston	b179537f2a	Move clean_room_for_join to master	2018-08-09 10:37:38 +01:00
Erik Johnston	484a0ebdfc	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-09 10:16:29 +01:00
Erik Johnston	5785b93711	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-09 10:16:16 +01:00
Erik Johnston	bf7598f582	Log when we 3pid/unbind request fails	2018-08-09 10:09:56 +01:00
Erik Johnston	2bdafaf3c1	Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet ... Add helper base class for generating new replication endpoints	2018-08-09 10:06:23 +01:00
Richard van der Hoff	bb89c84614	Merge pull request #3664 from matrix-org/rav/federation_metrics ... more metrics for the federation and appservice senders	2018-08-08 23:16:58 +01:00
Neil Johnson	e92fb00f32	sync auth blocking	2018-08-08 17:54:49 +01:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions ... Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	990fe9fc23	Merge pull request #3633 from matrix-org/neilj/mau_tracker ... API for monthly_active_users table	2018-08-08 12:44:15 +00:00
Erik Johnston	360ba89c50	Don't fail requests to unbind 3pids for non supporting ID servers ... Older identity servers may not support the unbind 3pid request, so we shouldn't fail the requests if we received one of 400/404/501. The request still fails if we receive e.g. 500 responses, allowing clients to retry requests on transient identity server errors that otherwise do support the API. Fixes #3661	2018-08-08 12:06:18 +01:00
Richard van der Hoff	53bca4690b	more metrics for the federation and appservice senders	2018-08-07 19:09:48 +01:00
Richard van der Hoff	3523f5432a	Don't expose default_room_version as config opt	2018-08-07 12:51:57 +01:00
Erik Johnston	495cb100d1	Allow profile changes to happen on workers	2018-08-07 10:50:26 +01:00
Erik Johnston	a3f5bf79a0	Add EDU/query handling over replication	2018-08-06 15:23:31 +01:00
Erik Johnston	e26dbd82ef	Add replication APIs for persisting federation events	2018-08-06 15:02:28 +01:00
Richard van der Hoff	f900d50824	include known room versions in outgoing make_joins	2018-08-06 13:45:37 +01:00
Neil Johnson	e40a510fbf	py3 fix	2018-08-03 23:19:13 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Will Hunt	16d9701892	Return M_NOT_FOUND when a profile could not be found. (#3596)	2018-08-03 19:08:05 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning ... This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Neil Johnson	897c51d274	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-03 13:40:47 +01:00
Erik Johnston	cb298ff623	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet	2018-08-03 09:25:15 +01:00
Richard van der Hoff	1fa98495d0	Merge pull request #3639 from matrix-org/rav/refactor_error_handling ... Clean up handling of errors from outbound requests	2018-08-02 17:38:24 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	0bf5ec0db7	Check room visibility for /event/ requests ... Make sure that the user has permission to view the requeseted event for /event/{eventId} and /room/{roomId}/event/{eventId} requests. Also check that the event is in the given room for /room/{roomId}/event/{eventId}, for sanity.	2018-08-02 15:03:27 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups ... Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	0a65450d04	Validation for events/rooms in fed requests ... When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)	2018-08-02 13:48:40 +01:00
Erik Johnston	40c1c59cf4	Merge pull request #3621 from matrix-org/erikj/split_fed_store ... Split out DB writes in federation handler	2018-08-02 10:41:42 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits ... Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Richard van der Hoff	01e93f48ed	Kill off MatrixCodeMessageException ... This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.	2018-08-01 16:02:46 +01:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	2018-08-02 00:54:06 +10:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Erik Johnston	a6d7b74915	update docs	2018-08-01 13:39:14 +01:00
Erik Johnston	4b256b9271	_persist_auth_tree no longer returns anything	2018-08-01 13:39:07 +01:00
Neil Johnson	0aba3d361a	count_monthly_users() async	2018-08-01 11:47:58 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Richard van der Hoff	bdbdceeafa	rename replication_layer to federation_client ... I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of classes.	2018-07-31 15:44:05 +01:00
Erik Johnston	443da003bc	Use new helper base class for membership requests	2018-07-31 14:32:23 +01:00
Erik Johnston	729b672823	Use new helper base class for ReplicationSendEventRestServlet	2018-07-31 14:32:23 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Erik Johnston	143f1a2532	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store	2018-07-30 09:56:18 +01:00
Matthew Hodgson	e9b2d047f6	make /context lazyload & filter aware (#3567) ... make /context lazyload & filter aware.	2018-07-27 15:12:50 +01:00
Richard van der Hoff	b0b5566f36	Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid ... if inviter_display_name == ""||None then default to inviter MXID	2018-07-27 10:08:39 +01:00
Matthew Hodgson	a75231b507	Deduplicate redundant lazy-loaded members (#3331) ... * attempt at deduplicating lazy-loaded members as per the proposal; we can deduplicate redundant lazy-loaded members which are sent in the same sync sequence. we do this heuristically rather than requiring the client to somehow tell us which members it has chosen to cache, by instead caching the last N members sent to a client, and not sending them again. For now we hardcode N to 100. Each cache for a given (user,device) tuple is in turn cached for up to X minutes (to avoid the caches building up). For now we hardcode X to 30. * add include_redundant_members filter option & make it work * remove stale todo * add tests for _get_some_state_from_cache * incorporate review	2018-07-26 22:51:30 +01:00
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	2018-07-26 11:48:08 +01:00
Matthew Hodgson	1bcd0490c2	Merge pull request #2970 from matrix-org/matthew/filter_members ... Implement the lazy_load_members room state filter parameter	2018-07-26 00:03:01 +01:00
Travis Ralston	d8e65ed7e1	Fix a minor documentation typo in on_make_leave	2018-07-25 15:44:41 -06:00
Matthew Hodgson	2565804030	Merge branch 'develop' into matthew/filter_members	2018-07-25 17:27:49 +01:00
Erik Johnston	78a691d005	Split out DB writes in federation handler ... This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.	2018-07-25 16:22:56 +01:00
Richard van der Hoff	55acd6856c	Fix updating of cached remote profiles ... _update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when it was called, would just return a generator object, without actually running any of the method body.	2018-07-25 10:34:48 +01:00
Michael Telatynski	87951d3891	Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid	2018-07-24 17:17:46 +01:00
Erik Johnston	8b8c4f34a3	Replace usage of get_current_toke with StreamToken.START ... This allows us to handle /context/ requests on the client_reader worker without having to pull in all the various stream handlers (e.g. precence, typing, pushers etc). The only thing the token gets used for is pagination, and that ignores everything but the room portion of the token.	2018-07-24 16:49:17 +01:00
Richard van der Hoff	a321f78991	Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups ... Fixes and optimisations for resolve_state_groups	2018-07-24 14:11:45 +01:00
Matthew Hodgson	1a01a5b964	clarify comment on p_ids	2018-07-24 14:03:15 +01:00
Matthew Hodgson	eb1d911ab7	rather than adding ll_ids, remove them from p_ids	2018-07-24 13:40:49 +01:00
Matthew Hodgson	d19fba3655	Merge branch 'develop' into matthew/filter_members	2018-07-24 12:39:54 +01:00
Matthew Hodgson	cd241d6bda	incorporate more review	2018-07-24 12:39:40 +01:00
Richard van der Hoff	a678145010	Merge branch 'develop' into rav/logcontext_fixes	2018-07-24 10:43:30 +01:00
Erik Johnston	536bc63a4e	Merge branch 'develop' into erikj/client_apis_move	2018-07-24 09:57:05 +01:00
Richard van der Hoff	cf2d15c6a9	another couple of logcontext leaks	2018-07-24 00:57:48 +01:00
Richard van der Hoff	8dff6e0322	Logcontext fixes ... Fix some random logcontext leaks.	2018-07-24 00:37:17 +01:00
Richard van der Hoff	c1f80effbe	Handle delta_ids being None in _update_context_for_auth_events ... it's easier to create the new state group as a delta from the existing one. (There's an outside chance this will help with https://github.com/matrix-org/synapse/issues/3364)	2018-07-23 22:06:50 +01:00
Matthew Hodgson	adfe29ec0b	Merge branch 'develop' into matthew/filter_members	2018-07-23 19:21:37 +01:00
Matthew Hodgson	254fb430d1	incorporate review	2018-07-23 19:21:20 +01:00
Erik Johnston	0faa3223cd	Fix missing attributes on workers. ... This was missed during the transition from attribute to getter for getting state from context.	2018-07-23 16:28:00 +01:00
Erik Johnston	0b0b24cb82	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move	2018-07-23 13:21:15 +01:00
Erik Johnston	027bc01a1b	Add support for updating state	2018-07-23 13:17:25 +01:00
Erik Johnston	e42510ba63	Use new getters	2018-07-23 13:17:22 +01:00
Amber Brown	3132b89f12	Make the rest of the .iterwhatever go away (#3562)	2018-07-21 15:47:18 +10:00
Erik Johnston	5c88bb722f	Move PaginationHandler to its own file	2018-07-20 15:32:23 +01:00
Erik Johnston	0ecf68aedc	Move check_in_room_or_world_readable to Auth	2018-07-20 15:30:59 +01:00
Richard van der Hoff	7c712f95bb	Combine Limiter and Linearizer ... Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.	2018-07-20 13:11:43 +01:00
Richard van der Hoff	8462c26485	Improvements to the Limiter ... * give them names, to improve logging * use a deque rather than a list for efficiency	2018-07-20 12:50:27 +01:00
Matthew Hodgson	bcaec2915a	incorporate review	2018-07-19 19:03:50 +01:00
Matthew Hodgson	924eb34d94	add a filtered_types param to limit filtering to specific types	2018-07-19 18:32:02 +01:00
Matthew Hodgson	be3adfc331	merge develop pydoc for _get_state_for_groups	2018-07-19 11:26:04 +01:00
Erik Johnston	bacdf0cbf9	Move RoomContextHandler out of Handlers ... This is in preparation for moving GET /context/ to a worker	2018-07-18 15:33:03 +01:00
Erik Johnston	8cb8df55e9	Split MessageHandler into read only and writers ... This will let us call the read only parts from workers, and so be able to move some APIs off of master, e.g. the `/state` API.	2018-07-18 15:33:03 +01:00
Richard van der Hoff	dab00faa83	Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks ... Remove event re-signing hacks	2018-07-18 12:46:27 +01:00
Richard van der Hoff	9c04b4abf9	Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server ... Refactor and optimze filter_events_for_server	2018-07-17 14:01:39 +01:00
Richard van der Hoff	ea69d35651	Move filter_events_for_server out of FederationHandler ... for easier unit testing.	2018-07-16 13:06:24 +01:00
Matthew Hodgson	ba22b6a456	typo	2018-07-13 12:03:39 +01:00
Matthew Hodgson	ea752bdd99	s/becuase/because/g	2018-07-10 17:58:18 +01:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Richard van der Hoff	a4ab491371	Merge branch 'develop' into rav/drop_re_signing_hacks	2018-07-04 07:13:38 +01:00
Erik Johnston	e3b4043800	Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking ... Check the state of prev_events a bit more thoroughly when coming over federation	2018-06-29 13:55:02 +01:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
Amber Brown	99800de63d	try and clean up	2018-06-27 11:40:27 +01:00
Amber Brown	f03a5d1a17	pep8	2018-06-27 11:38:14 +01:00
Amber Brown	a7ecf34b70	cleanups	2018-06-27 11:36:03 +01:00
Amber Brown	77078d6c8e	handle federation not telling us about prev_events	2018-06-27 11:27:32 +01:00
Matthew Hodgson	c7f6b420ae	Merge pull request #3448 from matrix-org/matthew/gdpr-deactivate-admin-api ... add GDPR erase param to deactivate API	2018-06-26 10:43:14 +01:00
Matthew Hodgson	1e788db430	add GDPR erase param to deactivate API	2018-06-26 10:26:54 +01:00
Erik Johnston	d72fb9a448	Merge pull request #3442 from matrix-org/matthew/allow-unconsented-parts ... allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 20:14:18 +01:00
Erik Johnston	df48f7ef37	Actually fix it	2018-06-25 20:03:41 +01:00
Erik Johnston	a0e8a53c6d	Comment	2018-06-25 19:57:38 +01:00
Erik Johnston	7bdc5c8fa3	Fix bug with assuming wrong type	2018-06-25 19:56:02 +01:00
Erik Johnston	ea7a9c0483	Add fast path to _filter_events_for_server ... Most rooms have a trivial history visibility like "shared" or "world_readable", especially large rooms, so lets not bother getting the full membership of those rooms in that case.	2018-06-25 19:49:13 +01:00
Matthew Hodgson	0269367f18	allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 17:56:10 +01:00
Matthew Hodgson	784189b1f4	typos	2018-06-25 17:37:16 +01:00
Matthew Hodgson	6eb861b67f	typo	2018-06-25 17:37:16 +01:00
Erik Johnston	33fdcfa957	Merge pull request #3441 from matrix-org/erikj/redo_erasure ... Fix user erasure and re-enable	2018-06-25 14:37:01 +01:00
Amber Brown	07cad26d65	Remove all global reactor imports & pass it around explicitly (#3424)	2018-06-25 14:08:28 +01:00
Erik Johnston	244484bf3c	Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" ... This reverts commit 1d009013b3.	2018-06-25 13:42:55 +01:00
Richard van der Hoff	1d009013b3	Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" ... This reverts commit ce0d911156, reversing changes made to b4a5d767a9.	2018-06-22 16:35:10 +01:00
Erik Johnston	28ddc6cfbe	Also log number of events for serach context	2018-06-22 15:42:11 +01:00
Erik Johnston	4b4cec3989	Add some logging to search queries	2018-06-22 15:42:11 +01:00
Erik Johnston	ce0d911156	Merge pull request #3431 from matrix-org/rav/erasure_visibility ... Support hiding events from deleted users	2018-06-22 15:06:44 +01:00
Erik Johnston	b4a5d767a9	Merge pull request #3428 from matrix-org/erikj/persisted_pdu ... Simplify get_persisted_pdu	2018-06-22 14:47:43 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	02bfc581f8	Merge pull request #3399 from costacruise/master ... Add error code to room creation error	2018-06-20 17:26:25 +01:00
Michael Wagner	19cd3120ec	Add error code to room creation error ... This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom	2018-06-14 14:08:40 +02:00
Amber Brown	a61738b316	Remove run_on_reactor (#3395)	2018-06-14 18:27:37 +10:00
Michael Telatynski	94700e55fa	if inviter_display_name == ""||None then default to inviter MXID ... to prevent email invite from "None"	2018-06-13 10:31:01 +01:00
Richard van der Hoff	b6faef2ad7	Filter out erased messages ... Redact any messges sent by erased users.	2018-06-12 09:53:18 +01:00
Richard van der Hoff	f1023ebf4b	mark accounts as erased when requested	2018-06-12 09:53:18 +01:00
Richard van der Hoff	9fc5b74b24	simplify get_persisted_pdu ... it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.	2018-06-12 09:51:31 +01:00
David Baker	187a546bff	Merge pull request #3276 from matrix-org/dbkr/unbind ... Remove email addresses / phone numbers from ID servers when they're removed from synapse	2018-06-11 16:02:00 +01:00
Matthew Hodgson	c96d882a02	Merge branch 'develop' into matthew/filter_members	2018-06-10 12:26:14 +03:00
Richard van der Hoff	c6b1441c52	Fix event filtering in get_missing_events handler	2018-06-08 14:15:31 +01:00
Richard van der Hoff	0834b49c6a	Fix event filtering in get_missing_events handler	2018-06-08 11:34:46 +01:00
Richard van der Hoff	8503dd0047	Remove event re-signing hacks ... These "temporary fixes" have been here three and a half years, and I can't find any events in the matrix.org database where the calculated signature differs from what's in the db. It's time for them to go away.	2018-06-07 16:08:29 +01:00
David Baker	d62162bbec	doc fixes	2018-06-05 18:09:13 +01:00
David Baker	6a29e815fc	Fix comment	2018-06-04 12:01:23 +01:00
David Baker	e44150a6de	Missing yield	2018-06-04 12:01:13 +01:00
David Baker	f731e42baf	docstring	2018-06-04 12:00:51 +01:00
Matthew Hodgson	28f09fcdd5	Merge branch 'develop' into matthew/filter_members	2018-06-04 00:09:17 +03:00
Matthew Hodgson	5f6122fe10	more comments	2018-06-04 00:08:52 +03:00
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	2018-05-31 19:03:47 +10:00
Richard van der Hoff	34b85df7f5	Update some comments and docstrings in SyncHandler	2018-05-29 22:31:18 +01:00
Richard van der Hoff	4a9cbdbc15	Exempt AS-registered users from doing gdpr	2018-05-29 19:54:32 +01:00
Matthew Hodgson	a6c8f7c875	add pydoc	2018-05-29 01:09:55 +01:00
Matthew Hodgson	7a6df013cc	merge develop	2018-05-29 00:25:22 +01:00
Amber Brown	754826a830	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-28 18:57:23 +10:00
Richard van der Hoff	41921ac01b	Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room ... Let users leave the server notice room after joining	2018-05-25 11:15:45 +01:00
Richard van der Hoff	757ed27258	Let users leave the server notice room after joining ... They still can't reject invites, but we let them leave it.	2018-05-25 11:07:21 +01:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Amber Brown	1f69693347	Merge pull request #3244 from NotAFile/py3-six-4 ... replace some iteritems with six	2018-05-24 13:04:07 -05:00
David Baker	77a23e2e05	Merge remote-tracking branch 'origin/develop' into dbkr/unbind	2018-05-24 16:20:53 +01:00
David Baker	7a1af504d7	Remove users from user directory on deactivate	2018-05-24 15:59:58 +01:00
Erik Johnston	f72d5a44d5	Merge pull request #3261 from matrix-org/erikj/pagination_fixes ... Fix federation backfill bugs	2018-05-24 14:52:03 +01:00
Erik Johnston	68399fc4de	Merge pull request #3267 from matrix-org/erikj/iter_filter ... Use iter* methods for _filter_events_for_server	2018-05-24 14:44:57 +01:00
David Baker	9700d15611	pep8	2018-05-24 11:23:15 +01:00
David Baker	a21a41bad7	comment	2018-05-24 11:19:59 +01:00
David Baker	b3bff53178	Unbind 3pids when they're deleted too	2018-05-24 11:08:05 +01:00
Richard van der Hoff	a0b3946fe2	Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri	2018-05-23 16:06:03 +01:00
Richard van der Hoff	3ff6f50eac	Use the localpart in the consent uri ... ... because it's shorter.	2018-05-23 15:28:23 +01:00
Richard van der Hoff	82191b08f6	Support for putting %(consent_uri)s in messages ... Make it possible to put the URI in the error message and the server notice that get sent by the server	2018-05-23 15:24:31 +01:00
David Baker	2c7866d664	Hit the 3pid unbind endpoint on deactivation	2018-05-23 14:38:56 +01:00
Erik Johnston	5aaa3189d5	s/values/itervalues/	2018-05-23 10:13:05 +01:00
Erik Johnston	0a4bca4134	Use iter* methods for _filter_events_for_server	2018-05-23 10:04:23 +01:00
Amber Brown	53cc2cde1f	cleanup	2018-05-22 17:32:57 -05:00
Erik Johnston	e85b5a0ff7	Use iter* methods	2018-05-22 19:02:48 +01:00
Erik Johnston	586b66b197	Fix that states is a dict of dicts	2018-05-22 19:02:36 +01:00
Richard van der Hoff	08bfc48abf	custom error code for not leaving server notices room	2018-05-22 17:27:27 +01:00
David Baker	0a078026ea	comment typo	2018-05-22 17:14:06 +01:00
Amber Brown	8f5a688d42	cleanups, self-registration	2018-05-22 10:56:03 -05:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Erik Johnston	cb2a2ad791	get_domains_from_state returns list of tuples	2018-05-22 16:23:39 +01:00
Richard van der Hoff	a5e2941aad	Reject attempts to send event before privacy consent is given ... Returns an M_CONSENT_NOT_GIVEN error (cf https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet given.	2018-05-22 12:00:47 +01:00
Richard van der Hoff	8810685df9	Stub out ServerNoticesSender on the workers ... ... and have the sync endpoints call it directly rather than obsure indirection via PresenceHandler	2018-05-22 11:54:51 +01:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Amber Brown	df9f72d9e5	replacing portions	2018-05-21 19:47:37 -05:00
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:59:26 +02:00
Richard van der Hoff	6d9dc67139	Merge pull request #3232 from matrix-org/rav/server_notices_room ... Infrastructure for a server notices room	2018-05-18 11:28:04 +01:00
Richard van der Hoff	67af392712	Merge pull request #3233 from matrix-org/rav/remove_dead_code ... Remove unused `update_external_syncs`	2018-05-18 11:22:43 +01:00
Richard van der Hoff	26305788fe	Make sure we reject attempts to invite the notices user	2018-05-18 11:18:39 +01:00
Erik Johnston	fa30ac38cc	Merge pull request #3221 from matrix-org/erikj/purge_token ... Make purge_history operate on tokens	2018-05-18 10:35:23 +01:00
Richard van der Hoff	88d3405332	fix missing yield for server_notices_room	2018-05-17 18:33:45 +01:00
Richard van der Hoff	d43d480d86	Remove unused update_external_syncs ... This method isn't used anywhere. Burninate it.	2018-05-17 18:22:19 +01:00
Richard van der Hoff	fed62e21ad	Infrastructure for a server notices room ... Server Notices use a special room which the user can't dismiss. They are created on demand when some other bit of the code calls send_notice. (This doesn't actually do much yet becuse we don't call send_notice anywhere)	2018-05-17 17:58:25 +01:00
Richard van der Hoff	c46367d0d7	Move RoomCreationHandler out of synapse.handlers.Handlers ... Handlers is deprecated nowadays, so let's move this out before I add a new dependency on it. Also fix the docstrings on create_room.	2018-05-17 09:08:42 +01:00
Erik Johnston	5f27ed75ad	Make purge_history operate on tokens ... As we're soon going to change how topological_ordering works	2018-05-15 16:23:50 +01:00
David Baker	8cbbfd16fb	Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate ... Part user from rooms on account deactivate	2018-05-14 11:31:48 +01:00
David Baker	6b49628e3b	Catch failure to part user from room	2018-05-10 12:23:53 +01:00
David Baker	217bc53c98	Many docstrings	2018-05-10 12:20:40 +01:00
David Baker	4d298506dd	Oops, don't call function passed to run_in_background	2018-05-10 11:57:13 +01:00
Erik Johnston	bfe1f73855	Merge pull request #3199 from matrix-org/erikj/pagination_sync ... Refactor sync APIs to reuse pagination API	2018-05-09 16:16:56 +01:00
Erik Johnston	d26bec8a43	Add comment to sync as to why code path is split	2018-05-09 15:56:07 +01:00
Erik Johnston	fcf55f2255	Fix returned token is no longer a tuple	2018-05-09 15:43:00 +01:00
David Baker	294e9a0c9b	Prefix internal functions	2018-05-09 15:10:37 +01:00
David Baker	52281e4c54	Indent fail	2018-05-09 15:06:16 +01:00
David Baker	7e8726b8fb	Part deactivated users in the background ... One room at a time so we don't take out the whole server with leave events, and restart at server restart.	2018-05-09 14:54:28 +01:00
Erik Johnston	e2accd7f1d	Refactor sync APIs to reuse pagination API ... The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.	2018-05-09 13:43:39 +01:00
Erik Johnston	c4af4c24ca	Refactor get_recent_events_for_room return type ... There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs	2018-05-09 11:55:34 +01:00
David Baker	bf98fa0864	Part user from rooms on account deactivate ... This implements this very crudely: this probably isn't viable because parting a user from all their rooms could take a long time, and if the HS gets restarted in that time the process will be aborted.	2018-05-08 15:58:35 +01:00
Richard van der Hoff	ca7211104e	Merge branch 'release-v0.28.1' into develop	2018-05-01 18:16:57 +01:00
Richard van der Hoff	33f469ba19	Apply some limits to depth to counter abuse ... * When creating a new event, cap its depth to 2^63 - 1 * When receiving events, reject any without a sensible depth As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI	2018-05-01 17:54:19 +01:00
Richard van der Hoff	63ae5cbf34	Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn ... Remove redundant call to preserve_fn	2018-04-30 10:23:59 +01:00
Richard van der Hoff	bc2017a594	Merge pull request #3153 from NotAFile/py3-httplib ... move httplib import to six	2018-04-30 01:26:42 +01:00
Richard van der Hoff	683149c1f9	Merge pull request #3151 from NotAFile/py3-xrange-1 ... Move more xrange to six	2018-04-30 01:20:06 +01:00
Richard van der Hoff	3b0e431c82	Merge pull request #3150 from NotAFile/py3-listcomp-yield ... Don't yield in list comprehensions	2018-04-30 01:11:41 +01:00
Richard van der Hoff	db75c86e84	Merge branch 'develop' into py3-xrange-1	2018-04-30 01:02:25 +01:00
Richard van der Hoff	af3cc50511	Remove redundant call to preserve_fn ... submit_event_for_as doesn't return a deferred anyway, so this is pointless.	2018-04-30 00:48:36 +01:00
Adrian Tschira	d82b6ea9e6	Move more xrange to six ... plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:57:00 +02:00
Adrian Tschira	94f4d7f49e	move httplib import to six	2018-04-28 13:43:34 +02:00
Adrian Tschira	cdb4647a80	Don't yield in list comprehensions ... I've tried to grep for more of this with no success. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:36:30 +02:00
Richard van der Hoff	453adf00b6	pep8; remove spurious import	2018-04-27 14:32:08 +01:00
Richard van der Hoff	fc149b4eeb	Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background	2018-04-27 14:31:23 +01:00
Richard van der Hoff	2a13af23bc	Use run_in_background in preference to preserve_fn ... While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.	2018-04-27 12:55:51 +01:00
Richard van der Hoff	41d4b07a53	Merge pull request #3142 from matrix-org/rav/reraise ... reraise exceptions more carefully	2018-04-27 12:16:19 +01:00
Richard van der Hoff	9c3da24561	Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions ... Improve exception handling for background processes	2018-04-27 11:47:49 +01:00
Richard van der Hoff	6493b22b42	reraise exceptions more carefully ... We need to be careful (under python 2, at least) that when we reraise an exception after doing some error handling, we actually reraise the original exception rather than anything that might have been raised (and handled) during the error handling.	2018-04-27 11:40:06 +01:00
Richard van der Hoff	605defb9e4	Add missing consumeErrors ... In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.	2018-04-27 11:16:28 +01:00
Richard van der Hoff	9255a6cb17	Improve exception handling for background processes ... There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.	2018-04-27 11:07:40 +01:00
Richard van der Hoff	77ebef9d43	Merge pull request #3118 from matrix-org/rav/reject_prev_events ... Reject events which have lots of prev_events	2018-04-23 17:51:38 +01:00
Richard van der Hoff	9b9c38373c	Remove spurious param	2018-04-23 12:00:06 +01:00
Richard van der Hoff	8dc4a6144b	Merge pull request #3107 from NotAFile/py3-bool-nonzero ... add __bool__ alias to __nonzero__ methods	2018-04-20 15:43:39 +01:00
Richard van der Hoff	3de7d9fe99	accept stupid events over backfill	2018-04-20 11:41:03 +01:00
Richard van der Hoff	11a67b7c9d	Merge pull request #3093 from matrix-org/rav/response_cache_wrap ... Refactor ResponseCache usage	2018-04-20 11:31:17 +01:00
Richard van der Hoff	bc381d5798	Merge pull request #3117 from matrix-org/rav/refactor_have_events ... Refactor store.have_events	2018-04-20 10:26:12 +01:00
Richard van der Hoff	b1dfbc3c40	Refactor store.have_events ... It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.	2018-04-20 10:25:56 +01:00
Richard van der Hoff	1f4b498b73	Add some comments	2018-04-18 00:15:36 +01:00
Richard van der Hoff	e585228860	Check events on backfill too	2018-04-18 00:06:42 +01:00
Richard van der Hoff	9b7794262f	Reject events which have too many auth_events or prev_events ... ... this should protect us from being dossed by people making silly events (deliberately or otherwise)	2018-04-18 00:06:42 +01:00
Richard van der Hoff	639480e14a	Avoid creating events with huge numbers of prev_events ... In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.	2018-04-16 18:41:37 +01:00
Adrian Tschira	f63ff73c7f	add __bool__ alias to __nonzero__ methods ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:40:47 +02:00
Richard van der Hoff	b78395b7fe	Refactor ResponseCache usage ... Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.	2018-04-12 13:02:15 +01:00
Richard van der Hoff	d5c74b9f6c	Merge pull request #3092 from matrix-org/rav/response_cache_metrics ... Add metrics for ResponseCache	2018-04-12 12:59:36 +01:00
Erik Johnston	19ceb4851f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag	2018-04-12 11:36:07 +01:00
Erik Johnston	f67e906e18	Set all metrics at the same time	2018-04-12 11:18:19 +01:00
Erik Johnston	971059a733	Merge pull request #3088 from matrix-org/erikj/as_parallel ... Send events to ASes concurrently	2018-04-12 10:42:36 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	92e34615c5	Track where event stream processing have gotten up to	2018-04-11 12:13:40 +01:00
Erik Johnston	121591568b	Send events to ASes concurrently	2018-04-11 09:56:00 +01:00
Richard van der Hoff	b3384232a0	Add metrics for ResponseCache	2018-04-10 23:14:47 +01:00
Erik Johnston	eaa2ebf20b	Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends ... Limit concurrent event sends for a room	2018-04-10 16:43:58 +01:00
Erik Johnston	f8e8ec013b	Note why we're limiting concurrent event sends	2018-04-10 14:00:46 +01:00
Richard van der Hoff	a3599dda97	Merge pull request #2996 from krombel/allow_auto_join_rooms ... move handling of auto_join_rooms to RegisterHandler	2018-04-10 01:11:00 +01:00
Richard van der Hoff	d2c6f4d626	Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin ... Return a 404 rather than a 500 on rejoining empty rooms	2018-04-09 17:32:36 +01:00
Richard van der Hoff	f3ef60662f	Return a 404 rather than a 500 on rejoining empty rooms ... Filter ourselves out of the server list before checking for an empty remote host list, to fix 500 error Fixes #2141	2018-04-09 12:56:22 +01:00
Erik Johnston	e5082494eb	Limit concurrent event sends for a room	2018-04-09 12:07:39 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Richard van der Hoff	725a72ec5a	Merge pull request #3000 from NotAFile/change-except-style ... Replace old style error catching with 'as' keyword	2018-04-04 10:45:22 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Richard van der Hoff	9a0db062af	Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors ... Fix error when claiming e2e keys from offline servers	2018-03-28 14:50:38 +01:00
David Baker	79452edeee	Add joinability for groups ... Adds API to set the 'joinable' flag, and corresponding flag in the table.	2018-03-28 14:03:37 +01:00
Krombel	6152e253d8	Merge branch 'develop' of into allow_auto_join_rooms	2018-03-28 14:45:28 +02:00
Richard van der Hoff	a134c572a6	Stringify exceptions for keys/{query,claim} ... Make sure we stringify any exceptions we return from keys/query and keys/claim, to avoid a 'not JSON serializable' error later Fixes #3010	2018-03-27 17:15:06 +01:00
Richard van der Hoff	c2a5cf2fe3	factor out exception handling for keys/claim and keys/query ... this stuff is badly c&p'ed	2018-03-27 17:11:23 +01:00
Erik Johnston	3e0c0660b3	Also do check inside linearizer	2018-03-27 13:01:34 +01:00
Erik Johnston	44cd6e1358	PEP8	2018-03-26 12:06:48 +01:00
Erik Johnston	a052aa42e7	Linearize calls to _generate_user_id	2018-03-26 12:02:20 +01:00
Erik Johnston	fa72803490	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-19 11:41:01 +00:00
Matthew Hodgson	b2f2282947	make lazy_load_members configurable in filters	2018-03-19 01:15:13 +00:00
Matthew Hodgson	478af0f720	reshuffle todo & comments	2018-03-19 01:00:12 +00:00
Matthew Hodgson	366f730bf6	only get member state IDs for incremental syncs if we're filtering	2018-03-18 21:40:43 +00:00
Matthew Hodgson	fc5397fdf5	remove debug	2018-03-16 01:44:55 +00:00
Matthew Hodgson	4f0493c850	fix tsm search again	2018-03-16 01:43:37 +00:00
Matthew Hodgson	f7dcc404f2	add state_ids for timeline entries	2018-03-16 01:37:53 +00:00
Matthew Hodgson	5b3b3aada8	simplify timeline_start_members	2018-03-16 01:17:34 +00:00
Matthew Hodgson	3bc5bd2d22	make incr syncs work	2018-03-16 00:52:04 +00:00
Erik Johnston	926ba76e23	Replace ujson with simplejson	2018-03-15 23:43:31 +00:00
NotAFile	2cc9f76bc3	replace old style error catching with 'as' keyword ... This is both easier to read and compatible with python3 (not that that matters) Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-15 16:11:17 +01:00
Krombel	91ea0202e6	move handling of auto_join_rooms to RegisterHandler ... Currently the handling of auto_join_rooms only works when a user registers itself via public register api. Registrations via registration_shared_secret and ModuleApi do not work This auto_joins the users in the registration handler which enables the auto join feature for all 3 registration paths. This is related to issue #2725 Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-14 16:45:37 +01:00
Matthew Hodgson	056a6df546	Merge branch 'develop' into matthew/filter_members	2018-03-14 15:38:05 +00:00
Erik Johnston	57db62e554	Merge pull request #2992 from matrix-org/erikj/implement_member_workre ... Implement RoomMemberWorkerHandler	2018-03-14 14:29:33 +00:00
Erik Johnston	62ad701326	s/join/joined/ in notify_user_membership_change	2018-03-14 14:17:43 +00:00
Erik Johnston	3f0f06cb31	Split RoomMemberWorkerHandler to separate file	2018-03-14 11:41:45 +00:00
Erik Johnston	3e839e0548	Merge pull request #2989 from matrix-org/erikj/profile_cache_master ... Only update remote profile cache on master	2018-03-14 09:42:27 +00:00
Matthew Hodgson	4d0cfef6ee	add copyright to nudge CI	2018-03-14 00:02:20 +00:00
Matthew Hodgson	c9d72e4571	oops	2018-03-13 23:46:45 +00:00
Matthew Hodgson	ccca02846d	make it work	2018-03-13 22:31:41 +00:00
Matthew Hodgson	12350e3f9a	merge proper fix to bug 2969	2018-03-13 22:11:58 +00:00
Matthew Hodgson	14a9d2f73d	ensure we always include the members for a given timeline block	2018-03-13 22:03:42 +00:00
Erik Johnston	b27320b550	Implement RoomMemberWorkerHandler	2018-03-13 18:26:00 +00:00
Erik Johnston	350331d466	_remote_join and co take a requester	2018-03-13 17:50:39 +00:00
Erik Johnston	1a69c6d590	Merge pull request #2987 from matrix-org/erikj/split_room_member_handler ... Split RoomMemberHandler into base and master class	2018-03-13 17:40:00 +00:00
Erik Johnston	df8ff682a7	Only update remote profile cache on master	2018-03-13 17:38:21 +00:00
Erik Johnston	6dbebef141	Add missing param to docstrings	2018-03-13 17:15:32 +00:00
Erik Johnston	16adb11cc0	Correct import order	2018-03-13 16:57:07 +00:00
Erik Johnston	82f16faa78	Move user_*_room distributor stuff to master class ... I added yields when calling user_left_room, but they shouldn't matter on the master process as they always return None anyway.	2018-03-13 16:38:15 +00:00
Erik Johnston	b78717b87b	Split RoomMemberHandler into base and master class ... The intention here is to split the class into the bits that can be done on workers and the bits that have to be done on the master. In future there will also be a class that can be run on the worker, which will delegate work to the master when necessary.	2018-03-13 16:37:41 +00:00
Erik Johnston	95cb401ae0	Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer ... Remove ReplicationLayer and user Client/Server directly	2018-03-13 15:45:08 +00:00
Erik Johnston	5d8476d8ff	Merge pull request #2981 from matrix-org/erikj/factor_remote_leave ... Factor out _remote_reject_invite in RoomMember	2018-03-13 15:44:56 +00:00
Erik Johnston	56e709857c	Merge pull request #2979 from matrix-org/erikj/no_handlers ... Don't build handlers on workers unnecessarily	2018-03-13 13:46:38 +00:00
Erik Johnston	cb9f8e527c	s/replication_client/federation_client/	2018-03-13 13:26:52 +00:00
Erik Johnston	bf8e97bd3c	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave	2018-03-13 13:17:08 +00:00
Erik Johnston	ea3442c15c	Add docstring	2018-03-13 13:16:21 +00:00