3531 Commits

Author	SHA1	Message	Date
Travis Ralston	d1e7b9c44c	Merge branch 'develop' into travis/login-terms	2018-10-31 13:15:14 -06:00
Richard van der Hoff	1729ba1650	Merge pull request #4101 from matrix-org/rav/aliases_for_upgrades ... Attempt to move room aliases on room upgrades	2018-10-31 17:52:18 +00:00
Richard van der Hoff	0f8591a5a8	Avoid else clause on exception for clarity	2018-10-31 15:43:57 +00:00
Richard van der Hoff	94c7fadc98	Attempt to move room aliases on room upgrades	2018-10-31 15:43:57 +00:00
Richard van der Hoff	9b827c40ca	Log some bits about event creation (#4121) ... I found these helpful in debugging my room upgrade tests.	2018-10-31 15:42:23 +00:00
Amber Brown	086e1a8f3e	Fix search 500ing (#4122)	2018-10-31 06:26:46 +11:00
Amber Brown	3bade14ec0	Fix search 500ing (#4122)	2018-10-31 04:33:41 +11:00
Amber Brown	4cd1c9f2ff	Delete the disused & unspecced identicon functionality (#4106)	2018-10-29 23:57:24 +11:00
Richard van der Hoff	7fbfea062e	Merge pull request #4100 from matrix-org/rav/room_upgrade_avatar ... Remember to copy the avatar on room upgrades	2018-10-29 12:49:21 +00:00
Richard van der Hoff	db24d7f15e	Better handling of odd PLs during room upgrades ... Fixes handling of rooms where we have permission to send the tombstone, but not other state. We need to (a) fail more gracefully when we can't send the PLs in the old room, and (b) not set the PLs in the new room until we are done with the other stuff.	2018-10-27 00:54:26 +01:00
Richard van der Hoff	5caf79b312	Remember to copy the avatar on room upgrades	2018-10-26 23:56:40 +01:00
Richard van der Hoff	54bbe71867	optimise state copying	2018-10-26 22:51:34 +01:00
Richard van der Hoff	193cadc988	Address review comments ... Improve comments, get old room state from the context we already have	2018-10-26 17:10:30 +01:00
Richard van der Hoff	e6babc27d5	restrict PLs in old room	2018-10-25 19:18:25 +01:00
Richard van der Hoff	3a263bf3ae	copy state	2018-10-25 19:18:25 +01:00
Richard van der Hoff	1b9f253e20	preserve PLs	2018-10-25 19:10:24 +01:00
Richard van der Hoff	4cda300058	preserve room visibility	2018-10-25 19:10:24 +01:00
Richard van der Hoff	0f7d1c9906	Basic initial support for room upgrades ... Currently just creates a new, empty, room, and sends a tombstone in the old room.	2018-10-25 19:10:24 +01:00
Richard van der Hoff	e1948175ee	Allow power_level_content_override=None for _send_events_for_new_room	2018-10-25 19:10:24 +01:00
Richard van der Hoff	7f7b2cd3de	Make room_member_handler a member of RoomCreationHandler ... ... to save passing it into `_send_events_for_new_room`	2018-10-25 19:10:18 +01:00
Richard van der Hoff	871c4abfec	Factor _generate_room_id out of create_room ... we're going to need this for room upgrades.	2018-10-25 18:23:09 +01:00
Erik Johnston	cb53ce9d64	Refactor state group lookup to reduce DB hits (#4011) ... Currently when fetching state groups from the data store we make two hits two the database: once for members and once for non-members (unless request is filtered to one or the other). This adds needless load to the datbase, so this PR refactors the lookup to make only a single database hit.	2018-10-25 17:49:55 +01:00
Erik Johnston	b94a43d5b5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/alias_disallow_list	2018-10-25 15:25:31 +01:00
Neil Johnson	f7f487e14c	Merge branch 'develop' of github.com:matrix-org/synapse into matthew/autocreate_autojoin	2018-10-25 14:40:06 +01:00
Travis Ralston	54def42c19	Merge branch 'develop' into travis/login-terms	2018-10-24 13:22:59 -06:00
Neil Johnson	9532caf6ef	remove trailing whiter space	2018-10-24 16:08:25 +01:00
Richard van der Hoff	94a49e0636	fix tuple ... Co-Authored-By: neilisfragile <neil@matrix.org>	2018-10-24 14:39:23 +01:00
Erik Johnston	3904cbf307	Merge pull request #4040 from matrix-org/erikj/states_res_v2_rebase ... Add v2 state resolution algorithm	2018-10-24 11:12:12 +01:00
Richard van der Hoff	e0b9d5f0af	Merge pull request #4075 from matrix-org/rav/fix_pusher_logcontexts ... Clean up the way logcontexts and threads work in the pushers	2018-10-24 09:53:57 +01:00
Erik Johnston	dacbeb2e03	Comment	2018-10-24 09:47:49 +01:00
Erik Johnston	810715f79a	Rename resolve_events_with_factory	2018-10-24 09:44:22 +01:00
Erik Johnston	cb23aa4c42	Merge pull request #4063 from matrix-org/erikj/cleaup_alias_creation ... Clean up room alias creation	2018-10-24 09:40:27 +01:00
Neil Johnson	a67d8ace9b	remove errant exception and style	2018-10-23 17:44:39 +01:00
Travis Ralston	43c3f0b02f	Merge pull request #3969 from turt2live/travis/fix-federated-group-requests ... Handle HttpResponseException more safely for federated groups	2018-10-23 10:41:04 -06:00
Travis Ralston	3e704822be	Comments help	2018-10-23 10:25:31 -06:00
Richard van der Hoff	b3f6dddad2	Give some more things logcontexts (#4077)	2018-10-23 14:29:17 +01:00
Richard van der Hoff	5c445114d3	Correctly account for cpu usage by background threads (#4074) ... Wrap calls to deferToThread() in a thing which uses a child logcontext to attribute CPU usage to the right request. While we're in the area, remove the logcontext_tracer stuff, which is never used, and afaik doesn't work. Fixes #4064	2018-10-23 13:12:32 +01:00
Richard van der Hoff	e7a16c6210	Remove redundant run_as_background_process() from pusherpool ... `on_new_notifications` and `on_new_receipts` in `HttpPusher` and `EmailPusher` now always return synchronously, so we can remove the `defer.gatherResults` on their results, and the `run_as_background_process` wrappers can be removed too because the PusherPool methods will now complete quickly enough.	2018-10-22 16:12:11 +01:00
Erik Johnston	084046456e	Add config option to control alias creation	2018-10-19 10:22:45 +01:00
Erik Johnston	74e7617083	Clean up room alias creation	2018-10-19 10:11:56 +01:00
Travis Ralston	49a044aa5f	Merge branch 'develop' into travis/login-terms	2018-10-18 09:57:58 -06:00
Will Hunt	d6a7797dd1	Fix roomlist since tokens on Python 3 (#4046) ... Thanks @Half-Shot !!!	2018-10-17 23:04:55 +11:00
Neil Johnson	c6584f4b5f	clean up config error logic and imports	2018-10-17 11:36:41 +01:00
Richard van der Hoff	fc0f13dd03	Fix incorrect truncation in get_missing_events ... It's quite important that get_missing_events returns the *latest* events in the room; however we were pulling event ids out of the database until we got *at least* 10, and then taking the *earliest* of the results. We also shouldn't really be relying on depth, and should be checking the room_id.	2018-10-16 21:10:04 +01:00
Erik Johnston	15133477ee	Fix up use of resolve_events_with_factory	2018-10-16 16:28:42 +01:00
Travis Ralston	a8ed93a4b5	pep8	2018-10-15 16:10:29 -06:00
Travis Ralston	442734ff9e	Ensure the terms params are actually provided	2018-10-15 14:56:13 -06:00
Travis Ralston	762a0982aa	Python is hard	2018-10-15 14:46:09 -06:00
Travis Ralston	f293d124b6	Merge branch 'develop' into travis/login-terms	2018-10-15 14:44:32 -06:00
David Baker	03c11032c3	Merge pull request #4019 from matrix-org/dbkr/e2e_backups ... E2E backups	2018-10-15 10:19:43 +01:00
Neil Johnson	1ccafb0c5e	no need to join room if creator	2018-10-13 21:14:21 +01:00
Travis Ralston	dd99db846d	Update login terms structure for the proposed language support	2018-10-12 18:03:27 -06:00
Travis Ralston	7ede650956	Merge branch 'develop' into travis/login-terms	2018-10-12 16:24:07 -06:00
Travis Ralston	164f8e4843	isort	2018-10-12 15:11:59 -06:00
Travis Ralston	7bb651de6a	More sane handling of group errors and pep8	2018-10-12 14:53:30 -06:00
Travis Ralston	e3586f7c06	Merge branch 'develop' into travis/fix-federated-group-requests	2018-10-12 14:49:58 -06:00
Neil Johnson	a2bfb778c8	improve auto room join logic, comments and tests	2018-10-12 18:17:36 +01:00
David Baker	8c0ff0287a	Linting soothes the savage PEP8 monster	2018-10-12 13:47:43 +01:00
David Baker	83e72bb2f0	PR feedback pt. 1	2018-10-12 11:26:18 +01:00
Erik Johnston	7e561b5c1a	Add description to counter metric	2018-10-10 11:41:15 +01:00
Erik Johnston	3cbe8331e6	Track number of non-empty sync responses instead	2018-10-10 11:23:17 +01:00
Erik Johnston	395276b405	Append _total to metric and fix up spelling	2018-10-10 09:24:39 +01:00
Erik Johnston	bdc27d6716	Add metric to count lazy member sync requests	2018-10-09 14:17:52 +01:00
David Baker	dc045ef202	Merge remote-tracking branch 'origin/develop' into dbkr/e2e_backups	2018-10-09 10:05:02 +01:00
David Baker	f4a4dbcad1	Apparently this blank line is Very Important	2018-10-09 09:47:04 +01:00
David Baker	0c905ee015	be python3 compatible	2018-10-09 09:39:13 +01:00
Erik Johnston	6982320572	Remove unnecessary extra function call layer	2018-10-08 14:06:19 +01:00
Erik Johnston	8a1817f0d2	Use errback pattern and catch async failures	2018-10-08 13:29:47 +01:00
Neil Johnson	ed82043efb	Merge branch 'develop' into matthew/autocreate_autojoin	2018-10-04 17:26:59 +01:00
Neil Johnson	2dadc092b8	move logic into register, fix room alias localpart bug, tests	2018-10-04 17:00:27 +01:00
Travis Ralston	537d0b7b36	Use a flag rather than a new route for the public policy ... This also means that the template now has optional parameters, which will need to be documented somehow.	2018-10-03 17:50:11 -06:00
Travis Ralston	149c4f1765	Supply params for terms auth stage ... As per https://github.com/matrix-org/matrix-doc/pull/1692	2018-10-03 15:57:42 -06:00
Travis Ralston	fd99787162	Incorporate Dave's work for GDPR login flows ... As per https://github.com/vector-im/riot-web/issues/7168#issuecomment-419996117	2018-10-03 15:57:42 -06:00
Erik Johnston	495a9d06bb	Fix exception handling in fetching remote profiles	2018-10-03 11:34:30 +01:00
Richard van der Hoff	8c41b0ca66	Merge pull request #3989 from matrix-org/rav/better_stacktraces ... Avoid reraise, to improve stacktraces	2018-10-02 10:47:38 +01:00
Erik Johnston	bc29946809	Merge pull request #3986 from matrix-org/erikj/fix_sync_with_redacted_state ... Fix lazy loaded sync with rejected state events	2018-10-02 11:38:35 +02:00
Richard van der Hoff	8174c6725b	Avoid reraise, to improve stacktraces	2018-10-01 18:50:34 +01:00
Richard van der Hoff	b5b93f45d5	Merge pull request #3968 from matrix-org/rav/fix_federation_errors ... Fix exceptions when handling incoming transactions	2018-10-01 15:54:24 +01:00
Erik Johnston	82f922b4af	Fix lazy loaded sync with rejected state events ... In particular, we assume that the name and canonical alias events in the state have not been rejected. In practice this may not be the case (though we should probably think about fixing that) so lets ensure that we gracefully handle that case, rather than 404'ing the sync request like we do now.	2018-10-01 14:19:40 +01:00
Matthew Hodgson	23b6a0537f	emit room aliases event	2018-09-29 02:19:37 +01:00
Matthew Hodgson	5b68f29f48	fix thinkos	2018-09-29 02:14:40 +01:00
Matthew Hodgson	8f646f2d04	fix UTs	2018-09-28 15:37:28 +01:00
Richard van der Hoff	3deaad2fb4	Merge pull request #3964 from matrix-org/rav/remove_localhost_checks ... remove spurious federation checks on localhost	2018-09-28 13:35:47 +01:00
Richard van der Hoff	965154d60a	Fix complete fail to do the right thing	2018-09-28 12:45:54 +01:00
Matthew Hodgson	07340cdaca	untested stab at autocreating autojoin rooms	2018-09-28 01:42:53 +01:00
Richard van der Hoff	f094f715cf	Merge remote-tracking branch 'origin/develop' into rav/fix_federation_errors	2018-09-27 15:18:21 +01:00
Richard van der Hoff	36c62a67c4	Merge pull request #3794 from matrix-org/erikj/faster_typing ... Improve performance of getting typing updates for replication	2018-09-27 15:14:26 +01:00
Richard van der Hoff	e1e3e77bfd	Merge pull request #3967 from matrix-org/rav/federation_handler_cleanups ... Clarifications in FederationHandler	2018-09-27 15:06:59 +01:00
Amber Brown	a512e637ac	Merge pull request #3970 from schnuffle/develop-py3 ... Replaced all occurences of e.message with str(e)	2018-09-27 23:39:45 +10:00
Schnuffle	dc5db01ff2	Replaced all occurences of e.message with str(e) ... Signed-off-by: Schnuffle <schnuffle@github.com>	2018-09-27 13:38:50 +02:00
Richard van der Hoff	333bee27f5	Include event when resolving state for missing prevs ... If we have a forward extremity for a room as `E`, and you receive `A`, `B`, s.t. `A -> B -> E`, and `B` also points to an unknown event `X`, then we need to do state res between `X` and `E`. When that happens, we need to make sure we include `X` in the state that goes into the state res alg. Fixes #3934.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	bd61c82bdf	Include state from remote servers in pdu handling ... If we've fetched state events from remote servers in order to resolve the state for a new event, we need to actually pass those events into resolve_events_with_factory (so that it can do the state res) and then persist the ones we need - otherwise other bits of the codebase get confused about why we have state groups pointing to non-existent events.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	a215b698c4	Fix "unhashable type: 'list'" exception in federation handling ... get_state_groups returns a map from state_group_id to a list of FrozenEvents, so was very much the wrong thing to be putting as one of the entries in the list passed to resolve_events_with_factory (which expects maps from (event_type, state_key) to event id). We actually want get_state_groups_ids().values() rather than get_state_groups(). This fixes the main problem in #3923, but there are other problems with this bit of code which get discovered once you do so.	2018-09-27 11:37:39 +01:00
Richard van der Hoff	28223841e0	more comments	2018-09-27 11:31:51 +01:00
Richard van der Hoff	e3c159863d	Clarifications in FederationHandler ... * add some comments on things that look a bit bogus * rename this `state` variable to avoid confusion with the `state` used elsewhere in this function. (There was no actual conflict, but it was a confusing bit of spaghetti.)	2018-09-27 11:31:51 +01:00
Richard van der Hoff	92abd3d6d6	Merge pull request #3966 from matrix-org/rav/rx_txn_logging_2 ... Logging improvements	2018-09-27 11:27:46 +01:00
Richard van der Hoff	4a15a3e4d5	Include eventid in log lines when processing incoming federation transactions (#3959) ... when processing incoming transactions, it can be hard to see what's going on, because we process a bunch of stuff in parallel, and because we may end up recursively working our way through a chain of three or four events. This commit creates a way to use logcontexts to add the relevant event ids to the log lines.	2018-09-27 11:25:34 +01:00
Travis Ralston	82fa31799c	Remove debugging statement	2018-09-26 14:01:02 -06:00
Travis Ralston	b4c3bc1734	Handle HttpResponseException more safely for federated groups	2018-09-26 13:48:04 -06:00
Richard van der Hoff	e70b4ce069	Logging improvements ... Some logging tweaks to help with debugging incoming federation transactions	2018-09-26 17:36:14 +01:00
Amber Brown	c2185f14d7	Merge pull request #3924 from matrix-org/rav/clean_up_on_receive_pdu ... Comments and interface cleanup for on_receive_pdu	2018-09-26 21:41:26 +10:00
Matthew Hodgson	787d22ed6c	Only lazy load self-members on initial sync ... Given we have disabled lazy loading for incr syncs in #3840, we can make self-LL more efficient by only doing it on initial sync. Also adds a bounds check for if/when we change our mind, so that we don't try to include LL members on sync responses with no timeline.	2018-09-25 00:49:26 +01:00
Matthew Hodgson	a2ddaa90f2	Always LL ourselves if we're in a room to simplify clients (#3916) ... Should fix https://github.com/vector-im/riot-web/issues/7209	2018-09-20 21:21:54 +01:00
Richard van der Hoff	703de4ec13	Comments and interface cleanup for on_receive_pdu ... Add some informative comments about what's going on here. Also, `sent_to_us_directly` and `get_missing` were doing the same thing (apart from in `_handle_queued_pdus`, which looks like a bug), so let's get rid of `get_missing` and use `sent_to_us_directly` consistently.	2018-09-20 13:06:55 +01:00
Richard van der Hoff	642199570c	Improve the logging when handling a federation transaction (#3904) ... Let's try to rationalise the logging that happens when we are processing an incoming transaction, to make it easier to figure out what is going wrong when they take ages. In particular: - make everything start with a [room_id event_id] prefix - make sure we log a warning when catching exceptions rather than just turning them into other, more cryptic, exceptions.	2018-09-19 17:28:18 +01:00
Erik Johnston	ce846bb620	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing	2018-09-19 15:08:36 +01:00
Amber Brown	f773ecbd61	Merge pull request #3903 from matrix-org/rav/increase_get_missing_events_timeout ... Bump timeout on get_missing_events request	2018-09-19 17:57:48 +10:00
Richard van der Hoff	a219ce8726	Use directory server for room joins (#3899) ... When we do a join, always try the server we used for the alias lookup first. Fixes #2418	2018-09-18 18:27:37 +01:00
Richard van der Hoff	550007cb0e	Bump timeout on get_missing_events request	2018-09-18 15:02:51 +01:00
Richard van der Hoff	286d6930b7	Merge pull request #3879 from matrix-org/matthew/fix-autojoin ... don't ratelimit autojoins	2018-09-18 13:07:01 +01:00
Richard van der Hoff	f75b9961c6	Reinstate missing null check	2018-09-17 16:52:02 +01:00
Richard van der Hoff	85a43f4167	Return a 404 when deleting unknown room alias ... As per https://github.com/matrix-org/matrix-doc/issues/1675 Fixes https://github.com/matrix-org/synapse/issues/2782	2018-09-17 13:19:00 +01:00
Matthew Hodgson	d42d79e3c3	don't ratelimit autojoins	2018-09-15 22:27:41 +01:00
Matthew Hodgson	024be6cf18	don't filter membership events based on history visibility (#3874) ... don't filter membership events based on history visibility as we will already have filtered the messages in the timeline, and state events are always visible. and because @erikjohnston said so.	2018-09-14 18:12:52 +01:00
Matthew Hodgson	2ac1abbc7e	show heroes if a room has a 'deleted' name/canonical_alias (#3851)	2018-09-12 17:11:05 +01:00
Matthew Hodgson	0403cf0783	argh pep8	2018-09-12 16:54:28 +01:00
Matthew Hodgson	0e200e366d	correctly log gappy sync metrics	2018-09-12 16:47:20 +01:00
Matthew Hodgson	11bfc2af1c	fix logline	2018-09-12 16:45:42 +01:00
Matthew Hodgson	b041115415	Speed up lazy loading (#3827) ... * speed up room summaries by pulling their data from room_memberships rather than room state * disable LL for incr syncs, and log incr sync stats (#3840)	2018-09-12 00:50:39 +01:00
Erik Johnston	f60c9e2a01	Don't send empty tags list down sync	2018-09-06 17:01:41 +01:00
Erik Johnston	7baf66ef5d	Send existing room tags down sync on join ... When a user joined a room any existing tags were not sent down the sync stream. Ordinarily this isn't a problem because the user needs to be in the room to have set tags in it, however synapse will sometimes add tags for a user to a room, e.g. for server notices, which need to come down sync.	2018-09-06 16:46:51 +01:00
Hubert Chathi	3801b8aa03	try to make flake8 and isort happy	2018-09-06 11:35:19 -04:00
Amber Brown	ee7c8bd2b5	Merge pull request #3795 from matrix-org/erikj/faster_sync_state ... User iter* during sync state calculations	2018-09-07 00:24:43 +10:00
Amber Brown	2608ebc04c	Port handlers/ to Python 3 (#3803)	2018-09-07 00:22:23 +10:00
Erik Johnston	7419764351	User iter* during sync state calculations	2018-09-05 16:19:50 +01:00
Erik Johnston	5f02017aea	Improve performance of getting typing updates for replication ... Fetching the list of all new typing notifications involved iterating over all rooms and comparing their serial. Lets move to using a stream change cache, like we do for other streams.	2018-09-05 10:20:40 +01:00
Erik Johnston	567363e497	Merge pull request #3737 from matrix-org/erikj/remove_redundant_state_func ... Remove unnecessary resolve_events_with_state_map	2018-09-03 16:19:41 +01:00
Neil Johnson	ea068d6f3c	fix bug where preserved threepid user comes to sign up and server is mau blocked	2018-08-31 10:49:14 +01:00
Matthew Hodgson	b1580f50fe	don't return non-LL-member state in incremental sync state blocks (#3760) ... don't return non-LL-member state in incremental sync state blocks	2018-08-28 23:25:58 +01:00
Hubert Chathi	16a31c6fce	update to newer Synapse APIs	2018-08-24 22:51:25 -04:00
Hubert Chathi	83caead95a	Merge branch 'develop' into e2e_backups	2018-08-24 11:44:26 -04:00
Erik Johnston	5c261107c9	Remove unnecessary resolve_events_with_state_map ... We only ever used the synchronous resolve_events_with_state_map in one place, which is trivial to replace with the async version.	2018-08-22 15:41:15 +01:00
Erik Johnston	764030cf63	Merge pull request #3659 from matrix-org/erikj/split_profiles ... Allow profile updates to happen on workers	2018-08-22 11:35:55 +01:00
Erik Johnston	8432e2ebd7	Rename WorkerProfileHandler to BaseProfileHandler	2018-08-22 10:13:40 +01:00
Erik Johnston	a81f140880	Add assert to ensure handler is only run on master	2018-08-22 10:11:21 +01:00
Erik Johnston	47b25ba5f3	Remove redundant vars	2018-08-22 10:09:05 +01:00
Erik Johnston	4d664278af	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler	2018-08-20 14:49:43 +01:00
Erik Johnston	8dee601054	Remove redundant room_version checks	2018-08-20 14:48:53 +01:00
Erik Johnston	e21c368b8b	Revert spurious change	2018-08-20 13:54:51 +01:00
Richard van der Hoff	c144252a8c	Merge pull request #3710 from matrix-org/rav/logcontext_for_pusher_updates ... Fix logcontexts for running pushers	2018-08-17 16:21:49 +01:00
Amber Brown	c334ca67bb	Integrate presence from hotfixes (#3694)	2018-08-18 01:08:45 +10:00
Erik Johnston	ab822a2d1f	Add some fixmes	2018-08-17 15:31:50 +01:00
Erik Johnston	91cdb6de08	Call UserDirectoryHandler methods directly ... Turns out that the user directory handling is fairly racey as a bunch of stuff assumes that the processing happens on master, which it doesn't when there is a synapse.app.user_dir worker. So lets just call the function directly until we actually get round to fixing it, since it doesn't make the situation any worse.	2018-08-17 15:26:13 +01:00
Erik Johnston	782689bd40	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-17 14:15:48 +01:00
Erik Johnston	ca87ad1def	Split ProfileHandler into master and worker	2018-08-17 14:15:14 +01:00
Richard van der Hoff	66f7dc8c87	Fix logcontexts for running pushers ... First of all, avoid resetting the logcontext before running the pushers, to fix the "Starting db txn 'get_all_updated_receipts' from sentinel context" warning. Instead, give them their own "background process" logcontexts.	2018-08-17 00:32:39 +01:00
Neil Johnson	372bf073c1	block event creation and room creation on hitting resource limits	2018-08-16 21:25:16 +01:00
Matthew Hodgson	762a758fea	lazyload aware /messages (#3589)	2018-08-16 14:22:47 +01:00
Matthew Hodgson	3f543dc021	initial cut at a room summary API (#3574)	2018-08-16 09:46:50 +01:00
Neil Johnson	81d727efa9	Merge pull request #3689 from matrix-org/neilj/fix_off_by_1+maus ... Fix Mau off by one errors	2018-08-15 16:19:41 +00:00
Matthew Hodgson	2f78f432c4	speed up /members and add at= and membership params (#3568)	2018-08-15 16:35:22 +01:00
Neil Johnson	86a00e05e1	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-15 16:27:08 +01:00
Erik Johnston	dc56c47dc0	Merge pull request #3653 from matrix-org/erikj/split_federation ... Move more federation APIs to workers	2018-08-15 14:59:02 +01:00
Neil Johnson	4601129c44	Merge pull request #3687 from matrix-org/neilj/admin_email ... support admin_email config and pass through into blocking errors,	2018-08-15 13:52:25 +00:00
Erik Johnston	ef184caf30	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-15 14:25:46 +01:00
Erik Johnston	488ffe6fdb	Use federation handler function rather than duplicate ... This involves renaming _persist_events to be a public function.	2018-08-15 14:17:18 +01:00
Erik Johnston	fef2e65d12	Merge pull request #3667 from matrix-org/erikj/fixup_unbind ... Don't fail requests to unbind 3pids for non supporting ID servers	2018-08-15 10:32:12 +01:00
Neil Johnson	19b433e3f4	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/admin_email	2018-08-14 17:44:46 +01:00
Neil Johnson	414d54b61a	Merge pull request #3670 from matrix-org/neilj/mau_sync_block ... Block ability to read via sync if mau limit exceeded	2018-08-14 15:21:31 +00:00
Neil Johnson	8f9a7eb58d	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-14 15:11:54 +01:00
Neil Johnson	c74c71128d	remove blank line	2018-08-14 15:06:24 +01:00
Neil Johnson	ed4bc3d2fc	fix off by 1s on mau	2018-08-14 15:04:48 +01:00
Neil Johnson	bd92c8eaa7	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:56:45 +01:00
Neil Johnson	99ebaed8e6	Update register.py ... remove comments	2018-08-14 14:55:55 +01:00
Neil Johnson	9b5bf3d858	Merge branch 'neilj/admin_email' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:51:38 +01:00
Neil Johnson	e25d87d97b	Merge branch 'neilj/mau_sync_block' of github.com:matrix-org/synapse into neilj/fix_off_by_1+maus	2018-08-14 14:32:18 +01:00
Neil Johnson	f4b49152e2	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 21:09:47 +01:00
Neil Johnson	ce7de9ae6b	Revert "support admin_email config and pass through into blocking errors, return AuthError in all cases" ... This reverts commit 0d43f991a1.	2018-08-13 18:06:18 +01:00
Neil Johnson	0d43f991a1	support admin_email config and pass through into blocking errors, return AuthError in all cases	2018-08-13 18:00:23 +01:00
Amber Brown	99dd975dae	Run tests under PostgreSQL (#3423)	2018-08-13 16:47:46 +10:00
Matthew Hodgson	72788cf9c1	support DELETE /version with no args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	fe87890b18	implement remaining tests and make them work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	5e42c45c96	switch get_current_version_info back to being get_version_info	2018-08-12 19:14:31 -04:00
Matthew Hodgson	982edca380	fix flakes	2018-08-12 19:14:31 -04:00
Matthew Hodgson	234611f347	fix typos	2018-08-12 19:14:31 -04:00
Matthew Hodgson	14b3da63a3	add a tonne of docstring; make upload_room_keys properly assert version	2018-08-12 19:14:31 -04:00
Matthew Hodgson	9f500cb39e	more docstring for the e2e_room_keys rest	2018-08-12 19:14:31 -04:00
Matthew Hodgson	ca0b052307	fix factoring out of _should_replace_room_key	2018-08-12 19:14:31 -04:00
Matthew Hodgson	cac0253799	rename room_key_version table correctly, and fix opt args	2018-08-12 19:14:31 -04:00
Matthew Hodgson	0abb205b47	blindly incorporate PR review - needs testing & fixing	2018-08-12 19:14:31 -04:00
Matthew Hodgson	69e51c7ba4	make /room_keys/version work	2018-08-12 19:14:31 -04:00
Matthew Hodgson	8ae64b270f	implement /room_keys/version too (untested)	2018-08-12 19:14:31 -04:00
Matthew Hodgson	6b8c07abc2	make it work and fix pep8	2018-08-12 19:13:09 -04:00
Matthew Hodgson	0bc4627a73	interim WIP checkin; doesn't build yet	2018-08-12 18:23:10 -04:00
Matthew Hodgson	53ace904b2	total WIP skeleton for /room_keys API	2018-08-12 18:23:10 -04:00
Amber Brown	b37c472419	Rename async to async_helpers because async is a keyword on Python 3.7 (#3678)	2018-08-10 23:50:21 +10:00
Neil Johnson	09cf130898	only block on sync where user is not part of the mau cohort	2018-08-09 17:39:12 +01:00
Erik Johnston	ce6db0e547	Choose state algorithm based on room version	2018-08-09 14:58:47 +01:00
Neil Johnson	69ce057ea6	block sync if auth checks fail	2018-08-09 12:26:27 +01:00
Neil Johnson	a5ef110749	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sync_block	2018-08-09 11:40:37 +01:00
Erik Johnston	b179537f2a	Move clean_room_for_join to master	2018-08-09 10:37:38 +01:00
Erik Johnston	484a0ebdfc	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_profiles	2018-08-09 10:16:29 +01:00
Erik Johnston	5785b93711	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_federation	2018-08-09 10:16:16 +01:00
Erik Johnston	bf7598f582	Log when we 3pid/unbind request fails	2018-08-09 10:09:56 +01:00
Erik Johnston	2bdafaf3c1	Merge pull request #3632 from matrix-org/erikj/refactor_repl_servlet ... Add helper base class for generating new replication endpoints	2018-08-09 10:06:23 +01:00
Richard van der Hoff	bb89c84614	Merge pull request #3664 from matrix-org/rav/federation_metrics ... more metrics for the federation and appservice senders	2018-08-08 23:16:58 +01:00
Neil Johnson	e92fb00f32	sync auth blocking	2018-08-08 17:54:49 +01:00
Richard van der Hoff	8521ae13e3	Merge pull request #3654 from matrix-org/rav/room_versions ... Support for room versioning	2018-08-08 17:10:53 +01:00
Neil Johnson	990fe9fc23	Merge pull request #3633 from matrix-org/neilj/mau_tracker ... API for monthly_active_users table	2018-08-08 12:44:15 +00:00
Erik Johnston	360ba89c50	Don't fail requests to unbind 3pids for non supporting ID servers ... Older identity servers may not support the unbind 3pid request, so we shouldn't fail the requests if we received one of 400/404/501. The request still fails if we receive e.g. 500 responses, allowing clients to retry requests on transient identity server errors that otherwise do support the API. Fixes #3661	2018-08-08 12:06:18 +01:00
Richard van der Hoff	53bca4690b	more metrics for the federation and appservice senders	2018-08-07 19:09:48 +01:00
Richard van der Hoff	3523f5432a	Don't expose default_room_version as config opt	2018-08-07 12:51:57 +01:00
Erik Johnston	495cb100d1	Allow profile changes to happen on workers	2018-08-07 10:50:26 +01:00
Erik Johnston	a3f5bf79a0	Add EDU/query handling over replication	2018-08-06 15:23:31 +01:00
Erik Johnston	e26dbd82ef	Add replication APIs for persisting federation events	2018-08-06 15:02:28 +01:00
Richard van der Hoff	f900d50824	include known room versions in outgoing make_joins	2018-08-06 13:45:37 +01:00
Neil Johnson	e40a510fbf	py3 fix	2018-08-03 23:19:13 +01:00
Neil Johnson	886be75ad1	bug fixes	2018-08-03 22:29:03 +01:00
Will Hunt	16d9701892	Return M_NOT_FOUND when a profile could not be found. (#3596)	2018-08-03 19:08:05 +01:00
Richard van der Hoff	0ca459ea33	Basic support for room versioning ... This is the first tranche of support for room versioning. It includes: * setting the default room version in the config file * new room_version param on the createRoom API * storing the version of newly-created rooms in the m.room.create event * fishing the version of existing rooms out of the m.room.create event	2018-08-03 16:08:32 +01:00
Neil Johnson	897c51d274	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker	2018-08-03 13:40:47 +01:00
Erik Johnston	cb298ff623	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_repl_servlet	2018-08-03 09:25:15 +01:00
Richard van der Hoff	1fa98495d0	Merge pull request #3639 from matrix-org/rav/refactor_error_handling ... Clean up handling of errors from outbound requests	2018-08-02 17:38:24 +01:00
Neil Johnson	74b1d46ad9	do mau checks based on monthly_active_users table	2018-08-02 16:57:35 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	0bf5ec0db7	Check room visibility for /event/ requests ... Make sure that the user has permission to view the requeseted event for /event/{eventId} and /room/{roomId}/event/{eventId} requests. Also check that the event is in the given room for /room/{roomId}/event/{eventId}, for sanity.	2018-08-02 15:03:27 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups ... Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	0a65450d04	Validation for events/rooms in fed requests ... When we get a federation request which refers to an event id, make sure that said event is in the room the caller claims it is in. (patch supplied by @turt2live)	2018-08-02 13:48:40 +01:00
Erik Johnston	40c1c59cf4	Merge pull request #3621 from matrix-org/erikj/split_fed_store ... Split out DB writes in federation handler	2018-08-02 10:41:42 +01:00
Neil Johnson	085435e13a	Merge pull request #3630 from matrix-org/neilj/mau_sign_in_log_in_limits ... Initial impl of capping MAU	2018-08-01 15:58:45 +00:00
Richard van der Hoff	01e93f48ed	Kill off MatrixCodeMessageException ... This code brings the SimpleHttpClient into line with the MatrixFederationHttpClient by having it raise HttpResponseExceptions when a request fails (rather than trying to parse for matrix errors and maybe raising MatrixCodeMessageException). Then, whenever we were checking for MatrixCodeMessageException and turning them into SynapseErrors, we now need to check for HttpResponseExceptions and call to_synapse_error.	2018-08-01 16:02:46 +01:00
Amber Brown	da7785147d	Python 3: Convert some unicode/bytes uses (#3569)	2018-08-02 00:54:06 +10:00
Neil Johnson	303f1c851f	Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_sign_in_log_in_limits	2018-08-01 13:42:50 +01:00
Erik Johnston	a6d7b74915	update docs	2018-08-01 13:39:14 +01:00
Erik Johnston	4b256b9271	_persist_auth_tree no longer returns anything	2018-08-01 13:39:07 +01:00
Neil Johnson	0aba3d361a	count_monthly_users() async	2018-08-01 11:47:58 +01:00
Neil Johnson	7931393495	make count_monthly_users async synapse/handlers/auth.py	2018-08-01 10:21:56 +01:00
Richard van der Hoff	bdbdceeafa	rename replication_layer to federation_client ... I have HAD ENOUGH of trying to remember wtf a replication layer is in terms of classes.	2018-07-31 15:44:05 +01:00
Erik Johnston	443da003bc	Use new helper base class for membership requests	2018-07-31 14:32:23 +01:00
Erik Johnston	729b672823	Use new helper base class for ReplicationSendEventRestServlet	2018-07-31 14:32:23 +01:00
Neil Johnson	df2235e7fa	coding style	2018-07-31 13:16:20 +01:00
Neil Johnson	251e6c1210	limit register and sign in on number of monthly users	2018-07-30 15:55:57 +01:00
Erik Johnston	143f1a2532	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_fed_store	2018-07-30 09:56:18 +01:00
Matthew Hodgson	e9b2d047f6	make /context lazyload & filter aware (#3567) ... make /context lazyload & filter aware.	2018-07-27 15:12:50 +01:00
Richard van der Hoff	b0b5566f36	Merge pull request #3391 from t3chguy/t3chguy/default_inviter_display_name_3pid ... if inviter_display_name == ""||None then default to inviter MXID	2018-07-27 10:08:39 +01:00
Matthew Hodgson	a75231b507	Deduplicate redundant lazy-loaded members (#3331) ... * attempt at deduplicating lazy-loaded members as per the proposal; we can deduplicate redundant lazy-loaded members which are sent in the same sync sequence. we do this heuristically rather than requiring the client to somehow tell us which members it has chosen to cache, by instead caching the last N members sent to a client, and not sending them again. For now we hardcode N to 100. Each cache for a given (user,device) tuple is in turn cached for up to X minutes (to avoid the caches building up). For now we hardcode X to 30. * add include_redundant_members filter option & make it work * remove stale todo * add tests for _get_some_state_from_cache * incorporate review	2018-07-26 22:51:30 +01:00
Richard van der Hoff	03751a6420	Fix some looping_call calls which were broken in #3604 ... It turns out that looping_call does check the deferred returned by its callback, and (at least in the case of client_ips), we were relying on this, and I broke it in #3604. Update run_as_background_process to return the deferred, and make sure we return it to clock.looping_call.	2018-07-26 11:48:08 +01:00
Matthew Hodgson	1bcd0490c2	Merge pull request #2970 from matrix-org/matthew/filter_members ... Implement the lazy_load_members room state filter parameter	2018-07-26 00:03:01 +01:00
Travis Ralston	d8e65ed7e1	Fix a minor documentation typo in on_make_leave	2018-07-25 15:44:41 -06:00
Matthew Hodgson	2565804030	Merge branch 'develop' into matthew/filter_members	2018-07-25 17:27:49 +01:00
Erik Johnston	78a691d005	Split out DB writes in federation handler ... This will allow us to easily add an internal replication API to proxy these reqeusts to master, so that we can move federation APIs to workers.	2018-07-25 16:22:56 +01:00
Richard van der Hoff	55acd6856c	Fix updating of cached remote profiles ... _update_remote_profile_cache was missing its `defer.inlineCallbacks`, so when it was called, would just return a generator object, without actually running any of the method body.	2018-07-25 10:34:48 +01:00
Michael Telatynski	87951d3891	Merge branch 'develop' of github.com:matrix-org/synapse into t3chguy/default_inviter_display_name_3pid	2018-07-24 17:17:46 +01:00
Erik Johnston	8b8c4f34a3	Replace usage of get_current_toke with StreamToken.START ... This allows us to handle /context/ requests on the client_reader worker without having to pull in all the various stream handlers (e.g. precence, typing, pushers etc). The only thing the token gets used for is pagination, and that ignores everything but the room portion of the token.	2018-07-24 16:49:17 +01:00
Richard van der Hoff	a321f78991	Merge pull request #3586 from matrix-org/rav/optimise_resolve_state_groups ... Fixes and optimisations for resolve_state_groups	2018-07-24 14:11:45 +01:00
Matthew Hodgson	1a01a5b964	clarify comment on p_ids	2018-07-24 14:03:15 +01:00
Matthew Hodgson	eb1d911ab7	rather than adding ll_ids, remove them from p_ids	2018-07-24 13:40:49 +01:00
Matthew Hodgson	d19fba3655	Merge branch 'develop' into matthew/filter_members	2018-07-24 12:39:54 +01:00
Matthew Hodgson	cd241d6bda	incorporate more review	2018-07-24 12:39:40 +01:00
Richard van der Hoff	a678145010	Merge branch 'develop' into rav/logcontext_fixes	2018-07-24 10:43:30 +01:00
Erik Johnston	536bc63a4e	Merge branch 'develop' into erikj/client_apis_move	2018-07-24 09:57:05 +01:00
Richard van der Hoff	cf2d15c6a9	another couple of logcontext leaks	2018-07-24 00:57:48 +01:00
Richard van der Hoff	8dff6e0322	Logcontext fixes ... Fix some random logcontext leaks.	2018-07-24 00:37:17 +01:00
Richard van der Hoff	c1f80effbe	Handle delta_ids being None in _update_context_for_auth_events ... it's easier to create the new state group as a delta from the existing one. (There's an outside chance this will help with https://github.com/matrix-org/synapse/issues/3364)	2018-07-23 22:06:50 +01:00
Matthew Hodgson	adfe29ec0b	Merge branch 'develop' into matthew/filter_members	2018-07-23 19:21:37 +01:00
Matthew Hodgson	254fb430d1	incorporate review	2018-07-23 19:21:20 +01:00
Erik Johnston	0faa3223cd	Fix missing attributes on workers. ... This was missed during the transition from attribute to getter for getting state from context.	2018-07-23 16:28:00 +01:00
Erik Johnston	0b0b24cb82	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move	2018-07-23 13:21:15 +01:00
Erik Johnston	027bc01a1b	Add support for updating state	2018-07-23 13:17:25 +01:00
Erik Johnston	e42510ba63	Use new getters	2018-07-23 13:17:22 +01:00
Amber Brown	3132b89f12	Make the rest of the .iterwhatever go away (#3562)	2018-07-21 15:47:18 +10:00
Erik Johnston	5c88bb722f	Move PaginationHandler to its own file	2018-07-20 15:32:23 +01:00
Erik Johnston	0ecf68aedc	Move check_in_room_or_world_readable to Auth	2018-07-20 15:30:59 +01:00
Richard van der Hoff	7c712f95bb	Combine Limiter and Linearizer ... Linearizer was effectively a Limiter with max_count=1, so rather than maintaining two sets of code, let's combine them.	2018-07-20 13:11:43 +01:00
Richard van der Hoff	8462c26485	Improvements to the Limiter ... * give them names, to improve logging * use a deque rather than a list for efficiency	2018-07-20 12:50:27 +01:00
Matthew Hodgson	bcaec2915a	incorporate review	2018-07-19 19:03:50 +01:00
Matthew Hodgson	924eb34d94	add a filtered_types param to limit filtering to specific types	2018-07-19 18:32:02 +01:00
Matthew Hodgson	be3adfc331	merge develop pydoc for _get_state_for_groups	2018-07-19 11:26:04 +01:00
Erik Johnston	bacdf0cbf9	Move RoomContextHandler out of Handlers ... This is in preparation for moving GET /context/ to a worker	2018-07-18 15:33:03 +01:00
Erik Johnston	8cb8df55e9	Split MessageHandler into read only and writers ... This will let us call the read only parts from workers, and so be able to move some APIs off of master, e.g. the `/state` API.	2018-07-18 15:33:03 +01:00
Richard van der Hoff	dab00faa83	Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks ... Remove event re-signing hacks	2018-07-18 12:46:27 +01:00
Richard van der Hoff	9c04b4abf9	Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server ... Refactor and optimze filter_events_for_server	2018-07-17 14:01:39 +01:00
Richard van der Hoff	ea69d35651	Move filter_events_for_server out of FederationHandler ... for easier unit testing.	2018-07-16 13:06:24 +01:00
Matthew Hodgson	ba22b6a456	typo	2018-07-13 12:03:39 +01:00
Matthew Hodgson	ea752bdd99	s/becuase/because/g	2018-07-10 17:58:18 +01:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Richard van der Hoff	a4ab491371	Merge branch 'develop' into rav/drop_re_signing_hacks	2018-07-04 07:13:38 +01:00
Erik Johnston	e3b4043800	Merge pull request #3456 from matrix-org/hawkowl/federation-prevevent-checking ... Check the state of prev_events a bit more thoroughly when coming over federation	2018-06-29 13:55:02 +01:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
Amber Brown	99800de63d	try and clean up	2018-06-27 11:40:27 +01:00
Amber Brown	f03a5d1a17	pep8	2018-06-27 11:38:14 +01:00
Amber Brown	a7ecf34b70	cleanups	2018-06-27 11:36:03 +01:00
Amber Brown	77078d6c8e	handle federation not telling us about prev_events	2018-06-27 11:27:32 +01:00
Matthew Hodgson	c7f6b420ae	Merge pull request #3448 from matrix-org/matthew/gdpr-deactivate-admin-api ... add GDPR erase param to deactivate API	2018-06-26 10:43:14 +01:00
Matthew Hodgson	1e788db430	add GDPR erase param to deactivate API	2018-06-26 10:26:54 +01:00
Erik Johnston	d72fb9a448	Merge pull request #3442 from matrix-org/matthew/allow-unconsented-parts ... allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 20:14:18 +01:00
Erik Johnston	df48f7ef37	Actually fix it	2018-06-25 20:03:41 +01:00
Erik Johnston	a0e8a53c6d	Comment	2018-06-25 19:57:38 +01:00
Erik Johnston	7bdc5c8fa3	Fix bug with assuming wrong type	2018-06-25 19:56:02 +01:00
Erik Johnston	ea7a9c0483	Add fast path to _filter_events_for_server ... Most rooms have a trivial history visibility like "shared" or "world_readable", especially large rooms, so lets not bother getting the full membership of those rooms in that case.	2018-06-25 19:49:13 +01:00
Matthew Hodgson	0269367f18	allow non-consented users to still part rooms (to let us autopart them)	2018-06-25 17:56:10 +01:00
Matthew Hodgson	784189b1f4	typos	2018-06-25 17:37:16 +01:00
Matthew Hodgson	6eb861b67f	typo	2018-06-25 17:37:16 +01:00
Erik Johnston	33fdcfa957	Merge pull request #3441 from matrix-org/erikj/redo_erasure ... Fix user erasure and re-enable	2018-06-25 14:37:01 +01:00
Amber Brown	07cad26d65	Remove all global reactor imports & pass it around explicitly (#3424)	2018-06-25 14:08:28 +01:00
Erik Johnston	244484bf3c	Revert "Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility"" ... This reverts commit 1d009013b3.	2018-06-25 13:42:55 +01:00
Richard van der Hoff	1d009013b3	Revert "Merge pull request #3431 from matrix-org/rav/erasure_visibility" ... This reverts commit ce0d911156, reversing changes made to b4a5d767a9.	2018-06-22 16:35:10 +01:00
Erik Johnston	28ddc6cfbe	Also log number of events for serach context	2018-06-22 15:42:11 +01:00
Erik Johnston	4b4cec3989	Add some logging to search queries	2018-06-22 15:42:11 +01:00
Erik Johnston	ce0d911156	Merge pull request #3431 from matrix-org/rav/erasure_visibility ... Support hiding events from deleted users	2018-06-22 15:06:44 +01:00
Erik Johnston	b4a5d767a9	Merge pull request #3428 from matrix-org/erikj/persisted_pdu ... Simplify get_persisted_pdu	2018-06-22 14:47:43 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	02bfc581f8	Merge pull request #3399 from costacruise/master ... Add error code to room creation error	2018-06-20 17:26:25 +01:00
Michael Wagner	19cd3120ec	Add error code to room creation error ... This error code is mentioned in the documentation at https://matrix.org/docs/api/client-server/#!/Room32creation/createRoom	2018-06-14 14:08:40 +02:00
Amber Brown	a61738b316	Remove run_on_reactor (#3395)	2018-06-14 18:27:37 +10:00
Michael Telatynski	94700e55fa	if inviter_display_name == ""||None then default to inviter MXID ... to prevent email invite from "None"	2018-06-13 10:31:01 +01:00
Richard van der Hoff	b6faef2ad7	Filter out erased messages ... Redact any messges sent by erased users.	2018-06-12 09:53:18 +01:00
Richard van der Hoff	f1023ebf4b	mark accounts as erased when requested	2018-06-12 09:53:18 +01:00
Richard van der Hoff	9fc5b74b24	simplify get_persisted_pdu ... it doesn't make much sense to use get_persisted_pdu on the receive path: just get the event straight from the store.	2018-06-12 09:51:31 +01:00
David Baker	187a546bff	Merge pull request #3276 from matrix-org/dbkr/unbind ... Remove email addresses / phone numbers from ID servers when they're removed from synapse	2018-06-11 16:02:00 +01:00
Matthew Hodgson	c96d882a02	Merge branch 'develop' into matthew/filter_members	2018-06-10 12:26:14 +03:00
Richard van der Hoff	c6b1441c52	Fix event filtering in get_missing_events handler	2018-06-08 14:15:31 +01:00
Richard van der Hoff	0834b49c6a	Fix event filtering in get_missing_events handler	2018-06-08 11:34:46 +01:00
Richard van der Hoff	8503dd0047	Remove event re-signing hacks ... These "temporary fixes" have been here three and a half years, and I can't find any events in the matrix.org database where the calculated signature differs from what's in the db. It's time for them to go away.	2018-06-07 16:08:29 +01:00
David Baker	d62162bbec	doc fixes	2018-06-05 18:09:13 +01:00
David Baker	6a29e815fc	Fix comment	2018-06-04 12:01:23 +01:00
David Baker	e44150a6de	Missing yield	2018-06-04 12:01:13 +01:00
David Baker	f731e42baf	docstring	2018-06-04 12:00:51 +01:00
Matthew Hodgson	28f09fcdd5	Merge branch 'develop' into matthew/filter_members	2018-06-04 00:09:17 +03:00
Matthew Hodgson	5f6122fe10	more comments	2018-06-04 00:08:52 +03:00
Amber Brown	c936a52a9e	Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307)	2018-05-31 19:03:47 +10:00
Richard van der Hoff	34b85df7f5	Update some comments and docstrings in SyncHandler	2018-05-29 22:31:18 +01:00
Richard van der Hoff	4a9cbdbc15	Exempt AS-registered users from doing gdpr	2018-05-29 19:54:32 +01:00
Matthew Hodgson	a6c8f7c875	add pydoc	2018-05-29 01:09:55 +01:00
Matthew Hodgson	7a6df013cc	merge develop	2018-05-29 00:25:22 +01:00
Amber Brown	754826a830	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-28 18:57:23 +10:00
Richard van der Hoff	41921ac01b	Merge pull request #3287 from matrix-org/rav/allow_leaving_server_notices_room ... Let users leave the server notice room after joining	2018-05-25 11:15:45 +01:00
Richard van der Hoff	757ed27258	Let users leave the server notice room after joining ... They still can't reject invites, but we let them leave it.	2018-05-25 11:07:21 +01:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Amber Brown	1f69693347	Merge pull request #3244 from NotAFile/py3-six-4 ... replace some iteritems with six	2018-05-24 13:04:07 -05:00
David Baker	77a23e2e05	Merge remote-tracking branch 'origin/develop' into dbkr/unbind	2018-05-24 16:20:53 +01:00
David Baker	7a1af504d7	Remove users from user directory on deactivate	2018-05-24 15:59:58 +01:00
Erik Johnston	f72d5a44d5	Merge pull request #3261 from matrix-org/erikj/pagination_fixes ... Fix federation backfill bugs	2018-05-24 14:52:03 +01:00
Erik Johnston	68399fc4de	Merge pull request #3267 from matrix-org/erikj/iter_filter ... Use iter* methods for _filter_events_for_server	2018-05-24 14:44:57 +01:00
David Baker	9700d15611	pep8	2018-05-24 11:23:15 +01:00
David Baker	a21a41bad7	comment	2018-05-24 11:19:59 +01:00
David Baker	b3bff53178	Unbind 3pids when they're deleted too	2018-05-24 11:08:05 +01:00
Richard van der Hoff	a0b3946fe2	Merge branch 'release-v0.30.0' into rav/localpart_in_consent_uri	2018-05-23 16:06:03 +01:00
Richard van der Hoff	3ff6f50eac	Use the localpart in the consent uri ... ... because it's shorter.	2018-05-23 15:28:23 +01:00
Richard van der Hoff	82191b08f6	Support for putting %(consent_uri)s in messages ... Make it possible to put the URI in the error message and the server notice that get sent by the server	2018-05-23 15:24:31 +01:00
David Baker	2c7866d664	Hit the 3pid unbind endpoint on deactivation	2018-05-23 14:38:56 +01:00
Erik Johnston	5aaa3189d5	s/values/itervalues/	2018-05-23 10:13:05 +01:00
Erik Johnston	0a4bca4134	Use iter* methods for _filter_events_for_server	2018-05-23 10:04:23 +01:00
Amber Brown	53cc2cde1f	cleanup	2018-05-22 17:32:57 -05:00
Erik Johnston	e85b5a0ff7	Use iter* methods	2018-05-22 19:02:48 +01:00
Erik Johnston	586b66b197	Fix that states is a dict of dicts	2018-05-22 19:02:36 +01:00
Richard van der Hoff	08bfc48abf	custom error code for not leaving server notices room	2018-05-22 17:27:27 +01:00
David Baker	0a078026ea	comment typo	2018-05-22 17:14:06 +01:00
Amber Brown	8f5a688d42	cleanups, self-registration	2018-05-22 10:56:03 -05:00
Amber Brown	a8990fa2ec	Merge remote-tracking branch 'origin/develop' into 3218-official-prom	2018-05-22 10:50:26 -05:00
Erik Johnston	cb2a2ad791	get_domains_from_state returns list of tuples	2018-05-22 16:23:39 +01:00
Richard van der Hoff	a5e2941aad	Reject attempts to send event before privacy consent is given ... Returns an M_CONSENT_NOT_GIVEN error (cf https://github.com/matrix-org/matrix-doc/issues/1252) if consent is not yet given.	2018-05-22 12:00:47 +01:00
Richard van der Hoff	8810685df9	Stub out ServerNoticesSender on the workers ... ... and have the sync endpoints call it directly rather than obsure indirection via PresenceHandler	2018-05-22 11:54:51 +01:00
Richard van der Hoff	9ea219c514	Send users a server notice about consent ... When a user first syncs, we will send them a server notice asking them to consent to the privacy policy if they have not already done so.	2018-05-22 11:54:51 +01:00
Amber Brown	df9f72d9e5	replacing portions	2018-05-21 19:47:37 -05:00
Adrian Tschira	933bf2dd35	replace some iteritems with six ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-19 17:59:26 +02:00
Richard van der Hoff	6d9dc67139	Merge pull request #3232 from matrix-org/rav/server_notices_room ... Infrastructure for a server notices room	2018-05-18 11:28:04 +01:00
Richard van der Hoff	67af392712	Merge pull request #3233 from matrix-org/rav/remove_dead_code ... Remove unused `update_external_syncs`	2018-05-18 11:22:43 +01:00
Richard van der Hoff	26305788fe	Make sure we reject attempts to invite the notices user	2018-05-18 11:18:39 +01:00
Erik Johnston	fa30ac38cc	Merge pull request #3221 from matrix-org/erikj/purge_token ... Make purge_history operate on tokens	2018-05-18 10:35:23 +01:00
Richard van der Hoff	88d3405332	fix missing yield for server_notices_room	2018-05-17 18:33:45 +01:00
Richard van der Hoff	d43d480d86	Remove unused update_external_syncs ... This method isn't used anywhere. Burninate it.	2018-05-17 18:22:19 +01:00
Richard van der Hoff	fed62e21ad	Infrastructure for a server notices room ... Server Notices use a special room which the user can't dismiss. They are created on demand when some other bit of the code calls send_notice. (This doesn't actually do much yet becuse we don't call send_notice anywhere)	2018-05-17 17:58:25 +01:00
Richard van der Hoff	c46367d0d7	Move RoomCreationHandler out of synapse.handlers.Handlers ... Handlers is deprecated nowadays, so let's move this out before I add a new dependency on it. Also fix the docstrings on create_room.	2018-05-17 09:08:42 +01:00
Erik Johnston	5f27ed75ad	Make purge_history operate on tokens ... As we're soon going to change how topological_ordering works	2018-05-15 16:23:50 +01:00
David Baker	8cbbfd16fb	Merge pull request #3201 from matrix-org/dbkr/leave_rooms_on_deactivate ... Part user from rooms on account deactivate	2018-05-14 11:31:48 +01:00
David Baker	6b49628e3b	Catch failure to part user from room	2018-05-10 12:23:53 +01:00
David Baker	217bc53c98	Many docstrings	2018-05-10 12:20:40 +01:00
David Baker	4d298506dd	Oops, don't call function passed to run_in_background	2018-05-10 11:57:13 +01:00
Erik Johnston	bfe1f73855	Merge pull request #3199 from matrix-org/erikj/pagination_sync ... Refactor sync APIs to reuse pagination API	2018-05-09 16:16:56 +01:00
Erik Johnston	d26bec8a43	Add comment to sync as to why code path is split	2018-05-09 15:56:07 +01:00
Erik Johnston	fcf55f2255	Fix returned token is no longer a tuple	2018-05-09 15:43:00 +01:00
David Baker	294e9a0c9b	Prefix internal functions	2018-05-09 15:10:37 +01:00
David Baker	52281e4c54	Indent fail	2018-05-09 15:06:16 +01:00
David Baker	7e8726b8fb	Part deactivated users in the background ... One room at a time so we don't take out the whole server with leave events, and restart at server restart.	2018-05-09 14:54:28 +01:00
Erik Johnston	e2accd7f1d	Refactor sync APIs to reuse pagination API ... The sync API often returns events in a topological rather than stream ordering, e.g. when the user joined the room or on initial sync. When this happens we can reuse existing pagination storage functions.	2018-05-09 13:43:39 +01:00
Erik Johnston	c4af4c24ca	Refactor get_recent_events_for_room return type ... There is no reason to return a tuple of tokens when the last token is always the token passed as an argument. Changing it makes it consistent with other storage APIs	2018-05-09 11:55:34 +01:00
David Baker	bf98fa0864	Part user from rooms on account deactivate ... This implements this very crudely: this probably isn't viable because parting a user from all their rooms could take a long time, and if the HS gets restarted in that time the process will be aborted.	2018-05-08 15:58:35 +01:00
Richard van der Hoff	ca7211104e	Merge branch 'release-v0.28.1' into develop	2018-05-01 18:16:57 +01:00
Richard van der Hoff	33f469ba19	Apply some limits to depth to counter abuse ... * When creating a new event, cap its depth to 2^63 - 1 * When receiving events, reject any without a sensible depth As per https://docs.google.com/document/d/1I3fi2S-XnpO45qrpCsowZv8P8dHcNZ4fsBsbOW7KABI	2018-05-01 17:54:19 +01:00
Richard van der Hoff	63ae5cbf34	Merge pull request #3143 from matrix-org/rav/remove_redundant_preserve_fn ... Remove redundant call to preserve_fn	2018-04-30 10:23:59 +01:00
Richard van der Hoff	bc2017a594	Merge pull request #3153 from NotAFile/py3-httplib ... move httplib import to six	2018-04-30 01:26:42 +01:00
Richard van der Hoff	683149c1f9	Merge pull request #3151 from NotAFile/py3-xrange-1 ... Move more xrange to six	2018-04-30 01:20:06 +01:00
Richard van der Hoff	3b0e431c82	Merge pull request #3150 from NotAFile/py3-listcomp-yield ... Don't yield in list comprehensions	2018-04-30 01:11:41 +01:00
Richard van der Hoff	db75c86e84	Merge branch 'develop' into py3-xrange-1	2018-04-30 01:02:25 +01:00
Richard van der Hoff	af3cc50511	Remove redundant call to preserve_fn ... submit_event_for_as doesn't return a deferred anyway, so this is pointless.	2018-04-30 00:48:36 +01:00
Adrian Tschira	d82b6ea9e6	Move more xrange to six ... plus a bonus next() Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:57:00 +02:00
Adrian Tschira	94f4d7f49e	move httplib import to six	2018-04-28 13:43:34 +02:00
Adrian Tschira	cdb4647a80	Don't yield in list comprehensions ... I've tried to grep for more of this with no success. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-28 13:36:30 +02:00
Richard van der Hoff	453adf00b6	pep8; remove spurious import	2018-04-27 14:32:08 +01:00
Richard van der Hoff	fc149b4eeb	Merge remote-tracking branch 'origin/develop' into rav/use_run_in_background	2018-04-27 14:31:23 +01:00
Richard van der Hoff	2a13af23bc	Use run_in_background in preference to preserve_fn ... While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.	2018-04-27 12:55:51 +01:00
Richard van der Hoff	41d4b07a53	Merge pull request #3142 from matrix-org/rav/reraise ... reraise exceptions more carefully	2018-04-27 12:16:19 +01:00
Richard van der Hoff	9c3da24561	Merge pull request #3138 from matrix-org/rav/catch_unhandled_exceptions ... Improve exception handling for background processes	2018-04-27 11:47:49 +01:00
Richard van der Hoff	6493b22b42	reraise exceptions more carefully ... We need to be careful (under python 2, at least) that when we reraise an exception after doing some error handling, we actually reraise the original exception rather than anything that might have been raised (and handled) during the error handling.	2018-04-27 11:40:06 +01:00
Richard van der Hoff	605defb9e4	Add missing consumeErrors ... In general we want defer.gatherResults to consumeErrors, rather than having exceptions hanging around and getting logged as CRITICAL unhandled errors.	2018-04-27 11:16:28 +01:00
Richard van der Hoff	9255a6cb17	Improve exception handling for background processes ... There were a bunch of places where we fire off a process to happen in the background, but don't have any exception handling on it - instead relying on the unhandled error being logged when the relevent deferred gets garbage-collected. This is unsatisfactory for a number of reasons: - logging on garbage collection is best-effort and may happen some time after the error, if at all - it can be hard to figure out where the error actually happened. - it is logged as a scary CRITICAL error which (a) I always forget to grep for and (b) it's not really CRITICAL if a background process we don't care about fails. So this is an attempt to add exception handling to everything we fire off into the background.	2018-04-27 11:07:40 +01:00
Richard van der Hoff	77ebef9d43	Merge pull request #3118 from matrix-org/rav/reject_prev_events ... Reject events which have lots of prev_events	2018-04-23 17:51:38 +01:00
Richard van der Hoff	9b9c38373c	Remove spurious param	2018-04-23 12:00:06 +01:00
Richard van der Hoff	8dc4a6144b	Merge pull request #3107 from NotAFile/py3-bool-nonzero ... add __bool__ alias to __nonzero__ methods	2018-04-20 15:43:39 +01:00
Richard van der Hoff	3de7d9fe99	accept stupid events over backfill	2018-04-20 11:41:03 +01:00
Richard van der Hoff	11a67b7c9d	Merge pull request #3093 from matrix-org/rav/response_cache_wrap ... Refactor ResponseCache usage	2018-04-20 11:31:17 +01:00
Richard van der Hoff	bc381d5798	Merge pull request #3117 from matrix-org/rav/refactor_have_events ... Refactor store.have_events	2018-04-20 10:26:12 +01:00
Richard van der Hoff	b1dfbc3c40	Refactor store.have_events ... It turns out that most of the time we were calling have_events, we were only using half of the result. Replace have_events with have_seen_events and get_rejection_reasons, so that we can see what's going on a bit more clearly.	2018-04-20 10:25:56 +01:00
Richard van der Hoff	1f4b498b73	Add some comments	2018-04-18 00:15:36 +01:00
Richard van der Hoff	e585228860	Check events on backfill too	2018-04-18 00:06:42 +01:00
Richard van der Hoff	9b7794262f	Reject events which have too many auth_events or prev_events ... ... this should protect us from being dossed by people making silly events (deliberately or otherwise)	2018-04-18 00:06:42 +01:00
Richard van der Hoff	639480e14a	Avoid creating events with huge numbers of prev_events ... In most cases, we limit the number of prev_events for a given event to 10 events. This fixes a particular code path which created events with huge numbers of prev_events.	2018-04-16 18:41:37 +01:00
Adrian Tschira	f63ff73c7f	add __bool__ alias to __nonzero__ methods ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:40:47 +02:00
Richard van der Hoff	b78395b7fe	Refactor ResponseCache usage ... Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.	2018-04-12 13:02:15 +01:00
Richard van der Hoff	d5c74b9f6c	Merge pull request #3092 from matrix-org/rav/response_cache_metrics ... Add metrics for ResponseCache	2018-04-12 12:59:36 +01:00
Erik Johnston	19ceb4851f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag	2018-04-12 11:36:07 +01:00
Erik Johnston	f67e906e18	Set all metrics at the same time	2018-04-12 11:18:19 +01:00
Erik Johnston	971059a733	Merge pull request #3088 from matrix-org/erikj/as_parallel ... Send events to ASes concurrently	2018-04-12 10:42:36 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	92e34615c5	Track where event stream processing have gotten up to	2018-04-11 12:13:40 +01:00
Erik Johnston	121591568b	Send events to ASes concurrently	2018-04-11 09:56:00 +01:00
Richard van der Hoff	b3384232a0	Add metrics for ResponseCache	2018-04-10 23:14:47 +01:00
Erik Johnston	eaa2ebf20b	Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends ... Limit concurrent event sends for a room	2018-04-10 16:43:58 +01:00
Erik Johnston	f8e8ec013b	Note why we're limiting concurrent event sends	2018-04-10 14:00:46 +01:00
Richard van der Hoff	a3599dda97	Merge pull request #2996 from krombel/allow_auto_join_rooms ... move handling of auto_join_rooms to RegisterHandler	2018-04-10 01:11:00 +01:00
Richard van der Hoff	d2c6f4d626	Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin ... Return a 404 rather than a 500 on rejoining empty rooms	2018-04-09 17:32:36 +01:00
Richard van der Hoff	f3ef60662f	Return a 404 rather than a 500 on rejoining empty rooms ... Filter ourselves out of the server list before checking for an empty remote host list, to fix 500 error Fixes #2141	2018-04-09 12:56:22 +01:00
Erik Johnston	e5082494eb	Limit concurrent event sends for a room	2018-04-09 12:07:39 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Richard van der Hoff	725a72ec5a	Merge pull request #3000 from NotAFile/change-except-style ... Replace old style error catching with 'as' keyword	2018-04-04 10:45:22 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Richard van der Hoff	9a0db062af	Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors ... Fix error when claiming e2e keys from offline servers	2018-03-28 14:50:38 +01:00
David Baker	79452edeee	Add joinability for groups ... Adds API to set the 'joinable' flag, and corresponding flag in the table.	2018-03-28 14:03:37 +01:00
Krombel	6152e253d8	Merge branch 'develop' of into allow_auto_join_rooms	2018-03-28 14:45:28 +02:00
Richard van der Hoff	a134c572a6	Stringify exceptions for keys/{query,claim} ... Make sure we stringify any exceptions we return from keys/query and keys/claim, to avoid a 'not JSON serializable' error later Fixes #3010	2018-03-27 17:15:06 +01:00
Richard van der Hoff	c2a5cf2fe3	factor out exception handling for keys/claim and keys/query ... this stuff is badly c&p'ed	2018-03-27 17:11:23 +01:00
Erik Johnston	3e0c0660b3	Also do check inside linearizer	2018-03-27 13:01:34 +01:00
Erik Johnston	44cd6e1358	PEP8	2018-03-26 12:06:48 +01:00
Erik Johnston	a052aa42e7	Linearize calls to _generate_user_id	2018-03-26 12:02:20 +01:00
Erik Johnston	fa72803490	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-19 11:41:01 +00:00
Matthew Hodgson	b2f2282947	make lazy_load_members configurable in filters	2018-03-19 01:15:13 +00:00
Matthew Hodgson	478af0f720	reshuffle todo & comments	2018-03-19 01:00:12 +00:00
Matthew Hodgson	366f730bf6	only get member state IDs for incremental syncs if we're filtering	2018-03-18 21:40:43 +00:00
Matthew Hodgson	fc5397fdf5	remove debug	2018-03-16 01:44:55 +00:00
Matthew Hodgson	4f0493c850	fix tsm search again	2018-03-16 01:43:37 +00:00
Matthew Hodgson	f7dcc404f2	add state_ids for timeline entries	2018-03-16 01:37:53 +00:00
Matthew Hodgson	5b3b3aada8	simplify timeline_start_members	2018-03-16 01:17:34 +00:00
Matthew Hodgson	3bc5bd2d22	make incr syncs work	2018-03-16 00:52:04 +00:00
Erik Johnston	926ba76e23	Replace ujson with simplejson	2018-03-15 23:43:31 +00:00
NotAFile	2cc9f76bc3	replace old style error catching with 'as' keyword ... This is both easier to read and compatible with python3 (not that that matters) Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-15 16:11:17 +01:00
Krombel	91ea0202e6	move handling of auto_join_rooms to RegisterHandler ... Currently the handling of auto_join_rooms only works when a user registers itself via public register api. Registrations via registration_shared_secret and ModuleApi do not work This auto_joins the users in the registration handler which enables the auto join feature for all 3 registration paths. This is related to issue #2725 Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-14 16:45:37 +01:00
Matthew Hodgson	056a6df546	Merge branch 'develop' into matthew/filter_members	2018-03-14 15:38:05 +00:00
Erik Johnston	57db62e554	Merge pull request #2992 from matrix-org/erikj/implement_member_workre ... Implement RoomMemberWorkerHandler	2018-03-14 14:29:33 +00:00
Erik Johnston	62ad701326	s/join/joined/ in notify_user_membership_change	2018-03-14 14:17:43 +00:00
Erik Johnston	3f0f06cb31	Split RoomMemberWorkerHandler to separate file	2018-03-14 11:41:45 +00:00
Erik Johnston	3e839e0548	Merge pull request #2989 from matrix-org/erikj/profile_cache_master ... Only update remote profile cache on master	2018-03-14 09:42:27 +00:00
Matthew Hodgson	4d0cfef6ee	add copyright to nudge CI	2018-03-14 00:02:20 +00:00
Matthew Hodgson	c9d72e4571	oops	2018-03-13 23:46:45 +00:00
Matthew Hodgson	ccca02846d	make it work	2018-03-13 22:31:41 +00:00
Matthew Hodgson	12350e3f9a	merge proper fix to bug 2969	2018-03-13 22:11:58 +00:00
Matthew Hodgson	14a9d2f73d	ensure we always include the members for a given timeline block	2018-03-13 22:03:42 +00:00
Erik Johnston	b27320b550	Implement RoomMemberWorkerHandler	2018-03-13 18:26:00 +00:00
Erik Johnston	350331d466	_remote_join and co take a requester	2018-03-13 17:50:39 +00:00
Erik Johnston	1a69c6d590	Merge pull request #2987 from matrix-org/erikj/split_room_member_handler ... Split RoomMemberHandler into base and master class	2018-03-13 17:40:00 +00:00
Erik Johnston	df8ff682a7	Only update remote profile cache on master	2018-03-13 17:38:21 +00:00
Erik Johnston	6dbebef141	Add missing param to docstrings	2018-03-13 17:15:32 +00:00
Erik Johnston	16adb11cc0	Correct import order	2018-03-13 16:57:07 +00:00
Erik Johnston	82f16faa78	Move user_*_room distributor stuff to master class ... I added yields when calling user_left_room, but they shouldn't matter on the master process as they always return None anyway.	2018-03-13 16:38:15 +00:00
Erik Johnston	b78717b87b	Split RoomMemberHandler into base and master class ... The intention here is to split the class into the bits that can be done on workers and the bits that have to be done on the master. In future there will also be a class that can be run on the worker, which will delegate work to the master when necessary.	2018-03-13 16:37:41 +00:00
Erik Johnston	95cb401ae0	Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer ... Remove ReplicationLayer and user Client/Server directly	2018-03-13 15:45:08 +00:00
Erik Johnston	5d8476d8ff	Merge pull request #2981 from matrix-org/erikj/factor_remote_leave ... Factor out _remote_reject_invite in RoomMember	2018-03-13 15:44:56 +00:00
Erik Johnston	56e709857c	Merge pull request #2979 from matrix-org/erikj/no_handlers ... Don't build handlers on workers unnecessarily	2018-03-13 13:46:38 +00:00
Erik Johnston	cb9f8e527c	s/replication_client/federation_client/	2018-03-13 13:26:52 +00:00
Erik Johnston	bf8e97bd3c	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave	2018-03-13 13:17:08 +00:00
Erik Johnston	ea3442c15c	Add docstring	2018-03-13 13:16:21 +00:00
Erik Johnston	16469a4f15	Merge pull request #2980 from matrix-org/erikj/rm_priv ... Make RoomMemberHandler functions private that can be	2018-03-13 13:11:11 +00:00
Erik Johnston	c82111a55f	Merge pull request #2982 from matrix-org/erikj/fix_extra_users ... extra_users is actually a list of UserIDs	2018-03-13 13:11:04 +00:00
Erik Johnston	da87791975	Merge pull request #2983 from matrix-org/erikj/rename_register_3pid ... Refactor get_or_register_3pid_guest	2018-03-13 13:10:52 +00:00
Erik Johnston	8b3573a8b2	Refactor get_or_register_3pid_guest	2018-03-13 12:08:58 +00:00
Erik Johnston	0f942f68c1	Factor out _remote_reject_invite in RoomMember	2018-03-13 11:22:45 +00:00
Erik Johnston	d0fcc48f9d	extra_users is actually a list of UserIDs	2018-03-13 11:20:06 +00:00
Erik Johnston	31becf4ac3	Make functions private that can be	2018-03-13 11:15:16 +00:00
Erik Johnston	265b993b8a	Split replication layer into two	2018-03-13 10:55:47 +00:00
Erik Johnston	c3f79c9da5	Split out edu/query registration to a separate class	2018-03-13 10:24:27 +00:00
Richard van der Hoff	e48c7aac4d	Add transactional API to history purge ... Make the purge request return quickly, and allow scripts to poll for updates.	2018-03-12 16:22:55 +00:00
Richard van der Hoff	1708412f56	Return an error when doing two purges on a room ... Queuing up purges doesn't sound like a good thing.	2018-03-12 16:22:54 +00:00
Matthew Hodgson	8713365265	typos	2018-03-11 20:10:25 +00:00
Matthew Hodgson	9b334b3f97	WIP experiment in lazyloading room members	2018-03-11 20:01:41 +00:00
Erik Johnston	a56d54dcb7	Fix up log message	2018-03-07 11:55:31 +00:00
Erik Johnston	02a1296ad6	Fix typo	2018-03-07 11:55:31 +00:00
Erik Johnston	8cb44da4aa	Fix race in sync when joining room ... The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.	2018-03-07 11:55:31 +00:00
Richard van der Hoff	8ffaacbee3	Merge pull request #2949 from krombel/use_bcrypt_checkpw ... use bcrypt.checkpw	2018-03-06 11:56:06 +00:00
Richard van der Hoff	b2932107bb	Merge pull request #2946 from matrix-org/rav/timestamp_to_purge ... Implement purge_history by timestamp	2018-03-06 11:20:23 +00:00
Krombel	ed9b5eced4	use bcrypt.checkpw ... in bcrypt 3.1.0 checkpw got introduced (already 2 years ago) This makes use of that with enhancements which might get introduced by that Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-05 18:02:59 +01:00
Erik Johnston	d4ffe61d4f	Remove ability for AS users to call /events and /sync ... This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144	2018-03-05 15:44:46 +00:00
Richard van der Hoff	f8bfcd7e0d	Provide a means to pass a timestamp to purge_history	2018-03-05 14:37:23 +00:00
Erik Johnston	d960d23830	Add missing yield during 3pid signature checks	2018-03-02 11:03:18 +00:00
Erik Johnston	9e08a93a7b	Merge pull request #2927 from matrix-org/erikj/read_marker_caches ... Improve caching for read_marker API	2018-03-01 17:12:34 +00:00
Erik Johnston	a83c514d1f	Improve caching for read_marker API ... We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.	2018-03-01 17:08:17 +00:00
Erik Johnston	529c026ac1	Move back to hs.is_mine	2018-03-01 16:49:12 +00:00
Erik Johnston	784f036306	Move RoomMemberHandler out of Handlers	2018-03-01 14:36:50 +00:00
Erik Johnston	f381d63813	Check event auth on the worker	2018-03-01 10:18:37 +00:00
Erik Johnston	6b8604239f	Correctly send ratelimit and extra_users params	2018-03-01 10:08:39 +00:00
Erik Johnston	f756f961ea	Fixup comments	2018-03-01 10:05:27 +00:00
Erik Johnston	28e973ac11	Calculate push actions on worker	2018-02-28 18:02:30 +00:00
Erik Johnston	d874d4f2d7	Delete from push_actions_staging in federation too	2018-02-20 11:37:52 +00:00
Erik Johnston	b96278d6fe	Ensure that we delete staging push actions on errors	2018-02-15 15:47:06 +00:00
Erik Johnston	c0c9327fe0	Merge pull request #2854 from matrix-org/erikj/event_create_worker ... Create a worker for event creation	2018-02-13 18:07:10 +00:00
Richard van der Hoff	10b34dbb9a	Merge pull request #2858 from matrix-org/rav/purge_updates ... delete_local_events for purge_room_history	2018-02-09 14:11:00 +00:00
Richard van der Hoff	74fcbf741b	delete_local_events for purge_history ... Add a flag which makes the purger delete local events	2018-02-09 13:07:41 +00:00
Richard van der Hoff	671540dccf	rename delete_old_state -> purge_history ... (beacause it deletes more than state)	2018-02-09 13:07:41 +00:00
Erik Johnston	50fe92cd26	Move presence handling into handle_new_client_event ... As we want to have it run on the main synapse instance	2018-02-07 10:34:09 +00:00
Erik Johnston	24dd73028a	Add replication http endpoint for event sending	2018-02-07 10:32:32 +00:00
Erik Johnston	e3624fad5f	Remove pointless ratelimit check ... The intention was for the check to be called as early as possible in the request, but actually was called just before the main ratelimit check, so was fairly pointless.	2018-02-07 10:30:25 +00:00
Erik Johnston	617199d73d	Merge pull request #2847 from matrix-org/erikj/separate_event_creation ... Split event creation into a separate handler	2018-02-06 17:01:17 +00:00
Erik Johnston	3e1e69ccaf	Update copyright	2018-02-06 16:40:38 +00:00
Erik Johnston	770b2252ca	s/_create_new_client_event/create_new_client_event/	2018-02-06 16:40:30 +00:00
Erik Johnston	3d33eef6fc	Store state groups separately from events (#2784) ... * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints	2018-02-06 14:31:24 +00:00
Erik Johnston	3fa362502c	Update places where we create events	2018-02-05 16:01:48 +00:00
Erik Johnston	5ff3d23564	Split event creation into a separate handler	2018-02-05 16:01:48 +00:00
Richard van der Hoff	0cbda53819	Rename resolve_state_groups -> resolve_state_groups_for_events ... (to make way for a method that actually just does the state group resolution)	2018-02-01 12:40:00 +00:00
Neil Johnson	73560237d6	add white space line	2018-01-26 00:15:10 +00:00
Neil Johnson	86c4f49a31	rather than try reconstruct the results object, better to guard against the xrange step argument being 0	2018-01-26 00:12:02 +00:00
Neil Johnson	f632083576	fix return type, should be a dict	2018-01-25 23:52:17 +00:00
Neil Johnson	6c6e197b0a	fix PEP8 violation	2018-01-25 23:47:46 +00:00
Neil Johnson	d02e43b15f	remove white space	2018-01-25 23:29:46 +00:00
Neil Johnson	349c739966	synapse 500s on a call to publicRooms in the case where the number of public rooms is zero, the specific cause is due to xrange trying to use a step value of zero, but if the total room number really is zero then it makes sense to just bail and save the extra processing	2018-01-25 23:28:44 +00:00
Matthew Hodgson	ab9f844aaf	Add federation_domain_whitelist option (#2820) ... Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network	2018-01-22 19:11:18 +01:00
Matthew Hodgson	447f4f0d5f	rewrite based on PR feedback: ... * [ ] split config options into allowed_local_3pids and registrations_require_3pid * [ ] simplify and comment logic for picking registration flows * [ ] fix docstring and move check_3pid_allowed into a new util module * [ ] use check_3pid_allowed everywhere @erikjohnston PTAL	2018-01-19 15:33:55 +00:00
Matthew Hodgson	9d332e0f79	fix up v1, and improve errors	2018-01-19 00:53:58 +00:00
Richard van der Hoff	a5213df1f7	Sanity checking for user ids ... Check the user_id passed to a couple of APIs for validity, to avoid "IndexError: list index out of range" exception which looks scary and results in a 500 rather than a more useful error. Fixes #1432, among other things	2018-01-17 14:28:54 +00:00
Richard van der Hoff	a027c2af8d	Metrics for events processed in appservice and fed sender ... More metrics I wished I'd had	2018-01-15 18:23:24 +00:00
Erik Johnston	825a07a974	Merge pull request #2773 from matrix-org/erikj/hash_bg ... Do bcrypt hashing in a background thread	2018-01-10 18:11:41 +00:00
Erik Johnston	f8e1ab5fee	Do bcrypt hashing in a background thread	2018-01-10 18:01:28 +00:00
Michael Telatynski	5f07f5694c	fix order of operations derp and also use .get to default to {} ... Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>	2018-01-10 15:11:35 +00:00
Michael Telatynski	8c9d5b4873	Fix publicised groups API (singular) over federation ... which was missing its fed client API, since there is no other API it might as well reuse the bulk one and unwrap it Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>	2018-01-10 15:04:51 +00:00
Erik Johnston	d8a6c734fa	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/createroom_content	2017-12-07 14:24:01 +00:00
Erik Johnston	ef045dcd71	Copy dict in update_membership too	2017-12-07 14:17:15 +00:00
Matthew Hodgson	33cb7ef0b7	Merge pull request #2723 from matrix-org/matthew/search-all-local-users ... Add all local users to the user_directory and optionally search them	2017-12-05 11:09:47 +00:00
Richard van der Hoff	da1010c83a	support custom login types for validating users ... Wire the custom login type support from password providers into the UI-auth user-validation flows.	2017-12-05 09:43:30 +00:00
Richard van der Hoff	d7ea8c4800	Factor out a validate_user_via_ui_auth method ... Collect together all the places that validate a logged-in user via UI auth.	2017-12-05 09:42:30 +00:00
Richard van der Hoff	d5f9fb06b0	Refactor UI auth implementation ... Instead of returning False when auth is incomplete, throw an exception which can be caught with a wrapper.	2017-12-05 09:40:05 +00:00
Matthew Hodgson	c22e73293a	speed up the rate of initial spam for users	2017-12-04 18:05:28 +00:00
Matthew Hodgson	74e0cc74ce	fix pep8 and tests	2017-12-04 15:11:38 +00:00
Matthew Hodgson	1bd40ca73e	switch to a simpler 'search_all_users' button as per review feedback	2017-12-04 14:58:39 +00:00
Matthew Hodgson	f397153dfc	Merge branch 'develop' into matthew/search-all-local-users	2017-11-30 01:51:38 +00:00
Matthew Hodgson	a4bb133b68	fix thinkos galore	2017-11-30 01:17:15 +00:00
Matthew Hodgson	cd3697e8b7	kick the user_directory index when new users register	2017-11-29 18:33:34 +00:00
Matthew Hodgson	3241c7aac3	untested WIP but might actually work	2017-11-29 18:27:05 +00:00
Matthew Hodgson	47d99a20d5	Add user_directory_include_pattern config param to expand search results to additional users ... Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive. It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users... ...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too. Committing it here as a WIP	2017-11-29 16:46:45 +00:00
Richard van der Hoff	ad7e570d07	Delete devices in various logout situations ... Make sure that we delete devices whenever a user is logged out due to any of the following situations: * /logout * /logout_all * change password * deactivate account (by the user or by an admin) * invalidate access token from a dynamic module Fixes #2672.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	ae31f8ce45	Move set_password into its own handler ... Non-functional refactoring to move set_password. This means that we'll be able to properly deactivate devices and access tokens without introducing a dependency loop.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	7ca5c68233	Move deactivate_account into its own handler ... Non-functional refactoring to move deactivate_account. This means that we'll be able to properly deactivate devices and access tokens without introducing a dependency loop.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	2c6d63922a	Remove pushers when deleting access tokens ... Whenever an access token is invalidated, we should remove the associated pushers.	2017-11-29 16:44:35 +00:00
Erik Johnston	dfbc45302e	PEP8	2017-11-28 15:23:26 +00:00
Erik Johnston	c4c1d170af	Fix wrong avatars when inviting multiple users when creating room ... We reused the `content` dictionary between invite requests, which meant they could end up reusing the profile info for a previous user	2017-11-28 15:19:15 +00:00
Luke Barnard	cdf6fc15b0	Merge pull request #2686 from matrix-org/luke/as-flair ... Add automagical AS Publicised Group(s)	2017-11-17 10:13:46 +00:00
Jurek	eded7084d2	Fix auth handler #2678	2017-11-17 10:07:27 +00:00
Matthew Hodgson	34c3d0a386	typo	2017-11-17 01:54:02 +00:00
Luke Barnard	5b48eec4a1	Make sure we check AS groups for lookup on bulk	2017-11-16 17:55:15 +00:00
Jurek	624a8bbd67	Fix auth handler #2678	2017-11-16 17:19:02 +00:00
Luke Barnard	97bd18af4e	Add automagical AS Publicised Group(s) ... via registration file "users" namespace: ```YAML ... namespaces: users: - exclusive: true regex: '.*luke.*' group_id: '+all_the_lukes:hsdomain' ... ``` This is part of giving App Services their own groups for matching users. With this, ghost users will be given the appeareance that they are in a group and that they have publicised the fact, but _only_ from the perspective of the `get_publicised_groups_for_user` API.	2017-11-16 16:44:55 +00:00
Richard van der Hoff	1b83c09c03	Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs ... Remove preserve_context_over_{fn, deferred}	2017-11-15 11:13:53 +00:00
Richard van der Hoff	7e6fa29cb5	Remove preserve_context_over_{fn, deferred} ... Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.	2017-11-14 11:22:42 +00:00
Richard van der Hoff	44a1bfd6a6	Reshuffle room list request code ... I'm not entirely sure if this will actually help anything, but it simplifies the code and might give further clues about why room list search requests are blowing out the get_current_state_ids caches.	2017-11-14 10:29:58 +00:00
Richard van der Hoff	1fc66c7460	Add a load of logging to the room_list handler ... So we can see what it gets up to.	2017-11-14 10:23:47 +00:00
Richard van der Hoff	9b803ccc98	Revert "Allow upper-case characters in mxids" ... This reverts commit b70b646903.	2017-11-09 21:57:24 +00:00
Richard van der Hoff	b70b646903	Allow upper-case characters in mxids ... Because we're never going to be able to fix this :'(	2017-11-09 19:36:13 +00:00
Erik Johnston	e8814410ef	Have an explicit API to update room config	2017-11-08 16:13:27 +00:00
Erik Johnston	94ff2cda73	Revert "Modify group room association API to allow modification of is_public"	2017-11-08 15:43:34 +00:00
Erik Johnston	d305987b40	Merge pull request #2631 from xyzz/fix_appservice_event_backlog ... Fix appservices being backlogged and not receiving new events due to a bug in notify_interested_services	2017-11-08 11:54:10 +00:00
Richard van der Hoff	02a9a93bde	Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res ... Fix bug in state group storage	2017-11-08 09:22:13 +00:00
Richard van der Hoff	e148438e97	s/items/iteritems/	2017-11-08 09:21:41 +00:00
Ilya Zhuravlev	d46386d57e	Remove useless assignment in notify_interested_services	2017-11-07 22:23:22 +03:00
Matthew Hodgson	228ccf1fe3	Merge pull request #2643 from matrix-org/matthew/user_dir_typos ... Fix various embarrassing typos around user_directory and add some doc.	2017-11-07 17:31:11 +00:00
Richard van der Hoff	780dbb378f	Update deltas when doing auth resolution ... Fixes a bug where the persisted state groups were different to those actually being used after auth resolution.	2017-11-07 16:43:00 +00:00
Richard van der Hoff	1ca4288135	factor out _update_context_for_auth_events ... This is duplicated, so let's factor it out before fixing it	2017-11-07 16:43:00 +00:00
Erik Johnston	5ef5e14ecc	Merge pull request #2636 from farialima/me-master ... Fix for #2635: correctly update rooms avatar/display name when modified by admin	2017-11-07 13:49:27 +00:00
Matthew Hodgson	d1622e080f	s/intial/initial/	2017-11-04 19:35:14 +00:00
Francois Granade	f103b91ffa	removed unused import flagged by flake8a	2017-11-03 18:45:49 +01:00
Francois Granade	fa4f337b49	Fix for issue 2635: correctly update rooms avatar/display name when modified by admin	2017-11-03 18:25:04 +01:00
Ilya Zhuravlev	45fbe4ff67	Fix appservices being backlogged and not receiving new events due to a bug in notify_interested_services	2017-11-02 22:49:43 +03:00
Richard van der Hoff	1189be43a2	Factor _AccountHandler proxy out to ModuleApi ... We're going to need to use this from places that aren't password auth, so let's move it to a proper class.	2017-11-02 14:36:11 +00:00
David Baker	b19d9e2174	Merge pull request #2624 from matrix-org/rav/password_provider_notify_logout ... Notify auth providers on logout	2017-11-02 10:55:17 +00:00
David Baker	1f080a6c97	Merge pull request #2623 from matrix-org/rav/callbacks_for_auth_providers ... Allow password_auth_providers to return a callback	2017-11-02 10:49:03 +00:00
David Baker	04897c9dc1	Merge pull request #2622 from matrix-org/rav/db_access_for_auth_providers ... Let auth providers get to the database	2017-11-02 10:41:25 +00:00
Richard van der Hoff	979eed4362	Fix user-interactive password auth ... this got broken in the previous commit	2017-11-01 17:03:20 +00:00
Richard van der Hoff	bc8a5c0330	Notify auth providers on logout ... Provide a hook by which auth providers can be notified of logouts.	2017-11-01 16:51:51 +00:00
Richard van der Hoff	4c8f94ac94	Allow password_auth_providers to return a callback ... ... so that they have a way to record access tokens.	2017-11-01 16:51:03 +00:00
Richard van der Hoff	846a94fbc9	Merge pull request #2620 from matrix-org/rav/auth_non_password ... Let password auth providers handle arbitrary login types	2017-11-01 16:45:33 +00:00
Richard van der Hoff	3cd6b22c7b	Let password auth providers handle arbitrary login types ... Provide a hook where password auth providers can say they know about other login types, and get passed the relevant parameters	2017-11-01 16:43:57 +00:00
David Baker	4f0488b307	Merge remote-tracking branch 'origin/develop' into rav/refactor_accesstoken_delete	2017-11-01 16:20:19 +00:00
David Baker	e5e930aec3	Merge pull request #2615 from matrix-org/rav/break_auth_device_dep ... Break dependency of auth_handler on device_handler	2017-11-01 16:06:31 +00:00
David Baker	fbbacb284e	Merge pull request #2613 from matrix-org/rav/kill_refresh_tokens ... Remove the last vestiges of refresh_tokens	2017-11-01 15:57:35 +00:00
Richard van der Hoff	dd13310fb8	Move access token deletion into auth handler ... Also move duplicated deactivation code into the auth handler. I want to add some hooks when we deactivate an access token, so let's bring it all in here so that there's somewhere to put it.	2017-11-01 15:46:22 +00:00
Richard van der Hoff	74c56f794c	Break dependency of auth_handler on device_handler ... I'm going to need to make the device_handler depend on the auth_handler, so I need to break this dependency to avoid a cycle. It turns out that the auth_handler was only using the device_handler in one place which was an edge case which we can more elegantly handle by throwing an error rather than fixing it up.	2017-11-01 10:27:06 +00:00
Richard van der Hoff	356bcafc44	Remove the last vestiges of refresh_tokens	2017-10-31 20:35:58 +00:00
Richard van der Hoff	3e0aaad190	Let auth providers get to the database ... Somewhat open to abuse, but also somewhat unavoidable :/	2017-10-31 17:22:29 +00:00
Luke Barnard	20fe347906	Modify group room association API to allow modification of is_public ... also includes renamings to make things more consistent.	2017-10-31 17:04:28 +00:00
Richard van der Hoff	1b65ae00ac	Refactor some logic from LoginRestServlet into AuthHandler ... I'm going to need some more flexibility in handling login types in password auth providers, so as a first step, move some stuff from LoginRestServlet into AuthHandler. In particular, we pass everything other than SAML, JWT and token logins down to the AuthHandler, which now has responsibility for checking the login type and fishing the password out of the login dictionary, as well as qualifying the user_id if need be. Ideally SAML, JWT and token would go that way too, but there's no real need for it right now and I'm trying to minimise impact. This commit *should* be non-functional.	2017-10-31 10:48:41 +00:00
Richard van der Hoff	110b373e9c	Merge pull request #2589 from matrix-org/rav/as_deactivate_account ... Allow ASes to deactivate their own users	2017-10-27 12:29:32 +01:00
Richard van der Hoff	785bd7fd75	Allow ASes to deactivate their own users	2017-10-27 00:01:00 +01:00
Erik Johnston	566e21eac8	Update room_list.py	2017-10-26 11:39:54 +01:00
Erik Johnston	2a7e9faeec	Do logcontexts outside ResponseCache	2017-10-25 15:21:08 +01:00
Erik Johnston	33a9026cdf	Add logging and fix log contexts for publicRooms	2017-10-25 10:26:06 +01:00
Richard van der Hoff	eaaabc6c4f	replace 'except:' with 'except Exception:' ... what could possibly go wrong	2017-10-23 15:52:32 +01:00
Richard van der Hoff	3267b81b81	Merge pull request #2561 from matrix-org/rav/id_checking ... Updates to ID checking	2017-10-23 14:39:20 +01:00
Richard van der Hoff	29812c628b	Allow = in mxids and groupids ... ... because the spec says we should.	2017-10-20 23:42:53 +01:00
Richard van der Hoff	58fbbe0f1d	Disallow capital letters in userids ... Factor out a common function for checking user ids and group ids, which forbids capitals.	2017-10-20 23:37:22 +01:00
Richard van der Hoff	631d7b87b5	Remove pointless create() method ... It just calls the constructor, so we may as well kill it rather than having random codepaths.	2017-10-20 22:14:55 +01:00
Luke Barnard	a3ac4f6b0a	_create_rererouter for get_invited_users_in_group	2017-10-16 15:41:03 +01:00
Luke Barnard	2c5972f87f	Implement GET /groups/$groupId/invited_users	2017-10-16 15:31:11 +01:00
Luke Barnard	6079d0027a	Log a warning when no profile for invited member ... And return empty profile	2017-10-16 14:20:45 +01:00
David Baker	b8d8ed1ba9	Merge pull request #2531 from matrix-org/dbkr/spamcheck_error_messages ... Allow error strings from spam checker	2017-10-12 10:31:03 +01:00
Erik Johnston	b752507b48	Fix fetching remote summaries	2017-10-11 16:59:18 +01:00
Richard van der Hoff	68fd82e840	Merge pull request #2530 from matrix-org/rav/fix_receipt_logcontext ... fix a logcontext leak in read receipt handling	2017-10-11 15:08:53 +01:00
David Baker	b78bae2d51	fix isinstance	2017-10-11 14:49:09 +01:00
David Baker	c3b7a45e84	Allow error strings from spam checker	2017-10-11 14:39:22 +01:00
Richard van der Hoff	c3e190ce67	fix a logcontext leak in read receipt handling	2017-10-11 14:37:20 +01:00
Erik Johnston	4ce4379235	Fix attestations to check correct server name	2017-10-11 14:11:43 +01:00
Erik Johnston	535cc49f27	Merge pull request #2466 from matrix-org/erikj/groups_merged ... Initial Group Implementation	2017-10-11 13:20:07 +01:00
Erik Johnston	bc7f3eb32f	Merge pull request #2483 from jeremycline/unfreeze-ujson-dump ... Unfreeze event before serializing with ujson	2017-10-11 13:18:52 +01:00
Erik Johnston	84e27a592d	Merge pull request #2490 from matrix-org/erikj/drop_left_room_events ... Ignore incoming events for rooms that we have left	2017-10-10 11:58:32 +01:00
Richard van der Hoff	c8f568ddf9	Fix up deferred handling in federation.py ... * Avoid preserve_context_over_deferred, which is broken * set consumeErrors=True on defer.gatherResults, to avoid spurious "unhandled failure" erros	2017-10-06 22:14:24 +01:00
David Baker	f878e6f8af	Spam checking: add the invitee to user_may_invite	2017-10-05 14:02:28 +01:00
David Baker	d8ce68b09b	spam check room publishing	2017-10-04 14:29:33 +01:00
David Baker	78d4ced829	un-double indent	2017-10-04 12:44:27 +01:00
David Baker	197c14dbcf	Add room creation checks to spam checker ... Lets the spam checker deny attempts to create rooms and add aliases to them.	2017-10-04 10:47:54 +01:00
David Baker	1e375468de	pass room id too	2017-10-03 17:13:14 +01:00
David Baker	c2c188b699	Federation was passing strings anyway ... so pass string everywhere	2017-10-03 15:46:19 +01:00
David Baker	c46a0d7eb4	this shouldn't be debug	2017-10-03 15:20:14 +01:00
David Baker	bd769a81e1	better logging	2017-10-03 15:16:40 +01:00
David Baker	41fd9989a2	Skip spam check for admin users	2017-10-03 14:17:44 +01:00
Erik Johnston	e4ab96021e	Update comments	2017-10-03 14:10:41 +01:00
David Baker	2a7ed700d5	Fix param name & lint	2017-10-03 14:04:10 +01:00
David Baker	84716d267c	Allow spam checker to reject invites too	2017-10-03 13:56:43 +01:00
Erik Johnston	f2da6df568	Remove spurious line feed	2017-10-03 11:31:06 +01:00
Erik Johnston	30848c0fcd	Ignore incoming events for rooms that we have left ... When synapse receives an event for a room its not in over federation, it double checks with the remote server to see if it is in fact in the room. This is done so that if the server has forgotten about the room (usually as a result of the database being dropped) it can recover from it. However, in the presence of state resets in large rooms, this can cause a lot of work for servers that have legitimately left. As a hacky solution that supports both cases we drop incoming events for rooms that we have explicitly left. This means that we no longer support the case of servers having forgotten that they've rejoined a room, but that is sufficiently rare that we're not going to support it for now.	2017-10-03 11:18:21 +01:00
David Baker	27955056e0	Merge branch 'develop' into erikj/groups_merged	2017-10-02 16:20:41 +01:00
Jeremy Cline	cafb8de132	Unfreeze event before serializing with ujson ... In newer versions of https://github.com/esnme/ultrajson, ujson does not serialize frozendicts (introduced in esnme/ultrajson@53f85b1). Although the PyPI version is still 1.35, Fedora ships with a build from commit esnme/ultrajson@2f1d487. This causes the serialization to fail if the distribution-provided package is used. This runs the event through the unfreeze utility before serializing it. Thanks to @ignatenkobrain for tracking down the root cause. fixes #2351 Signed-off-by: Jeremy Cline <jeremy@jcline.org>	2017-09-30 11:22:37 -04:00
Erik Johnston	8090fd4664	Fix /joined_members to work with AS users	2017-09-28 10:09:32 +01:00
Erik Johnston	adec03395d	Fix bug where /joined_members didn't check user was in room	2017-09-27 15:14:39 +01:00
David Baker	6cd5fcd536	Make the spam checker a module	2017-09-26 19:20:23 +01:00
Erik Johnston	17b8e2bd02	Add remove room API	2017-09-26 15:52:41 +01:00
Erik Johnston	95298783bb	Add is_publicised to group summary	2017-09-26 11:04:37 +01:00
Erik Johnston	e3edca3b5d	Refactor to speed up incremental syncs	2017-09-25 17:35:39 +01:00
Erik Johnston	3166ed55b2	Fix device list when rejoining room (#2461)	2017-09-22 14:44:17 +01:00
Richard van der Hoff	aa620d09a0	Add a config option to block all room invites (#2457) ... - allows sysadmins the ability to lock down their servers so that people can't send their users room invites.	2017-09-19 16:08:14 +01:00
Richard van der Hoff	2eabdf3f98	add some comments to on_exchange_third_party_invite_request	2017-09-19 12:20:36 +01:00
Richard van der Hoff	5ed109d59f	PoC for filtering spammy events (#2456) ... Demonstration of how you might add some hooks to filter out spammy events.	2017-09-19 12:20:11 +01:00
Erik Johnston	93e504d04e	Ensure that creator of group sees group down /sync	2017-09-19 11:08:16 +01:00
Erik Johnston	b5feaa5a49	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/groups_merged	2017-09-19 11:07:45 +01:00
Richard van der Hoff	3f405b34e9	Fix overzealous kicking of guest users (#2453) ... We should only kick guest users if the guest access event is authorised.	2017-09-19 08:52:52 +01:00
Erik Johnston	2d1b7955ae	Don't filter out current state events from timeline	2017-09-18 17:13:03 +01:00
Erik Johnston	d6dadd95ac	Correctly handle leaving room in /key/changes	2017-09-18 15:38:22 +01:00
Erik Johnston	4a94eb3ea4	Fix typo	2017-09-15 09:56:54 +01:00
Erik Johnston	3a0cee28d6	Actually hook leave notifs up	2017-09-14 11:49:37 +01:00
Erik Johnston	4f845a0713	Handle joining/leaving rooms in /keys/changes	2017-09-13 16:28:08 +01:00
Erik Johnston	473700f016	Get left rooms	2017-09-13 15:13:41 +01:00
Erik Johnston	9ce866ed4f	In sync handle device lists for newly joined/left rooms	2017-09-12 16:44:26 +01:00
Erik Johnston	69ef4987a6	Add left section to /keys/changes	2017-09-08 14:44:36 +01:00
Erik Johnston	53cc8ad35a	Send down device list change notif when member leaves/rejoins room	2017-09-07 15:08:39 +01:00
Erik Johnston	4a9b1cf253	Add user profiles to summary from group server	2017-08-25 16:23:58 +01:00
Erik Johnston	258409ef61	Fix typos and reinherit	2017-08-25 14:45:20 +01:00
Erik Johnston	bf81f3cf2c	Split out profile handler to fix tests	2017-08-25 14:34:56 +01:00
Erik Johnston	27ebc5c8f2	Add remote profile cache	2017-08-25 11:25:47 +01:00
Erik Johnston	ba3ff7918b	Fixup	2017-08-11 13:42:42 +01:00
Erik Johnston	ef8e578677	Add bulk group publicised lookup API	2017-08-09 13:36:22 +01:00
Erik Johnston	05e21285aa	Store whether the user wants to publicise their membership of a group	2017-08-08 13:01:46 +01:00
Erik Johnston	a1e67bcb97	Remove stale TODO comments	2017-08-04 10:07:10 +01:00
Erik Johnston	d5e32c843f	Correctly add joins to correct segment	2017-07-24 13:31:26 +01:00
Erik Johnston	96917d5552	Merge pull request #2378 from matrix-org/erikj/group_sync_support ... Add groups to sync stream	2017-07-21 11:05:39 +01:00
Erik Johnston	960dae3340	Add notifier	2017-07-20 17:14:44 +01:00
Erik Johnston	2cc998fed8	Fix replication. And notify	2017-07-20 17:13:18 +01:00
Erik Johnston	139fe30f47	Remember to cast to bool	2017-07-20 16:47:35 +01:00
Erik Johnston	c544188ee3	Add groups to sync stream	2017-07-20 16:36:42 +01:00
Erik Johnston	6f443a74cf	Add update group profile API	2017-07-20 09:46:33 +01:00
Erik Johnston	14a34f12d7	Comments	2017-07-18 17:28:42 +01:00
Erik Johnston	94ecd871a0	Fix typos	2017-07-18 16:38:54 +01:00
Erik Johnston	332839f6ea	Update federation client pokes	2017-07-18 14:45:37 +01:00
Erik Johnston	e5ea6dd021	Add client apis	2017-07-18 14:37:06 +01:00
Erik Johnston	cccfcfa7b9	Comments	2017-07-18 10:35:18 +01:00
Erik Johnston	68f34e85ce	Use transport client directly	2017-07-18 10:29:57 +01:00
Erik Johnston	2f9eafdd36	Add local group server support	2017-07-17 12:03:49 +01:00
Erik Johnston	e9aec001f4	Use less DB for device list handling in sync	2017-07-12 10:30:10 +01:00
Erik Johnston	28e8c46f29	Merge pull request #2352 from matrix-org/erikj/group_server_split ... Initial Group Server	2017-07-12 10:26:29 +01:00
Erik Johnston	9a6fd3ef29	Don't compute push actions for backfilled events	2017-07-11 10:02:21 +01:00
Erik Johnston	b8ca494ee9	Initial group server implementation	2017-07-10 15:44:15 +01:00
Erik Johnston	5a7f561a9b	Fix bug where pusherpool didn't start and broke some rooms ... Since we didn't instansiate the PusherPool at start time it could fail at run time, which it did for some users. This may or may not fix things for those users, but it should happen at start time and stop the server from starting.	2017-07-06 17:55:51 +01:00
Mark Haines	3e279411fe	Improve the error handling for bad invites received over federation	2017-06-30 16:20:30 +01:00
Erik Johnston	c72058bcc6	Use an ExpiringCache for storing registration sessions ... This is because pruning them was a significant performance drain on matrix.org	2017-06-29 14:08:37 +01:00
Erik Johnston	816605a137	Merge pull request #2307 from matrix-org/erikj/user_ip_batch ... Batch upsert user ips	2017-06-27 15:08:32 +01:00
Erik Johnston	a0a561ae85	Fix up client ips to read from pending data	2017-06-27 14:46:12 +01:00
Erik Johnston	1bce3e6b35	Remove unused variables	2017-06-26 14:03:27 +01:00
Erik Johnston	e3cbec10c1	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ensure_round_trip	2017-06-26 14:02:44 +01:00
Erik Johnston	44c722931b	Make some more params configurable	2017-06-22 14:59:52 +01:00
Erik Johnston	2d520a9826	Typo. ARGH.	2017-06-22 14:42:39 +01:00
Erik Johnston	24d894e2e2	Fix thinko in unhandled user spam	2017-06-22 14:39:05 +01:00
Erik Johnston	1217c7da91	Don't work out users who share room with appservice users	2017-06-21 12:00:41 +01:00
Erik Johnston	e5ae386ea4	Handle all cases of sending membership events	2017-06-19 16:07:54 +01:00
Erik Johnston	b490299a3b	Change to create new room and join other users	2017-06-19 14:10:13 +01:00
Erik Johnston	d7fe6b356c	Add shutdown room API	2017-06-19 12:37:27 +01:00
Erik Johnston	fcf01dd88e	Reject local events that don't round trip the DB	2017-06-19 11:33:40 +01:00
Erik Johnston	6aa5bc8635	Initial worker impl	2017-06-16 11:47:11 +01:00
Erik Johnston	a9d6fa8b2b	Include users who share room with requester in user directory	2017-06-15 10:17:21 +01:00
Erik Johnston	4564b05483	Implement updating users who share rooms on the fly	2017-06-15 10:17:17 +01:00
Erik Johnston	72613bc379	Implement initial population of users who share rooms table	2017-06-15 09:59:04 +01:00
Erik Johnston	d9fd937e39	Fix user directory insertion due to missing room_id	2017-06-13 11:50:24 +01:00
Erik Johnston	6ae8373d40	Don't assume existance of events when updating user directory	2017-06-13 10:19:26 +01:00
Erik Johnston	a837765e8c	Merge pull request #2266 from matrix-org/erikj/host_in_room ... Change is_host_joined to use current_state table	2017-06-12 09:49:51 +01:00
Erik Johnston	1664948e41	Comment	2017-06-09 13:05:05 +01:00
Erik Johnston	0185b75381	Change is_host_joined to use current_state table ... This bypasses a bug where using the state groups to figure out if a host is in a room sometimes errors if the servers isn't in the room. (For example when the server rejected an invite to a remote room)	2017-06-09 10:52:26 +01:00
Erik Johnston	7132e5cdff	Mark remote invite rejections as outliers	2017-06-09 10:08:18 +01:00
Erik Johnston	f45f07ab86	Merge pull request #2258 from matrix-org/erikj/user_dir ... Don't start user_directory handling on workers	2017-06-07 14:04:50 +01:00
Erik Johnston	a053ff3979	Merge pull request #2248 from matrix-org/erikj/state_fixup ... Faster cache for get_joined_hosts	2017-06-07 14:01:06 +01:00
Erik Johnston	ecdd2a3658	Don't start user_directory handling on workers	2017-06-07 12:02:53 +01:00
Erik Johnston	2f34ad31ac	Add some logging to user directory	2017-06-07 11:50:44 +01:00
Erik Johnston	09e4bc0501	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_fixup	2017-06-07 11:05:23 +01:00
Erik Johnston	1a01af079e	Handle profile updates in user directory	2017-06-01 15:39:51 +01:00
Erik Johnston	21e255a8f1	Split the table in two	2017-06-01 14:50:46 +01:00
Erik Johnston	7233341eac	Comments	2017-06-01 13:11:38 +01:00
Erik Johnston	8be6fd95a3	Check if host is still in room	2017-06-01 13:05:39 +01:00
Erik Johnston	9c7db2491b	Fix removing users	2017-06-01 11:36:50 +01:00
Erik Johnston	0fe6f3c521	Bug fixes and logging ... - Check if room is public when a user joins before adding to user dir - Fix typo of field name "content.join_rules" -> "content.join_rule"	2017-06-01 11:09:49 +01:00
Erik Johnston	cc7609aa9f	Comment briefly on how we keep user_directory up to date	2017-05-31 17:11:18 +01:00
Erik Johnston	b2d8d07109	Lifts things into separate function	2017-05-31 17:00:24 +01:00
Erik Johnston	f9791498ae	Typos	2017-05-31 16:50:57 +01:00
Erik Johnston	f091061711	Fix tests	2017-05-31 16:34:40 +01:00
Erik Johnston	dc51af3d03	Pull max id from correct table	2017-05-31 15:13:49 +01:00
Erik Johnston	350622a107	Handle the server leaving a public room	2017-05-31 15:11:36 +01:00
Erik Johnston	63fda37e20	Add comments	2017-05-31 15:00:29 +01:00
Erik Johnston	3b5f22ca40	Add search	2017-05-31 14:00:01 +01:00
Erik Johnston	b5db4ed5f6	Update room column when room becomes unpublic	2017-05-31 13:40:28 +01:00
Erik Johnston	168524543f	Add call later	2017-05-31 11:59:36 +01:00
Erik Johnston	3e123b8497	Start later	2017-05-31 11:56:27 +01:00
Erik Johnston	42137efde7	Don't go round in circles	2017-05-31 11:55:13 +01:00
Erik Johnston	eeb2f9e546	Add user_directory to database	2017-05-31 11:51:01 +01:00
Erik Johnston	23da638360	Fix typing tests	2017-05-26 10:02:04 +01:00
Erik Johnston	11f139a647	Merge pull request #2247 from matrix-org/erikj/auth_event ... Only store event_auth for state events	2017-05-24 16:46:34 +01:00
Erik Johnston	c049472b8a	Only store event_auth for state events	2017-05-24 15:23:31 +01:00
Erik Johnston	b4f59c7e27	Add count of one time keys to sync stream	2017-05-19 15:47:55 +01:00
Erik Johnston	cafe659f72	Store ActionGenerator in HomeServer	2017-05-19 10:09:56 +01:00
Erik Johnston	66d8ffabbd	Faster push rule calculation via push specific cache ... We add a push rule specific cache that ensures that we can reuse calculated push rules appropriately when a user join/leaves.	2017-05-17 16:55:40 +01:00
Erik Johnston	6fa8148ccb	Merge pull request #2228 from matrix-org/erikj/speed_up_get_hosts ... Speed up get_joined_hosts	2017-05-16 17:40:55 +01:00
Erik Johnston	13f540ef1b	Speed up get_joined_hosts	2017-05-16 16:05:22 +01:00
Erik Johnston	ec5c4499f4	Make presence use cached users/hosts in room	2017-05-16 16:01:43 +01:00
Luke Barnard	ccad2ed824	Modify condition on empty localpart	2017-05-10 17:34:30 +01:00
Luke Barnard	369195caa5	Modify register/available to be GET with query param ... - GET is now the method for register/available - a query parameter "username" is now used Also, empty usernames are now handled with an error message on registration or via register/available: `User ID cannot be empty`	2017-05-10 17:23:55 +01:00
Erik Johnston	a3648f84b2	Merge pull request #2208 from matrix-org/erikj/ratelimit_overrid ... Add per user ratelimiting overrides	2017-05-10 15:54:48 +01:00
Erik Johnston	b990b2fce5	Add per user ratelimiting overrides	2017-05-10 11:05:43 +01:00
Richard van der Hoff	aedaba018f	Replace some instances of preserve_context_over_deferred	2017-05-09 19:04:56 +01:00
Richard van der Hoff	de042b3b88	Do some logging when one-time-keys get claimed ... might help us figure out if https://github.com/vector-im/riot-web/issues/3868 has happened.	2017-05-09 19:04:56 +01:00
Richard van der Hoff	a7e9d8762d	Allow clients to upload one-time-keys with new sigs ... When a client retries a key upload, don't give an error if the signature has changed (but the key is the same). Fixes https://github.com/vector-im/riot-android/issues/1208, hopefully.	2017-05-09 19:04:56 +01:00
Erik Johnston	27c28eaa27	Merge pull request #2190 from matrix-org/erikj/mark_remote_as_back_more ... Always mark remotes as up if we receive a signed request from them	2017-05-05 14:08:12 +01:00
Erik Johnston	653d90c1a5	Comment	2017-05-05 14:01:17 +01:00
Erik Johnston	7b222fc56e	Remove redundant reset of destination timers	2017-05-05 11:14:09 +01:00
Erik Johnston	b843631d71	Add comment and TODO	2017-05-05 10:59:32 +01:00
Erik Johnston	f346048a6e	Handle exceptions thrown in handling remote device list updates	2017-05-05 10:34:10 +01:00
David Baker	60833c8978	Merge pull request #2147 from matrix-org/dbkr/http_request_propagate_error ... Propagate errors sensibly from proxied IS requests	2017-05-03 11:23:25 +01:00
Erik Johnston	0c27383dd7	Merge pull request #2170 from matrix-org/erikj/fed_hole_state ... Don't fetch state for missing events that we fetched	2017-05-03 10:49:37 +01:00
Erik Johnston	ef862186dd	Merge together redundant calculations/logging	2017-05-03 10:06:43 +01:00
Erik Johnston	3e5a62ecd8	Add more granular event send metrics	2017-05-02 14:23:26 +01:00
Erik Johnston	2347efc065	Fixup	2017-04-28 12:46:53 +01:00
Erik Johnston	9b147cd730	Remove unncessary call in _get_missing_events_for_pdu	2017-04-28 11:55:25 +01:00
Erik Johnston	3a9f5bf6dd	Don't fetch state for missing events that we fetched	2017-04-28 11:26:46 +01:00
Erik Johnston	25a96e0c63	Merge pull request #2163 from matrix-org/erikj/fix_invite_state ... Fix invite state to always include all events	2017-04-27 17:36:30 +01:00
Erik Johnston	87ae59f5e9	Typo	2017-04-27 15:16:21 +01:00
Erik Johnston	e42b4ebf0f	Add some extra logging for edge cases of federation	2017-04-27 14:38:21 +01:00
Erik Johnston	34e682d385	Fix invite state to always include all events	2017-04-26 16:18:08 +01:00
David Baker	82ae0238f9	Revert accidental commit	2017-04-26 11:43:16 +01:00
David Baker	81804909d3	Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error	2017-04-26 11:31:55 +01:00
David Baker	1a9255c12e	Use CodeMessageException subclass instead ... Parse json errors from get_json client methods and throw special errors.	2017-04-25 19:30:55 +01:00
David Baker	2e165295b7	Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error	2017-04-21 11:35:52 +01:00
David Baker	a90a0f5c8a	Propagate errors sensibly from proxied IS requests ... When we're proxying Matrix endpoints, parse out Matrix error responses and turn them into SynapseErrors so they can be propagated sensibly upstream.	2017-04-21 11:32:48 +01:00
Richard van der Hoff	0cdb32fc43	Remove redundant try/except clauses ... The `except SynapseError` clauses were pointless because the wrapped functions would never throw a `SynapseError` (they either throw a `CodeMessageException` or a `RuntimeError`). The `except CodeMessageException` is now also pointless because the caller treats all exceptions equally, so we may as well just throw the `CodeMessageException`.	2017-04-21 01:32:01 +01:00
Richard van der Hoff	838810b76a	Broaden the conditions for locally_rejecting invites ... The logic for marking invites as locally rejected was all well and good, but didn't happen when the remote server returned a 500, or wasn't reachable, or had no DNS, or whatever. Just expand the except clause to catch everything. Fixes https://github.com/matrix-org/synapse/issues/761.	2017-04-21 01:31:37 +01:00
Richard van der Hoff	736b9a4784	Remove redundant function ... inline `reject_remote_invite`, which only existed to make tracing the callflow more difficult.	2017-04-21 01:31:09 +01:00
Luke Barnard	3fb8784c92	m.read_marker -> m.fully_read (#2128) ... Also: - change the REST endpoint to have a "S" on the end (so it's now /read_markers) - change the content of the m.read_up_to event to have the key "event_id" instead of "marker".	2017-04-18 17:46:15 +01:00
Luke Barnard	78f0ddbfad	Merge pull request #2120 from matrix-org/luke/read-markers ... Implement Read Marker API	2017-04-13 14:21:31 +01:00
Luke Barnard	b9557064bf	Simplify is_event_after logic	2017-04-12 14:36:20 +01:00
Erik Johnston	247c736b9b	Merge pull request #2115 from matrix-org/erikj/dedupe_federation_repl ... Reduce federation replication traffic	2017-04-12 11:07:13 +01:00
Luke Barnard	69a18514e9	Only notify user, not entire room	2017-04-12 10:50:37 +01:00
Luke Barnard	122cd52ce4	Remove comment, simplify null-guard	2017-04-12 10:48:32 +01:00
Erik Johnston	c7ddb5ef7a	Reuse get_interested_parties	2017-04-12 10:16:26 +01:00
Luke Barnard	77fb2b72ae	Handle no previous RM	2017-04-12 09:47:29 +01:00
Luke Barnard	867822fa1e	flake8	2017-04-11 17:36:04 +01:00
Luke Barnard	73880268ef	Refactor event ordering check to events store	2017-04-11 17:34:09 +01:00
Luke Barnard	131485ef66	Copyright	2017-04-11 17:33:51 +01:00
Luke Barnard	0127423027	flake8	2017-04-11 17:07:07 +01:00
Erik Johnston	414522aed5	Move get_interested_parties	2017-04-11 15:33:26 +01:00
Erik Johnston	2be8a281d2	Comments	2017-04-11 15:28:24 +01:00
Erik Johnston	6308ac45b0	Move get_interested_remotes back to presence handler	2017-04-11 15:19:26 +01:00
Luke Barnard	d892079844	Finish implementing RM endpoint ... - This change causes a 405 to be sent if "m.read_marker" is set via /account_data - This also fixes-up the RM endpoint so that it actually Works.	2017-04-11 15:01:39 +01:00
lukebarnard	e263c26690	Initial commit of RM server-side impl ... (See https://docs.google.com/document/d/1UWqdS-e1sdwkLDUY0wA4gZyIkRp-ekjsLZ8k6g_Zvso/edit#heading=h.lndohpg8at5u)	2017-04-11 11:55:30 +01:00
Erik Johnston	f3cf3ff8b6	Merge branch 'master' of github.com:matrix-org/synapse into develop	2017-04-11 11:13:32 +01:00
Erik Johnston	29574fd5b3	Reduce federation presence replication traffic ... This is mainly done by moving the calculation of where to send presence updates from the presence handler to the transaction queue, so we only need to send the presence event (and not the destinations) across the replication connection. Before we were duplicating by sending the full state across once per destination.	2017-04-10 16:48:30 +01:00
Richard van der Hoff	64765e5199	When we do an invite rejection, save the signed leave event to the db ... During a rejection of an invite received over federation, we ask a remote server to make us a `leave` event, then sign it, then send that with `send_leave`. We were saving the *unsigned* version of the event (which has a different event id to the signed version) to our db (and sending it to the clients), whereas other servers in the room will have seen the *signed* version. We're not aware of any actual problems that caused, except that it makes the database confusing to look at and generally leaves the room in a weird state.	2017-04-07 14:39:32 +01:00
Erik Johnston	98ce212093	Merge pull request #2103 from matrix-org/erikj/no-double-encode ... Don't double encode replication data	2017-04-07 09:39:52 +01:00
Erik Johnston	877c029c16	Use iteritems	2017-04-06 15:51:22 +01:00
Erik Johnston	96b9b6c127	Don't double json encode typing replication data	2017-04-05 11:34:20 +01:00
Erik Johnston	62b89daac6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/repl_tcp_server	2017-04-04 09:46:16 +01:00
Richard van der Hoff	773e64cc1a	Merge pull request #2095 from matrix-org/rav/cull_log_preserves ... Cull spurious PreserveLoggingContexts	2017-04-03 17:02:25 +01:00
Richard van der Hoff	30bcbf775a	Accept join events from all servers ... Make sure that we accept join events from any server, rather than just the origin server, to make the federation join dance work correctly. (Fixes #1893).	2017-04-03 15:58:07 +01:00
Richard van der Hoff	7eb9f34cc3	Remove spurious yield ... In `MessageHandler`, remove `yield` on call to `Notifier.on_new_room_event`: it doesn't return anything anyway.	2017-04-03 15:44:19 +01:00
Richard van der Hoff	0b08c48fc5	Remove more spurious PreserveLoggingContexts ... Remove `PreserveLoggingContext` around calls to `Notifier.on_new_room_event`; there is no problem if the logcontext is set when calling it.	2017-04-03 15:43:37 +01:00
Erik Johnston	36d2b66f90	Add a timestamp to USER_SYNC command ... This timestamp is used to indicate when the user last sync'd	2017-03-31 15:42:22 +01:00
Erik Johnston	9d0170ac6c	Fix up presence	2017-03-31 11:36:32 +01:00
Erik Johnston	63fcc42990	Remove user from process_presence when stops syncing	2017-03-30 14:26:08 +01:00
Erik Johnston	e9dd8370b0	Add functions to presence to support remote syncs ... The TCP replication protocol streams deltas of who has started or stopped syncing. This is different from the HTTP API which periodically sends the full list of users who are syncing. This commit adds support for the new TCP style of sending deltas.	2017-03-30 13:25:14 +01:00
Erik Johnston	24d35ab47b	Add new storage functions for new replication ... The new replication protocol will keep all the streams separate, rather than muxing multiple streams into one.	2017-03-30 11:48:35 +01:00
Erik Johnston	7fc1f1e2b6	Cache hosts in room	2017-03-24 11:46:24 +00:00
Richard van der Hoff	9397edb28b	Merge pull request #2050 from matrix-org/rav/federation_backoff ... push federation retry limiter down to matrixfederationclient	2017-03-23 22:27:01 +00:00
Richard van der Hoff	5a16cb4bf0	Ignore backoff history for invites, aliases, and roomdirs ... Add a param to the federation client which lets us ignore historical backoff data for federation queries, and set it for a handful of operations.	2017-03-23 12:23:22 +00:00
Richard van der Hoff	4bd597d9fc	push federation retry limiter down to matrixfederationclient ... rather than having to instrument everywhere we make a federation call, make the MatrixFederationHttpClient manage the retry limiter.	2017-03-23 09:28:46 +00:00
Richard van der Hoff	e08f81d96a	Add a missing yield in device key upload ... (this would only very very rarely actually be a useful thing, so the main problem was the logcontext leak...)	2017-03-23 00:16:43 +00:00
Richard van der Hoff	eddce9d74a	Merge pull request #2027 from matrix-org/rav/logcontext_leaks ... A few fixes to logcontext things	2017-03-20 11:53:36 +00:00
Richard van der Hoff	f40c2db05a	Stop preserve_fn leaking context into the reactor ... Fix a bug in ``logcontext.preserve_fn`` which made it leak context into the reactor, and add a test for it. Also, get rid of ``logcontext.reset_context_after_deferred``, which tried to do the same thing but had its own, different, set of bugs.	2017-03-18 00:07:43 +00:00
Richard van der Hoff	5068fb16a5	Refactoring and cleanups ... A few non-functional changes: * A bunch of docstrings to document types * Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more readable. * Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid mind-bending conditional. * Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid mind-bending conditional.	2017-03-17 15:06:07 +00:00
Richard van der Hoff	2abe85d50e	Merge pull request #2016 from matrix-org/rav/queue_pdus_during_join ... Queue up federation PDUs while a room join is in progress	2017-03-17 11:32:44 +00:00
Erik Johnston	da146657c9	Comments	2017-03-16 13:04:07 +00:00
Erik Johnston	a158c36a8a	Comment	2017-03-16 11:57:45 +00:00
Erik Johnston	6957bfdca6	Don't recreate so many sets	2017-03-16 11:54:26 +00:00
Erik Johnston	2ccf3b241c	Implement no op for room stream in sync	2017-03-16 11:06:41 +00:00
Richard van der Hoff	9ce53a3861	Queue up federation PDUs while a room join is in progress ... This just takes the existing `room_queues` logic and moves it out to `on_receive_pdu` instead of `_process_received_pdu`, which ensures that we don't start trying to fetch prev_events and whathaveyou until the join has completed.	2017-03-15 18:01:11 +00:00
Erik Johnston	54d2b7e596	Merge pull request #2014 from Half-Shot/hs/fix-appservice-presence ... Add fallback to last_active_ts if it beats the last sync time on a presence timeout.	2017-03-15 17:37:15 +00:00
Erik Johnston	f83ac78201	Cache set of users whose presence the other user should see	2017-03-15 15:29:19 +00:00
Will Hunt	e6032054bf	Add a great comment to handle_timeout for active vs sync times.	2017-03-15 15:24:48 +00:00
Will Hunt	ebf5a6b14c	Add fallback to last_active_ts if it beats the last sync time.	2017-03-15 15:17:16 +00:00
Erik Johnston	e892457a03	Comment	2017-03-15 15:01:39 +00:00
Erik Johnston	6c82de5100	Format presence events on the edges instead of reformatting them multiple times	2017-03-15 14:27:34 +00:00
Erik Johnston	7827251daf	Merge pull request #1994 from matrix-org/dbkr/msisdn_signin_2 ... Phone number registration / login support v2	2017-03-15 09:59:54 +00:00
Erik Johnston	bb256ac96f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed	2017-03-14 11:35:05 +00:00
Luke Barnard	bbeeb97f75	Implement _simple_delete_many_txn, use it to delete devices ... (But this doesn't implement the same for deleting access tokens or e2e keys. Also respond to code review.	2017-03-13 17:53:23 +00:00
David Baker	73a5f06652	Support registration / login with phone number ... Changes from https://github.com/matrix-org/synapse/pull/1971	2017-03-13 17:27:51 +00:00
Richard van der Hoff	3c69f32402	Merge remote-tracking branch 'origin/develop' into rav/refactor_received_pdu	2017-03-13 12:20:47 +00:00
Erik Johnston	0162994983	Comments	2017-03-13 11:53:26 +00:00
Erik Johnston	7eae6eaa2f	Revert "Support registration & login with phone number"	2017-03-13 09:59:33 +00:00
Erik Johnston	79926e016e	Assume rooms likely haven't changed	2017-03-13 09:50:10 +00:00
Erik Johnston	8ffbe43ba1	Get current state by using current_state_events table	2017-03-10 17:39:35 +00:00
Erik Johnston	64d62e41b8	Noop repated delete device inbox calls from /sync	2017-03-10 10:36:43 +00:00
Richard van der Hoff	29235901b8	Move FederationServer._handle_new_pdu to FederationHandler ... Unfortunately this significantly increases the size of the already-rather-big FederationHandler, but the code fits more naturally here, and it paves the way for the tighter integration that I need between handling incoming PDUs and doing the join dance. Other than renaming the existing `FederationHandler.on_receive_pdu` to `_process_received_pdu` to make way for it, this just consists of the move, and replacing `self.handler` with `self` and `self` with `self.replication_layer`.	2017-03-09 16:20:13 +00:00
David Baker	663396e45d	Merge pull request #1971 from matrix-org/dbkr/msisdn_signin ... Support registration & login with phone number	2017-03-09 10:18:51 +00:00
David Baker	0e0aee25c4	Fix log line	2017-03-08 11:46:22 +00:00
David Baker	88df6c0c9a	Factor out msisdn canonicalisation ... Plus a couple of other minor fixes	2017-03-08 11:03:39 +00:00
David Baker	402a7bf63d	Fix pep8	2017-03-08 09:33:40 +00:00
Erik Johnston	f2581ee8b8	Don't keep around old stream IDs forever	2017-03-03 16:02:53 +00:00
Erik Johnston	9834367eea	Spelling	2017-03-03 15:31:57 +00:00
David Baker	ad882cd54d	Just return the deferred straight off ... defer.returnValue doth not maketh a generator: it would need a yield to be a generator, and this doesn't need a yield.	2017-03-01 18:08:51 +00:00
David Baker	3557cf34dc	Merge remote-tracking branch 'origin/develop' into dbkr/msisdn_signin	2017-03-01 17:20:37 +00:00
Erik Johnston	36be39b8b3	Fix device list update to not constantly resync	2017-03-01 14:12:11 +00:00
Erik Johnston	848cf95ea0	Pop with default value to stop throwing	2017-02-28 10:02:54 +00:00
Erik Johnston	64a2cef9bb	Pop rather than del from dict	2017-02-27 19:15:36 +00:00
Erik Johnston	17673404fb	Remove unused param	2017-02-20 15:02:01 +00:00
Erik Johnston	7f026792e1	Fix /context/ visibiltiy rules	2017-02-20 14:54:50 +00:00
David Baker	8c87bb550e	Merge pull request #1922 from matrix-org/dbkr/allow_forget_for_ban ... Allow forgetting rooms you're banned from	2017-02-17 10:52:30 +00:00
David Baker	474c9aadbe	Allow forgetting rooms you're banned from	2017-02-15 19:32:20 +00:00
David Baker	ce3e583d94	WIP support for msisdn 3pid proxy methods	2017-02-14 15:05:55 +00:00
Richard van der Hoff	fc2f29c1d0	Fix bugs in the /keys/changes api ... * `get_forward_extremeties_for_room` takes a numeric `stream_ordering`. We were passing a `RoomStreamToken`, which meant that it returned the *current* extremities, rather than those corresponding to the `from_token`. However: * `get_state_ids_for_events` required a second ('types') parameter; this meant that a `TypeError` was thrown and we ended up acting as though there was *no* prev state. * `get_state_ids_for_events` actually returns a map from event_id to state dictionary - just looking up the state keys in it again meant that we acted as though there was no prev state. We now check if each member's state has changed since *any* of the extremities. Also add/fix some comments.	2017-02-14 13:59:50 +00:00
Erik Johnston	9e617cd4c2	Cache get_presence storage	2017-02-13 13:50:03 +00:00
Erik Johnston	af6da6db2d	Merge pull request #1784 from morteza-araby/user-admin ... Administration functionalities	2017-02-06 16:21:10 +01:00
Erik Johnston	a597994fb6	Measure new device list stuff	2017-02-02 18:36:17 +00:00
Erik Johnston	1232ae41cf	Use new get_users_who_share_room_with_user	2017-02-02 15:25:00 +00:00
Erik Johnston	99fa03e8b5	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/device_list_fixes	2017-02-02 15:23:45 +00:00
Erik Johnston	a8331897aa	Merge pull request #1876 from matrix-org/erikj/shared_member_store ... Make presence.get_new_events a bit faster	2017-02-02 15:20:14 +00:00
Erik Johnston	46ecd9fd6d	Use stream_ordering_to_exterm for /keys/changes	2017-02-02 14:27:19 +00:00
Erik Johnston	54a79c1d37	Make presence.get_new_events a bit faster ... We do this by caching the set of users a user shares rooms with.	2017-02-02 13:07:18 +00:00
Morteza Araby	2849d3f29d	admin,storage: added more administrator functionalities ... administrators can now: - Set displayname of users - Update user avatars - Search for users by user_id - Browse all users in a paginated API - Reset user passwords - Deactivate users Helpers for doing paginated queries has also been added to storage Signed-off-by: Morteza Araby <morteza.araby@ericsson.com>	2017-02-02 14:02:26 +01:00
Erik Johnston	51adaac953	Fix email push in pusher worker ... This was broken when device list updates were implemented, as Mailer could no longer instantiate an AuthHandler due to a dependency on federation sending.	2017-02-02 10:53:36 +00:00
Erik Johnston	fbfe44bb4d	Doc args	2017-02-01 17:52:57 +00:00
Erik Johnston	7e919bdbd0	Include newly joined users in /keys/changes API	2017-02-01 17:33:16 +00:00
Erik Johnston	6d6591880e	Wake sync up for device changes	2017-02-01 15:15:16 +00:00
Erik Johnston	97479d0c54	Implement /keys/changes	2017-02-01 10:30:03 +00:00
Erik Johnston	bfb3d255b1	Merge pull request #1862 from matrix-org/erikj/presence_update ... Use DB cache of joined users for presence	2017-01-31 13:23:24 +00:00
Erik Johnston	c7a26b7c32	Fix unit tests	2017-01-30 17:11:24 +00:00
Erik Johnston	fd1c18c088	Use DB cache of joined users for presence	2017-01-30 17:00:24 +00:00
Erik Johnston	c2c9a78db9	Noop device key changes if they're the same	2017-01-30 16:55:04 +00:00
Erik Johnston	828db669ec	Use get_users_in_room and declare it iterable	2017-01-30 16:37:22 +00:00
Erik Johnston	9636b2407d	Merge pull request #1857 from matrix-org/erikj/device_list_stream ... Implement device lists updates over federation	2017-01-30 14:35:21 +00:00
Erik Johnston	252b503fc8	Hook device list updates to replication	2017-01-27 14:31:35 +00:00
Erik Johnston	888c59c955	Better name	2017-01-27 10:29:47 +00:00
Erik Johnston	76d40f4904	Handle users leaving rooms	2017-01-26 16:39:33 +00:00
Erik Johnston	fbfad76c03	Add comments	2017-01-26 16:33:21 +00:00
Erik Johnston	c974116f19	Implement device key caching over federation	2017-01-26 16:07:24 +00:00
Erik Johnston	51e9fe36e4	Fix up sending of m.device_list_update edus	2017-01-25 16:55:21 +00:00
Erik Johnston	2367c5568c	Add basic implementation of local device list changes	2017-01-25 14:27:27 +00:00
Paul "LeoNerd" Evans	10e48d8310	Don't clobber a displayname or avatar_url if provided by an m.room.member event	2017-01-24 18:06:07 +00:00
Erik Johnston	a55fa2047f	Insert delta of current_state_events to be more efficient	2017-01-20 17:10:18 +00:00
Erik Johnston	09eb08f910	Derive current_state_events from state groups	2017-01-20 11:52:51 +00:00
Erik Johnston	f9058ca785	Merge pull request #1822 from matrix-org/erikj/statE_logging ... Change resolve_state_groups call site logging to DEBUG	2017-01-18 11:02:03 +00:00
Erik Johnston	15f012032c	Merge pull request #1818 from matrix-org/erikj/state_auth_splitout_split ... Optimise state resolution	2017-01-18 10:53:00 +00:00
Erik Johnston	5f027d1fc5	Change resolve_state_groups call site logging to DEBUG	2017-01-17 17:07:15 +00:00
Erik Johnston	e6153e1bd1	Fix couple of federation state bugs	2017-01-17 13:22:34 +00:00
Erik Johnston	5d6bad1b3c	Optimise state resolution	2017-01-17 13:22:19 +00:00
Erik Johnston	7b62d0bc70	Add missing None check	2017-01-11 10:57:03 +00:00
Erik Johnston	657488113e	Merge pull request #1789 from matrix-org/erikj/decouple_presence ... Don't block messages sending on bumping presence	2017-01-10 14:06:05 +00:00
Erik Johnston	3b4de17d2b	Comment	2017-01-10 14:05:53 +00:00
Erik Johnston	7d0981b312	Merge pull request #1787 from matrix-org/erikj/linearize_member ... Linearize updates to membership via PUT /state/	2017-01-10 14:04:54 +00:00
Erik Johnston	586f474a44	Don't block messages sending on bumping presence	2017-01-10 12:46:00 +00:00
Erik Johnston	6823fe5241	Linearize updates to membership via PUT /state/	2017-01-09 18:25:13 +00:00
Erik Johnston	f7085ac84f	Name linearizer's for better logs	2017-01-09 17:17:10 +00:00
Mark Haines	06c00bd19b	Merge branch 'release-v0.18.6' into develop	2017-01-06 14:46:27 +00:00
Patrik Oldsberg	1ef6084b75	handlers/room_member: fix guest access check when joining rooms ... Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>	2017-01-06 14:36:56 +00:00
Matthew Hodgson	bd85434cb3	Merge branch 'release-v0.18.6' into develop	2017-01-05 13:58:19 +00:00
Matthew Hodgson	883ff92a7f	Fix case	2017-01-05 13:45:02 +00:00
Matthew Hodgson	d79d165761	add logging for all the places we call resolve_state_groups. my kingdom for a backtrace that actually works.	2017-01-05 13:40:39 +00:00
Matthew Hodgson	b31ed22738	Merge branch 'release-v0.18.6' into develop	2017-01-05 13:03:02 +00:00
Mark Haines	f784980d2b	Only send events that originate on this server. ... Or events that are sent via the federation "send_join" API. This should match the behaviour from before v0.18.5 and #1635 landed.	2017-01-05 11:26:30 +00:00
David Baker	84cf00c645	Fix another comment typo	2016-12-21 09:51:43 +00:00
David Baker	0c88ab1844	Add /account/3pid/delete endpoint ... Also fix a typo in a comment	2016-12-20 18:27:30 +00:00
Matthew Hodgson	f2a5aebf98	fix ability to change password to a non-ascii one ... https://github.com/vector-im/riot-web/issues/2658	2016-12-18 22:25:21 +00:00
Erik Johnston	b2f8642d3d	Cache network room list queries.	2016-12-16 16:11:43 +00:00
Richard van der Hoff	edc6a1e4f9	Add some logging for syncing to_device events ... Attempt to track down the loss of to_device events (https://github.com/vector-im/riot-web/issues/2711 etc).	2016-12-15 18:16:10 +00:00
Erik Johnston	35129ac998	Merge pull request #1698 from matrix-org/erikj/room_list ... Fix caching on public room list	2016-12-15 15:40:28 +00:00
Erik Johnston	b8255eba26	Comment	2016-12-14 13:49:54 +00:00
Erik Johnston	b2999a7055	Fix /backfill returning events it shouldn't	2016-12-14 13:41:45 +00:00
Erik Johnston	1de53a7a1a	Fix caching on public room list	2016-12-13 17:33:24 +00:00
Erik Johnston	1574b839e0	Merge pull request #1676 from matrix-org/erikj/room_list ... Add new API appservice specific public room list	2016-12-12 17:00:10 +00:00
Erik Johnston	d53a80af25	Merge pull request #1620 from matrix-org/erikj/concurrent_room_access ... Limit the number of events that can be created on a given room concurrently	2016-12-12 10:30:23 +00:00
Richard van der Hoff	85cd30b1fd	Merge pull request #1686 from matrix-org/rav/fix_federation_key_fails ... E2E key query: handle federation fails	2016-12-12 09:33:39 +00:00
Richard van der Hoff	deca951241	Remove unused import	2016-12-12 09:24:35 +00:00
Richard van der Hoff	efa4ccfaee	E2E key query: handle federation fails ... Don't fail the whole request if we can't connect to a particular server.	2016-12-09 18:31:01 +00:00
Erik Johnston	be14c24cea	Fix rare notifier bug where listeners dont timeout ... There was a race condition that caused the notifier to 'miss' the timeout notification, since there were no other checks for the timeout this caused listeners to get stuck in a loop until something happened.	2016-12-09 15:43:18 +00:00
Richard van der Hoff	b08f76bd23	Fix ignored read-receipts ... Don't ignore read-receipts which arrive in the same EDU as a read-receipt for an old event.	2016-12-08 12:13:01 +00:00
Erik Johnston	2687af82d4	Comments	2016-12-07 09:58:33 +00:00
Erik Johnston	f32fb65552	Add new API appservice specific public room list	2016-12-06 16:12:27 +00:00
Richard van der Hoff	1529c19675	Prevent user tokens being used as guest tokens (#1675) ... Make sure that a user cannot pretend to be a guest by adding 'guest = True' caveats.	2016-12-06 15:31:37 +00:00
Erik Johnston	9609c91e7d	Merge pull request #653 from matrix-org/erikj/preset_guest_join ... Enable guest access for private rooms by default	2016-12-05 17:47:14 +00:00
Erik Johnston	338df4f409	Merge pull request #1649 from matrix-org/dbkr/log_ui_auth_args ... Log the args that we have on UI auth completion	2016-12-05 16:40:58 +00:00
Richard van der Hoff	aa09d6b8f0	Rip out more refresh_token code ... We might as well treat all refresh_tokens as invalid. Just return a 403 from /tokenrefresh, so that we don't have a load of dead, untestable code hanging around. Still TODO: removing the table from the schema.	2016-11-30 17:40:18 +00:00
Richard van der Hoff	dc4b23e1a1	Merge branch 'develop' into rav/no_more_refresh_tokens	2016-11-30 17:10:04 +00:00
Richard van der Hoff	1c4f05db41	Stop putting a time caveat on access tokens ... The 'time' caveat on the access tokens was something of a lie, since we weren't enforcing it; more pertinently its presence stops us ever adding useful time caveats. Let's move in the right direction by not lying in our caveats.	2016-11-29 16:49:41 +00:00
Richard van der Hoff	5c4edc83b5	Stop generating refresh tokens ... Since we're not doing refresh tokens any more, we should start killing off the dead code paths. /tokenrefresh itself is a bit of a thornier subject, since there might be apps out there using it, but we can at least not generate refresh tokens on new logins.	2016-11-28 10:13:01 +00:00
David Baker	c9d4e7b716	Clarify that creds doesn not contain passwords.	2016-11-24 10:54:59 +00:00
David Baker	f681aab895	Log the args that we have on UI auth completion ... This will be super helpful for debugging if we have more registration woes.	2016-11-24 10:11:45 +00:00
Erik Johnston	feec718265	Shuffle receipt handler around so that worker apps don't need to load it	2016-11-23 15:14:24 +00:00
Erik Johnston	b69f76c106	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_fed_txn	2016-11-23 11:31:53 +00:00
Kegan Dougal	83bcdcee61	Return early on /sync code paths if a '*' filter is used ... This is currently very conservative in that it only does this if there is no `since` token. This limits the risk to clients likely to be doing one-off syncs (like bridges), but does mean that normal human clients won't benefit from the time savings here. If the savings are large enough, I would consider generalising this to just check the filter.	2016-11-22 16:38:35 +00:00
Erik Johnston	9687e039e7	Remove explicit calls to send_pdu	2016-11-21 14:48:51 +00:00
Erik Johnston	f8ee66250a	Handle sending events and device messages over federation	2016-11-17 15:48:04 +00:00
Erik Johnston	59ef517e6b	Use new federation_sender DI	2016-11-16 14:47:52 +00:00
Erik Johnston	d56c39cf24	Use external ldap auth pacakge	2016-11-15 13:03:19 +00:00
Erik Johnston	64038b806c	Comments	2016-11-11 10:42:08 +00:00
Erik Johnston	2bd4513a4d	Limit the number of events that can be created on a given room concurretnly	2016-11-10 16:44:35 +00:00
Erik Johnston	a4632783fb	Sample correctly	2016-11-08 11:20:26 +00:00
Erik Johnston	eeda4e618c	Limit the number of prev_events of new events	2016-11-08 11:02:29 +00:00
David Baker	9084720993	Don't error on non-ascii passwords	2016-11-03 10:42:14 +00:00
Erik Johnston	2ef617bc06	Fix infinite typing bug ... There's a bug somewhere that causes typing notifications to not be timed out properly. By adding a paranoia timer and using correct inequalities notifications should stop being stuck, even if it the root cause hasn't been fixed.	2016-10-24 15:51:22 +01:00
Erik Johnston	d04e2ff3a4	Fix incredubly slow back pagination query ... If a client didn't specify a from token when paginating backwards synapse would attempt to query the (global) maximum topological token. This a) doesn't make much sense since they're room specific and b) there are no indices that lets postgres do this efficiently.	2016-10-24 13:35:51 +01:00
Luke Barnard	e01a1bc92d	Merge pull request #1175 from matrix-org/luke/feature-configurable-as-rate-limiting ... Allow Configurable Rate Limiting Per AS	2016-10-20 16:21:10 +01:00
Luke Barnard	f09db236b1	as_user->app_service, less redundant comments, better positioned comments	2016-10-20 12:04:54 +01:00
Luke Barnard	1b17d1a106	Use real AS object by passing it through the requester ... This means synapse does not have to check if the AS is interested, but instead it effectively re-uses what it already knew about the requesting user	2016-10-20 11:43:05 +01:00
Erik Johnston	3aa8925091	Merge pull request #1176 from matrix-org/erikj/eager_ratelimit_check ... Check whether to ratelimit sooner to avoid work	2016-10-19 14:25:52 +01:00
Erik Johnston	f2f74ffce6	Comment	2016-10-19 14:21:28 +01:00
David Baker	df2a616c7b	Convert emails to lowercase when storing ... And db migration sql to convert existing addresses.	2016-10-19 11:13:55 +01:00
Erik Johnston	550308c7a1	Check whether to ratelimit sooner to avoid work	2016-10-19 10:45:24 +01:00
Luke Barnard	5b54d51d1e	Allow Configurable Rate Limiting Per AS ... This adds a flag loaded from the registration file of an AS that will determine whether or not its users are rate limited (by ratelimit in _base.py). Needed for IRC bridge reasons - see https://github.com/matrix-org/matrix-appservice-irc/issues/240.	2016-10-18 17:04:09 +01:00
Erik Johnston	35e2cc8b52	Merge pull request #1155 from matrix-org/erikj/pluggable_pwd_auth ... Implement pluggable password auth	2016-10-12 11:41:20 +01:00
Richard van der Hoff	0061e8744f	Merge pull request #1166 from matrix-org/rav/grandfather_broken_riot_signup ... Work around email-spamming Riot bug	2016-10-11 11:58:58 +01:00
Richard van der Hoff	fa74fcf512	Work around email-spamming Riot bug ... 5d9546f9 introduced a change to synapse behaviour, in that failures in the interactive-auth process would return the flows and params data as well as an error code (as specced in https://github.com/matrix-org/matrix-doc/pull/397). That change exposed a bug in Riot which would make it request a new validation token (and send a new email) each time it got a 401 with a `flows` parameter (see https://github.com/vector-im/vector-web/issues/2447 and the fix at https://github.com/matrix-org/matrix-react-sdk/pull/510). To preserve compatibility with broken versions of Riot, grandfather in the old behaviour for the email validation stage.	2016-10-11 11:34:40 +01:00
Erik Johnston	a2f2516199	Merge pull request #1157 from Rugvip/nolimit ... Remove rate limiting from app service senders and fix get_or_create_user requester	2016-10-11 11:20:54 +01:00
Richard van der Hoff	8681aff4f1	Merge pull request #1160 from matrix-org/rav/401_on_password_fail ... Interactive Auth: Return 401 from for incorrect password	2016-10-07 10:57:43 +01:00
Richard van der Hoff	5d9546f9f4	Interactive Auth: Return 401 from for incorrect password ... This requires a bit of fettling, because I want to return a helpful error message too but we don't want to distinguish between unknown user and invalid password. To avoid hardcoding the error message into 15 places in the code, I've had to refactor a few methods to return None instead of throwing. Fixes https://matrix.org/jira/browse/SYN-744	2016-10-07 00:00:00 +01:00
Patrik Oldsberg	7b5546d077	rest/client/v1/register: use the correct requester in createUser ... Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>	2016-10-06 22:12:32 +02:00
Patrik Oldsberg	3de7c8a4d0	handlers/profile: added admin override for set_displayname and set_avatar_url ... Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>	2016-10-06 15:24:59 +02:00
Patrik Oldsberg	2ff2d36b80	handers: do not ratelimit app service senders ... Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>	2016-10-06 15:24:59 +02:00
Patrik Oldsberg	9bfc617791	storage/appservice: make appservice methods only relying on the cache synchronous	2016-10-06 15:24:59 +02:00
Erik Johnston	850b103b36	Implement pluggable password auth ... Allows delegating the password auth to an external module. This also moves the LDAP auth to using this system, allowing it to be removed from the synapse tree entirely in the future.	2016-10-03 10:36:40 +01:00
Martin Weinelt	3027ea22b0	Restructure ldap authentication ... - properly parse return values of ldap bind() calls - externalize authentication methods - change control flow to be more error-resilient - unbind ldap connections in many places - improve log messages and loglevels	2016-09-29 15:30:08 +01:00
Erik Johnston	655891d179	Move FEDERATION_PING_INTERVAL timer. Update log line	2016-09-23 15:43:34 +01:00
Erik Johnston	22578545a0	Time out typing over federation	2016-09-23 14:00:52 +01:00
Erik Johnston	667fcd54e8	Merge pull request #1136 from matrix-org/erikj/fix_signed_3pid ... Allow invites via 3pid to bypass sender sig check	2016-09-22 13:41:49 +01:00
Erik Johnston	81964aeb90	Merge pull request #1132 from matrix-org/erikj/initial_sync_split ... Support /initialSync in synchrotron worker	2016-09-22 12:45:02 +01:00
Erik Johnston	2e9ee30969	Add comments	2016-09-22 11:59:46 +01:00
Erik Johnston	1168cbd54d	Allow invites via 3pid to bypass sender sig check ... When a server sends a third party invite another server may be the one that the inviting user registers with. In this case it is that remote server that will issue an actual invitation, and wants to do it "in the name of" the original invitee. However, the new proper invite will not be signed by the original server, and thus other servers would reject the invite if it was seen as coming from the original user. To fix this, a special case has been added to the auth rules whereby another server can send an invite "in the name of" another server's user, so long as that user had previously issued a third party invite that is now being accepted.	2016-09-22 10:56:53 +01:00
Erik Johnston	90c070c850	Add total_room_count_estimate to /publicRooms	2016-09-21 13:30:05 +01:00
Erik Johnston	87528f0756	Support /initialSync in synchrotron worker	2016-09-21 11:46:28 +01:00
Erik Johnston	39af634dd2	Merge pull request #1130 from matrix-org/erikj/fix_pubroom_pag ... Handle fact that _generate_room_entry may not return a room entry	2016-09-19 10:13:59 +01:00
Erik Johnston	4d49e0bdfd	PEP8	2016-09-17 18:09:22 +01:00
Erik Johnston	81570abfb2	Handle fact that _generate_room_entry may not return a room entry	2016-09-17 18:01:54 +01:00
Erik Johnston	eb24aecf8c	Merge pull request #1129 from matrix-org/erikj/fix_pubroom_pag ... Fix and clean up publicRooms pagination	2016-09-17 15:30:34 +01:00
Erik Johnston	a298331de4	Spelling	2016-09-17 14:59:40 +01:00
Erik Johnston	71edaae981	Fix and clean up publicRooms pagination	2016-09-17 14:46:19 +01:00
David Baker	5336acd46f	Make public room search case insensitive	2016-09-16 19:02:42 +01:00
Erik Johnston	b7b62bf9ea	Comment	2016-09-16 11:00:29 +01:00
Erik Johnston	d84319ae10	Add remote reoom cache	2016-09-16 10:31:59 +01:00
Erik Johnston	23b6701a28	Support filtering remote room lists	2016-09-16 10:24:15 +01:00
Erik Johnston	e58a9d781c	Filter remote rooms lists locally	2016-09-16 10:19:32 +01:00
Erik Johnston	74d4cdee25	Don't cache searched in /publicRooms	2016-09-16 09:05:11 +01:00
Erik Johnston	098db4aa52	Add very basic filter API to /publicRooms	2016-09-15 17:50:16 +01:00
Erik Johnston	c33b25fd8d	Change the way we calculate new_limit in /publicRooms and add POST API	2016-09-15 17:35:20 +01:00
Erik Johnston	e457034e99	Merge pull request #1121 from matrix-org/erikj/public_room_paginate ... Add pagination support to publicRooms	2016-09-15 13:27:09 +01:00
Erik Johnston	4fb65a1091	Base public room list off of public_rooms stream	2016-09-15 11:27:04 +01:00
Erik Johnston	5810cffd33	Pass since/from parameters over federation	2016-09-15 10:36:19 +01:00
Erik Johnston	f3eead0660	Allow paginating both forwards and backwards	2016-09-15 10:15:37 +01:00
Erik Johnston	4131381123	Remove support for aggregate room lists	2016-09-15 09:28:15 +01:00
Erik Johnston	4f181f361d	Accept optional token to public room list	2016-09-15 09:08:57 +01:00
Erik Johnston	c566f0ee17	Calculate the public room list from a stream_ordering	2016-09-14 17:42:47 +01:00
Erik Johnston	772c6067a3	Refactor public rooms to not pull out the full state for each room	2016-09-14 17:29:25 +01:00
Erik Johnston	264a48aedf	Merge pull request #1117 from matrix-org/erikj/fix_state ... Ensure we don't mutate state cache entries	2016-09-14 16:50:37 +01:00
Erik Johnston	a70a43bc51	Move RoomListHandler into a separate file	2016-09-14 14:07:37 +01:00
Erik Johnston	d5ae1f1291	Ensure we don't mutate state cache entries	2016-09-14 10:03:48 +01:00
Matthew Hodgson	1b01488d27	Merge pull request #1111 from matrix-org/matthew/device-ids ... make device IDs more useful for human disambiguation	2016-09-14 01:18:02 +01:00
Paul Evans	0f73f0e70e	Merge pull request #1116 from matrix-org/paul/tiny-fixes ... Fix typo "persiting"	2016-09-13 13:27:42 +01:00
Paul "LeoNerd" Evans	ca35e54d6b	Fix typo "persiting"	2016-09-13 13:26:33 +01:00
Mark Haines	ad816b0add	Limit how often we ask for keys from dead servers	2016-09-13 11:53:50 +01:00
Mark Haines	18ab019a4a	Move the E2E key handling into the e2e handler	2016-09-13 11:35:35 +01:00
Mark Haines	76b09c29b0	Merge pull request #1110 from matrix-org/markjh/e2e_timeout ... Add a timeout parameter for end2end key queries.	2016-09-13 10:50:45 +01:00
Matthew Hodgson	edbcb4152b	make device IDs more useful for human disambiguation	2016-09-13 00:02:39 +01:00
Mark Haines	949c2c5435	Add a timeout parameter for end2end key queries. ... Add a timeout parameter for controlling how long synapse will wait for responses from remote servers. For servers that fail include how they failed to make it easier to debug. Fetch keys from different servers in parallel rather than in series. Set the default timeout to 10s.	2016-09-12 18:17:09 +01:00
Erik Johnston	48a5a7552d	Add is_direct param to /createRoom	2016-09-12 16:34:20 +01:00
Erik Johnston	3265def8c7	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/batch_edus	2016-09-09 18:06:01 +01:00
Erik Johnston	52b2318777	Clobber EDUs in send queue	2016-09-09 15:59:08 +01:00
Paul Evans	56f38d1776	Merge pull request #1091 from matrix-org/paul/third-party-lookup ... Improvements to 3PE lookup API	2016-09-09 15:43:11 +01:00
Paul "LeoNerd" Evans	ed44c475d8	Reject malformed 3PE query metadata results earlier in AS API handling code	2016-09-09 15:07:04 +01:00
Paul "LeoNerd" Evans	f25d74f69c	Minor fixes from PR comments	2016-09-09 14:54:16 +01:00
Erik Johnston	6c4d582144	Deduplicate presence in _update_states	2016-09-09 14:28:22 +01:00
Paul "LeoNerd" Evans	25eb769b26	Efficiency fix for lookups of a single protocol	2016-09-09 13:25:02 +01:00
Paul "LeoNerd" Evans	033d43e419	Don't corrupt shared cache on subsequent protocol requests	2016-09-09 13:10:36 +01:00
Paul "LeoNerd" Evans	2f267ee160	Collect up all the "instances" lists of individual AS protocol results into one combined answer to the client	2016-09-08 17:43:53 +01:00
Erik Johnston	257025ac89	Merge pull request #1082 from matrix-org/erikj/remote_public_rooms ... Add server param to /publicRooms	2016-09-08 16:04:22 +01:00
Erik Johnston	8b93af662d	Check the user_id for presence/typing matches origin	2016-09-08 15:07:38 +01:00
Mark Haines	fa9d36e050	Merge branch 'develop' into markjh/direct_to_device_federation	2016-09-08 13:43:43 +01:00
Erik Johnston	791658b576	Add server param to /publicRooms	2016-09-08 11:53:05 +01:00
Mark Haines	43954d000e	Add a new method to enqueue the device messages rather than sending a dummy EDU	2016-09-07 16:10:51 +01:00
Mark Haines	d4a35ada28	Send device messages over federation	2016-09-06 18:16:20 +01:00
Erik Johnston	3c4208a057	Record counts of state changes	2016-09-06 11:31:01 +01:00
Erik Johnston	438ef47637	Short circuit if presence is the same	2016-09-06 10:28:35 +01:00
Erik Johnston	74a3b4a650	Fiddle should_notify to better report stats	2016-09-06 10:23:38 +01:00
Erik Johnston	8c93e0bae7	Merge pull request #1070 from matrix-org/erikj/presence_stats ... Record why we have chosen to notify	2016-09-05 15:01:30 +01:00
Erik Johnston	69054e3d4c	Record why we have chosen to notify	2016-09-05 14:12:11 +01:00
Erik Johnston	69a2d4e38c	Use get_joined_users_from_context instead of manually looking up hosts	2016-09-05 13:44:40 +01:00
Mark Haines	274c2f50a5	Merge pull request #1067 from matrix-org/markjh/idempotent ... Fix membership changes to be idempotent	2016-09-05 10:21:25 +01:00
Mark Haines	f2690c6423	Fix membership changes to be idempotent	2016-09-02 19:23:22 +01:00
Mark Haines	81b94c5750	Merge pull request #1066 from matrix-org/markjh/direct_to_device_lowerbound ... Only return new device messages in /sync	2016-09-02 16:18:34 +01:00
Mark Haines	65fa37ac5e	Only return new device messages in /sync	2016-09-02 15:50:37 +01:00
Erik Johnston	273b6bcf22	Merge pull request #1064 from matrix-org/erikj/on_receive_check ... Only check if host is in room if we have state and auth_chain	2016-09-02 14:43:35 +01:00
Erik Johnston	f7f1027d3d	Comment on when auth chain and state are None	2016-09-02 14:42:38 +01:00
Erik Johnston	34e5e17f91	Comment	2016-09-02 14:26:07 +01:00
Erik Johnston	b96c6c3185	Docstrings	2016-09-02 14:19:22 +01:00
Erik Johnston	bd9b8d87ae	Only check if host is in room if we have state and auth_chain	2016-09-02 13:40:28 +01:00
Erik Johnston	2854ee2a52	Only pull out IDs from DB for /state_ids/ request	2016-09-02 10:53:36 +01:00
Erik Johnston	265d847ffd	Fix typo in log line	2016-09-01 14:50:06 +01:00
Erik Johnston	c10cb581c6	Correctly handle the difference between prev and current state	2016-08-31 14:26:22 +01:00
Erik Johnston	c7f665d700	Merge pull request #1055 from matrix-org/erikj/occaisonally_persist ... Occaisonally persist unpersisted presence updates	2016-08-30 15:59:00 +01:00
Erik Johnston	097330bae8	Check correct variable	2016-08-30 15:50:20 +01:00
Erik Johnston	21b977ccfe	Occaisonally persist unpersisted presence updates	2016-08-30 15:39:50 +01:00
Erik Johnston	bc1a8b1f7a	Don't notify for online -> online transitions. ... Specifically, if currently_active remains true then we should not notify if only the last active time changes.	2016-08-30 15:05:32 +01:00
Erik Johnston	c8cd87b21b	Comment about message deletion	2016-08-30 11:23:26 +01:00
Erik Johnston	1ee6285905	Fix check	2016-08-30 11:17:46 +01:00
Erik Johnston	68e1a872fd	Noop get_new_messages_for_device if token hasn't changed	2016-08-30 10:58:46 +01:00