8247 Commits

Author	SHA1	Message	Date
Luke Barnard	6850f8aea3	Get group_info from existing call to check_group_is_ours	2018-04-06 15:43:27 +01:00
Luke Barnard	cd087a265d	Don't use redundant inlineCallbacks	2018-04-06 15:43:27 +01:00
Luke Barnard	87c864b698	join_rule -> join_policy	2018-04-06 15:43:27 +01:00
Luke Barnard	ae85c7804e	is_joinable -> join_rule	2018-04-06 15:43:27 +01:00
Luke Barnard	f8d1917fce	Fix federation client set_group_joinable typo	2018-04-06 15:43:27 +01:00
Luke Barnard	6eb3aa94b6	Factor out add_user from accept_invite and join_group	2018-04-06 15:43:27 +01:00
David Baker	edb45aae38	pep8	2018-04-06 15:43:27 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Erik Johnston	7b824f1475	Add response size metrics	2018-04-06 13:20:11 +01:00
Krombel	1d71f484d4	use PUT instead of POST for federating groups/m.join_policy	2018-04-06 12:54:09 +02:00
Richard van der Hoff	15e8ed874f	more verbosity in synctl	2018-04-06 09:28:36 +01:00
Richard van der Hoff	c7f0969731	Merge pull request #2986 from jplatte/join_reponse_room_id ... Add room_id to the response of `rooms/{roomId}/join`	2018-04-05 17:29:06 +01:00
Richard van der Hoff	3449da3bc7	Merge pull request #3068 from matrix-org/rav/fix_cache_invalidation ... Improve database cache performance	2018-04-05 17:21:44 +01:00
Richard van der Hoff	d1679a4ed7	Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics ... Remove redundant metrics which were deprecated in 0.27.0.	2018-04-05 17:21:18 +01:00
Richard van der Hoff	01afc563c3	Fix overzealous cache invalidation ... Fixes an issue where a cache invalidation would invalidate *all* pending entries, rather than just the entry that we intended to invalidate.	2018-04-05 16:24:04 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Neil Johnson	68b0ee4e8d	Merge pull request #3041 from matrix-org/r30_stats ... R30 stats	2018-04-05 15:37:37 +01:00
Richard van der Hoff	22284a6f65	Merge pull request #3060 from matrix-org/rav/kill_event_content ... Remove uses of events.content	2018-04-05 15:02:17 +01:00
Luke Barnard	917380e89d	NON NULL -> NOT NULL	2018-04-05 14:32:12 +01:00
Luke Barnard	104c0bc1d5	Use "/settings/" (plural)	2018-04-05 14:07:16 +01:00
Luke Barnard	700e5e7198	Use DEFAULT join_policy of "invite" in db	2018-04-05 14:01:17 +01:00
Luke Barnard	b214a04ffc	Document set_group_join_policy	2018-04-05 13:29:16 +01:00
Neil Johnson	0e5f479fc0	Review comments ... Use iteritems over item to loop over dict formatting	2018-04-05 12:16:46 +01:00
Richard van der Hoff	518f6de088	Remove redundant metrics which were deprecated in 0.27.0.	2018-04-04 19:46:28 +01:00
Jan Christian Grünhage	e4570c53dd	phone home cache size configurations	2018-04-04 16:46:58 +01:00
Adrian Tschira	6168351877	Add b prefixes to some strings that are bytes in py3 ... This has no effect on python2 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-04 13:48:51 +02:00
Richard van der Hoff	f92963f5db	Revert "improve mxid check performance"	2018-04-04 12:08:29 +01:00
Silke	72251d1b97	Remove address resolution of hosts in SRV records ... Signed-off-by: Silke Hofstra <silke@slxh.eu>	2018-04-04 12:26:50 +02:00
Richard van der Hoff	725a72ec5a	Merge pull request #3000 from NotAFile/change-except-style ... Replace old style error catching with 'as' keyword	2018-04-04 10:45:22 +01:00
Richard van der Hoff	a89f9f830c	Merge pull request #3044 from matrix-org/michaelk/performance_stats ... Add basic performance statistics to phone home	2018-04-04 10:37:25 +01:00
Richard van der Hoff	39ce38b024	Merge pull request #3053 from NotAFile/speedup-mxid-check ... improve mxid check performance	2018-04-04 10:19:52 +01:00
Richard van der Hoff	a9a74101a4	Document the behaviour of ResponseCache ... it looks like everything that uses ResponseCache expects to have to `make_deferred_yieldable` its results. It's debatable whether that is the best approach, but let's document it for now to avoid further confusion.	2018-04-04 09:06:22 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	3ee4ad09eb	Fix json encoding bug in replication ... json encoders have an encode method, not a dumps method.	2018-04-03 15:09:48 +01:00
Adrian Tschira	11597ddea5	improve mxid check performance ~4x ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-31 02:00:22 +02:00
Richard van der Hoff	2fe3f848b9	Remove uses of events.content	2018-03-29 23:17:12 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Neil Johnson	b4e37c6f50	pep8	2018-03-29 17:27:39 +01:00
Neil Johnson	9ee44a372d	Remove need for sqlite specific query	2018-03-29 16:45:34 +01:00
Erik Johnston	88cc9cc69e	Merge pull request #3043 from matrix-org/erikj/measure_state_group_creation ... Measure time it takes to calculate state group ID	2018-03-28 17:40:14 +01:00
Neil Johnson	dc7c020b33	fix pep8 errors	2018-03-28 17:25:15 +01:00
David Baker	c5de6987c2	This should probably be a PUT	2018-03-28 16:44:11 +01:00
Neil Johnson	241e4e8687	remove twisted deferral cruft	2018-03-28 16:25:53 +01:00
David Baker	929b34963d	OK, smallint it is then	2018-03-28 14:53:55 +01:00
Richard van der Hoff	9a0db062af	Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors ... Fix error when claiming e2e keys from offline servers	2018-03-28 14:50:38 +01:00
David Baker	a838444a70	Grr. Copy the definition from is_admin	2018-03-28 14:50:30 +01:00
Neil Johnson	4262aba17b	bump schema version	2018-03-28 14:40:03 +01:00
Neil Johnson	86932be2cb	Support multi client R30 for psql	2018-03-28 14:36:53 +01:00
David Baker	32260baa41	pep8	2018-03-28 14:29:42 +01:00
Michael Kaye	33f6195d9a	Handle review comments	2018-03-28 14:25:25 +01:00
David Baker	a164270833	Make column definition that works on both dbs	2018-03-28 14:23:00 +01:00
David Baker	352e1ff9ed	Add schema delta file	2018-03-28 14:07:57 +01:00
David Baker	79452edeee	Add joinability for groups ... Adds API to set the 'joinable' flag, and corresponding flag in the table.	2018-03-28 14:03:37 +01:00
Krombel	6152e253d8	Merge branch 'develop' of into allow_auto_join_rooms	2018-03-28 14:45:28 +02:00
Neil Johnson	792d340572	rename stat to future proof	2018-03-28 12:25:02 +01:00
Michael Kaye	4ceaa7433a	As daemonizing will make a new process, defer call to init.	2018-03-28 12:19:01 +01:00
Neil Johnson	788e69098c	Add user_ips last seen index	2018-03-28 12:03:13 +01:00
Neil Johnson	0f890f477e	No need to cast in count_daily_users	2018-03-28 11:49:57 +01:00
Neil Johnson	545001b9e4	Fix search_user_dir multiple sqlite versions do different things	2018-03-28 11:19:45 +01:00
Erik Johnston	01ccc9e6f2	Measure time it takes to calculate state group ID	2018-03-28 11:03:52 +01:00
Neil Johnson	a32d2548d9	query and call for r30 stats	2018-03-28 10:39:13 +01:00
Neil Johnson	9187e0762f	count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home	2018-03-28 10:02:32 +01:00
Erik Johnston	f879127aaa	Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id ... Linearize calls to _generate_user_id	2018-03-28 10:00:31 +01:00
Erik Johnston	e6d87c93f3	Merge pull request #3030 from matrix-org/erikj/no_ujson ... Remove last usage of ujson	2018-03-28 10:00:06 +01:00
Erik Johnston	004cc8a328	Merge pull request #3033 from matrix-org/erikj/calculate_state_metrics ... Add counter metrics for calculating state delta	2018-03-28 09:59:42 +01:00
Michael Kaye	ef520d8d0e	Include coarse CPU and Memory use in stats callbacks. ... This requires the psutil module, and is still opt-in based on the report_stats config option.	2018-03-27 17:56:03 +01:00
Richard van der Hoff	a134c572a6	Stringify exceptions for keys/{query,claim} ... Make sure we stringify any exceptions we return from keys/query and keys/claim, to avoid a 'not JSON serializable' error later Fixes #3010	2018-03-27 17:15:06 +01:00
Richard van der Hoff	c2a5cf2fe3	factor out exception handling for keys/claim and keys/query ... this stuff is badly c&p'ed	2018-03-27 17:11:23 +01:00
Erik Johnston	800cfd5774	Comment	2018-03-27 13:30:39 +01:00
Erik Johnston	152c2ac19e	Fix indent	2018-03-27 13:13:46 +01:00
Erik Johnston	e70287cff3	Comment	2018-03-27 13:13:38 +01:00
Erik Johnston	03a26e28d9	Merge pull request #3017 from matrix-org/erikj/add_cache_control_headers ... Add Cache-Control headers to all JSON APIs	2018-03-27 13:10:38 +01:00
Erik Johnston	3e0c0660b3	Also do check inside linearizer	2018-03-27 13:01:34 +01:00
Erik Johnston	3f49e131d9	Add counter metrics for calculating state delta ... This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event.	2018-03-27 10:57:35 +01:00
Erik Johnston	9b8c0fb162	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-26 21:41:55 +01:00
Erik Johnston	a9d7d98d3f	Bum version and changelog	2018-03-26 16:36:53 +01:00
Erik Johnston	bdbb1eec65	Merge branch 'erikj/simplejson_replication' of github.com:matrix-org/synapse into release-v0.27.0	2018-03-26 16:35:55 +01:00
Erik Johnston	9187862002	Bump version and changelog	2018-03-26 16:20:24 +01:00
Neil Johnson	51406dab96	version bump	2018-03-26 14:48:19 +01:00
Erik Johnston	fecb45e0c3	Remove last usage of ujson	2018-03-26 13:32:29 +01:00
Erik Johnston	44cd6e1358	PEP8	2018-03-26 12:06:48 +01:00
Erik Johnston	8d6dc106d1	Don't use _cursor_to_dict in find_next_generated_user_id_localpart	2018-03-26 12:02:44 +01:00
Erik Johnston	a052aa42e7	Linearize calls to _generate_user_id	2018-03-26 12:02:20 +01:00
Matthew Hodgson	8cbbfaefc1	404 correctly on missing paths via NoResource ... fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029	2018-03-23 10:32:50 +00:00
Erik Johnston	84b5cc69f5	Merge pull request #3006 from matrix-org/erikj/state_iter ... Use .iter* to avoid copies in StateHandler	2018-03-22 11:51:13 +00:00
Erik Johnston	fde8e8f09f	Fix s/iteriterms/itervalues	2018-03-22 11:42:16 +00:00
Erik Johnston	eb9fc021e3	Merge branch 'release-v0.27.0' of github.com:matrix-org/synapse into develop	2018-03-22 10:19:53 +00:00
Erik Johnston	1c41b05c8c	Add Cache-Control headers to all JSON APIs ... It is especially important that sync requests don't get cached, as if a sync returns the same token given then the client will call sync with the same parameters again. If the previous response was cached it will get reused, resulting in the client tight looping making the same request and never making any progress. In general, clients will expect to get up to date data when requesting APIs, and so its safer to do a blanket no cache policy than only whitelisting APIs that we know will break things if they get cached.	2018-03-21 17:46:26 +00:00
Neil Johnson	e66fbcbb02	fix merge conflicts	2018-03-20 14:25:31 +00:00
Erik Johnston	9aa5a0af51	Explicitly use simplejson	2018-03-20 09:58:13 +00:00
Erik Johnston	610accbb7f	Fix replication after switch to simplejson ... Turns out that simplejson serialises namedtuple's as dictionaries rather than tuples by default.	2018-03-19 16:12:48 +00:00
Neil Johnson	c384705ee8	Update __init__.py ... bump version	2018-03-19 15:11:58 +00:00
Erik Johnston	3f961e638a	Merge pull request #3005 from matrix-org/erikj/fix_cache_size ... Fix bug where state cache used lots of memory	2018-03-19 11:44:26 +00:00
Erik Johnston	fa72803490	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-19 11:41:01 +00:00
Erik Johnston	9a0d783c11	Add comments	2018-03-19 11:35:53 +00:00
Matthew Hodgson	38f952b9bc	spell out not to massively increase bcrypt rounds	2018-03-19 09:27:36 +00:00
Erik Johnston	a8ce159be4	Replace some ujson with simplejson to make it work	2018-03-16 00:27:09 +00:00
Erik Johnston	5b631ff41a	Remove wrong comment	2018-03-16 00:07:08 +00:00
Erik Johnston	ba48755d56	Bump version and changelog	2018-03-15 23:57:26 +00:00
Erik Johnston	926ba76e23	Replace ujson with simplejson	2018-03-15 23:43:31 +00:00
Richard van der Hoff	5a6e54264d	Make 'unexpected logging context' into warnings ... I think we've now fixed enough of these that the rest can be logged at warning.	2018-03-15 18:40:38 +00:00
Erik Johnston	9cf519769b	Use .iter* to avoid copies in StateHandler	2018-03-15 17:50:26 +00:00
Erik Johnston	7c7706f42b	Fix bug where state cache used lots of memory ... The state cache bases its size on the sum of the size of entries. The size of the entry is calculated once on insertion, so it is important that the size of entries does not change. The DictionaryCache modified the entries size, which caused the state cache to incorrectly think it was smaller than it actually was.	2018-03-15 15:46:54 +00:00
NotAFile	2cc9f76bc3	replace old style error catching with 'as' keyword ... This is both easier to read and compatible with python3 (not that that matters) Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-15 16:11:17 +01:00
Erik Johnston	ddb00efc1d	Bump version number	2018-03-15 14:41:30 +00:00
Krombel	91ea0202e6	move handling of auto_join_rooms to RegisterHandler ... Currently the handling of auto_join_rooms only works when a user registers itself via public register api. Registrations via registration_shared_secret and ModuleApi do not work This auto_joins the users in the registration handler which enables the auto join feature for all 3 registration paths. This is related to issue #2725 Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-14 16:45:37 +01:00
Erik Johnston	4f28018c83	Register membership/state servlets in event_creator	2018-03-14 14:30:06 +00:00
Erik Johnston	57db62e554	Merge pull request #2992 from matrix-org/erikj/implement_member_workre ... Implement RoomMemberWorkerHandler	2018-03-14 14:29:33 +00:00
Erik Johnston	0011ede3b0	Fix imports	2018-03-14 14:19:23 +00:00
Erik Johnston	62ad701326	s/join/joined/ in notify_user_membership_change	2018-03-14 14:17:43 +00:00
Erik Johnston	3f0f06cb31	Split RoomMemberWorkerHandler to separate file	2018-03-14 11:41:45 +00:00
Erik Johnston	3e839e0548	Merge pull request #2989 from matrix-org/erikj/profile_cache_master ... Only update remote profile cache on master	2018-03-14 09:42:27 +00:00
Erik Johnston	ebd0127999	Merge pull request #2988 from matrix-org/erikj/split_profile_store ... Split up ProfileStore	2018-03-14 09:41:06 +00:00
Erik Johnston	cfe75a9fb6	Merge pull request #2991 from matrix-org/erikj/fixup_rm ... _remote_join and co take a requester	2018-03-14 09:40:57 +00:00
Erik Johnston	f51565e023	Merge pull request #2993 from matrix-org/erikj/is_blocked ... Add is_blocked to worker store	2018-03-14 09:39:18 +00:00
Matthew Hodgson	d144ed6ffb	fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990) ... Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.	2018-03-13 22:36:04 +00:00
Erik Johnston	a08726fc42	Add is_blocked to worker store	2018-03-13 18:28:44 +00:00
Erik Johnston	b27320b550	Implement RoomMemberWorkerHandler	2018-03-13 18:26:00 +00:00
Erik Johnston	350331d466	_remote_join and co take a requester	2018-03-13 17:50:39 +00:00
Erik Johnston	1a69c6d590	Merge pull request #2987 from matrix-org/erikj/split_room_member_handler ... Split RoomMemberHandler into base and master class	2018-03-13 17:40:00 +00:00
Erik Johnston	df8ff682a7	Only update remote profile cache on master	2018-03-13 17:38:21 +00:00
Erik Johnston	3518d0ea8f	Split up ProfileStore	2018-03-13 17:36:50 +00:00
Erik Johnston	d45a114824	Raise, don't return, exception	2018-03-13 17:24:34 +00:00
Erik Johnston	6dbebef141	Add missing param to docstrings	2018-03-13 17:15:32 +00:00
Erik Johnston	16adb11cc0	Correct import order	2018-03-13 16:57:07 +00:00
Erik Johnston	82f16faa78	Move user_*_room distributor stuff to master class ... I added yields when calling user_left_room, but they shouldn't matter on the master process as they always return None anyway.	2018-03-13 16:38:15 +00:00
Erik Johnston	b78717b87b	Split RoomMemberHandler into base and master class ... The intention here is to split the class into the bits that can be done on workers and the bits that have to be done on the master. In future there will also be a class that can be run on the worker, which will delegate work to the master when necessary.	2018-03-13 16:37:41 +00:00
Erik Johnston	95cb401ae0	Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer ... Remove ReplicationLayer and user Client/Server directly	2018-03-13 15:45:08 +00:00
Erik Johnston	5d8476d8ff	Merge pull request #2981 from matrix-org/erikj/factor_remote_leave ... Factor out _remote_reject_invite in RoomMember	2018-03-13 15:44:56 +00:00
Jonas Platte	47ce527f45	Add room_id to the response of rooms/{roomId}/join ... Fixes #2349	2018-03-13 14:48:12 +01:00
Erik Johnston	56e709857c	Merge pull request #2979 from matrix-org/erikj/no_handlers ... Don't build handlers on workers unnecessarily	2018-03-13 13:46:38 +00:00
Erik Johnston	cb9f8e527c	s/replication_client/federation_client/	2018-03-13 13:26:52 +00:00
Erik Johnston	cea462e285	s/replication_server/federation_server	2018-03-13 13:22:21 +00:00
Erik Johnston	bf8e97bd3c	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave	2018-03-13 13:17:08 +00:00
Erik Johnston	ea3442c15c	Add docstring	2018-03-13 13:16:21 +00:00
Erik Johnston	16469a4f15	Merge pull request #2980 from matrix-org/erikj/rm_priv ... Make RoomMemberHandler functions private that can be	2018-03-13 13:11:11 +00:00
Erik Johnston	c82111a55f	Merge pull request #2982 from matrix-org/erikj/fix_extra_users ... extra_users is actually a list of UserIDs	2018-03-13 13:11:04 +00:00
Erik Johnston	da87791975	Merge pull request #2983 from matrix-org/erikj/rename_register_3pid ... Refactor get_or_register_3pid_guest	2018-03-13 13:10:52 +00:00
Erik Johnston	99e9b4f26c	Merge pull request #2984 from matrix-org/erikj/fix_rest_regeix ... RoomMembershipRestServlet doesn't handle /forget	2018-03-13 13:10:44 +00:00
Erik Johnston	f5160d4a3e	RoomMembershipRestServlet doesn't handle /forget ... Due to the order we register the REST handlers `/forget` was handled by the correct handler.	2018-03-13 12:12:55 +00:00
Erik Johnston	8b3573a8b2	Refactor get_or_register_3pid_guest	2018-03-13 12:08:58 +00:00
Richard van der Hoff	299fd740c7	Merge pull request #2975 from matrix-org/rav/measure_persist_events ... Add Measure block for persist_events	2018-03-13 12:06:25 +00:00
Erik Johnston	9a2d9b4789	Merge pull request #2977 from matrix-org/erikj/replication_move_props ... Move property setting from ReplicationLayer to base classes	2018-03-13 11:45:25 +00:00
Erik Johnston	f43b6d6d9b	Fix docstring types	2018-03-13 11:29:35 +00:00
Erik Johnston	0f942f68c1	Factor out _remote_reject_invite in RoomMember	2018-03-13 11:22:45 +00:00
Erik Johnston	d0fcc48f9d	extra_users is actually a list of UserIDs	2018-03-13 11:20:06 +00:00
Erik Johnston	31becf4ac3	Make functions private that can be	2018-03-13 11:15:16 +00:00
Erik Johnston	d023ecb810	Don't build handlers on workers unnecessarily	2018-03-13 11:08:10 +00:00
Erik Johnston	ea7b3c4b1b	Remove unused ReplicationLayer	2018-03-13 11:00:04 +00:00
Erik Johnston	265b993b8a	Split replication layer into two	2018-03-13 10:55:47 +00:00
Erik Johnston	e05bf34117	Move property setting from ReplicationLayer to FederationBase	2018-03-13 10:51:30 +00:00
Erik Johnston	c3f79c9da5	Split out edu/query registration to a separate class	2018-03-13 10:24:27 +00:00
Richard van der Hoff	889a2a853a	Add Measure block for persist_events ... This seems like a useful thing to measure.	2018-03-13 10:01:42 +00:00
Richard van der Hoff	e48c7aac4d	Add transactional API to history purge ... Make the purge request return quickly, and allow scripts to poll for updates.	2018-03-12 16:22:55 +00:00
Richard van der Hoff	1708412f56	Return an error when doing two purges on a room ... Queuing up purges doesn't sound like a good thing.	2018-03-12 16:22:54 +00:00
Richard van der Hoff	b984dd0b73	Merge pull request #2961 from matrix-org/rav/run_in_background ... Factor run_in_background out from preserve_fn	2018-03-12 16:19:13 +00:00
Richard van der Hoff	ba1d08bc4b	Merge pull request #2965 from matrix-org/rav/request_logging ... Add a metric which increments when a request is received	2018-03-12 09:45:33 +00:00
Richard van der Hoff	58dd148c4f	Add some docstrings to help figure this out	2018-03-09 18:05:41 +00:00
Richard van der Hoff	88541f9009	Add a metric which increments when a request is received ... It's useful to know when there are peaks in incoming requests - which isn't quite the same as there being peaks in outgoing responses, due to the time taken to handle requests.	2018-03-09 16:30:26 +00:00
Richard van der Hoff	dbe80a286b	refactor JsonResource ... rephrase the OPTIONS and unrecognised request handling so that they look similar to the common flow.	2018-03-09 16:22:16 +00:00
Richard van der Hoff	20f40348d4	Factor run_in_background out from preserve_fn ... It annoys me that we create temporary function objects when there's really no need for it. Let's factor the gubbins out of preserve_fn and start using it.	2018-03-08 11:50:11 +00:00
Erik Johnston	a56d54dcb7	Fix up log message	2018-03-07 11:55:31 +00:00
Erik Johnston	02a1296ad6	Fix typo	2018-03-07 11:55:31 +00:00
Erik Johnston	8cb44da4aa	Fix race in sync when joining room ... The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.	2018-03-07 11:55:31 +00:00
Richard van der Hoff	8ffaacbee3	Merge pull request #2949 from krombel/use_bcrypt_checkpw ... use bcrypt.checkpw	2018-03-06 11:56:06 +00:00
Richard van der Hoff	b2932107bb	Merge pull request #2946 from matrix-org/rav/timestamp_to_purge ... Implement purge_history by timestamp	2018-03-06 11:20:23 +00:00
Erik Johnston	7aed50a038	Merge pull request #2948 from matrix-org/erikj/kill_as_sync ... Remove ability for AS users to call /events and /sync	2018-03-06 11:10:09 +00:00
Erik Johnston	b6c4b851f1	Merge pull request #2947 from matrix-org/erikj/split_directory_store ... Split Directory store	2018-03-05 18:17:32 +00:00
Krombel	ed9b5eced4	use bcrypt.checkpw ... in bcrypt 3.1.0 checkpw got introduced (already 2 years ago) This makes use of that with enhancements which might get introduced by that Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-05 18:02:59 +01:00
Erik Johnston	d4ffe61d4f	Remove ability for AS users to call /events and /sync ... This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144	2018-03-05 15:44:46 +00:00
Erik Johnston	69ce365b79	Fix cache invalidation on deletion	2018-03-05 15:29:03 +00:00
Erik Johnston	2e223163ff	Split Directory store	2018-03-05 15:11:30 +00:00
Richard van der Hoff	f8bfcd7e0d	Provide a means to pass a timestamp to purge_history	2018-03-05 14:37:23 +00:00
Richard van der Hoff	d032785aa7	Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts ... Test and fix find_first_stream_ordering_after_ts	2018-03-05 12:26:14 +00:00
Richard van der Hoff	c818fcab11	Test and fix find_first_stream_ordering_after_ts ... It seemed to suffer from a bunch of off-by-one errors.	2018-03-05 12:04:02 +00:00
Richard van der Hoff	06a14876e5	Add find_first_stream_ordering_after_ts ... Expose this as a public function which can be called outside a txn	2018-03-05 11:53:39 +00:00
Erik Johnston	42174946f8	Merge pull request #2934 from matrix-org/erikj/cache_fix ... Fix bug with delayed cache invalidation stream	2018-03-05 11:33:17 +00:00
Erik Johnston	f394f5574d	Merge pull request #2929 from matrix-org/erikj/split_regististration_store ... Split registration store	2018-03-05 11:33:07 +00:00
dklug	af7ed8e1ef	Return 401 for invalid access_token on logout ... Signed-off-by: Duncan Klug <dklug@ucmerced.edu>	2018-03-02 22:01:27 -08:00
Erik Johnston	efb79820b4	Fix bug with delayed cache invalidation stream ... We poked the notifier before updated the current token for the cache invalidation stream. This mean that sometimes the update wouldn't be sent until the next time a cache was invalidated.	2018-03-02 14:45:15 +00:00
Erik Johnston	fafa3e7114	Split registration store	2018-03-02 13:48:27 +00:00
Erik Johnston	d960d23830	Add missing yield during 3pid signature checks	2018-03-02 11:03:18 +00:00
Erik Johnston	1a6c7cdf54	Merge pull request #2928 from matrix-org/erikj/read_marker_caches ... Fix typo in getting replication account data processing	2018-03-01 17:56:14 +00:00
Erik Johnston	89b7232ff8	Fix typo in getting replication account data processing	2018-03-01 17:50:30 +00:00
Erik Johnston	1773df0632	Merge pull request #2925 from matrix-org/erikj/split_sig_fed ... Split out SignatureStore and EventFederationStore	2018-03-01 17:32:58 +00:00
Erik Johnston	65cf454fd1	Remove unused DataStore	2018-03-01 17:27:53 +00:00
Erik Johnston	9e08a93a7b	Merge pull request #2927 from matrix-org/erikj/read_marker_caches ... Improve caching for read_marker API	2018-03-01 17:12:34 +00:00
Erik Johnston	4b44f05f19	Fewer lies are better	2018-03-01 17:08:17 +00:00
Erik Johnston	a83c514d1f	Improve caching for read_marker API ... We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.	2018-03-01 17:08:17 +00:00
Erik Johnston	33bebb63f3	Add some caches to help read marker API	2018-03-01 17:08:17 +00:00
Erik Johnston	483e8104db	Merge pull request #2926 from matrix-org/erikj/member_handler_move ... Move RoomMemberHandler out of Handlers	2018-03-01 17:01:25 +00:00
Erik Johnston	2ad4d5b5bb	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed	2018-03-01 16:59:39 +00:00
Erik Johnston	529c026ac1	Move back to hs.is_mine	2018-03-01 16:49:12 +00:00
Erik Johnston	7c371834cc	Stub out broken function only used for cache	2018-03-01 16:44:13 +00:00
Erik Johnston	64346be26d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store	2018-03-01 16:26:42 +00:00
Erik Johnston	22518e2833	Merge pull request #2923 from matrix-org/erikj/stream_ago_worker ... Calculate stream_ordering_month_ago correctly on workers	2018-03-01 16:23:54 +00:00
Erik Johnston	884b26ae41	Remove unused variables	2018-03-01 16:23:48 +00:00
Erik Johnston	1b2af11650	Document abstract class and method better	2018-03-01 16:20:57 +00:00
Erik Johnston	872ff95ed4	Default stream_ordering_*_ago to None	2018-03-01 16:00:05 +00:00
Erik Johnston	22004b524e	Fix comment typo	2018-03-01 15:59:40 +00:00
Erik Johnston	4bc4236faf	Merge pull request #2922 from matrix-org/erikj/split_room_store ... Split up RoomStore	2018-03-01 15:55:01 +00:00
Richard van der Hoff	2324124a72	Merge pull request #2921 from matrix-org/rav/unyielding_make_deferred_yieldable ... Rewrite make_deferred_yieldable avoiding inlineCallbacks	2018-03-01 15:39:10 +00:00
Erik Johnston	f793bc3877	Split out stream store	2018-03-01 15:13:08 +00:00
Erik Johnston	784f036306	Move RoomMemberHandler out of Handlers	2018-03-01 14:36:50 +00:00
Erik Johnston	6411f725be	Calculate stream_ordering_month_ago correctly on workers	2018-03-01 14:20:53 +00:00
Erik Johnston	a9a2d66cdd	Split out SignatureStore and EventFederationStore	2018-03-01 14:17:53 +00:00
Erik Johnston	0c8ba5dd1c	Split up RoomStore	2018-03-01 14:01:19 +00:00
Richard van der Hoff	3a75de923b	Rewrite make_deferred_yieldable avoiding inlineCallbacks ... ... because (a) it's actually simpler (b) it might be marginally more performant?	2018-03-01 12:40:05 +00:00
Erik Johnston	126b9bf96f	Log in the correct places	2018-03-01 12:05:33 +00:00
Erik Johnston	157298f986	Don't do preserve_fn for every request	2018-03-01 11:59:45 +00:00
Erik Johnston	89f90d808a	Add some logging	2018-03-01 11:59:16 +00:00
Erik Johnston	8ded8ba2c7	Make repl send_event idempotent and retry on timeouts ... If we treated timeouts as failures on the worker we would attempt to clean up e.g. push actions while the master might still process the event.	2018-03-01 11:20:34 +00:00
Erik Johnston	f381d63813	Check event auth on the worker	2018-03-01 10:18:37 +00:00
Erik Johnston	6b8604239f	Correctly send ratelimit and extra_users params	2018-03-01 10:08:39 +00:00
Erik Johnston	f756f961ea	Fixup comments	2018-03-01 10:05:27 +00:00
Erik Johnston	28e973ac11	Calculate push actions on worker	2018-02-28 18:02:30 +00:00
Erik Johnston	493e25d554	Move storage functions for push calculations ... This will allow push actions for an event to be calculated on workers.	2018-02-27 13:58:16 +00:00
Erik Johnston	3594dbc6dc	Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation ... Fix missing invalidations for receipt storage	2018-02-27 11:34:26 +00:00
Erik Johnston	2311189ee4	Merge pull request #2903 from matrix-org/erikj/split_roommember_store ... Split out RoomMemberStore	2018-02-27 11:32:10 +00:00
Erik Johnston	c57607874c	Merge pull request #2901 from matrix-org/erikj/split_as_stores ... Split AS stores	2018-02-27 10:07:07 +00:00
Erik Johnston	8956f0147a	Add comment	2018-02-27 10:06:51 +00:00
Erik Johnston	e5b4a208ce	Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions ... Batch inserts into event_push_actions_staging	2018-02-26 14:45:40 +00:00
Erik Johnston	45b5fe9122	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push	2018-02-26 13:49:24 +00:00
Erik Johnston	d62ce972f8	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store	2018-02-23 11:46:24 +00:00
Erik Johnston	6ae9a3d2a6	Update copyright	2018-02-23 11:44:49 +00:00
Erik Johnston	a90c60912f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions	2018-02-23 11:26:31 +00:00
Erik Johnston	50e8657867	Merge pull request #2902 from matrix-org/erikj/split_events_store ... Split out get_events and co into a worker store	2018-02-23 11:23:52 +00:00
Erik Johnston	1cf9e071dd	Merge pull request #2899 from matrix-org/erikj/split_pushers ... Split PusherStore	2018-02-23 11:23:35 +00:00
Erik Johnston	d0957753bf	Merge pull request #2898 from matrix-org/erikj/split_push_rules_store ... Split PushRulesStore	2018-02-23 11:23:23 +00:00
Erik Johnston	199dba6c15	Merge pull request #2897 from matrix-org/erikj/split_account_data ... Split AccountDataStore and TagStore	2018-02-23 11:23:11 +00:00
Erik Johnston	70349872c2	Update copyright	2018-02-23 11:14:35 +00:00
Erik Johnston	eba93b05bf	Split EventsWorkerStore into separate file	2018-02-23 11:01:21 +00:00
Erik Johnston	bf8a36e080	Update copyright	2018-02-23 10:52:10 +00:00
Erik Johnston	5d0f665848	Remove redundant clock	2018-02-23 10:49:58 +00:00
Erik Johnston	3bd760628b	_event_persist_queue shouldn't be in worker store	2018-02-23 10:49:18 +00:00
Erik Johnston	eb9b5eec81	Update copyright	2018-02-23 10:42:39 +00:00
Erik Johnston	c2ecfcc3a4	Update copyright	2018-02-23 10:41:34 +00:00
Erik Johnston	7e6cf89dc2	Update copyright	2018-02-23 10:39:19 +00:00
Erik Johnston	26d37f7a63	Update copyright	2018-02-23 10:33:55 +00:00
Erik Johnston	bb73f55fc6	Use absolute imports	2018-02-23 10:31:16 +00:00
Erik Johnston	faeb369f15	Fix missing invalidations for receipt storage	2018-02-21 15:19:54 +00:00
Erik Johnston	3dec9c66b3	Split out RoomMemberStore	2018-02-21 12:07:26 +00:00
Erik Johnston	46244b2759	Split AS stores	2018-02-21 11:49:34 +00:00
Erik Johnston	27b094f382	Split out get_events and co into a worker store	2018-02-21 11:41:48 +00:00
Erik Johnston	573712da6b	Update comments	2018-02-21 11:29:49 +00:00
Erik Johnston	c96d547f4d	Actually use new param	2018-02-21 11:03:42 +00:00
Erik Johnston	d15d237b0d	Split out EventPushActionWorkerStore	2018-02-21 11:01:13 +00:00
Erik Johnston	27939cbb0e	Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed ... Delete from push_actions_staging in federation too	2018-02-21 11:00:06 +00:00
Erik Johnston	6f72765371	Split PusherStore	2018-02-21 10:54:21 +00:00
Erik Johnston	cbaad969f9	Split PushRulesStore	2018-02-21 10:43:31 +00:00
Erik Johnston	ca9b9d9703	Split AccountDataStore and TagStore	2018-02-21 10:15:04 +00:00
Erik Johnston	8fbb4d0d19	Raise exception in abstract method	2018-02-20 17:59:23 +00:00
Erik Johnston	95e4cffd85	Fix comment	2018-02-20 17:58:40 +00:00
Erik Johnston	e316bbb4c0	Use abstract base class to access stream IDs	2018-02-20 17:43:57 +00:00
Erik Johnston	f5ac4dc2d4	Split ReceiptsStore	2018-02-20 16:28:28 +00:00
Erik Johnston	24087bffa9	Ensure all push actions are deleted from staging	2018-02-20 12:34:31 +00:00
Erik Johnston	ad0ccf15ea	Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts	2018-02-20 12:34:28 +00:00
Erik Johnston	d874d4f2d7	Delete from push_actions_staging in federation too	2018-02-20 11:37:52 +00:00
Erik Johnston	6ff8c87484	Batch inserts into event_push_actions_staging	2018-02-20 11:33:07 +00:00
Erik Johnston	324c3e9399	Merge pull request #2868 from matrix-org/erikj/refactor_media_storage ... Make store_file use store_into_file	2018-02-20 11:31:24 +00:00
Pascal Bach	3acd616979	Update pynacl dependency to 1.2.1 or higher ... Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>	2018-02-19 10:45:22 +01:00
Richard van der Hoff	d1a3325f99	(Really) fix tablescan of event_push_actions on purge ... commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/	2018-02-16 14:02:31 +00:00
Erik Johnston	6af025d3c4	Fix typo of double is_highlight	2018-02-16 11:35:31 +00:00
Erik Johnston	012e8e142a	Comments	2018-02-16 11:35:01 +00:00
Erik Johnston	b96278d6fe	Ensure that we delete staging push actions on errors	2018-02-15 15:47:06 +00:00
Erik Johnston	4810f7effd	Remove context.push_actions	2018-02-15 15:47:06 +00:00
Erik Johnston	c714c61853	Update event_push_actions table from staging table	2018-02-15 15:47:06 +00:00
Erik Johnston	acac21248c	Store push actions in staging area	2018-02-15 15:47:04 +00:00
Erik Johnston	106906a65e	Don't serialize current state over replication	2018-02-15 13:53:18 +00:00
Erik Johnston	5fb347fc41	Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log ... Don't log errors propogated from send_event	2018-02-15 12:31:49 +00:00
Erik Johnston	fd1601c596	Fix state group storage bug in workers ... We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.	2018-02-15 11:04:32 +00:00
Erik Johnston	ef344b10e5	Don't log errors propogated from send_event	2018-02-15 11:03:49 +00:00
Richard van der Hoff	b8d821aa68	Merge pull request #2867 from matrix-org/rav/rework_purge ... purge_history cleanups	2018-02-15 09:49:07 +00:00
Erik Johnston	92c52df702	Make store_file use store_into_file	2018-02-14 17:55:18 +00:00
Richard van der Hoff	d28ec43e15	Merge pull request #2769 from matrix-org/matthew/hit_the_gin ... switch back from GIST to GIN indexes	2018-02-14 16:59:03 +00:00
Richard van der Hoff	39bf47319f	purge_history: fix sqlite syntax error ... apparently sqlite insists on indexes being named	2018-02-14 16:42:19 +00:00
Richard van der Hoff	ac27f6a35e	purge_history: handle sqlite asshattery ... apparently creating a temporary table commits the transaction. because that's a useful thing.	2018-02-14 16:41:12 +00:00
Richard van der Hoff	5978dccff0	remove overzealous exception handling	2018-02-14 15:54:09 +00:00
Richard van der Hoff	278d21b5e4	purge_history: fix index use ... event_push_actions doesn't have an index on event_id, so we need to specify room_id.	2018-02-14 15:44:51 +00:00
Richard van der Hoff	5fcbf1e07c	Rework event purge to use a temporary table ... ... which should speed things up by reducing the amount of data being shuffled across the connection	2018-02-14 11:02:22 +00:00
Erik Johnston	c0c9327fe0	Merge pull request #2854 from matrix-org/erikj/event_create_worker ... Create a worker for event creation	2018-02-13 18:07:10 +00:00
Richard van der Hoff	d627174da2	Fix log message in purge_history ... (we don't just remove remote events)	2018-02-13 16:51:21 +00:00
Richard van der Hoff	ddb6a79b68	Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin	2018-02-13 16:45:36 +00:00
Richard van der Hoff	0b27ae8dc3	move search reindex to schema 47 ... We're up to schema v47 on develop now, so this will have to go in there to have an effect. This might cause an error if somebody has already run it in the v46 guise, and runs it again in the v47 guise, because it will cause a duplicate entry in the bbackground_updates table. On the other hand, the entry is removed once it is complete, and it is unlikely that anyone other than matrix.org has run it on v46. The update itself is harmless to re-run because it deliberately copes with the index already existing.	2018-02-13 16:44:46 +00:00
Richard van der Hoff	4a6d551704	GIN reindex: Fix syntax errors, improve exception handling	2018-02-13 16:44:46 +00:00
Richard van der Hoff	bfdf7b9237	Merge pull request #2864 from matrix-org/rav/persist_event_caching ... Use StateResolutionHandler to resolve state in persist_events	2018-02-13 14:45:57 +00:00
Richard van der Hoff	630caf8a70	style nit	2018-02-13 14:29:22 +00:00
Richard van der Hoff	8fd1a32456	Fix typos in purge api & doc ... * It's supposed to be purge_local_events, not ..._history * Fix the doc to have valid json	2018-02-13 13:09:39 +00:00
Richard van der Hoff	4d09366656	Merge pull request #2695 from okurz/feature/allow_recent_pysaml ... Allow use of higher versions of saml2	2018-02-13 12:24:08 +00:00
Richard van der Hoff	a9b712e9dc	Merge branch 'develop' into matthew/gin_work_mem	2018-02-13 12:16:01 +00:00
Erik Johnston	1026690cd2	Merge pull request #2857 from matrix-org/erikj/upload_store ... Tell storage providers about new file so they can upload	2018-02-12 13:52:58 +00:00
Richard van der Hoff	10b34dbb9a	Merge pull request #2858 from matrix-org/rav/purge_updates ... delete_local_events for purge_room_history	2018-02-09 14:11:00 +00:00
Richard van der Hoff	39a6b35496	purge: move room_depth update to end ... ... to avoid locking the table for too long	2018-02-09 13:07:41 +00:00
Richard van der Hoff	74fcbf741b	delete_local_events for purge_history ... Add a flag which makes the purger delete local events	2018-02-09 13:07:41 +00:00
Richard van der Hoff	e571aef06d	purge: Move cache invalidation to more appropriate place ... it was a bit of a non-sequitur there	2018-02-09 13:07:41 +00:00
Richard van der Hoff	61ffaa8137	bump purge logging to info ... this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.	2018-02-09 13:07:41 +00:00
Richard van der Hoff	671540dccf	rename delete_old_state -> purge_history ... (beacause it deletes more than state)	2018-02-09 13:07:41 +00:00
Erik Johnston	5fa571a91b	Tell storage providers about new file so they can upload	2018-02-07 13:35:08 +00:00
Erik Johnston	50fe92cd26	Move presence handling into handle_new_client_event ... As we want to have it run on the main synapse instance	2018-02-07 10:34:09 +00:00
Erik Johnston	8ec2e638be	Add event_creator worker	2018-02-07 10:32:32 +00:00
Erik Johnston	24dd73028a	Add replication http endpoint for event sending	2018-02-07 10:32:32 +00:00
Erik Johnston	e3624fad5f	Remove pointless ratelimit check ... The intention was for the check to be called as early as possible in the request, but actually was called just before the main ratelimit check, so was fairly pointless.	2018-02-07 10:30:25 +00:00
Erik Johnston	617199d73d	Merge pull request #2847 from matrix-org/erikj/separate_event_creation ... Split event creation into a separate handler	2018-02-06 17:01:17 +00:00
Erik Johnston	3e1e69ccaf	Update copyright	2018-02-06 16:40:38 +00:00
Erik Johnston	770b2252ca	s/_create_new_client_event/create_new_client_event/	2018-02-06 16:40:30 +00:00
Erik Johnston	3d33eef6fc	Store state groups separately from events (#2784) ... * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints	2018-02-06 14:31:24 +00:00
Richard van der Hoff	b31bf0bb51	Merge pull request #2849 from matrix-org/rav/clean_up_state_delta ... Remove redundant return value from _calculate_state_delta	2018-02-05 17:42:20 +01:00
Richard van der Hoff	ebfe64e3d6	Use StateResolutionHandler to resolve state in persist events ... ... and thus benefit (hopefully) from its cache.	2018-02-05 16:23:26 +00:00
Richard van der Hoff	225dc3b4cb	Flatten _get_new_state_after_events ... rejig the if statements to simplify the logic and reduce indentation	2018-02-05 16:23:25 +00:00
Richard van der Hoff	9fcbbe8e7d	Check that events being persisted have state_group	2018-02-05 16:23:25 +00:00
Richard van der Hoff	447aed42d2	Add event_map param to resolve_state_groups	2018-02-05 16:23:25 +00:00
Richard van der Hoff	ee6fb4cf85	Remove redundant return value from _calculate_state_delta ... we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.	2018-02-05 16:23:20 +00:00
Richard van der Hoff	3c7b480ba3	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-05 16:12:14 +00:00
Erik Johnston	3fa362502c	Update places where we create events	2018-02-05 16:01:48 +00:00
Erik Johnston	5ff3d23564	Split event creation into a separate handler	2018-02-05 16:01:48 +00:00
Richard van der Hoff	c46e75d3d8	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-05 15:43:22 +00:00
Richard van der Hoff	db91e72ade	Merge pull request #2844 from matrix-org/rav/evicted_metrics ... montoring metrics for number of cache evictions	2018-02-05 16:34:36 +01:00
Richard van der Hoff	bc496df192	report metrics on number of cache evictions	2018-02-05 15:34:01 +00:00
Erik Johnston	1f881e0746	Merge pull request #2791 from matrix-org/erikj/media_storage_refactor ... Ensure media is in local cache before thumbnailing	2018-02-05 11:28:52 +00:00
Richard van der Hoff	80b8a28100	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-04 00:23:06 +00:00
Richard van der Hoff	bd25f9cf36	Clean up work_mem handling ... Add some comments and improve exception handling when twiddling work_mem for the search update	2018-02-03 23:05:41 +00:00
Richard van der Hoff	4eeae7ad65	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-03 22:59:45 +00:00
Richard van der Hoff	bb9f0f3cdb	Merge branch 'develop' into matthew/gin_work_mem	2018-02-03 22:40:28 +00:00
Richard van der Hoff	6b02fc80d1	Reinstate event_search_postgres_gist handler ... People may have queued updates for this, so we can't just delete it.	2018-02-02 14:32:51 +00:00
Richard van der Hoff	9c9356512e	Merge pull request #2845 from matrix-org/rav/urlcache_error_handling ... Handle url_previews with no content-type	2018-02-02 15:27:52 +01:00
Richard van der Hoff	18eae413af	Merge pull request #2842 from matrix-org/rav/state_resolution_handler ... Factor out resolve_state_groups to a separate handler	2018-02-02 15:27:35 +01:00
Richard van der Hoff	78d6ddba86	Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta ... factor _get_new_state_after_events out of _calculate_state_delta	2018-02-02 15:27:15 +01:00
Richard van der Hoff	9dcd667ac2	Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring ... Docstring fixes	2018-02-02 15:26:49 +01:00
Travis Ralston	6e87b34f7b	Merge branch 'develop' into travis/admin-list-media	2018-02-01 18:05:47 -07:00
Richard van der Hoff	d5352cbba8	Handle url_previews with no content-type ... avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.	2018-02-02 00:53:46 +00:00
Richard van der Hoff	14737ba495	doc arg types for _seperate	2018-02-01 12:41:34 +00:00
Richard van der Hoff	e15d4ea248	More docstring fixes ... Fix a couple of errors in docstrings	2018-02-01 12:41:34 +00:00
Richard van der Hoff	a18828c129	Fix docstring for StateHandler.resolve_state_groups ... The return type was a complete lie, so fix it	2018-02-01 12:41:34 +00:00
Richard van der Hoff	6da4c4d3bd	Factor out resolve_state_groups to a separate handler ... We extract the storage-independent bits of the state group resolution out to a separate functiom, and stick it in a new handler, in preparation for its use from the storage layer.	2018-02-01 12:40:04 +00:00
Richard van der Hoff	0cbda53819	Rename resolve_state_groups -> resolve_state_groups_for_events ... (to make way for a method that actually just does the state group resolution)	2018-02-01 12:40:00 +00:00
Richard van der Hoff	77c0629ebc	Merge pull request #2837 from matrix-org/rav/fix_quarantine_media ... Fix sql error in quarantine_media	2018-02-01 11:45:58 +01:00
Travis Ralston	e16e45b1b4	pep8 ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 15:30:38 -07:00
Richard van der Hoff	e1e4ec9f9d	factor _get_new_state_after_events out of _calculate_state_delta ... This reduces the scope of a bunch of variables	2018-01-31 21:32:09 +00:00
Richard van der Hoff	78e7e05188	Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail ... Remove spurious log argument	2018-01-31 22:18:46 +01:00
Richard van der Hoff	ad48dfe73d	Merge pull request #2835 from matrix-org/rav/remove_event_type_param ... Remove unused "event_type" param on state.get_current_state_ids	2018-01-31 22:17:57 +01:00
Richard van der Hoff	518a74586c	Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling ... Improve exception handling in persist_event	2018-01-31 22:17:38 +01:00
Richard van der Hoff	d1fe4db882	Merge pull request #2833 from matrix-org/rav/pusher_hacks ... Logging and metrics for the http pusher	2018-01-31 22:16:59 +01:00
Richard van der Hoff	421d68ca8c	Merge pull request #2817 from matrix-org/rav/http_conn_pool ... Use a connection pool for the SimpleHttpClient	2018-01-31 22:14:22 +01:00
Travis Ralston	63c4383927	Documentation and naming ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 08:07:52 -07:00
Richard van der Hoff	af19f5e9aa	Remove spurious log argument ... ... which would cause scary-looking and unhelpful errors in the log on dns fail	2018-01-30 17:52:03 +00:00
Richard van der Hoff	773f0eed1e	Fix sql error in quarantine_media	2018-01-30 15:02:51 +00:00
Richard van der Hoff	adfc0c9539	docstring for get_current_state_ids	2018-01-29 17:39:55 +00:00
Richard van der Hoff	d413a2ba98	Remove unused "event_type" param on state.get_current_state_ids ... this param doesn't seem to be used, and is a bit pointless anyway because it can easily be replicated by the caller. It is also horrible, because it changes the return type of the method.	2018-01-29 17:06:57 +00:00
Richard van der Hoff	b387ee17b6	Improve exception handling in persist_event ... 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs	2018-01-29 17:05:33 +00:00
Richard van der Hoff	03dd745fe2	Better logging when pushes fail	2018-01-29 15:49:06 +00:00
Richard van der Hoff	e051abd20b	add appid/device_display_name to to pusher logging	2018-01-29 15:04:16 +00:00
Richard van der Hoff	02ba118f81	Increase http conn pool size	2018-01-29 14:30:15 +00:00
Richard van der Hoff	46022025ea	Fix SQL for user search ... fix some syntax errors for user search when search_all_users is enabled fixes #2801, hopefully	2018-01-27 17:21:57 +00:00
Richard van der Hoff	4b0f06e99c	Merge pull request #2830 from matrix-org/rav/factor_out_get_conn ... Factor out get_db_conn to HomeServer base class	2018-01-27 17:56:25 +01:00
Neil Johnson	e98f0f9112	Merge pull request #2827 from matrix-org/fix_server_500_on_public_rooms_call_when_no_rooms_exist ... Fix server 500 on public rooms call when no rooms exist	2018-01-26 20:22:40 +00:00
Richard van der Hoff	25adde9a04	Factor out get_db_conn to HomeServer base class ... This function is identical to all subclasses, so we may as well push it up to the base class to reduce duplication (and make use of it in the tests)	2018-01-26 00:56:49 +00:00
Neil Johnson	73560237d6	add white space line	2018-01-26 00:15:10 +00:00
Neil Johnson	86c4f49a31	rather than try reconstruct the results object, better to guard against the xrange step argument being 0	2018-01-26 00:12:02 +00:00
Neil Johnson	f632083576	fix return type, should be a dict	2018-01-25 23:52:17 +00:00
Neil Johnson	6c6e197b0a	fix PEP8 violation	2018-01-25 23:47:46 +00:00
Neil Johnson	d02e43b15f	remove white space	2018-01-25 23:29:46 +00:00
Neil Johnson	349c739966	synapse 500s on a call to publicRooms in the case where the number of public rooms is zero, the specific cause is due to xrange trying to use a step value of zero, but if the total room number really is zero then it makes sense to just bail and save the extra processing	2018-01-25 23:28:44 +00:00
Matthew Hodgson	9a72b70630	fix thinko on 3pid whitelisting	2018-01-24 11:07:47 +01:00
Richard van der Hoff	25e2456ee7	Merge pull request #2816 from matrix-org/rav/metrics_comments ... Add some comments about the reactor tick time metric	2018-01-23 19:39:26 +00:00
Matthew Hodgson	d32385336f	add ?ts massaging for ASes (#2754) ... blindly implement ?ts for AS. untested	2018-01-23 09:59:06 +01:00
Richard van der Hoff	4528dd2443	Fix logging and add user_id	2018-01-22 20:15:42 +00:00
Richard van der Hoff	93efd7eb04	logging and debug for http pusher	2018-01-22 18:14:10 +00:00
Matthew Hodgson	ab9f844aaf	Add federation_domain_whitelist option (#2820) ... Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network	2018-01-22 19:11:18 +01:00
Matthew Hodgson	d84f65255e	Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid ... add registrations_require_3pid and allow_local_3pids	2018-01-22 13:57:22 +00:00
Travis Ralston	a94d9b6b82	Appease the linter ... These are ids anyways, not mxc uris. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:49:46 -07:00
Travis Ralston	5552ed9a7f	Add an admin route to get all the media in a room ... This is intended to be used by administrators to monitor the media that is passing through their server, if they wish. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:37:53 -07:00
Richard van der Hoff	2c8526cac7	Use a connection pool for the SimpleHttpClient ... In particular I hope this will help the pusher, which makes many requests to sygnal, and is currently negotiating SSL for each one.	2018-01-20 00:55:44 +00:00
Richard van der Hoff	87b7d72760	Add some comments about the reactor tick time metric	2018-01-19 23:51:04 +00:00
Matthew Hodgson	49fce04624	fix typo (thanks sytest)	2018-01-19 19:55:38 +00:00
Richard van der Hoff	b0d9e633ee	Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs ... Use the right path for url_preview thumbnails	2018-01-19 18:57:15 +00:00
Richard van der Hoff	ad7ec63d08	Use the right path for url_preview thumbnails ... This was introduced by #2627: we were overwriting the original media for url previews with the thumbnails :/ (fixes https://github.com/vector-im/riot-web/issues/6012, hopefully)	2018-01-19 18:29:39 +00:00
Matthew Hodgson	62d7d66ae5	oops, check all login types	2018-01-19 18:23:56 +00:00
Matthew Hodgson	8fe253f19b	fix PR nitpicking	2018-01-19 18:23:45 +00:00
Matthew Hodgson	293380bef7	trailing commas	2018-01-19 15:38:53 +00:00
Matthew Hodgson	447f4f0d5f	rewrite based on PR feedback: ... * [ ] split config options into allowed_local_3pids and registrations_require_3pid * [ ] simplify and comment logic for picking registration flows * [ ] fix docstring and move check_3pid_allowed into a new util module * [ ] use check_3pid_allowed everywhere @erikjohnston PTAL	2018-01-19 15:33:55 +00:00
Matthew Hodgson	9d332e0f79	fix up v1, and improve errors	2018-01-19 00:53:58 +00:00
Matthew Hodgson	0af58f14ee	fix pep8	2018-01-19 00:33:51 +00:00
Matthew Hodgson	28a6ccb49c	add registrations_require_3pid ... lets homeservers specify a whitelist for 3PIDs that users are allowed to associate with. Typically useful for stopping people from registering with non-work emails	2018-01-19 00:19:58 +00:00
Erik Johnston	cd871a3057	Fix storage provider bug introduced when renamed to store_local	2018-01-18 18:37:59 +00:00
Erik Johnston	8ff6726c0d	Merge pull request #2812 from matrix-org/erikj/media_storage_provider_config ... Make storage providers configurable	2018-01-18 18:33:57 +00:00
Erik Johnston	d69768348f	Fix passing wrong config to provider constructor	2018-01-18 17:14:05 +00:00
Erik Johnston	8e85220373	Remove duplicate directory test	2018-01-18 17:12:35 +00:00
Erik Johnston	3fe2bae857	Missing staticmethod	2018-01-18 17:11:45 +00:00
Erik Johnston	aae77da73f	Fixup comments	2018-01-18 17:11:29 +00:00
Erik Johnston	b6dc7044a9	Merge pull request #2804 from matrix-org/erikj/file_consumer ... Add decent impl of a FileConsumer	2018-01-18 16:31:33 +00:00
Erik Johnston	9a89dae8c5	Fix typo in thumbnail resource causing access times to be incorrect	2018-01-18 15:06:24 +00:00
Erik Johnston	0af5dc63a8	Make storage providers more configurable	2018-01-18 14:07:21 +00:00
Richard van der Hoff	d57765fc8a	Fix bugs in block metrics ... ... which I introduced in #2785	2018-01-18 12:24:42 +00:00
Erik Johnston	2cf6a7bc20	Use better file consumer	2018-01-18 12:00:46 +00:00
Erik Johnston	4a53f3a3e8	Ensure media is in local cache before thumbnailing	2018-01-18 12:00:46 +00:00
Erik Johnston	be0dfcd4a2	Do logcontexts correctly	2018-01-18 11:57:57 +00:00
Erik Johnston	1432f7ccd5	Move test stuff to tests	2018-01-18 11:57:57 +00:00
Erik Johnston	2f18a2647b	Make all fields private	2018-01-18 11:57:54 +00:00
Richard van der Hoff	d6af5512bb	Merge pull request #2809 from matrix-org/rav/metrics_errors ... better exception logging in callbackmetrics	2018-01-18 11:46:37 +00:00
Richard van der Hoff	ce236f8ac8	better exception logging in callbackmetrics ... when we fail to render a metric, give a clue as to which metric it was	2018-01-18 11:30:49 +00:00
Erik Johnston	dc519602ac	Ensure we registerProducer isn't called twice	2018-01-18 11:07:17 +00:00
Erik Johnston	17b54389fe	Fix _notify_empty typo	2018-01-18 11:05:34 +00:00
Erik Johnston	28b338ed9b	Move definition of paused_producer to __init__	2018-01-18 11:04:41 +00:00
Erik Johnston	a177325b49	Fix comments	2018-01-18 11:02:43 +00:00
Richard van der Hoff	36da256cc6	Merge pull request #2805 from matrix-org/rav/log_state_res ... Log room when doing state resolution	2018-01-17 18:05:04 +00:00
Richard van der Hoff	1224612a79	Log room when doing state resolution ... Mostly because it helps figure out what is prompting the resolution	2018-01-17 17:11:59 +00:00
Erik Johnston	bc67e7d260	Add decent impl of a FileConsumer ... Twisted core doesn't have a general purpose one, so we need to write one ourselves. Features: - All writing happens in background thread - Supports both push and pull producers - Push producers get paused if the consumer falls behind	2018-01-17 16:43:03 +00:00
Erik Johnston	a87006f9c7	Merge pull request #2783 from matrix-org/erikj/media_last_accessed ... Keep track of last access time for local media	2018-01-17 16:39:02 +00:00
Matthew Hodgson	06db5c4b76	Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql ... fix SQL when searching all users	2018-01-17 16:27:54 +00:00
Matthew Hodgson	2d9ab533f9	fix SQL when searching all users	2018-01-17 15:58:52 +00:00
Richard van der Hoff	390093d45e	Split resolve_events into two functions ... ... so that the return type doesn't depend on the arg types	2018-01-17 15:44:31 +00:00
Erik Johnston	2fb3a28c98	Remove lost comment	2018-01-17 14:59:44 +00:00
Richard van der Hoff	a7e4ff9cca	Merge pull request #2795 from matrix-org/rav/track_db_scheduling ... Track DB scheduling delay per-request	2018-01-17 14:29:37 +00:00
Richard van der Hoff	f884cfffb9	Merge pull request #2797 from matrix-org/rav/user_id_checking ... Sanity checking for user ids	2018-01-17 14:29:12 +00:00
Richard van der Hoff	a5213df1f7	Sanity checking for user ids ... Check the user_id passed to a couple of APIs for validity, to avoid "IndexError: list index out of range" exception which looks scary and results in a 500 rather than a more useful error. Fixes #1432, among other things	2018-01-17 14:28:54 +00:00
Richard van der Hoff	e8f7541d3f	Merge remote-tracking branch 'origin/develop' into rav/track_db_scheduling	2018-01-17 14:01:57 +00:00
Richard van der Hoff	fb6563b4be	Merge pull request #2793 from matrix-org/rav/db_txn_time_in_millis ... Track db txn time in millisecs	2018-01-17 13:52:42 +00:00
Richard van der Hoff	1954e867b4	Merge pull request #2796 from matrix-org/rav/fix_closed_connection_errors ... Fix 'NoneType' object has no attribute 'writeHeaders'	2018-01-17 13:52:24 +00:00
Richard van der Hoff	11ab2f56f5	Merge branch 'develop' into rav/fix_closed_connection_errors	2018-01-17 11:25:11 +00:00
Richard van der Hoff	0486a7814a	Merge branch 'develop' into rav/rework_run_interaction	2018-01-17 11:24:38 +00:00
Richard van der Hoff	90c14da992	Merge branch 'develop' into rav/db_txn_time_in_millis	2018-01-17 11:19:55 +00:00
Richard van der Hoff	38506773eb	Merge remote-tracking branch 'origin/develop' into rav/optimise_logging_context	2018-01-17 10:57:13 +00:00
Erik Johnston	300edc2348	Update last access time when thumbnails are viewed	2018-01-17 10:24:43 +00:00
Erik Johnston	05f98a2224	Keep track of last access time for local media	2018-01-17 10:24:43 +00:00
Erik Johnston	3cb2dabaad	Merge pull request #2789 from matrix-org/erikj/fix_thumbnails ... Fix thumbnailing remote files	2018-01-17 10:22:20 +00:00
Erik Johnston	d728c47142	Add docstring	2018-01-17 10:06:14 +00:00
Richard van der Hoff	936482d507	Fix 'NoneType' object has no attribute 'writeHeaders' ... Avoid throwing a (harmless) exception when we try to write an error response to an http request where the client has disconnected. This comes up as a CRITICAL error in the logs which tends to mislead people into thinking there's an actual problem	2018-01-16 17:58:16 +00:00
Richard van der Hoff	3d12d97415	Track DB scheduling delay per-request ... For each request, track the amount of time spent waiting for a db connection. This entails adding it to the LoggingContext and we may as well add metrics for it while we are passing.	2018-01-16 17:23:32 +00:00
Richard van der Hoff	0f5d2cc37c	Merge branch 'rav/db_txn_time_in_millis' into rav/track_db_scheduling	2018-01-16 17:09:15 +00:00
Richard van der Hoff	8615f19d20	rework runInteraction in terms of runConnection ... ... so that we can share the code	2018-01-16 17:08:29 +00:00
Matthew Hodgson	5e97ca7ee6	fix typo	2018-01-16 16:52:35 +00:00
Erik Johnston	d863f68cab	Use local vars	2018-01-16 16:24:15 +00:00
Erik Johnston	6368e5c0ab	Change _generate_thumbnails to take media_type	2018-01-16 16:17:38 +00:00
Erik Johnston	0a90d9ede4	Move setting of file_id up to caller	2018-01-16 16:03:05 +00:00
Richard van der Hoff	6324b65f08	Track db txn time in millisecs ... ... to reduce the amount of floating-point foo we do.	2018-01-16 15:53:18 +00:00
Richard van der Hoff	44a498418c	Optimise LoggingContext creation and copying ... It turns out that the only thing we use the __dict__ of LoggingContext for is `request`, and given we create lots of LoggingContexts and then copy them every time we do a db transaction or log line, using the __dict__ seems a bit redundant. Let's try to optimise things by making the request attribute explicit.	2018-01-16 15:49:42 +00:00
Erik Johnston	5dfc83704b	Fix typo	2018-01-16 14:32:56 +00:00
Richard van der Hoff	febdca4b37	Merge pull request #2785 from matrix-org/rav/reorganise_metrics_again ... Reorganise request and block metrics	2018-01-16 14:10:21 +00:00
Richard van der Hoff	f5f89fda21	Merge pull request #2790 from matrix-org/rav/preserve_event_logcontext_leak ... Fix a logcontext leak in persist_events	2018-01-16 14:09:13 +00:00
Erik Johnston	307f88dfb6	Fix up log lines	2018-01-16 13:53:52 +00:00
Richard van der Hoff	807e848f0f	Merge pull request #2787 from matrix-org/rav/worker_event_counts ... Metrics for events processed in appservice and fed sender	2018-01-16 13:14:09 +00:00
Richard van der Hoff	4a31a61ef9	Merge pull request #2786 from matrix-org/rav/rdata_metrics ... Metrics for number of RDATA commands received	2018-01-16 13:13:53 +00:00
Erik Johnston	9795b9ebb1	Correctly use server_name/file_id when generating/fetching remote thumbnails	2018-01-16 12:02:06 +00:00
Erik Johnston	c5b589f2e8	Log when we respond with 404	2018-01-16 12:01:40 +00:00
Richard van der Hoff	64ddec1bc0	Fix a logcontext leak in persist_events ... ObserveableDeferred expects its callbacks to be called without any logcontexts, whereas it turns out we were calling them with the logcontext of the request which initiated the persistence loop. It seems wrong that we are attributing work done in the persistence loop to the request that happened to initiate it, so let's solve this by dropping the logcontext for it. (I'm not sure this actually causes any real problems other than messages in the debug log, but let's clean it up anyway)	2018-01-16 11:47:36 +00:00
Erik Johnston	a4c5e4a645	Fix thumbnailing remote files	2018-01-16 11:37:50 +00:00
Erik Johnston	1159abbdd2	Merge pull request #2767 from matrix-org/erikj/media_storage_refactor ... Refactor MediaRepository to separate out storage	2018-01-16 10:23:50 +00:00
Richard van der Hoff	a027c2af8d	Metrics for events processed in appservice and fed sender ... More metrics I wished I'd had	2018-01-15 18:23:24 +00:00
Richard van der Hoff	5c3c32f16f	Metrics for number of RDATA commands received ... I found myself wishing we had this.	2018-01-15 17:45:55 +00:00
Richard van der Hoff	39f4e29d01	Reorganise request and block metrics ... In order to circumvent the number of duplicate foo:count metrics increasing without bounds, it's time for a rearrangement. The following are all deprecated, and replaced with synapse_util_metrics_block_count: synapse_util_metrics_block_timer:count synapse_util_metrics_block_ru_utime:count synapse_util_metrics_block_ru_stime:count synapse_util_metrics_block_db_txn_count:count synapse_util_metrics_block_db_txn_duration:count The following are all deprecated, and replaced with synapse_http_server_response_count: synapse_http_server_requests synapse_http_server_response_time:count synapse_http_server_response_ru_utime:count synapse_http_server_response_ru_stime:count synapse_http_server_response_db_txn_count:count synapse_http_server_response_db_txn_duration:count The following are renamed (the old metrics are kept for now, but deprecated): synapse_util_metrics_block_timer:total -> synapse_util_metrics_block_time_seconds synapse_util_metrics_block_ru_utime:total -> synapse_util_metrics_block_ru_utime_seconds synapse_util_metrics_block_ru_stime:total -> synapse_util_metrics_block_ru_stime_seconds synapse_util_metrics_block_db_txn_count:total -> synapse_util_metrics_block_db_txn_count synapse_util_metrics_block_db_txn_duration:total -> synapse_util_metrics_block_db_txn_duration_seconds synapse_http_server_response_time:total -> synapse_http_server_response_time_seconds synapse_http_server_response_ru_utime:total -> synapse_http_server_response_ru_utime_seconds synapse_http_server_response_ru_stime:total -> synapse_http_server_response_ru_stime_seconds synapse_http_server_response_db_txn_count:total -> synapse_http_server_response_db_txn_count synapse_http_server_response_db_txn_duration:total synapse_http_server_response_db_txn_duration_seconds	2018-01-15 17:09:44 +00:00
Richard van der Hoff	992018d1c0	mechanism to render metrics with alternative names	2018-01-15 17:04:39 +00:00
Richard van der Hoff	80fa610f9c	Add some comments to metrics classes	2018-01-15 16:52:52 +00:00
Richard van der Hoff	19d274085f	Make Counter render floats ... Prometheus handles all metrics as floats, and sometimes we store non-integer values in them (notably, durations in seconds), so let's render them as floats too. (Note that the standard client libraries also treat Counters as floats.)	2018-01-12 23:49:44 +00:00
Richard van der Hoff	21bf87a146	Reinstate media download on thumbnail request ... We need to actually download the remote media when we get a request for a thumbnail.	2018-01-12 15:38:06 +00:00
Erik Johnston	694f1c1b18	Fix up comments	2018-01-12 15:02:46 +00:00
Erik Johnston	e21370ba54	Correctly reraise exception	2018-01-12 14:44:02 +00:00
Erik Johnston	85a4d78213	Make Responder a context manager	2018-01-12 13:32:03 +00:00
Erik Johnston	dcc8eded41	Add missing class var	2018-01-12 13:16:27 +00:00
Erik Johnston	fefeb0ab0e	Merge pull request #2774 from matrix-org/erikj/synctl ... When using synctl with workers, don't start the main synapse automatically	2018-01-12 12:00:07 +00:00
Erik Johnston	81391fa162	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/media_storage_refactor	2018-01-12 11:28:49 +00:00
Erik Johnston	1e4edd1717	Remove unnecessary condition	2018-01-12 11:28:32 +00:00
Erik Johnston	c6c009603c	Remove unused variables	2018-01-12 11:24:05 +00:00
Erik Johnston	4d88958cf6	Make class var local	2018-01-12 11:23:54 +00:00
Erik Johnston	227c491510	Comments	2018-01-12 11:22:41 +00:00
Erik Johnston	f4d93ae424	Actually make it work	2018-01-12 10:39:27 +00:00
Erik Johnston	f68e4cf690	Refactor	2018-01-12 10:11:12 +00:00
Richard van der Hoff	5f23b6d5ea	Merge pull request #2766 from matrix-org/rav/room_event ... Add /room/{id}/event/{id} to synapse	2018-01-11 14:47:18 +00:00
Erik Johnston	7cd34512d8	When using synctl with workers, don't start the main synapse automatically	2018-01-11 11:37:39 +00:00
Erik Johnston	07ab948c38	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-01-11 11:23:40 +00:00
Erik Johnston	825a07a974	Merge pull request #2773 from matrix-org/erikj/hash_bg ... Do bcrypt hashing in a background thread	2018-01-10 18:11:41 +00:00
Erik Johnston	f8e1ab5fee	Do bcrypt hashing in a background thread	2018-01-10 18:01:28 +00:00
Michael Telatynski	5f07f5694c	fix order of operations derp and also use .get to default to {} ... Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>	2018-01-10 15:11:35 +00:00
Michael Telatynski	8c9d5b4873	Fix publicised groups API (singular) over federation ... which was missing its fed client API, since there is no other API it might as well reuse the bulk one and unwrap it Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>	2018-01-10 15:04:51 +00:00
Richard van der Hoff	c175a5f0f2	Merge pull request #2770 from matrix-org/rav/fix_request_metrics ... Update http request metrics before calling servlet	2018-01-10 14:53:40 +00:00
Richard van der Hoff	d90e8ea444	Update http request metrics before calling servlet ... Make sure that we set the servlet name in the metrics object *before* calling the servlet, in case the servlet throws an exception.	2018-01-09 18:27:35 +00:00
hera	174eacc8ba	oops	2018-01-09 18:14:32 +00:00
Matthew Hodgson	a66f489678	fix GIST->GIN switch	2018-01-09 16:55:51 +00:00
Matthew Hodgson	e79db0a673	switch back from GIST to GIN indexes	2018-01-09 16:37:48 +00:00
Matthew Hodgson	e365ad329f	oops, tweak work_mem when actually storing	2018-01-09 16:30:30 +00:00
Matthew Hodgson	19f9227643	avoid 80s GIN inserts by tweaking work_mem ... see https://github.com/matrix-org/synapse/issues/2753 for details	2018-01-09 16:25:04 +00:00
Erik Johnston	8f03aa9f61	Add StorageProvider concept	2018-01-09 16:16:12 +00:00
Erik Johnston	2442e9876c	Make PreviewUrlResource use MediaStorage	2018-01-09 16:15:07 +00:00
Erik Johnston	9d30a7691c	Make ThumbnailResource use MediaStorage	2018-01-09 16:15:07 +00:00
Erik Johnston	9e20840e02	Use MediaStorage for remote media	2018-01-09 16:15:07 +00:00
Erik Johnston	dd3092c3a3	Use MediaStorage for local files	2018-01-09 16:15:07 +00:00
Erik Johnston	ada470bccb	Add MediaStorage class	2018-01-09 16:15:07 +00:00
Erik Johnston	1ee787912b	Add some helper classes	2018-01-09 16:15:07 +00:00
Erik Johnston	47ca5eb882	Split out add_file_headers	2018-01-09 16:15:07 +00:00
Erik Johnston	b6c9deffda	Remove dead TODO	2018-01-09 15:53:23 +00:00
Richard van der Hoff	51c9d9ed65	Add /room/{id}/event/{id} to synapse ... Turns out that there is a valid usecase for retrieving event by id (notably having received a push), but event ids should be scoped to room, so /event/{id} is wrong.	2018-01-09 14:39:12 +00:00
Erik Johnston	b30cd5b107	Remove dead code related to default thumbnails	2018-01-09 14:38:33 +00:00
Richard van der Hoff	cb66a2d387	Merge pull request #2763 from matrix-org/rav/fix_config_uts ... Fix broken config UTs	2018-01-09 12:08:08 +00:00
Richard van der Hoff	0211464ba2	Fix broken config UTs ... https://github.com/matrix-org/synapse/pull/2755 broke log-config generation, which in turn broke the unit tests.	2018-01-09 11:28:33 +00:00
Richard van der Hoff	3a556f1ea0	Make indentation of generated log config consistent ... (we had a mix of 2- and 4-space indents)	2018-01-09 11:27:19 +00:00
Travis Ralston	eccfc8e928	Fix templating error with unban permission message ... Fixes https://github.com/matrix-org/synapse/issues/2759 Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-07 19:52:58 -07:00
Richard van der Hoff	840f72356e	Remove 'verbosity'/'log_file' from generated cfg ... ... because these only really exist to confuse people nowadays. Also bring log config more into line with the generated log config, by making `level_for_storage` apply to the `synapse.storage.SQL` logger rather than `synapse.storage`.	2018-01-05 12:30:28 +00:00
Erik Johnston	864a6d2977	Bump version and changelog	2018-01-05 10:54:01 +00:00
Richard van der Hoff	6e375f4597	Merge pull request #2744 from matrix-org/rav/login_logging ... Better logging when login can't find a 3pid	2018-01-05 09:54:39 +00:00
Richard van der Hoff	bd91857028	Check missing fields in event_from_pdu_json ... Return a 400 rather than a 500 when somebody messes up their send_join	2017-12-30 18:40:19 +00:00
Richard van der Hoff	3079f80d4a	Factor out event_from_pdu_json ... turns out we have two copies of this, and neither needs to be an instance method	2017-12-30 18:40:19 +00:00
Richard van der Hoff	65abc90fb6	federation_server: clean up imports	2017-12-30 18:40:19 +00:00
Richard van der Hoff	a7b726ad18	federation_client: clean up imports	2017-12-30 18:40:19 +00:00
Richard van der Hoff	75c1b8df01	Better logging when login can't find a 3pid	2017-12-20 19:31:00 +00:00
Richard van der Hoff	3f9f1c50f3	Merge pull request #2683 from seckrv/fix_pwd_auth_prov_typo ... synapse/config/password_auth_providers: Fixed bracket typo	2017-12-18 22:37:21 +00:00
Silke	df0f602796	Implement listen_tcp method in remaining workers ... Signed-off-by: Silke <silke@slxh.eu>	2017-12-18 20:00:42 +01:00
Silke	26cd3f5690	Remove logger argument and do not catch replication listener ... Signed-off-by: Silke <silke@slxh.eu>	2017-12-18 20:00:42 +01:00
Silke Hofstra	ed48ecc58c	Add methods for listening on multiple addresses ... Add listen_tcp and listen_ssl which implement Twisted's reactor.listenTCP and reactor.listenSSL for multiple addresses. Signed-off-by: Silke Hofstra <silke@slxh.eu>	2017-12-17 13:15:48 +01:00
Silke Hofstra	37d1a90025	Allow binds to both :: and 0.0.0.0 ... Binding on 0.0.0.0 when :: is specified in the bind_addresses is now allowed. This causes a warning explaining the behaviour. Configuration changed to match. See #2232 Signed-off-by: Silke Hofstra <silke@slxh.eu>	2017-12-17 13:10:31 +01:00
Willem Mulder	3e59143ba8	Adapt the default config to bind on IPv6. ... Most deployments are on Linux (or Mac OS), so this would actually bind on both IPv4 and IPv6. Resolves #1886. Signed-off-by: Willem Mulder <willemmaster@hotmail.com>	2017-12-17 13:07:37 +01:00
Erik Johnston	069ae2a5d6	Bump changelog and version	2017-12-13 15:11:22 +00:00
Erik Johnston	d8a6c734fa	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/createroom_content	2017-12-07 14:24:01 +00:00
Erik Johnston	ef045dcd71	Copy dict in update_membership too	2017-12-07 14:17:15 +00:00
Matthew Hodgson	33cb7ef0b7	Merge pull request #2723 from matrix-org/matthew/search-all-local-users ... Add all local users to the user_directory and optionally search them	2017-12-05 11:09:47 +00:00
Matthew Hodgson	cdc2cb5d11	fix StoreError syntax	2017-12-05 11:09:31 +00:00
Richard van der Hoff	16ec3805e5	Fix error when deleting devices ... This was introduced in d7ea8c4 / PR #2728	2017-12-05 09:49:22 +00:00
Richard van der Hoff	da1010c83a	support custom login types for validating users ... Wire the custom login type support from password providers into the UI-auth user-validation flows.	2017-12-05 09:43:30 +00:00
Richard van der Hoff	d7ea8c4800	Factor out a validate_user_via_ui_auth method ... Collect together all the places that validate a logged-in user via UI auth.	2017-12-05 09:42:30 +00:00
Richard van der Hoff	d5f9fb06b0	Refactor UI auth implementation ... Instead of returning False when auth is incomplete, throw an exception which can be caught with a wrapper.	2017-12-05 09:40:05 +00:00
Matthew Hodgson	c22e73293a	speed up the rate of initial spam for users	2017-12-04 18:05:28 +00:00
Matthew Hodgson	b11dca2025	better doc	2017-12-04 17:51:33 +00:00
Matthew Hodgson	74e0cc74ce	fix pep8 and tests	2017-12-04 15:11:38 +00:00
Matthew Hodgson	1bd40ca73e	switch to a simpler 'search_all_users' button as per review feedback	2017-12-04 14:58:39 +00:00
Matthew Hodgson	f397153dfc	Merge branch 'develop' into matthew/search-all-local-users	2017-11-30 01:51:38 +00:00
Matthew Hodgson	5406392f8b	specify default user_directory_include_pattern	2017-11-30 01:45:34 +00:00
Matthew Hodgson	f61e107f63	remove null constraint on user_dir.room_id	2017-11-30 01:43:50 +00:00
Matthew Hodgson	4b1fceb913	fix alternation operator for FTS4 - how did this ever work!?	2017-11-30 01:34:03 +00:00
Matthew Hodgson	a4bb133b68	fix thinkos galore	2017-11-30 01:17:15 +00:00
Matthew Hodgson	cd3697e8b7	kick the user_directory index when new users register	2017-11-29 18:33:34 +00:00
Matthew Hodgson	3241c7aac3	untested WIP but might actually work	2017-11-29 18:27:05 +00:00
Richard van der Hoff	624c46eb06	Merge pull request #2721 from matrix-org/rav/get_user_by_access_token_comments ... Improve comments on get_user_by_access_token	2017-11-29 17:57:06 +00:00
Matthew Hodgson	47d99a20d5	Add user_directory_include_pattern config param to expand search results to additional users ... Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive. It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users... ...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too. Committing it here as a WIP	2017-11-29 16:46:45 +00:00
Richard van der Hoff	ad7e570d07	Delete devices in various logout situations ... Make sure that we delete devices whenever a user is logged out due to any of the following situations: * /logout * /logout_all * change password * deactivate account (by the user or by an admin) * invalidate access token from a dynamic module Fixes #2672.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	ae31f8ce45	Move set_password into its own handler ... Non-functional refactoring to move set_password. This means that we'll be able to properly deactivate devices and access tokens without introducing a dependency loop.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	7ca5c68233	Move deactivate_account into its own handler ... Non-functional refactoring to move deactivate_account. This means that we'll be able to properly deactivate devices and access tokens without introducing a dependency loop.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	2c6d63922a	Remove pushers when deleting access tokens ... Whenever an access token is invalidated, we should remove the associated pushers.	2017-11-29 16:44:35 +00:00
Richard van der Hoff	97d1a1dc01	Merge pull request #2718 from matrix-org/rav/notify_logcontexts ... Clear logcontext before starting fed txn queue runner	2017-11-29 16:01:46 +00:00
Richard van der Hoff	8b45de90a4	Merge pull request #2719 from matrix-org/rav/handle_missing_hashes ... Fix 500 when joining matrix-dev	2017-11-29 16:01:33 +00:00
Richard van der Hoff	7303ed65e1	Fix 500 when joining matrix-dev ... matrix-dev has an event (`$/6ANj/9QWQyd71N6DpRQPf+SDUu11+HVMeKSpMzBCwM:zemos.net`) which has no `hashes` member. Check for missing `hashes` element in events.	2017-11-29 16:00:46 +00:00
Richard van der Hoff	da562bd6a1	Improve comments on get_user_by_access_token ... because I have to reverse-engineer this every time.	2017-11-29 15:52:41 +00:00
Richard van der Hoff	d4fb4f7c52	Clear logcontext before starting fed txn queue runner ... These processes take a long time compared to the request, so there is lots of "Entering|Restoring dead context" in the logs. Let's try to shut it up a bit.	2017-11-28 15:26:14 +00:00
Erik Johnston	dfbc45302e	PEP8	2017-11-28 15:23:26 +00:00
Erik Johnston	c4c1d170af	Fix wrong avatars when inviting multiple users when creating room ... We reused the `content` dictionary between invite requests, which meant they could end up reusing the profile info for a previous user	2017-11-28 15:19:15 +00:00
Luke Barnard	ab1b2d0ff2	Allow guest access to group APIs for reading	2017-11-28 11:23:00 +00:00
Richard van der Hoff	5a4da5bf78	Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error ... Fix error on sqlite 3.7	2017-11-27 12:25:48 +00:00
Richard van der Hoff	84b31a3e7a	Merge pull request #2713 from matrix-org/rav/no_upsert_forever ... Avoid retrying forever on IntegrityError	2017-11-27 12:19:35 +00:00
Richard van der Hoff	df6c72ede3	Merge pull request #2711 from matrix-org/rav/fix_dns_errhandler ... Fix error handling on dns lookup	2017-11-27 12:19:18 +00:00
Richard van der Hoff	04bb79f139	Merge pull request #2710 from matrix-org/rav/remove_dead_code ... Tiny code cleanups	2017-11-27 12:15:44 +00:00
Richard van der Hoff	e828a7380a	Merge pull request #2708 from matrix-org/rav/replication_logcontext_leaks ... Fix some logcontext leaks in replication resource	2017-11-27 12:15:33 +00:00
Richard van der Hoff	7ef22a41a3	Merge pull request #2707 from matrix-org/rav/fix_urlpreview ... Fix OPTIONS on preview_url	2017-11-27 12:15:14 +00:00
Richard van der Hoff	6be01f599b	Improve tracebacks on exceptions ... Use failure.Failure to recover our failure, which will give us a useful stacktrace, unlike the rethrown exception.	2017-11-27 12:05:58 +00:00
Richard van der Hoff	63ccaa5873	Avoid retrying forever on IntegrityError	2017-11-27 12:00:07 +00:00
Richard van der Hoff	8b38096a89	Fix error handling on dns lookup ... pass the right arguments to the errback handler Fixes "TypeError('eb() takes exactly 2 arguments (1 given)',)"	2017-11-24 16:47:48 +00:00
Richard van der Hoff	795b0849f3	Add a comment which might save some confusion	2017-11-24 00:34:56 +00:00
Richard van der Hoff	7f14f0ae38	Remove dead sync_callback ... This is never used; let's remove it to stop confusing things.	2017-11-24 00:32:04 +00:00
Richard van der Hoff	0edf085b68	Fix some logcontext leaks in replication resource ... The @measure_func annotations rely on the wrapped function respecting the logcontext rules. Add the necessary yields to make this work.	2017-11-23 23:19:43 +00:00
Richard van der Hoff	8132a6b7ac	Fix OPTIONS on preview_url ... Fixes #2706	2017-11-23 17:52:31 +00:00
Richard van der Hoff	6b48b3e277	fix sql fails	2017-11-22 18:06:24 +00:00
Richard van der Hoff	2908f955d1	Check database in has_completed_background_updates ... so that the right thing happens on workers.	2017-11-22 18:02:15 +00:00
Richard van der Hoff	68ca864141	Add config option to disable media_repo on main synapse ... ... to stop us doing the cache cleanup jobs on the master.	2017-11-22 16:20:27 +00:00
Richard van der Hoff	e1fd4751de	Build MediaRepositoryResource as a homeserver dependency ... This avoids the scenario where we have four different PreviewUrlResources configured on a single app, each of which have their own caches and cache clearing jobs.	2017-11-22 16:19:49 +00:00
Richard van der Hoff	d5a7c56ef9	Merge pull request #2698 from matrix-org/rav/remove_dead_dependencies ... Clean up dependency list	2017-11-21 17:38:42 +00:00
Richard van der Hoff	0b4aa2dc21	Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert ... Avoid locking account_data tables for upserts	2017-11-21 13:39:14 +00:00
Richard van der Hoff	7298ed7c51	Clean up dependency list ... remove those that aren't used at all, and replace the ones that don't have builders with simple getters rather than dynamically-generated methods.	2017-11-21 11:15:41 +00:00
Richard van der Hoff	7098b65cb8	Fix error on sqlite 3.7 ... Create the url_cache index on local_media_repository as a background update, so that we can detect whether we are on sqlite or not and create a partial or complete index accordingly. To avoid running the cleanup job before we have built the index, add a bailout which will defer the cleanup if the bg updates are still running. Fixes https://github.com/matrix-org/synapse/issues/2572.	2017-11-21 11:14:17 +00:00
Oliver Kurz	83d8d4d8cd	Allow use of higher versions of saml2 ... The package was pinned to <4.0 with 07cf96eb because "from saml2 import config" did not work. This seems to have been fixed in the mean time in the saml2 package and therefore should not stop to use a more recent version. Signed-off-by: Oliver Kurz <okurz@suse.de>	2017-11-20 11:14:39 +01:00
Richard van der Hoff	59a7275258	Merge pull request #2688 from matrix-org/rav/unlock_more_upsert ... Avoid locking for upsert on pushers tables	2017-11-17 13:09:31 +00:00
Richard van der Hoff	d8a05418f9	Merge branch 'master' into develop	2017-11-17 12:13:14 +00:00
Luke Barnard	cdf6fc15b0	Merge pull request #2686 from matrix-org/luke/as-flair ... Add automagical AS Publicised Group(s)	2017-11-17 10:13:46 +00:00
Richard van der Hoff	74bbeb4373	Bump version in __init__.py	2017-11-17 10:10:53 +00:00
Jurek	eded7084d2	Fix auth handler #2678	2017-11-17 10:07:27 +00:00
Matthew Hodgson	34c3d0a386	typo	2017-11-17 01:54:02 +00:00
Luke Barnard	7fb0e98b03	Extract group_id from the dict for multiple use	2017-11-16 18:18:30 +00:00
Richard van der Hoff	c46139a17e	Avoid locking account_data tables for upserts	2017-11-16 18:08:01 +00:00
Luke Barnard	d8391f0541	Remove unused GROUP_ID_REGEX	2017-11-16 18:05:57 +00:00
Luke Barnard	4e8374856d	Document get_groups_for_user	2017-11-16 18:03:46 +00:00
Luke Barnard	270f9cd23a	Flake8	2017-11-16 18:03:31 +00:00
Luke Barnard	9d83d52027	Use a generator instead of a list	2017-11-16 17:57:34 +00:00
Luke Barnard	5b48eec4a1	Make sure we check AS groups for lookup on bulk	2017-11-16 17:55:15 +00:00
Luke Barnard	b1edf26051	Check group_id belongs to this domain	2017-11-16 17:54:27 +00:00
Richard van der Hoff	06e5bcfc83	Avoid locking for upsert on pushers tables ... * replace the upsert into deleted_pushers with an insert * no need to lock for upsert on pusher_throttle	2017-11-16 17:52:23 +00:00
Jurek	624a8bbd67	Fix auth handler #2678	2017-11-16 17:19:02 +00:00
Luke Barnard	97bd18af4e	Add automagical AS Publicised Group(s) ... via registration file "users" namespace: ```YAML ... namespaces: users: - exclusive: true regex: '.*luke.*' group_id: '+all_the_lukes:hsdomain' ... ``` This is part of giving App Services their own groups for matching users. With this, ghost users will be given the appeareance that they are in a group and that they have publicised the fact, but _only_ from the perspective of the `get_publicised_groups_for_user` API.	2017-11-16 16:44:55 +00:00
Richard van der Hoff	77a1227870	Fix broken ref to IntegrityError	2017-11-16 16:03:38 +00:00
Richard van der Hoff	7ab2b69e18	Avoid locking pushers table on upsert ... Now that _simple_upsert will retry on IntegrityError, we don't need to lock the table.	2017-11-16 15:32:01 +00:00
Richard van der Hoff	10aaa1bc15	_simple_upsert: retry on IntegrityError ... wrap the call to _simple_upsert_txn in a loop so that we retry on an integrityerror: this means we can avoid locking the table provided there is an unique index.	2017-11-16 15:30:15 +00:00
Richard van der Hoff	cdc9e50a5d	Cleanup in _simple_upsert_txn ... Bail out early to reduce indentation	2017-11-16 15:29:10 +00:00
Richard von Seck	6f05de0e5e	synapse/config/password_auth_providers: Fixed bracket typo ... Signed-off-by: Richard von Seck <richard.von-seck@gmx.net>	2017-11-16 15:59:38 +01:00
Richard van der Hoff	f959c01600	Merge pull request #2661 from matrix-org/rav/statereadstore ... Pull out bits of StateStore to a mixin	2017-11-15 17:23:01 +00:00
Richard van der Hoff	30d2730ee2	Declare support for r0.3.0	2017-11-15 16:24:22 +00:00
Richard van der Hoff	1b83c09c03	Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs ... Remove preserve_context_over_{fn, deferred}	2017-11-15 11:13:53 +00:00
David Baker	7190a550dc	Merge pull request #2650 from matrix-org/dbkr/push_include_content_option ... Rename redact_content option to include_content	2017-11-15 10:47:38 +00:00
Richard van der Hoff	b2cd6accf5	Remove __PreservingContextDeferred too	2017-11-14 23:00:10 +00:00
Richard van der Hoff	008aa2fc6d	Merge pull request #2671 from matrix-org/rav/room_list_fixes ... Reshuffle room list request code	2017-11-14 18:11:02 +00:00
Erik Johnston	03feb7a34d	Bump version and changelog	2017-11-14 14:51:25 +00:00
Richard van der Hoff	35a4b63240	Pull out bits of StateStore to a mixin ... ... so that we don't need to secretly gut-wrench it for use in the slaved stores. I haven't done the other stores yet, but we should. I'm tired of the workers breaking every time we tweak the stores because I forgot to gut-wrench the right method. fixes https://github.com/matrix-org/synapse/issues/2655.	2017-11-14 11:43:58 +00:00
Richard van der Hoff	4dd1bfa8c1	Revert "Revert "move _state_group_cache to statestore"" ... We're going to fix this properly on this branch, so that the _state_group_cache can end up in StateGroupReadStore. This reverts commit ab335edb02.	2017-11-14 11:43:58 +00:00
Richard van der Hoff	6caa379ba1	Merge pull request #2658 from matrix-org/rav/store_heirarchy_init ... Make __init__ consistent across Store hierarchy	2017-11-14 11:43:12 +00:00
Richard van der Hoff	7e6fa29cb5	Remove preserve_context_over_{fn, deferred} ... Both of these functions ae known to leak logcontexts. Replace the remaining calls to them and kill them off.	2017-11-14 11:22:42 +00:00
Richard van der Hoff	44a1bfd6a6	Reshuffle room list request code ... I'm not entirely sure if this will actually help anything, but it simplifies the code and might give further clues about why room list search requests are blowing out the get_current_state_ids caches.	2017-11-14 10:29:58 +00:00
Richard van der Hoff	1fc66c7460	Add a load of logging to the room_list handler ... So we can see what it gets up to.	2017-11-14 10:23:47 +00:00
Richard van der Hoff	7bd6c87eca	Merge pull request #2668 from turt2live/travis/whoami ... Add a route for determining who you are	2017-11-14 09:54:21 +00:00
Travis Ralston	812c191939	Remove redundent call ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2017-11-13 12:44:21 -07:00
Richard van der Hoff	c741ba59c9	Merge pull request #2669 from matrix-org/rav/cache_urlpreview_failure ... Cache failures in url_preview handler	2017-11-13 18:36:24 +00:00
David Baker	45ab288e07	Print instead of logging ... because we had to wait until the logger was set up	2017-11-13 18:32:08 +00:00
Richard van der Hoff	8b33ac8f6c	Merge branch 'develop' into rav/invalid_request_utf8	2017-11-13 11:56:22 +00:00
Richard van der Hoff	6cfee09be9	Make __init__ consitstent across Store heirarchy ... Add db_conn parameters to the `__init__` methods of the *Store classes, so that they are all consistent, which makes the multiple inheritance work correctly (and so that we can later extract mixins which can be used in the slavedstores)	2017-11-13 10:46:07 +00:00
Erik Johnston	ab335edb02	Revert "move _state_group_cache to statestore" ... This reverts commit f5cf3638e9.	2017-11-13 10:05:33 +00:00
Erik Johnston	bfbf1e1f1a	Up cache size of get_global_account_data_by_type_for_user	2017-11-13 09:52:11 +00:00
Travis Ralston	2d314b771f	Add a route for determining who you are ... Useful for applications which may have an access token, but no idea as to who owns it. Signed-off-by: Travis Ralston <travpc@gmail.com>	2017-11-12 23:39:38 -07:00
Richard van der Hoff	5d15abb120	Bit more logging	2017-11-10 16:58:04 +00:00
Richard van der Hoff	46790f50cf	Cache failures in url_preview handler ... Reshuffle the caching logic in the url_preview handler so that failures are cached (and to generally simplify things and fix the logcontext leaks).	2017-11-10 16:50:50 +00:00
Richard van der Hoff	e508145c9b	Add some more comments appservice user registration ... Explain why we don't validate userids registered via app services	2017-11-10 12:39:45 +00:00
Richard van der Hoff	e0ebd1e4bd	Downcase userids for shared-secret registration	2017-11-10 12:39:05 +00:00
Richard van der Hoff	f90649eb2b	Fix 500 on invalid utf-8 in request ... If somebody sends us a request where the the body is invalid utf-8, we should return a 400 rather than a 500. (json.loads throws a UnicodeError in this situation) We might as well catch all Exceptions here: it seems very unlikely that we would get a request that *isn't caused by invalid json.	2017-11-10 09:15:39 +00:00
Richard van der Hoff	9b599bc18d	Downcase userid on registration ... Force username to lowercase before attempting to register https://github.com/matrix-org/synapse/issues/2660	2017-11-09 22:20:01 +00:00
Richard van der Hoff	9b803ccc98	Revert "Allow upper-case characters in mxids" ... This reverts commit b70b646903.	2017-11-09 21:57:24 +00:00
Richard van der Hoff	b70b646903	Allow upper-case characters in mxids ... Because we're never going to be able to fix this :'(	2017-11-09 19:36:13 +00:00
Erik Johnston	2dce6b15c3	Fix typo	2017-11-09 15:56:16 +00:00
Erik Johnston	4e2b2508af	Register group servlet	2017-11-09 15:49:42 +00:00
Erik Johnston	13735843c7	Namespace visibility options for groups	2017-11-09 15:27:18 +00:00
Richard van der Hoff	618c7b816a	Merge pull request #2656 from matrix-org/rav/fix_deactivate ... Fix 'NoneType' not iterable in /deactivate	2017-11-09 15:20:35 +00:00
Richard van der Hoff	889102315e	Fix 'NoneType' not iterable in /deactivate ... make sure we actually return a value from user_delete_access_tokens	2017-11-09 15:15:33 +00:00
David Baker	b2a788e902	Make the commented config have the default	2017-11-09 10:11:42 +00:00
Erik Johnston	82e4bfb53d	Add brackets	2017-11-09 10:06:42 +00:00
Erik Johnston	e8814410ef	Have an explicit API to update room config	2017-11-08 16:13:27 +00:00
Erik Johnston	94ff2cda73	Revert "Modify group room association API to allow modification of is_public"	2017-11-08 15:43:34 +00:00
Erik Johnston	d305987b40	Merge pull request #2631 from xyzz/fix_appservice_event_backlog ... Fix appservices being backlogged and not receiving new events due to a bug in notify_interested_services	2017-11-08 11:54:10 +00:00
Erik Johnston	167eb01d83	Merge pull request #2637 from spantaleev/avoid-noop-media-deletes ... Avoid no-op media deletes	2017-11-08 11:53:27 +00:00
David Baker	ad408beb66	better comments	2017-11-08 11:50:08 +00:00
David Baker	1b870937ae	Log if any of the old config flags are set	2017-11-08 11:46:24 +00:00
David Baker	2a98ba0ed3	Rename redact_content option to include_content ... The redact_content option never worked because it read the wrong config section. The PR introducing it (https://github.com/matrix-org/synapse/pull/2301) had feedback suggesting the name be changed to not re-use the term 'redact' but this wasn't incorporated. This reanmes the option to give it a less confusing name, and also means that people who've set the redact_content option won't suddenly see a behaviour change when upgrading synapse, but instead can set include_content if they want to. This PR also updates the wording of the config comment to clarify that this has no effect on event_id_only push. Includes https://github.com/matrix-org/synapse/pull/2422	2017-11-08 10:35:30 +00:00
Richard van der Hoff	02a9a93bde	Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res ... Fix bug in state group storage	2017-11-08 09:22:13 +00:00
Richard van der Hoff	e148438e97	s/items/iteritems/	2017-11-08 09:21:41 +00:00
Ilya Zhuravlev	d46386d57e	Remove useless assignment in notify_interested_services	2017-11-07 22:23:22 +03:00
Matthew Hodgson	228ccf1fe3	Merge pull request #2643 from matrix-org/matthew/user_dir_typos ... Fix various embarrassing typos around user_directory and add some doc.	2017-11-07 17:31:11 +00:00
Richard van der Hoff	780dbb378f	Update deltas when doing auth resolution ... Fixes a bug where the persisted state groups were different to those actually being used after auth resolution.	2017-11-07 16:43:00 +00:00
Richard van der Hoff	1ca4288135	factor out _update_context_for_auth_events ... This is duplicated, so let's factor it out before fixing it	2017-11-07 16:43:00 +00:00
Richard van der Hoff	f5cf3638e9	move _state_group_cache to statestore ... this is internal to statestore, so let's keep it there.	2017-11-07 16:43:00 +00:00
Erik Johnston	5ef5e14ecc	Merge pull request #2636 from farialima/me-master ... Fix for #2635: correctly update rooms avatar/display name when modified by admin	2017-11-07 13:49:27 +00:00
Erik Johnston	76c9af193c	Revert "Merge branch 'master' of github.com:matrix-org/synapse into develop" ... This reverts commit f9b255cd62, reversing changes made to 1bd654dabd.	2017-11-07 13:32:35 +00:00
Erik Johnston	f9b255cd62	Merge branch 'master' of github.com:matrix-org/synapse into develop	2017-11-07 13:31:03 +00:00
Luke Barnard	38b265cb51	Remember to pick is_admin out of the db	2017-11-07 11:24:04 +00:00
Luke Barnard	5561c09091	Return whether a user is an admin within a group	2017-11-07 11:18:45 +00:00
Matthew Hodgson	3db5ff69b2	Merge pull request #2576 from maximevaillancourt/exclude-noscript-url-preview ... Ignore <noscript> tags when generating URL preview descriptions	2017-11-07 11:09:22 +00:00
Matthew Hodgson	631fa4a1b7	create new indexes before dropping old ones to keep safetynet in place	2017-11-07 10:41:55 +00:00
Richard van der Hoff	bf993db11c	Logging and logcontext fixes for Limiter ... Add some logging to the Limiter in a similar spirit to the Linearizer, to help debug issues. Also fix a logcontext leak. Also refactor slightly to avoid throwing exceptions.	2017-11-07 00:48:57 +00:00
Matthew Hodgson	4ad883398f	s/users_in_pubic_room/users_in_public_rooms/g	2017-11-04 19:39:40 +00:00
Matthew Hodgson	d802e8ca6a	s/users_in_pubic_room/users_in_public_rooms/g	2017-11-04 19:38:13 +00:00
Matthew Hodgson	a100700630	fix copyright....	2017-11-04 19:35:49 +00:00
Matthew Hodgson	b6b075fd49	s/popualte/populate/	2017-11-04 19:35:33 +00:00
Matthew Hodgson	d1622e080f	s/intial/initial/	2017-11-04 19:35:14 +00:00
Slavi Pantaleev	805196fbeb	Avoid no-op media deletes ... If there are no media entries to delete, avoid creating transactions, prepared statements and unnecessary log entries. Signed-off-by: Slavi Pantaleev <slavi@devture.com>	2017-11-04 09:50:15 +02:00
Francois Granade	f103b91ffa	removed unused import flagged by flake8a	2017-11-03 18:45:49 +01:00
Francois Granade	fa4f337b49	Fix for issue 2635: correctly update rooms avatar/display name when modified by admin	2017-11-03 18:25:04 +01:00
Ilya Zhuravlev	45fbe4ff67	Fix appservices being backlogged and not receiving new events due to a bug in notify_interested_services	2017-11-02 22:49:43 +03:00
David Baker	f851bc8182	Merge pull request #2630 from matrix-org/luke/fix-rooms-in-group ... Make the get_rooms_in_group API more sane	2017-11-02 17:23:17 +00:00
David Baker	9e09a1800b	Merge pull request #2629 from matrix-org/rav/register_inhibit_login ... support inhibit_login in /register	2017-11-02 16:51:35 +00:00
Luke Barnard	a34c586a89	Make the get_rooms_in_group API more sane ... Return entries with is_public = True when they're public and is_public = False otherwise.	2017-11-02 16:42:30 +00:00
Richard van der Hoff	6c3a02072b	support inhibit_login in /register ... Allow things to pass inhibit_login when registering to ... inhibit logins.	2017-11-02 16:31:07 +00:00
David Baker	4b36897cd9	Merge remote-tracking branch 'origin/develop' into rav/module_api_hooks	2017-11-02 15:19:17 +00:00
David Baker	d4553818a0	Merge pull request #2627 from matrix-org/rav/custom_rest_endpoints ... Add a hook for custom rest endpoints	2017-11-02 15:18:37 +00:00
David Baker	6b6f03ae05	Merge pull request #2626 from matrix-org/rav/refactor_module_api ... Factor _AccountHandler proxy out to ModuleApi	2017-11-02 15:15:30 +00:00
Richard van der Hoff	6b60f7dca0	Add more hooks to ModuleApi ... add `get_user_by_req` and `invalidate_access_token`	2017-11-02 14:37:39 +00:00
Richard van der Hoff	fcdfc911ee	Add a hook for custom rest endpoints ... Let the user specify custom modules which can be used for implementing extra endpoints.	2017-11-02 14:36:55 +00:00
Richard van der Hoff	1189be43a2	Factor _AccountHandler proxy out to ModuleApi ... We're going to need to use this from places that aren't password auth, so let's move it to a proper class.	2017-11-02 14:36:11 +00:00
Richard van der Hoff	6650a07ede	Factor out _configure_named_resource ... This was a bit of a code vomit, so let's factor it out to preserve some sanity	2017-11-02 14:33:37 +00:00
David Baker	b19d9e2174	Merge pull request #2624 from matrix-org/rav/password_provider_notify_logout ... Notify auth providers on logout	2017-11-02 10:55:17 +00:00
David Baker	1f080a6c97	Merge pull request #2623 from matrix-org/rav/callbacks_for_auth_providers ... Allow password_auth_providers to return a callback	2017-11-02 10:49:03 +00:00
David Baker	04897c9dc1	Merge pull request #2622 from matrix-org/rav/db_access_for_auth_providers ... Let auth providers get to the database	2017-11-02 10:41:25 +00:00
Richard van der Hoff	979eed4362	Fix user-interactive password auth ... this got broken in the previous commit	2017-11-01 17:03:20 +00:00
Richard van der Hoff	bc8a5c0330	Notify auth providers on logout ... Provide a hook by which auth providers can be notified of logouts.	2017-11-01 16:51:51 +00:00
Richard van der Hoff	4c8f94ac94	Allow password_auth_providers to return a callback ... ... so that they have a way to record access tokens.	2017-11-01 16:51:03 +00:00
Richard van der Hoff	846a94fbc9	Merge pull request #2620 from matrix-org/rav/auth_non_password ... Let password auth providers handle arbitrary login types	2017-11-01 16:45:33 +00:00
Richard van der Hoff	3cd6b22c7b	Let password auth providers handle arbitrary login types ... Provide a hook where password auth providers can say they know about other login types, and get passed the relevant parameters	2017-11-01 16:43:57 +00:00
David Baker	c9b9ef575b	Merge pull request #2621 from matrix-org/rav/refactor_accesstoken_delete ... Move access token deletion into auth handler	2017-11-01 16:26:06 +00:00
Matthew Hodgson	275826f234	Merge pull request #2617 from matrix-org/matthew/auto-displayname ... automatically set default displayname on register	2017-11-01 16:21:16 +00:00
David Baker	4f0488b307	Merge remote-tracking branch 'origin/develop' into rav/refactor_accesstoken_delete	2017-11-01 16:20:19 +00:00
David Baker	e5e930aec3	Merge pull request #2615 from matrix-org/rav/break_auth_device_dep ... Break dependency of auth_handler on device_handler	2017-11-01 16:06:31 +00:00
David Baker	fbbacb284e	Merge pull request #2613 from matrix-org/rav/kill_refresh_tokens ... Remove the last vestiges of refresh_tokens	2017-11-01 15:57:35 +00:00
Matthew Hodgson	9f7a555b4e	switch to setting default displayname in the storage layer ... to avoid clobbering guest user displaynames on registration	2017-11-01 15:51:30 +00:00
Richard van der Hoff	dd13310fb8	Move access token deletion into auth handler ... Also move duplicated deactivation code into the auth handler. I want to add some hooks when we deactivate an access token, so let's bring it all in here so that there's somewhere to put it.	2017-11-01 15:46:22 +00:00
David Baker	0bb253f37b	Apparently this is python	2017-11-01 14:02:52 +00:00
David Baker	59e7e62c4b	Log login requests ... Carefully though, to avoid logging passwords	2017-11-01 13:58:01 +00:00
Matthew Hodgson	f8420d6279	automatically set default displayname on register ... to avoid leaking ugly MXIDs and cluttering up the timeline with displayname changes as well as membership joins for autojoin rooms (e.g. the status autojoin rooms), automatically set the displayname to match the localpart of the mxid upon registration.	2017-11-01 13:15:41 +00:00
Luke Barnard	99354b430e	Merge pull request #2612 from matrix-org/luke/groups-room-relationship-is-public ... Modify group room association API to allow modification of is_public	2017-11-01 11:08:36 +00:00
Richard van der Hoff	74c56f794c	Break dependency of auth_handler on device_handler ... I'm going to need to make the device_handler depend on the auth_handler, so I need to break this dependency to avoid a cycle. It turns out that the auth_handler was only using the device_handler in one place which was an edge case which we can more elegantly handle by throwing an error rather than fixing it up.	2017-11-01 10:27:06 +00:00
Luke Barnard	318a249c8b	Leave is_public as required argument of update_room_group_association	2017-11-01 09:36:01 +00:00
Luke Barnard	207fabbc6a	Update docs for updating room group association	2017-11-01 09:35:15 +00:00
Richard van der Hoff	356bcafc44	Remove the last vestiges of refresh_tokens	2017-10-31 20:35:58 +00:00
Richard van der Hoff	3e0aaad190	Let auth providers get to the database ... Somewhat open to abuse, but also somewhat unavoidable :/	2017-10-31 17:22:29 +00:00
Luke Barnard	13b3d7b4a0	Flake8	2017-10-31 17:20:11 +00:00
Luke Barnard	20fe347906	Modify group room association API to allow modification of is_public ... also includes renamings to make things more consistent.	2017-10-31 17:04:28 +00:00
Richard van der Hoff	1650eb5847	DB schema interface for password auth providers ... Provide an interface by which password auth providers can register db schema files to be run at startup	2017-10-31 14:01:53 +00:00
David Baker	c31a7c3ff6	Merge pull request #2609 from matrix-org/rav/refactor_login ... Refactor some logic from LoginRestServlet into AuthHandler	2017-10-31 13:51:36 +00:00
Richard van der Hoff	1b65ae00ac	Refactor some logic from LoginRestServlet into AuthHandler ... I'm going to need some more flexibility in handling login types in password auth providers, so as a first step, move some stuff from LoginRestServlet into AuthHandler. In particular, we pass everything other than SAML, JWT and token logins down to the AuthHandler, which now has responsibility for checking the login type and fishing the password out of the login dictionary, as well as qualifying the user_id if need be. Ideally SAML, JWT and token would go that way too, but there's no real need for it right now and I'm trying to minimise impact. This commit *should* be non-functional.	2017-10-31 10:48:41 +00:00
Richard van der Hoff	ffc574a6f9	Clean up backwards-compat hacks for ldap ... try to make the backwards-compat flows follow the same code paths as the modern impl. This commit should be non-functional.	2017-10-31 10:47:02 +00:00
Luke Barnard	9bc17fc5fb	Fix wording on group creation error	2017-10-30 15:17:23 +00:00
Matthew Hodgson	208a6647f1	fix typo	2017-10-29 20:54:20 +00:00