8176 Commits

Author	SHA1	Message	Date
Adrian Tschira	bfc2ade9b3	Make event properties raise AttributeError instead ... They raised KeyError before. I'm changing this because the code uses hasattr() to check for the presence of a key. This worked accidentally before, because hasattr() silences all exceptions in python 2. However, in python3, this isn't the case anymore. I had a look around to see if anything depended on this raising a KeyError and I couldn't find anything. Of course, I could have simply missed it. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-15 20:16:59 +02:00
Neil Johnson	154b44c249	Merge branch 'master' of https://github.com/matrix-org/synapse into develop	2018-04-13 17:07:54 +01:00
Matthew Hodgson	78a9698650	fix federation_domain_whitelist ... we were checking the wrong server_name on inbound requests	2018-04-13 15:47:43 +01:00
Matthew Hodgson	25b0ba30b1	revert last to PR properly	2018-04-13 15:46:37 +01:00
Matthew Hodgson	f8d46cad3c	correctly auth inbound federation_domain_whitelist reqs	2018-04-13 15:41:52 +01:00
Neil Johnson	d4b2e05852	Merge branch 'master' of https://github.com/matrix-org/synapse into develop	2018-04-13 12:16:27 +01:00
Neil Johnson	51d628d28d	Bump version and Change log	2018-04-13 12:08:19 +01:00
Richard van der Hoff	d3347ad485	Revert "Use sortedcontainers instead of blist" ... This reverts commit 9fbe70a7dc. It turns out that sortedcontainers.SortedDict is not an exact match for blist.sorteddict; in particular, `popitem()` removes things from the opposite end of the dict. This is trivial to fix, but I want to add some unit tests, and potentially some more thought about it, before we do so.	2018-04-13 11:16:43 +01:00
Richard van der Hoff	fac3f9e678	Bump canonicaljson to 1.1.3 ... 1.1.2 was a bit broken too :/	2018-04-13 10:21:38 +01:00
Richard van der Hoff	60f6014bb7	ResponseCache: fix handling of completed results ... Turns out that ObservableDeferred.observe doesn't return a deferred if the result is already completed. Fix handling and improve documentation.	2018-04-13 07:32:29 +01:00
Richard van der Hoff	119596ab8f	Update canonicaljson dependency ... 1.1.0 and 1.1.1 were broken, so we're updating this to help people make sure they don't end up on a broken version. Also, 1.1.0 is speedier...	2018-04-12 17:31:44 +01:00
Richard van der Hoff	b78395b7fe	Refactor ResponseCache usage ... Adds a `.wrap` method to ResponseCache which wraps up the boilerplate of a (get, set) pair, and then use it throughout the codebase. This will be largely non-functional, but does include the following functional changes: * federation_server.on_context_state_request: drops use of _server_linearizer which looked redundant and could cause incorrect cache misses by yielding between the get and the set. * RoomListHandler.get_remote_public_room_list(): fixes logcontext leaks * the wrap function includes some logging. I'm hoping this won't be too noisy on production.	2018-04-12 13:02:15 +01:00
Richard van der Hoff	d5c74b9f6c	Merge pull request #3092 from matrix-org/rav/response_cache_metrics ... Add metrics for ResponseCache	2018-04-12 12:59:36 +01:00
Erik Johnston	415aeefd89	Format docstring	2018-04-12 12:07:09 +01:00
Erik Johnston	19ceb4851f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/processed_event_lag	2018-04-12 11:36:07 +01:00
Richard van der Hoff	261124396e	Merge pull request #3059 from matrix-org/rav/doc_response_cache ... Document the behaviour of ResponseCache	2018-04-12 11:22:30 +01:00
Erik Johnston	23a7f9d7f4	Doc we raise on unknown event	2018-04-12 11:20:51 +01:00
Erik Johnston	d7bf3a68f0	s/list/tuple	2018-04-12 11:19:04 +01:00
Erik Johnston	f67e906e18	Set all metrics at the same time	2018-04-12 11:18:19 +01:00
Erik Johnston	971059a733	Merge pull request #3088 from matrix-org/erikj/as_parallel ... Send events to ASes concurrently	2018-04-12 10:42:36 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	92e34615c5	Track where event stream processing have gotten up to	2018-04-11 12:13:40 +01:00
Erik Johnston	ab825aa328	Add GaugeMetric	2018-04-11 12:13:40 +01:00
Richard van der Hoff	233699c42e	Merge pull request #2760 from Valodim/pypy ... Synapse on PyPy	2018-04-11 11:20:01 +01:00
Neil Johnson	781cd8c54f	bump version/changelog	2018-04-11 10:54:43 +01:00
Neil Johnson	9ef0b179e0	Merge commit '11d2609da70af797405241cdf7d9df19db5628f2' of https://github.com/matrix-org/synapse into release-v0.27.0	2018-04-11 10:51:59 +01:00
Erik Johnston	121591568b	Send events to ASes concurrently	2018-04-11 09:56:00 +01:00
Richard van der Hoff	b3384232a0	Add metrics for ResponseCache	2018-04-10 23:14:47 +01:00
Neil Johnson	d54cfbb7a8	fix typo	2018-04-10 17:38:16 +01:00
Erik Johnston	eaa2ebf20b	Merge pull request #3079 from matrix-org/erikj/limit_concurrent_sends ... Limit concurrent event sends for a room	2018-04-10 16:43:58 +01:00
Erik Johnston	9daf82278f	Merge pull request #3078 from matrix-org/erikj/federation_sender ... Send federation events concurrently	2018-04-10 16:43:48 +01:00
Adrian Tschira	a3f9ddbede	Open certificate files as bytes ... That's what pyOpenSSL expects on python3 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-10 17:36:29 +02:00
Adrian Tschira	7f8eebc8ee	Open config file in non-bytes mode ... Nothing written into it is encoded, so it makes little sense, but it does break in python3 the way it was before. The variable names were adjusted to be less misleading. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-10 17:32:40 +02:00
Neil Johnson	dd723267b2	Merge branch 'release-v0.27.0' of https://github.com/matrix-org/synapse into develop	2018-04-10 15:03:29 +01:00
Erik Johnston	a060dfa132	Use run_in_background instead	2018-04-10 14:25:11 +01:00
Erik Johnston	f8e8ec013b	Note why we're limiting concurrent event sends	2018-04-10 14:00:46 +01:00
Erik Johnston	1246d23710	Preserve log contexts correctly	2018-04-10 12:04:32 +01:00
Erik Johnston	d49cbf712f	Log event ID on exception	2018-04-10 12:03:41 +01:00
Erik Johnston	ce72d590ed	Merge pull request #3082 from matrix-org/erikj/urlencode_paths ... URL quote path segments over federation	2018-04-10 11:31:16 +01:00
Erik Johnston	11d2609da7	Ensure slashes are escaped	2018-04-10 11:24:40 +01:00
Erik Johnston	dab87b84a3	URL quote path segments over federation	2018-04-10 11:16:08 +01:00
Vincent Breitmoser	6d7f0f8dd3	Don't disable GC when running on PyPy ... PyPy's incminimark GC can't be triggered manually. From what I observed there are no obvious issues with just letting it run normally. And unlike CPython, it actually returns unused RAM to the system. Signed-off-by: Vincent Breitmoser <look@my.amazin.horse>	2018-04-10 11:35:34 +02:00
Vincent Breitmoser	f4284d943a	In DomainSpecificString, override __repr__ in addition to __str__ ... For some reason, string interpolation on a DomainSpecificString object like "%r" % (domainSpecificStringObj) fails under PyPy, because the default __repr__ implementation wants to iterate over the object. I'm not sure why that happens, but overriding __repr__ instead of __str__ fixes this problem, and is arguably the more appropriate thing to do anyways.	2018-04-10 11:35:29 +02:00
Richard van der Hoff	d1e56cfcd1	Fix pep8 error on psycopg2cffi hack	2018-04-10 11:35:29 +02:00
Vincent Breitmoser	89de934981	Use psycopg2cffi module instead of psycopg2 if running on pypy ... The psycopg2 package isn't available for PyPy. This commit adds a check if the runtime is PyPy, and if it is uses psycopg2cffi module in favor of psycopg2. This is almost a drop-in replacement, except for one place where an additional cast to string is required.	2018-04-10 11:29:52 +02:00
Vincent Breitmoser	9fbe70a7dc	Use sortedcontainers instead of blist ... This commit drop-in replaces blist with SortedContainers. They are written in pure python so work with pypy, but perform as good as native implementations, at least in a couple benchmarks: http://www.grantjenks.com/docs/sortedcontainers/performance.html	2018-04-10 11:29:51 +02:00
Richard van der Hoff	a3599dda97	Merge pull request #2996 from krombel/allow_auto_join_rooms ... move handling of auto_join_rooms to RegisterHandler	2018-04-10 01:11:00 +01:00
Richard van der Hoff	87478c5a60	Merge pull request #3061 from NotAFile/add-some-byte-strings ... Add b prefixes to some strings that are bytes in py3	2018-04-09 23:54:05 +01:00
Richard van der Hoff	c508b2f2f0	Merge pull request #3073 from NotAFile/use-six-reraise ... Replace old-style raise with six.reraise	2018-04-09 23:53:40 +01:00
Richard van der Hoff	37354b55c9	Merge pull request #2938 from dklug/develop ... Return 401 for invalid access_token on logout	2018-04-09 23:52:56 +01:00
Richard van der Hoff	0e9aa1d091	Merge pull request #3074 from NotAFile/fix-py3-prints ... use python3-compatible prints	2018-04-09 23:44:41 +01:00
Richard van der Hoff	8eaa141d8f	Merge pull request #3075 from NotAFile/six-type-checks ... Replace some type checks with six type checks	2018-04-09 23:40:44 +01:00
Richard van der Hoff	664adb4236	Merge pull request #3016 from silkeh/improve-service-lookups ... Improve handling of SRV records for federation connections	2018-04-09 23:40:06 +01:00
Neil Johnson	41e0611895	remove errant print	2018-04-09 18:44:20 +01:00
Neil Johnson	61b439c904	Fix msec to sec, again	2018-04-09 18:43:48 +01:00
Neil Johnson	87770300d5	Fix msec to sec	2018-04-09 18:38:59 +01:00
Neil Johnson	64bc2162ef	Fix psycopg2 interpolation	2018-04-09 17:50:36 +01:00
Richard van der Hoff	d2c6f4d626	Merge pull request #3080 from matrix-org/rav/fix_500_on_rejoin ... Return a 404 rather than a 500 on rejoining empty rooms	2018-04-09 17:32:36 +01:00
Neil Johnson	5232d3bfb1	version bump v0.27.3-rc2	2018-04-09 17:25:57 +01:00
Neil Johnson	5e785d4d5b	Merge branch 'develop' of https://github.com/matrix-org/synapse into release-v0.27.0	2018-04-09 17:21:34 +01:00
Erik Johnston	6e025a97b4	Handle all events in a room correctly	2018-04-09 16:02:48 +01:00
Neil Johnson	d9ae2bc826	bump version to release candidate	2018-04-09 16:00:31 +01:00
Neil Johnson	072fb59446	bump version	2018-04-09 13:49:25 +01:00
Richard van der Hoff	13decdbf96	Revert "Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics" ... We aren't ready to release this yet, so I'm reverting it for now. This reverts commit d1679a4ed7, reversing changes made to e089100c62.	2018-04-09 12:59:12 +01:00
Richard van der Hoff	f3ef60662f	Return a 404 rather than a 500 on rejoining empty rooms ... Filter ourselves out of the server list before checking for an empty remote host list, to fix 500 error Fixes #2141	2018-04-09 12:56:22 +01:00
Erik Johnston	e5082494eb	Limit concurrent event sends for a room	2018-04-09 12:07:39 +01:00
Erik Johnston	56b0589865	Use create_and_send_nonmember_event everywhere	2018-04-09 12:04:18 +01:00
Erik Johnston	11974f3787	Send federation events concurrently	2018-04-09 11:47:10 +01:00
Erik Johnston	145d14656b	Handle exceptions in get_hosts_for_room when sending events over federation	2018-04-09 11:47:01 +01:00
Adrian Tschira	e54c202b81	Replace some type checks with six type checks ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-07 01:02:32 +02:00
Adrian Tschira	b0500d3774	use python3-compatible prints	2018-04-06 23:35:27 +02:00
Adrian Tschira	4f40d058cc	Replace old-style raise with six.reraise ... The old style raise is invalid syntax in python3. As noted in the docs, this adds one more frame in the traceback, but I think this is acceptable: <ipython-input-7-bcc5cba3de3f> in <module>() 16 except: 17 pass ---> 18 six.reraise(*x) /usr/lib/python3.6/site-packages/six.py in reraise(tp, value, tb) 691 if value.__traceback__ is not tb: 692 raise value.with_traceback(tb) --> 693 raise value 694 finally: 695 value = None <ipython-input-7-bcc5cba3de3f> in <module>() 9 10 try: ---> 11 x() 12 except: 13 x = sys.exc_info() Also note that this uses six, which is not formally a dependency yet, but is included indirectly since most packages depend on it. Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-06 23:06:24 +02:00
Luke Barnard	135fc5b9cd	Merge pull request #3046 from matrix-org/dbkr/join_group ... Implement group join API	2018-04-06 16:24:32 +01:00
Luke Barnard	020a501354	de-lint, quote consistency	2018-04-06 16:02:06 +01:00
Luke Barnard	db2fd801f7	Explicitly grab individual columns from group object	2018-04-06 15:57:25 +01:00
Luke Barnard	7945435587	When exposing group state, return is_openly_joinable ... as opposed to join_policy, which is really only pertinent to the synapse implementation of the group server. By doing this we keep the group server concept extensible by allowing arbitrarily complex rules for deciding whether a group is openly joinable.	2018-04-06 15:43:27 +01:00
Luke Barnard	6bd1b7053e	By default, join policy is "invite"	2018-04-06 15:43:27 +01:00
Luke Barnard	b4478e586f	add_user -> _add_user	2018-04-06 15:43:27 +01:00
Luke Barnard	112c2253e2	pep8	2018-04-06 15:43:27 +01:00
Luke Barnard	6850f8aea3	Get group_info from existing call to check_group_is_ours	2018-04-06 15:43:27 +01:00
Luke Barnard	cd087a265d	Don't use redundant inlineCallbacks	2018-04-06 15:43:27 +01:00
Luke Barnard	87c864b698	join_rule -> join_policy	2018-04-06 15:43:27 +01:00
Luke Barnard	ae85c7804e	is_joinable -> join_rule	2018-04-06 15:43:27 +01:00
Luke Barnard	f8d1917fce	Fix federation client set_group_joinable typo	2018-04-06 15:43:27 +01:00
Luke Barnard	6eb3aa94b6	Factor out add_user from accept_invite and join_group	2018-04-06 15:43:27 +01:00
David Baker	edb45aae38	pep8	2018-04-06 15:43:27 +01:00
David Baker	b370fe61c0	Implement group join API	2018-04-06 15:43:27 +01:00
Erik Johnston	7b824f1475	Add response size metrics	2018-04-06 13:20:11 +01:00
Krombel	1d71f484d4	use PUT instead of POST for federating groups/m.join_policy	2018-04-06 12:54:09 +02:00
Richard van der Hoff	15e8ed874f	more verbosity in synctl	2018-04-06 09:28:36 +01:00
Richard van der Hoff	c7f0969731	Merge pull request #2986 from jplatte/join_reponse_room_id ... Add room_id to the response of `rooms/{roomId}/join`	2018-04-05 17:29:06 +01:00
Richard van der Hoff	3449da3bc7	Merge pull request #3068 from matrix-org/rav/fix_cache_invalidation ... Improve database cache performance	2018-04-05 17:21:44 +01:00
Richard van der Hoff	d1679a4ed7	Merge pull request #3066 from matrix-org/rav/remove_redundant_metrics ... Remove redundant metrics which were deprecated in 0.27.0.	2018-04-05 17:21:18 +01:00
Richard van der Hoff	01afc563c3	Fix overzealous cache invalidation ... Fixes an issue where a cache invalidation would invalidate *all* pending entries, rather than just the entry that we intended to invalidate.	2018-04-05 16:24:04 +01:00
Luke Barnard	e089100c62	Merge pull request #3045 from matrix-org/dbkr/group_joinable ... Add joinability for groups	2018-04-05 15:57:49 +01:00
Neil Johnson	68b0ee4e8d	Merge pull request #3041 from matrix-org/r30_stats ... R30 stats	2018-04-05 15:37:37 +01:00
Richard van der Hoff	22284a6f65	Merge pull request #3060 from matrix-org/rav/kill_event_content ... Remove uses of events.content	2018-04-05 15:02:17 +01:00
Luke Barnard	917380e89d	NON NULL -> NOT NULL	2018-04-05 14:32:12 +01:00
Luke Barnard	104c0bc1d5	Use "/settings/" (plural)	2018-04-05 14:07:16 +01:00
Luke Barnard	700e5e7198	Use DEFAULT join_policy of "invite" in db	2018-04-05 14:01:17 +01:00
Luke Barnard	b214a04ffc	Document set_group_join_policy	2018-04-05 13:29:16 +01:00
Neil Johnson	0e5f479fc0	Review comments ... Use iteritems over item to loop over dict formatting	2018-04-05 12:16:46 +01:00
Richard van der Hoff	518f6de088	Remove redundant metrics which were deprecated in 0.27.0.	2018-04-04 19:46:28 +01:00
Jan Christian Grünhage	e4570c53dd	phone home cache size configurations	2018-04-04 16:46:58 +01:00
Adrian Tschira	6168351877	Add b prefixes to some strings that are bytes in py3 ... This has no effect on python2 Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-04-04 13:48:51 +02:00
Richard van der Hoff	f92963f5db	Revert "improve mxid check performance"	2018-04-04 12:08:29 +01:00
Silke	72251d1b97	Remove address resolution of hosts in SRV records ... Signed-off-by: Silke Hofstra <silke@slxh.eu>	2018-04-04 12:26:50 +02:00
Richard van der Hoff	725a72ec5a	Merge pull request #3000 from NotAFile/change-except-style ... Replace old style error catching with 'as' keyword	2018-04-04 10:45:22 +01:00
Richard van der Hoff	a89f9f830c	Merge pull request #3044 from matrix-org/michaelk/performance_stats ... Add basic performance statistics to phone home	2018-04-04 10:37:25 +01:00
Richard van der Hoff	39ce38b024	Merge pull request #3053 from NotAFile/speedup-mxid-check ... improve mxid check performance	2018-04-04 10:19:52 +01:00
Richard van der Hoff	a9a74101a4	Document the behaviour of ResponseCache ... it looks like everything that uses ResponseCache expects to have to `make_deferred_yieldable` its results. It's debatable whether that is the best approach, but let's document it for now to avoid further confusion.	2018-04-04 09:06:22 +01:00
Luke Barnard	eb8d8d6f57	Use join_policy API instead of joinable ... The API is now under /groups/$group_id/setting/m.join_policy and expects a JSON blob of the shape ```json { "m.join_policy": { "type": "invite" } } ``` where "invite" could alternatively be "open".	2018-04-03 16:16:40 +01:00
Richard van der Hoff	8da39ad98f	Merge pull request #3049 from matrix-org/rav/use_staticjson ... Use static JSONEncoders	2018-04-03 15:18:32 +01:00
Richard van der Hoff	3ee4ad09eb	Fix json encoding bug in replication ... json encoders have an encode method, not a dumps method.	2018-04-03 15:09:48 +01:00
Adrian Tschira	11597ddea5	improve mxid check performance ~4x ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-31 02:00:22 +02:00
Richard van der Hoff	2fe3f848b9	Remove uses of events.content	2018-03-29 23:17:12 +01:00
Richard van der Hoff	05630758f2	Use static JSONEncoders ... using json.dumps with custom options requires us to create a new JSONEncoder on each call. It's more efficient to create one upfront and reuse it.	2018-03-29 23:13:33 +01:00
Richard van der Hoff	fcfe7f6ad3	Use simplejson throughout ... Let's use simplejson rather than json, for consistency.	2018-03-29 22:45:52 +01:00
Neil Johnson	b4e37c6f50	pep8	2018-03-29 17:27:39 +01:00
Neil Johnson	9ee44a372d	Remove need for sqlite specific query	2018-03-29 16:45:34 +01:00
Erik Johnston	88cc9cc69e	Merge pull request #3043 from matrix-org/erikj/measure_state_group_creation ... Measure time it takes to calculate state group ID	2018-03-28 17:40:14 +01:00
Neil Johnson	dc7c020b33	fix pep8 errors	2018-03-28 17:25:15 +01:00
David Baker	c5de6987c2	This should probably be a PUT	2018-03-28 16:44:11 +01:00
Neil Johnson	241e4e8687	remove twisted deferral cruft	2018-03-28 16:25:53 +01:00
David Baker	929b34963d	OK, smallint it is then	2018-03-28 14:53:55 +01:00
Richard van der Hoff	9a0db062af	Merge pull request #3034 from matrix-org/rav/fix_key_claim_errors ... Fix error when claiming e2e keys from offline servers	2018-03-28 14:50:38 +01:00
David Baker	a838444a70	Grr. Copy the definition from is_admin	2018-03-28 14:50:30 +01:00
Neil Johnson	4262aba17b	bump schema version	2018-03-28 14:40:03 +01:00
Neil Johnson	86932be2cb	Support multi client R30 for psql	2018-03-28 14:36:53 +01:00
David Baker	32260baa41	pep8	2018-03-28 14:29:42 +01:00
Michael Kaye	33f6195d9a	Handle review comments	2018-03-28 14:25:25 +01:00
David Baker	a164270833	Make column definition that works on both dbs	2018-03-28 14:23:00 +01:00
David Baker	352e1ff9ed	Add schema delta file	2018-03-28 14:07:57 +01:00
David Baker	79452edeee	Add joinability for groups ... Adds API to set the 'joinable' flag, and corresponding flag in the table.	2018-03-28 14:03:37 +01:00
Krombel	6152e253d8	Merge branch 'develop' of into allow_auto_join_rooms	2018-03-28 14:45:28 +02:00
Neil Johnson	792d340572	rename stat to future proof	2018-03-28 12:25:02 +01:00
Michael Kaye	4ceaa7433a	As daemonizing will make a new process, defer call to init.	2018-03-28 12:19:01 +01:00
Neil Johnson	788e69098c	Add user_ips last seen index	2018-03-28 12:03:13 +01:00
Neil Johnson	0f890f477e	No need to cast in count_daily_users	2018-03-28 11:49:57 +01:00
Neil Johnson	545001b9e4	Fix search_user_dir multiple sqlite versions do different things	2018-03-28 11:19:45 +01:00
Erik Johnston	01ccc9e6f2	Measure time it takes to calculate state group ID	2018-03-28 11:03:52 +01:00
Neil Johnson	a32d2548d9	query and call for r30 stats	2018-03-28 10:39:13 +01:00
Neil Johnson	9187e0762f	count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home	2018-03-28 10:02:32 +01:00
Erik Johnston	f879127aaa	Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id ... Linearize calls to _generate_user_id	2018-03-28 10:00:31 +01:00
Erik Johnston	e6d87c93f3	Merge pull request #3030 from matrix-org/erikj/no_ujson ... Remove last usage of ujson	2018-03-28 10:00:06 +01:00
Erik Johnston	004cc8a328	Merge pull request #3033 from matrix-org/erikj/calculate_state_metrics ... Add counter metrics for calculating state delta	2018-03-28 09:59:42 +01:00
Michael Kaye	ef520d8d0e	Include coarse CPU and Memory use in stats callbacks. ... This requires the psutil module, and is still opt-in based on the report_stats config option.	2018-03-27 17:56:03 +01:00
Richard van der Hoff	a134c572a6	Stringify exceptions for keys/{query,claim} ... Make sure we stringify any exceptions we return from keys/query and keys/claim, to avoid a 'not JSON serializable' error later Fixes #3010	2018-03-27 17:15:06 +01:00
Richard van der Hoff	c2a5cf2fe3	factor out exception handling for keys/claim and keys/query ... this stuff is badly c&p'ed	2018-03-27 17:11:23 +01:00
Erik Johnston	800cfd5774	Comment	2018-03-27 13:30:39 +01:00
Erik Johnston	152c2ac19e	Fix indent	2018-03-27 13:13:46 +01:00
Erik Johnston	e70287cff3	Comment	2018-03-27 13:13:38 +01:00
Erik Johnston	03a26e28d9	Merge pull request #3017 from matrix-org/erikj/add_cache_control_headers ... Add Cache-Control headers to all JSON APIs	2018-03-27 13:10:38 +01:00
Erik Johnston	3e0c0660b3	Also do check inside linearizer	2018-03-27 13:01:34 +01:00
Erik Johnston	3f49e131d9	Add counter metrics for calculating state delta ... This will allow us to measure how often we calculate state deltas in event persistence that we would have been able to calculate at the same time we calculated the state for the event.	2018-03-27 10:57:35 +01:00
Erik Johnston	9b8c0fb162	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-26 21:41:55 +01:00
Erik Johnston	a9d7d98d3f	Bum version and changelog	2018-03-26 16:36:53 +01:00
Erik Johnston	bdbb1eec65	Merge branch 'erikj/simplejson_replication' of github.com:matrix-org/synapse into release-v0.27.0	2018-03-26 16:35:55 +01:00
Erik Johnston	9187862002	Bump version and changelog	2018-03-26 16:20:24 +01:00
Neil Johnson	51406dab96	version bump	2018-03-26 14:48:19 +01:00
Erik Johnston	fecb45e0c3	Remove last usage of ujson	2018-03-26 13:32:29 +01:00
Erik Johnston	44cd6e1358	PEP8	2018-03-26 12:06:48 +01:00
Erik Johnston	8d6dc106d1	Don't use _cursor_to_dict in find_next_generated_user_id_localpart	2018-03-26 12:02:44 +01:00
Erik Johnston	a052aa42e7	Linearize calls to _generate_user_id	2018-03-26 12:02:20 +01:00
Matthew Hodgson	8cbbfaefc1	404 correctly on missing paths via NoResource ... fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029	2018-03-23 10:32:50 +00:00
Erik Johnston	84b5cc69f5	Merge pull request #3006 from matrix-org/erikj/state_iter ... Use .iter* to avoid copies in StateHandler	2018-03-22 11:51:13 +00:00
Erik Johnston	fde8e8f09f	Fix s/iteriterms/itervalues	2018-03-22 11:42:16 +00:00
Erik Johnston	eb9fc021e3	Merge branch 'release-v0.27.0' of github.com:matrix-org/synapse into develop	2018-03-22 10:19:53 +00:00
Erik Johnston	1c41b05c8c	Add Cache-Control headers to all JSON APIs ... It is especially important that sync requests don't get cached, as if a sync returns the same token given then the client will call sync with the same parameters again. If the previous response was cached it will get reused, resulting in the client tight looping making the same request and never making any progress. In general, clients will expect to get up to date data when requesting APIs, and so its safer to do a blanket no cache policy than only whitelisting APIs that we know will break things if they get cached.	2018-03-21 17:46:26 +00:00
Neil Johnson	e66fbcbb02	fix merge conflicts	2018-03-20 14:25:31 +00:00
Erik Johnston	9aa5a0af51	Explicitly use simplejson	2018-03-20 09:58:13 +00:00
Erik Johnston	610accbb7f	Fix replication after switch to simplejson ... Turns out that simplejson serialises namedtuple's as dictionaries rather than tuples by default.	2018-03-19 16:12:48 +00:00
Neil Johnson	c384705ee8	Update __init__.py ... bump version	2018-03-19 15:11:58 +00:00
Erik Johnston	3f961e638a	Merge pull request #3005 from matrix-org/erikj/fix_cache_size ... Fix bug where state cache used lots of memory	2018-03-19 11:44:26 +00:00
Erik Johnston	fa72803490	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-03-19 11:41:01 +00:00
Erik Johnston	9a0d783c11	Add comments	2018-03-19 11:35:53 +00:00
Matthew Hodgson	38f952b9bc	spell out not to massively increase bcrypt rounds	2018-03-19 09:27:36 +00:00
Erik Johnston	a8ce159be4	Replace some ujson with simplejson to make it work	2018-03-16 00:27:09 +00:00
Erik Johnston	5b631ff41a	Remove wrong comment	2018-03-16 00:07:08 +00:00
Erik Johnston	ba48755d56	Bump version and changelog	2018-03-15 23:57:26 +00:00
Erik Johnston	926ba76e23	Replace ujson with simplejson	2018-03-15 23:43:31 +00:00
Richard van der Hoff	5a6e54264d	Make 'unexpected logging context' into warnings ... I think we've now fixed enough of these that the rest can be logged at warning.	2018-03-15 18:40:38 +00:00
Erik Johnston	9cf519769b	Use .iter* to avoid copies in StateHandler	2018-03-15 17:50:26 +00:00
Erik Johnston	7c7706f42b	Fix bug where state cache used lots of memory ... The state cache bases its size on the sum of the size of entries. The size of the entry is calculated once on insertion, so it is important that the size of entries does not change. The DictionaryCache modified the entries size, which caused the state cache to incorrectly think it was smaller than it actually was.	2018-03-15 15:46:54 +00:00
NotAFile	2cc9f76bc3	replace old style error catching with 'as' keyword ... This is both easier to read and compatible with python3 (not that that matters) Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-03-15 16:11:17 +01:00
Erik Johnston	ddb00efc1d	Bump version number	2018-03-15 14:41:30 +00:00
Krombel	91ea0202e6	move handling of auto_join_rooms to RegisterHandler ... Currently the handling of auto_join_rooms only works when a user registers itself via public register api. Registrations via registration_shared_secret and ModuleApi do not work This auto_joins the users in the registration handler which enables the auto join feature for all 3 registration paths. This is related to issue #2725 Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-14 16:45:37 +01:00
Erik Johnston	4f28018c83	Register membership/state servlets in event_creator	2018-03-14 14:30:06 +00:00
Erik Johnston	57db62e554	Merge pull request #2992 from matrix-org/erikj/implement_member_workre ... Implement RoomMemberWorkerHandler	2018-03-14 14:29:33 +00:00
Erik Johnston	0011ede3b0	Fix imports	2018-03-14 14:19:23 +00:00
Erik Johnston	62ad701326	s/join/joined/ in notify_user_membership_change	2018-03-14 14:17:43 +00:00
Erik Johnston	3f0f06cb31	Split RoomMemberWorkerHandler to separate file	2018-03-14 11:41:45 +00:00
Erik Johnston	3e839e0548	Merge pull request #2989 from matrix-org/erikj/profile_cache_master ... Only update remote profile cache on master	2018-03-14 09:42:27 +00:00
Erik Johnston	ebd0127999	Merge pull request #2988 from matrix-org/erikj/split_profile_store ... Split up ProfileStore	2018-03-14 09:41:06 +00:00
Erik Johnston	cfe75a9fb6	Merge pull request #2991 from matrix-org/erikj/fixup_rm ... _remote_join and co take a requester	2018-03-14 09:40:57 +00:00
Erik Johnston	f51565e023	Merge pull request #2993 from matrix-org/erikj/is_blocked ... Add is_blocked to worker store	2018-03-14 09:39:18 +00:00
Matthew Hodgson	d144ed6ffb	fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990) ... Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.	2018-03-13 22:36:04 +00:00
Erik Johnston	a08726fc42	Add is_blocked to worker store	2018-03-13 18:28:44 +00:00
Erik Johnston	b27320b550	Implement RoomMemberWorkerHandler	2018-03-13 18:26:00 +00:00
Erik Johnston	350331d466	_remote_join and co take a requester	2018-03-13 17:50:39 +00:00
Erik Johnston	1a69c6d590	Merge pull request #2987 from matrix-org/erikj/split_room_member_handler ... Split RoomMemberHandler into base and master class	2018-03-13 17:40:00 +00:00
Erik Johnston	df8ff682a7	Only update remote profile cache on master	2018-03-13 17:38:21 +00:00
Erik Johnston	3518d0ea8f	Split up ProfileStore	2018-03-13 17:36:50 +00:00
Erik Johnston	d45a114824	Raise, don't return, exception	2018-03-13 17:24:34 +00:00
Erik Johnston	6dbebef141	Add missing param to docstrings	2018-03-13 17:15:32 +00:00
Erik Johnston	16adb11cc0	Correct import order	2018-03-13 16:57:07 +00:00
Erik Johnston	82f16faa78	Move user_*_room distributor stuff to master class ... I added yields when calling user_left_room, but they shouldn't matter on the master process as they always return None anyway.	2018-03-13 16:38:15 +00:00
Erik Johnston	b78717b87b	Split RoomMemberHandler into base and master class ... The intention here is to split the class into the bits that can be done on workers and the bits that have to be done on the master. In future there will also be a class that can be run on the worker, which will delegate work to the master when necessary.	2018-03-13 16:37:41 +00:00
Erik Johnston	95cb401ae0	Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer ... Remove ReplicationLayer and user Client/Server directly	2018-03-13 15:45:08 +00:00
Erik Johnston	5d8476d8ff	Merge pull request #2981 from matrix-org/erikj/factor_remote_leave ... Factor out _remote_reject_invite in RoomMember	2018-03-13 15:44:56 +00:00
Jonas Platte	47ce527f45	Add room_id to the response of rooms/{roomId}/join ... Fixes #2349	2018-03-13 14:48:12 +01:00
Erik Johnston	56e709857c	Merge pull request #2979 from matrix-org/erikj/no_handlers ... Don't build handlers on workers unnecessarily	2018-03-13 13:46:38 +00:00
Erik Johnston	cb9f8e527c	s/replication_client/federation_client/	2018-03-13 13:26:52 +00:00
Erik Johnston	cea462e285	s/replication_server/federation_server	2018-03-13 13:22:21 +00:00
Erik Johnston	bf8e97bd3c	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave	2018-03-13 13:17:08 +00:00
Erik Johnston	ea3442c15c	Add docstring	2018-03-13 13:16:21 +00:00
Erik Johnston	16469a4f15	Merge pull request #2980 from matrix-org/erikj/rm_priv ... Make RoomMemberHandler functions private that can be	2018-03-13 13:11:11 +00:00
Erik Johnston	c82111a55f	Merge pull request #2982 from matrix-org/erikj/fix_extra_users ... extra_users is actually a list of UserIDs	2018-03-13 13:11:04 +00:00
Erik Johnston	da87791975	Merge pull request #2983 from matrix-org/erikj/rename_register_3pid ... Refactor get_or_register_3pid_guest	2018-03-13 13:10:52 +00:00
Erik Johnston	99e9b4f26c	Merge pull request #2984 from matrix-org/erikj/fix_rest_regeix ... RoomMembershipRestServlet doesn't handle /forget	2018-03-13 13:10:44 +00:00
Erik Johnston	f5160d4a3e	RoomMembershipRestServlet doesn't handle /forget ... Due to the order we register the REST handlers `/forget` was handled by the correct handler.	2018-03-13 12:12:55 +00:00
Erik Johnston	8b3573a8b2	Refactor get_or_register_3pid_guest	2018-03-13 12:08:58 +00:00
Richard van der Hoff	299fd740c7	Merge pull request #2975 from matrix-org/rav/measure_persist_events ... Add Measure block for persist_events	2018-03-13 12:06:25 +00:00
Erik Johnston	9a2d9b4789	Merge pull request #2977 from matrix-org/erikj/replication_move_props ... Move property setting from ReplicationLayer to base classes	2018-03-13 11:45:25 +00:00
Erik Johnston	f43b6d6d9b	Fix docstring types	2018-03-13 11:29:35 +00:00
Erik Johnston	0f942f68c1	Factor out _remote_reject_invite in RoomMember	2018-03-13 11:22:45 +00:00
Erik Johnston	d0fcc48f9d	extra_users is actually a list of UserIDs	2018-03-13 11:20:06 +00:00
Erik Johnston	31becf4ac3	Make functions private that can be	2018-03-13 11:15:16 +00:00
Erik Johnston	d023ecb810	Don't build handlers on workers unnecessarily	2018-03-13 11:08:10 +00:00
Erik Johnston	ea7b3c4b1b	Remove unused ReplicationLayer	2018-03-13 11:00:04 +00:00
Erik Johnston	265b993b8a	Split replication layer into two	2018-03-13 10:55:47 +00:00
Erik Johnston	e05bf34117	Move property setting from ReplicationLayer to FederationBase	2018-03-13 10:51:30 +00:00
Erik Johnston	c3f79c9da5	Split out edu/query registration to a separate class	2018-03-13 10:24:27 +00:00
Richard van der Hoff	889a2a853a	Add Measure block for persist_events ... This seems like a useful thing to measure.	2018-03-13 10:01:42 +00:00
Richard van der Hoff	e48c7aac4d	Add transactional API to history purge ... Make the purge request return quickly, and allow scripts to poll for updates.	2018-03-12 16:22:55 +00:00
Richard van der Hoff	1708412f56	Return an error when doing two purges on a room ... Queuing up purges doesn't sound like a good thing.	2018-03-12 16:22:54 +00:00
Richard van der Hoff	b984dd0b73	Merge pull request #2961 from matrix-org/rav/run_in_background ... Factor run_in_background out from preserve_fn	2018-03-12 16:19:13 +00:00
Richard van der Hoff	ba1d08bc4b	Merge pull request #2965 from matrix-org/rav/request_logging ... Add a metric which increments when a request is received	2018-03-12 09:45:33 +00:00
Richard van der Hoff	58dd148c4f	Add some docstrings to help figure this out	2018-03-09 18:05:41 +00:00
Richard van der Hoff	88541f9009	Add a metric which increments when a request is received ... It's useful to know when there are peaks in incoming requests - which isn't quite the same as there being peaks in outgoing responses, due to the time taken to handle requests.	2018-03-09 16:30:26 +00:00
Richard van der Hoff	dbe80a286b	refactor JsonResource ... rephrase the OPTIONS and unrecognised request handling so that they look similar to the common flow.	2018-03-09 16:22:16 +00:00
Richard van der Hoff	20f40348d4	Factor run_in_background out from preserve_fn ... It annoys me that we create temporary function objects when there's really no need for it. Let's factor the gubbins out of preserve_fn and start using it.	2018-03-08 11:50:11 +00:00
Erik Johnston	a56d54dcb7	Fix up log message	2018-03-07 11:55:31 +00:00
Erik Johnston	02a1296ad6	Fix typo	2018-03-07 11:55:31 +00:00
Erik Johnston	8cb44da4aa	Fix race in sync when joining room ... The race happens when the user joins a room at the same time as doing a sync. We fetch the current token and then get the rooms the user is in. If the join happens after the current token, but before we get the rooms we end up sending down a partial room entry in the sync. This is fixed by looking at the stream ordering of the membership returned by get_rooms_for_user, and handling the case when that stream ordering is after the current token.	2018-03-07 11:55:31 +00:00
Richard van der Hoff	8ffaacbee3	Merge pull request #2949 from krombel/use_bcrypt_checkpw ... use bcrypt.checkpw	2018-03-06 11:56:06 +00:00
Richard van der Hoff	b2932107bb	Merge pull request #2946 from matrix-org/rav/timestamp_to_purge ... Implement purge_history by timestamp	2018-03-06 11:20:23 +00:00
Erik Johnston	7aed50a038	Merge pull request #2948 from matrix-org/erikj/kill_as_sync ... Remove ability for AS users to call /events and /sync	2018-03-06 11:10:09 +00:00
Erik Johnston	b6c4b851f1	Merge pull request #2947 from matrix-org/erikj/split_directory_store ... Split Directory store	2018-03-05 18:17:32 +00:00
Krombel	ed9b5eced4	use bcrypt.checkpw ... in bcrypt 3.1.0 checkpw got introduced (already 2 years ago) This makes use of that with enhancements which might get introduced by that Signed-Off-by: Matthias Kesler <krombel@krombel.de>	2018-03-05 18:02:59 +01:00
Erik Johnston	d4ffe61d4f	Remove ability for AS users to call /events and /sync ... This functionality has been deprecated for a while as well as being broken for a while. Instead of fixing it lets just remove it entirely. See: https://github.com/matrix-org/matrix-doc/issues/1144	2018-03-05 15:44:46 +00:00
Erik Johnston	69ce365b79	Fix cache invalidation on deletion	2018-03-05 15:29:03 +00:00
Erik Johnston	2e223163ff	Split Directory store	2018-03-05 15:11:30 +00:00
Richard van der Hoff	f8bfcd7e0d	Provide a means to pass a timestamp to purge_history	2018-03-05 14:37:23 +00:00
Richard van der Hoff	d032785aa7	Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts ... Test and fix find_first_stream_ordering_after_ts	2018-03-05 12:26:14 +00:00
Richard van der Hoff	c818fcab11	Test and fix find_first_stream_ordering_after_ts ... It seemed to suffer from a bunch of off-by-one errors.	2018-03-05 12:04:02 +00:00
Richard van der Hoff	06a14876e5	Add find_first_stream_ordering_after_ts ... Expose this as a public function which can be called outside a txn	2018-03-05 11:53:39 +00:00
Erik Johnston	42174946f8	Merge pull request #2934 from matrix-org/erikj/cache_fix ... Fix bug with delayed cache invalidation stream	2018-03-05 11:33:17 +00:00
Erik Johnston	f394f5574d	Merge pull request #2929 from matrix-org/erikj/split_regististration_store ... Split registration store	2018-03-05 11:33:07 +00:00
dklug	af7ed8e1ef	Return 401 for invalid access_token on logout ... Signed-off-by: Duncan Klug <dklug@ucmerced.edu>	2018-03-02 22:01:27 -08:00
Erik Johnston	efb79820b4	Fix bug with delayed cache invalidation stream ... We poked the notifier before updated the current token for the cache invalidation stream. This mean that sometimes the update wouldn't be sent until the next time a cache was invalidated.	2018-03-02 14:45:15 +00:00
Erik Johnston	fafa3e7114	Split registration store	2018-03-02 13:48:27 +00:00
Erik Johnston	d960d23830	Add missing yield during 3pid signature checks	2018-03-02 11:03:18 +00:00
Erik Johnston	1a6c7cdf54	Merge pull request #2928 from matrix-org/erikj/read_marker_caches ... Fix typo in getting replication account data processing	2018-03-01 17:56:14 +00:00
Erik Johnston	89b7232ff8	Fix typo in getting replication account data processing	2018-03-01 17:50:30 +00:00
Erik Johnston	1773df0632	Merge pull request #2925 from matrix-org/erikj/split_sig_fed ... Split out SignatureStore and EventFederationStore	2018-03-01 17:32:58 +00:00
Erik Johnston	65cf454fd1	Remove unused DataStore	2018-03-01 17:27:53 +00:00
Erik Johnston	9e08a93a7b	Merge pull request #2927 from matrix-org/erikj/read_marker_caches ... Improve caching for read_marker API	2018-03-01 17:12:34 +00:00
Erik Johnston	4b44f05f19	Fewer lies are better	2018-03-01 17:08:17 +00:00
Erik Johnston	a83c514d1f	Improve caching for read_marker API ... We add a new storage function to get a paritcular type of room account data. This allows us to prefill the cache when updating that acount data.	2018-03-01 17:08:17 +00:00
Erik Johnston	33bebb63f3	Add some caches to help read marker API	2018-03-01 17:08:17 +00:00
Erik Johnston	483e8104db	Merge pull request #2926 from matrix-org/erikj/member_handler_move ... Move RoomMemberHandler out of Handlers	2018-03-01 17:01:25 +00:00
Erik Johnston	2ad4d5b5bb	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed	2018-03-01 16:59:39 +00:00
Erik Johnston	529c026ac1	Move back to hs.is_mine	2018-03-01 16:49:12 +00:00
Erik Johnston	7c371834cc	Stub out broken function only used for cache	2018-03-01 16:44:13 +00:00
Erik Johnston	64346be26d	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store	2018-03-01 16:26:42 +00:00
Erik Johnston	22518e2833	Merge pull request #2923 from matrix-org/erikj/stream_ago_worker ... Calculate stream_ordering_month_ago correctly on workers	2018-03-01 16:23:54 +00:00
Erik Johnston	884b26ae41	Remove unused variables	2018-03-01 16:23:48 +00:00
Erik Johnston	1b2af11650	Document abstract class and method better	2018-03-01 16:20:57 +00:00
Erik Johnston	872ff95ed4	Default stream_ordering_*_ago to None	2018-03-01 16:00:05 +00:00
Erik Johnston	22004b524e	Fix comment typo	2018-03-01 15:59:40 +00:00
Erik Johnston	4bc4236faf	Merge pull request #2922 from matrix-org/erikj/split_room_store ... Split up RoomStore	2018-03-01 15:55:01 +00:00
Richard van der Hoff	2324124a72	Merge pull request #2921 from matrix-org/rav/unyielding_make_deferred_yieldable ... Rewrite make_deferred_yieldable avoiding inlineCallbacks	2018-03-01 15:39:10 +00:00
Erik Johnston	f793bc3877	Split out stream store	2018-03-01 15:13:08 +00:00
Erik Johnston	784f036306	Move RoomMemberHandler out of Handlers	2018-03-01 14:36:50 +00:00
Erik Johnston	6411f725be	Calculate stream_ordering_month_ago correctly on workers	2018-03-01 14:20:53 +00:00
Erik Johnston	a9a2d66cdd	Split out SignatureStore and EventFederationStore	2018-03-01 14:17:53 +00:00
Erik Johnston	0c8ba5dd1c	Split up RoomStore	2018-03-01 14:01:19 +00:00
Richard van der Hoff	3a75de923b	Rewrite make_deferred_yieldable avoiding inlineCallbacks ... ... because (a) it's actually simpler (b) it might be marginally more performant?	2018-03-01 12:40:05 +00:00
Erik Johnston	126b9bf96f	Log in the correct places	2018-03-01 12:05:33 +00:00
Erik Johnston	157298f986	Don't do preserve_fn for every request	2018-03-01 11:59:45 +00:00
Erik Johnston	89f90d808a	Add some logging	2018-03-01 11:59:16 +00:00
Erik Johnston	8ded8ba2c7	Make repl send_event idempotent and retry on timeouts ... If we treated timeouts as failures on the worker we would attempt to clean up e.g. push actions while the master might still process the event.	2018-03-01 11:20:34 +00:00
Erik Johnston	f381d63813	Check event auth on the worker	2018-03-01 10:18:37 +00:00
Erik Johnston	6b8604239f	Correctly send ratelimit and extra_users params	2018-03-01 10:08:39 +00:00
Erik Johnston	f756f961ea	Fixup comments	2018-03-01 10:05:27 +00:00
Erik Johnston	28e973ac11	Calculate push actions on worker	2018-02-28 18:02:30 +00:00
Erik Johnston	493e25d554	Move storage functions for push calculations ... This will allow push actions for an event to be calculated on workers.	2018-02-27 13:58:16 +00:00
Erik Johnston	3594dbc6dc	Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation ... Fix missing invalidations for receipt storage	2018-02-27 11:34:26 +00:00
Erik Johnston	2311189ee4	Merge pull request #2903 from matrix-org/erikj/split_roommember_store ... Split out RoomMemberStore	2018-02-27 11:32:10 +00:00
Erik Johnston	c57607874c	Merge pull request #2901 from matrix-org/erikj/split_as_stores ... Split AS stores	2018-02-27 10:07:07 +00:00
Erik Johnston	8956f0147a	Add comment	2018-02-27 10:06:51 +00:00
Erik Johnston	e5b4a208ce	Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions ... Batch inserts into event_push_actions_staging	2018-02-26 14:45:40 +00:00
Erik Johnston	45b5fe9122	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push	2018-02-26 13:49:24 +00:00
Erik Johnston	d62ce972f8	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store	2018-02-23 11:46:24 +00:00
Erik Johnston	6ae9a3d2a6	Update copyright	2018-02-23 11:44:49 +00:00
Erik Johnston	a90c60912f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions	2018-02-23 11:26:31 +00:00
Erik Johnston	50e8657867	Merge pull request #2902 from matrix-org/erikj/split_events_store ... Split out get_events and co into a worker store	2018-02-23 11:23:52 +00:00
Erik Johnston	1cf9e071dd	Merge pull request #2899 from matrix-org/erikj/split_pushers ... Split PusherStore	2018-02-23 11:23:35 +00:00
Erik Johnston	d0957753bf	Merge pull request #2898 from matrix-org/erikj/split_push_rules_store ... Split PushRulesStore	2018-02-23 11:23:23 +00:00
Erik Johnston	199dba6c15	Merge pull request #2897 from matrix-org/erikj/split_account_data ... Split AccountDataStore and TagStore	2018-02-23 11:23:11 +00:00
Erik Johnston	70349872c2	Update copyright	2018-02-23 11:14:35 +00:00
Erik Johnston	eba93b05bf	Split EventsWorkerStore into separate file	2018-02-23 11:01:21 +00:00
Erik Johnston	bf8a36e080	Update copyright	2018-02-23 10:52:10 +00:00
Erik Johnston	5d0f665848	Remove redundant clock	2018-02-23 10:49:58 +00:00
Erik Johnston	3bd760628b	_event_persist_queue shouldn't be in worker store	2018-02-23 10:49:18 +00:00
Erik Johnston	eb9b5eec81	Update copyright	2018-02-23 10:42:39 +00:00
Erik Johnston	c2ecfcc3a4	Update copyright	2018-02-23 10:41:34 +00:00
Erik Johnston	7e6cf89dc2	Update copyright	2018-02-23 10:39:19 +00:00
Erik Johnston	26d37f7a63	Update copyright	2018-02-23 10:33:55 +00:00
Erik Johnston	bb73f55fc6	Use absolute imports	2018-02-23 10:31:16 +00:00
Erik Johnston	faeb369f15	Fix missing invalidations for receipt storage	2018-02-21 15:19:54 +00:00
Erik Johnston	3dec9c66b3	Split out RoomMemberStore	2018-02-21 12:07:26 +00:00
Erik Johnston	46244b2759	Split AS stores	2018-02-21 11:49:34 +00:00
Erik Johnston	27b094f382	Split out get_events and co into a worker store	2018-02-21 11:41:48 +00:00
Erik Johnston	573712da6b	Update comments	2018-02-21 11:29:49 +00:00
Erik Johnston	c96d547f4d	Actually use new param	2018-02-21 11:03:42 +00:00
Erik Johnston	d15d237b0d	Split out EventPushActionWorkerStore	2018-02-21 11:01:13 +00:00
Erik Johnston	27939cbb0e	Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed ... Delete from push_actions_staging in federation too	2018-02-21 11:00:06 +00:00
Erik Johnston	6f72765371	Split PusherStore	2018-02-21 10:54:21 +00:00
Erik Johnston	cbaad969f9	Split PushRulesStore	2018-02-21 10:43:31 +00:00
Erik Johnston	ca9b9d9703	Split AccountDataStore and TagStore	2018-02-21 10:15:04 +00:00
Erik Johnston	8fbb4d0d19	Raise exception in abstract method	2018-02-20 17:59:23 +00:00
Erik Johnston	95e4cffd85	Fix comment	2018-02-20 17:58:40 +00:00
Erik Johnston	e316bbb4c0	Use abstract base class to access stream IDs	2018-02-20 17:43:57 +00:00
Erik Johnston	f5ac4dc2d4	Split ReceiptsStore	2018-02-20 16:28:28 +00:00
Erik Johnston	24087bffa9	Ensure all push actions are deleted from staging	2018-02-20 12:34:31 +00:00
Erik Johnston	ad0ccf15ea	Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts	2018-02-20 12:34:28 +00:00
Erik Johnston	d874d4f2d7	Delete from push_actions_staging in federation too	2018-02-20 11:37:52 +00:00
Erik Johnston	6ff8c87484	Batch inserts into event_push_actions_staging	2018-02-20 11:33:07 +00:00
Erik Johnston	324c3e9399	Merge pull request #2868 from matrix-org/erikj/refactor_media_storage ... Make store_file use store_into_file	2018-02-20 11:31:24 +00:00
Pascal Bach	3acd616979	Update pynacl dependency to 1.2.1 or higher ... Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>	2018-02-19 10:45:22 +01:00
Richard van der Hoff	d1a3325f99	(Really) fix tablescan of event_push_actions on purge ... commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/	2018-02-16 14:02:31 +00:00
Erik Johnston	6af025d3c4	Fix typo of double is_highlight	2018-02-16 11:35:31 +00:00
Erik Johnston	012e8e142a	Comments	2018-02-16 11:35:01 +00:00
Erik Johnston	b96278d6fe	Ensure that we delete staging push actions on errors	2018-02-15 15:47:06 +00:00
Erik Johnston	4810f7effd	Remove context.push_actions	2018-02-15 15:47:06 +00:00
Erik Johnston	c714c61853	Update event_push_actions table from staging table	2018-02-15 15:47:06 +00:00
Erik Johnston	acac21248c	Store push actions in staging area	2018-02-15 15:47:04 +00:00
Erik Johnston	106906a65e	Don't serialize current state over replication	2018-02-15 13:53:18 +00:00
Erik Johnston	5fb347fc41	Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log ... Don't log errors propogated from send_event	2018-02-15 12:31:49 +00:00
Erik Johnston	fd1601c596	Fix state group storage bug in workers ... We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.	2018-02-15 11:04:32 +00:00
Erik Johnston	ef344b10e5	Don't log errors propogated from send_event	2018-02-15 11:03:49 +00:00
Richard van der Hoff	b8d821aa68	Merge pull request #2867 from matrix-org/rav/rework_purge ... purge_history cleanups	2018-02-15 09:49:07 +00:00
Erik Johnston	92c52df702	Make store_file use store_into_file	2018-02-14 17:55:18 +00:00
Richard van der Hoff	d28ec43e15	Merge pull request #2769 from matrix-org/matthew/hit_the_gin ... switch back from GIST to GIN indexes	2018-02-14 16:59:03 +00:00
Richard van der Hoff	39bf47319f	purge_history: fix sqlite syntax error ... apparently sqlite insists on indexes being named	2018-02-14 16:42:19 +00:00
Richard van der Hoff	ac27f6a35e	purge_history: handle sqlite asshattery ... apparently creating a temporary table commits the transaction. because that's a useful thing.	2018-02-14 16:41:12 +00:00
Richard van der Hoff	5978dccff0	remove overzealous exception handling	2018-02-14 15:54:09 +00:00
Richard van der Hoff	278d21b5e4	purge_history: fix index use ... event_push_actions doesn't have an index on event_id, so we need to specify room_id.	2018-02-14 15:44:51 +00:00
Richard van der Hoff	5fcbf1e07c	Rework event purge to use a temporary table ... ... which should speed things up by reducing the amount of data being shuffled across the connection	2018-02-14 11:02:22 +00:00
Erik Johnston	c0c9327fe0	Merge pull request #2854 from matrix-org/erikj/event_create_worker ... Create a worker for event creation	2018-02-13 18:07:10 +00:00
Richard van der Hoff	d627174da2	Fix log message in purge_history ... (we don't just remove remote events)	2018-02-13 16:51:21 +00:00
Richard van der Hoff	ddb6a79b68	Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin	2018-02-13 16:45:36 +00:00
Richard van der Hoff	0b27ae8dc3	move search reindex to schema 47 ... We're up to schema v47 on develop now, so this will have to go in there to have an effect. This might cause an error if somebody has already run it in the v46 guise, and runs it again in the v47 guise, because it will cause a duplicate entry in the bbackground_updates table. On the other hand, the entry is removed once it is complete, and it is unlikely that anyone other than matrix.org has run it on v46. The update itself is harmless to re-run because it deliberately copes with the index already existing.	2018-02-13 16:44:46 +00:00
Richard van der Hoff	4a6d551704	GIN reindex: Fix syntax errors, improve exception handling	2018-02-13 16:44:46 +00:00
Richard van der Hoff	bfdf7b9237	Merge pull request #2864 from matrix-org/rav/persist_event_caching ... Use StateResolutionHandler to resolve state in persist_events	2018-02-13 14:45:57 +00:00
Richard van der Hoff	630caf8a70	style nit	2018-02-13 14:29:22 +00:00
Richard van der Hoff	8fd1a32456	Fix typos in purge api & doc ... * It's supposed to be purge_local_events, not ..._history * Fix the doc to have valid json	2018-02-13 13:09:39 +00:00
Richard van der Hoff	4d09366656	Merge pull request #2695 from okurz/feature/allow_recent_pysaml ... Allow use of higher versions of saml2	2018-02-13 12:24:08 +00:00
Richard van der Hoff	a9b712e9dc	Merge branch 'develop' into matthew/gin_work_mem	2018-02-13 12:16:01 +00:00
Erik Johnston	1026690cd2	Merge pull request #2857 from matrix-org/erikj/upload_store ... Tell storage providers about new file so they can upload	2018-02-12 13:52:58 +00:00
Richard van der Hoff	10b34dbb9a	Merge pull request #2858 from matrix-org/rav/purge_updates ... delete_local_events for purge_room_history	2018-02-09 14:11:00 +00:00
Richard van der Hoff	39a6b35496	purge: move room_depth update to end ... ... to avoid locking the table for too long	2018-02-09 13:07:41 +00:00
Richard van der Hoff	74fcbf741b	delete_local_events for purge_history ... Add a flag which makes the purger delete local events	2018-02-09 13:07:41 +00:00
Richard van der Hoff	e571aef06d	purge: Move cache invalidation to more appropriate place ... it was a bit of a non-sequitur there	2018-02-09 13:07:41 +00:00
Richard van der Hoff	61ffaa8137	bump purge logging to info ... this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.	2018-02-09 13:07:41 +00:00
Richard van der Hoff	671540dccf	rename delete_old_state -> purge_history ... (beacause it deletes more than state)	2018-02-09 13:07:41 +00:00
Erik Johnston	5fa571a91b	Tell storage providers about new file so they can upload	2018-02-07 13:35:08 +00:00
Erik Johnston	50fe92cd26	Move presence handling into handle_new_client_event ... As we want to have it run on the main synapse instance	2018-02-07 10:34:09 +00:00
Erik Johnston	8ec2e638be	Add event_creator worker	2018-02-07 10:32:32 +00:00
Erik Johnston	24dd73028a	Add replication http endpoint for event sending	2018-02-07 10:32:32 +00:00
Erik Johnston	e3624fad5f	Remove pointless ratelimit check ... The intention was for the check to be called as early as possible in the request, but actually was called just before the main ratelimit check, so was fairly pointless.	2018-02-07 10:30:25 +00:00
Erik Johnston	617199d73d	Merge pull request #2847 from matrix-org/erikj/separate_event_creation ... Split event creation into a separate handler	2018-02-06 17:01:17 +00:00
Erik Johnston	3e1e69ccaf	Update copyright	2018-02-06 16:40:38 +00:00
Erik Johnston	770b2252ca	s/_create_new_client_event/create_new_client_event/	2018-02-06 16:40:30 +00:00
Erik Johnston	3d33eef6fc	Store state groups separately from events (#2784) ... * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints	2018-02-06 14:31:24 +00:00
Richard van der Hoff	b31bf0bb51	Merge pull request #2849 from matrix-org/rav/clean_up_state_delta ... Remove redundant return value from _calculate_state_delta	2018-02-05 17:42:20 +01:00
Richard van der Hoff	ebfe64e3d6	Use StateResolutionHandler to resolve state in persist events ... ... and thus benefit (hopefully) from its cache.	2018-02-05 16:23:26 +00:00
Richard van der Hoff	225dc3b4cb	Flatten _get_new_state_after_events ... rejig the if statements to simplify the logic and reduce indentation	2018-02-05 16:23:25 +00:00
Richard van der Hoff	9fcbbe8e7d	Check that events being persisted have state_group	2018-02-05 16:23:25 +00:00
Richard van der Hoff	447aed42d2	Add event_map param to resolve_state_groups	2018-02-05 16:23:25 +00:00
Richard van der Hoff	ee6fb4cf85	Remove redundant return value from _calculate_state_delta ... we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.	2018-02-05 16:23:20 +00:00
Richard van der Hoff	3c7b480ba3	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-05 16:12:14 +00:00
Erik Johnston	3fa362502c	Update places where we create events	2018-02-05 16:01:48 +00:00
Erik Johnston	5ff3d23564	Split event creation into a separate handler	2018-02-05 16:01:48 +00:00
Richard van der Hoff	c46e75d3d8	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-05 15:43:22 +00:00
Richard van der Hoff	db91e72ade	Merge pull request #2844 from matrix-org/rav/evicted_metrics ... montoring metrics for number of cache evictions	2018-02-05 16:34:36 +01:00
Richard van der Hoff	bc496df192	report metrics on number of cache evictions	2018-02-05 15:34:01 +00:00
Erik Johnston	1f881e0746	Merge pull request #2791 from matrix-org/erikj/media_storage_refactor ... Ensure media is in local cache before thumbnailing	2018-02-05 11:28:52 +00:00
Richard van der Hoff	80b8a28100	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-04 00:23:06 +00:00
Richard van der Hoff	bd25f9cf36	Clean up work_mem handling ... Add some comments and improve exception handling when twiddling work_mem for the search update	2018-02-03 23:05:41 +00:00
Richard van der Hoff	4eeae7ad65	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-03 22:59:45 +00:00
Richard van der Hoff	bb9f0f3cdb	Merge branch 'develop' into matthew/gin_work_mem	2018-02-03 22:40:28 +00:00
Richard van der Hoff	6b02fc80d1	Reinstate event_search_postgres_gist handler ... People may have queued updates for this, so we can't just delete it.	2018-02-02 14:32:51 +00:00
Richard van der Hoff	9c9356512e	Merge pull request #2845 from matrix-org/rav/urlcache_error_handling ... Handle url_previews with no content-type	2018-02-02 15:27:52 +01:00
Richard van der Hoff	18eae413af	Merge pull request #2842 from matrix-org/rav/state_resolution_handler ... Factor out resolve_state_groups to a separate handler	2018-02-02 15:27:35 +01:00
Richard van der Hoff	78d6ddba86	Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta ... factor _get_new_state_after_events out of _calculate_state_delta	2018-02-02 15:27:15 +01:00
Richard van der Hoff	9dcd667ac2	Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring ... Docstring fixes	2018-02-02 15:26:49 +01:00
Travis Ralston	6e87b34f7b	Merge branch 'develop' into travis/admin-list-media	2018-02-01 18:05:47 -07:00
Richard van der Hoff	d5352cbba8	Handle url_previews with no content-type ... avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.	2018-02-02 00:53:46 +00:00
Richard van der Hoff	14737ba495	doc arg types for _seperate	2018-02-01 12:41:34 +00:00
Richard van der Hoff	e15d4ea248	More docstring fixes ... Fix a couple of errors in docstrings	2018-02-01 12:41:34 +00:00
Richard van der Hoff	a18828c129	Fix docstring for StateHandler.resolve_state_groups ... The return type was a complete lie, so fix it	2018-02-01 12:41:34 +00:00
Richard van der Hoff	6da4c4d3bd	Factor out resolve_state_groups to a separate handler ... We extract the storage-independent bits of the state group resolution out to a separate functiom, and stick it in a new handler, in preparation for its use from the storage layer.	2018-02-01 12:40:04 +00:00
Richard van der Hoff	0cbda53819	Rename resolve_state_groups -> resolve_state_groups_for_events ... (to make way for a method that actually just does the state group resolution)	2018-02-01 12:40:00 +00:00
Richard van der Hoff	77c0629ebc	Merge pull request #2837 from matrix-org/rav/fix_quarantine_media ... Fix sql error in quarantine_media	2018-02-01 11:45:58 +01:00
Travis Ralston	e16e45b1b4	pep8 ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 15:30:38 -07:00
Richard van der Hoff	e1e4ec9f9d	factor _get_new_state_after_events out of _calculate_state_delta ... This reduces the scope of a bunch of variables	2018-01-31 21:32:09 +00:00
Richard van der Hoff	78e7e05188	Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail ... Remove spurious log argument	2018-01-31 22:18:46 +01:00
Richard van der Hoff	ad48dfe73d	Merge pull request #2835 from matrix-org/rav/remove_event_type_param ... Remove unused "event_type" param on state.get_current_state_ids	2018-01-31 22:17:57 +01:00
Richard van der Hoff	518a74586c	Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling ... Improve exception handling in persist_event	2018-01-31 22:17:38 +01:00
Richard van der Hoff	d1fe4db882	Merge pull request #2833 from matrix-org/rav/pusher_hacks ... Logging and metrics for the http pusher	2018-01-31 22:16:59 +01:00
Richard van der Hoff	421d68ca8c	Merge pull request #2817 from matrix-org/rav/http_conn_pool ... Use a connection pool for the SimpleHttpClient	2018-01-31 22:14:22 +01:00
Travis Ralston	63c4383927	Documentation and naming ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 08:07:52 -07:00
Richard van der Hoff	af19f5e9aa	Remove spurious log argument ... ... which would cause scary-looking and unhelpful errors in the log on dns fail	2018-01-30 17:52:03 +00:00
Richard van der Hoff	773f0eed1e	Fix sql error in quarantine_media	2018-01-30 15:02:51 +00:00
Richard van der Hoff	adfc0c9539	docstring for get_current_state_ids	2018-01-29 17:39:55 +00:00
Richard van der Hoff	d413a2ba98	Remove unused "event_type" param on state.get_current_state_ids ... this param doesn't seem to be used, and is a bit pointless anyway because it can easily be replicated by the caller. It is also horrible, because it changes the return type of the method.	2018-01-29 17:06:57 +00:00
Richard van der Hoff	b387ee17b6	Improve exception handling in persist_event ... 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs	2018-01-29 17:05:33 +00:00
Richard van der Hoff	03dd745fe2	Better logging when pushes fail	2018-01-29 15:49:06 +00:00
Richard van der Hoff	e051abd20b	add appid/device_display_name to to pusher logging	2018-01-29 15:04:16 +00:00
Richard van der Hoff	02ba118f81	Increase http conn pool size	2018-01-29 14:30:15 +00:00
Richard van der Hoff	46022025ea	Fix SQL for user search ... fix some syntax errors for user search when search_all_users is enabled fixes #2801, hopefully	2018-01-27 17:21:57 +00:00
Richard van der Hoff	4b0f06e99c	Merge pull request #2830 from matrix-org/rav/factor_out_get_conn ... Factor out get_db_conn to HomeServer base class	2018-01-27 17:56:25 +01:00
Neil Johnson	e98f0f9112	Merge pull request #2827 from matrix-org/fix_server_500_on_public_rooms_call_when_no_rooms_exist ... Fix server 500 on public rooms call when no rooms exist	2018-01-26 20:22:40 +00:00
Richard van der Hoff	25adde9a04	Factor out get_db_conn to HomeServer base class ... This function is identical to all subclasses, so we may as well push it up to the base class to reduce duplication (and make use of it in the tests)	2018-01-26 00:56:49 +00:00
Neil Johnson	73560237d6	add white space line	2018-01-26 00:15:10 +00:00
Neil Johnson	86c4f49a31	rather than try reconstruct the results object, better to guard against the xrange step argument being 0	2018-01-26 00:12:02 +00:00
Neil Johnson	f632083576	fix return type, should be a dict	2018-01-25 23:52:17 +00:00
Neil Johnson	6c6e197b0a	fix PEP8 violation	2018-01-25 23:47:46 +00:00
Neil Johnson	d02e43b15f	remove white space	2018-01-25 23:29:46 +00:00
Neil Johnson	349c739966	synapse 500s on a call to publicRooms in the case where the number of public rooms is zero, the specific cause is due to xrange trying to use a step value of zero, but if the total room number really is zero then it makes sense to just bail and save the extra processing	2018-01-25 23:28:44 +00:00
Matthew Hodgson	9a72b70630	fix thinko on 3pid whitelisting	2018-01-24 11:07:47 +01:00
Richard van der Hoff	25e2456ee7	Merge pull request #2816 from matrix-org/rav/metrics_comments ... Add some comments about the reactor tick time metric	2018-01-23 19:39:26 +00:00
Matthew Hodgson	d32385336f	add ?ts massaging for ASes (#2754) ... blindly implement ?ts for AS. untested	2018-01-23 09:59:06 +01:00
Richard van der Hoff	4528dd2443	Fix logging and add user_id	2018-01-22 20:15:42 +00:00
Richard van der Hoff	93efd7eb04	logging and debug for http pusher	2018-01-22 18:14:10 +00:00
Matthew Hodgson	ab9f844aaf	Add federation_domain_whitelist option (#2820) ... Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network	2018-01-22 19:11:18 +01:00
Matthew Hodgson	d84f65255e	Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid ... add registrations_require_3pid and allow_local_3pids	2018-01-22 13:57:22 +00:00
Travis Ralston	a94d9b6b82	Appease the linter ... These are ids anyways, not mxc uris. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:49:46 -07:00
Travis Ralston	5552ed9a7f	Add an admin route to get all the media in a room ... This is intended to be used by administrators to monitor the media that is passing through their server, if they wish. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:37:53 -07:00
Richard van der Hoff	2c8526cac7	Use a connection pool for the SimpleHttpClient ... In particular I hope this will help the pusher, which makes many requests to sygnal, and is currently negotiating SSL for each one.	2018-01-20 00:55:44 +00:00
Richard van der Hoff	87b7d72760	Add some comments about the reactor tick time metric	2018-01-19 23:51:04 +00:00
Matthew Hodgson	49fce04624	fix typo (thanks sytest)	2018-01-19 19:55:38 +00:00
Richard van der Hoff	b0d9e633ee	Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs ... Use the right path for url_preview thumbnails	2018-01-19 18:57:15 +00:00
Richard van der Hoff	ad7ec63d08	Use the right path for url_preview thumbnails ... This was introduced by #2627: we were overwriting the original media for url previews with the thumbnails :/ (fixes https://github.com/vector-im/riot-web/issues/6012, hopefully)	2018-01-19 18:29:39 +00:00
Matthew Hodgson	62d7d66ae5	oops, check all login types	2018-01-19 18:23:56 +00:00
Matthew Hodgson	8fe253f19b	fix PR nitpicking	2018-01-19 18:23:45 +00:00
Matthew Hodgson	293380bef7	trailing commas	2018-01-19 15:38:53 +00:00
Matthew Hodgson	447f4f0d5f	rewrite based on PR feedback: ... * [ ] split config options into allowed_local_3pids and registrations_require_3pid * [ ] simplify and comment logic for picking registration flows * [ ] fix docstring and move check_3pid_allowed into a new util module * [ ] use check_3pid_allowed everywhere @erikjohnston PTAL	2018-01-19 15:33:55 +00:00
Matthew Hodgson	9d332e0f79	fix up v1, and improve errors	2018-01-19 00:53:58 +00:00
Matthew Hodgson	0af58f14ee	fix pep8	2018-01-19 00:33:51 +00:00
Matthew Hodgson	28a6ccb49c	add registrations_require_3pid ... lets homeservers specify a whitelist for 3PIDs that users are allowed to associate with. Typically useful for stopping people from registering with non-work emails	2018-01-19 00:19:58 +00:00
Erik Johnston	cd871a3057	Fix storage provider bug introduced when renamed to store_local	2018-01-18 18:37:59 +00:00
Erik Johnston	8ff6726c0d	Merge pull request #2812 from matrix-org/erikj/media_storage_provider_config ... Make storage providers configurable	2018-01-18 18:33:57 +00:00
Erik Johnston	d69768348f	Fix passing wrong config to provider constructor	2018-01-18 17:14:05 +00:00
Erik Johnston	8e85220373	Remove duplicate directory test	2018-01-18 17:12:35 +00:00
Erik Johnston	3fe2bae857	Missing staticmethod	2018-01-18 17:11:45 +00:00
Erik Johnston	aae77da73f	Fixup comments	2018-01-18 17:11:29 +00:00
Erik Johnston	b6dc7044a9	Merge pull request #2804 from matrix-org/erikj/file_consumer ... Add decent impl of a FileConsumer	2018-01-18 16:31:33 +00:00
Erik Johnston	9a89dae8c5	Fix typo in thumbnail resource causing access times to be incorrect	2018-01-18 15:06:24 +00:00
Erik Johnston	0af5dc63a8	Make storage providers more configurable	2018-01-18 14:07:21 +00:00
Richard van der Hoff	d57765fc8a	Fix bugs in block metrics ... ... which I introduced in #2785	2018-01-18 12:24:42 +00:00
Erik Johnston	2cf6a7bc20	Use better file consumer	2018-01-18 12:00:46 +00:00
Erik Johnston	4a53f3a3e8	Ensure media is in local cache before thumbnailing	2018-01-18 12:00:46 +00:00
Erik Johnston	be0dfcd4a2	Do logcontexts correctly	2018-01-18 11:57:57 +00:00
Erik Johnston	1432f7ccd5	Move test stuff to tests	2018-01-18 11:57:57 +00:00
Erik Johnston	2f18a2647b	Make all fields private	2018-01-18 11:57:54 +00:00
Richard van der Hoff	d6af5512bb	Merge pull request #2809 from matrix-org/rav/metrics_errors ... better exception logging in callbackmetrics	2018-01-18 11:46:37 +00:00
Richard van der Hoff	ce236f8ac8	better exception logging in callbackmetrics ... when we fail to render a metric, give a clue as to which metric it was	2018-01-18 11:30:49 +00:00
Erik Johnston	dc519602ac	Ensure we registerProducer isn't called twice	2018-01-18 11:07:17 +00:00
Erik Johnston	17b54389fe	Fix _notify_empty typo	2018-01-18 11:05:34 +00:00
Erik Johnston	28b338ed9b	Move definition of paused_producer to __init__	2018-01-18 11:04:41 +00:00
Erik Johnston	a177325b49	Fix comments	2018-01-18 11:02:43 +00:00
Richard van der Hoff	36da256cc6	Merge pull request #2805 from matrix-org/rav/log_state_res ... Log room when doing state resolution	2018-01-17 18:05:04 +00:00
Richard van der Hoff	1224612a79	Log room when doing state resolution ... Mostly because it helps figure out what is prompting the resolution	2018-01-17 17:11:59 +00:00
Erik Johnston	bc67e7d260	Add decent impl of a FileConsumer ... Twisted core doesn't have a general purpose one, so we need to write one ourselves. Features: - All writing happens in background thread - Supports both push and pull producers - Push producers get paused if the consumer falls behind	2018-01-17 16:43:03 +00:00
Erik Johnston	a87006f9c7	Merge pull request #2783 from matrix-org/erikj/media_last_accessed ... Keep track of last access time for local media	2018-01-17 16:39:02 +00:00
Matthew Hodgson	06db5c4b76	Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql ... fix SQL when searching all users	2018-01-17 16:27:54 +00:00
Matthew Hodgson	2d9ab533f9	fix SQL when searching all users	2018-01-17 15:58:52 +00:00
Richard van der Hoff	390093d45e	Split resolve_events into two functions ... ... so that the return type doesn't depend on the arg types	2018-01-17 15:44:31 +00:00
Erik Johnston	2fb3a28c98	Remove lost comment	2018-01-17 14:59:44 +00:00
Richard van der Hoff	a7e4ff9cca	Merge pull request #2795 from matrix-org/rav/track_db_scheduling ... Track DB scheduling delay per-request	2018-01-17 14:29:37 +00:00
Richard van der Hoff	f884cfffb9	Merge pull request #2797 from matrix-org/rav/user_id_checking ... Sanity checking for user ids	2018-01-17 14:29:12 +00:00
Richard van der Hoff	a5213df1f7	Sanity checking for user ids ... Check the user_id passed to a couple of APIs for validity, to avoid "IndexError: list index out of range" exception which looks scary and results in a 500 rather than a more useful error. Fixes #1432, among other things	2018-01-17 14:28:54 +00:00
Richard van der Hoff	e8f7541d3f	Merge remote-tracking branch 'origin/develop' into rav/track_db_scheduling	2018-01-17 14:01:57 +00:00
Richard van der Hoff	fb6563b4be	Merge pull request #2793 from matrix-org/rav/db_txn_time_in_millis ... Track db txn time in millisecs	2018-01-17 13:52:42 +00:00
Richard van der Hoff	1954e867b4	Merge pull request #2796 from matrix-org/rav/fix_closed_connection_errors ... Fix 'NoneType' object has no attribute 'writeHeaders'	2018-01-17 13:52:24 +00:00
Richard van der Hoff	11ab2f56f5	Merge branch 'develop' into rav/fix_closed_connection_errors	2018-01-17 11:25:11 +00:00
Richard van der Hoff	0486a7814a	Merge branch 'develop' into rav/rework_run_interaction	2018-01-17 11:24:38 +00:00
Richard van der Hoff	90c14da992	Merge branch 'develop' into rav/db_txn_time_in_millis	2018-01-17 11:19:55 +00:00
Richard van der Hoff	38506773eb	Merge remote-tracking branch 'origin/develop' into rav/optimise_logging_context	2018-01-17 10:57:13 +00:00
Erik Johnston	300edc2348	Update last access time when thumbnails are viewed	2018-01-17 10:24:43 +00:00
Erik Johnston	05f98a2224	Keep track of last access time for local media	2018-01-17 10:24:43 +00:00
Erik Johnston	3cb2dabaad	Merge pull request #2789 from matrix-org/erikj/fix_thumbnails ... Fix thumbnailing remote files	2018-01-17 10:22:20 +00:00
Erik Johnston	d728c47142	Add docstring	2018-01-17 10:06:14 +00:00
Richard van der Hoff	936482d507	Fix 'NoneType' object has no attribute 'writeHeaders' ... Avoid throwing a (harmless) exception when we try to write an error response to an http request where the client has disconnected. This comes up as a CRITICAL error in the logs which tends to mislead people into thinking there's an actual problem	2018-01-16 17:58:16 +00:00
Richard van der Hoff	3d12d97415	Track DB scheduling delay per-request ... For each request, track the amount of time spent waiting for a db connection. This entails adding it to the LoggingContext and we may as well add metrics for it while we are passing.	2018-01-16 17:23:32 +00:00
Richard van der Hoff	0f5d2cc37c	Merge branch 'rav/db_txn_time_in_millis' into rav/track_db_scheduling	2018-01-16 17:09:15 +00:00
Richard van der Hoff	8615f19d20	rework runInteraction in terms of runConnection ... ... so that we can share the code	2018-01-16 17:08:29 +00:00
Matthew Hodgson	5e97ca7ee6	fix typo	2018-01-16 16:52:35 +00:00
Erik Johnston	d863f68cab	Use local vars	2018-01-16 16:24:15 +00:00
Erik Johnston	6368e5c0ab	Change _generate_thumbnails to take media_type	2018-01-16 16:17:38 +00:00
Erik Johnston	0a90d9ede4	Move setting of file_id up to caller	2018-01-16 16:03:05 +00:00
Richard van der Hoff	6324b65f08	Track db txn time in millisecs ... ... to reduce the amount of floating-point foo we do.	2018-01-16 15:53:18 +00:00
Richard van der Hoff	44a498418c	Optimise LoggingContext creation and copying ... It turns out that the only thing we use the __dict__ of LoggingContext for is `request`, and given we create lots of LoggingContexts and then copy them every time we do a db transaction or log line, using the __dict__ seems a bit redundant. Let's try to optimise things by making the request attribute explicit.	2018-01-16 15:49:42 +00:00
Erik Johnston	5dfc83704b	Fix typo	2018-01-16 14:32:56 +00:00
Richard van der Hoff	febdca4b37	Merge pull request #2785 from matrix-org/rav/reorganise_metrics_again ... Reorganise request and block metrics	2018-01-16 14:10:21 +00:00
Richard van der Hoff	f5f89fda21	Merge pull request #2790 from matrix-org/rav/preserve_event_logcontext_leak ... Fix a logcontext leak in persist_events	2018-01-16 14:09:13 +00:00
Erik Johnston	307f88dfb6	Fix up log lines	2018-01-16 13:53:52 +00:00
Richard van der Hoff	807e848f0f	Merge pull request #2787 from matrix-org/rav/worker_event_counts ... Metrics for events processed in appservice and fed sender	2018-01-16 13:14:09 +00:00
Richard van der Hoff	4a31a61ef9	Merge pull request #2786 from matrix-org/rav/rdata_metrics ... Metrics for number of RDATA commands received	2018-01-16 13:13:53 +00:00
Erik Johnston	9795b9ebb1	Correctly use server_name/file_id when generating/fetching remote thumbnails	2018-01-16 12:02:06 +00:00
Erik Johnston	c5b589f2e8	Log when we respond with 404	2018-01-16 12:01:40 +00:00
Richard van der Hoff	64ddec1bc0	Fix a logcontext leak in persist_events ... ObserveableDeferred expects its callbacks to be called without any logcontexts, whereas it turns out we were calling them with the logcontext of the request which initiated the persistence loop. It seems wrong that we are attributing work done in the persistence loop to the request that happened to initiate it, so let's solve this by dropping the logcontext for it. (I'm not sure this actually causes any real problems other than messages in the debug log, but let's clean it up anyway)	2018-01-16 11:47:36 +00:00
Erik Johnston	a4c5e4a645	Fix thumbnailing remote files	2018-01-16 11:37:50 +00:00
Erik Johnston	1159abbdd2	Merge pull request #2767 from matrix-org/erikj/media_storage_refactor ... Refactor MediaRepository to separate out storage	2018-01-16 10:23:50 +00:00
Richard van der Hoff	a027c2af8d	Metrics for events processed in appservice and fed sender ... More metrics I wished I'd had	2018-01-15 18:23:24 +00:00
Richard van der Hoff	5c3c32f16f	Metrics for number of RDATA commands received ... I found myself wishing we had this.	2018-01-15 17:45:55 +00:00
Richard van der Hoff	39f4e29d01	Reorganise request and block metrics ... In order to circumvent the number of duplicate foo:count metrics increasing without bounds, it's time for a rearrangement. The following are all deprecated, and replaced with synapse_util_metrics_block_count: synapse_util_metrics_block_timer:count synapse_util_metrics_block_ru_utime:count synapse_util_metrics_block_ru_stime:count synapse_util_metrics_block_db_txn_count:count synapse_util_metrics_block_db_txn_duration:count The following are all deprecated, and replaced with synapse_http_server_response_count: synapse_http_server_requests synapse_http_server_response_time:count synapse_http_server_response_ru_utime:count synapse_http_server_response_ru_stime:count synapse_http_server_response_db_txn_count:count synapse_http_server_response_db_txn_duration:count The following are renamed (the old metrics are kept for now, but deprecated): synapse_util_metrics_block_timer:total -> synapse_util_metrics_block_time_seconds synapse_util_metrics_block_ru_utime:total -> synapse_util_metrics_block_ru_utime_seconds synapse_util_metrics_block_ru_stime:total -> synapse_util_metrics_block_ru_stime_seconds synapse_util_metrics_block_db_txn_count:total -> synapse_util_metrics_block_db_txn_count synapse_util_metrics_block_db_txn_duration:total -> synapse_util_metrics_block_db_txn_duration_seconds synapse_http_server_response_time:total -> synapse_http_server_response_time_seconds synapse_http_server_response_ru_utime:total -> synapse_http_server_response_ru_utime_seconds synapse_http_server_response_ru_stime:total -> synapse_http_server_response_ru_stime_seconds synapse_http_server_response_db_txn_count:total -> synapse_http_server_response_db_txn_count synapse_http_server_response_db_txn_duration:total synapse_http_server_response_db_txn_duration_seconds	2018-01-15 17:09:44 +00:00
Richard van der Hoff	992018d1c0	mechanism to render metrics with alternative names	2018-01-15 17:04:39 +00:00
Richard van der Hoff	80fa610f9c	Add some comments to metrics classes	2018-01-15 16:52:52 +00:00
Richard van der Hoff	19d274085f	Make Counter render floats ... Prometheus handles all metrics as floats, and sometimes we store non-integer values in them (notably, durations in seconds), so let's render them as floats too. (Note that the standard client libraries also treat Counters as floats.)	2018-01-12 23:49:44 +00:00
Richard van der Hoff	21bf87a146	Reinstate media download on thumbnail request ... We need to actually download the remote media when we get a request for a thumbnail.	2018-01-12 15:38:06 +00:00
Erik Johnston	694f1c1b18	Fix up comments	2018-01-12 15:02:46 +00:00
Erik Johnston	e21370ba54	Correctly reraise exception	2018-01-12 14:44:02 +00:00
Erik Johnston	85a4d78213	Make Responder a context manager	2018-01-12 13:32:03 +00:00
Erik Johnston	dcc8eded41	Add missing class var	2018-01-12 13:16:27 +00:00
Erik Johnston	fefeb0ab0e	Merge pull request #2774 from matrix-org/erikj/synctl ... When using synctl with workers, don't start the main synapse automatically	2018-01-12 12:00:07 +00:00
Erik Johnston	81391fa162	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/media_storage_refactor	2018-01-12 11:28:49 +00:00
Erik Johnston	1e4edd1717	Remove unnecessary condition	2018-01-12 11:28:32 +00:00
Erik Johnston	c6c009603c	Remove unused variables	2018-01-12 11:24:05 +00:00
Erik Johnston	4d88958cf6	Make class var local	2018-01-12 11:23:54 +00:00
Erik Johnston	227c491510	Comments	2018-01-12 11:22:41 +00:00
Erik Johnston	f4d93ae424	Actually make it work	2018-01-12 10:39:27 +00:00
Erik Johnston	f68e4cf690	Refactor	2018-01-12 10:11:12 +00:00
Richard van der Hoff	5f23b6d5ea	Merge pull request #2766 from matrix-org/rav/room_event ... Add /room/{id}/event/{id} to synapse	2018-01-11 14:47:18 +00:00
Erik Johnston	7cd34512d8	When using synctl with workers, don't start the main synapse automatically	2018-01-11 11:37:39 +00:00
Erik Johnston	07ab948c38	Merge branch 'master' of github.com:matrix-org/synapse into develop	2018-01-11 11:23:40 +00:00