7488 Commits

Author	SHA1	Message	Date
Erik Johnston	d874d4f2d7	Delete from push_actions_staging in federation too	2018-02-20 11:37:52 +00:00
Erik Johnston	6ff8c87484	Batch inserts into event_push_actions_staging	2018-02-20 11:33:07 +00:00
Erik Johnston	324c3e9399	Merge pull request #2868 from matrix-org/erikj/refactor_media_storage ... Make store_file use store_into_file	2018-02-20 11:31:24 +00:00
Pascal Bach	3acd616979	Update pynacl dependency to 1.2.1 or higher ... Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>	2018-02-19 10:45:22 +01:00
Richard van der Hoff	d1a3325f99	(Really) fix tablescan of event_push_actions on purge ... commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/	2018-02-16 14:02:31 +00:00
Erik Johnston	6af025d3c4	Fix typo of double is_highlight	2018-02-16 11:35:31 +00:00
Erik Johnston	012e8e142a	Comments	2018-02-16 11:35:01 +00:00
Erik Johnston	b96278d6fe	Ensure that we delete staging push actions on errors	2018-02-15 15:47:06 +00:00
Erik Johnston	4810f7effd	Remove context.push_actions	2018-02-15 15:47:06 +00:00
Erik Johnston	c714c61853	Update event_push_actions table from staging table	2018-02-15 15:47:06 +00:00
Erik Johnston	acac21248c	Store push actions in staging area	2018-02-15 15:47:04 +00:00
Erik Johnston	106906a65e	Don't serialize current state over replication	2018-02-15 13:53:18 +00:00
Erik Johnston	5fb347fc41	Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log ... Don't log errors propogated from send_event	2018-02-15 12:31:49 +00:00
Erik Johnston	fd1601c596	Fix state group storage bug in workers ... We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.	2018-02-15 11:04:32 +00:00
Erik Johnston	ef344b10e5	Don't log errors propogated from send_event	2018-02-15 11:03:49 +00:00
Richard van der Hoff	b8d821aa68	Merge pull request #2867 from matrix-org/rav/rework_purge ... purge_history cleanups	2018-02-15 09:49:07 +00:00
Erik Johnston	92c52df702	Make store_file use store_into_file	2018-02-14 17:55:18 +00:00
Richard van der Hoff	d28ec43e15	Merge pull request #2769 from matrix-org/matthew/hit_the_gin ... switch back from GIST to GIN indexes	2018-02-14 16:59:03 +00:00
Richard van der Hoff	39bf47319f	purge_history: fix sqlite syntax error ... apparently sqlite insists on indexes being named	2018-02-14 16:42:19 +00:00
Richard van der Hoff	ac27f6a35e	purge_history: handle sqlite asshattery ... apparently creating a temporary table commits the transaction. because that's a useful thing.	2018-02-14 16:41:12 +00:00
Richard van der Hoff	5978dccff0	remove overzealous exception handling	2018-02-14 15:54:09 +00:00
Richard van der Hoff	278d21b5e4	purge_history: fix index use ... event_push_actions doesn't have an index on event_id, so we need to specify room_id.	2018-02-14 15:44:51 +00:00
Richard van der Hoff	5fcbf1e07c	Rework event purge to use a temporary table ... ... which should speed things up by reducing the amount of data being shuffled across the connection	2018-02-14 11:02:22 +00:00
Erik Johnston	c0c9327fe0	Merge pull request #2854 from matrix-org/erikj/event_create_worker ... Create a worker for event creation	2018-02-13 18:07:10 +00:00
Richard van der Hoff	d627174da2	Fix log message in purge_history ... (we don't just remove remote events)	2018-02-13 16:51:21 +00:00
Richard van der Hoff	ddb6a79b68	Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin	2018-02-13 16:45:36 +00:00
Richard van der Hoff	0b27ae8dc3	move search reindex to schema 47 ... We're up to schema v47 on develop now, so this will have to go in there to have an effect. This might cause an error if somebody has already run it in the v46 guise, and runs it again in the v47 guise, because it will cause a duplicate entry in the bbackground_updates table. On the other hand, the entry is removed once it is complete, and it is unlikely that anyone other than matrix.org has run it on v46. The update itself is harmless to re-run because it deliberately copes with the index already existing.	2018-02-13 16:44:46 +00:00
Richard van der Hoff	4a6d551704	GIN reindex: Fix syntax errors, improve exception handling	2018-02-13 16:44:46 +00:00
Richard van der Hoff	bfdf7b9237	Merge pull request #2864 from matrix-org/rav/persist_event_caching ... Use StateResolutionHandler to resolve state in persist_events	2018-02-13 14:45:57 +00:00
Richard van der Hoff	630caf8a70	style nit	2018-02-13 14:29:22 +00:00
Richard van der Hoff	8fd1a32456	Fix typos in purge api & doc ... * It's supposed to be purge_local_events, not ..._history * Fix the doc to have valid json	2018-02-13 13:09:39 +00:00
Richard van der Hoff	4d09366656	Merge pull request #2695 from okurz/feature/allow_recent_pysaml ... Allow use of higher versions of saml2	2018-02-13 12:24:08 +00:00
Richard van der Hoff	a9b712e9dc	Merge branch 'develop' into matthew/gin_work_mem	2018-02-13 12:16:01 +00:00
Erik Johnston	1026690cd2	Merge pull request #2857 from matrix-org/erikj/upload_store ... Tell storage providers about new file so they can upload	2018-02-12 13:52:58 +00:00
Richard van der Hoff	10b34dbb9a	Merge pull request #2858 from matrix-org/rav/purge_updates ... delete_local_events for purge_room_history	2018-02-09 14:11:00 +00:00
Richard van der Hoff	39a6b35496	purge: move room_depth update to end ... ... to avoid locking the table for too long	2018-02-09 13:07:41 +00:00
Richard van der Hoff	74fcbf741b	delete_local_events for purge_history ... Add a flag which makes the purger delete local events	2018-02-09 13:07:41 +00:00
Richard van der Hoff	e571aef06d	purge: Move cache invalidation to more appropriate place ... it was a bit of a non-sequitur there	2018-02-09 13:07:41 +00:00
Richard van der Hoff	61ffaa8137	bump purge logging to info ... this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.	2018-02-09 13:07:41 +00:00
Richard van der Hoff	671540dccf	rename delete_old_state -> purge_history ... (beacause it deletes more than state)	2018-02-09 13:07:41 +00:00
Erik Johnston	5fa571a91b	Tell storage providers about new file so they can upload	2018-02-07 13:35:08 +00:00
Erik Johnston	50fe92cd26	Move presence handling into handle_new_client_event ... As we want to have it run on the main synapse instance	2018-02-07 10:34:09 +00:00
Erik Johnston	8ec2e638be	Add event_creator worker	2018-02-07 10:32:32 +00:00
Erik Johnston	24dd73028a	Add replication http endpoint for event sending	2018-02-07 10:32:32 +00:00
Erik Johnston	e3624fad5f	Remove pointless ratelimit check ... The intention was for the check to be called as early as possible in the request, but actually was called just before the main ratelimit check, so was fairly pointless.	2018-02-07 10:30:25 +00:00
Erik Johnston	617199d73d	Merge pull request #2847 from matrix-org/erikj/separate_event_creation ... Split event creation into a separate handler	2018-02-06 17:01:17 +00:00
Erik Johnston	3e1e69ccaf	Update copyright	2018-02-06 16:40:38 +00:00
Erik Johnston	770b2252ca	s/_create_new_client_event/create_new_client_event/	2018-02-06 16:40:30 +00:00
Erik Johnston	3d33eef6fc	Store state groups separately from events (#2784) ... * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints	2018-02-06 14:31:24 +00:00
Richard van der Hoff	b31bf0bb51	Merge pull request #2849 from matrix-org/rav/clean_up_state_delta ... Remove redundant return value from _calculate_state_delta	2018-02-05 17:42:20 +01:00
Richard van der Hoff	ebfe64e3d6	Use StateResolutionHandler to resolve state in persist events ... ... and thus benefit (hopefully) from its cache.	2018-02-05 16:23:26 +00:00
Richard van der Hoff	225dc3b4cb	Flatten _get_new_state_after_events ... rejig the if statements to simplify the logic and reduce indentation	2018-02-05 16:23:25 +00:00
Richard van der Hoff	9fcbbe8e7d	Check that events being persisted have state_group	2018-02-05 16:23:25 +00:00
Richard van der Hoff	447aed42d2	Add event_map param to resolve_state_groups	2018-02-05 16:23:25 +00:00
Richard van der Hoff	ee6fb4cf85	Remove redundant return value from _calculate_state_delta ... we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.	2018-02-05 16:23:20 +00:00
Richard van der Hoff	3c7b480ba3	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-05 16:12:14 +00:00
Erik Johnston	3fa362502c	Update places where we create events	2018-02-05 16:01:48 +00:00
Erik Johnston	5ff3d23564	Split event creation into a separate handler	2018-02-05 16:01:48 +00:00
Richard van der Hoff	c46e75d3d8	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-05 15:43:22 +00:00
Richard van der Hoff	db91e72ade	Merge pull request #2844 from matrix-org/rav/evicted_metrics ... montoring metrics for number of cache evictions	2018-02-05 16:34:36 +01:00
Richard van der Hoff	bc496df192	report metrics on number of cache evictions	2018-02-05 15:34:01 +00:00
Erik Johnston	1f881e0746	Merge pull request #2791 from matrix-org/erikj/media_storage_refactor ... Ensure media is in local cache before thumbnailing	2018-02-05 11:28:52 +00:00
Richard van der Hoff	80b8a28100	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-04 00:23:06 +00:00
Richard van der Hoff	bd25f9cf36	Clean up work_mem handling ... Add some comments and improve exception handling when twiddling work_mem for the search update	2018-02-03 23:05:41 +00:00
Richard van der Hoff	4eeae7ad65	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-03 22:59:45 +00:00
Richard van der Hoff	bb9f0f3cdb	Merge branch 'develop' into matthew/gin_work_mem	2018-02-03 22:40:28 +00:00
Richard van der Hoff	6b02fc80d1	Reinstate event_search_postgres_gist handler ... People may have queued updates for this, so we can't just delete it.	2018-02-02 14:32:51 +00:00
Richard van der Hoff	9c9356512e	Merge pull request #2845 from matrix-org/rav/urlcache_error_handling ... Handle url_previews with no content-type	2018-02-02 15:27:52 +01:00
Richard van der Hoff	18eae413af	Merge pull request #2842 from matrix-org/rav/state_resolution_handler ... Factor out resolve_state_groups to a separate handler	2018-02-02 15:27:35 +01:00
Richard van der Hoff	78d6ddba86	Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta ... factor _get_new_state_after_events out of _calculate_state_delta	2018-02-02 15:27:15 +01:00
Richard van der Hoff	9dcd667ac2	Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring ... Docstring fixes	2018-02-02 15:26:49 +01:00
Travis Ralston	6e87b34f7b	Merge branch 'develop' into travis/admin-list-media	2018-02-01 18:05:47 -07:00
Richard van der Hoff	d5352cbba8	Handle url_previews with no content-type ... avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.	2018-02-02 00:53:46 +00:00
Richard van der Hoff	14737ba495	doc arg types for _seperate	2018-02-01 12:41:34 +00:00
Richard van der Hoff	e15d4ea248	More docstring fixes ... Fix a couple of errors in docstrings	2018-02-01 12:41:34 +00:00
Richard van der Hoff	a18828c129	Fix docstring for StateHandler.resolve_state_groups ... The return type was a complete lie, so fix it	2018-02-01 12:41:34 +00:00
Richard van der Hoff	6da4c4d3bd	Factor out resolve_state_groups to a separate handler ... We extract the storage-independent bits of the state group resolution out to a separate functiom, and stick it in a new handler, in preparation for its use from the storage layer.	2018-02-01 12:40:04 +00:00
Richard van der Hoff	0cbda53819	Rename resolve_state_groups -> resolve_state_groups_for_events ... (to make way for a method that actually just does the state group resolution)	2018-02-01 12:40:00 +00:00
Richard van der Hoff	77c0629ebc	Merge pull request #2837 from matrix-org/rav/fix_quarantine_media ... Fix sql error in quarantine_media	2018-02-01 11:45:58 +01:00
Travis Ralston	e16e45b1b4	pep8 ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 15:30:38 -07:00
Richard van der Hoff	e1e4ec9f9d	factor _get_new_state_after_events out of _calculate_state_delta ... This reduces the scope of a bunch of variables	2018-01-31 21:32:09 +00:00
Richard van der Hoff	78e7e05188	Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail ... Remove spurious log argument	2018-01-31 22:18:46 +01:00
Richard van der Hoff	ad48dfe73d	Merge pull request #2835 from matrix-org/rav/remove_event_type_param ... Remove unused "event_type" param on state.get_current_state_ids	2018-01-31 22:17:57 +01:00
Richard van der Hoff	518a74586c	Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling ... Improve exception handling in persist_event	2018-01-31 22:17:38 +01:00
Richard van der Hoff	d1fe4db882	Merge pull request #2833 from matrix-org/rav/pusher_hacks ... Logging and metrics for the http pusher	2018-01-31 22:16:59 +01:00
Richard van der Hoff	421d68ca8c	Merge pull request #2817 from matrix-org/rav/http_conn_pool ... Use a connection pool for the SimpleHttpClient	2018-01-31 22:14:22 +01:00
Travis Ralston	63c4383927	Documentation and naming ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 08:07:52 -07:00
Richard van der Hoff	af19f5e9aa	Remove spurious log argument ... ... which would cause scary-looking and unhelpful errors in the log on dns fail	2018-01-30 17:52:03 +00:00
Richard van der Hoff	773f0eed1e	Fix sql error in quarantine_media	2018-01-30 15:02:51 +00:00
Richard van der Hoff	adfc0c9539	docstring for get_current_state_ids	2018-01-29 17:39:55 +00:00
Richard van der Hoff	d413a2ba98	Remove unused "event_type" param on state.get_current_state_ids ... this param doesn't seem to be used, and is a bit pointless anyway because it can easily be replicated by the caller. It is also horrible, because it changes the return type of the method.	2018-01-29 17:06:57 +00:00
Richard van der Hoff	b387ee17b6	Improve exception handling in persist_event ... 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs	2018-01-29 17:05:33 +00:00
Richard van der Hoff	03dd745fe2	Better logging when pushes fail	2018-01-29 15:49:06 +00:00
Richard van der Hoff	e051abd20b	add appid/device_display_name to to pusher logging	2018-01-29 15:04:16 +00:00
Richard van der Hoff	02ba118f81	Increase http conn pool size	2018-01-29 14:30:15 +00:00
Richard van der Hoff	46022025ea	Fix SQL for user search ... fix some syntax errors for user search when search_all_users is enabled fixes #2801, hopefully	2018-01-27 17:21:57 +00:00
Richard van der Hoff	4b0f06e99c	Merge pull request #2830 from matrix-org/rav/factor_out_get_conn ... Factor out get_db_conn to HomeServer base class	2018-01-27 17:56:25 +01:00
Neil Johnson	e98f0f9112	Merge pull request #2827 from matrix-org/fix_server_500_on_public_rooms_call_when_no_rooms_exist ... Fix server 500 on public rooms call when no rooms exist	2018-01-26 20:22:40 +00:00
Richard van der Hoff	25adde9a04	Factor out get_db_conn to HomeServer base class ... This function is identical to all subclasses, so we may as well push it up to the base class to reduce duplication (and make use of it in the tests)	2018-01-26 00:56:49 +00:00
Neil Johnson	73560237d6	add white space line	2018-01-26 00:15:10 +00:00
Neil Johnson	86c4f49a31	rather than try reconstruct the results object, better to guard against the xrange step argument being 0	2018-01-26 00:12:02 +00:00
Neil Johnson	f632083576	fix return type, should be a dict	2018-01-25 23:52:17 +00:00
Neil Johnson	6c6e197b0a	fix PEP8 violation	2018-01-25 23:47:46 +00:00
Neil Johnson	d02e43b15f	remove white space	2018-01-25 23:29:46 +00:00
Neil Johnson	349c739966	synapse 500s on a call to publicRooms in the case where the number of public rooms is zero, the specific cause is due to xrange trying to use a step value of zero, but if the total room number really is zero then it makes sense to just bail and save the extra processing	2018-01-25 23:28:44 +00:00
Matthew Hodgson	9a72b70630	fix thinko on 3pid whitelisting	2018-01-24 11:07:47 +01:00
Richard van der Hoff	25e2456ee7	Merge pull request #2816 from matrix-org/rav/metrics_comments ... Add some comments about the reactor tick time metric	2018-01-23 19:39:26 +00:00
Matthew Hodgson	d32385336f	add ?ts massaging for ASes (#2754) ... blindly implement ?ts for AS. untested	2018-01-23 09:59:06 +01:00
Richard van der Hoff	4528dd2443	Fix logging and add user_id	2018-01-22 20:15:42 +00:00
Richard van der Hoff	93efd7eb04	logging and debug for http pusher	2018-01-22 18:14:10 +00:00
Matthew Hodgson	ab9f844aaf	Add federation_domain_whitelist option (#2820) ... Add federation_domain_whitelist gives a way to restrict which domains your HS is allowed to federate with. useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network	2018-01-22 19:11:18 +01:00
Matthew Hodgson	d84f65255e	Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid ... add registrations_require_3pid and allow_local_3pids	2018-01-22 13:57:22 +00:00
Travis Ralston	a94d9b6b82	Appease the linter ... These are ids anyways, not mxc uris. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:49:46 -07:00
Travis Ralston	5552ed9a7f	Add an admin route to get all the media in a room ... This is intended to be used by administrators to monitor the media that is passing through their server, if they wish. Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-20 22:37:53 -07:00
Richard van der Hoff	2c8526cac7	Use a connection pool for the SimpleHttpClient ... In particular I hope this will help the pusher, which makes many requests to sygnal, and is currently negotiating SSL for each one.	2018-01-20 00:55:44 +00:00
Richard van der Hoff	87b7d72760	Add some comments about the reactor tick time metric	2018-01-19 23:51:04 +00:00
Matthew Hodgson	49fce04624	fix typo (thanks sytest)	2018-01-19 19:55:38 +00:00
Richard van der Hoff	b0d9e633ee	Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs ... Use the right path for url_preview thumbnails	2018-01-19 18:57:15 +00:00
Richard van der Hoff	ad7ec63d08	Use the right path for url_preview thumbnails ... This was introduced by #2627: we were overwriting the original media for url previews with the thumbnails :/ (fixes https://github.com/vector-im/riot-web/issues/6012, hopefully)	2018-01-19 18:29:39 +00:00
Matthew Hodgson	62d7d66ae5	oops, check all login types	2018-01-19 18:23:56 +00:00
Matthew Hodgson	8fe253f19b	fix PR nitpicking	2018-01-19 18:23:45 +00:00
Matthew Hodgson	293380bef7	trailing commas	2018-01-19 15:38:53 +00:00
Matthew Hodgson	447f4f0d5f	rewrite based on PR feedback: ... * [ ] split config options into allowed_local_3pids and registrations_require_3pid * [ ] simplify and comment logic for picking registration flows * [ ] fix docstring and move check_3pid_allowed into a new util module * [ ] use check_3pid_allowed everywhere @erikjohnston PTAL	2018-01-19 15:33:55 +00:00
Matthew Hodgson	9d332e0f79	fix up v1, and improve errors	2018-01-19 00:53:58 +00:00
Matthew Hodgson	0af58f14ee	fix pep8	2018-01-19 00:33:51 +00:00
Matthew Hodgson	28a6ccb49c	add registrations_require_3pid ... lets homeservers specify a whitelist for 3PIDs that users are allowed to associate with. Typically useful for stopping people from registering with non-work emails	2018-01-19 00:19:58 +00:00
Erik Johnston	cd871a3057	Fix storage provider bug introduced when renamed to store_local	2018-01-18 18:37:59 +00:00
Erik Johnston	8ff6726c0d	Merge pull request #2812 from matrix-org/erikj/media_storage_provider_config ... Make storage providers configurable	2018-01-18 18:33:57 +00:00
Erik Johnston	d69768348f	Fix passing wrong config to provider constructor	2018-01-18 17:14:05 +00:00
Erik Johnston	8e85220373	Remove duplicate directory test	2018-01-18 17:12:35 +00:00
Erik Johnston	3fe2bae857	Missing staticmethod	2018-01-18 17:11:45 +00:00
Erik Johnston	aae77da73f	Fixup comments	2018-01-18 17:11:29 +00:00
Erik Johnston	b6dc7044a9	Merge pull request #2804 from matrix-org/erikj/file_consumer ... Add decent impl of a FileConsumer	2018-01-18 16:31:33 +00:00
Erik Johnston	9a89dae8c5	Fix typo in thumbnail resource causing access times to be incorrect	2018-01-18 15:06:24 +00:00
Erik Johnston	0af5dc63a8	Make storage providers more configurable	2018-01-18 14:07:21 +00:00
Richard van der Hoff	d57765fc8a	Fix bugs in block metrics ... ... which I introduced in #2785	2018-01-18 12:24:42 +00:00
Erik Johnston	2cf6a7bc20	Use better file consumer	2018-01-18 12:00:46 +00:00
Erik Johnston	4a53f3a3e8	Ensure media is in local cache before thumbnailing	2018-01-18 12:00:46 +00:00
Erik Johnston	be0dfcd4a2	Do logcontexts correctly	2018-01-18 11:57:57 +00:00
Erik Johnston	1432f7ccd5	Move test stuff to tests	2018-01-18 11:57:57 +00:00
Erik Johnston	2f18a2647b	Make all fields private	2018-01-18 11:57:54 +00:00
Richard van der Hoff	d6af5512bb	Merge pull request #2809 from matrix-org/rav/metrics_errors ... better exception logging in callbackmetrics	2018-01-18 11:46:37 +00:00
Richard van der Hoff	ce236f8ac8	better exception logging in callbackmetrics ... when we fail to render a metric, give a clue as to which metric it was	2018-01-18 11:30:49 +00:00
Erik Johnston	dc519602ac	Ensure we registerProducer isn't called twice	2018-01-18 11:07:17 +00:00
Erik Johnston	17b54389fe	Fix _notify_empty typo	2018-01-18 11:05:34 +00:00
Erik Johnston	28b338ed9b	Move definition of paused_producer to __init__	2018-01-18 11:04:41 +00:00
Erik Johnston	a177325b49	Fix comments	2018-01-18 11:02:43 +00:00
Richard van der Hoff	36da256cc6	Merge pull request #2805 from matrix-org/rav/log_state_res ... Log room when doing state resolution	2018-01-17 18:05:04 +00:00
Richard van der Hoff	1224612a79	Log room when doing state resolution ... Mostly because it helps figure out what is prompting the resolution	2018-01-17 17:11:59 +00:00
Erik Johnston	bc67e7d260	Add decent impl of a FileConsumer ... Twisted core doesn't have a general purpose one, so we need to write one ourselves. Features: - All writing happens in background thread - Supports both push and pull producers - Push producers get paused if the consumer falls behind	2018-01-17 16:43:03 +00:00
Erik Johnston	a87006f9c7	Merge pull request #2783 from matrix-org/erikj/media_last_accessed ... Keep track of last access time for local media	2018-01-17 16:39:02 +00:00
Matthew Hodgson	06db5c4b76	Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql ... fix SQL when searching all users	2018-01-17 16:27:54 +00:00
Matthew Hodgson	2d9ab533f9	fix SQL when searching all users	2018-01-17 15:58:52 +00:00
Richard van der Hoff	390093d45e	Split resolve_events into two functions ... ... so that the return type doesn't depend on the arg types	2018-01-17 15:44:31 +00:00
Erik Johnston	2fb3a28c98	Remove lost comment	2018-01-17 14:59:44 +00:00
Richard van der Hoff	a7e4ff9cca	Merge pull request #2795 from matrix-org/rav/track_db_scheduling ... Track DB scheduling delay per-request	2018-01-17 14:29:37 +00:00
Richard van der Hoff	f884cfffb9	Merge pull request #2797 from matrix-org/rav/user_id_checking ... Sanity checking for user ids	2018-01-17 14:29:12 +00:00
Richard van der Hoff	a5213df1f7	Sanity checking for user ids ... Check the user_id passed to a couple of APIs for validity, to avoid "IndexError: list index out of range" exception which looks scary and results in a 500 rather than a more useful error. Fixes #1432, among other things	2018-01-17 14:28:54 +00:00
Richard van der Hoff	e8f7541d3f	Merge remote-tracking branch 'origin/develop' into rav/track_db_scheduling	2018-01-17 14:01:57 +00:00
Richard van der Hoff	fb6563b4be	Merge pull request #2793 from matrix-org/rav/db_txn_time_in_millis ... Track db txn time in millisecs	2018-01-17 13:52:42 +00:00
Richard van der Hoff	1954e867b4	Merge pull request #2796 from matrix-org/rav/fix_closed_connection_errors ... Fix 'NoneType' object has no attribute 'writeHeaders'	2018-01-17 13:52:24 +00:00
Richard van der Hoff	11ab2f56f5	Merge branch 'develop' into rav/fix_closed_connection_errors	2018-01-17 11:25:11 +00:00
Richard van der Hoff	0486a7814a	Merge branch 'develop' into rav/rework_run_interaction	2018-01-17 11:24:38 +00:00
Richard van der Hoff	90c14da992	Merge branch 'develop' into rav/db_txn_time_in_millis	2018-01-17 11:19:55 +00:00
Richard van der Hoff	38506773eb	Merge remote-tracking branch 'origin/develop' into rav/optimise_logging_context	2018-01-17 10:57:13 +00:00
Erik Johnston	300edc2348	Update last access time when thumbnails are viewed	2018-01-17 10:24:43 +00:00
Erik Johnston	05f98a2224	Keep track of last access time for local media	2018-01-17 10:24:43 +00:00
Erik Johnston	3cb2dabaad	Merge pull request #2789 from matrix-org/erikj/fix_thumbnails ... Fix thumbnailing remote files	2018-01-17 10:22:20 +00:00
Erik Johnston	d728c47142	Add docstring	2018-01-17 10:06:14 +00:00
Richard van der Hoff	936482d507	Fix 'NoneType' object has no attribute 'writeHeaders' ... Avoid throwing a (harmless) exception when we try to write an error response to an http request where the client has disconnected. This comes up as a CRITICAL error in the logs which tends to mislead people into thinking there's an actual problem	2018-01-16 17:58:16 +00:00
Richard van der Hoff	3d12d97415	Track DB scheduling delay per-request ... For each request, track the amount of time spent waiting for a db connection. This entails adding it to the LoggingContext and we may as well add metrics for it while we are passing.	2018-01-16 17:23:32 +00:00
Richard van der Hoff	0f5d2cc37c	Merge branch 'rav/db_txn_time_in_millis' into rav/track_db_scheduling	2018-01-16 17:09:15 +00:00
Richard van der Hoff	8615f19d20	rework runInteraction in terms of runConnection ... ... so that we can share the code	2018-01-16 17:08:29 +00:00
Matthew Hodgson	5e97ca7ee6	fix typo	2018-01-16 16:52:35 +00:00
Erik Johnston	d863f68cab	Use local vars	2018-01-16 16:24:15 +00:00
Erik Johnston	6368e5c0ab	Change _generate_thumbnails to take media_type	2018-01-16 16:17:38 +00:00
Erik Johnston	0a90d9ede4	Move setting of file_id up to caller	2018-01-16 16:03:05 +00:00
Richard van der Hoff	6324b65f08	Track db txn time in millisecs ... ... to reduce the amount of floating-point foo we do.	2018-01-16 15:53:18 +00:00
Richard van der Hoff	44a498418c	Optimise LoggingContext creation and copying ... It turns out that the only thing we use the __dict__ of LoggingContext for is `request`, and given we create lots of LoggingContexts and then copy them every time we do a db transaction or log line, using the __dict__ seems a bit redundant. Let's try to optimise things by making the request attribute explicit.	2018-01-16 15:49:42 +00:00
Erik Johnston	5dfc83704b	Fix typo	2018-01-16 14:32:56 +00:00
Richard van der Hoff	febdca4b37	Merge pull request #2785 from matrix-org/rav/reorganise_metrics_again ... Reorganise request and block metrics	2018-01-16 14:10:21 +00:00
Richard van der Hoff	f5f89fda21	Merge pull request #2790 from matrix-org/rav/preserve_event_logcontext_leak ... Fix a logcontext leak in persist_events	2018-01-16 14:09:13 +00:00
Erik Johnston	307f88dfb6	Fix up log lines	2018-01-16 13:53:52 +00:00
Richard van der Hoff	807e848f0f	Merge pull request #2787 from matrix-org/rav/worker_event_counts ... Metrics for events processed in appservice and fed sender	2018-01-16 13:14:09 +00:00
Richard van der Hoff	4a31a61ef9	Merge pull request #2786 from matrix-org/rav/rdata_metrics ... Metrics for number of RDATA commands received	2018-01-16 13:13:53 +00:00
Erik Johnston	9795b9ebb1	Correctly use server_name/file_id when generating/fetching remote thumbnails	2018-01-16 12:02:06 +00:00
Erik Johnston	c5b589f2e8	Log when we respond with 404	2018-01-16 12:01:40 +00:00
Richard van der Hoff	64ddec1bc0	Fix a logcontext leak in persist_events ... ObserveableDeferred expects its callbacks to be called without any logcontexts, whereas it turns out we were calling them with the logcontext of the request which initiated the persistence loop. It seems wrong that we are attributing work done in the persistence loop to the request that happened to initiate it, so let's solve this by dropping the logcontext for it. (I'm not sure this actually causes any real problems other than messages in the debug log, but let's clean it up anyway)	2018-01-16 11:47:36 +00:00
Erik Johnston	a4c5e4a645	Fix thumbnailing remote files	2018-01-16 11:37:50 +00:00
Erik Johnston	1159abbdd2	Merge pull request #2767 from matrix-org/erikj/media_storage_refactor ... Refactor MediaRepository to separate out storage	2018-01-16 10:23:50 +00:00
Richard van der Hoff	a027c2af8d	Metrics for events processed in appservice and fed sender ... More metrics I wished I'd had	2018-01-15 18:23:24 +00:00
Richard van der Hoff	5c3c32f16f	Metrics for number of RDATA commands received ... I found myself wishing we had this.	2018-01-15 17:45:55 +00:00
Richard van der Hoff	39f4e29d01	Reorganise request and block metrics ... In order to circumvent the number of duplicate foo:count metrics increasing without bounds, it's time for a rearrangement. The following are all deprecated, and replaced with synapse_util_metrics_block_count: synapse_util_metrics_block_timer:count synapse_util_metrics_block_ru_utime:count synapse_util_metrics_block_ru_stime:count synapse_util_metrics_block_db_txn_count:count synapse_util_metrics_block_db_txn_duration:count The following are all deprecated, and replaced with synapse_http_server_response_count: synapse_http_server_requests synapse_http_server_response_time:count synapse_http_server_response_ru_utime:count synapse_http_server_response_ru_stime:count synapse_http_server_response_db_txn_count:count synapse_http_server_response_db_txn_duration:count The following are renamed (the old metrics are kept for now, but deprecated): synapse_util_metrics_block_timer:total -> synapse_util_metrics_block_time_seconds synapse_util_metrics_block_ru_utime:total -> synapse_util_metrics_block_ru_utime_seconds synapse_util_metrics_block_ru_stime:total -> synapse_util_metrics_block_ru_stime_seconds synapse_util_metrics_block_db_txn_count:total -> synapse_util_metrics_block_db_txn_count synapse_util_metrics_block_db_txn_duration:total -> synapse_util_metrics_block_db_txn_duration_seconds synapse_http_server_response_time:total -> synapse_http_server_response_time_seconds synapse_http_server_response_ru_utime:total -> synapse_http_server_response_ru_utime_seconds synapse_http_server_response_ru_stime:total -> synapse_http_server_response_ru_stime_seconds synapse_http_server_response_db_txn_count:total -> synapse_http_server_response_db_txn_count synapse_http_server_response_db_txn_duration:total synapse_http_server_response_db_txn_duration_seconds	2018-01-15 17:09:44 +00:00
Richard van der Hoff	992018d1c0	mechanism to render metrics with alternative names	2018-01-15 17:04:39 +00:00
Richard van der Hoff	80fa610f9c	Add some comments to metrics classes	2018-01-15 16:52:52 +00:00
Richard van der Hoff	19d274085f	Make Counter render floats ... Prometheus handles all metrics as floats, and sometimes we store non-integer values in them (notably, durations in seconds), so let's render them as floats too. (Note that the standard client libraries also treat Counters as floats.)	2018-01-12 23:49:44 +00:00
Richard van der Hoff	21bf87a146	Reinstate media download on thumbnail request ... We need to actually download the remote media when we get a request for a thumbnail.	2018-01-12 15:38:06 +00:00
Erik Johnston	694f1c1b18	Fix up comments	2018-01-12 15:02:46 +00:00
Erik Johnston	e21370ba54	Correctly reraise exception	2018-01-12 14:44:02 +00:00
Erik Johnston	85a4d78213	Make Responder a context manager	2018-01-12 13:32:03 +00:00