45 Commits

Author	SHA1	Message	Date
Amber Brown	463b072b12	Move logging utilities out of the side drawer of util/ and into logging/ (#5606)	2019-07-04 00:07:04 +10:00
Amber Brown	32e7c9e7f2	Run Black. (#5482)	2019-06-20 19:32:02 +10:00
Erik Johnston	75538813fc	Fix background updates to handle redactions/rejections (#5352) ... * Fix background updates to handle redactions/rejections In background updates based on current state delta stream we need to handle that we may not have all the events (or at least that `get_events` may raise an exception).	2019-06-06 00:45:46 +10:00
Erik Johnston	5ac75fc9a2	Join against events to use its room_id index	2019-05-30 15:26:55 +01:00
Amber Brown	46c8f7a517	Implement the SHHS complexity API (#5216)	2019-05-30 01:47:16 +10:00
Richard van der Hoff	85d1e03b9d	Simplifications and comments in do_auth (#5227) ... I was staring at this function trying to figure out wtf it was actually doing. This is (hopefully) a non-functional refactor which makes it a bit clearer.	2019-05-23 11:17:42 +01:00
Amber Brown	4a30e4acb4	Room Statistics (#4338)	2019-05-21 11:36:50 -05:00
Erik Johnston	52ddc6c0ed	Update docstring with correct type ... Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>	2019-05-15 09:52:15 +01:00
Erik Johnston	4fb44fb5b9	Expose DataStore._get_events as get_events_as_list ... This is in preparation for reaction work which requires it.	2019-05-14 13:37:44 +01:00
Amber Brown	7efd1d87c2	Run black on the rest of the storage module (#4996)	2019-04-03 10:07:29 +01:00
Richard van der Hoff	54a87a7b08	Collect room-version variations into one place (#4969) ... Collect all the things that make room-versions different to one another into one place, so that it's easier to define new room versions.	2019-04-01 10:24:38 +01:00
Richard van der Hoff	e707e7b38d	Fix infinite loop when an event is redacted in a v3 room (#4535)	2019-01-31 15:34:17 +00:00
Amber Brown	3f189c902e	Fix flake8 (#4519)	2019-01-30 10:53:17 +00:00
Erik Johnston	c21b7cbc09	Update synapse/storage/events_worker.py	2019-01-29 21:53:48 +00:00
Erik Johnston	4db252c073	Check redaction state when event is pulled out of the database	2019-01-29 21:48:36 +00:00
Erik Johnston	6d23ec2111	Fix typo	2019-01-29 21:45:53 +00:00
Erik Johnston	b5d510ad64	Remove unused arg	2019-01-29 21:45:28 +00:00
Richard van der Hoff	82165eeb05	Update synapse/storage/events_worker.py ... Co-Authored-By: erikjohnston <erikj@jki.re>	2019-01-29 21:14:39 +00:00
Erik Johnston	7709d2bd16	Implement rechecking of redactions	2019-01-29 11:56:20 +00:00
Erik Johnston	678a92cb56	Replace missed usages of FrozenEvent	2019-01-25 10:32:30 +00:00
Erik Johnston	be6a7e47fa	Revert "Require event format version to parse or create events"	2019-01-25 10:23:51 +00:00
Erik Johnston	e8c9f15397	Replace missed usages of FrozenEvent	2019-01-24 11:14:07 +00:00
Erik Johnston	be1065af59	isort	2019-01-23 11:48:16 +00:00
Erik Johnston	c5a296b10c	Add support for persisting event format versions ... Currently we only have the one event format version defined, but this adds the necessary infrastructure to persist and fetch the format versions alongside the events. We specify the format version rather than the room version as: 1. We don't necessarily know the room version, existing events may be either v1 or v2. 2. We'd need to be careful to prevent/handle correctly if different events in the same room reported to be of different versions, which sounds annoying.	2019-01-23 11:30:01 +00:00
Amber Brown	14e4d4f4bf	Port storage/ to Python 3 (#3725)	2018-08-31 00:19:58 +10:00
Erik Johnston	96a9a29645	Pull in necessary stores in federation_reader	2018-08-06 15:23:57 +01:00
Erik Johnston	62ace05c45	Move necessary storage functions to worker classes	2018-08-06 15:23:38 +01:00
Richard van der Hoff	704c3e6239	Merge branch 'master' into develop	2018-08-02 15:43:30 +01:00
Richard van der Hoff	14fa9d4d92	Avoid extra db lookups ... Since we're about to look up the events themselves anyway, we can skip the extra db queries here.	2018-08-02 13:55:51 +01:00
Richard van der Hoff	667fba68f3	Run things as background processes ... This fixes #3518, and ensures that we get useful logs and metrics for lots of things that happen in the background. (There are certainly more things that happen in the background; these are just the common ones I've found running a single-process synapse locally).	2018-07-18 20:55:05 +01:00
Richard van der Hoff	c3c29aa196	Attempt to include db threads in cpu usage stats (#3496) ... Let's try to include time spent in the DB threads in the per-request/block cpu usage metrics.	2018-07-10 16:12:36 +01:00
Richard van der Hoff	e31e5dee38	Add CPU metrics for _fetch_event_list ... add a Measure block on _fetch_event_list, in the hope that we can better measure CPU usage here.	2018-07-09 18:15:54 +01:00
Amber Brown	49af402019	run isort	2018-07-09 16:09:20 +10:00
Amber Brown	6350bf925e	Attempt to be more performant on PyPy (#3462)	2018-06-28 14:49:57 +01:00
Richard van der Hoff	70e6501913	Merge pull request #3419 from matrix-org/rav/events_per_request ... Log number of events fetched from DB	2018-06-22 11:17:56 +01:00
Richard van der Hoff	0495fe0035	Indirect evt_count updates via method call ... so that we can stub it for the sentinel and not have a billion failing UTs	2018-06-22 10:42:28 +01:00
Amber Brown	77ac14b960	Pass around the reactor explicitly (#3385)	2018-06-22 09:37:10 +01:00
Richard van der Hoff	b088aafcae	Log number of events fetched from DB ... When we finish processing a request, log the number of events we fetched from the database to handle it. [I'm trying to figure out which requests are responsible for large amounts of event cache churn. It may turn out to be more helpful to add counts to the prometheus per-request/block metrics, but that is an extension to this code anyway.]	2018-06-21 06:15:03 +01:00
Adrian Tschira	17a70cf6e9	Misc. py3 fixes ... Signed-off-by: Adrian Tschira <nota@notafile.com>	2018-05-24 20:20:33 +02:00
Richard van der Hoff	2a13af23bc	Use run_in_background in preference to preserve_fn ... While I was going through uses of preserve_fn for other PRs, I converted places which only use the wrapped function once to use run_in_background, to avoid creating the function object.	2018-04-27 12:55:51 +01:00
Erik Johnston	415aeefd89	Format docstring	2018-04-12 12:07:09 +01:00
Erik Johnston	23a7f9d7f4	Doc we raise on unknown event	2018-04-12 11:20:51 +01:00
Erik Johnston	4dae4a97ed	Track last processed event received_ts	2018-04-11 14:27:09 +01:00
Erik Johnston	a8ce159be4	Replace some ujson with simplejson to make it work	2018-03-16 00:27:09 +00:00
Erik Johnston	eba93b05bf	Split EventsWorkerStore into separate file	2018-02-23 11:01:21 +00:00