974 Commits

Author	SHA1	Message	Date
Erik Johnston	227f8ef031	Split out _get_event_from_row back into defer and _txn version	2015-05-20 13:00:57 +01:00
Erik Johnston	2bc60c55af	Fix _get_backfill_events to return events in the correct order	2015-05-20 12:57:00 +01:00
Erik Johnston	9084cdd70f	Ensure event_results is a set	2015-05-19 16:34:31 +01:00
Erik Johnston	aa729349dd	Fix event_backwards_extrem insertion to ignore outliers	2015-05-19 15:27:00 +01:00
Mark Haines	99914ec9f8	Merge pull request #152 from matrix-org/notifier_performance ... Notifier performance	2015-05-18 17:49:59 +01:00
Erik Johnston	165eb2dbe6	Comments and shuffle of functions	2015-05-18 15:18:41 +01:00
Erik Johnston	65878a2319	Remove unused metric	2015-05-18 14:06:30 +01:00
Erik Johnston	4d1b6f4ad1	Remove rejected events if we don't want rejected events	2015-05-18 14:03:46 +01:00
Mark Haines	0b0033c40b	Merge branch 'develop' into notifier_performance	2015-05-18 13:50:01 +01:00
Erik Johnston	10f1bdb9a2	Move get_events functions to storage.events	2015-05-18 10:21:40 +01:00
Erik Johnston	c71176858b	Newline, remove debug logging	2015-05-18 10:11:14 +01:00
Erik Johnston	c3b37abdfd	PEP8	2015-05-15 16:59:58 +01:00
Erik Johnston	6c74fd62a0	Revert limiting of fetching, it didn't help perf.	2015-05-15 16:45:35 +01:00
Erik Johnston	9ff7f66a2b	init j	2015-05-15 16:36:03 +01:00
Erik Johnston	70f272f71c	Don't completely drain the list	2015-05-15 16:34:17 +01:00
Erik Johnston	8763dd80ef	Don't fetch prev_content for current_state	2015-05-15 15:33:01 +01:00
Erik Johnston	acb12cc811	Make store.get_current_state fetch events asyncly	2015-05-15 15:20:05 +01:00
Erik Johnston	d62dee7eae	Remove more debug logging	2015-05-15 15:06:37 +01:00
Erik Johnston	0f29cfabc3	Remove debug logging	2015-05-15 14:06:42 +01:00
Erik Johnston	e275a9c0d9	preserve log context	2015-05-15 11:54:51 +01:00
Erik Johnston	aa32bd38e4	Add a wait	2015-05-15 11:35:04 +01:00
Erik Johnston	372d4c6d7b	Srsly. Don't use closures. Baaaaaad	2015-05-15 11:26:00 +01:00
Erik Johnston	575ec91d82	Correctly pass through params	2015-05-15 11:15:10 +01:00
Erik Johnston	de01438a57	Sort out error handling	2015-05-15 11:00:50 +01:00
Erik Johnston	a2c4f3f150	Fix daedlock	2015-05-15 10:54:04 +01:00
Erik Johnston	1d566edb81	Remove race condition	2015-05-14 16:54:35 +01:00
Erik Johnston	ef3d8754f5	Call from right thread	2015-05-14 15:41:55 +01:00
Erik Johnston	142934084a	Count and loop	2015-05-14 15:40:21 +01:00
Erik Johnston	96c5b9f87c	Don't start up more fetch_events	2015-05-14 15:36:04 +01:00
Erik Johnston	7cd6a6f6cf	Awful idea for speeding up fetching of events	2015-05-14 15:34:02 +01:00
Erik Johnston	7f4105a5c9	Turn off preemptive transactions	2015-05-14 14:51:06 +01:00
Erik Johnston	f4d58deba1	PEP8	2015-05-14 14:45:42 +01:00
Erik Johnston	386b7330d2	Move from _base to events	2015-05-14 14:45:22 +01:00
Erik Johnston	7d6a1dae31	Jump out early	2015-05-14 14:27:58 +01:00
Erik Johnston	656223fbd3	Actually, we probably want to run this in a transaction	2015-05-14 14:26:35 +01:00
Erik Johnston	2f7f8e1c2b	Preemptively jump into a transaction if we ask for get_prev_content	2015-05-14 14:17:36 +01:00
Erik Johnston	e1e9f0c5b2	loop -> gatherResults	2015-05-14 13:58:49 +01:00
Erik Johnston	ab78a8926e	Err, we probably want a bigger limit	2015-05-14 13:47:16 +01:00
Erik Johnston	f6f902d459	Move fetching of events into their own transactions	2015-05-14 13:45:48 +01:00
Erik Johnston	cdb3757942	Refactor _get_events	2015-05-14 13:31:55 +01:00
Erik Johnston	36ea26c5c0	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf	2015-05-14 12:01:38 +01:00
David Baker	7c549dd557	Add ID generator for push_rules_enable to #resolve SYN-378	2015-05-14 11:44:03 +01:00
Mark Haines	899d4675dd	Merge branch 'notifier_unify' into notifier_performance	2015-05-14 11:36:44 +01:00
Erik Johnston	4f1d984e56	Add index on events	2015-05-13 17:22:26 +01:00
Erik Johnston	968b01a91a	Actually use async method	2015-05-13 17:02:46 +01:00
Erik Johnston	4071f29653	Fetch events from events_id in their own transactions	2015-05-13 16:59:41 +01:00
Erik Johnston	a988361aea	Typo	2015-05-13 15:44:15 +01:00
Erik Johnston	8888982db3	Don't insert None	2015-05-13 15:43:32 +01:00
Erik Johnston	cf706cc6ef	Don't return None	2015-05-13 15:31:25 +01:00
Erik Johnston	5971d240d4	Limit batch size	2015-05-13 15:26:49 +01:00
Erik Johnston	ca4f458787	Fetch events in bulk	2015-05-13 15:13:42 +01:00
Erik Johnston	6edff11a88	Don't fetch redaction and rejection stuff for each event, so we can use index only scan	2015-05-13 14:39:05 +01:00
Mark Haines	63878c0379	Don't bother checking for updates if the stream token hasn't advanced for a user	2015-05-13 13:42:21 +01:00
Erik Johnston	02590c3e1d	Temp turn off checking for rejections and redactions	2015-05-13 11:31:28 +01:00
Erik Johnston	619a21812b	defer.gatherResults loop	2015-05-13 11:29:03 +01:00
Erik Johnston	fec4485e28	Batch fetching of events for state groups	2015-05-13 11:22:42 +01:00
Erik Johnston	409bcc76bd	Load events for state group seperately	2015-05-13 11:13:31 +01:00
Erik Johnston	e122685978	You need to call contextmanager	2015-05-12 16:12:37 +01:00
Mark Haines	a6fb2aa2a5	Merge pull request #144 from matrix-org/erikj/logging_context ... Preserving logging contexts	2015-05-12 15:23:50 +01:00
Mark Haines	1fce36b111	Merge pull request #149 from matrix-org/erikj/backfill ... Backfill support	2015-05-12 15:20:32 +01:00
Erik Johnston	8b28209c60	Err, delete the right stuff	2015-05-12 15:02:53 +01:00
Erik Johnston	30c72d377e	Newlines	2015-05-12 14:47:40 +01:00
Erik Johnston	e4eddf9b36	We do actually want to delete rows out of event_backward_extremities	2015-05-12 14:47:23 +01:00
Erik Johnston	c1779a79bc	Fix up _handle_prev_events to not try to insert duplicate rows	2015-05-12 14:41:50 +01:00
Erik Johnston	74850d7f75	Do state groups persistence /after/ checking if we have already persisted the event	2015-05-12 14:14:58 +01:00
Erik Johnston	da6a7bbdde	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context	2015-05-12 13:10:42 +01:00
Mark Haines	5002056b16	SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes.	2015-05-12 11:20:40 +01:00
Erik Johnston	4df11b5039	Make get_current_token accept a direction parameter, which tells whether the source whether we want a token for going 'forwards' or 'backwards'	2015-05-12 10:28:10 +01:00
Erik Johnston	84e6b4001f	Initial hack at wiring together pagination and backfill	2015-05-11 18:01:31 +01:00
Erik Johnston	17653a5dfe	Move storage.stream._StreamToken to types.RoomStreamToken	2015-05-11 18:01:01 +01:00
Erik Johnston	b036596b75	Prefer to use _simple_*.	2015-05-11 11:24:01 +01:00
Erik Johnston	cd525c0f5a	push_rules table expects an 'id' field	2015-05-11 11:24:01 +01:00
Erik Johnston	22d2f498fa	Fix push rule bug: can't insert bool into small int column	2015-05-10 10:50:51 +01:00
Erik Johnston	476899295f	Change the way we do logging contexts so that they survive divergences	2015-05-08 16:32:18 +01:00
Mark Haines	1c7912751e	Drop the old table not the new table	2015-05-08 16:04:32 +01:00
Mark Haines	9d36eb4eab	Rename unique constraint	2015-05-08 16:01:55 +01:00
Mark Haines	b0f71db3ff	Remove unsigned	2015-05-08 15:59:51 +01:00
Mark Haines	84e1cacea4	Bump schema version	2015-05-08 15:58:14 +01:00
Mark Haines	6538d445e8	Make the timestamps in server_keys_json bigints	2015-05-08 15:55:17 +01:00
Erik Johnston	0c4ac271ca	Merge branch 'erikj/executemany' of github.com:matrix-org/synapse into erikj/SYN-371	2015-05-05 18:21:19 +01:00
Erik Johnston	0cf7e480b4	And use buffer(...) there as well	2015-05-05 18:20:01 +01:00
Erik Johnston	ed2584050f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany	2015-05-05 18:15:20 +01:00
Erik Johnston	977338a7af	Use buffer(...) when inserting into bytea column	2015-05-05 18:12:53 +01:00
Mark Haines	31049c4d72	Merge pull request #139 from matrix-org/bugs/SYN-369 ... Fix race with cache invalidation. SYN-369	2015-05-05 17:46:13 +01:00
Mark Haines	deb0237166	Add some doc-string	2015-05-05 17:45:11 +01:00
Mark Haines	d18f37e026	Collect the invalidate callbacks on the transaction object rather than passing around a separate list	2015-05-05 17:32:21 +01:00
Erik Johnston	9951542393	Add a comment about the zip(*[zip(sorted(...),...)])	2015-05-05 17:06:55 +01:00
Mark Haines	041b6cba61	SYN-369: Add comments to the sequence number logic in the cache	2015-05-05 16:32:44 +01:00
Mark Haines	63075118a5	Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access	2015-05-05 16:24:04 +01:00
Erik Johnston	531d7955fd	Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely	2015-05-05 16:12:28 +01:00
Mark Haines	bfa4a7f8b0	Invalidate the room_member cache if the current state events updates	2015-05-05 15:43:49 +01:00
Mark Haines	d0fece8d3c	Missing return for when the event was already persisted	2015-05-05 15:39:09 +01:00
Erik Johnston	bdcd7693c8	Fix indentation	2015-05-05 15:14:48 +01:00
Erik Johnston	43c2e8deae	Add support for using executemany	2015-05-05 15:13:25 +01:00
Erik Johnston	1692dc019d	Don't call 'encode_parameter' no-op	2015-05-05 15:00:30 +01:00
Mark Haines	a9aea68fd5	Invalidate the caches from the correct thread	2015-05-05 14:57:08 +01:00
Mark Haines	261d809a47	Sequence the modifications to the cache so that selects don't race with inserts	2015-05-05 14:13:50 +01:00
Erik Johnston	d9cc5de9e5	Correctly name transaction	2015-05-05 10:24:10 +01:00
Erik Johnston	b8940cd902	Remove some unused indexes	2015-05-01 16:14:25 +01:00
David Baker	eb9bd2d949	user_id now in user_threepids	2015-05-01 15:04:37 +01:00
David Baker	9eeb03c0dd	Don't use self.execute: it's designed for fetching stuff	2015-05-01 14:21:25 +01:00
David Baker	7693f24792	No id field on user	2015-05-01 13:55:42 +01:00
David Baker	92b20713d7	More missed get_user_by_id API changes	2015-05-01 13:45:54 +01:00
Erik Johnston	4011cf1c42	Cache latest_event_ids_in_room	2015-05-01 13:06:26 +01:00
Erik Johnston	657298cebd	Don't lock user_ips table for upsert.	2015-05-01 10:46:48 +01:00
Erik Johnston	fabb7acd45	Fix bug where we reconnected to the database on every query.	2015-05-01 10:24:24 +01:00
Erik Johnston	23c639ff32	Split a storage function in two so that we don't have to do extra work.	2015-05-01 10:17:19 +01:00
Erik Johnston	8be5284e91	Remove pointless join	2015-04-30 18:49:26 +01:00
Erik Johnston	503e4d3d52	Fix broken SQL	2015-04-30 18:44:47 +01:00
Erik Johnston	00718ae7a9	Need more yield	2015-04-30 18:43:39 +01:00
Erik Johnston	0465560c1a	Add missing param	2015-04-30 18:42:44 +01:00
Erik Johnston	61d05daab1	More join conditions	2015-04-30 18:41:05 +01:00
Erik Johnston	6ead27ddda	Add more conditions on JOINs to make postgres go a little faster.	2015-04-30 18:32:03 +01:00
Erik Johnston	f383d5a801	Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction	2015-04-30 17:12:52 +01:00
Erik Johnston	69d4063651	Add get_rooms_for_user cache	2015-04-30 16:47:51 +01:00
Erik Johnston	3c4c229788	Don't use sub queries, it makes postgres sad	2015-04-30 16:16:53 +01:00
Mark Haines	29400b45b9	SYN-367: Use upsert rather than insert_or_replace	2015-04-30 15:21:31 +01:00
Erik Johnston	2366d28780	Don't needlessly join on state_events	2015-04-30 14:02:06 +01:00
Erik Johnston	1d7702833d	Make simple query rather than long one and then throw away half the results	2015-04-30 10:16:12 +01:00
Erik Johnston	b1ca784aca	Correctly decode, for sqlite and postgres, rows from pushers table	2015-04-29 19:41:14 +01:00
Erik Johnston	4a9dc5b2f5	pushkey' are also bytes.	2015-04-29 19:27:02 +01:00
Erik Johnston	0ade2712d1	Typo	2015-04-29 19:17:25 +01:00
Erik Johnston	50f96f256f	Also remove yield from within lock in the other generator	2015-04-29 19:17:00 +01:00
Erik Johnston	d2d61a8288	Fix deadlock in id_generators. No idea why this was an actual deadlock.	2015-04-29 19:15:23 +01:00
Erik Johnston	3864b3a8e6	Actually return rows	2015-04-29 18:07:36 +01:00
Erik Johnston	0618978238	Typo, args wrong way round	2015-04-29 18:04:35 +01:00
Erik Johnston	09177f4f2e	Decode buffers in same thread	2015-04-29 18:03:42 +01:00
Erik Johnston	472be88674	We store pusher data as bytes	2015-04-29 17:43:46 +01:00
David Baker	12d381bd5d	Decode the data json in the storage layer (was moved but this part was missed)	2015-04-29 17:13:51 +01:00
David Baker	f8c30faf25	Oops, update the contraint too	2015-04-29 16:58:42 +01:00
David Baker	61cd5d9045	Be more postgressive	2015-04-29 16:57:14 +01:00
David Baker	fb95035a65	Be postgressive	2015-04-29 16:53:41 +01:00
David Baker	4669def000	Oops, forgot the schema delta file	2015-04-29 16:50:16 +01:00
Erik Johnston	0337eaf321	txn.execute doesn't return cursors	2015-04-29 16:43:39 +01:00
Erik Johnston	884fb88e28	txn.execute doesn't return cursors	2015-04-29 16:35:20 +01:00
Erik Johnston	d76c058eea	Fix invalid SQL to work in postgres land	2015-04-29 16:30:25 +01:00
Erik Johnston	c8ed9bd278	pushers table requires an access_token	2015-04-29 15:33:27 +01:00
Erik Johnston	970a9b9d2b	We can't use REPLACE when upgrading databases now we have postgres	2015-04-29 13:55:44 +01:00
Erik Johnston	64991b0c8b	Merge pull request #129 from matrix-org/key_distribution ... Key distribution v2	2015-04-29 13:34:38 +01:00
Mark Haines	e26a3d8d9e	bump database schema version	2015-04-29 13:32:32 +01:00
Mark Haines	a9549fdce3	Use bytea rather than BLOB	2015-04-29 13:16:09 +01:00
Mark Haines	4ad8b45155	Merge branch 'develop' into key_distribution ... Conflicts: synapse/config/homeserver.py	2015-04-29 13:15:14 +01:00
Erik Johnston	cd0864121b	Make postgres database error slightly more helpful	2015-04-29 12:12:25 +01:00
Erik Johnston	4932a7e2d9	Use __all__ instead of assert to stop pyflakes from warning about unused import in __init__	2015-04-29 12:12:25 +01:00
Erik Johnston	a08bf11138	Appease PEP8	2015-04-29 11:44:48 +01:00
Erik Johnston	204132a998	Check that postgres database has correct charset set	2015-04-29 11:42:28 +01:00
Erik Johnston	327ca883ec	Merge branch 'develop' of github.com:matrix-org/synapse into postgres	2015-04-28 13:39:42 +01:00
Erik Johnston	18f8247701	Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit	2015-04-28 12:41:33 +01:00
Mark Haines	9182f87664	Merge pull request #126 from matrix-org/csauth ... Client / Server Auth Refactor	2015-04-28 11:00:27 +01:00
Mark Haines	55e1bc8920	And don't bump the schema version unnecessarily	2015-04-28 10:54:15 +01:00
Mark Haines	b96c133034	Add server_keys.sql to the current delta rather than creating a new delta	2015-04-28 10:50:00 +01:00
Erik Johnston	8558e1ec73	Make get_max_token into inlineCallbacks so that the lock works.	2015-04-27 15:19:44 +01:00
Erik Johnston	6f8e2d517e	Merge branch 'develop' of github.com:matrix-org/synapse into postgres	2015-04-27 14:41:40 +01:00
Erik Johnston	2732be83d9	Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible.	2015-04-27 13:22:30 +01:00
Erik Johnston	e4c4664d73	Handle the fact that postgres databases can be restarted from under us	2015-04-27 12:40:49 +01:00
David Baker	8a5be236e0	pep8	2015-04-27 11:49:18 +01:00
David Baker	b02e1006b9	Run database check before daemonizing, at the cost of database hygiene.	2015-04-27 11:46:00 +01:00
David Baker	f8152f2708	rename db method to be more informative	2015-04-27 10:16:26 +01:00
David Baker	2f475bd5d5	pep8	2015-04-24 18:15:07 +01:00
David Baker	a7b51f4539	Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266	2015-04-24 18:11:21 +01:00
Mark Haines	eede182df7	Merge branch 'develop' into key_distribution	2015-04-24 10:35:49 +01:00
Mark Haines	4e2f8b8722	Copyright notices	2015-04-24 10:35:29 +01:00
Mark Haines	f30d47c876	Implement remote key lookup api	2015-04-22 14:21:08 +01:00
David Baker	91c8f828e1	pep8	2015-04-17 19:56:04 +01:00
David Baker	8db6832db8	Password reset, finally.	2015-04-17 19:53:47 +01:00
David Baker	117f35ac4a	Add endpoint to get threepids from server	2015-04-17 17:20:18 +01:00
David Baker	4eea5cf6c2	pep8	2015-04-17 16:46:45 +01:00
David Baker	f96ab9d18d	make add3pid servlet work	2015-04-17 16:44:49 +01:00
Erik Johnston	e3417bbbe0	Revert needless change to storage.profile	2015-04-17 16:13:14 +01:00
David Baker	cb03fafdf1	Merge branch 'develop' into csauth	2015-04-17 13:51:10 +01:00
Erik Johnston	0ec346d942	Add unique index to room_aliases, remove duplicates on upgrade. Convert some columns back to TEXT from bytea	2015-04-17 11:10:20 +01:00
Erik Johnston	e1c0970c11	PEP8	2015-04-16 11:18:45 +01:00
Erik Johnston	b8092fbc82	Go back to storing JSON in TEXT	2015-04-16 11:17:52 +01:00
Erik Johnston	bc9e69e160	Move encoding and decoding of JSON into storage layer	2015-04-16 11:01:09 +01:00
Erik Johnston	ffad75bd62	Remove mysql/maria support	2015-04-15 17:00:50 +01:00
Mark Haines	a429515bdd	Add methods for storing and retrieving the raw key json	2015-04-15 16:58:35 +01:00
Erik Johnston	cf04cedf21	Change full_schemas/11 to work with postgres	2015-04-15 16:53:47 +01:00
Erik Johnston	5b31afcbd1	Remove debug logging	2015-04-15 16:27:04 +01:00
Erik Johnston	6e91f14d09	Add missing yield	2015-04-15 16:25:07 +01:00
Erik Johnston	ed26e4012b	pushers table requires a unique id.	2015-04-15 16:24:14 +01:00
Erik Johnston	a5c72780e6	Don't pass in removed flag	2015-04-15 15:13:22 +01:00
Erik Johnston	4af32a2817	Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction	2015-04-15 14:51:21 +01:00
Erik Johnston	25d2914fba	Turn off persistance of sent_transactions.response_json	2015-04-15 14:09:35 +01:00
Erik Johnston	cce5d057d3	Add index on events (topological_ordering, stream_ordering) to help with ORDER BY clauses	2015-04-15 11:43:25 +01:00
Erik Johnston	a971fa9d58	Use try..finally in contextlib.contextmanager	2015-04-15 10:25:43 +01:00
Erik Johnston	ded4128965	Use True for True rather than 1	2015-04-15 10:24:24 +01:00
Erik Johnston	f9e12f79ca	Add missing yield in storage func	2015-04-15 10:24:07 +01:00
Erik Johnston	c756dfeb14	Correctly identify deadlocks	2015-04-15 10:23:42 +01:00
Mark Haines	32e14d8181	Return a sha256 fingerprint rather than the entire tls certificate	2015-04-14 19:10:09 +01:00
Erik Johnston	127fad17dd	Add postgres database engine	2015-04-14 14:50:29 +01:00
Erik Johnston	5a95cd4442	Rename user_ips.user -> user_id	2015-04-14 13:54:09 +01:00
Erik Johnston	58d8339966	Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.	2015-04-14 13:53:20 +01:00
Erik Johnston	3c741682e5	Correctly increment the _next_id initially	2015-04-14 09:54:44 +01:00
Erik Johnston	86fc9b617c	For backwards compat, make state_groups.id have a type of int, not varchar	2015-04-13 17:03:49 +01:00
Erik Johnston	93937b2b31	Remove duplicate rows	2015-04-10 18:47:03 +01:00
Erik Johnston	c5365dee56	Use case sensitive collations	2015-04-10 18:46:33 +01:00
Erik Johnston	4103b1c470	DROP indexes before recreating them	2015-04-10 18:46:09 +01:00
Erik Johnston	4d5b098626	Use LONGBLOB and TEXT for arbitary length rows	2015-04-10 18:45:09 +01:00
Erik Johnston	7ed2ec3061	Handle the fact that in sqlite binary data might be stored as unicode or bytes	2015-04-10 13:41:54 +01:00
Erik Johnston	ce797ad373	Bump schema version	2015-04-10 11:22:30 +01:00