2361 Commits

Author	SHA1	Message	Date
Erik Johnston	b811c98574	Remove failed deferreds from cache	2015-08-06 14:01:27 +01:00
Erik Johnston	433314cc34	Re-implement DEBUG_CACHES flag	2015-08-06 14:01:05 +01:00
Erik Johnston	f596ff402e	Merge branch 'erikj/cache_deferreds' into erikj/dictionary_cache	2015-08-06 13:37:56 +01:00
Erik Johnston	2efb93af52	Merge branch 'erikj/cached_keyword_args' into erikj/cache_deferreds	2015-08-06 13:35:28 +01:00
Erik Johnston	953dbd28a7	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/cached_keyword_args	2015-08-06 13:35:03 +01:00
Erik Johnston	7eea3e356f	Make @cached cache deferreds rather than the deferreds' values	2015-08-06 13:33:34 +01:00
Erik Johnston	3e1b77efc2	Merge branch 'erikj/cached_keyword_args' of github.com:matrix-org/synapse into erikj/dictionary_cache	2015-08-05 16:45:56 +01:00
Erik Johnston	b52b4a84ec	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/dictionary_cache	2015-08-05 15:41:20 +01:00
Erik Johnston	1e62a3d3a9	Up the cache size for 'get_joined_hosts_for_room' and 'get_users_in_room'	2015-08-05 15:40:40 +01:00
Erik Johnston	a89559d797	Use LRU cache by default	2015-08-05 15:39:47 +01:00
Erik Johnston	07507643cb	Use dictionary cache to do group -> state fetching	2015-08-05 15:11:42 +01:00
Erik Johnston	c67ba143fa	Move DictionaryCache	2015-08-04 15:58:28 +01:00
Erik Johnston	e7768e77f5	Add basic dictionary cache	2015-08-04 15:56:56 +01:00
David Baker	e124128542	Bump schema version	2015-08-04 14:50:31 +01:00
David Baker	c77048e12f	Add endpoint that proxies ID server request token and errors if the given email is in use on this Home Server.	2015-08-04 14:37:09 +01:00
Erik Johnston	413a4c289b	Add comment	2015-08-04 11:08:07 +01:00
Erik Johnston	4d6cb8814e	Speed up event filtering (for ACL) logic	2015-08-04 09:32:23 +01:00
Erik Johnston	39e21ea51c	Add support for using keyword arguments with cached functions	2015-07-27 13:57:29 +01:00
David Baker	20c0324e9c	Dodesn't seem to make any difference: guess it does work with the object reference	2015-07-21 16:21:37 -07:00
David Baker	cf7a40b08a	I think this was what was intended...	2015-07-21 16:08:00 -07:00
Mark Haines	3b5823c74d	s/take/claim/ for end to end key APIs	2015-07-20 18:23:54 +01:00
Matrix	4f973eb657	Up default cache size for _RoomStreamChangeCache	2015-07-18 19:07:33 +01:00
Erik Johnston	b6d4a4c6d8	Merge pull request #199 from matrix-org/erikj/receipts ... Implement read receipts.	2015-07-16 18:18:36 +01:00
Erik Johnston	baa55fb69e	Merge pull request #193 from matrix-org/erikj/bulk_persist_event ... Add bulk insert events API	2015-07-14 10:49:24 +01:00
Erik Johnston	4624d6035e	Docs	2015-07-14 10:19:07 +01:00
Erik Johnston	5989637f37	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts	2015-07-13 13:50:57 +01:00
Erik Johnston	e5991af629	Comments	2015-07-13 13:30:43 +01:00
Erik Johnston	17bb9a7eb9	Remove commented out code	2015-07-10 14:07:57 +01:00
Erik Johnston	532fcc997a	Merge pull request #196 from matrix-org/erikj/room_history ... Add ability to restrict room history.	2015-07-10 13:47:04 +01:00
Erik Johnston	0b3389bcd2	Merge pull request #194 from matrix-org/erikj/bulk_verify_sigs ... Implement bulk verify_signed_json API	2015-07-10 13:46:53 +01:00
Erik Johnston	0d7f0febf4	Uniquely name unique constraint	2015-07-10 13:43:03 +01:00
Erik Johnston	ed88720952	Handle error slightly better	2015-07-09 16:14:46 +01:00
Mark Haines	bf0d59ed30	Don't bother with a timeout for one time keys on the server.	2015-07-09 14:04:03 +01:00
Erik Johnston	1af188209a	Change format of receipts to allow inclusion of data	2015-07-09 11:39:30 +01:00
Mark Haines	8fb79eeea4	Only remove one time keys when new one time keys are added	2015-07-08 17:04:29 +01:00
Erik Johnston	304343f4d7	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts	2015-07-08 15:37:33 +01:00
Erik Johnston	af812b68dd	Add a cache to fetching of receipt streams	2015-07-08 15:35:00 +01:00
Erik Johnston	87311d1b8c	Hook up receipts to v1 initialSync	2015-07-08 11:02:04 +01:00
Erik Johnston	ca041d5526	Wire together receipts and the notifer/federation	2015-07-07 15:25:30 +01:00
Erik Johnston	716e426933	Fix various typos	2015-07-07 10:55:31 +01:00
Mark Haines	b5770f8947	Add store for client end to end keys	2015-07-06 18:46:47 +01:00
Erik Johnston	65e69dec8b	Don't explode if we don't recognize one of the event_ids in the backfill request	2015-07-06 09:33:03 +01:00
Erik Johnston	1a60545626	Add basic impl for room history ACL on GET /messages client API	2015-07-02 16:20:10 +01:00
Erik Johnston	ac78e60de6	Add stream_id index	2015-07-02 13:18:41 +01:00
Erik Johnston	80a61330ee	Add basic storage functions for handling of receipts	2015-07-01 17:19:12 +01:00
Erik Johnston	901f56fa63	Add tables for receipts	2015-06-30 15:29:47 +01:00
Erik Johnston	b5f55a1d85	Implement bulk verify_signed_json API	2015-06-26 10:39:34 +01:00
Erik Johnston	5130d80d79	Add bulk insert events API	2015-06-25 17:29:34 +01:00
Erik Johnston	6924852592	Batch SELECTs in _get_auth_chain_ids_txn	2015-06-23 11:01:04 +01:00
Erik Johnston	2eda996a63	Add a dummy.sql into delta/20 as pip isn't packinging the pushers.py	2015-06-13 12:21:58 +01:00
Erik Johnston	1e365e88bd	Bump schema version	2015-06-11 15:50:39 +01:00
David Baker	da84946de4	pep8	2015-06-04 16:43:45 +01:00
David Baker	63a7b3ad1e	Add script to (re)convert the pushers table to changing the unique key. Also give the python db upgrade scripts the database engine so they can convert parameter strings, and add *args **kwargs to the upgrade function so we can add more args in future and previous scripts will ignore them.	2015-06-04 16:16:01 +01:00
Erik Johnston	0d7d9c37b6	Add cache to get_state_groups	2015-06-03 14:45:55 +01:00
Erik Johnston	d8866d7277	Caches should be bound to instances. ... Before, caches were global and so different instances of the stores would share caches. This caused problems in the unit tests.	2015-06-03 14:45:17 +01:00
Erik Johnston	6e17463228	Don't explode if we don't have the event	2015-06-01 16:39:43 +01:00
Erik Johnston	a7b65bdedf	Add config option to turn off freezing events. Use new encode_json api and ujson.loads	2015-05-29 12:17:33 +01:00
Mark Haines	d9f60e8dc8	Merge pull request #163 from matrix-org/markjh/presence_list_cache ... Add a cache for the presence list	2015-05-22 17:02:23 +01:00
Mark Haines	0e42dfbe22	Merge pull request #164 from matrix-org/markjh/pusher_performance_2 ... Add a cache for get_push rules for user, fix cache invalidation	2015-05-22 17:01:56 +01:00
Mark Haines	1a9a9abcc7	Add a cache for getting the presence list for a user	2015-05-22 16:11:17 +01:00
Erik Johnston	74b7de83ec	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes	2015-05-22 16:10:42 +01:00
Mark Haines	36317f3dad	Merge pull request #156 from matrix-org/erikj/join_perf ... Make joining #matrix:matrix.org over federation quicker	2015-05-22 16:09:54 +01:00
Mark Haines	b6adfc59f5	Invalidate the get_latest_event_ids_in_room cache when deleting from event_forward_extremities	2015-05-22 13:01:03 +01:00
Mark Haines	a04cde613e	Add a cache for get_push rules for user, fix cache invalidation	2015-05-22 10:39:45 +01:00
Erik Johnston	4429e720ae	Merge branch 'master' of github.com:matrix-org/synapse into develop	2015-05-22 10:33:00 +01:00
Mark Haines	2043527b9b	Don't try to use a txn when not in one, remove spurious debug logging	2015-05-21 16:53:03 +01:00
Mark Haines	53447e9cd3	Add caches for things requested by the pushers	2015-05-21 16:41:39 +01:00
Mark Haines	d61ce3f670	Add a cache for get_current_state with state_key	2015-05-21 16:41:39 +01:00
Erik Johnston	a910984b58	Actually return something from lambda	2015-05-21 15:58:41 +01:00
Erik Johnston	e309b1045d	Sort backfill events	2015-05-21 15:57:35 +01:00
Erik Johnston	0180bfe4aa	Remove dead code	2015-05-21 15:53:41 +01:00
Erik Johnston	1f3d1d85a9	Only get non-state	2015-05-21 15:52:29 +01:00
Erik Johnston	39a3340f73	Skip events we've already seen	2015-05-21 15:48:56 +01:00
Erik Johnston	ae3bff3491	Correctly prepopulate queue	2015-05-21 15:46:07 +01:00
Erik Johnston	dc085ddf8c	Don't prepopulate event_results	2015-05-21 15:44:05 +01:00
Erik Johnston	73d23c6ae8	Don't readd things that are already in event_results	2015-05-21 15:40:22 +01:00
Erik Johnston	6189d8e54d	PriorityQueue gives lowest first	2015-05-21 15:38:08 +01:00
Erik Johnston	115ef3ddac	Correctly capture Queue.Empty exception	2015-05-21 15:37:43 +01:00
Erik Johnston	c2633907c5	Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes	2015-05-21 14:58:47 +01:00
Erik Johnston	ebfdd2eb5b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf	2015-05-21 14:54:52 +01:00
Erik Johnston	a551c5dad7	Merge pull request #155 from matrix-org/erikj/perf ... Bulk and batch retrieval of events.	2015-05-21 14:54:40 +01:00
Erik Johnston	27e4b45c06	s/for events/for requests for events/	2015-05-21 14:52:23 +01:00
Erik Johnston	ac5f2bf9db	s/for events/for requests for events/	2015-05-21 14:50:57 +01:00
Erik Johnston	80a167b1f0	Add comments	2015-05-21 11:19:04 +01:00
Erik Johnston	9118a92862	Split up _get_events into defer and txn versions	2015-05-20 13:27:16 +01:00
Erik Johnston	ab45e12d31	Make not return a deferred _get_event_from_row_txn	2015-05-20 13:07:19 +01:00
Erik Johnston	f407cbd2f1	PEP8	2015-05-20 13:02:01 +01:00
Erik Johnston	227f8ef031	Split out _get_event_from_row back into defer and _txn version	2015-05-20 13:00:57 +01:00
Erik Johnston	2bc60c55af	Fix _get_backfill_events to return events in the correct order	2015-05-20 12:57:00 +01:00
Erik Johnston	9084cdd70f	Ensure event_results is a set	2015-05-19 16:34:31 +01:00
Erik Johnston	aa729349dd	Fix event_backwards_extrem insertion to ignore outliers	2015-05-19 15:27:00 +01:00
Mark Haines	99914ec9f8	Merge pull request #152 from matrix-org/notifier_performance ... Notifier performance	2015-05-18 17:49:59 +01:00
Erik Johnston	165eb2dbe6	Comments and shuffle of functions	2015-05-18 15:18:41 +01:00
Erik Johnston	65878a2319	Remove unused metric	2015-05-18 14:06:30 +01:00
Erik Johnston	4d1b6f4ad1	Remove rejected events if we don't want rejected events	2015-05-18 14:03:46 +01:00
Mark Haines	0b0033c40b	Merge branch 'develop' into notifier_performance	2015-05-18 13:50:01 +01:00
Erik Johnston	10f1bdb9a2	Move get_events functions to storage.events	2015-05-18 10:21:40 +01:00
Erik Johnston	c71176858b	Newline, remove debug logging	2015-05-18 10:11:14 +01:00
Erik Johnston	c3b37abdfd	PEP8	2015-05-15 16:59:58 +01:00
Erik Johnston	6c74fd62a0	Revert limiting of fetching, it didn't help perf.	2015-05-15 16:45:35 +01:00
Erik Johnston	9ff7f66a2b	init j	2015-05-15 16:36:03 +01:00
Erik Johnston	70f272f71c	Don't completely drain the list	2015-05-15 16:34:17 +01:00
Erik Johnston	8763dd80ef	Don't fetch prev_content for current_state	2015-05-15 15:33:01 +01:00
Erik Johnston	acb12cc811	Make store.get_current_state fetch events asyncly	2015-05-15 15:20:05 +01:00
Erik Johnston	d62dee7eae	Remove more debug logging	2015-05-15 15:06:37 +01:00
Erik Johnston	0f29cfabc3	Remove debug logging	2015-05-15 14:06:42 +01:00
Erik Johnston	e275a9c0d9	preserve log context	2015-05-15 11:54:51 +01:00
Erik Johnston	aa32bd38e4	Add a wait	2015-05-15 11:35:04 +01:00
Erik Johnston	372d4c6d7b	Srsly. Don't use closures. Baaaaaad	2015-05-15 11:26:00 +01:00
Erik Johnston	575ec91d82	Correctly pass through params	2015-05-15 11:15:10 +01:00
Erik Johnston	de01438a57	Sort out error handling	2015-05-15 11:00:50 +01:00
Erik Johnston	a2c4f3f150	Fix daedlock	2015-05-15 10:54:04 +01:00
Erik Johnston	1d566edb81	Remove race condition	2015-05-14 16:54:35 +01:00
Erik Johnston	ef3d8754f5	Call from right thread	2015-05-14 15:41:55 +01:00
Erik Johnston	142934084a	Count and loop	2015-05-14 15:40:21 +01:00
Erik Johnston	96c5b9f87c	Don't start up more fetch_events	2015-05-14 15:36:04 +01:00
Erik Johnston	7cd6a6f6cf	Awful idea for speeding up fetching of events	2015-05-14 15:34:02 +01:00
Erik Johnston	7f4105a5c9	Turn off preemptive transactions	2015-05-14 14:51:06 +01:00
Erik Johnston	f4d58deba1	PEP8	2015-05-14 14:45:42 +01:00
Erik Johnston	386b7330d2	Move from _base to events	2015-05-14 14:45:22 +01:00
Erik Johnston	7d6a1dae31	Jump out early	2015-05-14 14:27:58 +01:00
Erik Johnston	656223fbd3	Actually, we probably want to run this in a transaction	2015-05-14 14:26:35 +01:00
Erik Johnston	2f7f8e1c2b	Preemptively jump into a transaction if we ask for get_prev_content	2015-05-14 14:17:36 +01:00
Erik Johnston	e1e9f0c5b2	loop -> gatherResults	2015-05-14 13:58:49 +01:00
Erik Johnston	ab78a8926e	Err, we probably want a bigger limit	2015-05-14 13:47:16 +01:00
Erik Johnston	f6f902d459	Move fetching of events into their own transactions	2015-05-14 13:45:48 +01:00
Erik Johnston	cdb3757942	Refactor _get_events	2015-05-14 13:31:55 +01:00
Erik Johnston	36ea26c5c0	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf	2015-05-14 12:01:38 +01:00
David Baker	7c549dd557	Add ID generator for push_rules_enable to #resolve SYN-378	2015-05-14 11:44:03 +01:00
Mark Haines	899d4675dd	Merge branch 'notifier_unify' into notifier_performance	2015-05-14 11:36:44 +01:00
Erik Johnston	4f1d984e56	Add index on events	2015-05-13 17:22:26 +01:00
Erik Johnston	968b01a91a	Actually use async method	2015-05-13 17:02:46 +01:00
Erik Johnston	4071f29653	Fetch events from events_id in their own transactions	2015-05-13 16:59:41 +01:00
Erik Johnston	a988361aea	Typo	2015-05-13 15:44:15 +01:00
Erik Johnston	8888982db3	Don't insert None	2015-05-13 15:43:32 +01:00
Erik Johnston	cf706cc6ef	Don't return None	2015-05-13 15:31:25 +01:00
Erik Johnston	5971d240d4	Limit batch size	2015-05-13 15:26:49 +01:00
Erik Johnston	ca4f458787	Fetch events in bulk	2015-05-13 15:13:42 +01:00
Erik Johnston	6edff11a88	Don't fetch redaction and rejection stuff for each event, so we can use index only scan	2015-05-13 14:39:05 +01:00
Mark Haines	63878c0379	Don't bother checking for updates if the stream token hasn't advanced for a user	2015-05-13 13:42:21 +01:00
Erik Johnston	02590c3e1d	Temp turn off checking for rejections and redactions	2015-05-13 11:31:28 +01:00
Erik Johnston	619a21812b	defer.gatherResults loop	2015-05-13 11:29:03 +01:00
Erik Johnston	fec4485e28	Batch fetching of events for state groups	2015-05-13 11:22:42 +01:00
Erik Johnston	409bcc76bd	Load events for state group seperately	2015-05-13 11:13:31 +01:00
Erik Johnston	e122685978	You need to call contextmanager	2015-05-12 16:12:37 +01:00
Mark Haines	a6fb2aa2a5	Merge pull request #144 from matrix-org/erikj/logging_context ... Preserving logging contexts	2015-05-12 15:23:50 +01:00
Mark Haines	1fce36b111	Merge pull request #149 from matrix-org/erikj/backfill ... Backfill support	2015-05-12 15:20:32 +01:00
Erik Johnston	8b28209c60	Err, delete the right stuff	2015-05-12 15:02:53 +01:00
Erik Johnston	30c72d377e	Newlines	2015-05-12 14:47:40 +01:00
Erik Johnston	e4eddf9b36	We do actually want to delete rows out of event_backward_extremities	2015-05-12 14:47:23 +01:00
Erik Johnston	c1779a79bc	Fix up _handle_prev_events to not try to insert duplicate rows	2015-05-12 14:41:50 +01:00
Erik Johnston	74850d7f75	Do state groups persistence /after/ checking if we have already persisted the event	2015-05-12 14:14:58 +01:00
Erik Johnston	da6a7bbdde	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context	2015-05-12 13:10:42 +01:00
Mark Haines	5002056b16	SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes.	2015-05-12 11:20:40 +01:00
Erik Johnston	4df11b5039	Make get_current_token accept a direction parameter, which tells whether the source whether we want a token for going 'forwards' or 'backwards'	2015-05-12 10:28:10 +01:00
Erik Johnston	84e6b4001f	Initial hack at wiring together pagination and backfill	2015-05-11 18:01:31 +01:00
Erik Johnston	17653a5dfe	Move storage.stream._StreamToken to types.RoomStreamToken	2015-05-11 18:01:01 +01:00
Erik Johnston	b036596b75	Prefer to use _simple_*.	2015-05-11 11:24:01 +01:00
Erik Johnston	cd525c0f5a	push_rules table expects an 'id' field	2015-05-11 11:24:01 +01:00
Erik Johnston	22d2f498fa	Fix push rule bug: can't insert bool into small int column	2015-05-10 10:50:51 +01:00
Erik Johnston	476899295f	Change the way we do logging contexts so that they survive divergences	2015-05-08 16:32:18 +01:00
Mark Haines	1c7912751e	Drop the old table not the new table	2015-05-08 16:04:32 +01:00
Mark Haines	9d36eb4eab	Rename unique constraint	2015-05-08 16:01:55 +01:00
Mark Haines	b0f71db3ff	Remove unsigned	2015-05-08 15:59:51 +01:00
Mark Haines	84e1cacea4	Bump schema version	2015-05-08 15:58:14 +01:00
Mark Haines	6538d445e8	Make the timestamps in server_keys_json bigints	2015-05-08 15:55:17 +01:00
Erik Johnston	0c4ac271ca	Merge branch 'erikj/executemany' of github.com:matrix-org/synapse into erikj/SYN-371	2015-05-05 18:21:19 +01:00
Erik Johnston	0cf7e480b4	And use buffer(...) there as well	2015-05-05 18:20:01 +01:00
Erik Johnston	ed2584050f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany	2015-05-05 18:15:20 +01:00
Erik Johnston	977338a7af	Use buffer(...) when inserting into bytea column	2015-05-05 18:12:53 +01:00
Mark Haines	31049c4d72	Merge pull request #139 from matrix-org/bugs/SYN-369 ... Fix race with cache invalidation. SYN-369	2015-05-05 17:46:13 +01:00
Mark Haines	deb0237166	Add some doc-string	2015-05-05 17:45:11 +01:00
Mark Haines	d18f37e026	Collect the invalidate callbacks on the transaction object rather than passing around a separate list	2015-05-05 17:32:21 +01:00
Erik Johnston	9951542393	Add a comment about the zip(*[zip(sorted(...),...)])	2015-05-05 17:06:55 +01:00
Mark Haines	041b6cba61	SYN-369: Add comments to the sequence number logic in the cache	2015-05-05 16:32:44 +01:00
Mark Haines	63075118a5	Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access	2015-05-05 16:24:04 +01:00
Erik Johnston	531d7955fd	Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely	2015-05-05 16:12:28 +01:00
Mark Haines	bfa4a7f8b0	Invalidate the room_member cache if the current state events updates	2015-05-05 15:43:49 +01:00
Mark Haines	d0fece8d3c	Missing return for when the event was already persisted	2015-05-05 15:39:09 +01:00
Erik Johnston	bdcd7693c8	Fix indentation	2015-05-05 15:14:48 +01:00
Erik Johnston	43c2e8deae	Add support for using executemany	2015-05-05 15:13:25 +01:00
Erik Johnston	1692dc019d	Don't call 'encode_parameter' no-op	2015-05-05 15:00:30 +01:00
Mark Haines	a9aea68fd5	Invalidate the caches from the correct thread	2015-05-05 14:57:08 +01:00
Mark Haines	261d809a47	Sequence the modifications to the cache so that selects don't race with inserts	2015-05-05 14:13:50 +01:00
Erik Johnston	d9cc5de9e5	Correctly name transaction	2015-05-05 10:24:10 +01:00
Erik Johnston	b8940cd902	Remove some unused indexes	2015-05-01 16:14:25 +01:00
David Baker	eb9bd2d949	user_id now in user_threepids	2015-05-01 15:04:37 +01:00
David Baker	9eeb03c0dd	Don't use self.execute: it's designed for fetching stuff	2015-05-01 14:21:25 +01:00
David Baker	7693f24792	No id field on user	2015-05-01 13:55:42 +01:00
David Baker	92b20713d7	More missed get_user_by_id API changes	2015-05-01 13:45:54 +01:00
Erik Johnston	4011cf1c42	Cache latest_event_ids_in_room	2015-05-01 13:06:26 +01:00
Erik Johnston	657298cebd	Don't lock user_ips table for upsert.	2015-05-01 10:46:48 +01:00
Erik Johnston	fabb7acd45	Fix bug where we reconnected to the database on every query.	2015-05-01 10:24:24 +01:00
Erik Johnston	23c639ff32	Split a storage function in two so that we don't have to do extra work.	2015-05-01 10:17:19 +01:00
Erik Johnston	8be5284e91	Remove pointless join	2015-04-30 18:49:26 +01:00
Erik Johnston	503e4d3d52	Fix broken SQL	2015-04-30 18:44:47 +01:00
Erik Johnston	00718ae7a9	Need more yield	2015-04-30 18:43:39 +01:00
Erik Johnston	0465560c1a	Add missing param	2015-04-30 18:42:44 +01:00
Erik Johnston	61d05daab1	More join conditions	2015-04-30 18:41:05 +01:00
Erik Johnston	6ead27ddda	Add more conditions on JOINs to make postgres go a little faster.	2015-04-30 18:32:03 +01:00
Erik Johnston	f383d5a801	Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction	2015-04-30 17:12:52 +01:00
Erik Johnston	69d4063651	Add get_rooms_for_user cache	2015-04-30 16:47:51 +01:00
Erik Johnston	3c4c229788	Don't use sub queries, it makes postgres sad	2015-04-30 16:16:53 +01:00
Mark Haines	29400b45b9	SYN-367: Use upsert rather than insert_or_replace	2015-04-30 15:21:31 +01:00
Erik Johnston	2366d28780	Don't needlessly join on state_events	2015-04-30 14:02:06 +01:00
Erik Johnston	1d7702833d	Make simple query rather than long one and then throw away half the results	2015-04-30 10:16:12 +01:00
Erik Johnston	b1ca784aca	Correctly decode, for sqlite and postgres, rows from pushers table	2015-04-29 19:41:14 +01:00
Erik Johnston	4a9dc5b2f5	pushkey' are also bytes.	2015-04-29 19:27:02 +01:00
Erik Johnston	0ade2712d1	Typo	2015-04-29 19:17:25 +01:00
Erik Johnston	50f96f256f	Also remove yield from within lock in the other generator	2015-04-29 19:17:00 +01:00
Erik Johnston	d2d61a8288	Fix deadlock in id_generators. No idea why this was an actual deadlock.	2015-04-29 19:15:23 +01:00
Erik Johnston	3864b3a8e6	Actually return rows	2015-04-29 18:07:36 +01:00
Erik Johnston	0618978238	Typo, args wrong way round	2015-04-29 18:04:35 +01:00
Erik Johnston	09177f4f2e	Decode buffers in same thread	2015-04-29 18:03:42 +01:00
Erik Johnston	472be88674	We store pusher data as bytes	2015-04-29 17:43:46 +01:00
David Baker	12d381bd5d	Decode the data json in the storage layer (was moved but this part was missed)	2015-04-29 17:13:51 +01:00
David Baker	f8c30faf25	Oops, update the contraint too	2015-04-29 16:58:42 +01:00
David Baker	61cd5d9045	Be more postgressive	2015-04-29 16:57:14 +01:00
David Baker	fb95035a65	Be postgressive	2015-04-29 16:53:41 +01:00
David Baker	4669def000	Oops, forgot the schema delta file	2015-04-29 16:50:16 +01:00
Erik Johnston	0337eaf321	txn.execute doesn't return cursors	2015-04-29 16:43:39 +01:00
Erik Johnston	884fb88e28	txn.execute doesn't return cursors	2015-04-29 16:35:20 +01:00
Erik Johnston	d76c058eea	Fix invalid SQL to work in postgres land	2015-04-29 16:30:25 +01:00
Erik Johnston	c8ed9bd278	pushers table requires an access_token	2015-04-29 15:33:27 +01:00
Erik Johnston	970a9b9d2b	We can't use REPLACE when upgrading databases now we have postgres	2015-04-29 13:55:44 +01:00
Erik Johnston	64991b0c8b	Merge pull request #129 from matrix-org/key_distribution ... Key distribution v2	2015-04-29 13:34:38 +01:00
Mark Haines	e26a3d8d9e	bump database schema version	2015-04-29 13:32:32 +01:00
Mark Haines	a9549fdce3	Use bytea rather than BLOB	2015-04-29 13:16:09 +01:00
Mark Haines	4ad8b45155	Merge branch 'develop' into key_distribution ... Conflicts: synapse/config/homeserver.py	2015-04-29 13:15:14 +01:00
Erik Johnston	cd0864121b	Make postgres database error slightly more helpful	2015-04-29 12:12:25 +01:00
Erik Johnston	4932a7e2d9	Use __all__ instead of assert to stop pyflakes from warning about unused import in __init__	2015-04-29 12:12:25 +01:00
Erik Johnston	a08bf11138	Appease PEP8	2015-04-29 11:44:48 +01:00
Erik Johnston	204132a998	Check that postgres database has correct charset set	2015-04-29 11:42:28 +01:00
Erik Johnston	327ca883ec	Merge branch 'develop' of github.com:matrix-org/synapse into postgres	2015-04-28 13:39:42 +01:00
Erik Johnston	18f8247701	Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit	2015-04-28 12:41:33 +01:00
Mark Haines	9182f87664	Merge pull request #126 from matrix-org/csauth ... Client / Server Auth Refactor	2015-04-28 11:00:27 +01:00
Mark Haines	55e1bc8920	And don't bump the schema version unnecessarily	2015-04-28 10:54:15 +01:00
Mark Haines	b96c133034	Add server_keys.sql to the current delta rather than creating a new delta	2015-04-28 10:50:00 +01:00
Erik Johnston	8558e1ec73	Make get_max_token into inlineCallbacks so that the lock works.	2015-04-27 15:19:44 +01:00
Erik Johnston	6f8e2d517e	Merge branch 'develop' of github.com:matrix-org/synapse into postgres	2015-04-27 14:41:40 +01:00
Erik Johnston	2732be83d9	Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible.	2015-04-27 13:22:30 +01:00
Erik Johnston	e4c4664d73	Handle the fact that postgres databases can be restarted from under us	2015-04-27 12:40:49 +01:00
David Baker	8a5be236e0	pep8	2015-04-27 11:49:18 +01:00
David Baker	b02e1006b9	Run database check before daemonizing, at the cost of database hygiene.	2015-04-27 11:46:00 +01:00
David Baker	f8152f2708	rename db method to be more informative	2015-04-27 10:16:26 +01:00
David Baker	2f475bd5d5	pep8	2015-04-24 18:15:07 +01:00
David Baker	a7b51f4539	Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266	2015-04-24 18:11:21 +01:00
Mark Haines	eede182df7	Merge branch 'develop' into key_distribution	2015-04-24 10:35:49 +01:00
Mark Haines	4e2f8b8722	Copyright notices	2015-04-24 10:35:29 +01:00
Mark Haines	f30d47c876	Implement remote key lookup api	2015-04-22 14:21:08 +01:00
David Baker	91c8f828e1	pep8	2015-04-17 19:56:04 +01:00
David Baker	8db6832db8	Password reset, finally.	2015-04-17 19:53:47 +01:00
David Baker	117f35ac4a	Add endpoint to get threepids from server	2015-04-17 17:20:18 +01:00
David Baker	4eea5cf6c2	pep8	2015-04-17 16:46:45 +01:00
David Baker	f96ab9d18d	make add3pid servlet work	2015-04-17 16:44:49 +01:00
Erik Johnston	e3417bbbe0	Revert needless change to storage.profile	2015-04-17 16:13:14 +01:00
David Baker	cb03fafdf1	Merge branch 'develop' into csauth	2015-04-17 13:51:10 +01:00
Erik Johnston	0ec346d942	Add unique index to room_aliases, remove duplicates on upgrade. Convert some columns back to TEXT from bytea	2015-04-17 11:10:20 +01:00
Erik Johnston	e1c0970c11	PEP8	2015-04-16 11:18:45 +01:00
Erik Johnston	b8092fbc82	Go back to storing JSON in TEXT	2015-04-16 11:17:52 +01:00
Erik Johnston	bc9e69e160	Move encoding and decoding of JSON into storage layer	2015-04-16 11:01:09 +01:00
Erik Johnston	ffad75bd62	Remove mysql/maria support	2015-04-15 17:00:50 +01:00
Mark Haines	a429515bdd	Add methods for storing and retrieving the raw key json	2015-04-15 16:58:35 +01:00
Erik Johnston	cf04cedf21	Change full_schemas/11 to work with postgres	2015-04-15 16:53:47 +01:00
Erik Johnston	5b31afcbd1	Remove debug logging	2015-04-15 16:27:04 +01:00
Erik Johnston	6e91f14d09	Add missing yield	2015-04-15 16:25:07 +01:00
Erik Johnston	ed26e4012b	pushers table requires a unique id.	2015-04-15 16:24:14 +01:00
Erik Johnston	a5c72780e6	Don't pass in removed flag	2015-04-15 15:13:22 +01:00
Erik Johnston	4af32a2817	Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction	2015-04-15 14:51:21 +01:00
Erik Johnston	25d2914fba	Turn off persistance of sent_transactions.response_json	2015-04-15 14:09:35 +01:00
Erik Johnston	cce5d057d3	Add index on events (topological_ordering, stream_ordering) to help with ORDER BY clauses	2015-04-15 11:43:25 +01:00
Erik Johnston	a971fa9d58	Use try..finally in contextlib.contextmanager	2015-04-15 10:25:43 +01:00
Erik Johnston	ded4128965	Use True for True rather than 1	2015-04-15 10:24:24 +01:00
Erik Johnston	f9e12f79ca	Add missing yield in storage func	2015-04-15 10:24:07 +01:00
Erik Johnston	c756dfeb14	Correctly identify deadlocks	2015-04-15 10:23:42 +01:00
Mark Haines	32e14d8181	Return a sha256 fingerprint rather than the entire tls certificate	2015-04-14 19:10:09 +01:00
Erik Johnston	127fad17dd	Add postgres database engine	2015-04-14 14:50:29 +01:00
Erik Johnston	5a95cd4442	Rename user_ips.user -> user_id	2015-04-14 13:54:09 +01:00
Erik Johnston	58d8339966	Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS.	2015-04-14 13:53:20 +01:00
Erik Johnston	3c741682e5	Correctly increment the _next_id initially	2015-04-14 09:54:44 +01:00
Erik Johnston	86fc9b617c	For backwards compat, make state_groups.id have a type of int, not varchar	2015-04-13 17:03:49 +01:00
Erik Johnston	93937b2b31	Remove duplicate rows	2015-04-10 18:47:03 +01:00
Erik Johnston	c5365dee56	Use case sensitive collations	2015-04-10 18:46:33 +01:00
Erik Johnston	4103b1c470	DROP indexes before recreating them	2015-04-10 18:46:09 +01:00
Erik Johnston	4d5b098626	Use LONGBLOB and TEXT for arbitary length rows	2015-04-10 18:45:09 +01:00
Erik Johnston	7ed2ec3061	Handle the fact that in sqlite binary data might be stored as unicode or bytes	2015-04-10 13:41:54 +01:00
Erik Johnston	ce797ad373	Bump schema version	2015-04-10 11:22:30 +01:00
Erik Johnston	7e863c51e6	Use unsigned bigint	2015-04-10 11:22:04 +01:00
Erik Johnston	0f12772e32	SQLite wants INTEGER and not BIGINT for primary keys	2015-04-10 11:16:09 +01:00
Erik Johnston	d5d4281647	Update full_schemas/16 to match delta files. Add delta/16 scripts	2015-04-10 10:59:46 +01:00
Erik Johnston	cda4a6f93f	Revert non-trivial changes to upgrade scripts	2015-04-10 10:19:50 +01:00
Erik Johnston	e2722f58ee	Fix schema again	2015-04-10 10:16:29 +01:00
Erik Johnston	a1665c5094	Revert non-trivial schema changes and move them to a new schema version.	2015-04-10 10:05:44 +01:00
Erik Johnston	2ded344620	Remove unused import	2015-04-09 13:46:06 +01:00
Erik Johnston	9707acfc40	Remove spurious spaces	2015-04-09 13:45:20 +01:00
Erik Johnston	8bf285e082	Merge branch 'develop' of github.com:matrix-org/synapse into mysql	2015-04-09 13:13:26 +01:00
Erik Johnston	8ad0f4912e	Stream ordering and out of order insertions. ... Handle the fact that events can be persisted out of order, and so to get the "current max" stream token becomes non trivial - as we need to make sure that *all* stream tokens less than the current max have also successfully been persisted.	2015-04-09 11:41:36 +01:00
Erik Johnston	279a547a8b	Use generic db exceptions rather than sqlite3 specific ones	2015-04-08 16:53:48 +01:00
Mark Haines	83f5125d52	Merge pull request #114 from matrix-org/improve_get_event_cache ... Improve get event cache	2015-04-08 16:50:27 +01:00
Erik Johnston	3887350e47	Merge branch 'develop' of github.com:matrix-org/synapse into mysql	2015-04-08 14:11:07 +01:00
Erik Johnston	c1b34af441	Move database timer logging to seperate logger	2015-04-08 13:31:19 +01:00
Erik Johnston	9a05795619	Retry transaction, not SQL query	2015-04-08 13:11:28 +01:00
Erik Johnston	24d8134ac1	Fix maria engine to correctly recognize deadlocks	2015-04-08 13:10:54 +01:00
Erik Johnston	4fe95094d1	Merge branch 'develop' of github.com:matrix-org/synapse into mysql	2015-04-07 18:05:39 +01:00
Erik Johnston	49d6aa1394	Retry on deadlock	2015-04-07 15:28:37 +01:00
Erik Johnston	0bfa78b39b	PEP8	2015-04-07 12:16:05 +01:00
Erik Johnston	6bc9edd8b2	Fix prepare_sqlite3_database's convert_param_style	2015-04-07 12:13:58 +01:00
Erik Johnston	05a35d62b6	Bump database version	2015-04-07 12:10:15 +01:00
Erik Johnston	8574bf62dc	Add index to presence table	2015-04-07 12:09:36 +01:00
Erik Johnston	0af5f5efaf	Don't use multiple UNIQUE constraints; it will cause deadlocks	2015-04-07 12:08:35 +01:00
Erik Johnston	c8d3f6486d	Implement or_ignore flag on inserts	2015-04-07 12:06:22 +01:00
Erik Johnston	304111afd0	Don't use AUTOINCREMENT, use an in memory version	2015-04-07 12:05:36 +01:00
Erik Johnston	779f7b0f44	Fix unicode support	2015-04-02 10:06:22 +01:00
Erik Johnston	9236136f3a	Make work in both Maria and SQLite. Fix tests	2015-04-01 14:12:33 +01:00
Kegan Dougal	5e88a09a42	Add same user_id char checks as registration.	2015-03-31 14:00:25 +01:00
Kegan Dougal	cf1fa59f4b	Use a sender localpart instead of a user ID. ... Form the user ID at runtime instead, This gives less room for error in AS config files since they cannot specify the domain of another HS.	2015-03-31 13:48:03 +01:00
Kegan Dougal	c217504949	Edit SQL schema to use string IDs not ints. Use token as ID. Update tests.	2015-03-31 12:07:56 +01:00
Kegan Dougal	b59aa74556	Fix tests and missing returns on deferreds.	2015-03-31 11:35:45 +01:00
Kegan Dougal	d33ae65efc	Remove more reg/unreg methods. Read config not database for cache.	2015-03-31 11:00:00 +01:00
Paul "LeoNerd" Evans	3e420aebd8	Revert "Add another @cached wrapper, this time on get_presence_state()" ... This reverts commit ff1fa0fbf8.	2015-03-27 16:16:58 +00:00
Paul "LeoNerd" Evans	ff1fa0fbf8	Add another @cached wrapper, this time on get_presence_state()	2015-03-27 15:57:16 +00:00
David Baker	a32e876ef4	Delete pushers when changing password	2015-03-26 13:40:16 +00:00
Paul "LeoNerd" Evans	a198894bf7	Appease pep8	2015-03-26 11:53:58 +00:00
Kegan Dougal	32206dde3f	Fixes from PR comments	2015-03-26 10:11:52 +00:00
Kegan Dougal	4edcbcee3b	Merge branch 'develop' into application-services-txn-reliability ... Conflicts: synapse/storage/__init__.py	2015-03-26 10:07:59 +00:00
Paul "LeoNerd" Evans	953e40f9dc	Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one	2015-03-25 19:12:16 +00:00
David Baker	c1a256cc4c	Allow multiple pushers for a single app ID & pushkey, honouring the 'append' flag in the API.	2015-03-25 19:06:22 +00:00
Paul "LeoNerd" Evans	f173d40a32	Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key	2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans	1b988b051b	Store the rejected reason in (Frozen)Event structs	2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans	033a517feb	Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file	2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans	9ba6487b3f	Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict()	2015-03-25 19:05:34 +00:00
Erik Johnston	0e8f5095c7	Fix unicode database support	2015-03-25 17:15:20 +00:00
David Baker	c7023f2155	1) Pushers are now associated with an access token ... 2) Change places where we mean unauthenticated to 401, not 403, in C/S v2: hack so it stays as 403 in v1 because web client relies on it.	2015-03-24 17:24:15 +00:00
Erik Johnston	0ba393924a	Escape non printing ascii character	2015-03-24 16:31:52 +00:00
Erik Johnston	f488293d96	Don't reinsert into event_edges	2015-03-24 16:20:26 +00:00
Erik Johnston	1aa44939fc	Fix bugs in transactions storage	2015-03-24 16:20:05 +00:00
Erik Johnston	5a447098dd	Don't use room hosts table	2015-03-24 16:19:24 +00:00
Erik Johnston	9e98f1022a	Don't order by rowid	2015-03-24 16:19:01 +00:00
Erik Johnston	9115421ace	Use _simple_upsert	2015-03-24 16:17:39 +00:00
David Baker	d19e79ecc9	Make deleting other access tokens when you change your password actually work	2015-03-24 15:33:48 +00:00
Erik Johnston	6e7131f02f	Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable.	2015-03-23 15:38:56 +00:00
Erik Johnston	9a7f496298	Sanitize RoomMemberStore	2015-03-23 15:29:04 +00:00
David Baker	d98660a60d	Implement password changing (finally) along with a start on making client/server auth more general.	2015-03-23 14:20:28 +00:00
Erik Johnston	d5272b1d2c	Use 'update or insert' rather than on 'conflict replace'	2015-03-23 14:02:34 +00:00
Erik Johnston	278149f533	Sanitize TransactionStore	2015-03-23 13:43:21 +00:00
Paul "LeoNerd" Evans	72d8406409	Put a cache on get_aliases_for_room	2015-03-20 19:21:13 +00:00
Paul "LeoNerd" Evans	a63b4f7101	Remember the 'last seen' time for a given user/IP/device combination and only bother INSERTing another if it's stale	2015-03-20 18:25:49 +00:00
Paul "LeoNerd" Evans	0f86312c4c	Pull out the cache logic from the @cached wrapper into its own class we can reuse	2015-03-20 18:25:42 +00:00
Paul "LeoNerd" Evans	b1022ed8b5	func(*EXPR) is valid Python syntax, really...	2015-03-20 17:47:45 +00:00
Erik Johnston	f6583796fe	Merge branch 'develop' of github.com:matrix-org/synapse into mysql	2015-03-20 16:31:48 +00:00
Erik Johnston	80cd08c190	PEP8	2015-03-20 16:03:25 +00:00
Erik Johnston	9517f4da4d	Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement	2015-03-20 16:02:47 +00:00
Erik Johnston	dc0c989ef4	Give sensible names for '_simple_...' transactions	2015-03-20 15:59:18 +00:00
Paul "LeoNerd" Evans	ceb61daa70	Add the tiniest of tiny one-element caches to get_room_events_max_id() as it's read every time someone hits eventstream	2015-03-20 15:44:06 +00:00
Erik Johnston	fce0114005	Start removing Tables	2015-03-20 15:05:44 +00:00
Erik Johnston	7e282a53a5	Tidy up _simple_... methods	2015-03-20 15:05:10 +00:00
Paul "LeoNerd" Evans	91cb46191d	Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though	2015-03-20 14:59:45 +00:00
Erik Johnston	87db64b839	Rearrange storage modules	2015-03-20 14:11:38 +00:00
Erik Johnston	cb8162d3d1	Rearrange storage modules	2015-03-20 13:52:56 +00:00
Erik Johnston	d288d273e1	Generate transaction id in code	2015-03-20 10:57:44 +00:00
Erik Johnston	d4f50f3ae5	decode_result takes an iterable	2015-03-20 10:57:26 +00:00
Erik Johnston	0d0610870d	Fix up schemas some more	2015-03-20 10:55:31 +00:00
Erik Johnston	d7a0496f3e	Convert storage layer to be mysql compatible	2015-03-19 15:59:48 +00:00
Erik Johnston	58ed393235	Remove redundant key	2015-03-19 15:12:05 +00:00
Erik Johnston	fae059cc18	Fix up schemas to work with mariadb	2015-03-19 13:42:39 +00:00
Erik Johnston	bb24609158	Clean out event_forward_extremities table when the server rejoins the room	2015-03-18 11:19:47 +00:00
Erik Johnston	89036579ed	Update schema to work with mariadb	2015-03-18 11:18:49 +00:00
Paul "LeoNerd" Evans	93978c5e2b	@cached() annotate get_user_by_token() - achieves a minor DB performance improvement	2015-03-17 17:24:51 +00:00
Paul "LeoNerd" Evans	1489521ee5	Be polite and ensure we use @functools.wraps() when creating a function decorator	2015-03-17 17:19:22 +00:00
Paul "LeoNerd" Evans	be170b1426	Add a metric for the scheduling latency of SQL queries	2015-03-16 17:21:59 +00:00
Erik Johnston	cd2539ab2a	Merge pull request #110 from matrix-org/fix_ban ... Fix ban	2015-03-16 15:36:52 +00:00
Kegan Dougal	f0d6f724a2	Set the service ID as soon as it is known.	2015-03-16 15:24:32 +00:00
Erik Johnston	f1d2b94e0b	Copy dict of context.current_state before changing it.	2015-03-16 15:13:05 +00:00
Kegan Dougal	835e01fc70	Minor PR comment tweaks.	2015-03-16 10:16:59 +00:00
Kegan Dougal	f9232c7917	Merge branch 'develop' into application-services-txn-reliability ... Conflicts: synapse/storage/appservice.py	2015-03-16 10:09:15 +00:00
Paul "LeoNerd" Evans	e75fa8bbbf	Bugfix to sql_txn_timer increment - add only the per-TXN duration, not the total time ever spent since boot	2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans	63cb7ece62	Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer	2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans	f1fbe3e09f	Rename TimerMetric to DistributionMetric; as it could count more than just time	2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans	642f725fd7	Pretend the 'getEvent' cache is just another cache in the set of all the others for metric	2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans	b0cf867319	Use _ instead of . as a metric namespacing separator, for Prometheus	2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans	f9478e475b	Rename Metrics' "keys" to "labels"	2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans	fa319a5786	Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely	2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans	25187ab674	Collect per-SQL-verb timer stats on query execution time	2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans	e1a7e3564f	Delete a couple of TODO markers of monitoring stats now done	2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans	8664599af7	Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache	2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans	b0cdf097f4	Sprinkle some CacheCounterMetrics around the synapse.storage layer	2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans	a7a28f85ae	Appease pep8	2015-03-11 17:32:43 +00:00
Paul "LeoNerd" Evans	59a5f012cc	Also give _execute() a description	2015-03-11 17:19:17 +00:00
Paul "LeoNerd" Evans	099e4b88d8	Add a description to storage layer's _execute_and_decode()	2015-03-11 17:08:57 +00:00
David Baker	04f8478aaa	Add the master push rule for the break-my-push button. Allow server default rules to be disabled by default.	2015-03-10 17:26:25 +00:00
David Baker	8916acbc13	These aren't defined for redacted events so don't crash	2015-03-10 11:21:37 +00:00
Kegan Dougal	db1fbc6c6f	Fix remaining scheduler bugs. Add more informative logging.	2015-03-10 10:04:20 +00:00
Kegan Dougal	7e0bba555c	Remove unused import	2015-03-09 17:48:37 +00:00
Kegan Dougal	04c9751f24	Bug fixes whilst putting it all together	2015-03-09 17:45:41 +00:00
Kegan Dougal	b98cd03193	Use event IDs instead of dumping event content in the txns table.	2015-03-09 17:25:20 +00:00
Kegan Dougal	21fd84dcb8	Use seconds; start gluing in the AS scheduler into the AS handler.	2015-03-09 17:01:19 +00:00
Kegan Dougal	0a60bbf4fa	Finish appservice txn storage impl and tests.	2015-03-09 15:53:03 +00:00
Kegan Dougal	1ead1caa18	Implement create_appservice_txn with tests.	2015-03-09 13:54:20 +00:00
Kegan Dougal	1c2dcf762a	Partially implement txn store methods with tests.	2015-03-09 13:10:31 +00:00
Kegan Dougal	406d32f8b5	Start implementing ApplicationServiceTransactionStore	2015-03-06 17:35:14 +00:00
Kegan Dougal	34ce2ca62f	Merge branch 'develop' into application-services-txn-reliability	2015-03-06 17:28:49 +00:00
Kegan Dougal	4a6afa6abf	Assign the AS ID from the database; replace old placeholder txn id.	2015-03-06 17:27:55 +00:00
Kegan Dougal	01c099d9ef	Add appservice txns sql schema	2015-03-06 17:16:47 +00:00
Kegan Dougal	2602ddc379	Apply clarity and docstrings	2015-03-06 16:16:14 +00:00
Kegan Dougal	0354659f9d	Finish synapse.appservice.scheduler implementation. ... With tests to assert behaviour. Not hooked up yet. Stub datastore methods not implemented yet.	2015-03-06 16:09:05 +00:00
Kegan Dougal	f260cb72cd	Flesh out more stub functions.	2015-03-06 15:12:24 +00:00
Kegan Dougal	141ec04d19	Add stub ApplicationServiceTransactionStore. Bootstrap Recoverers. Fill in stub Transaction functions.	2015-03-06 14:53:35 +00:00
David Baker	cf66ddc1b4	Schema change as delta in v14	2015-03-06 14:11:49 +00:00
David Baker	ed72fc3a50	Merge branch 'develop' into pushrules2 ... Conflicts: synapse/storage/schema/pusher.sql	2015-03-04 15:24:21 +00:00
Erik Johnston	d56c01fff4	Note that we don't specify execution order	2015-03-04 15:10:05 +00:00
Erik Johnston	17d319a20d	s/schema_deltas/applied_schema_deltas/	2015-03-04 15:06:22 +00:00
David Baker	92b3dc3219	Merge branch 'develop' into pushrules2	2015-03-04 14:56:41 +00:00
Erik Johnston	5681264faa	s/%r/%s/	2015-03-04 14:21:53 +00:00
Erik Johnston	f701197227	Add example directory structures in doc	2015-03-04 14:20:14 +00:00
David Baker	2a45f3d448	Use if not results rather than len, as per feedback.	2015-03-04 14:17:59 +00:00
Erik Johnston	16dd87d848	Don't assume db conn is a Context Manager. ... Twisted adbapi wrapped connections aren't context managers.	2015-03-04 14:03:41 +00:00
Erik Johnston	5eefd1f618	Add unique constraint on schema_version.lock schema. Use conflict clause in sql.	2015-03-04 13:52:18 +00:00
Erik Johnston	b4c38738f4	Change to use logger in db upgrade script	2015-03-04 13:43:35 +00:00
Erik Johnston	640e53935d	Use context manager with db conn to correctly commit and rollback	2015-03-04 13:43:17 +00:00
Erik Johnston	8c8354e85a	Actually add full_schemas dir	2015-03-04 13:34:38 +00:00
Erik Johnston	c3530c3fb3	More docs. Rename 'schema/current' to 'schema/full_schemas'	2015-03-04 13:34:11 +00:00
Erik Johnston	811355ccd0	Add some docs and remove unused variables	2015-03-04 13:11:01 +00:00
Erik Johnston	82b34e813d	SYN-67: Finish up implementing new database schema management	2015-03-04 12:04:19 +00:00
Erik Johnston	8d33adfbbb	SYN-67: Begin changing the way we handle schema versioning	2015-03-02 18:23:55 +00:00
David Baker	6fab7bd2c1	s/user_name/user/ as per mjark's comment	2015-03-02 18:17:19 +00:00
Erik Johnston	9f03553f48	Add missing comma	2015-03-02 16:38:40 +00:00
Erik Johnston	b41dc68773	We purposefully don't have a version 14 delta script.	2015-03-02 16:36:19 +00:00
David Baker	20436cdf75	Blank lines	2015-03-02 15:58:12 +00:00
Kegsay	8ad024ea80	Merge pull request #93 from matrix-org/application-services-exclusive ... Application services exclusive flag support	2015-03-02 14:56:32 +00:00
Erik Johnston	0a036944bd	Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu	2015-03-02 13:53:30 +00:00
Kegan Dougal	c3c01641d2	Run deltas and bump user_version in upgrade script	2015-03-02 13:38:57 +00:00
Kegan Dougal	cb97ea3ec2	PEP8	2015-03-02 11:23:46 +00:00
Kegan Dougal	377ae369c1	Wrap all of get_app_service_rooms in a txn.	2015-03-02 11:20:51 +00:00
Kegan Dougal	b216b36892	JOIN state_events rather than parsing unrecognized_keys to pull out member state_keys	2015-03-02 10:41:35 +00:00
Kegan Dougal	3d73383d18	Modify _simple_select_list to allow an empty WHERE clause. Use it for get_all_rooms and get_all_users.	2015-03-02 10:16:24 +00:00
Kegan Dougal	ebc4830666	PR tweaks: set earlier on and use 'as json' for compat	2015-03-02 09:53:00 +00:00
Kegan Dougal	16b90764ad	Convert expected format for AS regex to include exclusivity. ... Previously you just specified the regex as a string, now it expects a JSON object with a 'regex' key and an 'exclusive' boolean, as per spec.	2015-02-27 10:44:32 +00:00
Kegan Dougal	806a6c886a	PEP8	2015-02-27 09:48:57 +00:00
David Baker	1959088156	Add API for getting/setting enabled-ness of push rules.	2015-02-26 18:07:44 +00:00
Kegan Dougal	f0995436e7	Check for membership invite events correctly.	2015-02-26 17:21:17 +00:00
Kegan Dougal	dcec7175dc	Finish impl to get new events for AS. ASes should now be able to poll /events	2015-02-26 16:23:01 +00:00
Kegan Dougal	92478e96d6	Finish impl to extract all room IDs an AS may be interested in when polling the event stream.	2015-02-26 14:35:28 +00:00
David Baker	944003021b	whitespace	2015-02-26 13:43:05 +00:00
David Baker	94fa334b01	Add enable/disable overlay for push rules (REST API not yet hooked up)	2015-02-25 19:17:07 +00:00
Kegan Dougal	29267cf9d7	PEP8 and pyflakes	2015-02-25 17:42:28 +00:00
Kegan Dougal	978ce87c86	Comment unused variables.	2015-02-25 17:37:48 +00:00
Kegan Dougal	2c79c4dc7f	Fix alias query.	2015-02-25 17:37:14 +00:00
Kegan Dougal	2b8ca84296	Add support for extracting matching room_ids and room_aliases for a given AS.	2015-02-25 17:15:25 +00:00
Kegan Dougal	2d20466f9a	Add stub functions and work out execution flow to implement AS event stream polling.	2015-02-25 15:00:59 +00:00
Paul "LeoNerd" Evans	9640510de2	Use OrderedDict for @cached backing store, so we can evict the oldest key unbiased	2015-02-23 18:41:58 +00:00
Paul "LeoNerd" Evans	f53fcbce97	Use cache.pop() instead of a separate membership test + del []	2015-02-23 18:30:45 +00:00
Mark Haines	27080698e7	Fix code style warning	2015-02-23 18:19:13 +00:00
Mark Haines	74048bdd41	Remove unused import	2015-02-23 18:17:43 +00:00
Paul "LeoNerd" Evans	a09e59a698	Pull the _get_event_cache.setdefault() call out of the try block, as it doesn't need to be there and is confusing	2015-02-23 16:55:57 +00:00
Paul "LeoNerd" Evans	044d813ef7	Use the @cached decorator to implement the destination_retry_timings cache	2015-02-23 16:04:40 +00:00
Paul "LeoNerd" Evans	357fba2c24	RoomMemberStore no longer needs a _user_rooms_cache member	2015-02-23 15:57:41 +00:00
Paul "LeoNerd" Evans	e76d485e29	Allow @cached-wrapped functions to have a prefill method for setting entries	2015-02-23 15:41:54 +00:00
Paul "LeoNerd" Evans	4631b737fd	Squash out the now-redundant ApplicationServicesCache object class	2015-02-23 14:38:44 +00:00
Erik Johnston	42b972bccd	Revert get_auth_chain changes	2015-02-23 14:35:23 +00:00
Paul "LeoNerd" Evans	55022d6ca5	Remove a TODO note	2015-02-19 18:38:09 +00:00
Paul "LeoNerd" Evans	ebc3db295b	Take named arguments to @cached() decorator, add a 'max_entries' limit	2015-02-19 18:36:02 +00:00
Paul "LeoNerd" Evans	077d200342	Move @cached decorator out into synapse.storage._base; add minimal docs	2015-02-19 17:29:39 +00:00
Erik Johnston	0ac2a79faa	Initial stab at implementing a batched get_missing_pdus request	2015-02-19 17:24:14 +00:00
Paul "LeoNerd" Evans	61959928bb	Pull out the 'get_rooms_for_user' cache logic into a reüsable @cached decorator	2015-02-19 14:58:07 +00:00
Erik Johnston	2c29ed3e84	Use absolute path when loading delta sql files	2015-02-17 17:22:24 +00:00
Erik Johnston	1a989c436c	Bump schema version	2015-02-17 15:45:55 +00:00
Erik Johnston	fa6c93bd26	Merge branch 'consumeErrors' of github.com:matrix-org/synapse into develop	2015-02-17 15:18:17 +00:00
Erik Johnston	72a4de2ce6	Use consumeErrors=True on all DeferredLists. ... This is so that the DeferredLists actually consume the error instead of propogating down the non-existent errback chain. This should reduce the number of unhandled errors we are seeing.	2015-02-17 10:07:01 +00:00
David Baker	64def4f953	Merge branch 'hotfixes-0.7.0e' into develop	2015-02-13 16:18:34 +00:00
David Baker	8d5cce62ab	Update pushers by app id and pushkey, not user id and pushkey	2015-02-13 16:16:16 +00:00
Mark Haines	0d872f5aa6	Merge pull request #50 from matrix-org/application-services ... Application Services	2015-02-13 15:06:14 +00:00
Mark Haines	183b3d4e47	Prepare the database whenever a connection is opened from the db_pool so that in-memory databases will work	2015-02-13 14:38:24 +00:00
Erik Johnston	6b186a57ba	Merge branch 'fix' of github.com:matrix-org/synapse into hotfixes-v0.7.0a	2015-02-12 14:51:46 +00:00
Erik Johnston	717687e1fc	Get an auth query one at a time	2015-02-12 14:39:31 +00:00
Erik Johnston	83d41f25d8	Set database schema version in delta	2015-02-12 10:05:47 +00:00
Erik Johnston	8bbdf32849	Convert get_rooms to use runInteraction so the transacion has a more helpful description	2015-02-11 18:56:13 +00:00
Erik Johnston	42bc56dad3	Merge branch 'develop' of github.com:matrix-org/synapse into use-simplejson	2015-02-11 17:01:38 +00:00
Erik Johnston	7eef84a95b	pyflakes	2015-02-11 16:52:22 +00:00
Kegan Dougal	f2fdcb7c4b	Merge branch 'develop' into application-services	2015-02-11 16:43:26 +00:00
Paul "LeoNerd" Evans	b164e0896c	Merge branch 'bugs/SYN-264' into develop	2015-02-11 16:23:30 +00:00
Paul "LeoNerd" Evans	7f47ba7383	Added another TODO note	2015-02-11 16:18:21 +00:00
Paul "LeoNerd" Evans	45b56609ae	Cache the result of a get_rooms_for_user query, to make user_rooms_intersect() much lighter in the read-common case	2015-02-11 16:04:08 +00:00
Paul "LeoNerd" Evans	7be0f6594e	First step of making user_rooms_intersect() faster - implement in intersection logic in Python code terms of a DB query that is cacheable per user	2015-02-11 15:53:56 +00:00
Erik Johnston	ddb816cf60	Don't unfreeze when using FreezeEvent.get_dict, as we are using a JSONEncoder that understands FrozenDict	2015-02-11 15:44:28 +00:00
Mark Haines	ddc25cf4e2	Invalidate the cache for an event if it is redacted	2015-02-11 15:23:28 +00:00
Mark Haines	aff892ce79	Fix formatting	2015-02-11 15:02:35 +00:00
Mark Haines	f5a70e0d2e	Add a cache for get_event	2015-02-11 15:01:15 +00:00
Erik Johnston	4ebbaf0d43	Blunty replace json with simplejson	2015-02-11 14:23:10 +00:00
Kegan Dougal	fd40d992ad	PEP8-ify	2015-02-11 10:41:33 +00:00
Kegan Dougal	9978c5c103	Merge branch 'develop' into application-services	2015-02-11 10:03:24 +00:00
Mark Haines	b085fac735	Code-style fixes	2015-02-10 16:30:48 +00:00
Mark Haines	fda4422bc9	Fix pyflakes	2015-02-10 14:54:07 +00:00
Mark Haines	d7c7efb691	Add performance counters for different stages of loading events	2015-02-10 14:50:53 +00:00
Mark Haines	0c4536da8f	Use the transaction 'desc' rather than 'name', increment the txn_ids in ... txn names	2015-02-09 18:06:31 +00:00
Mark Haines	347b497db0	Formatting	2015-02-09 17:57:09 +00:00
Mark Haines	3a5ad7dbd5	Performance counters for database transaction names	2015-02-09 17:55:56 +00:00
Kegan Dougal	53557fc532	Merge branch 'develop' into application-services	2015-02-09 15:20:56 +00:00
Erik Johnston	c4ee4ce93e	Fix typo	2015-02-09 15:00:37 +00:00
Erik Johnston	66fde49f07	Log database time every 10s and log as percentage	2015-02-09 14:45:15 +00:00
Erik Johnston	75656712e3	Time how long we're spending on the database thread	2015-02-09 14:22:52 +00:00
Kegan Dougal	5a7dd05818	Modify auth.get_user_by_req for authing appservices directly. ... Add logic to map the appservice token to the autogenned appservice user ID. Add unit tests for all forms of get_user_by_req (user/appservice, valid/bad/missing tokens)	2015-02-09 14:14:15 +00:00
Kegan Dougal	ac3183caaa	Register a user account for the AS when the AS registers. Add 'sender' column to AS table.	2015-02-09 12:03:37 +00:00
Erik Johnston	0cd66885e3	Move delta/v13.sql to delta/v12.sql	2015-02-06 14:38:04 +00:00
Kegan Dougal	a3c6010718	Add delta sql file.	2015-02-05 16:48:57 +00:00
Kegan Dougal	fc8bcc809d	Merge branch 'develop' into application-services	2015-02-05 15:32:45 +00:00
David Baker	aaf50bf6f3	Give server default rules the 'default' attribute and fix various brokenness.	2015-02-05 15:11:38 +00:00
Kegan Dougal	951690e54d	Merge branch 'develop' into application-services	2015-02-05 14:28:03 +00:00
Erik Johnston	f08bd95880	Merge pull request #47 from matrix-org/signature_failures ... Federation fixes.	2015-02-05 14:03:00 +00:00
Kegan Dougal	27091f146a	Add hs_token column and generate a different token f.e application service.	2015-02-05 10:08:12 +00:00
Kegan Dougal	89f2e8fbdf	Fix bug in store defer. Add more unit tests.	2015-02-04 15:21:03 +00:00
Kegan Dougal	17753f0c20	Add stub ApplicationServiceApi and glue it with the handler.	2015-02-04 11:19:18 +00:00
Erik Johnston	03d415a6a2	Brief comment on why we do some things on every call to persist_event and not others	2015-02-04 10:40:59 +00:00
Erik Johnston	c0462dbf15	Rearrange persist_event so that do all the queries that need to be done before returning early if we have already persisted that event.	2015-02-04 10:16:51 +00:00
Erik Johnston	02be8da5e1	Add doc to get_event	2015-02-03 17:34:07 +00:00
David Baker	dc7bb70f22	s/instance_handle/profile_tag/	2015-02-03 16:51:07 +00:00
Kegan Dougal	94a5db9f4d	Add appservice package and move ApplicationService into it.	2015-02-03 14:44:16 +00:00
Kegan Dougal	f2c039bfb9	Implement restricted namespace checks. Begin fleshing out the main hook for notifying application services.	2015-02-03 13:29:27 +00:00
Kegan Dougal	a060b47b13	Add namespace constants. Add restrict_to option to limit namespace checks.	2015-02-03 13:17:28 +00:00
Kegan Dougal	3bd2841fdb	Everyone loves SQL typos	2015-02-03 11:37:52 +00:00
Kegan Dougal	197f3ea4ba	Implement regex checks for app services. ... Expose handler.get_services_for_event which manages the checks for all services.	2015-02-03 11:26:33 +00:00
Erik Johnston	e7ca813dd4	Try to ensure we don't persist an event we have already persisted. In persist_event check if we already have the event, if so then update instead of replacing so that we don't cause a bump of the stream_ordering.	2015-02-03 10:39:41 +00:00
Kegan Dougal	9ff349a3cb	Add defers in the right places.	2015-02-02 17:42:49 +00:00
Kegan Dougal	1a2de0c5fe	Implement txns for AS (un)registration.	2015-02-02 17:39:41 +00:00
Erik Johnston	941f59101b	Don't fail an entire request if one of the returned events fails a signature check. If an event does fail a signature check, look in the local database and request it from the originator.	2015-02-02 16:56:01 +00:00
Kegan Dougal	a006d168c5	Actually merge into develop.	2015-02-02 16:05:34 +00:00
Kegan Dougal	c059c9fea5	Merge branch 'develop' into application-services ... Conflicts: synapse/handlers/__init__.py synapse/storage/__init__.py	2015-02-02 15:57:59 +00:00
Erik Johnston	7d897f5bfc	Merge pull request #43 from matrix-org/rejections ... Rejections	2015-01-30 16:11:14 +00:00
Mark Haines	6dc92d3427	Merge pull request #41 from matrix-org/client_v2_sync ... Client v2 sync	2015-01-30 15:54:25 +00:00
Erik Johnston	1bd540ef79	Merge branch 'develop' of github.com:matrix-org/synapse into rejections ... Conflicts: synapse/storage/schema/im.sql	2015-01-30 15:16:38 +00:00
Mark Haines	9ec9d6f2cb	Merge pull request #42 from matrix-org/replication_split ... Replication split	2015-01-30 15:14:10 +00:00
Erik Johnston	91015ad008	Remove merge conflict	2015-01-30 14:58:54 +00:00
Erik Johnston	4f7fe63b6d	Remember to add schema file to list	2015-01-30 14:57:53 +00:00
Erik Johnston	fdd2ac495a	Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage ... Conflicts: synapse/storage/__init__.py	2015-01-30 14:57:33 +00:00
Mark Haines	8bc3066e0b	Merge branch 'client_v2_filter' into client_v2_sync	2015-01-30 14:57:04 +00:00
Erik Johnston	471c47441d	Merge pull request #37 from matrix-org/client_v2_filter ... Client v2 filter	2015-01-30 14:56:08 +00:00
Mark Haines	e97f756a05	Use 'in' to test if the key exists, remove unused _filters_for_user	2015-01-30 14:54:06 +00:00
Erik Johnston	2f4cb04f45	Be more specific in naming columns in selects.	2015-01-30 14:48:11 +00:00
Erik Johnston	472cf532b7	Put CREATE rejections into seperate .sql	2015-01-30 14:48:03 +00:00
David Baker	322a047502	Add room member count condition and default rule to make a noise on rooms of only 2 people.	2015-01-30 14:46:45 +00:00
Erik Johnston	c515d37797	Merge branch 'replication_split' of github.com:matrix-org/synapse into rejections ... Conflicts: synapse/storage/schema/delta/v12.sql	2015-01-30 14:19:49 +00:00
Erik Johnston	84b78c3b5f	Merge branch 'rejections_storage' of github.com:matrix-org/synapse into replication_split	2015-01-30 14:17:47 +00:00
Erik Johnston	e0b7c521cb	Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage ... Conflicts: synapse/storage/__init__.py synapse/storage/schema/delta/v12.sql	2015-01-30 14:08:28 +00:00
Mark Haines	8498d348d8	Fix token formatting	2015-01-30 11:42:09 +00:00
Mark Haines	22dd1cde2d	Filter the recent events before applying the limit when doing an incremental sync with a gap	2015-01-30 11:32:35 +00:00
Erik Johnston	2ebf795c0a	Merge branch 'develop' of github.com:matrix-org/synapse into rejections ... Conflicts: synapse/storage/__init__.py synapse/storage/schema/delta/v12.sql	2015-01-30 11:10:37 +00:00
Erik Johnston	78015948a7	Initial implementation of auth conflict resolution	2015-01-29 16:52:33 +00:00
Mark Haines	8e571cbed8	Merge branch 'client_v2_filter' into client_v2_sync	2015-01-29 16:18:59 +00:00
Mark Haines	295322048d	Merge branch 'develop' into client_v2_filter	2015-01-29 16:18:34 +00:00
Mark Haines	acb68a39e0	Code style fixes.	2015-01-29 16:12:40 +00:00
Mark Haines	396a67a09a	Merge branch 'client_v2_filter' into client_v2_sync ... Conflicts: synapse/rest/client/v2_alpha/__init__.py	2015-01-29 14:58:00 +00:00
Mark Haines	9d8f798a3f	Merge changes from develop	2015-01-29 14:55:27 +00:00
Mark Haines	e4f50fa0aa	Move bump schema delta	2015-01-29 14:53:18 +00:00
Mark Haines	e016f4043b	Use get_room_events_stream to get changes to the rooms if the number of changes is small	2015-01-29 14:40:28 +00:00
Kegan Dougal	3773759c0f	Also edit the filter column on the delta SQL	2015-01-29 09:15:33 +00:00
Mark Haines	b0b80074e0	SYN-252: Supply the stream and topological parts in the correct order to the constructor	2015-01-29 01:48:48 +00:00
David Baker	d5bdf3c0c7	Allow the push rule delete method to take more specifiers.	2015-01-28 18:06:04 +00:00
Kegan Dougal	11634017f4	s/definition/filter_json/ since definition is now used to mean a component of the filter, rather than the complete json	2015-01-28 17:42:19 +00:00
Mark Haines	9c61556504	Merge branch 'develop' into client_v2_sync	2015-01-28 17:29:30 +00:00
Mark Haines	26c8fff19e	Merge pull request #36 from matrix-org/device_id_from_access_token ... Extract the device id and token id from the access token when autheniticating users	2015-01-28 17:19:28 +00:00
Mark Haines	388581e087	Extract the id token of the token when authing users, include the token and device_id in the internal meta data for the event along with the transaction id when sending events	2015-01-28 16:58:23 +00:00
Kegan Dougal	c23e3db544	Add filter JSON sanity checks.	2015-01-28 16:45:18 +00:00
Erik Johnston	0ef5bfd6a9	Start implementing auth conflict res	2015-01-28 16:16:53 +00:00
David Baker	60b143a52e	Move pushers delta to v12 and bump schema version	2015-01-28 15:48:28 +00:00
David Baker	e78dd33292	Use %s instead of +	2015-01-28 14:52:58 +00:00
David Baker	93aac9bb7b	Newline	2015-01-28 14:51:01 +00:00
David Baker	445ad9941e	Redundant parens	2015-01-28 14:49:59 +00:00
David Baker	6d485dd1c7	unnecessary newlines	2015-01-28 14:48:42 +00:00
David Baker	fb0928097a	More magic commas (including the place I copied it from...)	2015-01-28 14:48:07 +00:00
David Baker	0cbb6b0f52	Google doc style	2015-01-28 14:44:41 +00:00
David Baker	c291a4d522	Merge branch 'develop' into pushers ... Conflicts: synapse/handlers/events.py synapse/server.py	2015-01-28 12:51:05 +00:00
Kegan Dougal	42876969b9	Add basic application_services SQL, and hook up parts of the appservice store to read from it.	2015-01-28 11:59:38 +00:00
Kegan Dougal	b46fa8603e	Remove unused import	2015-01-28 09:17:48 +00:00
Mark Haines	e020574d65	Fix Formatting	2015-01-27 20:19:36 +00:00
Paul "LeoNerd" Evans	8398f19bce	Created schema delta	2015-01-27 19:00:09 +00:00
Paul "LeoNerd" Evans	06cc147012	Initial stab at real SQL storage implementation of user filter definitions	2015-01-27 18:46:03 +00:00
Paul "LeoNerd" Evans	54e513b4e6	Move storage of user filters into real datastore layer; now have to mock it out in the REST-level tests	2015-01-27 17:48:13 +00:00
Kegan Dougal	fbeaeb8689	Log when ASes are registered/unregistered.	2015-01-27 17:34:40 +00:00
Kegan Dougal	ec3719b583	Use ApplicationService when registering.	2015-01-27 17:15:06 +00:00
Kegan Dougal	92171f9dd1	Add stub methods, TODOs and docstrings for application services.	2015-01-27 16:53:59 +00:00
Mark Haines	a56008842b	Start implementing incremental initial sync	2015-01-27 16:24:22 +00:00
Kegan Dougal	7331d34839	Add AS specific classes with docstrings.	2015-01-27 16:23:46 +00:00
Kegan Dougal	51449e0665	Add appservice handler and store. Glue together rest > handler > store.	2015-01-27 15:50:28 +00:00
David Baker	69a75b7ebe	Add brackets to make get room name / alias work	2015-01-26 16:52:47 +00:00
Erik Johnston	7b88619241	Split up replication_layer module into client, server and transaction queue	2015-01-26 10:45:24 +00:00
David Baker	5f84ba8ea1	Add API to delete push rules.	2015-01-23 17:49:37 +00:00
Erik Johnston	9b1e552b51	Merge branch 'develop' of github.com:matrix-org/synapse into rejections	2015-01-23 15:51:48 +00:00
Erik Johnston	3b9cc882a5	Add storage method have_events	2015-01-23 15:42:52 +00:00
Mark Haines	5759bec43c	Replace hs.parse_userid with UserID.from_string	2015-01-23 11:47:15 +00:00
David Baker	bcd48b9636	Fix adding rules without before/after & add the rule that we couldn't find to the error	2015-01-23 10:28:25 +00:00
David Baker	8a850573c9	As yet fairly untested GET API for push rules	2015-01-22 19:32:17 +00:00
David Baker	673773b217	oops, this is not its own schema file	2015-01-22 18:27:07 +00:00
David Baker	7ecb49ef25	Insufficient newlines	2015-01-22 17:53:30 +00:00
David Baker	5c6189ea3e	Merge branch 'develop' into pushers ... Conflicts: synapse/rest/__init__.py	2015-01-22 17:46:16 +00:00
David Baker	ede491b4e0	Oops: second part of commit dc938606	2015-01-22 17:38:53 +00:00
David Baker	dc93860619	Add rest API & store for creating push rules ... Also make unrecognised request error look more like synapse errors because it makes it easier to throw them from within rest classes.	2015-01-22 17:37:12 +00:00
Erik Johnston	b1b85753d7	Add support for storing rejected events in EventContext and data stores	2015-01-22 15:50:17 +00:00
David Baker	afb714f7be	add instance_handles to pushers so we have a way to refer to them even if the push token changes.	2015-01-20 11:49:48 +00:00
David Baker	2d2953cf5f	Require device language when adding a pusher. ... Because this seems like it might be useful to do sooner rather than later.	2015-01-16 11:24:10 +00:00
David Baker	2ca2dbc821	Send room name and first alias in notification poke.	2015-01-15 16:56:18 +00:00
David Baker	2cb30767fa	Honour the 'rejected' return from push gateways ... Add a timestamp to push tokens so we know the last time they we got them from the device. Send it to the push gateways so it can determine whether its failure is more recent than the token. Stop and remove pushers that have been rejected.	2015-01-13 19:48:37 +00:00
Mark Haines	3891597eb3	Remove unused functions	2015-01-13 15:57:26 +00:00
Mark Haines	fda63064fc	get_room_events isn't called anywhere	2015-01-13 14:43:26 +00:00
Mark Haines	895fcb377e	Fix stream token ordering	2015-01-13 14:38:53 +00:00
David Baker	c06a9063e1	Merge branch 'develop' into pushers	2015-01-13 13:15:51 +00:00
Kegan Dougal	c43d898119	SYN-178: Fix off by one.	2015-01-12 17:38:40 +00:00
Erik Johnston	36a2a877e2	Use time.time() instead of time.clock()	2015-01-06 16:34:41 +00:00
Erik Johnston	fd9a8db7ea	Only fetch the columns we need.	2015-01-06 15:59:31 +00:00
Erik Johnston	a01416cf21	Add delta and bump DB version	2015-01-06 15:42:18 +00:00
Erik Johnston	f6da237c35	Add index on transaction_id to sent_transcations	2015-01-06 15:40:38 +00:00
Erik Johnston	9bd07bed23	Actually time that function	2015-01-06 15:28:56 +00:00
Erik Johnston	03a501456c	Time how long calls to _get_destination_retry_timings take	2015-01-06 15:22:28 +00:00
Erik Johnston	52b2c6c9c7	Don't include None's in _get_events_txn	2015-01-06 14:56:57 +00:00
Erik Johnston	8a12df8cf3	Merge branch 'erikj-perf' of github.com:matrix-org/synapse into develop	2015-01-06 14:45:57 +00:00
Erik Johnston	96707ed718	Name 'user_rooms_intersect' transaction	2015-01-06 14:44:27 +00:00
Erik Johnston	76ec154e95	We don't need the full events for get_rooms_for_user_where_membership_is	2015-01-06 14:37:00 +00:00
Mark Haines	5e23a19204	Merge pull request #28 from matrix-org/erikj-perf ... Database performance improvements.	2015-01-06 13:33:40 +00:00
Mark Haines	adb04b1e57	Update copyright notices	2015-01-06 13:21:39 +00:00
Erik Johnston	af1c7c7808	PEP8	2015-01-06 13:13:17 +00:00
Erik Johnston	12819d5082	Remove debug lines	2015-01-06 13:12:30 +00:00
Erik Johnston	52d8519008	Don't do batching when getting events.	2015-01-06 13:10:27 +00:00
Erik Johnston	98933e3db6	Only fetch prev_content when a client is streaming/paginating. Use transactions for event streams.	2015-01-06 13:03:23 +00:00
Erik Johnston	3e26720e05	Temporarily turn off 'redacted_because' and 'prev_content' keys	2015-01-06 11:26:58 +00:00
Erik Johnston	f4ea78e9e2	More debug logging	2015-01-06 11:24:18 +00:00
Erik Johnston	753126b8cc	Add some debug logging	2015-01-06 11:18:12 +00:00
Erik Johnston	d7e8ea67b3	Reformat	2015-01-06 11:18:02 +00:00
Erik Johnston	f0128f9600	Add RoomMemberStore.get_users_in_room, so that we can get the list of joined users without having to retrieve the full events	2015-01-06 10:55:43 +00:00
Erik Johnston	d61109f578	Merge branch 'hotfixes-v0.6.0' of github.com:matrix-org/synapse into erikj-perf	2014-12-19 16:37:08 +00:00
Mark Haines	9c71d945d6	Look for name, topic in the event content rather than the event itself when persisting room name and topic events	2014-12-19 15:16:48 +00:00
David Baker	b56730bb6e	Merge branch 'develop' into pushers ... Conflicts: synapse/api/errors.py synapse/server.py synapse/storage/__init__.py	2014-12-18 15:15:22 +00:00
David Baker	afa953a293	schema version is now 10	2014-12-18 15:11:06 +00:00
David Baker	0a6664493a	Merge branch 'master' into pushers	2014-12-18 15:06:11 +00:00
David Baker	4c7ad50f6e	Thank you, pyflakes	2014-12-18 14:55:04 +00:00
David Baker	173264b656	...and bump SCHEMA_VERSION	2014-12-18 14:53:10 +00:00
David Baker	fc7c5e9cd7	Rename the pusher SQL delta to v9 which the next free one	2014-12-18 14:51:29 +00:00
David Baker	9728c305a3	after a few rethinks, a working implementation of pushers.	2014-12-18 14:49:22 +00:00
Erik Johnston	f3788e3c78	Test some ideas that might help performance a bit	2014-12-17 23:37:08 +00:00
Erik Johnston	dec5b62339	Use _get_events_txn instead of _parse_events_txn	2014-12-16 19:16:41 +00:00
Erik Johnston	2215faa361	Merge branch 'hotfixes-v0.5.4a' of github.com:matrix-org/synapse into release-v0.6.0	2014-12-16 19:11:13 +00:00
Erik Johnston	96779d2490	Fix bug where we did not send the full auth chain to people that joined over federation	2014-12-16 18:57:36 +00:00
Erik Johnston	f76269392b	Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.6.0 ... Conflicts: synapse/state.py	2014-12-16 18:35:46 +00:00
Erik Johnston	52f99243ab	Use is_outlier() so that we don't get AttributeError	2014-12-16 18:33:50 +00:00
Erik Johnston	5b39cfff69	Don't assume an event exists	2014-12-16 18:25:24 +00:00
Mark Haines	56db465047	Merge branch 'release-v0.6.0' into develop	2014-12-16 17:29:49 +00:00
Mark Haines	627e4f01d2	Remove send_message since nothing was calling it. Remove Snapshot because only send_message was using it	2014-12-16 16:07:41 +00:00
Erik Johnston	42b725ce52	Fix upgrade script to run all the missing deltas.	2014-12-16 15:13:34 +00:00
Erik Johnston	b3c793e362	Do run all deltas up to missing delta 10	2014-12-16 14:44:53 +00:00
Erik Johnston	2e44714214	Make failure to run appropraite upgrade scripts more helpful.	2014-12-16 14:20:32 +00:00
Erik Johnston	ef5a141050	Bump database version	2014-12-16 13:57:47 +00:00
Erik Johnston	35f4f6b070	Update upgrade script	2014-12-16 13:27:53 +00:00
Erik Johnston	882dc8dcab	Persist internal_metadata	2014-12-16 13:17:09 +00:00
Erik Johnston	3c77d13aa5	Kill off synapse.api.events.*	2014-12-16 11:29:05 +00:00
Erik Johnston	f280929a12	Use frozenutils	2014-12-15 17:31:36 +00:00
Erik Johnston	c8dd3314d6	Fix bug where we ignored event_edge_hashes table	2014-12-15 13:55:22 +00:00
Erik Johnston	c39beb5559	Store json as UTF-8 and not bytes	2014-12-12 14:53:37 +00:00
Erik Johnston	75085bb4d1	Pyflakes	2014-12-12 14:34:34 +00:00
Erik Johnston	cde840a82c	Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor ... Conflicts: setup.py	2014-12-11 17:48:48 +00:00
Erik Johnston	0b04369238	Fix public room joining by making sure replaces_state never points to itself.	2014-12-11 15:56:06 +00:00
Erik Johnston	9191292b0f	Fix prev_content	2014-12-11 15:16:55 +00:00
Erik Johnston	8cdebce470	Fix redactions. Fix 'age' key	2014-12-11 13:25:19 +00:00
Erik Johnston	02e4c18171	Remove dead code	2014-12-10 18:00:36 +00:00
Mark Haines	4f37c0ea9d	Merge branch 'develop' into media_repository	2014-12-10 16:55:06 +00:00
Mark Haines	61fc37e467	Merge branch 'develop' into media_repository	2014-12-10 16:14:17 +00:00
Erik Johnston	1d2a0040cf	Fix bug where we clobbered old state group values	2014-12-10 15:55:03 +00:00
Mark Haines	e5275d856e	Get the code actually working	2014-12-10 15:46:18 +00:00
Mark Haines	cc84d3ea78	Thumbnail uploaded and cached images	2014-12-10 15:40:52 +00:00
Erik Johnston	02db7eb209	Fix bug when uploading state with empty state_key	2014-12-10 14:02:48 +00:00
Erik Johnston	aae8a37e63	Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor	2014-12-10 13:18:40 +00:00
Erik Johnston	95aa903ffa	Try and figure out how and why signatures are being changed.	2014-12-10 11:37:47 +00:00
Erik Johnston	b8d30899b1	Code style.	2014-12-10 10:16:09 +00:00
Matthew Hodgson	2b1acb7671	squidge to 79 columns as per pep8	2014-12-10 00:03:55 +00:00
Erik Johnston	aa3f66cf7f	Change the way we implement get_events to be less sucky	2014-12-09 13:35:26 +00:00
Matthew Hodgson	8529fba02d	fix a million stupid bugs and make it actually work	2014-12-08 19:34:51 +00:00
Matthew Hodgson	0d3fa1ac6e	add a write-through cache on the retry schedule	2014-12-08 17:48:57 +00:00
Erik Johnston	ba3d1e2fc0	Remove unused import	2014-12-08 12:01:25 +00:00
Erik Johnston	d044121168	Various typos and bug fixes.	2014-12-08 09:08:26 +00:00
Matthew Hodgson	5cd43d4b9f	fix stupid syntax thinkos	2014-12-07 23:44:16 +00:00
Matthew Hodgson	aed62a3583	track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover.	2014-12-07 02:26:07 +00:00
Mark Haines	a953be097f	Add a method field to thumbnail storage	2014-12-05 16:31:56 +00:00
Erik Johnston	6630e1b579	Start making more things use EventContext rather than event.*	2014-12-05 16:20:48 +00:00
Erik Johnston	c31dba86ec	Convert rest and handlers to use new event structure	2014-12-04 15:50:01 +00:00
Mark Haines	c01fd5573c	Implement download support for media_repository	2014-12-04 14:22:31 +00:00
Erik Johnston	5d7c9ab789	Begin converting things to use the new Event structure	2014-12-04 11:27:59 +00:00
David Baker	88af58d41d	Update to app_id / app_instance_id (partially) and mangle to be PEP8 compliant.	2014-12-03 13:37:02 +00:00
Erik Johnston	6941a19715	Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor	2014-12-03 11:56:49 +00:00
Paul "LeoNerd" Evans	10eb8f070c	Workaround for non-uniqueness of room member events in the database confusing HAVING COUNT() test of room membership intersection (with thanks to Tom Molesworth)	2014-12-02 21:36:53 +00:00
Mark Haines	2f804a7072	Fix pyflakes and pep8 warnings	2014-12-02 19:55:18 +00:00
Mark Haines	5da65085d1	Get uploads working with new media repo	2014-12-02 19:51:47 +00:00
Mark Haines	279c48c8b4	Write the upload portion of version 1 of the media repository	2014-12-02 17:13:14 +00:00
David Baker	7642d95d5e	Merge branch 'develop' into pushers	2014-12-02 13:50:05 +00:00
Erik Johnston	ec2b5d8c28	Store full JSON of events in db	2014-12-01 16:22:07 +00:00
Erik Johnston	1505055334	Don't return outliers when we get recent events for rooms.	2014-11-27 16:38:50 +00:00
Erik Johnston	4e2ffe79a4	Don't delete the entire current_state_events table	2014-11-26 15:17:08 +00:00
Erik Johnston	cb76945688	Add update delta for schema change	2014-11-26 11:17:19 +00:00
Erik Johnston	87538711b6	Update schema to support multiple signatures	2014-11-26 11:14:30 +00:00
Erik Johnston	3598c11c8d	Correctly handle the case where we get an event for an unknown room, which turns out we are actually in	2014-11-26 10:41:08 +00:00
Erik Johnston	a46e5ef621	SYN-163: Add an order by rowid to selects. ... This should fix the bug where the edges of the graph get returned in a different order than they were inserted in, and so no get_event no longer returned the exact same JSON as was inserted. This meant that signature checks failed.	2014-11-24 10:56:36 +00:00
David Baker	eb6aedf92c	More work on pushers. Attempt to do HTTP pokes. Not sure if the actual HTTP pokes work or not yet but the retry semantics are pretty good.	2014-11-21 12:21:00 +00:00
David Baker	23465a30b6	Merge branch 'develop' into pushers	2014-11-20 18:17:46 +00:00
Mark Haines	db9ce032a4	Fix pep8 codestyle warnings	2014-11-20 17:26:36 +00:00
Mark Haines	dfdda2c871	Use module loggers rather than the root logger. Exceptions caused by bad clients shouldn't cause ERROR level logging. Fix sql logging to use 'repr' rather than 'str'	2014-11-20 17:10:37 +00:00
Mark Haines	32090aee16	Add a few missing yields, Move deferred lists inside PreserveLoggingContext because they don't interact well with the logging contexts	2014-11-20 16:24:00 +00:00
David Baker	74c3879760	Start creating a module to do generic notifications (just prints them to stdout currently!)	2014-11-19 18:20:59 +00:00
Erik Johnston	5b46ce579b	Bump version, changelog and upgrade.rst	2014-11-19 18:00:57 +00:00
Mark Haines	1731af3f29	SYN-104: When going backwards the end token should be before the last event	2014-11-18 16:45:06 +00:00
Mark Haines	ae9c2ab165	SYN-149: Send join event immediately after the room create event	2014-11-18 15:29:48 +00:00
Mark Haines	a5b88c489e	Split out sending the room alias events from creating the alias so that we can do them in the right point when creating a room	2014-11-18 15:03:13 +00:00
Mark Haines	cb4b6c844a	Merge PDUs and Events into one object	2014-11-14 21:25:02 +00:00
Mark Haines	8c2b5ea7c4	Fix PDU and event signatures	2014-11-14 19:11:04 +00:00
Mark Haines	e903c941cb	Merge branch 'develop' into request_logging ... Conflicts: setup.py synapse/storage/_base.py synapse/util/async.py	2014-11-14 11:16:50 +00:00
Erik Johnston	f04b3d5042	Store all signatures on events rather than just dropping them	2014-11-12 17:02:34 +00:00
Erik Johnston	e715741abc	Update some of the docs in event_federation	2014-11-12 16:20:30 +00:00
Erik Johnston	e24d5cb97d	Document StateStore and use transactions	2014-11-12 14:33:48 +00:00
Erik Johnston	58c0ef90c9	Add indices to state group tables	2014-11-12 14:33:48 +00:00
Erik Johnston	6fea478d2e	Fix bugs with invites/joins across federatiom. ... Both in terms of auth and not trying to fetch missing PDUs for invites, joins etc.	2014-11-12 11:24:11 +00:00
Erik Johnston	5ff0bfb81d	Fix bug where we /always/ created a new state group	2014-11-11 14:16:41 +00:00
Erik Johnston	a8e565eca8	Add an EventValidator. Fix bugs in auth ++ storage	2014-11-10 18:25:42 +00:00
Erik Johnston	cdc1b5d629	Fix regression where we did not return redacted events.	2014-11-10 15:21:30 +00:00
Erik Johnston	5d439b127b	PEP8	2014-11-10 13:46:44 +00:00
Erik Johnston	6447db063a	Fix backfill to work. Add auth to backfill request	2014-11-10 11:59:51 +00:00
Erik Johnston	65f846ade0	Notify users about invites.	2014-11-10 11:15:02 +00:00
Erik Johnston	6cb6cb9e69	Tidy up some of the unused sql tables	2014-11-10 10:31:00 +00:00
Erik Johnston	1c06806f90	Finish redaction algorithm.	2014-11-10 10:21:32 +00:00
Erik Johnston	02c3b1c9e2	Add '/event_auth/' federation api	2014-11-07 15:35:53 +00:00
Erik Johnston	16a0815fac	Fix bug in _get_auth_chain_txn	2014-11-07 11:21:20 +00:00
Erik Johnston	49948d72f3	Fix joining over federation	2014-11-07 10:53:38 +00:00
Erik Johnston	8b0e96474b	Implement method to get auth_chain from a given event_id	2014-11-07 10:53:38 +00:00
Erik Johnston	bf6b72eb55	Start implementing auth chains	2014-11-07 10:53:38 +00:00
Erik Johnston	351c64e99e	Amalgamate all power levels. ... Remove concept of reqired power levels, something similiar can be done using the new power level event.	2014-11-06 16:59:13 +00:00
Erik Johnston	4317c8e583	Implement new replace_state and changed prev_state ... `prev_state` is now a list of previous state ids, similiar to prev_events. `replace_state` now points to what we think was replaced.	2014-11-06 15:10:55 +00:00
Erik Johnston	cc44ecc62f	Get correct prev_events	2014-11-05 13:23:35 +00:00
Erik Johnston	aa76bf39ab	Remove unused imports	2014-11-04 14:14:02 +00:00
Erik Johnston	d59aa6af25	For now, don't store txn -> pdu mappings.	2014-11-03 11:35:19 +00:00
Erik Johnston	f139c02e95	Formatting	2014-11-03 11:34:49 +00:00
Erik Johnston	ecabff7eb4	Sign evnets	2014-10-31 17:08:52 +00:00
Erik Johnston	80b2710e6f	Remove unused signature storage methods	2014-10-31 17:08:36 +00:00
Erik Johnston	fb3a01fa3a	Remove unused sql file.	2014-10-31 16:04:27 +00:00
Erik Johnston	d30d79b5be	Make prev_event signing work again.	2014-10-31 15:35:39 +00:00
Erik Johnston	2f39dc19a2	Remove more references to dead PDU tables	2014-10-31 14:27:14 +00:00
Erik Johnston	bfa36a72b9	Remove PDU tables.	2014-10-31 14:00:32 +00:00
Erik Johnston	21fe249d62	Actually don't store any PDUs	2014-10-31 10:47:34 +00:00
Erik Johnston	841df4da71	Don't store any PDUs	2014-10-31 09:59:59 +00:00
Erik Johnston	f2de2d644a	Move the impl of backfill to use events.	2014-10-31 09:59:02 +00:00
Erik Johnston	ef9c4476a0	Merge branch 'develop' of github.com:matrix-org/synapse into federation_authorization	2014-10-30 11:18:28 +00:00
Mark Haines	7c06399512	Merge branch 'develop' into request_logging ... Conflicts: synapse/config/logger.py	2014-10-30 11:13:58 +00:00
Mark Haines	7d709542ca	Fix pep8 warnings	2014-10-30 11:10:17 +00:00
Erik Johnston	aa80900a8e	Fix SQL so that accepts we may want to persist events twice.	2014-10-30 10:11:06 +00:00
Mark Haines	b29517bd01	Add a request-id to each log line	2014-10-30 01:21:33 +00:00
Erik Johnston	e7858b6d7e	Start filling out and using new events tables	2014-10-29 16:59:24 +00:00
Erik Johnston	a10c2ec88d	Don't reference PDU when persisting event	2014-10-28 17:15:32 +00:00
Erik Johnston	2d1dfb3b34	Begin implementing all the PDU storage stuff in Events land	2014-10-28 16:42:35 +00:00
Erik Johnston	da1dda3e1d	Add transaction level logging and timing information. Add a _simple_delete method	2014-10-28 11:18:04 +00:00
Erik Johnston	967ce43b59	Clean up LoggingTransaction	2014-10-28 10:53:11 +00:00
Erik Johnston	8e358ef35a	Add timer to LoggingTransaction	2014-10-28 10:34:05 +00:00
Erik Johnston	ad9226eeec	Merge branch 'event_signing' of github.com:matrix-org/synapse into federation_authorization ... Conflicts: synapse/storage/__init__.py	2014-10-27 11:58:32 +00:00
Mark Haines	5e2236f9ff	fix pyflakes warnings	2014-10-27 11:19:15 +00:00
Mark Haines	acb2d171e8	Merge branch 'develop' into event_signing	2014-10-27 11:14:11 +00:00
Mark Haines	d56e389a95	Fix pyflakes warnings	2014-10-27 10:33:17 +00:00
Erik Johnston	bb4a20174c	Merge branch 'develop' of github.com:matrix-org/synapse into federation_authorization ... Conflicts: synapse/federation/transport.py synapse/handlers/message.py	2014-10-27 10:20:44 +00:00
Mark Haines	db2e350e29	Wrap preparing the database in a transaction. Otherwise it will take many seconds to complete because sqlite will create a transaction per statement	2014-10-24 19:04:26 +01:00
Erik Johnston	5662be894e	Bump database version number.	2014-10-17 20:26:18 +01:00
Erik Johnston	5ffe5ab43f	Use state groups to get current state. Make join dance actually work.	2014-10-17 18:56:42 +01:00
Mark Haines	dc3c2823ac	Merge branch 'develop' into event_signing ... Conflicts: synapse/federation/replication.py	2014-10-17 17:33:58 +01:00
Mark Haines	82c5820767	keep 'origin_server_ts' as 'ts' in the database to avoid needlessly updating schema	2014-10-17 17:31:48 +01:00
Mark Haines	f5cf7ac25b	SPEC-7: Rename 'ts' to 'origin_server_ts'	2014-10-17 17:12:25 +01:00
Mark Haines	c8f996e29f	Hash the same content covered by the signature when referencing previous PDUs rather than reusing the PDU content hashes	2014-10-17 11:40:35 +01:00
Mark Haines	bb04447c44	Include hashes of previous pdus when referencing them	2014-10-16 23:25:12 +01:00
Mark Haines	66104da10c	Sign outgoing PDUs.	2014-10-16 00:09:48 +01:00
Mark Haines	1c445f88f6	persist hashes and origin signatures for PDUs	2014-10-15 17:09:04 +01:00
Erik Johnston	e7bc1291a0	Begin making auth use event.old_state_events	2014-10-15 16:06:59 +01:00
Erik Johnston	80472ac198	Add missing package storate.state	2014-10-15 10:04:55 +01:00
Erik Johnston	5fefc12d1e	Begin implementing state groups.	2014-10-14 16:59:51 +01:00
Mark Haines	3dac27a8a9	Storage for pdu signatures	2014-10-14 14:58:31 +01:00
Mark Haines	07639c79d9	Respond with more helpful error messages for unsigned requests	2014-10-13 16:39:15 +01:00
Mark Haines	9605593d11	Merge branch 'develop' into server2server_signing ... Conflicts: synapse/storage/__init__.py tests/rest/test_presence.py	2014-09-30 17:55:06 +01:00
Mark Haines	b95a178584	SYN-75 Verify signatures on server to server transactions	2014-09-30 15:15:10 +01:00
Erik Johnston	fbf6320614	pyflakes cleanup	2014-09-30 12:38:38 +01:00
Erik Johnston	e06adc6d7e	SYN-2: Allow server admins to delete room aliases	2014-09-30 11:31:42 +01:00
Erik Johnston	7151615260	Update docstring	2014-09-29 15:35:54 +01:00
Erik Johnston	1550ab9e2f	SYN-48: Delete dead code	2014-09-29 15:04:47 +01:00
Erik Johnston	3ccb17ce59	SYN-48: Implement WHOIS rest servlet	2014-09-29 14:59:52 +01:00
Erik Johnston	c65306f877	Add auth check to test if a user is an admin or not.	2014-09-29 13:35:38 +01:00
Erik Johnston	f7d80930f2	SYN-48: Track User-Agents as well as IPs for client devices.	2014-09-29 13:35:15 +01:00
Erik Johnston	0fdf308874	Track the IP users connect with. Add an admin column to users table.	2014-09-26 16:36:24 +01:00
Erik Johnston	37bfe44046	Merge branch 'deletions' of github.com:matrix-org/synapse into develop	2014-09-25 17:02:53 +01:00
Erik Johnston	c818aa13eb	Add LIMIT to scalar subquery	2014-09-25 15:51:21 +01:00
Erik Johnston	ba87eb6753	Fix bug where we tried to insert state events with null state key	2014-09-25 14:45:27 +01:00
Erik Johnston	70899d3ab2	Rename deletions to redactions	2014-09-24 15:27:59 +01:00
Erik Johnston	7d9a84a445	Make deleting deletes not undelete	2014-09-24 14:18:08 +01:00
Erik Johnston	1e6c5b205c	Fix bug where we didn't correctly pull out the event_id of the deletion	2014-09-24 13:29:20 +01:00
Erik Johnston	4354590a69	Add v4 deltas to current sql.	2014-09-24 11:06:41 +01:00
Erik Johnston	efea61dc50	Rename 'pruned' to 'pruned_because'	2014-09-23 17:40:58 +01:00
Erik Johnston	bc250a6afa	SYN-12: Implement auth for deletion by adding a 'delete_level' on the ops levels event ... SYN-12 # comment Auth has been added.	2014-09-23 17:36:24 +01:00
Erik Johnston	b99f6eb904	Make sure we don't persist the 'pruned' key	2014-09-23 15:29:27 +01:00
Erik Johnston	78af6bbb98	Add m.room.deletion. If an event is deleted it will be returned to clients 'pruned', i.e. all client specified keys will be removed.	2014-09-23 15:28:32 +01:00
Erik Johnston	231afe464a	Add a deletions table	2014-09-22 13:42:52 +01:00
Erik Johnston	28bcd01e8d	SYN-47: Fix bug where we still returned events for rooms we had left. ... SYN-47 #resolve	2014-09-19 14:45:21 +01:00
Paul "LeoNerd" Evans	4571cf7baa	Merge branch 'develop' into test-sqlite-memory	2014-09-17 18:27:47 +01:00
Erik Johnston	59516a8bb1	Correctly handle receiving 'missing' Pdus from federation, rather than just discarding them.	2014-09-15 16:40:44 +01:00
Paul "LeoNerd" Evans	b0406b9ead	Merge remote-tracking branch 'origin/develop' into test-sqlite-memory	2014-09-15 14:15:10 +01:00
Erik Johnston	5bd9369a62	Correctly handle the 'age' key in events and pdus	2014-09-15 13:26:11 +01:00
Erik Johnston	14975ce5bc	Fix bug where we relied on the current_state_events being updated when we are handling type specific persistence	2014-09-12 17:57:02 +01:00
Erik Johnston	667e747ed1	Fix bug where we no longer stored user_id on Pdus	2014-09-12 17:56:21 +01:00
Paul "LeoNerd" Evans	1c51c8ab7d	Merge remote-tracking branch 'origin/develop' into test-sqlite-memory ... Conflicts: synapse/storage/pdu.py	2014-09-12 17:20:06 +01:00
Erik Johnston	39e3fc69e5	Make the state resolution use actual power levels rather than taking them from a Pdu key.	2014-09-12 17:11:09 +01:00
Erik Johnston	b42fe05c51	Fix bug where we incorrectly removed a remote host from the list of hosts in a room when any user from that host left that room even if they weren't the last user from that host in that room	2014-09-12 17:11:09 +01:00
Erik Johnston	ca1ae7cf9b	Fix bug where we didn't return a tuple when expected.	2014-09-12 17:11:09 +01:00
Paul "LeoNerd" Evans	aa525e4a63	More accurate docs / clearer paramter names in RoomMemberStore	2014-09-12 16:43:49 +01:00
Paul "LeoNerd" Evans	a87eac4308	Revert recent changes to RoomMemberStore	2014-09-12 15:51:51 +01:00
Paul "LeoNerd" Evans	a840ff8f3f	Now don't need the other logger.debug() call in _execute	2014-09-12 14:38:27 +01:00
Paul "LeoNerd" Evans	1c20249884	Logging of all SQL queries via the 'synapse.storage.SQL' logger	2014-09-12 14:37:55 +01:00
Paul "LeoNerd" Evans	e53d77b501	Add a .runInteraction() method on SQLBaseStore itself to wrap the .db_pool	2014-09-12 14:28:07 +01:00
Paul "LeoNerd" Evans	249e8f2277	Add a better _store_room_member_txn() method that takes separated fields instead of an event object; also add FIXME comment about a big bug in the logic	2014-09-11 18:52:35 +01:00
Paul "LeoNerd" Evans	aaf9ab68c6	Rename _store_room_member_txn to _store_room_member_from_event_txn so we can create another, more sensible function of that name	2014-09-11 18:44:04 +01:00
Paul "LeoNerd" Evans	55397f6347	prepare_database() on db_conn, not plain name, so we can pass in the connection from outside	2014-09-10 16:23:58 +01:00
Paul "LeoNerd" Evans	ce55a8cc4b	Move database preparing code out of homserver.py into storage where it belongs	2014-09-10 15:42:15 +01:00
Erik Johnston	a75f8686ba	Fix bug where we used an unbound local variable if we ended up rolling back the persist_event transaction	2014-09-09 16:27:59 +01:00
Erik Johnston	e062f2dfa8	Apparently we can't do txn.rollback(), so raise and catch an exception instead.	2014-09-08 22:37:19 +01:00
Erik Johnston	83ce57302d	Fix bug in state handling where we incorrectly identified a missing pdu. Update tests to catch this case.	2014-09-08 19:50:59 +01:00
Erik Johnston	c0577ea87a	Rollback if we try and insert duplicate events	2014-09-08 18:34:18 +01:00
Erik Johnston	781ff713ba	When getting a state event also include the previous content	2014-09-06 02:23:36 +01:00
Erik Johnston	9dd4570b68	Generate m.room.aliases event when the HS creates a room alias	2014-09-05 21:35:56 +01:00
Paul "LeoNerd" Evans	d703e712f7	Add support to _simple_insert() to do INSERT OR REPLACE	2014-09-03 18:18:41 +01:00
Matthew Hodgson	8a7c1d6a00	fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org hasn't been incorporated in time for launch.	2014-09-03 17:31:57 +01:00
Paul "LeoNerd" Evans	756e171ad0	Store SQL DDL deltas as well; attempt to upgrade the database on startup if it's too old	2014-09-03 14:14:39 +01:00
Paul "LeoNerd" Evans	da3e04df8b	Rename the 'do_users_share_a_room' to something slightly less verb-sounding	2014-09-03 10:40:48 +01:00
Mark Haines	c6eafdfbaf	Add copyright notices and fix pyflakes errors	2014-09-03 09:43:11 +01:00
Erik Johnston	4b8244fbf8	Fix bug where we didn't correctly store the ops power levels event.	2014-09-02 12:12:14 +01:00
Erik Johnston	f452899fe2	Merge branch 'develop' of github.com:matrix-org/synapse into room_config	2014-09-01 19:57:28 +01:00
Erik Johnston	b8ab9f1c0a	Add all the necessary checks to make banning work.	2014-09-01 18:24:56 +01:00
Erik Johnston	db7109c43b	Add beginnings of ban support.	2014-09-01 16:15:34 +01:00
Mark Haines	9ea1de432d	Fix homeserver config parsing	2014-09-01 15:51:15 +01:00
Erik Johnston	865469f233	Implement power level lists, default power levels and send_evnet_level/add_state_level events.	2014-09-01 14:39:09 +01:00
Mark Haines	3eb45eba0e	Merge branch 'develop' into server2server_tls	2014-08-31 16:08:20 +01:00
Erik Johnston	1118f02689	Start adding storage for new events.	2014-08-29 15:18:30 +01:00
Erik Johnston	ee06023573	Get the equalities right.	2014-08-29 13:28:06 +01:00
Mark Haines	d7ae9b90a0	Add store for server certificates and keys	2014-08-28 18:19:47 +01:00
Paul "LeoNerd" Evans	2c7c12bc6e	Initial room event stream token must be s0, not s1, or everyone will miss the very first room event	2014-08-28 17:39:34 +01:00
Mark Haines	2aeaa7b77c	Merge branch 'develop' into storage_transactions ... Conflicts: synapse/handlers/room.py synapse/storage/stream.py	2014-08-27 17:15:58 +01:00
Mark Haines	f54b70520a	Return the store_id from persist_event	2014-08-27 17:03:45 +01:00
Mark Haines	b30358f439	add _get_room_member, fix datastore methods	2014-08-27 16:51:54 +01:00
Mark Haines	a03c7f27a8	Fill out prev_events before calling persist_event	2014-08-27 14:32:19 +01:00
Erik Johnston	47519cd8c2	Merge branch 'develop' of github.com:matrix-org/synapse into stream_refactor ... Conflicts: synapse/handlers/events.py synapse/rest/events.py synapse/rest/room.py	2014-08-27 14:13:06 +01:00
Mark Haines	474dcecb11	Remove unused populate_previous_pdus	2014-08-27 11:34:31 +01:00
Mark Haines	bf05218c4b	Merge branch 'develop' into storage_transactions	2014-08-27 11:19:37 +01:00
Kegan Dougal	c65885e166	Added support for GET /events/$eventid with auth checks.	2014-08-27 10:33:01 +01:00
Mark Haines	d2798de660	Fold federation/handler into handlers/federation	2014-08-26 19:49:42 +01:00
Erik Johnston	93cff1668c	Merge branch 'develop' of github.com:matrix-org/synapse into stream_refactor	2014-08-26 18:57:55 +01:00
Erik Johnston	3a2a5b959c	WIP: Completely change how event streaming and pagination work. This reflects the change in the underlying storage model.	2014-08-26 18:57:46 +01:00
Mark Haines	64e2a5d58e	Move pdu and event persistence into a single persist_event function	2014-08-26 18:01:36 +01:00
Kegan Dougal	5a3df1d029	Feedback: Removed FeedbackRestServlet. Modified keys on FeedbackEvent. Expanded the feedback constants to fully explain what type of feedback they are.	2014-08-26 17:49:46 +01:00
Mark Haines	4b63b06cad	Merge branch 'develop' into storage_transactions ... Conflicts: synapse/api/auth.py synapse/handlers/room.py synapse/storage/__init__.py	2014-08-26 16:07:05 +01:00
Mark Haines	ac21dfff6d	Fix pyflakes errors	2014-08-26 16:02:59 +01:00
Mark Haines	32347bfcc9	fix a few pyflakes errors	2014-08-26 16:01:29 +01:00
Mark Haines	4b2ad549d5	Move the event storage into a single transaction	2014-08-26 14:36:03 +01:00
Erik Johnston	a664ec20e0	Add a do_users_share_a_room method and use that in the presence handler.	2014-08-26 13:38:26 +01:00
Kegan Dougal	47c3a089c5	Merge branch 'develop' of github.com:matrix-org/synapse into client_server_url_rename	2014-08-26 09:26:33 +01:00
Kegan Dougal	cab3095803	Removed member list servlet: now using generic state paths.	2014-08-26 09:26:07 +01:00
Erik Johnston	be6abdff19	Order 'get_recent_events_for_room' correctly.	2014-08-26 09:22:58 +01:00
Erik Johnston	9d86c8c7a6	Add a unique constraint on the room hosts table	2014-08-24 11:29:29 +01:00
Erik Johnston	537ecd4e99	Turn off spammy logging	2014-08-22 18:12:38 +01:00
Erik Johnston	9f514915af	Add indices to schema	2014-08-22 18:03:39 +01:00
Mark Haines	1379dcae6f	Take a snapshot of the state of the room before performing updates	2014-08-22 17:00:10 +01:00
Erik Johnston	808f663ed1	Don't return state event outlier's when paginating.	2014-08-22 13:06:07 +01:00
Erik Johnston	e7ee0b9fc1	Change IM sync api to also return the current presence list.	2014-08-21 16:40:21 +01:00
Erik Johnston	063e1b22e6	Stop internal keys from getting into SynapseEvents	2014-08-21 15:06:00 +01:00
Erik Johnston	ebd3c41ede	Make event stream storage return all membership events about the user, regardless of if they were in the room or not.	2014-08-20 17:09:44 +01:00
Erik Johnston	e8244c23ba	Give the event_id of the failed event	2014-08-20 15:53:07 +01:00
Erik Johnston	d4fb1c8a92	Only hit get_room_events_stream if we have a valid user_id	2014-08-19 17:18:19 +01:00
Erik Johnston	ae493c9418	Fix token to correct format	2014-08-19 16:45:55 +01:00
Erik Johnston	eea2dc7dde	Remove debug logging from token parsing funcs.	2014-08-19 16:40:38 +01:00
Erik Johnston	d94765999d	Add comment about what strorage.stream does	2014-08-19 16:40:25 +01:00
Erik Johnston	840771190f	Fix bug where we sometimes set min_token to None.	2014-08-19 14:32:47 +01:00
Erik Johnston	75b6d982a0	Add a 'backfill room' button	2014-08-19 14:20:03 +01:00
Erik Johnston	598a1d8ff9	Change the way pagination works to support out of order events.	2014-08-19 14:19:48 +01:00
Erik Johnston	1422a22970	Fix typos in SQL and where we still had rowid's (which no longer exist)	2014-08-18 16:25:18 +01:00
Erik Johnston	4eb8f84aa8	Make snapshot_all_rooms return results in the correct form, including start and end tokens.	2014-08-18 16:20:21 +01:00
Erik Johnston	709a92cee8	SQL doesn't allow AUTOINCREMENT on non PRIMARY KEY columns.	2014-08-18 16:00:46 +01:00
Erik Johnston	fc26275bb3	Add two different columns for ordering the events table, one which can be used for pagination and one which can be as tokens for notifying clients. Also add a 'processed' field which is currently always set to True	2014-08-18 15:50:41 +01:00
Erik Johnston	1c2caacd67	Merge branch 'master' of github.com:matrix-org/synapse into sql_refactor	2014-08-18 11:14:57 +01:00
Matthew Hodgson	0f9b633af7	retcols is mandatory. i have no idea how this could ever have worked?	2014-08-16 01:46:35 +01:00
Erik Johnston	0e938b1ff7	Rename method name to not clash with other ones in storage.	2014-08-15 16:47:48 +01:00
Erik Johnston	506711749f	We no longer need to special case room config events.	2014-08-15 16:45:16 +01:00
Erik Johnston	d260a42ca2	PEP8 cleanups	2014-08-15 16:17:36 +01:00
Erik Johnston	8fa3cc37f9	Comment.	2014-08-15 16:11:25 +01:00
Erik Johnston	cd2967d271	Fix bug when generating a key when get_room_events_stream returned zero rows	2014-08-15 16:05:46 +01:00
Erik Johnston	86be66c34e	Actually use MAX_STREAM_SIZE constant.	2014-08-15 16:04:54 +01:00
Erik Johnston	8d1f763209	Fix pagination to work with new db schema	2014-08-15 15:53:06 +01:00
Paul "LeoNerd" Evans	3c532314ec	Fix imsync's SELECT query to only find the rooms I'm actually joined in, not every room I have ever joined	2014-08-15 15:44:53 +01:00
Erik Johnston	01f089d9fb	Correctly return new token when returning events. Serialize events correctly.	2014-08-15 15:28:54 +01:00
Erik Johnston	114984a236	Start chagning the events stream to work with the new DB schema	2014-08-15 13:58:28 +01:00
Erik Johnston	d72f897f07	Merge branch 'master' of github.com:matrix-org/synapse into sql_refactor ... Conflicts: synapse/storage/stream.py	2014-08-15 11:50:14 +01:00
Erik Johnston	5002efa31b	Reimplement the get public rooms api to work with new DB schema	2014-08-15 10:26:35 +01:00
Erik Johnston	2c46bb6208	Fix up typos and correct sql queries	2014-08-14 18:40:50 +01:00
Erik Johnston	7e681ad778	Update StreamStore	2014-08-14 18:01:39 +01:00
Erik Johnston	661c711765	Start fixing places that use the data store.	2014-08-14 17:34:37 +01:00
Erik Johnston	78b501eba6	Fix typo	2014-08-14 17:09:28 +01:00
Erik Johnston	2529f2bc01	Rename _execute_query	2014-08-14 16:58:51 +01:00
Paul "LeoNerd" Evans	93a8be7bef	We really don't need debug logging of all the SQL statements we execute; we're quite happy these all work now	2014-08-14 16:15:57 +01:00
Kegan Dougal	fb93e14e53	Be more helpful when failing to register/login, stating why (communication error, user in user, wrong credentials, etc). Make the HS send M_USER_IN_USE.	2014-08-14 16:03:04 +01:00
Erik Johnston	937c175029	Fix up RoomMemberStore to work with the new schema.	2014-08-14 16:02:10 +01:00
Erik Johnston	6d6a1c3454	Actually encode dicts as json in the DB	2014-08-14 14:30:25 +01:00
Erik Johnston	10294b6082	Merge branch 'master' of github.com:matrix-org/synapse into sql_refactor ... Conflicts: synapse/storage/_base.py	2014-08-14 10:01:04 +01:00
Paul "LeoNerd" Evans	d05aa651f8	An initial hack at storing presence state-change mtimes in database and presenting age durations to clients/federation events	2014-08-13 19:19:15 +01:00
Paul "LeoNerd" Evans	baf04be5cf	Set datastore's .hs field in SQLBaseStore rather than in the toplevel DataStore mixed-in result class	2014-08-13 19:19:15 +01:00
Matthew Hodgson	59dfbaba3b	when we're talking about backfilling data in federation, call it backfilling - not pagination.	2014-08-13 18:17:09 +01:00
Erik Johnston	beaf4384d9	Make feedback table also store sender.	2014-08-13 18:03:41 +01:00
Erik Johnston	336987bb8d	Initial stab at refactoring the SQL tables, including rejigging some of the storage layer.	2014-08-13 18:03:41 +01:00
Erik Johnston	3dfa84bec8	Convert im schema to a 'one' table structure	2014-08-13 18:03:41 +01:00
Kegan Dougal	9b3702c924	Return the room_alias when GETing public rooms.	2014-08-13 11:57:49 +01:00
Matthew Hodgson	f98e6380f1	add in whitespace after copyright statements to improve legibility	2014-08-13 03:14:34 +01:00
matrix.org	4f475c7697	Reference Matrix Home Server	2014-08-12 15:10:52 +01:00