2379 Commits

Author	SHA1	Message	Date
Erik Johnston	c45d8e9ba2	Add profile data to the room_membership table for joins	2016-12-08 13:08:41 +00:00
Richard van der Hoff	bd07a35c29	Fix result of insert_receipt ... This should fix the absence of notifications when new receipts arrive.	2016-12-08 12:11:34 +00:00
Erik Johnston	2687af82d4	Comments	2016-12-07 09:58:33 +00:00
Erik Johnston	f32fb65552	Add new API appservice specific public room list	2016-12-06 16:12:27 +00:00
Richard van der Hoff	aa09d6b8f0	Rip out more refresh_token code ... We might as well treat all refresh_tokens as invalid. Just return a 403 from /tokenrefresh, so that we don't have a load of dead, untestable code hanging around. Still TODO: removing the table from the schema.	2016-11-30 17:40:18 +00:00
Erik Johnston	46ca345b06	Don't send old events as federation	2016-11-24 12:29:02 +00:00
Erik Johnston	1985860c6e	Comment	2016-11-23 15:59:59 +00:00
Erik Johnston	2ac516850b	More efficient notif count queries	2016-11-23 15:57:04 +00:00
Erik Johnston	302fbd218d	Merge pull request #1635 from matrix-org/erikj/split_out_fed_txn ... Split out federation transaction sending to a worker	2016-11-23 15:39:12 +00:00
Erik Johnston	b69f76c106	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_fed_txn	2016-11-23 11:31:53 +00:00
Erik Johnston	0163466d72	Ignore AS users when fetching push rules ... By ignoring AS users early on when fetching push rules for a room we can avoid needlessly hitting the DB and filling up the caches.	2016-11-23 11:01:01 +00:00
Erik Johnston	54fed21c04	Fix tests and flake8	2016-11-22 18:18:31 +00:00
Erik Johnston	90565d015e	Invalidate retry cache in both directions	2016-11-22 17:45:44 +00:00
Kegan Dougal	a2a6c1c22f	Fail with a coherent error message if /sync?filter= is invalid	2016-11-21 13:15:25 +00:00
Erik Johnston	524d61bf7e	Fix tests	2016-11-21 11:53:02 +00:00
Erik Johnston	7c9cdb2245	Store federation stream positions in the database	2016-11-21 11:33:08 +00:00
Erik Johnston	f8ee66250a	Handle sending events and device messages over federation	2016-11-17 15:48:04 +00:00
Erik Johnston	ed787cf09e	Hook up the send queue and create a federation sender worker	2016-11-16 17:34:44 +00:00
Erik Johnston	c7546b3cdb	Merge pull request #1617 from matrix-org/erikj/intern_state_dict ... Correctly intern keys in state cache	2016-11-15 16:45:55 +00:00
Erik Johnston	f9d156d270	New Flake8 fixes	2016-11-15 11:22:29 +00:00
Erik Johnston	587d8ac60f	Correctly intern keys in state cache	2016-11-08 11:53:25 +00:00
Erik Johnston	63772443e6	Comment	2016-11-04 10:53:42 +00:00
Erik Johnston	a3f6576084	Remove unused but buggy function	2016-11-04 10:48:20 +00:00
Erik Johnston	64c6566980	Remove spurious comment	2016-11-03 15:04:32 +00:00
Erik Johnston	8fd4d9129f	Replace postgres GIN with GIST ... This is because GIN can be slow to write too, especially when the table gets large.	2016-11-03 15:00:03 +00:00
Erik Johnston	760469c812	Continue to clean up received_transactions	2016-11-01 11:42:08 +00:00
Erik Johnston	f09d2b692f	Removed unused stuff	2016-10-31 17:10:56 +00:00
Erik Johnston	4c3eb14d68	Increase batching of sent transaction inserts ... This should further reduce the number of individual inserts, transactions and updates that are required for keeping sent_transactions up to date.	2016-10-31 16:07:45 +00:00
Erik Johnston	a9111786f9	Use most recently added binding, not most recently seen user.	2016-10-27 14:32:45 +01:00
Erik Johnston	1fc1bc2a51	Fix user_threepids schema delta ... The delta `37/user_threepids.sql` aimed to update all the email addresses to be lower case, however duplicate emails may exist in the table already. This commit adds a step where the delta moves the duplicate emails to a new `medium` `email_old`. Only the most recently used account keeps the binding intact. We move rather than delete so that we retain some record of which emails were associated with which account.	2016-10-27 14:14:44 +01:00
Erik Johnston	d04e2ff3a4	Fix incredubly slow back pagination query ... If a client didn't specify a from token when paginating backwards synapse would attempt to query the (global) maximum topological token. This a) doesn't make much sense since they're room specific and b) there are no indices that lets postgres do this efficiently.	2016-10-24 13:35:51 +01:00
Erik Johnston	78c083f159	Merge pull request #1164 from pik/error-codes ... Clarify Error codes for GET /filter/	2016-10-19 14:26:17 +01:00
David Baker	0108ed8ae6	Latest delta is now 37	2016-10-19 11:40:35 +01:00
David Baker	a7f48320b1	Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive	2016-10-19 11:28:56 +01:00
David Baker	df2a616c7b	Convert emails to lowercase when storing ... And db migration sql to convert existing addresses.	2016-10-19 11:13:55 +01:00
pik	e8b1d2a452	Refactor test_filter to use real DataStore ... * add tests for filter api errors	2016-10-18 12:17:38 -05:00
Erik Johnston	6942d68247	Bump schema version	2016-10-17 11:17:45 +01:00
Erik Johnston	b59994b454	Remove TODO	2016-10-17 11:17:02 +01:00
Erik Johnston	816988baaa	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth	2016-10-17 11:10:37 +01:00
Erik Johnston	2869a29fd7	Drop some unused indices	2016-10-17 11:08:19 +01:00
David Baker	29c5922021	Revert part of 6207399 ... older sqlite doesn't support indexes on expressions, lets just store things lowercase in the db	2016-10-14 16:20:24 +01:00
David Baker	bcb1245a2d	Merge remote-tracking branch 'origin/develop' into dbkr/password_reset_case_insensitive	2016-10-14 15:10:38 +01:00
David Baker	62073992c5	Make password reset email field case insensitive	2016-10-14 13:56:53 +01:00
Erik Johnston	3061dac53e	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/replication_noop	2016-10-11 14:08:29 +01:00
Patrik Oldsberg	9bfc617791	storage/appservice: make appservice methods only relying on the cache synchronous	2016-10-06 15:24:59 +02:00
Erik Johnston	5875a65253	Merge pull request #1145 from matrix-org/erikj/fix_reindex ... Fix background reindex of origin_server_ts	2016-09-29 13:53:48 +01:00
Erik Johnston	9040c9ffa1	Fix background reindex of origin_server_ts ... The storage function `_get_events_txn` was removed everywhere except from this background reindex. The function was removed due to it being (almost) completely unused while also being large and complex. Therefore, instead of resurrecting `_get_events_txn` we manually reimplement the bits that are needed directly.	2016-09-27 11:23:49 +01:00
Erik Johnston	4974147aa3	Remove duplication	2016-09-27 09:27:54 +01:00
Erik Johnston	13122e5e24	Remove unused variable	2016-09-27 09:21:51 +01:00
Erik Johnston	cf3e1cc200	Fix perf of fetching state in SQLite	2016-09-26 17:16:24 +01:00
Erik Johnston	748d8fdc7b	Reduce DB hits for replication ... Some streams will occaisonally advance their positions without actually having any new rows to send over federation. Currently this means that the token will not advance on the workers, leading to them repeatedly sending a slightly out of date token. This in turns requires the master to hit the DB to check if there are any new rows, rather than hitting the no op logic where we check if the given token matches the current token. This commit changes the API to always return an entry if the position for a stream has changed, allowing workers to advance their tokens correctly.	2016-09-23 16:49:21 +01:00
Erik Johnston	8009d84364	Match against event_id, rather than room_id	2016-09-21 16:46:59 +01:00
Erik Johnston	dc692556d6	Remove spurious AS clause	2016-09-21 16:28:47 +01:00
Erik Johnston	dc78db8c56	Update correct table	2016-09-21 15:52:44 +01:00
Erik Johnston	4f78108d8c	Readd entries to public_room_list_stream that were deleted	2016-09-21 15:24:22 +01:00
Erik Johnston	0b78d8adf2	Fix _delete_old_forward_extrem_cache query	2016-09-21 15:20:56 +01:00
Erik Johnston	a68807d426	Comment	2016-09-16 11:36:20 +01:00
Erik Johnston	e58a9d781c	Filter remote rooms lists locally	2016-09-16 10:19:32 +01:00
Erik Johnston	de4f798f01	Handling expiring stream extrems correctly.	2016-09-15 17:34:59 +01:00
Erik Johnston	955f34d23e	Change get_pos_of_last_change to return upper bound	2016-09-15 15:12:07 +01:00
Erik Johnston	cb3edec6af	Use stream_change cache to make get_forward_extremeties_for_room cache more effective	2016-09-15 14:28:13 +01:00
Erik Johnston	55e6fc917c	Add cache to get_forward_extremeties_for_room	2016-09-15 14:04:28 +01:00
Erik Johnston	e457034e99	Merge pull request #1121 from matrix-org/erikj/public_room_paginate ... Add pagination support to publicRooms	2016-09-15 13:27:09 +01:00
Erik Johnston	211786ecd6	Stream public room changes down replication	2016-09-15 11:47:23 +01:00
Erik Johnston	4fb65a1091	Base public room list off of public_rooms stream	2016-09-15 11:27:04 +01:00
Erik Johnston	c566f0ee17	Calculate the public room list from a stream_ordering	2016-09-14 17:42:47 +01:00
Erik Johnston	baffe96d95	Add a room visibility stream	2016-09-14 17:29:19 +01:00
Erik Johnston	ed992ae6ba	Add a DB index to figure out past state at a stream ordering in a room	2016-09-14 16:20:27 +01:00
Erik Johnston	00f51493f5	Fix reindex	2016-09-14 10:18:30 +01:00
Erik Johnston	d5ae1f1291	Ensure we don't mutate state cache entries	2016-09-14 10:03:48 +01:00
Erik Johnston	43253c10b8	Remove redundant event_auth index	2016-09-13 11:47:48 +01:00
Erik Johnston	b17af156c7	Remove where clause	2016-09-12 17:05:54 +01:00
Erik Johnston	0b32bb20bb	Index contains_url for file search queries	2016-09-12 16:57:05 +01:00
Erik Johnston	c94de0ab60	Add WHERE clause support to index creation	2016-09-12 16:55:01 +01:00
Erik Johnston	7c679b1118	Merge pull request #1106 from matrix-org/erikj/state_reindex_concurrent ... Create new index concurrently	2016-09-12 14:38:40 +01:00
Erik Johnston	d080b3425c	Merge pull request #1105 from matrix-org/erikj/make_notif_highlight_query_fast ... Optimise /notifications query	2016-09-12 14:34:12 +01:00
Erik Johnston	03a98aff3c	Create new index concurrently	2016-09-12 14:27:01 +01:00
Erik Johnston	fa20c9ce94	Change the index to be stream_ordering, highlight	2016-09-12 14:04:08 +01:00
Erik Johnston	5ef5435529	Remove unused import	2016-09-12 13:32:58 +01:00
Mark Haines	aa7b890cfe	Merge pull request #1104 from matrix-org/markjh/direct_to_device_federation_sync ... Fix direct to device messages recieved over federation to notify sync	2016-09-12 13:25:23 +01:00
Erik Johnston	7cd6edb947	Use register_background_index_update	2016-09-12 12:54:48 +01:00
Erik Johnston	0294c14ec4	Add back in query change	2016-09-12 12:43:56 +01:00
Erik Johnston	7fe42cf949	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/make_notif_highlight_query_fast	2016-09-12 12:37:09 +01:00
Erik Johnston	15ca0c6a4d	Make reindex happen in bg	2016-09-12 12:36:36 +01:00
Mark Haines	a232e06100	Fix direct to device messages recieved over federation to notify sync	2016-09-12 12:30:46 +01:00
Mark Haines	31f85f9db9	Add comments to existing schema deltas that used "CREATE INDEX" directly	2016-09-12 11:00:26 +01:00
Erik Johnston	54417999b6	Revert "Add index to event_push_actions"	2016-09-12 10:39:55 +01:00
Erik Johnston	897d57bc58	Change state fetch query for postgres to be faster ... It turns out that postgres doesn't like doing a list of OR's and is about 1000x slower, so we just issue a query for each specific type seperately.	2016-09-12 10:05:07 +01:00
David Baker	b91e2833b3	Merge remote-tracking branch 'origin/develop' into dbkr/make_notif_highlight_query_fast	2016-09-09 19:11:34 +01:00
David Baker	f2acc3dcf9	Add index to event_push_actions ... and remove room_id caluse so it uses it Mostly from @negativemjark	2016-09-09 18:54:54 +01:00
Mark Haines	647c724573	Use the previous MAX value if any to set the stream_id	2016-09-09 11:52:44 +01:00
Mark Haines	6a6cbfcf1e	Track the max_stream_device_id in a separate table, since we delete from the inbox table	2016-09-09 11:48:23 +01:00
Mark Haines	303b6f29f0	Merge pull request #1087 from matrix-org/markjh/reapply_delta ... Reapply 34/device_outbox in 35/device_outbox_again.py since the schem…	2016-09-09 11:24:27 +01:00
Erik Johnston	1fe7ca1362	Merge branch 'release-v0.17.3' of github.com:matrix-org/synapse into develop	2016-09-09 11:15:40 +01:00
Mark Haines	0877157353	Just move the schema and add some DROPs	2016-09-09 11:04:47 +01:00
Erik Johnston	2ffec928e2	Reduce batch size to be under SQL limit	2016-09-09 11:03:31 +01:00
Matthew Hodgson	b8f84f99ff	Merge pull request #1081 from matrix-org/dbkr/notifications_only_highlight ... Implement `only=highlight` on `/notifications`	2016-09-09 00:09:51 +01:00
Mark Haines	43b77c5d97	Only catch databas errors	2016-09-08 17:44:21 +01:00
Mark Haines	7d5b142547	Add a stub run_upgrade	2016-09-08 17:39:11 +01:00
David Baker	c3276aef25	Merge pull request #1080 from matrix-org/dbkr/fix_notifications_api_with_from ... Fix /notifications API when used with `from` param	2016-09-08 17:35:35 +01:00
Mark Haines	fa722a699c	Reapply 34/device_outbox in 35/device_outbox_again.py since the schema was bumped before it landed on develop	2016-09-08 17:35:16 +01:00
Erik Johnston	5c688739d6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/reindex_state_groups	2016-09-08 16:52:09 +01:00
Erik Johnston	ebb46497ba	Add delta file	2016-09-08 16:38:54 +01:00
Erik Johnston	5beda10bbd	Reindex state_groups_state after pruning	2016-09-08 16:18:01 +01:00
Mark Haines	a1c8f268e5	Support wildcard device_ids for direct to device messages	2016-09-08 15:13:05 +01:00
Mark Haines	fa9d36e050	Merge branch 'develop' into markjh/direct_to_device_federation	2016-09-08 13:43:43 +01:00
David Baker	4ef222ab61	Implement only=highlight on /notifications	2016-09-08 13:43:35 +01:00
Erik Johnston	61cd9af09b	Log delta files we're applying	2016-09-08 13:40:46 +01:00
David Baker	c5b49eb7ca	Fix /notifications API when used with from param	2016-09-08 09:40:10 +01:00
Erik Johnston	b568ca309c	Temporarily disable sequential scans for state fetching	2016-09-08 09:38:54 +01:00
Mark Haines	3c320c006c	Merge pull request #1077 from matrix-org/markjh/device_logging ... Log the types and values when failing to store devices	2016-09-07 18:24:24 +01:00
Mark Haines	85b51fdd6b	Log the types and values when failing to store devices	2016-09-07 17:19:18 +01:00
Mark Haines	2a0159b8ae	Fix the stream change cache to work over replication	2016-09-07 15:58:00 +01:00
Mark Haines	31a07d2335	Add stream change caches for device messages	2016-09-07 15:27:07 +01:00
Erik Johnston	513188aa56	Comment	2016-09-07 14:53:23 +01:00
Erik Johnston	fadb01551a	Add appopriate framing clause	2016-09-07 14:39:01 +01:00
Erik Johnston	d25c20ccbe	Use windowing function to make use of index	2016-09-07 14:22:22 +01:00
Mark Haines	7d893beebe	Comment the add_messages storage functions	2016-09-07 12:03:37 +01:00
Mark Haines	d4a35ada28	Send device messages over federation	2016-09-06 18:16:20 +01:00
Mark Haines	e020834e4f	Add storage methods for federated device messages	2016-09-06 15:12:13 +01:00
Mark Haines	2ad72da931	Add tables for federated device messages ... Adds tables for storing the messages that need to be sent to a remote device and for deduplicating messages received.	2016-09-06 15:10:40 +01:00
Mark Haines	f4164edb70	Move _add_messages_to_device_inbox_txn into a separate method	2016-09-06 11:26:37 +01:00
Erik Johnston	0595413c0f	Scale the batch size so that we're not bitten by the minimum	2016-09-05 15:49:57 +01:00
Erik Johnston	a7032abb2e	Correctly handle reindexing state groups that already have an edge	2016-09-05 15:07:23 +01:00
Erik Johnston	9e6d88f4e2	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage	2016-09-05 15:01:33 +01:00
Erik Johnston	70332a12dd	Take value in a better way	2016-09-05 14:57:14 +01:00
Erik Johnston	373654c635	Comment about sqlite and WITH RECURSIVE	2016-09-05 14:50:36 +01:00
Erik Johnston	485d999c8a	Correctly delete old state groups in purge history API	2016-09-05 14:49:08 +01:00
Erik Johnston	bb36b93f71	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage	2016-09-05 11:53:11 +01:00
Erik Johnston	caed150363	Remove unused imports	2016-09-05 10:52:01 +01:00
Erik Johnston	80a6a445fa	Only fetch local pushers	2016-09-05 10:43:32 +01:00
Erik Johnston	628e65721b	Add comments	2016-09-05 10:41:27 +01:00
Erik Johnston	a99e933550	Add upgrade script that will slowly prune state_groups_state entries	2016-09-05 10:05:36 +01:00
Erik Johnston	3847fa38c4	Make bulk_get_push_rules_for_room use get_joined_users_from_context cache	2016-09-05 10:02:38 +01:00
Mark Haines	81b94c5750	Merge pull request #1066 from matrix-org/markjh/direct_to_device_lowerbound ... Only return new device messages in /sync	2016-09-02 16:18:34 +01:00
Mark Haines	65fa37ac5e	Only return new device messages in /sync	2016-09-02 15:50:37 +01:00
Erik Johnston	3baf641a48	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage	2016-09-02 14:54:07 +01:00
Erik Johnston	cce957e254	Bump max_entries on get_destination_retry_timings	2016-09-02 14:08:33 +01:00
Erik Johnston	657847e4c6	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/state_storage	2016-09-02 11:04:48 +01:00
Erik Johnston	598317927c	Limit the length of state chains	2016-09-02 10:41:38 +01:00
Erik Johnston	9e25443db8	Move to storing state_groups_state as deltas	2016-09-01 14:31:26 +01:00
Erik Johnston	44982606ee	Merge pull request #1060 from matrix-org/erikj/state_ids ... Assign state groups in state handler.	2016-09-01 14:20:42 +01:00
Erik Johnston	0cfd6c3161	Use state_groups table to test existence	2016-08-31 16:25:57 +01:00
Erik Johnston	5405351b14	Lower get_linearized_receipts_for_room cache size	2016-08-31 16:19:44 +01:00
Erik Johnston	1671913287	Merge pull request #1059 from matrix-org/erikj/sent_transaction_delete ... Clean up old sent transactions	2016-08-31 15:07:33 +01:00
Erik Johnston	c10cb581c6	Correctly handle the difference between prev and current state	2016-08-31 14:26:22 +01:00
Erik Johnston	ef0cc648cf	Clean up old sent transactions	2016-08-31 11:12:02 +01:00
Mark Haines	a662252758	Return the current stream position from add_messages_to_device_inbox	2016-08-31 10:42:52 +01:00
Mark Haines	1aa3e1d287	Add a replication stream for direct to device messages	2016-08-31 10:38:58 +01:00
Erik Johnston	1bb8ec296d	Generate state group ids in state layer	2016-08-31 10:09:46 +01:00
Erik Johnston	5dc2a702cf	Make _state_groups_id_gen a normal IdGenerator	2016-08-30 16:55:11 +01:00
Erik Johnston	55fc17cf4b	Merge pull request #1049 from matrix-org/erikj/presence_users_in_room ... Use state handler instead of get_users_in_room/get_joined_hosts	2016-08-30 10:50:37 +01:00
Erik Johnston	bed10f9880	Use state handler instead of get_users_in_room/get_joined_hosts	2016-08-26 14:54:30 +01:00
Mark Haines	4bbef62124	Merge remote-tracking branch 'origin/develop' into markjh/direct_to_device	2016-08-26 14:35:31 +01:00
Erik Johnston	4daa397a00	Add is_host_joined to slave storage	2016-08-26 13:10:56 +01:00
Erik Johnston	1ccdc1e93a	Cache check_host_in_room	2016-08-26 10:59:40 +01:00
Erik Johnston	778fa85f47	Make sync not pull out full state	2016-08-25 18:59:44 +01:00
Mark Haines	ab34fdecb7	Merge branch 'develop' into markjh/direct_to_device	2016-08-25 18:34:46 +01:00
Mark Haines	641efb6a39	Fix the deduplication of incoming direct-to-device messages	2016-08-25 18:14:02 +01:00
Erik Johnston	721414d98a	Add desc	2016-08-25 17:49:05 +01:00
Mark Haines	e993925279	Add store-and-forward direct-to-device messaging	2016-08-25 17:35:37 +01:00
Erik Johnston	a3dc1e9cbe	Replace context.current_state with context.current_state_ids	2016-08-25 17:32:22 +01:00
Erik Johnston	17f4f14df7	Pull out event ids rather than full events for state	2016-08-25 13:42:44 +01:00
Erik Johnston	9219139351	Preserve some logcontexts	2016-08-24 11:58:40 +01:00
Erik Johnston	122c7a43c9	Merge pull request #1038 from matrix-org/erikj/receved_txn_purge ... Delete old received_transactions rows	2016-08-23 11:02:40 +01:00
Erik Johnston	d143f211c8	Merge pull request #1028 from matrix-org/dbkr/notifications_api ... Add the Notifications API	2016-08-22 18:23:24 +01:00
Erik Johnston	aad8a1a825	Delete old received_transactions	2016-08-22 16:29:46 +01:00
Erik Johnston	e65bc7d315	Merge pull request #1031 from matrix-org/erikj/measure_notifier ... Add more Measure blocks	2016-08-22 12:13:07 +01:00
Matthew Hodgson	6e80c03d45	Merge branch 'develop' into dbkr/notifications_api	2016-08-20 00:16:18 +01:00
Erik Johnston	37adde32dc	Move defer.returnValue out of Measure	2016-08-19 18:23:44 +01:00
Erik Johnston	47dd8f02a1	Measure _get_event_from_row	2016-08-19 18:23:44 +01:00
Erik Johnston	e6784daf07	Merge pull request #1030 from matrix-org/erikj/cache_contexts ... Add concept of cache contexts	2016-08-19 16:29:58 +01:00
Erik Johnston	c0d7d9d642	Rename to on_invalidate	2016-08-19 15:13:58 +01:00
Erik Johnston	dc76a3e909	Make cache_context an explicit option	2016-08-19 15:02:38 +01:00
Erik Johnston	f164fd9220	Move _bulk_get_push_rules_for_room to storage layer	2016-08-19 14:29:20 +01:00
Erik Johnston	ba214a5e32	Remove lru option	2016-08-19 14:17:11 +01:00
Erik Johnston	b770435389	Make get_new_events_for_appservice use indices	2016-08-19 10:28:42 +01:00
David Baker	1e4217c90c	Explicit join	2016-08-18 17:53:44 +01:00
David Baker	602c84cd9c	Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api	2016-08-18 17:15:26 +01:00
Erik Johnston	07229bbdae	Add appservice worker	2016-08-18 14:59:55 +01:00
Erik Johnston	9da84a9a1e	Make AppserviceHandler stream events from database ... This is for two reasons: 1. Suppresses duplicates correctly, as the notifier doesn't do any duplicate suppression. 2. Makes it easier to connect the AppserviceHandler to the replication stream.	2016-08-18 11:54:41 +01:00
Erik Johnston	732cf72b86	Fix push_display_name_rename schema update	2016-08-17 18:12:21 +01:00
David Baker	0d7cef0943	Merge pull request #1011 from matrix-org/dbkr/contains_display_name_override ... Move display name rule	2016-08-17 16:15:45 +01:00
Erik Johnston	949629291c	Do it in storage function	2016-08-16 17:05:34 +01:00
David Baker	ad42322257	Add migration script ... To port existing rule actions & enable entries to the new name	2016-08-16 16:56:30 +01:00
Erik Johnston	25c2332071	Merge pull request #1010 from matrix-org/erikj/refactor_deletions ... Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.	2016-08-16 11:37:53 +01:00
Erik Johnston	a2427981b7	Use cached get_user_by_access_token in slaves	2016-08-16 11:24:32 +01:00
Erik Johnston	6cbd1b495e	Merge branch 'fix_integrity_retry' of https://github.com/Ralith/synapse into Ralith-fix_integrity_retry	2016-08-16 10:50:24 +01:00
Erik Johnston	dc3a00f24f	Refactor user_delete_access_tokens. Invalidate get_user_by_access_token to slaves.	2016-08-15 17:04:39 +01:00
Erik Johnston	89e786bd85	Doc get_next() context manager usage	2016-08-15 13:45:26 +01:00
Erik Johnston	d9664344ec	Rename table. Add docs.	2016-08-15 11:45:57 +01:00
Erik Johnston	64e7e11853	Implement cache replication stream	2016-08-15 11:16:45 +01:00
Erik Johnston	4d70d1f80e	Add some invalidations to a cache_stream	2016-08-15 11:15:17 +01:00
Erik Johnston	99bbd90b0d	Always run txn.after_callbacks	2016-08-15 09:45:44 +01:00
Benjamin Saunders	8a57cc3123	Add missing database corruption recovery case ... Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>	2016-08-14 11:50:22 -07:00
David Baker	b4ecf0b886	Merge remote-tracking branch 'origin/develop' into dbkr/notifications_api	2016-08-11 14:09:13 +01:00
Erik Johnston	a8b946decb	Raise 404 when couldn't find event	2016-08-05 15:31:02 +01:00
Erik Johnston	f0fa66f495	Delete more tables	2016-08-05 10:40:08 +01:00
Benjamin Saunders	a2b7102eea	Tweak integrity error recovery to work as intended	2016-08-04 20:38:08 -07:00
Erik Johnston	8ad8490cff	Fix typo	2016-08-04 15:21:29 +01:00
Erik Johnston	59fa91fe88	Retry event persistence on IntegrityError ... Due to a bug in the porting script some backfilled events were not correctly persisted, causing irrecoverable IntegrityErrors on future attempts to persist those events. This commit adds a retry mechanism invoked upon IntegrityError, where when retried the tables are purged for all references to the events being persisted.	2016-08-04 15:02:15 +01:00
Richard van der Hoff	98385888b8	PEP8	2016-08-03 15:42:08 +01:00
Richard van der Hoff	68264d7404	Include device name in /keys/query response ... Add an 'unsigned' section which includes the device display name.	2016-08-03 15:42:08 +01:00
Richard van der Hoff	530bc862dc	Merge branch 'rav/null_default_device_displayname' into develop	2016-08-03 14:30:32 +01:00
Erik Johnston	80ad710217	Remove other bit of deduplication	2016-08-03 13:25:59 +01:00
Richard van der Hoff	4fec5e57be	Default device_display_name to null ... It turns out that it's more useful to return a null device display name (and let clients decide how to handle it: eg, falling back to device_id) than using a constant string like "unknown device".	2016-08-03 11:53:00 +01:00
Erik Johnston	a8a32d2714	Ensure we only persist an event once at a time	2016-08-03 11:23:39 +01:00
Erik Johnston	5aa024e501	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader	2016-07-29 11:24:56 +01:00
Mark Haines	8dad08a950	Fix SQL to supply arguments in the same order	2016-07-29 09:57:13 +01:00
Mark Haines	0a7d3cd00f	Create separate methods for getting messages to push ... for the email and http pushers rather than trying to make a single method that will work with their conflicting requirements. The http pusher needs to get the messages in ascending stream order, and doesn't want to miss a message. The email pusher needs to get the messages in descending timestamp order, and doesn't mind if it misses messages.	2016-07-28 20:24:24 +01:00
Kegan Dougal	370135ad0b	Comment get_unread_push_actions_for_user_in_range function	2016-07-28 16:47:37 +01:00
Erik Johnston	6ede23ff1b	Add more key storage funcs into slave store	2016-07-28 15:41:26 +01:00
Richard van der Hoff	40e539683c	Merge pull request #956 from matrix-org/rav/check_device_id_on_key_upload ... Make the device id on e2e key upload optional	2016-07-27 18:16:57 +01:00
Richard van der Hoff	26cb0efa88	SQL syntax fix	2016-07-27 12:30:22 +01:00
Richard van der Hoff	d47115ff8b	Delete e2e keys on device delete	2016-07-27 12:24:52 +01:00
Richard van der Hoff	2e3d90d67c	Make the device id on e2e key upload optional ... We should now be able to get our device_id from the access_token, so the device_id on the upload request is optional. Where it is supplied, we should check that it matches. For active access_tokens without an associated device_id, we ought to register the device in the devices table. Also update the table on upgrade so that all of the existing e2e keys are associated with real devices.	2016-07-26 23:38:12 +01:00
Mark Haines	87ffd21b29	Fix a couple of bugs in the transaction and keyring code	2016-07-26 19:19:08 +01:00
Richard van der Hoff	33d7776473	Fix typo	2016-07-26 13:32:15 +01:00
Richard van der Hoff	9ad8d9b17c	Merge branch 'develop' into rav/delete_refreshtoken_on_delete_device	2016-07-26 13:29:46 +01:00
Mark Haines	9c4cf83259	Merge pull request #948 from matrix-org/markjh/auth_fixes ... Don't add rejections to the state_group, persist all rejections	2016-07-26 13:22:57 +01:00
Richard van der Hoff	8e02494166	Delete refresh tokens when deleting devices	2016-07-26 11:10:37 +01:00
Mark Haines	a6f06ce3e2	Fix how push_actions are redacted.	2016-07-26 11:05:39 +01:00
Mark Haines	efeb6176c1	Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically	2016-07-26 10:49:52 +01:00
Richard van der Hoff	012b4c1913	Implement updating devices ... You can update the displayname of devices now.	2016-07-26 07:35:48 +01:00
Richard van der Hoff	436bffd15f	Implement deleting devices	2016-07-26 07:35:48 +01:00
Mark Haines	1b3c3e6d68	Only update the events and event_json tables for rejected events	2016-07-25 18:44:30 +01:00
Mark Haines	8f7f4cb92b	Don't add the events to forward extremities if the event is rejected	2016-07-25 17:13:37 +01:00
Mark Haines	2623cec874	Don't add rejections to the state_group, persist all rejections	2016-07-25 16:12:16 +01:00
Mark Haines	955ef1f06c	fix: defer.returnValue takes one argument	2016-07-25 16:04:45 +01:00
Richard van der Hoff	2ee4c9ee02	background updates: fix assert again	2016-07-25 16:01:46 +01:00
Richard van der Hoff	9dbd903f41	background updates: Fix assertion to do something	2016-07-25 14:05:23 +01:00
Richard van der Hoff	465117d7ca	Fix background_update tests ... A bit of a cleanup for background_updates, and make sure that the real background updates have run before we start the unit tests, so that they don't interfere with the tests.	2016-07-25 12:10:42 +01:00
Richard van der Hoff	363786845b	PEP8	2016-07-22 13:21:07 +01:00
Richard van der Hoff	ec5717caf5	Create index on user_ips in the background ... user_ips is kinda big, so really we want to add the index in the background once we're running. Replace the schema delta with one which will do that. I've done this in a way that's reasonably easy to reuse as there a few other indexes I need, and I don't suppose they will be the last.	2016-07-22 13:16:39 +01:00
Erik Johnston	aede7248ab	Split out a FederationReader process	2016-07-21 17:37:44 +01:00
Richard van der Hoff	c445f5fec7	storage/client_ips: remove some dead code	2016-07-21 11:58:47 +01:00
Richard van der Hoff	7314bf4682	Merge branch 'develop' into rav/get_devices_api ... (pick up PR #938 in the hope of fixing the UTs)	2016-07-20 17:40:00 +01:00
Richard van der Hoff	bc8f265f0a	GET /devices endpoint ... implement a GET /devices endpoint which lists all of the user's devices. It also returns the last IP where we saw that device, so there is some dancing to fish that out of the user_ips table.	2016-07-20 16:42:32 +01:00
Richard van der Hoff	ec041b335e	Record device_id in client_ips ... Record the device_id when we add a client ip; it's somewhat redundant as we could get it via the access_token, but it will make querying rather easier.	2016-07-20 16:41:03 +01:00
Richard van der Hoff	053e83dafb	More doc-comments ... Fix some more comments on some things	2016-07-20 16:40:28 +01:00
David Baker	e967bc86e7	Merge pull request #932 from matrix-org/rav/register_refactor ... Further registration refactoring	2016-07-20 11:03:33 +01:00
Erik Johnston	1e2a7f18a1	Merge pull request #922 from matrix-org/erikj/file_api2 ... Feature: Add filter to /messages. Add 'contains_url' to filter.	2016-07-20 10:40:48 +01:00
Erik Johnston	f91faf09b3	Comment	2016-07-20 10:18:09 +01:00
Richard van der Hoff	40cbffb2d2	Further registration refactoring ... * `RegistrationHandler.appservice_register` no longer issues an access token: instead it is left for the caller to do it. (There are two of these, one in `synapse/rest/client/v1/register.py`, which now simply calls `AuthHandler.issue_access_token`, and the other in `synapse/rest/client/v2_alpha/register.py`, which is covered below). * In `synapse/rest/client/v2_alpha/register.py`, move the generation of access_tokens into `_create_registration_details`. This means that the normal flow no longer needs to call `AuthHandler.issue_access_token`; the shared-secret flow can tell `RegistrationHandler.register` not to generate a token; and the appservice flow continues to work despite the above change.	2016-07-19 18:46:19 +01:00
Richard van der Hoff	022b9176fe	schema fix ... device_id should be text, not bigint.	2016-07-19 11:44:05 +01:00
Richard van der Hoff	f863a52cea	Add device_id support to /login ... Add a 'devices' table to the storage, as well as a 'device_id' column to refresh_tokens. Allow the client to pass a device_id, and initial_device_display_name, to /login. If login is successful, then register the device in the devices table if it wasn't known already. If no device_id was supplied, make one up. Associate the device_id with the access token and refresh token, so that we can get at it again later. Ensure that the device_id is copied from the refresh token to the access_token when the token is refreshed.	2016-07-18 16:39:44 +01:00
Erik Johnston	f52565de50	Fix /purge_history bug ... This was caused by trying to insert duplicate backward extremeties	2016-07-15 14:23:15 +01:00
Erik Johnston	978fa53cc2	Pull out min stream_ordering from ex_outlier_stream	2016-07-15 10:22:30 +01:00
Erik Johnston	eec9609e96	event_backwards_extremeties may not be empty	2016-07-15 10:22:09 +01:00
Erik Johnston	d554ca5e1d	Add support for filters in paginate_room_events	2016-07-14 15:59:04 +01:00
Erik Johnston	b64aa6d687	Add sender and contains_url field to events table	2016-07-14 15:35:43 +01:00
Mark Haines	dfde67a6fe	Add a comment explaining allow_none	2016-07-08 15:57:06 +01:00
Mark Haines	10c843fcfb	Ensure that the guest user is in the database when upgrading accounts	2016-07-08 15:15:55 +01:00
Erik Johnston	58930da52b	Merge branch 'master' of github.com:matrix-org/synapse into develop	2016-07-08 14:11:37 +01:00
Erik Johnston	b92e7955be	Comment	2016-07-07 11:42:15 +01:00
Erik Johnston	c98e1479bd	Return 400 rather than 500	2016-07-07 11:41:07 +01:00
Erik Johnston	67f2c901ea	Add rest servlet. Fix SQL.	2016-07-06 15:56:59 +01:00
Erik Johnston	a17e7caeb7	Merge branch 'erikj/shared_secret' into erikj/test2	2016-07-06 14:46:31 +01:00
Erik Johnston	be3548f7e1	Remove spurious txn	2016-07-05 17:46:51 +01:00
Erik Johnston	4adf93e0f7	Fix for postgres	2016-07-05 17:34:25 +01:00
Erik Johnston	651faee698	Add an admin option to shared secret registration	2016-07-05 17:30:22 +01:00
Erik Johnston	8f8798bc0d	Add ReadWriteLock for pagination and history prune	2016-07-05 15:30:25 +01:00
Mark Haines	dd2ccee27d	Fix typo	2016-07-05 14:06:07 +01:00
Mark Haines	b6b0132ac7	Make get_events_around more efficient on sqlite3	2016-07-05 13:55:18 +01:00
Mark Haines	d44d11d864	Use true/false for boolean parameter inclusive to avoid potential for sqli, and possibly make the code clearer	2016-07-05 10:39:13 +01:00
Erik Johnston	2d21d43c34	Add purge_history API	2016-07-05 10:28:51 +01:00
Mark Haines	0fb76c71ac	Use different SQL for postgres and sqlite3 for when using multicolumn indexes	2016-07-04 19:44:55 +01:00
Erik Johnston	a67bf0b074	Add storage function to purge history for a room	2016-07-04 16:02:50 +01:00
Mark Haines	f18d7546c6	Use a query that postgresql optimises better for get_events_around	2016-07-04 15:48:25 +01:00
Erik Johnston	f328d95cef	Feature: Add deactivate account admin API ... Allows server admins to "deactivate" accounts, which: - Revokes all access tokens - Removes all threepids - Removes password The API is a POST to `/admin/deactivate/<user_id>`	2016-06-30 15:40:58 +01:00
Mark Haines	6783534a0f	Merge pull request #886 from matrix-org/markjh/async_commit ... Optionally make committing to postgres asynchronous.	2016-06-29 15:21:58 +01:00
Erik Johnston	a70688445d	Implement purge_media_cache admin API	2016-06-29 14:57:59 +01:00
Erik Johnston	314b146b2e	Track approximate last access time for remote media	2016-06-29 11:41:20 +01:00
David Baker	aa3a4944d5	more pep8	2016-06-24 11:45:23 +01:00
David Baker	870c45913e	Use similar naming we use in email notifs for push ... Fixes https://github.com/vector-im/vector-web/issues/1654	2016-06-24 11:41:11 +01:00
David Baker	b5fb7458d5	Actually we need to order these properly ... otherwise we'll end up returning the wrong 20	2016-06-22 18:07:14 +01:00
David Baker	f73fdb04a6	Style	2016-06-22 17:51:40 +01:00
David Baker	3a4120e49a	Put most recent 20 messages in notif ... Fixes https://github.com/vector-im/vector-web/issues/1648	2016-06-22 17:47:18 +01:00
Mark Haines	d5fb561709	Optionally make committing to postgres asynchronous. ... Useful when running tests when you don't care whether the server will lose data that it claims that it has committed.	2016-06-20 17:53:38 +01:00
Mark Haines	41e4b2efea	Add the create_profile method back since the tests use it	2016-06-17 19:20:47 +01:00
Mark Haines	9f1800fba8	Remove registered_users from the distributor. ... The only place that was observed was to set the profile. I've made it so that the profile is set within store.register in the same transaction that creates the user. This required some slight changes to the registration code for upgrading guest users, since it previously relied on the distributor swallowing errors if the profile already existed.	2016-06-17 19:14:16 +01:00
Erik Johnston	1a815fb04f	Don't hit DB for noop replications queries	2016-06-08 11:33:30 +01:00
Mark Haines	0b2158719c	Remove dead code. ... Loading push rules now happens in the datastore, so we can remove the methods that loaded them outside the datastore. The ``waiting_for_join_list`` in federation handler is populated by anything, so can be removed. The ``_get_members_events_txn`` method isn't called from anywhere so can be removed.	2016-06-07 15:07:11 +01:00
Erik Johnston	84379062f9	Fix AS retries, but with correct ordering	2016-06-07 10:24:50 +01:00
Erik Johnston	310197bab5	Fix AS retries	2016-06-07 09:34:50 +01:00
Erik Johnston	e4134c5e13	Merge pull request #841 from matrix-org/erikj/event_counter ... Add metric counter for number of persisted events	2016-06-06 14:17:40 +01:00
Erik Johnston	7aa778fba9	Add metric counter for number of persisted events	2016-06-06 11:58:09 +01:00
Erik Johnston	70aee0717c	Add events to cache when we persist them	2016-06-06 11:34:53 +01:00
Erik Johnston	cffe46408f	Don't rely on options when inserting event into cache	2016-06-03 18:25:21 +01:00
Erik Johnston	10ea3f46ba	Change the way we cache events	2016-06-03 17:57:50 +01:00
Erik Johnston	05e01f21d7	Remove event fetching from DB threads	2016-06-03 17:22:13 +01:00
Mark Haines	eef541a291	Move insert_client_ip to a separate class	2016-06-03 14:42:35 +01:00
Erik Johnston	d06f2a229e	Merge pull request #828 from matrix-org/erikj/joined_hosts_for_room ... Make get_joined_hosts_for_room use get_users_in_room	2016-06-03 11:50:30 +01:00
Erik Johnston	722472b48c	Merge pull request #825 from matrix-org/erikj/cache_push_rules ... Load push rules in storage layer so that they get cached	2016-06-03 11:44:32 +01:00
Mark Haines	b09f348530	Merge pull request #824 from matrix-org/markjh/slaved_presence_store ... Add a slaved store for presence	2016-06-03 11:26:33 +01:00
Erik Johnston	ccb56fc24b	Make get_joined_hosts_for_room use get_users_in_room	2016-06-03 11:20:23 +01:00
Erik Johnston	6a0afa582a	Load push rules in storage layer, so that they get cached	2016-06-03 11:10:00 +01:00
Mark Haines	3ae915b27e	Add a slaved store for presence	2016-06-03 11:05:53 +01:00
Erik Johnston	9c26b390a2	Only get local users	2016-06-03 11:04:31 +01:00
Erik Johnston	065e739d6e	Merge pull request #811 from matrix-org/erikj/state_users_in_room ... Use state to calculate get_users_in_room	2016-06-03 10:58:27 +01:00
Erik Johnston	696d7c5937	Merge pull request #809 from matrix-org/erikj/cache_receipts_in_room ... Add get_users_with_read_receipts_in_room cache	2016-06-03 10:58:24 +01:00
Mark Haines	80f34d7b57	Fix setting the _clock in SQLBaseStore	2016-06-02 15:23:56 +01:00
Erik Johnston	a04a2d043c	Merge pull request #807 from matrix-org/erikj/push_rules_cache ... Ensure we always return boolean in push rules	2016-06-01 18:07:48 +01:00
Erik Johnston	0f06b496d1	Merge pull request #806 from matrix-org/erikj/hash_cache ... Cache get_event_reference_hashes	2016-06-01 18:07:42 +01:00
Erik Johnston	c8285564a3	Use state to calculate get_users_in_room	2016-06-01 15:25:25 +01:00
David Baker	d60eed0710	Limit number of notifications in an email notification	2016-06-01 11:45:43 +01:00
Erik Johnston	43db0d9f6a	Add get_users_with_read_receipts_in_room cache	2016-06-01 10:54:32 +01:00
Erik Johnston	dea9f20f8c	Force boolean	2016-05-31 17:24:30 +01:00
Mark Haines	c8c5bf950a	Fix synapse/storage/schema/delta/30/as_users.py	2016-05-31 17:10:40 +01:00
Erik Johnston	c9ca285d33	Merge pull request #805 from matrix-org/erikj/push_rules_cache ... Fix GET /push_rules	2016-05-31 16:42:21 +01:00
Erik Johnston	1d4ee854e2	Fix typo	2016-05-31 15:45:53 +01:00
Erik Johnston	cca0093fa9	Change fix	2016-05-31 15:44:08 +01:00
Erik Johnston	aefd2d1cbc	Cache get_event_reference_hashes	2016-05-31 15:32:32 +01:00
Erik Johnston	10de8c2631	Merge pull request #804 from matrix-org/erikj/push_rules_cache ... Add caches to bulk_get_push_rules*	2016-05-31 15:04:40 +01:00
Erik Johnston	e5b0bbcd33	Add caches to bulk_get_push_rules*	2016-05-31 13:46:58 +01:00
Erik Johnston	85b992f621	Fix to allow start with postgres	2016-05-27 10:44:44 +01:00
David Baker	37b7e84620	Include the ts the notif was received at	2016-05-24 11:33:32 +01:00
David Baker	a24bc5b2dc	Add GET /notifications API	2016-05-23 18:33:51 +01:00
David Baker	31b5395ab6	Remove debug logging	2016-05-23 16:32:01 +01:00
David Baker	c2da3406fc	Oops, missing comma	2016-05-20 18:03:31 +01:00
David Baker	ccffb0965d	Remove stale line	2016-05-20 17:59:10 +01:00
David Baker	18d68bfee4	Handle empty events table	2016-05-20 17:58:09 +01:00
David Baker	d4503e25ed	Make deleting push actions more efficient ... There's no index on received_ts, so manually binary search using the stream_ordering index, and only update it once an hour.	2016-05-20 17:56:10 +01:00
David Baker	149fa411e2	Only delete push actions after 30 days	2016-05-20 15:25:12 +01:00
Erik Johnston	42368ea8db	Add desc to get_presence_for_users	2016-05-18 11:38:10 +01:00
Mark Haines	6a30a0bfd3	Move the functions for parsing app service config	2016-05-17 11:28:58 +01:00
Mark Haines	816df9f267	get_room_members is unused now	2016-05-16 19:51:43 +01:00
Mark Haines	1a3a2002ff	Spell "domain" correctly ... s/domian/domain/g	2016-05-16 19:17:23 +01:00
Mark Haines	b7381d5338	Allow receipts for events we haven't seen in the db	2016-05-13 15:46:41 +01:00
Erik Johnston	ba5c616ff4	Merge pull request #778 from matrix-org/erikj/add_pusher ... Fixup add_pusher	2016-05-13 14:43:23 +01:00
Erik Johnston	0c11c1be88	Spelling	2016-05-13 14:42:25 +01:00
Erik Johnston	e00e8f2166	Merge pull request #769 from matrix-org/erikj/push_actions_delete ... Delete old pushers	2016-05-13 14:41:36 +01:00
Erik Johnston	fd8e921b6e	Merge pull request #779 from matrix-org/erikj/receipts ... Use tree cache for get_linearized_receipts_for_room	2016-05-13 14:41:21 +01:00
Erik Johnston	c9aff0736c	Remove topics table	2016-05-13 14:40:38 +01:00
Mark Haines	9295fa30a8	Annotate the removed indicies with why they were removed.	2016-05-13 14:16:57 +01:00
Erik Johnston	5e50058473	Remove unused indices ... This includes removing both unused indices and indices that are subsets of other indices.	2016-05-13 13:28:07 +01:00
Erik Johnston	6da7f39d95	Use tree cache for get_linearized_receipts_for_room	2016-05-13 11:41:23 +01:00
Erik Johnston	13d37c3c56	Fixup add_pusher	2016-05-13 11:25:02 +01:00
Matrix	3b1930e8ec	unbreak schema	2016-05-10 16:42:37 +01:00
Matthew Hodgson	fe97b81c09	Merge pull request #759 from matrix-org/dbkr/email_notifs ... Send email notifications for missed messages	2016-05-10 16:30:05 +02:00
David Baker	997db04648	Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs	2016-05-10 14:40:19 +02:00
Erik Johnston	c9eb6dfc1b	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ignore_user	2016-05-09 13:21:06 +01:00
Erik Johnston	3f84da139c	Merge pull request #773 from matrix-org/erikj/get_domian_from_id ... Add and use get_domain_from_id	2016-05-09 13:21:00 +01:00
Erik Johnston	def64d6ef3	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ignore_user	2016-05-09 13:05:09 +01:00
Erik Johnston	8715731559	Merge pull request #772 from matrix-org/erikj/get_user_cache ... Add cache to get_user_by_id	2016-05-09 11:12:11 +01:00
Erik Johnston	08dfa8eee2	Add and use get_domian_from_id	2016-05-09 10:36:03 +01:00
Erik Johnston	4ea762c1a2	Add cache to get_user_by_id	2016-05-09 10:08:21 +01:00
Erik Johnston	012cb5416c	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/push_actions_delete	2016-05-06 15:59:20 +01:00
Erik Johnston	fcb2c3f0db	Remove unused import	2016-05-06 15:47:40 +01:00
Erik Johnston	fd85b167ec	Pull loop one level up	2016-05-06 15:38:42 +01:00
Erik Johnston	b6e0be701e	Queue events for persistence	2016-05-06 14:31:44 +01:00
Erik Johnston	d13459636f	Pull prev txn from in memory	2016-05-06 11:30:55 +01:00
Erik Johnston	1d275dba69	Don't needlessly enter transaction	2016-05-06 11:25:58 +01:00
Erik Johnston	56b5e83e36	Reduce database inserts when sending transactions	2016-05-06 11:20:18 +01:00
Mark Haines	9c272da05f	Add an openidish mechanism for proving to third parties that you own a given user_id	2016-05-05 13:42:44 +01:00
Erik Johnston	1f0f5ffa1e	Add bulk fetch storage API	2016-05-05 10:03:15 +01:00
Erik Johnston	fee1118a20	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/ignore_user	2016-05-04 19:08:27 +01:00
Erik Johnston	fcd1eb642d	Add primary key	2016-05-04 16:51:51 +01:00
Erik Johnston	8e6a163f27	Add timestamp and auto incrementing ID	2016-05-04 15:19:12 +01:00
David Baker	80be396464	Correct SQL statement for postgres ... In standard sql, join binds tighter than comma, so we were joining on the wrong table. Postgres follows the standard (apparently).	2016-05-04 13:19:59 +01:00
Erik Johnston	984d4a2c0f	Add /report endpoint	2016-05-04 11:28:10 +01:00
Erik Johnston	a438a6d2bc	Implement basic ignore user	2016-05-04 10:16:46 +01:00
Erik Johnston	183f23f10d	Delete old pushers	2016-05-03 14:22:33 +01:00
David Baker	35b7b8e4bc	Remove unused function	2016-04-29 20:10:34 +01:00
David Baker	b0a1036d93	Use explicit join	2016-04-29 19:28:56 +01:00
David Baker	8f99cd5996	Oops, actually specify the user id	2016-04-29 19:27:03 +01:00
David Baker	60f44c098d	Remove unnecessary if	2016-04-29 19:17:10 +01:00
David Baker	50ad8005e4	Put spaces at start of line	2016-04-29 19:16:15 +01:00
David Baker	acded821c4	Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs	2016-04-29 10:05:20 +01:00
Mark Haines	871357d539	Check that somethign has happend before running the selects	2016-04-27 11:54:13 +01:00
Erik Johnston	183cacac90	Simplify query and handle finishing correctly	2016-04-22 10:01:57 +01:00
Erik Johnston	ae571810f2	Order NULLs first	2016-04-21 18:14:18 +01:00
Erik Johnston	3ddbb1687c	Fix query	2016-04-21 18:02:36 +01:00
Erik Johnston	8fae3d7b1e	Use special UPDATE syntax	2016-04-21 18:01:49 +01:00
Erik Johnston	b57dcb4b51	Typo	2016-04-21 17:49:00 +01:00
Erik Johnston	26db18bc90	Need to do _background_update_progress_txn in actual transaction	2016-04-21 17:45:56 +01:00
Erik Johnston	e395eb1108	Update progress when creating index	2016-04-21 17:39:24 +01:00
Erik Johnston	3b0fa77f50	Fix SQL statement	2016-04-21 17:37:42 +01:00
Erik Johnston	129e403487	Create index must be on a conn	2016-04-21 17:35:51 +01:00
Erik Johnston	51bb339ab2	Create index concurrently	2016-04-21 17:16:11 +01:00
Erik Johnston	b743c1237e	Add missing run_upgrade	2016-04-21 17:12:04 +01:00
Erik Johnston	c877f0f034	Optimise event_search in postgres	2016-04-21 16:56:14 +01:00
David Baker	f63bd4ff47	Send a rather basic email notif ... Also pep8 fixes	2016-04-20 13:02:01 +01:00
Erik Johnston	61c7edfd34	Add cache to _get_state_groups_from_groups	2016-04-19 17:22:03 +01:00
David Baker	07d765209d	First bits of emailpusher ... Mostly logic of when to send an email	2016-04-19 14:24:36 +01:00
Erik Johnston	d213d69fe3	Add desc arg	2016-04-14 11:36:23 +01:00
Erik Johnston	96bcfb29c7	Add index	2016-04-14 11:26:33 +01:00
Erik Johnston	a2546b9082	Fix query for get_unread_push_actions_for_user_in_range	2016-04-14 11:08:31 +01:00
David Baker	767fc0b739	pep8	2016-04-13 14:23:27 +01:00
David Baker	54d08c8868	Only return unread notifications ... Make get_unread_push_actions_for_user_in_range only return unread event actions, being more true to its name. Done in two separate sql queries to get actions after a read receipt and those in a room wiht no receipt at all. SQL queries by Erik.	2016-04-13 14:16:45 +01:00
David Baker	d33d623f0d	Merge pull request #716 from matrix-org/dbkr/get_pushers ... Add get endpoint for pushers	2016-04-12 14:40:37 +01:00
David Baker	7984ffdc6a	Unneccessarywhitespaceisunnecessary	2016-04-12 13:55:57 +01:00
David Baker	c1267d04c5	Oops, forgot the desc.	2016-04-12 13:55:32 +01:00
David Baker	44891b4a0a	Tidy up get_pusher functions ... Decodes pushers rows on the main thread rather than the db thread and uses _simple_select_list. Also do the same to the function I copied and factor out the duplication into a helper function.	2016-04-12 13:47:17 +01:00
David Baker	7b39bcdaae	Mis-named function	2016-04-12 13:35:08 +01:00
Erik Johnston	762ada1e07	Add back backfilled parameter that was removed	2016-04-12 11:58:04 +01:00
Erik Johnston	cccf86dd05	Check if we've already backfilled events	2016-04-12 11:19:32 +01:00
David Baker	8a76094965	Add get endpoint for pushers ... As per https://github.com/matrix-org/matrix-doc/pull/308	2016-04-11 18:00:03 +01:00
David Baker	2547dffccc	Merge pull request #705 from matrix-org/dbkr/pushers_use_event_actions ... Change pushers to use the event_actions table	2016-04-11 12:58:55 +01:00
Matthew Hodgson	4bd3d25218	Merge pull request #688 from matrix-org/matthew/preview_urls ... URL previewing support	2016-04-11 10:40:29 +01:00
Matthew Hodgson	6ff7a79308	move local_media_repository_url_cache.sql to schema v31	2016-04-08 19:09:02 +01:00
Matthew Hodgson	2460d904bd	fix error checking for new SQL	2016-04-08 19:04:29 +01:00
Matthew Hodgson	fb83f6a1fc	fix SQL based on PR feedback	2016-04-08 18:55:38 +01:00
David Baker	ed3979df5f	Fix invite pushes ... * If the event is an invite event, add the invitee to list of user we run push rules for (if they have a pusher etc) * Move invite_for_me to be higher prio than member events otherwise member events matches them * Spell override right	2016-04-08 15:29:59 +01:00
Mark Haines	7e2f971c08	Remove some unused functions (#711) ... * Remove some unused functions * get_room_events_stream is only used in tests * is_exclusive_room might actually be something we want	2016-04-08 14:01:56 +01:00
David Baker	d9f38561c8	Literally a dictionary	2016-04-07 17:45:01 +01:00
David Baker	4836864f56	generate id in the main thread	2016-04-07 17:38:48 +01:00
David Baker	3fb35cbd6f	Oops, inequality fail	2016-04-07 17:33:37 +01:00
Matthew Hodgson	d6e7333ae4	Merge branch 'develop' into matthew/preview_urls	2016-04-07 17:26:44 +01:00
Mark Haines	ceb599e789	Add tests for redactions	2016-04-07 16:52:07 +01:00
David Baker	05d044aac3	pep8	2016-04-07 16:45:38 +01:00
David Baker	2d5c693fd3	Fix port script for changes merged from develop	2016-04-07 16:43:54 +01:00
David Baker	9c99ab4572	Merge remote-tracking branch 'origin/develop' into dbkr/pushers_use_event_actions	2016-04-07 16:35:22 +01:00
David Baker	92e3071623	Send badge count pushes. ... Also fix bugs with retrying.	2016-04-07 15:39:53 +01:00
David Baker	0fd1cd2400	pep8	2016-04-06 16:50:47 +01:00
Mark Haines	e815763b7f	Merge pull request #697 from matrix-org/markjh/slaveI ... Add a slaved events store class	2016-04-06 16:19:25 +01:00
David Baker	7e2c89a37f	Make pushers use the event_push_actions table instead of listening on an event stream & running the rules again. Sytest passes, but remaining to do: ... * Make badges work again * Remove old, unused code	2016-04-06 15:42:15 +01:00
Erik Johnston	b713934b2e	Merge pull request #698 from matrix-org/erikj/port_script_fix ... Don't require config to create database	2016-04-06 14:32:45 +01:00
Mark Haines	75fb9ac1be	Add a slaved events store class ... Add a test to check that get_room_names_and_aliases does the same thing on both the master and on the slave data store.	2016-04-06 14:18:35 +01:00
Erik Johnston	8aab9d87fa	Don't require config to create database	2016-04-06 14:15:45 +01:00
Mark Haines	7d11f825aa	Merge pull request #694 from matrix-org/markjh/caches ... Move _get_cache_dict into the SQLBaseStore	2016-04-06 13:21:25 +01:00
Mark Haines	87f2dec8d4	Make the cache objects be per instance rather than being global	2016-04-06 13:08:05 +01:00
Mark Haines	a1e0d316ea	Move _get_cache_dict into the SQLBaseStore	2016-04-06 13:05:19 +01:00
Erik Johnston	b29f98377d	Merge pull request #691 from matrix-org/erikj/member ... Fix stuck invites	2016-04-05 12:44:39 +01:00
Erik Johnston	df727f2126	Fix stuck invites ... If rejecting a remote invite fails with an error response don't fail the entire request; instead mark the invite as locally rejected. This fixes the bug where users can get stuck invites which they can neither accept nor reject.	2016-04-05 11:13:24 +01:00
Erik Johnston	7a77f8b6d5	Merge pull request #690 from matrix-org/erikj/member ... Store invites in a separate table.	2016-04-05 09:12:27 +01:00
Erik Johnston	0c53d750e7	Docs and indents	2016-04-04 18:02:48 +01:00
Erik Johnston	92ab45a330	Add upgrade path, rename table	2016-04-04 17:07:43 +01:00
Erik Johnston	3d76b7cb2b	Store invites in a separate table.	2016-04-04 16:30:15 +01:00
Erik Johnston	bf14883a04	Merge pull request #689 from matrix-org/erikj/member ... Do checks for memberships before creating events	2016-04-04 11:56:40 +01:00
Matthew Hodgson	9f7dc2bef7	Merge branch 'develop' into matthew/preview_urls	2016-04-04 00:38:21 +01:00
Matthew Hodgson	8b98a7e8c3	pep8	2016-04-03 12:56:29 +01:00
Matthew Hodgson	7426c86eb8	add a persistent cache of URL lookups, and fix up the in-memory one to work	2016-04-03 00:31:57 +01:00
Erik Johnston	d76d89323c	Use computed prev event ids	2016-04-01 17:39:32 +01:00
Mark Haines	89e6839a48	Merge pull request #686 from matrix-org/markjh/doc_strings ... Use google style doc strings.	2016-04-01 16:20:09 +01:00
Mark Haines	2a37467fa1	Use google style doc strings. ... pycharm supports them so there is no need to use the other format. Might as well convert the existing strings to reduce the risk of people accidentally cargo culting the wrong doc string format.	2016-04-01 16:12:07 +01:00
Mark Haines	f2b916534b	Merge pull request #684 from matrix-org/markjh/backfill_id_gen ... Use a stream id generator for backfilled ids	2016-04-01 15:13:14 +01:00
Mark Haines	9bc5b4c663	Assert that the step != 0	2016-04-01 15:08:20 +01:00
Mark Haines	35b5c4ba1b	use google style doc strings	2016-04-01 15:07:01 +01:00
Mark Haines	a2866e2e6a	Rename direction to step, apply checks consistently	2016-04-01 13:50:54 +01:00
Mark Haines	e36bfbab38	Use a stream id generator for backfilled ids	2016-04-01 13:29:05 +01:00
Mark Haines	7753fc6570	Fix the invalidation of the names and aliases cache	2016-04-01 10:34:51 +01:00
Mark Haines	5d06929169	Move the check for backfilled outside the for loop	2016-03-31 15:09:09 +01:00
Mark Haines	76503f95ed	Remove the is_new_state argument to persist event. ... Move the checks for whether an event is new state inside persist event itself. This was harder than expected because there wasn't enough information passed to persist event to correctly handle invites from remote servers for new rooms.	2016-03-31 15:00:42 +01:00
Mark Haines	62e395f0e3	Merge pull request #676 from matrix-org/markjh/replicate_stateIII ... Add replication streams for ex outliers and current state resets	2016-03-31 11:20:57 +01:00
Mark Haines	2ec5426035	Use a namedtuple rather than tuple unpacking	2016-03-31 10:33:02 +01:00
Mark Haines	1fbb094c6f	Add replication streams for ex outliers and current state resets	2016-03-30 17:19:56 +01:00
Mark Haines	8b8052909f	return the state_group for backfill	2016-03-30 16:20:07 +01:00
Mark Haines	61407986b4	Add a entry to current_state_resets table when the current state is reset	2016-03-30 16:18:46 +01:00
Mark Haines	31a9eceda5	Add a replication stream for state groups	2016-03-30 16:01:58 +01:00
Mark Haines	1e25f62ee6	Use a stream id generator to assign state group ids	2016-03-30 12:55:02 +01:00
Erik Johnston	2f0180b09e	Don't bother interning keys that are already interned	2016-03-23 16:29:46 +00:00
Erik Johnston	acdfef7b14	Intern all the things	2016-03-23 16:25:54 +00:00
Erik Johnston	75daede92f	String intern	2016-03-23 14:53:53 +00:00
Erik Johnston	fbdeb1778d	Merge pull request #664 from matrix-org/erikj/public_room_list ... Don't require alias in public room list.	2016-03-23 14:42:01 +00:00
Erik Johnston	b275765545	Comment about weird SQL	2016-03-23 14:15:32 +00:00
Erik Johnston	0c1a27b787	SQLite and postgres doesn't share a true literal	2016-03-23 14:10:49 +00:00
Erik Johnston	84afeb41f3	Ensure all old public rooms have aliases	2016-03-23 13:59:34 +00:00
Erik Johnston	2749da542c	Merge pull request #663 from matrix-org/erikj/invite_for_user ... Make get_invites return RoomsForUser	2016-03-23 13:19:26 +00:00
Erik Johnston	e14baa7a3b	Merge pull request #665 from matrix-org/erikj/dont_cache_events ... Only cache events in the event cache	2016-03-23 13:19:12 +00:00
Erik Johnston	d87a846ebc	Don't cache events in get_recent_events_for_room	2016-03-23 11:42:50 +00:00
Erik Johnston	8b0dfc9fc4	Don't cache events in get_current_state_for_key	2016-03-23 11:42:17 +00:00
Erik Johnston	b6507869cd	Make get_invites return RoomsForUser	2016-03-23 10:32:10 +00:00
Erik Johnston	99f929f36b	Make StateHandler._state_cache only store event_ids.	2016-03-22 16:06:04 +00:00
Erik Johnston	6cf0ba1466	Bump get_unread_event_push_actions_by_room_for_user cache	2016-03-22 14:18:21 +00:00
Erik Johnston	76d18a5776	Bump get_aliases_for_room cache	2016-03-22 14:08:13 +00:00
Erik Johnston	fa2f96c2e3	Merge pull request #660 from matrix-org/erikj/state_cache ... Don't cache events in _state_group_cache	2016-03-22 13:12:06 +00:00
Erik Johnston	f93304e77f	Merge pull request #659 from matrix-org/erikj/state_cache_factor ... Make stateGroupCache honour CACHE_SIZE_FACTOR	2016-03-22 13:12:01 +00:00
Erik Johnston	2c86187a1b	Don't cache events in _state_group_cache ... Instead, simply cache the event ids, relying on the event cache to cache the actual events. The problem was that while the state groups cache was limited in the number of groups it could hold, each individual group could consist of thousands of events.	2016-03-22 12:00:09 +00:00
Erik Johnston	d6ac752538	Merge pull request #657 from matrix-org/erikj/roomlist ... Add published room list edit API	2016-03-22 11:57:39 +00:00
Erik Johnston	b591277620	Make stateGroupCache honour CACHE_SIZE_FACTOR	2016-03-22 10:32:50 +00:00
Mark Haines	5244c0b48e	Remove unused backfilled parameter from persist_event	2016-03-21 18:06:08 +00:00
Erik Johnston	3e7fac0d56	Add published room list edit API	2016-03-21 15:06:07 +00:00
Erik Johnston	67ed8065db	Dedupe requested event list in _get_events	2016-03-18 14:31:31 +00:00
Mark Haines	673c96ce97	Remove dead code left over from presence changes	2016-03-17 11:03:47 +00:00
Mark Haines	ba660ecde2	Add a comment to offer a hint to an explanation for why we have a unique constraint on (app_id, pushkey, user_id)	2016-03-16 10:35:00 +00:00
Mark Haines	ee32d622ce	Fix a couple of errors when deleting pushers	2016-03-15 17:47:36 +00:00
Mark Haines	b6e8420aee	Add replication stream for pushers	2016-03-15 17:33:10 +00:00
Erik Johnston	15122da0e2	Thats not how transactions work.	2016-03-11 16:45:27 +00:00
Erik Johnston	b13035cc91	Implement logout	2016-03-11 16:27:50 +00:00
Erik Johnston	c081228439	Fix SQL statement	2016-03-11 15:09:17 +00:00
Erik Johnston	b5afe6bc38	Merge pull request #641 from matrix-org/dbkr/fix_change_password ... Fix logout on password change	2016-03-11 14:48:20 +00:00
David Baker	2dee03aee5	more pep8	2016-03-11 14:38:23 +00:00
David Baker	af59826a2f	Make select more sensible when dseleting access tokens, rename pusher deletion to match access token deletion and make exception arg optional.	2016-03-11 14:34:09 +00:00
David Baker	f523177850	Delete old, unused methods and rename new one to just be user_delete_access_tokens with an except_token_ids argument doing what it says on the tin.	2016-03-11 14:29:01 +00:00
David Baker	57c444b3ad	Dear PyCharm, please indent sensibly for me. Thx.	2016-03-11 14:25:05 +00:00
Erik Johnston	58443a022d	Merge pull request #635 from matrix-org/erikj/sync_order ... Use topological orders for initial sync timeline	2016-03-11 13:17:49 +00:00
David Baker	aa11db5f11	Fix cache invalidation so deleting access tokens (which we did when changing password) actually takes effect without HS restart. Reinstate the code to avoid logging out the session that changed the password, removed in 415c2f0549	2016-03-11 13:14:18 +00:00
Daniel Wagner-Hall	465605d616	Store appservice ID on register	2016-03-10 15:58:22 +00:00
Erik Johnston	9669a99d1a	Update users table in a batched manner	2016-03-10 15:12:19 +00:00
Erik Johnston	8a88684736	Add comment	2016-03-09 16:51:22 +00:00
Erik Johnston	af2fe6110c	Return the correct token form	2016-03-09 16:11:53 +00:00
Erik Johnston	3ecaabc7fd	Use topological orders for initial sync timeline	2016-03-09 15:45:34 +00:00
Erik Johnston	158a322e82	Ensure integer is an integer	2016-03-09 10:20:48 +00:00
Mark Haines	5f5817ab05	Merge pull request #632 from matrix-org/markjh/py3v2 ... Fix relative imports so they work in both py3 and py27	2016-03-08 18:11:10 +00:00
Daniel Wagner-Hall	3b97797c8d	Merge branch 'develop' into daniel/ick	2016-03-08 17:35:09 +00:00
Daniel Wagner-Hall	edca2d9891	Idempotent-ise schema update script ... If any ASes don't have an ID, the schema will fail, and then it will error when trying to add the column again.	2016-03-08 17:32:29 +00:00
Mark Haines	7076082ae6	Fix relative imports so they work in both py3 and py27	2016-03-08 11:45:50 +00:00
Mark Haines	9848b54cac	Prefill from the correct stream	2016-03-04 16:20:22 +00:00
Mark Haines	deda48068c	prefill the push rules stream change cache	2016-03-04 16:19:42 +00:00
Mark Haines	ebcbb23226	s/stream_ordering/event_stream_ordering/ in push	2016-03-04 16:15:23 +00:00
Mark Haines	7e9fc9b6af	/FNARG/d	2016-03-04 15:54:09 +00:00
Mark Haines	ec7460b4f2	Merge branch 'develop' into markjh/pushrule_stream	2016-03-04 14:44:34 +00:00
Mark Haines	1b4f4a936f	Hook up the push rules stream to account_data in /sync	2016-03-04 14:44:01 +00:00
Erik Johnston	dd463e246d	Merge pull request #614 from matrix-org/erikj/alias_delete ... Allow alias creators to delete aliases	2016-03-04 09:02:33 +00:00
Daniel Wagner-Hall	95481e7ba7	Merge pull request #571 from matrix-org/daniel/asids ... Mark AS users with their AS's ID	2016-03-03 13:56:28 +00:00
Mark Haines	2223204eba	Hook push rules up to the replication API	2016-03-02 17:26:20 +00:00
Mark Haines	a1cf9e3bf3	Add a stream for push rule updates	2016-03-01 18:16:37 +00:00
Mark Haines	a612ce6659	Merge pull request #489 from matrix-org/markjh/replication ... Add a /replication API for extracting the updates that happened on synapse.	2016-03-01 15:08:24 +00:00
Mark Haines	60a0f81c7a	Add a /replication API for extracting the updates that happened on ... synapse This is necessary for replicating the data in synapse to be visible to a separate service because presence and typing notifications aren't stored in a database so won't be visible to another process. This API can be used to either get the raw data by requesting the tables themselves or to just receive notifications for updates by following the streams meta-stream. Returns updates for each table requested a JSON array of arrays with a row for each row in the table. Each table is prefixed by a header row with the: name of the table, current stream_id position for the table, number of rows, number of columns and the names of the columns. This is followed by the rows that have been added to the server since the requester last asked. The API has a timeout and is hooked up to the notifier so that a slave can long poll for updates.	2016-03-01 14:49:41 +00:00
Erik Johnston	f9af8962f8	Allow alias creators to delete aliases	2016-03-01 14:46:31 +00:00
Mark Haines	54172924c8	Load the current id in the IdGenerator constructor ... Rather than loading them lazily. This allows us to remove all the yield statements and spurious arguments for the get_next methods. It also allows us to replace all instances of get_next_txn with get_next since get_next no longer needs to access the db.	2016-03-01 14:32:56 +00:00
Mark Haines	de27f7fc79	Add support for changing the actions for default rules ... See matrix-org/matrix-doc#283 Works by adding dummy rules to the push rules table with a negative priority class and then using those rules to clobber the default rule actions when adding the default rules in ``list_with_base_rules``	2016-02-26 14:28:19 +00:00
Daniel Wagner-Hall	33300673b7	Generate guest access token on 3pid invites ... This means that following the same link across multiple sessions or devices can re-use the same guest account. Note that this is somewhat of an abuse vector; we can't throw up captchas on this flow, so this is a way of registering ephemeral accounts for spam, whose sign-up we don't rate limit.	2016-02-24 14:41:25 +00:00
Erik Johnston	6451fcd085	Create a new stream_id per presence update	2016-02-23 15:51:39 +00:00
Erik Johnston	e5ad2e5267	Merge pull request #582 from matrix-org/erikj/presence ... Rewrite presence for performance.	2016-02-19 09:37:50 +00:00
Erik Johnston	e6c5e3f28a	Close cursor	2016-02-18 16:39:28 +00:00
Erik Johnston	42109a62a4	Remove unused param from get_max_token	2016-02-18 16:37:28 +00:00
Mark Haines	b9977ea667	Remove dead code for setting device specific rules. ... It wasn't possible to hit the code from the API because of a typo in parsing the request path. Since no-one was using the feature we might as well remove the dead code.	2016-02-18 16:05:13 +00:00
Erik Johnston	112283e230	Prefix TS fields with _ts	2016-02-18 10:11:43 +00:00
Daniel Wagner-Hall	7e90fb6a57	Merge branch 'develop' into daniel/roomcleanupincremental ... Conflicts: synapse/rest/client/v1/room.py	2016-02-17 15:53:59 +00:00
Erik Johnston	e5999bfb1a	Initial cut	2016-02-17 15:40:50 +00:00
Daniel Wagner-Hall	a4e278bfe7	Respond to federated invite with non-empty context ... Currently, we magically perform an extra database hit to find the inviter, and use this to guess where we should send the event. Instead, fill in a valid context, so that other callers relying on the context actually have one.	2016-02-17 15:25:12 +00:00
Mark Haines	a9c9868957	Make adding push rules idempotent ... Also remove the **kwargs from the add_push_rule method. Fixes https://matrix.org/jira/browse/SYN-391	2016-02-16 15:53:38 +00:00
Daniel Wagner-Hall	763360594d	Mark AS users with their AS's ID	2016-02-11 17:26:42 +00:00
Erik Johnston	ce14c7a995	Fix SYN-627, events are in incorrect room in /sync	2016-02-11 15:02:56 +00:00
Erik Johnston	0eff740523	Return events in correct order for /events	2016-02-11 10:07:27 +00:00
Erik Johnston	5189bfdef4	Batch fetch _get_state_groups_from_groups	2016-02-10 13:24:42 +00:00
Erik Johnston	24f00a6c33	Use _simple_select_many for _get_state_group_for_events	2016-02-10 12:57:50 +00:00
Erik Johnston	f7ef5c1d57	Merge pull request #568 from matrix-org/erikj/unread_notif ... Atomically persit push actions when we persist the event	2016-02-10 11:25:32 +00:00
Erik Johnston	0214745239	Rename functions	2016-02-10 11:09:56 +00:00
Erik Johnston	7b0d846407	Atomically persit push actions when we persist the event	2016-02-09 16:19:15 +00:00
Erik Johnston	78d6c1b5be	Change a log from debug to info	2016-02-09 14:44:12 +00:00
Erik Johnston	70a8608749	Invalidate get_last_receipt_event_id_for_user cache	2016-02-09 14:27:29 +00:00
Erik Johnston	eff12e838c	Don't load all ephemeral state for a room on every sync	2016-02-09 13:55:59 +00:00
Erik Johnston	2c1fbea531	Fix up logcontexts	2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall	79a1c0574b	Allocate guest user IDs numericcally ... The current random IDs are ugly and confusing when presented in UIs. This makes them prettier and easier to read. Also, disable non-automated registration of numeric IDs so that we don't need to worry so much about people carving out our automated address space and us needing to keep retrying ID registration.	2016-02-05 11:22:30 +00:00
Erik Johnston	4d36e73230	Actually return something sensible	2016-02-03 16:35:00 +00:00
Erik Johnston	aa4af94c69	We return dicts now.	2016-02-03 16:29:32 +00:00
Erik Johnston	b84d59c5f0	Add descriptions	2016-02-03 16:22:35 +00:00
Erik Johnston	8450114098	Merge pull request #554 from matrix-org/erikj/event_push ... Change event_push_actions_rm_tokens schema	2016-02-03 15:02:47 +00:00
Erik Johnston	f8aae79a72	Simplify get_rooms	2016-02-03 13:24:35 +00:00
Erik Johnston	6f52e90065	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_room_fix	2016-02-03 11:06:29 +00:00
Erik Johnston	771528ab13	Change event_push_actions_rm_tokens schema	2016-02-03 10:50:49 +00:00
Erik Johnston	b32121a5d1	Unused import	2016-02-03 10:30:56 +00:00
Daniel Wagner-Hall	d83d004ccd	Fix flake8 warnings for new flake8	2016-02-02 17:18:50 +00:00
Erik Johnston	8a391e33ae	s/get_room_changes_for_user/get_membership_changes_for_user/	2016-02-02 16:12:10 +00:00
Erik Johnston	477b1ed6cf	Fetch events in a separate transaction. ... This has a couple of benefits: - It reduces the time of transactions, allowing other database requests to run. - Fetching events is given a dedicated database thread, and so can't starve other database requests.	2016-02-02 15:58:14 +00:00
Erik Johnston	65e92eca49	Change the way we do public room list fetching	2016-02-02 15:21:10 +00:00
Erik Johnston	89b40b225c	Order things correctly	2016-02-01 16:32:46 +00:00
Erik Johnston	4bf448be25	Switch over /events to use per room caches	2016-02-01 16:26:51 +00:00
Erik Johnston	2d3837bec7	Merge pull request #543 from matrix-org/erikj/sync ... Cache get_room_changes_for_user	2016-02-01 15:05:06 +00:00
Erik Johnston	ceb6b8680a	Only use room_ids if in get_room_events_stream if is_guest	2016-02-01 10:33:52 +00:00
Erik Johnston	25c311eaf6	Cache get_room_changes_for_user	2016-01-29 16:52:48 +00:00
Erik Johnston	cc9c97e0dc	Invalidate _account_data_stream_cache correctly	2016-01-29 16:41:51 +00:00
Erik Johnston	8da95b6f1b	Comment. Remove superfluous order by	2016-01-29 15:39:17 +00:00
Erik Johnston	b5dbced938	Don't prefill account data	2016-01-29 14:53:59 +00:00
Erik Johnston	3d60686c0c	Actually use cache	2016-01-29 14:49:11 +00:00
Erik Johnston	45488e0ffa	Max is not a function	2016-01-29 14:42:01 +00:00
Erik Johnston	f67d60496a	Convert param style	2016-01-29 14:41:16 +00:00
Erik Johnston	18579534ea	Prefill stream change caches	2016-01-29 14:37:59 +00:00
Erik Johnston	ebc5f00efe	Bump AccountDataAndTagsChangeCache size	2016-01-29 13:37:40 +00:00
Erik Johnston	b18114e19e	Merge pull request #536 from matrix-org/erikj/sync ... Make /sync "better".	2016-01-29 13:04:51 +00:00
Erik Johnston	467c27a1f9	Amalgamate tags and account data stream caches	2016-01-28 18:20:00 +00:00
Erik Johnston	03b2c2577c	Don't use defer.returnValue	2016-01-28 17:29:24 +00:00
Erik Johnston	8fe8951a8d	Cache filters	2016-01-28 17:09:09 +00:00
Erik Johnston	fdca8ec418	Add events index	2016-01-28 16:41:59 +00:00
Erik Johnston	45cf827c8f	Change name and doc has_entity_changed	2016-01-28 16:39:18 +00:00
Erik Johnston	00cb3eb24b	Cache tags and account data	2016-01-28 16:37:41 +00:00
Erik Johnston	c23a8c7833	Ensure keys to RoomStreamChangeCache are ints	2016-01-28 15:55:26 +00:00
Erik Johnston	e1941442d4	Invalidate caches properly. Remove unused arg	2016-01-28 15:02:41 +00:00
Erik Johnston	19fd425928	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sync	2016-01-28 14:46:08 +00:00
Erik Johnston	167d1df699	Merge pull request #534 from matrix-org/erikj/setup ... Add a Homeserver.setup method	2016-01-28 14:45:36 +00:00
Erik Johnston	7ed2bbeb11	Clean up a bit. Add comment	2016-01-28 14:32:05 +00:00
Erik Johnston	4e7948b47a	Allow paginating backwards from stream token	2016-01-28 11:52:34 +00:00
Erik Johnston	ba8931829b	Return correct type of token	2016-01-28 11:34:17 +00:00
Erik Johnston	61eaa6ec64	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/sync	2016-01-28 10:50:57 +00:00
Erik Johnston	c5e7c0e436	Up get_rooms_for_user cache size	2016-01-28 09:58:45 +00:00
Erik Johnston	e7febf4fbb	PEP8	2016-01-27 17:33:27 +00:00
Erik Johnston	aca3193efb	Use the same path for incremental with gap or without gap	2016-01-27 17:33:27 +00:00
Erik Johnston	b97f6626b6	Add cache to room stream	2016-01-27 17:33:26 +00:00
Erik Johnston	a955cbfa49	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/setup	2016-01-27 17:22:35 +00:00
Mark Haines	8c94833b72	Fix adding push rules relative to other rules	2016-01-27 10:24:20 +00:00
Erik Johnston	87f9477b10	Add a Homeserver.setup method. ... This is for setting up dependencies that require work on startup. This is useful for the DataStore that wants to read a bunch from the database before initiliazing.	2016-01-26 15:51:06 +00:00
Erik Johnston	8431f62ebb	Merge pull request #525 from matrix-org/erikj/select_many ... Implement a `_simple_select_many_batch`	2016-01-25 16:30:36 +00:00
Erik Johnston	f091b73e69	Merge pull request #527 from matrix-org/erikj/push_cache ... Push: Use storage apis that are cached	2016-01-25 16:16:34 +00:00
Erik Johnston	aea5da0ef6	Guard against empty iterables	2016-01-25 15:59:29 +00:00
Erik Johnston	1ebf5e3d03	Correct docstring	2016-01-25 15:53:36 +00:00
Erik Johnston	86896408b0	Add index to event_push_actions	2016-01-25 15:30:32 +00:00
Erik Johnston	53cb173663	Push: Use storage apis that are cached	2016-01-25 13:55:18 +00:00
Erik Johnston	ddd25def01	Implement a _simple_select_many_batch	2016-01-25 13:36:02 +00:00
David Baker	52bdd1b834	Overlay the push_rules_enabled map for users, otherwise they won't be able to disable server default rules.	2016-01-22 14:58:19 +00:00
David Baker	10f76dc5da	Make LRU cache not default to treecache & add options to use it	2016-01-22 12:10:33 +00:00
David Baker	42eae4634f	Use new invalidate_many cache invalidation to invalidate the event_push_actions cache appropriately.	2016-01-22 11:22:48 +00:00
Erik Johnston	8f66fe6392	Cache get_unread_event_push_actions_by_room_for_user	2016-01-21 15:02:07 +00:00
David Baker	367cfab4e6	peppate	2016-01-20 16:05:09 +00:00
David Baker	d4315bbf6b	Add index by user id on receipts_linearized	2016-01-20 15:33:27 +00:00
David Baker	3fa344c037	Add storage function to get all receipts for a user. Also add some cache invalidation to the receipts storage because there wasn't any, and remove a method that was unused.	2016-01-20 15:30:31 +00:00
Erik Johnston	5a7d1ecffc	Add regex cache. Only caculate push actions for users that have sent read receipts, and are on that server	2016-01-19 16:01:05 +00:00
Erik Johnston	3adcc4c86a	Return highlight_count in /sync	2016-01-19 11:35:50 +00:00
Erik Johnston	5cd2126a6a	Remove dead code	2016-01-18 16:49:46 +00:00
Erik Johnston	2c176e02ae	Make unit tests work	2016-01-18 14:48:50 +00:00
Erik Johnston	f59b564507	Make notifications go quicker	2016-01-18 14:48:29 +00:00
Erik Johnston	cc66a9a5e3	Allow filtering events for multiple users at once	2016-01-18 14:43:50 +00:00
Daniel Wagner-Hall	2680043bc6	Require ID and as_token be unique for ASs ... Defaults ID to as_token if not specified. This will change when IDs are fully supported.	2016-01-14 14:34:01 +00:00
Daniel Wagner-Hall	27927463a1	Merge pull request #494 from matrix-org/daniel/2 ... Don't start server if ASes are invalidly configured	2016-01-14 11:06:19 +00:00
Daniel Wagner-Hall	fcb6df45e5	Merge pull request #493 from matrix-org/daniel/1 ... Delete unused code	2016-01-14 11:06:14 +00:00
Daniel Wagner-Hall	f6fcff3602	Don't start server if ASes are invalidly configured	2016-01-13 17:09:24 +00:00
Daniel Wagner-Hall	244b356a37	Delete unused code	2016-01-13 17:03:58 +00:00
Mark Haines	9c1f853d58	Rename 'user_name' to 'user_id' in push to make it consistent with the rest of the code	2016-01-13 13:32:59 +00:00
Mark Haines	f4dad9f639	Merge remote-tracking branch 'origin/erikj/bulk_get_push_rules' into markjh/table_name ... Conflicts: synapse/storage/push_rule.py	2016-01-13 11:46:07 +00:00
Erik Johnston	8740e4e94a	bulk_get_push_rules should handle empty lists	2016-01-13 11:37:17 +00:00
Mark Haines	c0a279e808	Delete the table objects from TransactionStore	2016-01-13 11:15:20 +00:00
Mark Haines	96e400fee5	Remove the RoomsTable object	2016-01-13 11:07:32 +00:00
Mark Haines	a8e9e0b916	Remove the PushersTable and EventPushActionsTable objects	2016-01-12 14:41:26 +00:00
Mark Haines	31de2953a3	Remove the PushRuleTable and PushRuleEnableTable objects	2016-01-12 14:36:16 +00:00
David Baker	b5d33a656f	Postgres doesn't like booleans	2016-01-11 17:13:52 +00:00
David Baker	fe56138142	Remove rogue 'admin'	2016-01-11 17:09:03 +00:00
David Baker	c232780081	Merge pull request #456 from matrix-org/store_event_actions ... Send unread notification counts	2016-01-08 14:47:15 +00:00
David Baker	daadcf36c0	This comma is actually important	2016-01-07 10:15:35 +00:00
Matthew Hodgson	6c28ac260c	copyrights	2016-01-07 04:26:29 +00:00
David Baker	442fcc02f7	Merge remote-tracking branch 'origin/develop' into store_event_actions	2016-01-06 17:28:55 +00:00
David Baker	b6a585348a	Adding is_guest here won't work because it just constructs a dict of uid -> password hash	2016-01-06 17:16:02 +00:00
Daniel Wagner-Hall	5880de186b	Log when we skip daily messages	2016-01-06 13:52:16 +00:00
David Baker	992928304f	Delete notifications for redacted events	2016-01-06 11:58:46 +00:00
David Baker	ae1262a241	Add schema change file for is_guest flag	2016-01-06 11:58:20 +00:00
David Baker	c79f221192	Add is_guest flag to users db to track whether a user is a guest user or not. Use this so we can run _filter_events_for_client when calculating event_push_actions.	2016-01-06 11:38:09 +00:00
David Baker	eb03625626	Merge remote-tracking branch 'origin/develop' into store_event_actions	2016-01-05 18:39:50 +00:00
Daniel Wagner-Hall	cfd07aafff	Allow guests to upgrade their accounts	2016-01-05 18:01:18 +00:00
David Baker	4eb7b950c8	= not == in sql	2016-01-04 18:11:17 +00:00
David Baker	c914d67cda	Rename event-actions to event_push_actions as per PR request	2016-01-04 14:05:37 +00:00
David Baker	3051c9d002	Address minor PR issues	2016-01-04 13:39:29 +00:00
David Baker	4c8f6a7e42	Insert push actions in a single db query rather than one per user/profile_tag	2015-12-22 17:04:31 +00:00
David Baker	65c451cb38	Add bulk push rule evaluator which actually still evaluates rules one by one, but does far fewer db queries to fetch the rules	2015-12-22 15:19:34 +00:00
David Baker	c061b47c57	Merge remote-tracking branch 'origin/develop' into store_event_actions	2015-12-21 15:30:26 +00:00
David Baker	091c545c4f	pep8	2015-12-21 10:14:57 +00:00
David Baker	413d0d6a24	Make unread notification count sending work: put the correct count in incremental syncs too, where necessary, and fix silly bugs like only select the event actions for that user...	2015-12-18 17:47:00 +00:00
Richard van der Hoff	a64f9bbfe0	Fix 500 error when back-paginating search results ... We were mistakenly adding pagination clauses to the count query, which then failed because the count query doesn't join to the events table.	2015-12-17 12:50:46 +00:00
David Baker	42ad49f5b7	still very WIP, but now sends unread_notifications_count in the room object on sync (only actually corrrect in a full sync: hardcoded to 0 in incremental syncs).	2015-12-16 18:42:09 +00:00
Mark Haines	4fab578b43	Merge branch 'release-v0.12.0' into develop	2015-12-16 11:18:45 +00:00
Mark Haines	dcfc70e8ed	Allow users to change which account a 3pid is bound to	2015-12-15 17:02:21 +00:00
Mark Haines	15e9885197	Merge branch 'release-v0.12.0' into develop	2015-12-14 14:46:55 +00:00
Mark Haines	98dfa7d24f	Skip events that where the body, name or topic isn't a string when back populating the FTS index	2015-12-14 13:55:46 +00:00
Daniel Wagner-Hall	76e69cc8de	Fix typo	2015-12-14 12:38:55 +00:00
Mark Haines	bfc52a2342	Fix typo in sql for full text search on sqlite3	2015-12-14 11:38:11 +00:00
Mark Haines	deeebbfcb7	Merge branch 'release-v0.12.0' into develop	2015-12-12 14:21:49 +00:00
Erik Johnston	d9a5c56930	Include approximate count of search results	2015-12-11 11:40:23 +00:00
Erik Johnston	51fb590c0e	Use more efficient query form	2015-12-11 11:12:57 +00:00
David Baker	aa667ee396	Save event actions to the db	2015-12-10 17:51:15 +00:00
Mark Haines	7d6b313312	Add caches for whether a room has been forgotten by a user	2015-12-10 17:49:34 +00:00