Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-10-01 11:49:51 -04:00
Code Issues Packages Projects Releases Wiki Activity
14,130 Commits 4 Branches 615 Tags 63 MiB
5ac75fc9a2
Commit Graph

3028 Commits

Author SHA1 Message Date
Erik Johnston
1af188209a Change format of receipts to allow inclusion of data 2015-07-09 11:39:30 +01:00
Mark Haines
8fb79eeea4 Only remove one time keys when new one time keys are added 2015-07-08 17:04:29 +01:00
Erik Johnston
304343f4d7 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/receipts 2015-07-08 15:37:33 +01:00
Erik Johnston
af812b68dd Add a cache to fetching of receipt streams 2015-07-08 15:35:00 +01:00
Erik Johnston
87311d1b8c Hook up receipts to v1 initialSync 2015-07-08 11:02:04 +01:00
Erik Johnston
ca041d5526 Wire together receipts and the notifer/federation 2015-07-07 15:25:30 +01:00
Erik Johnston
716e426933 Fix various typos 2015-07-07 10:55:31 +01:00
Mark Haines
b5770f8947 Add store for client end to end keys 2015-07-06 18:46:47 +01:00
Erik Johnston
65e69dec8b Don't explode if we don't recognize one of the event_ids in the backfill request 2015-07-06 09:33:03 +01:00
Erik Johnston
1a60545626 Add basic impl for room history ACL on GET /messages client API 2015-07-02 16:20:10 +01:00
Erik Johnston
ac78e60de6 Add stream_id index 2015-07-02 13:18:41 +01:00
Erik Johnston
80a61330ee Add basic storage functions for handling of receipts 2015-07-01 17:19:12 +01:00
Erik Johnston
901f56fa63 Add tables for receipts 2015-06-30 15:29:47 +01:00
Erik Johnston
b5f55a1d85 Implement bulk verify_signed_json API 2015-06-26 10:39:34 +01:00
Erik Johnston
5130d80d79 Add bulk insert events API 2015-06-25 17:29:34 +01:00
Erik Johnston
6924852592 Batch SELECTs in _get_auth_chain_ids_txn 2015-06-23 11:01:04 +01:00
Erik Johnston
2eda996a63 Add a dummy.sql into delta/20 as pip isn't packinging the pushers.py 2015-06-13 12:21:58 +01:00
Erik Johnston
1e365e88bd Bump schema version 2015-06-11 15:50:39 +01:00
David Baker
da84946de4 pep8 2015-06-04 16:43:45 +01:00
David Baker
63a7b3ad1e Add script to (re)convert the pushers table to changing the unique key. Also give the python db upgrade scripts the database engine so they can convert parameter strings, and add *args **kwargs to the upgrade function so we can add more args in future and previous scripts will ignore them. 2015-06-04 16:16:01 +01:00
Erik Johnston
0d7d9c37b6 Add cache to get_state_groups 2015-06-03 14:45:55 +01:00
Erik Johnston
d8866d7277 Caches should be bound to instances. 
Before, caches were global and so different instances of the stores
would share caches. This caused problems in the unit tests.
 2015-06-03 14:45:17 +01:00
Erik Johnston
6e17463228 Don't explode if we don't have the event 2015-06-01 16:39:43 +01:00
Erik Johnston
a7b65bdedf Add config option to turn off freezing events. Use new encode_json api and ujson.loads 2015-05-29 12:17:33 +01:00
Mark Haines
d9f60e8dc8 Merge pull request #163 from matrix-org/markjh/presence_list_cache 
Add a cache for the presence list
 2015-05-22 17:02:23 +01:00
Mark Haines
0e42dfbe22 Merge pull request #164 from matrix-org/markjh/pusher_performance_2 
Add a cache for get_push rules for user, fix cache invalidation
 2015-05-22 17:01:56 +01:00
Mark Haines
1a9a9abcc7 Add a cache for getting the presence list for a user 2015-05-22 16:11:17 +01:00
Erik Johnston
74b7de83ec Merge branch 'develop' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-22 16:10:42 +01:00
Mark Haines
36317f3dad Merge pull request #156 from matrix-org/erikj/join_perf 
Make joining #matrix:matrix.org over federation quicker
 2015-05-22 16:09:54 +01:00
Mark Haines
b6adfc59f5 Invalidate the get_latest_event_ids_in_room cache when deleting from event_forward_extremities 2015-05-22 13:01:03 +01:00
Mark Haines
a04cde613e Add a cache for get_push rules for user, fix cache invalidation 2015-05-22 10:39:45 +01:00
Erik Johnston
4429e720ae Merge branch 'master' of github.com:matrix-org/synapse into develop 2015-05-22 10:33:00 +01:00
Mark Haines
2043527b9b Don't try to use a txn when not in one, remove spurious debug logging 2015-05-21 16:53:03 +01:00
Mark Haines
53447e9cd3 Add caches for things requested by the pushers 2015-05-21 16:41:39 +01:00
Mark Haines
d61ce3f670 Add a cache for get_current_state with state_key 2015-05-21 16:41:39 +01:00
Erik Johnston
a910984b58 Actually return something from lambda 2015-05-21 15:58:41 +01:00
Erik Johnston
e309b1045d Sort backfill events 2015-05-21 15:57:35 +01:00
Erik Johnston
0180bfe4aa Remove dead code 2015-05-21 15:53:41 +01:00
Erik Johnston
1f3d1d85a9 Only get non-state 2015-05-21 15:52:29 +01:00
Erik Johnston
39a3340f73 Skip events we've already seen 2015-05-21 15:48:56 +01:00
Erik Johnston
ae3bff3491 Correctly prepopulate queue 2015-05-21 15:46:07 +01:00
Erik Johnston
dc085ddf8c Don't prepopulate event_results 2015-05-21 15:44:05 +01:00
Erik Johnston
73d23c6ae8 Don't readd things that are already in event_results 2015-05-21 15:40:22 +01:00
Erik Johnston
6189d8e54d PriorityQueue gives lowest first 2015-05-21 15:38:08 +01:00
Erik Johnston
115ef3ddac Correctly capture Queue.Empty exception 2015-05-21 15:37:43 +01:00
Erik Johnston
c2633907c5 Merge branch 'erikj/join_perf' of github.com:matrix-org/synapse into erikj/backfill_fixes 2015-05-21 14:58:47 +01:00
Erik Johnston
ebfdd2eb5b Merge branch 'develop' of github.com:matrix-org/synapse into erikj/join_perf 2015-05-21 14:54:52 +01:00
Erik Johnston
a551c5dad7 Merge pull request #155 from matrix-org/erikj/perf 
Bulk and batch retrieval of events.
 2015-05-21 14:54:40 +01:00
Erik Johnston
27e4b45c06 s/for events/for requests for events/ 2015-05-21 14:52:23 +01:00
Erik Johnston
ac5f2bf9db s/for events/for requests for events/ 2015-05-21 14:50:57 +01:00
Erik Johnston
80a167b1f0 Add comments 2015-05-21 11:19:04 +01:00
Erik Johnston
9118a92862 Split up _get_events into defer and txn versions 2015-05-20 13:27:16 +01:00
Erik Johnston
ab45e12d31 Make not return a deferred _get_event_from_row_txn 2015-05-20 13:07:19 +01:00
Erik Johnston
f407cbd2f1 PEP8 2015-05-20 13:02:01 +01:00
Erik Johnston
227f8ef031 Split out _get_event_from_row back into defer and _txn version 2015-05-20 13:00:57 +01:00
Erik Johnston
2bc60c55af Fix _get_backfill_events to return events in the correct order 2015-05-20 12:57:00 +01:00
Erik Johnston
9084cdd70f Ensure event_results is a set 2015-05-19 16:34:31 +01:00
Erik Johnston
aa729349dd Fix event_backwards_extrem insertion to ignore outliers 2015-05-19 15:27:00 +01:00
Mark Haines
99914ec9f8 Merge pull request #152 from matrix-org/notifier_performance 
Notifier performance
 2015-05-18 17:49:59 +01:00
Erik Johnston
165eb2dbe6 Comments and shuffle of functions 2015-05-18 15:18:41 +01:00
Erik Johnston
65878a2319 Remove unused metric 2015-05-18 14:06:30 +01:00
Erik Johnston
4d1b6f4ad1 Remove rejected events if we don't want rejected events 2015-05-18 14:03:46 +01:00
Mark Haines
0b0033c40b Merge branch 'develop' into notifier_performance 2015-05-18 13:50:01 +01:00
Erik Johnston
10f1bdb9a2 Move get_events functions to storage.events 2015-05-18 10:21:40 +01:00
Erik Johnston
c71176858b Newline, remove debug logging 2015-05-18 10:11:14 +01:00
Erik Johnston
c3b37abdfd PEP8 2015-05-15 16:59:58 +01:00
Erik Johnston
6c74fd62a0 Revert limiting of fetching, it didn't help perf. 2015-05-15 16:45:35 +01:00
Erik Johnston
9ff7f66a2b init j 2015-05-15 16:36:03 +01:00
Erik Johnston
70f272f71c Don't completely drain the list 2015-05-15 16:34:17 +01:00
Erik Johnston
8763dd80ef Don't fetch prev_content for current_state 2015-05-15 15:33:01 +01:00
Erik Johnston
acb12cc811 Make store.get_current_state fetch events asyncly 2015-05-15 15:20:05 +01:00
Erik Johnston
d62dee7eae Remove more debug logging 2015-05-15 15:06:37 +01:00
Erik Johnston
0f29cfabc3 Remove debug logging 2015-05-15 14:06:42 +01:00
Erik Johnston
e275a9c0d9 preserve log context 2015-05-15 11:54:51 +01:00
Erik Johnston
aa32bd38e4 Add a wait 2015-05-15 11:35:04 +01:00
Erik Johnston
372d4c6d7b Srsly. Don't use closures. Baaaaaad 2015-05-15 11:26:00 +01:00
Erik Johnston
575ec91d82 Correctly pass through params 2015-05-15 11:15:10 +01:00
Erik Johnston
de01438a57 Sort out error handling 2015-05-15 11:00:50 +01:00
Erik Johnston
a2c4f3f150 Fix daedlock 2015-05-15 10:54:04 +01:00
Erik Johnston
1d566edb81 Remove race condition 2015-05-14 16:54:35 +01:00
Erik Johnston
ef3d8754f5 Call from right thread 2015-05-14 15:41:55 +01:00
Erik Johnston
142934084a Count and loop 2015-05-14 15:40:21 +01:00
Erik Johnston
96c5b9f87c Don't start up more fetch_events 2015-05-14 15:36:04 +01:00
Erik Johnston
7cd6a6f6cf Awful idea for speeding up fetching of events 2015-05-14 15:34:02 +01:00
Erik Johnston
7f4105a5c9 Turn off preemptive transactions 2015-05-14 14:51:06 +01:00
Erik Johnston
f4d58deba1 PEP8 2015-05-14 14:45:42 +01:00
Erik Johnston
386b7330d2 Move from _base to events 2015-05-14 14:45:22 +01:00
Erik Johnston
7d6a1dae31 Jump out early 2015-05-14 14:27:58 +01:00
Erik Johnston
656223fbd3 Actually, we probably want to run this in a transaction 2015-05-14 14:26:35 +01:00
Erik Johnston
2f7f8e1c2b Preemptively jump into a transaction if we ask for get_prev_content 2015-05-14 14:17:36 +01:00
Erik Johnston
e1e9f0c5b2 loop -> gatherResults 2015-05-14 13:58:49 +01:00
Erik Johnston
ab78a8926e Err, we probably want a bigger limit 2015-05-14 13:47:16 +01:00
Erik Johnston
f6f902d459 Move fetching of events into their own transactions 2015-05-14 13:45:48 +01:00
Erik Johnston
cdb3757942 Refactor _get_events 2015-05-14 13:31:55 +01:00
Erik Johnston
36ea26c5c0 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/perf 2015-05-14 12:01:38 +01:00
David Baker
7c549dd557 Add ID generator for push_rules_enable to #resolve SYN-378 2015-05-14 11:44:03 +01:00
Mark Haines
899d4675dd Merge branch 'notifier_unify' into notifier_performance 2015-05-14 11:36:44 +01:00
Erik Johnston
4f1d984e56 Add index on events 2015-05-13 17:22:26 +01:00
Erik Johnston
968b01a91a Actually use async method 2015-05-13 17:02:46 +01:00
Erik Johnston
4071f29653 Fetch events from events_id in their own transactions 2015-05-13 16:59:41 +01:00
Erik Johnston
a988361aea Typo 2015-05-13 15:44:15 +01:00
Erik Johnston
8888982db3 Don't insert None 2015-05-13 15:43:32 +01:00
Erik Johnston
cf706cc6ef Don't return None 2015-05-13 15:31:25 +01:00
Erik Johnston
5971d240d4 Limit batch size 2015-05-13 15:26:49 +01:00
Erik Johnston
ca4f458787 Fetch events in bulk 2015-05-13 15:13:42 +01:00
Erik Johnston
6edff11a88 Don't fetch redaction and rejection stuff for each event, so we can use index only scan 2015-05-13 14:39:05 +01:00
Mark Haines
63878c0379 Don't bother checking for updates if the stream token hasn't advanced for a user 2015-05-13 13:42:21 +01:00
Erik Johnston
02590c3e1d Temp turn off checking for rejections and redactions 2015-05-13 11:31:28 +01:00
Erik Johnston
619a21812b defer.gatherResults loop 2015-05-13 11:29:03 +01:00
Erik Johnston
fec4485e28 Batch fetching of events for state groups 2015-05-13 11:22:42 +01:00
Erik Johnston
409bcc76bd Load events for state group seperately 2015-05-13 11:13:31 +01:00
Erik Johnston
e122685978 You need to call contextmanager 2015-05-12 16:12:37 +01:00
Mark Haines
a6fb2aa2a5 Merge pull request #144 from matrix-org/erikj/logging_context 
Preserving logging contexts
 2015-05-12 15:23:50 +01:00
Mark Haines
1fce36b111 Merge pull request #149 from matrix-org/erikj/backfill 
Backfill support
 2015-05-12 15:20:32 +01:00
Erik Johnston
8b28209c60 Err, delete the right stuff 2015-05-12 15:02:53 +01:00
Erik Johnston
30c72d377e Newlines 2015-05-12 14:47:40 +01:00
Erik Johnston
e4eddf9b36 We do actually want to delete rows out of event_backward_extremities 2015-05-12 14:47:23 +01:00
Erik Johnston
c1779a79bc Fix up _handle_prev_events to not try to insert duplicate rows 2015-05-12 14:41:50 +01:00
Erik Johnston
74850d7f75 Do state groups persistence /after/ checking if we have already persisted the event 2015-05-12 14:14:58 +01:00
Erik Johnston
da6a7bbdde Merge branch 'develop' of github.com:matrix-org/synapse into erikj/logging_context 2015-05-12 13:10:42 +01:00
Mark Haines
5002056b16 SYN-377: Make sure that the StreamIdGenerator.get_next.__exit__ is called from the main thread after the transaction completes, not from database thread before the transaction completes. 2015-05-12 11:20:40 +01:00
Erik Johnston
4df11b5039 Make get_current_token accept a direction parameter, which tells whether the source whether we want a token for going 'forwards' or 'backwards' 2015-05-12 10:28:10 +01:00
Erik Johnston
84e6b4001f Initial hack at wiring together pagination and backfill 2015-05-11 18:01:31 +01:00
Erik Johnston
17653a5dfe Move storage.stream._StreamToken to types.RoomStreamToken 2015-05-11 18:01:01 +01:00
Erik Johnston
b036596b75 Prefer to use _simple_*. 2015-05-11 11:24:01 +01:00
Erik Johnston
cd525c0f5a push_rules table expects an 'id' field 2015-05-11 11:24:01 +01:00
Erik Johnston
22d2f498fa Fix push rule bug: can't insert bool into small int column 2015-05-10 10:50:51 +01:00
Erik Johnston
476899295f Change the way we do logging contexts so that they survive divergences 2015-05-08 16:32:18 +01:00
Mark Haines
1c7912751e Drop the old table not the new table 2015-05-08 16:04:32 +01:00
Mark Haines
9d36eb4eab Rename unique constraint 2015-05-08 16:01:55 +01:00
Mark Haines
b0f71db3ff Remove unsigned 2015-05-08 15:59:51 +01:00
Mark Haines
84e1cacea4 Bump schema version 2015-05-08 15:58:14 +01:00
Mark Haines
6538d445e8 Make the timestamps in server_keys_json bigints 2015-05-08 15:55:17 +01:00
Erik Johnston
0c4ac271ca Merge branch 'erikj/executemany' of github.com:matrix-org/synapse into erikj/SYN-371 2015-05-05 18:21:19 +01:00
Erik Johnston
0cf7e480b4 And use buffer(...) there as well 2015-05-05 18:20:01 +01:00
Erik Johnston
ed2584050f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/executemany 2015-05-05 18:15:20 +01:00
Erik Johnston
977338a7af Use buffer(...) when inserting into bytea column 2015-05-05 18:12:53 +01:00
Mark Haines
31049c4d72 Merge pull request #139 from matrix-org/bugs/SYN-369 
Fix race with cache invalidation. SYN-369
 2015-05-05 17:46:13 +01:00
Mark Haines
deb0237166 Add some doc-string 2015-05-05 17:45:11 +01:00
Mark Haines
d18f37e026 Collect the invalidate callbacks on the transaction object rather than passing around a separate list 2015-05-05 17:32:21 +01:00
Erik Johnston
9951542393 Add a comment about the zip(*[zip(sorted(...),...)]) 2015-05-05 17:06:55 +01:00
Mark Haines
041b6cba61 SYN-369: Add comments to the sequence number logic in the cache 2015-05-05 16:32:44 +01:00
Mark Haines
63075118a5 Add debug flag in synapse/storage/_base.py for debugging the cache logic by comparing what is in the cache with what was in the database on every access 2015-05-05 16:24:04 +01:00
Erik Johnston
531d7955fd Don't insert without deduplication. In this case we never actually use this table, so simply remove the insert entirely 2015-05-05 16:12:28 +01:00
Mark Haines
bfa4a7f8b0 Invalidate the room_member cache if the current state events updates 2015-05-05 15:43:49 +01:00
Mark Haines
d0fece8d3c Missing return for when the event was already persisted 2015-05-05 15:39:09 +01:00
Erik Johnston
bdcd7693c8 Fix indentation 2015-05-05 15:14:48 +01:00
Erik Johnston
43c2e8deae Add support for using executemany 2015-05-05 15:13:25 +01:00
Erik Johnston
1692dc019d Don't call 'encode_parameter' no-op 2015-05-05 15:00:30 +01:00
Mark Haines
a9aea68fd5 Invalidate the caches from the correct thread 2015-05-05 14:57:08 +01:00
Mark Haines
261d809a47 Sequence the modifications to the cache so that selects don't race with inserts 2015-05-05 14:13:50 +01:00
Erik Johnston
d9cc5de9e5 Correctly name transaction 2015-05-05 10:24:10 +01:00
Erik Johnston
b8940cd902 Remove some unused indexes 2015-05-01 16:14:25 +01:00
David Baker
eb9bd2d949 user_id now in user_threepids 2015-05-01 15:04:37 +01:00
David Baker
9eeb03c0dd Don't use self.execute: it's designed for fetching stuff 2015-05-01 14:21:25 +01:00
David Baker
7693f24792 No id field on user 2015-05-01 13:55:42 +01:00
David Baker
92b20713d7 More missed get_user_by_id API changes 2015-05-01 13:45:54 +01:00
Erik Johnston
4011cf1c42 Cache latest_event_ids_in_room 2015-05-01 13:06:26 +01:00
Erik Johnston
657298cebd Don't lock user_ips table for upsert. 2015-05-01 10:46:48 +01:00
Erik Johnston
fabb7acd45 Fix bug where we reconnected to the database on every query. 2015-05-01 10:24:24 +01:00
Erik Johnston
23c639ff32 Split a storage function in two so that we don't have to do extra work. 2015-05-01 10:17:19 +01:00
Erik Johnston
8be5284e91 Remove pointless join 2015-04-30 18:49:26 +01:00
Erik Johnston
503e4d3d52 Fix broken SQL 2015-04-30 18:44:47 +01:00
Erik Johnston
00718ae7a9 Need more yield 2015-04-30 18:43:39 +01:00
Erik Johnston
0465560c1a Add missing param 2015-04-30 18:42:44 +01:00
Erik Johnston
61d05daab1 More join conditions 2015-04-30 18:41:05 +01:00
Erik Johnston
6ead27ddda Add more conditions on JOINs to make postgres go a little faster. 2015-04-30 18:32:03 +01:00
Erik Johnston
f383d5a801 Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction 2015-04-30 17:12:52 +01:00
Erik Johnston
69d4063651 Add get_rooms_for_user cache 2015-04-30 16:47:51 +01:00
Erik Johnston
3c4c229788 Don't use sub queries, it makes postgres sad 2015-04-30 16:16:53 +01:00
Mark Haines
29400b45b9 SYN-367: Use upsert rather than insert_or_replace 2015-04-30 15:21:31 +01:00
Erik Johnston
2366d28780 Don't needlessly join on state_events 2015-04-30 14:02:06 +01:00
Erik Johnston
1d7702833d Make simple query rather than long one and then throw away half the results 2015-04-30 10:16:12 +01:00
Erik Johnston
b1ca784aca Correctly decode, for sqlite and postgres, rows from pushers table 2015-04-29 19:41:14 +01:00
Erik Johnston
4a9dc5b2f5 pushkey' are also bytes. 2015-04-29 19:27:02 +01:00
Erik Johnston
0ade2712d1 Typo 2015-04-29 19:17:25 +01:00
Erik Johnston
50f96f256f Also remove yield from within lock in the other generator 2015-04-29 19:17:00 +01:00
Erik Johnston
d2d61a8288 Fix deadlock in id_generators. No idea why this was an actual deadlock. 2015-04-29 19:15:23 +01:00
Erik Johnston
3864b3a8e6 Actually return rows 2015-04-29 18:07:36 +01:00
Erik Johnston
0618978238 Typo, args wrong way round 2015-04-29 18:04:35 +01:00
Erik Johnston
09177f4f2e Decode buffers in same thread 2015-04-29 18:03:42 +01:00
Erik Johnston
472be88674 We store pusher data as bytes 2015-04-29 17:43:46 +01:00
David Baker
12d381bd5d Decode the data json in the storage layer (was moved but this part was missed) 2015-04-29 17:13:51 +01:00
David Baker
f8c30faf25 Oops, update the contraint too 2015-04-29 16:58:42 +01:00
David Baker
61cd5d9045 Be more postgressive 2015-04-29 16:57:14 +01:00
David Baker
fb95035a65 Be postgressive 2015-04-29 16:53:41 +01:00
David Baker
4669def000 Oops, forgot the schema delta file 2015-04-29 16:50:16 +01:00
Erik Johnston
0337eaf321 txn.execute doesn't return cursors 2015-04-29 16:43:39 +01:00
Erik Johnston
884fb88e28 txn.execute doesn't return cursors 2015-04-29 16:35:20 +01:00
Erik Johnston
d76c058eea Fix invalid SQL to work in postgres land 2015-04-29 16:30:25 +01:00
Erik Johnston
c8ed9bd278 pushers table requires an access_token 2015-04-29 15:33:27 +01:00
Erik Johnston
970a9b9d2b We can't use REPLACE when upgrading databases now we have postgres 2015-04-29 13:55:44 +01:00
Erik Johnston
64991b0c8b Merge pull request #129 from matrix-org/key_distribution 
Key distribution v2
 2015-04-29 13:34:38 +01:00
Mark Haines
e26a3d8d9e bump database schema version 2015-04-29 13:32:32 +01:00
Mark Haines
a9549fdce3 Use bytea rather than BLOB 2015-04-29 13:16:09 +01:00
Mark Haines
4ad8b45155 Merge branch 'develop' into key_distribution 
Conflicts:
	synapse/config/homeserver.py
 2015-04-29 13:15:14 +01:00
Erik Johnston
cd0864121b Make postgres database error slightly more helpful 2015-04-29 12:12:25 +01:00
Erik Johnston
4932a7e2d9 Use __all__ instead of assert to stop pyflakes from warning about unused import in __init__ 2015-04-29 12:12:25 +01:00
Erik Johnston
a08bf11138 Appease PEP8 2015-04-29 11:44:48 +01:00
Erik Johnston
204132a998 Check that postgres database has correct charset set 2015-04-29 11:42:28 +01:00
Erik Johnston
327ca883ec Merge branch 'develop' of github.com:matrix-org/synapse into postgres 2015-04-28 13:39:42 +01:00
Erik Johnston
18f8247701 Use TEXT instead of VARCHAR(n), since PostgreSQL treats them the same except for a limit 2015-04-28 12:41:33 +01:00
Mark Haines
9182f87664 Merge pull request #126 from matrix-org/csauth 
Client / Server Auth Refactor
 2015-04-28 11:00:27 +01:00
Mark Haines
55e1bc8920 And don't bump the schema version unnecessarily 2015-04-28 10:54:15 +01:00
Mark Haines
b96c133034 Add server_keys.sql to the current delta rather than creating a new delta 2015-04-28 10:50:00 +01:00
Erik Johnston
8558e1ec73 Make get_max_token into inlineCallbacks so that the lock works. 2015-04-27 15:19:44 +01:00
Erik Johnston
6f8e2d517e Merge branch 'develop' of github.com:matrix-org/synapse into postgres 2015-04-27 14:41:40 +01:00
Erik Johnston
2732be83d9 Shuffle operations so that locking upsert happens last in the txn. This ensures the lock is held for the least amount of time possible. 2015-04-27 13:22:30 +01:00
Erik Johnston
e4c4664d73 Handle the fact that postgres databases can be restarted from under us 2015-04-27 12:40:49 +01:00
David Baker
8a5be236e0 pep8 2015-04-27 11:49:18 +01:00
David Baker
b02e1006b9 Run database check before daemonizing, at the cost of database hygiene. 2015-04-27 11:46:00 +01:00
David Baker
f8152f2708 rename db method to be more informative 2015-04-27 10:16:26 +01:00
David Baker
2f475bd5d5 pep8 2015-04-24 18:15:07 +01:00
David Baker
a7b51f4539 Check users in our table aren't on a different domain to the one we're configured with to try & fix SYN-266 2015-04-24 18:11:21 +01:00
Mark Haines
eede182df7 Merge branch 'develop' into key_distribution 2015-04-24 10:35:49 +01:00
Mark Haines
4e2f8b8722 Copyright notices 2015-04-24 10:35:29 +01:00
Mark Haines
f30d47c876 Implement remote key lookup api 2015-04-22 14:21:08 +01:00
David Baker
91c8f828e1 pep8 2015-04-17 19:56:04 +01:00
David Baker
8db6832db8 Password reset, finally. 2015-04-17 19:53:47 +01:00
David Baker
117f35ac4a Add endpoint to get threepids from server 2015-04-17 17:20:18 +01:00
David Baker
4eea5cf6c2 pep8 2015-04-17 16:46:45 +01:00
David Baker
f96ab9d18d make add3pid servlet work 2015-04-17 16:44:49 +01:00
Erik Johnston
e3417bbbe0 Revert needless change to storage.profile 2015-04-17 16:13:14 +01:00
David Baker
cb03fafdf1 Merge branch 'develop' into csauth 2015-04-17 13:51:10 +01:00
Erik Johnston
0ec346d942 Add unique index to room_aliases, remove duplicates on upgrade. Convert some columns back to TEXT from bytea 2015-04-17 11:10:20 +01:00
Erik Johnston
e1c0970c11 PEP8 2015-04-16 11:18:45 +01:00
Erik Johnston
b8092fbc82 Go back to storing JSON in TEXT 2015-04-16 11:17:52 +01:00
Erik Johnston
bc9e69e160 Move encoding and decoding of JSON into storage layer 2015-04-16 11:01:09 +01:00
Erik Johnston
ffad75bd62 Remove mysql/maria support 2015-04-15 17:00:50 +01:00
Mark Haines
a429515bdd Add methods for storing and retrieving the raw key json 2015-04-15 16:58:35 +01:00
Erik Johnston
cf04cedf21 Change full_schemas/11 to work with postgres 2015-04-15 16:53:47 +01:00
Erik Johnston
5b31afcbd1 Remove debug logging 2015-04-15 16:27:04 +01:00
Erik Johnston
6e91f14d09 Add missing yield 2015-04-15 16:25:07 +01:00
Erik Johnston
ed26e4012b pushers table requires a unique id. 2015-04-15 16:24:14 +01:00
Erik Johnston
a5c72780e6 Don't pass in removed flag 2015-04-15 15:13:22 +01:00
Erik Johnston
4af32a2817 Postgres does not allow you to continue using a cursor after a DB exception has been raised, so move _simple_insert or_ignore flag out of transaction 2015-04-15 14:51:21 +01:00
Erik Johnston
25d2914fba Turn off persistance of sent_transactions.response_json 2015-04-15 14:09:35 +01:00
Erik Johnston
cce5d057d3 Add index on events (topological_ordering, stream_ordering) to help with ORDER BY clauses 2015-04-15 11:43:25 +01:00
Erik Johnston
a971fa9d58 Use try..finally in contextlib.contextmanager 2015-04-15 10:25:43 +01:00
Erik Johnston
ded4128965 Use True for True rather than 1 2015-04-15 10:24:24 +01:00
Erik Johnston
f9e12f79ca Add missing yield in storage func 2015-04-15 10:24:07 +01:00
Erik Johnston
c756dfeb14 Correctly identify deadlocks 2015-04-15 10:23:42 +01:00
Mark Haines
32e14d8181 Return a sha256 fingerprint rather than the entire tls certificate 2015-04-14 19:10:09 +01:00
Erik Johnston
127fad17dd Add postgres database engine 2015-04-14 14:50:29 +01:00
Erik Johnston
5a95cd4442 Rename user_ips.user -> user_id 2015-04-14 13:54:09 +01:00
Erik Johnston
58d8339966 Add support for postgres instead of mysql. Change sql accourdingly. blob + varbinary -> bytea. No support for UNSIGNED or CREATE INDEX IF NOT EXISTS. 2015-04-14 13:53:20 +01:00
Erik Johnston
3c741682e5 Correctly increment the _next_id initially 2015-04-14 09:54:44 +01:00
Erik Johnston
86fc9b617c For backwards compat, make state_groups.id have a type of int, not varchar 2015-04-13 17:03:49 +01:00
Erik Johnston
93937b2b31 Remove duplicate rows 2015-04-10 18:47:03 +01:00
Erik Johnston
c5365dee56 Use case sensitive collations 2015-04-10 18:46:33 +01:00
Erik Johnston
4103b1c470 DROP indexes before recreating them 2015-04-10 18:46:09 +01:00
Erik Johnston
4d5b098626 Use LONGBLOB and TEXT for arbitary length rows 2015-04-10 18:45:09 +01:00
Erik Johnston
7ed2ec3061 Handle the fact that in sqlite binary data might be stored as unicode or bytes 2015-04-10 13:41:54 +01:00
Erik Johnston
ce797ad373 Bump schema version 2015-04-10 11:22:30 +01:00
Erik Johnston
7e863c51e6 Use unsigned bigint 2015-04-10 11:22:04 +01:00
Erik Johnston
0f12772e32 SQLite wants INTEGER and not BIGINT for primary keys 2015-04-10 11:16:09 +01:00
Erik Johnston
d5d4281647 Update full_schemas/16 to match delta files. Add delta/16 scripts 2015-04-10 10:59:46 +01:00
Erik Johnston
cda4a6f93f Revert non-trivial changes to upgrade scripts 2015-04-10 10:19:50 +01:00
Erik Johnston
e2722f58ee Fix schema again 2015-04-10 10:16:29 +01:00
Erik Johnston
a1665c5094 Revert non-trivial schema changes and move them to a new schema version. 2015-04-10 10:05:44 +01:00
Erik Johnston
2ded344620 Remove unused import 2015-04-09 13:46:06 +01:00
Erik Johnston
9707acfc40 Remove spurious spaces 2015-04-09 13:45:20 +01:00
Erik Johnston
8bf285e082 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-09 13:13:26 +01:00
Erik Johnston
8ad0f4912e
Stream ordering and out of order insertions. 
Handle the fact that events can be persisted out of order, and so to get
the "current max" stream token becomes non trivial - as we need to make
sure that *all* stream tokens less than the current max have also
successfully been persisted.
 2015-04-09 11:41:36 +01:00
Erik Johnston
279a547a8b Use generic db exceptions rather than sqlite3 specific ones 2015-04-08 16:53:48 +01:00
Mark Haines
83f5125d52 Merge pull request #114 from matrix-org/improve_get_event_cache 
Improve get event cache
 2015-04-08 16:50:27 +01:00
Erik Johnston
3887350e47 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-08 14:11:07 +01:00
Erik Johnston
c1b34af441 Move database timer logging to seperate logger 2015-04-08 13:31:19 +01:00
Erik Johnston
9a05795619 Retry transaction, not SQL query 2015-04-08 13:11:28 +01:00
Erik Johnston
24d8134ac1 Fix maria engine to correctly recognize deadlocks 2015-04-08 13:10:54 +01:00
Erik Johnston
4fe95094d1 Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-04-07 18:05:39 +01:00
Erik Johnston
49d6aa1394 Retry on deadlock 2015-04-07 15:28:37 +01:00
Erik Johnston
0bfa78b39b PEP8 2015-04-07 12:16:05 +01:00
Erik Johnston
6bc9edd8b2 Fix prepare_sqlite3_database's convert_param_style 2015-04-07 12:13:58 +01:00
Erik Johnston
05a35d62b6 Bump database version 2015-04-07 12:10:15 +01:00
Erik Johnston
8574bf62dc Add index to presence table 2015-04-07 12:09:36 +01:00
Erik Johnston
0af5f5efaf Don't use multiple UNIQUE constraints; it will cause deadlocks 2015-04-07 12:08:35 +01:00
Erik Johnston
c8d3f6486d Implement or_ignore flag on inserts 2015-04-07 12:06:22 +01:00
Erik Johnston
304111afd0 Don't use AUTOINCREMENT, use an in memory version 2015-04-07 12:05:36 +01:00
Erik Johnston
779f7b0f44 Fix unicode support 2015-04-02 10:06:22 +01:00
Erik Johnston
9236136f3a Make work in both Maria and SQLite. Fix tests 2015-04-01 14:12:33 +01:00
Kegan Dougal
5e88a09a42 Add same user_id char checks as registration. 2015-03-31 14:00:25 +01:00
Kegan Dougal
cf1fa59f4b Use a sender localpart instead of a user ID. 
Form the user ID at runtime instead, This gives less room for error in AS
config files since they cannot specify the domain of another HS.
 2015-03-31 13:48:03 +01:00
Kegan Dougal
c217504949 Edit SQL schema to use string IDs not ints. Use token as ID. Update tests. 2015-03-31 12:07:56 +01:00
Kegan Dougal
b59aa74556 Fix tests and missing returns on deferreds. 2015-03-31 11:35:45 +01:00
Kegan Dougal
d33ae65efc Remove more reg/unreg methods. Read config not database for cache. 2015-03-31 11:00:00 +01:00
Paul "LeoNerd" Evans
3e420aebd8 Revert "Add another @cached wrapper, this time on get_presence_state()" 
This reverts commit ff1fa0fbf8.
 2015-03-27 16:16:58 +00:00
Paul "LeoNerd" Evans
ff1fa0fbf8 Add another @cached wrapper, this time on get_presence_state() 2015-03-27 15:57:16 +00:00
David Baker
a32e876ef4 Delete pushers when changing password 2015-03-26 13:40:16 +00:00
Paul "LeoNerd" Evans
a198894bf7 Appease pep8 2015-03-26 11:53:58 +00:00
Kegan Dougal
32206dde3f Fixes from PR comments 2015-03-26 10:11:52 +00:00
Kegan Dougal
4edcbcee3b Merge branch 'develop' into application-services-txn-reliability 
Conflicts:
	synapse/storage/__init__.py
 2015-03-26 10:07:59 +00:00
Paul "LeoNerd" Evans
953e40f9dc Implement the main getEvent cache using Cache() instead of a custom application of LruCache; also unify its two-level structure into just one 2015-03-25 19:12:16 +00:00
David Baker
c1a256cc4c Allow multiple pushers for a single app ID & pushkey, honouring the 'append' flag in the API. 2015-03-25 19:06:22 +00:00
Paul "LeoNerd" Evans
f173d40a32 Use FrozenEvent's reject_reason to decide whether to return it; don't include allow_rejected in the main getEvents cache key 2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
1b988b051b Store the rejected reason in (Frozen)Event structs 2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
033a517feb Indirect invalidations of _get_event_cache via a helper method to keep all uses of the cache lexically within one .py file 2015-03-25 19:06:05 +00:00
Paul "LeoNerd" Evans
9ba6487b3f Allow a choice of LRU behaviour for Cache() by using LruCache() or OrderedDict() 2015-03-25 19:05:34 +00:00
Erik Johnston
0e8f5095c7 Fix unicode database support 2015-03-25 17:15:20 +00:00
David Baker
c7023f2155 1) Pushers are now associated with an access token 
2) Change places where we mean unauthenticated to 401, not 403, in C/S v2: hack so it stays as 403 in v1 because web client relies on it.
 2015-03-24 17:24:15 +00:00
Erik Johnston
0ba393924a Escape non printing ascii character 2015-03-24 16:31:52 +00:00
Erik Johnston
f488293d96 Don't reinsert into event_edges 2015-03-24 16:20:26 +00:00
Erik Johnston
1aa44939fc Fix bugs in transactions storage 2015-03-24 16:20:05 +00:00
Erik Johnston
5a447098dd Don't use room hosts table 2015-03-24 16:19:24 +00:00
Erik Johnston
9e98f1022a Don't order by rowid 2015-03-24 16:19:01 +00:00
Erik Johnston
9115421ace Use _simple_upsert 2015-03-24 16:17:39 +00:00
David Baker
d19e79ecc9 Make deleting other access tokens when you change your password actually work 2015-03-24 15:33:48 +00:00
Erik Johnston
6e7131f02f Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable. 2015-03-23 15:38:56 +00:00
Erik Johnston
9a7f496298 Sanitize RoomMemberStore 2015-03-23 15:29:04 +00:00
David Baker
d98660a60d Implement password changing (finally) along with a start on making client/server auth more general. 2015-03-23 14:20:28 +00:00
Erik Johnston
d5272b1d2c Use 'update or insert' rather than on 'conflict replace' 2015-03-23 14:02:34 +00:00
Erik Johnston
278149f533 Sanitize TransactionStore 2015-03-23 13:43:21 +00:00
Paul "LeoNerd" Evans
72d8406409 Put a cache on get_aliases_for_room 2015-03-20 19:21:13 +00:00
Paul "LeoNerd" Evans
a63b4f7101 Remember the 'last seen' time for a given user/IP/device combination and only bother INSERTing another if it's stale 2015-03-20 18:25:49 +00:00
Paul "LeoNerd" Evans
0f86312c4c Pull out the cache logic from the @cached wrapper into its own class we can reuse 2015-03-20 18:25:42 +00:00
Paul "LeoNerd" Evans
b1022ed8b5 func(*EXPR) is valid Python syntax, really... 2015-03-20 17:47:45 +00:00
Erik Johnston
f6583796fe Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-03-20 16:31:48 +00:00
Erik Johnston
80cd08c190 PEP8 2015-03-20 16:03:25 +00:00
Erik Johnston
9517f4da4d Merge branch 'develop' of github.com:matrix-org/synapse into store_rearrangement 2015-03-20 16:02:47 +00:00
Erik Johnston
dc0c989ef4 Give sensible names for '_simple_...' transactions 2015-03-20 15:59:18 +00:00
Paul "LeoNerd" Evans
ceb61daa70 Add the tiniest of tiny one-element caches to get_room_events_max_id() as it's read every time someone hits eventstream 2015-03-20 15:44:06 +00:00
Erik Johnston
fce0114005 Start removing Tables 2015-03-20 15:05:44 +00:00
Erik Johnston
7e282a53a5 Tidy up _simple_... methods 2015-03-20 15:05:10 +00:00
Paul "LeoNerd" Evans
91cb46191d Allow @cached-wrapped functions to have more or fewer than 1 argument; assert on the total count of them though 2015-03-20 14:59:45 +00:00
Erik Johnston
87db64b839 Rearrange storage modules 2015-03-20 14:11:38 +00:00
Erik Johnston
cb8162d3d1 Rearrange storage modules 2015-03-20 13:52:56 +00:00
Erik Johnston
d288d273e1 Generate transaction id in code 2015-03-20 10:57:44 +00:00
Erik Johnston
d4f50f3ae5 decode_result takes an iterable 2015-03-20 10:57:26 +00:00
Erik Johnston
0d0610870d Fix up schemas some more 2015-03-20 10:55:31 +00:00
Erik Johnston
d7a0496f3e Convert storage layer to be mysql compatible 2015-03-19 15:59:48 +00:00
Erik Johnston
58ed393235 Remove redundant key 2015-03-19 15:12:05 +00:00
Erik Johnston
fae059cc18 Fix up schemas to work with mariadb 2015-03-19 13:42:39 +00:00
Erik Johnston
bb24609158 Clean out event_forward_extremities table when the server rejoins the room 2015-03-18 11:19:47 +00:00
Erik Johnston
89036579ed Update schema to work with mariadb 2015-03-18 11:18:49 +00:00
Paul "LeoNerd" Evans
93978c5e2b @cached() annotate get_user_by_token() - achieves a minor DB performance improvement 2015-03-17 17:24:51 +00:00
Paul "LeoNerd" Evans
1489521ee5 Be polite and ensure we use @functools.wraps() when creating a function decorator 2015-03-17 17:19:22 +00:00
Paul "LeoNerd" Evans
be170b1426 Add a metric for the scheduling latency of SQL queries 2015-03-16 17:21:59 +00:00
Erik Johnston
cd2539ab2a Merge pull request #110 from matrix-org/fix_ban 
Fix ban
 2015-03-16 15:36:52 +00:00
Kegan Dougal
f0d6f724a2 Set the service ID as soon as it is known. 2015-03-16 15:24:32 +00:00
Erik Johnston
f1d2b94e0b Copy dict of context.current_state before changing it. 2015-03-16 15:13:05 +00:00
Kegan Dougal
835e01fc70 Minor PR comment tweaks. 2015-03-16 10:16:59 +00:00
Kegan Dougal
f9232c7917 Merge branch 'develop' into application-services-txn-reliability 
Conflicts:
	synapse/storage/appservice.py
 2015-03-16 10:09:15 +00:00
Paul "LeoNerd" Evans
e75fa8bbbf Bugfix to sql_txn_timer increment - add only the per-TXN duration, not the total time ever spent since boot 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
63cb7ece62 Rename the timer metrics exported by synapse.storage to append _time, so the meaning of ':total' is clearer 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f1fbe3e09f Rename TimerMetric to DistributionMetric; as it could count more than just time 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
642f725fd7 Pretend the 'getEvent' cache is just another cache in the set of all the others for metric 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
b0cf867319 Use _ instead of . as a metric namespacing separator, for Prometheus 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f9478e475b Rename Metrics' "keys" to "labels" 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
fa319a5786 Add TimerMetrics to shadow the PerformanceCounters in synapse.storage; with the view to eventually replacing them entirely 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
25187ab674 Collect per-SQL-verb timer stats on query execution time 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
e1a7e3564f Delete a couple of TODO markers of monitoring stats now done 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
8664599af7 Rename CacheCounterMetric to just CacheMetric; add a CallbackMetric component to give the size of the cache 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
b0cdf097f4 Sprinkle some CacheCounterMetrics around the synapse.storage layer 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
a7a28f85ae Appease pep8 2015-03-11 17:32:43 +00:00
Paul "LeoNerd" Evans
59a5f012cc Also give _execute() a description 2015-03-11 17:19:17 +00:00
Paul "LeoNerd" Evans
099e4b88d8 Add a description to storage layer's _execute_and_decode() 2015-03-11 17:08:57 +00:00
David Baker
04f8478aaa Add the master push rule for the break-my-push button. Allow server default rules to be disabled by default. 2015-03-10 17:26:25 +00:00
David Baker
8916acbc13 These aren't defined for redacted events so don't crash 2015-03-10 11:21:37 +00:00
Kegan Dougal
db1fbc6c6f Fix remaining scheduler bugs. Add more informative logging. 2015-03-10 10:04:20 +00:00
Kegan Dougal
7e0bba555c Remove unused import 2015-03-09 17:48:37 +00:00
Kegan Dougal
04c9751f24 Bug fixes whilst putting it all together 2015-03-09 17:45:41 +00:00
Kegan Dougal
b98cd03193 Use event IDs instead of dumping event content in the txns table. 2015-03-09 17:25:20 +00:00
Kegan Dougal
21fd84dcb8 Use seconds; start gluing in the AS scheduler into the AS handler. 2015-03-09 17:01:19 +00:00
Kegan Dougal
0a60bbf4fa Finish appservice txn storage impl and tests. 2015-03-09 15:53:03 +00:00
Kegan Dougal
1ead1caa18 Implement create_appservice_txn with tests. 2015-03-09 13:54:20 +00:00
Kegan Dougal
1c2dcf762a Partially implement txn store methods with tests. 2015-03-09 13:10:31 +00:00
Kegan Dougal
406d32f8b5 Start implementing ApplicationServiceTransactionStore 2015-03-06 17:35:14 +00:00
Kegan Dougal
34ce2ca62f Merge branch 'develop' into application-services-txn-reliability 2015-03-06 17:28:49 +00:00
Kegan Dougal
4a6afa6abf Assign the AS ID from the database; replace old placeholder txn id. 2015-03-06 17:27:55 +00:00
Kegan Dougal
01c099d9ef Add appservice txns sql schema 2015-03-06 17:16:47 +00:00
Kegan Dougal
2602ddc379 Apply clarity and docstrings 2015-03-06 16:16:14 +00:00
Kegan Dougal
0354659f9d Finish synapse.appservice.scheduler implementation. 
With tests to assert behaviour. Not hooked up yet. Stub datastore methods
not implemented yet.
 2015-03-06 16:09:05 +00:00
Kegan Dougal
f260cb72cd Flesh out more stub functions. 2015-03-06 15:12:24 +00:00
Kegan Dougal
141ec04d19 Add stub ApplicationServiceTransactionStore. Bootstrap Recoverers. Fill in stub Transaction functions. 2015-03-06 14:53:35 +00:00
David Baker
cf66ddc1b4 Schema change as delta in v14 2015-03-06 14:11:49 +00:00
David Baker
ed72fc3a50 Merge branch 'develop' into pushrules2 
Conflicts:
	synapse/storage/schema/pusher.sql
 2015-03-04 15:24:21 +00:00
Erik Johnston
d56c01fff4 Note that we don't specify execution order 2015-03-04 15:10:05 +00:00
Erik Johnston
17d319a20d s/schema_deltas/applied_schema_deltas/ 2015-03-04 15:06:22 +00:00
David Baker
92b3dc3219 Merge branch 'develop' into pushrules2 2015-03-04 14:56:41 +00:00
Erik Johnston
5681264faa s/%r/%s/ 2015-03-04 14:21:53 +00:00
Erik Johnston
f701197227 Add example directory structures in doc 2015-03-04 14:20:14 +00:00
David Baker
2a45f3d448 Use if not results rather than len, as per feedback. 2015-03-04 14:17:59 +00:00
Erik Johnston
16dd87d848 Don't assume db conn is a Context Manager. 
Twisted adbapi wrapped connections aren't context managers.
 2015-03-04 14:03:41 +00:00
Erik Johnston
5eefd1f618 Add unique constraint on schema_version.lock schema. Use conflict clause in sql. 2015-03-04 13:52:18 +00:00
Erik Johnston
b4c38738f4 Change to use logger in db upgrade script 2015-03-04 13:43:35 +00:00
Erik Johnston
640e53935d Use context manager with db conn to correctly commit and rollback 2015-03-04 13:43:17 +00:00
Erik Johnston
8c8354e85a Actually add full_schemas dir 2015-03-04 13:34:38 +00:00
Erik Johnston
c3530c3fb3 More docs. Rename 'schema/current' to 'schema/full_schemas' 2015-03-04 13:34:11 +00:00
Erik Johnston
811355ccd0 Add some docs and remove unused variables 2015-03-04 13:11:01 +00:00
Erik Johnston
82b34e813d SYN-67: Finish up implementing new database schema management 2015-03-04 12:04:19 +00:00
Erik Johnston
8d33adfbbb SYN-67: Begin changing the way we handle schema versioning 2015-03-02 18:23:55 +00:00
David Baker
6fab7bd2c1 s/user_name/user/ as per mjark's comment 2015-03-02 18:17:19 +00:00
Erik Johnston
9f03553f48 Add missing comma 2015-03-02 16:38:40 +00:00
Erik Johnston
b41dc68773 We purposefully don't have a version 14 delta script. 2015-03-02 16:36:19 +00:00
David Baker
20436cdf75 Blank lines 2015-03-02 15:58:12 +00:00
Kegsay
8ad024ea80 Merge pull request #93 from matrix-org/application-services-exclusive 
Application services exclusive flag support
 2015-03-02 14:56:32 +00:00
Erik Johnston
0a036944bd Merge branch 'develop' of github.com:matrix-org/synapse into batched_get_pdu 2015-03-02 13:53:30 +00:00
Kegan Dougal
c3c01641d2 Run deltas and bump user_version in upgrade script 2015-03-02 13:38:57 +00:00
Kegan Dougal
cb97ea3ec2 PEP8 2015-03-02 11:23:46 +00:00
Kegan Dougal
377ae369c1 Wrap all of get_app_service_rooms in a txn. 2015-03-02 11:20:51 +00:00
Kegan Dougal
b216b36892 JOIN state_events rather than parsing unrecognized_keys to pull out member state_keys 2015-03-02 10:41:35 +00:00
Kegan Dougal
3d73383d18 Modify _simple_select_list to allow an empty WHERE clause. Use it for get_all_rooms and get_all_users. 2015-03-02 10:16:24 +00:00
Kegan Dougal
ebc4830666 PR tweaks: set earlier on and use 'as json' for compat 2015-03-02 09:53:00 +00:00
Kegan Dougal
16b90764ad Convert expected format for AS regex to include exclusivity. 
Previously you just specified the regex as a string, now it expects a JSON
object with a 'regex' key and an 'exclusive' boolean, as per spec.
 2015-02-27 10:44:32 +00:00
Kegan Dougal
806a6c886a PEP8 2015-02-27 09:48:57 +00:00
David Baker
1959088156 Add API for getting/setting enabled-ness of push rules. 2015-02-26 18:07:44 +00:00
Kegan Dougal
f0995436e7 Check for membership invite events correctly. 2015-02-26 17:21:17 +00:00
Kegan Dougal
dcec7175dc Finish impl to get new events for AS. ASes should now be able to poll /events 2015-02-26 16:23:01 +00:00
Kegan Dougal
92478e96d6 Finish impl to extract all room IDs an AS may be interested in when polling the event stream. 2015-02-26 14:35:28 +00:00
David Baker
944003021b whitespace 2015-02-26 13:43:05 +00:00
David Baker
94fa334b01 Add enable/disable overlay for push rules (REST API not yet hooked up) 2015-02-25 19:17:07 +00:00
Kegan Dougal
29267cf9d7 PEP8 and pyflakes 2015-02-25 17:42:28 +00:00
Kegan Dougal
978ce87c86 Comment unused variables. 2015-02-25 17:37:48 +00:00
Kegan Dougal
2c79c4dc7f Fix alias query. 2015-02-25 17:37:14 +00:00
Kegan Dougal
2b8ca84296 Add support for extracting matching room_ids and room_aliases for a given AS. 2015-02-25 17:15:25 +00:00
Kegan Dougal
2d20466f9a Add stub functions and work out execution flow to implement AS event stream polling. 2015-02-25 15:00:59 +00:00
Paul "LeoNerd" Evans
9640510de2 Use OrderedDict for @cached backing store, so we can evict the oldest key unbiased 2015-02-23 18:41:58 +00:00
Paul "LeoNerd" Evans
f53fcbce97 Use cache.pop() instead of a separate membership test + del [] 2015-02-23 18:30:45 +00:00
Mark Haines
27080698e7 Fix code style warning 2015-02-23 18:19:13 +00:00
Mark Haines
74048bdd41 Remove unused import 2015-02-23 18:17:43 +00:00
Paul "LeoNerd" Evans
a09e59a698 Pull the _get_event_cache.setdefault() call out of the try block, as it doesn't need to be there and is confusing 2015-02-23 16:55:57 +00:00
Paul "LeoNerd" Evans
044d813ef7 Use the @cached decorator to implement the destination_retry_timings cache 2015-02-23 16:04:40 +00:00
Paul "LeoNerd" Evans
357fba2c24 RoomMemberStore no longer needs a _user_rooms_cache member 2015-02-23 15:57:41 +00:00
Paul "LeoNerd" Evans
e76d485e29 Allow @cached-wrapped functions to have a prefill method for setting entries 2015-02-23 15:41:54 +00:00
Paul "LeoNerd" Evans
4631b737fd Squash out the now-redundant ApplicationServicesCache object class 2015-02-23 14:38:44 +00:00
Erik Johnston
42b972bccd Revert get_auth_chain changes 2015-02-23 14:35:23 +00:00
Paul "LeoNerd" Evans
55022d6ca5 Remove a TODO note 2015-02-19 18:38:09 +00:00
Paul "LeoNerd" Evans
ebc3db295b Take named arguments to @cached() decorator, add a 'max_entries' limit 2015-02-19 18:36:02 +00:00
Paul "LeoNerd" Evans
077d200342 Move @cached decorator out into synapse.storage._base; add minimal docs 2015-02-19 17:29:39 +00:00
Erik Johnston
0ac2a79faa Initial stab at implementing a batched get_missing_pdus request 2015-02-19 17:24:14 +00:00
Paul "LeoNerd" Evans
61959928bb Pull out the 'get_rooms_for_user' cache logic into a reüsable @cached decorator 2015-02-19 14:58:07 +00:00
Erik Johnston
2c29ed3e84 Use absolute path when loading delta sql files 2015-02-17 17:22:24 +00:00
Erik Johnston
1a989c436c Bump schema version 2015-02-17 15:45:55 +00:00
Erik Johnston
fa6c93bd26 Merge branch 'consumeErrors' of github.com:matrix-org/synapse into develop 2015-02-17 15:18:17 +00:00
Erik Johnston
72a4de2ce6 Use consumeErrors=True on all DeferredLists. 
This is so that the DeferredLists actually consume the error instead of
propogating down the non-existent errback chain. This should reduce the
number of unhandled errors we are seeing.
 2015-02-17 10:07:01 +00:00
David Baker
64def4f953 Merge branch 'hotfixes-0.7.0e' into develop 2015-02-13 16:18:34 +00:00
David Baker
8d5cce62ab Update pushers by app id and pushkey, not user id and pushkey 2015-02-13 16:16:16 +00:00
Mark Haines
0d872f5aa6 Merge pull request #50 from matrix-org/application-services 
Application Services
 2015-02-13 15:06:14 +00:00
Mark Haines
183b3d4e47 Prepare the database whenever a connection is opened from the db_pool so that in-memory databases will work 2015-02-13 14:38:24 +00:00
Erik Johnston
6b186a57ba Merge branch 'fix' of github.com:matrix-org/synapse into hotfixes-v0.7.0a 2015-02-12 14:51:46 +00:00
Erik Johnston
717687e1fc Get an auth query one at a time 2015-02-12 14:39:31 +00:00
Erik Johnston
83d41f25d8 Set database schema version in delta 2015-02-12 10:05:47 +00:00
Erik Johnston
8bbdf32849 Convert get_rooms to use runInteraction so the transacion has a more helpful description 2015-02-11 18:56:13 +00:00
Erik Johnston
42bc56dad3 Merge branch 'develop' of github.com:matrix-org/synapse into use-simplejson 2015-02-11 17:01:38 +00:00
Erik Johnston
7eef84a95b pyflakes 2015-02-11 16:52:22 +00:00
Kegan Dougal
f2fdcb7c4b Merge branch 'develop' into application-services 2015-02-11 16:43:26 +00:00
Paul "LeoNerd" Evans
b164e0896c Merge branch 'bugs/SYN-264' into develop 2015-02-11 16:23:30 +00:00
Paul "LeoNerd" Evans
7f47ba7383 Added another TODO note 2015-02-11 16:18:21 +00:00
Paul "LeoNerd" Evans
45b56609ae Cache the result of a get_rooms_for_user query, to make user_rooms_intersect() much lighter in the read-common case 2015-02-11 16:04:08 +00:00
Paul "LeoNerd" Evans
7be0f6594e First step of making user_rooms_intersect() faster - implement in intersection logic in Python code terms of a DB query that is cacheable per user 2015-02-11 15:53:56 +00:00
Erik Johnston
ddb816cf60 Don't unfreeze when using FreezeEvent.get_dict, as we are using a JSONEncoder that understands FrozenDict 2015-02-11 15:44:28 +00:00
Mark Haines
ddc25cf4e2 Invalidate the cache for an event if it is redacted 2015-02-11 15:23:28 +00:00
Mark Haines
aff892ce79 Fix formatting 2015-02-11 15:02:35 +00:00
Mark Haines
f5a70e0d2e Add a cache for get_event 2015-02-11 15:01:15 +00:00
Erik Johnston
4ebbaf0d43 Blunty replace json with simplejson 2015-02-11 14:23:10 +00:00
Kegan Dougal
fd40d992ad PEP8-ify 2015-02-11 10:41:33 +00:00
Kegan Dougal
9978c5c103 Merge branch 'develop' into application-services 2015-02-11 10:03:24 +00:00
Mark Haines
b085fac735 Code-style fixes 2015-02-10 16:30:48 +00:00
Mark Haines
fda4422bc9 Fix pyflakes 2015-02-10 14:54:07 +00:00
Mark Haines
d7c7efb691 Add performance counters for different stages of loading events 2015-02-10 14:50:53 +00:00
Mark Haines
0c4536da8f Use the transaction 'desc' rather than 'name', increment the txn_ids in 
txn names
 2015-02-09 18:06:31 +00:00
Mark Haines
347b497db0 Formatting 2015-02-09 17:57:09 +00:00
Mark Haines
3a5ad7dbd5 Performance counters for database transaction names 2015-02-09 17:55:56 +00:00
Kegan Dougal
53557fc532 Merge branch 'develop' into application-services 2015-02-09 15:20:56 +00:00
Erik Johnston
c4ee4ce93e Fix typo 2015-02-09 15:00:37 +00:00
Erik Johnston
66fde49f07 Log database time every 10s and log as percentage 2015-02-09 14:45:15 +00:00
Erik Johnston
75656712e3 Time how long we're spending on the database thread 2015-02-09 14:22:52 +00:00
Kegan Dougal
5a7dd05818 Modify auth.get_user_by_req for authing appservices directly. 
Add logic to map the appservice token to the autogenned appservice user ID.
Add unit tests for all forms of get_user_by_req (user/appservice,
valid/bad/missing tokens)
 2015-02-09 14:14:15 +00:00
Kegan Dougal
ac3183caaa Register a user account for the AS when the AS registers. Add 'sender' column to AS table. 2015-02-09 12:03:37 +00:00
Erik Johnston
0cd66885e3 Move delta/v13.sql to delta/v12.sql 2015-02-06 14:38:04 +00:00
Kegan Dougal
a3c6010718 Add delta sql file. 2015-02-05 16:48:57 +00:00
Kegan Dougal
fc8bcc809d Merge branch 'develop' into application-services 2015-02-05 15:32:45 +00:00
David Baker
aaf50bf6f3 Give server default rules the 'default' attribute and fix various brokenness. 2015-02-05 15:11:38 +00:00
Kegan Dougal
951690e54d Merge branch 'develop' into application-services 2015-02-05 14:28:03 +00:00
Erik Johnston
f08bd95880 Merge pull request #47 from matrix-org/signature_failures 
Federation fixes.
 2015-02-05 14:03:00 +00:00
Kegan Dougal
27091f146a Add hs_token column and generate a different token f.e application service. 2015-02-05 10:08:12 +00:00
Kegan Dougal
89f2e8fbdf Fix bug in store defer. Add more unit tests. 2015-02-04 15:21:03 +00:00
Kegan Dougal
17753f0c20 Add stub ApplicationServiceApi and glue it with the handler. 2015-02-04 11:19:18 +00:00
Erik Johnston
03d415a6a2 Brief comment on why we do some things on every call to persist_event and not others 2015-02-04 10:40:59 +00:00
Erik Johnston
c0462dbf15 Rearrange persist_event so that do all the queries that need to be done before returning early if we have already persisted that event. 2015-02-04 10:16:51 +00:00
Erik Johnston
02be8da5e1 Add doc to get_event 2015-02-03 17:34:07 +00:00
David Baker
dc7bb70f22 s/instance_handle/profile_tag/ 2015-02-03 16:51:07 +00:00
Kegan Dougal
94a5db9f4d Add appservice package and move ApplicationService into it. 2015-02-03 14:44:16 +00:00
Kegan Dougal
f2c039bfb9 Implement restricted namespace checks. Begin fleshing out the main hook for notifying application services. 2015-02-03 13:29:27 +00:00
Kegan Dougal
a060b47b13 Add namespace constants. Add restrict_to option to limit namespace checks. 2015-02-03 13:17:28 +00:00
Kegan Dougal
3bd2841fdb Everyone loves SQL typos 2015-02-03 11:37:52 +00:00
Kegan Dougal
197f3ea4ba Implement regex checks for app services. 
Expose handler.get_services_for_event which manages the checks for all
services.
 2015-02-03 11:26:33 +00:00
Erik Johnston
e7ca813dd4 Try to ensure we don't persist an event we have already persisted. In persist_event check if we already have the event, if so then update instead of replacing so that we don't cause a bump of the stream_ordering. 2015-02-03 10:39:41 +00:00
Kegan Dougal
9ff349a3cb Add defers in the right places. 2015-02-02 17:42:49 +00:00
Kegan Dougal
1a2de0c5fe Implement txns for AS (un)registration. 2015-02-02 17:39:41 +00:00
Erik Johnston
941f59101b Don't fail an entire request if one of the returned events fails a signature check. If an event does fail a signature check, look in the local database and request it from the originator. 2015-02-02 16:56:01 +00:00
Kegan Dougal
a006d168c5 Actually merge into develop. 2015-02-02 16:05:34 +00:00
Kegan Dougal
c059c9fea5 Merge branch 'develop' into application-services 
Conflicts:
	synapse/handlers/__init__.py
	synapse/storage/__init__.py
 2015-02-02 15:57:59 +00:00
Erik Johnston
7d897f5bfc Merge pull request #43 from matrix-org/rejections 
Rejections
 2015-01-30 16:11:14 +00:00
Mark Haines
6dc92d3427 Merge pull request #41 from matrix-org/client_v2_sync 
Client v2 sync
 2015-01-30 15:54:25 +00:00
Erik Johnston
1bd540ef79 Merge branch 'develop' of github.com:matrix-org/synapse into rejections 
Conflicts:
	synapse/storage/schema/im.sql
 2015-01-30 15:16:38 +00:00
Mark Haines
9ec9d6f2cb Merge pull request #42 from matrix-org/replication_split 
Replication split
 2015-01-30 15:14:10 +00:00
Erik Johnston
91015ad008 Remove merge conflict 2015-01-30 14:58:54 +00:00
Erik Johnston
4f7fe63b6d Remember to add schema file to list 2015-01-30 14:57:53 +00:00
Erik Johnston
fdd2ac495a Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage 
Conflicts:
	synapse/storage/__init__.py
 2015-01-30 14:57:33 +00:00
Mark Haines
8bc3066e0b Merge branch 'client_v2_filter' into client_v2_sync 2015-01-30 14:57:04 +00:00
Erik Johnston
471c47441d Merge pull request #37 from matrix-org/client_v2_filter 
Client v2 filter
 2015-01-30 14:56:08 +00:00
Mark Haines
e97f756a05 Use 'in' to test if the key exists, remove unused _filters_for_user 2015-01-30 14:54:06 +00:00
Erik Johnston
2f4cb04f45 Be more specific in naming columns in selects. 2015-01-30 14:48:11 +00:00
Erik Johnston
472cf532b7 Put CREATE rejections into seperate .sql 2015-01-30 14:48:03 +00:00
David Baker
322a047502 Add room member count condition and default rule to make a noise on rooms of only 2 people. 2015-01-30 14:46:45 +00:00
Erik Johnston
c515d37797 Merge branch 'replication_split' of github.com:matrix-org/synapse into rejections 
Conflicts:
	synapse/storage/schema/delta/v12.sql
 2015-01-30 14:19:49 +00:00
Erik Johnston
84b78c3b5f Merge branch 'rejections_storage' of github.com:matrix-org/synapse into replication_split 2015-01-30 14:17:47 +00:00
Erik Johnston
e0b7c521cb Merge branch 'develop' of github.com:matrix-org/synapse into rejections_storage 
Conflicts:
	synapse/storage/__init__.py
	synapse/storage/schema/delta/v12.sql
 2015-01-30 14:08:28 +00:00
Mark Haines
8498d348d8 Fix token formatting 2015-01-30 11:42:09 +00:00
Mark Haines
22dd1cde2d Filter the recent events before applying the limit when doing an incremental sync with a gap 2015-01-30 11:32:35 +00:00
Erik Johnston
2ebf795c0a Merge branch 'develop' of github.com:matrix-org/synapse into rejections 
Conflicts:
	synapse/storage/__init__.py
	synapse/storage/schema/delta/v12.sql
 2015-01-30 11:10:37 +00:00
Erik Johnston
78015948a7 Initial implementation of auth conflict resolution 2015-01-29 16:52:33 +00:00
Mark Haines
8e571cbed8 Merge branch 'client_v2_filter' into client_v2_sync 2015-01-29 16:18:59 +00:00
Mark Haines
295322048d Merge branch 'develop' into client_v2_filter 2015-01-29 16:18:34 +00:00
Mark Haines
acb68a39e0 Code style fixes. 2015-01-29 16:12:40 +00:00
Mark Haines
396a67a09a Merge branch 'client_v2_filter' into client_v2_sync 
Conflicts:
	synapse/rest/client/v2_alpha/__init__.py
 2015-01-29 14:58:00 +00:00
Mark Haines
9d8f798a3f Merge changes from develop 2015-01-29 14:55:27 +00:00
Mark Haines
e4f50fa0aa Move bump schema delta 2015-01-29 14:53:18 +00:00
Mark Haines
e016f4043b Use get_room_events_stream to get changes to the rooms if the number of changes is small 2015-01-29 14:40:28 +00:00
Kegan Dougal
3773759c0f Also edit the filter column on the delta SQL 2015-01-29 09:15:33 +00:00
Mark Haines
b0b80074e0 SYN-252: Supply the stream and topological parts in the correct order to the constructor 2015-01-29 01:48:48 +00:00
David Baker
d5bdf3c0c7 Allow the push rule delete method to take more specifiers. 2015-01-28 18:06:04 +00:00
Kegan Dougal
11634017f4 s/definition/filter_json/ since definition is now used to mean a component of the filter, rather than the complete json 2015-01-28 17:42:19 +00:00
Mark Haines
9c61556504 Merge branch 'develop' into client_v2_sync 2015-01-28 17:29:30 +00:00
Mark Haines
26c8fff19e Merge pull request #36 from matrix-org/device_id_from_access_token 
Extract the device id and token id from the access token when autheniticating users
 2015-01-28 17:19:28 +00:00
Mark Haines
388581e087 Extract the id token of the token when authing users, include the token and device_id in the internal meta data for the event along with the transaction id when sending events 2015-01-28 16:58:23 +00:00
Kegan Dougal
c23e3db544 Add filter JSON sanity checks. 2015-01-28 16:45:18 +00:00
Erik Johnston
0ef5bfd6a9 Start implementing auth conflict res 2015-01-28 16:16:53 +00:00
David Baker
60b143a52e Move pushers delta to v12 and bump schema version 2015-01-28 15:48:28 +00:00
David Baker
e78dd33292 Use %s instead of + 2015-01-28 14:52:58 +00:00
David Baker
93aac9bb7b Newline 2015-01-28 14:51:01 +00:00
David Baker
445ad9941e Redundant parens 2015-01-28 14:49:59 +00:00
David Baker
6d485dd1c7 unnecessary newlines 2015-01-28 14:48:42 +00:00
David Baker
fb0928097a More magic commas (including the place I copied it from...) 2015-01-28 14:48:07 +00:00
David Baker
0cbb6b0f52 Google doc style 2015-01-28 14:44:41 +00:00
David Baker
c291a4d522 Merge branch 'develop' into pushers 
Conflicts:
	synapse/handlers/events.py
	synapse/server.py
 2015-01-28 12:51:05 +00:00
Kegan Dougal
42876969b9 Add basic application_services SQL, and hook up parts of the appservice store to read from it. 2015-01-28 11:59:38 +00:00
Kegan Dougal
b46fa8603e Remove unused import 2015-01-28 09:17:48 +00:00
Mark Haines
e020574d65 Fix Formatting 2015-01-27 20:19:36 +00:00
Paul "LeoNerd" Evans
8398f19bce Created schema delta 2015-01-27 19:00:09 +00:00
Paul "LeoNerd" Evans
06cc147012 Initial stab at real SQL storage implementation of user filter definitions 2015-01-27 18:46:03 +00:00
Paul "LeoNerd" Evans
54e513b4e6 Move storage of user filters into real datastore layer; now have to mock it out in the REST-level tests 2015-01-27 17:48:13 +00:00
Kegan Dougal
fbeaeb8689 Log when ASes are registered/unregistered. 2015-01-27 17:34:40 +00:00
Kegan Dougal
ec3719b583 Use ApplicationService when registering. 2015-01-27 17:15:06 +00:00
Kegan Dougal
92171f9dd1 Add stub methods, TODOs and docstrings for application services. 2015-01-27 16:53:59 +00:00
Mark Haines
a56008842b Start implementing incremental initial sync 2015-01-27 16:24:22 +00:00
Kegan Dougal
7331d34839 Add AS specific classes with docstrings. 2015-01-27 16:23:46 +00:00
Kegan Dougal
51449e0665 Add appservice handler and store. Glue together rest > handler > store. 2015-01-27 15:50:28 +00:00
David Baker
69a75b7ebe Add brackets to make get room name / alias work 2015-01-26 16:52:47 +00:00
Erik Johnston
7b88619241 Split up replication_layer module into client, server and transaction queue 2015-01-26 10:45:24 +00:00
David Baker
5f84ba8ea1 Add API to delete push rules. 2015-01-23 17:49:37 +00:00
Erik Johnston
9b1e552b51 Merge branch 'develop' of github.com:matrix-org/synapse into rejections 2015-01-23 15:51:48 +00:00
Erik Johnston
3b9cc882a5 Add storage method have_events 2015-01-23 15:42:52 +00:00
Mark Haines
5759bec43c Replace hs.parse_userid with UserID.from_string 2015-01-23 11:47:15 +00:00
David Baker
bcd48b9636 Fix adding rules without before/after & add the rule that we couldn't find to the error 2015-01-23 10:28:25 +00:00
David Baker
8a850573c9 As yet fairly untested GET API for push rules 2015-01-22 19:32:17 +00:00
David Baker
673773b217 oops, this is not its own schema file 2015-01-22 18:27:07 +00:00
David Baker
7ecb49ef25 Insufficient newlines 2015-01-22 17:53:30 +00:00
David Baker
5c6189ea3e Merge branch 'develop' into pushers 
Conflicts:
	synapse/rest/__init__.py
 2015-01-22 17:46:16 +00:00
David Baker
ede491b4e0 Oops: second part of commit dc938606 2015-01-22 17:38:53 +00:00
David Baker
dc93860619 Add rest API & store for creating push rules 
Also make unrecognised request error look more like synapse errors
because it makes it easier to throw them from within rest classes.
 2015-01-22 17:37:12 +00:00
Erik Johnston
b1b85753d7 Add support for storing rejected events in EventContext and data stores 2015-01-22 15:50:17 +00:00
David Baker
afb714f7be add instance_handles to pushers so we have a way to refer to them even if the push token changes. 2015-01-20 11:49:48 +00:00
David Baker
2d2953cf5f Require device language when adding a pusher. 
Because this seems like it might be useful to do sooner rather
than later.
 2015-01-16 11:24:10 +00:00
David Baker
2ca2dbc821 Send room name and first alias in notification poke. 2015-01-15 16:56:18 +00:00
David Baker
2cb30767fa Honour the 'rejected' return from push gateways 
Add a timestamp to push tokens so we know the last time they we
got them from the device. Send it to the push gateways so it can
determine whether its failure is more recent than the token.
Stop and remove pushers that have been rejected.
 2015-01-13 19:48:37 +00:00
Mark Haines
3891597eb3 Remove unused functions 2015-01-13 15:57:26 +00:00
Mark Haines
fda63064fc get_room_events isn't called anywhere 2015-01-13 14:43:26 +00:00
Mark Haines
895fcb377e Fix stream token ordering 2015-01-13 14:38:53 +00:00
David Baker
c06a9063e1 Merge branch 'develop' into pushers 2015-01-13 13:15:51 +00:00
Kegan Dougal
c43d898119 SYN-178: Fix off by one. 2015-01-12 17:38:40 +00:00
Erik Johnston
36a2a877e2 Use time.time() instead of time.clock() 2015-01-06 16:34:41 +00:00
Erik Johnston
fd9a8db7ea Only fetch the columns we need. 2015-01-06 15:59:31 +00:00
Erik Johnston
a01416cf21 Add delta and bump DB version 2015-01-06 15:42:18 +00:00
Erik Johnston
f6da237c35 Add index on transaction_id to sent_transcations 2015-01-06 15:40:38 +00:00
Erik Johnston
9bd07bed23 Actually time that function 2015-01-06 15:28:56 +00:00
Erik Johnston
03a501456c Time how long calls to _get_destination_retry_timings take 2015-01-06 15:22:28 +00:00
Erik Johnston
52b2c6c9c7 Don't include None's in _get_events_txn 2015-01-06 14:56:57 +00:00
Erik Johnston
8a12df8cf3 Merge branch 'erikj-perf' of github.com:matrix-org/synapse into develop 2015-01-06 14:45:57 +00:00
Erik Johnston
96707ed718 Name 'user_rooms_intersect' transaction 2015-01-06 14:44:27 +00:00
Erik Johnston
76ec154e95 We don't need the full events for get_rooms_for_user_where_membership_is 2015-01-06 14:37:00 +00:00
Mark Haines
5e23a19204 Merge pull request #28 from matrix-org/erikj-perf 
Database performance improvements.
 2015-01-06 13:33:40 +00:00
Mark Haines
adb04b1e57 Update copyright notices 2015-01-06 13:21:39 +00:00
Erik Johnston
af1c7c7808 PEP8 2015-01-06 13:13:17 +00:00
Erik Johnston
12819d5082 Remove debug lines 2015-01-06 13:12:30 +00:00
Erik Johnston
52d8519008 Don't do batching when getting events. 2015-01-06 13:10:27 +00:00
Erik Johnston
98933e3db6 Only fetch prev_content when a client is streaming/paginating. Use transactions for event streams. 2015-01-06 13:03:23 +00:00
Erik Johnston
3e26720e05 Temporarily turn off 'redacted_because' and 'prev_content' keys 2015-01-06 11:26:58 +00:00
Erik Johnston
f4ea78e9e2 More debug logging 2015-01-06 11:24:18 +00:00
Erik Johnston
753126b8cc Add some debug logging 2015-01-06 11:18:12 +00:00
Erik Johnston
d7e8ea67b3 Reformat 2015-01-06 11:18:02 +00:00
Erik Johnston
f0128f9600 Add RoomMemberStore.get_users_in_room, so that we can get the list of joined users without having to retrieve the full events 2015-01-06 10:55:43 +00:00
Erik Johnston
d61109f578 Merge branch 'hotfixes-v0.6.0' of github.com:matrix-org/synapse into erikj-perf 2014-12-19 16:37:08 +00:00
Mark Haines
9c71d945d6 Look for name, topic in the event content rather than the event itself when persisting room name and topic events 2014-12-19 15:16:48 +00:00
David Baker
b56730bb6e Merge branch 'develop' into pushers 
Conflicts:
	synapse/api/errors.py
	synapse/server.py
	synapse/storage/__init__.py
 2014-12-18 15:15:22 +00:00
David Baker
afa953a293 schema version is now 10 2014-12-18 15:11:06 +00:00
David Baker
0a6664493a Merge branch 'master' into pushers 2014-12-18 15:06:11 +00:00
David Baker
4c7ad50f6e Thank you, pyflakes 2014-12-18 14:55:04 +00:00
David Baker
173264b656 ...and bump SCHEMA_VERSION 2014-12-18 14:53:10 +00:00
David Baker
fc7c5e9cd7 Rename the pusher SQL delta to v9 which the next free one 2014-12-18 14:51:29 +00:00
David Baker
9728c305a3 after a few rethinks, a working implementation of pushers. 2014-12-18 14:49:22 +00:00
Erik Johnston
f3788e3c78 Test some ideas that might help performance a bit 2014-12-17 23:37:08 +00:00
Erik Johnston
dec5b62339 Use _get_events_txn instead of _parse_events_txn 2014-12-16 19:16:41 +00:00
Erik Johnston
2215faa361 Merge branch 'hotfixes-v0.5.4a' of github.com:matrix-org/synapse into release-v0.6.0 2014-12-16 19:11:13 +00:00
Erik Johnston
96779d2490 Fix bug where we did not send the full auth chain to people that joined over federation 2014-12-16 18:57:36 +00:00
Erik Johnston
f76269392b Merge branch 'develop' of github.com:matrix-org/synapse into release-v0.6.0 
Conflicts:
	synapse/state.py
 2014-12-16 18:35:46 +00:00
Erik Johnston
52f99243ab Use is_outlier() so that we don't get AttributeError 2014-12-16 18:33:50 +00:00
Erik Johnston
5b39cfff69 Don't assume an event exists 2014-12-16 18:25:24 +00:00
Mark Haines
56db465047 Merge branch 'release-v0.6.0' into develop 2014-12-16 17:29:49 +00:00
Mark Haines
627e4f01d2 Remove send_message since nothing was calling it. Remove Snapshot because only send_message was using it 2014-12-16 16:07:41 +00:00
Erik Johnston
42b725ce52 Fix upgrade script to run all the missing deltas. 2014-12-16 15:13:34 +00:00
Erik Johnston
b3c793e362 Do run all deltas up to missing delta 10 2014-12-16 14:44:53 +00:00
Erik Johnston
2e44714214 Make failure to run appropraite upgrade scripts more helpful. 2014-12-16 14:20:32 +00:00
Erik Johnston
ef5a141050 Bump database version 2014-12-16 13:57:47 +00:00
Erik Johnston
35f4f6b070 Update upgrade script 2014-12-16 13:27:53 +00:00
Erik Johnston
882dc8dcab Persist internal_metadata 2014-12-16 13:17:09 +00:00
Erik Johnston
3c77d13aa5 Kill off synapse.api.events.* 2014-12-16 11:29:05 +00:00
Erik Johnston
f280929a12 Use frozenutils 2014-12-15 17:31:36 +00:00
Erik Johnston
c8dd3314d6 Fix bug where we ignored event_edge_hashes table 2014-12-15 13:55:22 +00:00
Erik Johnston
c39beb5559 Store json as UTF-8 and not bytes 2014-12-12 14:53:37 +00:00
Erik Johnston
75085bb4d1 Pyflakes 2014-12-12 14:34:34 +00:00
Erik Johnston
cde840a82c Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor 
Conflicts:
	setup.py
 2014-12-11 17:48:48 +00:00
Erik Johnston
0b04369238 Fix public room joining by making sure replaces_state never points to itself. 2014-12-11 15:56:06 +00:00
Erik Johnston
9191292b0f Fix prev_content 2014-12-11 15:16:55 +00:00
Erik Johnston
8cdebce470 Fix redactions. Fix 'age' key 2014-12-11 13:25:19 +00:00
Erik Johnston
02e4c18171 Remove dead code 2014-12-10 18:00:36 +00:00
Mark Haines
4f37c0ea9d Merge branch 'develop' into media_repository 2014-12-10 16:55:06 +00:00
Mark Haines
61fc37e467 Merge branch 'develop' into media_repository 2014-12-10 16:14:17 +00:00
Erik Johnston
1d2a0040cf Fix bug where we clobbered old state group values 2014-12-10 15:55:03 +00:00
Mark Haines
e5275d856e Get the code actually working 2014-12-10 15:46:18 +00:00
Mark Haines
cc84d3ea78 Thumbnail uploaded and cached images 2014-12-10 15:40:52 +00:00
Erik Johnston
02db7eb209 Fix bug when uploading state with empty state_key 2014-12-10 14:02:48 +00:00
Erik Johnston
aae8a37e63 Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor 2014-12-10 13:18:40 +00:00
Erik Johnston
95aa903ffa Try and figure out how and why signatures are being changed. 2014-12-10 11:37:47 +00:00
Erik Johnston
b8d30899b1 Code style. 2014-12-10 10:16:09 +00:00
Matthew Hodgson
2b1acb7671 squidge to 79 columns as per pep8 2014-12-10 00:03:55 +00:00
Erik Johnston
aa3f66cf7f Change the way we implement get_events to be less sucky 2014-12-09 13:35:26 +00:00
Matthew Hodgson
8529fba02d fix a million stupid bugs and make it actually work 2014-12-08 19:34:51 +00:00
Matthew Hodgson
0d3fa1ac6e add a write-through cache on the retry schedule 2014-12-08 17:48:57 +00:00
Erik Johnston
ba3d1e2fc0 Remove unused import 2014-12-08 12:01:25 +00:00
Erik Johnston
d044121168 Various typos and bug fixes. 2014-12-08 09:08:26 +00:00
Matthew Hodgson
5cd43d4b9f fix stupid syntax thinkos 2014-12-07 23:44:16 +00:00
Matthew Hodgson
aed62a3583 track replication destination health, and perform exponential back-off when sending transactions. does *not* yet retry transactions, but drops them on the floor if waiting for a server to recover. 2014-12-07 02:26:07 +00:00
Mark Haines
a953be097f Add a method field to thumbnail storage 2014-12-05 16:31:56 +00:00
Erik Johnston
6630e1b579 Start making more things use EventContext rather than event.* 2014-12-05 16:20:48 +00:00
Erik Johnston
c31dba86ec Convert rest and handlers to use new event structure 2014-12-04 15:50:01 +00:00
Mark Haines
c01fd5573c Implement download support for media_repository 2014-12-04 14:22:31 +00:00
Erik Johnston
5d7c9ab789 Begin converting things to use the new Event structure 2014-12-04 11:27:59 +00:00
David Baker
88af58d41d Update to app_id / app_instance_id (partially) and mangle to be PEP8 compliant. 2014-12-03 13:37:02 +00:00
Erik Johnston
6941a19715 Merge branch 'develop' of github.com:matrix-org/synapse into events_refactor 2014-12-03 11:56:49 +00:00
Paul "LeoNerd" Evans
10eb8f070c Workaround for non-uniqueness of room member events in the database confusing HAVING COUNT() test of room membership intersection (with thanks to Tom Molesworth) 2014-12-02 21:36:53 +00:00
Mark Haines
2f804a7072 Fix pyflakes and pep8 warnings 2014-12-02 19:55:18 +00:00
Mark Haines
5da65085d1 Get uploads working with new media repo 2014-12-02 19:51:47 +00:00
Mark Haines
279c48c8b4 Write the upload portion of version 1 of the media repository 2014-12-02 17:13:14 +00:00
David Baker
7642d95d5e Merge branch 'develop' into pushers 2014-12-02 13:50:05 +00:00
Erik Johnston
ec2b5d8c28 Store full JSON of events in db 2014-12-01 16:22:07 +00:00
Erik Johnston
1505055334 Don't return outliers when we get recent events for rooms. 2014-11-27 16:38:50 +00:00
Erik Johnston
4e2ffe79a4 Don't delete the entire current_state_events table 2014-11-26 15:17:08 +00:00
Erik Johnston
cb76945688 Add update delta for schema change 2014-11-26 11:17:19 +00:00
Erik Johnston
87538711b6 Update schema to support multiple signatures 2014-11-26 11:14:30 +00:00
Erik Johnston
3598c11c8d Correctly handle the case where we get an event for an unknown room, which turns out we are actually in 2014-11-26 10:41:08 +00:00
Erik Johnston
a46e5ef621 SYN-163: Add an order by rowid to selects. 
This should fix the bug where the edges of the graph get returned in a
different order than they were inserted in, and so no get_event no
longer returned the exact same JSON as was inserted. This meant that
signature checks failed.
 2014-11-24 10:56:36 +00:00
David Baker
eb6aedf92c More work on pushers. Attempt to do HTTP pokes. Not sure if the actual HTTP pokes work or not yet but the retry semantics are pretty good. 2014-11-21 12:21:00 +00:00
David Baker
23465a30b6 Merge branch 'develop' into pushers 2014-11-20 18:17:46 +00:00
Mark Haines
db9ce032a4 Fix pep8 codestyle warnings 2014-11-20 17:26:36 +00:00
Mark Haines
dfdda2c871 Use module loggers rather than the root logger. Exceptions caused by bad clients shouldn't cause ERROR level logging. Fix sql logging to use 'repr' rather than 'str' 2014-11-20 17:10:37 +00:00
Mark Haines
32090aee16 Add a few missing yields, Move deferred lists inside PreserveLoggingContext because they don't interact well with the logging contexts 2014-11-20 16:24:00 +00:00
David Baker
74c3879760 Start creating a module to do generic notifications (just prints them to stdout currently!) 2014-11-19 18:20:59 +00:00
Erik Johnston
5b46ce579b Bump version, changelog and upgrade.rst 2014-11-19 18:00:57 +00:00
Mark Haines
1731af3f29 SYN-104: When going backwards the end token should be before the last event 2014-11-18 16:45:06 +00:00
Mark Haines
ae9c2ab165 SYN-149: Send join event immediately after the room create event 2014-11-18 15:29:48 +00:00
Mark Haines
a5b88c489e Split out sending the room alias events from creating the alias so that we can do them in the right point when creating a room 2014-11-18 15:03:13 +00:00
Mark Haines
cb4b6c844a Merge PDUs and Events into one object 2014-11-14 21:25:02 +00:00
Mark Haines
8c2b5ea7c4 Fix PDU and event signatures 2014-11-14 19:11:04 +00:00
Mark Haines
e903c941cb Merge branch 'develop' into request_logging 
Conflicts:
	setup.py
	synapse/storage/_base.py
	synapse/util/async.py
 2014-11-14 11:16:50 +00:00
Erik Johnston
f04b3d5042 Store all signatures on events rather than just dropping them 2014-11-12 17:02:34 +00:00
Erik Johnston
e715741abc Update some of the docs in event_federation 2014-11-12 16:20:30 +00:00
Erik Johnston
e24d5cb97d Document StateStore and use transactions 2014-11-12 14:33:48 +00:00
Erik Johnston
58c0ef90c9 Add indices to state group tables 2014-11-12 14:33:48 +00:00
Erik Johnston
6fea478d2e Fix bugs with invites/joins across federatiom. 
Both in terms of auth and not trying to fetch missing PDUs for invites,
joins etc.
 2014-11-12 11:24:11 +00:00
Erik Johnston
5ff0bfb81d Fix bug where we /always/ created a new state group 2014-11-11 14:16:41 +00:00
Erik Johnston
a8e565eca8 Add an EventValidator. Fix bugs in auth ++ storage 2014-11-10 18:25:42 +00:00
Erik Johnston
cdc1b5d629 Fix regression where we did not return redacted events. 2014-11-10 15:21:30 +00:00
Erik Johnston
5d439b127b PEP8 2014-11-10 13:46:44 +00:00
Erik Johnston
6447db063a Fix backfill to work. Add auth to backfill request 2014-11-10 11:59:51 +00:00
Erik Johnston
65f846ade0 Notify users about invites. 2014-11-10 11:15:02 +00:00
Erik Johnston
6cb6cb9e69 Tidy up some of the unused sql tables 2014-11-10 10:31:00 +00:00
Erik Johnston
1c06806f90 Finish redaction algorithm. 2014-11-10 10:21:32 +00:00
Erik Johnston
02c3b1c9e2 Add '/event_auth/' federation api 2014-11-07 15:35:53 +00:00
Erik Johnston
16a0815fac Fix bug in _get_auth_chain_txn 2014-11-07 11:21:20 +00:00
Erik Johnston
49948d72f3 Fix joining over federation 2014-11-07 10:53:38 +00:00
Erik Johnston
8b0e96474b Implement method to get auth_chain from a given event_id 2014-11-07 10:53:38 +00:00
Erik Johnston
bf6b72eb55 Start implementing auth chains 2014-11-07 10:53:38 +00:00
Erik Johnston
351c64e99e Amalgamate all power levels. 
Remove concept of reqired power levels, something similiar can be done
using the new power level event.
 2014-11-06 16:59:13 +00:00
Erik Johnston
4317c8e583 Implement new replace_state and changed prev_state 
`prev_state` is now a list of previous state ids, similiar to
prev_events. `replace_state` now points to what we think was replaced.
 2014-11-06 15:10:55 +00:00
Erik Johnston
cc44ecc62f Get correct prev_events 2014-11-05 13:23:35 +00:00
Erik Johnston
aa76bf39ab Remove unused imports 2014-11-04 14:14:02 +00:00
Erik Johnston
d59aa6af25 For now, don't store txn -> pdu mappings. 2014-11-03 11:35:19 +00:00
Erik Johnston
f139c02e95 Formatting 2014-11-03 11:34:49 +00:00
Erik Johnston
ecabff7eb4 Sign evnets 2014-10-31 17:08:52 +00:00
Erik Johnston
80b2710e6f Remove unused signature storage methods 2014-10-31 17:08:36 +00:00
Erik Johnston
fb3a01fa3a Remove unused sql file. 2014-10-31 16:04:27 +00:00
Erik Johnston
d30d79b5be Make prev_event signing work again. 2014-10-31 15:35:39 +00:00
Erik Johnston
2f39dc19a2 Remove more references to dead PDU tables 2014-10-31 14:27:14 +00:00
Erik Johnston
bfa36a72b9 Remove PDU tables. 2014-10-31 14:00:32 +00:00
Erik Johnston
21fe249d62 Actually don't store any PDUs 2014-10-31 10:47:34 +00:00
Erik Johnston
841df4da71 Don't store any PDUs 2014-10-31 09:59:59 +00:00
Erik Johnston
f2de2d644a Move the impl of backfill to use events. 2014-10-31 09:59:02 +00:00
Erik Johnston
ef9c4476a0 Merge branch 'develop' of github.com:matrix-org/synapse into federation_authorization 2014-10-30 11:18:28 +00:00
Mark Haines
7c06399512 Merge branch 'develop' into request_logging 
Conflicts:
	synapse/config/logger.py
 2014-10-30 11:13:58 +00:00
Mark Haines
7d709542ca Fix pep8 warnings 2014-10-30 11:10:17 +00:00
Erik Johnston
aa80900a8e Fix SQL so that accepts we may want to persist events twice. 2014-10-30 10:11:06 +00:00
Mark Haines
b29517bd01 Add a request-id to each log line 2014-10-30 01:21:33 +00:00
Erik Johnston
e7858b6d7e Start filling out and using new events tables 2014-10-29 16:59:24 +00:00
Erik Johnston
a10c2ec88d Don't reference PDU when persisting event 2014-10-28 17:15:32 +00:00
Erik Johnston
2d1dfb3b34 Begin implementing all the PDU storage stuff in Events land 2014-10-28 16:42:35 +00:00
Erik Johnston
da1dda3e1d Add transaction level logging and timing information. Add a _simple_delete method 2014-10-28 11:18:04 +00:00
Erik Johnston
967ce43b59 Clean up LoggingTransaction 2014-10-28 10:53:11 +00:00
Erik Johnston
8e358ef35a Add timer to LoggingTransaction 2014-10-28 10:34:05 +00:00
Erik Johnston
ad9226eeec Merge branch 'event_signing' of github.com:matrix-org/synapse into federation_authorization 
Conflicts:
	synapse/storage/__init__.py
 2014-10-27 11:58:32 +00:00
Mark Haines
5e2236f9ff fix pyflakes warnings 2014-10-27 11:19:15 +00:00
Mark Haines
acb2d171e8 Merge branch 'develop' into event_signing 2014-10-27 11:14:11 +00:00
Mark Haines
d56e389a95 Fix pyflakes warnings 2014-10-27 10:33:17 +00:00
Erik Johnston
bb4a20174c Merge branch 'develop' of github.com:matrix-org/synapse into federation_authorization 
Conflicts:
	synapse/federation/transport.py
	synapse/handlers/message.py
 2014-10-27 10:20:44 +00:00
Mark Haines
db2e350e29 Wrap preparing the database in a transaction. Otherwise it will take many seconds to complete because sqlite will create a transaction per statement 2014-10-24 19:04:26 +01:00
Erik Johnston
5662be894e Bump database version number. 2014-10-17 20:26:18 +01:00
Erik Johnston
5ffe5ab43f Use state groups to get current state. Make join dance actually work. 2014-10-17 18:56:42 +01:00
Mark Haines
dc3c2823ac Merge branch 'develop' into event_signing 
Conflicts:
	synapse/federation/replication.py
 2014-10-17 17:33:58 +01:00
Mark Haines
82c5820767 keep 'origin_server_ts' as 'ts' in the database to avoid needlessly updating schema 2014-10-17 17:31:48 +01:00
Mark Haines
f5cf7ac25b SPEC-7: Rename 'ts' to 'origin_server_ts' 2014-10-17 17:12:25 +01:00
Mark Haines
c8f996e29f Hash the same content covered by the signature when referencing previous PDUs rather than reusing the PDU content hashes 2014-10-17 11:40:35 +01:00
Mark Haines
bb04447c44 Include hashes of previous pdus when referencing them 2014-10-16 23:25:12 +01:00
Mark Haines
66104da10c Sign outgoing PDUs. 2014-10-16 00:09:48 +01:00
Mark Haines
1c445f88f6 persist hashes and origin signatures for PDUs 2014-10-15 17:09:04 +01:00
Erik Johnston
e7bc1291a0 Begin making auth use event.old_state_events 2014-10-15 16:06:59 +01:00
Erik Johnston
80472ac198 Add missing package storate.state 2014-10-15 10:04:55 +01:00
Erik Johnston
5fefc12d1e Begin implementing state groups. 2014-10-14 16:59:51 +01:00
Mark Haines
3dac27a8a9 Storage for pdu signatures 2014-10-14 14:58:31 +01:00
Mark Haines
07639c79d9 Respond with more helpful error messages for unsigned requests 2014-10-13 16:39:15 +01:00
Mark Haines
9605593d11 Merge branch 'develop' into server2server_signing 
Conflicts:
	synapse/storage/__init__.py
	tests/rest/test_presence.py
 2014-09-30 17:55:06 +01:00
Mark Haines
b95a178584 SYN-75 Verify signatures on server to server transactions 2014-09-30 15:15:10 +01:00
Erik Johnston
fbf6320614 pyflakes cleanup 2014-09-30 12:38:38 +01:00
Erik Johnston
e06adc6d7e SYN-2: Allow server admins to delete room aliases 2014-09-30 11:31:42 +01:00
Erik Johnston
7151615260 Update docstring 2014-09-29 15:35:54 +01:00
Erik Johnston
1550ab9e2f SYN-48: Delete dead code 2014-09-29 15:04:47 +01:00
Erik Johnston
3ccb17ce59 SYN-48: Implement WHOIS rest servlet 2014-09-29 14:59:52 +01:00
Erik Johnston
c65306f877 Add auth check to test if a user is an admin or not. 2014-09-29 13:35:38 +01:00
Erik Johnston
f7d80930f2 SYN-48: Track User-Agents as well as IPs for client devices. 2014-09-29 13:35:15 +01:00
Erik Johnston
0fdf308874 Track the IP users connect with. Add an admin column to users table. 2014-09-26 16:36:24 +01:00
Erik Johnston
37bfe44046 Merge branch 'deletions' of github.com:matrix-org/synapse into develop 2014-09-25 17:02:53 +01:00
Erik Johnston
c818aa13eb Add LIMIT to scalar subquery 2014-09-25 15:51:21 +01:00
Erik Johnston
ba87eb6753 Fix bug where we tried to insert state events with null state key 2014-09-25 14:45:27 +01:00
Erik Johnston
70899d3ab2 Rename deletions to redactions 2014-09-24 15:27:59 +01:00
Erik Johnston
7d9a84a445 Make deleting deletes not undelete 2014-09-24 14:18:08 +01:00
Erik Johnston
1e6c5b205c Fix bug where we didn't correctly pull out the event_id of the deletion 2014-09-24 13:29:20 +01:00
Erik Johnston
4354590a69 Add v4 deltas to current sql. 2014-09-24 11:06:41 +01:00
Erik Johnston
efea61dc50 Rename 'pruned' to 'pruned_because' 2014-09-23 17:40:58 +01:00
Erik Johnston
bc250a6afa SYN-12: Implement auth for deletion by adding a 'delete_level' on the ops levels event 
SYN-12 # comment Auth has been added.
 2014-09-23 17:36:24 +01:00
Erik Johnston
b99f6eb904 Make sure we don't persist the 'pruned' key 2014-09-23 15:29:27 +01:00
Erik Johnston
78af6bbb98 Add m.room.deletion. If an event is deleted it will be returned to clients 'pruned', i.e. all client specified keys will be removed. 2014-09-23 15:28:32 +01:00
Erik Johnston
231afe464a Add a deletions table 2014-09-22 13:42:52 +01:00
Erik Johnston
28bcd01e8d SYN-47: Fix bug where we still returned events for rooms we had left. 
SYN-47 #resolve
 2014-09-19 14:45:21 +01:00
Paul "LeoNerd" Evans
4571cf7baa Merge branch 'develop' into test-sqlite-memory 2014-09-17 18:27:47 +01:00
Erik Johnston
59516a8bb1 Correctly handle receiving 'missing' Pdus from federation, rather than just discarding them. 2014-09-15 16:40:44 +01:00
Paul "LeoNerd" Evans
b0406b9ead Merge remote-tracking branch 'origin/develop' into test-sqlite-memory 2014-09-15 14:15:10 +01:00
Erik Johnston
5bd9369a62 Correctly handle the 'age' key in events and pdus 2014-09-15 13:26:11 +01:00
Erik Johnston
14975ce5bc Fix bug where we relied on the current_state_events being updated when we are handling type specific persistence 2014-09-12 17:57:02 +01:00
Erik Johnston
667e747ed1 Fix bug where we no longer stored user_id on Pdus 2014-09-12 17:56:21 +01:00
Paul "LeoNerd" Evans
1c51c8ab7d Merge remote-tracking branch 'origin/develop' into test-sqlite-memory 
Conflicts:
	synapse/storage/pdu.py
 2014-09-12 17:20:06 +01:00
Erik Johnston
39e3fc69e5 Make the state resolution use actual power levels rather than taking them from a Pdu key. 2014-09-12 17:11:09 +01:00
Erik Johnston
b42fe05c51 Fix bug where we incorrectly removed a remote host from the list of hosts in a room when any user from that host left that room even if they weren't the last user from that host in that room 2014-09-12 17:11:09 +01:00
Erik Johnston
ca1ae7cf9b Fix bug where we didn't return a tuple when expected. 2014-09-12 17:11:09 +01:00
Paul "LeoNerd" Evans
aa525e4a63 More accurate docs / clearer paramter names in RoomMemberStore 2014-09-12 16:43:49 +01:00
Paul "LeoNerd" Evans
a87eac4308 Revert recent changes to RoomMemberStore 2014-09-12 15:51:51 +01:00
Paul "LeoNerd" Evans
a840ff8f3f Now don't need the other logger.debug() call in _execute 2014-09-12 14:38:27 +01:00
Paul "LeoNerd" Evans
1c20249884 Logging of all SQL queries via the 'synapse.storage.SQL' logger 2014-09-12 14:37:55 +01:00
Paul "LeoNerd" Evans
e53d77b501 Add a .runInteraction() method on SQLBaseStore itself to wrap the .db_pool 2014-09-12 14:28:07 +01:00
Paul "LeoNerd" Evans
249e8f2277 Add a better _store_room_member_txn() method that takes separated fields instead of an event object; also add FIXME comment about a big bug in the logic 2014-09-11 18:52:35 +01:00
Paul "LeoNerd" Evans
aaf9ab68c6 Rename _store_room_member_txn to _store_room_member_from_event_txn so we can create another, more sensible function of that name 2014-09-11 18:44:04 +01:00
Paul "LeoNerd" Evans
55397f6347 prepare_database() on db_conn, not plain name, so we can pass in the connection from outside 2014-09-10 16:23:58 +01:00
Paul "LeoNerd" Evans
ce55a8cc4b Move database preparing code out of homserver.py into storage where it belongs 2014-09-10 15:42:15 +01:00
Erik Johnston
a75f8686ba Fix bug where we used an unbound local variable if we ended up rolling back the persist_event transaction 2014-09-09 16:27:59 +01:00
Erik Johnston
e062f2dfa8 Apparently we can't do txn.rollback(), so raise and catch an exception instead. 2014-09-08 22:37:19 +01:00
Erik Johnston
83ce57302d Fix bug in state handling where we incorrectly identified a missing pdu. Update tests to catch this case. 2014-09-08 19:50:59 +01:00
Erik Johnston
c0577ea87a Rollback if we try and insert duplicate events 2014-09-08 18:34:18 +01:00
Erik Johnston
781ff713ba When getting a state event also include the previous content 2014-09-06 02:23:36 +01:00
Erik Johnston
9dd4570b68 Generate m.room.aliases event when the HS creates a room alias 2014-09-05 21:35:56 +01:00
Paul "LeoNerd" Evans
d703e712f7 Add support to _simple_insert() to do INSERT OR REPLACE 2014-09-03 18:18:41 +01:00
Matthew Hodgson
8a7c1d6a00 fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org hasn't been incorporated in time for launch. 2014-09-03 17:31:57 +01:00
Paul "LeoNerd" Evans
756e171ad0 Store SQL DDL deltas as well; attempt to upgrade the database on startup if it's too old 2014-09-03 14:14:39 +01:00
Paul "LeoNerd" Evans
da3e04df8b Rename the 'do_users_share_a_room' to something slightly less verb-sounding 2014-09-03 10:40:48 +01:00
Mark Haines
c6eafdfbaf Add copyright notices and fix pyflakes errors 2014-09-03 09:43:11 +01:00
Erik Johnston
4b8244fbf8 Fix bug where we didn't correctly store the ops power levels event. 2014-09-02 12:12:14 +01:00
Erik Johnston
f452899fe2 Merge branch 'develop' of github.com:matrix-org/synapse into room_config 2014-09-01 19:57:28 +01:00
Erik Johnston
b8ab9f1c0a Add all the necessary checks to make banning work. 2014-09-01 18:24:56 +01:00
Erik Johnston
db7109c43b Add beginnings of ban support. 2014-09-01 16:15:34 +01:00
Mark Haines
9ea1de432d Fix homeserver config parsing 2014-09-01 15:51:15 +01:00
Erik Johnston
865469f233 Implement power level lists, default power levels and send_evnet_level/add_state_level events. 2014-09-01 14:39:09 +01:00
Mark Haines
3eb45eba0e Merge branch 'develop' into server2server_tls 2014-08-31 16:08:20 +01:00
Erik Johnston
1118f02689 Start adding storage for new events. 2014-08-29 15:18:30 +01:00
Erik Johnston
ee06023573 Get the equalities right. 2014-08-29 13:28:06 +01:00
Mark Haines
d7ae9b90a0 Add store for server certificates and keys 2014-08-28 18:19:47 +01:00
Paul "LeoNerd" Evans
2c7c12bc6e Initial room event stream token must be s0, not s1, or everyone will miss the very first room event 2014-08-28 17:39:34 +01:00
Mark Haines
2aeaa7b77c Merge branch 'develop' into storage_transactions 
Conflicts:
	synapse/handlers/room.py
	synapse/storage/stream.py
 2014-08-27 17:15:58 +01:00
Mark Haines
f54b70520a Return the store_id from persist_event 2014-08-27 17:03:45 +01:00
Mark Haines
b30358f439 add _get_room_member, fix datastore methods 2014-08-27 16:51:54 +01:00
Mark Haines
a03c7f27a8 Fill out prev_events before calling persist_event 2014-08-27 14:32:19 +01:00
Erik Johnston
47519cd8c2 Merge branch 'develop' of github.com:matrix-org/synapse into stream_refactor 
Conflicts:
	synapse/handlers/events.py
	synapse/rest/events.py
	synapse/rest/room.py
 2014-08-27 14:13:06 +01:00
Mark Haines
474dcecb11 Remove unused populate_previous_pdus 2014-08-27 11:34:31 +01:00
Mark Haines
bf05218c4b Merge branch 'develop' into storage_transactions 2014-08-27 11:19:37 +01:00
Kegan Dougal
c65885e166 Added support for GET /events/$eventid with auth checks. 2014-08-27 10:33:01 +01:00
Mark Haines
d2798de660 Fold federation/handler into handlers/federation 2014-08-26 19:49:42 +01:00
Erik Johnston
93cff1668c Merge branch 'develop' of github.com:matrix-org/synapse into stream_refactor 2014-08-26 18:57:55 +01:00
Erik Johnston
3a2a5b959c WIP: Completely change how event streaming and pagination work. This reflects the change in the underlying storage model. 2014-08-26 18:57:46 +01:00
Mark Haines
64e2a5d58e Move pdu and event persistence into a single persist_event function 2014-08-26 18:01:36 +01:00
Kegan Dougal
5a3df1d029 Feedback: Removed FeedbackRestServlet. Modified keys on FeedbackEvent. Expanded the feedback constants to fully explain what type of feedback they are. 2014-08-26 17:49:46 +01:00
Mark Haines
4b63b06cad Merge branch 'develop' into storage_transactions 
Conflicts:
	synapse/api/auth.py
	synapse/handlers/room.py
	synapse/storage/__init__.py
 2014-08-26 16:07:05 +01:00
Mark Haines
ac21dfff6d Fix pyflakes errors 2014-08-26 16:02:59 +01:00
Mark Haines
32347bfcc9 fix a few pyflakes errors 2014-08-26 16:01:29 +01:00
Mark Haines
4b2ad549d5 Move the event storage into a single transaction 2014-08-26 14:36:03 +01:00
Erik Johnston
a664ec20e0 Add a do_users_share_a_room method and use that in the presence handler. 2014-08-26 13:38:26 +01:00
Kegan Dougal
47c3a089c5 Merge branch 'develop' of github.com:matrix-org/synapse into client_server_url_rename 2014-08-26 09:26:33 +01:00
Kegan Dougal
cab3095803 Removed member list servlet: now using generic state paths. 2014-08-26 09:26:07 +01:00
Erik Johnston
be6abdff19 Order 'get_recent_events_for_room' correctly. 2014-08-26 09:22:58 +01:00
Erik Johnston
9d86c8c7a6 Add a unique constraint on the room hosts table 2014-08-24 11:29:29 +01:00
Erik Johnston
537ecd4e99 Turn off spammy logging 2014-08-22 18:12:38 +01:00
Erik Johnston
9f514915af Add indices to schema 2014-08-22 18:03:39 +01:00
Mark Haines
1379dcae6f Take a snapshot of the state of the room before performing updates 2014-08-22 17:00:10 +01:00
Erik Johnston
808f663ed1 Don't return state event outlier's when paginating. 2014-08-22 13:06:07 +01:00
Erik Johnston
e7ee0b9fc1 Change IM sync api to also return the current presence list. 2014-08-21 16:40:21 +01:00
Erik Johnston
063e1b22e6 Stop internal keys from getting into SynapseEvents 2014-08-21 15:06:00 +01:00
Erik Johnston
ebd3c41ede Make event stream storage return all membership events about the user, regardless of if they were in the room or not. 2014-08-20 17:09:44 +01:00
Erik Johnston
e8244c23ba Give the event_id of the failed event 2014-08-20 15:53:07 +01:00
Erik Johnston
d4fb1c8a92 Only hit get_room_events_stream if we have a valid user_id 2014-08-19 17:18:19 +01:00
Erik Johnston
ae493c9418 Fix token to correct format 2014-08-19 16:45:55 +01:00
Erik Johnston
eea2dc7dde Remove debug logging from token parsing funcs. 2014-08-19 16:40:38 +01:00
Erik Johnston
d94765999d Add comment about what strorage.stream does 2014-08-19 16:40:25 +01:00
Erik Johnston
840771190f Fix bug where we sometimes set min_token to None. 2014-08-19 14:32:47 +01:00
Erik Johnston
75b6d982a0 Add a 'backfill room' button 2014-08-19 14:20:03 +01:00
Erik Johnston
598a1d8ff9 Change the way pagination works to support out of order events. 2014-08-19 14:19:48 +01:00
Erik Johnston
1422a22970 Fix typos in SQL and where we still had rowid's (which no longer exist) 2014-08-18 16:25:18 +01:00
Erik Johnston
4eb8f84aa8 Make snapshot_all_rooms return results in the correct form, including start and end tokens. 2014-08-18 16:20:21 +01:00
Erik Johnston
709a92cee8 SQL doesn't allow AUTOINCREMENT on non PRIMARY KEY columns. 2014-08-18 16:00:46 +01:00
Erik Johnston
fc26275bb3 Add two different columns for ordering the events table, one which can be used for pagination and one which can be as tokens for notifying clients. Also add a 'processed' field which is currently always set to True 2014-08-18 15:50:41 +01:00
Erik Johnston
1c2caacd67 Merge branch 'master' of github.com:matrix-org/synapse into sql_refactor 2014-08-18 11:14:57 +01:00
Matthew Hodgson
0f9b633af7 retcols is mandatory. i have no idea how this could ever have worked? 2014-08-16 01:46:35 +01:00
Erik Johnston
0e938b1ff7 Rename method name to not clash with other ones in storage. 2014-08-15 16:47:48 +01:00
Erik Johnston
506711749f We no longer need to special case room config events. 2014-08-15 16:45:16 +01:00
Erik Johnston
d260a42ca2 PEP8 cleanups 2014-08-15 16:17:36 +01:00
Erik Johnston
8fa3cc37f9 Comment. 2014-08-15 16:11:25 +01:00
Erik Johnston
cd2967d271 Fix bug when generating a key when get_room_events_stream returned zero rows 2014-08-15 16:05:46 +01:00
Erik Johnston
86be66c34e Actually use MAX_STREAM_SIZE constant. 2014-08-15 16:04:54 +01:00
Erik Johnston
8d1f763209 Fix pagination to work with new db schema 2014-08-15 15:53:06 +01:00
Paul "LeoNerd" Evans
3c532314ec Fix imsync's SELECT query to only find the rooms I'm actually joined in, not every room I have ever joined 2014-08-15 15:44:53 +01:00
Erik Johnston
01f089d9fb Correctly return new token when returning events. Serialize events correctly. 2014-08-15 15:28:54 +01:00
Erik Johnston
114984a236 Start chagning the events stream to work with the new DB schema 2014-08-15 13:58:28 +01:00
Erik Johnston
d72f897f07 Merge branch 'master' of github.com:matrix-org/synapse into sql_refactor 
Conflicts:
	synapse/storage/stream.py
 2014-08-15 11:50:14 +01:00
Erik Johnston
5002efa31b Reimplement the get public rooms api to work with new DB schema 2014-08-15 10:26:35 +01:00
Erik Johnston
2c46bb6208 Fix up typos and correct sql queries 2014-08-14 18:40:50 +01:00
Erik Johnston
7e681ad778 Update StreamStore 2014-08-14 18:01:39 +01:00
Erik Johnston
661c711765 Start fixing places that use the data store. 2014-08-14 17:34:37 +01:00
Erik Johnston
78b501eba6 Fix typo 2014-08-14 17:09:28 +01:00
Erik Johnston
2529f2bc01 Rename _execute_query 2014-08-14 16:58:51 +01:00
Paul "LeoNerd" Evans
93a8be7bef We really don't need debug logging of all the SQL statements we execute; we're quite happy these all work now 2014-08-14 16:15:57 +01:00
Kegan Dougal
fb93e14e53 Be more helpful when failing to register/login, stating why (communication error, user in user, wrong credentials, etc). Make the HS send M_USER_IN_USE. 2014-08-14 16:03:04 +01:00
Erik Johnston
937c175029 Fix up RoomMemberStore to work with the new schema. 2014-08-14 16:02:10 +01:00
Erik Johnston
6d6a1c3454 Actually encode dicts as json in the DB 2014-08-14 14:30:25 +01:00
Erik Johnston
10294b6082 Merge branch 'master' of github.com:matrix-org/synapse into sql_refactor 
Conflicts:
	synapse/storage/_base.py
 2014-08-14 10:01:04 +01:00
Paul "LeoNerd" Evans
d05aa651f8 An initial hack at storing presence state-change mtimes in database and presenting age durations to clients/federation events 2014-08-13 19:19:15 +01:00
Paul "LeoNerd" Evans
baf04be5cf Set datastore's .hs field in SQLBaseStore rather than in the toplevel DataStore mixed-in result class 2014-08-13 19:19:15 +01:00
Matthew Hodgson
59dfbaba3b when we're talking about backfilling data in federation, call it backfilling - not pagination. 2014-08-13 18:17:09 +01:00
Erik Johnston
beaf4384d9 Make feedback table also store sender. 2014-08-13 18:03:41 +01:00
Erik Johnston
336987bb8d Initial stab at refactoring the SQL tables, including rejigging some of the storage layer. 2014-08-13 18:03:41 +01:00
Erik Johnston
3dfa84bec8 Convert im schema to a 'one' table structure 2014-08-13 18:03:41 +01:00
Kegan Dougal
9b3702c924 Return the room_alias when GETing public rooms. 2014-08-13 11:57:49 +01:00
Matthew Hodgson
f98e6380f1 add in whitespace after copyright statements to improve legibility 2014-08-13 03:14:34 +01:00
matrix.org
4f475c7697 Reference Matrix Home Server 2014-08-12 15:10:52 +01:00