Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-29 19:36:14 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,789 Commits 4 Branches 609 Tags 60 MiB
48a910e128
Commit Graph

207 Commits

Author SHA1 Message Date
Erik Johnston
9e25443db8 Move to storing state_groups_state as deltas 2016-09-01 14:31:26 +01:00
Erik Johnston
0cfd6c3161 Use state_groups table to test existence 2016-08-31 16:25:57 +01:00
Erik Johnston
c10cb581c6 Correctly handle the difference between prev and current state 2016-08-31 14:26:22 +01:00
Erik Johnston
1bb8ec296d Generate state group ids in state layer 2016-08-31 10:09:46 +01:00
Erik Johnston
5dc2a702cf Make _state_groups_id_gen a normal IdGenerator 2016-08-30 16:55:11 +01:00
Erik Johnston
778fa85f47 Make sync not pull out full state 2016-08-25 18:59:44 +01:00
Erik Johnston
a3dc1e9cbe Replace context.current_state with context.current_state_ids 2016-08-25 17:32:22 +01:00
Erik Johnston
17f4f14df7 Pull out event ids rather than full events for state 2016-08-25 13:42:44 +01:00
Erik Johnston
ba214a5e32 Remove lru option 2016-08-19 14:17:11 +01:00
Erik Johnston
61c7edfd34 Add cache to _get_state_groups_from_groups 2016-04-19 17:22:03 +01:00
Mark Haines
87f2dec8d4 Make the cache objects be per instance rather than being global 2016-04-06 13:08:05 +01:00
Mark Haines
89e6839a48 Merge pull request #686 from matrix-org/markjh/doc_strings 
Use google style doc strings.
 2016-04-01 16:20:09 +01:00
Mark Haines
2a37467fa1 Use google style doc strings. 
pycharm supports them so there is no need to use the other format.

Might as well convert the existing strings to reduce the risk of
people accidentally cargo culting the wrong doc string format.
 2016-04-01 16:12:07 +01:00
Mark Haines
e36bfbab38 Use a stream id generator for backfilled ids 2016-04-01 13:29:05 +01:00
Mark Haines
31a9eceda5 Add a replication stream for state groups 2016-03-30 16:01:58 +01:00
Mark Haines
1e25f62ee6 Use a stream id generator to assign state group ids 2016-03-30 12:55:02 +01:00
Erik Johnston
2f0180b09e Don't bother interning keys that are already interned 2016-03-23 16:29:46 +00:00
Erik Johnston
acdfef7b14 Intern all the things 2016-03-23 16:25:54 +00:00
Erik Johnston
75daede92f String intern 2016-03-23 14:53:53 +00:00
Erik Johnston
8b0dfc9fc4 Don't cache events in get_current_state_for_key 2016-03-23 11:42:17 +00:00
Erik Johnston
2c86187a1b Don't cache events in _state_group_cache 
Instead, simply cache the event ids, relying on the event cache to cache
the actual events.

The problem was that while the state groups cache was limited in the
number of groups it could hold, each individual group could consist of
thousands of events.
 2016-03-22 12:00:09 +00:00
Mark Haines
54172924c8 Load the current id in the IdGenerator constructor 
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.

It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
 2016-03-01 14:32:56 +00:00
Erik Johnston
5189bfdef4 Batch fetch _get_state_groups_from_groups 2016-02-10 13:24:42 +00:00
Erik Johnston
24f00a6c33 Use _simple_select_many for _get_state_group_for_events 2016-02-10 12:57:50 +00:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Richard van der Hoff
fddedd51d9 Fix a few race conditions in the state calculation 
Be a bit more careful about how we calculate the state to be returned by
/sync. In a few places, it was possible for /sync to return slightly later
state than that represented by the next_batch token and the timeline. In
particular, the following cases were susceptible:

* On a full state sync, for an active room
* During a per-room incremental sync with a timeline gap
* When the user has just joined a room. (Refactor check_joined_room to make it
  less magical)

Also, use store.get_state_for_events() (and thus the existing stategroups) to
calculate the state corresponding to a particular sync position, rather than
state_handler.compute_event_context(), which recalculates from first principles
(and tends to miss some state).

Merged from PR https://github.com/matrix-org/synapse/pull/372
 2015-11-13 10:39:09 +00:00
Erik Johnston
927004e349 Remove unused room_id parameter 2015-10-12 15:06:14 +01:00
Mark Haines
1cd65a8d1e synapse/storage/state.py: _make_group_id was unused 2015-09-23 10:37:58 +01:00
Erik Johnston
1bd1a43073 Actually check if event_id isn't returned by _get_state_groups 2015-08-21 14:30:34 +01:00
Erik Johnston
a82938416d Remove newline because vertical whitespace makes mjark sad 2015-08-18 16:28:13 +01:00
Erik Johnston
0bfdaf1f4f Rejig the code to make it nicer 2015-08-18 16:26:07 +01:00
Erik Johnston
8199475ce0 Ensure we never return a None event from _get_state_for_groups 2015-08-18 11:44:10 +01:00
Erik Johnston
85d0bc3bdc Reduce cache size from obscenely large to quite large 2015-08-18 11:00:38 +01:00
Erik Johnston
f9d4da7f45 Fix bug where we were leaking None into state event lists 2015-08-17 09:39:45 +01:00
Erik Johnston
2bb2c02571 Remove some vertical space 2015-08-13 17:11:30 +01:00
Erik Johnston
57877b01d7 Replace list comprehension 2015-08-13 17:00:17 +01:00
Erik Johnston
0fbed2a8fa Comment 2015-08-12 17:22:54 +01:00
Erik Johnston
c10ac7806e Explain why we're prefilling dict with Nones 2015-08-12 17:16:30 +01:00
Erik Johnston
101ee3fd00 Better variable name 2015-08-12 17:08:05 +01:00
Erik Johnston
df361d08f7 Split _get_state_for_group_from_cache into two 2015-08-12 17:06:21 +01:00
Erik Johnston
2df8dd9b37 Move all the caches into their own package, synapse.util.caches 2015-08-11 18:00:59 +01:00
Erik Johnston
53a817518b Comments 2015-08-11 11:40:40 +01:00
Erik Johnston
1b994a97dd Fix application of ACLs 2015-08-11 10:41:40 +01:00
Erik Johnston
10b874067b Fix state cache 2015-08-11 09:12:41 +01:00
Erik Johnston
017b798e4f Clean up StateStore 2015-08-10 15:01:06 +01:00
Erik Johnston
2c019eea11 Remove unused function 2015-08-10 14:44:41 +01:00
Erik Johnston
aa88582e00 Do bounds check 2015-08-10 10:08:15 +01:00
Erik Johnston
5119e416e8 Line length 2015-08-10 10:05:30 +01:00
Erik Johnston
8f04b6fa7a Merge branch 'erikj/cache_varargs_interface' of github.com:matrix-org/synapse into erikj/dictionary_cache 2015-08-07 19:30:25 +01:00
Erik Johnston
3213ff630c Remove unnecessary cache 2015-08-07 19:14:05 +01:00
Erik Johnston
9eb5b23d3a Batch up various DB requests for event -> state 2015-08-07 18:16:02 +01:00
Erik Johnston
b3768ec10a Remove unncessary cache 2015-08-07 13:41:05 +01:00
Erik Johnston
fe994e728f Store absence of state in cache 2015-08-07 10:17:38 +01:00
Erik Johnston
3e1b77efc2 Merge branch 'erikj/cached_keyword_args' of github.com:matrix-org/synapse into erikj/dictionary_cache 2015-08-05 16:45:56 +01:00
Erik Johnston
07507643cb Use dictionary cache to do group -> state fetching 2015-08-05 15:11:42 +01:00
Erik Johnston
413a4c289b Add comment 2015-08-04 11:08:07 +01:00
Erik Johnston
4d6cb8814e Speed up event filtering (for ACL) logic 2015-08-04 09:32:23 +01:00
Erik Johnston
39e21ea51c Add support for using keyword arguments with cached functions 2015-07-27 13:57:29 +01:00
Erik Johnston
baa55fb69e Merge pull request #193 from matrix-org/erikj/bulk_persist_event 
Add bulk insert events API
 2015-07-14 10:49:24 +01:00
Erik Johnston
1a60545626 Add basic impl for room history ACL on GET /messages client API 2015-07-02 16:20:10 +01:00
Erik Johnston
5130d80d79 Add bulk insert events API 2015-06-25 17:29:34 +01:00
Erik Johnston
0d7d9c37b6 Add cache to get_state_groups 2015-06-03 14:45:55 +01:00
Erik Johnston
4429e720ae Merge branch 'master' of github.com:matrix-org/synapse into develop 2015-05-22 10:33:00 +01:00
Mark Haines
d61ce3f670 Add a cache for get_current_state with state_key 2015-05-21 16:41:39 +01:00
Erik Johnston
8763dd80ef Don't fetch prev_content for current_state 2015-05-15 15:33:01 +01:00
Erik Johnston
acb12cc811 Make store.get_current_state fetch events asyncly 2015-05-15 15:20:05 +01:00
Erik Johnston
f6f902d459 Move fetching of events into their own transactions 2015-05-14 13:45:48 +01:00
Erik Johnston
cdb3757942 Refactor _get_events 2015-05-14 13:31:55 +01:00
Erik Johnston
4071f29653 Fetch events from events_id in their own transactions 2015-05-13 16:59:41 +01:00
Erik Johnston
ca4f458787 Fetch events in bulk 2015-05-13 15:13:42 +01:00
Erik Johnston
6edff11a88 Don't fetch redaction and rejection stuff for each event, so we can use index only scan 2015-05-13 14:39:05 +01:00
Erik Johnston
02590c3e1d Temp turn off checking for rejections and redactions 2015-05-13 11:31:28 +01:00
Erik Johnston
619a21812b defer.gatherResults loop 2015-05-13 11:29:03 +01:00
Erik Johnston
fec4485e28 Batch fetching of events for state groups 2015-05-13 11:22:42 +01:00
Erik Johnston
409bcc76bd Load events for state group seperately 2015-05-13 11:13:31 +01:00
Erik Johnston
43c2e8deae Add support for using executemany 2015-05-05 15:13:25 +01:00
Erik Johnston
8be5284e91 Remove pointless join 2015-04-30 18:49:26 +01:00
Erik Johnston
00718ae7a9 Need more yield 2015-04-30 18:43:39 +01:00
Erik Johnston
0465560c1a Add missing param 2015-04-30 18:42:44 +01:00
Erik Johnston
f383d5a801 Fix up get_current_state and get_room_name_and_aliases queries to parse events in transaction 2015-04-30 17:12:52 +01:00
Erik Johnston
3c4c229788 Don't use sub queries, it makes postgres sad 2015-04-30 16:16:53 +01:00
Erik Johnston
2366d28780 Don't needlessly join on state_events 2015-04-30 14:02:06 +01:00
Erik Johnston
86fc9b617c For backwards compat, make state_groups.id have a type of int, not varchar 2015-04-13 17:03:49 +01:00
Erik Johnston
0bfa78b39b PEP8 2015-04-07 12:16:05 +01:00
Erik Johnston
304111afd0 Don't use AUTOINCREMENT, use an in memory version 2015-04-07 12:05:36 +01:00
Erik Johnston
6e7131f02f Remove uses of REPLACE and ON CONFLICT IGNORE to make the SQL more portable. 2015-03-23 15:38:56 +00:00
Erik Johnston
f6583796fe Merge branch 'develop' of github.com:matrix-org/synapse into mysql 2015-03-20 16:31:48 +00:00
Erik Johnston
87db64b839 Rearrange storage modules 2015-03-20 14:11:38 +00:00
Erik Johnston
cb8162d3d1 Rearrange storage modules 2015-03-20 13:52:56 +00:00
Erik Johnston
d7a0496f3e Convert storage layer to be mysql compatible 2015-03-19 15:59:48 +00:00
Erik Johnston
f1d2b94e0b Copy dict of context.current_state before changing it. 2015-03-16 15:13:05 +00:00
Mark Haines
3891597eb3 Remove unused functions 2015-01-13 15:57:26 +00:00
Mark Haines
5e23a19204 Merge pull request #28 from matrix-org/erikj-perf 
Database performance improvements.
 2015-01-06 13:33:40 +00:00
Mark Haines
adb04b1e57 Update copyright notices 2015-01-06 13:21:39 +00:00
Erik Johnston
12819d5082 Remove debug lines 2015-01-06 13:12:30 +00:00
Erik Johnston
753126b8cc Add some debug logging 2015-01-06 11:18:12 +00:00
Erik Johnston
f3788e3c78 Test some ideas that might help performance a bit 2014-12-17 23:37:08 +00:00
Erik Johnston
d044121168 Various typos and bug fixes. 2014-12-08 09:08:26 +00:00
Erik Johnston
6630e1b579 Start making more things use EventContext rather than event.* 2014-12-05 16:20:48 +00:00
Erik Johnston
3598c11c8d Correctly handle the case where we get an event for an unknown room, which turns out we are actually in 2014-11-26 10:41:08 +00:00
Erik Johnston
e24d5cb97d Document StateStore and use transactions 2014-11-12 14:33:48 +00:00
Erik Johnston
6fea478d2e Fix bugs with invites/joins across federatiom. 
Both in terms of auth and not trying to fetch missing PDUs for invites,
joins etc.
 2014-11-12 11:24:11 +00:00
Erik Johnston
5ff0bfb81d Fix bug where we /always/ created a new state group 2014-11-11 14:16:41 +00:00
Erik Johnston
da1dda3e1d Add transaction level logging and timing information. Add a _simple_delete method 2014-10-28 11:18:04 +00:00
Erik Johnston
5ffe5ab43f Use state groups to get current state. Make join dance actually work. 2014-10-17 18:56:42 +01:00
Erik Johnston
e7bc1291a0 Begin making auth use event.old_state_events 2014-10-15 16:06:59 +01:00
Erik Johnston
80472ac198 Add missing package storate.state 2014-10-15 10:04:55 +01:00