Erik Johnston
21b7375778
Add an index to make membership queries faster
2017-01-31 15:15:57 +00:00
Erik Johnston
a55fa2047f
Insert delta of current_state_events to be more efficient
2017-01-20 17:10:18 +00:00
Erik Johnston
897f8752da
Up cache max entries for state
2017-01-16 15:08:17 +00:00
Erik Johnston
01521299c7
Increase cache size limit
2017-01-16 11:56:51 +00:00
Erik Johnston
2fae34bd2c
Optionally measure size of cache by sum of length of values
2017-01-13 17:46:17 +00:00
Matthew Hodgson
8cfc0165e9
fix annoying typos
2017-01-05 13:39:43 +00:00
Erik Johnston
587d8ac60f
Correctly intern keys in state cache
2016-11-08 11:53:25 +00:00
Erik Johnston
4974147aa3
Remove duplication
2016-09-27 09:27:54 +01:00
Erik Johnston
13122e5e24
Remove unused variable
2016-09-27 09:21:51 +01:00
Erik Johnston
cf3e1cc200
Fix perf of fetching state in SQLite
2016-09-26 17:16:24 +01:00
Erik Johnston
00f51493f5
Fix reindex
2016-09-14 10:18:30 +01:00
Erik Johnston
d5ae1f1291
Ensure we don't mutate state cache entries
2016-09-14 10:03:48 +01:00
Erik Johnston
03a98aff3c
Create new index concurrently
2016-09-12 14:27:01 +01:00
Erik Johnston
897d57bc58
Change state fetch query for postgres to be faster
...
It turns out that postgres doesn't like doing a list of OR's and is
about 1000x slower, so we just issue a query for each specific type
seperately.
2016-09-12 10:05:07 +01:00
Erik Johnston
5beda10bbd
Reindex state_groups_state after pruning
2016-09-08 16:18:01 +01:00
Erik Johnston
b568ca309c
Temporarily disable sequential scans for state fetching
2016-09-08 09:38:54 +01:00
Erik Johnston
513188aa56
Comment
2016-09-07 14:53:23 +01:00
Erik Johnston
fadb01551a
Add appopriate framing clause
2016-09-07 14:39:01 +01:00
Erik Johnston
d25c20ccbe
Use windowing function to make use of index
2016-09-07 14:22:22 +01:00
Erik Johnston
0595413c0f
Scale the batch size so that we're not bitten by the minimum
2016-09-05 15:49:57 +01:00
Erik Johnston
a7032abb2e
Correctly handle reindexing state groups that already have an edge
2016-09-05 15:07:23 +01:00
Erik Johnston
70332a12dd
Take value in a better way
2016-09-05 14:57:14 +01:00
Erik Johnston
373654c635
Comment about sqlite and WITH RECURSIVE
2016-09-05 14:50:36 +01:00
Erik Johnston
628e65721b
Add comments
2016-09-05 10:41:27 +01:00
Erik Johnston
a99e933550
Add upgrade script that will slowly prune state_groups_state entries
2016-09-05 10:05:36 +01:00
Erik Johnston
598317927c
Limit the length of state chains
2016-09-02 10:41:38 +01:00
Erik Johnston
9e25443db8
Move to storing state_groups_state as deltas
2016-09-01 14:31:26 +01:00
Erik Johnston
0cfd6c3161
Use state_groups table to test existence
2016-08-31 16:25:57 +01:00
Erik Johnston
c10cb581c6
Correctly handle the difference between prev and current state
2016-08-31 14:26:22 +01:00
Erik Johnston
1bb8ec296d
Generate state group ids in state layer
2016-08-31 10:09:46 +01:00
Erik Johnston
5dc2a702cf
Make _state_groups_id_gen a normal IdGenerator
2016-08-30 16:55:11 +01:00
Erik Johnston
778fa85f47
Make sync not pull out full state
2016-08-25 18:59:44 +01:00
Erik Johnston
a3dc1e9cbe
Replace context.current_state with context.current_state_ids
2016-08-25 17:32:22 +01:00
Erik Johnston
17f4f14df7
Pull out event ids rather than full events for state
2016-08-25 13:42:44 +01:00
Erik Johnston
ba214a5e32
Remove lru option
2016-08-19 14:17:11 +01:00
Erik Johnston
61c7edfd34
Add cache to _get_state_groups_from_groups
2016-04-19 17:22:03 +01:00
Mark Haines
87f2dec8d4
Make the cache objects be per instance rather than being global
2016-04-06 13:08:05 +01:00
Mark Haines
89e6839a48
Merge pull request #686 from matrix-org/markjh/doc_strings
...
Use google style doc strings.
2016-04-01 16:20:09 +01:00
Mark Haines
2a37467fa1
Use google style doc strings.
...
pycharm supports them so there is no need to use the other format.
Might as well convert the existing strings to reduce the risk of
people accidentally cargo culting the wrong doc string format.
2016-04-01 16:12:07 +01:00
Mark Haines
e36bfbab38
Use a stream id generator for backfilled ids
2016-04-01 13:29:05 +01:00
Mark Haines
31a9eceda5
Add a replication stream for state groups
2016-03-30 16:01:58 +01:00
Mark Haines
1e25f62ee6
Use a stream id generator to assign state group ids
2016-03-30 12:55:02 +01:00
Erik Johnston
2f0180b09e
Don't bother interning keys that are already interned
2016-03-23 16:29:46 +00:00
Erik Johnston
acdfef7b14
Intern all the things
2016-03-23 16:25:54 +00:00
Erik Johnston
75daede92f
String intern
2016-03-23 14:53:53 +00:00
Erik Johnston
8b0dfc9fc4
Don't cache events in get_current_state_for_key
2016-03-23 11:42:17 +00:00
Erik Johnston
2c86187a1b
Don't cache events in _state_group_cache
...
Instead, simply cache the event ids, relying on the event cache to cache
the actual events.
The problem was that while the state groups cache was limited in the
number of groups it could hold, each individual group could consist of
thousands of events.
2016-03-22 12:00:09 +00:00
Mark Haines
54172924c8
Load the current id in the IdGenerator constructor
...
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.
It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
2016-03-01 14:32:56 +00:00
Erik Johnston
5189bfdef4
Batch fetch _get_state_groups_from_groups
2016-02-10 13:24:42 +00:00
Erik Johnston
24f00a6c33
Use _simple_select_many for _get_state_group_for_events
2016-02-10 12:57:50 +00:00
