Commit Graph

220 Commits

Author SHA1 Message Date
Will Hunt
0708f437cc Log roomid along with Unknown room (#4297) 2018-12-24 10:49:42 +01:00
Erik Johnston
ad88460e0d Move _find_unreferenced_groups 2018-10-29 14:24:19 +00:00
Erik Johnston
664b192a3b Fix set operations thinko 2018-10-29 14:21:43 +00:00
Erik Johnston
f4f223aa44 Don't make temporary list 2018-10-29 14:01:49 +00:00
Erik Johnston
b2399f6281 Make SQL a bit cleaner 2018-10-29 14:01:11 +00:00
Erik Johnston
03e634dad4 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/purge_state_groups 2018-10-26 16:22:45 +01:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011)
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
2018-10-25 17:49:55 +01:00
Erik Johnston
47a9da28ca Batch process handling state groups 2018-10-19 15:48:15 +01:00
Erik Johnston
67a1e315cc Fix up comments 2018-10-12 13:49:48 +01:00
Erik Johnston
4917ff5523 Add state_group index to event_to_state_groups
This is needed to efficiently check for unreferenced state groups during
purge.
2018-10-04 16:03:08 +01:00
Erik Johnston
17d585753f Delete unreferened state groups during purge 2018-10-04 16:03:06 +01:00
Richard van der Hoff
ae6ad4cf41
docstrings and unittests for storage.state (#3958)
I spent ages trying to figure out how I was going mad...
2018-09-27 11:22:25 +01:00
Matthew Hodgson
bb81e78ec6
Split the state_group_cache in two (#3726)
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
2018-08-22 00:56:37 +02:00
Matthew Hodgson
3f543dc021 initial cut at a room summary API (#3574) 2018-08-16 09:46:50 +01:00
Matthew Hodgson
2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Richard van der Hoff
0ca459ea33 Basic support for room versioning
This is the first tranche of support for room versioning. It includes:
 * setting the default room version in the config file
 * new room_version param on the createRoom API
 * storing the version of newly-created rooms in the m.room.create event
 * fishing the version of existing rooms out of the m.room.create event
2018-08-03 16:08:32 +01:00
Matthew Hodgson
bc7944e6d2 switch missing_types to be a bool 2018-07-25 23:36:31 +01:00
Matthew Hodgson
7d9fb88617 incorporate more review. 2018-07-25 16:33:50 +01:00
Matthew Hodgson
cb5c37a57c handle the edge case for _get_some_state_from_cache where types is [] 2018-07-24 20:34:45 +01:00
Matthew Hodgson
e22700c3dd consider non-filter_type types as wildcards, thus missing from the state-group-cache 2018-07-24 13:59:07 +01:00
Matthew Hodgson
cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Matthew Hodgson
efcdacad7d handle case where types is [] on postgres correctly 2018-07-23 22:41:05 +01:00
Matthew Hodgson
254fb430d1 incorporate review 2018-07-23 19:21:20 +01:00
Matthew Hodgson
2f558300cc fix thinkos; unbreak tests 2018-07-19 19:22:27 +01:00
Matthew Hodgson
bcaec2915a incorporate review 2018-07-19 19:03:50 +01:00
Matthew Hodgson
924eb34d94 add a filtered_types param to limit filtering to specific types 2018-07-19 18:32:02 +01:00
Matthew Hodgson
be3adfc331 merge develop pydoc for _get_state_for_groups 2018-07-19 11:26:04 +01:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
43e02c409d Disable partial state group caching for wildcard lookups
When _get_state_for_groups is given a wildcard filter, just do a complete
lookup. Hopefully this will give us the best of both worlds by not filling up
the ram if we only need one or two keys, but also making the cache still work
for the federation reader usecase.
2018-06-22 11:52:07 +01:00
Richard van der Hoff
240f192523
Merge pull request #3382 from matrix-org/rav/optimise_state_groups
Optimise state_group_cache update
2018-06-22 11:20:20 +01:00
Richard van der Hoff
eb32b2ca20 Optimise state_group_cache update
(1) matrix-org-hotfixes has removed the intern calls; let's do the same here.
(2) remove redundant iteritems() so we can used an optimised db update.
2018-06-11 22:56:11 +01:00
Matthew Hodgson
c96d882a02 Merge branch 'develop' into matthew/filter_members 2018-06-10 12:26:14 +03:00
Richard van der Hoff
d3a8c9c55e Fix sql error in _get_state_groups_from_groups
If this was called with a `(type, None)` entry in types (which is supposed to
return all state of type `type`), it would explode with a sql error.
2018-06-06 14:19:01 +01:00
Erik Johnston
042eedfa2b Add hacky cache factor override system 2018-06-04 15:39:28 +01:00
Matthew Hodgson
28f09fcdd5 Merge branch 'develop' into matthew/filter_members 2018-06-04 00:09:17 +03:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy (#3307) 2018-05-31 19:03:47 +10:00
Matthew Hodgson
a6c8f7c875 add pydoc 2018-05-29 01:09:55 +01:00
Matthew Hodgson
9f77001e27 pep8 2018-03-14 00:07:47 +00:00
Matthew Hodgson
d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990)
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
2018-03-13 22:36:04 +00:00
Matthew Hodgson
ccca02846d make it work 2018-03-13 22:31:41 +00:00
Matthew Hodgson
f0f9a0605b remove comment now #2969 is fixed 2018-03-13 22:12:15 +00:00
Matthew Hodgson
12350e3f9a merge proper fix to bug 2969 2018-03-13 22:11:58 +00:00
Matthew Hodgson
afbf4d3dcc typoe 2018-03-13 19:48:04 +00:00
Matthew Hodgson
865377a70d disable optimisation for searching for state groups
when type filter includes wildcards on state_key
2018-03-13 19:46:04 +00:00
Matthew Hodgson
b2aba9e430 build where_clause sanely 2018-03-13 18:13:44 +00:00
Matthew Hodgson
52f7e23c72 PR feedbackz 2018-03-13 18:07:55 +00:00
Matthew Hodgson
1b1c137771 fix bug #2926 2018-03-13 17:52:52 +00:00
Matthew Hodgson
fdedcd1f4d correctly handle None state_keys
and fix include_other_types thinko
2018-03-12 01:39:06 +00:00
Matthew Hodgson
97c0496cfa fix sqlite where clause 2018-03-12 00:27:06 +00:00
Matthew Hodgson
8713365265 typos 2018-03-11 20:10:25 +00:00