Commit Graph

262 Commits

Author SHA1 Message Date
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496 purge: move room_depth update to end
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b delete_local_events for purge_history
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d purge: Move cache invalidation to more appropriate place
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137 bump purge logging to info
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf rename delete_old_state -> purge_history
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events (#2784)
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff
ee6fb4cf85 Remove redundant return value from _calculate_state_delta
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
Richard van der Hoff
e1e4ec9f9d factor _get_new_state_after_events out of _calculate_state_delta
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
Richard van der Hoff
b387ee17b6 Improve exception handling in persist_event
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.

2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
Richard van der Hoff
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res
Log room when doing state resolution
2018-01-17 18:05:04 +00:00
Richard van der Hoff
1224612a79 Log room when doing state resolution
Mostly because it helps figure out what is prompting the resolution
2018-01-17 17:11:59 +00:00
Richard van der Hoff
390093d45e Split resolve_events into two functions
... so that the return type doesn't depend on the arg types
2018-01-17 15:44:31 +00:00
Richard van der Hoff
64ddec1bc0 Fix a logcontext leak in persist_events
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.

It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.

(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
2018-01-16 11:47:36 +00:00
Richard van der Hoff
6cfee09be9 Make __init__ consitstent across Store heirarchy
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Richard van der Hoff
eaaabc6c4f replace 'except:' with 'except Exception:'
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff
2e9f5ea31a Fix logcontext handling for persist_events
* don't use preserve_context_over_deferred, which is known broken.

* remove a redundant preserve_fn.

* add/improve some comments
2017-10-17 10:59:30 +01:00
Erik Johnston
11d62f43c9 Invalidate cache 2017-10-03 14:12:28 +01:00
Erik Johnston
6ff14ddd2e Make into list 2017-06-29 15:47:37 +01:00
Erik Johnston
5946aa0877 Prefill forward extrems and event to state groups 2017-06-29 15:38:48 +01:00
Erik Johnston
dfeca6cf40 Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
Erik Johnston
6aa5bc8635 Initial worker impl 2017-06-16 11:47:11 +01:00
Erik Johnston
052c5d19d5 Merge pull request #2281 from matrix-org/erikj/phone_home_stats
Fix phone home stats
2017-06-15 12:46:23 +01:00
Erik Johnston
4b461a6931 Add some more stats 2017-06-15 09:39:39 +01:00
Erik Johnston
617304b2cf Fix phone home stats 2017-06-14 19:47:15 +01:00
Erik Johnston
d53fe399eb Add cache for is_host_joined 2017-06-13 09:56:18 +01:00
Erik Johnston
c62c480dc6 Merge pull request #2259 from matrix-org/erikj/fix_state_woes
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
Erik Johnston
197bd126f0 Fix bug where state_group tables got corrupted
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.

We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
Erik Johnston
4abcff0177 Fix typo 2017-05-31 16:22:36 +01:00
Erik Johnston
304880d185 Add stream change cache 2017-05-31 15:46:36 +01:00
Erik Johnston
04095f7581 Add clobbered event_id 2017-05-30 14:53:01 +01:00
Erik Johnston
a584a81b3e Add current_state_delta_stream table 2017-05-30 14:44:09 +01:00
Erik Johnston
c049472b8a Only store event_auth for state events 2017-05-24 15:23:31 +01:00
Erik Johnston
b8492b6c2f Merge pull request #2224 from matrix-org/erikj/prefill_state
Prefill state caches
2017-05-16 15:50:11 +01:00
Erik Johnston
331570ea6f Remove spurious merge artifacts 2017-05-16 15:33:07 +01:00
Erik Johnston
871605f4e2 Comments 2017-05-15 15:11:30 +01:00
Erik Johnston
e0d2f6d5b0 Add more granular event send metrics 2017-05-15 15:11:30 +01:00
Erik Johnston
bfbc907cec Prefill state caches 2017-05-15 15:11:13 +01:00
Richard van der Hoff
34194aaff7 Don't create event_search index on sqlite
... because the table is virtual
2017-05-11 12:46:55 +01:00
Richard van der Hoff
114f290947 Add more logging for purging
Log the number of events we will be deleting at info.
2017-05-11 12:08:47 +01:00
Richard van der Hoff
baafb85ba4 Add an index to event_search
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
Richard van der Hoff
dc026bb16f Tidy purge code and add some comments
Try to make this clearer with more comments and some variable renames
2017-05-11 10:56:12 +01:00
Richard van der Hoff
8e345ce465 Don't de-delta state groups we're about to delete 2017-05-10 18:44:22 +01:00
Richard van der Hoff
b64d312421 add some logging to purge_history 2017-05-10 18:44:22 +01:00
Erik Johnston
093f7e47cc Expand docstring a bit 2017-05-08 16:14:46 +01:00
Erik Johnston
ffad4fe35b Don't update event cache hit ratio from get_joined_users
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
Erik Johnston
587f07543f Revert "Prefill state caches" 2017-05-04 15:07:27 +01:00
Erik Johnston
1827057acc Comments 2017-05-03 09:56:05 +01:00
Erik Johnston
8346e6e696 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state 2017-05-03 09:46:40 +01:00
Erik Johnston
3e5a62ecd8 Add more granular event send metrics 2017-05-02 14:23:26 +01:00