Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
Richard van der Hoff
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
Richard van der Hoff
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
Richard van der Hoff
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
Richard van der Hoff
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
Richard van der Hoff
e1e4ec9f9d
factor _get_new_state_after_events out of _calculate_state_delta
...
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
Richard van der Hoff
b387ee17b6
Improve exception handling in persist_event
...
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.
2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
Richard van der Hoff
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res
...
Log room when doing state resolution
2018-01-17 18:05:04 +00:00
Richard van der Hoff
1224612a79
Log room when doing state resolution
...
Mostly because it helps figure out what is prompting the resolution
2018-01-17 17:11:59 +00:00
Richard van der Hoff
390093d45e
Split resolve_events into two functions
...
... so that the return type doesn't depend on the arg types
2018-01-17 15:44:31 +00:00
Richard van der Hoff
64ddec1bc0
Fix a logcontext leak in persist_events
...
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.
It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.
(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
2018-01-16 11:47:36 +00:00
Richard van der Hoff
6cfee09be9
Make __init__ consitstent across Store heirarchy
...
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
2017-11-13 10:46:07 +00:00
Richard van der Hoff
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Richard van der Hoff
2e9f5ea31a
Fix logcontext handling for persist_events
...
* don't use preserve_context_over_deferred, which is known broken.
* remove a redundant preserve_fn.
* add/improve some comments
2017-10-17 10:59:30 +01:00
Erik Johnston
11d62f43c9
Invalidate cache
2017-10-03 14:12:28 +01:00
Erik Johnston
6ff14ddd2e
Make into list
2017-06-29 15:47:37 +01:00
Erik Johnston
5946aa0877
Prefill forward extrems and event to state groups
2017-06-29 15:38:48 +01:00
Erik Johnston
dfeca6cf40
Merge pull request #2286 from matrix-org/erikj/split_out_user_dir
...
Split out user directory to a separate process
2017-06-16 13:01:19 +01:00
Erik Johnston
6aa5bc8635
Initial worker impl
2017-06-16 11:47:11 +01:00
Erik Johnston
052c5d19d5
Merge pull request #2281 from matrix-org/erikj/phone_home_stats
...
Fix phone home stats
2017-06-15 12:46:23 +01:00
Erik Johnston
4b461a6931
Add some more stats
2017-06-15 09:39:39 +01:00
Erik Johnston
617304b2cf
Fix phone home stats
2017-06-14 19:47:15 +01:00
Erik Johnston
d53fe399eb
Add cache for is_host_joined
2017-06-13 09:56:18 +01:00
Erik Johnston
c62c480dc6
Merge pull request #2259 from matrix-org/erikj/fix_state_woes
...
Fix bug where state_group tables got corrupted
2017-06-07 17:51:25 +01:00
Erik Johnston
197bd126f0
Fix bug where state_group tables got corrupted
...
This is due to the fact that we prefilled caches using txn.call_after,
which always gets called including on error.
We fix this by making txn.call_after only fire when a transaction
completes successfully, which is what we want most of the time anyway.
2017-06-07 17:39:36 +01:00
Erik Johnston
4abcff0177
Fix typo
2017-05-31 16:22:36 +01:00
Erik Johnston
304880d185
Add stream change cache
2017-05-31 15:46:36 +01:00
Erik Johnston
04095f7581
Add clobbered event_id
2017-05-30 14:53:01 +01:00
Erik Johnston
a584a81b3e
Add current_state_delta_stream table
2017-05-30 14:44:09 +01:00
Erik Johnston
c049472b8a
Only store event_auth for state events
2017-05-24 15:23:31 +01:00
Erik Johnston
b8492b6c2f
Merge pull request #2224 from matrix-org/erikj/prefill_state
...
Prefill state caches
2017-05-16 15:50:11 +01:00
Erik Johnston
331570ea6f
Remove spurious merge artifacts
2017-05-16 15:33:07 +01:00
Erik Johnston
871605f4e2
Comments
2017-05-15 15:11:30 +01:00
Erik Johnston
e0d2f6d5b0
Add more granular event send metrics
2017-05-15 15:11:30 +01:00
Erik Johnston
bfbc907cec
Prefill state caches
2017-05-15 15:11:13 +01:00
Richard van der Hoff
34194aaff7
Don't create event_search index on sqlite
...
... because the table is virtual
2017-05-11 12:46:55 +01:00
Richard van der Hoff
114f290947
Add more logging for purging
...
Log the number of events we will be deleting at info.
2017-05-11 12:08:47 +01:00
Richard van der Hoff
baafb85ba4
Add an index to event_search
...
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
Richard van der Hoff
dc026bb16f
Tidy purge code and add some comments
...
Try to make this clearer with more comments and some variable renames
2017-05-11 10:56:12 +01:00
Richard van der Hoff
8e345ce465
Don't de-delta state groups we're about to delete
2017-05-10 18:44:22 +01:00
Richard van der Hoff
b64d312421
add some logging to purge_history
2017-05-10 18:44:22 +01:00
Erik Johnston
093f7e47cc
Expand docstring a bit
2017-05-08 16:14:46 +01:00
Erik Johnston
ffad4fe35b
Don't update event cache hit ratio from get_joined_users
...
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
Erik Johnston
587f07543f
Revert "Prefill state caches"
2017-05-04 15:07:27 +01:00
Erik Johnston
1827057acc
Comments
2017-05-03 09:56:05 +01:00
Erik Johnston
8346e6e696
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state
2017-05-03 09:46:40 +01:00
Erik Johnston
3e5a62ecd8
Add more granular event send metrics
2017-05-02 14:23:26 +01:00
Erik Johnston
a2c89a225c
Prefill state caches
2017-05-02 10:40:31 +01:00
Luke Barnard
6a70647d45
Correct logic in is_event_after
2017-04-13 13:46:17 +01:00
Luke Barnard
b9557064bf
Simplify is_event_after logic
2017-04-12 14:36:20 +01:00
Luke Barnard
7f94709066
travis flake8..
2017-04-11 18:35:45 +01:00
Luke Barnard
73880268ef
Refactor event ordering check to events store
2017-04-11 17:34:09 +01:00
Erik Johnston
24d35ab47b
Add new storage functions for new replication
...
The new replication protocol will keep all the streams separate, rather
than muxing multiple streams into one.
2017-03-30 11:48:35 +01:00
Erik Johnston
bac9bf1b12
Typo
2017-03-27 18:02:17 +01:00
Erik Johnston
d82c42837f
Short circuit if all new events have same state group
2017-03-27 18:00:47 +01:00
Erik Johnston
e71940aa64
Use iter(items|values)
2017-03-24 10:57:02 +00:00
Erik Johnston
00957d1aa4
User Cursor.__iter__ instead of fetchall
...
This prevents unnecessary construction of lists
2017-03-23 17:53:49 +00:00
Erik Johnston
37a187bfab
Merge pull request #2033 from matrix-org/erikj/repl_speed
...
Don't send the full event json over replication
2017-03-21 13:11:15 +00:00
Erik Johnston
61f471f779
Don't send the full event json over replication
2017-03-17 15:50:01 +00:00
Richard van der Hoff
0c01f829ae
Avoid resetting state on rejected events
...
When we get a rejected event, give it the same state_group as its prev_event,
rather than no state_group at all.
This should fix https://github.com/matrix-org/synapse/issues/1935 .
2017-03-17 15:06:08 +00:00
Richard van der Hoff
5068fb16a5
Refactoring and cleanups
...
A few non-functional changes:
* A bunch of docstrings to document types
* Split `EventsStore._persist_events_txn` up a bit. Hopefully it's a bit more
readable.
* Rephrase `EventFederationStore._update_min_depth_for_room_txn` to avoid
mind-bending conditional.
* Rephrase rejected/outlier conditional in `_update_outliers_txn` to avoid
mind-bending conditional.
2017-03-17 15:06:07 +00:00
Erik Johnston
bb256ac96f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/public_list_speed
2017-03-14 11:35:05 +00:00
Erik Johnston
cc7a294e2e
Fix current_state_events table to not lie
...
If we try and persist two state events that have the same ancestor we
calculate the wrong current state when persisting those events.
2017-03-14 10:57:43 +00:00
Erik Johnston
8ffbe43ba1
Get current state by using current_state_events table
2017-03-10 17:39:35 +00:00
Erik Johnston
a41dce8f8a
Remove needless check
2017-02-27 18:54:43 +00:00
Erik Johnston
c0d6045776
It should be all
2017-02-27 18:45:24 +00:00
Erik Johnston
49f4bc4709
Don't fetch current state in common case
...
Currently we fetch the list of current state events whenever we send
something in a room. This is overkill for the common case of persisting
a simple chain of non-state events, so lets handle that case specially.
2017-02-27 18:33:41 +00:00
Erik Johnston
7723b4caa4
Ignore new rejected events when working out forward extremeties.
2017-02-08 14:48:06 +00:00
Erik Johnston
ebfaff84c9
Merge pull request #1870 from matrix-org/erikj/cache_get_all_new_events
...
Add a small cache get_all_new_events
2017-02-01 13:22:02 +00:00
Erik Johnston
368c88c487
Add a small cache get_all_new_events
2017-02-01 10:50:44 +00:00
Erik Johnston
458b6f4733
Only invalidate membership caches based on the cache stream
...
Before we completely invalidated get_users_in_room whenever we updated
any current_state_events table. This was way too aggressive.
2017-01-31 16:09:03 +00:00
Erik Johnston
4b3403ca9b
Stream cache invalidations for room membership storage functions
2017-01-30 17:28:22 +00:00
Erik Johnston
fdf2a31a51
Typo
2017-01-23 16:14:14 +00:00
Erik Johnston
c77b24c092
Refactor to calculate state delta outside transaction
2017-01-23 14:51:33 +00:00
Erik Johnston
5d2134d485
Comments
2017-01-20 17:13:24 +00:00
Erik Johnston
a55fa2047f
Insert delta of current_state_events to be more efficient
2017-01-20 17:10:18 +00:00
Erik Johnston
d0897dead5
Spelling
2017-01-20 15:05:11 +00:00
Erik Johnston
567aa35b67
Update all call sites after rename
2017-01-20 14:40:31 +00:00
Erik Johnston
f2f40e64a9
Comments
2017-01-20 14:38:13 +00:00
Erik Johnston
4c6a31cd6e
Calculate the forward extremeties once
2017-01-20 14:28:53 +00:00
Erik Johnston
09eb08f910
Derive current_state_events from state groups
2017-01-20 11:52:51 +00:00
Erik Johnston
4ec1cf49e2
Lower loading events log to DEBUG
2017-01-17 17:28:32 +00:00
Mark Haines
dd3df11c55
More logging for the linearizer and for get_events
2017-01-05 12:32:47 +00:00
Erik Johnston
f9d156d270
New Flake8 fixes
2016-11-15 11:22:29 +00:00
Erik Johnston
816988baaa
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/remove_auth
2016-10-17 11:10:37 +01:00
Erik Johnston
2869a29fd7
Drop some unused indices
2016-10-17 11:08:19 +01:00
Erik Johnston
9040c9ffa1
Fix background reindex of origin_server_ts
...
The storage function `_get_events_txn` was removed everywhere except
from this background reindex. The function was removed due to it being
(almost) completely unused while also being large and complex.
Therefore, instead of resurrecting `_get_events_txn` we manually
reimplement the bits that are needed directly.
2016-09-27 11:23:49 +01:00
Erik Johnston
0b32bb20bb
Index contains_url for file search queries
2016-09-12 16:57:05 +01:00
Erik Johnston
70332a12dd
Take value in a better way
2016-09-05 14:57:14 +01:00
Erik Johnston
485d999c8a
Correctly delete old state groups in purge history API
2016-09-05 14:49:08 +01:00
Erik Johnston
598317927c
Limit the length of state chains
2016-09-02 10:41:38 +01:00
Erik Johnston
1bb8ec296d
Generate state group ids in state layer
2016-08-31 10:09:46 +01:00
Erik Johnston
5dc2a702cf
Make _state_groups_id_gen a normal IdGenerator
2016-08-30 16:55:11 +01:00
Erik Johnston
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
Erik Johnston
9219139351
Preserve some logcontexts
2016-08-24 11:58:40 +01:00
Erik Johnston
37adde32dc
Move defer.returnValue out of Measure
2016-08-19 18:23:44 +01:00
Erik Johnston
47dd8f02a1
Measure _get_event_from_row
2016-08-19 18:23:44 +01:00
Benjamin Saunders
8a57cc3123
Add missing database corruption recovery case
...
Signed-off-by: Benjamin Saunders <ben.e.saunders@gmail.com>
2016-08-14 11:50:22 -07:00
Erik Johnston
a8b946decb
Raise 404 when couldn't find event
2016-08-05 15:31:02 +01:00
Erik Johnston
f0fa66f495
Delete more tables
2016-08-05 10:40:08 +01:00
Benjamin Saunders
a2b7102eea
Tweak integrity error recovery to work as intended
2016-08-04 20:38:08 -07:00
Erik Johnston
8ad8490cff
Fix typo
2016-08-04 15:21:29 +01:00
Erik Johnston
59fa91fe88
Retry event persistence on IntegrityError
...
Due to a bug in the porting script some backfilled events were not
correctly persisted, causing irrecoverable IntegrityErrors on future
attempts to persist those events.
This commit adds a retry mechanism invoked upon IntegrityError,
where when retried the tables are purged for all references to the
events being persisted.
2016-08-04 15:02:15 +01:00
Erik Johnston
80ad710217
Remove other bit of deduplication
2016-08-03 13:25:59 +01:00
Erik Johnston
a8a32d2714
Ensure we only persist an event once at a time
2016-08-03 11:23:39 +01:00
Mark Haines
a6f06ce3e2
Fix how push_actions are redacted.
2016-07-26 11:05:39 +01:00
Mark Haines
efeb6176c1
Don't add rejected events if we've seen them befrore. Add some comments to explain what the code is doing mechanically
2016-07-26 10:49:52 +01:00
Mark Haines
1b3c3e6d68
Only update the events and event_json tables for rejected events
2016-07-25 18:44:30 +01:00
Mark Haines
8f7f4cb92b
Don't add the events to forward extremities if the event is rejected
2016-07-25 17:13:37 +01:00
Mark Haines
2623cec874
Don't add rejections to the state_group, persist all rejections
2016-07-25 16:12:16 +01:00
Erik Johnston
1e2a7f18a1
Merge pull request #922 from matrix-org/erikj/file_api2
...
Feature: Add filter to /messages. Add 'contains_url' to filter.
2016-07-20 10:40:48 +01:00
Erik Johnston
f52565de50
Fix /purge_history bug
...
This was caused by trying to insert duplicate backward extremeties
2016-07-15 14:23:15 +01:00
Erik Johnston
eec9609e96
event_backwards_extremeties may not be empty
2016-07-15 10:22:09 +01:00
Erik Johnston
b64aa6d687
Add sender and contains_url field to events table
2016-07-14 15:35:43 +01:00
Erik Johnston
b92e7955be
Comment
2016-07-07 11:42:15 +01:00
Erik Johnston
c98e1479bd
Return 400 rather than 500
2016-07-07 11:41:07 +01:00
Erik Johnston
67f2c901ea
Add rest servlet. Fix SQL.
2016-07-06 15:56:59 +01:00
Erik Johnston
2d21d43c34
Add purge_history API
2016-07-05 10:28:51 +01:00
Erik Johnston
a67bf0b074
Add storage function to purge history for a room
2016-07-04 16:02:50 +01:00
David Baker
870c45913e
Use similar naming we use in email notifs for push
...
Fixes https://github.com/vector-im/vector-web/issues/1654
2016-06-24 11:41:11 +01:00
Erik Johnston
e4134c5e13
Merge pull request #841 from matrix-org/erikj/event_counter
...
Add metric counter for number of persisted events
2016-06-06 14:17:40 +01:00
Erik Johnston
7aa778fba9
Add metric counter for number of persisted events
2016-06-06 11:58:09 +01:00
Erik Johnston
70aee0717c
Add events to cache when we persist them
2016-06-06 11:34:53 +01:00
Erik Johnston
cffe46408f
Don't rely on options when inserting event into cache
2016-06-03 18:25:21 +01:00
Erik Johnston
10ea3f46ba
Change the way we cache events
2016-06-03 17:57:50 +01:00
Erik Johnston
05e01f21d7
Remove event fetching from DB threads
2016-06-03 17:22:13 +01:00
Erik Johnston
c8285564a3
Use state to calculate get_users_in_room
2016-06-01 15:25:25 +01:00
David Baker
997db04648
Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs
2016-05-10 14:40:19 +02:00
Erik Johnston
fcb2c3f0db
Remove unused import
2016-05-06 15:47:40 +01:00
Erik Johnston
fd85b167ec
Pull loop one level up
2016-05-06 15:38:42 +01:00
Erik Johnston
b6e0be701e
Queue events for persistence
2016-05-06 14:31:44 +01:00
David Baker
acded821c4
Merge remote-tracking branch 'origin/develop' into dbkr/email_notifs
2016-04-29 10:05:20 +01:00
Mark Haines
871357d539
Check that somethign has happend before running the selects
2016-04-27 11:54:13 +01:00
David Baker
07d765209d
First bits of emailpusher
...
Mostly logic of when to send an email
2016-04-19 14:24:36 +01:00
Erik Johnston
762ada1e07
Add back backfilled parameter that was removed
2016-04-12 11:58:04 +01:00
Erik Johnston
cccf86dd05
Check if we've already backfilled events
2016-04-12 11:19:32 +01:00
David Baker
9c99ab4572
Merge remote-tracking branch 'origin/develop' into dbkr/pushers_use_event_actions
2016-04-07 16:35:22 +01:00
David Baker
0fd1cd2400
pep8
2016-04-06 16:50:47 +01:00
David Baker
7e2c89a37f
Make pushers use the event_push_actions table instead of listening on an event stream & running the rules again. Sytest passes, but remaining to do:
...
* Make badges work again
* Remove old, unused code
2016-04-06 15:42:15 +01:00
Mark Haines
75fb9ac1be
Add a slaved events store class
...
Add a test to check that get_room_names_and_aliases does the same
thing on both the master and on the slave data store.
2016-04-06 14:18:35 +01:00
Erik Johnston
3d76b7cb2b
Store invites in a separate table.
2016-04-04 16:30:15 +01:00
Mark Haines
f2b916534b
Merge pull request #684 from matrix-org/markjh/backfill_id_gen
...
Use a stream id generator for backfilled ids
2016-04-01 15:13:14 +01:00
Mark Haines
e36bfbab38
Use a stream id generator for backfilled ids
2016-04-01 13:29:05 +01:00
Mark Haines
7753fc6570
Fix the invalidation of the names and aliases cache
2016-04-01 10:34:51 +01:00
Mark Haines
5d06929169
Move the check for backfilled outside the for loop
2016-03-31 15:09:09 +01:00
Mark Haines
76503f95ed
Remove the is_new_state argument to persist event.
...
Move the checks for whether an event is new state inside persist
event itself.
This was harder than expected because there wasn't enough information
passed to persist event to correctly handle invites from remote servers
for new rooms.
2016-03-31 15:00:42 +01:00
Mark Haines
62e395f0e3
Merge pull request #676 from matrix-org/markjh/replicate_stateIII
...
Add replication streams for ex outliers and current state resets
2016-03-31 11:20:57 +01:00
Mark Haines
2ec5426035
Use a namedtuple rather than tuple unpacking
2016-03-31 10:33:02 +01:00
Mark Haines
1fbb094c6f
Add replication streams for ex outliers and current state resets
2016-03-30 17:19:56 +01:00
Mark Haines
8b8052909f
return the state_group for backfill
2016-03-30 16:20:07 +01:00
Mark Haines
61407986b4
Add a entry to current_state_resets table when the current state is reset
2016-03-30 16:18:46 +01:00
Mark Haines
31a9eceda5
Add a replication stream for state groups
2016-03-30 16:01:58 +01:00
Mark Haines
1e25f62ee6
Use a stream id generator to assign state group ids
2016-03-30 12:55:02 +01:00
Erik Johnston
e14baa7a3b
Merge pull request #665 from matrix-org/erikj/dont_cache_events
...
Only cache events in the event cache
2016-03-23 13:19:12 +00:00
Erik Johnston
8b0dfc9fc4
Don't cache events in get_current_state_for_key
2016-03-23 11:42:17 +00:00
Erik Johnston
99f929f36b
Make StateHandler._state_cache only store event_ids.
2016-03-22 16:06:04 +00:00
Mark Haines
5244c0b48e
Remove unused backfilled parameter from persist_event
2016-03-21 18:06:08 +00:00
Erik Johnston
67ed8065db
Dedupe requested event list in _get_events
2016-03-18 14:31:31 +00:00
Mark Haines
7076082ae6
Fix relative imports so they work in both py3 and py27
2016-03-08 11:45:50 +00:00
Mark Haines
a612ce6659
Merge pull request #489 from matrix-org/markjh/replication
...
Add a /replication API for extracting the updates that happened on synapse.
2016-03-01 15:08:24 +00:00
Mark Haines
60a0f81c7a
Add a /replication API for extracting the updates that happened on
...
synapse
This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.
This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.
Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.
Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.
The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Mark Haines
54172924c8
Load the current id in the IdGenerator constructor
...
Rather than loading them lazily. This allows us to remove all
the yield statements and spurious arguments for the get_next
methods.
It also allows us to replace all instances of get_next_txn with
get_next since get_next no longer needs to access the db.
2016-03-01 14:32:56 +00:00
Erik Johnston
42109a62a4
Remove unused param from get_max_token
2016-02-18 16:37:28 +00:00
Erik Johnston
0214745239
Rename functions
2016-02-10 11:09:56 +00:00
Erik Johnston
7b0d846407
Atomically persit push actions when we persist the event
2016-02-09 16:19:15 +00:00
Erik Johnston
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
Erik Johnston
00cb3eb24b
Cache tags and account data
2016-01-28 16:37:41 +00:00
Erik Johnston
e1941442d4
Invalidate caches properly. Remove unused arg
2016-01-28 15:02:41 +00:00
Erik Johnston
e7febf4fbb
PEP8
2016-01-27 17:33:27 +00:00
Erik Johnston
aca3193efb
Use the same path for incremental with gap or without gap
2016-01-27 17:33:27 +00:00
Erik Johnston
b97f6626b6
Add cache to room stream
2016-01-27 17:33:26 +00:00
Erik Johnston
87f9477b10
Add a Homeserver.setup method.
...
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
Matthew Hodgson
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
5880de186b
Log when we skip daily messages
2016-01-06 13:52:16 +00:00
Mark Haines
6a5ff5f223
Track the time spent in the database per request.
...
and track the number of transactions that request started.
2015-12-07 17:56:11 +00:00
Erik Johnston
4dcaa42b6d
Allow paginating search ordered by recents
2015-11-30 17:45:31 +00:00
Richard van der Hoff
e4d622aaaf
Implementation of state rollback in /sync
...
Implementation of SPEC-254: roll back the state dictionary to how it looked at
the start of the timeline.
Merged PR https://github.com/matrix-org/synapse/pull/373
2015-11-13 10:58:56 +00:00
Daniel Wagner-Hall
cf437900e0
Return world_readable and guest_can_join in /publicRooms
2015-11-10 17:10:27 +00:00
Daniel Wagner-Hall
ca2f90742d
Open up /events to anonymous users for room events only
...
Squash-merge of PR #345 from daniel/anonymousevents
2015-11-05 14:32:26 +00:00
Erik Johnston
61561b9df7
Keep FTS indexes up to date. Only search through rooms currently joined
2015-10-12 10:49:53 +01:00
Mark Haines
92d8d724c5
Remove unused functions from synapse/storage/events.py
2015-09-23 10:33:06 +01:00
Daniel Wagner-Hall
eb011cd99b
Add docstring
2015-09-22 13:29:36 +01:00
Daniel Wagner-Hall
7213588083
Implement configurable stats reporting
...
SYN-287
This requires that HS owners either opt in or out of stats reporting.
When --generate-config is passed, --report-stats must be specified
If an already-generated config is used, and doesn't have the
report_stats key, it is requested to be set.
2015-09-22 12:57:40 +01:00
Erik Johnston
2048388cfd
Merge pull request #281 from matrix-org/erikj/update_extremeties
...
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:57:25 +01:00
Erik Johnston
3b05b67c89
When updating a stored event from outlier to non-outlier, remember to update the extremeties
2015-09-15 16:34:42 +01:00
Mark Haines
01fc3943f1
Fix indent
2015-08-24 17:18:58 +01:00
Mark Haines
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
Erik Johnston
1f7642efa9
Fix bug where we didn't correctly serialize the redacted_because key over federation
2015-08-21 09:36:07 +01:00
Erik Johnston
20addfa358
Change Cache to not use *args in its interface
2015-08-07 18:32:47 +01:00
Erik Johnston
17bb9a7eb9
Remove commented out code
2015-07-10 14:07:57 +01:00
