Richard van der Hoff
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
Richard van der Hoff
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
Richard van der Hoff
8fd1a32456
Fix typos in purge api & doc
...
* It's supposed to be purge_local_events, not ..._history
* Fix the doc to have valid json
2018-02-13 13:09:39 +00:00
Richard van der Hoff
4d09366656
Merge pull request #2695 from okurz/feature/allow_recent_pysaml
...
Allow use of higher versions of saml2
2018-02-13 12:24:08 +00:00
Richard van der Hoff
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
Erik Johnston
32c7b8e48b
Update workers docs to include http port
2018-02-12 17:21:23 +00:00
Erik Johnston
1026690cd2
Merge pull request #2857 from matrix-org/erikj/upload_store
...
Tell storage providers about new file so they can upload
2018-02-12 13:52:58 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
Erik Johnston
5fa571a91b
Tell storage providers about new file so they can upload
2018-02-07 13:35:08 +00:00
Erik Johnston
053255f36c
Merge pull request #2856 from matrix-org/erikj/remove_ratelimit
...
Remove pointless ratelimit check
2018-02-07 11:12:14 +00:00
Erik Johnston
f133228cb3
Add note in docs/workers.rst
2018-02-07 10:34:31 +00:00
Erik Johnston
50fe92cd26
Move presence handling into handle_new_client_event
...
As we want to have it run on the main synapse instance
2018-02-07 10:34:09 +00:00
Erik Johnston
8ec2e638be
Add event_creator worker
2018-02-07 10:32:32 +00:00
Erik Johnston
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
Erik Johnston
e3624fad5f
Remove pointless ratelimit check
...
The intention was for the check to be called as early as possible in the
request, but actually was called just before the main ratelimit check,
so was fairly pointless.
2018-02-07 10:30:25 +00:00
Erik Johnston
617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation
...
Split event creation into a separate handler
2018-02-06 17:01:17 +00:00
Erik Johnston
3e1e69ccaf
Update copyright
2018-02-06 16:40:38 +00:00
Erik Johnston
770b2252ca
s/_create_new_client_event/create_new_client_event/
2018-02-06 16:40:30 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
...
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
Richard van der Hoff
9a304ef2b0
Merge pull request #2848 from matrix-org/rav/refactor_search_insert
...
Factor out common code for search insert
2018-02-05 17:42:09 +01:00
Richard van der Hoff
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
Richard van der Hoff
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
Richard van der Hoff
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
Richard van der Hoff
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
Richard van der Hoff
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
Richard van der Hoff
3c7b480ba3
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
Erik Johnston
25c0a020f4
Updates tests
2018-02-05 16:01:48 +00:00
Erik Johnston
3fa362502c
Update places where we create events
2018-02-05 16:01:48 +00:00
Erik Johnston
5ff3d23564
Split event creation into a separate handler
2018-02-05 16:01:48 +00:00
Richard van der Hoff
c46e75d3d8
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
Richard van der Hoff
db91e72ade
Merge pull request #2844 from matrix-org/rav/evicted_metrics
...
montoring metrics for number of cache evictions
2018-02-05 16:34:36 +01:00
Richard van der Hoff
bc496df192
report metrics on number of cache evictions
2018-02-05 15:34:01 +00:00
Erik Johnston
a1beca0e25
Fix broken unit test for media storage
2018-02-05 12:44:03 +00:00
Erik Johnston
b5049d2e5c
Add .vscode to gitignore
2018-02-05 12:06:46 +00:00
Erik Johnston
1f881e0746
Merge pull request #2791 from matrix-org/erikj/media_storage_refactor
...
Ensure media is in local cache before thumbnailing
2018-02-05 11:28:52 +00:00
Richard van der Hoff
80b8a28100
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
Richard van der Hoff
bd25f9cf36
Clean up work_mem handling
...
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
Richard van der Hoff
4eeae7ad65
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
Richard van der Hoff
bb9f0f3cdb
Merge branch 'develop' into matthew/gin_work_mem
2018-02-03 22:40:28 +00:00
Richard van der Hoff
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
Richard van der Hoff
9c9356512e
Merge pull request #2845 from matrix-org/rav/urlcache_error_handling
...
Handle url_previews with no content-type
2018-02-02 15:27:52 +01:00
Richard van der Hoff
18eae413af
Merge pull request #2842 from matrix-org/rav/state_resolution_handler
...
Factor out resolve_state_groups to a separate handler
2018-02-02 15:27:35 +01:00
