Matrix-Mirrors/anonymousland-synapse
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse.git synced 2024-12-27 03:59:25 -05:00
Code Issues Packages Projects Releases Wiki Activity
10,143 Commits 4 Branches 615 Tags 63 MiB
6411f725be
Commit Graph

2339 Commits

Author SHA1 Message Date
Richard van der Hoff
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin 
switch back from GIST to GIN indexes
 2018-02-14 16:59:03 +00:00
Richard van der Hoff
39bf47319f purge_history: fix sqlite syntax error 
apparently sqlite insists on indexes being named
 2018-02-14 16:42:19 +00:00
Richard van der Hoff
ac27f6a35e purge_history: handle sqlite asshattery 
apparently creating a temporary table commits the transaction. because that's a
useful thing.
 2018-02-14 16:41:12 +00:00
Richard van der Hoff
5978dccff0 remove overzealous exception handling 2018-02-14 15:54:09 +00:00
Richard van der Hoff
278d21b5e4 purge_history: fix index use 
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
 2018-02-14 15:44:51 +00:00
Richard van der Hoff
5fcbf1e07c Rework event purge to use a temporary table 
... which should speed things up by reducing the amount of data being shuffled
across the connection
 2018-02-14 11:02:22 +00:00
Erik Johnston
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker 
Create a worker for event creation
 2018-02-13 18:07:10 +00:00
Richard van der Hoff
d627174da2 Fix log message in purge_history 
(we don't just remove remote events)
 2018-02-13 16:51:21 +00:00
Richard van der Hoff
ddb6a79b68 Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin 2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3 move search reindex to schema 47 
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.

This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
 2018-02-13 16:44:46 +00:00
Richard van der Hoff
4a6d551704 GIN reindex: Fix syntax errors, improve exception handling 2018-02-13 16:44:46 +00:00
Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching 
Use StateResolutionHandler to resolve state in persist_events
 2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit 2018-02-13 14:29:22 +00:00
Richard van der Hoff
a9b712e9dc Merge branch 'develop' into matthew/gin_work_mem 2018-02-13 12:16:01 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates 
delete_local_events for purge_room_history
 2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496 purge: move room_depth update to end 
... to avoid locking the table for too long
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b delete_local_events for purge_history 
Add a flag which makes the purger delete local events
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d purge: Move cache invalidation to more appropriate place 
it was a bit of a non-sequitur there
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137 bump purge logging to info 
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf rename delete_old_state -> purge_history 
(beacause it deletes more than state)
 2018-02-09 13:07:41 +00:00
Erik Johnston
24dd73028a Add replication http endpoint for event sending 2018-02-07 10:32:32 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events (#2784) 
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
 2018-02-06 14:31:24 +00:00
Richard van der Hoff
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta 
Remove redundant return value from _calculate_state_delta
 2018-02-05 17:42:20 +01:00
Richard van der Hoff
ebfe64e3d6 Use StateResolutionHandler to resolve state in persist events 
... and thus benefit (hopefully) from its cache.
 2018-02-05 16:23:26 +00:00
Richard van der Hoff
225dc3b4cb Flatten _get_new_state_after_events 
rejig the if statements to simplify the logic and reduce indentation
 2018-02-05 16:23:25 +00:00
Richard van der Hoff
9fcbbe8e7d Check that events being persisted have state_group 2018-02-05 16:23:25 +00:00
Richard van der Hoff
447aed42d2 Add event_map param to resolve_state_groups 2018-02-05 16:23:25 +00:00
Richard van der Hoff
ee6fb4cf85 Remove redundant return value from _calculate_state_delta 
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
 2018-02-05 16:23:20 +00:00
Richard van der Hoff
3c7b480ba3 Factor out common code for search insert 
we can reuse the same code as is used for event insert, for doing the
background index population.
 2018-02-05 16:12:14 +00:00
Richard van der Hoff
c46e75d3d8 Move store_event_search_txn to SearchStore 
... as a precursor to making event storing and doing the bg update share some
code.
 2018-02-05 15:43:22 +00:00
Richard van der Hoff
80b8a28100 Factor out common code for search insert 
we can reuse the same code as is used for event insert, for doing the
background index population.
 2018-02-04 00:23:06 +00:00
Richard van der Hoff
bd25f9cf36 Clean up work_mem handling 
Add some comments and improve exception handling when twiddling work_mem for
the search update
 2018-02-03 23:05:41 +00:00
Richard van der Hoff
4eeae7ad65 Move store_event_search_txn to SearchStore 
... as a precursor to making event storing and doing the bg update share some
code.
 2018-02-03 22:59:45 +00:00
Richard van der Hoff
bb9f0f3cdb Merge branch 'develop' into matthew/gin_work_mem 2018-02-03 22:40:28 +00:00
Richard van der Hoff
6b02fc80d1 Reinstate event_search_postgres_gist handler 
People may have queued updates for this, so we can't just delete it.
 2018-02-02 14:32:51 +00:00
Richard van der Hoff
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta 
factor _get_new_state_after_events out of _calculate_state_delta
 2018-02-02 15:27:15 +01:00
Travis Ralston
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media 2018-02-01 18:05:47 -07:00
Richard van der Hoff
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media 
Fix sql error in quarantine_media
 2018-02-01 11:45:58 +01:00
Travis Ralston
e16e45b1b4 pep8 
Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-31 15:30:38 -07:00
Richard van der Hoff
e1e4ec9f9d factor _get_new_state_after_events out of _calculate_state_delta 
This reduces the scope of a bunch of variables
 2018-01-31 21:32:09 +00:00
Travis Ralston
63c4383927 Documentation and naming 
Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-31 08:07:52 -07:00
Richard van der Hoff
773f0eed1e Fix sql error in quarantine_media 2018-01-30 15:02:51 +00:00
Richard van der Hoff
b387ee17b6 Improve exception handling in persist_event 
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.

2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
 2018-01-29 17:05:33 +00:00
Richard van der Hoff
46022025ea Fix SQL for user search 
fix some syntax errors for user search when search_all_users is enabled

fixes #2801, hopefully
 2018-01-27 17:21:57 +00:00
Travis Ralston
a94d9b6b82 Appease the linter 
These are ids anyways, not mxc uris.

Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-20 22:49:46 -07:00
Travis Ralston
5552ed9a7f Add an admin route to get all the media in a room 
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.

Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-20 22:37:53 -07:00
Richard van der Hoff
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res 
Log room when doing state resolution
 2018-01-17 18:05:04 +00:00
Richard van der Hoff
1224612a79 Log room when doing state resolution 
Mostly because it helps figure out what is prompting the resolution
 2018-01-17 17:11:59 +00:00
Erik Johnston
a87006f9c7
Merge pull request #2783 from matrix-org/erikj/media_last_accessed 
Keep track of last access time for local media
 2018-01-17 16:39:02 +00:00
Matthew Hodgson
06db5c4b76
Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql 
fix SQL when searching all users
 2018-01-17 16:27:54 +00:00
Matthew Hodgson
2d9ab533f9 fix SQL when searching all users 2018-01-17 15:58:52 +00:00
Richard van der Hoff
390093d45e Split resolve_events into two functions 
... so that the return type doesn't depend on the arg types
 2018-01-17 15:44:31 +00:00
Erik Johnston
2fb3a28c98
Remove lost comment 2018-01-17 14:59:44 +00:00
Erik Johnston
05f98a2224 Keep track of last access time for local media 2018-01-17 10:24:43 +00:00
Richard van der Hoff
3d12d97415 Track DB scheduling delay per-request 
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
 2018-01-16 17:23:32 +00:00
Richard van der Hoff
8615f19d20 rework runInteraction in terms of runConnection 
... so that we can share the code
 2018-01-16 17:08:29 +00:00
Richard van der Hoff
64ddec1bc0 Fix a logcontext leak in persist_events 
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.

It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.

(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
 2018-01-16 11:47:36 +00:00
hera
174eacc8ba oops 2018-01-09 18:14:32 +00:00
Matthew Hodgson
a66f489678 fix GIST->GIN switch 2018-01-09 16:55:51 +00:00
Matthew Hodgson
e79db0a673 switch back from GIST to GIN indexes 2018-01-09 16:37:48 +00:00
Matthew Hodgson
e365ad329f oops, tweak work_mem when actually storing 2018-01-09 16:30:30 +00:00
Matthew Hodgson
19f9227643 avoid 80s GIN inserts by tweaking work_mem 
see https://github.com/matrix-org/synapse/issues/2753 for details
 2018-01-09 16:25:04 +00:00
Erik Johnston
b30cd5b107 Remove dead code related to default thumbnails 2018-01-09 14:38:33 +00:00
Matthew Hodgson
cdc2cb5d11 fix StoreError syntax 2017-12-05 11:09:31 +00:00
Matthew Hodgson
74e0cc74ce fix pep8 and tests 2017-12-04 15:11:38 +00:00
Matthew Hodgson
1bd40ca73e switch to a simpler 'search_all_users' button as per review feedback 2017-12-04 14:58:39 +00:00
Matthew Hodgson
f397153dfc Merge branch 'develop' into matthew/search-all-local-users 2017-11-30 01:51:38 +00:00
Matthew Hodgson
f61e107f63 remove null constraint on user_dir.room_id 2017-11-30 01:43:50 +00:00
Matthew Hodgson
4b1fceb913 fix alternation operator for FTS4 - how did this ever work!? 2017-11-30 01:34:03 +00:00
Matthew Hodgson
a4bb133b68 fix thinkos galore 2017-11-30 01:17:15 +00:00
Matthew Hodgson
3241c7aac3 untested WIP but might actually work 2017-11-29 18:27:05 +00:00
Matthew Hodgson
47d99a20d5 Add user_directory_include_pattern config param to expand search results to additional users 
Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive.
It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users...
...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too.

Committing it here as a WIP
 2017-11-29 16:46:45 +00:00
Richard van der Hoff
2c6d63922a Remove pushers when deleting access tokens 
Whenever an access token is invalidated, we should remove the associated
pushers.
 2017-11-29 16:44:35 +00:00
Richard van der Hoff
5a4da5bf78
Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error 
Fix error on sqlite 3.7
 2017-11-27 12:25:48 +00:00
Richard van der Hoff
63ccaa5873 Avoid retrying forever on IntegrityError 2017-11-27 12:00:07 +00:00
Richard van der Hoff
6b48b3e277 fix sql fails 2017-11-22 18:06:24 +00:00
Richard van der Hoff
2908f955d1 Check database in has_completed_background_updates 
so that the right thing happens on workers.
 2017-11-22 18:02:15 +00:00
Richard van der Hoff
0b4aa2dc21
Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert 
Avoid locking account_data tables for upserts
 2017-11-21 13:39:14 +00:00
Richard van der Hoff
7098b65cb8 Fix error on sqlite 3.7 
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.

To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.

Fixes https://github.com/matrix-org/synapse/issues/2572.
 2017-11-21 11:14:17 +00:00
Richard van der Hoff
c46139a17e Avoid locking account_data tables for upserts 2017-11-16 18:08:01 +00:00
Richard van der Hoff
06e5bcfc83 Avoid locking for upsert on pushers tables 
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
 2017-11-16 17:52:23 +00:00
Richard van der Hoff
77a1227870 Fix broken ref to IntegrityError 2017-11-16 16:03:38 +00:00
Richard van der Hoff
7ab2b69e18 Avoid locking pushers table on upsert 
Now that _simple_upsert will retry on IntegrityError, we don't need to lock the
table.
 2017-11-16 15:32:01 +00:00
Richard van der Hoff
10aaa1bc15 _simple_upsert: retry on IntegrityError 
wrap the call to _simple_upsert_txn in a loop so that we retry on an
integrityerror: this means we can avoid locking the table provided there is an
unique index.
 2017-11-16 15:30:15 +00:00
Richard van der Hoff
cdc9e50a5d Cleanup in _simple_upsert_txn 
Bail out early to reduce indentation
 2017-11-16 15:29:10 +00:00
Richard van der Hoff
f959c01600
Merge pull request #2661 from matrix-org/rav/statereadstore 
Pull out bits of StateStore to a mixin
 2017-11-15 17:23:01 +00:00
Richard van der Hoff
1b83c09c03
Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs 
Remove preserve_context_over_{fn, deferred}
 2017-11-15 11:13:53 +00:00
Richard van der Hoff
35a4b63240 Pull out bits of StateStore to a mixin 
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.

fixes https://github.com/matrix-org/synapse/issues/2655.
 2017-11-14 11:43:58 +00:00
Richard van der Hoff
4dd1bfa8c1 Revert "Revert "move _state_group_cache to statestore"" 
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.

This reverts commit ab335edb02.
 2017-11-14 11:43:58 +00:00
Richard van der Hoff
7e6fa29cb5 Remove preserve_context_over_{fn, deferred} 
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
 2017-11-14 11:22:42 +00:00
Richard van der Hoff
6cfee09be9 Make __init__ consitstent across Store heirarchy 
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
 2017-11-13 10:46:07 +00:00
Erik Johnston
ab335edb02 Revert "move _state_group_cache to statestore" 
This reverts commit f5cf3638e9.
 2017-11-13 10:05:33 +00:00
Erik Johnston
bfbf1e1f1a Up cache size of get_global_account_data_by_type_for_user 2017-11-13 09:52:11 +00:00
Erik Johnston
2dce6b15c3 Fix typo 2017-11-09 15:56:16 +00:00
Richard van der Hoff
618c7b816a
Merge pull request #2656 from matrix-org/rav/fix_deactivate 
Fix 'NoneType' not iterable in /deactivate
 2017-11-09 15:20:35 +00:00
Richard van der Hoff
889102315e Fix 'NoneType' not iterable in /deactivate 
make sure we actually return a value from user_delete_access_tokens
 2017-11-09 15:15:33 +00:00
Erik Johnston
e8814410ef Have an explicit API to update room config 2017-11-08 16:13:27 +00:00
Erik Johnston
94ff2cda73
Revert "Modify group room association API to allow modification of is_public" 2017-11-08 15:43:34 +00:00
Erik Johnston
167eb01d83
Merge pull request #2637 from spantaleev/avoid-noop-media-deletes 
Avoid no-op media deletes
 2017-11-08 11:53:27 +00:00
Richard van der Hoff
02a9a93bde
Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res 
Fix bug in state group storage
 2017-11-08 09:22:13 +00:00