Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
10,300 Commits 5 Branches 723 Tags 124 MiB
152c2ac19e
Commit Graph

7509 Commits

Author SHA1 Message Date
Erik Johnston
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches 
Improve caching for read_marker API
 2018-03-01 17:12:34 +00:00
Erik Johnston
4b44f05f19 Fewer lies are better 2018-03-01 17:08:17 +00:00
Erik Johnston
a83c514d1f Improve caching for read_marker API 
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
 2018-03-01 17:08:17 +00:00
Erik Johnston
33bebb63f3 Add some caches to help read marker API 2018-03-01 17:08:17 +00:00
Erik Johnston
483e8104db
Merge pull request #2926 from matrix-org/erikj/member_handler_move 
Move RoomMemberHandler out of Handlers
 2018-03-01 17:01:25 +00:00
Erik Johnston
2ad4d5b5bb Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed 2018-03-01 16:59:39 +00:00
Erik Johnston
529c026ac1 Move back to hs.is_mine 2018-03-01 16:49:12 +00:00
Erik Johnston
7c371834cc Stub out broken function only used for cache 2018-03-01 16:44:13 +00:00
Erik Johnston
64346be26d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store 2018-03-01 16:26:42 +00:00
Erik Johnston
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker 
Calculate stream_ordering_month_ago correctly on workers
 2018-03-01 16:23:54 +00:00
Erik Johnston
884b26ae41 Remove unused variables 2018-03-01 16:23:48 +00:00
Erik Johnston
1b2af11650 Document abstract class and method better 2018-03-01 16:20:57 +00:00
Erik Johnston
872ff95ed4 Default stream_ordering_*_ago to None 2018-03-01 16:00:05 +00:00
Erik Johnston
22004b524e Fix comment typo 2018-03-01 15:59:40 +00:00
Erik Johnston
4bc4236faf
Merge pull request #2922 from matrix-org/erikj/split_room_store 
Split up RoomStore
 2018-03-01 15:55:01 +00:00
Richard van der Hoff
2324124a72
Merge pull request #2921 from matrix-org/rav/unyielding_make_deferred_yieldable 
Rewrite make_deferred_yieldable avoiding inlineCallbacks
 2018-03-01 15:39:10 +00:00
Erik Johnston
f793bc3877 Split out stream store 2018-03-01 15:13:08 +00:00
Erik Johnston
784f036306 Move RoomMemberHandler out of Handlers 2018-03-01 14:36:50 +00:00
Erik Johnston
6411f725be Calculate stream_ordering_month_ago correctly on workers 2018-03-01 14:20:53 +00:00
Erik Johnston
a9a2d66cdd Split out SignatureStore and EventFederationStore 2018-03-01 14:17:53 +00:00
Erik Johnston
0c8ba5dd1c Split up RoomStore 2018-03-01 14:01:19 +00:00
Richard van der Hoff
3a75de923b Rewrite make_deferred_yieldable avoiding inlineCallbacks 
... because (a) it's actually simpler (b) it might be marginally more
performant?
 2018-03-01 12:40:05 +00:00
Erik Johnston
126b9bf96f Log in the correct places 2018-03-01 12:05:33 +00:00
Erik Johnston
157298f986 Don't do preserve_fn for every request 2018-03-01 11:59:45 +00:00
Erik Johnston
89f90d808a Add some logging 2018-03-01 11:59:16 +00:00
Erik Johnston
8ded8ba2c7 Make repl send_event idempotent and retry on timeouts 
If we treated timeouts as failures on the worker we would attempt to
clean up e.g. push actions while the master might still process the
event.
 2018-03-01 11:20:34 +00:00
Erik Johnston
f381d63813 Check event auth on the worker 2018-03-01 10:18:37 +00:00
Erik Johnston
6b8604239f Correctly send ratelimit and extra_users params 2018-03-01 10:08:39 +00:00
Erik Johnston
f756f961ea Fixup comments 2018-03-01 10:05:27 +00:00
Erik Johnston
28e973ac11 Calculate push actions on worker 2018-02-28 18:02:30 +00:00
Erik Johnston
493e25d554 Move storage functions for push calculations 
This will allow push actions for an event to be calculated on workers.
 2018-02-27 13:58:16 +00:00
Erik Johnston
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation 
Fix missing invalidations for receipt storage
 2018-02-27 11:34:26 +00:00
Erik Johnston
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store 
Split out RoomMemberStore
 2018-02-27 11:32:10 +00:00
Erik Johnston
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores 
Split AS stores
 2018-02-27 10:07:07 +00:00
Erik Johnston
8956f0147a Add comment 2018-02-27 10:06:51 +00:00
Erik Johnston
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions 
Batch inserts into event_push_actions_staging
 2018-02-26 14:45:40 +00:00
Erik Johnston
45b5fe9122 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push 2018-02-26 13:49:24 +00:00
Erik Johnston
d62ce972f8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store 2018-02-23 11:46:24 +00:00
Erik Johnston
6ae9a3d2a6 Update copyright 2018-02-23 11:44:49 +00:00
Erik Johnston
a90c60912f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions 2018-02-23 11:26:31 +00:00
Erik Johnston
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store 
Split out get_events and co into a worker store
 2018-02-23 11:23:52 +00:00
Erik Johnston
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers 
Split PusherStore
 2018-02-23 11:23:35 +00:00
Erik Johnston
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store 
Split PushRulesStore
 2018-02-23 11:23:23 +00:00
Erik Johnston
199dba6c15
Merge pull request #2897 from matrix-org/erikj/split_account_data 
Split AccountDataStore and TagStore
 2018-02-23 11:23:11 +00:00
Erik Johnston
70349872c2 Update copyright 2018-02-23 11:14:35 +00:00
Erik Johnston
eba93b05bf Split EventsWorkerStore into separate file 2018-02-23 11:01:21 +00:00
Erik Johnston
bf8a36e080 Update copyright 2018-02-23 10:52:10 +00:00
Erik Johnston
5d0f665848 Remove redundant clock 2018-02-23 10:49:58 +00:00
Erik Johnston
3bd760628b _event_persist_queue shouldn't be in worker store 2018-02-23 10:49:18 +00:00
Erik Johnston
eb9b5eec81 Update copyright 2018-02-23 10:42:39 +00:00
Erik Johnston
c2ecfcc3a4 Update copyright 2018-02-23 10:41:34 +00:00
Erik Johnston
7e6cf89dc2 Update copyright 2018-02-23 10:39:19 +00:00
Erik Johnston
26d37f7a63 Update copyright 2018-02-23 10:33:55 +00:00
Erik Johnston
bb73f55fc6 Use absolute imports 2018-02-23 10:31:16 +00:00
Erik Johnston
faeb369f15 Fix missing invalidations for receipt storage 2018-02-21 15:19:54 +00:00
Erik Johnston
3dec9c66b3 Split out RoomMemberStore 2018-02-21 12:07:26 +00:00
Erik Johnston
46244b2759 Split AS stores 2018-02-21 11:49:34 +00:00
Erik Johnston
27b094f382 Split out get_events and co into a worker store 2018-02-21 11:41:48 +00:00
Erik Johnston
573712da6b Update comments 2018-02-21 11:29:49 +00:00
Erik Johnston
c96d547f4d Actually use new param 2018-02-21 11:03:42 +00:00
Erik Johnston
d15d237b0d Split out EventPushActionWorkerStore 2018-02-21 11:01:13 +00:00
Erik Johnston
27939cbb0e
Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed 
Delete from push_actions_staging in federation too
 2018-02-21 11:00:06 +00:00
Erik Johnston
6f72765371 Split PusherStore 2018-02-21 10:54:21 +00:00
Erik Johnston
cbaad969f9 Split PushRulesStore 2018-02-21 10:43:31 +00:00
Erik Johnston
ca9b9d9703 Split AccountDataStore and TagStore 2018-02-21 10:15:04 +00:00
Erik Johnston
8fbb4d0d19 Raise exception in abstract method 2018-02-20 17:59:23 +00:00
Erik Johnston
95e4cffd85 Fix comment 2018-02-20 17:58:40 +00:00
Erik Johnston
e316bbb4c0 Use abstract base class to access stream IDs 2018-02-20 17:43:57 +00:00
Erik Johnston
f5ac4dc2d4 Split ReceiptsStore 2018-02-20 16:28:28 +00:00
Erik Johnston
24087bffa9 Ensure all push actions are deleted from staging 2018-02-20 12:34:31 +00:00
Erik Johnston
ad0ccf15ea Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts 2018-02-20 12:34:28 +00:00
Erik Johnston
d874d4f2d7 Delete from push_actions_staging in federation too 2018-02-20 11:37:52 +00:00
Erik Johnston
6ff8c87484 Batch inserts into event_push_actions_staging 2018-02-20 11:33:07 +00:00
Erik Johnston
324c3e9399
Merge pull request #2868 from matrix-org/erikj/refactor_media_storage 
Make store_file use store_into_file
 2018-02-20 11:31:24 +00:00
Pascal Bach
3acd616979 Update pynacl dependency to 1.2.1 or higher 
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
 2018-02-19 10:45:22 +01:00
Richard van der Hoff
d1a3325f99 (Really) fix tablescan of event_push_actions on purge 
commit 278d21b5 added new code to avoid the tablescan, but didn't remove the
old :/
 2018-02-16 14:02:31 +00:00
Erik Johnston
6af025d3c4 Fix typo of double is_highlight 2018-02-16 11:35:31 +00:00
Erik Johnston
012e8e142a Comments 2018-02-16 11:35:01 +00:00
Erik Johnston
b96278d6fe Ensure that we delete staging push actions on errors 2018-02-15 15:47:06 +00:00
Erik Johnston
4810f7effd Remove context.push_actions 2018-02-15 15:47:06 +00:00
Erik Johnston
c714c61853 Update event_push_actions table from staging table 2018-02-15 15:47:06 +00:00
Erik Johnston
acac21248c Store push actions in staging area 2018-02-15 15:47:04 +00:00
Erik Johnston
106906a65e Don't serialize current state over replication 2018-02-15 13:53:18 +00:00
Erik Johnston
5fb347fc41
Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log 
Don't log errors propogated from send_event
 2018-02-15 12:31:49 +00:00
Erik Johnston
fd1601c596 Fix state group storage bug in workers 
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
 2018-02-15 11:04:32 +00:00
Erik Johnston
ef344b10e5 Don't log errors propogated from send_event 2018-02-15 11:03:49 +00:00
Richard van der Hoff
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge 
purge_history cleanups
 2018-02-15 09:49:07 +00:00
Erik Johnston
92c52df702 Make store_file use store_into_file 2018-02-14 17:55:18 +00:00
Richard van der Hoff
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin 
switch back from GIST to GIN indexes
 2018-02-14 16:59:03 +00:00
Richard van der Hoff
39bf47319f purge_history: fix sqlite syntax error 
apparently sqlite insists on indexes being named
 2018-02-14 16:42:19 +00:00
Richard van der Hoff
ac27f6a35e purge_history: handle sqlite asshattery 
apparently creating a temporary table commits the transaction. because that's a
useful thing.
 2018-02-14 16:41:12 +00:00
Richard van der Hoff
5978dccff0 remove overzealous exception handling 2018-02-14 15:54:09 +00:00
Richard van der Hoff
278d21b5e4 purge_history: fix index use 
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
 2018-02-14 15:44:51 +00:00
Richard van der Hoff
5fcbf1e07c Rework event purge to use a temporary table 
... which should speed things up by reducing the amount of data being shuffled
across the connection
 2018-02-14 11:02:22 +00:00
Erik Johnston
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker 
Create a worker for event creation
 2018-02-13 18:07:10 +00:00
Richard van der Hoff
d627174da2 Fix log message in purge_history 
(we don't just remove remote events)
 2018-02-13 16:51:21 +00:00
Richard van der Hoff
ddb6a79b68 Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin 2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3 move search reindex to schema 47 
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.

This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
 2018-02-13 16:44:46 +00:00
Richard van der Hoff
4a6d551704 GIN reindex: Fix syntax errors, improve exception handling 2018-02-13 16:44:46 +00:00
Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching 
Use StateResolutionHandler to resolve state in persist_events
 2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit 2018-02-13 14:29:22 +00:00
Richard van der Hoff
8fd1a32456 Fix typos in purge api & doc 
* It's supposed to be purge_local_events, not ..._history
* Fix the doc to have valid json
 2018-02-13 13:09:39 +00:00
Richard van der Hoff
4d09366656
Merge pull request #2695 from okurz/feature/allow_recent_pysaml 
Allow use of higher versions of saml2
 2018-02-13 12:24:08 +00:00
Richard van der Hoff
a9b712e9dc Merge branch 'develop' into matthew/gin_work_mem 2018-02-13 12:16:01 +00:00
Erik Johnston
1026690cd2
Merge pull request #2857 from matrix-org/erikj/upload_store 
Tell storage providers about new file so they can upload
 2018-02-12 13:52:58 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates 
delete_local_events for purge_room_history
 2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496 purge: move room_depth update to end 
... to avoid locking the table for too long
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b delete_local_events for purge_history 
Add a flag which makes the purger delete local events
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d purge: Move cache invalidation to more appropriate place 
it was a bit of a non-sequitur there
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137 bump purge logging to info 
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf rename delete_old_state -> purge_history 
(beacause it deletes more than state)
 2018-02-09 13:07:41 +00:00
Erik Johnston
5fa571a91b Tell storage providers about new file so they can upload 2018-02-07 13:35:08 +00:00
Erik Johnston
50fe92cd26 Move presence handling into handle_new_client_event 
As we want to have it run on the main synapse instance
 2018-02-07 10:34:09 +00:00
Erik Johnston
8ec2e638be Add event_creator worker 2018-02-07 10:32:32 +00:00
Erik Johnston
24dd73028a Add replication http endpoint for event sending 2018-02-07 10:32:32 +00:00
Erik Johnston
e3624fad5f Remove pointless ratelimit check 
The intention was for the check to be called as early as possible in the
request, but actually was called just before the main ratelimit check,
so was fairly pointless.
 2018-02-07 10:30:25 +00:00
Erik Johnston
617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation 
Split event creation into a separate handler
 2018-02-06 17:01:17 +00:00
Erik Johnston
3e1e69ccaf Update copyright 2018-02-06 16:40:38 +00:00
Erik Johnston
770b2252ca s/_create_new_client_event/create_new_client_event/ 2018-02-06 16:40:30 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events (#2784) 
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
 2018-02-06 14:31:24 +00:00
Richard van der Hoff
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta 
Remove redundant return value from _calculate_state_delta
 2018-02-05 17:42:20 +01:00
Richard van der Hoff
ebfe64e3d6 Use StateResolutionHandler to resolve state in persist events 
... and thus benefit (hopefully) from its cache.
 2018-02-05 16:23:26 +00:00
Richard van der Hoff
225dc3b4cb Flatten _get_new_state_after_events 
rejig the if statements to simplify the logic and reduce indentation
 2018-02-05 16:23:25 +00:00
Richard van der Hoff
9fcbbe8e7d Check that events being persisted have state_group 2018-02-05 16:23:25 +00:00
Richard van der Hoff
447aed42d2 Add event_map param to resolve_state_groups 2018-02-05 16:23:25 +00:00
Richard van der Hoff
ee6fb4cf85 Remove redundant return value from _calculate_state_delta 
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
 2018-02-05 16:23:20 +00:00
Richard van der Hoff
3c7b480ba3 Factor out common code for search insert 
we can reuse the same code as is used for event insert, for doing the
background index population.
 2018-02-05 16:12:14 +00:00
Erik Johnston
3fa362502c Update places where we create events 2018-02-05 16:01:48 +00:00
Erik Johnston
5ff3d23564 Split event creation into a separate handler 2018-02-05 16:01:48 +00:00
Richard van der Hoff
c46e75d3d8 Move store_event_search_txn to SearchStore 
... as a precursor to making event storing and doing the bg update share some
code.
 2018-02-05 15:43:22 +00:00
Richard van der Hoff
db91e72ade
Merge pull request #2844 from matrix-org/rav/evicted_metrics 
montoring metrics for number of cache evictions
 2018-02-05 16:34:36 +01:00
Richard van der Hoff
bc496df192 report metrics on number of cache evictions 2018-02-05 15:34:01 +00:00
Erik Johnston
1f881e0746
Merge pull request #2791 from matrix-org/erikj/media_storage_refactor 
Ensure media is in local cache before thumbnailing
 2018-02-05 11:28:52 +00:00
Richard van der Hoff
80b8a28100 Factor out common code for search insert 
we can reuse the same code as is used for event insert, for doing the
background index population.
 2018-02-04 00:23:06 +00:00
Richard van der Hoff
bd25f9cf36 Clean up work_mem handling 
Add some comments and improve exception handling when twiddling work_mem for
the search update
 2018-02-03 23:05:41 +00:00
Richard van der Hoff
4eeae7ad65 Move store_event_search_txn to SearchStore 
... as a precursor to making event storing and doing the bg update share some
code.
 2018-02-03 22:59:45 +00:00
Richard van der Hoff
bb9f0f3cdb Merge branch 'develop' into matthew/gin_work_mem 2018-02-03 22:40:28 +00:00
Richard van der Hoff
6b02fc80d1 Reinstate event_search_postgres_gist handler 
People may have queued updates for this, so we can't just delete it.
 2018-02-02 14:32:51 +00:00
Richard van der Hoff
9c9356512e
Merge pull request #2845 from matrix-org/rav/urlcache_error_handling 
Handle url_previews with no content-type
 2018-02-02 15:27:52 +01:00
Richard van der Hoff
18eae413af
Merge pull request #2842 from matrix-org/rav/state_resolution_handler 
Factor out resolve_state_groups to a separate handler
 2018-02-02 15:27:35 +01:00
Richard van der Hoff
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta 
factor _get_new_state_after_events out of _calculate_state_delta
 2018-02-02 15:27:15 +01:00
Richard van der Hoff
9dcd667ac2
Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring 
Docstring fixes
 2018-02-02 15:26:49 +01:00
Travis Ralston
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media 2018-02-01 18:05:47 -07:00
Richard van der Hoff
d5352cbba8 Handle url_previews with no content-type 
avoid failing with an exception if the remote server doesn't give us a
Content-Type header.

Also, clean up the exception handling a bit.
 2018-02-02 00:53:46 +00:00
Richard van der Hoff
14737ba495 doc arg types for _seperate 2018-02-01 12:41:34 +00:00
Richard van der Hoff
e15d4ea248 More docstring fixes 
Fix a couple of errors in docstrings
 2018-02-01 12:41:34 +00:00
Richard van der Hoff
a18828c129 Fix docstring for StateHandler.resolve_state_groups 
The return type was a complete lie, so fix it
 2018-02-01 12:41:34 +00:00
Richard van der Hoff
6da4c4d3bd Factor out resolve_state_groups to a separate handler 
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
 2018-02-01 12:40:04 +00:00
Richard van der Hoff
0cbda53819 Rename resolve_state_groups -> resolve_state_groups_for_events 
(to make way for a method that actually just does the state group resolution)
 2018-02-01 12:40:00 +00:00
Richard van der Hoff
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media 
Fix sql error in quarantine_media
 2018-02-01 11:45:58 +01:00