Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-21 01:14:19 -05:00
Code Issues Packages Projects Releases Wiki Activity
10,606 Commits 4 Branches 609 Tags 60 MiB
99a54bf2af
Commit Graph

7746 Commits

Author SHA1 Message Date
David Baker
32260baa41 pep8 2018-03-28 14:29:42 +01:00
Michael Kaye
33f6195d9a Handle review comments 2018-03-28 14:25:25 +01:00
David Baker
a164270833 Make column definition that works on both dbs 2018-03-28 14:23:00 +01:00
David Baker
352e1ff9ed Add schema delta file 2018-03-28 14:07:57 +01:00
David Baker
79452edeee Add joinability for groups 
Adds API to set the 'joinable' flag, and corresponding flag in the
table.
 2018-03-28 14:03:37 +01:00
Krombel
6152e253d8 Merge branch 'develop' of into allow_auto_join_rooms 2018-03-28 14:45:28 +02:00
Neil Johnson
792d340572 rename stat to future proof 2018-03-28 12:25:02 +01:00
Michael Kaye
4ceaa7433a As daemonizing will make a new process, defer call to init. 2018-03-28 12:19:01 +01:00
Neil Johnson
788e69098c Add user_ips last seen index 2018-03-28 12:03:13 +01:00
Neil Johnson
0f890f477e No need to cast in count_daily_users 2018-03-28 11:49:57 +01:00
Neil Johnson
545001b9e4 Fix search_user_dir multiple sqlite versions do different things 2018-03-28 11:19:45 +01:00
Erik Johnston
01ccc9e6f2 Measure time it takes to calculate state group ID 2018-03-28 11:03:52 +01:00
Neil Johnson
a32d2548d9 query and call for r30 stats 2018-03-28 10:39:13 +01:00
Neil Johnson
9187e0762f count_daily_users failed if db was sqlite due to type failure - presumably this prevcented all sqlite homeservers reporting home 2018-03-28 10:02:32 +01:00
Erik Johnston
f879127aaa
Merge pull request #3029 from matrix-org/erikj/linearize_generate_user_id 
Linearize calls to _generate_user_id
 2018-03-28 10:00:31 +01:00
Erik Johnston
e6d87c93f3
Merge pull request #3030 from matrix-org/erikj/no_ujson 
Remove last usage of ujson
 2018-03-28 10:00:06 +01:00
Erik Johnston
004cc8a328
Merge pull request #3033 from matrix-org/erikj/calculate_state_metrics 
Add counter metrics for calculating state delta
 2018-03-28 09:59:42 +01:00
Michael Kaye
ef520d8d0e Include coarse CPU and Memory use in stats callbacks. 
This requires the psutil module, and is still opt-in based on the report_stats
config option.
 2018-03-27 17:56:03 +01:00
Richard van der Hoff
a134c572a6 Stringify exceptions for keys/{query,claim} 
Make sure we stringify any exceptions we return from keys/query and keys/claim,
to avoid a 'not JSON serializable' error later

Fixes #3010
 2018-03-27 17:15:06 +01:00
Richard van der Hoff
c2a5cf2fe3 factor out exception handling for keys/claim and keys/query 
this stuff is badly c&p'ed
 2018-03-27 17:11:23 +01:00
Erik Johnston
800cfd5774 Comment 2018-03-27 13:30:39 +01:00
Erik Johnston
152c2ac19e Fix indent 2018-03-27 13:13:46 +01:00
Erik Johnston
e70287cff3 Comment 2018-03-27 13:13:38 +01:00
Erik Johnston
03a26e28d9
Merge pull request #3017 from matrix-org/erikj/add_cache_control_headers 
Add Cache-Control headers to all JSON APIs
 2018-03-27 13:10:38 +01:00
Erik Johnston
3e0c0660b3 Also do check inside linearizer 2018-03-27 13:01:34 +01:00
Erik Johnston
3f49e131d9 Add counter metrics for calculating state delta 
This will allow us to measure how often we calculate state deltas in
event persistence that we would have been able to calculate at the same
time we calculated the state for the event.
 2018-03-27 10:57:35 +01:00
Erik Johnston
9b8c0fb162 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-26 21:41:55 +01:00
Erik Johnston
a9d7d98d3f Bum version and changelog 2018-03-26 16:36:53 +01:00
Erik Johnston
bdbb1eec65 Merge branch 'erikj/simplejson_replication' of github.com:matrix-org/synapse into release-v0.27.0 2018-03-26 16:35:55 +01:00
Erik Johnston
9187862002 Bump version and changelog 2018-03-26 16:20:24 +01:00
Neil Johnson
51406dab96 version bump 2018-03-26 14:48:19 +01:00
Erik Johnston
fecb45e0c3 Remove last usage of ujson 2018-03-26 13:32:29 +01:00
Erik Johnston
44cd6e1358 PEP8 2018-03-26 12:06:48 +01:00
Erik Johnston
8d6dc106d1 Don't use _cursor_to_dict in find_next_generated_user_id_localpart 2018-03-26 12:02:44 +01:00
Erik Johnston
a052aa42e7 Linearize calls to _generate_user_id 2018-03-26 12:02:20 +01:00
Matthew Hodgson
8cbbfaefc1 404 correctly on missing paths via NoResource 
fixes https://github.com/matrix-org/synapse/issues/2043 and https://github.com/matrix-org/synapse/issues/2029
 2018-03-23 10:32:50 +00:00
Erik Johnston
84b5cc69f5
Merge pull request #3006 from matrix-org/erikj/state_iter 
Use .iter* to avoid copies in StateHandler
 2018-03-22 11:51:13 +00:00
Erik Johnston
fde8e8f09f Fix s/iteriterms/itervalues 2018-03-22 11:42:16 +00:00
Erik Johnston
eb9fc021e3 Merge branch 'release-v0.27.0' of github.com:matrix-org/synapse into develop 2018-03-22 10:19:53 +00:00
Erik Johnston
1c41b05c8c Add Cache-Control headers to all JSON APIs 
It is especially important that sync requests don't get cached, as if a
sync returns the same token given then the client will call sync with
the same parameters again. If the previous response was cached it will
get reused, resulting in the client tight looping making the same
request and never making any progress.

In general, clients will expect to get up to date data when requesting
APIs, and so its safer to do a blanket no cache policy than only
whitelisting APIs that we know will break things if they get cached.
 2018-03-21 17:46:26 +00:00
Neil Johnson
e66fbcbb02 fix merge conflicts 2018-03-20 14:25:31 +00:00
Erik Johnston
9aa5a0af51 Explicitly use simplejson 2018-03-20 09:58:13 +00:00
Erik Johnston
610accbb7f Fix replication after switch to simplejson 
Turns out that simplejson serialises namedtuple's as dictionaries rather
than tuples by default.
 2018-03-19 16:12:48 +00:00
Neil Johnson
c384705ee8
Update __init__.py 
bump version
 2018-03-19 15:11:58 +00:00
Erik Johnston
3f961e638a
Merge pull request #3005 from matrix-org/erikj/fix_cache_size 
Fix bug where state cache used lots of memory
 2018-03-19 11:44:26 +00:00
Erik Johnston
fa72803490 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-03-19 11:41:01 +00:00
Erik Johnston
9a0d783c11 Add comments 2018-03-19 11:35:53 +00:00
Matthew Hodgson
38f952b9bc spell out not to massively increase bcrypt rounds 2018-03-19 09:27:36 +00:00
Erik Johnston
a8ce159be4 Replace some ujson with simplejson to make it work 2018-03-16 00:27:09 +00:00
Erik Johnston
5b631ff41a Remove wrong comment 2018-03-16 00:07:08 +00:00
Erik Johnston
ba48755d56 Bump version and changelog 2018-03-15 23:57:26 +00:00
Erik Johnston
926ba76e23 Replace ujson with simplejson 2018-03-15 23:43:31 +00:00
Erik Johnston
9cf519769b Use .iter* to avoid copies in StateHandler 2018-03-15 17:50:26 +00:00
Erik Johnston
7c7706f42b Fix bug where state cache used lots of memory 
The state cache bases its size on the sum of the size of entries. The
size of the entry is calculated once on insertion, so it is important
that the size of entries does not change.

The DictionaryCache modified the entries size, which caused the state
cache to incorrectly think it was smaller than it actually was.
 2018-03-15 15:46:54 +00:00
NotAFile
2cc9f76bc3 replace old style error catching with 'as' keyword 
This is both easier to read and compatible with python3 (not that that
matters)

Signed-off-by: Adrian Tschira <nota@notafile.com>
 2018-03-15 16:11:17 +01:00
Erik Johnston
ddb00efc1d Bump version number 2018-03-15 14:41:30 +00:00
Krombel
91ea0202e6 move handling of auto_join_rooms to RegisterHandler 
Currently the handling of auto_join_rooms only works when a user
registers itself via public register api. Registrations via
registration_shared_secret and ModuleApi do not work

This auto_joins the users in the registration handler which enables
the auto join feature for all 3 registration paths.

This is related to issue #2725

Signed-Off-by: Matthias Kesler <krombel@krombel.de>
 2018-03-14 16:45:37 +01:00
Erik Johnston
4f28018c83 Register membership/state servlets in event_creator 2018-03-14 14:30:06 +00:00
Erik Johnston
57db62e554
Merge pull request #2992 from matrix-org/erikj/implement_member_workre 
Implement RoomMemberWorkerHandler
 2018-03-14 14:29:33 +00:00
Erik Johnston
0011ede3b0 Fix imports 2018-03-14 14:19:23 +00:00
Erik Johnston
62ad701326 s/join/joined/ in notify_user_membership_change 2018-03-14 14:17:43 +00:00
Erik Johnston
3f0f06cb31 Split RoomMemberWorkerHandler to separate file 2018-03-14 11:41:45 +00:00
Erik Johnston
3e839e0548
Merge pull request #2989 from matrix-org/erikj/profile_cache_master 
Only update remote profile cache on master
 2018-03-14 09:42:27 +00:00
Erik Johnston
ebd0127999
Merge pull request #2988 from matrix-org/erikj/split_profile_store 
Split up ProfileStore
 2018-03-14 09:41:06 +00:00
Erik Johnston
cfe75a9fb6
Merge pull request #2991 from matrix-org/erikj/fixup_rm 
_remote_join and co take a requester
 2018-03-14 09:40:57 +00:00
Erik Johnston
f51565e023
Merge pull request #2993 from matrix-org/erikj/is_blocked 
Add is_blocked to worker store
 2018-03-14 09:39:18 +00:00
Matthew Hodgson
d144ed6ffb
fix bug #2926 (loading all state for a given type from the DB if the state_key is None) (#2990) 
Fixes a regression that had crept in where the caching layer upholds requests for loading state which is filtered by type (but not by state_key), but the DB layer itself would interpret a missing state_key as a request to filter by null state_key rather than returning all state_keys.
 2018-03-13 22:36:04 +00:00
Erik Johnston
a08726fc42 Add is_blocked to worker store 2018-03-13 18:28:44 +00:00
Erik Johnston
b27320b550 Implement RoomMemberWorkerHandler 2018-03-13 18:26:00 +00:00
Erik Johnston
350331d466 _remote_join and co take a requester 2018-03-13 17:50:39 +00:00
Erik Johnston
1a69c6d590
Merge pull request #2987 from matrix-org/erikj/split_room_member_handler 
Split RoomMemberHandler into base and master class
 2018-03-13 17:40:00 +00:00
Erik Johnston
df8ff682a7 Only update remote profile cache on master 2018-03-13 17:38:21 +00:00
Erik Johnston
3518d0ea8f Split up ProfileStore 2018-03-13 17:36:50 +00:00
Erik Johnston
d45a114824 Raise, don't return, exception 2018-03-13 17:24:34 +00:00
Erik Johnston
6dbebef141 Add missing param to docstrings 2018-03-13 17:15:32 +00:00
Erik Johnston
16adb11cc0 Correct import order 2018-03-13 16:57:07 +00:00
Erik Johnston
82f16faa78 Move user_*_room distributor stuff to master class 
I added yields when calling user_left_room, but they shouldn't matter on
the master process as they always return None anyway.
 2018-03-13 16:38:15 +00:00
Erik Johnston
b78717b87b Split RoomMemberHandler into base and master class 
The intention here is to split the class into the bits that can be done
on workers and the bits that have to be done on the master.

In future there will also be a class that can be run on the worker,
which will delegate work to the master when necessary.
 2018-03-13 16:37:41 +00:00
Erik Johnston
95cb401ae0
Merge pull request #2978 from matrix-org/erikj/refactor_replication_layer 
Remove ReplicationLayer and user Client/Server directly
 2018-03-13 15:45:08 +00:00
Erik Johnston
5d8476d8ff
Merge pull request #2981 from matrix-org/erikj/factor_remote_leave 
Factor out _remote_reject_invite in RoomMember
 2018-03-13 15:44:56 +00:00
Jonas Platte
47ce527f45 Add room_id to the response of rooms/{roomId}/join 
Fixes #2349
 2018-03-13 14:48:12 +01:00
Erik Johnston
56e709857c
Merge pull request #2979 from matrix-org/erikj/no_handlers 
Don't build handlers on workers unnecessarily
 2018-03-13 13:46:38 +00:00
Erik Johnston
cb9f8e527c s/replication_client/federation_client/ 2018-03-13 13:26:52 +00:00
Erik Johnston
cea462e285 s/replication_server/federation_server 2018-03-13 13:22:21 +00:00
Erik Johnston
bf8e97bd3c Merge branch 'develop' of github.com:matrix-org/synapse into erikj/factor_remote_leave 2018-03-13 13:17:08 +00:00
Erik Johnston
ea3442c15c Add docstring 2018-03-13 13:16:21 +00:00
Erik Johnston
16469a4f15
Merge pull request #2980 from matrix-org/erikj/rm_priv 
Make RoomMemberHandler functions private that can be
 2018-03-13 13:11:11 +00:00
Erik Johnston
c82111a55f
Merge pull request #2982 from matrix-org/erikj/fix_extra_users 
extra_users is actually a list of UserIDs
 2018-03-13 13:11:04 +00:00
Erik Johnston
da87791975
Merge pull request #2983 from matrix-org/erikj/rename_register_3pid 
Refactor get_or_register_3pid_guest
 2018-03-13 13:10:52 +00:00
Erik Johnston
99e9b4f26c
Merge pull request #2984 from matrix-org/erikj/fix_rest_regeix 
RoomMembershipRestServlet doesn't handle /forget
 2018-03-13 13:10:44 +00:00
Erik Johnston
f5160d4a3e RoomMembershipRestServlet doesn't handle /forget 
Due to the order we register the REST handlers `/forget` was handled by
the correct handler.
 2018-03-13 12:12:55 +00:00
Erik Johnston
8b3573a8b2 Refactor get_or_register_3pid_guest 2018-03-13 12:08:58 +00:00
Richard van der Hoff
299fd740c7
Merge pull request #2975 from matrix-org/rav/measure_persist_events 
Add Measure block for persist_events
 2018-03-13 12:06:25 +00:00
Erik Johnston
9a2d9b4789
Merge pull request #2977 from matrix-org/erikj/replication_move_props 
Move property setting from ReplicationLayer to base classes
 2018-03-13 11:45:25 +00:00
Erik Johnston
f43b6d6d9b Fix docstring types 2018-03-13 11:29:35 +00:00
Erik Johnston
0f942f68c1 Factor out _remote_reject_invite in RoomMember 2018-03-13 11:22:45 +00:00
Erik Johnston
d0fcc48f9d extra_users is actually a list of UserIDs 2018-03-13 11:20:06 +00:00
Erik Johnston
31becf4ac3 Make functions private that can be 2018-03-13 11:15:16 +00:00
Erik Johnston
d023ecb810 Don't build handlers on workers unnecessarily 2018-03-13 11:08:10 +00:00
Erik Johnston
ea7b3c4b1b Remove unused ReplicationLayer 2018-03-13 11:00:04 +00:00
Erik Johnston
265b993b8a Split replication layer into two 2018-03-13 10:55:47 +00:00
Erik Johnston
e05bf34117 Move property setting from ReplicationLayer to FederationBase 2018-03-13 10:51:30 +00:00
Erik Johnston
c3f79c9da5 Split out edu/query registration to a separate class 2018-03-13 10:24:27 +00:00
Richard van der Hoff
889a2a853a Add Measure block for persist_events 
This seems like a useful thing to measure.
 2018-03-13 10:01:42 +00:00
Richard van der Hoff
e48c7aac4d Add transactional API to history purge 
Make the purge request return quickly, and allow scripts to poll for updates.
 2018-03-12 16:22:55 +00:00
Richard van der Hoff
1708412f56 Return an error when doing two purges on a room 
Queuing up purges doesn't sound like a good thing.
 2018-03-12 16:22:54 +00:00
Richard van der Hoff
b984dd0b73
Merge pull request #2961 from matrix-org/rav/run_in_background 
Factor run_in_background out from preserve_fn
 2018-03-12 16:19:13 +00:00
Richard van der Hoff
ba1d08bc4b
Merge pull request #2965 from matrix-org/rav/request_logging 
Add a metric which increments when a request is received
 2018-03-12 09:45:33 +00:00
Richard van der Hoff
58dd148c4f Add some docstrings to help figure this out 2018-03-09 18:05:41 +00:00
Richard van der Hoff
88541f9009 Add a metric which increments when a request is received 
It's useful to know when there are peaks in incoming requests - which isn't
quite the same as there being peaks in outgoing responses, due to the time
taken to handle requests.
 2018-03-09 16:30:26 +00:00
Richard van der Hoff
dbe80a286b refactor JsonResource 
rephrase the OPTIONS and unrecognised request handling so that they look
similar to the common flow.
 2018-03-09 16:22:16 +00:00
Richard van der Hoff
20f40348d4 Factor run_in_background out from preserve_fn 
It annoys me that we create temporary function objects when there's really no
need for it. Let's factor the gubbins out of preserve_fn and start using it.
 2018-03-08 11:50:11 +00:00
Erik Johnston
a56d54dcb7 Fix up log message 2018-03-07 11:55:31 +00:00
Erik Johnston
02a1296ad6 Fix typo 2018-03-07 11:55:31 +00:00
Erik Johnston
8cb44da4aa Fix race in sync when joining room 
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.

This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
 2018-03-07 11:55:31 +00:00
Richard van der Hoff
8ffaacbee3
Merge pull request #2949 from krombel/use_bcrypt_checkpw 
use bcrypt.checkpw
 2018-03-06 11:56:06 +00:00
Richard van der Hoff
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge 
Implement purge_history by timestamp
 2018-03-06 11:20:23 +00:00
Erik Johnston
7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync 
Remove ability for AS users to call /events and /sync
 2018-03-06 11:10:09 +00:00
Erik Johnston
b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store 
Split Directory store
 2018-03-05 18:17:32 +00:00
Krombel
ed9b5eced4 use bcrypt.checkpw 
in bcrypt 3.1.0 checkpw got introduced (already 2 years ago)
This makes use of that with enhancements which might get introduced
by that

Signed-Off-by: Matthias Kesler <krombel@krombel.de>
 2018-03-05 18:02:59 +01:00
Erik Johnston
d4ffe61d4f Remove ability for AS users to call /events and /sync 
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.

See: https://github.com/matrix-org/matrix-doc/issues/1144
 2018-03-05 15:44:46 +00:00
Erik Johnston
69ce365b79 Fix cache invalidation on deletion 2018-03-05 15:29:03 +00:00
Erik Johnston
2e223163ff Split Directory store 2018-03-05 15:11:30 +00:00
Richard van der Hoff
f8bfcd7e0d Provide a means to pass a timestamp to purge_history 2018-03-05 14:37:23 +00:00
Richard van der Hoff
d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts 
Test and fix find_first_stream_ordering_after_ts
 2018-03-05 12:26:14 +00:00
Richard van der Hoff
c818fcab11 Test and fix find_first_stream_ordering_after_ts 
It seemed to suffer from a bunch of off-by-one errors.
 2018-03-05 12:04:02 +00:00
Richard van der Hoff
06a14876e5 Add find_first_stream_ordering_after_ts 
Expose this as a public function which can be called outside a txn
 2018-03-05 11:53:39 +00:00
Erik Johnston
42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix 
Fix bug with delayed cache invalidation stream
 2018-03-05 11:33:17 +00:00
Erik Johnston
f394f5574d
Merge pull request #2929 from matrix-org/erikj/split_regististration_store 
Split registration store
 2018-03-05 11:33:07 +00:00
dklug
af7ed8e1ef Return 401 for invalid access_token on logout 
Signed-off-by: Duncan Klug <dklug@ucmerced.edu>
 2018-03-02 22:01:27 -08:00
Erik Johnston
efb79820b4 Fix bug with delayed cache invalidation stream 
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
 2018-03-02 14:45:15 +00:00
Erik Johnston
fafa3e7114 Split registration store 2018-03-02 13:48:27 +00:00
Erik Johnston
d960d23830 Add missing yield during 3pid signature checks 2018-03-02 11:03:18 +00:00
Erik Johnston
1a6c7cdf54
Merge pull request #2928 from matrix-org/erikj/read_marker_caches 
Fix typo in getting replication account data processing
 2018-03-01 17:56:14 +00:00
Erik Johnston
89b7232ff8 Fix typo in getting replication account data processing 2018-03-01 17:50:30 +00:00
Erik Johnston
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed 
Split out SignatureStore and EventFederationStore
 2018-03-01 17:32:58 +00:00
Erik Johnston
65cf454fd1 Remove unused DataStore 2018-03-01 17:27:53 +00:00
Erik Johnston
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches 
Improve caching for read_marker API
 2018-03-01 17:12:34 +00:00
Erik Johnston
4b44f05f19 Fewer lies are better 2018-03-01 17:08:17 +00:00
Erik Johnston
a83c514d1f Improve caching for read_marker API 
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
 2018-03-01 17:08:17 +00:00
Erik Johnston
33bebb63f3 Add some caches to help read marker API 2018-03-01 17:08:17 +00:00
Erik Johnston
483e8104db
Merge pull request #2926 from matrix-org/erikj/member_handler_move 
Move RoomMemberHandler out of Handlers
 2018-03-01 17:01:25 +00:00
Erik Johnston
2ad4d5b5bb Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed 2018-03-01 16:59:39 +00:00
Erik Johnston
529c026ac1 Move back to hs.is_mine 2018-03-01 16:49:12 +00:00
Erik Johnston
7c371834cc Stub out broken function only used for cache 2018-03-01 16:44:13 +00:00
Erik Johnston
64346be26d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store 2018-03-01 16:26:42 +00:00
Erik Johnston
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker 
Calculate stream_ordering_month_ago correctly on workers
 2018-03-01 16:23:54 +00:00
Erik Johnston
884b26ae41 Remove unused variables 2018-03-01 16:23:48 +00:00
Erik Johnston
1b2af11650 Document abstract class and method better 2018-03-01 16:20:57 +00:00
Erik Johnston
872ff95ed4 Default stream_ordering_*_ago to None 2018-03-01 16:00:05 +00:00
Erik Johnston
22004b524e Fix comment typo 2018-03-01 15:59:40 +00:00
Erik Johnston
4bc4236faf
Merge pull request #2922 from matrix-org/erikj/split_room_store 
Split up RoomStore
 2018-03-01 15:55:01 +00:00
Richard van der Hoff
2324124a72
Merge pull request #2921 from matrix-org/rav/unyielding_make_deferred_yieldable 
Rewrite make_deferred_yieldable avoiding inlineCallbacks
 2018-03-01 15:39:10 +00:00
Erik Johnston
f793bc3877 Split out stream store 2018-03-01 15:13:08 +00:00
Erik Johnston
784f036306 Move RoomMemberHandler out of Handlers 2018-03-01 14:36:50 +00:00
Erik Johnston
6411f725be Calculate stream_ordering_month_ago correctly on workers 2018-03-01 14:20:53 +00:00
Erik Johnston
a9a2d66cdd Split out SignatureStore and EventFederationStore 2018-03-01 14:17:53 +00:00
Erik Johnston
0c8ba5dd1c Split up RoomStore 2018-03-01 14:01:19 +00:00
Richard van der Hoff
3a75de923b Rewrite make_deferred_yieldable avoiding inlineCallbacks 
... because (a) it's actually simpler (b) it might be marginally more
performant?
 2018-03-01 12:40:05 +00:00
Erik Johnston
126b9bf96f Log in the correct places 2018-03-01 12:05:33 +00:00
Erik Johnston
157298f986 Don't do preserve_fn for every request 2018-03-01 11:59:45 +00:00
Erik Johnston
89f90d808a Add some logging 2018-03-01 11:59:16 +00:00
Erik Johnston
8ded8ba2c7 Make repl send_event idempotent and retry on timeouts 
If we treated timeouts as failures on the worker we would attempt to
clean up e.g. push actions while the master might still process the
event.
 2018-03-01 11:20:34 +00:00
Erik Johnston
f381d63813 Check event auth on the worker 2018-03-01 10:18:37 +00:00
Erik Johnston
6b8604239f Correctly send ratelimit and extra_users params 2018-03-01 10:08:39 +00:00
Erik Johnston
f756f961ea Fixup comments 2018-03-01 10:05:27 +00:00
Erik Johnston
28e973ac11 Calculate push actions on worker 2018-02-28 18:02:30 +00:00
Erik Johnston
493e25d554 Move storage functions for push calculations 
This will allow push actions for an event to be calculated on workers.
 2018-02-27 13:58:16 +00:00
Erik Johnston
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation 
Fix missing invalidations for receipt storage
 2018-02-27 11:34:26 +00:00
Erik Johnston
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store 
Split out RoomMemberStore
 2018-02-27 11:32:10 +00:00
Erik Johnston
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores 
Split AS stores
 2018-02-27 10:07:07 +00:00
Erik Johnston
8956f0147a Add comment 2018-02-27 10:06:51 +00:00
Erik Johnston
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions 
Batch inserts into event_push_actions_staging
 2018-02-26 14:45:40 +00:00
Erik Johnston
45b5fe9122 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push 2018-02-26 13:49:24 +00:00
Erik Johnston
d62ce972f8 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store 2018-02-23 11:46:24 +00:00
Erik Johnston
6ae9a3d2a6 Update copyright 2018-02-23 11:44:49 +00:00
Erik Johnston
a90c60912f Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions 2018-02-23 11:26:31 +00:00
Erik Johnston
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store 
Split out get_events and co into a worker store
 2018-02-23 11:23:52 +00:00
Erik Johnston
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers 
Split PusherStore
 2018-02-23 11:23:35 +00:00
Erik Johnston
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store 
Split PushRulesStore
 2018-02-23 11:23:23 +00:00
Erik Johnston
199dba6c15
Merge pull request #2897 from matrix-org/erikj/split_account_data 
Split AccountDataStore and TagStore
 2018-02-23 11:23:11 +00:00
Erik Johnston
70349872c2 Update copyright 2018-02-23 11:14:35 +00:00
Erik Johnston
eba93b05bf Split EventsWorkerStore into separate file 2018-02-23 11:01:21 +00:00
Erik Johnston
bf8a36e080 Update copyright 2018-02-23 10:52:10 +00:00
Erik Johnston
5d0f665848 Remove redundant clock 2018-02-23 10:49:58 +00:00
Erik Johnston
3bd760628b _event_persist_queue shouldn't be in worker store 2018-02-23 10:49:18 +00:00
Erik Johnston
eb9b5eec81 Update copyright 2018-02-23 10:42:39 +00:00
Erik Johnston
c2ecfcc3a4 Update copyright 2018-02-23 10:41:34 +00:00
Erik Johnston
7e6cf89dc2 Update copyright 2018-02-23 10:39:19 +00:00
Erik Johnston
26d37f7a63 Update copyright 2018-02-23 10:33:55 +00:00
Erik Johnston
bb73f55fc6 Use absolute imports 2018-02-23 10:31:16 +00:00
Erik Johnston
faeb369f15 Fix missing invalidations for receipt storage 2018-02-21 15:19:54 +00:00
Erik Johnston
3dec9c66b3 Split out RoomMemberStore 2018-02-21 12:07:26 +00:00
Erik Johnston
46244b2759 Split AS stores 2018-02-21 11:49:34 +00:00
Erik Johnston
27b094f382 Split out get_events and co into a worker store 2018-02-21 11:41:48 +00:00
Erik Johnston
573712da6b Update comments 2018-02-21 11:29:49 +00:00
Erik Johnston
c96d547f4d Actually use new param 2018-02-21 11:03:42 +00:00
Erik Johnston
d15d237b0d Split out EventPushActionWorkerStore 2018-02-21 11:01:13 +00:00
Erik Johnston
27939cbb0e
Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed 
Delete from push_actions_staging in federation too
 2018-02-21 11:00:06 +00:00
Erik Johnston
6f72765371 Split PusherStore 2018-02-21 10:54:21 +00:00
Erik Johnston
cbaad969f9 Split PushRulesStore 2018-02-21 10:43:31 +00:00
Erik Johnston
ca9b9d9703 Split AccountDataStore and TagStore 2018-02-21 10:15:04 +00:00
Erik Johnston
8fbb4d0d19 Raise exception in abstract method 2018-02-20 17:59:23 +00:00
Erik Johnston
95e4cffd85 Fix comment 2018-02-20 17:58:40 +00:00
Erik Johnston
e316bbb4c0 Use abstract base class to access stream IDs 2018-02-20 17:43:57 +00:00
Erik Johnston
f5ac4dc2d4 Split ReceiptsStore 2018-02-20 16:28:28 +00:00
Erik Johnston
24087bffa9 Ensure all push actions are deleted from staging 2018-02-20 12:34:31 +00:00
Erik Johnston
ad0ccf15ea Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts 2018-02-20 12:34:28 +00:00
Erik Johnston
d874d4f2d7 Delete from push_actions_staging in federation too 2018-02-20 11:37:52 +00:00
Erik Johnston
6ff8c87484 Batch inserts into event_push_actions_staging 2018-02-20 11:33:07 +00:00
Erik Johnston
324c3e9399
Merge pull request #2868 from matrix-org/erikj/refactor_media_storage 
Make store_file use store_into_file
 2018-02-20 11:31:24 +00:00
Pascal Bach
3acd616979 Update pynacl dependency to 1.2.1 or higher 
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
 2018-02-19 10:45:22 +01:00
Richard van der Hoff
d1a3325f99 (Really) fix tablescan of event_push_actions on purge 
commit 278d21b5 added new code to avoid the tablescan, but didn't remove the
old :/
 2018-02-16 14:02:31 +00:00
Erik Johnston
6af025d3c4 Fix typo of double is_highlight 2018-02-16 11:35:31 +00:00
Erik Johnston
012e8e142a Comments 2018-02-16 11:35:01 +00:00
Erik Johnston
b96278d6fe Ensure that we delete staging push actions on errors 2018-02-15 15:47:06 +00:00
Erik Johnston
4810f7effd Remove context.push_actions 2018-02-15 15:47:06 +00:00
Erik Johnston
c714c61853 Update event_push_actions table from staging table 2018-02-15 15:47:06 +00:00
Erik Johnston
acac21248c Store push actions in staging area 2018-02-15 15:47:04 +00:00
Erik Johnston
106906a65e Don't serialize current state over replication 2018-02-15 13:53:18 +00:00
Erik Johnston
5fb347fc41
Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log 
Don't log errors propogated from send_event
 2018-02-15 12:31:49 +00:00
Erik Johnston
fd1601c596 Fix state group storage bug in workers 
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
 2018-02-15 11:04:32 +00:00
Erik Johnston
ef344b10e5 Don't log errors propogated from send_event 2018-02-15 11:03:49 +00:00
Richard van der Hoff
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge 
purge_history cleanups
 2018-02-15 09:49:07 +00:00
Erik Johnston
92c52df702 Make store_file use store_into_file 2018-02-14 17:55:18 +00:00
Richard van der Hoff
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin 
switch back from GIST to GIN indexes
 2018-02-14 16:59:03 +00:00
Richard van der Hoff
39bf47319f purge_history: fix sqlite syntax error 
apparently sqlite insists on indexes being named
 2018-02-14 16:42:19 +00:00
Richard van der Hoff
ac27f6a35e purge_history: handle sqlite asshattery 
apparently creating a temporary table commits the transaction. because that's a
useful thing.
 2018-02-14 16:41:12 +00:00
Richard van der Hoff
5978dccff0 remove overzealous exception handling 2018-02-14 15:54:09 +00:00
Richard van der Hoff
278d21b5e4 purge_history: fix index use 
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
 2018-02-14 15:44:51 +00:00
Richard van der Hoff
5fcbf1e07c Rework event purge to use a temporary table 
... which should speed things up by reducing the amount of data being shuffled
across the connection
 2018-02-14 11:02:22 +00:00
Erik Johnston
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker 
Create a worker for event creation
 2018-02-13 18:07:10 +00:00
Richard van der Hoff
d627174da2 Fix log message in purge_history 
(we don't just remove remote events)
 2018-02-13 16:51:21 +00:00
Richard van der Hoff
ddb6a79b68 Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin 2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3 move search reindex to schema 47 
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.

This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
 2018-02-13 16:44:46 +00:00
Richard van der Hoff
4a6d551704 GIN reindex: Fix syntax errors, improve exception handling 2018-02-13 16:44:46 +00:00
Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching 
Use StateResolutionHandler to resolve state in persist_events
 2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit 2018-02-13 14:29:22 +00:00
Richard van der Hoff
8fd1a32456 Fix typos in purge api & doc 
* It's supposed to be purge_local_events, not ..._history
* Fix the doc to have valid json
 2018-02-13 13:09:39 +00:00
Richard van der Hoff
4d09366656
Merge pull request #2695 from okurz/feature/allow_recent_pysaml 
Allow use of higher versions of saml2
 2018-02-13 12:24:08 +00:00
Richard van der Hoff
a9b712e9dc Merge branch 'develop' into matthew/gin_work_mem 2018-02-13 12:16:01 +00:00
Erik Johnston
1026690cd2
Merge pull request #2857 from matrix-org/erikj/upload_store 
Tell storage providers about new file so they can upload
 2018-02-12 13:52:58 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates 
delete_local_events for purge_room_history
 2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496 purge: move room_depth update to end 
... to avoid locking the table for too long
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b delete_local_events for purge_history 
Add a flag which makes the purger delete local events
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d purge: Move cache invalidation to more appropriate place 
it was a bit of a non-sequitur there
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137 bump purge logging to info 
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf rename delete_old_state -> purge_history 
(beacause it deletes more than state)
 2018-02-09 13:07:41 +00:00
Erik Johnston
5fa571a91b Tell storage providers about new file so they can upload 2018-02-07 13:35:08 +00:00
Erik Johnston
50fe92cd26 Move presence handling into handle_new_client_event 
As we want to have it run on the main synapse instance
 2018-02-07 10:34:09 +00:00