Erik Johnston
ea7b3c4b1b
Remove unused ReplicationLayer
2018-03-13 11:00:04 +00:00
Erik Johnston
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
Erik Johnston
e05bf34117
Move property setting from ReplicationLayer to FederationBase
2018-03-13 10:51:30 +00:00
Erik Johnston
c3f79c9da5
Split out edu/query registration to a separate class
2018-03-13 10:24:27 +00:00
Richard van der Hoff
889a2a853a
Add Measure block for persist_events
...
This seems like a useful thing to measure.
2018-03-13 10:01:42 +00:00
Richard van der Hoff
e48c7aac4d
Add transactional API to history purge
...
Make the purge request return quickly, and allow scripts to poll for updates.
2018-03-12 16:22:55 +00:00
Richard van der Hoff
1708412f56
Return an error when doing two purges on a room
...
Queuing up purges doesn't sound like a good thing.
2018-03-12 16:22:54 +00:00
Richard van der Hoff
b984dd0b73
Merge pull request #2961 from matrix-org/rav/run_in_background
...
Factor run_in_background out from preserve_fn
2018-03-12 16:19:13 +00:00
Richard van der Hoff
ba1d08bc4b
Merge pull request #2965 from matrix-org/rav/request_logging
...
Add a metric which increments when a request is received
2018-03-12 09:45:33 +00:00
Richard van der Hoff
58dd148c4f
Add some docstrings to help figure this out
2018-03-09 18:05:41 +00:00
Richard van der Hoff
88541f9009
Add a metric which increments when a request is received
...
It's useful to know when there are peaks in incoming requests - which isn't
quite the same as there being peaks in outgoing responses, due to the time
taken to handle requests.
2018-03-09 16:30:26 +00:00
Richard van der Hoff
dbe80a286b
refactor JsonResource
...
rephrase the OPTIONS and unrecognised request handling so that they look
similar to the common flow.
2018-03-09 16:22:16 +00:00
Richard van der Hoff
20f40348d4
Factor run_in_background out from preserve_fn
...
It annoys me that we create temporary function objects when there's really no
need for it. Let's factor the gubbins out of preserve_fn and start using it.
2018-03-08 11:50:11 +00:00
Erik Johnston
a56d54dcb7
Fix up log message
2018-03-07 11:55:31 +00:00
Erik Johnston
02a1296ad6
Fix typo
2018-03-07 11:55:31 +00:00
Erik Johnston
8cb44da4aa
Fix race in sync when joining room
...
The race happens when the user joins a room at the same time as doing a
sync. We fetch the current token and then get the rooms the user is in.
If the join happens after the current token, but before we get the rooms
we end up sending down a partial room entry in the sync.
This is fixed by looking at the stream ordering of the membership
returned by get_rooms_for_user, and handling the case when that stream
ordering is after the current token.
2018-03-07 11:55:31 +00:00
Richard van der Hoff
8ffaacbee3
Merge pull request #2949 from krombel/use_bcrypt_checkpw
...
use bcrypt.checkpw
2018-03-06 11:56:06 +00:00
Richard van der Hoff
b2932107bb
Merge pull request #2946 from matrix-org/rav/timestamp_to_purge
...
Implement purge_history by timestamp
2018-03-06 11:20:23 +00:00
Erik Johnston
7aed50a038
Merge pull request #2948 from matrix-org/erikj/kill_as_sync
...
Remove ability for AS users to call /events and /sync
2018-03-06 11:10:09 +00:00
Erik Johnston
b6c4b851f1
Merge pull request #2947 from matrix-org/erikj/split_directory_store
...
Split Directory store
2018-03-05 18:17:32 +00:00
Krombel
ed9b5eced4
use bcrypt.checkpw
...
in bcrypt 3.1.0 checkpw got introduced (already 2 years ago)
This makes use of that with enhancements which might get introduced
by that
Signed-Off-by: Matthias Kesler <krombel@krombel.de>
2018-03-05 18:02:59 +01:00
Erik Johnston
d4ffe61d4f
Remove ability for AS users to call /events and /sync
...
This functionality has been deprecated for a while as well as being
broken for a while. Instead of fixing it lets just remove it entirely.
See: https://github.com/matrix-org/matrix-doc/issues/1144
2018-03-05 15:44:46 +00:00
Erik Johnston
69ce365b79
Fix cache invalidation on deletion
2018-03-05 15:29:03 +00:00
Erik Johnston
2e223163ff
Split Directory store
2018-03-05 15:11:30 +00:00
Richard van der Hoff
f8bfcd7e0d
Provide a means to pass a timestamp to purge_history
2018-03-05 14:37:23 +00:00
Richard van der Hoff
d032785aa7
Merge pull request #2943 from matrix-org/rav/fix_find_first_stream_ordering_after_ts
...
Test and fix find_first_stream_ordering_after_ts
2018-03-05 12:26:14 +00:00
Richard van der Hoff
c818fcab11
Test and fix find_first_stream_ordering_after_ts
...
It seemed to suffer from a bunch of off-by-one errors.
2018-03-05 12:04:02 +00:00
Richard van der Hoff
06a14876e5
Add find_first_stream_ordering_after_ts
...
Expose this as a public function which can be called outside a txn
2018-03-05 11:53:39 +00:00
Erik Johnston
42174946f8
Merge pull request #2934 from matrix-org/erikj/cache_fix
...
Fix bug with delayed cache invalidation stream
2018-03-05 11:33:17 +00:00
Erik Johnston
f394f5574d
Merge pull request #2929 from matrix-org/erikj/split_regististration_store
...
Split registration store
2018-03-05 11:33:07 +00:00
dklug
af7ed8e1ef
Return 401 for invalid access_token on logout
...
Signed-off-by: Duncan Klug <dklug@ucmerced.edu>
2018-03-02 22:01:27 -08:00
Erik Johnston
efb79820b4
Fix bug with delayed cache invalidation stream
...
We poked the notifier before updated the current token for the cache
invalidation stream. This mean that sometimes the update wouldn't be
sent until the next time a cache was invalidated.
2018-03-02 14:45:15 +00:00
Erik Johnston
fafa3e7114
Split registration store
2018-03-02 13:48:27 +00:00
Erik Johnston
d960d23830
Add missing yield during 3pid signature checks
2018-03-02 11:03:18 +00:00
Erik Johnston
1a6c7cdf54
Merge pull request #2928 from matrix-org/erikj/read_marker_caches
...
Fix typo in getting replication account data processing
2018-03-01 17:56:14 +00:00
Erik Johnston
89b7232ff8
Fix typo in getting replication account data processing
2018-03-01 17:50:30 +00:00
Erik Johnston
1773df0632
Merge pull request #2925 from matrix-org/erikj/split_sig_fed
...
Split out SignatureStore and EventFederationStore
2018-03-01 17:32:58 +00:00
Erik Johnston
65cf454fd1
Remove unused DataStore
2018-03-01 17:27:53 +00:00
Erik Johnston
9e08a93a7b
Merge pull request #2927 from matrix-org/erikj/read_marker_caches
...
Improve caching for read_marker API
2018-03-01 17:12:34 +00:00
Erik Johnston
4b44f05f19
Fewer lies are better
2018-03-01 17:08:17 +00:00
Erik Johnston
a83c514d1f
Improve caching for read_marker API
...
We add a new storage function to get a paritcular type of room account
data. This allows us to prefill the cache when updating that acount
data.
2018-03-01 17:08:17 +00:00
Erik Johnston
33bebb63f3
Add some caches to help read marker API
2018-03-01 17:08:17 +00:00
Erik Johnston
483e8104db
Merge pull request #2926 from matrix-org/erikj/member_handler_move
...
Move RoomMemberHandler out of Handlers
2018-03-01 17:01:25 +00:00
Erik Johnston
2ad4d5b5bb
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_sig_fed
2018-03-01 16:59:39 +00:00
Erik Johnston
529c026ac1
Move back to hs.is_mine
2018-03-01 16:49:12 +00:00
Erik Johnston
7c371834cc
Stub out broken function only used for cache
2018-03-01 16:44:13 +00:00
Erik Johnston
64346be26d
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_stream_store
2018-03-01 16:26:42 +00:00
Erik Johnston
22518e2833
Merge pull request #2923 from matrix-org/erikj/stream_ago_worker
...
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 16:23:54 +00:00
Erik Johnston
884b26ae41
Remove unused variables
2018-03-01 16:23:48 +00:00
Erik Johnston
1b2af11650
Document abstract class and method better
2018-03-01 16:20:57 +00:00
Erik Johnston
872ff95ed4
Default stream_ordering_*_ago to None
2018-03-01 16:00:05 +00:00
Erik Johnston
22004b524e
Fix comment typo
2018-03-01 15:59:40 +00:00
Erik Johnston
4bc4236faf
Merge pull request #2922 from matrix-org/erikj/split_room_store
...
Split up RoomStore
2018-03-01 15:55:01 +00:00
Richard van der Hoff
2324124a72
Merge pull request #2921 from matrix-org/rav/unyielding_make_deferred_yieldable
...
Rewrite make_deferred_yieldable avoiding inlineCallbacks
2018-03-01 15:39:10 +00:00
Erik Johnston
f793bc3877
Split out stream store
2018-03-01 15:13:08 +00:00
Erik Johnston
784f036306
Move RoomMemberHandler out of Handlers
2018-03-01 14:36:50 +00:00
Erik Johnston
6411f725be
Calculate stream_ordering_month_ago correctly on workers
2018-03-01 14:20:53 +00:00
Erik Johnston
a9a2d66cdd
Split out SignatureStore and EventFederationStore
2018-03-01 14:17:53 +00:00
Erik Johnston
0c8ba5dd1c
Split up RoomStore
2018-03-01 14:01:19 +00:00
Richard van der Hoff
3a75de923b
Rewrite make_deferred_yieldable avoiding inlineCallbacks
...
... because (a) it's actually simpler (b) it might be marginally more
performant?
2018-03-01 12:40:05 +00:00
Erik Johnston
126b9bf96f
Log in the correct places
2018-03-01 12:05:33 +00:00
Erik Johnston
157298f986
Don't do preserve_fn for every request
2018-03-01 11:59:45 +00:00
Erik Johnston
89f90d808a
Add some logging
2018-03-01 11:59:16 +00:00
Erik Johnston
8ded8ba2c7
Make repl send_event idempotent and retry on timeouts
...
If we treated timeouts as failures on the worker we would attempt to
clean up e.g. push actions while the master might still process the
event.
2018-03-01 11:20:34 +00:00
Erik Johnston
f381d63813
Check event auth on the worker
2018-03-01 10:18:37 +00:00
Erik Johnston
6b8604239f
Correctly send ratelimit and extra_users params
2018-03-01 10:08:39 +00:00
Erik Johnston
f756f961ea
Fixup comments
2018-03-01 10:05:27 +00:00
Erik Johnston
28e973ac11
Calculate push actions on worker
2018-02-28 18:02:30 +00:00
Erik Johnston
493e25d554
Move storage functions for push calculations
...
This will allow push actions for an event to be calculated on workers.
2018-02-27 13:58:16 +00:00
Erik Johnston
3594dbc6dc
Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation
...
Fix missing invalidations for receipt storage
2018-02-27 11:34:26 +00:00
Erik Johnston
2311189ee4
Merge pull request #2903 from matrix-org/erikj/split_roommember_store
...
Split out RoomMemberStore
2018-02-27 11:32:10 +00:00
Erik Johnston
c57607874c
Merge pull request #2901 from matrix-org/erikj/split_as_stores
...
Split AS stores
2018-02-27 10:07:07 +00:00
Erik Johnston
8956f0147a
Add comment
2018-02-27 10:06:51 +00:00
Erik Johnston
e5b4a208ce
Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions
...
Batch inserts into event_push_actions_staging
2018-02-26 14:45:40 +00:00
Erik Johnston
45b5fe9122
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push
2018-02-26 13:49:24 +00:00
Erik Johnston
d62ce972f8
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store
2018-02-23 11:46:24 +00:00
Erik Johnston
6ae9a3d2a6
Update copyright
2018-02-23 11:44:49 +00:00
Erik Johnston
a90c60912f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions
2018-02-23 11:26:31 +00:00
Erik Johnston
50e8657867
Merge pull request #2902 from matrix-org/erikj/split_events_store
...
Split out get_events and co into a worker store
2018-02-23 11:23:52 +00:00
Erik Johnston
1cf9e071dd
Merge pull request #2899 from matrix-org/erikj/split_pushers
...
Split PusherStore
2018-02-23 11:23:35 +00:00
Erik Johnston
d0957753bf
Merge pull request #2898 from matrix-org/erikj/split_push_rules_store
...
Split PushRulesStore
2018-02-23 11:23:23 +00:00
Erik Johnston
199dba6c15
Merge pull request #2897 from matrix-org/erikj/split_account_data
...
Split AccountDataStore and TagStore
2018-02-23 11:23:11 +00:00
Erik Johnston
70349872c2
Update copyright
2018-02-23 11:14:35 +00:00
Erik Johnston
eba93b05bf
Split EventsWorkerStore into separate file
2018-02-23 11:01:21 +00:00
Erik Johnston
bf8a36e080
Update copyright
2018-02-23 10:52:10 +00:00
Erik Johnston
5d0f665848
Remove redundant clock
2018-02-23 10:49:58 +00:00
Erik Johnston
3bd760628b
_event_persist_queue shouldn't be in worker store
2018-02-23 10:49:18 +00:00
Erik Johnston
eb9b5eec81
Update copyright
2018-02-23 10:42:39 +00:00
Erik Johnston
c2ecfcc3a4
Update copyright
2018-02-23 10:41:34 +00:00
Erik Johnston
7e6cf89dc2
Update copyright
2018-02-23 10:39:19 +00:00
Erik Johnston
26d37f7a63
Update copyright
2018-02-23 10:33:55 +00:00
Erik Johnston
bb73f55fc6
Use absolute imports
2018-02-23 10:31:16 +00:00
Erik Johnston
faeb369f15
Fix missing invalidations for receipt storage
2018-02-21 15:19:54 +00:00
Erik Johnston
3dec9c66b3
Split out RoomMemberStore
2018-02-21 12:07:26 +00:00
Erik Johnston
46244b2759
Split AS stores
2018-02-21 11:49:34 +00:00
Erik Johnston
27b094f382
Split out get_events and co into a worker store
2018-02-21 11:41:48 +00:00
Erik Johnston
573712da6b
Update comments
2018-02-21 11:29:49 +00:00
Erik Johnston
c96d547f4d
Actually use new param
2018-02-21 11:03:42 +00:00
Erik Johnston
d15d237b0d
Split out EventPushActionWorkerStore
2018-02-21 11:01:13 +00:00
Erik Johnston
27939cbb0e
Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed
...
Delete from push_actions_staging in federation too
2018-02-21 11:00:06 +00:00
Erik Johnston
6f72765371
Split PusherStore
2018-02-21 10:54:21 +00:00
Erik Johnston
cbaad969f9
Split PushRulesStore
2018-02-21 10:43:31 +00:00
Erik Johnston
ca9b9d9703
Split AccountDataStore and TagStore
2018-02-21 10:15:04 +00:00
Erik Johnston
8fbb4d0d19
Raise exception in abstract method
2018-02-20 17:59:23 +00:00
Erik Johnston
95e4cffd85
Fix comment
2018-02-20 17:58:40 +00:00
Erik Johnston
e316bbb4c0
Use abstract base class to access stream IDs
2018-02-20 17:43:57 +00:00
Erik Johnston
f5ac4dc2d4
Split ReceiptsStore
2018-02-20 16:28:28 +00:00
Erik Johnston
24087bffa9
Ensure all push actions are deleted from staging
2018-02-20 12:34:31 +00:00
Erik Johnston
ad0ccf15ea
Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts
2018-02-20 12:34:28 +00:00
Erik Johnston
d874d4f2d7
Delete from push_actions_staging in federation too
2018-02-20 11:37:52 +00:00
Erik Johnston
6ff8c87484
Batch inserts into event_push_actions_staging
2018-02-20 11:33:07 +00:00
Erik Johnston
324c3e9399
Merge pull request #2868 from matrix-org/erikj/refactor_media_storage
...
Make store_file use store_into_file
2018-02-20 11:31:24 +00:00
Pascal Bach
3acd616979
Update pynacl dependency to 1.2.1 or higher
...
Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>
2018-02-19 10:45:22 +01:00
Richard van der Hoff
d1a3325f99
(Really) fix tablescan of event_push_actions on purge
...
commit 278d21b5
2018-02-16 14:02:31 +00:00
Erik Johnston
6af025d3c4
Fix typo of double is_highlight
2018-02-16 11:35:31 +00:00
Erik Johnston
012e8e142a
Comments
2018-02-16 11:35:01 +00:00
Erik Johnston
b96278d6fe
Ensure that we delete staging push actions on errors
2018-02-15 15:47:06 +00:00
Erik Johnston
4810f7effd
Remove context.push_actions
2018-02-15 15:47:06 +00:00
Erik Johnston
c714c61853
Update event_push_actions table from staging table
2018-02-15 15:47:06 +00:00
Erik Johnston
acac21248c
Store push actions in staging area
2018-02-15 15:47:04 +00:00
Erik Johnston
106906a65e
Don't serialize current state over replication
2018-02-15 13:53:18 +00:00
Erik Johnston
5fb347fc41
Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log
...
Don't log errors propogated from send_event
2018-02-15 12:31:49 +00:00
Erik Johnston
fd1601c596
Fix state group storage bug in workers
...
We needed to move `_count_state_group_hops_txn` to the
StateGroupWorkerStore.
2018-02-15 11:04:32 +00:00
Erik Johnston
ef344b10e5
Don't log errors propogated from send_event
2018-02-15 11:03:49 +00:00
Richard van der Hoff
b8d821aa68
Merge pull request #2867 from matrix-org/rav/rework_purge
...
purge_history cleanups
2018-02-15 09:49:07 +00:00
Erik Johnston
92c52df702
Make store_file use store_into_file
2018-02-14 17:55:18 +00:00
Richard van der Hoff
d28ec43e15
Merge pull request #2769 from matrix-org/matthew/hit_the_gin
...
switch back from GIST to GIN indexes
2018-02-14 16:59:03 +00:00
Richard van der Hoff
39bf47319f
purge_history: fix sqlite syntax error
...
apparently sqlite insists on indexes being named
2018-02-14 16:42:19 +00:00
Richard van der Hoff
ac27f6a35e
purge_history: handle sqlite asshattery
...
apparently creating a temporary table commits the transaction. because that's a
useful thing.
2018-02-14 16:41:12 +00:00
Richard van der Hoff
5978dccff0
remove overzealous exception handling
2018-02-14 15:54:09 +00:00
Richard van der Hoff
278d21b5e4
purge_history: fix index use
...
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
2018-02-14 15:44:51 +00:00
Richard van der Hoff
5fcbf1e07c
Rework event purge to use a temporary table
...
... which should speed things up by reducing the amount of data being shuffled
across the connection
2018-02-14 11:02:22 +00:00
Erik Johnston
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker
...
Create a worker for event creation
2018-02-13 18:07:10 +00:00
Richard van der Hoff
d627174da2
Fix log message in purge_history
...
(we don't just remove remote events)
2018-02-13 16:51:21 +00:00
Richard van der Hoff
ddb6a79b68
Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin
2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3
move search reindex to schema 47
...
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.
This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
2018-02-13 16:44:46 +00:00
Richard van der Hoff
4a6d551704
GIN reindex: Fix syntax errors, improve exception handling
2018-02-13 16:44:46 +00:00
Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching
...
Use StateResolutionHandler to resolve state in persist_events
2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit
2018-02-13 14:29:22 +00:00
Richard van der Hoff
8fd1a32456
Fix typos in purge api & doc
...
* It's supposed to be purge_local_events, not ..._history
* Fix the doc to have valid json
2018-02-13 13:09:39 +00:00
Richard van der Hoff
4d09366656
Merge pull request #2695 from okurz/feature/allow_recent_pysaml
...
Allow use of higher versions of saml2
2018-02-13 12:24:08 +00:00
Richard van der Hoff
a9b712e9dc
Merge branch 'develop' into matthew/gin_work_mem
2018-02-13 12:16:01 +00:00
Erik Johnston
1026690cd2
Merge pull request #2857 from matrix-org/erikj/upload_store
...
Tell storage providers about new file so they can upload
2018-02-12 13:52:58 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates
...
delete_local_events for purge_room_history
2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496
purge: move room_depth update to end
...
... to avoid locking the table for too long
2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b
delete_local_events for purge_history
...
Add a flag which makes the purger delete local events
2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d
purge: Move cache invalidation to more appropriate place
...
it was a bit of a non-sequitur there
2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137
bump purge logging to info
...
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf
rename delete_old_state -> purge_history
...
(beacause it deletes more than state)
2018-02-09 13:07:41 +00:00
Erik Johnston
5fa571a91b
Tell storage providers about new file so they can upload
2018-02-07 13:35:08 +00:00
Erik Johnston
50fe92cd26
Move presence handling into handle_new_client_event
...
As we want to have it run on the main synapse instance
2018-02-07 10:34:09 +00:00
Erik Johnston
8ec2e638be
Add event_creator worker
2018-02-07 10:32:32 +00:00
Erik Johnston
24dd73028a
Add replication http endpoint for event sending
2018-02-07 10:32:32 +00:00
Erik Johnston
e3624fad5f
Remove pointless ratelimit check
...
The intention was for the check to be called as early as possible in the
request, but actually was called just before the main ratelimit check,
so was fairly pointless.
2018-02-07 10:30:25 +00:00
Erik Johnston
617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation
...
Split event creation into a separate handler
2018-02-06 17:01:17 +00:00
Erik Johnston
3e1e69ccaf
Update copyright
2018-02-06 16:40:38 +00:00
Erik Johnston
770b2252ca
s/_create_new_client_event/create_new_client_event/
2018-02-06 16:40:30 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events ( #2784 )
...
* Split state group persist into seperate storage func
* Add per database engine code for state group id gen
* Move store_state_group to StateReadStore
This allows other workers to use it, and so resolve state.
* Hook up store_state_group
* Fix tests
* Rename _store_mult_state_groups_txn
* Rename StateGroupReadStore
* Remove redundant _have_persisted_state_group_txn
* Update comments
* Comment compute_event_context
* Set start val for state_group_id_seq
... otherwise we try to recreate old state groups
* Update comments
* Don't store state for outliers
* Update comment
* Update docstring as state groups are ints
2018-02-06 14:31:24 +00:00
Richard van der Hoff
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta
...
Remove redundant return value from _calculate_state_delta
2018-02-05 17:42:20 +01:00
Richard van der Hoff
ebfe64e3d6
Use StateResolutionHandler to resolve state in persist events
...
... and thus benefit (hopefully) from its cache.
2018-02-05 16:23:26 +00:00
Richard van der Hoff
225dc3b4cb
Flatten _get_new_state_after_events
...
rejig the if statements to simplify the logic and reduce indentation
2018-02-05 16:23:25 +00:00
Richard van der Hoff
9fcbbe8e7d
Check that events being persisted have state_group
2018-02-05 16:23:25 +00:00
Richard van der Hoff
447aed42d2
Add event_map param to resolve_state_groups
2018-02-05 16:23:25 +00:00
Richard van der Hoff
ee6fb4cf85
Remove redundant return value from _calculate_state_delta
...
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
2018-02-05 16:23:20 +00:00
Richard van der Hoff
3c7b480ba3
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-05 16:12:14 +00:00
Erik Johnston
3fa362502c
Update places where we create events
2018-02-05 16:01:48 +00:00
Erik Johnston
5ff3d23564
Split event creation into a separate handler
2018-02-05 16:01:48 +00:00
Richard van der Hoff
c46e75d3d8
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-05 15:43:22 +00:00
Richard van der Hoff
db91e72ade
Merge pull request #2844 from matrix-org/rav/evicted_metrics
...
montoring metrics for number of cache evictions
2018-02-05 16:34:36 +01:00
Richard van der Hoff
bc496df192
report metrics on number of cache evictions
2018-02-05 15:34:01 +00:00
Erik Johnston
1f881e0746
Merge pull request #2791 from matrix-org/erikj/media_storage_refactor
...
Ensure media is in local cache before thumbnailing
2018-02-05 11:28:52 +00:00
Richard van der Hoff
80b8a28100
Factor out common code for search insert
...
we can reuse the same code as is used for event insert, for doing the
background index population.
2018-02-04 00:23:06 +00:00
Richard van der Hoff
bd25f9cf36
Clean up work_mem handling
...
Add some comments and improve exception handling when twiddling work_mem for
the search update
2018-02-03 23:05:41 +00:00
Richard van der Hoff
4eeae7ad65
Move store_event_search_txn to SearchStore
...
... as a precursor to making event storing and doing the bg update share some
code.
2018-02-03 22:59:45 +00:00
Richard van der Hoff
bb9f0f3cdb
Merge branch 'develop' into matthew/gin_work_mem
2018-02-03 22:40:28 +00:00
Richard van der Hoff
6b02fc80d1
Reinstate event_search_postgres_gist handler
...
People may have queued updates for this, so we can't just delete it.
2018-02-02 14:32:51 +00:00
Richard van der Hoff
9c9356512e
Merge pull request #2845 from matrix-org/rav/urlcache_error_handling
...
Handle url_previews with no content-type
2018-02-02 15:27:52 +01:00
Richard van der Hoff
18eae413af
Merge pull request #2842 from matrix-org/rav/state_resolution_handler
...
Factor out resolve_state_groups to a separate handler
2018-02-02 15:27:35 +01:00
Richard van der Hoff
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta
...
factor _get_new_state_after_events out of _calculate_state_delta
2018-02-02 15:27:15 +01:00
Richard van der Hoff
9dcd667ac2
Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring
...
Docstring fixes
2018-02-02 15:26:49 +01:00
Travis Ralston
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media
2018-02-01 18:05:47 -07:00
Richard van der Hoff
d5352cbba8
Handle url_previews with no content-type
...
avoid failing with an exception if the remote server doesn't give us a
Content-Type header.
Also, clean up the exception handling a bit.
2018-02-02 00:53:46 +00:00
Richard van der Hoff
14737ba495
doc arg types for _seperate
2018-02-01 12:41:34 +00:00
Richard van der Hoff
e15d4ea248
More docstring fixes
...
Fix a couple of errors in docstrings
2018-02-01 12:41:34 +00:00
Richard van der Hoff
a18828c129
Fix docstring for StateHandler.resolve_state_groups
...
The return type was a complete lie, so fix it
2018-02-01 12:41:34 +00:00
Richard van der Hoff
6da4c4d3bd
Factor out resolve_state_groups to a separate handler
...
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
2018-02-01 12:40:04 +00:00
Richard van der Hoff
0cbda53819
Rename resolve_state_groups -> resolve_state_groups_for_events
...
(to make way for a method that actually just does the state group resolution)
2018-02-01 12:40:00 +00:00
Richard van der Hoff
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media
...
Fix sql error in quarantine_media
2018-02-01 11:45:58 +01:00
Travis Ralston
e16e45b1b4
pep8
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 15:30:38 -07:00
Richard van der Hoff
e1e4ec9f9d
factor _get_new_state_after_events out of _calculate_state_delta
...
This reduces the scope of a bunch of variables
2018-01-31 21:32:09 +00:00
Richard van der Hoff
78e7e05188
Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail
...
Remove spurious log argument
2018-01-31 22:18:46 +01:00
Richard van der Hoff
ad48dfe73d
Merge pull request #2835 from matrix-org/rav/remove_event_type_param
...
Remove unused "event_type" param on state.get_current_state_ids
2018-01-31 22:17:57 +01:00
Richard van der Hoff
518a74586c
Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling
...
Improve exception handling in persist_event
2018-01-31 22:17:38 +01:00
Richard van der Hoff
d1fe4db882
Merge pull request #2833 from matrix-org/rav/pusher_hacks
...
Logging and metrics for the http pusher
2018-01-31 22:16:59 +01:00
Richard van der Hoff
421d68ca8c
Merge pull request #2817 from matrix-org/rav/http_conn_pool
...
Use a connection pool for the SimpleHttpClient
2018-01-31 22:14:22 +01:00
Travis Ralston
63c4383927
Documentation and naming
...
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-31 08:07:52 -07:00
Richard van der Hoff
af19f5e9aa
Remove spurious log argument
...
... which would cause scary-looking and unhelpful errors in the log on dns fail
2018-01-30 17:52:03 +00:00
Richard van der Hoff
773f0eed1e
Fix sql error in quarantine_media
2018-01-30 15:02:51 +00:00
Richard van der Hoff
adfc0c9539
docstring for get_current_state_ids
2018-01-29 17:39:55 +00:00
Richard van der Hoff
d413a2ba98
Remove unused "event_type" param on state.get_current_state_ids
...
this param doesn't seem to be used, and is a bit pointless anyway because it
can easily be replicated by the caller. It is also horrible, because it changes
the return type of the method.
2018-01-29 17:06:57 +00:00
Richard van der Hoff
b387ee17b6
Improve exception handling in persist_event
...
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.
2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
2018-01-29 17:05:33 +00:00
Richard van der Hoff
03dd745fe2
Better logging when pushes fail
2018-01-29 15:49:06 +00:00
Richard van der Hoff
e051abd20b
add appid/device_display_name to to pusher logging
2018-01-29 15:04:16 +00:00
Richard van der Hoff
02ba118f81
Increase http conn pool size
2018-01-29 14:30:15 +00:00
Richard van der Hoff
46022025ea
Fix SQL for user search
...
fix some syntax errors for user search when search_all_users is enabled
fixes #2801 , hopefully
2018-01-27 17:21:57 +00:00
Richard van der Hoff
4b0f06e99c
Merge pull request #2830 from matrix-org/rav/factor_out_get_conn
...
Factor out get_db_conn to HomeServer base class
2018-01-27 17:56:25 +01:00
Neil Johnson
e98f0f9112
Merge pull request #2827 from matrix-org/fix_server_500_on_public_rooms_call_when_no_rooms_exist
...
Fix server 500 on public rooms call when no rooms exist
2018-01-26 20:22:40 +00:00
Richard van der Hoff
25adde9a04
Factor out get_db_conn to HomeServer base class
...
This function is identical to all subclasses, so we may as well push it up to
the base class to reduce duplication (and make use of it in the tests)
2018-01-26 00:56:49 +00:00
Neil Johnson
73560237d6
add white space line
2018-01-26 00:15:10 +00:00
Neil Johnson
86c4f49a31
rather than try reconstruct the results object, better to guard against the xrange step argument being 0
2018-01-26 00:12:02 +00:00
Neil Johnson
f632083576
fix return type, should be a dict
2018-01-25 23:52:17 +00:00
Neil Johnson
6c6e197b0a
fix PEP8 violation
2018-01-25 23:47:46 +00:00
Neil Johnson
d02e43b15f
remove white space
2018-01-25 23:29:46 +00:00
Neil Johnson
349c739966
synapse 500s on a call to publicRooms in the case where the number of public rooms is zero, the specific cause is due to xrange trying to use a step value of zero, but if the total room number really is zero then it makes sense to just bail and save the extra processing
2018-01-25 23:28:44 +00:00
Matthew Hodgson
9a72b70630
fix thinko on 3pid whitelisting
2018-01-24 11:07:47 +01:00
Richard van der Hoff
25e2456ee7
Merge pull request #2816 from matrix-org/rav/metrics_comments
...
Add some comments about the reactor tick time metric
2018-01-23 19:39:26 +00:00
Matthew Hodgson
d32385336f
add ?ts massaging for ASes ( #2754 )
...
blindly implement ?ts for AS. untested
2018-01-23 09:59:06 +01:00
Richard van der Hoff
4528dd2443
Fix logging and add user_id
2018-01-22 20:15:42 +00:00
Richard van der Hoff
93efd7eb04
logging and debug for http pusher
2018-01-22 18:14:10 +00:00
Matthew Hodgson
ab9f844aaf
Add federation_domain_whitelist option ( #2820 )
...
Add federation_domain_whitelist
gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
2018-01-22 19:11:18 +01:00
Matthew Hodgson
d84f65255e
Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid
...
add registrations_require_3pid and allow_local_3pids
2018-01-22 13:57:22 +00:00
Travis Ralston
a94d9b6b82
Appease the linter
...
These are ids anyways, not mxc uris.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:49:46 -07:00
Travis Ralston
5552ed9a7f
Add an admin route to get all the media in a room
...
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.
Signed-off-by: Travis Ralston <travpc@gmail.com>
2018-01-20 22:37:53 -07:00
Richard van der Hoff
2c8526cac7
Use a connection pool for the SimpleHttpClient
...
In particular I hope this will help the pusher, which makes many requests to
sygnal, and is currently negotiating SSL for each one.
2018-01-20 00:55:44 +00:00
Richard van der Hoff
87b7d72760
Add some comments about the reactor tick time metric
2018-01-19 23:51:04 +00:00
Matthew Hodgson
49fce04624
fix typo (thanks sytest)
2018-01-19 19:55:38 +00:00
Richard van der Hoff
b0d9e633ee
Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs
...
Use the right path for url_preview thumbnails
2018-01-19 18:57:15 +00:00
Richard van der Hoff
ad7ec63d08
Use the right path for url_preview thumbnails
...
This was introduced by #2627 : we were overwriting the original media for url
previews with the thumbnails :/
(fixes https://github.com/vector-im/riot-web/issues/6012 , hopefully)
2018-01-19 18:29:39 +00:00
Matthew Hodgson
62d7d66ae5
oops, check all login types
2018-01-19 18:23:56 +00:00
Matthew Hodgson
8fe253f19b
fix PR nitpicking
2018-01-19 18:23:45 +00:00
Matthew Hodgson
293380bef7
trailing commas
2018-01-19 15:38:53 +00:00
Matthew Hodgson
447f4f0d5f
rewrite based on PR feedback:
...
* [ ] split config options into allowed_local_3pids and registrations_require_3pid
* [ ] simplify and comment logic for picking registration flows
* [ ] fix docstring and move check_3pid_allowed into a new util module
* [ ] use check_3pid_allowed everywhere
@erikjohnston PTAL
2018-01-19 15:33:55 +00:00
Matthew Hodgson
9d332e0f79
fix up v1, and improve errors
2018-01-19 00:53:58 +00:00
Matthew Hodgson
0af58f14ee
fix pep8
2018-01-19 00:33:51 +00:00
Matthew Hodgson
28a6ccb49c
add registrations_require_3pid
...
lets homeservers specify a whitelist for 3PIDs that users are allowed to associate with.
Typically useful for stopping people from registering with non-work emails
2018-01-19 00:19:58 +00:00
Erik Johnston
cd871a3057
Fix storage provider bug introduced when renamed to store_local
2018-01-18 18:37:59 +00:00
Erik Johnston
8ff6726c0d
Merge pull request #2812 from matrix-org/erikj/media_storage_provider_config
...
Make storage providers configurable
2018-01-18 18:33:57 +00:00
Erik Johnston
d69768348f
Fix passing wrong config to provider constructor
2018-01-18 17:14:05 +00:00
Erik Johnston
8e85220373
Remove duplicate directory test
2018-01-18 17:12:35 +00:00
Erik Johnston
3fe2bae857
Missing staticmethod
2018-01-18 17:11:45 +00:00
Erik Johnston
aae77da73f
Fixup comments
2018-01-18 17:11:29 +00:00
Erik Johnston
b6dc7044a9
Merge pull request #2804 from matrix-org/erikj/file_consumer
...
Add decent impl of a FileConsumer
2018-01-18 16:31:33 +00:00
Erik Johnston
9a89dae8c5
Fix typo in thumbnail resource causing access times to be incorrect
2018-01-18 15:06:24 +00:00
Erik Johnston
0af5dc63a8
Make storage providers more configurable
2018-01-18 14:07:21 +00:00
Richard van der Hoff
d57765fc8a
Fix bugs in block metrics
...
... which I introduced in #2785
2018-01-18 12:24:42 +00:00
Erik Johnston
2cf6a7bc20
Use better file consumer
2018-01-18 12:00:46 +00:00
Erik Johnston
4a53f3a3e8
Ensure media is in local cache before thumbnailing
2018-01-18 12:00:46 +00:00
Erik Johnston
be0dfcd4a2
Do logcontexts correctly
2018-01-18 11:57:57 +00:00
Erik Johnston
1432f7ccd5
Move test stuff to tests
2018-01-18 11:57:57 +00:00
Erik Johnston
2f18a2647b
Make all fields private
2018-01-18 11:57:54 +00:00
