Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-10-01 08:25:44 -04:00
Code Issues Packages Projects Releases Wiki Activity
10,120 Commits 4 Branches 609 Tags 60 MiB
e5b4a208ce
Commit Graph

7368 Commits

Author SHA1 Message Date
Richard van der Hoff
5978dccff0 remove overzealous exception handling 2018-02-14 15:54:09 +00:00
Richard van der Hoff
278d21b5e4 purge_history: fix index use 
event_push_actions doesn't have an index on event_id, so we need to specify
room_id.
 2018-02-14 15:44:51 +00:00
Richard van der Hoff
5fcbf1e07c Rework event purge to use a temporary table 
... which should speed things up by reducing the amount of data being shuffled
across the connection
 2018-02-14 11:02:22 +00:00
Erik Johnston
c0c9327fe0
Merge pull request #2854 from matrix-org/erikj/event_create_worker 
Create a worker for event creation
 2018-02-13 18:07:10 +00:00
Richard van der Hoff
d627174da2 Fix log message in purge_history 
(we don't just remove remote events)
 2018-02-13 16:51:21 +00:00
Richard van der Hoff
ddb6a79b68 Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin 2018-02-13 16:45:36 +00:00
Richard van der Hoff
0b27ae8dc3 move search reindex to schema 47 
We're up to schema v47 on develop now, so this will have to go in there to have
an effect.

This might cause an error if somebody has already run it in the v46 guise, and
runs it again in the v47 guise, because it will cause a duplicate entry in the
bbackground_updates table. On the other hand, the entry is removed once it is
complete, and it is unlikely that anyone other than matrix.org has run it on
v46. The update itself is harmless to re-run because it deliberately copes with
the index already existing.
 2018-02-13 16:44:46 +00:00
Richard van der Hoff
4a6d551704 GIN reindex: Fix syntax errors, improve exception handling 2018-02-13 16:44:46 +00:00
Richard van der Hoff
bfdf7b9237
Merge pull request #2864 from matrix-org/rav/persist_event_caching 
Use StateResolutionHandler to resolve state in persist_events
 2018-02-13 14:45:57 +00:00
Richard van der Hoff
630caf8a70
style nit 2018-02-13 14:29:22 +00:00
Richard van der Hoff
8fd1a32456 Fix typos in purge api & doc 
* It's supposed to be purge_local_events, not ..._history
* Fix the doc to have valid json
 2018-02-13 13:09:39 +00:00
Richard van der Hoff
4d09366656
Merge pull request #2695 from okurz/feature/allow_recent_pysaml 
Allow use of higher versions of saml2
 2018-02-13 12:24:08 +00:00
Richard van der Hoff
a9b712e9dc Merge branch 'develop' into matthew/gin_work_mem 2018-02-13 12:16:01 +00:00
Erik Johnston
1026690cd2
Merge pull request #2857 from matrix-org/erikj/upload_store 
Tell storage providers about new file so they can upload
 2018-02-12 13:52:58 +00:00
Richard van der Hoff
10b34dbb9a
Merge pull request #2858 from matrix-org/rav/purge_updates 
delete_local_events for purge_room_history
 2018-02-09 14:11:00 +00:00
Richard van der Hoff
39a6b35496 purge: move room_depth update to end 
... to avoid locking the table for too long
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
74fcbf741b delete_local_events for purge_history 
Add a flag which makes the purger delete local events
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
e571aef06d purge: Move cache invalidation to more appropriate place 
it was a bit of a non-sequitur there
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
61ffaa8137 bump purge logging to info 
this thing takes ages and the only sign of any progress is the logs, so having
some logs is useful.
 2018-02-09 13:07:41 +00:00
Richard van der Hoff
671540dccf rename delete_old_state -> purge_history 
(beacause it deletes more than state)
 2018-02-09 13:07:41 +00:00
Erik Johnston
5fa571a91b Tell storage providers about new file so they can upload 2018-02-07 13:35:08 +00:00
Erik Johnston
50fe92cd26 Move presence handling into handle_new_client_event 
As we want to have it run on the main synapse instance
 2018-02-07 10:34:09 +00:00
Erik Johnston
8ec2e638be Add event_creator worker 2018-02-07 10:32:32 +00:00
Erik Johnston
24dd73028a Add replication http endpoint for event sending 2018-02-07 10:32:32 +00:00
Erik Johnston
e3624fad5f Remove pointless ratelimit check 
The intention was for the check to be called as early as possible in the
request, but actually was called just before the main ratelimit check,
so was fairly pointless.
 2018-02-07 10:30:25 +00:00
Erik Johnston
617199d73d
Merge pull request #2847 from matrix-org/erikj/separate_event_creation 
Split event creation into a separate handler
 2018-02-06 17:01:17 +00:00
Erik Johnston
3e1e69ccaf Update copyright 2018-02-06 16:40:38 +00:00
Erik Johnston
770b2252ca s/_create_new_client_event/create_new_client_event/ 2018-02-06 16:40:30 +00:00
Erik Johnston
3d33eef6fc
Store state groups separately from events (#2784) 
* Split state group persist into seperate storage func

* Add per database engine code for state group id gen

* Move store_state_group to StateReadStore

This allows other workers to use it, and so resolve state.

* Hook up store_state_group

* Fix tests

* Rename _store_mult_state_groups_txn

* Rename StateGroupReadStore

* Remove redundant _have_persisted_state_group_txn

* Update comments

* Comment compute_event_context

* Set start val for state_group_id_seq

... otherwise we try to recreate old state groups

* Update comments

* Don't store state for outliers

* Update comment

* Update docstring as state groups are ints
 2018-02-06 14:31:24 +00:00
Richard van der Hoff
b31bf0bb51
Merge pull request #2849 from matrix-org/rav/clean_up_state_delta 
Remove redundant return value from _calculate_state_delta
 2018-02-05 17:42:20 +01:00
Richard van der Hoff
ebfe64e3d6 Use StateResolutionHandler to resolve state in persist events 
... and thus benefit (hopefully) from its cache.
 2018-02-05 16:23:26 +00:00
Richard van der Hoff
225dc3b4cb Flatten _get_new_state_after_events 
rejig the if statements to simplify the logic and reduce indentation
 2018-02-05 16:23:25 +00:00
Richard van der Hoff
9fcbbe8e7d Check that events being persisted have state_group 2018-02-05 16:23:25 +00:00
Richard van der Hoff
447aed42d2 Add event_map param to resolve_state_groups 2018-02-05 16:23:25 +00:00
Richard van der Hoff
ee6fb4cf85 Remove redundant return value from _calculate_state_delta 
we already have the state from _get_new_state_after_events, so returning it
from _calculate_state_delta is just confusing.
 2018-02-05 16:23:20 +00:00
Richard van der Hoff
3c7b480ba3 Factor out common code for search insert 
we can reuse the same code as is used for event insert, for doing the
background index population.
 2018-02-05 16:12:14 +00:00
Erik Johnston
3fa362502c Update places where we create events 2018-02-05 16:01:48 +00:00
Erik Johnston
5ff3d23564 Split event creation into a separate handler 2018-02-05 16:01:48 +00:00
Richard van der Hoff
c46e75d3d8 Move store_event_search_txn to SearchStore 
... as a precursor to making event storing and doing the bg update share some
code.
 2018-02-05 15:43:22 +00:00
Richard van der Hoff
db91e72ade
Merge pull request #2844 from matrix-org/rav/evicted_metrics 
montoring metrics for number of cache evictions
 2018-02-05 16:34:36 +01:00
Richard van der Hoff
bc496df192 report metrics on number of cache evictions 2018-02-05 15:34:01 +00:00
Erik Johnston
1f881e0746
Merge pull request #2791 from matrix-org/erikj/media_storage_refactor 
Ensure media is in local cache before thumbnailing
 2018-02-05 11:28:52 +00:00
Richard van der Hoff
80b8a28100 Factor out common code for search insert 
we can reuse the same code as is used for event insert, for doing the
background index population.
 2018-02-04 00:23:06 +00:00
Richard van der Hoff
bd25f9cf36 Clean up work_mem handling 
Add some comments and improve exception handling when twiddling work_mem for
the search update
 2018-02-03 23:05:41 +00:00
Richard van der Hoff
4eeae7ad65 Move store_event_search_txn to SearchStore 
... as a precursor to making event storing and doing the bg update share some
code.
 2018-02-03 22:59:45 +00:00
Richard van der Hoff
bb9f0f3cdb Merge branch 'develop' into matthew/gin_work_mem 2018-02-03 22:40:28 +00:00
Richard van der Hoff
6b02fc80d1 Reinstate event_search_postgres_gist handler 
People may have queued updates for this, so we can't just delete it.
 2018-02-02 14:32:51 +00:00
Richard van der Hoff
9c9356512e
Merge pull request #2845 from matrix-org/rav/urlcache_error_handling 
Handle url_previews with no content-type
 2018-02-02 15:27:52 +01:00
Richard van der Hoff
18eae413af
Merge pull request #2842 from matrix-org/rav/state_resolution_handler 
Factor out resolve_state_groups to a separate handler
 2018-02-02 15:27:35 +01:00
Richard van der Hoff
78d6ddba86
Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta 
factor _get_new_state_after_events out of _calculate_state_delta
 2018-02-02 15:27:15 +01:00
Richard van der Hoff
9dcd667ac2
Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring 
Docstring fixes
 2018-02-02 15:26:49 +01:00
Travis Ralston
6e87b34f7b
Merge branch 'develop' into travis/admin-list-media 2018-02-01 18:05:47 -07:00
Richard van der Hoff
d5352cbba8 Handle url_previews with no content-type 
avoid failing with an exception if the remote server doesn't give us a
Content-Type header.

Also, clean up the exception handling a bit.
 2018-02-02 00:53:46 +00:00
Richard van der Hoff
14737ba495 doc arg types for _seperate 2018-02-01 12:41:34 +00:00
Richard van der Hoff
e15d4ea248 More docstring fixes 
Fix a couple of errors in docstrings
 2018-02-01 12:41:34 +00:00
Richard van der Hoff
a18828c129 Fix docstring for StateHandler.resolve_state_groups 
The return type was a complete lie, so fix it
 2018-02-01 12:41:34 +00:00
Richard van der Hoff
6da4c4d3bd Factor out resolve_state_groups to a separate handler 
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
 2018-02-01 12:40:04 +00:00
Richard van der Hoff
0cbda53819 Rename resolve_state_groups -> resolve_state_groups_for_events 
(to make way for a method that actually just does the state group resolution)
 2018-02-01 12:40:00 +00:00
Richard van der Hoff
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media 
Fix sql error in quarantine_media
 2018-02-01 11:45:58 +01:00
Travis Ralston
e16e45b1b4 pep8 
Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-31 15:30:38 -07:00
Richard van der Hoff
e1e4ec9f9d factor _get_new_state_after_events out of _calculate_state_delta 
This reduces the scope of a bunch of variables
 2018-01-31 21:32:09 +00:00
Richard van der Hoff
78e7e05188
Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail 
Remove spurious log argument
 2018-01-31 22:18:46 +01:00
Richard van der Hoff
ad48dfe73d
Merge pull request #2835 from matrix-org/rav/remove_event_type_param 
Remove unused "event_type" param on state.get_current_state_ids
 2018-01-31 22:17:57 +01:00
Richard van der Hoff
518a74586c
Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling 
Improve exception handling in persist_event
 2018-01-31 22:17:38 +01:00
Richard van der Hoff
d1fe4db882
Merge pull request #2833 from matrix-org/rav/pusher_hacks 
Logging and metrics for the http pusher
 2018-01-31 22:16:59 +01:00
Richard van der Hoff
421d68ca8c
Merge pull request #2817 from matrix-org/rav/http_conn_pool 
Use a connection pool for the SimpleHttpClient
 2018-01-31 22:14:22 +01:00
Travis Ralston
63c4383927 Documentation and naming 
Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-31 08:07:52 -07:00
Richard van der Hoff
af19f5e9aa Remove spurious log argument 
... which would cause scary-looking and unhelpful errors in the log on dns fail
 2018-01-30 17:52:03 +00:00
Richard van der Hoff
773f0eed1e Fix sql error in quarantine_media 2018-01-30 15:02:51 +00:00
Richard van der Hoff
adfc0c9539 docstring for get_current_state_ids 2018-01-29 17:39:55 +00:00
Richard van der Hoff
d413a2ba98 Remove unused "event_type" param on state.get_current_state_ids 
this param doesn't seem to be used, and is a bit pointless anyway because it
can easily be replicated by the caller. It is also horrible, because it changes
the return type of the method.
 2018-01-29 17:06:57 +00:00
Richard van der Hoff
b387ee17b6 Improve exception handling in persist_event 
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.

2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
 2018-01-29 17:05:33 +00:00
Richard van der Hoff
03dd745fe2 Better logging when pushes fail 2018-01-29 15:49:06 +00:00
Richard van der Hoff
e051abd20b add appid/device_display_name to to pusher logging 2018-01-29 15:04:16 +00:00
Richard van der Hoff
02ba118f81 Increase http conn pool size 2018-01-29 14:30:15 +00:00
Richard van der Hoff
46022025ea Fix SQL for user search 
fix some syntax errors for user search when search_all_users is enabled

fixes #2801, hopefully
 2018-01-27 17:21:57 +00:00
Richard van der Hoff
4b0f06e99c
Merge pull request #2830 from matrix-org/rav/factor_out_get_conn 
Factor out get_db_conn to HomeServer base class
 2018-01-27 17:56:25 +01:00
Neil Johnson
e98f0f9112
Merge pull request #2827 from matrix-org/fix_server_500_on_public_rooms_call_when_no_rooms_exist 
Fix server 500 on public rooms call when no rooms exist
 2018-01-26 20:22:40 +00:00
Richard van der Hoff
25adde9a04 Factor out get_db_conn to HomeServer base class 
This function is identical to all subclasses, so we may as well push it up to
the base class to reduce duplication (and make use of it in the tests)
 2018-01-26 00:56:49 +00:00
Neil Johnson
73560237d6 add white space line 2018-01-26 00:15:10 +00:00
Neil Johnson
86c4f49a31 rather than try reconstruct the results object, better to guard against the xrange step argument being 0 2018-01-26 00:12:02 +00:00
Neil Johnson
f632083576 fix return type, should be a dict 2018-01-25 23:52:17 +00:00
Neil Johnson
6c6e197b0a fix PEP8 violation 2018-01-25 23:47:46 +00:00
Neil Johnson
d02e43b15f remove white space 2018-01-25 23:29:46 +00:00
Neil Johnson
349c739966 synapse 500s on a call to publicRooms in the case where the number of public rooms is zero, the specific cause is due to xrange trying to use a step value of zero, but if the total room number really is zero then it makes sense to just bail and save the extra processing 2018-01-25 23:28:44 +00:00
Matthew Hodgson
9a72b70630 fix thinko on 3pid whitelisting 2018-01-24 11:07:47 +01:00
Richard van der Hoff
25e2456ee7
Merge pull request #2816 from matrix-org/rav/metrics_comments 
Add some comments about the reactor tick time metric
 2018-01-23 19:39:26 +00:00
Matthew Hodgson
d32385336f
add ?ts massaging for ASes (#2754) 
blindly implement ?ts for AS. untested
 2018-01-23 09:59:06 +01:00
Richard van der Hoff
4528dd2443 Fix logging and add user_id 2018-01-22 20:15:42 +00:00
Richard van der Hoff
93efd7eb04 logging and debug for http pusher 2018-01-22 18:14:10 +00:00
Matthew Hodgson
ab9f844aaf
Add federation_domain_whitelist option (#2820) 
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
 2018-01-22 19:11:18 +01:00
Matthew Hodgson
d84f65255e
Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid 
add registrations_require_3pid and allow_local_3pids
 2018-01-22 13:57:22 +00:00
Travis Ralston
a94d9b6b82 Appease the linter 
These are ids anyways, not mxc uris.

Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-20 22:49:46 -07:00
Travis Ralston
5552ed9a7f Add an admin route to get all the media in a room 
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.

Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-20 22:37:53 -07:00
Richard van der Hoff
2c8526cac7 Use a connection pool for the SimpleHttpClient 
In particular I hope this will help the pusher, which makes many requests to
sygnal, and is currently negotiating SSL for each one.
 2018-01-20 00:55:44 +00:00
Richard van der Hoff
87b7d72760 Add some comments about the reactor tick time metric 2018-01-19 23:51:04 +00:00
Matthew Hodgson
49fce04624 fix typo (thanks sytest) 2018-01-19 19:55:38 +00:00
Richard van der Hoff
b0d9e633ee
Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs 
Use the right path for url_preview thumbnails
 2018-01-19 18:57:15 +00:00
Richard van der Hoff
ad7ec63d08 Use the right path for url_preview thumbnails 
This was introduced by #2627: we were overwriting the original media for url
previews with the thumbnails :/

(fixes https://github.com/vector-im/riot-web/issues/6012, hopefully)
 2018-01-19 18:29:39 +00:00
Matthew Hodgson
62d7d66ae5 oops, check all login types 2018-01-19 18:23:56 +00:00