Matrix-Mirrors/synapse-product
1
0
Fork 0
mirror of https://git.anonymousland.org/anonymousland/synapse-product.git synced 2024-12-24 05:19:22 -05:00
Code Issues Packages Projects Releases Wiki Activity
11,480 Commits 4 Branches 609 Tags 60 MiB
2e9c73e8ca
Commit Graph

8336 Commits

Author SHA1 Message Date
Matthew Hodgson
eb1d911ab7 rather than adding ll_ids, remove them from p_ids 2018-07-24 13:40:49 +01:00
Matthew Hodgson
d19fba3655 Merge branch 'develop' into matthew/filter_members 2018-07-24 12:39:54 +01:00
Matthew Hodgson
cd241d6bda incorporate more review 2018-07-24 12:39:40 +01:00
Richard van der Hoff
30bfed5aa5 Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room 2018-07-24 11:46:09 +01:00
Erik Johnston
97acd385a3 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/speed_up_calculate_state_delta 2018-07-24 11:32:13 +01:00
Erik Johnston
0fa73e4a63 Remove unnecessary if 2018-07-24 11:19:23 +01:00
Erik Johnston
ff5426f6b8 Speed up _calculate_state_delta 2018-07-24 10:55:11 +01:00
Richard van der Hoff
a678145010
Merge branch 'develop' into rav/logcontext_fixes 2018-07-24 10:43:30 +01:00
Erik Johnston
d436ad332c
Merge pull request #3555 from matrix-org/erikj/client_apis_move 
Make client_reader support some more read only APIs
 2018-07-24 10:42:28 +01:00
Erik Johnston
536bc63a4e
Merge branch 'develop' into erikj/client_apis_move 2018-07-24 09:57:05 +01:00
Richard van der Hoff
cf2d15c6a9 another couple of logcontext leaks 2018-07-24 00:57:48 +01:00
Richard van der Hoff
8dff6e0322 Logcontext fixes 
Fix some random logcontext leaks.
 2018-07-24 00:37:17 +01:00
Richard van der Hoff
69fb5dbdab fix idiocy 2018-07-24 00:04:44 +01:00
Richard van der Hoff
1938cffaea Add some measure blocks to persist_events 
... to help us figure out where 40% of CPU is going
 2018-07-23 23:48:19 +01:00
Matthew Hodgson
efcdacad7d handle case where types is [] on postgres correctly 2018-07-23 22:41:05 +01:00
Richard van der Hoff
ce0c18dec5 Improve logging for exceptions handling PDUs 
when we get an exception handling a federation PDU, log the whole stacktrace.
 2018-07-23 22:13:19 +01:00
Richard van der Hoff
c1f80effbe Handle delta_ids being None in _update_context_for_auth_events 
it's easier to create the new state group as a delta from the existing one.

(There's an outside chance this will help with
https://github.com/matrix-org/synapse/issues/3364)
 2018-07-23 22:06:50 +01:00
Matthew Hodgson
adfe29ec0b Merge branch 'develop' into matthew/filter_members 2018-07-23 19:21:37 +01:00
Matthew Hodgson
254fb430d1 incorporate review 2018-07-23 19:21:20 +01:00
Richard van der Hoff
5c705f70c9 Fixes and optimisations for resolve_state_groups 
First of all, fix the logic which looks for identical input state groups so
that we actually use them. This turned out to be most easily done by factoring
the relevant code out to a separate function so that we could do an early
return.

Secondly, avoid building the whole `conflicted_state` dict (which was only ever
used as a boolean flag).

Thirdly, replace the construction of the `state` dict (which mapped from keys
to events that set them), with an optimistic construction of the resolution
result assuming there will be no conflicts. This should be no slower than
building the old `state` dict, and:
  - in the conflicted case, we'll short-cut it, saving part of the work
  - in the unconflicted case, it saves rebuilding the resolution from the
    `state` dict.

Finally, do a couple of s/values/itervalues/.
 2018-07-23 19:10:11 +01:00
Erik Johnston
8b9f164fff Comments 2018-07-23 17:43:01 +01:00
Erik Johnston
50c60e5fad Only get cached state from context in persist_event 
We don't want to bother pulling out the current state from the DB since
until we know we have to. Checking the context for state is just an
optimisation.
 2018-07-23 17:21:40 +01:00
Richard van der Hoff
4f5cc8e4e7 Merge remote-tracking branch 'origin/develop' into rav/remove_who_forgot_in_room 2018-07-23 17:15:12 +01:00
Richard van der Hoff
dae6dc1e77 Remove redundant checks on room forgottenness 
Fixes #3550
 2018-07-23 17:13:34 +01:00
Erik Johnston
0faa3223cd Fix missing attributes on workers. 
This was missed during the transition from attribute to getter for
getting state from context.
 2018-07-23 16:28:00 +01:00
Erik Johnston
999bcf9d01 Fix EventContext when using workers 
We were:
  1. Not correctly setting all attributes
  2. Using defer.inlineCallbacks in a non-generator
 2018-07-23 15:24:21 +01:00
Erik Johnston
4797ed000e Update docstrings to make sense 2018-07-23 15:05:56 +01:00
Erik Johnston
0b0b24cb82 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/client_apis_move 2018-07-23 13:21:15 +01:00
Erik Johnston
027bc01a1b Add support for updating state 2018-07-23 13:17:25 +01:00
Erik Johnston
e42510ba63 Use new getters 2018-07-23 13:17:22 +01:00
Erik Johnston
440b8845b5 Make EventContext lazy load state 2018-07-23 12:56:56 +01:00
Erik Johnston
842cdece42 pep8 2018-07-23 12:31:35 +01:00
Erik Johnston
acbfdc3442 Refcator EventContext to accept state during init 2018-07-23 12:20:55 +01:00
Matthew Hodgson
9b34f3ea3a
Merge branch 'develop' into matthew/sync_deleted_devices 2018-07-23 10:03:28 +01:00
Matthew Hodgson
c1bf2b587e
add trailing comma 2018-07-23 09:56:23 +01:00
Amber Brown
3132b89f12
Make the rest of the .iterwhatever go away (#3562) 2018-07-21 15:47:18 +10:00
Erik Johnston
5c88bb722f Move PaginationHandler to its own file 2018-07-20 15:32:23 +01:00
Erik Johnston
0ecf68aedc Move check_in_room_or_world_readable to Auth 2018-07-20 15:30:59 +01:00
Richard van der Hoff
3d6df84658 Test and fix support for cancellation in Linearizer 2018-07-20 13:59:55 +01:00
Richard van der Hoff
4f67623674
Merge pull request #3571 from matrix-org/rav/limiter_fixes 
A set of improvements to the Limiter
 2018-07-20 13:53:03 +01:00
Amber Brown
e1a237eaab
Admin API for creating new users (#3415) 2018-07-20 22:41:13 +10:00
Richard van der Hoff
7c712f95bb Combine Limiter and Linearizer 
Linearizer was effectively a Limiter with max_count=1, so rather than
maintaining two sets of code, let's combine them.
 2018-07-20 13:11:43 +01:00
Richard van der Hoff
8462c26485 Improvements to the Limiter 
* give them names, to improve logging
* use a deque rather than a list for efficiency
 2018-07-20 12:50:27 +01:00
Richard van der Hoff
d7275eecf3 Add a sleep to the Limiter to fix stack overflows. 
Fixes #3570
 2018-07-20 12:37:12 +01:00
Matthew Hodgson
2f558300cc fix thinkos; unbreak tests 2018-07-19 19:22:27 +01:00
Matthew Hodgson
bcaec2915a incorporate review 2018-07-19 19:03:50 +01:00
Matthew Hodgson
924eb34d94 add a filtered_types param to limit filtering to specific types 2018-07-19 18:32:02 +01:00
Amber Brown
9a8acdc720 Merge branch 'master' into develop 2018-07-19 21:16:44 +10:00
Amber Brown
38f53399a2 0.33 final 2018-07-19 21:11:40 +10:00
Amber Brown
95ccb6e2ec
Don't spew errors because we can't save metrics (#3563) 2018-07-19 20:58:18 +10:00
Matthew Hodgson
be3adfc331 merge develop pydoc for _get_state_for_groups 2018-07-19 11:26:04 +01:00
Matthew Hodgson
9e40834f74 yes, we do need to invalidate the device_id_exists_cache when deleting a remote device 2018-07-19 11:15:10 +01:00
Richard van der Hoff
f1a15ea206 revert 00bc979 
... we've fixed the things that caused the warnings, so we should reinstate the
warning.
 2018-07-19 11:14:20 +01:00
Richard van der Hoff
1ffb7bec20 Merge remote-tracking branch 'origin/release-v0.33.0' into develop 2018-07-19 11:12:33 +01:00
Richard van der Hoff
c754e006f4
Merge pull request #3556 from matrix-org/rav/background_processes 
Run things as background processes
 2018-07-19 11:04:18 +01:00
Amber Brown
a97c845271
Move v1-only APIs into their own module & isolate deprecated ones (#3460) 2018-07-19 20:03:33 +10:00
Matthew Hodgson
c0685f67c0 spell out that include_deleted_devices requires include_all_devices 2018-07-19 10:59:02 +01:00
Richard van der Hoff
00bc979137 Disable logcontext warning 
Temporary workaround to #3518 while we release 0.33.0.
 2018-07-19 10:51:15 +01:00
Erik Johnston
6f62a6ef21
Merge pull request #3554 from matrix-org/erikj/response_metrics_code 
Add response code to response timer metrics
 2018-07-19 10:25:18 +01:00
Richard van der Hoff
8c69b735e3 Make Distributor run its processes as a background process 
This is more involved than it might otherwise be, because the current
implementation just drops its logcontexts and runs everything in the sentinel
context.

It turns out that we aren't actually using a bunch of the functionality here
(notably suppress_failures and the fact that Distributor.fire returns a
deferred), so the easiest way to fix this is actually by simplifying a bunch of
code.
 2018-07-18 20:55:05 +01:00
Richard van der Hoff
667fba68f3 Run things as background processes 
This fixes #3518, and ensures that we get useful logs and metrics for lots of
things that happen in the background.

(There are certainly more things that happen in the background; these are just
the common ones I've found running a single-process synapse locally).
 2018-07-18 20:55:05 +01:00
Erik Johnston
9b596177ae Add some room read only APIs to client_reader 2018-07-18 15:33:03 +01:00
Erik Johnston
bacdf0cbf9 Move RoomContextHandler out of Handlers 
This is in preparation for moving GET /context/ to a worker
 2018-07-18 15:33:03 +01:00
Erik Johnston
8cb8df55e9 Split MessageHandler into read only and writers 
This will let us call the read only parts from workers, and so be able
to move some APIs off of master, e.g. the `/state` API.
 2018-07-18 15:33:03 +01:00
Richard van der Hoff
65d6a0e477
Merge pull request #3553 from matrix-org/rav/background_process_tracking 
Resource tracking for background processes
 2018-07-18 14:27:50 +01:00
Erik Johnston
5bd0a47fcd pep8 2018-07-18 14:19:00 +01:00
Erik Johnston
e45a46b6e4 Add response code to response timer metrics 2018-07-18 13:59:36 +01:00
Richard van der Hoff
dab00faa83
Merge pull request #3367 from matrix-org/rav/drop_re_signing_hacks 
Remove event re-signing hacks
 2018-07-18 12:46:27 +01:00
David Baker
c91a44572e
Merge pull request #3514 from matrix-org/dbkr/turn_dont_add_defaults 
Comment dummy TURN parameters in default config
 2018-07-18 11:56:18 +01:00
Richard van der Hoff
6e3fc657b4 Resource tracking for background processes 
This introduces a mechanism for tracking resource usage by background
processes, along with an example of how it will be used.

This will help address #3518, but more importantly will give us better insights
into things which are happening but not being shown up by the request metrics.

We *could* do this with Measure blocks, but:
 - I think having them pulled out as a completely separate metric class will
   make it easier to distinguish top-level processes from those which are
   nested.

 - I want to be able to report on in-flight background processes, and I don't
   think we want to do this for *all* Measure blocks.
 2018-07-18 10:50:33 +01:00
Amber Brown
5f3d02f6eb bump to 0.33.0rc1 2018-07-18 12:52:56 +10:00
Richard van der Hoff
79eb339c66 add a comment 2018-07-17 14:53:34 +01:00
Richard van der Hoff
d897be6a98 Fix visibility of events from erased users over federation 2018-07-17 14:02:07 +01:00
Richard van der Hoff
9c04b4abf9
Merge pull request #3541 from matrix-org/rav/optimize_filter_events_for_server 
Refactor and optimze filter_events_for_server
 2018-07-17 14:01:39 +01:00
Richard van der Hoff
94440ae994 fix imports 2018-07-17 11:51:26 +01:00
Amber Brown
bc006b3c9d
Refactor REST API tests to use explicit reactors (#3351) 2018-07-17 20:43:18 +10:00
Richard van der Hoff
2172a3d8cb add a comment 2018-07-17 11:13:57 +01:00
Erik Johnston
b2aa05a8d6 Use efficient .intersection 2018-07-17 11:07:04 +01:00
Erik Johnston
547b1355d3 Fix perf regression in PR #3530 
The get_entities_changed function was changed to return all changed
entities since the given stream position, rather than only those changed
from a given list of entities. This resulted in the function incorrectly
returning large numbers of entities that, for example, caused large
increases in database usage.
 2018-07-17 10:27:51 +01:00
Amber Brown
3fe0938b76
Merge pull request #3530 from matrix-org/erikj/stream_cache 
Don't return unknown entities in get_entities_changed
 2018-07-17 13:44:46 +10:00
Richard van der Hoff
09e29fb58b Attempt to make _filter_events_for_server more efficient 2018-07-16 14:06:09 +01:00
Krombel
78a9ddcf9a rerun isort with latest version 2018-07-16 14:23:25 +02:00
Richard van der Hoff
ea69d35651 Move filter_events_for_server out of FederationHandler 
for easier unit testing.
 2018-07-16 13:06:24 +01:00
Krombel
4a27000548 check isort by travis 2018-07-16 13:57:33 +02:00
Jeroen
505530f36a Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests 
# Conflicts:
#	synapse/crypto/context_factory.py
 2018-07-14 20:24:46 +02:00
Amber Brown
8a4f05fefb
Fix develop because I broke it :( (#3535) 2018-07-14 09:51:00 +10:00
Amber Brown
8532953c04
Merge pull request #3534 from krombel/use_parse_and_asserts_from_servlet 
Use parse and asserts from http.servlet
 2018-07-14 09:09:19 +10:00
Amber Brown
a2374b2c7f
fix sytests 2018-07-14 07:52:58 +10:00
Amber Brown
33b60c01b5
Make auth & transactions more testable (#3499) 2018-07-14 07:34:49 +10:00
Krombel
516f960ad8 add changelog 2018-07-13 22:19:19 +02:00
Krombel
3366b9c534 rename assert_params_in_request to assert_params_in_dict 
the method "assert_params_in_request" does handle dicts and not
requests. A request body has to be parsed to json before this method
can be used
 2018-07-13 21:53:01 +02:00
Krombel
32fd6910d0 Use parse_{int,str} and assert from http.servlet 
parse_integer and parse_string can take a request and raise errors
in case we have wrong or missing params.
This PR tries to use them more to deduplicate some code and make it
better readable
 2018-07-13 21:40:14 +02:00
Richard van der Hoff
2aba1f549c
Merge pull request #3533 from matrix-org/rav/fix_federation_ratelimite_queue 
Make FederationRateLimiter queue requests properly
 2018-07-13 16:59:18 +01:00
Richard van der Hoff
33b40d0a25 Make FederationRateLimiter queue requests properly 
popitem removes the *most recent* item by default [1]. We want the oldest.

Fixes #3524

[1]: https://docs.python.org/2/library/collections.html#collections.OrderedDict.popitem
 2018-07-13 16:19:40 +01:00
Erik Johnston
77b692e65d Don't return unknown entities in get_entities_changed 
The stream cache keeps track of all entities that have changed since
a particular stream position, so get_entities_changed does not need to
return unknown entites when given a larger stream position.

This makes it consistent with the behaviour of has_entity_changed.
 2018-07-13 15:26:10 +01:00
Matthew Hodgson
ba22b6a456 typo 2018-07-13 12:03:39 +01:00
Richard van der Hoff
6dff49b8a9
Merge pull request #3521 from matrix-org/rav/optimise_stream_change_cache 
Reduce set building in get_entities_changed
 2018-07-12 12:08:49 +01:00
Matthew Hodgson
12ec58301f shift to using an explicit deleted flag on m.device_list_update EDUs 
and generally make it work.
 2018-07-12 11:39:43 +01:00
Richard van der Hoff
fa5c2bc082 Reduce set building in get_entities_changed 
This line shows up as about 5% of cpu time on a synchrotron:

    not_known_entities = set(entities) - set(self._entity_to_key)

Presumably the problem here is that _entity_to_key can be largeish, and
building a set for its keys every time this function is called is slow.

Here we rewrite the logic to avoid building so many sets.
 2018-07-12 11:37:44 +01:00
Richard van der Hoff
482d17b58b Merge branch 'develop' into rav/enforce_report_api 2018-07-12 09:56:28 +01:00