Commit Graph

713 Commits

Author SHA1 Message Date
Jorik Schellekens
1d65292e94 Link the send loop with the edus contexts
The contexts were being filtered too early so  the send loop wasn't
being linked to them unless the destination
was whitelisted.
2019-09-05 14:42:37 +01:00
Andrew Morgan
4548d1f87e
Remove unnecessary parentheses around return statements (#5931)
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
reivilibre
7ccc251415
Merge pull request #5859 from matrix-org/rei/msc2197
MSC2197 Search Filters over Federation
2019-08-28 09:00:21 +01:00
Jorik Schellekens
8767b63a82
Propagate opentracing contexts through EDUs (#5852)
Propagate opentracing contexts through EDUs
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-22 18:21:10 +01:00
Jorik Schellekens
812ed6b0d5
Opentracing across workers (#5771)
Propagate opentracing contexts across workers


Also includes some Convenience modifications to opentracing for servlets, notably:
- Add boolean to skip the whitelisting check on inject
  extract methods. - useful when injecting into carriers
  locally. Otherwise we'd always have to include our
  own servername and whitelist our servername
- start_active_span_from_request instead of header
- Add boolean to decide whether to extract context
  from a request to a servlet
2019-08-22 18:08:07 +01:00
Jorik Schellekens
9a6f2be572
Opentrace e2e keys (#5855)
Add opentracing tags and logs for e2e keys
2019-08-22 11:28:12 +01:00
Olivier Wilkinson (reivilibre)
bb29bc2937 Use MSC2197 on stable prefix as it has almost finished FCP
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-20 08:49:31 +01:00
Jorik Schellekens
87fa26006b
Opentracing misc (#5856)
Add authenticated_entity and servlet_names tags.

Functionally:
- Add a tag for authenticated_entity
- Add a tag for servlet_names

Stylistically:
Moved to importing methods directly from opentracing.
2019-08-16 16:13:25 +01:00
Olivier Wilkinson (reivilibre)
2253b083d9 Add support for inbound MSC2197 requests on unstable Federation API
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-15 11:06:21 +01:00
Olivier Wilkinson (reivilibre)
6fadb560fc Support MSC2197 outbound with unstable prefix
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-15 10:59:37 +01:00
Olivier Wilkinson (reivilibre)
1b63ccd848 Wrap get_local_public_room_list call in maybeDeferred because it
is cached and so does not always return a `Deferred`.
`await` does not silently pass-through non-Deferreds like `yield` used to.

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-14 14:53:49 +01:00
Erik Johnston
a8f40a8302 Return 502 not 500 when failing to reach any remote server. 2019-08-01 13:47:31 +01:00
Amber Brown
865077f1d1
Room Complexity Client Implementation (#5783) 2019-07-30 02:47:27 +10:00
Richard van der Hoff
1a93daf353
Merge pull request #5744 from matrix-org/erikj/log_leave_origin_mismatch
Log when we receive a /make_* request from a different origin
2019-07-26 12:38:37 +01:00
Richard van der Hoff
d1020653fc Log when we receive a /make_* request from a different origin 2019-07-26 10:08:22 +01:00
Andrew Morgan
baf081cd3b Bugfixes
--------
 
 - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl04Ur0THGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9F4oD/0TY6S/SEd2uAmzor64ojmbX5BOwPzf
 j/wzUTrfvuf40EvkNPDpnejNZSvy/ysbaGQaQusv0SQKlV3xrvdn4RuMvnOWVWck
 kBsO+lvzOaUTR0KHDxN4y9F5eI2NdPbub4847PPVzyqSIHAd+kolxXS8kSBBhwpL
 yfaICWV/AOy5L7xN+JZ9IQpnegVAvUj5DmgXzDHd6VdeiHDVJuARaBgrR5uCkwVS
 ZoLRqZ95XV/qiguMAUvPOwyEqht2mwO64989MswP16YYm8oMkB5QA6I5nYnACsTP
 qk9YcN/oNvEfQXUhttku6MxK1/4yUMPUhEoDBDH7ebc0440QDtWN+IHTdA6oPVZB
 IuStL9YGY16m7Ltx37ZUA4URfNMiSeLHo3zKc/mCAcwxN4HyOjJewtxbG5zKQAOZ
 SMs8UcDwGR4zL1hnt8ZDNYtWwfzJBQIdGjoHvjXJEY7/1csTv2lmAwewFTXiqSAr
 30GW5ews94kotqBK53zZT6V0F5gHNqgGHniOz1ZpqLLxYLqO3LSAGe97CrqlWUdX
 GkhA9tZyweknociD9fyyBmKdcFJ4mL4a+oGI5CMnSMph8UvCY8Y5XMb1T+iYEABI
 tA9G3mBvgkLPj+5V+8QggNkBafSigW2Q4FX7enGsDmiiskZOtfeKrAcVkapD4ooi
 3I7IW5aetZr2IQ==
 =+JBn
 -----END PGP SIGNATURE-----

Merge tag 'v1.2.0rc2' into develop

Bugfixes
--------

- Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
2019-07-24 13:47:51 +01:00
Jorik Schellekens
cf2972c818
Fix servlet metric names (#5734)
* Fix servlet metric names

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Remove redundant check

* Cover all return paths
2019-07-24 13:07:35 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Richard van der Hoff
fa8271c5ac
Convert synapse.federation.transport.server to async (#5689)
* Convert BaseFederationServlet._wrap to async

Empirically, this fixes some lost stacktraces. It should be safe because the
wrapped function is called from JsonResource._async_render, which is already
async.

* Convert the rest of synapse.federation.transport.server to async

We may as well do the whole file while we're here.

* changelog

* flake8
2019-07-18 11:46:47 +01:00
Jorik Schellekens
38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d9067bf9889b86c360c05ac88618b85c4f

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
Richard van der Hoff
a6a776f3d8
remove dead transaction persist code (#5622)
this hasn't done anything for years
2019-07-05 12:59:42 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Brendan Abolivier
bfe84e051e Split public rooms directory auth config in two 2019-06-24 15:42:31 +01:00
Richard van der Hoff
2f8491daef
Fix logging error when a tampered event is detected. (#5500) 2019-06-21 15:11:42 +01:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
b42f90470f Add experimental option to reduce extremities.
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.

THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
Richard van der Hoff
5c15039e06
Clean up code for sending federation EDUs. (#5381)
This code confused the hell out of me today. Split _get_new_device_messages
into its two (unrelated) parts.
2019-06-13 13:52:08 +01:00
Andrew Morgan
2d1d7b7e6f Prevent multiple device list updates from breaking a batch send (#5156)
fixes #5153
2019-06-06 23:54:00 +01:00
Richard van der Hoff
cea9750d11 Associate a request_name with each verify request, for logging
Also:
* rename VerifyKeyRequest->VerifyJsonRequest
* calculate key_ids on VerifyJsonRequest construction
* refactor things to pass around VerifyJsonRequests instead of 4-tuples
2019-06-05 10:46:26 +01:00
Richard van der Hoff
14f13babb0
Add a test room version where we enforce key validity (#5348) 2019-06-05 10:38:25 +01:00
Richard van der Hoff
2615c6bd9e
Clean up debug logging (#5347)
Remove some spurious stuff, clarify some other stuff
2019-06-05 10:35:40 +01:00
Richard van der Hoff
dae224a73f
Fix failure to fetch batches of PDUs (#5342)
FederationClient.get_pdu is called in a loop to fetch a batch of PDUs. A
failure to fetch one should not result in a failure of the whole batch. Add the
missing `continue`.
2019-06-04 18:05:06 +01:00
Richard van der Hoff
b4189b112f
Rename get_events->get_events_from_store_or_dest (#5344)
We have too many things called get_event, and it's hard to figure out what we
mean. Also remove some unused params from the signature, and add some logging.
2019-06-04 18:01:09 +01:00
Richard van der Hoff
fec2dcb1a5
Enforce validity period on server_keys for fed requests. (#5321)
When handling incoming federation requests, make sure that we have an
up-to-date copy of the signing key.

We do not yet enforce the validity period for event signatures.
2019-06-03 22:59:51 +01:00
Amber Brown
46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Richard van der Hoff
04d53794d6
Fix error handling for rooms whose versions are unknown. (#5219)
If we remove support for a particular room version, we should behave more
gracefully. This should make client requests fail with a 400 rather than a 500,
and will ignore individiual PDUs in a federation transaction, rather than the
whole transaction.
2019-05-21 13:47:25 +01:00
Amber Brown
f1e5b41388
Make all the rate limiting options more consistent (#5181) 2019-05-15 12:06:04 -05:00
Richard van der Hoff
130f932cbc Run black on per_destination_queue
... mostly to fix pep8 fails
2019-05-09 16:27:02 +01:00
Quentin Dufour
11ea16777f Limit the number of EDUs in transactions to 100 as expected by receiver (#5138)
Fixes #3951.
2019-05-09 11:01:41 +01:00
Matthew Hodgson
c0e0740bef add options to require an access_token to GET /profile and /publicRooms on CS API (#5083)
This commit adds two config options:

* `restrict_public_rooms_to_local_users`

Requires auth to fetch the public rooms directory through the CS API and disables fetching it through the federation API.

* `require_auth_for_profile_requests`

When set to `true`, requires that requests to `/profile` over the CS API are authenticated, and only returns the user's profile if the requester shares a room with the profile's owner, as per MSC1301.

MSC1301 also specifies a behaviour for federation (only returning the profile if the server asking for it shares a room with the profile's owner), but that's currently really non-trivial to do in a not too expensive way. Next step is writing down a MSC that allows a HS to specify which user sent the profile query. In this implementation, Synapse won't send a profile query over federation if it doesn't believe it already shares a room with the profile's owner, though.

Groups have been intentionally omitted from this commit.
2019-05-08 18:26:56 +01:00
Richard van der Hoff
837d7f85a9 more logging improvements 2019-04-25 22:17:59 +01:00
Richard van der Hoff
fd8fb32bdd remove extraneous exception logging 2019-04-25 22:02:03 +01:00
Richard van der Hoff
7ca638c761 Clarify logging when PDU signature checking fails 2019-04-25 20:55:12 +01:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969)
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
2019-04-01 10:24:38 +01:00
Erik Johnston
197fae1639 Use event streams to calculate presence
Primarily this fixes a bug in the handling of remote users joining a
room where the server sent out the presence for all local users in the
room to all servers in the room.

We also change to using the state delta stream, rather than the
distributor, as it will make it easier to split processing out of the
master process (as well as being more flexible).

Finally, when sending presence states to newly joined servers we filter
out old presence states to reduce the number sent. Initially we filter
out states that are offline and have a last active more than a week ago,
though this can be changed down the line.

Fixes #3962
2019-03-27 13:41:36 +00:00
Andrew Morgan
4a125be138
Make federation endpoints more tolerant of trailing slashes v2 (#4935)
Redo of https://github.com/matrix-org/synapse/pull/4840
2019-03-26 11:35:29 +00:00
Andrew Morgan
7bef97dfb7
Remove trailing slashes from outbound federation requests and retry on 400 (#4840)
As per #3622, we remove trailing slashes from outbound federation requests. However, to ensure that we remain backwards compatible with previous versions of Synapse, if we receive a HTTP 400 with `M_UNRECOGNIZED`, then we are likely talking to an older version of Synapse in which case we retry with a trailing slash appended to the request path.
2019-03-21 15:07:28 +00:00
Richard van der Hoff
a902d13180
Batch up outgoing read-receipts to reduce federation traffic. (#4890)
Rate-limit outgoing read-receipts as per #4730.
2019-03-20 16:02:25 +00:00
Richard van der Hoff
3b7ceb2c69
Merge pull request #4855 from matrix-org/rav/refactor_transaction_queue
Split TransactionQueue up
2019-03-15 12:32:11 +00:00
Richard van der Hoff
2dee441bdb
Merge pull request #4852 from matrix-org/rav/move_rr_sending_to_worker
Move client receipt processing to federation sender worker.
2019-03-15 12:30:30 +00:00