Commit Graph

606 Commits

Author SHA1 Message Date
Erik Johnston
1dec31560e Change jitter to be a factor rather than absolute value 2019-08-20 11:46:00 +01:00
Jorik Schellekens
87fa26006b
Opentracing misc (#5856)
Add authenticated_entity and servlet_names tags.

Functionally:
- Add a tag for authenticated_entity
- Add a tag for servlet_names

Stylistically:
Moved to importing methods directly from opentracing.
2019-08-16 16:13:25 +01:00
Erik Johnston
861d663c15 Fixup changelog and remove debug logging 2019-08-16 13:15:26 +01:00
Erik Johnston
f299c5414c Refactor MatrixFederationAgent to retry SRV.
This refactors MatrixFederationAgent to move the SRV lookup into the
endpoint code, this has two benefits:
	1. Its easier to retry different host/ports in the same way as
	   HostnameEndpoint.
	2. We avoid SRV lookups if we have a free connection in the pool
2019-08-15 15:43:22 +01:00
Erik Johnston
e6e136decc Retry well known on fail.
If we have recently seen a valid well-known for a domain we want to
retry on (non-final) errors a few times, to handle temporary blips in
networking/etc.
2019-08-15 09:28:58 +01:00
Erik Johnston
17e1e80726 Retry well-known lookup before expiry.
This gives a bit of a grace period where we can attempt to refetch a
remote `well-known`, while still using the cached result if that fails.

Hopefully this will make the well-known resolution a bit more torelant
of failures, rather than it immediately treating failures as "no result"
and caching that for an hour.
2019-08-13 16:20:38 +01:00
Erik Johnston
f218705d2a Make default well known cache global again. 2019-08-13 10:06:51 +01:00
Erik Johnston
107ad133fc Move well known lookup into a separate clas 2019-08-07 15:36:38 +01:00
Erik Johnston
af9f1c0764 Add a lower bound for TTL on well known results.
It costs both us and the remote server for us to fetch the well known
for every single request we send, so we add a minimum cache period. This
is set to 5m so that we still honour the basic premise of "refetch
frequently".
2019-08-06 17:01:23 +01:00
Erik Johnston
d2e3d5b9db Handle incorrectly encoded query params correctly 2019-08-01 13:23:00 +01:00
Erik Johnston
a9bcae9f50 Share SSL options for well-known requests 2019-07-31 10:39:24 +01:00
Andrew Morgan
baf081cd3b Bugfixes
--------
 
 - Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
 -----BEGIN PGP SIGNATURE-----
 
 iQJHBAABCgAxFiEEgQG31Z317NrSMt0QiISIDS7+X/QFAl04Ur0THGFuZHJld0Bh
 bW9yZ2FuLnh5egAKCRCIhIgNLv5f9F4oD/0TY6S/SEd2uAmzor64ojmbX5BOwPzf
 j/wzUTrfvuf40EvkNPDpnejNZSvy/ysbaGQaQusv0SQKlV3xrvdn4RuMvnOWVWck
 kBsO+lvzOaUTR0KHDxN4y9F5eI2NdPbub4847PPVzyqSIHAd+kolxXS8kSBBhwpL
 yfaICWV/AOy5L7xN+JZ9IQpnegVAvUj5DmgXzDHd6VdeiHDVJuARaBgrR5uCkwVS
 ZoLRqZ95XV/qiguMAUvPOwyEqht2mwO64989MswP16YYm8oMkB5QA6I5nYnACsTP
 qk9YcN/oNvEfQXUhttku6MxK1/4yUMPUhEoDBDH7ebc0440QDtWN+IHTdA6oPVZB
 IuStL9YGY16m7Ltx37ZUA4URfNMiSeLHo3zKc/mCAcwxN4HyOjJewtxbG5zKQAOZ
 SMs8UcDwGR4zL1hnt8ZDNYtWwfzJBQIdGjoHvjXJEY7/1csTv2lmAwewFTXiqSAr
 30GW5ews94kotqBK53zZT6V0F5gHNqgGHniOz1ZpqLLxYLqO3LSAGe97CrqlWUdX
 GkhA9tZyweknociD9fyyBmKdcFJ4mL4a+oGI5CMnSMph8UvCY8Y5XMb1T+iYEABI
 tA9G3mBvgkLPj+5V+8QggNkBafSigW2Q4FX7enGsDmiiskZOtfeKrAcVkapD4ooi
 3I7IW5aetZr2IQ==
 =+JBn
 -----END PGP SIGNATURE-----

Merge tag 'v1.2.0rc2' into develop

Bugfixes
--------

- Fix a regression introduced in v1.2.0rc1 which led to incorrect labels on some prometheus metrics. ([\#5734](https://github.com/matrix-org/synapse/issues/5734))
2019-07-24 13:47:51 +01:00
Jorik Schellekens
cf2972c818
Fix servlet metric names (#5734)
* Fix servlet metric names

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Remove redundant check

* Cover all return paths
2019-07-24 13:07:35 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Jorik Schellekens
38a6d3eea7
Add basic opentracing support (#5544)
* Configure and initialise tracer

Includes config options for the tracer and sets up JaegerClient.

* Scope manager using LogContexts

We piggy-back our tracer scopes by using log context.
The current log context gives us the current scope. If new scope is
created we create a stack of scopes in the context.

* jaeger is a dependency now

* Carrier inject and extraction for Twisted Headers

* Trace federation requests on the way in and out.

The span is created in _started_processing and closed in
_finished_processing because we need a meaningful log context.

* Create logcontext for new scope.

Instead of having a stack of scopes in a logcontext we create a new
context for a new scope if the current logcontext already has a scope.

* Remove scope from logcontext if logcontext is top level

* Disable tracer if not configured

* typo

* Remove dependence on jaeger internals

* bools

* Set service name

* :Explicitely state that the tracer is disabled

* Black is the new black

* Newsfile

* Code style

* Use the new config setup.

* Generate config.

* Copyright

* Rename config to opentracing

* Remove user whitelisting

* Empty whitelist by default

* User ConfigError instead of RuntimeError

* Use isinstance

* Use tag constants for opentracing.

* Remove debug comment and no need to explicitely record error

* Two errors a "s(c)entry"

* Docstrings!

* Remove debugging brainslip

* Homeserver Whitlisting

* Better opentracing config comment

* linting

* Inclue worker name in service_name

* Make opentracing an optional dependency

* Neater config retreival

* Clean up dummy tags

* Instantiate tracing as object instead of global class

* Inlcude opentracing as a homeserver member.

* Thread opentracing to the request level

* Reference opetnracing through hs

* Instantiate dummy opentracin g for tests.

* About to revert, just keeping the unfinished changes just in case

* Revert back to global state, commit number:

9ce4a3d9067bf9889b86c360c05ac88618b85c4f

* Use class level methods in tracerutils

* Start and stop requests spans in a place where we
have access to the authenticated entity

* Seen it, isort it

* Make sure to close the active span.

* I'm getting black and blue from this.

* Logger formatting

Co-Authored-By: Erik Johnston <erik@matrix.org>

* Outdated comment

* Import opentracing at the top

* Return a contextmanager

* Start tracing client requests from the servlet

* Return noop context manager if not tracing

* Explicitely say that these are federation requests

* Include servlet name in client requests

* Use context manager

* Move opentracing to logging/

* Seen it, isort it again!

* Ignore twisted return exceptions on context exit

* Escape the scope

* Scopes should be entered to make them useful.

* Nicer decorator names

* Just one init, init?

* Don't need to close something that isn't open

* Docs make you smarter
2019-07-11 10:36:03 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Amber Brown
0ee9076ffe Fix media repo breaking (#5593) 2019-07-02 19:01:28 +01:00
Amber Brown
f40a7dc41f
Make the http server handle coroutine-making REST servlets (#5475) 2019-06-29 17:06:55 +10:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
95d38afe96 Don't log exception when failing to fetch remote content.
In particular, let's not log stack traces when we stop processing
becuase the response body was too large.
2019-06-07 12:39:10 +01:00
Richard van der Hoff
5bdb189f86
Improve docstrings on MatrixFederationClient. (#5332) 2019-06-04 11:14:16 +01:00
Richard van der Hoff
8d92329214
Remove spurious debug from MatrixFederationHttpClient.get_json (#5287)
This is just unhelpful spam
2019-05-29 19:31:52 +01:00
Aaron Raimist
2d4853039f
Fix error code for invalid parameter
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2019-05-24 17:13:16 -05:00
Andrew Morgan
5a4b328f52 Add ability to blacklist ip ranges for federation traffic (#5043) 2019-05-13 19:05:06 +01:00
Andrew Morgan
2f48c4e1ae
URL preview blacklisting fixes (#5155)
Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
2019-05-10 10:32:44 -07:00
Andrew Morgan
6824ddd93d Config option for verifying federation certificates (MSC 1711) (#4967) 2019-04-25 14:22:49 +01:00
Andrew Morgan
4a125be138
Make federation endpoints more tolerant of trailing slashes v2 (#4935)
Redo of https://github.com/matrix-org/synapse/pull/4840
2019-03-26 11:35:29 +00:00
Andrew Morgan
b41c2eaadc Clean up backoff_on_404 and metehod calls 2019-03-21 14:32:47 +00:00
Andrew Morgan
2150151abe kwargs doesn't like commas on calling funcs either. TIL 2019-03-20 14:13:32 +00:00
Andrew Morgan
bb52a2e653 lint 2019-03-20 14:08:57 +00:00
Andrew Morgan
cd36a1283b New test, fix issues 2019-03-20 14:00:39 +00:00
Andrew Morgan
c69df5d5d3 Fix comments. v0.99.2 -> v0.99.3 2019-03-20 11:27:18 +00:00
Andrew Morgan
551ea11559 Just return if not doing any trailing slash shennanigans 2019-03-20 11:07:36 +00:00
Andrew Morgan
a8ad39eec7 lint 2019-03-18 17:47:39 +00:00
Andrew Morgan
621e7f37f1 Better exception handling 2019-03-18 17:45:54 +00:00
Andrew Morgan
ecea5af491 Correct var name 2019-03-13 21:21:03 +00:00
Andrew Morgan
b2df0e8e2c receiving a 400 caused an exception. handle it 2019-03-13 21:08:10 +00:00
Andrew Morgan
9a2e22fd41 is this what purgatory feels like 2019-03-13 20:29:38 +00:00
Andrew Morgan
45524f2f5e i should have given up x2 2019-03-13 20:17:39 +00:00
Andrew Morgan
8d16ffaf7a i should have given up 2019-03-13 20:03:10 +00:00
Andrew Morgan
26f8e2d099 there comes a time when you should give up. but you dont 2019-03-13 19:49:08 +00:00
Andrew Morgan
5ca857ad84 as above 2019-03-13 19:35:23 +00:00
Andrew Morgan
7c0295f13c no kwargs today 2019-03-13 19:27:10 +00:00
Andrew Morgan
66cdb840a6 Or perhaps I was the one who was drunk 2019-03-13 19:18:25 +00:00
Andrew Morgan
bec313818c go home python, you're drunk 2019-03-13 19:10:56 +00:00
Andrew Morgan
c991e7aec7 Syntax checker is bork 2019-03-13 19:08:08 +00:00
Andrew Morgan
c2d848b80d Destructure again 2019-03-13 19:04:43 +00:00
Andrew Morgan
ee8ba397e8 Are you happy now 2019-03-13 18:48:31 +00:00
Andrew Morgan
9dd0e34679 Syntax test 2019-03-13 18:45:17 +00:00
Andrew Morgan
220607a618 Remove testing code 2019-03-13 18:43:40 +00:00
Andrew Morgan
660b77f362 Add missing docstring detail 2019-03-13 18:38:16 +00:00
Andrew Morgan
5526b054aa Fix syntax issues 2019-03-13 18:35:21 +00:00
Andrew Morgan
09626bfd39 Switch to wrapper function around _send_request 2019-03-13 18:26:06 +00:00
Andrew Morgan
7d053cfe10 Retry on 400:M_UNRECOGNIZED 2019-03-13 12:10:33 +00:00
Andrew Morgan
7e75d9644b Fix paranthesis indent 2019-03-13 11:15:23 +00:00
Andrew Morgan
cf301e37d8 Add workaround note 2019-03-13 11:14:43 +00:00
Andrew Morgan
0ea8582f8b Cleaner way of implementing trailing slashes 2019-03-12 14:11:11 +00:00
Andrew Morgan
64ff11019e Retry certain federation requests on 404 2019-03-08 18:22:47 +00:00
Andrew Morgan
ac61b45a75 Minor docstring fixes for MatrixFederationAgent (#4765) 2019-02-28 16:24:01 +00:00
Erik Johnston
c7b333c545 Log tracebacks correctly 2019-02-25 16:56:41 +00:00
Richard van der Hoff
c88bc53903
Fix TaskStopped exceptions when outbound requests time out (#4690) 2019-02-20 11:35:52 +00:00
Erik Johnston
fe725f7e45 Cleanup top level request exception logging
Firstly, we always logged that the request was being handled via
`JsonResource._async_render`, so we change that to use the servlet name
we add to the request.

Secondly, we pass the exception information to the logger rather than
formatting it manually. This makes it consistent with other exception
logging, allwoing logging hooks and formatters to access the exception
information.
2019-02-18 15:11:04 +00:00
Richard van der Hoff
5d27730a73
Move ClientTLSOptionsFactory init out of refresh_certificates (#4611)
It's nothing to do with refreshing the certificates. No idea why it was here.
2019-02-11 18:03:30 +00:00
Richard van der Hoff
ef43a03fc5
Merge pull request #4546 from matrix-org/rav/silence_critical_error_from_federation
Fix noisy "twisted.internet.task.TaskStopped" errors in logs
2019-02-01 14:37:22 +00:00
Richard van der Hoff
fa794980ec
Merge pull request #4544 from matrix-org/rav/skip_invalid_well_known
Treat an invalid .well-known the same as an absent one
2019-02-01 13:18:36 +00:00
Richard van der Hoff
e9779a6f8f Fix b'ab' noise in logs 2019-02-01 12:34:31 +00:00
Richard van der Hoff
f0ba34f581 Fix noisy "twisted.internet.task.TaskStopped" errors in logs
Fixes #4003
2019-02-01 12:22:57 +00:00
Richard van der Hoff
8a21b03fba Treat an invalid .well-known the same as an absent one
... basically, carry on and fall back to SRV etc.
2019-02-01 11:37:31 +00:00
Richard van der Hoff
3c8a41140e Cache failures to parse .well-known
Also add a Measure block around the .well-known fetch
2019-02-01 00:37:52 +00:00
Richard van der Hoff
24d59c7568 better logging for federation connections 2019-01-31 23:18:20 +00:00
Richard van der Hoff
d428b46346 Update federation routing logic to check .well-known before SRV 2019-01-31 23:14:18 +00:00
Richard van der Hoff
a79034aedf
Merge pull request #4521 from matrix-org/rav/fed_routing/cleanups
Tiny .well-known fixes
2019-01-30 11:47:24 +00:00
Richard van der Hoff
c7b24ac3d0
Follow redirects on .well-known (#4520) 2019-01-30 11:43:33 +00:00
Richard van der Hoff
09a1a6b55e fix exception text 2019-01-30 10:58:52 +00:00
Richard van der Hoff
928c50b59a Also jitter the invalid cache period 2019-01-30 10:58:52 +00:00
Richard van der Hoff
bc5f6e1797
Add a caching layer to .well-known responses (#4516) 2019-01-30 10:55:25 +00:00
Richard van der Hoff
cc2d650ef7
Relax requirement for a content-type on .well-known (#4511) 2019-01-29 16:49:17 +00:00
Richard van der Hoff
99e36d5e24
Implement MSC1708 (.well-known lookups for server routing) (#4489) 2019-01-29 13:53:02 +00:00
Richard van der Hoff
3bd0f1a4a3 docstrings for _RoutingResult 2019-01-28 12:43:09 +00:00
Richard van der Hoff
0fd5b3b53e Handle IP literals explicitly
We don't want to be doing .well-known lookups on these guys.
2019-01-28 10:34:30 +00:00
Richard van der Hoff
51958df766 MatrixFederationAgent: factor out routing logic
This is going to get too big and unmanageable.
2019-01-28 10:34:30 +00:00
Richard van der Hoff
d840019192
Fix idna and ipv6 literal handling in MatrixFederationAgent (#4487)
Turns out that the library does a better job of parsing URIs than our
reinvented wheel. Who knew.

There are two things going on here. The first is that, unlike
parse_server_name, URI.fromBytes will strip off square brackets from IPv6
literals, which means that it is valid input to ClientTLSOptionsFactory and
HostnameEndpoint.

The second is that we stay in `bytes` throughout (except for the argument to
ClientTLSOptionsFactory), which avoids the weirdness of (sometimes) ending up
with idna-encoded values being held in `unicode` variables. TBH it probably
would have been ok but it made the tests fragile.
2019-01-28 09:56:59 +00:00
Richard van der Hoff
8520bc3109
Fix Host header sent by MatrixFederationAgent (#4468)
Move the Host header logic down here so that (a) it is used if we reuse the
agent elsewhere, and (b) we can mess about with it with .well-known.
2019-01-25 12:38:16 +00:00
Richard van der Hoff
4a6e863843
Merge pull request #4464 from matrix-org/rav/fix_srv_lookup
MatrixFederationAgent: Look up the right SRV record
2019-01-24 13:57:51 +00:00
Richard van der Hoff
f4697b5ec1
Fix UnboundLocalError in post_urlencoded_get_json (#4460)
This could cause exceptions if the id server returned 4xx responses.
2019-01-24 13:38:29 +00:00
Richard van der Hoff
afd69a0920 Look up the right SRV record 2019-01-24 13:31:43 +00:00
Richard van der Hoff
7021784d46 put resolve_service in an object
this makes it easier to stub things out for tests.
2019-01-22 20:35:12 +00:00
Richard van der Hoff
53a327b4d5 Require that service_name be a byte string
it is only ever a bytes now, so let's enforce that.
2019-01-22 20:35:12 +00:00
Richard van der Hoff
fe212bbe4a Kill off matrix_federation_endpoint
this thing is now redundant.
2019-01-22 20:34:35 +00:00
Richard van der Hoff
7871146667 Make MatrixFederationClient use MatrixFederationAgent
... instead of the matrix_federation_endpoint
2019-01-22 20:34:35 +00:00
Richard van der Hoff
44be7513bf MatrixFederationAgent
Pull the magic that is currently in matrix_federation_endpoint and friends into
an agent-like thing
2019-01-22 20:34:35 +00:00
Richard van der Hoff
33a55289cb
Refactor and bugfix for resove_service (#4427) 2019-01-22 10:59:27 +00:00
Richard van der Hoff
de6888e7ce Remove redundant WrappedConnection (#4409)
* Remove redundant WrappedConnection

The matrix federation client uses an HTTP connection pool, which times out its
idle HTTP connections, so there is no need for any of this business.
2019-01-18 23:07:38 +11:00
Richard van der Hoff
9feb5d0b71
sign_request -> build_auth_headers (#4408)
Just got very confused about the fact that the headers are only an output, not
an input.
2019-01-17 12:40:09 +00:00
Erik Johnston
34ea14139d Fixup docstrings for matrixfederationclient 2019-01-09 09:25:59 +00:00
Erik Johnston
d422570e9b Use RequestSendFailed when fail to parse content type headers 2019-01-08 14:22:18 +00:00
Erik Johnston
b970cb0e96 Refactor request sending to have better excpetions (#4358)
* Correctly retry and back off if we get a HTTPerror response

* Refactor request sending to have better excpetions

MatrixFederationHttpClient blindly reraised exceptions to the caller
without differentiating "expected" failures (e.g. connection timeouts
etc) versus more severe problems (e.g. programming errors).

This commit adds a RequestSendFailed exception that is raised when
"expected" failures happen, allowing the TransactionQueue to log them as
warnings while allowing us to log other exceptions as actual exceptions.
2019-01-08 11:04:28 +00:00
Amber Brown
ea6abf6724
Fix IP URL previews on Python 3 (#4215) 2018-12-22 01:56:13 +11:00
Richard van der Hoff
0a1fc52971 fix parse_string docstring 2018-11-08 11:12:29 +00:00
Amber Brown
b3708830b8
Fix URL preview bugs (type error when loading cache from db, content-type including quotes) (#4157) 2018-11-08 01:37:43 +11:00