Commit Graph

496 Commits

Author SHA1 Message Date
Erik Johnston
a334e1cace Update to use new timeout function everywhere.
The existing deferred timeout helper function (and the one into twisted)
suffer from a bug when a deferred's canceller throws an exception, .

The new helper function doesn't suffer from this problem.
2018-09-19 10:39:40 +01:00
Amber Brown
47c02e6332
Merge pull request from turt2live/travis/fix-logging-1
Fix matrixfederationclient.py logging: Destination is a string
2018-09-19 18:14:47 +10:00
Amber Brown
3d6b24fb1b
Merge pull request from matrix-org/rav/set_sni_to_server_name
Set SNI to the server_name, not whatever was in the SRV record
2018-09-19 17:59:33 +10:00
Travis Ralston
35aec19f0a Destination is a string 2018-09-18 15:29:30 -06:00
Richard van der Hoff
31c15dcb80
Refactor matrixfederationclient to fix logging ()
We want to wait until we have read the response body before we log the request
as complete, otherwise a confusing thing happens where the request appears to
have completed, but we later fail it.

To do this, we factor the salient details of a request out to a separate
object, which can then keep track of the txn_id, so that it can be logged.
2018-09-18 18:17:15 +01:00
Richard van der Hoff
b3097396e7 Set SNI to the server_name, not whatever was in the SRV record
Fixes 
2018-09-18 17:01:12 +01:00
Richard van der Hoff
ac80cb08fe Fix more b'abcd' noise in metrics 2018-09-17 17:16:50 +01:00
Richard van der Hoff
f00a9d2636 Fix some b'abcd' noise in logs and metrics
Python 3 compatibility: make sure that we decode some byte sequences before we
use them to create log lines and metrics labels.
2018-09-17 16:15:42 +01:00
Erik Johnston
fcfe7a850d Add an awful secondary timeout to fix wedged requests
This is an attempt to mitigate  by adding yet-another-timeout
2018-09-14 19:23:07 +01:00
Erik Johnston
3e6e94fe9f
Merge pull request from matrix-org/hawkowl/timeouts-2
timeouts 2: electric boogaloo
2018-09-14 16:58:44 +01:00
Amber Brown
bc9af88a2d fix 2018-09-15 00:26:00 +10:00
Erik Johnston
9e2f9a7b57 Measure outbound requests 2018-09-14 15:11:26 +01:00
Amber Brown
c971aa7b9d fix 2018-09-14 03:57:02 +10:00
Amber Brown
8f08d848f5 fix 2018-09-14 03:53:56 +10:00
Amber Brown
7c33ab76da redact better 2018-09-14 03:45:34 +10:00
Amber Brown
63755fa4c2 we do that higher up 2018-09-14 03:21:47 +10:00
Amber Brown
73884ebac5 Merge remote-tracking branch 'origin/develop' into hawkowl/timeouts-2 2018-09-14 03:11:25 +10:00
Amber Brown
7c27c4d51c
merge () 2018-09-14 03:11:11 +10:00
Amber Brown
1c3f4d9ca5 buffer? 2018-09-14 03:09:13 +10:00
Amber Brown
bfa0b759e0
Attempt to figure out what's going on with timeouts () 2018-09-14 00:15:51 +10:00
Erik Johnston
8c5b84441b Log outbound requests when we retry 2018-09-12 16:22:14 +01:00
Erik Johnston
4084a774a8 Timeout reading body for outbound HTTP requests 2018-09-12 10:10:20 +01:00
Amber Brown
2d2828dcbc
Port http/ to Python 3 () 2018-09-06 00:10:47 +10:00
Erik Johnston
1d67b13674 Fix bug when federation_domain_whitelist is an emtpy list
Outbound federation were incorrectly allowed when the config option was
set to an empty list
2018-08-24 12:13:12 +01:00
Richard van der Hoff
a52f276990 Bugfixes
--------
 
 - Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
 ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEIQBQJ4l+yK4dlKkFIwi0edOSShEFAlt6/yMACgkQIwi0edOS
 ShFrzAgAhLA74fClydGLfhPNCXIw12Amv2ASA6+r0ukqjYQ/9MZ0CRsL+PMZJTk6
 Z75PK3TkX38zSpPaUx9HdMQO/yE+rpI99ZxlBM+8jzsf1jyF8ljqxSfixXUlaXX9
 698PQQ3f+IDoSs59NKrHvoWhUThaBpf/1GwRxG5mUcm2oKJItNMHk/uLurNCaIIZ
 YSwWcxvZdluOMX0WrlF/S524DqtgULcRf3E9x3mumobkVGxI3UH73hXgGUcPi7XV
 ttLuy7fVbV2Cun/wvjBqD7kc8LNbKwSaT9y+/xS0D5QYjph4rpQiTG7S5rtPLPMU
 Z2lL1C5ggBCkD43g3orDvehbNWCL5g==
 =0vEe
 -----END PGP SIGNATURE-----

Merge tag 'v0.33.3rc2' into develop

Bugfixes
--------

- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs
([\#3723](https://github.com/matrix-org/synapse/issues/3723))
2018-08-21 20:30:09 +01:00
Erik Johnston
cd6937fb26 Fix typo 2018-08-21 16:28:10 +01:00
Richard van der Hoff
be6527325a Fix exceptions when a connection is closed before we read the headers
This fixes bugs introduced in , by making sure that we behave sanely
when an incoming connection is closed before the headers are read.
2018-08-20 18:21:10 +01:00
Erik Johnston
b01a755498 Make the in flight requests metrics thread safe 2018-08-20 17:27:52 +01:00
Amber Brown
324525f40c
Port over enough to get some sytests running on Python 3 () 2018-08-20 23:54:49 +10:00
Richard van der Hoff
63260397c6
Merge pull request from matrix-org/rav/use_producer_for_responses
Use a producer to stream back responses
2018-08-17 14:58:45 +01:00
Richard van der Hoff
afcd655ab6 Use a producer to stream back responses
The problem with dumping all of the json response into the Request object at
once is that doing so starts the timeout for the next request to be received:
so if it takes longer than 60s to stream back the response to the client, the
client never gets it.

The correct solution is to use a Producer; then the timeout is only started
once all of the content is sent over the TCP connection.
2018-08-15 15:04:16 +01:00
Richard van der Hoff
eaaa2248ff Refactor request logging code
This commit moves a bunch of the logic for deciding when to log the receipt and
completion of HTTP requests into SynapseRequest, rather than in the request
handling wrappers.

Advantages of this are:
 * we get logs for *all* requests (including OPTIONS and HEADs), rather than
   just those that end up hitting handlers we've remembered to decorate
   correctly.

 * when a request handler wires up a Producer (as the media stuff does
   currently, and as other things will do soon), we log at the point that all
   of the traffic has been sent to the client.
2018-08-15 13:47:52 +01:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 () 2018-08-10 23:50:21 +10:00
Richard van der Hoff
3c0213a217
Merge pull request from vojeroen/send_sni_for_federation_requests
send SNI for federation requests
2018-08-10 12:23:54 +01:00
Richard van der Hoff
f900d50824 include known room versions in outgoing make_joins 2018-08-06 13:45:37 +01:00
Richard van der Hoff
01e93f48ed Kill off MatrixCodeMessageException
This code brings the SimpleHttpClient into line with the
MatrixFederationHttpClient by having it raise HttpResponseExceptions when a
request fails (rather than trying to parse for matrix errors and maybe raising
MatrixCodeMessageException).

Then, whenever we were checking for MatrixCodeMessageException and turning them
into SynapseErrors, we now need to check for HttpResponseExceptions and call
to_synapse_error.
2018-08-01 16:02:46 +01:00
Richard van der Hoff
fa7dc889f1 Be more careful which errors we send back over the C-S API
We really shouldn't be sending all CodeMessageExceptions back over the C-S API;
it will include things like 401s which we shouldn't proxy.

That means that we need to explicitly turn a few HttpResponseExceptions into
SynapseErrors in the federation layer.

The effect of the latter is that the matrix errcode will get passed through
correctly to calling clients, which might help with some of the random
M_UNKNOWN errors when trying to join rooms.
2018-08-01 16:02:38 +01:00
Amber Brown
da7785147d
Python 3: Convert some unicode/bytes uses () 2018-08-02 00:54:06 +10:00
Richard van der Hoff
7041cd872b
Merge branch 'develop' into send_sni_for_federation_requests 2018-07-27 09:17:11 +01:00
Amber Brown
a97c845271
Move v1-only APIs into their own module & isolate deprecated ones () 2018-07-19 20:03:33 +10:00
Erik Johnston
5bd0a47fcd pep8 2018-07-18 14:19:00 +01:00
Erik Johnston
e45a46b6e4 Add response code to response timer metrics 2018-07-18 13:59:36 +01:00
Amber Brown
bc006b3c9d
Refactor REST API tests to use explicit reactors () 2018-07-17 20:43:18 +10:00
Krombel
78a9ddcf9a rerun isort with latest version 2018-07-16 14:23:25 +02:00
Krombel
4a27000548 check isort by travis 2018-07-16 13:57:33 +02:00
Jeroen
505530f36a Merge remote-tracking branch 'upstream/develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/crypto/context_factory.py
2018-07-14 20:24:46 +02:00
Krombel
3366b9c534 rename assert_params_in_request to assert_params_in_dict
the method "assert_params_in_request" does handle dicts and not
requests. A request body has to be parsed to json before this method
can be used
2018-07-13 21:53:01 +02:00
Richard van der Hoff
55370331da
Refactor logcontext resource usage tracking ()
Factor out the resource usage tracking out to a separate object, which can be
passed around and copied independently of the logcontext itself.
2018-07-10 13:56:07 +01:00
Jeroen
b5e157d895 Merge branch 'develop' into send_sni_for_federation_requests
# Conflicts:
#	synapse/http/endpoint.py
2018-07-09 08:51:11 +02:00
Amber Brown
49af402019 run isort 2018-07-09 16:09:20 +10:00
Richard van der Hoff
546bc9e28b More server_name validation
We need to do a bit more validation when we get a server name, but don't want
to be re-doing it all over the shop, so factor out a separate
parse_and_validate_server_name, and do the extra validation.

Also, use it to verify the server name in the config file.
2018-07-04 18:59:51 +01:00
Richard van der Hoff
508196e08a
Reject invalid server names ()
Make sure that server_names used in auth headers are sane, and reject them with
a sensible error code, before they disappear off into the depths of the system.
2018-07-03 14:36:14 +01:00
Matthew Hodgson
fc4f8f33be replace invalid utf8 with \ufffd 2018-07-02 11:33:02 +01:00
Matthew Hodgson
1c867f5391 a fix which doesn't NPE everywhere 2018-07-01 11:56:33 +01:00
Matthew Hodgson
f131bf8d3e don't mix unicode strings with utf8-in-byte-strings
otherwise we explode with:

```
Traceback (most recent call last):
  File /usr/lib/python2.7/logging/handlers.py, line 78, in emit
    logging.FileHandler.emit(self, record)
  File /usr/lib/python2.7/logging/__init__.py, line 950, in emit
    StreamHandler.emit(self, record)
  File /usr/lib/python2.7/logging/__init__.py, line 887, in emit
    self.handleError(record)
  File /usr/lib/python2.7/logging/__init__.py, line 810, in handleError
    None, sys.stderr)
  File /usr/lib/python2.7/traceback.py, line 124, in print_exception
    _print(file, 'Traceback (most recent call last):')
  File /usr/lib/python2.7/traceback.py, line 13, in _print
    file.write(str+terminator)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_io.py, line 170, in write
    self.log.emit(self.level, format=u{log_io}, log_io=line)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 144, in emit
    self.observer(event)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 136, in __call__
    errorLogger = self._errorLoggerForObserver(brokenObserver)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 156, in _errorLoggerForObserver
    if obs is not observer
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_observer.py, line 81, in __init__
    self.log = Logger(observer=self)
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 64, in __init__
    namespace = self._namespaceFromCallingContext()
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/logger/_logger.py, line 42, in _namespaceFromCallingContext
    return currentframe(2).f_globals[__name__]
  File /home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/compat.py, line 93, in currentframe
    for x in range(n + 1):
RuntimeError: maximum recursion depth exceeded while calling a Python object
Logged from file site.py, line 129
  File /usr/lib/python2.7/logging/__init__.py, line 859, in emit
    msg = self.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 732, in format
    return fmt.format(record)
  File /usr/lib/python2.7/logging/__init__.py, line 471, in format
    record.message = record.getMessage()
  File /usr/lib/python2.7/logging/__init__.py, line 335, in getMessage
    msg = msg % self.args
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 4: ordinal not in range(128)
Logged from file site.py, line 129
```

...where the logger apparently recurses whilst trying to log the error, hitting the
maximum recursion depth and killing everything badly.
2018-07-01 05:08:58 +01:00
Amber Brown
6350bf925e
Attempt to be more performant on PyPy () 2018-06-28 14:49:57 +01:00
Jeroen
b7f34ee348 allow self-signed certificates 2018-06-26 20:41:05 +02:00
Erik Johnston
947fea67cb Need to pass reactor to endpoint fac 2018-06-25 15:22:57 +01:00
Amber Brown
07cad26d65
Remove all global reactor imports & pass it around explicitly () 2018-06-25 14:08:28 +01:00
Jeroen
07b4f88de9 formatting changes for pep8 2018-06-25 12:31:16 +02:00
Jeroen
3d605853c8 send SNI for federation requests 2018-06-24 22:38:43 +02:00
Richard van der Hoff
70e6501913
Merge pull request from matrix-org/rav/events_per_request
Log number of events fetched from DB
2018-06-22 11:17:56 +01:00
Amber Brown
77ac14b960
Pass around the reactor explicitly () 2018-06-22 09:37:10 +01:00
Richard van der Hoff
b088aafcae Log number of events fetched from DB
When we finish processing a request, log the number of events we fetched from
the database to handle it.

[I'm trying to figure out which requests are responsible for large amounts of
event cache churn. It may turn out to be more helpful to add counts to the
prometheus per-request/block metrics, but that is an extension to this code
anyway.]
2018-06-21 06:15:03 +01:00
Amber Brown
f6c4d74f96
Fix inflight requests metric (incorrect name & traceback) () 2018-06-20 11:18:57 +01:00
David Baker
187a546bff
Merge pull request from matrix-org/dbkr/unbind
Remove email addresses / phone numbers from ID servers when they're removed from synapse
2018-06-11 16:02:00 +01:00
Michael Telatynski
330432031b
redact_uri in two missed log paths 2018-06-06 10:25:48 +01:00
David Baker
bf54c1cf6c pep8 2018-06-06 10:15:33 +01:00
David Baker
3e4bc4488c More doc fixes 2018-06-06 09:44:10 +01:00
Michael Telatynski
e6cbf47773
factor out uri redaction into a method on http 2018-06-05 18:31:40 +01:00
David Baker
607bd27c83 fix pep8 2018-06-05 18:10:35 +01:00
David Baker
d62162bbec doc fixes 2018-06-05 18:09:13 +01:00
David Baker
c5930d513a Docstring 2018-06-04 12:05:58 +01:00
Michael Telatynski
09503126df
Strip access_token from outgoing requests using existing regex 2018-06-02 23:25:13 +01:00
Amber Brown
3ef5cd74a6 update to more consistently use seconds in any metrics or logging 2018-05-28 19:39:27 +10:00
Amber Brown
a2eb5db4a0 update metrics to be in seconds 2018-05-28 19:10:27 +10:00
Amber Brown
754826a830 Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-28 18:57:23 +10:00
Amber Brown
c4fb15a06c
Merge pull request from NotAFile/py3-repr-string
use repr, not str
2018-05-24 13:00:20 -05:00
David Baker
77a23e2e05 Merge remote-tracking branch 'origin/develop' into dbkr/unbind 2018-05-24 16:20:53 +01:00
Erik Johnston
46345187cc
Merge pull request from NotAFile/py3-six-3
Replace some more comparisons with six
2018-05-24 16:08:57 +01:00
Amber Brown
e987079037 fixes 2018-05-23 13:03:51 -05:00
David Baker
2c7866d664 Hit the 3pid unbind endpoint on deactivation 2018-05-23 14:38:56 +01:00
Amber Brown
53cc2cde1f cleanup 2018-05-22 17:32:57 -05:00
Amber Brown
071206304d cleanup pep8 errors 2018-05-22 16:54:22 -05:00
Amber Brown
8f5a688d42 cleanups, self-registration 2018-05-22 10:56:03 -05:00
Amber Brown
a8990fa2ec Merge remote-tracking branch 'origin/develop' into 3218-official-prom 2018-05-22 10:50:26 -05:00
Erik Johnston
c435b0b441 Don't store context 2018-05-22 09:34:26 +01:00
Erik Johnston
fb2806b186 Move in_flight_requests_count to be a callback metric 2018-05-22 09:31:53 +01:00
Amber Brown
fcc525b0b7 rest of the changes 2018-05-21 19:48:57 -05:00
Amber Brown
df9f72d9e5 replacing portions 2018-05-21 19:47:37 -05:00
Erik Johnston
dfa70adc33 Add in flight request metrics
This tracks CPU and DB usage while requests are in flight, rather than
when we write the response.
2018-05-21 16:23:06 +01:00
Adrian Tschira
d9fe2b2d9d Replace some more comparisons with six
plus a bonus b"" string I missed last time

Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:56:31 +02:00
Adrian Tschira
b932b4ea25 use repr, not str
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:28:42 +02:00
Richard van der Hoff
8030a825c8
Merge pull request from matrix-org/rav/consent_handler
ConsentResource to gather policy consent from users
2018-05-16 07:19:18 +01:00
Richard van der Hoff
47815edcfa ConsentResource to gather policy consent from users
Hopefully there are enough comments and docs in this that it makes sense on its
own.
2018-05-15 15:11:59 +01:00
Richard van der Hoff
318711e139 Set Server header in SynapseRequest
(instead of everywhere that writes a response. Or rather, the subset of places
which write responses where we haven't forgotten it).

This also means that we don't have to have the mysterious version_string
attribute in anything with a request handler.

Unfortunately it does mean that we have to pass the version string wherever we
instantiate a SynapseSite, which has been c&ped 150 times, but that is code
that ought to be cleaned up anyway really.
2018-05-10 18:50:27 +01:00
Richard van der Hoff
645cb4bf06 Remove redundant request_handler decorator
This is needless complexity; we might as well use the wrapper directly.

Also rename wrap_request_handler->wrap_json_request_handler.
2018-05-10 12:19:53 +01:00
Richard van der Hoff
09f570b935 Factor wrap_request_handler_with_logging out of wrap_request_handler
... so that it can be used on non-JSON endpoints
2018-05-10 12:19:52 +01:00
Richard van der Hoff
9589a1925e Remove include_metrics param
The metrics are now available via the request, so this is redundant and can go
away at last.
2018-05-10 12:19:52 +01:00
Richard van der Hoff
49e5a613f1 Move outgoing_responses_counter handling to RequestMetrics
it's much neater there.
2018-05-10 12:19:52 +01:00