Commit Graph

657 Commits

Author SHA1 Message Date
Erik Johnston
aa4d8c1f9a Merge branch 'master' into develop 2021-01-13 10:36:55 +00:00
Marcus
e385c8b473
Don't apply the IP range blacklist to proxy connections (#9084)
It is expected that the proxy would be on a private IP address so the
configured proxy should be connected to regardless of the IP range
blacklist.
2021-01-12 12:20:30 -05:00
Patrick Cloke
723b19748a
Handle bad JSON data being returned from the federation API. (#9070) 2021-01-12 11:07:01 -05:00
Richard van der Hoff
2ec8ca5e60
Remove SynapseRequest.get_user_agent (#9069)
SynapseRequest is in danger of becoming a bit of a dumping-ground for "useful stuff relating to Requests",
which isn't really its intention (its purpose is to override render, finished and connectionLost to set up the 
LoggingContext and write the right entries to the request log).

Putting utility functions inside SynapseRequest means that lots of our code ends up requiring a
SynapseRequest when there is nothing synapse-specific about the Request at all, and any old
twisted.web.iweb.IRequest will do. This increases code coupling and makes testing more difficult.

In short: move get_user_agent out to a utility function.
2021-01-12 12:34:16 +00:00
Patrick Cloke
ff5c4da128
Add a maximum size for well-known lookups. (#8950) 2020-12-16 17:25:24 -05:00
Patrick Cloke
1619802228
Various clean-ups to the logging context code (#8935) 2020-12-14 14:19:47 -05:00
Richard van der Hoff
1821f7cc26
Fix buglet in DirectRenderJsonResource (#8897)
this was using `canonical_json` without setting it, so when you used it as a
standalone class, you would get exceptions.
2020-12-10 12:42:55 +00:00
Patrick Cloke
30fba62108
Apply an IP range blacklist to push and key revocation requests. (#8821)
Replaces the `federation_ip_range_blacklist` configuration setting with an
`ip_range_blacklist` setting with wider scope. It now applies to:

* Federation
* Identity servers
* Push notifications
* Checking key validitity for third-party invite events

The old `federation_ip_range_blacklist` setting is still honored if present, but
with reduced scope (it only applies to federation and identity servers).
2020-12-02 11:09:24 -05:00
Nicolas Chamo
3f0cba657c
Allow Date header through CORS (#8804) 2020-12-01 13:24:56 +00:00
Tulir Asokan
856eab606b
Remove special case of pretty printing JSON responses for curl (#8833)
* Remove special case of pretty printing JSON responses for curl

Signed-off-by: Tulir Asokan <tulir@maunium.net>
2020-11-27 12:37:55 +00:00
Patrick Cloke
968939bdac
Add additional type hints to HTTP client. (#8812)
This also removes some duplicated code between the simple
HTTP client and matrix federation client.
2020-11-25 13:30:47 -05:00
Patrick Cloke
f38676d161
Add type hints to matrix federation client / agent. (#8806) 2020-11-25 07:07:21 -05:00
Andrew Morgan
eedaf90c84
Better error message when a remote resource uses invalid Content-Type (#8719) 2020-11-11 14:22:40 +00:00
Erik Johnston
f21e24ffc2
Add ability for access tokens to belong to one user but grant access to another user. (#8616)
We do it this way round so that only the "owner" can delete the access token (i.e. `/logout/all` by the "owner" also deletes that token, but `/logout/all` by the "target user" doesn't).

A future PR will add an API for creating such a token.

When the target user and authenticated entity are different the `Processed request` log line will be logged with a: `{@admin:server as @bob:server} ...`. I'm not convinced by that format (especially since it adds spaces in there, making it harder to use `cut -d ' '` to chop off the start of log lines). Suggestions welcome.
2020-10-29 15:58:44 +00:00
Richard van der Hoff
c97da1e45d
Merge pull request #8678 from matrix-org/rav/fix_frozen_events
Fix serialisation errors when using third-party event rules.
2020-10-28 20:41:42 +00:00
Richard van der Hoff
e80eb69887 remove unused imports 2020-10-28 16:18:05 +00:00
Richard van der Hoff
b6ca69e4f1 Remove frozendict_json_encoder and support frozendicts everywhere
Not being able to serialise `frozendicts` is fragile, and it's annoying to have
to think about which serialiser you want. There's no real downside to
supporting frozendicts, so let's just have one json encoder.
2020-10-28 15:56:57 +00:00
Dan Callahan
aff1eb7c67
Tell Black to format code for Python 3.5 (#8664)
This allows trailing commas in multi-line arg lists.

Minor, but we might as well keep our formatting current with regard to
our minimum supported Python version.

Signed-off-by: Dan Callahan <danc@element.io>
2020-10-27 23:26:36 +00:00
Patrick Cloke
34a5696f93
Fix typos and spelling errors. (#8639) 2020-10-23 12:38:40 -04:00
Erik Johnston
c850dd9a8e
Fix handling of User-Agent headers with bad utf-8. (#8632) 2020-10-23 17:12:59 +01:00
Patrick Cloke
514a240aed
Remove unused OPTIONS handlers. (#8621)
The handling of OPTIONS requests was consolidated in #7534, but the endpoint
specific handlers were not removed.
2020-10-22 08:35:55 -04:00
Andrew Morgan
3e58ce72b4
Don't bother responding to client requests that have already disconnected (#8465)
This PR ports the quick fix from https://github.com/matrix-org/synapse/pull/2796 to further methods which handle media, URL preview and `/key/v2/server` requests. This prevents a harmless `ERROR` that comes up in the logs when we were unable to respond to a client request when the client had already disconnected. In this case we simply bail out if the client has already done so.

This is the 'simple fix' as suggested by https://github.com/matrix-org/synapse/issues/5304#issuecomment-574740003.

Fixes https://github.com/matrix-org/synapse/issues/6700
Fixes https://github.com/matrix-org/synapse/issues/5304
2020-10-06 10:03:39 +01:00
Patrick Cloke
4ff0201e62
Enable mypy checking for unreachable code and fix instances. (#8432) 2020-10-01 08:09:18 -04:00
Richard van der Hoff
1c262431f9
Fix handling of connection timeouts in outgoing http requests (#8400)
* Remove `on_timeout_cancel` from `timeout_deferred`

The `on_timeout_cancel` param to `timeout_deferred` wasn't always called on a
timeout (in particular if the canceller raised an exception), so it was
unreliable. It was also only used in one place, and to be honest it's easier to
do what it does a different way.

* Fix handling of connection timeouts in outgoing http requests

Turns out that if we get a timeout during connection, then a different
exception is raised, which wasn't always handled correctly.

To fix it, catch the exception in SimpleHttpClient and turn it into a
RequestTimedOutError (which is already a documented exception).

Also add a description to RequestTimedOutError so that we can see which stage
it failed at.

* Fix incorrect handling of timeouts reading federation responses

This was trapping the wrong sort of TimeoutError, so was never being hit.

The effect was relatively minor, but we should fix this so that it does the
expected thing.

* Fix inconsistent handling of `timeout` param between methods

`get_json`, `put_json` and `delete_json` were applying a different timeout to
the response body to `post_json`; bring them in line and test.

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
2020-09-29 10:29:21 +01:00
Richard van der Hoff
450ec48445
A pair of tiny cleanups in the federation request code. (#8401) 2020-09-28 13:15:00 +01:00
Richard van der Hoff
11c9e17738
Add type annotations to SimpleHttpClient (#8372) 2020-09-24 15:47:20 +01:00
Patrick Cloke
8a4a4186de
Simplify super() calls to Python 3 syntax. (#8344)
This converts calls like super(Foo, self) -> super().

Generated with:

    sed -i "" -Ee 's/super\([^\(]+\)/super()/g' **/*.py
2020-09-18 09:56:44 -04:00
Patrick Cloke
aec294ee0d
Use slots in attrs classes where possible (#8296)
slots use less memory (and attribute access is faster) while slightly
limiting the flexibility of the class attributes. This focuses on objects
which are instantiated "often" and for short periods of time.
2020-09-14 12:50:06 -04:00
Patrick Cloke
b86764662b
Fix the exception that is raised when invalid JSON is encountered. (#8291) 2020-09-10 14:55:25 -04:00
Patrick Cloke
c619253db8
Stop sub-classing object (#8249) 2020-09-04 06:54:56 -04:00
Patrick Cloke
5bf8e5f55b
Convert the well known resolver to async (#8214) 2020-09-01 09:15:22 -04:00
Patrick Cloke
f594e434c3
Switch the JSON byte producer from a pull to a push producer. (#8116) 2020-08-19 08:07:57 -04:00
Patrick Cloke
eebf52be06
Be stricter about JSON that is accepted by Synapse (#8106) 2020-08-19 07:26:03 -04:00
Patrick Cloke
2f4d60a5ba
Iteratively encode JSON responses to avoid blocking the reactor. (#8013) 2020-08-18 08:49:59 -04:00
Erik Johnston
a0f574f3c2
Reduce INFO logging (#8050)
c.f. #8021 

A lot of the code here is to change the `Completed 200 OK` logging to include the request URI so that we can drop the `Sending request...` log line.

Some notes:

1. We won't log retries, which may be confusing considering the time taken log line includes retries and sleeps.
2. The `_send_request_with_optional_trailing_slash` will always be logged *without* the forward slash, even if it succeeded only with the forward slash.
2020-08-11 18:10:07 +01:00
Erik Johnston
2f9fd5ab00
Don't log OPTIONS request at INFO (#8049) 2020-08-07 14:53:05 +01:00
Erik Johnston
7620912d84
Add health check endpoint (#8048) 2020-08-07 14:21:24 +01:00
David Vo
4dd27e6d11
Reduce unnecessary whitespace in JSON. (#7372) 2020-08-07 08:02:55 -04:00
Patrick Cloke
c36228c403
Convert run_as_background_process inner function to async. (#8032) 2020-08-06 08:20:42 -04:00
Patrick Cloke
88a3ff12f0
Convert the SimpleHttpClient to async. (#8016) 2020-08-04 07:22:04 -04:00
Patrick Cloke
6812509807
Implement handling of HTTP HEAD requests. (#7999) 2020-08-03 08:45:42 -04:00
Patrick Cloke
c978f6c451
Convert federation client to async/await. (#7975) 2020-07-30 08:01:33 -04:00
Patrick Cloke
a53e0160a2
Ensure the msg property of HttpResponseException is a string. (#7979) 2020-07-29 13:56:06 -04:00
Patrick Cloke
6a080ea184
Return an empty body for OPTIONS requests. (#7886) 2020-07-24 07:08:07 -04:00
Richard van der Hoff
1ec688bf21
Downgrade warning on client disconnect to INFO (#7928)
Clients disconnecting before we finish processing the request happens from time
to time. We don't need to yell about it
2020-07-24 09:55:47 +01:00
Patrick Cloke
68cd935826
Convert the federation agent and related code to async/await. (#7874) 2020-07-23 07:05:57 -04:00
Richard van der Hoff
2ccd48e921 fix an incorrect comment 2020-07-22 00:24:56 +01:00
Patrick Cloke
35450519de
Ensure that calls to json.dumps are compatible with the standard library json. (#7836) 2020-07-15 13:40:54 -04:00
Erik Johnston
f13061d515
Fix client reader sharding tests (#7853)
* Fix client reader sharding tests

* Newsfile

* Fix typing

* Update changelog.d/7853.misc

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Move mocking of http_client to tests

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2020-07-15 15:27:35 +01:00
Patrick Cloke
66a4af8d96
Do not use canonicaljson to magically handle decoding bytes from JSON. (#7802) 2020-07-10 14:30:08 -04:00