Richard van der Hoff
0d8e3ad48b
Fix logcontext leaks in httpclient
...
`preserve_context_over_fn` is borked
2017-10-26 18:17:10 +01:00
Richard van der Hoff
12ef02dc3d
SimpleHTTPClient: add support for headers
...
Sometimes we need to pass headers into these methods
2017-10-26 17:59:50 +01:00
Kegan Dougal
d0debb2116
Remember how twisted works
2017-05-05 11:00:21 +01:00
Kegan Dougal
c2ddd773bc
Include the clock
2017-05-05 10:52:46 +01:00
Kegan Dougal
7dd3bf5e24
Rewrite SimpleHttpClient.request to include timeouts
...
Fixes #2191
2017-05-05 10:49:19 +01:00
David Baker
482a2ad122
No need for the exception variable
2017-05-03 11:02:59 +01:00
David Baker
c0380402bc
List caught expection types
2017-05-03 10:56:22 +01:00
David Baker
5fd12dce01
Remove debugging
2017-04-26 12:36:26 +01:00
David Baker
c366276056
Fix get_json
2017-04-26 10:07:01 +01:00
David Baker
1a9255c12e
Use CodeMessageException subclass instead
...
Parse json errors from get_json client methods and throw special
errors.
2017-04-25 19:30:55 +01:00
David Baker
a46982cee9
Need the HTTP status code
2017-04-21 16:20:12 +01:00
David Baker
70caf49914
Do the same for get_json
2017-04-21 16:09:03 +01:00
David Baker
a1595cec78
Don't error for 3xx responses
2017-04-21 11:51:17 +01:00
David Baker
a90a0f5c8a
Propagate errors sensibly from proxied IS requests
...
When we're proxying Matrix endpoints, parse out Matrix error
responses and turn them into SynapseErrors so they can be
propagated sensibly upstream.
2017-04-21 11:32:48 +01:00
Johannes Löthberg
0648e76979
Remove spurious newline
...
Apparently I just removed the spaces instead...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-12 18:41:30 +01:00
Johannes Löthberg
d3bd94805f
Fixup for #1689 and #1690
...
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2016-12-12 16:32:47 +01:00
Glyph
6e18805ac2
IPv6 support for client.py
...
This is an (untested) general sketch of how to use wrapClientTLS to implement TLS over IPv6, as well as faster connections over IPv4.
2016-12-11 11:10:32 +01:00
Richard van der Hoff
255c229f23
Work around TLS bug in twisted
...
Wrap up twisted's FileBodyProducer to work around
https://twistedmatrix.com/trac/ticket/8473 . Hopefully this fixes
https://matrix.org/jira/browse/SYN-700 .
2016-06-15 10:39:08 +01:00
Mark Haines
eb79110beb
Clean up the blacklist/whitelist handling.
...
Always set the config key with an empty list, even if a list isn't specified.
This means that the codepaths are the same for both the empty list and
for a missing key. Since the behaviour is the same for both cases this
makes the code somewhat easier to reason about.
2016-05-16 13:03:59 +01:00
Matthew Hodgson
792def4928
add a url_preview_ip_range_whitelist config param so we can whitelist the matrix.org IP space
2016-05-01 12:44:24 +01:00
Erik Johnston
f505575f69
Make InsecureInterceptableContextFactory work with SpiderEndpoint
2016-04-19 16:08:14 +01:00
Matthew Hodgson
83b2f83da0
actually throw meaningful errors
2016-04-08 21:36:59 +01:00
Matthew Hodgson
dafef5a688
Add url_preview_enabled config option to turn on/off preview_url endpoint. defaults to off.
...
Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered.
Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered.
Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist
Add commentary and generally address PR feedback
2016-04-08 18:37:15 +01:00
Matthew Hodgson
8b98a7e8c3
pep8
2016-04-03 12:56:29 +01:00
Matthew Hodgson
7426c86eb8
add a persistent cache of URL lookups, and fix up the in-memory one to work
2016-04-03 00:31:57 +01:00
Matthew Hodgson
d1b154a10f
support gzip compression, and don't pass through error msgs
2016-04-02 03:06:39 +01:00
Matthew Hodgson
683e564815
handle spidered relative images correctly
2016-03-31 23:52:58 +01:00
Matthew Hodgson
0d3d7de6fc
sync in changes from matrixfederationclient
2016-03-31 12:42:27 +01:00
Matthew Hodgson
a8a5dd3b44
handle requests with missing content-length headers (e.g. YouTube)
2016-03-31 01:55:21 +01:00
Matthew Hodgson
721b2bfa85
implement redirects
2016-03-29 03:32:52 +01:00
Matthew Hodgson
64b4aead15
make it work
2016-03-29 03:13:25 +01:00
Matthew Hodgson
dd4287ca5d
make it build
2016-03-29 02:07:57 +01:00
Matthew Hodgson
d9d48aad2d
Merge branch 'develop' into matthew/preview_urls
2016-03-27 22:54:42 +01:00
David Baker
9a3c80a348
pep8
2016-03-08 10:09:07 +00:00
David Baker
7bcee4733a
Encode unicode objects given to post_urlencode* otherwise urllib.urlencode chokes.
2016-03-08 10:04:38 +00:00
Matthew Hodgson
7dd0c1730a
initial WIP of a tentative preview_url endpoint - incomplete, untested, experimental, etc. just putting it here for safekeeping for now
2016-01-24 18:47:27 -05:00
Matthew Hodgson
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
b8dd5b1a2d
Verify third party ID server certificates
2015-10-16 14:54:54 +01:00
Erik Johnston
a80ef851f7
Fix previous merge to s/version_string/user_agent/
2015-10-10 12:35:39 +01:00
Steven Hammerton
625e13bfde
Add get_raw method to SimpleHttpClient, use this in CAS auth rather than requests
2015-10-10 10:54:19 +01:00
Daniel Wagner-Hall
492beb62a8
Use space not dash as delimiter
2015-10-06 09:53:33 -05:00
Daniel Wagner-Hall
e0b466bcfd
Use space not dash as delimiter
2015-10-06 09:32:26 -05:00
Daniel Wagner-Hall
b28c7da0a4
Preserve version string in user agent
2015-10-05 20:49:39 -05:00
Daniel Wagner-Hall
8fc52bc56a
Allow synapse's useragent to be customized
...
This will allow me to write tests which verify which server made HTTP
requests in a federation context.
2015-10-02 17:13:51 -05:00
Daniel Wagner-Hall
3bcbabc9fb
Rename context factory
...
Mjark is officially no fun.
2015-09-15 15:46:22 +01:00
Daniel Wagner-Hall
6485f03d91
Fix random formatting
2015-09-09 13:05:00 +01:00
Daniel Wagner-Hall
81a93ddcc8
Allow configuration to ignore invalid SSL certs
...
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
Mark Haines
78323ccdb3
Remove syutil dependency in favour of smaller single-purpose libraries
2015-08-24 16:17:38 +01:00
Erik Johnston
653533a3da
Fix log context when sending requests
2015-06-19 11:46:49 +01:00
Erik Johnston
83f119a84a
Log requests and responses sent via http.client
2015-06-15 13:14:12 +01:00
Erik Johnston
79e37a7ecb
Correctly pass connection pool parameter
2015-05-28 16:48:53 +01:00
Erik Johnston
11f51e6ded
Up maxPersistentPerHost count
2015-05-28 15:45:46 +01:00
Erik Johnston
086df80790
Add connection pooling to SimpleHttpClient
2015-05-28 15:43:21 +01:00
Erik Johnston
476899295f
Change the way we do logging contexts so that they survive divergences
2015-05-08 16:32:18 +01:00
David Baker
59bf16eddc
New registration for C/S API v2. Only ReCAPTCHA working currently.
2015-03-30 18:13:10 +01:00
Paul "LeoNerd" Evans
128cf2daf7
Appease pep8
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
b0cf867319
Use _ instead of . as a metric namespacing separator, for Prometheus
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f9478e475b
Rename Metrics' "keys" to "labels"
2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
7d72e44eb9
Add vector counters to HTTP clients and servers; count the requests by method and responses by method and response code
2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
c53ec53d80
Pull out all uses of the underlying HTTP user agent .request() method into a single wrapper function, to make adding metrics easier
2015-03-12 16:24:50 +00:00
Kegan Dougal
2de5b14fe0
Fix bug which prevented the HS pushing events to the AS due to FrozenEvents
2015-03-02 15:36:37 +00:00
Erik Johnston
5358966a87
Use git aware version string in User-Agent and Server headers
2015-02-18 16:52:04 +00:00
Mark Haines
0d872f5aa6
Merge pull request #50 from matrix-org/application-services
...
Application Services
2015-02-13 15:06:14 +00:00
Erik Johnston
2bf0e85f3d
Use encode_canonical_json for http client
2015-02-11 17:34:23 +00:00
Kegan Dougal
f518324426
Minor tweaks based on PR feedback.
2015-02-11 16:41:16 +00:00
Erik Johnston
4ebbaf0d43
Blunty replace json with simplejson
2015-02-11 14:23:10 +00:00
Kegan Dougal
543e84fe70
Add SimpleHttpClient.put_json with the same semantics as get_json.
2015-02-04 17:39:51 +00:00
Kegan Dougal
6d3e4f4d0a
Update user/alias query APIs to use new format of SimpleHttpClient.get_json
2015-02-04 17:32:44 +00:00
Kegan Dougal
96d4bf9012
Modify API for SimpleHttpClient.get_json and update usages.
...
Previously, this would only return the HTTP body as JSON, and discard other
response information (e.g. the HTTP response code). This has now been changed
to throw a CodeMessageException on a non-2xx response, with the response code
and body, which can then be parsed as JSON.
Affected modules include:
- Registration/Login (when using an email for IS auth)
2015-02-04 17:07:31 +00:00
David Baker
c06a9063e1
Merge branch 'develop' into pushers
2015-01-13 13:15:51 +00:00
Mark Haines
c9d2cecac9
SYN-231: User agent header broken
2015-01-08 09:41:11 +00:00
Mark Haines
adb04b1e57
Update copyright notices
2015-01-06 13:21:39 +00:00
Mark Haines
24b5d01853
Include version in User-Agent and Server headers
2014-12-22 10:16:02 +00:00
David Baker
eb6aedf92c
More work on pushers. Attempt to do HTTP pokes. Not sure if the actual HTTP pokes work or not yet but the retry semantics are pretty good.
2014-11-21 12:21:00 +00:00
Mark Haines
610c2ea131
Fix pep8 and pyflakes warnings
2014-11-20 18:00:10 +00:00
David Baker
e377d33652
Separate out the matrix http client completely because just about all of its code it now separate from the simple case we need for standard HTTP(S)
2014-11-20 17:41:56 +00:00
David Baker
dc60eee50e
Refactor the HTTP clients a little.
2014-11-20 13:53:34 +00:00
David Baker
cf66532ac1
CaptchaServerHttpClient should extend the base, not matrix http client.
2014-11-20 12:48:21 +00:00
Mark Haines
3e54d70ae2
SYN-141: Encode query params as UTF-8.
2014-11-18 19:43:08 +00:00
Mark Haines
7c06399512
Merge branch 'develop' into request_logging
...
Conflicts:
synapse/config/logger.py
2014-10-30 11:13:58 +00:00
Mark Haines
7d709542ca
Fix pep8 warnings
2014-10-30 11:10:17 +00:00
Mark Haines
b29517bd01
Add a request-id to each log line
2014-10-30 01:21:33 +00:00
Mark Haines
4be99c2989
Add get_json method to 3pid http client. Better logging for errors in 3pid requests
2014-10-20 14:10:08 +01:00
Mark Haines
f74e850b5c
remove debugging logging for signing requests
2014-10-14 11:46:13 +01:00
Mark Haines
6684855767
Verify signatures for server2server requests
2014-10-13 14:37:46 +01:00
Mark Haines
10ef8e6e4b
SYN-75 sign at the request level rather than the transaction level
2014-10-13 11:49:55 +01:00
Mark Haines
693d0b8f45
Replace on_send_callback with something a bit clearer so that we can sign messages
2014-10-13 10:49:04 +01:00
Mark Haines
b9cdc443d7
Fix pyflakes errors
2014-10-02 14:37:30 +01:00
Mark Haines
574377636e
Add a keyword argument to get_json to avoid retrying on DNS failures. Rather than passing MatrixHttpClient.RETRY_DNS_LOOKUP_FAILURES as a fake query string parameter
2014-10-02 14:26:13 +01:00
Mark Haines
4f11518934
Split PlainHttpClient into separate clients for talking to Identity servers and talking to Capatcha servers
2014-10-02 14:03:26 +01:00
Mark Haines
c8d67beb9c
remove "red", "blue" and "green" server_name mappings
2014-10-01 15:52:07 +01:00
Erik Johnston
6ac0b4ade8
Fix 'age' key to update on retries
2014-09-15 16:24:03 +01:00
Kegan Dougal
1829b55bb0
Captchas now work on registration. Missing x-forwarded-for config arg support. Missing reloading a new captcha on the web client / displaying a sensible error message.
2014-09-05 19:18:23 -07:00
David Baker
d72ce4da64
Merge branch 'develop' of github.com:matrix-org/synapse into develop
...
Conflicts:
synapse/http/client.py
2014-09-03 18:25:17 +01:00
David Baker
d6ecbbdf0a
Add support for registering with a threepid to the HS (get credentials from the client and check them against an ID server).
2014-09-03 18:22:27 +01:00
Matthew Hodgson
8a7c1d6a00
fix the copyright holder from matrix.org to OpenMarket Ltd, as matrix.org hasn't been incorporated in time for launch.
2014-09-03 17:31:57 +01:00
Kegan Dougal
7fc84c7019
Make retrying requests on DNS failures configurable, and turn off retrying only in directory.get_association
2014-09-03 14:26:52 +01:00
Kegan Dougal
b1195c125f
hs: Updated synapse.http.client to handle DNSLookupErrors and bail immediately.
2014-09-03 13:51:17 +01:00
Mark Haines
5452a8ee29
Fix SSL for federation http client
2014-09-01 18:43:08 +01:00
Mark Haines
a53946a8a1
Enable SSL for s2s http client
2014-09-01 18:30:00 +01:00
Matthew Hodgson
a4da962bab
fix http client GET parameters; somehow missing named param. how could this have ever worked!?
2014-08-14 02:59:54 +01:00
Paul "LeoNerd" Evans
1ddae5f40b
Update get_json()'s documentation to match the actual observed behaviour
2014-08-13 17:23:49 +01:00
Matthew Hodgson
f98e6380f1
add in whitespace after copyright statements to improve legibility
2014-08-13 03:14:34 +01:00
matrix.org
4f475c7697
Reference Matrix Home Server
2014-08-12 15:10:52 +01:00
