Commit Graph

87 Commits

Author SHA1 Message Date
Richard van der Hoff
255c229f23 Work around TLS bug in twisted
Wrap up twisted's FileBodyProducer to work around
https://twistedmatrix.com/trac/ticket/8473. Hopefully this fixes
https://matrix.org/jira/browse/SYN-700.
2016-06-15 10:39:08 +01:00
Mark Haines
eb79110beb Clean up the blacklist/whitelist handling.
Always set the config key with an empty list, even if a list isn't specified.
This means that the codepaths are the same for both the empty list and
for a missing key. Since the behaviour is the same for both cases this
makes the code somewhat easier to reason about.
2016-05-16 13:03:59 +01:00
Matthew Hodgson
792def4928 add a url_preview_ip_range_whitelist config param so we can whitelist the matrix.org IP space 2016-05-01 12:44:24 +01:00
Erik Johnston
f505575f69 Make InsecureInterceptableContextFactory work with SpiderEndpoint 2016-04-19 16:08:14 +01:00
Matthew Hodgson
83b2f83da0 actually throw meaningful errors 2016-04-08 21:36:59 +01:00
Matthew Hodgson
dafef5a688 Add url_preview_enabled config option to turn on/off preview_url endpoint. defaults to off.
Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered.
Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered.
Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist
Add commentary and generally address PR feedback
2016-04-08 18:37:15 +01:00
Matthew Hodgson
8b98a7e8c3 pep8 2016-04-03 12:56:29 +01:00
Matthew Hodgson
7426c86eb8 add a persistent cache of URL lookups, and fix up the in-memory one to work 2016-04-03 00:31:57 +01:00
Matthew Hodgson
d1b154a10f support gzip compression, and don't pass through error msgs 2016-04-02 03:06:39 +01:00
Matthew Hodgson
683e564815 handle spidered relative images correctly 2016-03-31 23:52:58 +01:00
Matthew Hodgson
0d3d7de6fc sync in changes from matrixfederationclient 2016-03-31 12:42:27 +01:00
Matthew Hodgson
a8a5dd3b44 handle requests with missing content-length headers (e.g. YouTube) 2016-03-31 01:55:21 +01:00
Matthew Hodgson
721b2bfa85 implement redirects 2016-03-29 03:32:52 +01:00
Matthew Hodgson
64b4aead15 make it work 2016-03-29 03:13:25 +01:00
Matthew Hodgson
dd4287ca5d make it build 2016-03-29 02:07:57 +01:00
Matthew Hodgson
d9d48aad2d Merge branch 'develop' into matthew/preview_urls 2016-03-27 22:54:42 +01:00
David Baker
9a3c80a348 pep8 2016-03-08 10:09:07 +00:00
David Baker
7bcee4733a Encode unicode objects given to post_urlencode* otherwise urllib.urlencode chokes. 2016-03-08 10:04:38 +00:00
Matthew Hodgson
7dd0c1730a initial WIP of a tentative preview_url endpoint - incomplete, untested, experimental, etc. just putting it here for safekeeping for now 2016-01-24 18:47:27 -05:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
b8dd5b1a2d Verify third party ID server certificates 2015-10-16 14:54:54 +01:00
Erik Johnston
a80ef851f7 Fix previous merge to s/version_string/user_agent/ 2015-10-10 12:35:39 +01:00
Steven Hammerton
625e13bfde Add get_raw method to SimpleHttpClient, use this in CAS auth rather than requests 2015-10-10 10:54:19 +01:00
Daniel Wagner-Hall
492beb62a8 Use space not dash as delimiter 2015-10-06 09:53:33 -05:00
Daniel Wagner-Hall
e0b466bcfd Use space not dash as delimiter 2015-10-06 09:32:26 -05:00
Daniel Wagner-Hall
b28c7da0a4 Preserve version string in user agent 2015-10-05 20:49:39 -05:00
Daniel Wagner-Hall
8fc52bc56a Allow synapse's useragent to be customized
This will allow me to write tests which verify which server made HTTP
requests in a federation context.
2015-10-02 17:13:51 -05:00
Daniel Wagner-Hall
3bcbabc9fb Rename context factory
Mjark is officially no fun.
2015-09-15 15:46:22 +01:00
Daniel Wagner-Hall
6485f03d91 Fix random formatting 2015-09-09 13:05:00 +01:00
Daniel Wagner-Hall
81a93ddcc8 Allow configuration to ignore invalid SSL certs
This will be useful for sytest, and sytest only, hence the aggressive
config key name.
2015-09-09 12:02:07 +01:00
Mark Haines
78323ccdb3 Remove syutil dependency in favour of smaller single-purpose libraries 2015-08-24 16:17:38 +01:00
Erik Johnston
653533a3da Fix log context when sending requests 2015-06-19 11:46:49 +01:00
Erik Johnston
83f119a84a Log requests and responses sent via http.client 2015-06-15 13:14:12 +01:00
Erik Johnston
79e37a7ecb Correctly pass connection pool parameter 2015-05-28 16:48:53 +01:00
Erik Johnston
11f51e6ded Up maxPersistentPerHost count 2015-05-28 15:45:46 +01:00
Erik Johnston
086df80790 Add connection pooling to SimpleHttpClient 2015-05-28 15:43:21 +01:00
Erik Johnston
476899295f Change the way we do logging contexts so that they survive divergences 2015-05-08 16:32:18 +01:00
David Baker
59bf16eddc New registration for C/S API v2. Only ReCAPTCHA working currently. 2015-03-30 18:13:10 +01:00
Paul "LeoNerd" Evans
128cf2daf7 Appease pep8 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
b0cf867319 Use _ instead of . as a metric namespacing separator, for Prometheus 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
f9478e475b Rename Metrics' "keys" to "labels" 2015-03-12 16:24:51 +00:00
Paul "LeoNerd" Evans
7d72e44eb9 Add vector counters to HTTP clients and servers; count the requests by method and responses by method and response code 2015-03-12 16:24:50 +00:00
Paul "LeoNerd" Evans
c53ec53d80 Pull out all uses of the underlying HTTP user agent .request() method into a single wrapper function, to make adding metrics easier 2015-03-12 16:24:50 +00:00
Kegan Dougal
2de5b14fe0 Fix bug which prevented the HS pushing events to the AS due to FrozenEvents 2015-03-02 15:36:37 +00:00
Erik Johnston
5358966a87 Use git aware version string in User-Agent and Server headers 2015-02-18 16:52:04 +00:00
Mark Haines
0d872f5aa6 Merge pull request #50 from matrix-org/application-services
Application Services
2015-02-13 15:06:14 +00:00
Erik Johnston
2bf0e85f3d Use encode_canonical_json for http client 2015-02-11 17:34:23 +00:00
Kegan Dougal
f518324426 Minor tweaks based on PR feedback. 2015-02-11 16:41:16 +00:00
Erik Johnston
4ebbaf0d43 Blunty replace json with simplejson 2015-02-11 14:23:10 +00:00
Kegan Dougal
543e84fe70 Add SimpleHttpClient.put_json with the same semantics as get_json. 2015-02-04 17:39:51 +00:00