9 Commits

Author	SHA1	Message	Date
Noah Levitt	f5acb6c34b	make requests library dependency explicity	2015-08-01 01:25:07 +00:00
Noah Levitt	a0a0b0ff2c	use nlevitt brozzler branch of youtube-dl	2015-07-28 01:10:39 +00:00
Noah Levitt	88f352efea	use new fork of youtube-dl with support for extra http headers on every request	2015-07-21 19:23:01 +00:00
Noah Levitt	38ddfe498d	require my "tweaks" branch of websocket-client	2015-07-20 16:06:47 -07:00
Noah Levitt	2ba5bd4d4b	support adding extra http request headers	2015-07-17 13:45:27 -07:00
Noah Levitt	140a441eb5	honor site proxy setting; remove brozzler-worker options that are now configured at the site level (and in the case of ignore_cert_errors, always on, no longer an option); use "reppy" library for robots.txt handling; fix some bugs	2015-07-16 17:19:12 -07:00
Noah Levitt	fd0c3322ee	update readme, s/umbra/brozzler/ in most places, delete non-brozzler stuff	2015-07-13 17:09:39 -07:00
Noah Levitt	3eff099b16	determine if youtube-dl can do something with a url	2015-07-13 16:40:56 -07:00
Noah Levitt	783794ca37	basic of site/seed crawling with scoping	2015-07-09 18:36:07 -07:00