5 Commits

Author	SHA1	Message	Date
Noah Levitt	2ba5bd4d4b	support adding extra http request headers	2015-07-17 13:45:27 -07:00
Noah Levitt	140a441eb5	honor site proxy setting; remove brozzler-worker options that are now configured at the site level (and in the case of ignore_cert_errors, always on, no longer an option); use "reppy" library for robots.txt handling; fix some bugs	2015-07-16 17:19:12 -07:00
Noah Levitt	fd0c3322ee	update readme, s/umbra/brozzler/ in most places, delete non-brozzler stuff	2015-07-13 17:09:39 -07:00
Noah Levitt	3eff099b16	determine if youtube-dl can do something with a url	2015-07-13 16:40:56 -07:00
Noah Levitt	783794ca37	basic of site/seed crawling with scoping	2015-07-09 18:36:07 -07:00