Noah Levitt 140a441eb5 honor site proxy setting; remove brozzler-worker options that are now configured at the site level (and in the case of ignore_cert_errors, always on, no longer an option); use "reppy" library for robots.txt handling; fix some bugs

2015-07-16 17:19:12 -07:00

..

browse-url

update readme, s/umbra/brozzler/ in most places, delete non-brozzler stuff

2015-07-13 17:09:39 -07:00

brozzler-add-site

add support for supplying json blob defining site with configuration to brozzler-add-site

2015-07-16 14:48:01 -07:00

brozzler-hq

save screenshots as metadata records using warcprox PUTMETA (same format as kenji's wide crawl)

2015-07-15 16:32:02 -07:00

brozzler-worker

honor site proxy setting; remove brozzler-worker options that are now configured at the site level (and in the case of ignore_cert_errors, always on, no longer an option); use "reppy" library for robots.txt handling; fix some bugs

2015-07-16 17:19:12 -07:00