Noah Levitt 72816d1058 don't check robots.txt when scheduling a new site to be crawled, but mark the seed Page as needs_robots_check, and delegate the robots check to brozzler-worker; new test of robots.txt adherence		2016-11-16 12:23:59 -08:00
..
htdocs	robots.txt for testing	2016-11-16 12:12:17 -08:00
test_cluster.py	don't check robots.txt when scheduling a new site to be crawled, but mark the seed Page as needs_robots_check, and delegate the robots check to brozzler-worker; new test of robots.txt adherence	2016-11-16 12:23:59 -08:00
test_units.py	monkey-patch reppy to support substring user-agent matching	2016-11-16 11:41:34 -08:00