7 Commits

Author	SHA1	Message	Date
Noah Levitt	72816d1058	don't check robots.txt when scheduling a new site to be crawled, but mark the seed Page as needs_robots_check, and delegate the robots check to brozzler-worker; new test of robots.txt adherence	2016-11-16 12:23:59 -08:00
Noah Levitt	5ac8994a24	rename webconsole to dashboard	2016-11-04 17:46:23 -07:00
Mouse Reeve	2215aaab21	Use warcprox if enable_warcprox_features is true	2016-10-18 17:39:33 -07:00
Noah Levitt	a370e7b987	tiny fix, and now the test passes for me	2016-10-14 19:21:26 -07:00
Noah Levitt	27452990ee	toward getting initial tests to pass	2016-10-14 18:26:48 -07:00
Noah Levitt	56e651baeb	working on basic integration tests	2016-10-13 17:12:35 -07:00
Noah Levitt	c864499a64	starting to create a framework for testing	2016-09-14 17:06:49 -07:00