9 Commits

Author	SHA1	Message	Date
Noah Levitt	18ca996216	rudimentary robots.txt support	2015-07-13 15:56:54 -07:00
Noah Levitt	b0f3b8a5e3	clean shutdown for brozzler-hq	2015-07-11 18:18:54 -07:00
Noah Levitt	384120928c	set in_progress=0 for completed url	2015-07-11 13:24:38 -07:00
Noah Levitt	610f9c8cf4	add missing file hq.py, improve some logging, fix little race condition bug	2015-07-11 13:09:45 -07:00
Noah Levitt	bb3561a690	check scope (on hq side), fix buglets	2015-07-11 12:33:19 -07:00
Noah Levitt	1fb336cb2e	crawling outlinks not totally working	2015-07-11 02:29:19 -07:00
Noah Levitt	fd99764baa	brozzler-worker partially working	2015-07-10 21:07:47 -07:00
Noah Levitt	8aa1e6715a	feed seed url to the crawl url queue	2015-07-10 20:12:33 -07:00
Noah Levitt	1d068f4f86	starting work on brozzler crawl hq	2015-07-10 18:01:54 -07:00