14 Commits

Author	SHA1	Message	Date
Noah Levitt	85a863b1e3	change argument to --amqp-url for clarity and consistency	2015-07-23 00:39:57 +00:00
Noah Levitt	b5cb94fc8b	some additional logging and error handling to avoid mysterious messages	2015-07-21 06:33:02 +00:00
Noah Levitt	923cd98652	save screenshots as metadata records using warcprox PUTMETA (same format as kenji's wide crawl)	2015-07-15 16:32:02 -07:00
Noah Levitt	9b13f0c34c	refactor hq code into hq module	2015-07-15 14:27:21 -07:00
Noah Levitt	fd0c3322ee	update readme, s/umbra/brozzler/ in most places, delete non-brozzler stuff	2015-07-13 17:09:39 -07:00
Noah Levitt	18ca996216	rudimentary robots.txt support	2015-07-13 15:56:54 -07:00
Noah Levitt	b0f3b8a5e3	clean shutdown for brozzler-hq	2015-07-11 18:18:54 -07:00
Noah Levitt	384120928c	set in_progress=0 for completed url	2015-07-11 13:24:38 -07:00
Noah Levitt	610f9c8cf4	add missing file hq.py, improve some logging, fix little race condition bug	2015-07-11 13:09:45 -07:00
Noah Levitt	bb3561a690	check scope (on hq side), fix buglets	2015-07-11 12:33:19 -07:00
Noah Levitt	1fb336cb2e	crawling outlinks not totally working	2015-07-11 02:29:19 -07:00
Noah Levitt	fd99764baa	brozzler-worker partially working	2015-07-10 21:07:47 -07:00
Noah Levitt	8aa1e6715a	feed seed url to the crawl url queue	2015-07-10 20:12:33 -07:00
Noah Levitt	1d068f4f86	starting work on brozzler crawl hq	2015-07-10 18:01:54 -07:00