Noah Levitt d8904dc9e7 Merge branch 'master' into qa ...

* master:
  implement resilience to warcprox outage, i.e. deal with brozzler.ProxyError in brozzler-worker
  have _warcprox_write_record also raise ProxyError when appropriate, and test this

2017-04-18 17:54:21 -07:00

..

dashboard

parameterize command line entry points and add tests of --version, a rudimentary check that the commands at least run

2017-04-14 11:46:26 -07:00

js-templates

Merge branch 'master' into qa

2017-04-05 12:09:58 -07:00

__init__.py

raise new exception brozzler.ProxyError in case of proxy error browsing a page

2017-04-17 18:14:02 -07:00

behaviors.yaml

Merge branch 'ari-4960' into qa

2017-04-18 15:24:14 -07:00

browser.py

Merge branch 'master' into qa

2017-04-17 18:15:32 -07:00

chrome.py

let the OS pick an available port, to avoid what appear to be timing issues causing multiple browsers to choose the same port

2017-02-22 12:44:19 -08:00

cli.py

make brozzle-page respect --proxy (no test for this!)

2017-04-17 18:11:09 -07:00

easy.py

parameterize command line entry points and add tests of --version, a rudimentary check that the commands at least run

2017-04-14 11:46:26 -07:00

frontier.py

new command line utility brozzler-stop-crawl, with tests

2017-04-14 18:06:15 -07:00

job_schema.yaml

Refactor the way the proxy is configured. Job/site settings "proxy" and "enable_warcprox_features" are gone. Brozzler-worker now has mutually exclusive options --proxy and --warcprox-auto. --warcprox-auto means find an instance of warcprox in the service registry, and enable warcprox features. --proxy is provided, determines if proxy is warcprox by consulting http://{proxy_address}/status (see https://github.com/internetarchive/warcprox/commit/8caae0d7d3), and enables warcprox features if so.

2017-03-24 13:55:23 -07:00

model.py

consolidate job.py and site.py into model.py, and let Job and Site share the elapsed() method by way of a mixin

2017-03-29 18:49:04 -07:00

pywb.py

use urlcanon library for canonicalization, surtification, scope match rules

2017-03-15 14:59:51 -07:00

robots.py

raise brozzler.ProxyError in case of proxy error fetching robots.txt, doing youtube-dl, or doing raw fetch

2017-04-17 18:15:22 -07:00

worker.py

implement resilience to warcprox outage, i.e. deal with brozzler.ProxyError in brozzler-worker

2017-04-18 17:54:12 -07:00