Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-06-19 12:24:20 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
brozzler - distributed browser-based web crawler
108 commits 62 branches 20 tags 8.9 MiB
Find a file
vonrosen 2dc30cc8bc Merge pull request #23 from nlevitt/master 
improve robustness, refactor, tweaks
2014-05-28 12:38:11 -07:00
bin sigterm and sigint both shutdown request shutdown, which stops consuming urls and waits for active browsers to finish; a second sigint/sigterm immediately shuts down active browsers 2014-05-24 01:52:22 -07:00
umbra sigterm and sigint both shutdown request shutdown, which stops consuming urls and waits for active browsers to finish; a second sigint/sigterm immediately shuts down active browsers 2014-05-24 01:52:22 -07:00
.gitignore not sure why /bin/ et al were in .gitignore... replace with a couple of useful things 2014-05-20 17:06:26 -07:00
README.md improve readme, mentioning archive-it per kristine 2014-05-23 13:34:51 -07:00
setup.py improve helper utilities 2014-05-20 16:44:13 -07:00

README.md

umbra

Umbra is a browser automation tool, developed for the web archiving service https://archive-it.org/.

Umbra receives urls via AMQP. It opens them in the chrome or chromium browser, with which it communicates using the chrome remote debug protocol (see https://developer.chrome.com/devtools/docs/debugger-protocol). It runs javascript behaviors to simulate user interaction with the page. It publishes information about the the urls requested by the browser back to AMQP. The format of the incoming and outgoing AMQP messages is described in pydoc umbra.controller.

Umbra can be used with the Heritrix web crawler, using these heritrix modules:

Install

Install via pip from this repo, e.g.

pip install git+https://github.com/internetarchive/umbra.git

Umbra requires an AMQP messaging service like RabbitMQ. On Ubuntu, sudo apt-get install rabbitmq-server will install and start RabbitMQ at amqp://guest:guest@localhost:5672/%2f, which the default AMQP url for umbra.

Run

The command umbra will start umbra with default configuration. umbra --help describes all command line options.

Umbra also comes with these utilities:

  • browse-url - open urls in chrome/chromium and run behaviors (without involving AMQP)
  • queue-url - send url to umbra via AMQP
  • drain-queue - consume messages from AMQP queue