Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-05-12 11:32:16 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
brozzler - distributed browser-based web crawler
33 commits 64 branches 16 tags 7.5 MiB
Find a file
Noah Levitt 02fbe725cb cache parent url metadata and send back via amqp with child urls
2014-02-10 20:40:06 -08:00
bin Replace js evaluation with direct page navigation, add default for dump_queue 2014-01-28 00:10:31 -05:00
umbra cache parent url metadata and send back via amqp with child urls 2014-02-10 20:40:06 -08:00
.gitignore Some refactor/testing and utility scripts 2014-01-22 18:03:02 +00:00
README.md Update readme 2014-01-28 00:12:33 -05:00
setup.py Inccognito messes with currently running chromium instances, disable it 2014-01-23 18:26:20 -05:00

README.md

umbra

Browser automation via chrome debug protocol

Install

Install via pip from this repo.

Run

"umbra" script should be in bin/. load_url.py takes urls as arguments and puts them onto a rabbitmq queue dump_queue.py prints resources discovered by the browser and sent over the return queue.

On ubuntu, rabbitmq install with sudo apt-get install rabbitmq-server should automatically be set up for these three scripts to function on localhost ( the default amqp url ).