Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 08:39:59 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
447 Commits 69 Branches 15 Tags
Commit Graph

447 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt
1577cd8926 call clearInterval when umbraBehaviorFinished is about to return true (see 1ef528eea7) 2016-06-16 13:55:17 -05:00
Noah Levitt
98acc8dc92 copy over fec.gov behavior from umbra master 2016-06-16 13:53:28 -05:00
Noah Levitt
d75e8c394a switch flask requirement to recent release, suggest gunicorn for running the app 2016-06-15 22:00:39 +00:00
Noah Levitt
b0ed4b8128 Merge pull request #7 from galgeek/disable_extensions 
disable browser extensions
 2016-06-06 11:57:46 -07:00
Noah Levitt
c63c21c30a Merge pull request #6 from ato/document-config 
Document the job config format
 2016-06-06 11:56:45 -07:00
Barbara Miller
1c1237d07e disable browser extensions 2016-05-27 22:51:38 -07:00
Noah Levitt
92f8f7c16d Merge pull request #5 from ato/fix-brozzler-new-site 
brozzler-new-site: Fix warcprox_meta's default value and json import
 2016-05-17 10:14:48 -07:00
Alex Osborne
484805fbda proxy is not supposed to have http:// prefix 
Looks like the prefixes are added by BrozzleWorker._fetch_url()
 2016-05-17 16:20:38 +10:00
Alex Osborne
02af30edd4 Document the job config format 2016-05-17 15:20:09 +10:00
Alex Osborne
a939689d44 Fix warcprox_meta's default value and json import 2016-05-17 13:51:42 +10:00
Noah Levitt
182cbfd0ce bump version so we can upload to pypi and fix the readme 2016-05-11 12:10:23 -07:00
Noah Levitt
dd2211df31 yes you can install brozzler from the outside world now! 2016-05-11 12:09:09 -07:00
Noah Levitt
6f6216e432 catch exception from rethinkdb when unregistering from the service registry at shutdown 2016-05-11 00:46:50 +00:00
Noah Levitt
c6e0e7c507 correctly handle site with no pages (which means the seed was blocked by robots.txt) in frontier.seed_page 2016-05-11 00:45:47 +00:00
Noah Levitt
317a5eb99d without sudo, psutil.net_connections() raises psutil.AccessDenied on mac; in this case, silently try running chrome on the unvetted configured port 2016-05-09 17:25:14 -07:00
Noah Levitt
1141c5951e add psutil dependency 2016-05-09 17:19:53 -07:00
Noah Levitt
c12090b3ef oops, no "+" there 2016-05-07 01:46:36 +00:00
Noah Levitt
464da5c3a6 avoid errors with old versions of pip or non-utf-8 locales by specifying the encoding of README.rst 2016-05-07 01:46:15 +00:00
Noah Levitt
1445aa9976 make Site.warcprox_meta a special thing, replacing Site.extra_headers; this way, warcprox_meta is a dictionary in rethinkdb rather than a long json string 2016-05-05 23:24:10 +00:00
Noah Levitt
07e15e26bd Merge pull request #3 from internetarchive/AITFIVE-859 
browser.py - Check for open ports before starting Chrome. Open next a…
 2016-05-05 16:00:38 -07:00
Adam Miller
1f7f55a14a browser.py - Fix port search logic 2016-05-05 22:55:45 +00:00
Adam Miller
8e84465ff9 browser.py - Check for open ports before starting Chrome. Open next available on conflict 2016-05-05 22:31:07 +00:00
Noah Levitt
053767d393 bump version again 2016-05-05 10:37:58 -07:00
Noah Levitt
8d618ed135 refactor post-behavior stuff into separate interval function for clarity 2016-05-05 10:37:00 -07:00
Noah Levitt
1ef528eea7 do the clearInterval thing when umbraBehaviorFinished is about to return true on all the behaviors (that have that function)... for the record the impetus for this is to stop scrolling so we can take the screenshot 2016-05-05 10:35:06 -07:00
Noah Levitt
5b492ac6f1 remove old facebook behavior, replaced by facebook.js.template (missed this on commit cea192b) 2016-05-05 10:28:01 -07:00
Noah Levitt
5a2ea2cea4 make brozzle-page utility save the screenshot to a file 2016-05-05 10:10:53 -07:00
Noah Levitt
87af7eaa73 Merge pull request #2 from internetarchive/AITFIVE-832 
Restructure browser.py to take screenshot after behavior script.
 2016-05-05 10:08:21 -07:00
Noah Levitt
31356d526a Merge branch 'master' into AITFIVE-832 
* master:
  copy over latest behaviors and stuff from umbra
  support for host rules in outlink scoping
  recover from rethinkdb error updating service registry
 2016-05-05 10:06:12 -07:00
Noah Levitt
cea192b4b3 copy over latest behaviors and stuff from umbra 2016-05-05 00:58:26 -07:00
Adam Miller
6e4e28d2df Modifying default.js behavior to stop the interval function when umbraBehaviorFinished returns true 
We should do this in all behaviors ultimately to stop the behavior script upon completion
 2016-05-05 01:03:57 +00:00
Adam Miller
61cec15fff Restructure browser.py to take screenshot after behavior script. 2016-05-03 22:06:03 +00:00
Noah Levitt
0af00bb3d5 support for host rules in outlink scoping 2016-05-03 20:52:22 +00:00
Noah Levitt
1d21f2c307 recover from rethinkdb error updating service registry 2016-05-03 08:02:59 +00:00
Noah Levitt
f285be71fb new generator site_pages() iterates over a site's pages 2016-04-28 00:29:22 +00:00
Noah Levitt
abe2c244eb fix brozzler.svg symlink 2016-04-25 20:03:02 +00:00
Noah Levitt
df61e55b6b add license headers 2016-04-25 20:02:11 +00:00
Noah Levitt
e210d417fb add methods to get all sites for a job, seed page for a site 2016-04-25 17:01:56 +00:00
Noah Levitt
2c7c713f00 add "metadata" field to site object 2016-04-25 17:01:22 +00:00
Noah Levitt
8d9fc7d3e3 working on avoiding race condition resulting in multiple brozzler-workers claiming the same site 2016-04-22 01:27:50 +00:00
Noah Levitt
2825ffea15 support for extra "blocks" and "accepts" scope rules 2016-04-21 22:22:44 +00:00
Noah Levitt
68abb3cb94 log "behavior finished"/"hard timeout" only once 2016-04-21 22:02:50 +00:00
Noah Levitt
568a553432 use the uncanonicalized url as part of the sha1 input to generate the page id, since canonicalization was stripping off the #fragment, and we might want to crawl the same url with different fragments (and there's no option to GoogleURLCanonicalizer to not strip the fragment) 2016-04-21 22:01:49 +00:00
Noah Levitt
dd8f0d525d set read_mode=majority when claiming a site to brozzle, to avoid weird thing where brozzler keeps claiming site it's already working on (not sure this is the cause of the problem but i don't see why else it might happen) 2016-04-21 20:43:25 +00:00
Noah Levitt
1e52d1cf98 restore scoping out of urls with unsupported schemes 2016-04-21 11:40:08 -07:00
Noah Levitt
fee008266f support for one-hop-off (or n-hop-off) scoping 2016-04-21 17:41:59 +00:00
Noah Levitt
7bc726f717 fix bug preventing links from being extracted if hard timeout is reached 2016-04-20 17:24:18 -07:00
Noah Levitt
4bbbbcf138 fix bug where the first time a site was claimed, another brozzler-worker would claim it anyway (and find no pages to brozzle) 2016-04-21 00:21:08 +00:00
Noah Levitt
416aa064f8 don't know why some jobs were missing from the list, but with this change they all show up 2016-04-19 22:41:48 +00:00
Noah Levitt
b5f5581477 only list available services (ones with recent heartbeats) 2016-04-19 21:14:20 +00:00