Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 00:29:53 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
560 Commits 69 Branches 15 Tags
Commit Graph

132 Commits

Author SHA1 Message Date
Noah Levitt
d82b40be68 fix typo 2016-10-06 18:20:41 -07:00
Noah Levitt
59a15d7f5c bump dev version after merging pull requests 2016-10-04 14:43:43 -07:00
Alex Osborne
5ac67fe513 Validate job conf against a Cerberus schema 2016-10-04 21:19:25 +11:00
Noah Levitt
15633be612 finish vagrant-brozzler-new-job.py 2016-10-03 18:17:35 -07:00
Noah Levitt
de5c520ad7 pin psutil version, too 2016-10-03 17:12:00 -07:00
Noah Levitt
f220707aa4 pin pillow to version 3.3.0, primarily because we have a wheel in devpi for that version 2016-10-03 17:06:58 -07:00
Noah Levitt
0e096dd4e4 don't try to read the browser's cookie database if the browser hasn't been started (which can happen if the page is simply fetched rather than browsed because it's not html) 2016-10-03 15:03:08 -07:00
Noah Levitt
bfd4c1f8c6 document a bunch of job settings 2016-09-29 16:15:44 -07:00
Noah Levitt
8c9a9c5666 starting on documenting job configuration 2016-09-29 12:03:16 -07:00
Noah Levitt
2462efc4ed replace vagrant-brozzler-new-site with python script that fills in default options and passes through others 2016-09-22 01:47:23 +01:00
Noah Levitt
cc9517cb45 add missing rethinkdb config file to ansible config 2016-09-22 01:45:28 +01:00
Noah Levitt
253122d061 new script runs brozzler-new-site queues a new site to brozzle on the vagrant brozzler deployment 2016-09-16 16:35:44 -07:00
Noah Levitt
38af0f347b working on including pywb in vagrant environment (not finished) 2016-09-14 17:08:00 -07:00
Noah Levitt
c864499a64 starting to create a framework for testing 2016-09-14 17:06:49 -07:00
Noah Levitt
be27b4e16e header comment tweak 2016-09-14 17:04:32 -07:00
Noah Levitt
85073ab82b new prog "brozzler-wayback" runs monkey-patched pywb 2016-09-14 17:04:01 -07:00
Noah Levitt
1c5c9417d2 avoid "Uncaught TypeError: Cannot read property 'querySelectorAll' of undefined" from outlinks script 2016-08-25 13:10:30 -07:00
Noah Levitt
ed7e01210d little readme fix 2016-08-12 17:02:41 -07:00
Noah Levitt
c9bc9fb67d for vagrant, static ansible inventory file, add brozzler-webconsole 2016-08-10 18:41:23 -07:00
Noah Levitt
f671cf4f11 add info to display of jobless sites in brozzler-webconsole; fix creation of "least_hops" index on the rethinkdb table "pages" 2016-08-09 11:24:58 -07:00
Noah Levitt
74b229cfb0 add arguments --webconsole-address --webconsole-port --pywb-address and change default ports 2016-08-09 10:43:52 -07:00
Noah Levitt
94a8e70226 list jobless sites on brozzler-webconsole front page 2016-08-08 17:44:41 -07:00
Noah Levitt
4fa1571bc5 run brozzler-webconsole inside brozzler-easy 2016-08-08 17:43:38 -07:00
Noah Levitt
531b26aabb add section about brozzler-easy to the readme 2016-08-05 18:28:30 -07:00
Noah Levitt
c04bf85f4e add --help to brozzler-webconsole 2016-08-05 18:19:15 -07:00
Noah Levitt
ba6b342e28 fix exception happening now that we have binary data in rethinkdb (the cookie db) "TypeError: <binary, 7168 bytes, '53 51 4c 69 74 65...'> is not JSON serializable" 2016-08-05 17:12:22 -07:00
Noah Levitt
a211cc0514 dev version number again 2016-08-04 17:34:58 -07:00
Noah Levitt
ae63369c3c another version for pypi 2016-08-04 17:33:47 -07:00
Noah Levitt
20f9934dd9 avoid "Uncaught RangeError: Maximum call stack size exceeded" compiling outlinks 2016-08-04 17:33:06 -07:00
Noah Levitt
7734399a22 back to a dev version number 2016-08-04 16:00:42 -07:00
Noah Levitt
57c0d84fbd bump version to 1.1b4 for pypi upload 2016-08-04 15:55:56 -07:00
Noah Levitt
e62055d7d6 logging tweak 2016-08-04 15:54:05 -07:00
Noah Levitt
65d97caa9a install brozzler.webconsole package 2016-07-29 12:56:10 -05:00
Noah Levitt
cfc18e6845 add docstring to _chain_chrome_messages, remove debug logging, tweak name of websock thread 2016-07-28 20:29:11 -05:00
Noah Levitt
2046ee36e0 add a timeout to the one post-behavior step that didn't already have one (getting a screenshot), and majorly refactored the post-behavior code to incorporate timeouts automatically into each step, and hopefully make it easier to follow 2016-07-28 19:59:28 -05:00
Noah Levitt
b2b07b79a9 logging tweaks 2016-07-28 10:19:30 -05:00
Noah Levitt
dd2d8c89e3 reduce log level of messages from chrome, since it spews stuff that looks bad but usually isn't 2016-07-27 18:48:13 -05:00
Noah Levitt
041a4970ce back to a dev version number 2016-07-27 16:57:42 -05:00
Noah Levitt
d94a7c23b9 1.1b3 for upload to pypi 2016-07-27 16:53:10 -05:00
Noah Levitt
c4bdb6c1fd pass behavior template parameters on to behavior - fixes umbra's ability to log in with parameters received from amqp 2016-07-26 19:47:09 -05:00
Noah Levitt
127002b77d brozzler[easy] requires warcprox>=2.0b1 2016-07-21 19:14:11 -05:00
Noah Levitt
37bff5328b look for a sensible default chromium/chrome executable 2016-07-19 15:57:24 -05:00
Noah Levitt
c902a70450 tweak thread names 2016-07-19 14:33:57 -05:00
Noah Levitt
ac3a71742d convert domain specific rule url prefixes to our style of surt 2016-07-19 14:31:43 -05:00
Noah Levitt
7d9f019e67 have pywb support loading warc records from warc files still being written (look for foo.warc.gz.open) 2016-07-17 20:09:56 -05:00
Noah Levitt
b62d5a6350 install flash plugin for chromium 2016-07-13 15:23:50 -05:00
Noah Levitt
04e1e5277e make state dumping signal handler more robust (now you can kill -QUIT a thousand times in a row without causing problems) 2016-07-13 14:52:05 -05:00
Noah Levitt
c6e6b34e82 handle case where websocket connection is unexpectedly closed during the post-behavior phase 2016-07-06 18:17:01 -05:00
Noah Levitt
3bf3c80720 implement timeout and retries to work around issue where sometimes we receive no result message after requesting outlinks 2016-07-06 17:54:36 -05:00
Noah Levitt
be58fb46f7 forgot to commit easy.py, add pywb.py with support for pywb rethinkdb index, and make brozzler-easy also run pywb 2016-07-06 14:52:00 -05:00