820 Commits

Author SHA1 Message Date
Barbara Miller
66dabbadb3 Merge branch 'simpledo' into qa 2017-01-06 16:31:18 -08:00
Barbara Miller
377fab785b simpledo first draft 2017-01-06 16:28:59 -08:00
Noah Levitt
4e7f9f8690 Merge branch 'master' into qa
* master:
  fix oversight including username/password in site config when starting a new job
  restore BrozzlerWorker built-in support for managing its own thread
  restore handling of 420 Reached limit, with a rudimentary test
  add import missing from test
  restore support for on_response and on_request, with an automated test for on_response
2017-01-06 13:03:25 -08:00
Noah Levitt
76b658747e fix oversight including username/password in site config when starting a new job 2017-01-06 13:03:09 -08:00
Noah Levitt
c2704b18be restore BrozzlerWorker built-in support for managing its own thread 2017-01-04 14:57:34 -08:00
Noah Levitt
70b67942a5 restore handling of 420 Reached limit, with a rudimentary test 2016-12-22 13:44:09 -08:00
Noah Levitt
e5fb6cb4b9 add import missing from test 2016-12-21 19:19:34 -08:00
Noah Levitt
eabb0fb114 restore support for on_response and on_request, with an automated test for on_response 2016-12-21 18:35:55 -08:00
Noah Levitt
3922258c50 convert clickGetPDFs and noguchi to jinja2 2016-12-21 18:23:53 -08:00
Noah Levitt
422a5ad726 Merge branch 'master' into qa
* master:
  need $DISPLAY set for test_brozzling.py
  restore handling of "aw snap" or "he's dead jim"
  add seed username/password parameters to job config schema
  loosen the find_available_port test slightly, since it seems to be not 100% predictable for reasons i haven't investigated
  convert mouseovers and simpleclicks to jinja2
  remove obsolete facebook login code
  convert behaviors to jinja2, move them to new subdir js-templates, along with javascript previously stored as a string in browser.py
  add hack for submitting a login form containing an element with name or id "submit", which masks the form submit() method
  how did i miss this file?
  forgot to git add new test data
  detect <input type="email"> as potential username field for login
  generalized support for login doing automatic detection of login form on a page
  yet more refactoring of browser.py, clearer separation of purpose, Browser class manages browsing, sends most of the messages to chrome, WebsockReceiverThread handles messages that come back from chrome
  bump version number in setup.py
  major refactoring of browsing code to make it easier to add functionality
  back to dev version number
  i dub thee 1.1b8
2016-12-21 18:11:56 -08:00
Noah Levitt
c90c73372e need $DISPLAY set for test_brozzling.py 2016-12-21 15:15:03 -08:00
Noah Levitt
f7427219cf restore handling of "aw snap" or "he's dead jim" 2016-12-21 14:21:20 -08:00
Noah Levitt
a5d48a9fdb add seed username/password parameters to job config schema 2016-12-20 18:06:20 -08:00
Noah Levitt
86d6060a2d loosen the find_available_port test slightly, since it seems to be not 100% predictable for reasons i haven't investigated 2016-12-20 17:52:21 -08:00
Noah Levitt
edf0a3a50d convert mouseovers and simpleclicks to jinja2 2016-12-20 17:34:29 -08:00
Noah Levitt
e2dbf68ccd remove obsolete facebook login code 2016-12-20 16:38:11 -08:00
Noah Levitt
a0b61408b9 convert behaviors to jinja2, move them to new subdir js-templates, along with javascript previously stored as a string in browser.py 2016-12-20 16:33:25 -08:00
Noah Levitt
06fd0a0d79 add hack for submitting a login form containing an element with name or id "submit", which masks the form submit() method 2016-12-20 11:24:26 -08:00
Noah Levitt
b24b229cb2 how did i miss this file? 2016-12-20 11:13:48 -08:00
Noah Levitt
7a40822e64 forgot to git add new test data 2016-12-19 18:10:07 -08:00
Noah Levitt
2f8f20bbb4 detect <input type="email"> as potential username field for login 2016-12-19 18:08:10 -08:00
Noah Levitt
86ac48d6c3 generalized support for login doing automatic detection of login form on a page 2016-12-19 17:30:09 -08:00
Noah Levitt
bc6e0d243f yet more refactoring of browser.py, clearer separation of purpose, Browser class manages browsing, sends most of the messages to chrome, WebsockReceiverThread handles messages that come back from chrome 2016-12-16 13:52:12 -08:00
Noah Levitt
534d2e63d6 bump version number in setup.py 2016-12-15 16:43:27 -08:00
Noah Levitt
c71854127d major refactoring of browsing code to make it easier to add functionality 2016-12-15 16:42:45 -08:00
Noah Levitt
ef8bc83928 Merge branch 'refactor-browsing' into qa
* refactor-browsing:
  don't log every little message from chrome
2016-12-15 13:21:38 -08:00
Noah Levitt
cb6a00f4f0 don't log every little message from chrome 2016-12-15 13:21:30 -08:00
Noah Levitt
f6333df6ef back to dev version number 2016-12-15 12:34:26 -08:00
Noah Levitt
85de2fad6a i dub thee 1.1b8 1.1b8 2016-12-15 12:33:34 -08:00
Noah Levitt
7a68599057 Merge branch 'refactor-browsing' into qa
* refactor-browsing:
  more shutdown tweaks
  improving shutdown process
  working on major refactoring of browser management
2016-12-15 12:28:21 -08:00
Noah Levitt
4186869bf9 Merge branch 'master' into qa
* master:
  fix bug handling page with zero outlinks
  avoid infinite loop in case youtube-dl encounters redirect loop (which can be ok if cookies have been set or something)
  brozzler logo svg with small default size
  travis-ci slack integration
  fix _find_available_port and its unit test
  little fixes
  avoid broken version of websocket-client to fix https://github.com/internetarchive/brozzler/issues/28
  wrong branch of warcprox in ansible install
  move cookie db management code into chrome.py
  move _find_available_ports to chrome.py, changing the way it works so that browser:9200 doesn't get stuck at 9201 forever, which pushes 9201 to 9202 etc, and add a unit test
  split Chrome class into its own module
  new utility brozzler-list-captures for looking up entries in the "captures" table
2016-12-15 12:07:29 -08:00
Noah Levitt
4bdad4729a more shutdown tweaks 2016-12-14 16:13:14 -08:00
Noah Levitt
5fa96b6438 improving shutdown process 2016-12-14 14:49:41 -08:00
Noah Levitt
f23f928c16 working on major refactoring of browser management 2016-12-09 16:50:11 -08:00
Noah Levitt
d68053764c fix bug handling page with zero outlinks 2016-12-09 16:43:23 -08:00
Noah Levitt
af1e1c75ec avoid infinite loop in case youtube-dl encounters redirect loop (which can be ok if cookies have been set or something) 2016-12-09 14:16:27 -08:00
Noah Levitt
f6a25aa4f0 brozzler logo svg with small default size 2016-12-08 15:16:02 -08:00
Noah Levitt
40b4d9bfe8 travis-ci slack integration 2016-12-07 14:46:29 -08:00
Noah Levitt
9bcec54f4b fix _find_available_port and its unit test 2016-12-07 14:08:34 -08:00
Noah Levitt
eed8b9ec30 little fixes 2016-12-07 11:20:10 -08:00
Noah Levitt
0b6c5346bd avoid broken version of websocket-client to fix https://github.com/internetarchive/brozzler/issues/28 2016-12-07 11:18:41 -08:00
Noah Levitt
e250c4ca89 wrong branch of warcprox in ansible install 2016-12-07 09:33:06 -08:00
Noah Levitt
d3063fbd2b move cookie db management code into chrome.py 2016-12-06 18:04:51 -08:00
Noah Levitt
ce03381b92 move _find_available_ports to chrome.py, changing the way it works so that browser:9200 doesn't get stuck at 9201 forever, which pushes 9201 to 9202 etc, and add a unit test 2016-12-06 17:12:20 -08:00
Noah Levitt
74009852d6 split Chrome class into its own module 2016-12-06 12:50:38 -08:00
Noah Levitt
3c43fdaced new utility brozzler-list-captures for looking up entries in the "captures" table 2016-11-30 00:52:14 +00:00
Noah Levitt
2eea50dcfb Merge branch 'master' into qa
* master:
  in warcprox 2.0b2, captures table field has been renamed to "record_length"
  remove flickr behavior, flickr is better off with the default behavior for now
  Update README.rst
  add travis-ci badge
2016-11-21 16:21:30 -08:00
Noah Levitt
9567c088c8 in warcprox 2.0b2, captures table field has been renamed to "record_length" 2016-11-21 16:21:21 -08:00
Noah Levitt
55c9ae07b7 remove flickr behavior, flickr is better off with the default behavior for now 2016-11-16 17:16:48 -08:00
Noah Levitt
899ee8a8dd Update README.rst 2016-11-16 12:26:50 -08:00