Noah Levitt
87de7750e6
Merge branch 'master' into qa
...
* master:
stopgap fix for problem where an attempt to save a screenshot of a url with a hash tag containing spaces or non-ascii characters would fail, causing the whole brozzle of the page to fail, and end up in a retry loop (better handling of hash tags is planned which will obviate this change)
2017-02-01 22:39:29 +00:00
Noah Levitt
ed2d58d87d
stopgap fix for problem where an attempt to save a screenshot of a url with a hash tag containing spaces or non-ascii characters would fail, causing the whole brozzle of the page to fail, and end up in a retry loop (better handling of hash tags is planned which will obviate this change)
2017-02-01 22:39:12 +00:00
Noah Levitt
1874434d60
Merge branch 'master' into qa
...
* master:
pywb support for thumbnail: and screenshot: urls
fix oops
2017-01-31 10:26:48 -08:00
Noah Levitt
5c684779e5
pywb support for thumbnail: and screenshot: urls
2017-01-31 10:26:38 -08:00
Noah Levitt
8f5003b784
fix oops
2017-01-30 23:47:39 -08:00
Barbara Miller
6bf8cfe893
Merge branch 'qa' of github.com:internetarchive/brozzler into qa
2017-01-30 20:52:58 -08:00
Noah Levitt
8df96ceb59
Merge branch 'master' into qa
...
* master:
new flag Page.blocked_by_robots
be more patient to avoid spurious warnings waiting for browser to start up
2017-01-30 11:39:37 -08:00
Noah Levitt
4b6831b464
new flag Page.blocked_by_robots
2017-01-30 10:43:25 -08:00
Noah Levitt
a8b564f100
be more patient to avoid spurious warnings waiting for browser to start up
2017-01-24 10:06:37 -08:00
Noah Levitt
8c116295ea
Merge branch 'master' into qa
...
* master:
restore ping_timeout argument to WebSocketApp.run_forever to fix problem of leaking websocket receiver threads hanging forever on select()
missed a spot
improve brozzler-dashboard logging; fix default wayback baseurl in brozzler dashboard (https://github.com/internetarchive/brozzler/issues/31 ); tweak arg parsing related stuff
avoid js errors in case site or job is not configured to keep stats
add travis-ci slack notification to internetarchive/brozzler channel
2017-01-24 09:56:14 -08:00
Noah Levitt
d22cc075e0
restore ping_timeout argument to WebSocketApp.run_forever to fix problem of leaking websocket receiver threads hanging forever on select()
2017-01-24 09:55:56 -08:00
Noah Levitt
5375b819dd
missed a spot
2017-01-20 23:59:31 -08:00
Noah Levitt
c3b637d244
improve brozzler-dashboard logging; fix default wayback baseurl in brozzler dashboard ( https://github.com/internetarchive/brozzler/issues/31 ); tweak arg parsing related stuff
2017-01-20 23:41:59 -08:00
Noah Levitt
095456aa27
avoid js errors in case site or job is not configured to keep stats
2017-01-20 23:36:23 -08:00
Noah Levitt
65f818e901
add travis-ci slack notification to internetarchive/brozzler channel
2017-01-16 12:44:12 -08:00
Noah Levitt
049c2688bf
Merge branch 'master' into qa
...
* master:
support for BROZZLER_RETHINKDB_SERVERS and BROZZLER_RETHINKDB_DB environment variables, honored by all the brozzler-* commands
2017-01-13 20:27:18 +00:00
Noah Levitt
037723fe2b
support for BROZZLER_RETHINKDB_SERVERS and BROZZLER_RETHINKDB_DB environment variables, honored by all the brozzler-* commands
2017-01-13 20:27:09 +00:00
Noah Levitt
58bac8fc83
Merge branch 'master' into qa
...
* master:
adapt to exception message from newer versions of chromium (e.g. 57.0.2981.0)
2017-01-13 12:08:09 -08:00
Noah Levitt
77c4dc1116
adapt to exception message from newer versions of chromium (e.g. 57.0.2981.0)
2017-01-13 12:08:00 -08:00
Noah Levitt
87eeaf7888
Merge branch 'master' into qa
...
* master:
tests for dismissal of javascript dialogs (alert, prompt, confirm)
dismiss alerts from the page being browsed (avoids hanging)
2017-01-13 11:46:52 -08:00
Noah Levitt
011d814ee2
tests for dismissal of javascript dialogs (alert, prompt, confirm)
2017-01-13 11:46:42 -08:00
Barbara Miller
c9982e8727
Merge branch 'simpledo' into qa
2017-01-13 11:19:55 -08:00
Barbara Miller
de55b54d14
isVisible
2017-01-13 11:19:24 -08:00
Noah Levitt
d2ed6b97a2
dismiss alerts from the page being browsed (avoids hanging)
2017-01-13 10:27:37 -08:00
Noah Levitt
77b34bd5b2
Merge branch 'master' into qa
...
* master:
simpleclicks - only click if element is visible, fixes spinning on moma.org sites
implement brozzler-list-pages
new cli utils brozzler-list-jobs and brozzler-list-sites
2017-01-12 23:23:57 -08:00
Noah Levitt
766441e65c
simpleclicks - only click if element is visible, fixes spinning on moma.org sites
2017-01-12 23:23:46 -08:00
Noah Levitt
38d9eee68d
implement brozzler-list-pages
2017-01-12 08:22:45 +00:00
Noah Levitt
184612332e
new cli utils brozzler-list-jobs and brozzler-list-sites
2017-01-12 07:50:58 +00:00
Noah Levitt
3a3eccf00e
Merge branch 'master' into qa
...
* master:
implement sha1 lookup and url prefix lookup for brozzler-list-captures
2017-01-12 01:26:19 +00:00
Noah Levitt
64a0ea879a
implement sha1 lookup and url prefix lookup for brozzler-list-captures
2017-01-12 01:26:09 +00:00
Noah Levitt
a99fc2fbe6
Merge branch 'master' into qa
...
* master:
catch exceptions parsing funky urls when scoping and extracting outlinks
2017-01-09 15:18:27 -08:00
Noah Levitt
32097a8f8b
catch exceptions parsing funky urls when scoping and extracting outlinks
2017-01-09 15:18:19 -08:00
Noah Levitt
755ca045b6
Merge branch 'master' into qa
...
* master:
fix bug where login form would not be detected in some cases when there was a non-login form earlier on the page
2017-01-09 11:40:46 -08:00
Noah Levitt
2486768830
fix bug where login form would not be detected in some cases when there was a non-login form earlier on the page
2017-01-09 11:40:30 -08:00
Noah Levitt
9f2efcbb4c
Merge branch 'master' into qa
...
* master:
reset browser shutdown flag when starting up
2017-01-06 17:57:21 -08:00
Noah Levitt
d0022fe7bf
reset browser shutdown flag when starting up
2017-01-06 17:57:11 -08:00
Noah Levitt
bd5d4a8c97
fix behaviors.yaml, switch simpledo behavior to jinja2 template
2017-01-06 17:30:24 -08:00
Noah Levitt
54e6001c49
fix behaviors.yaml, switch simpledo behavior to jinja2 template
2017-01-06 17:19:00 -08:00
Barbara Miller
501aec77fc
update for new behaviors set up
2017-01-06 17:12:54 -08:00
Barbara Miller
281bd538bf
Merge branch 'simpledo' into qa
2017-01-06 16:34:56 -08:00
Barbara Miller
d861888864
simpledo first draft
2017-01-06 16:33:04 -08:00
Barbara Miller
66dabbadb3
Merge branch 'simpledo' into qa
2017-01-06 16:31:18 -08:00
Barbara Miller
377fab785b
simpledo first draft
2017-01-06 16:28:59 -08:00
Noah Levitt
4e7f9f8690
Merge branch 'master' into qa
...
* master:
fix oversight including username/password in site config when starting a new job
restore BrozzlerWorker built-in support for managing its own thread
restore handling of 420 Reached limit, with a rudimentary test
add import missing from test
restore support for on_response and on_request, with an automated test for on_response
2017-01-06 13:03:25 -08:00
Noah Levitt
76b658747e
fix oversight including username/password in site config when starting a new job
2017-01-06 13:03:09 -08:00
Noah Levitt
c2704b18be
restore BrozzlerWorker built-in support for managing its own thread
2017-01-04 14:57:34 -08:00
Noah Levitt
70b67942a5
restore handling of 420 Reached limit, with a rudimentary test
2016-12-22 13:44:09 -08:00
Noah Levitt
e5fb6cb4b9
add import missing from test
2016-12-21 19:19:34 -08:00
Noah Levitt
eabb0fb114
restore support for on_response and on_request, with an automated test for on_response
2016-12-21 18:35:55 -08:00
Noah Levitt
3922258c50
convert clickGetPDFs and noguchi to jinja2
2016-12-21 18:23:53 -08:00
