Noah Levitt
|
d916b68ab9
|
use the new api with brozzler.thread_accept_exceptions()
|
2017-04-24 20:02:34 -07:00 |
|
Noah Levitt
|
0953e6972e
|
refactor thread_raise safety to use a context manager
|
2017-04-24 19:51:51 -07:00 |
|
Noah Levitt
|
f140e5bdbd
|
allow this stupid test to fail
|
2017-04-21 12:17:11 -07:00 |
|
Noah Levitt
|
ba519d7288
|
improve messaging when brozzler-stop-crawl is passed nonexistent seed/job id
|
2017-04-20 18:04:17 -07:00 |
|
Noah Levitt
|
7706bab8b8
|
safen up brozzler.thread_raise() to avoid interrupting rethinkdb transactions and such
|
2017-04-20 17:08:16 -07:00 |
|
Noah Levitt
|
b3fa7a4e39
|
quote that shell meta character
|
2017-04-18 18:46:59 -07:00 |
|
Noah Levitt
|
426916a238
|
need warcprox in python path for travis tests now
|
2017-04-18 18:10:18 -07:00 |
|
Noah Levitt
|
8256a34b4f
|
implement resilience to warcprox outage, i.e. deal with brozzler.ProxyError in brozzler-worker
|
2017-04-18 17:54:12 -07:00 |
|
Noah Levitt
|
5603ff5380
|
have _warcprox_write_record also raise ProxyError when appropriate, and test this
|
2017-04-18 16:58:51 -07:00 |
|
Neil Minton
|
f541dce5c3
|
Crawl Google Calendar for fortstjames.ca
|
2017-04-18 15:22:33 -07:00 |
|
Noah Levitt
|
ac972d399f
|
fix robots.txt proxy down test by setting site.id (cached robots is stored by site.id, and other tests that ran earlier with no site.id were interfering); and test another kind of connection error, for whatever that's worth
|
2017-04-18 12:00:23 -07:00 |
|
Noah Levitt
|
dc43794363
|
raise brozzler.ProxyError in case of proxy error fetching robots.txt, doing youtube-dl, or doing raw fetch
|
2017-04-17 18:15:22 -07:00 |
|
Noah Levitt
|
349b41ab32
|
raise new exception brozzler.ProxyError in case of proxy error browsing a page
|
2017-04-17 18:14:02 -07:00 |
|
Noah Levitt
|
87a7301f4d
|
make brozzle-page respect --proxy (no test for this!)
|
2017-04-17 18:11:09 -07:00 |
|
Noah Levitt
|
0e90950de2
|
oops, version bump for previous commit
|
2017-04-17 18:10:56 -07:00 |
|
Noah Levitt
|
0884b4cd56
|
bubble up proxy errors fetching robots.txt, with unit test, and documentation
|
2017-04-17 16:47:05 -07:00 |
|
Noah Levitt
|
df7734f2ca
|
new command line utility brozzler-stop-crawl, with tests
|
2017-04-14 18:06:15 -07:00 |
|
Noah Levitt
|
fae60e9960
|
parameterize command line entry points and add tests of --version, a rudimentary check that the commands at least run
|
2017-04-14 11:46:26 -07:00 |
|
Noah Levitt
|
b3cf746f53
|
stupid version number bump
|
2017-04-05 17:01:52 -07:00 |
|
Noah Levitt
|
62917a6f1a
|
Revert "bump version number for last pull request"
This reverts commit d192fc269eddeb8b06888e95bb6e4a6639e34415.
|
2017-04-05 17:01:06 -07:00 |
|
Noah Levitt
|
d192fc269e
|
bump version number for last pull request
|
2017-04-05 16:15:24 -07:00 |
|
Barbara Miller
|
537eb1cf7f
|
Merge pull request #34 from galgeek/ARI-5193
mouseover for ky.gov sites
|
2017-04-05 16:13:57 -07:00 |
|
Noah Levitt
|
5bcd10c228
|
extract area/@href links, and add test for outlink extraction
|
2017-04-05 12:09:48 -07:00 |
|
Barbara Miller
|
847b68eaf4
|
add JIRA info
|
2017-04-04 15:52:03 -07:00 |
|
Barbara Miller
|
901321199c
|
mouseover for ky.gov sites
|
2017-03-31 15:48:01 -07:00 |
|
Noah Levitt
|
d4d3ef4fd3
|
ugh fix version number
|
2017-03-30 17:53:36 -07:00 |
|
Noah Levitt
|
125d77b8c4
|
consolidate job.py and site.py into model.py, and let Job and Site share the elapsed() method by way of a mixin
|
2017-03-29 18:49:04 -07:00 |
|
Noah Levitt
|
3d47805ec1
|
new model for crawling hashtags, each one is no longer a top-level page
|
2017-03-27 12:15:49 -07:00 |
|
Noah Levitt
|
a836269e95
|
remove some vestiges of old proxy stuff
|
2017-03-24 16:04:43 -07:00 |
|
Noah Levitt
|
a826fdc7ef
|
new test of frontier.seed_page
|
2017-03-24 15:45:40 -07:00 |
|
Noah Levitt
|
0e35de43b6
|
actually respect --proxy and --warcprox-auto options to brozzler-worker
|
2017-03-24 22:27:52 +00:00 |
|
Noah Levitt
|
934190084c
|
Refactor the way the proxy is configured. Job/site settings "proxy" and "enable_warcprox_features" are gone. Brozzler-worker now has mutually exclusive options --proxy and --warcprox-auto. --warcprox-auto means find an instance of warcprox in the service registry, and enable warcprox features. --proxy is provided, determines if proxy is warcprox by consulting http://{proxy_address}/status (see https://github.com/internetarchive/warcprox/commit/8caae0d7d3), and enables warcprox features if so.
|
2017-03-24 13:55:23 -07:00 |
|
Noah Levitt
|
9a2f181eb6
|
back to a dev version number
|
2017-03-22 16:12:39 -07:00 |
|
Noah Levitt
|
613dca29dc
|
1.1b10 since 1.1b9 has bugs :(
1.1b10
|
2017-03-22 16:11:26 -07:00 |
|
Noah Levitt
|
4ba25db684
|
ugh, avoid infinite recursion
|
2017-03-22 15:53:58 -07:00 |
|
Noah Levitt
|
34bb64297f
|
fix frontier tests now that enable_warcprox_features is simply omitted by default
|
2017-03-22 15:46:12 -07:00 |
|
Noah Levitt
|
4aa611af52
|
i dub thee 1.1b9
1.1b9
|
2017-03-22 15:25:55 -07:00 |
|
Noah Levitt
|
b63badea53
|
github didn't like that, how about a width in pixels
|
2017-03-22 15:23:47 -07:00 |
|
Noah Levitt
|
2e6fe9ccc0
|
maybe pypi supports RST image "scale"
|
2017-03-22 15:20:35 -07:00 |
|
Noah Levitt
|
aae810cc6e
|
fix brozzler-easy so that warcprox features are enabled automatically (feature was already there but broken)
|
2017-03-22 15:15:07 -07:00 |
|
Noah Levitt
|
603956ec41
|
restore accidentally deleted line of code
|
2017-03-21 13:08:18 -07:00 |
|
Noah Levitt
|
95ba334b89
|
initialize page.videos correctly in all cases
|
2017-03-21 11:10:57 -07:00 |
|
Noah Levitt
|
eeee523b18
|
three-value "brozzled" parameter for frontier.site_pages(); fix thing where every Site got a list of all the seeds from the job; and some more frontier tests to catch these kinds of things
|
2017-03-20 17:28:16 -07:00 |
|
Noah Levitt
|
0e9f4a0c26
|
forgot to add the new test data
|
2017-03-20 12:33:52 -07:00 |
|
Noah Levitt
|
e9c7606318
|
oops remove pdb call
|
2017-03-20 12:14:11 -07:00 |
|
Noah Levitt
|
13130bd9d9
|
save info about embedded videos in page document in rethinkdb
|
2017-03-20 11:49:11 -07:00 |
|
Noah Levitt
|
94ba56dca5
|
actually implement the brozzler-list-jobs --job option
|
2017-03-17 11:14:45 -07:00 |
|
Noah Levitt
|
0685c77d01
|
always save outlinks info on rethinkdb page object, get rid of 'remember_outlinks' option, to keep config simple, and because it's not a very expensive thing
|
2017-03-17 10:04:10 -07:00 |
|
Noah Levitt
|
701f7654a8
|
make brozzler-list-* a little more intuitive, maybe
|
2017-03-16 13:01:41 -07:00 |
|
Noah Levitt
|
6c81b40e28
|
if parent page has a redirect_url, check scope rules both with the parent_page original url and with the redirect url, with automated tests
|
2017-03-16 12:12:33 -07:00 |
|