Barbara Miller
c1d9419093
minimal yt-dlp updates
2022-01-19 16:53:46 -08:00
Barbara Miller
040a942ef2
Merge branch 'master' of github.com:internetarchive/brozzler
2022-01-03 16:37:40 -08:00
Alex Dempsey
427908e821
Merge pull request #233 from cclauss/codespell
...
Fix typos
2021-10-12 12:34:37 -07:00
Christian Clauss
a5ed291e65
Fix typos
2021-10-12 10:19:48 +02:00
Barbara Miller
4f301f4e03
Merge pull request #225 from internetarchive/wt-376-yt-user-page-fix
...
Added new extractor type to brozzler's youtube-dl playlist handling
2021-06-08 14:43:42 -07:00
Barbara Miller
c311fbb41f
bump version, update copyright
2021-05-25 17:14:21 -07:00
Barbara Miller
b59c4395ed
Merge pull request #223 from vbanos/fix-AddressValueError
...
Skip invalid outlink
2021-05-25 17:12:35 -07:00
Vangelis Banos
7aabc5f655
Skip invalid outlink
...
When one of the outlinks is `http://-1/ ` `urlcanon.whatwg` raises an
unhandled exception `ipaddress.AddressValueError` and the capture fails.
We can skip the problematic outlink and keep the rest without crashing.
2021-05-23 11:31:47 +00:00
Pravin Visakan
eabdeb0238
Added user page extractor type to ytdl monkeypatch
2021-05-04 16:50:38 -07:00
Barbara Miller
6290692ac4
Merge branch 'master' of github.com:internetarchive/brozzler
2020-12-02 14:34:30 -08:00
Barbara Miller
0f27c9995a
bump version
2020-10-29 17:12:14 -07:00
jkafader
5005c619f6
Merge pull request #211 from internetarchive/galgeek-websocket-url-timeout
...
configurable websocket url timeout, default 60
2020-10-29 17:08:48 -07:00
Barbara Miller
11c5cfa865
add param for Chrome.start
2020-10-21 15:39:46 -07:00
Barbara Miller
2d380cc8f2
Merge branch 'master' of github.com:internetarchive/brozzler
2020-10-14 22:06:11 -07:00
Barbara Miller
dc50fe1db2
Merge pull request #212 from internetarchive/bump-version-to-1.5.23
...
bump version after merge
2020-10-13 15:21:18 -07:00
Barbara Miller
052c3552ca
bump version after merge
2020-10-13 15:19:50 -07:00
Barbara Miller
f2ebdca597
configurable websocket url timeout, default 60
2020-10-13 15:12:32 -07:00
Barbara Miller
bb7594a14d
Merge pull request #209 from vbanos/outlinks-timeout
...
Thanks, @vbanos!
2020-10-13 15:01:55 -07:00
Barbara Miller
4372752bd6
Merge branch 'qa' of github.com:internetarchive/brozzler
2020-10-05 14:37:07 -07:00
Vangelis Banos
8addaf31d5
Add option extract_outlinks_timeout
...
`Browser.extract_outlinks` has a default `timeout=60` parm that cannot be
changed in any way. (It is always invoked using `extract_outlinks()`.
We add param `extract_outlinks_timeout=60` to `BrozzlerWorker` and
`Browser.browse_page` to allow that.
2020-10-04 15:39:30 +00:00
Barbara Miller
18d3f5f930
Merge pull request #208 from internetarchive/galgeek-patch-2
...
based on PR #207 — thanks @cclaus!
2020-09-21 18:06:03 -07:00
Barbara Miller
297eaac6dd
update travis.yml and test!
2020-09-21 17:08:39 -07:00
Barbara Miller
c744bb2f92
update copyright
2020-09-01 19:05:21 -07:00
Barbara Miller
d599778c27
Merge pull request #206 from internetarchive/galgeek-patch-1
...
bump version after merge
2020-08-05 09:24:28 -07:00
Barbara Miller
84d6bb43fa
bump version after merge
2020-08-05 09:23:58 -07:00
Barbara Miller
5a6ecb09d5
Merge pull request #205 from vbanos/behavior-timeout-zero
...
Skip loading behavior when behavior_timeout=0
behavior_timeout is an existing parameter to `Browser.browse_page`
2020-08-04 16:18:58 -07:00
Neil Minton
12913cccf0
Merge pull request #204 from galgeek/noplaylist-ydl
...
youtube-dl option noplaylist: True
2020-08-04 14:12:14 -04:00
Vangelis Banos
8b10587031
Skip loading behavior when behavior_timeout=0
...
The user may set `behavior_timeout=0`. This means that they don't want
to run the behavior. As it is now, Brozzler will invoke
`brozzler.behavior_script` to load the script and `self.run_behavior`
to execute it.
We will run the behavior using `Runtime.evaluate` but then it will be
terminated immediately because of timeout=0.
It is better to skip behavior loading and running when
`behavior_timeout=0`.
2020-08-04 06:27:21 +00:00
Barbara Miller
dc0d99470a
Merge pull request #203 from miku/update-readme-proxy
...
Thank you, @miku!
2020-07-28 13:43:19 -07:00
Martin Czygan
8e670ca814
readme: remove proxy from job configuration
...
It has been removed in 934190084c73699747cf3f4c4d2ee7e268927eae.
2020-07-28 22:21:05 +02:00
Barbara Miller
fa2fe1eadf
Merge branch 'noplaylist-ydl' into qa
2020-07-24 16:24:36 -07:00
Barbara Miller
e3a067cf60
youtube-dl option noplaylist: True
2020-07-24 16:22:50 -07:00
jkafader
1b9ebca13c
Merge pull request #202 from galgeek/limit_downloadThroughput
...
configurable limit for Chromium download throughput
2020-07-23 14:14:20 -07:00
Barbara Miller
c482dc4d3b
Merge branch 'fix_test_ydl_stitching' into qa
2020-07-14 10:53:40 -07:00
Barbara Miller
7f2d3485a6
try waiting a little
2020-07-14 10:53:07 -07:00
Barbara Miller
35b239c984
Merge branch 'limit_downloadThroughput' into qa
2020-07-14 10:13:17 -07:00
Barbara Miller
739d09294e
make configurable
2020-07-14 10:12:28 -07:00
Barbara Miller
36b4f80350
try SPN2 downloadThroughput limit
2020-07-14 10:12:28 -07:00
Barbara Miller
b6bf392773
Merge branch 'limit_downloadThroughput' into qa
2020-07-11 14:17:16 -07:00
Barbara Miller
d40bcb0497
Revert "travis_wait"
...
This reverts commit f462f9c8cdff1bad7fdded14169b80393dfeb4a2.
2020-07-11 14:14:26 -07:00
Barbara Miller
5329218f22
Un-Update .travis.yml
2020-07-09 14:39:33 -07:00
Barbara Miller
17ea3ea1c5
Update .travis.yml
2020-07-09 12:46:04 -07:00
Barbara Miller
c86ad357a8
Merge branch 'limit_downloadThroughput' into qa
2020-07-09 12:37:53 -07:00
Barbara Miller
f462f9c8cd
travis_wait
2020-07-09 12:36:55 -07:00
Barbara Miller
98d624acbe
Merge branch 'limit_downloadThroughput' into qa
2020-07-09 12:28:37 -07:00
Barbara Miller
dc1b809a6c
travis_wait
2020-07-09 12:26:15 -07:00
Barbara Miller
bef2730e7e
Merge branch 'limit_downloadThroughput' into qa
2020-06-27 10:22:48 -07:00
Barbara Miller
4936ce09d7
make configurable
2020-06-26 18:32:11 -07:00
Barbara Miller
754d024f7a
update (C)
2020-06-26 11:49:27 -07:00
Barbara Miller
96696099f6
Merge branch 'limit_downloadThroughput' into qa
2020-06-25 17:26:11 -07:00
