Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 08:39:59 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,700 Commits 69 Branches 15 Tags
Commit Graph

1700 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Adam Miller
6abc23d441 Merge branch 'adam/patch-yt-dlp-infinite-loop-bug' into qa 2025-01-06 11:38:44 -08:00
Adam Miller
426570b084 feat: Handle too many redirects as well 2025-01-06 11:30:46 -08:00
Barbara Miller
28009b5844 Merge remote-tracking branch 'upstream/adam/patch-yt-dlp-infinite-loop-bug' into qa 2025-01-02 17:49:27 -08:00
Adam Miller
2f86f1f853 feat: override yt-dlp generic extractor to add redirect loop detection logic 2025-01-02 16:55:35 -08:00
Barbara Miller
63c53641b8 remove cassandra references (currently unused) 2024-12-13 15:47:15 -08:00
Barbara Miller
eb922f5155
bump version to 1.6.5 2024-12-12 14:01:21 -08:00
Barbara Miller
bfc4aac76a
Merge pull request #307 from galgeek/ytdlp_tmpdir 
brozzler yt-dlp should be able to specify a separate tempdir
 2024-12-12 13:59:51 -08:00
Barbara Miller
135f623fda
Merge pull request #308 from galgeek/retry_only_proxied 
retry only proxied urls
 2024-12-12 13:59:09 -08:00
Barbara Miller
513f661501 Merge branch 'retry_only_proxied' into qa 2024-12-12 12:56:11 -08:00
Barbara Miller
5f82b8871d if ydl.is_youtube_host 2024-12-12 12:54:57 -08:00
Barbara Miller
d8ebf2824e minor edits 2024-12-12 12:42:12 -08:00
Barbara Miller
7f39dad627 Merge branch 'ytdlp_tmpdir' into qa 2024-12-12 11:56:56 -08:00
Barbara Miller
de1890eedc black'd 2024-12-12 11:56:03 -08:00
Barbara Miller
1603d4584b Merge branch 'ytdlp_tmpdir' into qa 2024-12-12 11:49:31 -08:00
Barbara Miller
789fe8116c skip separate edits mostly 2024-12-12 11:48:26 -08:00
Barbara Miller
a86962e886 cli param for ytldp_tmpdir 2024-12-12 11:20:20 -08:00
Barbara Miller
c2d6221b66 valid version id 2024-12-10 17:37:52 -08:00
Barbara Miller
d235b02abd PROXY_ATTEMPTS constant 2024-12-10 09:40:24 -08:00
Barbara Miller
a390c3fcac if ydl.isyoutubehost 2024-12-09 20:48:30 -08:00
Barbara Miller
47d59970a7 ... if isyoutubehost 2024-12-09 20:41:44 -08:00
Barbara Miller
8747bf9fd8 local var max_attempts 2024-12-09 20:38:23 -08:00
Barbara Miller
36d6a1e5ab define MAX_YTDLP_ATTEMPTS at head of file 2024-12-09 18:45:25 -08:00
Barbara Miller
1af418c192 black'd 2024-12-09 17:43:37 -08:00
Barbara Miller
b04fe476f7 skip caps for non-constants 2024-12-09 17:25:39 -08:00
Barbara Miller
a49b978c60 add ytdlp_tmp, and more... 2024-12-09 17:21:21 -08:00
Barbara Miller
8d0652bfed Merge branch 'retry_only_proxied' into qa 2024-12-06 18:50:19 -08:00
Barbara Miller
db56c8fb00 Merge branch 'ytdlp_tmpdir' into qa 2024-12-06 18:50:01 -08:00
Barbara Miller
f7938fe4bc retry only proxied urls 2024-12-06 17:34:28 -08:00
Barbara Miller
905c11d795 ytdlp should use /tmp 2024-12-06 17:17:57 -08:00
Adam Miller
73a9c9f823
Merge pull request #306 from internetarchive/adam/update_brozzler_version 
chore: Bump version to 1.6.4
 2024-12-05 09:22:06 -08:00
Adam Miller
42e2452363 chore: Bump version to 1.6.4 2024-12-05 09:14:11 -08:00
Adam Miller
18d3c8a697
Merge pull request #285 from internetarchive/adam/connection_failure_retry 
feat: Detect connection failures forwarded from warcprox and retry th…
 2024-12-04 15:38:28 -08:00
Adam Miller
210491c867
Merge pull request #305 from galgeek/chrome130fixes 
small fixes for chrome v.130
 2024-12-04 15:37:06 -08:00
Barbara Miller
c781f53e3f black'd 2024-12-02 15:19:50 -08:00
Barbara Miller
60c870d731 tweak qa version 2024-12-02 14:49:58 -08:00
Barbara Miller
edb3e51f6b fixes for chrome v.130 2024-12-02 14:48:22 -08:00
Adam Miller
59d9ece0f8 Merge branch 'adam/connection_failure_retry' into qa 2024-11-26 13:47:52 -08:00
Adam Miller
1ffa6cbe0b fix: Ensure page is None when we're done retrying, so site gets disclaimed 2024-11-26 13:47:14 -08:00
Adam Miller
f9f68caaba resolve merge conflicts 2024-11-26 10:52:06 -08:00
Adam Miller
cb355d9159 Merge branch 'master' into adam/connection_failure_retry 2024-11-26 10:42:27 -08:00
Adam Miller
4b20d81b19 chore: Comment retry delay backoff calculation. 2024-11-26 10:26:47 -08:00
Barbara Miller
d9d09996b0
bump version to 1.6.3 2024-11-12 16:22:53 -08:00
Barbara Miller
f0c158b283
Merge pull request #303 from vbanos/http-https 
Disable automatic http to https transformation

My QA crawl looks good.
 2024-11-12 16:21:49 -08:00
Adam Miller
05ddc28ea8 Comment frontier query change 2024-11-12 15:05:49 -08:00
Adam Miller
f332e569be Merge branch 'master' into adam/connection_failure_retry 2024-11-12 14:59:49 -08:00
Adam Miller
ab488968c5 Adjust backoff timing to 60, 90, 135 depending on number of attempts 2024-11-12 13:07:07 -08:00
Adam Miller
0f100e6a71 revert browser.py and pull status from browser.websock_thread.page_status instead. 2024-11-12 13:06:09 -08:00
Vangelis Banos
a52765d2ca Disable automatic http to https transformation 
Chrome 130 automatically converts http to https even if the target URL
is http. We disable this behavior because some target sites simply don't
have https.
 2024-11-12 21:59:49 +02:00
Barbara Miller
d079409a5b
Merge pull request #302 from internetarchive/pypi_1dot6dot2 
minor updates for v.1.6.2, on pypi at https://pypi.org/project/brozzler/1.6.2/
 2024-11-05 17:37:32 -08:00
Barbara Miller
c4e5dc63fc minor updates for v.1.6.2 (for pypi) 2024-11-05 17:26:59 -08:00