Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-23 08:09:48 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,589 Commits 69 Branches 15 Tags
Commit Graph

1589 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Adam Miller
0feac5cd07 feat: add timeout to header check 2025-02-04 16:21:28 -08:00
Barbara Miller
df4bd148d5
bump version and update copyright
Some checks failed
Publish Artifacts / Build distribution 📦 (push) Has been cancelled
Details
Python Formatting Check / formatting (push) Has been cancelled
Details
2025-01-23 16:26:16 -08:00
Barbara Miller
a749b2968b
Merge pull request #316 from galgeek/bmiller/shorter_behavior_timeout 
shorter behavior timeout
 2025-01-23 15:37:29 -08:00
Barbara Miller
5e701e9dbe
Merge pull request #315 from galgeek/bmiller/proxy_select 
yt-dlp proxy handling update
 2025-01-23 15:37:01 -08:00
Adam Miller
1e30b4f478
Merge pull request #312 from internetarchive/adam/patch-yt-dlp-infinite-loop-bug 
feat: override yt-dlp generic extractor to add redirect loop detectio…
 2025-01-23 15:30:56 -08:00
Barbara Miller
2905324435 behavior_timeout=300seconds 2025-01-23 14:56:44 -08:00
Barbara Miller
9e09782984 ytdlp_proxy_file param 2025-01-23 14:35:34 -08:00
Barbara Miller
b22349e281 black'd 2025-01-23 12:37:56 -08:00
Barbara Miller
baa33e3079 ytdlp_proxy 2025-01-23 12:17:07 -08:00
Barbara Miller
854970f4dd black'd 2025-01-23 11:21:05 -08:00
Barbara Miller
170377fe89 yt-dlp proxy handling update 2025-01-23 10:58:32 -08:00
Adam Miller
493587ca2c fix: return ie_result and cleanup variable names to properly represent hop depth instead of redirects 2025-01-15 12:00:07 -08:00
Adam Miller
a250eb2b68 fix: ensure url is not a video when determining if we are in a redirect 2025-01-06 18:56:22 -08:00
Adam Miller
5be1b3b22a chore: formatting 2025-01-06 18:23:17 -08:00
Adam Miller
1596667919 chore: rewrite approach using process_ie_result 2025-01-06 18:20:30 -08:00
Adam Miller
426570b084 feat: Handle too many redirects as well 2025-01-06 11:30:46 -08:00
Adam Miller
2f86f1f853 feat: override yt-dlp generic extractor to add redirect loop detection logic 2025-01-02 16:55:35 -08:00
Barbara Miller
eb922f5155
bump version to 1.6.5 2024-12-12 14:01:21 -08:00
Barbara Miller
bfc4aac76a
Merge pull request #307 from galgeek/ytdlp_tmpdir 
brozzler yt-dlp should be able to specify a separate tempdir
 2024-12-12 13:59:51 -08:00
Barbara Miller
135f623fda
Merge pull request #308 from galgeek/retry_only_proxied 
retry only proxied urls
 2024-12-12 13:59:09 -08:00
Barbara Miller
5f82b8871d if ydl.is_youtube_host 2024-12-12 12:54:57 -08:00
Barbara Miller
d8ebf2824e minor edits 2024-12-12 12:42:12 -08:00
Barbara Miller
de1890eedc black'd 2024-12-12 11:56:03 -08:00
Barbara Miller
789fe8116c skip separate edits mostly 2024-12-12 11:48:26 -08:00
Barbara Miller
a86962e886 cli param for ytldp_tmpdir 2024-12-12 11:20:20 -08:00
Barbara Miller
d235b02abd PROXY_ATTEMPTS constant 2024-12-10 09:40:24 -08:00
Barbara Miller
a390c3fcac if ydl.isyoutubehost 2024-12-09 20:48:30 -08:00
Barbara Miller
47d59970a7 ... if isyoutubehost 2024-12-09 20:41:44 -08:00
Barbara Miller
8747bf9fd8 local var max_attempts 2024-12-09 20:38:23 -08:00
Barbara Miller
36d6a1e5ab define MAX_YTDLP_ATTEMPTS at head of file 2024-12-09 18:45:25 -08:00
Barbara Miller
1af418c192 black'd 2024-12-09 17:43:37 -08:00
Barbara Miller
b04fe476f7 skip caps for non-constants 2024-12-09 17:25:39 -08:00
Barbara Miller
a49b978c60 add ytdlp_tmp, and more... 2024-12-09 17:21:21 -08:00
Barbara Miller
f7938fe4bc retry only proxied urls 2024-12-06 17:34:28 -08:00
Barbara Miller
905c11d795 ytdlp should use /tmp 2024-12-06 17:17:57 -08:00
Adam Miller
73a9c9f823
Merge pull request #306 from internetarchive/adam/update_brozzler_version 
chore: Bump version to 1.6.4
 2024-12-05 09:22:06 -08:00
Adam Miller
42e2452363 chore: Bump version to 1.6.4 2024-12-05 09:14:11 -08:00
Adam Miller
18d3c8a697
Merge pull request #285 from internetarchive/adam/connection_failure_retry 
feat: Detect connection failures forwarded from warcprox and retry th…
 2024-12-04 15:38:28 -08:00
Adam Miller
210491c867
Merge pull request #305 from galgeek/chrome130fixes 
small fixes for chrome v.130
 2024-12-04 15:37:06 -08:00
Barbara Miller
c781f53e3f black'd 2024-12-02 15:19:50 -08:00
Adam Miller
1ffa6cbe0b fix: Ensure page is None when we're done retrying, so site gets disclaimed 2024-11-26 13:47:14 -08:00
Adam Miller
cb355d9159 Merge branch 'master' into adam/connection_failure_retry 2024-11-26 10:42:27 -08:00
Adam Miller
4b20d81b19 chore: Comment retry delay backoff calculation. 2024-11-26 10:26:47 -08:00
Barbara Miller
d9d09996b0
bump version to 1.6.3 2024-11-12 16:22:53 -08:00
Barbara Miller
f0c158b283
Merge pull request #303 from vbanos/http-https 
Disable automatic http to https transformation

My QA crawl looks good.
 2024-11-12 16:21:49 -08:00
Adam Miller
05ddc28ea8 Comment frontier query change 2024-11-12 15:05:49 -08:00
Adam Miller
f332e569be Merge branch 'master' into adam/connection_failure_retry 2024-11-12 14:59:49 -08:00
Adam Miller
ab488968c5 Adjust backoff timing to 60, 90, 135 depending on number of attempts 2024-11-12 13:07:07 -08:00
Adam Miller
0f100e6a71 revert browser.py and pull status from browser.websock_thread.page_status instead. 2024-11-12 13:06:09 -08:00
Vangelis Banos
a52765d2ca Disable automatic http to https transformation 
Chrome 130 automatically converts http to https even if the target URL
is http. We disable this behavior because some target sites simply don't
have https.
 2024-11-12 21:59:49 +02:00