Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-23 08:09:48 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,578 Commits 69 Branches 15 Tags
Commit Graph

1578 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Adam Miller
493587ca2c fix: return ie_result and cleanup variable names to properly represent hop depth instead of redirects 2025-01-15 12:00:07 -08:00
Adam Miller
a250eb2b68 fix: ensure url is not a video when determining if we are in a redirect 2025-01-06 18:56:22 -08:00
Adam Miller
5be1b3b22a chore: formatting 2025-01-06 18:23:17 -08:00
Adam Miller
1596667919 chore: rewrite approach using process_ie_result 2025-01-06 18:20:30 -08:00
Adam Miller
426570b084 feat: Handle too many redirects as well 2025-01-06 11:30:46 -08:00
Adam Miller
2f86f1f853 feat: override yt-dlp generic extractor to add redirect loop detection logic 2025-01-02 16:55:35 -08:00
Barbara Miller
eb922f5155
bump version to 1.6.5 2024-12-12 14:01:21 -08:00
Barbara Miller
bfc4aac76a
Merge pull request #307 from galgeek/ytdlp_tmpdir 
brozzler yt-dlp should be able to specify a separate tempdir
 2024-12-12 13:59:51 -08:00
Barbara Miller
135f623fda
Merge pull request #308 from galgeek/retry_only_proxied 
retry only proxied urls
 2024-12-12 13:59:09 -08:00
Barbara Miller
5f82b8871d if ydl.is_youtube_host 2024-12-12 12:54:57 -08:00
Barbara Miller
d8ebf2824e minor edits 2024-12-12 12:42:12 -08:00
Barbara Miller
de1890eedc black'd 2024-12-12 11:56:03 -08:00
Barbara Miller
789fe8116c skip separate edits mostly 2024-12-12 11:48:26 -08:00
Barbara Miller
a86962e886 cli param for ytldp_tmpdir 2024-12-12 11:20:20 -08:00
Barbara Miller
d235b02abd PROXY_ATTEMPTS constant 2024-12-10 09:40:24 -08:00
Barbara Miller
a390c3fcac if ydl.isyoutubehost 2024-12-09 20:48:30 -08:00
Barbara Miller
47d59970a7 ... if isyoutubehost 2024-12-09 20:41:44 -08:00
Barbara Miller
8747bf9fd8 local var max_attempts 2024-12-09 20:38:23 -08:00
Barbara Miller
36d6a1e5ab define MAX_YTDLP_ATTEMPTS at head of file 2024-12-09 18:45:25 -08:00
Barbara Miller
1af418c192 black'd 2024-12-09 17:43:37 -08:00
Barbara Miller
b04fe476f7 skip caps for non-constants 2024-12-09 17:25:39 -08:00
Barbara Miller
a49b978c60 add ytdlp_tmp, and more... 2024-12-09 17:21:21 -08:00
Barbara Miller
f7938fe4bc retry only proxied urls 2024-12-06 17:34:28 -08:00
Barbara Miller
905c11d795 ytdlp should use /tmp 2024-12-06 17:17:57 -08:00
Adam Miller
73a9c9f823
Merge pull request #306 from internetarchive/adam/update_brozzler_version 
chore: Bump version to 1.6.4
 2024-12-05 09:22:06 -08:00
Adam Miller
42e2452363 chore: Bump version to 1.6.4 2024-12-05 09:14:11 -08:00
Adam Miller
18d3c8a697
Merge pull request #285 from internetarchive/adam/connection_failure_retry 
feat: Detect connection failures forwarded from warcprox and retry th…
 2024-12-04 15:38:28 -08:00
Adam Miller
210491c867
Merge pull request #305 from galgeek/chrome130fixes 
small fixes for chrome v.130
 2024-12-04 15:37:06 -08:00
Barbara Miller
c781f53e3f black'd 2024-12-02 15:19:50 -08:00
Adam Miller
1ffa6cbe0b fix: Ensure page is None when we're done retrying, so site gets disclaimed 2024-11-26 13:47:14 -08:00
Adam Miller
cb355d9159 Merge branch 'master' into adam/connection_failure_retry 2024-11-26 10:42:27 -08:00
Adam Miller
4b20d81b19 chore: Comment retry delay backoff calculation. 2024-11-26 10:26:47 -08:00
Barbara Miller
d9d09996b0
bump version to 1.6.3 2024-11-12 16:22:53 -08:00
Barbara Miller
f0c158b283
Merge pull request #303 from vbanos/http-https 
Disable automatic http to https transformation

My QA crawl looks good.
 2024-11-12 16:21:49 -08:00
Adam Miller
05ddc28ea8 Comment frontier query change 2024-11-12 15:05:49 -08:00
Adam Miller
f332e569be Merge branch 'master' into adam/connection_failure_retry 2024-11-12 14:59:49 -08:00
Adam Miller
ab488968c5 Adjust backoff timing to 60, 90, 135 depending on number of attempts 2024-11-12 13:07:07 -08:00
Adam Miller
0f100e6a71 revert browser.py and pull status from browser.websock_thread.page_status instead. 2024-11-12 13:06:09 -08:00
Vangelis Banos
a52765d2ca Disable automatic http to https transformation 
Chrome 130 automatically converts http to https even if the target URL
is http. We disable this behavior because some target sites simply don't
have https.
 2024-11-12 21:59:49 +02:00
Barbara Miller
d079409a5b
Merge pull request #302 from internetarchive/pypi_1dot6dot2 
minor updates for v.1.6.2, on pypi at https://pypi.org/project/brozzler/1.6.2/
 2024-11-05 17:37:32 -08:00
Barbara Miller
c4e5dc63fc minor updates for v.1.6.2 (for pypi) 2024-11-05 17:26:59 -08:00
Barbara Miller
07a6b9845a
bump version 2024-11-02 10:20:13 -07:00
Barbara Miller
b5177865c1
Merge pull request #300 from galgeek/extra_think 
updates for pypi & library use
 2024-11-02 10:16:43 -07:00
Barbara Miller
413fc13120 for pypi 2024-11-02 09:36:12 -07:00
Barbara Miller
2b4f40a3e6 facilitate library use 2024-11-02 09:35:03 -07:00
Barbara Miller
09ec0b1078 update copyright notice 2024-11-02 09:33:28 -07:00
Barbara Miller
0db478f704
Merge pull request #299 from galgeek/doublethink_up 
update doublethink dependency
 2024-10-31 11:21:41 -07:00
Barbara Miller
d1e6e0f632 update doublethink dependency 2024-10-31 11:04:40 -07:00
Barbara Miller
f4e6f6c1c3
Merge pull request #298 from internetarchive/galgeek-patch-3 
Update workflow publish-artifacts.yml
 2024-10-30 15:59:29 -07:00
Barbara Miller
781d7efc62
Merge pull request #297 from galgeek/master 
update README, yt-dlp, not youtube-dl
 2024-10-30 15:58:31 -07:00