brozzler

Git-Mirrors/brozzler

Fork 0

mirror of https://github.com/internetarchive/brozzler.git synced 2026-01-14 07:21:02 -05:00

Commit graph

2a9b68c55c try-login: use requestSubmit for form submissions master Armando Biagioni Neto 2025-12-17 11:56:22 -03:00
e2b70623a7

try-login: use requestSubmit for form submissions Armando Biagioni Neto 2025-12-17 11:56:22 -03:00
32bd27f998

Merge 1164bc68ef into eb25858b7b Wolfgang Faust 2026-01-13 05:41:16 +00:00
d13fef92ff

Merge e872a83d6b into eb25858b7b Wolfgang Faust 2026-01-13 05:17:26 +00:00
0f41cbf939

Merge e28236c5f8 into eb25858b7b Wolfgang Faust 2026-01-13 04:44:53 +00:00
eb0d0f6d70

Merge ab27094ac7 into eb25858b7b Adam Miller 2026-01-13 04:36:53 +00:00
5ea05897c4

Merge 8f5d0ba528 into eb25858b7b Misty De Méo 2026-01-13 02:40:27 +00:00
764883cd47

Merge 7ef4a04e50 into eb25858b7b Adam Miller 2026-01-13 01:24:59 +00:00
b757ad2c94

Merge c77c245f42 into eb25858b7b Karl Blumenthal 2026-01-12 21:47:23 +00:00
bbaf043eaa

Merge dd11f0b20d into eb25858b7b Vangelis Banos 2026-01-12 19:26:09 +00:00
eb25858b7b docs: remove proxy field from job-conf Misty De Méo 2026-01-12 10:50:29 -08:00
7419660979 docs: remove proxy field from job-conf Misty De Méo 2026-01-12 10:50:29 -08:00
114626841e

Merge a449b1e75f into 9d0c61fee6 Vangelis Banos 2026-01-10 21:49:50 -05:00
847c689a9e

Merge 5fa3c20455 into 9d0c61fee6 Corentin Barreau 2026-01-10 21:49:49 -05:00
62485f6fd1 fix: reverting retry count change qa Adam Miller 2026-01-06 13:20:28 -08:00
e6cf2148cc chore: log chrome error page, and fix retry attempt comparison Adam Miller 2026-01-06 11:40:31 -08:00
227838a7b8 chore: handle PageConnectionError in CLI mode Adam Miller 2026-01-06 11:39:50 -08:00
d647620e85 fix: attempting to detect chrome error pages and trigger retry logic Adam Miller 2025-12-23 15:06:03 -08:00
9d0c61fee6 deps: yt-dlp 2025.12.08 Misty De Méo 2025-12-19 12:28:37 -08:00
b74d7d6925

Merge pull request #438 from internetarchive/adam/handle-chrome-error-pages Adam Miller 2026-01-06 13:35:56 -08:00
7512c84e67 fix: reverting retry count change Adam Miller 2026-01-06 13:20:28 -08:00
d0ef974313 chore: log chrome error page, and fix retry attempt comparison Adam Miller 2026-01-06 11:40:31 -08:00
85267b8edb chore: handle PageConnectionError in CLI mode Adam Miller 2026-01-06 11:39:50 -08:00
991f1741fa fix: attempting to detect chrome error pages and trigger retry logic Adam Miller 2025-12-23 15:06:03 -08:00
99551f853a deps: yt-dlp 2025.12.08 Misty De Méo 2025-12-19 12:28:37 -08:00
2f623f7e6d behaviors: optionally collect outlinks while looping Misty De Méo 2025-12-17 15:32:20 -08:00
b6078ceee7 behaviors: allow extracting outlinks Misty De Méo 2025-12-16 11:04:26 -08:00
874163beec browser: correctly set outlinks type Misty De Méo 2025-12-18 16:21:17 -08:00
ae779eefa8 outlinks: simplify outlink parsing Misty De Méo 2025-12-17 16:22:23 -08:00
9da38cc40b behaviors: add loop control selector Misty De Méo 2025-12-16 17:58:46 -08:00
cca8eea285 browser: make outlink gathering reusable Misty De Méo 2025-12-17 12:12:08 -08:00
dc2873a3bc deps: urllib3 2.6.1 / brotli 1.2.0 / requests 2.32.5 Misty De Méo 2025-12-10 09:22:59 -08:00
7e9d7f1130 behaviors: optionally collect outlinks while looping Misty De Méo 2025-12-17 15:32:20 -08:00
93309f17d7 behaviors: optionally collect outlinks while looping Misty De Méo 2025-12-17 15:32:20 -08:00
00edb56aa1 behaviors: allow extracting outlinks Misty De Méo 2025-12-16 11:04:26 -08:00
023d7eea56 behaviors: allow extracting outlinks Misty De Méo 2025-12-16 11:04:26 -08:00
eec9234264 browser: correctly set outlinks type Misty De Méo 2025-12-18 16:21:17 -08:00
58e8e6ba13

Merge 45c7e6522c into 33fffdfefd TheTechRobo 2025-12-19 09:41:13 -05:00
95118f45b0 browser: correctly set outlinks type Misty De Méo 2025-12-18 16:21:17 -08:00
33fffdfefd outlinks: simplify outlink parsing Misty De Méo 2025-12-17 16:22:23 -08:00
cc64fb0587 outlinks: simplify outlink parsing Misty De Méo 2025-12-17 16:22:23 -08:00
93bb1a9a35 behaviors: add loop control selector Misty De Méo 2025-12-16 17:58:46 -08:00
42722db029 behaviors: add loop control selector Misty De Méo 2025-12-16 17:58:46 -08:00
bf692bf144 browser: make outlink gathering reusable Misty De Méo 2025-12-17 12:12:08 -08:00
1632f8bf05 browser: make outlink gathering reusable Misty De Méo 2025-12-17 12:12:08 -08:00
6578b84abf

Merge 4310912c74 into 47f5c06ee4 Adam Miller 2025-12-17 12:10:08 -05:00
ab27094ac7 chore: note that max_claimed_sites is optional adam/fix_reclaiming_of_abandoned_claims Adam Miller 2025-12-16 13:34:57 -08:00
935896e37c fix: allow reclaiming of sites claimed over an hour ago Adam Miller 2025-12-16 11:54:44 -08:00
47f5c06ee4 deps: urllib3 2.6.1 / brotli 1.2.0 / requests 2.32.5 Misty De Méo 2025-12-10 09:22:59 -08:00
f6947e5daa deps: urllib3 2.6.1 / brotli 1.2.0 / requests 2.32.5 Misty De Méo 2025-12-10 09:22:59 -08:00
2ef65ca0dd fix(deps): update dependency yt-dlp to v2025.12.8 renovate[bot] 2025-12-08 00:33:09 +00:00
8dfd25f61f

fix(deps): update dependency yt-dlp to v2025.12.8 renovate[bot] 2025-12-08 00:33:09 +00:00
a449b1e75f Use chrome -v only when running in DEBUG logging level vbanos 2025-12-04 12:04:33 +01:00
dd11f0b20d Remove chrome flag --disable-direct-npapi-requests vbanos 2025-12-03 19:35:11 +01:00
4e2098009f

Merge pull request #424 from vbanos/get-version-opt Alex Dempsey 2025-12-02 14:38:34 -08:00
7bf52b0b79 Optimize check_version vbanos 2025-12-02 22:45:30 +01:00
45c7e6522c Resolve upstream conflicts TheTechRobo 2025-11-21 15:51:42 -05:00
d12ed3af6a

Merge pull request #420 from vbanos/screenshot-limits Alex Dempsey 2025-11-19 10:44:26 -08:00
63dd934fa7 Set default height to 20k and fix formatting vbanos 2025-11-19 13:24:16 +01:00
c7f179438c

Merge pull request #423 from internetarchive/misty/release_1_8_1 v1.8.1 Adam Miller 2025-11-13 16:45:45 -08:00
aa36cb930b chore: address ci formatting Adam Miller 2025-11-13 11:19:04 -08:00
58ba082a21 fix: We were applying the max_sites_to_claim filter too early. Many sites in a single crawl prevent claimable sites from getting through. Adam Miller 2025-11-13 11:14:11 -08:00
771f553572 release: 1.8.1 Misty De Méo 2025-11-13 15:37:51 -08:00
78263af2f7

Merge pull request #422 from internetarchive/adam/fix_claim_sites_pre_filter Adam Miller 2025-11-13 11:30:20 -08:00
4430605ef1 chore: address ci formatting adam/fix_claim_sites_pre_filter Adam Miller 2025-11-13 11:19:04 -08:00
96942d40f9 fix: We were applying the max_sites_to_claim filter too early. Many sites in a single crawl prevent claimable sites from getting through. Adam Miller 2025-11-13 11:14:11 -08:00
5222fdacdc fix(deps): update dependency yt-dlp to v2025.11.12 renovate[bot] 2025-11-12 03:38:48 +00:00
789c7c56a4

fix(deps): update dependency yt-dlp to v2025.11.12 renovate[bot] 2025-11-12 03:38:48 +00:00
d5b66b8e78 Limit screenshot width and height vbanos 2025-11-10 12:27:06 +01:00
4d1fb31bc6 ci: install deno Misty De Méo 2025-11-07 14:48:15 -08:00
d54ddd8d67 ci: install deno Misty De Méo 2025-11-07 14:48:15 -08:00
3757d0681a ci: install deno Misty De Méo 2025-11-07 14:48:15 -08:00
5898d8604e note places we might check for duplicate display_id Barbara Miller 2025-10-30 11:31:49 -07:00
0f8ea3cb6d fix(deps): update dependency yt-dlp to v2025.10.22 renovate[bot] 2025-10-27 19:58:52 +00:00
1cfc80d670

fix(deps): update dependency yt-dlp to v2025.10.22 renovate[bot] 2025-10-27 19:58:52 +00:00
90a8dc5ebe deps: bump minimum python to 3.10 Misty De Méo 2025-10-27 09:31:10 -07:00
9a47de68ee deps: bump minimum python to 3.10 Misty De Méo 2025-10-27 09:31:10 -07:00
e5836a4de1 deps: bump minimum python to 3.10 Misty De Méo 2025-10-27 09:31:10 -07:00
47d8d4d19e fix(deps): update dependency yt-dlp to v2025.10.14 renovate[bot] 2025-10-15 01:50:05 +00:00
3405e611b3

fix(deps): update dependency yt-dlp to v2025.10.14 renovate[bot] 2025-10-15 01:50:05 +00:00
1678d163d4 deps: bump pluggy Misty De Méo 2025-10-08 13:54:30 -07:00
23de803f6f deps: bump pluggy Misty De Méo 2025-10-08 13:54:30 -07:00
3803ee7a55 Remove unused msg_id variable TheTechRobo 2025-10-07 21:42:02 -04:00
952260bc45 Automatically attach to new targets TheTechRobo 2025-10-07 21:38:40 -04:00
de7a1bcab4 Use global WebSocket instead of target-specific WebSocket TheTechRobo 2025-10-06 00:01:56 -04:00
44484583cb release: 1.8.0 v1.8.0 Misty De Méo 2025-10-07 12:04:34 -07:00
3587f6e486 pyproject: add Misty Misty De Méo 2025-10-07 12:04:20 -07:00
b377869c37 release: 1.8.0 Misty De Méo 2025-10-07 12:04:34 -07:00
61786a99b2 pyproject: add Misty Misty De Méo 2025-10-07 12:04:20 -07:00
04d06ca49e deps: bump locked yt-dlp Misty De Méo 2025-10-07 11:55:55 -07:00
ba9e4f1be7

Merge pull request #367 from galgeek/barbara/header_request_timeout_60 Barbara Miller 2025-10-07 12:00:15 -07:00
a95c754f90 deps: bump locked yt-dlp Misty De Méo 2025-10-07 11:55:55 -07:00
35153266a1

Merge branch 'master' into barbara/header_request_timeout_60 Barbara Miller 2025-10-07 11:00:39 -07:00
98a829f269 ssl: allow fetching pages needing legacy renegotiation Misty De Méo 2025-10-01 11:35:45 -07:00
da8b00016a ssl: allow fetching pages needing legacy renegotiation Misty De Méo 2025-10-01 11:35:45 -07:00
5cc8dd8ad0 ssl: allow fetching pages needing legacy renegotiation Misty De Méo 2025-10-01 11:35:45 -07:00
4aeca4ad91 fix(deps): update dependency yt-dlp to v2025.9.26 renovate[bot] 2025-09-27 00:28:59 +00:00
525fe5d047

fix(deps): update dependency yt-dlp to v2025.9.26 renovate[bot] 2025-09-27 00:28:59 +00:00
31cb217751 fix(deps): update dependency yt-dlp to v2025.9.23 renovate[bot] 2025-09-23 10:29:53 +00:00
d47ce51819

fix(deps): update dependency yt-dlp to v2025.9.23 renovate[bot] 2025-09-23 10:29:53 +00:00