Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-08-03 12:06:28 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1729 commits 62 branches 20 tags 10 MiB
Commit graph

1729 commits

This branch
This branch
All branches
Author SHA1 Message Date
Misty De Méo
a3471a2d73 Merge branch 'misty/simplify_mime_type_check' into qa 2025-07-18 14:06:07 -07:00
Misty De Méo
8f5d0ba528 worker: simplify response check 
Pre-fetching this with .get() calls lets us skip some repetitive
indexing.
 2025-07-18 13:55:00 -07:00
Barbara Miller
c3d7f408b5 Merge branch 'predup_type_playlist' into qa 2025-07-17 19:10:22 -07:00
Barbara Miller
f979bbf154 even more updates post-QA-deploy 2025-07-17 19:09:58 -07:00
Barbara Miller
55d8ae06fb Merge branch 'predup_type_playlist' into qa 2025-07-16 16:36:26 -07:00
Barbara Miller
3adae49be7 more updates post-QA-deploy 2025-07-16 16:30:05 -07:00
Barbara Miller
acf2bebb89 bump qa version 2025-07-15 19:32:22 -07:00
Barbara Miller
43b3e86589 Merge branch 'predup_type_playlist' into qa 2025-07-15 19:31:31 -07:00
Barbara Miller
3d377e7943 updates from qa deploy 2025-07-15 19:31:08 -07:00
Barbara Miller
fd8b1570a4 bump qa version 2025-07-15 17:21:54 -07:00
Barbara Miller
935241102b Merge branch 'predup_type_playlist' into qa 2025-07-15 17:16:30 -07:00
Barbara Miller
5b967b9738 log urls we skipped adding to outlinks 2025-07-15 17:16:00 -07:00
Barbara Miller
5b92338d98 Merge branch 'predup_type_playlist' into qa 2025-07-15 14:43:58 -07:00
Barbara Miller
3af8247277 updates for QA deploy 2025-07-15 14:42:53 -07:00
Barbara Miller
bc987b2a27 add get_recent_video_capture, mostly 2025-07-14 17:07:33 -07:00
Barbara Miller
3c1328ff53 save_video_capture_record 2025-07-14 16:19:38 -07:00
Barbara Miller
701707c7ce save video_record here? 2025-07-14 16:19:38 -07:00
Barbara Miller
825de5f728 worker._video_data and seed_id mostly 2025-07-14 16:19:38 -07:00
Barbara Miller
d4e0aa67ec more new fields 2025-07-14 16:19:38 -07:00
Barbara Miller
2422e9da04 def create_video_capture_record minimally 2025-07-14 16:19:38 -07:00
Barbara Miller
2fc30b029c dataclass VideoCaptureRecord instead 2025-07-14 16:19:38 -07:00
Barbara Miller
43151027f3 dataclass VideoDataRecord 2025-07-14 16:19:38 -07:00
Barbara Miller
b4b950c0fc self._video_data in worker 2025-07-14 16:19:38 -07:00
Barbara Miller
db17335ffb fix github ruff issues 2025-07-14 16:19:34 -07:00
Barbara Miller
7d58a9ae3b keep it simple for now 2025-07-14 16:17:29 -07:00
Barbara Miller
046db4b6cc VideoDataClient, generalized 2025-07-14 16:17:29 -07:00
Barbara Miller
de4e7e0c08 VideoDataWrapper refined 2025-07-14 16:17:29 -07:00
Barbara Miller
f21d312ca9 initial interface update 2025-07-14 16:17:29 -07:00
Barbara Miller
667feae559 ruff import block fix 2025-07-14 16:17:29 -07:00
Barbara Miller
8dcac47ae8 type hint get_video_captures 2025-07-14 16:17:29 -07:00
Barbara Miller
af1aaeee34 containing_page_url_pattern update 2025-07-14 16:17:29 -07:00
Barbara Miller
0526eb816c make psycopg dependency optional 2025-07-14 16:17:29 -07:00
Barbara Miller
203d86f402 use job_conf.get() 2025-07-14 16:17:29 -07:00
Barbara Miller
fe5ad0c31d VIDEO_DATA_SOURCE 2025-07-14 16:17:29 -07:00
Barbara Miller
f925660eb4 skip ternary op for now 2025-07-14 16:17:29 -07:00
Barbara Miller
fd0e0d3f30 variable VIDEO_DATA 2025-07-14 16:17:29 -07:00
Barbara Miller
c0db5b9403 CONCAT 2025-07-14 16:17:29 -07:00
Barbara Miller
92a6cacb5f ruff format updates 2025-07-14 16:17:29 -07:00
Barbara Miller
03b329cd2a formatting fix 2025-07-14 16:17:29 -07:00
Barbara Miller
55e446a41a initial commit 2025-07-14 16:17:29 -07:00
Misty De Méo
f9cc2ea48e ci: test with 3.14 beta
Some checks failed
Python Formatting Check / formatting (push) Has been cancelled
Details
Tests / Run tests (push) Has been cancelled
Details 
3.14 beta 4 is very late in the cycle, so it seems like a good time
for us to start testing with it to make sure we're ready.
 2025-07-10 09:47:18 -07:00
Misty De Méo
aea4286bd1 ci: use uv 2025-07-10 09:41:09 -07:00
Misty De Méo
7b691fe397 worker: skip audio content-types for media exclusion
Some checks failed
Python Formatting Check / formatting (push) Has been cancelled
Details
Tests / Run tests (push) Has been cancelled
Details
2025-07-07 14:41:03 -07:00
Misty De Méo
a0f60c1051 Video exclusion: skip YouTube UMP packets too
Some checks failed
Python Formatting Check / formatting (push) Has been cancelled
Details
Tests / Run tests (push) Has been cancelled
Details 
In testing a page with an embedded YouTube video with video
exclusion enabled, I found that brozzler ended up capturing about
30MB of UMP packets. We should be filtering those out too.
 2025-06-26 17:13:24 -07:00
Misty De Méo
5ff893ddaf brozzler-new-site: add flag to disable videos 
This makes it easier to test the new video exclusion work.
 2025-06-26 14:38:15 -07:00
Misty De Méo
38f164dbc4 Makefile: remove target-version
Some checks are pending
Python Formatting Check / formatting (push) Waiting to run
Details
Tests / Run tests (push) Waiting to run
Details 
This can be inferred from our pyproject.toml.
 2025-06-26 09:04:51 -07:00
Misty De Méo
f9848efc1e tests: recognize CI=true 2025-06-26 09:04:51 -07:00
Misty De Méo
a4e5418e13 tests: enable format check 2025-06-26 09:04:51 -07:00
Misty De Méo
0f2c166e2a tests: use github-format in ci 2025-06-26 09:04:51 -07:00
Misty De Méo
422527d7e4 tests: ruff fixes
Some checks are pending
Python Formatting Check / formatting (push) Waiting to run
Details
Tests / Run tests (push) Waiting to run
Details
2025-06-25 15:50:39 -07:00