Misty De Méo
10db4bd19f
renovate: disable everything but yt-dlp
2025-08-21 12:39:20 -07:00
Misty De Méo
9b7999989b
renovate: customizations
2025-08-21 12:39:20 -07:00
renovate[bot]
b889cedf64
Add renovate.json
2025-08-21 12:39:20 -07:00
Misty De Méo
972b816878
deps: warctools 5.0.1
...
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
Silences a noisy warning; no other changes.
2025-08-18 15:32:43 -07:00
Misty De Méo
6261ea15ad
tests: add some silenced warnings
...
These come from a dependency we can't affect right now.
2025-08-18 15:20:12 -07:00
Misty De Méo
940dadfc12
worker: add missing import
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
2025-07-30 14:17:30 -07:00
Misty De Méo
5ee31cd879
browser: fix json separators
2025-07-30 14:17:30 -07:00
TheTechRobo
08bb09ff06
Add --no-headless option to brozzle-page and brozzler-worker CLI
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
2025-07-28 15:04:00 -07:00
TheTechRobo
7d7968e833
Add headless option to Chrome.start
2025-07-28 15:04:00 -07:00
Misty De Méo
f719b61983
docs: bump README copyright year
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (push) Waiting to run
2025-07-28 14:19:46 -07:00
Misty De Méo
43b7e57147
docs: remove outdated README comment
2025-07-28 14:19:46 -07:00
Misty De Méo
4c77515063
deps: warctools 5.0.0
...
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
Needed for the warcprox import to work.
2025-07-21 12:40:11 -07:00
Misty De Méo
99575b03b4
ci: always run full test suite
...
We previously ran the full suite, including test_brozzling, on a daily
timer because it took an enormous amount of time to run. I'd been under
the impression this was because it *had* to take that long to do the
work it was performing, but it looks like it hadn't been necessary and
the suite has been sped up massively since. We can now run it in about
six and a half minutes, which is perfectly fine to run on every PR.
2025-07-21 12:40:11 -07:00
Misty De Méo
f54e9e382a
tests: fix invalid escape
...
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
This made the common mistake of putting `\.` instead of `\\.`
in a non-raw string.
2025-07-18 16:51:51 -07:00
Misty De Méo
60f363ca89
tests: mark frontier perf test xfail
...
This is failing for me in CI, but passing locally.
2025-07-18 16:32:55 -07:00
Misty De Méo
db5cc6758a
ci: run frontier tests
...
This was skipped before due to flakiness, but it seems to be both
reliable and fast enough to be tolerable. It takes about 30 seconds
to complete on my local machine.
2025-07-18 16:32:55 -07:00
Misty De Méo
dfcfed8ace
ci: skip manpage generation
...
This should speed up dependency installs.
2025-07-18 16:07:59 -07:00
Misty De Méo
cb2ee89aee
tests: fix out of date frontier fixture
2025-07-18 15:34:59 -07:00
Misty De Méo
306e55d61a
ci: fix daily run
...
I migrated our regular tests to use `uv`, but neglected to update
this config too.
2025-07-18 15:34:59 -07:00
Misty De Méo
0f0ae4fbc3
remove unnecessary imports, use find_spec
...
This was flagged by ruff check - if we just want to find out if a
package is available, and don't need to actually import it, we can
use importlib.util.find_spec() to resolve it. This can lead to a
moderate speedup too, since the import might be slow.
2025-07-18 15:09:45 -07:00
Misty De Méo
85ae741b5d
deps: bump ruff
2025-07-18 14:45:48 -07:00
Misty De Méo
f9cc2ea48e
ci: test with 3.14 beta
...
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
3.14 beta 4 is very late in the cycle, so it seems like a good time
for us to start testing with it to make sure we're ready.
2025-07-10 09:47:18 -07:00
Misty De Méo
aea4286bd1
ci: use uv
2025-07-10 09:41:09 -07:00
Misty De Méo
7b691fe397
worker: skip audio content-types for media exclusion
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
2025-07-07 14:41:03 -07:00
Misty De Méo
a0f60c1051
Video exclusion: skip YouTube UMP packets too
...
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
In testing a page with an embedded YouTube video with video
exclusion enabled, I found that brozzler ended up capturing about
30MB of UMP packets. We should be filtering those out too.
2025-06-26 17:13:24 -07:00
Misty De Méo
5ff893ddaf
brozzler-new-site: add flag to disable videos
...
This makes it easier to test the new video exclusion work.
2025-06-26 14:38:15 -07:00
Misty De Méo
38f164dbc4
Makefile: remove target-version
...
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (push) Waiting to run
This can be inferred from our pyproject.toml.
2025-06-26 09:04:51 -07:00
Misty De Méo
f9848efc1e
tests: recognize CI=true
2025-06-26 09:04:51 -07:00
Misty De Méo
a4e5418e13
tests: enable format check
2025-06-26 09:04:51 -07:00
Misty De Méo
0f2c166e2a
tests: use github-format in ci
2025-06-26 09:04:51 -07:00
Misty De Méo
422527d7e4
tests: ruff fixes
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (push) Waiting to run
2025-06-25 15:50:39 -07:00
Misty De Méo
70e4c3d7f6
worker: fix possibly-unbound status code
...
We assigned this inside an exception handler, and allow
processing to continue on after catching the exception.
2025-06-25 15:42:56 -07:00
Misty De Méo
d33df40283
gitignore: ignore warcprox files
...
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
These are created by some tests.
2025-06-12 15:45:42 -07:00
Misty De Méo
bee01d32b8
deps: yt-dlp 2025.05.22
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (push) Waiting to run
2025-06-12 13:14:38 -07:00
Misty De Méo
8b20ea91bb
Move classifiers from setup.py
2025-06-12 13:05:49 -07:00
Misty De Méo
33f60ce609
Drop Python 3.8 support
...
Python 3.8 is EOL since October. It's no longer supported by new versions
of yt-dlp, limiting video capture support. It's also no longer supported
by setuptools, which has complicated distribution - it's preventing us
from keeping packaging configuration up to date.
2025-06-12 12:55:17 -07:00
Misty De Méo
0227da6530
brozzler 1.7.0
2025-06-12 10:52:25 -07:00
Gretchen Leigh Miller
40613e35b4
WT-2950 Implement Seed-level video capture setting handling + Job-level PDF-only option
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
2025-06-04 13:25:48 -07:00
Misty De Méo
14ccd6f4e7
deps: specify more extras for yt-dlp
2025-06-04 13:21:34 -07:00
Misty De Méo
94920b8b98
ci: two fixes to dependabot action
...
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
These were necessary in qa for the automated yt-dlp action to run to completion.
2025-05-30 13:27:42 -07:00
Misty De Méo
f8ede3d605
ci: remember to merge after approving
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (push) Waiting to run
2025-05-29 16:40:55 -07:00
Misty De Méo
944dc4c478
ci: install chrome before uv sync
2025-05-29 16:40:55 -07:00
Misty De Méo
3513da068d
tests: separate out youtube tests
...
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
Right now we expect these YouTube tests to fail for reasons unrelated to
yt-dlp. We still want to try them, but we won't count them towards
capture failures.
2025-05-28 13:06:54 -07:00
Misty De Méo
7d3155652e
pyproject: remove dynamic fields
...
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (push) Waiting to run
At this point might as well move the rest of these into the pyproject.toml,
taking them out of setup.py entirely.
2025-05-27 16:53:44 -07:00
Misty De Méo
189f669998
deps: move to pyproject.toml
...
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (push) Waiting to run
Dependabot seems to be having trouble parsing our extras; see if
this fixes it.
2025-05-27 15:41:59 -07:00
Misty De Méo
943acd35d6
fix: dependabot.yml location
2025-05-27 15:18:43 -07:00
Misty De Méo
794f7dd98d
ci: set up a yt-dlp test script
...
This runs every time we get a new yt-dlp version - we test to see if
this script is able to download at least 3/5 out of a set of videos
we've defined. If it succeeds, we go ahead and automatically merge
the new yt-dlp version into the qa branch so that we can test
further.
2025-05-27 15:09:15 -07:00
Misty De Méo
b4d2726e54
dependabot: add yt-dlp in qa
...
Also sets up an auto-approve for dependabot PRs.
2025-05-27 15:09:15 -07:00
Misty De Méo
2984bd955b
warcprox 2.7.0
Python Formatting Check / formatting (push) Has been cancelled
Tests / Run tests (push) Has been cancelled
2025-05-23 15:51:53 -07:00
Misty De Méo
bed0599d6e
ci: fix publish needs write permissions
...
Applies fixes from https://github.com/internetarchive/warcprox/pull/220
2025-05-23 15:47:07 -07:00
