1856 Commits

Author SHA1 Message Date
Misty De Méo
4627901a37 deps: install setuptools on python 3.12+
distutils was removed beginning in Python 3.12, but it's used at
runtime by rethinkdb 2.4.9. setuptools provides a copy of distutils,
so we should make sure to install it when we're on Python 3.12 or
newer until we're able to upgrade to a version of rethinkdb that
no longer needs it.

See: https://www.python.org/downloads/release/python-3120/
2025-03-13 09:46:49 -07:00
Gretchen Leigh Miller
d13add09e5 ruff import sorting pass + adding uv.lock (#342)
* ruff import sorting pass

* add uv.lock

* move comment back to its proper place
2025-03-13 09:46:31 -07:00
Misty De Méo
eae68a5b65 setup.py: bump qa version 2025-03-11 09:19:42 -07:00
Misty De Méo
c9c9bda869 fix: bind worker_id inside BrozzlerWorker
This ensures the parameter remains available within a multithreaded context.
2025-03-10 14:43:54 -07:00
Misty De Méo
bae29f7b9b setup.py: bump qa version 2025-03-07 11:04:16 -08:00
Misty De Méo
5aee7f50bf cli: use worker's yt-dlp check 2025-03-05 11:39:30 -08:00
Misty De Méo
96459fe766 Merge branch 'doublethink-optional' into qa 2025-03-05 11:18:58 -08:00
Misty De Méo
e7f6c874b7 CLI: add new --worker-id option
This adds a new commandline flag allowing the worker ID to be specified.
If present, it will be added to the global context so that it will be
included in every logging statement.

Previously, we only had some indirect values to tie logging statements
to specific workers, so this should make it easier to follow.
2025-03-05 11:12:58 -08:00
Misty De Méo
e6d0de8f3c CLI: enable log prefixing
This adds a commandline option which enables log level prefixing.
These prefixes enable log level-based filtering in journalctl when
present so long as logs are going to the journal, and
`SyslogLevelPrefix=` is set to `true` (which it is by default).

For documentation: https://manpages.debian.org/testing/libsystemd-dev/sd-daemon.3.en.html
2025-03-05 11:12:54 -08:00
Misty De Méo
a23dd6a923 chore: use ruff for formatting
There are a few minor changes here compared to black; it flagged
unnecessary string concatenations, and has slightly different
opinions on line length.
2025-03-05 11:12:46 -08:00
Misty De Méo
c5aa46174d ydl: never try if extra missing 2025-03-05 11:02:16 -08:00
Alex Dempsey
72e549694c Only import yt-dlp if we're using it 2025-03-05 11:02:16 -08:00
Alex Dempsey
01e19fdf70 Make tz-aware datetime of the epoch with stdlib 2025-03-05 11:02:16 -08:00
Misty De Méo
7b896be372 __init__.py: rework imports
Although doublethink is an optional dependency to allow brozzler to be
used as a library without it, in practice we had some mandatory import
statements that prevented brozzler from being imported without it.
This fixes that by gating off some of the imports and exports.

If doublethink is available, brozzler works as it is now. But if it
isn't, we make a few changes:

* brozzler.worker, brozzler.cli and brozzler.model reexports are
  disabled
* One brozzler.cli function, which is used outside brozzler's own cli,
  has been moved into brozzler's __init__.py. For compatibility, it's
  reexported from brozzler.cli.
2025-03-05 11:02:16 -08:00
Misty De Méo
b45e5dc096 CLI: add new --worker-id option
This adds a new commandline flag allowing the worker ID to be specified.
If present, it will be added to the global context so that it will be
included in every logging statement.

Previously, we only had some indirect values to tie logging statements
to specific workers, so this should make it easier to follow.
2025-03-05 11:01:50 -08:00
Misty De Méo
af0f3ed378 CLI: enable log prefixing
This adds a commandline option which enables log level prefixing.
These prefixes enable log level-based filtering in journalctl when
present so long as logs are going to the journal, and
`SyslogLevelPrefix=` is set to `true` (which it is by default).

For documentation: https://manpages.debian.org/testing/libsystemd-dev/sd-daemon.3.en.html
2025-03-05 11:01:50 -08:00
Misty De Méo
f384d0b830 deps: add dev deps to pyproject.toml 2025-03-05 10:07:29 -08:00
Misty De Méo
ffeaee7a01 chore: use ruff for formatting
Some checks are pending
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (3.12) (push) Waiting to run
Tests / Run tests (3.8) (push) Waiting to run
There are a few minor changes here compared to black; it flagged
unnecessary string concatenations, and has slightly different
opinions on line length.
2025-03-05 09:43:17 -08:00
Misty De Méo
ab8970ff3e test: add CI (#329)
This adds two CI runs: a quick one that happens for every pull
request and merge to master, and a longer one that happens daily.

This also adds a new installation group to setup.py because the
`easy` group isn't currently installable, and some of the dependencies
specified there need to be present for the tests to run.
2025-03-04 09:34:53 -08:00
Misty De Méo
c59b08df33
test: add CI (#329)
Some checks are pending
Python Formatting Check / formatting (push) Waiting to run
Tests / Run tests (3.12) (push) Waiting to run
Tests / Run tests (3.8) (push) Waiting to run
This adds two CI runs: a quick one that happens for every pull
request and merge to master, and a longer one that happens daily.

This also adds a new installation group to setup.py because the
`easy` group isn't currently installable, and some of the dependencies
specified there need to be present for the tests to run.
2025-03-04 09:34:23 -08:00
Barbara Miller
984a129b43
Merge pull request #333 from galgeek/barbara/rm_workflow_maybe
Some checks are pending
Python Formatting Check / formatting (push) Waiting to run
remove unused publish-artifacts workflow
2025-03-03 14:53:57 -08:00
Barbara Miller
968cac0e10 Merge branch 'barbara/rm_workflow_maybe' into qa 2025-03-03 14:46:47 -08:00
Barbara Miller
e6332c7f94 unused 2025-03-03 14:41:30 -08:00
Adam Miller
39b695dc70
Merge pull request #320 from internetarchive/adam/chrome-flags-disable-optimization-model-downloads
feat: add chrome flag to disable download of autocomplete optimizatio…
2025-03-03 14:09:29 -08:00
Barbara Miller
f4b4b4d6bb fix merge error 2025-03-03 14:07:30 -08:00
Barbara Miller
7580a32783 Merge remote-tracking branch 'upstream/adam/chrome-flags-disable-optimization-model-downloads' into qa 2025-03-03 14:04:47 -08:00
Adam Miller
ccfadc87c8 combine disable-features flags 2025-03-03 13:58:48 -08:00
Misty De Méo
3ddb1beb17 deps: remove easy group (#331)
This group isn't actually installable right now because of the jinja
dependency conflict.
2025-03-03 09:25:17 -08:00
Misty De Méo
e37c0ad78c
deps: remove easy group (#331)
This group isn't actually installable right now because of the jinja
dependency conflict.
2025-03-03 09:24:52 -08:00
Gretchen Leigh Miller
6ca1a62489
update minimum Python version in README (#327)
Some checks failed
Publish Artifacts / Build distribution 📦 (push) Has been cancelled
Python Formatting Check / formatting (push) Has been cancelled
2025-02-25 12:08:35 -08:00
Misty De Méo
c0a4412d80 Merge branch 'logging' into qa 2025-02-24 16:31:26 -08:00
Misty De Méo
23cee477dc
feat: set up structlog logging (#325)
Some checks are pending
Publish Artifacts / Build distribution 📦 (push) Waiting to run
Python Formatting Check / formatting (push) Waiting to run
This ports the logging from `logging` to `structlog`. This updates
all of the logger instantiations along with all of the places
`logging` was called. Data that was being inlined into log statements
has been broken out so that it's now structured arguments to the
log statements instead.
2025-02-24 16:31:09 -08:00
Misty De Méo
c6001d8799 cli: move logger name comment to docstring 2025-02-24 16:28:53 -08:00
Misty De Méo
72e4c29982 setup.py: bump qa version 2025-02-24 11:42:24 -08:00
Misty De Méo
3cdb62ab81 Merge branch 'logging' into qa 2025-02-24 11:42:05 -08:00
Misty De Méo
e141543667 logging: exception is a magic keyword 2025-02-24 11:41:51 -08:00
Misty De Méo
464aa201e4 Stop newline-joining log params 2025-02-21 16:18:06 -08:00
Misty De Méo
25256db003 Merge branch 'logging' into qa 2025-02-21 10:15:12 -08:00
Misty De Méo
baefea7b8e logging: use UTC 2025-02-21 10:14:38 -08:00
Misty De Méo
4b93e20737 chrome: clarify units 2025-02-21 10:08:12 -08:00
Misty De Méo
6f8ad56618 remove unnecessary json encode 2025-02-21 10:08:01 -08:00
Misty De Méo
e1fccb261b pass logger_name in remaining places 2025-02-21 10:07:43 -08:00
Misty De Méo
a35ad276aa bump qa version 2025-02-21 09:28:02 -08:00
Misty De Méo
5f1e92c23f Merge branch 'logging' into qa 2025-02-21 09:27:44 -08:00
Misty De Méo
b79054d27e logging: decorate location with callsite 2025-02-21 09:22:54 -08:00
Misty De Méo
f21245384e cli: add comment to logging.basicConfig 2025-02-21 09:17:05 -08:00
Misty De Méo
45f352dc86 add logger name to event dict 2025-02-21 09:17:05 -08:00
Misty De Méo
564307c7f8 worker: add thread to the binding 2025-02-21 09:17:05 -08:00
Misty De Méo
5bf831605a tests: remove unneeded logging setup
This was added in 7706bab8b824e0a18c2e883b712966faf2c5b271 to support
some test logging in the same commit. The log statements have since
been removed.
2025-02-21 09:17:05 -08:00
Misty De Méo
626a9530c9 remove trace monkeypatch 2025-02-21 09:17:05 -08:00