Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-07-26 16:25:43 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch '2407-deps-update' into qa

Browse source
This commit is contained in:
Barbara Miller 2024-07-23 17:59:33 -07:00
commit d8e2add554
2 changed files with 5 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

4
job-conf.rst
View file

@ -3,7 +3,7 @@ Brozzler Job Configuration
Jobs are used to brozzle multiple seeds and/or apply settings and scope rules, Jobs are used to brozzle multiple seeds and/or apply settings and scope rules,
as defined byusing YAML files. At least one seed URL must be specified. as defined byusing YAML files. At least one seed URL must be specified.
All other configurartions are optional. All other configurations are optional.
.. contents:: .. contents::
@ -44,7 +44,7 @@ How inheritance works
Most of the settings that apply to seeds can also be specified at the top Most of the settings that apply to seeds can also be specified at the top
level, in which case all seeds inherit those settings. If an option is level, in which case all seeds inherit those settings. If an option is
specified both at the top level and at the seed level, the results are merged. specified both at the top level and at the seed level, the results are merged.
In cases of coflict, the seed-level value takes precedence. In cases of conflict, the seed-level value takes precedence.
In the example yaml above, ``warcprox_meta`` is specified at the top level and In the example yaml above, ``warcprox_meta`` is specified at the top level and
at the seed level for the seed http://one.example.org/. At the top level we at the seed level for the seed http://one.example.org/. At the top level we

6
setup.py
View file

@ -34,7 +34,7 @@ def find_package_data(package):
setuptools.setup( setuptools.setup(
name="brozzler", name="brozzler",
version="1.5.51a1", version="1.5.52a1",
description="Distributed web crawling with browsers", description="Distributed web crawling with browsers",
url="https://github.com/internetarchive/brozzler", url="https://github.com/internetarchive/brozzler",
author="Noah Levitt", author="Noah Levitt",
@ -66,11 +66,11 @@ setuptools.setup(
}, },
install_requires=[ install_requires=[
"PyYAML>=5.1", "PyYAML>=5.1",
"yt_dlp<2023.11.16",
"cassandra-driver==3.29.1", "cassandra-driver==3.29.1",
"yt_dlp==2024.7.16",
"reppy==0.3.4", "reppy==0.3.4",
"requests>=2.21", "requests>=2.21",
"websocket-client>=0.39.0,<=0.48.0", "websocket-client==1.8.0",
"pillow>=5.2.0", "pillow>=5.2.0",
"urlcanon>=0.1.dev23", "urlcanon>=0.1.dev23",
"doublethink @ git+https://github.com/internetarchive/doublethink.git@Py311", "doublethink @ git+https://github.com/internetarchive/doublethink.git@Py311",