brozzler/job-conf.rst

brozzler job configuration
**************************

Jobs are defined using yaml files. Options may be specified either at the
top-level or on individual seeds. At least one seed url must be specified,
everything else is optional.

an example
==========

::

    id: myjob
    time_limit: 60 # seconds
    ignore_robots: false
    max_claimed_sites: 2
    warcprox_meta:
      warc-prefix: job1
      stats:
        buckets:
        - job1-stats
    metadata: {}
    seeds:
    - url: http://one.example.org/
      warcprox_meta:
        warc-prefix: job1-seed1
        stats:
          buckets:
          - job1-seed1-stats
    - url: http://two.example.org/
      time_limit: 30
    - url: http://three.example.org/
      time_limit: 10
      ignore_robots: true
      scope:
        surt: http://(org,example,

how inheritance works
=====================

Most of the available options apply to seeds. Such options can also be
specified at the top level, in which case the seeds inherit the options. If
an option is specified both at the top level and at the level of an individual
seed, the results are merged with the seed-level value taking precedence in
case of conflicts. It's probably easiest to make sense of this by way of an
example.

In the example yaml above, ``warcprox_meta`` is specified at the top level and
at the seed level for the seed http://one.example.org/. At the top level we
have::

  warcprox_meta:
    warc-prefix: job1
    stats:
      buckets:
      - job1-stats

At the seed level we have::

    warcprox_meta:
      warc-prefix: job1-seed1
      stats:
        buckets:
        - job1-seed1-stats

The merged configuration as applied to the seed http://one.example.org/ will
be::

    warcprox_meta:
      warc-prefix: job1-seed1
      stats:
        buckets:
        - job1-stats
        - job1-seed1-stats

Notice that:

- There is a collision on ``warc-prefix`` and the seed-level value wins.
- Since ``buckets`` is a list, the merged result includes all the values from
  both the top level and the seed level.

settings reference
==================

``id``
------
+-----------+--------+----------+--------------------------+
| scope     | type   | required | default                  |
+===========+========+==========+==========================+
| top-level | string | no       | *generated by rethinkdb* |
+-----------+--------+----------+--------------------------+
An arbitrary identifier for this job. Must be unique across this deployment of
brozzler.

``seeds``
---------
+-----------+------------------------+----------+---------+
| scope     | type                   | required | default |
+===========+========================+==========+=========+
| top-level | list (of dictionaries) | yes      | *n/a*   |
+-----------+------------------------+----------+---------+
List of seeds. Each item in the list is a dictionary (associative array) which
defines the seed. It must specify ``url`` (see below) and can additionally
specify any of the settings of scope *seed-level*.

``max_claimed_sites``
---------------------
+-----------+--------+----------+---------+
| scope     | type   | required | default |
+===========+========+==========+=========+
| top-level | number | no       | *none*  |
+-----------+--------+----------+---------+
Puts a cap on the number of sites belonging to a given job that can be brozzled
simultaneously across the cluster. Addresses the problem of a job with many
seeds starving out other jobs.

``url``
-------
+------------+--------+----------+---------+
| scope      | type   | required | default |
+============+========+==========+=========+
| seed-level | string | yes      | *n/a*   |
+------------+--------+----------+---------+
The seed url.

``metadata``
------------
+-----------------------+------------+----------+---------+
| scope                 | type       | required | default |
+=======================+============+==========+=========+
| seed-level, top-level | dictionary | no      | *none*   |
+-----------------------+------------+----------+---------+
Arbitrary information about the crawl job or site. Merely informative, not used
by brozzler for anything. Could be of use to some external process.

``time_limit``
--------------
+-----------------------+--------+----------+---------+
| scope                 | type   | required | default |
+=======================+========+==========+=========+
| seed-level, top-level | number | no       | *none*  |
+-----------------------+--------+----------+---------+
Time limit in seconds. If not specified, there no time limit. Time limit is
enforced at the seed level. If a time limit is specified at the top level, it
is inherited by each seed as described above, and enforced individually on each
seed.

``ignore_robots``
-----------------
+-----------------------+---------+----------+-----------+
| scope                 | type    | required | default   |
+=======================+=========+==========+===========+
| seed-level, top-level | boolean | no       | ``false`` |
+-----------------------+---------+----------+-----------+
If set to ``true``, brozzler will happily crawl pages that would otherwise be
blocked by robots.txt rules.

``user_agent``
--------------
+-----------------------+---------+----------+---------+
| scope                 | type    | required | default |
+=======================+=========+==========+=========+
| seed-level, top-level | string  | no       | *none*  |
+-----------------------+---------+----------+---------+
The ``User-Agent`` header brozzler will send to identify itself to web servers.
It's good ettiquette to include a project URL with a notice to webmasters that
explains why you're crawling, how to block the crawler robots.txt and how to
contact the operator if the crawl is causing problems.

``warcprox_meta``
-----------------
+-----------------------+------------+----------+-----------+
| scope                 | type       | required | default   |
+=======================+============+==========+===========+
| seed-level, top-level | dictionary | no       | ``false`` |
+-----------------------+------------+----------+-----------+
Specifies the Warcprox-Meta header to send with every request, if ``proxy`` is
configured. The value of the Warcprox-Meta header is a json blob. It is used to
pass settings and information to warcprox. Warcprox does not forward the header
on to the remote site. See the warcprox docs for more information (XXX not yet
written).

Brozzler takes the configured value of ``warcprox_meta``, converts it to
json and populates the Warcprox-Meta header with that value. For example::

    warcprox_meta:
      warc-prefix: job1-seed1
      stats:
        buckets:
        - job1-stats
        - job1-seed1-stats

becomes::

    Warcprox-Meta: {"warc-prefix":"job1-seed1","stats":{"buckets":["job1-stats","job1-seed1-stats"]}}

``scope``
---------
+-----------------------+------------+----------+-----------+
| scope                 | type       | required | default   |
+=======================+============+==========+===========+
| seed-level, top-level | dictionary | no       | ``false`` |
+-----------------------+------------+----------+-----------+
Scope rules. *TODO*

``surt``
--------
+-------------+--------+----------+---------------------------+
| scope       | type   | required | default                   |
+=============+========+==========+===========================+
| scope-level | string | no       | *generated from seed url* |
+-------------+--------+----------+---------------------------+

``accepts``
-----------
+-------------+------+----------+---------+
| scope       | type | required | default |
+=============+======+==========+=========+
| scope-level | list | no       | *none*  |
+-------------+------+----------+---------+

``blocks``
-----------
+-------------+------+----------+---------+
| scope       | type | required | default |
+=============+======+==========+=========+
| scope-level | list | no       | *none*  |
+-------------+------+----------+---------+