From eebbc1d2793a3b155098bb1631d8683c80974410 Mon Sep 17 00:00:00 2001 From: Karl-Rainer Blumenthal Date: Thu, 28 Jun 2018 12:59:22 -0400 Subject: [PATCH 1/3] Copy edits --- readme.rst | 44 ++++++++++---------------------------------- 1 file changed, 10 insertions(+), 34 deletions(-) diff --git a/readme.rst b/readme.rst index 6aeb5e9..9496508 100644 --- a/readme.rst +++ b/readme.rst @@ -8,14 +8,7 @@ =============== "browser" \| "crawler" = "brozzler" -Brozzler is a distributed web crawler (爬虫) that uses a real browser (chrome -or chromium) to fetch pages and embedded urls and to extract links. It also -uses `youtube-dl `_ to enhance media -capture capabilities. - -Brozzler is designed to work in conjunction with -`warcprox `_ for web -archiving. +Brozzler is a distributed web crawler (爬虫) that uses a real browser (Chrome or Chromium) to fetch pages and embedded URLs and to extract links. It employs `youtube-dl `_ to enhance media capture capabilities, warcprox to write content to Web ARChive (WARC) files, `rethinkdb `_ to index captured URLs, a native dashboard for crawl job monitoring, and a customized Python Wayback interface for archival replay. Requirements ------------ @@ -24,20 +17,12 @@ Requirements - RethinkDB deployment - Chromium or Google Chrome >= version 64 -Worth noting is that the browser requires a graphical environment to run. You -already have this on your laptop, but on a server it will probably require -deploying some additional infrastructure (typically X11; note that Xvfb does -not support screenshots; Xvnc4, from package vnc4server, does). The vagrant -configuration in the brozzler repository (still a work in progress) has an -example setup. +Note: The browser requires a graphical environment to run. When brozzler is run on a server, this may require deploying some additional infrastructure (typically X11; Xvfb does not support screenshots, however Xvnc4 from package vnc4server, does). The `vagrant configuration `_ in the brozzler repository (still a work in progress) has an example setup. Getting Started --------------- -The easiest way to get started with brozzler for web archiving is with -``brozzler-easy``. Brozzler-easy runs brozzler-worker, warcprox, -`pywb `_, and brozzler-dashboard, configured -to work with each other, in a single process. +The easiest way to get started with brozzler for web archiving is with ``brozzler-easy``. Brozzler-easy runs brozzler-worker, warcprox, brozzler wayback, and brozzler-dashboard, configured to work with each other in a single process. Mac instructions: @@ -60,8 +45,7 @@ Mac instructions: # start brozzler-easy brozzler-easy -At this point brozzler-easy will start brozzling your site. Results will be -immediately available for playback in pywb at http://localhost:8880/brozzler/. +At this point brozzler-easy will start archiving your site. Results will be immediately available for playback in pywb at http://localhost:8880/brozzler/. *Brozzler-easy demonstrates the full brozzler archival crawling workflow, but does not take advantage of brozzler's distributed nature.* @@ -88,9 +72,7 @@ Submit sites not tied to a job:: Job Configuration ----------------- -Jobs are defined using yaml files. Options may be specified either at the -top-level or on individual seeds. At least one seed url must be specified, -everything else is optional. For details, see ``_. +Brozzler jobs are defined using YAML files. Options may be specified either at the top-level or on individual seeds. At least one seed URL must be specified, however everything else is optional. For details, see ``_. :: @@ -127,14 +109,14 @@ To start the app, run brozzler-dashboard +At this point Brozzler Dashboard will be accessible at http://localhost:8000/. + See ``brozzler-dashboard --help`` for configuration options. Brozzler Wayback ---------------- -Brozzler comes with a customized version of -`pywb `_ which supports using the rethinkdb -"captures" table (populated by warcprox) as its index. +Brozzler comes with a customized version of `pywb `_, which supports using the rethinkdb "captures" table (populated by warcprox) as its index. To use, first install dependencies. @@ -172,9 +154,7 @@ Then browse http://localhost:8880/brozzler/. Headless Chrome (experimental) -------------------------------- -`Headless Chromium `_ -is now available in stable Chrome releases for 64-bit Linux and may be -used to run the browser without a visible window or X11 at all. +`Headless Chromium `_ is now available in stable Chrome releases for 64-bit Linux and may be used to run the browser without a visible window or X11. To try this out, create a wrapper script like ~/bin/chrome-headless.sh: @@ -191,11 +171,7 @@ option: chmod +x ~/bin/chrome-headless.sh brozzler-worker --chrome-exe ~/bin/chrome-headless.sh -Beware: Chrome's headless mode is still very new and has a number of -`unresolved issues. `_ -You may experience hangs or crashes with some types of content. Brozzler -has not had much testing with it. For the moment we recommend using -Chrome's regular mode instead. +Beware: Chrome's headless mode is still very new and has `unresolved issues `_. Its use with brozzler has not yet been extensively tested. You may experience hangs or crashes with some types of content. For the moment we recommend using Chrome's regular mode instead. License ------- From bd78e0723279f89c4ed33491f7a1c1c41842f4dd Mon Sep 17 00:00:00 2001 From: Karl-Rainer Blumenthal Date: Fri, 6 Jul 2018 15:24:12 -0400 Subject: [PATCH 2/3] Copy edits to job-conf readme Good reading and rampant pedantry! --- job-conf.rst | 156 ++++++++++++++++++++++++++------------------------- 1 file changed, 80 insertions(+), 76 deletions(-) diff --git a/job-conf.rst b/job-conf.rst index 403e821..5d1eb0a 100644 --- a/job-conf.rst +++ b/job-conf.rst @@ -1,8 +1,9 @@ Brozzler Job Configuration ************************** -Jobs are defined using yaml files. At least one seed url must be specified, -everything else is optional. +Jobs are used to brozzle multiple seeds and/or apply settings and scope rules, +as defined byusing YAML files. At least one seed URL must be specified. +All other configurartions are optional. .. contents:: @@ -42,9 +43,8 @@ How inheritance works Most of the settings that apply to seeds can also be specified at the top level, in which case all seeds inherit those settings. If an option is -specified both at the top level and at seed level, the results are merged with -the seed-level value taking precedence in case of conflicts. It's probably -easiest to make sense of this by way of an example. +specified both at the top level and at the seed level, the results are merged. +In cases of coflict, the seed-level value takes precedence. In the example yaml above, ``warcprox_meta`` is specified at the top level and at the seed level for the seed http://one.example.org/. At the top level we @@ -74,7 +74,7 @@ be:: - job1-stats - job1-seed1-stats -Notice that: +In this example: - There is a collision on ``warc-prefix`` and the seed-level value wins. - Since ``buckets`` is a list, the merged result includes all the values from @@ -120,8 +120,8 @@ specify any seed settings. Seed-level-only settings ------------------------ -These settings can be specified only at the seed level, unlike most seed -settings, which can also be specified at the top level. +These settings can be specified only at the seed level, unlike the settings +below that can also be specified at the top level. ``url`` ~~~~~~~ @@ -130,7 +130,7 @@ settings, which can also be specified at the top level. +========+==========+=========+ | string | yes | *n/a* | +--------+----------+---------+ -The seed url. Crawling starts here. +The seed URL. Brozzling starts here. ``username`` ~~~~~~~~~~~~ @@ -153,14 +153,14 @@ If set, used to populate automatically detected login forms. If ``username`` and ``password`` are configured for a seed, brozzler will look for a login form on each page it crawls for that seed. A form that has a single text or email field (the username), a single password field (````), -and has ``method="POST"`` is considered to be a login form. The form may have -other fields like checkboxes and hidden fields. For these, brozzler will leave +and has ``method="POST"`` is considered to be a login form. When forms have +other fields like checkboxes and/or hidden fields, brozzler will leave the default values in place. Brozzler submits login forms after page load. Then brozzling proceeds as usual. Seed-level / top-level settings ------------------------------- -These are seed settings that can also be speficied at the top level, in which +These are seed settings that can also be specified at the top level, in which case they are inherited by all seeds. ``metadata`` @@ -170,8 +170,9 @@ case they are inherited by all seeds. +============+==========+=========+ | dictionary | no | *none* | +------------+----------+---------+ -Arbitrary information about the crawl job or site. Merely informative, not used -by brozzler for anything. Could be of use to some external process. +Information about the crawl job or site. Could be useful for external +descriptive or informative metadata, but not used by brozzler in the course of +archiving. ``time_limit`` ~~~~~~~~~~~~~~ @@ -202,8 +203,9 @@ warcprox for archival crawling. +=========+==========+===========+ | boolean | no | ``false`` | +---------+----------+-----------+ -If set to ``true``, brozzler will happily crawl pages that would otherwise be -blocked by robots.txt rules. +If set to ``true``, brozzler will fetch pages that would otherwise be blocked +by `robots.txt rules +`_. ``user_agent`` ~~~~~~~~~~~~~~ @@ -213,9 +215,9 @@ blocked by robots.txt rules. | string | no | *none* | +---------+----------+---------+ The ``User-Agent`` header brozzler will send to identify itself to web servers. -It's good ettiquette to include a project URL with a notice to webmasters that -explains why you're crawling, how to block the crawler robots.txt and how to -contact the operator if the crawl is causing problems. +It is good ettiquette to include a project URL with a notice to webmasters that +explains why you are crawling, how to block the crawler via robots.txt, and how +to contact the operator if the crawl is causing problems. ``warcprox_meta`` ~~~~~~~~~~~~~~~~~ @@ -227,8 +229,9 @@ contact the operator if the crawl is causing problems. Specifies the ``Warcprox-Meta`` header to send with every request, if ``proxy`` is configured. The value of the ``Warcprox-Meta`` header is a json blob. It is used to pass settings and information to warcprox. Warcprox does not forward -the header on to the remote site. For full documentation on ``warcprox-meta`` -see https://github.com/internetarchive/warcprox/blob/master/api.rst#warcprox-meta-http-request-header +the header on to the remote site. For further explanation of this field and +its uses see +https://github.com/internetarchive/warcprox/blob/master/api.rst Brozzler takes the configured value of ``warcprox_meta``, converts it to json and populates the Warcprox-Meta header with that value. For example:: @@ -256,8 +259,8 @@ Scope specificaion for the seed. See the "Scoping" section which follows. Scoping ======= -The scope of a seed determines which links are scheduled for crawling and which -are not. Example:: +The scope of a seed determines which links are scheduled for crawling ("in +scope") and which are not. For example:: scope: accepts: @@ -288,71 +291,69 @@ then the scope rule as a whole matches. For example:: - domain: youngscholars.unimelb.edu.au substring: wp-login.php?action=logout -This rule applies if the domain of the url is "youngscholars.unimelb.edu.au" or +This rule applies if the domain of the URL is "youngscholars.unimelb.edu.au" or a subdomain, and the string "wp-login.php?action=logout" is found somewhere in -the url. +the URL. -Brozzler applies these logical steps to decide whether a url is in or out of +Brozzler applies these logical steps to decide whether a URL is in or out of scope: -1. If the number of hops from seed is greater than ``max_hops``, the url is +1. If the number of hops from seed is greater than ``max_hops``, the URL is **out of scope**. -2. Otherwise, if any ``block`` rule matches, the url is **out of scope**. -3. Otherwise, if any ``accept`` rule matches, the url is **in scope**. -4. Otherwise, if the url is at most ``max_hops_off`` hops from the last page - that was in scope thanks to an ``accept`` rule, the url is **in scope**. +2. Otherwise, if any ``block`` rule matches, the URL is **out of scope**. +3. Otherwise, if any ``accept`` rule matches, the URL is **in scope**. +4. Otherwise, if the URL is at most ``max_hops_off`` hops from the last page + that was in scope because of an ``accept`` rule, the url is **in scope**. 5. Otherwise (no rules match), the url is **out of scope**. -Notably, ``block`` rules take precedence over ``accept`` rules. +In cases of conflict, ``block`` rules take precedence over ``accept`` rules. -It may also be helpful to think about a list of scope rules as a boolean -expression. For example:: +Scope rules may be conceived as a boolean expression. For example:: blocks: - domain: youngscholars.unimelb.edu.au substring: wp-login.php?action=logout - domain: malware.us -means block the url IF:: +means block the URL IF:: ("domain: youngscholars.unimelb.edu.au" AND "substring: wp-login.php?action=logout") OR "domain: malware.us" -Automatic scoping based on seed urls +Automatic scoping based on seed URLs ------------------------------------ -Brozzler usually generates an ``accept`` scope rule based on the seed url. It +Brozzler usually generates an ``accept`` scope rule based on the seed URL. It does this to fulfill the usual expectation that everything "under" the seed will be crawled. -To generate the rule, brozzler canonicalizes the seed url using the `urlcanon +To generate the rule, brozzler canonicalizes the seed URL using the `urlcanon `_ library's "semantic" canonicalizer, then removes the query string if any, and finally serializes the result in SSURT -[1]_ form. For example, a seed url of +[1]_ form. For example, a seed URL of ``https://www.EXAMPLE.com:443/foo//bar?a=b&c=d#fdiap`` becomes ``com,example,www,//https:/foo/bar?a=b&c=d``. -If the url in the browser location bar at the end of brozzling the seed page -differs from the seed url, brozzler automatically adds a second ``accept`` rule -to ensure the site is in scope, as if the new url were the original seed url. -It does this so that, for example, if ``http://example.com/`` redirects to -``http://www.example.com/``, the rest of the ``www.example.com`` is in scope. - -Brozzler derives its general approach to the seed surt from Heritrix, but -differs in a few respects. +Brozzler derives its general approach to the seed surt from `heritrix +`_, but differs in a few respects. 1. Unlike heritrix, brozzler does not strip the path segment after the last slash. 2. Canonicalization does not attempt to match heritrix exactly, though it usually does match. -3. When generating a surt for an https url, heritrix changes the scheme to - http. For example, the heritrix surt for ``https://www.example.com/`` is +3. When generating a SURT for an HTTPS URL, heritrix changes the scheme to + HTTP. For example, the heritrix SURT for ``https://www.example.com/`` is ``http://(com,example,www,)`` and this means that all of ``http://www.example.com/*`` and ``https://www.example.com/*`` are in - scope. It also means that a manually specified surt with scheme "https" does + scope. It also means that a manually specified SURT with scheme "https" does not match anything. Brozzler does no scheme munging. -4. Brozzler identifies seed "redirects" by retrieving the url from the +4. Brozzler identifies seed "redirects" by retrieving the URL from the browser's location bar at the end of brozzling the seed page, whereas - heritrix follows http 3xx redirects. -5. Brozzler uses ssurt instead of surt. + heritrix follows HTTP 3XX redirects. If the URL in the browser + location bar at the end of brozzling the seed page differs from the seed + URL, brozzler automatically adds a second ``accept`` rule to ensure the + site is in scope, as if the new URL were the original seed URL. For example, + if ``http://example.com/`` redirects to ``http://www.example.com/``, the + rest of the ``www.example.com`` is in scope. +5. Brozzler uses SSURT instead of SURT. 6. There is currently no brozzler option to disable the automatically generated ``accept`` rules. @@ -366,9 +367,9 @@ Scope settings +======+==========+=========+ | list | no | *none* | +------+----------+---------+ -List of scope rules. If any of the rules match, and the url is within -``max_hops`` from seed, and none of the ``block`` rules apply, the url is in -scope. +List of scope rules. If any of the rules match, the URL is within +``max_hops`` from seed, and none of the ``block`` rules apply, then the URL is +in scope and brozzled. ``blocks`` ~~~~~~~~~~~ @@ -377,7 +378,8 @@ scope. +======+==========+=========+ | list | no | *none* | +------+----------+---------+ -List of scope rules. If any of the rules match, the url is deemed out of scope. +List of scope rules. If any of the rules match, then the URL is deemed out +of scope and NOT brozzled. ``max_hops`` ~~~~~~~~~~~~ @@ -395,8 +397,8 @@ Maximum number of hops from seed. +========+==========+=========+ | number | no | 0 | +--------+----------+---------+ -Expands the scope to include urls up to this many hops from the last page that -was in scope thanks to an ``accept`` rule. +Expands the scope to include URLs up to this many hops from the last page that +was in scope because of an ``accept`` rule. Scope rule conditions --------------------- @@ -408,7 +410,7 @@ Scope rule conditions +========+==========+=========+ | string | no | *none* | +--------+----------+---------+ -Matches if the host part of the canonicalized url is ``domain`` or a +Matches if the host part of the canonicalized URL is ``domain`` or a subdomain. ``substring`` @@ -418,7 +420,7 @@ subdomain. +========+==========+=========+ | string | no | *none* | +--------+----------+---------+ -Matches if ``substring`` is found anywhere in the canonicalized url. +Matches if ``substring`` value is found anywhere in the canonicalized URL. ``regex`` ~~~~~~~~~ @@ -427,7 +429,7 @@ Matches if ``substring`` is found anywhere in the canonicalized url. +========+==========+=========+ | string | no | *none* | +--------+----------+---------+ -Matches if the full canonicalized url matches ``regex``. +Matches if the full canonicalized URL matches a regular expression. ``ssurt`` ~~~~~~~~~ @@ -436,7 +438,8 @@ Matches if the full canonicalized url matches ``regex``. +========+==========+=========+ | string | no | *none* | +--------+----------+---------+ -Matches if the canonicalized url in SSURT [1]_ form starts with ``ssurt``. +Matches if the canonicalized URL in SSURT [1]_ form starts with the ``ssurt`` +value. ``surt`` ~~~~~~~~ @@ -445,7 +448,8 @@ Matches if the canonicalized url in SSURT [1]_ form starts with ``ssurt``. +========+==========+=========+ | string | no | *none* | +--------+----------+---------+ -Matches if the canonicalized url in SURT [2]_ form starts with ``surt``. +Matches if the canonicalized URL in SURT [2]_ form starts with the ``surt`` +value. ``parent_url_regex`` ~~~~~~~~~~~~~~~~~~~~ @@ -454,15 +458,15 @@ Matches if the canonicalized url in SURT [2]_ form starts with ``surt``. +========+==========+=========+ | string | no | *none* | +--------+----------+---------+ -Matches if the full canonicalized parent url matches ``regex``. The parent url -is the url of the page in which the link was found. +Matches if the full canonicalized parent URL matches a regular expression. +The parent URL is the URL of the page in which a link is found. Using ``warcprox_meta`` ======================= -``warcprox_meta`` deserves some more discussion. It plays a very important role -in brozzler job configuration. ``warcprox_meta`` is the way you set the -filenames of the warcs for your crawl. For example, if each seed should have a -different warc name prefix, you might have a job configured this way:: +``warcprox_meta`` plays a very important role in brozzler job configuration. +It sets the filenames of the WARC files created by a job. For example, if each +seed should have a different WARC filename prefix, you might configure a job +this way:: seeds: - url: https://example.com/ @@ -472,9 +476,9 @@ different warc name prefix, you might have a job configured this way:: warcprox_meta: warc-prefix: seed2 -``warcprox_meta`` is also the way to put limits on the size of the crawl job. -For example, this configuration will stop the crawl after about 100 MB of novel -content has been crawled:: +``warcprox_meta`` may also be used to limit the size of the job. For example, +this configuration will stop the crawl after about 100 MB of novel content has +been archived:: seeds: - url: https://example.com/ @@ -486,10 +490,10 @@ content has been crawled:: limits: my-job/new/wire_bytes: 100000000 -To prevent any urls from a host from being captured, it's not sufficient to use +To prevent any URLs from a host from being captured, it is not sufficient to use a ``scope`` rule as described above. That kind of scoping only applies to -navigational links discovered in crawled pages. To make absolutely sure no url -from a given host is fetched, not even (say) an image embedded in a page, use +navigational links discovered in crawled pages. To make absolutely sure that no +url from a given host is fetched--not even an image embedded in a page--use ``warcprox_meta`` like so:: warcprox_meta: From f7407a87c1c0b89b1bc2827ecba4b717a0efb545 Mon Sep 17 00:00:00 2001 From: Noah Levitt Date: Mon, 23 Jul 2018 23:32:56 +0000 Subject: [PATCH 3/3] reformat readme to 80 columns --- README.rst | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/README.rst b/README.rst index 491f0b8..27d25e9 100644 --- a/README.rst +++ b/README.rst @@ -8,7 +8,13 @@ =============== "browser" \| "crawler" = "brozzler" -Brozzler is a distributed web crawler (爬虫) that uses a real browser (Chrome or Chromium) to fetch pages and embedded URLs and to extract links. It employs `youtube-dl `_ to enhance media capture capabilities, warcprox to write content to Web ARChive (WARC) files, `rethinkdb `_ to index captured URLs, a native dashboard for crawl job monitoring, and a customized Python Wayback interface for archival replay. +Brozzler is a distributed web crawler (爬虫) that uses a real browser (Chrome +or Chromium) to fetch pages and embedded URLs and to extract links. It employs +`youtube-dl `_ to enhance media capture +capabilities, warcprox to write content to Web ARChive (WARC) files, `rethinkdb +`_ to index captured URLs, a native +dashboard for crawl job monitoring, and a customized Python Wayback interface +for archival replay. Requirements ------------ @@ -17,12 +23,19 @@ Requirements - RethinkDB deployment - Chromium or Google Chrome >= version 64 -Note: The browser requires a graphical environment to run. When brozzler is run on a server, this may require deploying some additional infrastructure (typically X11; Xvfb does not support screenshots, however Xvnc4 from package vnc4server, does). The `vagrant configuration `_ in the brozzler repository (still a work in progress) has an example setup. +Note: The browser requires a graphical environment to run. When brozzler is run +on a server, this may require deploying some additional infrastructure +(typically X11; Xvfb does not support screenshots, however Xvnc4 from package +vnc4server, does). The `vagrant configuration `_ in the brozzler +repository (still a work in progress) has an example setup. Getting Started --------------- -The easiest way to get started with brozzler for web archiving is with ``brozzler-easy``. Brozzler-easy runs brozzler-worker, warcprox, brozzler wayback, and brozzler-dashboard, configured to work with each other in a single process. +The easiest way to get started with brozzler for web archiving is with +``brozzler-easy``. Brozzler-easy runs brozzler-worker, warcprox, brozzler +wayback, and brozzler-dashboard, configured to work with each other in a single +process. Mac instructions: @@ -45,7 +58,8 @@ Mac instructions: # start brozzler-easy brozzler-easy -At this point brozzler-easy will start archiving your site. Results will be immediately available for playback in pywb at http://localhost:8880/brozzler/. +At this point brozzler-easy will start archiving your site. Results will be +immediately available for playback in pywb at http://localhost:8880/brozzler/. *Brozzler-easy demonstrates the full brozzler archival crawling workflow, but does not take advantage of brozzler's distributed nature.* @@ -72,7 +86,9 @@ Submit sites not tied to a job:: Job Configuration ----------------- -Brozzler jobs are defined using YAML files. Options may be specified either at the top-level or on individual seeds. At least one seed URL must be specified, however everything else is optional. For details, see ``_. +Brozzler jobs are defined using YAML files. Options may be specified either at +the top-level or on individual seeds. At least one seed URL must be specified, +however everything else is optional. For details, see ``_. :: @@ -116,7 +132,9 @@ See ``brozzler-dashboard --help`` for configuration options. Brozzler Wayback ---------------- -Brozzler comes with a customized version of `pywb `_, which supports using the rethinkdb "captures" table (populated by warcprox) as its index. +Brozzler comes with a customized version of `pywb +`_, which supports using the rethinkdb +"captures" table (populated by warcprox) as its index. To use, first install dependencies. @@ -154,7 +172,10 @@ Then browse http://localhost:8880/brozzler/. Headless Chrome (experimental) -------------------------------- -`Headless Chromium `_ is now available in stable Chrome releases for 64-bit Linux and may be used to run the browser without a visible window or X11. +`Headless Chromium +`_ +is now available in stable Chrome releases for 64-bit Linux and may be used to +run the browser without a visible window or X11. To try this out, create a wrapper script like ~/bin/chrome-headless.sh: @@ -171,7 +192,11 @@ option: chmod +x ~/bin/chrome-headless.sh brozzler-worker --chrome-exe ~/bin/chrome-headless.sh -Beware: Chrome's headless mode is still very new and has `unresolved issues `_. Its use with brozzler has not yet been extensively tested. You may experience hangs or crashes with some types of content. For the moment we recommend using Chrome's regular mode instead. +Beware: Chrome's headless mode is still very new and has `unresolved issues +`_. +Its use with brozzler has not yet been extensively tested. You may experience +hangs or crashes with some types of content. For the moment we recommend using +Chrome's regular mode instead. License -------