Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 08:39:59 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity

use nlevitt fork of surt library for less stupid handling of mailto: urls, etc

Browse Source
This commit is contained in:
Noah Levitt 2015-08-20 21:23:59 +00:00
parent ee50818dca
commit 231d019659
2 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
brozzler/robots.py
View File

@ -44,7 +44,7 @@ def is_permitted_by_robots(site, url):
raise e.args[0]
else:
if tries_left > 0:
logging.warn("caught exception %s fetching robots.txt (%s tries left) for %s", repr(e), tries_left, url)
logging.warn("caught exception fetching robots.txt (%s tries left) for %s: %s", tries_left, url, repr(e))
tries_left -= 1
else:
logging.error("caught exception fetching robots.txt (0 tries left) for %s: %s", url, repr(e), exc_info=True)

2
requirements.txt
View File

@ -1,6 +1,6 @@
argparse
PyYAML
git+https://github.com/ikreymer/surt.git@py3
git+https://github.com/nlevitt/surt.git@py3
# youtube_dl
git+https://github.com/nlevitt/youtube-dl.git@brozzler
git+https://github.com/seomoz/reppy.git