Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-07-21 22:18:55 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

debugging travis-ci test failure

Browse source
This commit is contained in:
Noah Levitt 2017-04-18 10:23:45 -07:00
parent dc43794363
commit 0880ab615b
1 changed files with 3 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

3
brozzler/robots.py
View file

@ -102,6 +102,9 @@ def is_permitted_by_robots(site, url, proxy=None):
url, site.user_agent or "brozzler")
return result
except Exception as e:
logging.warn(
"caught exception fetching robots.txt: e=%s vars(e)=%s",
repr(e), vars(e))
if isinstance(e, reppy.exceptions.ServerError) and isinstance(
e.args[0], brozzler.ReachedLimit):
raise e.args[0]