Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-04-20 23:56:34 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

initial commit

Browse Source
This commit is contained in:
Barbara Miller 2019-07-11 12:52:52 -07:00
parent 8107abd804
commit 5368a84066
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
brozzler/robots.py
View File

@ -90,7 +90,7 @@ def is_permitted_by_robots(site, url, proxy=None):
brozzler.ReachedLimit: if warcprox responded with 420 Reached Limit
requests.exceptions.ProxyError: if the proxy is down
'''
if site.ignore_robots:
if site.ignore_robots and not site.user_agent:
return True
try: