Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-09-16 02:41:44 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Revert "initial commit"

Browse source 
This reverts commit 5368a84066.
This commit is contained in:
Barbara Miller 2019-08-08 14:03:30 -07:00
parent d0c46db746
commit c6308fe754
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
brozzler/robots.py
View file

@ -90,7 +90,7 @@ def is_permitted_by_robots(site, url, proxy=None):
brozzler.ReachedLimit: if warcprox responded with 420 Reached Limit
requests.exceptions.ProxyError: if the proxy is down
'''
if site.ignore_robots and not site.user_agent:
if site.ignore_robots:
return True
try: