Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-09-20 20:54:45 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Revert "initial commit"

Browse source 
This reverts commit 5368a84066.
This commit is contained in:
Barbara Miller 2019-08-08 14:03:30 -07:00
parent d0c46db746
commit c6308fe754
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
brozzler/robots.py
View file

@ -90,7 +90,7 @@ def is_permitted_by_robots(site, url, proxy=None):
brozzler.ReachedLimit: if warcprox responded with 420 Reached Limit brozzler.ReachedLimit: if warcprox responded with 420 Reached Limit
requests.exceptions.ProxyError: if the proxy is down requests.exceptions.ProxyError: if the proxy is down
''' '''
if site.ignore_robots and not site.user_agent: if site.ignore_robots:
return True return True
try: try: