Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 08:39:59 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity

correctly handle site with no pages (which means the seed was blocked by robots.txt) in frontier.seed_page

Browse Source
This commit is contained in:
Noah Levitt 2016-05-11 00:45:47 +00:00
parent 317a5eb99d
commit c6e0e7c507
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
brozzler/frontier.py
View File

@ -313,6 +313,8 @@ class RethinkDbFrontier:
if len(pages) > 1:
self.logger.warn(
"more than one seed page for site_id %s ?", site_id)
if len(pages) < 1:
return None
return brozzler.Page(**pages[0])
def site_pages(self, site_id, unbrozzled_only=False):