Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-04-20 23:56:34 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

fix bug clobbering own changes to parent_page?

Browse Source
This commit is contained in:
Noah Levitt 2019-10-17 13:25:47 -07:00
parent c64f45728d
commit b01227b986
1 changed files with 6 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
brozzler/frontier.py
View File

@ -386,6 +386,12 @@ class RethinkDbFrontier:
pages[fresh_page.id] = fresh_page
counts['added'] += 1
# make sure we're not stepping on our own toes in case we have a link
# back to parent_page, which I think happens because of hashtags
if parent_page.id in pages:
self._merge_page(parent_page, pages[parent_page.id])
del pages[parent_page.id]
# insert/replace in batches of 50 to try to avoid this error:
# "rethinkdb.errors.ReqlDriverError: Query size (167883036) greater than maximum (134217727) in:"
# there can be many pages and each one can be very large (many videos,