Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-05-02 14:46:18 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

use the uncanonicalized url as part of the sha1 input to generate the page id, since canonicalization was stripping off the #fragment, and we might want to crawl the same url with different fragments (and there's no option to GoogleURLCanonicalizer to not strip the fragment)

Browse source
This commit is contained in:
Noah Levitt 2016-04-21 22:01:49 +00:00
parent dd8f0d525d
commit 568a553432
2 changed files with 2 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

2
setup.py
View file

@ -2,7 +2,7 @@ import setuptools
import glob
setuptools.setup(name='brozzler',
version='1.1.dev4',
version='1.1.dev5',
description='Distributed web crawling with browsers',
url='https://github.com/nlevitt/brozzler',
author='Noah Levitt',