Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 08:39:59 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity

Use urlcanon.whatwg in extracted outlinks

Browse Source 
The aim is to improve outlink quality.
This commit is contained in:
Vangelis Banos 2019-10-31 21:27:55 +00:00
parent 8beb96817e
commit 33b7a7f564
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
brozzler/browser.py
View File

@ -594,8 +594,8 @@ class Browser:
if ('result' in message and 'result' in message['result']
and 'value' in message['result']['result']):
if message['result']['result']['value']:
return frozenset(
message['result']['result']['value'].split('\n'))
return frozenset([str(urlcanon.whatwg(link)) for link in
message['result']['result']['value'].split('\n')])
else:
# no links found
return frozenset()