Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-07-26 08:15:43 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

honor stop request before choosing proxy

Browse source 
makes test_warcprox_outage_resiliency pass again
This commit is contained in:
Noah Levitt 2018-03-05 16:08:24 -08:00
parent 2639d7b991
commit 318ae13bcb
1 changed files with 4 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

4
brozzler/worker.py
View file

@ -506,6 +506,10 @@ class BrozzlerWorker:
site.save() site.save()
start = time.time() start = time.time()
page = None page = None
self._frontier.enforce_time_limit(site)
self._frontier.honor_stop_request(site)
# _proxy_for() call in log statement can raise brozzler.ProxyError
# which is why we honor time limit and stop request first☝🏻
self.logger.info( self.logger.info(
"brozzling site (proxy=%r) %r", "brozzling site (proxy=%r) %r",
self._proxy_for(site), site) self._proxy_for(site), site)