Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-09-27 11:41:42 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

chore: Comment retry delay backoff calculation.

Browse source
This commit is contained in:
Adam Miller 2024-11-26 10:26:47 -08:00
parent 05ddc28ea8
commit 4b20d81b19
1 changed files with 2 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

2
brozzler/worker.py
View file

@ -552,6 +552,8 @@ class BrozzlerWorker:
"unexpected exception site=%r page=%r", site, page, exc_info=True "unexpected exception site=%r page=%r", site, page, exc_info=True
) )
if page: if page:
# Calculate backoff in seconds based on number of failed attempts.
# Minimum of 60, max of 135 giving delays of 60, 90, 135, 135...
retry_delay = min(135, 60 * (1.5**page.failed_attempts)) retry_delay = min(135, 60 * (1.5**page.failed_attempts))
page.retry_after = doublethink.utcnow() + datetime.timedelta( page.retry_after = doublethink.utcnow() + datetime.timedelta(
seconds=retry_delay seconds=retry_delay