Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 08:39:59 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity

sleep for 0.5 sec before attempting to reconnect to amqp; documentation tweaks

Browse Source
This commit is contained in:
Noah Levitt 2014-05-23 13:34:07 -07:00
parent 155db96461
commit a7cd872b95
1 changed files with 16 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
umbra/controller.py
View File

@ -8,9 +8,12 @@ import kombu
from umbra.browser import Browser from umbra.browser import Browser
class AmqpBrowserController: class AmqpBrowserController:
"""Consumes amqp messages representing requests to browse urls, from the """
amqp queue "urls" on exchange "umbra". Incoming amqp message is a json Consumes amqp messages representing requests to browse urls, from the
object with 3 attributes: specified amqp queue (default: "urls") on the specified amqp exchange
(default: "umbra"). Incoming amqp message is a json object with 3
attributes:
{ {
"clientId": "umbra.client.123", "clientId": "umbra.client.123",
"url": "http://example.com/my_fancy_page", "url": "http://example.com/my_fancy_page",
@ -19,20 +22,20 @@ class AmqpBrowserController:
"url" is the url to browse. "url" is the url to browse.
"clientId" uniquely identifies the client of "clientId" uniquely identifies the client of umbra. Umbra uses the clientId
umbra. Umbra uses the clientId to direct information via amqp back to the as the amqp routing key, to direct information via amqp back to the client.
client. It sends this information on that same "umbra" exchange, and uses It sends this information on the same specified amqp exchange (default:
the clientId as the amqp routing key. "umbra").
Each url requested in the browser is published to amqp this way. The Each url requested in the browser is published to amqp this way. The
outgoing amqp message is a json object: outgoing amqp message is a json object:
{ {
'url': 'http://example.com/images/embedded_thing.jpg', "url": "http://example.com/images/embedded_thing.jpg",
'method': 'GET', "method": "GET",
'headers': {'User-Agent': '...', 'Accept': '...'} "headers": {"User-Agent": "...", "Accept": "...", ...},
'parentUrl': 'http://example.com/my_fancy_page', "parentUrl": "http://example.com/my_fancy_page",
'parentUrlMetadata': {"arbitrary":"fields", "etc":4}, "parentUrlMetadata": {"arbitrary":"fields", "etc":4, ...}
} }
POST requests have an additional field, postData. POST requests have an additional field, postData.
@ -102,6 +105,7 @@ class AmqpBrowserController:
pass pass
except BaseException as e: except BaseException as e:
self.logger.error("amqp exception {}".format(e)) self.logger.error("amqp exception {}".format(e))
time.sleep(0.5)
self.logger.error("attempting to reopen amqp connection") self.logger.error("attempting to reopen amqp connection")
def _browse_page_requested(self, body, message): def _browse_page_requested(self, body, message):