Noah Levitt
|
ab86426475
|
properly handle socket.error from amqp conn.drain_events (was previously diagnosed as error starting browser)
|
2014-11-03 11:54:10 -08:00 |
|
Noah Levitt
|
c886b57d3a
|
reject (discard) bad messages
|
2014-08-19 18:51:43 -07:00 |
|
Noah Levitt
|
ae838af25d
|
set amqp prefetch count to the number of urls we can handle at a time, i.e. max_active_browsers (with prefetch=1 umbra was only browsing one url at a time, after quickly burning through urls already on the queue when started)
|
2014-07-02 10:30:51 -07:00 |
|
Noah Levitt
|
6306c16698
|
kill -HUP to immediately close and reopen amqp consumer connection
|
2014-06-23 17:18:27 -07:00 |
|
Noah Levitt
|
17ef9d9f28
|
close and reopen the amqp consumer connection only every 2.5 hours instead of every 15 minutes, because now that we have to wait for all browsers to close when we do the reconnection, it slows us down a lot
|
2014-06-18 14:58:44 -07:00 |
|
Noah Levitt
|
025db91dea
|
get rid of --browser-wait and --routing-key in favor of sensible defaults, some other tweaks
|
2014-06-11 10:58:08 -07:00 |
|
Noah Levitt
|
a78e60f1da
|
wait for a browser to become available and start it up before reading the next url from amqp; ack the message only after completing the browsing process successfully, and requeue if it's not successful; some refactoring to make the timing work for this
|
2014-06-09 13:15:05 -07:00 |
|
Noah Levitt
|
e619e013b6
|
sleep for 5 seconds after starting a browser, since starting 20 at once brings the computer to its knees
|
2014-06-03 15:57:12 -07:00 |
|
Noah Levitt
|
56a721f059
|
dump stack trace and don't return browser to pool on critical error where chrome process might still be running
|
2014-05-30 23:07:39 -07:00 |
|
Noah Levitt
|
94c2e4390b
|
debugging to and mitigation for problem "[Errno 98] Address already in use"
|
2014-05-28 18:57:21 -07:00 |
|
Noah Levitt
|
9c08be2699
|
sigterm and sigint both shutdown request shutdown, which stops consuming urls and waits for active browsers to finish; a second sigint/sigterm immediately shuts down active browsers
|
2014-05-24 01:52:22 -07:00 |
|
Noah Levitt
|
b67d9fadf0
|
log ports chose for browsers, and give threads nice names to make logs easier to understand
|
2014-05-23 22:30:25 -07:00 |
|
Noah Levitt
|
2c4ba005b5
|
make umbra amenable to clustering by using a pool of n browsers and removing the browser-clientId affinity (not useful currently since we start a fresh browser instance for each page browsed), and set prefetch_count=1 on amqp consumers to round-robin incoming urls among umbra instances
|
2014-05-23 21:59:34 -07:00 |
|
Noah Levitt
|
a7cd872b95
|
sleep for 0.5 sec before attempting to reconnect to amqp; documentation tweaks
|
2014-05-23 13:34:07 -07:00 |
|
Noah Levitt
|
bf3afcccb9
|
oops, Browser.__init__ doesn't take client_id anymore
|
2014-05-20 19:27:53 -07:00 |
|
Noah Levitt
|
b59e76a5b9
|
clean shutdown without draining entire amqp queue (only consume urls from amqp when browser activity isn't saturated)
|
2014-05-20 03:02:48 -07:00 |
|
Noah Levitt
|
3e4232f32c
|
refactor umbra.py into controller.py and browser.py, improve class names
|
2014-05-20 02:42:40 -07:00 |
|