Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 00:29:53 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
174 Commits 69 Branches 15 Tags
Commit Graph

174 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Adam Miller
ce47461656 Making scrolling and image loading more tolerant of slow loading. 2015-01-30 16:55:53 -08:00
Noah Levitt
cdcef934e7 rewrite instagram behavior to be more like a state machine; update css selectors for current instagram; refactor as a sort of singleton class for cleaner namespacing 2015-01-16 13:21:12 -08:00
Noah Levitt
ddc7064585 Merge branch 'master' into ARI-3904 2015-01-15 18:37:28 -08:00
Noah Levitt
ffd60d35e6 Merge pull request #36 from vonrosen/ari-4150 
Allow scrolling down a timeline in the facebook plugin so as to capture content in third party embedded timelines.
 2014-12-22 21:47:31 -08:00
Hunter Stern
5ea12fd053 More refinements. 2014-12-19 15:52:13 -08:00
Hunter Stern
8d225b8859 More debugging. 2014-12-19 15:13:02 -08:00
Hunter Stern
5304f2909d Less verbose logging. 2014-12-19 14:35:11 -08:00
Hunter Stern
ae60205648 Fix for https://webarchive.jira.com/browse/ARI-4150 2014-12-19 14:17:50 -08:00
Hunter Stern
cf88b9968c Merge branch 'master' of github.com:internetarchive/umbra 2014-12-12 15:59:25 -08:00
Noah Levitt
1108ef9362 Merge pull request #33 from adam-miller/ARI-4016 
ARI-4016 - Support: embedded videos on marquette.edu
 2014-11-21 15:10:53 -08:00
Adam Miller
7f8e6802de Implementing suggestions in pull request. 2014-11-07 15:56:05 -08:00
vonrosen
8e6859ef56 Merge pull request #35 from nlevitt/amqp-socket-error 
properly handle socket.error from amqp conn.drain_events (was previously...
 2014-11-03 12:09:27 -08:00
Noah Levitt
9053279b4e change default routing key to "urls" 2014-11-03 11:54:59 -08:00
Noah Levitt
ab86426475 properly handle socket.error from amqp conn.drain_events (was previously diagnosed as error starting browser) 2014-11-03 11:54:10 -08:00
Noah Levitt
f40bd39e1a Merge pull request #34 from dhamaniasad/patch-1 
Update README.md
 2014-10-30 19:04:24 -07:00
Asad Dhamani
9231cc2b5c Update README.md 2014-10-31 07:02:49 +05:30
Asad Dhamani
e264f09c27 Update README.md 2014-10-29 12:42:43 +05:30
Hunter Stern
52bb02cbbe Merge branch 'master' of github.com:internetarchive/umbra 2014-10-16 20:09:42 +00:00
vonrosen
01ed5a7d4d Merge pull request #28 from internetarchive/ari-3940 
Ari 3940 - prioritize scrolling all the way to the bottom
 2014-10-09 21:21:02 +00:00
Adam Miller
bdf3e73062 Wait until big image is loaded before clicking to next image. 2014-10-03 14:17:07 -07:00
Adam Miller
916f1b990e Cleanup instagram timeout and state handling 2014-09-17 16:26:53 -07:00
Adam Miller
eb3ea95b87 Cleanup timeout logic 2014-09-17 15:26:13 -07:00
Adam Miller
5a3c8e9a05 ARI-4016 - Support: embedded videos on marquette.edu 2014-09-15 11:06:33 -07:00
Adam Miller
7afdd7b50b Added behavior for instagram to scroll past two pages, and click to enlarge images. 2014-09-02 17:02:30 -07:00
Noah Levitt
9052fd8569 add license section 2014-09-02 16:11:49 -07:00
Noah Levitt
51d6b1a4e2 apache license 2014-09-02 16:10:00 -07:00
Hunter Stern
eb8c9faf89 Merge remote-tracking branch 'internetarchive/master' 2014-08-28 10:56:27 -07:00
Adam Miller
ce2957269f Merge pull request #31 from nlevitt/drain-republish 
new utility queue-json, and another change to help with draining from and republishing to amqp
 2014-08-26 16:58:21 -07:00
Noah Levitt
2ab767eaa9 make drain-queue output actual json instead of python dict syntax 2014-08-26 23:46:00 +00:00
Noah Levitt
fe1d9e01eb utility queue-json to publish an arbitrary json blob to amqp 2014-08-26 23:45:42 +00:00
vonrosen
bbba344886 Merge pull request #29 from nlevitt/handle-bad-message 
reject (discard) bad messages
 2014-08-20 08:21:29 -07:00
Noah Levitt
c886b57d3a reject (discard) bad messages 2014-08-19 18:51:43 -07:00
Hunter Stern
b110a57938 Merge remote-tracking branch 'internetarchive/master' 2014-08-14 15:26:15 -07:00
Noah Levitt
9d90b5830a facebook - scroll all the to the bottom before scrolling back up to click more stuff 2014-08-01 16:53:13 -07:00
Noah Levitt
dd9ef50484 suppress logging of umbraBehaviorFinished() message which is sent a lot 2014-08-01 16:22:45 -07:00
Adam Miller
8e44e18053 Merge pull request #26 from nlevitt/dev 
stability!
 2014-07-21 13:18:24 -07:00
Noah Levitt
ae838af25d set amqp prefetch count to the number of urls we can handle at a time, i.e. max_active_browsers (with prefetch=1 umbra was only browsing one url at a time, after quickly burning through urls already on the queue when started) 2014-07-02 10:30:51 -07:00
Noah Levitt
6306c16698 kill -HUP to immediately close and reopen amqp consumer connection 2014-06-23 17:18:27 -07:00
Noah Levitt
02c054c284 do not wait forever for zombie websocket threads (this change should also reveal how we get these sometimes) 2014-06-20 18:13:45 -07:00
Noah Levitt
9b32f9a3d1 ugh, it was better with the default width, in spite of the ridiculous behavior.script 2014-06-20 14:40:12 -07:00
Noah Levitt
2cf69bdaff seriously, don't try to wrap any lines, pprint 2014-06-20 14:37:33 -07:00
Noah Levitt
c6fa00812c when dumping state on SIGQUIT, build the whole string before printing to avoid stuff getting intermingled with other logging and stuff 2014-06-20 14:33:01 -07:00
Noah Levitt
ead46d5716 more elaborate dumping of state on SIGQUIT to replace faulthandler 2014-06-20 14:05:33 -07:00
Noah Levitt
ebb14ff889 get rid of chrome_wait straggler 2014-06-18 17:31:28 -07:00
Noah Levitt
17ef9d9f28 close and reopen the amqp consumer connection only every 2.5 hours instead of every 15 minutes, because now that we have to wait for all browsers to close when we do the reconnection, it slows us down a lot 2014-06-18 14:58:44 -07:00
Noah Levitt
025db91dea get rid of --browser-wait and --routing-key in favor of sensible defaults, some other tweaks 2014-06-11 10:58:08 -07:00
Noah Levitt
a78e60f1da wait for a browser to become available and start it up before reading the next url from amqp; ack the message only after completing the browsing process successfully, and requeue if it's not successful; some refactoring to make the timing work for this 2014-06-09 13:15:05 -07:00
Noah Levitt
e3c23a0f2b Merge pull request #25 from vonrosen/ari-3724 
Allow flash requests to be detected. For https://webarchive.jira.com/browse/ARI-3724
 2014-06-06 15:15:24 -07:00
vonrosen
d40b542ffe Merge pull request #1 from vonrosen/ari-3724 
Allow flash requests to be detected.
 2014-06-06 10:51:09 -07:00
Hunter Stern
41270af223 Allow flash requests to be detected. 2014-06-06 10:47:29 -07:00