Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-25 00:59:52 -05:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
204 Commits 69 Branches 15 Tags
Commit Graph

131 Commits

Author SHA1 Message Date
Hunter Stern
f282213981 Add fix for https://webarchive.jira.com/browse/ARI-4518 2015-09-17 08:43:30 -07:00
Hunter Stern
5ccc535f51 More changes 2015-09-16 09:23:13 -07:00
Hunter Stern
3467670900 More changes for handling psu24 site 2015-09-15 18:03:08 -07:00
Hunter Stern
ea41653c44 Pulled in changes from https://github.com/nlevitt/umbra/tree/aitfive-451-alt 2015-09-15 11:53:53 -07:00
Noah Levitt
b30cc2d68b simpler implementation for https://github.com/internetarchive/umbra/pull/42/files 2015-09-14 17:57:01 -07:00
Lauren Ko
d4a783285e Adds routing_key to queue Queue creation 2015-07-31 14:15:18 -05:00
Noah Levitt
73bbd87d5d merge in latest from master and adjust config as needed 2015-02-02 14:52:56 -08:00
Noah Levitt
776a6dac68 Merge branch 'master' into simple-behaviors 2015-02-02 14:49:34 -08:00
Noah Levitt
48b8754f40 Merge branch 'master' into simple-behaviors 2015-02-02 14:48:26 -08:00
Noah Levitt
db759f1066 Merge pull request #32 from adam-miller/ARI-3904 
ARI-3904 Instagram behavior to scroll past two pages, and click to enla...
 2015-02-02 14:47:44 -08:00
Adam Miller
ce47461656 Making scrolling and image loading more tolerant of slow loading. 2015-01-30 16:55:53 -08:00
Noah Levitt
9e5900c61f ARI-3956 simple behavior for usask.ca slideshows (which also required enhancing the simple behavior logic) 2015-01-27 16:03:58 -08:00
Noah Levitt
0901cac2e0 Merge pull request #38 from nlevitt/bump-browser-timeout 
increase browser start and stop timeouts, since sometimes we strand brow...
 2015-01-26 21:22:18 -08:00
Noah Levitt
e9c2fc61dd increase browser start and stop timeouts, since sometimes we strand browser processes after starting them, when the machine is very busy 2015-01-26 21:09:56 -08:00
Noah Levitt
c5c642a990 support for simple behavior that clicks on elements matching configured css selector; and one such behavior for acalog sites ARI-3775 2015-01-26 16:58:12 -08:00
Noah Levitt
0647df1ab9 behaviors.yaml to configure behaviors, in preparation for "simple" behavior support 2015-01-26 16:01:53 -08:00
Hunter Stern
91f9788eb2 Add iframe css path to target id for soundcloud buttons. 2015-01-21 16:28:29 -08:00
Hunter Stern
e9451f88d8 Merge branch 'master' of github.com:internetarchive/umbra into ari-3774 2015-01-21 16:21:13 -08:00
Noah Levitt
cdcef934e7 rewrite instagram behavior to be more like a state machine; update css selectors for current instagram; refactor as a sort of singleton class for cleaner namespacing 2015-01-16 13:21:12 -08:00
Noah Levitt
ddc7064585 Merge branch 'master' into ARI-3904 2015-01-15 18:37:28 -08:00
Hunter Stern
5ea12fd053 More refinements. 2014-12-19 15:52:13 -08:00
Hunter Stern
8d225b8859 More debugging. 2014-12-19 15:13:02 -08:00
Hunter Stern
5304f2909d Less verbose logging. 2014-12-19 14:35:11 -08:00
Hunter Stern
ae60205648 Fix for https://webarchive.jira.com/browse/ARI-4150 2014-12-19 14:17:50 -08:00
Noah Levitt
1108ef9362 Merge pull request #33 from adam-miller/ARI-4016 
ARI-4016 - Support: embedded videos on marquette.edu
 2014-11-21 15:10:53 -08:00
Adam Miller
7f8e6802de Implementing suggestions in pull request. 2014-11-07 15:56:05 -08:00
Noah Levitt
ab86426475 properly handle socket.error from amqp conn.drain_events (was previously diagnosed as error starting browser) 2014-11-03 11:54:10 -08:00
vonrosen
01ed5a7d4d Merge pull request #28 from internetarchive/ari-3940 
Ari 3940 - prioritize scrolling all the way to the bottom
 2014-10-09 21:21:02 +00:00
Adam Miller
bdf3e73062 Wait until big image is loaded before clicking to next image. 2014-10-03 14:17:07 -07:00
Hunter Stern
1ee45053c5 Even more formatting changes. 2014-09-22 14:22:52 -07:00
Hunter Stern
6af3455dbf Improve formatting. 2014-09-22 14:21:00 -07:00
Adam Miller
916f1b990e Cleanup instagram timeout and state handling 2014-09-17 16:26:53 -07:00
Adam Miller
eb3ea95b87 Cleanup timeout logic 2014-09-17 15:26:13 -07:00
Adam Miller
5a3c8e9a05 ARI-4016 - Support: embedded videos on marquette.edu 2014-09-15 11:06:33 -07:00
Hunter Stern
a2ea2501db More soundcloud changes. 2014-09-12 16:07:32 -07:00
Hunter Stern
e320654d1e Allow selector to detect https and http soundcloud widget. 2014-09-12 09:56:41 -07:00
Adam Miller
7afdd7b50b Added behavior for instagram to scroll past two pages, and click to enlarge images. 2014-09-02 17:02:30 -07:00
Hunter Stern
0e7fd93967 Merge remote-tracking branch 'internetarchive/master' into ari-3774 2014-08-26 15:12:13 -07:00
Noah Levitt
c886b57d3a reject (discard) bad messages 2014-08-19 18:51:43 -07:00
Noah Levitt
9d90b5830a facebook - scroll all the to the bottom before scrolling back up to click more stuff 2014-08-01 16:53:13 -07:00
Noah Levitt
dd9ef50484 suppress logging of umbraBehaviorFinished() message which is sent a lot 2014-08-01 16:22:45 -07:00
Hunter Stern
6a5d1e2266 Disable web security in chromium so iframes on different domains can be accessed by behavior javascript. 2014-07-24 16:46:06 -07:00
Hunter Stern
80f3a4a067 Enhancement to allow embedded soundcloud audio files to be detected 2014-07-24 16:44:05 -07:00
Noah Levitt
ae838af25d set amqp prefetch count to the number of urls we can handle at a time, i.e. max_active_browsers (with prefetch=1 umbra was only browsing one url at a time, after quickly burning through urls already on the queue when started) 2014-07-02 10:30:51 -07:00
Noah Levitt
6306c16698 kill -HUP to immediately close and reopen amqp consumer connection 2014-06-23 17:18:27 -07:00
Noah Levitt
02c054c284 do not wait forever for zombie websocket threads (this change should also reveal how we get these sometimes) 2014-06-20 18:13:45 -07:00
Noah Levitt
ead46d5716 more elaborate dumping of state on SIGQUIT to replace faulthandler 2014-06-20 14:05:33 -07:00
Noah Levitt
17ef9d9f28 close and reopen the amqp consumer connection only every 2.5 hours instead of every 15 minutes, because now that we have to wait for all browsers to close when we do the reconnection, it slows us down a lot 2014-06-18 14:58:44 -07:00
Noah Levitt
025db91dea get rid of --browser-wait and --routing-key in favor of sensible defaults, some other tweaks 2014-06-11 10:58:08 -07:00
Noah Levitt
a78e60f1da wait for a browser to become available and start it up before reading the next url from amqp; ack the message only after completing the browsing process successfully, and requeue if it's not successful; some refactoring to make the timing work for this 2014-06-09 13:15:05 -07:00