Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-02-24 08:39:59 -05:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,125 Commits 69 Branches 15 Tags
Commit Graph

1125 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Noah Levitt
aacb886b62 ARI-3814 try to recover from rabbitmq communication problems 2014-04-07 11:45:12 -07:00
Eldon
4e72cbae58 Merge pull request #18 from nlevitt/ari-3771 
to address ARI-3771 "Lasalle Facebook last scrolldown doesn't work", scr...
 2014-04-04 16:04:38 -04:00
Eldon
beeb4a2a2c Merge pull request #17 from nlevitt/ari-3811 
thread dump on SIGQUIT a la java
 2014-04-04 15:21:41 -04:00
Noah Levitt
be9115fd11 to address ARI-3771 "Lasalle Facebook last scrolldown doesn't work", scroll by 200 pixels each time instead of 100 on facebook, which avoids hitting the 15 second idle timeout in my tests; also detect when unclicked targets are above the screen/viewport and not below and log it as such, instead of trying to continue scrolling down 2014-04-04 12:16:00 -07:00
Noah Levitt
da975bc586 thread dump on SIGQUIT a la java 2014-04-03 21:19:08 -07:00
Eldon
e1c297269c Merge pull request #15 from nlevitt/master 
setup.py - include behaviors.d/*.js in installation
 2014-03-13 11:09:34 -04:00
Noah Levitt
f3a540b92d setup.py - include behaviors.d/*.js in installation 2014-03-13 00:00:32 -07:00
vonrosen
b3bd959ab2 Merge pull request #14 from eldondev/master 
Check to see if the object has a click method before calling it
 2014-03-10 12:01:20 -07:00
Eldon
427b74ebfc Check to see if the object has a click method before calling it 2014-03-10 14:58:16 -04:00
vonrosen
a16ce4abeb Merge pull request #13 from nlevitt/master 
facebook, logging, timeout tweaks
 2014-03-09 16:47:19 -07:00
Noah Levitt
3fd792fddb lengthen timeouts and improve timeout handling; log js console messages from browser 2014-03-07 19:39:27 -08:00
Noah Levitt
5637e7111f use *rel=["theater"] to click on photos and videos that won't navigate to a new page; don't click on comments links for now, since it might interfere with other stuff; more verbose logging of click targets 2014-03-07 19:37:43 -08:00
vonrosen
a0f8474a73 Merge pull request #12 from nlevitt/master 
vimeo, tweaks
 2014-03-07 11:32:14 -08:00
Noah Levitt
5a7a24083f simplify checking for *.js 2014-03-07 11:29:43 -08:00
Noah Levitt
a30b5d8dd2 only reset idle timer on Network.requestWillBeSent instead of all events (otherwise long-running videos keep the browser open unnecessarily) 2014-03-06 18:35:04 -08:00
Noah Levitt
9d9014c864 start the hard stop timer 2014-03-06 18:32:30 -08:00
Noah Levitt
52db581a3c restore logging 2014-03-06 18:25:46 -08:00
Noah Levitt
12d66982d1 only load behaviors files named like *.js (avoids vim .swp files and stuff); tweak logging 2014-03-06 18:25:35 -08:00
Noah Levitt
9cb9172a4d behavior for vimeo - click on <video> elements 2014-03-06 18:24:12 -08:00
Noah Levitt
9848c41d5f make regexes the same that crawlman puts in crawler-beans.cxml 2014-03-06 18:23:31 -08:00
vonrosen
5b1992a8c0 Merge pull request #11 from eldondev/master 
Convert behaviors to independent, runnable javascript files, hard timeout on pages
 2014-03-06 11:08:45 -08:00
Eldon
393df3f16e Update behaviors for facebook theater 2014-03-05 23:44:52 -05:00
Eldon
f2f78d2ced Convert from one big json file, to js files with a regex as a comment at the top. 2014-03-05 23:19:09 -05:00
Eldon
4c22891093 Merge pull request #10 from nlevitt/master 
remove unused function
 2014-02-25 17:35:26 -05:00
Noah Levitt
b763d6550f remove unused function 2014-02-25 14:26:10 -08:00
Eldon
b4675a7cd2 Merge pull request #9 from nlevitt/master 
add behaviors, handle multiple browsers
 2014-02-25 16:23:27 -05:00
Noah Levitt
11da122ec2 remove old commented out line of code 2014-02-18 13:20:18 -08:00
Noah Levitt
b96d8856d4 create temp dir for user profile rather than rely on --temp-profile 2014-02-14 19:45:16 -08:00
Noah Levitt
b4846e1063 scrolldown seems to get everything for flickr and facebook at the moment 2014-02-14 17:57:04 -08:00
Noah Levitt
28282641f2 add a little logging 2014-02-14 15:18:10 -08:00
Noah Levitt
2368688fbe Merge remote-tracking branch 'eldondev/master' into nlevitt-master (add behaviors) 
Conflicts:
	umbra/umbra.py
 2014-02-14 15:10:23 -08:00
Noah Levitt
3389c5a66d remove some extraneous debug logging 2014-02-13 18:36:08 -08:00
Eldon
fe15932c26 Click on photos in gallery behavior 2014-02-13 13:37:08 -05:00
Eldon
af01fcbcfe Add more flickr behavior 2014-02-13 13:32:34 -05:00
Eldon
445288d5e7 First few behaviors 2014-02-13 13:10:19 -05:00
Noah Levitt
f69edd5a87 handle multiple clients, browsers 2014-02-13 01:59:09 -08:00
Noah Levitt
4dbe111aee Merge branch 'master' into nlevitt-master 2014-02-12 18:15:05 -08:00
Noah Levitt
fe1c68af90 Merge pull request #7 from eldondev/master 
Refactor to pull Chrome execution inside of umbra, simplify some things
 2014-02-12 18:13:52 -08:00
Eldon
bdf00cc515 Refactor to pull Chrome execution inside of umbra, simplify some things 2014-02-12 19:31:03 -05:00
Eldon
dd871f3a6a Merge pull request #6 from nlevitt/master 
progress and formatting changes
 2014-02-12 16:09:04 -05:00
Noah Levitt
4935d55b6e specify classifier 'Programming Language :: Python :: 3.3' since websocket-client-py3 requires python 3.3, doesn't work with 3.2 2014-02-12 12:17:41 -08:00
Noah Levitt
f9d56d3071 formatting change only - indent with 4 spaces 2014-02-10 20:45:18 -08:00
Noah Levitt
02fbe725cb cache parent url metadata and send back via amqp with child urls 2014-02-10 20:40:06 -08:00
Noah Levitt
f8c5a08c1b Merge pull request #5 from eldondev/master 
A couple of updates and intelligent defaults.
 2014-01-28 14:57:20 -08:00
Eldon
5588eedbe2 Update readme 2014-01-28 00:12:33 -05:00
Eldon
8afe7d90a2 Replace js evaluation with direct page navigation, add default for dump_queue 2014-01-28 00:10:31 -05:00
Eldon
7d89d1bed1 Merge pull request #4 from nlevitt/master 
get rid of multiple window handling, add more logging, etc
 2014-01-27 20:32:59 -08:00
Noah Levitt
8eb92b28e6 make load_url handle arguments similarly to umbra 2014-01-27 19:34:54 -08:00
Noah Levitt
e9f6408881 improved browser ownership locking 2014-01-27 19:24:28 -08:00
Noah Levitt
ae2c3668a9 get rid of multiple window handling, add more logging, clean shutdown on ctrl+c, other tweaks 2014-01-27 19:15:31 -08:00