45 Commits

Author	SHA1	Message	Date
Noah Levitt	3fd792fddb	lengthen timeouts and improve timeout handling; log js console messages from browser	2014-03-07 19:39:27 -08:00
Noah Levitt	5637e7111f	use *rel=["theater"] to click on photos and videos that won't navigate to a new page; don't click on comments links for now, since it might interfere with other stuff; more verbose logging of click targets	2014-03-07 19:37:43 -08:00
Noah Levitt	5a7a24083f	simplify checking for *.js	2014-03-07 11:29:43 -08:00
Noah Levitt	a30b5d8dd2	only reset idle timer on Network.requestWillBeSent instead of all events (otherwise long-running videos keep the browser open unnecessarily)	2014-03-06 18:35:04 -08:00
Noah Levitt	9d9014c864	start the hard stop timer	2014-03-06 18:32:30 -08:00
Noah Levitt	52db581a3c	restore logging	2014-03-06 18:25:46 -08:00
Noah Levitt	12d66982d1	only load behaviors files named like *.js (avoids vim .swp files and stuff); tweak logging	2014-03-06 18:25:35 -08:00
Noah Levitt	9cb9172a4d	behavior for vimeo - click on <video> elements	2014-03-06 18:24:12 -08:00
Noah Levitt	9848c41d5f	make regexes the same that crawlman puts in crawler-beans.cxml	2014-03-06 18:23:31 -08:00
Eldon	393df3f16e	Update behaviors for facebook theater	2014-03-05 23:44:52 -05:00
Eldon	f2f78d2ced	Convert from one big json file, to js files with a regex as a comment at the top.	2014-03-05 23:19:09 -05:00
Noah Levitt	b763d6550f	remove unused function	2014-02-25 14:26:10 -08:00
Noah Levitt	11da122ec2	remove old commented out line of code	2014-02-18 13:20:18 -08:00
Noah Levitt	b96d8856d4	create temp dir for user profile rather than rely on --temp-profile	2014-02-14 19:45:16 -08:00
Noah Levitt	b4846e1063	scrolldown seems to get everything for flickr and facebook at the moment	2014-02-14 17:57:04 -08:00
Noah Levitt	28282641f2	add a little logging	2014-02-14 15:18:10 -08:00
Noah Levitt	2368688fbe	Merge remote-tracking branch 'eldondev/master' into nlevitt-master (add behaviors) ... Conflicts: umbra/umbra.py	2014-02-14 15:10:23 -08:00
Noah Levitt	3389c5a66d	remove some extraneous debug logging	2014-02-13 18:36:08 -08:00
Eldon	fe15932c26	Click on photos in gallery behavior	2014-02-13 13:37:08 -05:00
Eldon	af01fcbcfe	Add more flickr behavior	2014-02-13 13:32:34 -05:00
Eldon	445288d5e7	First few behaviors	2014-02-13 13:10:19 -05:00
Noah Levitt	f69edd5a87	handle multiple clients, browsers	2014-02-13 01:59:09 -08:00
Eldon	bdf00cc515	Refactor to pull Chrome execution inside of umbra, simplify some things	2014-02-12 19:31:03 -05:00
Noah Levitt	f9d56d3071	formatting change only - indent with 4 spaces	2014-02-10 20:45:18 -08:00
Noah Levitt	02fbe725cb	cache parent url metadata and send back via amqp with child urls	2014-02-10 20:40:06 -08:00
Eldon	8afe7d90a2	Replace js evaluation with direct page navigation, add default for dump_queue	2014-01-28 00:10:31 -05:00
Noah Levitt	e9f6408881	improved browser ownership locking	2014-01-27 19:24:28 -08:00
Noah Levitt	ae2c3668a9	get rid of multiple window handling, add more logging, clean shutdown on ctrl+c, other tweaks	2014-01-27 19:15:31 -08:00
Eldon	bd0183058d	Inccognito messes with currently running chromium instances, disable it	2014-01-23 18:26:20 -05:00
Eldon	0c25bbb4c3	Raise exception if we are never able to connect to the browser	2014-01-23 16:28:52 -05:00
Eldon	ca3d6d5999	Raise an exception if no browser starts	2014-01-23 16:24:53 -05:00
Eldon	4852fbf29f	Update setup.py, get rid of unused dependency	2014-01-23 16:18:13 -05:00
Eldon	fdfaf496a9	Incognito mode prevents local caching	2014-01-23 19:25:55 +00:00
Eldon	09b0060137	Add additional checks for weird chrome behavior, serialize requests	2014-01-23 18:39:17 +00:00
Eldon	6dc20e660f	Remove debugging output, improve support scripts	2014-01-22 18:41:00 +00:00
Eldon	4e38a142d4	Some refactor/testing and utility scripts	2014-01-22 18:03:02 +00:00
Eldon	e0a2917b81	Add amqp connectivity	2014-01-22 07:38:03 +00:00
Eldon	5a6749777b	It appears to be possible for the debug port to open without any tabs, so we should handle that.	2014-01-22 02:38:51 +00:00
Eldon	428d6cb7da	Rework executable script so that it uses a main	2014-01-22 02:30:12 +00:00
Eldon	7b219ab011	Fix parameter passing and work with chromiums wrapper stuff	2014-01-22 02:22:16 +00:00
Eldon	db9eee5f2b	Should be full python 3 now	2014-01-22 01:32:41 +00:00
Eldon	5900e90e7d	Moving to python3	2014-01-21 22:53:28 +00:00
Eldon	f90b4db690	Add additional option to control invoked executable and wait time	2014-01-21 22:47:49 +00:00
Eldon	25a028a35c	Functioning window creation and navigation based on a single url	2014-01-21 21:32:33 +00:00
Eldon	fdb62be2ba	First commit of umbra	2014-01-21 06:41:46 +00:00