Noah Levitt
eaa32ad3fc
Merge branch 'master' into qa
...
* master:
don't check robots.txt when scheduling a new site to be crawled, but mark the seed Page as needs_robots_check, and delegate the robots check to brozzler-worker; new test of robots.txt adherence
robots.txt for testing
monkey-patch reppy to support substring user-agent matching
give vagrant vm enough memory so that tests pass consistently
need warcprox to listen on public address because that's what it puts in the service registry
looks like the problem may have been a bug in ansible 2.2.0.0, so pin to 2.1.3.0
2016-11-16 12:24:30 -08:00
Noah Levitt
72816d1058
don't check robots.txt when scheduling a new site to be crawled, but mark the seed Page as needs_robots_check, and delegate the robots check to brozzler-worker; new test of robots.txt adherence
2016-11-16 12:23:59 -08:00
Noah Levitt
24cc8377fb
robots.txt for testing
2016-11-16 12:12:17 -08:00
Noah Levitt
3aead6de93
monkey-patch reppy to support substring user-agent matching
2016-11-16 11:41:34 -08:00
Noah Levitt
398871d46b
give vagrant vm enough memory so that tests pass consistently
2016-11-14 18:26:00 -08:00
Noah Levitt
2b0a47c914
Merge pull request #27 from internetarchive/i2
...
update Instagram behavior, mostly css selectors
2016-11-14 12:40:55 -08:00
Noah Levitt
a74247412c
need warcprox to listen on public address because that's what it puts in the service registry
2016-11-14 10:03:40 -08:00
Noah Levitt
c9b45a7e76
looks like the problem may have been a bug in ansible 2.2.0.0, so pin to 2.1.3.0
2016-11-14 09:58:13 -08:00
Barbara Miller
e01739743f
Merge branch 'i2' into qa
2016-11-14 09:25:58 -08:00
Barbara Miller
12a054e6dc
update behavior, mostly css selectors
2016-11-14 09:20:40 -08:00
Noah Levitt
28b010a2ba
back to dev version number
2016-11-11 14:58:55 -08:00
Noah Levitt
7aca046905
1.1b7
2016-11-11 14:58:07 -08:00
Barbara Miller
eb3fad9c84
cp feature branch instagram.js
2016-11-11 14:51:11 -08:00
Barbara Miller
54ec6cf15b
Merge branch 'i2' into qa
2016-11-11 14:44:10 -08:00
Barbara Miller
bb9334d757
jslint edits
2016-11-11 14:21:08 -08:00
Barbara Miller
d162a85a65
update markup, & simplify big image browse?
2016-11-11 14:21:08 -08:00
Noah Levitt
a80d6bcc9a
Merge branch 'master' into qa
...
* master:
use \n to delimit outlinks because urls can contain spaces (and anything else except [\n\t\0]) in the fragment part even after browser canonicalization
2016-11-11 14:19:37 -08:00
Noah Levitt
26b571219b
use \n to delimit outlinks because urls can contain spaces (and anything else except [\n\t\0]) in the fragment part even after browser canonicalization
2016-11-11 14:14:47 -08:00
Barbara Miller
7093e66360
Merge branch 'i2' into qa
2016-11-11 13:34:44 -08:00
Barbara Miller
51dfb2a899
jslint edits
2016-11-11 13:33:09 -08:00
Barbara Miller
3c3a09f5c0
Merge branch 'i2' into qa
2016-11-10 17:21:33 -08:00
Barbara Miller
2f6767627b
update markup, & simplify big image browse?
2016-11-10 17:21:20 -08:00
Noah Levitt
0eb07c9ca2
Merge branch 'master' into qa
...
* master:
pass behavior_parameters from job configuration into Site objects
add --behavior-parameters argument to brozzler-new-site
fix bug in final_bounces (not sure what I was thinking)
restore accidentally removed functionality handling page redirects and friends
cat logs on travis-ci failure
reppy 0.4.1 has a significantly different api apparently, so for now let's go back to 0.3.4
still trying to get installation of pip to work on travis-ci
update for reppy api change and pin to current version of reppy
tweaks to ansible config to try to get the deployment to run on travis-ci
2016-11-09 13:43:24 -08:00
Noah Levitt
02bf23059e
pass behavior_parameters from job configuration into Site objects
2016-11-09 13:43:10 -08:00
Noah Levitt
8e115b44fa
add --behavior-parameters argument to brozzler-new-site
2016-11-09 13:12:36 -08:00
Noah Levitt
953e50d9a6
fix bug in final_bounces (not sure what I was thinking)
2016-11-09 13:12:14 -08:00
Noah Levitt
8889e4ab20
restore accidentally removed functionality handling page redirects and friends
2016-11-08 18:17:48 -08:00
Noah Levitt
054cb255ac
cat logs on travis-ci failure
2016-11-08 14:26:12 -08:00
Noah Levitt
125a31165a
reppy 0.4.1 has a significantly different api apparently, so for now let's go back to 0.3.4
2016-11-08 14:11:46 -08:00
Noah Levitt
fe18d915f5
still trying to get installation of pip to work on travis-ci
2016-11-08 13:50:12 -08:00
Noah Levitt
f10b4c71e6
update for reppy api change and pin to current version of reppy
2016-11-08 13:39:32 -08:00
Noah Levitt
cba5fa4a0b
tweaks to ansible config to try to get the deployment to run on travis-ci
2016-11-08 13:31:52 -08:00
Noah Levitt
fbd540244b
Merge branch 'master' into qa
...
* master:
move behavior_parameters into top level of site configuration
install the virtualenv package with pip because the apt version is old and conflicts with the recent version of pip we're using
logging tweak
rename webconsole to dashboard
add login details to behavior parameters
initial login additions
2016-11-07 18:16:18 -08:00
Noah Levitt
9d66f294ec
move behavior_parameters into top level of site configuration
2016-11-07 18:16:04 -08:00
Noah Levitt
185d65bd5b
Merge remote-tracking branch 'galgeek/login'
...
* galgeek/login:
add login details to behavior parameters
initial login additions
2016-11-07 18:15:43 -08:00
Noah Levitt
abca90a128
install the virtualenv package with pip because the apt version is old and conflicts with the recent version of pip we're using
2016-11-07 17:51:43 -08:00
Noah Levitt
99feeab581
logging tweak
2016-11-04 17:53:02 -07:00
Noah Levitt
5ac8994a24
rename webconsole to dashboard
2016-11-04 17:46:23 -07:00
Barbara Miller
b87128f6fe
Merge branch 'i2' into qa
2016-11-04 17:26:38 -07:00
Barbara Miller
d5849d251b
update markup, & simplify big image browse?
2016-11-04 17:21:18 -07:00
Barbara Miller
65a46f4558
Merge branch 'instagram' into qa
2016-11-03 15:49:26 -07:00
Barbara Miller
509a81cdf1
new instagram behavior for new page markup
2016-11-03 15:48:28 -07:00
Barbara Miller
c670bd060e
add login details to behavior parameters
2016-11-02 16:51:19 -07:00
Barbara Miller
6c7f88c171
initial login additions
2016-11-02 16:04:18 -07:00
Noah Levitt
fef7d6a9fa
Merge pull request #25 from ato/mouseovers-behavior
...
Add a mouseovers behavior
2016-10-31 11:55:46 -07:00
Noah Levitt
0c8ea52b08
Merge pull request #26 from ato/flash-doc
...
Update Flash plugin instructions
2016-10-31 11:55:02 -07:00
Alex Osborne
a1591a169a
Update Flash plugin instructions
...
libpepflashplayer.so is no longer included in the Chrome release
package. Adobe are resuming Linux releases and the plugin is
now available from their download site.
2016-10-29 14:12:33 +11:00
Noah Levitt
5bd4908e1d
punycode host part of url to avoid errors doing WARCPROX_WRITE_RECORD
2016-10-26 13:50:23 -07:00
Noah Levitt
f30c143c66
avoid exception in case of url without host part
2016-10-26 12:45:24 -07:00
Noah Levitt
332912acd7
apparently response.status doesn't work sometimes; response.getcode() is documented so hopefully it keeps working
2016-10-25 17:50:49 -07:00
