Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-04-20 23:56:34 -04:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Commit Graph

  • bc6e0d243f yet more refactoring of browser.py, clearer separation of purpose, Browser class manages browsing, sends most of the messages to chrome, WebsockReceiverThread handles messages that come back from chrome Noah Levitt 2016-12-16 13:52:12 -08:00
  • 534d2e63d6 bump version number in setup.py Noah Levitt 2016-12-15 16:43:27 -08:00
  • c71854127d major refactoring of browsing code to make it easier to add functionality Noah Levitt 2016-12-15 16:42:45 -08:00
  • ef8bc83928 Merge branch 'refactor-browsing' into qa Noah Levitt 2016-12-15 13:21:38 -08:00
  • cb6a00f4f0 don't log every little message from chrome refactor-browsing Noah Levitt 2016-12-15 13:21:30 -08:00
  • f6333df6ef back to dev version number Noah Levitt 2016-12-15 12:34:26 -08:00
  • 85de2fad6a i dub thee 1.1b8 1.1b8 Noah Levitt 2016-12-15 12:33:34 -08:00
  • 7a68599057 Merge branch 'refactor-browsing' into qa Noah Levitt 2016-12-15 12:28:21 -08:00
  • 4186869bf9 Merge branch 'master' into qa Noah Levitt 2016-12-15 12:07:29 -08:00
  • 4bdad4729a more shutdown tweaks Noah Levitt 2016-12-14 16:13:14 -08:00
  • 5fa96b6438 improving shutdown process Noah Levitt 2016-12-14 14:49:41 -08:00
  • 1dcbaeda39 improve naming Barbara Miller 2016-12-11 10:21:51 -08:00
  • 013adbc0c4 simpledo first draft Barbara Miller 2016-12-10 17:23:01 -08:00
  • f23f928c16 working on major refactoring of browser management Noah Levitt 2016-12-09 16:28:11 -08:00
  • d68053764c fix bug handling page with zero outlinks Noah Levitt 2016-12-09 16:43:23 -08:00
  • af1e1c75ec avoid infinite loop in case youtube-dl encounters redirect loop (which can be ok if cookies have been set or something) Noah Levitt 2016-12-09 14:16:27 -08:00
  • f6a25aa4f0 brozzler logo svg with small default size Noah Levitt 2016-12-08 15:16:02 -08:00
  • 40b4d9bfe8 travis-ci slack integration Noah Levitt 2016-12-07 14:46:29 -08:00
  • 9bcec54f4b fix _find_available_port and its unit test Noah Levitt 2016-12-07 14:08:34 -08:00
  • eed8b9ec30 little fixes Noah Levitt 2016-12-07 11:20:10 -08:00
  • 0b6c5346bd avoid broken version of websocket-client to fix https://github.com/internetarchive/brozzler/issues/28 Noah Levitt 2016-12-07 11:18:41 -08:00
  • e250c4ca89 wrong branch of warcprox in ansible install Noah Levitt 2016-12-07 09:33:06 -08:00
  • d3063fbd2b move cookie db management code into chrome.py Noah Levitt 2016-12-06 18:04:51 -08:00
  • ce03381b92 move _find_available_ports to chrome.py, changing the way it works so that browser:9200 doesn't get stuck at 9201 forever, which pushes 9201 to 9202 etc, and add a unit test Noah Levitt 2016-12-06 17:12:20 -08:00
  • 74009852d6 split Chrome class into its own module Noah Levitt 2016-12-06 12:50:38 -08:00
  • 3c43fdaced new utility brozzler-list-captures for looking up entries in the "captures" table Noah Levitt 2016-11-30 00:52:14 +00:00
  • 2eea50dcfb Merge branch 'master' into qa Noah Levitt 2016-11-21 16:21:30 -08:00
  • 9567c088c8 in warcprox 2.0b2, captures table field has been renamed to "record_length" Noah Levitt 2016-11-21 16:21:21 -08:00
  • 55c9ae07b7 remove flickr behavior, flickr is better off with the default behavior for now Noah Levitt 2016-11-16 17:16:48 -08:00
  • 899ee8a8dd Update README.rst Noah Levitt 2016-11-16 12:26:50 -08:00
  • 6bb9d68dce add travis-ci badge Noah Levitt 2016-11-16 12:26:33 -08:00
  • eaa32ad3fc Merge branch 'master' into qa Noah Levitt 2016-11-16 12:24:30 -08:00
  • 72816d1058 don't check robots.txt when scheduling a new site to be crawled, but mark the seed Page as needs_robots_check, and delegate the robots check to brozzler-worker; new test of robots.txt adherence Noah Levitt 2016-11-16 12:23:59 -08:00
  • 24cc8377fb robots.txt for testing Noah Levitt 2016-11-16 12:12:17 -08:00
  • 3aead6de93 monkey-patch reppy to support substring user-agent matching Noah Levitt 2016-11-16 11:41:34 -08:00
  • 398871d46b give vagrant vm enough memory so that tests pass consistently Noah Levitt 2016-11-14 18:26:00 -08:00
  • 2b0a47c914 Merge pull request #27 from internetarchive/i2 Noah Levitt 2016-11-14 12:40:55 -08:00
  • a74247412c need warcprox to listen on public address because that's what it puts in the service registry Noah Levitt 2016-11-14 10:03:40 -08:00
  • c9b45a7e76 looks like the problem may have been a bug in ansible 2.2.0.0, so pin to 2.1.3.0 Noah Levitt 2016-11-11 16:05:41 -08:00
  • e01739743f Merge branch 'i2' into qa Barbara Miller 2016-11-14 09:25:58 -08:00
  • 12a054e6dc update behavior, mostly css selectors Barbara Miller 2016-11-04 17:21:18 -07:00
  • c5ff31c906 another dumb fix fix-travis Noah Levitt 2016-11-11 18:14:47 -08:00
  • 9a6fc0b12d oops Noah Levitt 2016-11-11 17:57:24 -08:00
  • 9e109b84b6 guessing the test is failing because warcprox listens on localhost but the address from the service registry is not localhost Noah Levitt 2016-11-11 17:43:01 -08:00
  • 5aca4c2f23 remove some debugging stuff Noah Levitt 2016-11-11 16:59:14 -08:00
  • 465595ae11 looks like the problem may have been a bug in ansible 2.2.0.0, so pin to 2.1.3.0 Noah Levitt 2016-11-11 16:05:41 -08:00
  • 28b010a2ba back to dev version number Noah Levitt 2016-11-11 14:58:55 -08:00
  • 7aca046905 1.1b7 1.1b7 Noah Levitt 2016-11-11 14:58:07 -08:00
  • eb3fad9c84 cp feature branch instagram.js Barbara Miller 2016-11-11 14:51:11 -08:00
  • 54ec6cf15b Merge branch 'i2' into qa Barbara Miller 2016-11-11 14:44:10 -08:00
  • bb9334d757 jslint edits Barbara Miller 2016-11-11 13:10:43 -08:00
  • d162a85a65 update markup, & simplify big image browse? Barbara Miller 2016-11-04 17:21:18 -07:00
  • a80d6bcc9a Merge branch 'master' into qa Noah Levitt 2016-11-11 14:19:37 -08:00
  • 3ac2dabb74 Merge branch 'master' into fix-travis Noah Levitt 2016-11-11 14:15:52 -08:00
  • 26b571219b use \n to delimit outlinks because urls can contain spaces (and anything else except [\n\t\0]) in the fragment part even after browser canonicalization Noah Levitt 2016-11-11 14:14:47 -08:00
  • 7093e66360 Merge branch 'i2' into qa Barbara Miller 2016-11-11 13:34:44 -08:00
  • 51dfb2a899 jslint edits Barbara Miller 2016-11-11 13:10:43 -08:00
  • 3c3a09f5c0 Merge branch 'i2' into qa Barbara Miller 2016-11-10 17:21:33 -08:00
  • 2f6767627b update markup, & simplify big image browse? Barbara Miller 2016-11-04 17:21:18 -07:00
  • 4592cfebc1 what happens if we deactivate travis's virtualenv Noah Levitt 2016-11-10 11:49:16 -08:00
  • bd93de44e6 ugh fix typo Noah Levitt 2016-11-10 11:26:58 -08:00
  • e531b105b6 looks like installation issue due to virtualenv-within-a-virtualenv, add some more debugging related to that Noah Levitt 2016-11-10 11:03:24 -08:00
  • 96394618f4 python packages are not installing in the virtualenvs in /opt? ansible wtf? Noah Levitt 2016-11-10 10:28:17 -08:00
  • 18142703ec if we specify full paths to the executable maybe it will work (though i don't get why we would need to do that in travis but not elsewhere) Noah Levitt 2016-11-09 16:37:49 -08:00
  • 5e977a4953 could this issue be solved by "sudo initctl reload-configuration"? Noah Levitt 2016-11-09 16:18:51 -08:00
  • 119fa5ae5d Merge branch 'master' into fix-travis Noah Levitt 2016-11-09 16:16:16 -08:00
  • 0eb07c9ca2 Merge branch 'master' into qa Noah Levitt 2016-11-09 13:43:24 -08:00
  • 02bf23059e pass behavior_parameters from job configuration into Site objects Noah Levitt 2016-11-09 13:43:10 -08:00
  • 8e115b44fa add --behavior-parameters argument to brozzler-new-site Noah Levitt 2016-11-09 13:12:36 -08:00
  • 953e50d9a6 fix bug in final_bounces (not sure what I was thinking) Noah Levitt 2016-11-09 13:12:14 -08:00
  • a5eff2395b more travis-ci debugging info Noah Levitt 2016-11-09 10:01:50 -08:00
  • 97ac66f590 some stupid stuff for debugging travis Noah Levitt 2016-11-09 09:46:51 -08:00
  • 8889e4ab20 restore accidentally removed functionality handling page redirects and friends Noah Levitt 2016-11-08 18:17:48 -08:00
  • 054cb255ac cat logs on travis-ci failure Noah Levitt 2016-11-08 14:26:12 -08:00
  • 125a31165a reppy 0.4.1 has a significantly different api apparently, so for now let's go back to 0.3.4 Noah Levitt 2016-11-08 14:11:46 -08:00
  • fe18d915f5 still trying to get installation of pip to work on travis-ci Noah Levitt 2016-11-08 13:50:12 -08:00
  • f10b4c71e6 update for reppy api change and pin to current version of reppy Noah Levitt 2016-11-08 13:39:32 -08:00
  • cba5fa4a0b tweaks to ansible config to try to get the deployment to run on travis-ci Noah Levitt 2016-11-08 13:31:52 -08:00
  • fbd540244b Merge branch 'master' into qa Noah Levitt 2016-11-07 18:16:18 -08:00
  • 9d66f294ec move behavior_parameters into top level of site configuration Noah Levitt 2016-11-07 18:16:04 -08:00
  • 185d65bd5b Merge remote-tracking branch 'galgeek/login' Noah Levitt 2016-11-07 18:15:43 -08:00
  • abca90a128 install the virtualenv package with pip because the apt version is old and conflicts with the recent version of pip we're using Noah Levitt 2016-11-07 17:51:43 -08:00
  • 99feeab581 logging tweak Noah Levitt 2016-11-04 17:53:02 -07:00
  • 5ac8994a24 rename webconsole to dashboard Noah Levitt 2016-11-04 17:46:23 -07:00
  • b87128f6fe Merge branch 'i2' into qa Barbara Miller 2016-11-04 17:26:38 -07:00
  • d5849d251b update markup, & simplify big image browse? Barbara Miller 2016-11-04 17:21:18 -07:00
  • 65a46f4558 Merge branch 'instagram' into qa Barbara Miller 2016-11-03 15:49:26 -07:00
  • 509a81cdf1 new instagram behavior for new page markup Barbara Miller 2016-11-03 15:48:28 -07:00
  • c670bd060e add login details to behavior parameters Barbara Miller 2016-11-02 16:51:19 -07:00
  • 6c7f88c171 initial login additions Barbara Miller 2016-10-28 18:20:25 -07:00
  • fef7d6a9fa Merge pull request #25 from ato/mouseovers-behavior Noah Levitt 2016-10-31 11:55:46 -07:00
  • 0c8ea52b08 Merge pull request #26 from ato/flash-doc Noah Levitt 2016-10-31 11:55:02 -07:00
  • a1591a169a Update Flash plugin instructions Alex Osborne 2016-10-29 14:03:32 +11:00
  • 5bd4908e1d punycode host part of url to avoid errors doing WARCPROX_WRITE_RECORD Noah Levitt 2016-10-26 13:50:23 -07:00
  • f30c143c66 avoid exception in case of url without host part Noah Levitt 2016-10-26 12:45:24 -07:00
  • 332912acd7 apparently response.status doesn't work sometimes; response.getcode() is documented so hopefully it keeps working Noah Levitt 2016-10-25 17:50:49 -07:00
  • 73893f41f7 Merge branch 'noguchi' into qa Barbara Miller 2016-10-21 19:19:53 -07:00
  • 5c67c36474 jslint'd Barbara Miller 2016-10-21 19:02:36 -07:00
  • b93c86d9c8 Merge branch 'noguchi' into qa Barbara Miller 2016-10-21 15:30:35 -07:00
  • 04944e2f94 use site's nav JS Barbara Miller 2016-10-21 15:29:56 -07:00