1128 Commits

Author SHA1 Message Date
Noah Levitt
8de3e21103 fix another oversight 2018-09-28 14:45:45 -07:00
Noah Levitt
1ee36c38b9 ugh. oops 2018-09-28 13:44:54 -07:00
Noah Levitt
7137918005 Revert "add a github PR template for this repo"
This reverts commit 83552eb444fc5ef9bd4b9d1772820b62aae67e46.
2018-09-28 13:16:46 -07:00
Noah Levitt
7980b40ee3 improve performance of brozzler-new-job
by inserting pages and sites in bulk
2018-09-28 13:13:22 -07:00
Noah Levitt
2386e85a37 bump doublethink dependency 2018-09-27 14:25:49 -07:00
jkafader
f8ce9858fb
Merge pull request #121 from nlevitt/purge
brozzler-purge - purge crawl state from rethinkdb
2018-09-25 16:45:24 -07:00
Noah Levitt
f4fad934a7 verbiage tweaks 2018-09-25 15:19:33 -07:00
Noah Levitt
560981c1ad safety check and --force for brozzler-purge 2018-09-25 15:17:45 -07:00
Noah Levitt
174178e02e new command brozzler-purge 2018-09-25 14:56:26 -07:00
Noah Levitt
48bf185746 bump version after merge 2018-09-18 11:08:44 -07:00
Noah Levitt
dceee8bdbd
Merge pull request #119 from nlevitt/ydl-stitch-fix
WIP youtube-dl stitching fixes
2018-09-18 11:08:21 -07:00
Noah Levitt
60cd69e2bd send warcprox-meta when pushing stitched up video
also put locking around monkey patching to avoid race condition
2018-09-18 01:07:52 -07:00
Noah Levitt
1ef717fa75 test exposing bug that we don't send warcprox-meta
when pushing stitched-up video with WARCPROX_WRITE_RECORD
2018-09-18 01:05:18 -07:00
Noah Levitt
efb0696833 bump version number after merge 2018-09-06 16:17:59 -07:00
jkafader
8368cd2bcb
Merge pull request #115 from nlevitt/ydl-stitched
Ydl stitched
2018-09-06 16:15:52 -07:00
Noah Levitt
a4eacb5b8f oops, back to dev version number 2018-09-04 10:52:34 -07:00
jkafader
e38b867ff5
Merge pull request #118 from nlevitt/relax-claiming
Relax claiming
2018-09-04 10:45:11 -07:00
Noah Levitt
2d5c6681cf wait 20 seconds to claim sites if none were avail-
able last time, up from 0.5 seconds
this should lighten the load on rethinkdb considerably
2018-08-31 15:23:59 -07:00
Noah Levitt
d0f5cd7168 tweak logging 2018-08-31 15:23:48 -07:00
Noah Levitt
88d3d3b310
why did those tests fail??? (#117)
1.4 for pypi
1.4
2018-08-22 14:35:39 -07:00
Noah Levitt
02e98f101d
Merge pull request #116 from kblumenthal/master
Add screenshots
2018-08-22 14:34:52 -07:00
Karl-Rainer Blumenthal
ff1645ef7d
Add screenshots
Add Brozzler Dashboard and Wayback screenshots to readme
2018-08-22 13:02:08 -04:00
Karl-Rainer Blumenthal
7c8b597ad3
Add screenshots
Add screenshots of Brozzler Dashboard and Wayback
2018-08-22 12:55:10 -04:00
Noah Levitt
2a2952e810 back to dev version 2018-08-21 15:18:18 -07:00
Noah Levitt
b63661ea70 1.4 for pypi 2018-08-21 15:15:38 -07:00
Noah Levitt
eaf7ef74be explain --warcprox-auto briefly 2018-08-17 12:06:04 -07:00
Karl-Rainer Blumenthal
2081e6388a
Merge pull request #2 from internetarchive/master
Updating to upstream origin
2018-08-17 14:26:46 -04:00
Noah Levitt
8cdc3dee21 Merge branch 'master' into ydl-stitched
* master:
  vagrant readme fixes (thanks funkyfuture)
  update cryptography dep version
2018-08-17 10:34:00 -07:00
Noah Levitt
d19e139101 vagrant readme fixes (thanks funkyfuture) 2018-08-17 10:31:01 -07:00
Noah Levitt
ffa8021968 update cryptography dep version
github tells me there's a vulnerability <2.3
2018-08-16 14:32:03 -07:00
Noah Levitt
e7d2273856 fix failing tests 2018-08-16 11:40:54 -07:00
Noah Levitt
3c27132aaa test for youtube-dl stitch-up 2018-08-15 17:42:53 -07:00
Noah Levitt
c2ad8427e1 add missing imports and fix mimetype issue 2018-08-15 17:41:35 -07:00
Noah Levitt
33520da8f9 move youtube-dl code into separate file 2018-08-14 15:10:48 -07:00
Noah Levitt
39155ebcc5 push youtube-dl's stitched up videos to warcprox
(no tests yet)
2018-08-13 15:40:48 -07:00
Noah Levitt
4e398e1da2 expose more brozzle-page args 2018-08-13 15:38:24 -07:00
Noah Levitt
b44a444dc2 update pillow dependency to get rid of github vul-
nerability warning
2018-07-24 16:37:25 -05:00
Noah Levitt
771d6aa626 more readme edits 2018-07-23 19:05:49 -05:00
Noah Levitt
073fc713f4
Merge pull request #113 from nlevitt/karl-readme
Karl readme copy edits
2018-07-23 18:36:00 -05:00
Noah Levitt
f7407a87c1 reformat readme to 80 columns 2018-07-23 23:32:56 +00:00
Noah Levitt
a7fb7bcc37 Merge branch 'master' into karl
* master:
  bump up heartbeat interval (see comment)
  back to dev version
  version 1.3 (messed up 1.2)
  setuptools wants README not readme
  back to dev version number
  version 1.2
  bump dev version after merge
  is test_time_limit is failing because of timing?
  fix bug in test, add another one
  treat any error fetching robots.txt as "allow all"
  update instagram behavior
2018-07-23 23:28:42 +00:00
Karl-Rainer Blumenthal
bd78e07232
Copy edits to job-conf readme
Good reading and rampant pedantry!
2018-07-06 15:24:12 -04:00
Noah Levitt
9d18dc6aeb bump up heartbeat interval (see comment) 2018-07-03 18:35:08 -05:00
Karl-Rainer Blumenthal
eebbc1d279
Copy edits 2018-06-28 12:59:22 -04:00
Noah Levitt
783fd0ea87 back to dev version 2018-06-25 19:32:27 +00:00
Noah Levitt
bd63908fb9 version 1.3 (messed up 1.2) 1.3 2018-06-25 19:30:39 +00:00
Noah Levitt
2780c92569 setuptools wants README not readme 2018-06-25 19:10:57 +00:00
Noah Levitt
032c7d2898 back to dev version number 2018-06-25 12:33:34 -05:00
Noah Levitt
442d02b26a version 1.2 1.2 2018-06-25 12:21:00 -05:00
Noah Levitt
196cd555ea bump dev version after merge 2018-06-25 11:44:45 -05:00