Noah Levitt
196cd2c5eb
will this fix the travis build?
2017-11-08 17:41:39 -08:00
Noah Levitt
a24fac0194
Merge pull request #70 from internetarchive/skipDashManifest
...
skip remembering dash manifests
2017-11-08 17:12:44 -08:00
Noah Levitt
b81cc4eb0a
remove stray pdb line
2017-11-08 17:03:54 -08:00
Noah Levitt
133726e942
test a real-ish mpd
2017-11-08 17:01:27 -08:00
Barbara Miller
e8fdf84db8
add test--not a Video
2017-11-07 17:23:51 -08:00
Barbara Miller
91527f12df
comment referencing PR
2017-11-07 16:05:35 -08:00
Barbara Miller
31e54c94e7
skip remembering dash manifests
2017-11-06 16:43:43 -08:00
Barbara Miller
7f4deacdf7
Merge pull request #69 from BitBaron/ari-5426
...
Thanks, Neil!
2017-10-25 15:37:37 -07:00
Noah Levitt
19b67196ab
Merge pull request #68 from danielbicho/master
...
fix resume_job
2017-10-17 09:51:54 -07:00
Daniel Bicho
c4fa612547
fix some errors in test_resume_job
2017-10-17 10:33:26 +01:00
Noah Levitt
d40390f938
cryptography lib version 2.1.1 is causing problems
2017-10-16 10:52:09 -07:00
Daniel Bicho
bb98a43c8c
fix and test both job stop request and site stop requests
2017-10-16 11:46:35 +01:00
Daniel Bicho
8aa10962bc
test resume_job adding a simulation of a crawl job stopped and then resumed.
2017-10-15 19:11:46 +01:00
Daniel Bicho
378c097c29
add verification change to test_resume_job
2017-10-13 12:13:51 +01:00
Daniel Bicho
36e323c942
fix resume_job function, the job was not able to resume because the job stop_requested value was not reset.
2017-10-12 19:21:13 +01:00
Noah Levitt
554dbe821b
Merge pull request #67 from internetarchive/skip_youtube_dl
...
skip_youtube_dl
2017-09-29 15:10:10 -07:00
Barbara Miller
a86bde734f
skip unnecessary assignment too
2017-09-29 15:06:36 -07:00
Barbara Miller
e6bb6791af
skip unnecessary assignment
2017-09-29 14:53:24 -07:00
Barbara Miller
5e7b3b73dd
skip_youtube_dl
2017-09-29 14:33:23 -07:00
Noah Levitt
ec847e48bc
fix problem where each hashtag visited causes a page load if page url redirects
2017-09-27 14:11:20 -07:00
Noah Levitt
384c877e9a
new test exposing problem where each hashtag visited causes a page load, if page redirects
2017-09-27 14:08:28 -07:00
Noah Levitt
519ce4c733
Merge pull request #66 from internetarchive/ARI-5259
...
ARI-5259 blog.sina.com.cn pagination
2017-09-07 13:07:50 -07:00
Barbara Miller
eb1f79271f
blog.sin.com.cn pagination
2017-09-05 14:20:36 -07:00
Barbara Miller
71d54faae0
Merge pull request #65 from vbanos/behavior_timeout
...
Make behavior_timeout configurable
2017-08-31 14:39:39 -07:00
Vangelis Banos
bb93b04c23
Make behavior_timeout configurable
...
``behavior_timeout`` is hardcoded to 900s. With this MR we make it
configurable with a default value of 900. We add a new variable to
``BrozzlerWorker`` and ``Browser``.
2017-08-31 08:06:26 +00:00
Barbara Miller
18a52f0b15
Merge pull request #64 from galgeek/typo
...
fix typo
2017-08-26 16:58:58 -07:00
Barbara Miller
e786013b1b
fix typo
2017-08-26 16:58:00 -07:00
Barbara Miller
00b57ed87a
Merge pull request #61 from internetarchive/x11-support
...
screenshots don't work with Xvfb
2017-08-26 16:45:50 -07:00
Barbara Miller
f810603cdf
Merge pull request #63 from vbanos/configurable-page-timeout
...
Thank you, @vbanos!
2017-08-23 13:31:29 -07:00
Vangelis Banos
00513af877
Configurable page timeout
...
The page loading timeout was hard-coded to 300s. With this change,
we make it configurable with a default value of 300.
2017-08-23 08:05:14 +00:00
Neil Minton
4733b0ac7d
Update SoundCloud.com behavior selectors.
2017-08-18 14:16:51 -07:00
Neil Minton
a8a624fbbf
Add Archive.org playlists to default behavior.
2017-08-18 14:16:51 -07:00
Neil Minton
b0fd1df1ef
Generalize default behavior.
2017-08-18 14:16:51 -07:00
Neil Minton
12e02ae401
Merge pull request #62 from internetarchive/ARI-5460
...
update Instagram selectors
2017-08-17 16:08:44 -07:00
Barbara Miller
c181f4bcc3
screenshots don't work w/Xvfb
2017-08-16 15:20:43 -07:00
Barbara Miller
6391e7b40f
Merge pull request #60 from galgeek/ARI-5453
...
simpleclicks for wixsite.com
2017-08-14 17:14:09 -07:00
Barbara Miller
901995c6cf
Merge pull request #58 from internetarchive/ARI-5379
...
ARI 5379 URL regex update
2017-08-14 16:54:17 -07:00
Barbara Miller
36b7e4f3d6
Merge pull request #59 from galgeek/ARI-5465
...
skip a.uiMorePagerPrimary after all
2017-08-14 16:50:39 -07:00
Barbara Miller
b5121c26a8
simpleclicks for wixsite.com
2017-08-14 16:47:49 -07:00
Barbara Miller
12cca540bf
Merge branch 'ARI-5379' of github.com:internetarchive/brozzler into ARI-5379
2017-08-11 17:17:07 -07:00
Barbara Miller
4d99ebc1b4
only div.teaser, for *pm.gc.ca*
2017-08-11 17:16:02 -07:00
Barbara Miller
6952bf76fb
improve/expand url_regex
2017-08-11 17:16:02 -07:00
Barbara Miller
05c0cd7914
skip a.uiMorePagerPrimary after all
2017-08-11 10:41:15 -07:00
Noah Levitt
f785adb944
Merge pull request #57 from internetarchive/ARI-5409
...
simpleclicks for tuebingen.de
2017-08-10 10:59:38 -07:00
Barbara Miller
61bbb9774f
sign up button and updated selectors
2017-08-09 16:23:35 -07:00
Noah Levitt
55856d2180
Merge pull request #54 from vbanos/worker_on_request
...
Pass missing on_request callback to BrozzlerWorker methods
2017-08-03 14:17:02 -07:00
Barbara Miller
101257effe
simpleclicks for tuebingen.de
2017-08-03 13:21:46 -07:00
Noah Levitt
bf250194b4
bump dev version number after some PR merges
2017-08-01 12:04:56 -07:00
Noah Levitt
50881ab38b
Merge pull request #55 from galgeek/ARI-5242
...
ARI-5242
2017-08-01 12:04:18 -07:00
Noah Levitt
f7fe06874d
Merge pull request #53 from vbanos/bugfix-needs-browsing
...
bugfix for BrozzlerWorker._needs_browsing
2017-08-01 10:14:56 -07:00
