Git-Mirrors/brozzler
1
0
Fork 0
mirror of https://github.com/internetarchive/brozzler.git synced 2025-08-09 23:13:07 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions
1361 commits 62 branches 20 tags 10 MiB
Commit graph

1361 commits

This branch
This branch
All branches
Author SHA1 Message Date
Barbara Miller
eb1f79271f blog.sin.com.cn pagination 2017-09-05 14:20:36 -07:00
Barbara Miller
71d54faae0 Merge pull request #65 from vbanos/behavior_timeout 
Make behavior_timeout configurable
 2017-08-31 14:39:39 -07:00
Barbara Miller
17d410f000 Merge branch 'behavior_timeout' into qa 2017-08-31 10:31:55 -07:00
Vangelis Banos
bb93b04c23 Make behavior_timeout configurable 
``behavior_timeout`` is hardcoded to 900s. With this MR we make it
configurable with a default value of 900. We add a new variable to
``BrozzlerWorker`` and ``Browser``.
 2017-08-31 08:06:26 +00:00
Barbara Miller
18a52f0b15 Merge pull request #64 from galgeek/typo 
fix typo
 2017-08-26 16:58:58 -07:00
Barbara Miller
e786013b1b fix typo 2017-08-26 16:58:00 -07:00
Barbara Miller
00b57ed87a Merge pull request #61 from internetarchive/x11-support 
screenshots don't work with Xvfb
 2017-08-26 16:45:50 -07:00
Neil Minton
337945004c Merge branch 'aitfive-1295' into qa 2017-08-25 10:34:51 -07:00
Neil Minton
5ad7c9c7cc Revert "Log oulinks for all users of Browser." 
This reverts commit 58b95fa7bf.

It was decided that this change didn't make sense for Brozzler.
 2017-08-25 10:30:42 -07:00
Barbara Miller
f810603cdf Merge pull request #63 from vbanos/configurable-page-timeout 
Thank you, @vbanos!
 2017-08-23 13:31:29 -07:00
Barbara Miller
96ba4f1a78 Merge branch 'configurable-page-timeout' into qa 2017-08-23 11:11:32 -07:00
Vangelis Banos
00513af877 Configurable page timeout 
The page loading timeout was hard-coded to 300s. With this change,
we make it configurable with a default value of 300.
 2017-08-23 08:05:14 +00:00
Neil Minton
60d7d59442 Merge branch 'ari-5426' into qa 2017-08-18 14:18:58 -07:00
Neil Minton
4733b0ac7d Update SoundCloud.com behavior selectors. 2017-08-18 14:16:51 -07:00
Neil Minton
a8a624fbbf Add Archive.org playlists to default behavior. 2017-08-18 14:16:51 -07:00
Neil Minton
b0fd1df1ef Generalize default behavior. 2017-08-18 14:16:51 -07:00
Neil Minton
12e02ae401 Merge pull request #62 from internetarchive/ARI-5460 
update Instagram selectors
 2017-08-17 16:08:44 -07:00
Barbara Miller
c181f4bcc3 screenshots don't work w/Xvfb 2017-08-16 15:20:43 -07:00
Barbara Miller
6391e7b40f Merge pull request #60 from galgeek/ARI-5453 
simpleclicks for wixsite.com
 2017-08-14 17:14:09 -07:00
Barbara Miller
901995c6cf Merge pull request #58 from internetarchive/ARI-5379 
ARI 5379 URL regex update
 2017-08-14 16:54:17 -07:00
Barbara Miller
36b7e4f3d6 Merge pull request #59 from galgeek/ARI-5465 
skip a.uiMorePagerPrimary after all
 2017-08-14 16:50:39 -07:00
Barbara Miller
b5121c26a8 simpleclicks for wixsite.com 2017-08-14 16:47:49 -07:00
Barbara Miller
12cca540bf Merge branch 'ARI-5379' of github.com:internetarchive/brozzler into ARI-5379 2017-08-11 17:17:07 -07:00
Barbara Miller
4d99ebc1b4 only div.teaser, for *pm.gc.ca* 2017-08-11 17:16:02 -07:00
Barbara Miller
6952bf76fb improve/expand url_regex 2017-08-11 17:16:02 -07:00
Barbara Miller
fdf07d006d Merge branch 'ARI-5465' into qa 2017-08-11 10:41:57 -07:00
Barbara Miller
05c0cd7914 skip a.uiMorePagerPrimary after all 2017-08-11 10:41:15 -07:00
Noah Levitt
f785adb944 Merge pull request #57 from internetarchive/ARI-5409 
simpleclicks for tuebingen.de
 2017-08-10 10:59:38 -07:00
Barbara Miller
44e2b8579a Merge branch 'ARI-5460' into qa 2017-08-09 16:24:13 -07:00
Barbara Miller
61bbb9774f sign up button and updated selectors 2017-08-09 16:23:35 -07:00
Barbara Miller
abd024f091 Merge branch 'ARI-5453' into qa 2017-08-08 11:37:50 -07:00
Barbara Miller
ec710f34be simpleclicks for wixsite.com 2017-08-08 11:37:24 -07:00
Noah Levitt
55856d2180 Merge pull request #54 from vbanos/worker_on_request 
Pass missing on_request callback to BrozzlerWorker methods
 2017-08-03 14:17:02 -07:00
Barbara Miller
101257effe simpleclicks for tuebingen.de 2017-08-03 13:21:46 -07:00
Neil Minton
3e8e699661 Merge branch 'aitfive-1295' into qa 2017-08-02 10:59:00 -07:00
Noah Levitt
5be7dd4407 Merge branch 'master' into qa 
* master:
  bump dev version number after some PR merges
  bugfix for BrozzlerWorker._needs_browsing
  Remove redundant method parameter.
  bugfix
  Make youtube-dl optional in BrozzlerWorker.brozzle_page
 2017-08-01 12:05:07 -07:00
Noah Levitt
bf250194b4 bump dev version number after some PR merges 2017-08-01 12:04:56 -07:00
Noah Levitt
50881ab38b Merge pull request #55 from galgeek/ARI-5242 
ARI-5242
 2017-08-01 12:04:18 -07:00
Noah Levitt
f7fe06874d Merge pull request #53 from vbanos/bugfix-needs-browsing 
bugfix for BrozzlerWorker._needs_browsing
 2017-08-01 10:14:56 -07:00
Vangelis Banos
78b9d61654 Pass missing on_request callback to BrozzlerWorker methods 
``Browser.browser_page`` has the ``on_request`` parameter but it is on
used by ``BrozzlerWorker._browse_page`` where we invoke it.

I add this to ``BrozzlerWorker.brozzle_page`` and pass it also to
``BrozzlerWorker._browser_page``. Now, it is possible to use this
callback from other applications when calling
``BrozzlerWorker.brozzle_page``.
 2017-08-01 13:58:42 +00:00
Vangelis Banos
ae73edb244 bugfix for BrozzlerWorker._needs_browsing 
I'm sorry but with my previous commit I introduced a bug in
``BrozzlerWorker._needs_browsing`` method.

More specifically, if the ``brozzler_spy`` param is False (this happens
when ``youtube_dl`` is disabled), ``_needs_browsing`` method returns
always ``False`` and this messes with ``brozzle_page`` workflow. The
browser is nevenr visiting the page.

I'm fixing this here.
 2017-08-01 11:43:42 +00:00
Neil Minton
58b95fa7bf Log oulinks for all users of Browser. 2017-07-31 15:43:53 -07:00
Noah Levitt
895bfbf913 Merge pull request #51 from vbanos/youtube-dl-option 
Make youtube-dl optional in BrozzlerWorker.brozzle_page
 2017-07-31 11:32:51 -07:00
Noah Levitt
7d62eb6525 Merge pull request #52 from vbanos/remove-redundant-param 
Remove redundant method parameter.
 2017-07-31 11:23:28 -07:00
Vangelis Banos
0343969807 Remove redundant method parameter. 
``ignore_cert_errors`` is passed to ``Chrome`` via ``Browser`` via
``BrowserPool` here:

https://github.com/internetarchive/brozzler/blob/master/brozzler/worker.py#L120

it is not doing anything in ``Browser.browser_page``.
 2017-07-31 12:36:17 +00:00
Vangelis Banos
6259d03be1 bugfix 2017-07-31 10:36:35 +00:00
Vangelis Banos
9c81a7bbda Make youtube-dl optional in BrozzlerWorker.brozzle_page 
Enabled by default (of course).
Speed is significantly improved when disabled.
 2017-07-31 08:57:47 +00:00
Barbara Miller
d743dcd53e Merge branch 'behavior-refactor' into qa 2017-07-26 15:43:08 -07:00
Barbara Miller
2e300166f2 reset didSomething for each action 2017-07-26 15:42:25 -07:00
Barbara Miller
ef1bd179c5 Merge branch 'behavior-refactor' into qa 2017-07-21 16:40:02 -07:00