Other approach was too slow and caused db contention.
New approach avoids (slow) rethinkdb join by max_claimed_sites job
parameter to each of the job's sites. Uses rethinkdb fold() to count
claimed sites and enforce max_claimed_sites within a single query.
By default chromium creates multiple renderer processes (each running
multiple threads) for each instance of a site the user visits. What we
see from `ps auxcf` output is the following:
```
\_ chromium-browse
\_ chromium-browse
| \_ chromium-browse
| \_ chromium-browse
| \_ chromium-browse
| \_ chromium-browse
```
Using the `--single-process` option, we run all renderers in the same
process, saving the overhead of running multiple processes. `ps auxcf`
output is the following:
```
\_ chromium-browse
\_ chromium-browse
\_ chromium-browse
```
Performance is improved a bit and I guess that using this in large scale
Brozzler deployments will have even better performance effects.
The potential problem of `--single-process` is stability (if a renderer
crashes, the whole browser also crashes) but since we use very short-lived
instances of chromium, we don't worry about this.
Details on chromium process models:
https://www.chromium.org/developers/design-documents/process-models
* max-claimed-sites:
new job setting max_claimed_sites
move time limit enforcement
Invalid syntax in WebsockReceiverThread._javascript_dialog_open
Make Browser._wait_for sleep time a varible
Send more compact JSON to browser
Remove google safebrowsing flags
try to get chromium 64? (#92)
Add chromium CLI flags to improve capture performance
Puts a cap on the number of sites belonging to a given job that can be brozzled
simultaneously across the cluster. Addresses the problem of a job with many
seeds starving out other jobs. For AITFIVE-1578.
Every time we run a JS behavior, we load a Jinja2 template.
By default, Jinja2 has option `auto_reload=True`. This mean that
every time a template is requested the loader checks if the source file changed
and if yes, it will reload the template. For higher performance it’s possible
to disable that.
Also note that Jinja caches 400 templates by default.
Ref: http://jinja.pocoo.org/docs/2.10/api/
In Brozzler, we don't make changes to JS templates while the system is
running. So, there is no point in having auto_reload=True.
We've been seeing some of this:
2018-02-14 20:16:44,011 17816 CRITICAL BrozzlingThread:36444 brozzler.worker.BrozzlerWorker.brozzle_site(worker.py:559) unexpected exception
Traceback (most recent call last):
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/worker.py", line 528, in brozzle_site
enable_youtube_dl=not self._skip_youtube_dl)
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/worker.py", line 385, in brozzle_page
on_request)
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/worker.py", line 447, in _browse_page
cookie_db=site.get('cookie_db'))
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/browser.py", line 338, in start
self._wait_for(lambda: self.websock_thread.is_open, timeout=10)
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/browser.py", line 311, in _wait_for
elapsed, callback))
brozzler.browser.BrowsingTimeout: timed out after 11.1s waiting for: <function Browser.start.<locals>.<lambda> at 0x7fb2dc772bd8>
Mostly at startup. Now that brozzler claims sites in batches for
brozzling, we have situations where we start up a whole bunch of
browsers at the same time. That's probably why in some cases they are
slow to establish the websocket connection.
We've been seeing a lot of this:
2018-02-14 20:06:01,472 13286 CRITICAL BrozzlingThread:44789 brozzler.worker.BrozzlerWorker.brozzle_site(worker.py:559) unexpected exception
Traceback (most recent call last):
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/worker.py", line 528, in brozzle_site
enable_youtube_dl=not self._skip_youtube_dl)
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/worker.py", line 385, in brozzle_page
on_request)
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/worker.py", line 459, in _browse_page
behavior_timeout=self._behavior_timeout)
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/browser.py", line 463, in browse_page
jpeg_bytes = self.screenshot()
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/browser.py", line 565, in screenshot
timeout=timeout)
File "/opt/brozzler-ve3/lib/python3.5/site-packages/brozzler/browser.py", line 311, in _wait_for
elapsed, callback))
brozzler.browser.BrowsingTimeout: timed out after 90.5s waiting for: <function Browser.screenshot.<locals>.<lambda> at 0x7f5ab0076a68>
Browser bug, maybe? To work around it, reduce timeout to 45 seconds, try
getting screenshot 3 times, and if it fails proceed anyway, don't queue
the page for recrawling.
* master:
fix attempt for deadlock-ish situation
fix unclosed file warnings when running python in debug mode
give vagrant vm a name in virtualbox
add note to readme about browser version
check browser version at startup
Reinstate logging
Fix typo and block legacy google-analytics.com/ga.js
Use Network.setBlockedUrls instead of Debugger to block URLs
bump dev version after PR merge
back to dev version number
commit for beta release
this should fix travis build?
fix tests
update brozzler-easy for current warcprox api
claim sites to brozzle in batches to reduce contention over sites table
lengthen site session brozzling time to 15 minutes
fix needs_browsing check
new test test_needs_browsing
increase timeout waiting for screenshot
Use TCP_NODELAY in websocket connection to improve performance
``--disable-background-timer-throttling`` and ``--disable-renderer-backgrounding``:
karma JS test runner uses these to improve chrome performance
https://github.com/karma-runner/karma-chrome-launcher/issues/123
``--disable-hang-monitor``: Suppresses hang monitor dialogs in renderer
processes. This may allow slow unload handlers on a page to prevent the
tab from closing, but the Task Manager can be used to terminate the
offending process in this case.
``--mute-audio``: obvious.
The following are part of google safe browsing features:
``--disable-client-side-phishing-detection``
``--safebrowsing-disable-auto-update``
``--safebrowsing-disable-download-protection``
Reference: https://peter.sh/experiments/chromium-command-line-switches/
