Git-Mirrors/qubes-doc
1
0
Fork 0
mirror of https://github.com/QubesOS/qubes-doc.git synced 2025-09-20 13:04:45 -04:00
Code Issues Projects Releases Packages Wiki Activity

Convert to RST

Browse source 
This is done using tools at
https://github.com/maiska/qubes-translation-utilz, commit
4c8e2a7f559fd37e29b51769ed1ab1c6cf92e00d.
This commit is contained in:
Marek Marczykowski-Górecki 2025-07-04 14:23:09 +02:00
parent e14921b8a3
commit 918eaa479d
No known key found for this signature in database
GPG key ID: F32894BE9684938A
248 changed files with 23904 additions and 20774 deletions
Download patch file Download diff file Expand all files Collapse all files

25
.readthedocs.yaml Normal file
View file

@ -0,0 +1,25 @@
version: 2
build:
os: "ubuntu-lts-latest"
tools:
python: "3.13"
submodules:
include: all
recursive: true
sphinx:
builder: html
configuration: conf.py
fail_on_warning: false
python:
install:
- requirements: requirements.txt
formats:
- pdf
- epub

23
_dev/index.rst
View file

@ -1,9 +1,26 @@
==============================================
Welcome to Qubes OS developer's documentation!
==============================================
This is documentation for the source code. Please choose specific repostitory:
* `core-admin </projects/core-admin>`_
* `core-admin-client </projects/core-admin-client>`_
.. toctree::
:maxdepth: 1
Or see `the main Qubes OS documentation <https://www.qubes-os.org/doc/>`_.
core-admin <>
.. _core-admin: /projects/core-admin
core-admin-client <>
.. _core-admin-client: /projects/core-admin-client
Or see the main Qubes OS documentation <https://www.qubes-os.org/doc/>
.. _the main qubes os documentation: https://www.qubes-os.org/doc/
.

133
conf.py Normal file
View file

@ -0,0 +1,133 @@
"""
ReST directive for embedding Youtube and Vimeo videos.
There are two directives added: ``youtube`` and ``vimeo``. The only
argument is the video id of the video to include.
Both directives have three optional arguments: ``height``, ``width``
and ``align``. Default height is 281 and default width is 500.
Example::
.. youtube:: anwy2MPT5RE
:height: 315
:width: 560
:align: left
:copyright: (c) 2012 by Danilo Bargen.
:license: BSD 3-clause
"""
from __future__ import absolute_import
from docutils import nodes
from docutils.parsers.rst import Directive, directives
# -- Project information -----------------------------------------------------
project = 'Qubes OS'
copyright = '2025, Qubes OS Project'
author = 'Qubes OS Project'
title = "Qubes Docs"
html_title = "Qubes Docs"
# The full version, including alpha/beta/rc tags
release = '0.1'
# -- General configuration ---------------------------------------------------
html_static_path = ['attachment/doc']
extensions = [
'sphinx.ext.autosectionlabel',
'sphinxnotes.strike',
'sphinx_reredirects'
]
redirects = {
"user/hardware/hcl": "https://www.qubes-os.org/hcl/",
"user/downloading-installing-upgrading/downloads:mirrors":"https://www.qubes-os.org/downloads/mirrors/",
"developer/general/visual-style-guide": "https://www.qubes-os.org/doc/visual-style-guide/",
"developer/general/website-style-guide":"https://www.qubes-os.org/doc/website-style-guide/",
"user/downloading-installing-upgrading/downloads":"https://www.qubes-os.org/downloads/",
"developer/general/how-to-edit-the-documentation":"https://www.qubes-os.org/doc/how-to-edit-the-documentation/"
}
autosectionlabel_prefix_document = True
source_suffix = {
'.rst': 'restructuredtext',
}
templates_path = ['_templates']
root_doc = "index"
exclude_patterns = [
'_dev/*',
'attachment/*',
'**/*.txt'
]
html_theme = 'sphinx_rtd_theme'
# html_theme = 'default'
# html_theme = 'classic'
html_theme_options = {
'externalrefs': True,
'bgcolor': 'white',
'linkcolor': '#99bfff',
'textcolor': '#000000',
'visitedlinkcolor': '#7b7b7b',
'bodyfont': '"Open Sans", Arial, sans-serif',
'codebgcolor': '$color-qube-light',
'codebgcolor': 'grey',
'body_min_width': '50%',
'body_max_width': '90%',
'collapse_navigation': True,
}
gettext_uuid = True
gettext_compact = False
# epub_show_urls = 'footnote'
# latex_show_urls ='footnote'
locale_dirs = ['_translated']
# from https://gist.github.com/ehles/bed012d78aad5d3cd6c35a49bef32f9f
def align(argument):
"""Conversion function for the "align" option."""
return directives.choice(argument, ('left', 'center', 'right'))
class IframeVideo(Directive):
has_content = False
required_arguments = 1
optional_arguments = 0
final_argument_whitespace = False
option_spec = {
'height': directives.nonnegative_int,
'width': directives.nonnegative_int,
'align': align,
}
default_width = 500
default_height = 281
def run(self):
self.options['video_id'] = directives.uri(self.arguments[0])
if not self.options.get('width'):
self.options['width'] = self.default_width
if not self.options.get('height'):
self.options['height'] = self.default_height
if not self.options.get('align'):
self.options['align'] = 'left'
return [nodes.raw('', self.html % self.options, format='html')]
class GeneralVid(IframeVideo):
html = '<iframe src="%(video_id)s" width="%(width)u" height="%(height)u" frameborder="0" webkitAllowFullScreen mozallowfullscreen allowfullscreen class="responsive" referrerpolicy="no-referrer" scrolling="no"></iframe>'
class Youtube(IframeVideo):
html = '<iframe src="https://www.youtube-nocookie.com/embed/%(video_id)s" \
width="%(width)u" height="%(height)u" frameborder="0" \
webkitAllowFullScreen mozallowfullscreen allowfullscreen \
class="responsive" referrerpolicy="no-referrer" scrolling="no"></iframe>'
def setup(builder):
directives.register_directive('youtube', Youtube)
directives.register_directive('generalvid', GeneralVid)

801
developer/building/development-workflow.rst
View file

@ -1,524 +1,559 @@
---
lang: en
layout: doc
permalink: /doc/development-workflow/
redirect_from:
- /en/doc/development-workflow/
- /doc/DevelopmentWorkflow/
- /wiki/DevelopmentWorkflow/
ref: 66
title: Development workflow
---
====================
Development workflow
====================
A workflow for developing Qubes OS+
To begin, setup [QubesBuilder](/doc/qubes-builder-v2/). This guide
assumes you're using qubes-builder v2 to build Qubes.
To begin, setup :doc:`QubesBuilder </developer/building/qubes-builder-v2>`. This guide assumes you’re using qubes-builder v2 to build Qubes.
## Repositories and committing Code
Repositories and committing Code
--------------------------------
Qubes source code is split into many git repos. These are all contained in the
`artifacts/sources` directory under qubes-builder. Subdirectories there are
separate components, stored in separate git repositories.
The best way to write and contribute code is to create a git repo somewhere
(e.g., GitHub) for the repo you are interested in editing (e.g.,
`qubes-manager`, `core-agent-linux`, etc). To integrate your repo with the rest
of Qubes, cd to the repo directory and add your repository as a remote in git
Qubes source code is split into many git repos. These are all contained in the ``artifacts/sources`` directory under qubes-builder. Subdirectories there are separate components, stored in separate git repositories.
The best way to write and contribute code is to create a git repo somewhere (e.g., GitHub) for the repo you are interested in editing (e.g., ``qubes-manager``, ``core-agent-linux``, etc). To integrate your repo with the rest of Qubes, cd to the repo directory and add your repository as a remote in git
**Example:**
~~~
$ cd qubes-builder/artifacts/sources/qubes-manager
$ git remote add abel git@GitHub.com:abeluck/qubes-manager.git
~~~
.. code:: bash
You can then proceed to easily develop in your own branches, pull in new
commits from the dev branches, merge them, and eventually push to your own repo.
$ cd qubes-builder/artifacts/sources/qubes-manager
$ git remote add abel git@GitHub.com:abeluck/qubes-manager.git
When you are ready to submit your changes to Qubes to be merged, push your
changes, then create a signed git tag (using `git tag -s`). Finally, send a
letter to the Qubes listserv describing the changes, and including a link to
your repository. If you are using GitHub you can instead create a pull request.
Don't forget to include the public PGP key you use to sign your tags.
### Kernel-specific notes
#### Prepare fresh version of kernel sources, with Qubes-specific patches applied
You can then proceed to easily develop in your own branches, pull in new commits from the dev branches, merge them, and eventually push to your own repo.
In `qubes-builder/artifacts/sources/linux-kernel`:
When you are ready to submit your changes to Qubes to be merged, push your changes, then create a signed git tag (using ``git tag -s``). Finally, send a letter to the Qubes listserv describing the changes, and including a link to your repository. If you are using GitHub you can instead create a pull request. Don’t forget to include the public PGP key you use to sign your tags.
~~~
make prep
~~~
Kernel-specific notes
^^^^^^^^^^^^^^^^^^^^^
The resulting tree will be in kernel-\<VERSION\>/linux-\<VERSION\>:
~~~
ls -ltrd kernel*/linux*
~~~
Prepare fresh version of kernel sources, with Qubes-specific patches applied
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
~~~
drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18
drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj
~~~
#### Go to the kernel tree and update the version
In ``qubes-builder/artifacts/sources/linux-kernel``:
In `qubes-builder/artifacts/sources/linux-kernel`:
.. code:: bash
~~~
cd kernel-3.4.18/linux-3.4.18
~~~
make prep
#### Changing the config
In `kernel-3.4.18/linux-3.4.18`:
~~~
cp ../../config .config
make oldconfig
~~~
The resulting tree will be in kernel-<VERSION>/linux-<VERSION>:
.. code:: bash
ls -ltrd kernel*/linux*
.. code:: bash
drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18
drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj
Go to the kernel tree and update the version
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In ``qubes-builder/artifacts/sources/linux-kernel``:
.. code:: bash
cd kernel-3.4.18/linux-3.4.18
Changing the config
^^^^^^^^^^^^^^^^^^^
In ``kernel-3.4.18/linux-3.4.18``:
.. code:: bash
cp ../../config .config
make oldconfig
Now change the configuration. For example, in ``kernel-3.4.18/linux-3.4.18``:
.. code:: bash
make menuconfig
Now change the configuration. For example, in `kernel-3.4.18/linux-3.4.18`:
~~~
make menuconfig
~~~
Copy the modified config back into the kernel tree:
~~~
cp .config ../../../config
~~~
.. code:: bash
cp .config ../../../config
Patching the code
^^^^^^^^^^^^^^^^^
#### Patching the code
TODO: describe the workflow for patching the code, below are some random notes, not working well
~~~
ln -s ../../patches.xen
export QUILT_PATCHES=patches.xen
export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
export QUILT_SERIES=../../series-pvops.conf
.. code:: bash
quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch
quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \
drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \
include/xen/interface/io/usbif.h
ln -s ../../patches.xen
export QUILT_PATCHES=patches.xen
export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
export QUILT_SERIES=../../series-pvops.conf
quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch
quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \
drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \
include/xen/interface/io/usbif.h
*edit something*
quilt refresh
cd ../..
vi series.conf
*edit something*
quilt refresh
cd ../..
vi series.conf
~~~
#### Building RPMs
Building RPMs
^^^^^^^^^^^^^
Now is a good moment to make sure you have changed the kernel release name in
rel file. For example, if you change it to '1debug201211116c' the
resulting RPMs will be named
'kernel-3.4.18-1debug20121116c.pvops.qubes.x86\_64.rpm'. This will help
distinguish between different versions of the same package.
You might want to take a moment here to review (git diff, git status), commit
your changes locally.
Now is a good moment to make sure you have changed the kernel release name in rel file. For example, if you change it to ‘1debug201211116c’ the resulting RPMs will be named ‘kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm’. This will help distinguish between different versions of the same package.
You might want to take a moment here to review (git diff, git status), commit your changes locally.
To actually build RPMs, in qubes-builder:
~~~
./qb -c linux-kernel package fetch prep build
~~~
.. code:: bash
RPMs will appear in
`artifacts/repository/destination_name/package_name`
(for example `artifacts/repository/host-fc37/linux-kernel-6.6.31-1.1/`
./qb -c linux-kernel package fetch prep build
### Useful [QubesBuilder](/doc/qubes-builder-v2/) commands
1. `./qb package diff` - show uncommitted changes
2. ` ./qb repository check-release-status-for-component` and
`./qb repository check-release-status-for-template` - show version of each
component/template (based on git tags)
3. `./qb package sign` - sign built packages
4. `./qb package publish` and `./qb package upload` - publish signed packages
and upload published
repository
## Copying Code to dom0
RPMs will appear in ``artifacts/repository/destination_name/package_name`` (for example ``artifacts/repository/host-fc37/linux-kernel-6.6.31-1.1/``
When developing it is convenient to be able to rapidly test changes. Assuming
you're developing Qubes on Qubes, you should be working in a special VM for
Qubes and occasionally you will want to transfer code or RPMs back to dom0 for
testing.
Useful :doc:`QubesBuilder </developer/building/qubes-builder-v2>` commands
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1. ``./qb package diff`` - show uncommitted changes
2. ``./qb repository check-release-status-for-component`` and ``./qb repository check-release-status-for-template`` - show version of each component/template (based on git tags)
3. ``./qb package sign`` - sign built packages
4. ``./qb package publish`` and ``./qb package upload`` - publish signed packages and upload published repository
Copying Code to dom0
--------------------
When developing it is convenient to be able to rapidly test changes. Assuming you’re developing Qubes on Qubes, you should be working in a special VM for Qubes and occasionally you will want to transfer code or RPMs back to dom0 for testing.
Here are some handy scripts Marek has shared to facilitate this.
You may also like to run your [test environment on separate
machine](/doc/test-bench/).
You may also like to run your :doc:`test environment on separate machine </developer/debugging/test-bench>`.
Syncing dom0 files
^^^^^^^^^^^^^^^^^^
### Syncing dom0 files
TODO: edit this script to be more generic
~~~
#!/bin/sh
.. code:: bash
set -x
set -e
#!/bin/sh
set -x
set -e
QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes
QUBES_PY=$QUBES_PY_DIR/qubes.py
QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py
qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv
cp $QUBES_PY qubes.py.bak$$
cp $QUBESUTILS_PY qubesutils.py.bak$$
cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$
sudo cp qvm-core/qubes.py $QUBES_PY
sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY
sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/
sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/
sudo cp misc/vm-template-hvm.conf /usr/share/qubes/
sudo cp misc/qubes-start.desktop /usr/share/qubes/
sudo cp misc/block-snapshot /etc/xen/scripts/
sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/
# FIXME(Abel Luck): I hope to
QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes
QUBES_PY=$QUBES_PY_DIR/qubes.py
QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py
qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv
cp $QUBES_PY qubes.py.bak$$
cp $QUBESUTILS_PY qubesutils.py.bak$$
cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$
sudo cp qvm-core/qubes.py $QUBES_PY
sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY
sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/
sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/
sudo cp misc/vm-template-hvm.conf /usr/share/qubes/
sudo cp misc/qubes-start.desktop /usr/share/qubes/
sudo cp misc/block-snapshot /etc/xen/scripts/
sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/
# FIXME(Abel Luck): I hope to
~~~
### Apply qvm-tools
Apply qvm-tools
^^^^^^^^^^^^^^^
TODO: make it more generic
~~~
#!/bin/sh
.. code:: bash
BAK=qvm-tools.bak$$
mkdir -p $BAK
cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/
sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/
~~~
#!/bin/sh
BAK=qvm-tools.bak$$
mkdir -p $BAK
cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/
sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/
### Copy from dom0 to an appvm
~~~
#/bin/sh
#
# usage ./cp-domain <vm_name> <file_to_copy>
#
domain=$1
file=$2
fname=`basename $file`
qvm-run $domain 'mkdir /home/user/incoming/dom0 -p'
cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname"
~~~
Copy from dom0 to an appvm
^^^^^^^^^^^^^^^^^^^^^^^^^^
## Git connection between VMs
Sometimes it's useful to transfer git commits between VMs. You can use `git
format-patch` for that and simply copy the files. But you can also setup
custom qrexec service for it.
.. code:: bash
Below example assumes that you use `builder-RX` directory in target VM to
store sources in qubes-builder layout (where `X` is some number). Make sure that
all the scripts are executable.
#/bin/sh
#
# usage ./cp-domain <vm_name> <file_to_copy>
#
domain=$1
file=$2
fname=`basename $file`
qvm-run $domain 'mkdir /home/user/incoming/dom0 -p'
cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname"
Service file (save in `/usr/local/etc/qubes-rpc/local.Git` in target VM):
~~~
#!/bin/sh
exec 2>/tmp/log2
Git connection between VMs
--------------------------
read service rel repo
echo "Params: $service $rel $repo" >&2
# Adjust regexps if needed
echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1
echo "$rel" | grep -q '^[0-9.]\+$' || exit 1
path="/home/user/builder-R$rel/qubes-src/$repo"
if [ "$repo" = "builder" ]; then
path="/home/user/builder-R$rel"
fi
case $service in
git-receive-pack|git-upload-pack)
echo "starting $service $path" >&2
exec $service $path
;;
*)
echo "Unsupported service: $service" >&2
;;
esac
~~~
Client script (save in `~/bin/git-qrexec` in source VM):
Sometimes it’s useful to transfer git commits between VMs. You can use ``git format-patch`` for that and simply copy the files. But you can also setup custom qrexec service for it.
~~~
#!/bin/sh
Below example assumes that you use ``builder-RX`` directory in target VM to store sources in qubes-builder layout (where ``X`` is some number). Make sure that all the scripts are executable.
VMNAME=$1
Service file (save in ``/usr/local/etc/qubes-rpc/local.Git`` in target VM):
(echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git
~~~
.. code:: bash
You will also need to setup qrexec policy in dom0 (`/etc/qubes-rpc/policy/local.Git`).
#!/bin/sh
exec 2>/tmp/log2
read service rel repo
echo "Params: $service $rel $repo" >&2
# Adjust regexps if needed
echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1
echo "$rel" | grep -q '^[0-9.]\+$' || exit 1
path="/home/user/builder-R$rel/qubes-src/$repo"
if [ "$repo" = "builder" ]; then
path="/home/user/builder-R$rel"
fi
case $service in
git-receive-pack|git-upload-pack)
echo "starting $service $path" >&2
exec $service $path
;;
*)
echo "Unsupported service: $service" >&2
;;
esac
Client script (save in ``~/bin/git-qrexec`` in source VM):
.. code:: bash
#!/bin/sh
VMNAME=$1
(echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git
You will also need to setup qrexec policy in dom0 (``/etc/qubes-rpc/policy/local.Git``).
Usage:
~~~
[user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux"
[user@source core-agent-linux]$ git push testbuilder master
~~~
.. code:: bash
You can create `~/bin/add-remote` script to ease adding remotes:
[user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux"
[user@source core-agent-linux]$ git push testbuilder master
~~~
#!/bin/sh
[ -n "$1" ] || exit 1
if [ "$1" = "tb" ]; then
git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`"
exit $?
fi
You can create ``~/bin/add-remote`` script to ease adding remotes:
git remote add $1 git@GitHub.com:$1/qubes-`basename $PWD`
~~~
.. code:: bash
It should be executed from component top level directory. This script takes one
argument - remote name. If it is `tb`, then it creates qrexec-based git remote
to `testbuilder` VM. Otherwise it creates remote pointing at GitHub account of
the same name. In any case it points at repository matching current directory
name.
#!/bin/sh
[ -n "$1" ] || exit 1
if [ "$1" = "tb" ]; then
git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`"
exit $?
fi
git remote add $1 git@GitHub.com:$1/qubes-`basename $PWD`
## Sending packages to different VM
Other useful script(s) can be used to setup local package repository hosted in
some VM. This way you can keep your development VM behind firewall, while
having an option to expose some yum/apt repository to the local network (to
have them installed on test machine).
To achieve this goal, a dummy repository can be created, which instead of
populating metadata locally, will upload the packages to some other VM and
trigger repository update there (using qrexec). You can use `unstable`
repository flavor, because there is no release managing rules bundled (unlike
current and current-testing).
It should be executed from component top level directory. This script takes one argument - remote name. If it is ``tb``, then it creates qrexec-based git remote to ``testbuilder`` VM. Otherwise it creates remote pointing at GitHub account of the same name. In any case it points at repository matching current directory name.
### RPM packages - yum repo
Sending packages to different VM
--------------------------------
In source VM, grab [linux-yum](https://GitHub.com/QubesOS/qubes-linux-yum) repository (below is assumed you've made it in
`~/repo-yum-upload` directory) and replace `update_repo.sh` script with:
~~~
#!/bin/sh
Other useful script(s) can be used to setup local package repository hosted in some VM. This way you can keep your development VM behind firewall, while having an option to expose some yum/apt repository to the local network (to have them installed on test machine).
VMNAME=repo-vm
To achieve this goal, a dummy repository can be created, which instead of populating metadata locally, will upload the packages to some other VM and trigger repository update there (using qrexec). You can use ``unstable`` repository flavor, because there is no release managing rules bundled (unlike current and current-testing).
set -e
qvm-copy-to-vm $VMNAME $1
# remove only files, leave directory structure
find -type f -name '*.rpm' -delete
# trigger repo update
qrexec-client-vm $VMNAME local.UpdateYum
~~~
RPM packages - yum repo
^^^^^^^^^^^^^^^^^^^^^^^
In target VM, setup actual yum repository (also based on [linux-yum](https://GitHub.com/QubesOS/qubes-linux-yum), this time
without modifications). You will also need to setup some gpg key for signing
packages (it is possible to force yum to install unsigned packages, but it
isn't possible for `qubes-dom0-update` tool). Fill `~/.rpmmacros` with
key description:
~~~
%_gpg_name Test packages signing key
~~~
In source VM, grab `linux-yum <https://GitHub.com/QubesOS/qubes-linux-yum>`__ repository (below is assumed you’ve made it in ``~/repo-yum-upload`` directory) and replace ``update_repo.sh`` script with:
Then setup `local.UpdateYum` qrexec service (`/usr/local/etc/qubes-rpc/local.UpdateYum`):
.. code:: bash
~~~
#!/bin/sh
#!/bin/sh
VMNAME=repo-vm
set -e
qvm-copy-to-vm $VMNAME $1
# remove only files, leave directory structure
find -type f -name '*.rpm' -delete
# trigger repo update
qrexec-client-vm $VMNAME local.UpdateYum
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
exit 1
fi
real_repository=/home/user/linux-yum
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1
In target VM, setup actual yum repository (also based on `linux-yum <https://GitHub.com/QubesOS/qubes-linux-yum>`__, this time without modifications). You will also need to setup some gpg key for signing packages (it is possible to force yum to install unsigned packages, but it isn’t possible for ``qubes-dom0-update`` tool). Fill ``~/.rpmmacros`` with key description:
rsync -lr --remove-source-files $incoming/ $real_repository
cd $real_repository
export SKIP_REPO_CHECK=1
if [ -d $incoming/r3.1 ]; then
./update_repo-unstable.sh r3.1
fi
.. code:: bash
if [ -d $incoming/r3.0 ]; then
./update_repo-unstable.sh r3.0
fi
%_gpg_name Test packages signing key
if [ -d $incoming/r2 ]; then
./update_repo-unstable.sh r2
fi
find $incoming -type d -empty -delete
exit 0
~~~
Of course you will also need to setup qrexec policy in dom0
`/etc/qubes-rpc/policy/local.UpdateYum`.
If you want to access the repository from network, you need to setup HTTP
server serving it, and configure the system to let other machines actually
reach this HTTP server. You can use for example using [port
forwarding](/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world) or setting up Tor hidden service. Configuration
details of those services are outside of the scope of this page.
Then setup ``local.UpdateYum`` qrexec service (``/usr/local/etc/qubes-rpc/local.UpdateYum``):
Usage: setup `builder.conf` in source VM to use your dummy-uploader repository:
.. code:: bash
~~~
LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1
~~~
#!/bin/sh
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
exit 1
fi
real_repository=/home/user/linux-yum
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1
rsync -lr --remove-source-files $incoming/ $real_repository
cd $real_repository
export SKIP_REPO_CHECK=1
if [ -d $incoming/r3.1 ]; then
./update_repo-unstable.sh r3.1
fi
if [ -d $incoming/r3.0 ]; then
./update_repo-unstable.sh r3.0
fi
if [ -d $incoming/r2 ]; then
./update_repo-unstable.sh r2
fi
find $incoming -type d -empty -delete
exit 0
Then use `make update-repo-unstable` to upload the packages. You can also
specify selected components on command line, then build them and upload to the
repository:
~~~
make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable
~~~
On the test machine, add yum repository (`/etc/yum.repos.d`) pointing at just
configured HTTP server. For example:
Of course you will also need to setup qrexec policy in dom0 ``/etc/qubes-rpc/policy/local.UpdateYum``.
~~~
[local-test]
name=Test
baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20
~~~
If you want to access the repository from network, you need to setup HTTP server serving it, and configure the system to let other machines actually reach this HTTP server. You can use for example using :ref:`port forwarding <user/security-in-qubes/firewall:port forwarding to a qube from the outside world>` or setting up Tor hidden service. Configuration details of those services are outside of the scope of this page.
Remember to also import gpg public key using `rpm --import`.
Usage: setup ``builder.conf`` in source VM to use your dummy-uploader repository:
.. code:: bash
LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1
Then use ``make update-repo-unstable`` to upload the packages. You can also specify selected components on command line, then build them and upload to the repository:
.. code:: bash
make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable
On the test machine, add yum repository (``/etc/yum.repos.d``) pointing at just configured HTTP server. For example:
.. code:: bash
[local-test]
name=Test
baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20
Remember to also import gpg public key using ``rpm --import``.
Deb packages - Apt repo
^^^^^^^^^^^^^^^^^^^^^^^
### Deb packages - Apt repo
Steps are mostly the same as in the case of yum repo. The only details that differ:
- use [linux-deb](https://GitHub.com/QubesOS/qubes-linux-deb) instead of [linux-yum](https://GitHub.com/QubesOS/qubes-linux-yum) as a base - both in source and target VM
- use different `update_repo.sh` script in source VM (below)
- use `local.UpdateApt` qrexec service in target VM (code below)
- in target VM additionally place `update-local-repo.sh` script in repository dir (code below)
- use `linux-deb <https://GitHub.com/QubesOS/qubes-linux-deb>`__ instead of `linux-yum <https://GitHub.com/QubesOS/qubes-linux-yum>`__ as a base - both in source and target VM
`update_repo.sh` script:
- use different ``update_repo.sh`` script in source VM (below)
~~~
#!/bin/sh
- use ``local.UpdateApt`` qrexec service in target VM (code below)
set -e
- in target VM additionally place ``update-local-repo.sh`` script in repository dir (code below)
current_release=$1
VMNAME=repo-vm
qvm-copy-to-vm $VMNAME $1
find $current_release -type f -name '*.deb' -delete
rm -f $current_release/vm/db/*
qrexec-client-vm $VMNAME local.UpdateApt
~~~
`local.UpdateApt` service code (`/usr/local/etc/qubes-rpc/local.UpdateApt` in repo-serving VM):
``update_repo.sh`` script:
~~~
#!/bin/sh
.. code:: bash
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
exit 1
fi
#!/bin/sh
set -e
current_release=$1
VMNAME=repo-vm
qvm-copy-to-vm $VMNAME $1
find $current_release -type f -name '*.deb' -delete
rm -f $current_release/vm/db/*
qrexec-client-vm $VMNAME local.UpdateApt
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/
cd /home/user/linux-deb
export SKIP_REPO_CHECK=1
if [ -d $incoming/r3.1 ]; then
for dist in `ls r3.1/vm/dists`; do
./update-local-repo.sh r3.1/vm $dist
done
fi
if [ -d $incoming/r3.0 ]; then
for dist in `ls r3.0/vm/dists`; do
./update-local-repo.sh r3.0/vm $dist
done
fi
``local.UpdateApt`` service code (``/usr/local/etc/qubes-rpc/local.UpdateApt`` in repo-serving VM):
if [ -d $incoming/r2 ]; then
for dist in `ls r2/vm/dists`; do
./update-local-repo.sh r2/vm $dist
done
fi
find $incoming -type d -empty -delete
exit 0
~~~
.. code:: bash
`update-local-repo.sh`:
#!/bin/sh
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
exit 1
fi
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/
cd /home/user/linux-deb
export SKIP_REPO_CHECK=1
if [ -d $incoming/r3.1 ]; then
for dist in `ls r3.1/vm/dists`; do
./update-local-repo.sh r3.1/vm $dist
done
fi
if [ -d $incoming/r3.0 ]; then
for dist in `ls r3.0/vm/dists`; do
./update-local-repo.sh r3.0/vm $dist
done
fi
if [ -d $incoming/r2 ]; then
for dist in `ls r2/vm/dists`; do
./update-local-repo.sh r2/vm $dist
done
fi
find $incoming -type d -empty -delete
exit 0
~~~
#!/bin/sh
set -e
# Set this to your local repository signing key
SIGN_KEY=01ABCDEF
``update-local-repo.sh``:
[ -z "$1" ] && { echo "Usage: $0 <repo> <dist>"; exit 1; }
.. code:: bash
REPO_DIR=$1
DIST=$2
#!/bin/sh
set -e
# Set this to your local repository signing key
SIGN_KEY=01ABCDEF
[ -z "$1" ] && { echo "Usage: $0 <repo> <dist>"; exit 1; }
REPO_DIR=$1
DIST=$2
if [ "$DIST" = "wheezy-unstable" ]; then
DIST_TAG=deb7
elif [ "$DIST" = "jessie-unstable" ]; then
DIST_TAG=deb8
elif [ "$DIST" = "stretch-unstable" ]; then
DIST_TAG=deb9
fi
pushd $REPO_DIR
mkdir -p dists/$DIST/main/binary-amd64
dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages
gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz
cat > dists/$DIST/Release <<EOF
Label: Test repo
Suite: $DIST
Codename: $DIST
Date: `date -R`
Architectures: amd64
Components: main
SHA1:
EOF
function calc_sha1() {
f=dists/$DIST/$1
echo -n " "
echo -n `sha1sum $f|cut -d' ' -f 1` ""
echo -n `stat -c %s $f` ""
echo $1
}
calc_sha1 main/binary-amd64/Packages >> dists/$DIST/Release
rm -f $DIST/Release.gpg
rm -f $DIST/InRelease
gpg -abs -u "$SIGN_KEY" \
< dists/$DIST/Release > dists/$DIST/Release.gpg
gpg -a -s --clearsign -u "$SIGN_KEY" \
< dists/$DIST/Release > dists/$DIST/InRelease
popd
if [ `id -u` -eq 0 ]; then
chown -R --reference=$REPO_DIR $REPO_DIR
fi
if [ "$DIST" = "wheezy-unstable" ]; then
DIST_TAG=deb7
elif [ "$DIST" = "jessie-unstable" ]; then
DIST_TAG=deb8
elif [ "$DIST" = "stretch-unstable" ]; then
DIST_TAG=deb9
fi
pushd $REPO_DIR
mkdir -p dists/$DIST/main/binary-amd64
dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages
gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz
cat > dists/$DIST/Release <<EOF
Label: Test repo
Suite: $DIST
Codename: $DIST
Date: `date -R`
Architectures: amd64
Components: main
SHA1:
EOF
function calc_sha1() {
f=dists/$DIST/$1
echo -n " "
echo -n `sha1sum $f|cut -d' ' -f 1` ""
echo -n `stat -c %s $f` ""
echo $1
}
calc_sha1 main/binary-amd64/Packages >> dists/$DIST/Release
rm -f $DIST/Release.gpg
rm -f $DIST/InRelease
gpg -abs -u "$SIGN_KEY" \
< dists/$DIST/Release > dists/$DIST/Release.gpg
gpg -a -s --clearsign -u "$SIGN_KEY" \
< dists/$DIST/Release > dists/$DIST/InRelease
popd
Usage: add this line to ``/etc/apt/sources.list`` on test machine (adjust host and path):
if [ `id -u` -eq 0 ]; then
chown -R --reference=$REPO_DIR $REPO_DIR
fi
~~~
.. code:: bash
deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main
Usage: add this line to `/etc/apt/sources.list` on test machine (adjust host and path):
~~~
deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main
~~~

87
developer/building/qubes-builder-details.rst
View file

@ -1,54 +1,70 @@
---
lang: en
layout: doc
permalink: /doc/qubes-builder-details/
redirect_from:
- /en/doc/qubes-builder-details/
- /doc/QubesBuilderDetails/
- /wiki/QubesBuilderDetails/
ref: 65
title: Qubes builder details
---
=====================
Qubes builder details
=====================
<div class="alert alert-warning" role="alert">
<i class="fa fa-exclamation-circle"></i>
<b>Note:</b> This information concerns the old Qubes builder (v1). It supports
only building Qubes 4.1 or earlier.<br>The build process has been completely rewritten in <a href="https://github.com/QubesOS/qubes-builderv2/">qubes-builder v2</a>. This can be used for building Qubes R4.1 and later, and all related components.</div>
.. warning::
**Note:** This information concerns the old Qubes builder (v1). It supports only building Qubes 4.1 or earlier.The build process has been completely rewritten in `qubes-builder v2 <https://github.com/QubesOS/qubes-builderv2/>`__ . This can be used for building Qubes R4.1 and later, and all related components.
Components Makefile.builder file
--------------------------------
[QubesBuilder](/doc/qubes-builder/) expects that each component have *Makefile.builder* file in its root directory. This file specifies what should be done to build the package. As name suggests, this is normal makefile, which is included by builder as its configuration. Its main purpose is to set some variables. Generally all available variables/settings are described as comments at the beginning of Makefile.\* in [QubesBuilder](/doc/qubes-builder/).
:doc:`QubesBuilder </developer/building/qubes-builder>` expects that each component have *Makefile.builder* file in its root directory. This file specifies what should be done to build the package. As name suggests, this is normal makefile, which is included by builder as its configuration. Its main purpose is to set some variables. Generally all available variables/settings are described as comments at the beginning of Makefile.* in :doc:`QubesBuilder </developer/building/qubes-builder>`.
Variables for Linux build:
- `RPM_SPEC_FILES` List (space separated) of spec files for RPM package build. Path should be relative to component root directory. [QubesBuilder](/doc/qubes-builder/) will install all BuildRequires (in chroot environment) before the build. In most Qubes components all spec files are kept in *rpm\_spec* directory. This is mainly used for Fedora packages build.
- `ARCH_BUILD_DIRS` List (space separated) of directories with PKGBUILD files for Archlinux package build. Similar to RPM build, [QubesBuilder](/doc/qubes-builder/) will install all makedepends, then build the package.
- ``RPM_SPEC_FILES`` List (space separated) of spec files for RPM package build. Path should be relative to component root directory. :doc:`QubesBuilder </developer/building/qubes-builder>` will install all BuildRequires (in chroot environment) before the build. In most Qubes components all spec files are kept in *rpm_spec* directory. This is mainly used for Fedora packages build.
- ``ARCH_BUILD_DIRS`` List (space separated) of directories with PKGBUILD files for Archlinux package build. Similar to RPM build, :doc:`QubesBuilder </developer/building/qubes-builder>` will install all makedepends, then build the package.
Most components uses *archlinux* directory for this purpose, so its good to keep this style.
Variables for Windows build:
- `WIN_COMPILER` Choose which compiler should be used for this component, thus which build scripts. Currently two options available:
- `WDK` - Windows Driver Kit (default). Command used to build: *build -cZg*.
- `mingw` - MinGW (Windows gcc port). Command used to build: *make all*
- `WIN_SOURCE_SUBDIRS` List of directories in which above command should be run. In most cases it will be only one entry: current directory (*.*).
- `WIN_PREBUILD_CMD` Command to run before build, mostly useful for WDK build (in mingw case, you can use makefile for this purpose). Can be used to set some variables, preprocess some files etc.
- `WIN_SIGN_CMD` Command used to sign resulting binaries. Note that default value is *sign.bat*. If you don't want to sign binaries, specify some placeholder here (eg. *true*). Check existing components (eg. vmm-xen-windows-pvdrivers) for example scripts. This command will be run with certain environment variables:
- `CERT_FILENAME` Path to key file (pfx format)
- `CERT_PASSWORD` Key password
- `CERT_PUBLIC_FILENAME` Certificate path in the case of self-signed cert
- `CERT_CROSS_CERT_FILENAME` Certificate path in the case of correct autheticode cert
- `SIGNTOOL` Path to signtool
- `WIN_PACKAGE_CMD` Command used to produce installation package (msi or msm). Default value is *wix.bat*, similar to above - use *true* if you don't want this command.
- `WIN_OUTPUT_HEADERS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with public headers of the package - for use in other components.
- `WIN_OUTPUT_LIBS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with libraries (both DLL and implib) of the package - for use in other components. Note that [QubesBuilder](/doc/qubes-builder/) will copy files specified as *\$(WIN\_OUTPUT\_LIBS)/\*/\** to match WDK directory layout (*\<specified directory\>/\<arch directory\>/\<actual libraries\>*), so you in mingw build you need to place libraries in some additional subdirectory.
- `WIN_BUILD_DEPS` List of components required to build this one. [QubesBuilder](/doc/qubes-builder/) will copy files specified with `WIN_OUTPUT_HEADERS` and `WIN_OUTPUT_LIBS` of those components to some directory and provide its path with `QUBES_INCLUDES` and `QUBES_LIBS` variables. Use those variables in your build scripts (*sources* or *Makefile* - depending on selected compiler). You can assume that the variables are always set and directories always exists, even if empty.
- ``WIN_COMPILER`` Choose which compiler should be used for this component, thus which build scripts. Currently two options available:
- ``WDK`` - Windows Driver Kit (default). Command used to build: *build -cZg*.
- ``mingw`` - MinGW (Windows gcc port). Command used to build: *make all*
- ``WIN_SOURCE_SUBDIRS`` List of directories in which above command should be run. In most cases it will be only one entry: current directory (*.*).
- ``WIN_PREBUILD_CMD`` Command to run before build, mostly useful for WDK build (in mingw case, you can use makefile for this purpose). Can be used to set some variables, preprocess some files etc.
- ``WIN_SIGN_CMD`` Command used to sign resulting binaries. Note that default value is *sign.bat*. If you don’t want to sign binaries, specify some placeholder here (eg. *true*). Check existing components (eg. vmm-xen-windows-pvdrivers) for example scripts. This command will be run with certain environment variables:
- ``CERT_FILENAME`` Path to key file (pfx format)
- ``CERT_PASSWORD`` Key password
- ``CERT_PUBLIC_FILENAME`` Certificate path in the case of self-signed cert
- ``CERT_CROSS_CERT_FILENAME`` Certificate path in the case of correct autheticode cert
- ``SIGNTOOL`` Path to signtool
- ``WIN_PACKAGE_CMD`` Command used to produce installation package (msi or msm). Default value is *wix.bat*, similar to above - use *true* if you don’t want this command.
- ``WIN_OUTPUT_HEADERS`` Directory (relative to ``WIN_SOURCE_SUBDIRS`` element) with public headers of the package - for use in other components.
- ``WIN_OUTPUT_LIBS`` Directory (relative to ``WIN_SOURCE_SUBDIRS`` element) with libraries (both DLL and implib) of the package - for use in other components. Note that :doc:`QubesBuilder </developer/building/qubes-builder>` will copy files specified as *$(WIN_OUTPUT_LIBS)/\*/\** to match WDK directory layout (*<specified directory>/<arch directory>/<actual libraries>*), so you in mingw build you need to place libraries in some additional subdirectory.
- ``WIN_BUILD_DEPS`` List of components required to build this one. :doc:`QubesBuilder </developer/building/qubes-builder>` will copy files specified with ``WIN_OUTPUT_HEADERS`` and ``WIN_OUTPUT_LIBS`` of those components to some directory and provide its path with ``QUBES_INCLUDES`` and ``QUBES_LIBS`` variables. Use those variables in your build scripts (*sources* or *Makefile* - depending on selected compiler). You can assume that the variables are always set and directories always exists, even if empty.
builder.conf settings
---------------------
Most settings are documented in *builder.conf.default* file, which can be used as template the actual configuration.
**TODO**
@ -56,4 +72,7 @@ Most settings are documented in *builder.conf.default* file, which can be used a
Notes
-----
* For a list of custom TemplateVMs available in QubesBuilder look at [Supported Versions page](/doc/supported-releases/).
- For a list of custom TemplateVMs available in QubesBuilder look at :doc:`Supported Versions page </user/downloading-installing-upgrading/supported-releases>`.

247
developer/building/qubes-builder-v2.rst
View file

@ -1,166 +1,155 @@
---
lang: en
layout: doc
permalink: /doc/qubes-builder-v2/
redirect_from:
- /en/doc/qubes-builder-v2/
- /doc/QubesBuilder2/
- /wiki/QubesBuilder2/
ref: 311
title: Qubes builder v2
---
This is a brief introduction to using Qubes Builder v2 to work with Qubes OS
sources. It will walk you through installing and configuring Builder v2, and
using it to fetch and build Qubes OS packages.
For details and customization, use [Qubes OS v2 builder documentation](https://github.com/QubesOS/qubes-builderv2/).
# Overview
In the second generation of Qubes OS builder, container or disposable qube
isolation is used to perform every stage of the build and release process.
From fetching sources to building, everything is executed inside an isolated
*cage* (either a disposable or a container) using an *executor*. For every
command that needs to perform an action on sources, like cloning and
verifying Git repos, rendering a SPEC file, generating SRPM or Debian
source packages, a new cage is used. Only the signing, publishing, and
uploading stages are executed locally outside a cage.
================
Qubes builder v2
================
# Setup
This is a brief introduction to using Qubes Builder v2 to work with Qubes OS sources. It will walk you through installing and configuring Builder v2, and using it to fetch and build Qubes OS packages.
This is a simple setup using a docker executor. This is a good default choice;
if you don't know which executor to use, use docker.
For details and customization, use `Qubes OS v2 builder documentation <https://github.com/QubesOS/qubes-builderv2/>`__.
1. First, decide what qube you are going to use when working with Qubes
Builder v2. It can be an AppVM or a Standalone qube, with some steps
different between the two.
Overview
--------
In the second generation of Qubes OS builder, container or disposable qube isolation is used to perform every stage of the build and release process. From fetching sources to building, everything is executed inside an isolated *cage* (either a disposable or a container) using an *executor*. For every command that needs to perform an action on sources, like cloning and verifying Git repos, rendering a SPEC file, generating SRPM or Debian source packages, a new cage is used. Only the signing, publishing, and uploading stages are executed locally outside a cage.
Setup
-----
This is a simple setup using a docker executor. This is a good default choice; if you don’t know which executor to use, use docker.
1. First, decide what qube you are going to use when working with Qubes Builder v2. It can be an AppVM or a Standalone qube, with some steps different between the two.
2. Installing dependencies
- If you want to use an app qube for developing, install dependencies in the template.
If you are using a standalone, install them in the qube itself.
Dependencies are specified in `dependencies-*.
txt` files in the main builder directory, and you can install them easily
in the following ways:
- If you want to use an app qube for developing, install dependencies in the template. If you are using a standalone, install them in the qube itself. Dependencies are specified in ``dependencies-*. txt`` files in the main builder directory, and you can install them easily in the following ways:
1. for Fedora, use:
```shell
$ sudo dnf install $(cat dependencies-fedora.txt)
$ test -f /usr/share/qubes/marker-vm && sudo dnf install qubes-gpg-split
```
2. for Debian (note: some Debian packages require Debian version 13 or
later), use:
```shell
$ sudo apt install $(cat dependencies-debian.txt)
$ test -f /usr/share/qubes/marker-vm && sudo apt install qubes-gpg-split
```
If you have installed dependencies in the template, close it, and
(re)start the development qube.
.. code:: bash
$ sudo dnf install $(cat dependencies-fedora.txt)
$ test -f /usr/share/qubes/marker-vm && sudo dnf install qubes-gpg-split
2. for Debian (note: some Debian packages require Debian version 13 or later), use:
.. code:: bash
$ sudo apt install $(cat dependencies-debian.txt)
$ test -f /usr/share/qubes/marker-vm && sudo apt install qubes-gpg-split
If you have installed dependencies in the template, close it, and (re)start the development qube.
3. Clone the qubes-builder v2 repository into a location of your choice:
```shell
git clone https://github.com/QubesOS/qubes-builderv2
cd qubes-builderv2/
```
.. code:: bash
4. If you haven't previously used docker in the current qube, you need to set up
some permissions. In particular, the user has to be added to the `docker`
group:
git clone https://github.com/QubesOS/qubes-builderv2
cd qubes-builderv2/
```shell
$ sudo usermod -aG docker user
```
Next, **restart the qube**.
4. If you haven’t previously used docker in the current qube, you need to set up some permissions. In particular, the user has to be added to the ``docker`` group:
.. code:: bash
$ sudo usermod -aG docker user
Next, **restart the qube**.
5. Finally, you need to generate a docker image:
```shell
$ tools/generate-container-image.sh docker
```
.. code:: bash
In an app qube, as `/var/lib/docker` is not persistent by default, you also
need to use [bind-dirs](/doc/bind-dirs/) to avoid repeating this step after reboot, adding
the following to the `/rw/config/qubes-bind-dirs.d/docker.conf` file in
this qube:
$ tools/generate-container-image.sh docker
```
binds+=( '/var/lib/docker' )
```
In an app qube, as ``/var/lib/docker`` is not persistent by default, you also need to use :doc:`bind-dirs </user/advanced-topics/bind-dirs>` to avoid repeating this step after reboot, adding the following to the ``/rw/config/qubes-bind-dirs.d/docker.conf`` file in this qube:
# Configuration
.. code:: bash
To use Qubes OS Builder v2, you need to have a `builder.yml` configuration file.
You can use one of the sample files from the `example-configs/` directory; for a
more readable `builder.yml`, you can also include one of the files from that
directory in your `builder.yml`. An example `builder.yml` is:
```
# include configuration relevant for the current release
include:
- example-configs/qubes-os-r4.2.yml
# which repository to use to fetch sources
use-qubes-repo:
version: 4.2
testing: true
# each package built will have local build number appended to package release
# number. It makes it easier to update in testing environment
increment-devel-versions: true
# reduce output
debug: false
# this can be set to true if you do not want sources to be automatically
# fetched from git
skip-git-fetch: false
# executor configuration
executor:
type: docker
options:
image: "qubes-builder-fedora:latest"
```
binds+=( '/var/lib/docker' )
# Using Builder v2
To fetch sources - in this example, for the `core-admin-client` package, you
can use the following command:
```shell
$ ./qb -c core-admin-client package fetch
```
This will fetch the sources for the listed package and place them in
`artifacts/sources` directory.
Configuration
-------------
To build a package (from sources in the `artifacts/sources` directory), use:
```shell
$ ./qb -c core-admin-client package fetch prep build
```
To use Qubes OS Builder v2, you need to have a ``builder.yml`` configuration file. You can use one of the sample files from the ``example-configs/`` directory; for a more readable ``builder.yml``, you can also include one of the files from that directory in your ``builder.yml``. An example ``builder.yml`` is:
or, if you want to build for a specific target (`host-fc37` is a `dom0`
using Fedora 37, `vm-fc40` would be a qube using Fedora 40 etc.), use:
.. code:: bash
# include configuration relevant for the current release
include:
- example-configs/qubes-os-r4.2.yml
# which repository to use to fetch sources
use-qubes-repo:
version: 4.2
testing: true
# each package built will have local build number appended to package release
# number. It makes it easier to update in testing environment
increment-devel-versions: true
# reduce output
debug: false
# this can be set to true if you do not want sources to be automatically
# fetched from git
skip-git-fetch: false
# executor configuration
executor:
type: docker
options:
image: "qubes-builder-fedora:latest"
Using Builder v2
----------------
To fetch sources - in this example, for the ``core-admin-client`` package, you can use the following command:
.. code:: bash
$ ./qb -c core-admin-client package fetch
This will fetch the sources for the listed package and place them in ``artifacts/sources`` directory.
To build a package (from sources in the ``artifacts/sources`` directory), use:
.. code:: bash
$ ./qb -c core-admin-client package fetch prep build
or, if you want to build for a specific target (``host-fc37`` is a ``dom0`` using Fedora 37, ``vm-fc40`` would be a qube using Fedora 40 etc.), use:
.. code:: bash
$ ./qb -c core-admin-client -d host-fc37 package fetch prep build
```shell
$ ./qb -c core-admin-client -d host-fc37 package fetch prep build
```
If you want to fetch the entire Qubes OS source use the following:
```shell
$ ./qb package fetch
```
.. code:: bash
**caution**: some repositories might have additional requirements. You can
disable repositories that are not needed in the `example-configs/*.yml`
file you are using by commenting them out. In particular, `python-fido2`,
`lvm` and `windows`-related repositories have special requirements.
$ ./qb package fetch
**caution**: some repositories might have additional requirements. You can disable repositories that are not needed in the ``example-configs/*.yml`` file you are using by commenting them out. In particular, ``python-fido2``, ``lvm`` and ``windows``-related repositories have special requirements.

301
developer/building/qubes-builder.rst
View file

@ -1,218 +1,241 @@
---
lang: en
layout: doc
permalink: /doc/qubes-builder/
redirect_from:
- /en/doc/qubes-builder/
- /doc/QubesBuilder/
- /wiki/QubesBuilder/
ref: 64
title: Qubes builder
---
<div class="alert alert-warning" role="alert">
<i class="fa fa-exclamation-circle"></i>
<b>Note:</b> These instructions concern the older Qubes builder (v1). It supports
only building Qubes 4.2 or earlier.<br>The build process has been completely rewritten in <a href="https://github.com/QubesOS/qubes-builderv2/">qubes-builder v2</a>. This can be used for building Qubes R4.2 and later, and all related components.
</div>
**Note: See [ISO building instructions](/doc/qubes-iso-building/) for a streamlined overview on how to use the build system.**
=============
Qubes builder
=============
We have a fully automated build system for Qubes, that downloads, builds and
packages all the Qubes components, and finally should spit out a ready-to-use
installation ISO, all in a [secure](/news/2016/05/30/build-security/) way.
.. warning::
**Note:** These instructions concern the older Qubes builder (v1). It supports only building Qubes 4.2 or earlier.The build process has been completely rewritten in `qubes-builder v2 <https://github.com/QubesOS/qubes-builderv2/>`__ . This can be used for building Qubes R4.2 and later, and all related components.
**Note: See** :doc:`ISO building instructions </developer/building/qubes-iso-building>` **for a streamlined overview on how to use the build system.**
We have a fully automated build system for Qubes, that downloads, builds and packages all the Qubes components, and finally should spit out a ready-to-use installation ISO, all in a `secure <https://www.qubes-os.org/news/2016/05/30/build-security/>`__ way.
In order to use it, you should use an rpm-based distro, like Fedora :), and should ensure the following packages are installed:
- sudo
- gnupg
- git
- createrepo
- rpm-build
- dnf-plugins-core
- make
- wget
- rpmdevtools
- python3-sh
- dialog
- rpm-sign
- dpkg-dev
- debootstrap
- python3-pyyaml
- devscripts
- perl-Digest-MD5
- perl-Digest-SHA
Usually you can install those packages by just issuing:
```shell
sudo dnf install gnupg git createrepo rpm-build make wget rpmdevtools python3-sh dialog rpm-sign dpkg-dev debootstrap python3-pyyaml devscripts perl-Digest-MD5 perl-Digest-SHA
```
.. code:: bash
The build system creates build environments in chroots and so no other packages are needed on the host.
All files created by the build system are contained within the qubes-builder directory.
The full build requires some 25GB of free space, so keep that in mind when deciding where to place this directory.
sudo dnf install gnupg git createrepo rpm-build make wget rpmdevtools python3-sh dialog rpm-sign dpkg-dev debootstrap python3-pyyaml devscripts perl-Digest-MD5 perl-Digest-SHA
The build system is configured via builder.conf file.
You can use the setup.sh script to create and modify this file.
Alternatively, you can copy the provided default builder.conf, and modify it as needed, e.g.:
```shell
cp example-configs/qubes-os-master.conf builder.conf
# edit the builder.conf file and set the following variables:
NO_SIGN=1
```
The build system creates build environments in chroots and so no other packages are needed on the host. All files created by the build system are contained within the qubes-builder directory. The full build requires some 25GB of free space, so keep that in mind when deciding where to place this directory.
One additional useful requirement is that 'sudo root' must work without any prompt, which is default on most distros (e.g. 'sudo bash' brings you the root shell without asking for any password).
This is important as the builder needs to switch to root and then back to user several times during the build process.
The build system is configured via builder.conf file. You can use the setup.sh script to create and modify this file. Alternatively, you can copy the provided default builder.conf, and modify it as needed, e.g.:
Additionally, if building with signing enabled (NO\_SIGN is not set), you must adjust `\~/.rpmmacros` file so that it points to the GPG key used for package signing, e.g.:
.. code:: bash
```bash
%_signature gpg
%_gpg_path /home/user/.gnupg
%_gpg_name AC1BF9B3 # <-- Key ID used for signing
```
cp example-configs/qubes-os-master.conf builder.conf
# edit the builder.conf file and set the following variables:
NO_SIGN=1
It is also recommended that you use an empty passphrase for the private key used for signing.
Contrary to a popular belief, this doesn't affect your key or sources security -- if somebody compromised your system, then the game is over anyway, whether you have used an additional passphrase for the key or not.
One additional useful requirement is that ‘sudo root’ must work without any prompt, which is default on most distros (e.g. ‘sudo bash’ brings you the root shell without asking for any password). This is important as the builder needs to switch to root and then back to user several times during the build process.
Additionally, if building with signing enabled (NO_SIGN is not set), you must adjust ``\~/.rpmmacros`` file so that it points to the GPG key used for package signing, e.g.:
.. code:: bash
%_signature gpg
%_gpg_path /home/user/.gnupg
%_gpg_name AC1BF9B3 # <-- Key ID used for signing
It is also recommended that you use an empty passphrase for the private key used for signing. Contrary to a popular belief, this doesn’t affect your key or sources security – if somebody compromised your system, then the game is over anyway, whether you have used an additional passphrase for the key or not.
So, to build Qubes you would do:
```shell
# Import the Qubes master key
gpg --recv-keys 0xDDFA1A3E36879494
.. code:: bash
# Verify its fingerprint, set as 'trusted'.
# This is described here:
# https://www.qubes-os.org/doc/VerifyingSignatures
# Import the Qubes master key
gpg --recv-keys 0xDDFA1A3E36879494
# Verify its fingerprint, set as 'trusted'.
# This is described here:
# https://www.qubes-os.org/doc/VerifyingSignatures
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
gpg --import qubes-developers-keys.asc
git clone https://github.com/QubesOS/qubes-builder.git qubes-builder
cd qubes-builder
# Verify its integrity:
git tag -v `git describe`
cp example-configs/qubes-os-master.conf builder.conf
# edit the builder.conf file and set the following variables:
# NO_SIGN="1"
# Download all components:
make get-sources
# And now to build all Qubes RPMs (this will take a few hours):
make qubes
# ... and then to build the ISO
make iso
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
gpg --import qubes-developers-keys.asc
git clone https://github.com/QubesOS/qubes-builder.git qubes-builder
cd qubes-builder
# Verify its integrity:
git tag -v `git describe`
cp example-configs/qubes-os-master.conf builder.conf
# edit the builder.conf file and set the following variables:
# NO_SIGN="1"
# Download all components:
make get-sources
# And now to build all Qubes RPMs (this will take a few hours):
make qubes
# ... and then to build the ISO
make iso
```
And this should produce a shiny new ISO.
You can also build selected component separately. Eg. to compile only gui virtualization agent/daemon:
```shell
make gui-daemon
```
.. code:: bash
make gui-daemon
You can get a full list from make help.
## Making customized build
Making customized build
-----------------------
### Manual source modification
You can also modify sources somehow if you wish.
Here are some basic steps:
Manual source modification
^^^^^^^^^^^^^^^^^^^^^^^^^^
You can also modify sources somehow if you wish. Here are some basic steps:
1. Download qubes-builder as described above (if you want to use marmarek’s branches, you should also download qubes-builder from his repo - replace ‘QubesOS’ with ‘marmarek’ in above git clone command)
1. Download qubes-builder as described above (if you want to use marmarek's branches, you should also download qubes-builder from his repo - replace 'QubesOS' with 'marmarek' in above git clone command)
2. Edit builder.conf (still the same as above), some useful additions:
- You can also set GIT\_PREFIX="marmarek/qubes-" to use marmarek's repo instead of "mainstream" - it contains newer (but less tested) versions
- You can also set GIT_PREFIX=“marmarek/qubes-” to use marmarek’s repo instead of “mainstream” - it contains newer (but less tested) versions
3. Download unmodified sources
```shell
make get-sources
```
.. code:: bash
make get-sources
4. **Make your modifications here**
5. Build the Qubes
`make qubes` actually is just meta target which builds all required
components in correct order. The list of components is configured in
builder.conf. You can also check the current value at the end of `make
help`, or using `make build-info`.
5. Build the Qubes ``make qubes`` actually is just meta target which builds all required components in correct order. The list of components is configured in builder.conf. You can also check the current value at the end of ``make help``, or using ``make build-info``.
6. `get-sources` is already done, so continue with the next one. You can skip `sign-all` if you've disabled signing
6. ``get-sources`` is already done, so continue with the next one. You can skip ``sign-all`` if you’ve disabled signing
.. code:: bash
make vmm-xen core-admin linux-kernel gui-daemon template desktop-linux-kde installer-qubes-os manager linux-dom0-updates
```shell
make vmm-xen core-admin linux-kernel gui-daemon template desktop-linux-kde installer-qubes-os manager linux-dom0-updates
```
7. build iso installation image
```shell
make iso
```
.. code:: bash
### Use pre-built Qubes packages
make iso
For building just a few selected packages, it's very useful to download pre-built qubes-specific dependencies from `{yum,deb}.qubes-os.org`.
This is especially true for `gcc`, which takes several hours to build.
Before creating the `chroot`, add this to your `builder.conf`:
```
USE_QUBES_REPO_VERSION = $(RELEASE)
```
It will add the 'current' Qubes repository to your `chroot` environment.
Next, specify which components (`gcc`, for example) you want to download instead of compiling:
```
COMPONENTS := $(filter-out gcc,$(COMPONENTS))
```
Alternatively, edit the actual COMPONENTS list which is defined in the included version-dependent config from example-configs (see series of include directives near the beginning of `builder.conf`).
This way, you can build only the packages in which you are interested.
Use pre-built Qubes packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you also want to use the 'current-testing' repository, add this to your configuration:
```
USE_QUBES_REPO_TESTING = 1
```
For building just a few selected packages, it’s very useful to download pre-built qubes-specific dependencies from ``{yum,deb}.qubes-os.org``. This is especially true for ``gcc``, which takes several hours to build.
In the case of an existing `chroot`, for mock-enabled builds, this works immediately because `chroot` is constructed each time separately.
For legacy builds, it will not add the necessary configuration into the build environment unless a specific builder change or configuration would force rebuilding chroot.
Before creating the ``chroot``, add this to your ``builder.conf``:
Also, once enabled, disabling this setting will not disable repositories in relevant chroots.
And even if it did, there could be some leftover packages installed from those repos (which may or may not be desirable).
.. code:: bash
**Note**
If you are building Ubuntu templates, you cannot use this option.
This is because Qubes does not provide official packages for Ubuntu templates.
USE_QUBES_REPO_VERSION = $(RELEASE)
## Code verification keys management
[QubesBuilder](/doc/qubes-builder/) by default verifies signed tags on every downloaded code.
Public keys used for that are stored in `keyrings/git`.
By default Qubes developers' keys are imported automatically, but if you need some additional keys (for example your own), you can add them using:
```shell
GNUPGHOME=$PWD/keyrings/git gpg --import /path/to/key.asc
GNUPGHOME=$PWD/keyrings/git gpg --edit-key ID_OF_JUST_IMPORTED_KEY
# here use "trust" command to set key fully or ultimately trusted - only those keys are accepted by QubesBuilder
```
It will add the ‘current’ Qubes repository to your ``chroot`` environment. Next, specify which components (``gcc``, for example) you want to download instead of compiling:
All Qubes developers' keys are signed by the Qubes Master Signing Key (which is set as ultimately trusted key), so are trusted automatically.
.. code:: bash
If you are the owner of Master key and want to revoke such signature, use the `revsig` gpg key edit command and update the key in qubes-developers-keys.asc - now the key will be no longer trusted (unless manually set as such).
COMPONENTS := $(filter-out gcc,$(COMPONENTS))
## Further information
For advanced [QubesBuilder](/doc/qubes-builder/) use, and preparing sources, take a look at [QubesBuilderDetails](/doc/qubes-builder-details/) page, or [QubesBuilder's doc directory](https://github.com/marmarek/qubes-builder/tree/master/doc).
Alternatively, edit the actual COMPONENTS list which is defined in the included version-dependent config from example-configs (see series of include directives near the beginning of ``builder.conf``). This way, you can build only the packages in which you are interested.
If you also want to use the ‘current-testing’ repository, add this to your configuration:
.. code:: bash
USE_QUBES_REPO_TESTING = 1
In the case of an existing ``chroot``, for mock-enabled builds, this works immediately because ``chroot`` is constructed each time separately. For legacy builds, it will not add the necessary configuration into the build environment unless a specific builder change or configuration would force rebuilding chroot.
Also, once enabled, disabling this setting will not disable repositories in relevant chroots. And even if it did, there could be some leftover packages installed from those repos (which may or may not be desirable).
**Note** If you are building Ubuntu templates, you cannot use this option. This is because Qubes does not provide official packages for Ubuntu templates.
Code verification keys management
---------------------------------
:doc:`QubesBuilder </developer/building/qubes-builder>` by default verifies signed tags on every downloaded code. Public keys used for that are stored in ``keyrings/git``. By default Qubes developers’ keys are imported automatically, but if you need some additional keys (for example your own), you can add them using:
.. code:: bash
GNUPGHOME=$PWD/keyrings/git gpg --import /path/to/key.asc
GNUPGHOME=$PWD/keyrings/git gpg --edit-key ID_OF_JUST_IMPORTED_KEY
# here use "trust" command to set key fully or ultimately trusted - only those keys are accepted by QubesBuilder
All Qubes developers’ keys are signed by the Qubes Master Signing Key (which is set as ultimately trusted key), so are trusted automatically.
If you are the owner of Master key and want to revoke such signature, use the ``revsig`` gpg key edit command and update the key in qubes-developers-keys.asc - now the key will be no longer trusted (unless manually set as such).
Further information
-------------------
For advanced :doc:`QubesBuilder </developer/building/qubes-builder>` use, and preparing sources, take a look at :doc:`QubesBuilderDetails </developer/building/qubes-builder-details>` page, or `QubesBuilder’s doc directory <https://github.com/marmarek/qubes-builder/tree/master/doc>`__.

313
developer/building/qubes-iso-building.rst
View file

@ -1,219 +1,226 @@
---
lang: en
layout: doc
permalink: /doc/qubes-iso-building/
redirect_from:
- /doc/qubes-r3-building/
- /en/doc/qubes-r3-building/
- /en/doc/qubes-iso-building/
- /doc/QubesR3Building/
- /wiki/QubesR3Building/
ref: 63
title: Qubes ISO building
---
==================
Qubes ISO building
==================
<div class="alert alert-warning" role="alert">
<i class="fa fa-exclamation-circle"></i>
<b>Note:</b> These instructions concern the older Qubes builder (v1). It supports
only building Qubes 4.2 or earlier.<br>The build process has been completely rewritten in <a href="https://github.com/QubesOS/qubes-builderv2/">qubes-builder v2</a>. This can be used for building Qubes R4.2 and later versions, and all related components.
</div>
.. warning::
**Note:** These instructions concern the older Qubes builder (v1). It supports only building Qubes 4.2 or earlier.The build process has been completely rewritten in `qubes-builder v2 <https://github.com/QubesOS/qubes-builderv2/>`__ . This can be used for building Qubes R4.2 and later versions, and all related components.
Build Environment
-----------------
Fedora 36 (and 37) has been successfully used to build Qubes R4.1 with the below steps.
Other rpm-based operating systems may also work.
Travis-CI uses Ubuntu 18.04 to perform test builds, except it can not test the `./setup` script.
**Notes:** On modern Fedora system (like Fedora 37) SeLinux is enforced by
default and is blocking the build system. You would get error like
"can't create transaction lock on /.../rpm/.rpm.lock (Permission denied)".
You can set SeLinux to permissive mode with
Fedora 36 (and 37) has been successfully used to build Qubes R4.1 with the below steps. Other rpm-based operating systems may also work. Travis-CI uses Ubuntu 18.04 to perform test builds, except it can not test the ``./setup`` script.
~~~bash
sudo setenforce 0
~~~
**Notes:** On modern Fedora system (like Fedora 37) SeLinux is enforced by default and is blocking the build system. You would get error like “can’t create transaction lock on /…/rpm/.rpm.lock (Permission denied)”. You can set SeLinux to permissive mode with
.. code:: bash
sudo setenforce 0
In `dom0`, install the Fedora 36 (or 37) template if you don't already have it.
In ``dom0``, install the Fedora 36 (or 37) template if you don’t already have it.
~~~
sudo qubes-dom0-update qubes-template-fedora-36
~~~
.. code:: bash
Create a standalone AppVM from the Fedora template.
Set private storage to at least 60 GB if you will be building only the default templates; 100 GB or more if you plan on additional.
It's not required, but if you allocate additional CPU cores, the build process can utilize them at some steps such as the kernel build.
Likewise, more memory (up to 16 GB) can help.
Last, you may want to disable memory balancing, but keep in mind the impact on your other qubes.
sudo qubes-dom0-update qubes-template-fedora-36
Create a standalone AppVM from the Fedora template. Set private storage to at least 60 GB if you will be building only the default templates; 100 GB or more if you plan on additional. It’s not required, but if you allocate additional CPU cores, the build process can utilize them at some steps such as the kernel build. Likewise, more memory (up to 16 GB) can help. Last, you may want to disable memory balancing, but keep in mind the impact on your other qubes.
Once you’ve built the development AppVM, open a Terminal window to it and install the necessary dependencies (see :doc:`QubesBuilder </developer/building/qubes-builder>` for more info):
.. code:: bash
$ sudo dnf install git createrepo rpm-build rpm-sign make python3-sh rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5 perl-Digest-SHA
Once you've built the development AppVM, open a Terminal window to it and install the necessary dependencies (see [QubesBuilder](/doc/qubes-builder/) for more info):
~~~
$ sudo dnf install git createrepo rpm-build rpm-sign make python3-sh rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5 perl-Digest-SHA
~~~
Get the necessary keys to verify the sources (run these and other commands below as a regular user, not root):
~~~
wget https://keys.qubes-os.org/keys/qubes-master-signing-key.asc
gpg --import qubes-master-signing-key.asc
gpg --edit-key 36879494
fpr
# Verify fingerprint! See Note below!
# Once verified, set trust to *ultimate*
# (Typical sequence is trust, 5, Y, q)
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
gpg --import qubes-developers-keys.asc
~~~
.. code:: bash
**Note** In the above process, we do *not* rely on the security of our server (keys.qubes-os.org) nor the connection (ssl, cert) -- we only rely on you getting the Qubes Master Signing Key fingerprint *somehow* and ensuring they match!
See [verifying signatures](/security/verifying-signatures/#how-to-import-and-authenticate-the-qubes-master-signing-key) for verification sources.
wget https://keys.qubes-os.org/keys/qubes-master-signing-key.asc
gpg --import qubes-master-signing-key.asc
gpg --edit-key 36879494
fpr
# Verify fingerprint! See Note below!
# Once verified, set trust to *ultimate*
# (Typical sequence is trust, 5, Y, q)
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
gpg --import qubes-developers-keys.asc
Now let's bootstrap the builder. Unfortunately, the builder cannot verify itself (the classic Chicken and Egg problem), so we need to verify the signature manually:
~~~
git clone https://github.com/QubesOS/qubes-builder.git
cd qubes-builder
git tag -v `git describe`
~~~
**Note** It's very important to check if the verification message contains "Good signature from ..." and does not contain "WARNING: This key is not certified with a trusted signature!".
**Note** In the above process, we do *not* rely on the security of our server (keys.qubes-os.org) nor the connection (ssl, cert) – we only rely on you getting the Qubes Master Signing Key fingerprint *somehow* and ensuring they match! See :ref:`verifying signatures <project-security/verifying-signatures:how to import and authenticate the qubes master signing key>` for verification sources.
Assuming the verification went fine, we're good to go with all the rest without ever thinking more about verifying digital signatures on all the rest of the components, apart from an additional step if doing a non-scripted build.
The builder will do that for us for each component, every time we build, even for all auxiliary files (e.g. Xen or Linux kernel sources).
Now let’s bootstrap the builder. Unfortunately, the builder cannot verify itself (the classic Chicken and Egg problem), so we need to verify the signature manually:
.. code:: bash
git clone https://github.com/QubesOS/qubes-builder.git
cd qubes-builder
git tag -v `git describe`
**Note** It’s very important to check if the verification message contains “Good signature from …” and does not contain “WARNING: This key is not certified with a trusted signature!”.
Assuming the verification went fine, we’re good to go with all the rest without ever thinking more about verifying digital signatures on all the rest of the components, apart from an additional step if doing a non-scripted build. The builder will do that for us for each component, every time we build, even for all auxiliary files (e.g. Xen or Linux kernel sources).
Build using setup script
-----------------
------------------------
Let's configure the builder first (see [procedure](/doc/qubes-iso-building/#build-using-manual-steps) at bottom if you would prefer to manually configure):
~~~
cd ~/qubes-builder
./setup
# Select Yes to add Qubes Master Signing Key
# Select Yes to add Qubes OS Signing Key
# Select 4.1 for version
# Stable
# Select Yes for fast Git cloning
# Select Current (if you want the option to use pre-built packages)
# Select No (we want a full build)
# Select fc36 and bullseye (for the currently shipping templates)
# Select builder-rpm, builder-debian, template-whonix, mgmt-salt
# Select Yes to add adrelanos's third party key
# Select Yes (to download)
~~~
Let’s configure the builder first (see :ref:`procedure <developer/building/qubes-iso-building:build using manual steps>` at bottom if you would prefer to manually configure):
.. code:: bash
cd ~/qubes-builder
./setup
# Select Yes to add Qubes Master Signing Key
# Select Yes to add Qubes OS Signing Key
# Select 4.1 for version
# Stable
# Select Yes for fast Git cloning
# Select Current (if you want the option to use pre-built packages)
# Select No (we want a full build)
# Select fc36 and bullseye (for the currently shipping templates)
# Select builder-rpm, builder-debian, template-whonix, mgmt-salt
# Select Yes to add adrelanos's third party key
# Select Yes (to download)
Once it completes downloading, re-run ``setup`` to add the Whonix templates:
.. code:: bash
./setup
# Choose the same options as above, except at templates select:
# fc36, bullseye, whonix-gateway-16, whonix-workstation-16
Once it completes downloading, re-run `setup` to add the Whonix templates:
~~~
./setup
# Choose the same options as above, except at templates select:
# fc36, bullseye, whonix-gateway-16, whonix-workstation-16
~~~
Continue the build process with:
~~~
make install-deps
make get-sources
~~~
.. code:: bash
When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable in `builder.conf` at this stage to specify the currently shipping Tor Browser version.
See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/).
make install-deps
make get-sources
You may also want to add `COMPONENTS := $(filter-out gcc,$(COMPONENTS))` to bypass a multiple hour compile step.
See [QubesBuilder](/doc/qubes-builder/#use-pre-built-qubes-packages) for more detail.
When building the Whonix templates, you will often need to add/update the ``WHONIX_TBB_VERSION`` variable in ``builder.conf`` at this stage to specify the currently shipping Tor Browser version. See the related note under `Extra Whonix Build Options <https://forum.qubes-os.org/t/18981>`__.
You may also want to add ``COMPONENTS := $(filter-out gcc,$(COMPONENTS))`` to bypass a multiple hour compile step. See :ref:`QubesBuilder <developer/building/qubes-builder:use pre-built qubes packages>` for more detail.
Finally, if you are making a test build, use:
~~~
make qubes
make iso
~~~
.. code:: bash
make qubes
make iso
Or for a fully signed build (this requires setting ``SIGN_KEY`` in ``builder.conf``):
.. code:: bash
make qubes
make sign-all
make iso
Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`):
~~~
make qubes
make sign-all
make iso
~~~
Enjoy your new ISO!
Build using manual steps
-----------------
------------------------
Instead of using `./setup`, you can manually configure the build.
The script takes care of a lot of the keyring preparation for us, so we first need to set that up.
Instead of using ``./setup``, you can manually configure the build. The script takes care of a lot of the keyring preparation for us, so we first need to set that up.
If you will be building Whonix templates:
~~~
cd ~
gpg --keyserver pgp.mit.edu --recv-keys 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
gpg --fingerprint 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
~~~
.. code:: bash
**Note:** It's very important to check the fingerprint displayed against multiple sources such as the [Whonix web site](https://www.whonix.org/wiki/Whonix_Signing_Key), etc.
It should look something like this:
cd ~
gpg --keyserver pgp.mit.edu --recv-keys 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
gpg --fingerprint 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
**Note:** It’s very important to check the fingerprint displayed against multiple sources such as the `Whonix web site <https://www.whonix.org/wiki/Whonix_Signing_Key>`__, etc. It should look something like this:
.. code:: bash
pub rsa4096 2014-01-16 [SC] [expires: 2026-01-23]
Key fingerprint = 916B 8D99 C38E AF5E 8ADC 7A2A 8D66 066A 2EEA CCDA
uid [ unknown] Patrick Schleizer <adrelanos@kicksecure.com>
uid [ unknown] Patrick Schleizer <adrelanos@riseup.net>
uid [ unknown] Patrick Schleizer <adrelanos@whonix.org>
sub rsa4096 2014-01-16 [E] [expires: 2026-01-23]
sub rsa4096 2014-01-16 [A] [expires: 2026-01-23]
sub rsa4096 2014-01-16 [S] [expires: 2026-01-23]
~~~
pub rsa4096 2014-01-16 [SC] [expires: 2026-01-23]
Key fingerprint = 916B 8D99 C38E AF5E 8ADC 7A2A 8D66 066A 2EEA CCDA
uid [ unknown] Patrick Schleizer <adrelanos@kicksecure.com>
uid [ unknown] Patrick Schleizer <adrelanos@riseup.net>
uid [ unknown] Patrick Schleizer <adrelanos@whonix.org>
sub rsa4096 2014-01-16 [E] [expires: 2026-01-23]
sub rsa4096 2014-01-16 [A] [expires: 2026-01-23]
sub rsa4096 2014-01-16 [S] [expires: 2026-01-23]
~~~
Next, prepare the Git keyring directory and copy them in:
~~~
export GNUPGHOME=~/qubes-builder/keyrings/git
mkdir --parents "$GNUPGHOME"
cp ~/.gnupg/pubring.gpg "$GNUPGHOME"
cp ~/.gnupg/trustdb.gpg "$GNUPGHOME"
chmod --recursive 700 "$GNUPGHOME"
~~~
.. code:: bash
export GNUPGHOME=~/qubes-builder/keyrings/git
mkdir --parents "$GNUPGHOME"
cp ~/.gnupg/pubring.gpg "$GNUPGHOME"
cp ~/.gnupg/trustdb.gpg "$GNUPGHOME"
chmod --recursive 700 "$GNUPGHOME"
Copy one of the example configurations:
~~~
cd ~/qubes-builder
cp example-configs/qubes-os-master.conf builder.conf
~~~
.. code:: bash
Edit `builder.conf`, referring to `doc/Configuration.md` for a description of all available options.
cd ~/qubes-builder
cp example-configs/qubes-os-master.conf builder.conf
Edit ``builder.conf``, referring to ``doc/Configuration.md`` for a description of all available options.
Continue the build process with:
~~~
make install-deps
make get-sources
unset GNUPGHOME
~~~
.. code:: bash
When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable at this stage to specify the currently shipping Tor Browser version.
See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/).
make install-deps
make get-sources
unset GNUPGHOME
When building the Whonix templates, you will often need to add/update the ``WHONIX_TBB_VERSION`` variable at this stage to specify the currently shipping Tor Browser version. See the related note under `Extra Whonix Build Options <https://forum.qubes-os.org/t/18981>`__.
Finally, if you are making a test build, use:
~~~
make qubes
make iso
~~~
.. code:: bash
make qubes
make iso
Or for a fully signed build (this requires setting ``SIGN_KEY`` in ``builder.conf``):
.. code:: bash
make qubes
make sign-all
make iso
Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`):
~~~
make qubes
make sign-all
make iso
~~~
Enjoy your new ISO!

8
developer/building/qubes-template-configs.md
View file

@ -1,8 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/qubes-template-configs/
redirect_to: https://github.com/QubesOS/qubes-template-configs
ref: 248
title: Qubes template configs
---

292
developer/code/code-signing.rst
View file

@ -1,199 +1,209 @@
---
lang: en
layout: doc
permalink: /doc/code-signing/
ref: 51
title: Code signing
---
============
Code signing
============
All contributions to the Qubes OS [source code](/doc/source-code/) must be cryptographically signed by the author's PGP key.
## Generating a Key
All contributions to the Qubes OS :doc:`source code </developer/code/source-code>` must be cryptographically signed by the author’s PGP key.
(Note: If you already have a PGP key, you may skip this step.)
Generating a Key
----------------
Alex Cabal has written an excellent [guide](https://alexcabal.com/creating-the-perfect-gpg-keypair/) on creating a PGP keypair.
Below, we reproduce just the minimum steps in generating a keypair using GnuPG.
Please read Cabal's full guide for further important details.
~~~
$ gpg --gen-key
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
(**Note:** If you already have a PGP key, you may skip this step.)
gpg: directory '/home/user/.gnupg' created
gpg: keybox '/home/user/.gnupg/pubring.kbx' created
Note: Use "gpg --full-generate-key" for a full featured key generation dialog.
Alex Cabal has written an excellent `guide <https://alexcabal.com/creating-the-perfect-gpg-keypair/>`__ on creating a PGP keypair. Below, we reproduce just the minimum steps in generating a keypair using GnuPG. Please read Cabal’s full guide for further important details.
GnuPG needs to construct a user ID to identify your key.
.. code:: bash
Real name: Bilbo Baggins
Email address: bilbo@shire.org
You selected this USER-ID:
"Bilbo Baggins <bilbo@shire.org>"
$ gpg --gen-key
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
gpg: directory '/home/user/.gnupg' created
gpg: keybox '/home/user/.gnupg/pubring.kbx' created
**Note:** Use "gpg --full-generate-key" for a full featured key generation dialog.
GnuPG needs to construct a user ID to identify your key.
Real name: Bilbo Baggins
Email address: bilbo@shire.org
You selected this USER-ID:
"Bilbo Baggins <bilbo@shire.org>"
Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
<type your passphrase>
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /home/user/.gnupg/trustdb.gpg: trustdb created
gpg: key 6E2F4E7AF50A5827 marked as ultimately trusted
gpg: directory '/home/user/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/87975838063F97A968D503266E2F4E7AF50A5827.rev'
public and secret key created and signed.
pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30]
87975838063F97A968D503266E2F4E7AF50A5827
uid Bilbo Baggins <bilbo@shire.org>
sub rsa3072 2021-12-30 [E] [expires: 2023-12-30]
Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
<type your passphrase>
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /home/user/.gnupg/trustdb.gpg: trustdb created
gpg: key 6E2F4E7AF50A5827 marked as ultimately trusted
gpg: directory '/home/user/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/user/.gnupg/openpgp-revocs.d/87975838063F97A968D503266E2F4E7AF50A5827.rev'
public and secret key created and signed.
Upload the Key
--------------
pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30]
87975838063F97A968D503266E2F4E7AF50A5827
uid Bilbo Baggins <bilbo@shire.org>
sub rsa3072 2021-12-30 [E] [expires: 2023-12-30]
~~~
## Upload the Key
For others to find the public key, please upload it to a server.
Currently, [these](https://github.com/marmarek/signature-checker/blob/master/check-git-signature#L133-L135) are the recognized servers.
Currently, `these <https://github.com/marmarek/signature-checker/blob/master/check-git-signature#L133-L135>`__ are the recognized servers.
In the example below, we will use `keyserver.ubuntu.com`.
In the example below, we will use ``keyserver.ubuntu.com``.
Replace 6E2F4E7AF50A5827 with your key ID, preferably the **long keyID**
which is the last 16 hex digits of the long number in the second line
of the output above:
```
pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30]
87975838063F97A968D503266E2F4E7AF50A5827
```
Replace 6E2F4E7AF50A5827 with your key ID, preferably the **long keyID** which is the last 16 hex digits of the long number in the second line of the output above:
```shell_session
$ gpg --send-keys --keyserver hkps://keyserver.ubuntu.com 6E2F4E7AF50A5827
gpg: sending key 6E2F4E7AF50A5827 to hkps://keyserver.ubuntu.com
```
.. code:: bash
## Using PGP with Git
pub rsa3072 2021-12-30 [SC] [expires: 2023-12-30]
87975838063F97A968D503266E2F4E7AF50A5827
If you're submitting a patch via GitHub (or a similar Git server), please sign
your Git commits.
.. code:: bash
$ gpg --send-keys --keyserver hkps://keyserver.ubuntu.com 6E2F4E7AF50A5827
gpg: sending key 6E2F4E7AF50A5827 to hkps://keyserver.ubuntu.com
Using PGP with Git
------------------
If you’re submitting a patch via GitHub (or a similar Git server), please sign your Git commits.
1. Set up Git to use your key:
~~~
git config --global user.signingkey <KEYID>
~~~
.. code:: bash
git config --global user.signingkey <KEYID>
2. Set up Git to sign your commits with your key:
~~~
git config --global commit.gpgsign true
~~~
.. code:: bash
git config --global commit.gpgsign true
Alternatively, manually specify when a commit is to be signed:
~~~
git commit -S
~~~
.. code:: bash
3. (Optional) Create signed tags.
Signed commits are totally sufficient to contribute to Qubes OS.
However, if you have commits which are not signed and you do not want to change them,
you can create a signed tag for the commit and push it before the check.
git commit -S
This is useful for example, if you have a commit back in the git history which
you like to sign now without rewriting the history.
~~~
git tag -s <tag_name> -m "<tag_message>"
~~~
You can also create an alias to make this easier.
Edit your `~/.gitconfig` file.
In the `[alias]` section, add `stag` to create signed tags and `spush` to create signed tags and push them.
3. (Optional) Create signed tags. Signed commits are totally sufficient to contribute to Qubes OS. However, if you have commits which are not signed and you do not want to change them, you can create a signed tag for the commit and push it before the check.
This is useful for example, if you have a commit back in the git history which you like to sign now without rewriting the history.
~~~
[alias]
stag = "!bash -c 'id=\"`git rev-parse --verify HEAD`\"; tag_name="signed_tag_for_${id:0:8}"; git tag -s "$tag_name" -m \"Tag for commit $id\"; echo \"$tag_name\"'"
spush = "!bash -c 'git push origin `git stag`'"
~~~
.. code:: bash
You may also find it convenient to have an alias for verifying the tag on the
latest commit:
git tag -s <tag_name> -m "<tag_message>"
~~~
vtag = !git tag -v `git describe`
~~~
## GitHub Signature Verification (optional)
You can also create an alias to make this easier. Edit your ``~/.gitconfig`` file. In the ``[alias]`` section, add ``stag`` to create signed tags and ``spush`` to create signed tags and push them.
GitHub shows a green `Verified` label indicating that the GPG signature could be
verified using any of the contributor’s GPG keys uploaded to GitHub. You can
upload your public key on GitHub by adding your public GPG key on the [New GPG
key](https://github.com/settings/gpg/new) under the [SSH GPG keys page](https://github.com/settings/keys).
.. code:: bash
## Code Signature Checks
[alias]
stag = "!bash -c 'id=\"`git rev-parse --verify HEAD`\"; tag_name="signed_tag_for_${id:0:8}"; git tag -s "$tag_name" -m \"Tag for commit $id\"; echo \"$tag_name\"'"
spush = "!bash -c 'git push origin `git stag`'"
The [signature-checker](https://github.com/marmarek/signature-checker) checks if code contributions are signed.
Although GitHub adds a little green `Verified` button next to the commit, the [signature-checker](https://github.com/marmarek/signature-checker) uses this algorithm to check if a commit is correctly signed:
1. Is the commit signed?
If the commit is not signed, you can see the message
> policy/qubesos/code-signing — No signature found
2. If the commit is signed, the key is downloaded from a GPG key server.
If you can see the following error message, please check if you have uploaded the key to a key server.
> policy/qubesos/code-signing — Unable to verify (no valid key found)
You may also find it convenient to have an alias for verifying the tag on the latest commit:
### No Signature Found
.. code:: bash
> policy/qubesos/code-signing — No signature found
vtag = !git tag -v `git describe`
GitHub Signature Verification (optional)
----------------------------------------
GitHub shows a green ``Verified`` label indicating that the GPG signature could be verified using any of the contributor’s GPG keys uploaded to GitHub. You can upload your public key on GitHub by adding your public GPG key on the `New GPG key <https://github.com/settings/gpg/new>`__ under the `SSH GPG keys page <https://github.com/settings/keys>`__.
Code Signature Checks
---------------------
The `signature-checker <https://github.com/marmarek/signature-checker>`__ checks if code contributions are signed. Although GitHub adds a little green ``Verified`` button next to the commit, the `signature-checker <https://github.com/marmarek/signature-checker>`__ uses this algorithm to check if a commit is correctly signed:
1. Is the commit signed? If the commit is not signed, you can see the message
``policy/qubesos/code-signing — No signature found``
2. If the commit is signed, the key is downloaded from a GPG key server. If you can see the following error message, please check if you have uploaded the key to a key server.
``policy/qubesos/code-signing — Unable to verify (no valid key found)``
No Signature Found
^^^^^^^^^^^^^^^^^^
``policy/qubesos/code-signing — No signature found``
In this case, you have several options to sign the commit:
1. Amend the commit and replace it with a signed commit.
You can use this command to create a new signed commit:
1. Amend the commit and replace it with a signed commit. You can use this command to create a new signed commit:
.. code:: bash
git commit --amend -S
```
git commit --amend -S
```
This also rewrites the commit so you need to push it forcefully:
```
git push -f
```
.. code:: bash
2. Create a signed tag for the unsigned commit.
If the commit is back in history and you do not want to change it,
you can create a signed tag for this commit and push the signature.
You can use the alias from above:
git push -f
```
git checkout <commit>
git spush
```
Now, the signature checker needs to re-check the signature.
Please comment on the pull request that you would like to have the signatures checked again.
### Unable To Verify
2. Create a signed tag for the unsigned commit. If the commit is back in history and you do not want to change it, you can create a signed tag for this commit and push the signature. You can use the alias from above:
> policy/qubesos/code-signing — Unable to verify (no valid key found)
.. code:: bash
This means that the [signature-checker](https://github.com/marmarek/signature-checker) has found a signature for the commit
but is not able to verify it using the any key available.
This might be that you forgot to upload the key to a key server.
Please upload it.
git checkout <commit>
git spush
## Using PGP with Email
If you're submitting a patch by emailing the [developer mailing list](/support/#qubes-devel), simply sign your email with your PGP key.
One good way to do this is with a program like [Enigmail](https://www.enigmail.net/).
Enigmail is a security addon for the Mozilla Thunderbird email client that allows you to easily digitally encrypt and sign your emails.
Now, the signature checker needs to re-check the signature. Please comment on the pull request that you would like to have the signatures checked again.
Unable To Verify
^^^^^^^^^^^^^^^^
``policy/qubesos/code-signing — Unable to verify (no valid key found)``
This means that the `signature-checker <https://github.com/marmarek/signature-checker>`__ has found a signature for the commit but is not able to verify it using the any key available. This might be that you forgot to upload the key to a key server. Please upload it.
Using PGP with Email
--------------------
If you’re submitting a patch by emailing the :ref:`developer mailing list <introduction/support:qubes-devel>`, simply sign your email with your PGP key. One good way to do this is with a program like `Enigmail <https://www.enigmail.net/>`__. Enigmail is a security addon for the Mozilla Thunderbird email client that allows you to easily digitally encrypt and sign your emails.

257
developer/code/coding-style.rst
View file

@ -1,185 +1,262 @@
---
lang: en
layout: doc
permalink: /doc/coding-style/
redirect_from:
- /en/doc/coding-style/
- /doc/CodingStyle/
- /wiki/CodingStyle/
- /trac/wiki/CodingStyle/
ref: 53
title: Coding style
---
============
Coding style
============
Rationale
---------
Maintaining proper coding style is very important for any large software project, such as Qubes. Here's why:
Maintaining proper coding style is very important for any large software project, such as Qubes. Here’s why:
- It eases maintenance tasks, such as adding new functionality or generalizing code later,
- It allows others (as well as the future you!) to easily understand fragments of code and what they were supposed to do, and thus makes it easier to later extend them with newer functionality or bug fixes,
- It allows others to easily review the code and catch various bugs,
- It provides for an aesthetically pleasing experience when one reads the code...
Often, developers, usually smart ones, undersell the value of proper coding style, thinking that it's much more important how their code works. These developers expect that if their code solves some problem using a nice and neat trick, then that's all that is really required. Such thinking shows, however, immaturity and is a signal that the developer, no matter how bright and smart, might not be a good fit for larger projects. Writing a clever exploit for a Black Hat show is one thing - writing useful software supposed to be used and maintained for years is quite a different story. If you want to show off what a smart programmer you are, then you should become a researcher and write exploits. If, on the other hand, you want to be part of a team that makes real, useful software, you should ensure your coding style is impeccable. At Qubes project, we often took shortcuts and wrote nasty code, and this has always back fired at us, sometime months, sometime years later, the net result being we had to spend time fixing code, rather than implementing new functionality.
- It provides for an aesthetically pleasing experience when one reads the code…
And here's a [link to the real case](https://groups.google.com/forum/#!msg/qubes-devel/XgTo6L8-5XA/JLOadvBqnqMJ) (one Qubes Security Bulletin) demonstrating how the lackadaisical coding style lead to a real security bug. Never assume you're smart enough to disregard clean and rigorous coding!
Often, developers, usually smart ones, undersell the value of proper coding style, thinking that it’s much more important how their code works. These developers expect that if their code solves some problem using a nice and neat trick, then that’s all that is really required. Such thinking shows, however, immaturity and is a signal that the developer, no matter how bright and smart, might not be a good fit for larger projects. Writing a clever exploit for a Black Hat show is one thing - writing useful software supposed to be used and maintained for years is quite a different story. If you want to show off what a smart programmer you are, then you should become a researcher and write exploits. If, on the other hand, you want to be part of a team that makes real, useful software, you should ensure your coding style is impeccable. At Qubes project, we often took shortcuts and wrote nasty code, and this has always back fired at us, sometime months, sometime years later, the net result being we had to spend time fixing code, rather than implementing new functionality.
And here’s a `link to the real case <https://groups.google.com/forum/#!msg/qubes-devel/XgTo6L8-5XA/JLOadvBqnqMJ>`__ (one Qubes Security Bulletin) demonstrating how the lackadaisical coding style lead to a real security bug. Never assume you’re smart enough to disregard clean and rigorous coding!
General typographic conventions
-------------------------------
- **Use space-expanded tabs that equal 4 spaces.** Yes, we know, there are many arguments for using "real" tabs instead of space-expanded tabs, but we need to pick one convention to make the project consistent. One argument for using space-expanded tabs is that this way the programmer is in control of how the code will look like, despite how other users have configured their editors to visualize the tabs (of course, we assume any sane person uses a fixed-width font for viewing the source code). If it makes you feel any better, assume this is just an arbitrary choice made to enforce a unified style.
- **Maintain max. line length of 80 characters**. Even though today's monitors often are very wide and it's often not a problem to have 120 characters displayed in an editor, maintaining shorter line lengths improves readability. It also allows others to have two parallel windows open, side by side, each with different parts of the source code.
- **Use space-expanded tabs that equal 4 spaces.** Yes, we know, there are many arguments for using “real” tabs instead of space-expanded tabs, but we need to pick one convention to make the project consistent. One argument for using space-expanded tabs is that this way the programmer is in control of how the code will look like, despite how other users have configured their editors to visualize the tabs (of course, we assume any sane person uses a fixed-width font for viewing the source code). If it makes you feel any better, assume this is just an arbitrary choice made to enforce a unified style.
- **Naming conventions for any OS *other than Windows***:
- `ClassName`
- `some_variable`, `some_function`, `some_argument`
- **Maintain max. line length of 80 characters**. Even though today’s monitors often are very wide and it’s often not a problem to have 120 characters displayed in an editor, maintaining shorter line lengths improves readability. It also allows others to have two parallel windows open, side by side, each with different parts of the source code.
- **Naming convention *for Windows OS*** -- exceptionally to preserve Windows conventions please use the following:
- `ClassName`, `FunctionName`
- `pszArgumentOne`, `hPipe` -- use Hungarian notation for argument and variables
- **Naming conventions for any OS other than Windows**:
- **Maintain a decent amount of horizontal spacing**, e.g. add a space after `if` or before `{` in C, and similar in other languages. Whether and where to also use spaces within expressions, such as (x\*2+5) vs. (x \* 2 + 5) is left to the developer's judgment. Do not put spaces immediately after or before the brackets in expressions, so avoid constructs like this: `if ( condition )` and use ones like this: `if (condition)` instead.
- ``ClassName``
- **Use single new lines** ('\\n' aka LF) in any non-Windows source code. On Windows, exceptionally, use the CRLF line endings (--). This will allow the source code to be easily viewable in various Windows-based programs.
- ``some_variable``, ``some_function``, ``some_argument``
- **Naming convention for Windows OS** – exceptionally to preserve Windows conventions please use the following:
- ``ClassName``, ``FunctionName``
- ``pszArgumentOne``, ``hPipe`` – use Hungarian notation for argument and variables
- **Maintain a decent amount of horizontal spacing**, e.g. add a space after ``if`` or before ``{`` in C, and similar in other languages. Whether and where to also use spaces within expressions, such as (x*2+5) vs. (x * 2 + 5) is left to the developer’s judgment. Do not put spaces immediately after or before the brackets in expressions, so avoid constructs like this: ``if ( condition )`` and use ones like this: ``if (condition)`` instead.
- **Use single new lines** (‘\n’ aka LF) in any non-Windows source code. On Windows, exceptionally, use the CRLF line endings (–). This will allow the source code to be easily viewable in various Windows-based programs.
- **Use descriptive names for variables and functions**! Really, at a time when most editors have auto-completion features, there is no excuse for using short variable names.
- **Comments should be indented together with the code**, e.g. like this:
- **Comments should be indented together with the code**, e.g. like this:
~~~
for (...) {
// The following code finds PGP private key matching the given public key in O(1)
while (key_found) {
(...)
.. code:: c
for (...) {
// The following code finds PGP private key matching the given public key in O(1)
while (key_found) {
(...)
}
}
}
~~~
File naming conventions
-----------------------
- All file names written with small letters, use dash to separate words, rather than underscores, e.g. `qubes-dom0-update`. Never use spaces!
- All file names written with small letters, use dash to separate words, rather than underscores, e.g. ``qubes-dom0-update``. Never use spaces!
**File naming in Linux/Unix-like systems:**
- User commands that operate on particular VMs (also those accessible in VMs): `/usr/bin/qvm-*`
- User commands that apply to the whole system (Dom0 only): `/usr/bin/qubes-*`
- Auxiliary executables and scripts in `/usr/libexec/qubes/` (Note: previously we used `/usr/lib/qubes` but decided to change that)
- Helper, non-executable files in `/usr/share/qubes/`
- Various config files in `/etc/qubes`
- Qubes RPC services in `/etc/qubes-rpc`. Qubes RPC Policy definitions (only in Dom0) in `/etc/qubes-rpc/policy/`
- All VM-related configs, images, and other files in `/var/lib/qubes/`
- System-wide temporary files which reflect the current state of system in `/var/run/qubes`
- Logs: either log to the system-wide messages, or to `/var/log/qubes/`
- User commands that operate on particular VMs (also those accessible in VMs): ``/usr/bin/qvm-*``
- User commands that apply to the whole system (Dom0 only): ``/usr/bin/qubes-*``
- Auxiliary executables and scripts in ``/usr/libexec/qubes/`` (**Note:** previously we used ``/usr/lib/qubes`` but decided to change that)
- Helper, non-executable files in ``/usr/share/qubes/``
- Various config files in ``/etc/qubes``
- Qubes RPC services in ``/etc/qubes-rpc``. Qubes RPC Policy definitions (only in Dom0) in ``/etc/qubes-rpc/policy/``
- All VM-related configs, images, and other files in ``/var/lib/qubes/``
- System-wide temporary files which reflect the current state of system in ``/var/run/qubes``
- Logs: either log to the system-wide messages, or to ``/var/log/qubes/``
**File naming in Windows systems:**
- All base qubes-related files in `C:\Program Files\Invisible Things Lab\Qubes\` (Exceptionally spaces are allowed here to adhere to Windows naming conventions)
- Other, third-party files, not Qubes-specific, such as e.g. Xen PV drivers might be in different vendor subdirs, e.g. `C:\Program Files\Xen PV Drivers`
- All base qubes-related files in ``C:\Program Files\Invisible Things Lab\Qubes\`` (Exceptionally spaces are allowed here to adhere to Windows naming conventions)
- Other, third-party files, not Qubes-specific, such as e.g. Xen PV drivers might be in different vendor subdirs, e.g. ``C:\Program Files\Xen PV Drivers``
General programming style guidelines
------------------------------------
- Do not try to impress with your coding kung-fu, do not use tricks to save 2 lines of code, always prefer readability over trickiness!
- Make sure your code compiles and builds without warnings.
- Always think first about interfaces (e.g. function arguments, or class methods) and data structures before you start writing the actual code.
- Always think first about interfaces (e.g. function arguments, or class methods) and data structures before you start writing the actual code.
- Use comments to explain non-trivial code fragments, or expected behavior of more complex functions, if it is not clear from their name.
- Do **not** use comments for code fragments where it is immediately clear what the code does. E.g. avoid constructs like this:
~~~
// Return window id
int get_window_id (...) {
(...)
return id;
}
~~~
.. code:: c
- Do **not** use comments to disable code fragments. In production code there should really be no commented or disabled code fragments. If you really, really have a good reason to retain some fragment of unused code, use \#if or \#ifdef to disable it, e.g.:
// Return window id
int get_window_id (...) {
(...)
return id;
}
~~~
#if 0
(...) // Some unused code here
#endif
~~~
... and preferably use some descriptive macro instead of just `0`, e.g.:
~~~
#if USE_OLD_WINDOW_TRAVERSING
(...) // Some unused code here
#endif
~~~
- Do **not** use comments to disable code fragments. In production code there should really be no commented or disabled code fragments. If you really, really have a good reason to retain some fragment of unused code, use #if or #ifdef to disable it, e.g.:
Not sure how to do similar thing in Python... Anyone?
.. code:: c
#if 0
(...) // Some unused code here
#endif
… and preferably use some descriptive macro instead of just ``0``, e.g.:
.. code:: c
#if USE_OLD_WINDOW_TRAVERSING
(...) // Some unused code here
#endif
Not sure how to do similar thing in Python… Anyone?
But generally, there is little excuse to keep old, unused code fragments in the code. One should really use the functionality provided by the source code management system, such as git, instead. E.g. create a special branch for storing the old, unused code – this way you will always be able to merge this code into upstream in the future.
- Do not hardcode values in the code! The only three numbers that are an exception here and for which it is acceptable to hardcode them are: ``0``, ``1`` and ``-1``, and frankly the last two are still controversial…
> But generally, there is little excuse to keep old, unused code fragments in the code. One should really use the functionality provided by the source code management system, such as git, instead. E.g. create a special branch for storing the old, unused code -- this way you will always be able to merge this code into upstream in the future.
- Do not hardcode values in the code! The only three numbers that are an exception here and for which it is acceptable to hardcode them are: `0`, `1` and `-1`, and frankly the last two are still controversial...
Source Code management (Git) guidelines
---------------------------------------
- Use git to maintain all code for Qubes project.
- Before you start using git, make sure you understand that git is a decentralized Source Code Management system, and that it doesn't behave like traditional, centralized source code management systems, such as SVN. Here's a good [introductory book on git](https://git-scm.com/book). Read it.
- Before you start using git, make sure you understand that git is a decentralized Source Code Management system, and that it doesn’t behave like traditional, centralized source code management systems, such as SVN. Here’s a good `introductory book on git <https://git-scm.com/book>`__. Read it.
- Qubes code is divided into many git repositories. There are several reasons for that:
- This creates natural boundaries between different code blocks, enforcing proper interfaces, and easing independent development to be conducted on various code parts at the same time, without the fear of running into conflicts.
- By maintaining relatively small git repositories, it is easy for new developers to understand the code and contribute new patches, without the need to understand all the other code.
- Code repositories represent also licensing boundaries. So, e.g. because `core-agent-linux` and `core-agent-windows` are maintained in two different repositories, it is possible to have the latter under a proprietary, non-GPL license, while keeping the former fully open source.
- We have drastically changed the layout and naming of the code repositories shortly after Qubes OS R2 Beta 2 release. For details on the current code layout, please read [this article](https://blog.invisiblethings.org/2013/03/21/introducing-qubes-odyssey-framework.html).
- Code repositories represent also licensing boundaries. So, e.g. because ``core-agent-linux`` and ``core-agent-windows`` are maintained in two different repositories, it is possible to have the latter under a proprietary, non-GPL license, while keeping the former fully open source.
- We have drastically changed the layout and naming of the code repositories shortly after Qubes OS R2 Beta 2 release. For details on the current code layout, please read `this article <https://blog.invisiblethings.org/2013/03/21/introducing-qubes-odyssey-framework.html>`__.
Commit message guidelines
-------------------------
Please attempt to follow these conventions when writing your Git commit messages:
- Separate the subject line from the body with a blank line.
- Limit the subject line to approximately 50 characters.
- Capitalize the subject line.
- Do not end the subject line with a period.
- Use the imperative mood in the subject line.
- Wrap the body at 72 characters.
- Use the body to explain *what* and *why* rather than *how*.
For details, examples, and the rationale behind each of these conventions, please see [this blog post](https://chris.beams.io/posts/git-commit/), which is the source of this list.
For details, examples, and the rationale behind each of these conventions, please see `this blog post <https://chris.beams.io/posts/git-commit/>`__, which is the source of this list.
Security coding guidelines
--------------------------
- As a general rule: **untrusted input** is our \#1 enemy!
- As a general rule: **untrusted input** is our #1 enemy!
- Any input that comes from untrusted, or less trusted, or just differently-trusted, entity should always be considered as malicious and should always be sanitized and verified. So, if your software runs in Dom0 and processes some input from any of the VMs, this input should be considered to be malicious. Even if your software runs in a VM, and processes input from some other VM, you should also assume that the input is malicious and verify it.
- Use `untrusted_` prefix for all variables that hold values read from untrusted party and which have not yet been verified to be decent, e.g.:
~~~
read_struct(untrusted_conf);
/* sanitize start */
if (untrusted_conf.width > MAX_WINDOW_WIDTH)
untrusted_conf.width = MAX_WINDOW_WIDTH;
if (untrusted_conf.height > MAX_WINDOW_HEIGHT)
untrusted_conf.height = MAX_WINDOW_HEIGHT;
width = untrusted_conf.width;
height = untrusted_conf.height;
~~~
- Use ``untrusted_`` prefix for all variables that hold values read from untrusted party and which have not yet been verified to be decent, e.g.:
.. code:: c
read_struct(untrusted_conf);
/* sanitize start */
if (untrusted_conf.width > MAX_WINDOW_WIDTH)
untrusted_conf.width = MAX_WINDOW_WIDTH;
if (untrusted_conf.height > MAX_WINDOW_HEIGHT)
untrusted_conf.height = MAX_WINDOW_HEIGHT;
width = untrusted_conf.width;
height = untrusted_conf.height;
- Use others variables, without the ``untrusted_`` prefix to hold the sanitized values, as shown above.
- Use others variables, without the `untrusted_` prefix to hold the sanitized values, as shown above.
Python-specific guidelines
--------------------------
- Please follow the guidelines [here](https://peps.python.org/pep-0008/), unless they were in conflict with what is written on this page.
- Please follow the guidelines `here <https://peps.python.org/pep-0008/>`__, unless they were in conflict with what is written on this page.
C and C++ specific guidelines
-----------------------------
- Do not place code in `*.h` files.
- Use `const` whenever possible, e.g. in function arguments passed via pointers.
- Do not mix procedural and objective code together -- if you write in C++, use classes and objects.
- Do not place code in ``*.h`` files.
- Use ``const`` whenever possible, e.g. in function arguments passed via pointers.
- Do not mix procedural and objective code together – if you write in C++, use classes and objects.
- Think about classes hierarchy, before starting to implement specific methods.
Bash-specific guidelines
------------------------
- Avoid writing scripts in bash whenever possible. Use python instead. Bash-scripts are Unix-specific and will not work under Windows VMs, or in Windows admin domain, or Windows gui domain.

28
developer/code/license.rst
View file

@ -1,24 +1,10 @@
---
lang: en
layout: doc
permalink: /doc/license/
redirect_from:
- /en/doc/license/
- /doc/QubesLicensing/
- /wiki/QubesLicensing/
ref: 52
title: Software license
---
================
Software license
================
Qubes OS is a compilation of software packages, each under its own license. The
compilation is made available under the GNU General Public License version 2
(GPLv2). However, the license for this compilation does not supersede the
license of any package included in the compilation.
The source code of Qubes OS is contained in repositories under the
[@QubesOS](https://github.com/QubesOS) account on GitHub. This source code is
made available under GPLv2, unless there is a `LICENSE` file in the root of the
containing repository that specifies a different license.
Qubes OS is a compilation of software packages, each under its own license. The compilation is made available under the GNU General Public License version 2 (GPLv2). However, the license for this compilation does not supersede the license of any package included in the compilation.
The full text of the GPLv2 license can be found
[here](https://www.gnu.org/licenses/gpl-2.0.html).
The source code of Qubes OS is contained in repositories under the `@QubesOS <https://github.com/QubesOS>`__ account on GitHub. This source code is made available under GPLv2, unless there is a ``LICENSE`` file in the root of the containing repository that specifies a different license.
The full text of the GPLv2 license can be found `here <https://www.gnu.org/licenses/gpl-2.0.html>`__.

109
developer/code/source-code.rst
View file

@ -1,83 +1,80 @@
---
lang: en
layout: doc
permalink: /doc/source-code/
redirect_from:
- /en/doc/source-code/
- /doc/SourceCode/
- /wiki/SourceCode/
ref: 54
title: Source code
---
===========
Source code
===========
All the Qubes code is kept in Git repositories. We have divided the project into
several components, each of which has its own separate repository, for example:
* `core-admin.git` -- The core Qubes infrastructure, responsible for VM
management, VM templates, fs sharing, etc.
* `gui-daemon.git` -- GUI virtualization, Dom0 side.
* `gui-agent-linux.git` -- GUI virtualization, Linux VM side.
* `linux-template-builder.git` -- Scripts and other files used to create Qubes
template images.
All the Qubes code is kept in Git repositories. We have divided the project into several components, each of which has its own separate repository, for example:
All of our repositories are available under the [QubesOS GitHub account](https://github.com/QubesOS/).
- ``core-admin.git`` – The core Qubes infrastructure, responsible for VM management, VM templates, fs sharing, etc.
- ``gui-daemon.git`` – GUI virtualization, Dom0 side.
- ``gui-agent-linux.git`` – GUI virtualization, Linux VM side.
- ``linux-template-builder.git`` – Scripts and other files used to create Qubes template images.
All of our repositories are available under the `QubesOS GitHub account <https://github.com/QubesOS/>`__.
To clone a repository:
~~~
git clone https://github.com/QubesOS/qubes-<repo_name>.git <repo_name>
~~~
.. code:: bash
git clone https://github.com/QubesOS/qubes-<repo_name>.git <repo_name>
e.g.:
~~~
git clone https://github.com/QubesOS/qubes-core-admin.git core-admin
~~~
.. code:: bash
To build Qubes you do not need to download all these repositories.
If you use [qubes builder](/doc/QubesBuilder/) you can specify *what* you want to build, and download only the repositories needed to build that target.
git clone https://github.com/QubesOS/qubes-core-admin.git core-admin
To build Qubes you do not need to download all these repositories. If you use :doc:`qubes builder </developer/building/qubes-builder>` you can specify *what* you want to build, and download only the repositories needed to build that target.
If you really do want to clone **all** of the repositories, you can use these commands:
~~~
curl "https://api.github.com/orgs/QubesOS/repos?page=1&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone
curl "https://api.github.com/orgs/QubesOS/repos?page=2&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone
~~~
.. code:: bash
curl "https://api.github.com/orgs/QubesOS/repos?page=1&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone
curl "https://api.github.com/orgs/QubesOS/repos?page=2&per_page=100" | grep -e 'clone_url*' | cut -d \" -f 4 | xargs -L1 git clone
To update (git fetch) **all** of these repositories :
~~~
find . -mindepth 1 -maxdepth 1 -type d -exec git -C {} fetch --tags --recurse-submodules=on-demand --all \;
~~~
.. code:: bash
find . -mindepth 1 -maxdepth 1 -type d -exec git -C {} fetch --tags --recurse-submodules=on-demand --all \;
(Alternatively, you can pull instead of just fetching.)
How to Send Patches
-------------------
If you want to [contribute code](/doc/contributing/#contributing-code) to the project, there are two ways. Whichever
method you choose, you must [sign your code](/doc/code-signing/) before it can be accepted.
* **Preferred**: Use GitHub's [fork & pull requests](https://guides.github.com/activities/forking/).
If you want to :ref:`contribute code <introduction/contributing:contributing code>` to the project, there are two ways. Whichever method you choose, you must :doc:`sign your code </developer/code/code-signing>` before it can be accepted.
- **Preferred**: Use GitHub’s `fork & pull requests <https://guides.github.com/activities/forking/>`__.
Opening a pull request on GitHub greatly eases the code review and tracking process. In addition, especially for bigger changes, it’s a good idea to send a message to the :ref:`qubes-devel mailing list <introduction/support:qubes-devel>` in order to notify people who do not receive GitHub notifications.
- Send a patch to the :ref:`qubes-devel mailing list <introduction/support:qubes-devel>` (``git format-patch``).
1. Make all the changes in your working directory, i.e. edit files, move them around (you can use ‘git mv’ for this), etc.
2. Add the changes and commit them (``git add``, ``git commit``). Never mix different changes into one commit! Write a good description of the commit. The first line should contain a short summary, and then, if you feel like more explanations are needed, enter an empty new line, and then start the long, detailed description (optional).
3. Test your changes NOW: check if RPMs build fine, etc.
4. Create the patch using ``git format-patch``. This has an advantage over ``git diff``, because the former will also include your commit message, your name and email, so that *your* name will be used as a commit’s author.
5. Send your patch to ``qubes-devel``. Start the message subject with ``[PATCH]``.
Opening a pull request on GitHub greatly eases the code review and tracking
process. In addition, especially for bigger changes, it's a good idea to send
a message to the [qubes-devel mailing list](/support/#qubes-devel) in order to notify people who
do not receive GitHub notifications.
* Send a patch to the [qubes-devel mailing list](/support/#qubes-devel) (`git format-patch`).
1. Make all the changes in your working directory, i.e. edit files, move them
around (you can use 'git mv' for this), etc.
2. Add the changes and commit them (`git add`, `git commit`). Never mix
different changes into one commit! Write a good description of the commit.
The first line should contain a short summary, and then, if you feel like
more explanations are needed, enter an empty new line, and then start the
long, detailed description (optional).
3. Test your changes NOW: check if RPMs build fine, etc.
4. Create the patch using `git format-patch`. This has an advantage over
`git diff`, because the former will also include your commit message, your
name and email, so that *your* name will be used as a commit's author.
5. Send your patch to `qubes-devel`. Start the message subject with
`[PATCH]`.

413
developer/debugging/automated-tests.rst
View file

@ -1,281 +1,296 @@
---
lang: en
layout: doc
permalink: /doc/automated-tests/
redirect_from:
- /en/doc/automated-tests/
- /doc/AutomatedTests/
ref: 45
title: Automated tests
---
===============
Automated tests
===============
## Unit and Integration Tests
Starting with Qubes R3 we use [python unittest](https://docs.python.org/3/library/unittest.html) to perform automatic tests of Qubes OS.
Despite the name, we use it for both [unit tests](https://en.wikipedia.org/wiki/Unit_tests) and [integration tests](https://en.wikipedia.org/wiki/Integration_tests).
The main purpose is, of course, to deliver much more stable releases.
Unit and Integration Tests
--------------------------
Starting with Qubes R3 we use `python unittest <https://docs.python.org/3/library/unittest.html>`__ to perform automatic tests of Qubes OS. Despite the name, we use it for both `unit tests <https://en.wikipedia.org/wiki/Unit_tests>`__ and `integration tests <https://en.wikipedia.org/wiki/Integration_tests>`__. The main purpose is, of course, to deliver much more stable releases.
The integration tests must be run in dom0, but some unit tests can run inside a VM as well.
### Integration & unit testing in dom0
Integration & unit testing in dom0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Integration tests are written with the assumption that they will be executed on dedicated hardware and must be run in dom0. All other unit tests can also be run in dom0.
**Do not run the tests on installations with important data, because you might lose it.**
All the VMs with a name starting with `test-` on the installation are removed during the process, and all the tests are recklessly started from dom0, even when testing (& possibly breaking) VM components.
All the VMs with a name starting with ``test-`` on the installation are removed during the process, and all the tests are recklessly started from dom0, even when testing (& possibly breaking) VM components.
First you need to build all packages that you want to test. Please do not mix branches as this will inevitably lead to failures. Then setup Qubes OS with these packages installed.
For testing you'll have to stop the `qubesd` service as the tests will use its own custom variant of the service:
`sudo systemctl stop qubesd`
For testing you’ll have to stop the ``qubesd`` service as the tests will use its own custom variant of the service: ``sudo systemctl stop qubesd``
Don't forget to start it after testing again.
Don’t forget to start it after testing again.
To start testing you can then use the standard python unittest runner:
`sudo -E python3 -m unittest -v qubes.tests`
``sudo -E python3 -m unittest -v qubes.tests``
Alternatively, use the custom Qubes OS test runner:
`sudo -E python3 -m qubes.tests.run -v`
``sudo -E python3 -m qubes.tests.run -v``
Our test runner runs mostly the same as the standard one, but it has some nice additional features like colored output and not needing the "qubes.test" prefix.
Our test runner runs mostly the same as the standard one, but it has some nice additional features like colored output and not needing the “qubes.test” prefix.
You can use `python3 -m qubes.tests.run -h` to get usage information:
You can use ``python3 -m qubes.tests.run -h`` to get usage information:
```
[user@dom0 ~]$ python3 -m qubes.tests.run -h
usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast]
[--do-not-clean] [--do-clean] [--loglevel LEVEL]
[--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg]
[TESTNAME [TESTNAME ...]]
.. code:: bash
positional arguments:
TESTNAME list of tests to run named like in description
(default: run all tests)
[user@dom0 ~]$ python3 -m qubes.tests.run -h
usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast]
[--do-not-clean] [--do-clean] [--loglevel LEVEL]
[--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg]
[TESTNAME [TESTNAME ...]]
positional arguments:
TESTNAME list of tests to run named like in description
(default: run all tests)
optional arguments:
-h, --help show this help message and exit
--verbose, -v increase console verbosity level
--quiet, -q decrease console verbosity level
--list, -l list all available tests and exit
--failfast, -f stop on the first fail, error or unexpected success
--no-failfast disable --failfast
--loglevel LEVEL, -L LEVEL
logging level for file and syslog forwarding (one of:
NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL;
default: DEBUG)
--logfile FILE, -o FILE
if set, test run will be also logged to file
--syslog reenable logging to syslog
--no-syslog disable logging to syslog
--kmsg, --very-brave-or-very-stupid
log most important things to kernel ring-buffer
--no-kmsg, --i-am-smarter-than-kay-sievers
do not abuse kernel ring-buffer
--allow-running-along-qubesd
allow running in parallel with qubesd; this is
DANGEROUS and WILL RESULT IN INCONSISTENT SYSTEM STATE
--break-to-repl break to REPL after tests
When running only specific tests, write their names like in log, in format:
MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.".
Example: basic/TC_00_Basic/test_000_create
optional arguments:
-h, --help show this help message and exit
--verbose, -v increase console verbosity level
--quiet, -q decrease console verbosity level
--list, -l list all available tests and exit
--failfast, -f stop on the first fail, error or unexpected success
--no-failfast disable --failfast
--loglevel LEVEL, -L LEVEL
logging level for file and syslog forwarding (one of:
NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL;
default: DEBUG)
--logfile FILE, -o FILE
if set, test run will be also logged to file
--syslog reenable logging to syslog
--no-syslog disable logging to syslog
--kmsg, --very-brave-or-very-stupid
log most important things to kernel ring-buffer
--no-kmsg, --i-am-smarter-than-kay-sievers
do not abuse kernel ring-buffer
--allow-running-along-qubesd
allow running in parallel with qubesd; this is
DANGEROUS and WILL RESULT IN INCONSISTENT SYSTEM STATE
--break-to-repl break to REPL after tests
When running only specific tests, write their names like in log, in format:
MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.".
Example: basic/TC_00_Basic/test_000_create
```
For instance, to run only the tests for the fedora-21 template, you can use the ``-l`` option, then filter the list:
For instance, to run only the tests for the fedora-21 template, you can use the `-l` option, then filter the list:
.. code:: bash
[user@dom0 ~]$ python3 -m qubes.tests.run -l | grep fedora-21
network/VmNetworking_fedora-21/test_000_simple_networking
network/VmNetworking_fedora-21/test_010_simple_proxyvm
network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm
network/VmNetworking_fedora-21/test_030_firewallvm_firewall
network/VmNetworking_fedora-21/test_040_inter_vm
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_000_start_shutdown
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_010_run_gui_app
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_050_qrexec_simple_eof
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_051_qrexec_simple_eof_reverse
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_052_qrexec_vm_service_eof
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_053_qrexec_vm_service_eof_reverse
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_060_qrexec_exit_code_dom0
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_065_qrexec_exit_code_vm
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_100_qrexec_filecopy
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_110_qrexec_filecopy_deny
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_120_qrexec_filecopy_self
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_000_prepare_dvm
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file
[user@dom0 ~]$ sudo -E python3 -m qubes.tests.run -v `python3 -m qubes.tests.run -l | grep fedora-21`
```
[user@dom0 ~]$ python3 -m qubes.tests.run -l | grep fedora-21
network/VmNetworking_fedora-21/test_000_simple_networking
network/VmNetworking_fedora-21/test_010_simple_proxyvm
network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm
network/VmNetworking_fedora-21/test_030_firewallvm_firewall
network/VmNetworking_fedora-21/test_040_inter_vm
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_000_start_shutdown
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_010_run_gui_app
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_050_qrexec_simple_eof
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_051_qrexec_simple_eof_reverse
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_052_qrexec_vm_service_eof
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_053_qrexec_vm_service_eof_reverse
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_060_qrexec_exit_code_dom0
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_065_qrexec_exit_code_vm
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_100_qrexec_filecopy
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_110_qrexec_filecopy_deny
vm_qrexec_gui/TC_00_AppVM_fedora-21/test_120_qrexec_filecopy_self
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_000_prepare_dvm
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file
[user@dom0 ~]$ sudo -E python3 -m qubes.tests.run -v `python3 -m qubes.tests.run -l | grep fedora-21`
```
Example test run:
![snapshot-tests2.png](/attachment/doc/snapshot-tests2.png)
.. figure:: /attachment/doc/snapshot-tests2.png
:alt: snapshot-tests2.png
Tests are also compatible with nose2 test runner, so you can use this instead:
```bash
sudo systemctl stop qubesd; sudo -E nose2 -v --plugin nose2.plugins.loader.loadtests qubes.tests; sudo systemctl start qubesd
```
.. code:: bash
This may be especially useful together with various nose2 plugins to store tests results (for example `nose2.plugins.junitxml`), to ease presenting results. This is what we use on [OpenQA](https://open.qa/).
sudo systemctl stop qubesd; sudo -E nose2 -v --plugin nose2.plugins.loader.loadtests qubes.tests; sudo systemctl start qubesd
This may be especially useful together with various nose2 plugins to store tests results (for example ``nose2.plugins.junitxml``), to ease presenting results. This is what we use on `OpenQA <https://open.qa/>`__.
Unit testing inside a VM
^^^^^^^^^^^^^^^^^^^^^^^^
### Unit testing inside a VM
Many unit tests will also work inside a VM. However all of the tests requiring a dedicated VM to be run (mostly the integration tests) will be skipped.
Whereas integration tests are mostly stored in the [qubes-core-admin](https://github.com/QubesOS/qubes-core-admin) repository, unit tests can be found in each of the Qubes OS repositories.
Whereas integration tests are mostly stored in the `qubes-core-admin <https://github.com/QubesOS/qubes-core-admin>`__ repository, unit tests can be found in each of the Qubes OS repositories.
To for example run the `qubes-core-admin` unit tests, you currently have to clone at least [qubes-core-admin](https://github.com/QubesOS/qubes-core-admin) and
its dependency [qubes-core-qrexec](https://github.com/QubesOS/qubes-core-qrexec) repository in the branches that you want to test.
To for example run the ``qubes-core-admin`` unit tests, you currently have to clone at least `qubes-core-admin <https://github.com/QubesOS/qubes-core-admin>`__ and its dependency `qubes-core-qrexec <https://github.com/QubesOS/qubes-core-qrexec>`__ repository in the branches that you want to test.
The below example however will assume that you set up a build environment as described in the [Qubes Builder documentation](/doc/qubes-builder-v2/).
The below example however will assume that you set up a build environment as described in the :doc:`Qubes Builder documentation </developer/building/qubes-builder-v2>`.
Assuming you cloned the `qubes-builder` repository to your home directory inside a fedora VM, you can use the following commands to run the unit tests:
Assuming you cloned the ``qubes-builder`` repository to your home directory inside a fedora VM, you can use the following commands to run the unit tests:
```{.bash}
cd ~
sudo dnf install python3-pip lvm2 python35 python3-virtualenv
virtualenv -p /usr/bin/python35 python35
source python35/bin/activate
python3 -V
cd ~/qubes-builder/qubes-src/core-admin
pip3 install -r ci/requirements.txt
export PYTHONPATH=../core-qrexec:test-packages
./run-tests
```
.. code:: bash
To run only the tests related to e.g. `lvm`, you may use:
cd ~
sudo dnf install python3-pip lvm2 python35 python3-virtualenv
virtualenv -p /usr/bin/python35 python35
source python35/bin/activate
python3 -V
cd ~/qubes-builder/qubes-src/core-admin
pip3 install -r ci/requirements.txt
export PYTHONPATH=../core-qrexec:test-packages
./run-tests
`./run-tests -v $(python3 -m qubes.tests.run -l | grep lvm)`
You can later re-use the created virtual environment including all of the via `pip3` installed packages with `source ~/python35/bin/activate`.
To run only the tests related to e.g. ``lvm``, you may use:
We recommend to run the unit tests with the Python version that the code is meant to be run with in dom0 (3.5 was just an example above). For instance, the `release4.0` (Qubes 4.0) branch is intended
to be run with Python 3.5 whereas the Qubes 4.1 branch (`master` as of 2020-07) is intended to be run with Python 3.7 or higher. You can always check your dom0 installation for the Python version of
the current stable branch.
``./run-tests -v $(python3 -m qubes.tests.run -l | grep lvm)``
You can later re-use the created virtual environment including all of the via ``pip3`` installed packages with ``source ~/python35/bin/activate``.
We recommend to run the unit tests with the Python version that the code is meant to be run with in dom0 (3.5 was just an example above). For instance, the ``release4.0`` (Qubes 4.0) branch is intended to be run with Python 3.5 whereas the Qubes 4.1 branch (``master`` as of 2020-07) is intended to be run with Python 3.7 or higher. You can always check your dom0 installation for the Python version of the current stable branch.
Tests configuration
^^^^^^^^^^^^^^^^^^^
### Tests configuration
Test runs can be altered using environment variables:
- `DEFAULT_LVM_POOL` - LVM thin pool to use for tests, in `VolumeGroup/ThinPool` format
- `QUBES_TEST_PCIDEV` - PCI device to be used in PCI passthrough tests (for example sound card)
- `QUBES_TEST_TEMPLATES` - space separated list of templates to run tests on; if not set, all installed templates are tested
- `QUBES_TEST_LOAD_ALL` - load all tests (including tests for all templates) when relevant test modules are imported; this needs to be set for test runners not supporting [load_tests protocol](https://docs.python.org/3/library/unittest.html#load-tests-protocol)
- ``DEFAULT_LVM_POOL`` - LVM thin pool to use for tests, in ``VolumeGroup/ThinPool`` format
### Adding a new test to core-admin
- ``QUBES_TEST_PCIDEV`` - PCI device to be used in PCI passthrough tests (for example sound card)
After adding a new unit test to [core-admin/qubes/tests](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) you'll have to include it in [core-admin/qubes/tests/\_\_init\_\_.py](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests/__init__.py)
- ``QUBES_TEST_TEMPLATES`` - space separated list of templates to run tests on; if not set, all installed templates are tested
#### Editing `__init__.py`
- ``QUBES_TEST_LOAD_ALL`` - load all tests (including tests for all templates) when relevant test modules are imported; this needs to be set for test runners not supporting `load_tests protocol <https://docs.python.org/3/library/unittest.html#load-tests-protocol>`__
You'll also need to add your test at the bottom of the `__init__.py` file, in the method `def load_tests`, in the for loop with `modname`.
Again, given the hypothetical `example.py` test:
~~~python
for modname in (
'qubes.tests.basic',
'qubes.tests.dom0_update',
'qubes.tests.network',
'qubes.tests.vm_qrexec_gui',
'qubes.tests.backup',
'qubes.tests.backupcompatibility',
'qubes.tests.regressions',
'qubes.tests.example', # This is our newly added test
):
~~~
### Testing PyQt applications
Adding a new test to core-admin
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When testing (Py)QT applications, it's useful to create a separate QApplication object for each test.
But QT framework does not allow multiple QApplication objects in the same process at the same time.
This means it's critical to reliably cleanup the previous instance before creating a new one.
This turns out to be a non-trivial task, especially if _any_ test uses the event loop.
Failure to perform proper cleanup in many cases results in SEGV.
Below you can find steps for the proper cleanup:
~~~python
import asyncio
import quamash
import unittest
import gc
After adding a new unit test to `core-admin/qubes/tests <https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests>`__ you’ll have to include it in `core-admin/qubes/tests/__init__.py <https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests/__init__.py>`__
class SomeTestCase(unittest.TestCase):
def setUp(self):
[...]
Editing ``__init__.py``
^^^^^^^^^^^^^^^^^^^^^^^
# force "cleanlooks" style, the default one on Xfce (GtkStyle) use
# static variable internally and caches pointers to later destroyed
# objects (result: SEGV)
self.qtapp = QtGui.QApplication(["test", "-style", "cleanlooks"])
# construct event loop even if this particular test doesn't use it,
# otherwise events with qtapp references will be queued there anyway and the
# first test that actually use event loop will try to dereference (already
# destroyed) objects, resulting in SEGV
self.loop = quamash.QEventLoop(self.qtapp)
You’ll also need to add your test at the bottom of the ``__init__.py`` file, in the method ``def load_tests``, in the for loop with ``modname``. Again, given the hypothetical ``example.py`` test:
def tearDown(self):
[...]
# process any pending events before destroying the object
self.qtapp.processEvents()
.. code:: python
# queue destroying the QApplication object, do that for any other QT
# related objects here too
self.qtapp.deleteLater()
for modname in (
'qubes.tests.basic',
'qubes.tests.dom0_update',
'qubes.tests.network',
'qubes.tests.vm_qrexec_gui',
'qubes.tests.backup',
'qubes.tests.backupcompatibility',
'qubes.tests.regressions',
'qubes.tests.example', # This is our newly added test
):
# process any pending events (other than just queued destroy), just in case
self.qtapp.processEvents()
# execute main loop, which will process all events, _including just queued destroy_
self.loop.run_until_complete(asyncio.sleep(0))
Testing PyQt applications
^^^^^^^^^^^^^^^^^^^^^^^^^
# at this point it QT objects are destroyed, cleanup all remaining references;
# del other QT object here too
self.loop.close()
del self.qtapp
del self.loop
gc.collect()
~~~
## Automated tests with openQA
When testing (Py)QT applications, it’s useful to create a separate QApplication object for each test. But QT framework does not allow multiple QApplication objects in the same process at the same time. This means it’s critical to reliably cleanup the previous instance before creating a new one. This turns out to be a non-trivial task, especially if *any* test uses the event loop. Failure to perform proper cleanup in many cases results in SEGV. Below you can find steps for the proper cleanup:
**URL:** <https://openqa.qubes-os.org/>
**Tests:** <https://github.com/marmarek/openqa-tests-qubesos>
.. code:: python
Manually testing Qubes OS and its installation is a time-consuming process.
We use [OpenQA](https://open.qa/) to automate this process.
It works by installing Qubes in KVM and interacting with it as a user would, including simulating mouse clicks and keyboard presses.
Then, it checks the output to see whether various tests were passed, e.g. by comparing the virtual screen output to screenshots of a successful installation.
import asyncio
import quamash
import unittest
import gc
class SomeTestCase(unittest.TestCase):
def setUp(self):
[...]
# force "cleanlooks" style, the default one on Xfce (GtkStyle) use
# static variable internally and caches pointers to later destroyed
# objects (result: SEGV)
self.qtapp = QtGui.QApplication(["test", "-style", "cleanlooks"])
# construct event loop even if this particular test doesn't use it,
# otherwise events with qtapp references will be queued there anyway and the
# first test that actually use event loop will try to dereference (already
# destroyed) objects, resulting in SEGV
self.loop = quamash.QEventLoop(self.qtapp)
def tearDown(self):
[...]
# process any pending events before destroying the object
self.qtapp.processEvents()
# queue destroying the QApplication object, do that for any other QT
# related objects here too
self.qtapp.deleteLater()
# process any pending events (other than just queued destroy), just in case
self.qtapp.processEvents()
# execute main loop, which will process all events, _including just queued destroy_
self.loop.run_until_complete(asyncio.sleep(0))
# at this point it QT objects are destroyed, cleanup all remaining references;
# del other QT object here too
self.loop.close()
del self.qtapp
del self.loop
gc.collect()
Using openQA to automatically test the Qubes installation process works as of Qubes 4.0-rc4 on 2018-01-26, provided that the versions of KVM and QEMU are new enough and the hardware has VT-x and EPT.
KVM also supports nested virtualization, so HVM should theoretically work.
In practice, however, either Xen or QEMU crashes when this is attempted.
Nonetheless, PV works well, which is sufficient for automated installation testing.
Thanks to present and past donors who have provided the infrastructure for Qubes' openQA system with hardware that meets these requirements.
Automated tests with openQA
---------------------------
### Looking for patterns in tests
In order to better visualize patterns in tests the [`openqa_investigator`](https://github.com/QubesOS/openqa-tests-qubesos/blob/master/utils/openqa_investigator.py) script can be used.
It feeds off of the openQA test data to make graph plots. Here is an example:
**URL:** https://openqa.qubes-os.org/ **Tests:** https://github.com/marmarek/openqa-tests-qubesos
Manually testing Qubes OS and its installation is a time-consuming process. We use `OpenQA <https://open.qa/>`__ to automate this process. It works by installing Qubes in KVM and interacting with it as a user would, including simulating mouse clicks and keyboard presses. Then, it checks the output to see whether various tests were passed, e.g. by comparing the virtual screen output to screenshots of a successful installation.
Using openQA to automatically test the Qubes installation process works as of Qubes 4.0-rc4 on 2018-01-26, provided that the versions of KVM and QEMU are new enough and the hardware has VT-x and EPT. KVM also supports nested virtualization, so HVM should theoretically work. In practice, however, either Xen or QEMU crashes when this is attempted. Nonetheless, PV works well, which is sufficient for automated installation testing.
Thanks to present and past donors who have provided the infrastructure for Qubes’ openQA system with hardware that meets these requirements.
Looking for patterns in tests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order to better visualize patterns in tests the `openqa_investigator <https://github.com/QubesOS/openqa-tests-qubesos/blob/master/utils/openqa_investigator.py>`__ script can be used. It feeds off of the openQA test data to make graph plots. Here is an example:
.. figure:: /attachment/doc/openqa-investigator-splitgpg-example.png
:alt: openqa-investigator-splitgpg-example.png
![openqa-investigator-splitgpg-example.png](/attachment/doc/openqa-investigator-splitgpg-example.png)
Some outputs:
- plot by tests
- plot by errors
- markdown
- plot by tests
- plot by errors
- markdown
Some filters:
- filter by error
- filter by test name
- filter by error
Check out the script's help with `python3 openqa_investigator.py --help`
to see all available options.
- filter by test name
Check out the script’s help with ``python3 openqa_investigator.py --help`` to see all available options.

68
developer/debugging/mount-lvm-image.rst
View file

@ -1,54 +1,58 @@
---
lang: en
layout: doc
permalink: /doc/mount-lvm-image/
ref: 46
title: How to mount LVM images
---
=======================
How to mount LVM images
=======================
You want to read your LVM image (e.g., there is a problem where you can't start any VMs except dom0).
1: make the image available for qubesdb.
From dom0 terminal:
You want to read your LVM image (e.g., there is a problem where you can’t start any VMs except dom0).
1: make the image available for qubesdb. From dom0 terminal:
.. code:: bash
# Example: /dev/qubes_dom0/vm-debian-9-tmp-root
[user@dom0]$ dev=$(basename $(readlink /dev/YOUR_LVM_VG/YOUR_LVM_IMAGE))
[user@dom0]$ qubesdb-write /qubes-block-devices/$dev/desc "YOUR_LVM_IMAGE"
```bash
# Example: /dev/qubes_dom0/vm-debian-9-tmp-root
[user@dom0]$ dev=$(basename $(readlink /dev/YOUR_LVM_VG/YOUR_LVM_IMAGE))
[user@dom0]$ qubesdb-write /qubes-block-devices/$dev/desc "YOUR_LVM_IMAGE"
```
2: Create a new disposable VM
```bash
[user@dom0]$ qvm-run -v --dispvm=YOUR_DVM_TEMPLATE --service qubes.StartApp+xterm &
```
.. code:: bash
[user@dom0]$ qvm-run -v --dispvm=YOUR_DVM_TEMPLATE --service qubes.StartApp+xterm &
3: Attach the device to your newly created disp VM
From the GUI, or from the command line:
```bash
[user@dom0]$ qvm-block attach NEWLY_CREATED_DISPVM dom0:$dev
```
.. code:: bash
[user@dom0]$ qvm-block attach NEWLY_CREATED_DISPVM dom0:$dev
4: Mount the partition you want to, and do what you want with it
```bash
[user@dispXXXX]$ mount /dev/xvdiX /mnt/
```
.. code:: bash
[user@dispXXXX]$ mount /dev/xvdiX /mnt/
5: Umount and kill the VM
```bash
[user@dispXXXX]$ umount /mnt/
```
.. code:: bash
[user@dispXXXX]$ umount /mnt/
6: Remove the image from qubesdb
```bash
[user@dom0]$ qubesdb-rm /qubes-block-devices/$dev/
```
.. code:: bash
# References
[user@dom0]$ qubesdb-rm /qubes-block-devices/$dev/
Please consult this issue's [comment](https://github.com/QubesOS/qubes-issues/issues/4687#issuecomment-451626625).
References
----------
Please consult this issue’s `comment <https://github.com/QubesOS/qubes-issues/issues/4687#issuecomment-451626625>`__.

85
developer/debugging/safe-remote-ttys.rst
View file

@ -1,76 +1,77 @@
---
lang: en
layout: doc
permalink: /doc/safe-remote-ttys/
redirect_from:
- /en/doc/safe-remote-ttys/
ref: 49
title: Safe remote dom0 terminals
---
==========================
Safe remote dom0 terminals
==========================
If you do not have working graphics in Dom0, then using a terminal can be quite annoying!
This was the case for the author while trying to debug PCI-passthrough of a machine's primary (only) GPU.
Your first thought might be to just allow network access to Dom0, enable ssh, and connect in remotely.
But, this gravely violates the Qubes security model.
If you do not have working graphics in Dom0, then using a terminal can be quite annoying! This was the case for the author while trying to debug PCI-passthrough of a machine’s primary (only) GPU.
Instead, a better solution is to split the input and output paths of using a terminal.
Use your normal keyboard for input, but have the output go to a remote machine in a unidirectional manner.
Your first thought might be to just allow network access to Dom0, enable ssh, and connect in remotely. But, this gravely violates the Qubes security model.
Instead, a better solution is to split the input and output paths of using a terminal. Use your normal keyboard for input, but have the output go to a remote machine in a unidirectional manner.
To do this, we make use of script(1), qvm-run, and optionally your network transport of choice.
To a different VM
-----------------
As an example of forwarding terminal output to another VM on the same machine:
~~~
$ mkfifo /tmp/foo
$ qvm-run -p some-vm 'xterm -e "cat 0<&5" 5<&0' </tmp/foo >/dev/null 2>&1 &
$ script -f /tmp/foo
~~~
.. code:: bash
$ mkfifo /tmp/foo
$ qvm-run -p some-vm 'xterm -e "cat 0<&5" 5<&0' </tmp/foo >/dev/null 2>&1 &
$ script -f /tmp/foo
To a different machine
----------------------
In this case over SSH (from a network-connected VM):
~~~
$ mkfifo /tmp/foo
$ qvm-run -p some-vm \
'ssh user@host sh -c "DISPLAY=:0 xterm -e \"cat 0<&5\" 5<&0"' \
</tmp/foo >/dev/null 2>&1 &
$ script -f /tmp/foo
~~~
.. code:: bash
Note that no data received over SSH is ever treated as terminal input in Dom0.
The input path remains only from your trusted local keyboard.
$ mkfifo /tmp/foo
$ qvm-run -p some-vm \
'ssh user@host sh -c "DISPLAY=:0 xterm -e \"cat 0<&5\" 5<&0"' \
</tmp/foo >/dev/null 2>&1 &
$ script -f /tmp/foo
Note that no data received over SSH is ever treated as terminal input in Dom0. The input path remains only from your trusted local keyboard.
Multiple terminals
------------------
For multiple terminals, you may find it easier to just use tmux than to try to blindly switch to the correct window.
Terminal size
-------------
It is up to you to ensure the sizes of the local and remote terminal are the same, otherwise things may display incorrectly (especially in interactive programs).
Depending on your shell, the size of your local (blind) terminal is likely stored in the `$LINES` and `$COLUMNS` variables.
~~~
$ echo $COLUMNS $LINES
80 24
~~~
It is up to you to ensure the sizes of the local and remote terminal are the same, otherwise things may display incorrectly (especially in interactive programs). Depending on your shell, the size of your local (blind) terminal is likely stored in the ``$LINES`` and ``$COLUMNS`` variables.
.. code:: bash
$ echo $COLUMNS $LINES
80 24
A note on serial consoles
-------------------------
If your machine has a serial console, you may with to use that, but note that a similar split-I/O model should be used to ensure Dom0 integrity.
If you use the serial console as normal (via e.g. getty on ttyX, and logging in as normal), then the machine at the end of the serial cable could compromise your machine!
Ideally, you would take input from your trusted keyboard, and only send the output over the serial cable via e.g. disabling getty and using:
~~~
script -f /dev/ttyS0
~~~
If your machine has a serial console, you may with to use that, but note that a similar split-I/O model should be used to ensure Dom0 integrity. If you use the serial console as normal (via e.g. getty on ttyX, and logging in as normal), then the machine at the end of the serial cable could compromise your machine! Ideally, you would take input from your trusted keyboard, and only send the output over the serial cable via e.g. disabling getty and using:
You don't even need to connect the TX pin.
.. code:: bash
script -f /dev/ttyS0
You don’t even need to connect the TX pin.

366
developer/debugging/test-bench.rst
View file

@ -1,236 +1,266 @@
---
lang: en
layout: doc
permalink: /doc/test-bench/
redirect_from:
- /en/doc/test-bench/
- /doc/TestBench/
- /wiki/TestBench/
ref: 44
title: How to set up a test bench
---
==========================
How to set up a test bench
==========================
This guide shows how to set up simple test bench that automatically test your code you're about to push. It is written especially for `core3` branch of `core-admin.git` repo, but some ideas are universal.
We will set up a spare machine (bare metal, not a virtual) that will be hosting our experimental Dom0. We will communicate with it via Ethernet and SSH. This tutorial assumes you are familiar with [QubesBuilder](/doc/qubes-builder/) and you have it set up and running flawlessly.
This guide shows how to set up simple test bench that automatically test your code you’re about to push. It is written especially for ``core3`` branch of ``core-admin.git`` repo, but some ideas are universal.
> **Notice:**
> This setup intentionally weakens some security properties in the testing system. So make sure you understand the risks and use exclusively for testing.
We will set up a spare machine (bare metal, not a virtual) that will be hosting our experimental Dom0. We will communicate with it via Ethernet and SSH. This tutorial assumes you are familiar with :doc:`QubesBuilder </developer/building/qubes-builder>` and you have it set up and running flawlessly.
## Setting up the Machine
**Notice:** This setup intentionally weakens some security properties in the testing system. So make sure you understand the risks and use exclusively for testing.
### Install ISO
First, do a clean install from the `.iso` [you built](/doc/qubes-iso-building/) or grabbed elsewhere (for example [here](https://forum.qubes-os.org/t/qubesos-4-1-alpha-signed-weekly-builds/3601)).
Setting up the Machine
----------------------
### Enabling Network Access in Dom0
Internet access is intentionally disabled by default in dom0. But to ease the deployment process we will give it access. The following steps should be done in `dom0`.
Install ISO
^^^^^^^^^^^
> **Note:** the following assume you have only one network card. If you have two, pick one and leave the other attached to `sys-net`.
1. Remove the network card (PCI device) from `sys-net`
First, do a clean install from the ``.iso`` :doc:`you built </developer/building/qubes-iso-building>` or grabbed elsewhere (for example `here <https://forum.qubes-os.org/t/qubesos-4-1-alpha-signed-weekly-builds/3601>`__).
Enabling Network Access in Dom0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Internet access is intentionally disabled by default in dom0. But to ease the deployment process we will give it access. The following steps should be done in ``dom0``.
**Note:** the following assume you have only one network card. If you have two, pick one and leave the other attached to ``sys-net``.
1. Remove the network card (PCI device) from ``sys-net``
2. Restart your computer (for the removal to take effect)
3. Install `dhcp-client` and `openssh-server` on your testbench's dom0.
4. Save the following script in `/home/user/bin/dom0_network.sh` and make it executable. It should enable your network card in dom0. *Be sure to adjust the script's variables to suit your needs.*
```bash
#!/bin/sh
3. Install ``dhcp-client`` and ``openssh-server`` on your testbench’s dom0.
# adjust this for your NIC (run lspci)
BDF=0000:02:00.0
4. Save the following script in ``/home/user/bin/dom0_network.sh`` and make it executable. It should enable your network card in dom0. *Be sure to adjust the script’s variables to suit your needs.*
# adjust this for your network driver
DRIVER=e1000e
.. code:: bash
prog=$(basename $0)
#!/bin/sh
# adjust this for your NIC (run lspci)
BDF=0000:02:00.0
# adjust this for your network driver
DRIVER=e1000e
prog=$(basename $0)
pciunbind() {
local path
path=/sys/bus/pci/devices/${1}/driver/unbind
if ! [ -w ${path} ]; then
echo "${prog}: Device ${1} not bound"
return 1
fi
echo -n ${1} >${path}
}
pcibind() {
local path
path=/sys/bus/pci/drivers/${2}/bind
if ! [ -w ${path} ]; then
echo "${prog}: Driver ${2} not found"
return 1
fi
echo ${1} >${path}
}
pciunbind ${BDF}
pcibind ${BDF} ${DRIVER}
sleep 1
dhclient
pciunbind() {
local path
path=/sys/bus/pci/devices/${1}/driver/unbind
if ! [ -w ${path} ]; then
echo "${prog}: Device ${1} not bound"
return 1
fi
echo -n ${1} >${path}
}
pcibind() {
local path
path=/sys/bus/pci/drivers/${2}/bind
if ! [ -w ${path} ]; then
echo "${prog}: Driver ${2} not found"
return 1
fi
echo ${1} >${path}
}
pciunbind ${BDF}
pcibind ${BDF} ${DRIVER}
sleep 1
dhclient
```
5. Configure your DHCP server so your testbench gets static IP and connect your machine to your local network. You should ensure that your testbench can reach the Internet.
6. You'll need to run the above script on every startup. To automate this save the following systemd service `/etc/systemd/system/dom0-network-direct.service`
6. You’ll need to run the above script on every startup. To automate this save the following systemd service ``/etc/systemd/system/dom0-network-direct.service``
```
[Unit]
Description=Connect network to dom0
.. code:: bash
[Unit]
Description=Connect network to dom0
[Service]
Type=oneshot
ExecStart=/home/user/bin/dom0_network.sh
[Install]
WantedBy=multi-user.target
[Service]
Type=oneshot
ExecStart=/home/user/bin/dom0_network.sh
[Install]
WantedBy=multi-user.target
```
7. Then, enable and start the SSH Server and the script on boot:
```bash
sudo systemctl enable sshd
sudo systemctl start sshd
sudo systemctl enable dom0-network-direct
sudo systemctl start dom0-network-direct
```
.. code:: bash
> **Note:** If you want to install additional software in dom0 and your only network card was assigned to dom0, then _instead_ of the usual `sudo qubes-dom0-update <PACKAGE>` now you run `sudo dnf --setopt=reposdir=/etc/yum.repos.d install <PACKAGE>`.
sudo systemctl enable sshd
sudo systemctl start sshd
sudo systemctl enable dom0-network-direct
sudo systemctl start dom0-network-direct
### Install Tests and Their Dependencies
A regular Qubes installation isn't ready to run the full suite of tests. For example, in order to run the [Split GPG tests](https://github.com/QubesOS/qubes-app-linux-split-gpg/blob/4bc201bb70c011119eed19df25dc5b46120d04ed/tests/splitgpg/tests.py) you need to have the `qubes-gpg-split-tests` package installed in your app qubes.
Because of the above reason, some additional configurations need to be done to your testing environment. This can be done in an automated manner with the help of the [Salt](/doc/salt) configuration that provisions the [automated testing environment](/doc/automated-tests/).
The following commands should work for you, but do keep in mind that the provisioning scripts are designed for the [openQA environment](https://openqa.qubes-os.org/) and not your specific local testing system. Run the following in `dom0`:
**Note:** If you want to install additional software in dom0 and your only network card was assigned to dom0, then *instead* of the usual ``sudo qubes-dom0-update <PACKAGE>`` now you run ``sudo dnf --setopt=reposdir=/etc/yum.repos.d install <PACKAGE>``.
```bash
# For future reference the following commands are an adaptation of
# https://github.com/marmarek/openqa-tests-qubesos/blob/master/tests/update.pm
Install Tests and Their Dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Install git
sudo qubes-dom0-update git || sudo dnf --setopt=reposdir=/etc/yum.repos.d install git
# Download the openQA automated testing environment Salt configuration
git clone https://github.com/marmarek/openqa-tests-qubesos/
cd openqa-tests-qubesos/extra-files
sudo cp -a system-tests/ /srv/salt/
sudo qubesctl top.enable system-tests
# Install the same configuration as the one in openQA
QUBES_VERSION=4.1
PILLAR_DIR=/srv/pillar/base/update
sudo mkdir -p $PILLAR_DIR
printf 'update:\n qubes_ver: '$QUBES_VERSION'\n' | sudo tee $PILLAR_DIR/init.sls
printf "base:\n '*':\n - update\n" | sudo tee $PILLAR_DIR/init.top
sudo qubesctl top.enable update pillar=True
A regular Qubes installation isn’t ready to run the full suite of tests. For example, in order to run the `Split GPG tests <https://github.com/QubesOS/qubes-app-linux-split-gpg/blob/4bc201bb70c011119eed19df25dc5b46120d04ed/tests/splitgpg/tests.py>`__ you need to have the ``qubes-gpg-split-tests`` package installed in your app qubes.
# Apply states to dom0 and VMs
# NOTE: These commands can take several minutes (if not more) without showing output
sudo qubesctl --show-output state.highstate
sudo qubesctl --max-concurrency=2 --skip-dom0 --templates --show-output state.highstate
```
Because of the above reason, some additional configurations need to be done to your testing environment. This can be done in an automated manner with the help of the :doc:`Salt </user/advanced-topics/salt>` configuration that provisions the :doc:`automated testing environment </developer/debugging/automated-tests>`.
## Development VM
The following commands should work for you, but do keep in mind that the provisioning scripts are designed for the `openQA environment <https://openqa.qubes-os.org/>`__ and not your specific local testing system. Run the following in ``dom0``:
### SSH
.. code:: bash
Arrange firewall so you can reach the testbench from your `qubes-dev` VM. Generate SSH key in `qubes-dev`:
# For future reference the following commands are an adaptation of
# https://github.com/marmarek/openqa-tests-qubesos/blob/master/tests/update.pm
# Install git
sudo qubes-dom0-update git || sudo dnf --setopt=reposdir=/etc/yum.repos.d install git
# Download the openQA automated testing environment Salt configuration
git clone https://github.com/marmarek/openqa-tests-qubesos/
cd openqa-tests-qubesos/extra-files
sudo cp -a system-tests/ /srv/salt/
sudo qubesctl top.enable system-tests
# Install the same configuration as the one in openQA
QUBES_VERSION=4.1
PILLAR_DIR=/srv/pillar/base/update
sudo mkdir -p $PILLAR_DIR
printf 'update:\n qubes_ver: '$QUBES_VERSION'\n' | sudo tee $PILLAR_DIR/init.sls
printf "base:\n '*':\n - update\n" | sudo tee $PILLAR_DIR/init.top
sudo qubesctl top.enable update pillar=True
# Apply states to dom0 and VMs
# NOTE: These commands can take several minutes (if not more) without showing output
sudo qubesctl --show-output state.highstate
sudo qubesctl --max-concurrency=2 --skip-dom0 --templates --show-output state.highstate
~~~
ssh-keygen -t ecdsa -b 521
~~~
Add the following section in `.ssh/config` in `qubes-dev`:
Development VM
--------------
~~~
Host testbench
# substitute username in testbench
User user
# substitute address of your testbench
HostName 192.168.123.45
~~~
#### Passwordless SSH Login
SSH
^^^
To log to your testbench without entering password every time, copy your newly generated public key (`id_ecdsa.pub`) to `~/.ssh/authorized_keys` on your testbench. You can do this easily by running this command on `qubes-dev`: `ssh-copy-id -i ~/.ssh/id_ecdsa.pub user@192.168.123.45` (substituting with the actual username address of your testbench).
### Scripting
Arrange firewall so you can reach the testbench from your ``qubes-dev`` VM. Generate SSH key in ``qubes-dev``:
This step is optional, but very helpful. Put these scripts somewhere in your `${PATH}`, like `/usr/local/bin`.
.. code:: bash
`qtb-runtests`:
ssh-keygen -t ecdsa -b 521
```bash
#!/bin/sh
ssh testbench python -m qubes.tests.run
```
`qtb-install`:
Add the following section in ``.ssh/config`` in ``qubes-dev``:
```bash
#!/bin/sh
.. code:: bash
TMPDIR=/tmp/qtb-rpms
Host testbench
# substitute username in testbench
User user
# substitute address of your testbench
HostName 192.168.123.45
if [ $# -eq 0 ]; then
echo "usage: $(basename $0) <rpmfile> ..."
exit 2
fi
set -e
ssh testbench mkdir -p "${TMPDIR}"
scp "${@}" testbench:"${TMPDIR}" || echo "check if you have 'scp' installed on your testbench"
Passwordless SSH Login
^^^^^^^^^^^^^^^^^^^^^^
while [ $# -gt 0 ]; do
ssh testbench sudo rpm -i --replacepkgs --replacefiles "${TMPDIR}/$(basename ${1})"
shift
done
```
`qtb-iterate`:
To log to your testbench without entering password every time, copy your newly generated public key (``id_ecdsa.pub``) to ``~/.ssh/authorized_keys`` on your testbench. You can do this easily by running this command on ``qubes-dev``: ``ssh-copy-id -i ~/.ssh/id_ecdsa.pub user@192.168.123.45`` (substituting with the actual username address of your testbench).
```bash
#!/bin/sh
Scripting
^^^^^^^^^
set -e
# substitute path to your builder installation
pushd ${HOME}/builder >/dev/null
This step is optional, but very helpful. Put these scripts somewhere in your ``${PATH}``, like ``/usr/local/bin``.
# the following are needed only if you have sources outside builder
#rm -rf qubes-src/core-admin
#qb -c core-admin package fetch
``qtb-runtests``:
qb -c core-admin -d host-fc41 prep build
# update your dom0 fedora distribution as appropriate
qtb-install qubes-src/core-admin/rpm/x86_64/qubes-core-dom0-*.rpm
qtb-runtests
```
.. code:: bash
### Hooking git
#!/bin/sh
ssh testbench python -m qubes.tests.run
I (woju) have those two git hooks. They ensure tests are passing (or are marked as expected failure) when committing and pushing. For committing it is only possible to run tests that may be executed from git repo (even if the rest were available, I probably wouldn't want to do that). For pushing, I also install RPM and run tests on testbench.
`core-admin/.git/hooks/pre-commit`: (you may retain also the default hook, here omitted for readability)
``qtb-install``:
```bash
#!/bin/sh
.. code:: bash
set -e
#!/bin/sh
TMPDIR=/tmp/qtb-rpms
if [ $# -eq 0 ]; then
echo "usage: $(basename $0) <rpmfile> ..."
exit 2
fi
set -e
ssh testbench mkdir -p "${TMPDIR}"
scp "${@}" testbench:"${TMPDIR}" || echo "check if you have 'scp' installed on your testbench"
while [ $# -gt 0 ]; do
ssh testbench sudo rpm -i --replacepkgs --replacefiles "${TMPDIR}/$(basename ${1})"
shift
done
python -c "import sys, qubes.tests.run; sys.exit(not qubes.tests.run.main())"
```
`core-admin/.git/hooks/pre-push`:
``qtb-iterate``:
```bash
#!/bin/sh
.. code:: bash
#!/bin/sh
set -e
# substitute path to your builder installation
pushd ${HOME}/builder >/dev/null
# the following are needed only if you have sources outside builder
#rm -rf qubes-src/core-admin
#qb -c core-admin package fetch
qb -c core-admin -d host-fc41 prep build
# update your dom0 fedora distribution as appropriate
qtb-install qubes-src/core-admin/rpm/x86_64/qubes-core-dom0-*.rpm
qtb-runtests
Hooking git
^^^^^^^^^^^
I (woju) have those two git hooks. They ensure tests are passing (or are marked as expected failure) when committing and pushing. For committing it is only possible to run tests that may be executed from git repo (even if the rest were available, I probably wouldn’t want to do that). For pushing, I also install RPM and run tests on testbench.
``core-admin/.git/hooks/pre-commit``: (you may retain also the default hook, here omitted for readability)
.. code:: bash
#!/bin/sh
set -e
python -c "import sys, qubes.tests.run; sys.exit(not qubes.tests.run.main())"
``core-admin/.git/hooks/pre-push``:
.. code:: bash
#!/bin/sh
exec qtb-iterate
exec qtb-iterate
```

432
developer/debugging/vm-interface.rst
View file

@ -1,226 +1,298 @@
---
lang: en
layout: doc
permalink: /doc/vm-interface/
redirect_from:
- /en/doc/vm-interface/
- /doc/VMInterface/
- /doc/SystemDoc/VMInterface/
- /wiki/SystemDoc/VMInterface/
ref: 47
title: Qube configuration interface
---
============================
Qube configuration interface
============================
Qubes VM have some settings set by dom0 based on VM settings. There are multiple configuration channels, which includes:
- QubesDB
- XenStore (in Qubes 2, data the same as in QubesDB, keys without leading `/`)
- XenStore (in Qubes 2, data the same as in QubesDB, keys without leading ``/``)
- Qubes RPC (called at VM startup, or when configuration changed)
- GUI protocol
## QubesDB
### Keys exposed by dom0 to VM
- `/qubes-base-template` - base template
- `/qubes-vm-type` - VM type, the same as `type` field in `qvm-prefs`. One of `AppVM`, `ProxyVM`, `NetVM`, `TemplateVM`, `HVM`, `TemplateHVM`
- `/qubes-vm-updatable` - flag whether VM is updatable (whether changes in root.img will survive VM restart). One of `True`, `False`
- `/qubes-vm-persistence` - what data do persist between VM restarts:
- `full` - all disks
- `rw-only` - only `/rw` disk
- `none` - none
- `/qubes-timezone` - name of timezone based on dom0 timezone. For example `Europe/Warsaw`
- `/qubes-keyboard` (deprecated in R4.1) - keyboard layout based on dom0 layout. Its syntax is suitable for `xkbcomp` command (after expanding escape sequences like `\n` or `\t`). This is meant only as some default value, VM can ignore this option and choose its own keyboard layout (this is what keyboard setting from Qubes Manager does). This entry is created as part of gui-daemon initialization (so not available when gui-daemon disabled, or not started yet).
- `/keyboard-layout` - keyboard layout based on GuiVM layout. Its syntax can be `layout+variant+options`, `layout+variant`, `layout++options` or simply `layout`. For example, `fr+oss`, `pl++compose:caps` or `fr`. This is meant only as some default value, VM can ignore this option and choose its own keyboard layout (this is what keyboard setting from Qubes Manager does).
- `/qubes-debug-mode` - flag whether VM has debug mode enabled (qvm-prefs setting). One of `1`, `0`
- `/qubes-service/SERVICE_NAME` - subtree for VM services controlled from dom0 (using the `qvm-service` command or Qubes Manager). One of `1`, `0`. Note that not every service will be listed here, if entry is missing, it means "use VM default". A list of currently supported services is in the `qvm-service` man page.
- `/qubes-netm ask` - network mask (only when VM has netvm set); currently hardcoded "255.255.255.0"
- `/qubes-ip` - IP address for this VM (only when VM has netvm set)
- `/qubes-gateway` - default gateway IP (only when VM has netvm set); VM should add host route to this address directly via eth0 (or whatever default interface name is)
- `/qubes-primary-dns` - primary DNS address (only when VM has netvm set)
- `/qubes-secondary-dns` - secondary DNS address (only when VM has netvm set)
- `/qubes-netvm-gateway` - same as `qubes-gateway` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM)
- `/qubes-netvm-netmask` - same as `qubes-netmask` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM)
- `/qubes-netvm-network` - network address (only when VM serves as network backend - ProxyVM and NetVM); can be also calculated from qubes-netvm-gateway and qubes-netvm-netmask
- `/qubes-netvm-primary-dns` - same as `qubes-primary-dns` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM); traffic sent to this IP on port 53 should be redirected to primary DNS server
- `/qubes-netvm-secondary-dns` - same as `qubes-secondary-dns` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM); traffic sent to this IP on port 53 should be redirected to secondary DNS server
- `/guivm-windows-prefix` - title prefix for any window not originating from another qube. This means windows of applications running in GuiVM itself
QubesDB
-------
#### Firewall rules in 3.x
QubesDB is also used to configure firewall in ProxyVMs. Rules are stored in
separate key for each target VM. Entries:
Keys exposed by dom0 to VM
^^^^^^^^^^^^^^^^^^^^^^^^^^
- `/qubes-iptables` - control entry - dom0 writing `reload` here signals `qubes-firewall` service to reload rules
- `/qubes-iptables-header` - rules not related to any particular VM, should be applied before domains rules
- `/qubes-iptables-domainrules/NNN` - rules for domain `NNN` (arbitrary number)
in `iptables-save` format. Rules are self-contained - fill `FORWARD` iptables
chain and contains all required matches (source IP address etc), as well as
final default action (`DROP`/`ACCEPT`)
VM after applying rules may signal some error, writing a message to
`/qubes-iptables-error` key. This does not exclude any other way of
communicating problems - like a popup.
- ``/qubes-base-template`` - base template
#### Firewall rules in 4.x
- ``/qubes-vm-type`` - VM type, the same as ``type`` field in ``qvm-prefs``. One of ``AppVM``, ``ProxyVM``, ``NetVM``, ``TemplateVM``, ``HVM``, ``TemplateHVM``
QubesDB is also used to configure firewall in ProxyVMs. Each rule is stored as
a separate entry, grouped on target VM:
- ``/qubes-vm-updatable`` - flag whether VM is updatable (whether changes in root.img will survive VM restart). One of ``True``, ``False``
- `/qubes-firewall/SOURCE_IP` - base tree under which rules are placed. All
rules there should be applied to filter traffic coming from `SOURCE_IP`. This
can be either IPv4 or IPv6 address.
Dom0 will do an empty write to this top level entry after finishing rules
update, so VM can setup a watch here to trigger rules reload.
- `/qubes-firewall/SOURCE_IP/policy` - default action if no rule matches:
`drop` or `accept`.
- `/qubes-firewall/SOURCE_IP/NNNN` - rule number `NNNN` - decimal number,
padded with zeros. Se below for rule format. All the rules should be
applied in order of rules implied by those numbers. Note that QubesDB
itself does not impose any ordering (you need to sort the rules after
retrieving them). The first rule has number `0000`.
- ``/qubes-vm-persistence`` - what data do persist between VM restarts:
Each rule is a single QubesDB entry, consisting of pairs `key=value` separated
by space. QubesDB enforces limit on a single entry length - 3072 bytes.
Possible options for a single rule:
- ``full`` - all disks
- `action`, values: `accept`, `drop`; this is present in every rule
- `dst4`, value: destination IPv4 address with a mask; for example: `192.168.0.0/24`
- `dst6`, value: destination IPv6 address with a mask; for example: `2000::/3`
- `dsthost`, value: DNS hostname of destination host
- `proto`, values: `tcp`, `udp`, `icmp`
- `specialtarget`, value: One of predefined target, currently defined values:
- `dns` - such option should match DNS traffic to default DNS server (but
not any DNS server), on both TCP and UDP
- `dstports`, value: destination ports range separated with `-`, valid only
together with `proto=tcp` or `proto=udp`; for example `1-1024`, `80-80`
- `icmptype`, value: numeric (decimal) icmp message type, for example `8` for
echo request, valid only together with `proto=icmp`
- `dpi`, value: Deep Packet Inspection protocol (like: HTTP, SSL, SMB, SSH, SMTP) or the default 'NO' as no DPI, only packet filtering
- ``rw-only`` - only ``/rw`` disk
Options must appear in the rule in the order listed above. Duplicated options
are forbidden.
- ``none`` - none
A rule matches only when all predicates match. Only one of `dst4`, `dst6` or
`dsthost` can be used in a single rule.
If tool applying firewall encounters any parse error (unknown option, invalid
value, duplicated option, etc), it should drop all the traffic coming from that `SOURCE_IP`,
regardless of properly parsed rules.
- ``/qubes-timezone`` - name of timezone based on dom0 timezone. For example ``Europe/Warsaw``
- ``/qubes-keyboard`` (deprecated in R4.1) - keyboard layout based on dom0 layout. Its syntax is suitable for ``xkbcomp`` command (after expanding escape sequences like ``\n`` or ``\t``). This is meant only as some default value, VM can ignore this option and choose its own keyboard layout (this is what keyboard setting from Qubes Manager does). This entry is created as part of gui-daemon initialization (so not available when gui-daemon disabled, or not started yet).
- ``/keyboard-layout`` - keyboard layout based on GuiVM layout. Its syntax can be ``layout+variant+options``, ``layout+variant``, ``layout++options`` or simply ``layout``. For example, ``fr+oss``, ``pl++compose:caps`` or ``fr``. This is meant only as some default value, VM can ignore this option and choose its own keyboard layout (this is what keyboard setting from Qubes Manager does).
- ``/qubes-debug-mode`` - flag whether VM has debug mode enabled (qvm-prefs setting). One of ``1``, ``0``
- ``/qubes-service/SERVICE_NAME`` - subtree for VM services controlled from dom0 (using the ``qvm-service`` command or Qubes Manager). One of ``1``, ``0``. Note that not every service will be listed here, if entry is missing, it means “use VM default”. A list of currently supported services is in the ``qvm-service`` man page.
- ``/qubes-netm ask`` - network mask (only when VM has netvm set); currently hardcoded “255.255.255.0”
- ``/qubes-ip`` - IP address for this VM (only when VM has netvm set)
- ``/qubes-gateway`` - default gateway IP (only when VM has netvm set); VM should add host route to this address directly via eth0 (or whatever default interface name is)
- ``/qubes-primary-dns`` - primary DNS address (only when VM has netvm set)
- ``/qubes-secondary-dns`` - secondary DNS address (only when VM has netvm set)
- ``/qubes-netvm-gateway`` - same as ``qubes-gateway`` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM)
- ``/qubes-netvm-netmask`` - same as ``qubes-netmask`` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM)
- ``/qubes-netvm-network`` - network address (only when VM serves as network backend - ProxyVM and NetVM); can be also calculated from qubes-netvm-gateway and qubes-netvm-netmask
- ``/qubes-netvm-primary-dns`` - same as ``qubes-primary-dns`` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM); traffic sent to this IP on port 53 should be redirected to primary DNS server
- ``/qubes-netvm-secondary-dns`` - same as ``qubes-secondary-dns`` in connected VMs (only when VM serves as network backend - ProxyVM and NetVM); traffic sent to this IP on port 53 should be redirected to secondary DNS server
- ``/guivm-windows-prefix`` - title prefix for any window not originating from another qube. This means windows of applications running in GuiVM itself
Firewall rules in 3.x
^^^^^^^^^^^^^^^^^^^^^
QubesDB is also used to configure firewall in ProxyVMs. Rules are stored in separate key for each target VM. Entries:
- ``/qubes-iptables`` - control entry - dom0 writing ``reload`` here signals ``qubes-firewall`` service to reload rules
- ``/qubes-iptables-header`` - rules not related to any particular VM, should be applied before domains rules
- ``/qubes-iptables-domainrules/NNN`` - rules for domain ``NNN`` (arbitrary number) in ``iptables-save`` format. Rules are self-contained - fill ``FORWARD`` iptables chain and contains all required matches (source IP address etc), as well as final default action (``DROP``/``ACCEPT``)
VM after applying rules may signal some error, writing a message to ``/qubes-iptables-error`` key. This does not exclude any other way of communicating problems - like a popup.
Firewall rules in 4.x
^^^^^^^^^^^^^^^^^^^^^
QubesDB is also used to configure firewall in ProxyVMs. Each rule is stored as a separate entry, grouped on target VM:
- ``/qubes-firewall/SOURCE_IP`` - base tree under which rules are placed. All rules there should be applied to filter traffic coming from ``SOURCE_IP``. This can be either IPv4 or IPv6 address. Dom0 will do an empty write to this top level entry after finishing rules update, so VM can setup a watch here to trigger rules reload.
- ``/qubes-firewall/SOURCE_IP/policy`` - default action if no rule matches: ``drop`` or ``accept``.
- ``/qubes-firewall/SOURCE_IP/NNNN`` - rule number ``NNNN`` - decimal number, padded with zeros. Se below for rule format. All the rules should be applied in order of rules implied by those numbers. Note that QubesDB itself does not impose any ordering (you need to sort the rules after retrieving them). The first rule has number ``0000``.
Each rule is a single QubesDB entry, consisting of pairs ``key=value`` separated by space. QubesDB enforces limit on a single entry length - 3072 bytes. Possible options for a single rule:
- ``action``, values: ``accept``, ``drop``; this is present in every rule
- ``dst4``, value: destination IPv4 address with a mask; for example: ``192.168.0.0/24``
- ``dst6``, value: destination IPv6 address with a mask; for example: ``2000::/3``
- ``dsthost``, value: DNS hostname of destination host
- ``proto``, values: ``tcp``, ``udp``, ``icmp``
- ``specialtarget``, value: One of predefined target, currently defined values:
- ``dns`` - such option should match DNS traffic to default DNS server (but not any DNS server), on both TCP and UDP
- ``dstports``, value: destination ports range separated with ``-``, valid only together with ``proto=tcp`` or ``proto=udp``; for example ``1-1024``, ``80-80``
- ``icmptype``, value: numeric (decimal) icmp message type, for example ``8`` for echo request, valid only together with ``proto=icmp``
- ``dpi``, value: Deep Packet Inspection protocol (like: HTTP, SSL, SMB, SSH, SMTP) or the default ‘NO’ as no DPI, only packet filtering
Options must appear in the rule in the order listed above. Duplicated options are forbidden.
A rule matches only when all predicates match. Only one of ``dst4``, ``dst6`` or ``dsthost`` can be used in a single rule.
If tool applying firewall encounters any parse error (unknown option, invalid value, duplicated option, etc), it should drop all the traffic coming from that ``SOURCE_IP``, regardless of properly parsed rules.
Example valid rules:
- `action=accept dst4=8.8.8.8 proto=udp dstports=53-53`
- `action=drop dst6=2a00:1450:4000::/37 proto=tcp`
- `action=accept specialtarget=dns`
- `action=drop proto=tcp specialtarget=dns` - drop DNS queries sent using TCP
- `action=drop`
- ``action=accept dst4=8.8.8.8 proto=udp dstports=53-53``
### Keys set by VM for passing info to dom0
- ``action=drop dst6=2a00:1450:4000::/37 proto=tcp``
- ``action=accept specialtarget=dns``
- ``action=drop proto=tcp specialtarget=dns`` - drop DNS queries sent using TCP
- ``action=drop``
Keys set by VM for passing info to dom0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- ``memory/meminfo`` (**xenstore**) - used memory (updated by qubes-meminfo-writer), input information for qmemman;
- Qubes 3.x format: 6 lines (EOL encoded as ``\n``), each in format “FIELD: VALUE kB”; fields: ``MemTotal``, ``MemFree``, ``Buffers``, ``Cached``, ``SwapTotal``, ``SwapFree``; meaning the same as in ``/proc/meminfo`` in Linux.
- `memory/meminfo` (**xenstore**) - used memory (updated by qubes-meminfo-writer), input information for qmemman;
- Qubes 3.x format: 6 lines (EOL encoded as `\n`), each in format "FIELD: VALUE kB"; fields: `MemTotal`, `MemFree`, `Buffers`, `Cached`, `SwapTotal`, `SwapFree`; meaning the same as in `/proc/meminfo` in Linux.
- Qubes 4.0+ format: used memory size in the VM, in kbytes
- `/qubes-block-devices` - list of block devices exposed by this VM, each device (subdirectory) should be named in a way that VM can attach the device based on it. Each should contain these entries:
- `desc` - device description (ASCII text)
- `size` - device size in bytes
- `mode` - default connection mode; `r` for read-only, `w` for read-write
- `/qubes-usb-devices` - list of USB devices exposed by this VM, each device (subdirectory) should contain:
- `desc` - device description (ASCII text)
- `usb-ver` - USB version (1, 2 or 3)
## Qubes RPC
- ``/qubes-block-devices`` - list of block devices exposed by this VM, each device (subdirectory) should be named in a way that VM can attach the device based on it. Each should contain these entries:
- ``desc`` - device description (ASCII text)
- ``size`` - device size in bytes
- ``mode`` - default connection mode; ``r`` for read-only, ``w`` for read-write
- ``/qubes-usb-devices`` - list of USB devices exposed by this VM, each device (subdirectory) should contain:
- ``desc`` - device description (ASCII text)
- ``usb-ver`` - USB version (1, 2 or 3)
Qubes RPC
---------
Services called by dom0 to provide some VM configuration:
- `qubes.SetMonitorLayout` - provide list of monitors, one per line. Each line contains four numbers: `width height X Y width_mm height_mm` (physical dimensions - `width_mm` and `height_mm` - are optional)
- `qubes.WaitForSession` - called to wait for full VM startup
- `qubes.GetAppmenus` - receive appmenus from given VM (template); TODO: describe format here
- `qubes.GetImageRGBA` - receive image/application icon. Protocol:
- ``qubes.SetMonitorLayout`` - provide list of monitors, one per line. Each line contains four numbers: ``width height X Y width_mm height_mm`` (physical dimensions - ``width_mm`` and ``height_mm`` - are optional)
- ``qubes.WaitForSession`` - called to wait for full VM startup
- ``qubes.GetAppmenus`` - receive appmenus from given VM (template); TODO: describe format here
- ``qubes.GetImageRGBA`` - receive image/application icon. Protocol:
1. Caller sends name of requested icon. This can be one of:
* `xdgicon:NAME` - search for NAME in standard icons theme
* `-` - get icon data from stdin (the caller), can be prefixed with format name, for example `png:-`
* file name
2. The service responds with image dimensions: width and height as
decimal numbers, separated with space and with EOL marker at the and; then
image data in RGBA format (32 bits per pixel)
- `qubes.SetDateTime` - set VM time, called periodically by dom0 (can be
triggered manually from dom0 by calling `qvm-sync-clock`). The service
receives one line at stdin - time in format of `date -u -Iseconds`, for
example `2015-07-31T16:10:43+0000`.
- `qubes.SetGuiMode` - called in HVM to switch between fullscreen and seamless
GUI mode. The service receives a single word on stdin - either `FULLSCREEN`
or `SEAMLESS`
- `qubes.ResizeDisk` - called to inform that underlying disk was resized.
Name of disk image is passed on standard input (`root`, `private`, `volatile`,
or other). This is used starting with Qubes 4.0.
- ``xdgicon:NAME`` - search for NAME in standard icons theme
- ``-`` - get icon data from stdin (the caller), can be prefixed with format name, for example ``png:-``
- file name
2. The service responds with image dimensions: width and height as decimal numbers, separated with space and with EOL marker at the and; then image data in RGBA format (32 bits per pixel)
- ``qubes.SetDateTime`` - set VM time, called periodically by dom0 (can be triggered manually from dom0 by calling ``qvm-sync-clock``). The service receives one line at stdin - time in format of ``date -u -Iseconds``, for example ``2015-07-31T16:10:43+0000``.
- ``qubes.SetGuiMode`` - called in HVM to switch between fullscreen and seamless GUI mode. The service receives a single word on stdin - either ``FULLSCREEN`` or ``SEAMLESS``
- ``qubes.ResizeDisk`` - called to inform that underlying disk was resized. Name of disk image is passed on standard input (``root``, ``private``, ``volatile``, or other). This is used starting with Qubes 4.0.
Other Qrexec services installed by default:
- `qubes.Backup` - store Qubes backup. The service receives location chosen by
the user (one line, terminated by `\n`), the backup archive ([description of
backup format](/doc/BackupEmergencyRestoreV2/))
- `qubes.DetachPciDevice` - service called in reaction to `qvm-pci -d` call on
running VM. The service receives one word - BDF of device to detach. When the
service call ends, the device will be detached
- `qubes.Filecopy` - receive some files from other VM. Files sent in [qfile format](/doc/qfilecopy/)
- `qubes.OpenInVM` - open a file in called VM. Service receives a single file on stdin (in
[qfile format](/doc/qfilecopy/). After a file viewer/editor is terminated, if
the file was modified, can be sent back (just raw content, without any
headers); otherwise service should just terminate without sending anything.
This service is used by both `qvm-open-in-vm` and `qvm-open-in-dvm` tools. When
called in DispVM, service termination will trigger DispVM cleanup.
- `qubes.Restore` - retrieve Qubes backup. The service receives backup location
entered by the user (one line, terminated by `\n`), then should output backup
archive in [qfile format](/doc/qfilecopy/) (core-agent-linux component contains
`tar2qfile` utility to do the conversion)
- `qubes.SelectDirectory`, `qubes.SelectFile` - services which should show
file/directory selection dialog and return (to stdout) a single line
containing selected path, or nothing in the case of cancellation
- `qubes.SuspendPre` - service called in every VM with PCI device attached just
before system suspend
- `qubes.SuspendPost` - service called in every VM with PCI device attached just
after system resume
- `qubes.SyncNtpClock` - service called to trigger network time synchronization.
Service should synchronize local VM time and terminate when done.
- `qubes.WindowIconUpdater` - service called by VM to send icons of individual
windows. The protocol there is simple one direction stream: VM sends window ID
followed by icon in `qubes.GetImageRGBA` format, then next window ID etc. VM
can send icon for the same window multiple times to replace previous one (for
example for animated icons)
- `qubes.VMShell` - call any command in the VM; the command(s) is passed one per line
- `qubes.VMShell+WaitForSession` waits for full VM startup first
- `qubes.VMExec` - call any command in the VM, without using shell, the command
needs to be passed as argument and encoded as follows:
- the executable name and arguments are separated by `+`
- everything except alphanumeric characters, `.` and `_` needs to be
escaped
- bytes are escaped as `-HH` (where `HH` is hex code, capital letters only)
- `-` itself can be escaped as `--`
- example: to run `ls -a /home/user`, use
`qubes.VMExec+ls+--a+-2Fhome-2Fuser`
- `qubes.VMExecGUI` - a variant of `qubes.VMExec` that waits for full VM
startup first
- ``qubes.Backup`` - store Qubes backup. The service receives location chosen by the user (one line, terminated by ``\n``), the backup archive (:doc:`description of backup format </user/how-to-guides/backup-emergency-restore-v2>`)
- ``qubes.DetachPciDevice`` - service called in reaction to ``qvm-pci -d`` call on running VM. The service receives one word - BDF of device to detach. When the service call ends, the device will be detached
- ``qubes.Filecopy`` - receive some files from other VM. Files sent in :doc:`qfile format </developer/services/qfilecopy>`
- ``qubes.OpenInVM`` - open a file in called VM. Service receives a single file on stdin (in :doc:`qfile format </developer/services/qfilecopy>`. After a file viewer/editor is terminated, if the file was modified, can be sent back (just raw content, without any headers); otherwise service should just terminate without sending anything. This service is used by both ``qvm-open-in-vm`` and ``qvm-open-in-dvm`` tools. When called in DispVM, service termination will trigger DispVM cleanup.
- ``qubes.Restore`` - retrieve Qubes backup. The service receives backup location entered by the user (one line, terminated by ``\n``), then should output backup archive in :doc:`qfile format </developer/services/qfilecopy>` (core-agent-linux component contains ``tar2qfile`` utility to do the conversion)
- ``qubes.SelectDirectory``, ``qubes.SelectFile`` - services which should show file/directory selection dialog and return (to stdout) a single line containing selected path, or nothing in the case of cancellation
- ``qubes.SuspendPre`` - service called in every VM with PCI device attached just before system suspend
- ``qubes.SuspendPost`` - service called in every VM with PCI device attached just after system resume
- ``qubes.SyncNtpClock`` - service called to trigger network time synchronization. Service should synchronize local VM time and terminate when done.
- ``qubes.WindowIconUpdater`` - service called by VM to send icons of individual windows. The protocol there is simple one direction stream: VM sends window ID followed by icon in ``qubes.GetImageRGBA`` format, then next window ID etc. VM can send icon for the same window multiple times to replace previous one (for example for animated icons)
- ``qubes.VMShell`` - call any command in the VM; the command(s) is passed one per line
- ``qubes.VMShell+WaitForSession`` waits for full VM startup first
- ``qubes.VMExec`` - call any command in the VM, without using shell, the command needs to be passed as argument and encoded as follows:
- the executable name and arguments are separated by ``+``
- everything except alphanumeric characters, ``.`` and ``_`` needs to be escaped
- bytes are escaped as ``-HH`` (where ``HH`` is hex code, capital letters only)
- ``-`` itself can be escaped as ``--``
- example: to run ``ls -a /home/user``, use ``qubes.VMExec+ls+--a+-2Fhome-2Fuser``
- ``qubes.VMExecGUI`` - a variant of ``qubes.VMExec`` that waits for full VM startup first
Services called in GuiVM:
- `policy.Ask`, `policy.Notify` - confirmation prompt and notifications for
Qubes RPC calls, see [qrexec-policy implementation](/doc/qrexec-internals/#qrexec-policy-implementation)
for a detailed description.
- ``policy.Ask``, ``policy.Notify`` - confirmation prompt and notifications for Qubes RPC calls, see :ref:`qrexec-policy implementation <developer/services/qrexec-internals:\`\`qrexec-policy\`\` implementation>` for a detailed description.
Currently Qubes still calls few tools in VM directly, not using service abstraction. This will change in the future. Those tools are:
- ``/usr/lib/qubes/qubes-download-dom0-updates.sh`` - script to download updates (or new packages to be installed) for dom0 (``qubes-dom0-update`` tool)
- ``date -u -Iseconds`` - called directly to retrieve time after calling ``qubes.SyncNtpClock`` service (``qvm-sync-clock`` tool)
- ``nm-online -x`` - called before ``qubes.SyncNtpClock`` service call by ``qvm-sync-clock`` tool
- ``resize2fs`` - called to resize filesystem on /rw partition by ``qvm-grow-private`` tool
- ``gpk-update-viewer`` - called by Qubes Manager to display available updates in a TemplateVM
- ``systemctl start qubes-update-check.timer`` (and similarly stop) - called when enabling/disabling updates checking in given VM (``qubes-update-check`` :doc:`qvm-service </user/advanced-topics/qubes-service>`)
Currently Qubes still calls few tools in VM directly, not using service
abstraction. This will change in the future. Those tools are:
- `/usr/lib/qubes/qubes-download-dom0-updates.sh` - script to download updates (or new packages to be installed) for dom0 (`qubes-dom0-update` tool)
- `date -u -Iseconds` - called directly to retrieve time after calling `qubes.SyncNtpClock` service (`qvm-sync-clock` tool)
- `nm-online -x` - called before `qubes.SyncNtpClock` service call by `qvm-sync-clock` tool
- `resize2fs` - called to resize filesystem on /rw partition by `qvm-grow-private` tool
- `gpk-update-viewer` - called by Qubes Manager to display available updates in a TemplateVM
- `systemctl start qubes-update-check.timer` (and similarly stop) - called when enabling/disabling updates checking in given VM (`qubes-update-check` [qvm-service](/doc/qubes-service/))
Additionally, automatic tests extensively run various commands directly in VMs. We do not plan to change that.
## GUI protocol
GUI protocol
------------
GUI initialization includes passing the whole screen dimensions from dom0 to VM. This will most likely be overwritten by qubes.SetMonitorLayout Qubes RPC call.

176
developer/debugging/windows-debugging.rst
View file

@ -1,89 +1,131 @@
---
lang: en
layout: doc
permalink: /doc/windows-debugging/
redirect_from:
- /en/doc/windows-debugging/
- /doc/WindowsDebugging/
- /wiki/WindowsDebugging/
ref: 50
title: Windows debugging
---
=================
Windows debugging
=================
Debugging Windows code can be tricky in a virtualized environment. The guide below assumes Qubes 4.2 and Windows 7 or later VMs.
User-mode debugging is usually straightforward if it can be done on one machine. Just duplicate your normal debugging environment in the VM.
Things get complicated if you need to perform kernel debugging or troubleshoot problems that only manifest on system boot, user logoff or similar. For that you need two Windows VMs: the *host* and the *target*. The *host* will contain the debugger, your source code and private symbols. The *target* will run the code being debugged. We will use kernel debugging over network which is supported from Windows 7 onwards. The main caveat is that Windows kernel supports only specific network adapters for this, and the default one in Qubes won't work.
Things get complicated if you need to perform kernel debugging or troubleshoot problems that only manifest on system boot, user logoff or similar. For that you need two Windows VMs: the *host* and the *target*. The *host* will contain the debugger, your source code and private symbols. The *target* will run the code being debugged. We will use kernel debugging over network which is supported from Windows 7 onwards. The main caveat is that Windows kernel supports only specific network adapters for this, and the default one in Qubes won’t work.
## Important note
Important note
--------------
- Do not install Xen network PV drivers in the target VM. Network kernel debugging needs a specific type of NIC or it won't work, the network PV drivers interfere with that.
- If you have kernel debugging active when the Xen PV drivers are being installed, make sure to disable it before rebooting (`bcdedit /set debug off`). You can re-enable debugging after the reboot. The OS won't boot otherwise. I'm not sure what's the exact cause. I know that busparams for the debugging NIC change when PV drivers are installed (see later), but even changing that accordingly in the debug settings doesn't help -- so it's best to disable debug for this one reboot.
- Do not install Xen network PV drivers in the target VM. Network kernel debugging needs a specific type of NIC or it won’t work, the network PV drivers interfere with that.
## Modifying the NIC of the target VM
- If you have kernel debugging active when the Xen PV drivers are being installed, make sure to disable it before rebooting (``bcdedit /set debug off``). You can re-enable debugging after the reboot. The OS won’t boot otherwise. I’m not sure what’s the exact cause. I know that busparams for the debugging NIC change when PV drivers are installed (see later), but even changing that accordingly in the debug settings doesn’t help – so it’s best to disable debug for this one reboot.
You will need to create a custom libvirt config for the target VM. See [the documentation](https://dev.qubes-os.org/projects/core-admin/en/latest/libvirt.html) for overview of how libvirt templates work in Qubes. The following assumes the target VM is named `target-vm`.
- Edit `/usr/share/qubes/templates/libvirt/xen.xml` to prepare our custom config to override just the NIC part of the global template:
- add `{% raw %}{% block network %}{% endraw %}` before `{% raw %}{% if vm.netvm %}{% endraw %}`
- add `{% raw %}{% endblock %}{% endraw %}` after the matching `{% raw %}{% endif %}{% endraw %}`
- Copy `/usr/share/qubes/templates/libvirt/devices/net.xml` to `/etc/qubes/templates/libvirt/xen/by-name/target-vm.xml`.
- Add `<model type='e1000'/>` to the `<interface>` section.
- Enclose everything within `{% raw %}{% block network %}{% endraw %}` + `{% raw %}{% endblock %}{% endraw %}`.
- Add `{% raw %}{% extends 'libvirt/xen.xml' %}{% endraw %}` at the start.
- The final `target-vm.xml` should look something like this:
~~~
{% raw %}
{% extends 'libvirt/xen.xml' %}
{% block network %}
<interface type='ethernet'>
<mac address="{{ vm.mac }}" />
<ip address="{{ vm.ip }}" />
<backenddomain name="{{ vm.netvm.name }}" />
<script path='vif-route-qubes' />
<model type='e1000' />
</interface>
{% endblock %}
{% endraw %}
~~~
Modifying the NIC of the target VM
----------------------------------
- Start `target-vm` and verify in the device manager that a "Intel PRO/1000 MT" adapter is present.
## Host and target preparation
You will need to create a custom libvirt config for the target VM. See `the documentation <https://dev.qubes-os.org/projects/core-admin/en/latest/libvirt.html>`__ for overview of how libvirt templates work in Qubes. The following assumes the target VM is named ``target-vm``.
- Edit ``/usr/share/qubes/templates/libvirt/xen.xml`` to prepare our custom config to override just the NIC part of the global template:
- add ``{% block network %}`` before ``{% if vm.netvm %}``
- add ``{% endblock %}`` after the matching ``{% endif %}``
- Copy ``/usr/share/qubes/templates/libvirt/devices/net.xml`` to ``/etc/qubes/templates/libvirt/xen/by-name/target-vm.xml``.
- Add ``<model type='e1000'/>`` to the ``<interface>`` section.
- Enclose everything within ``{% block network %}`` + ``{% endblock %}``.
- Add ``{% extends 'libvirt/xen.xml' %}`` at the start.
- The final ``target-vm.xml`` should look something like this:
.. code:: bash
{% extends 'libvirt/xen.xml' %}
{% block network %}
<interface type='ethernet'>
<mac address="{{ vm.mac }}" />
<ip address="{{ vm.ip }}" />
<backenddomain name="{{ vm.netvm.name }}" />
<script path='vif-route-qubes' />
<model type='e1000' />
</interface>
{% endblock %}
- Start ``target-vm`` and verify in the device manager that a “Intel PRO/1000 MT” adapter is present.
Host and target preparation
---------------------------
- On ``host-vm`` you will need WinDbg, which is a part of the `Windows SDK <https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/>`__.
- Copy the ``Debuggers`` directory from Windows SDK to ``target-vm``.
- In both ``host-vm`` and ``target-vm`` switch the windows network to private (it tends to be public by default).
- On `host-vm` you will need WinDbg, which is a part of the [Windows SDK](https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/).
- Copy the `Debuggers` directory from Windows SDK to `target-vm`.
- In both `host-vm` and `target-vm` switch the windows network to private (it tends to be public by default).
- Either turn off the windows firewall or enable all ICMP-in rules in both VMs.
- In `firewall-vm` edit `/rw/config/qubes-firewall-user-script` to connect both Windows VMs, add:
- `iptables -I FORWARD 2 -s <target-vm-ip> -d <host-vm-ip> -j ACCEPT`
- `iptables -I FORWARD 2 -s <host-vm-ip> -d <target-vm-ip> -j ACCEPT`
- run `/rw/config/qubes-firewall-user-script` so the changes take effect immediately
- In ``firewall-vm`` edit ``/rw/config/qubes-firewall-user-script`` to connect both Windows VMs, add:
- ``iptables -I FORWARD 2 -s <target-vm-ip> -d <host-vm-ip> -j ACCEPT``
- ``iptables -I FORWARD 2 -s <host-vm-ip> -d <target-vm-ip> -j ACCEPT``
- run ``/rw/config/qubes-firewall-user-script`` so the changes take effect immediately
- Make sure both VMs can ping each other.
- In `target-vm`:
- start elevated `cmd` session
- `cd sdk\Debuggers\x64`
- `kdnet` should show that the NIC is supported, note the busparams:
~~~
Network debugging is supported on the following NICs:
busparams=0.6.0, Intel(R) PRO/1000 MT Network Connection, KDNET is running on this NIC.
~~~
- In ``target-vm``:
- start elevated ``cmd`` session
- ``cd sdk\Debuggers\x64``
- ``kdnet`` should show that the NIC is supported, note the busparams:
.. code:: bash
Network debugging is supported on the following NICs:
busparams=0.6.0, Intel(R) PRO/1000 MT Network Connection, KDNET is running on this NIC.
- ``bcdedit /debug on``
- ``bcdedit /dbgsettings net hostip:<host-vm-ip> port:50000 key:1.1.1.1`` (you can customize the key)
- ``bcdedit /set "{dbgsettings}" busparams x.y.z`` (use the busparams ``kdnet`` has shown earlier)
- In ``host-vm`` start WinDbg: ``windbg -k net:port=50000,key=1.1.1.1``. It will listen for target’s connection.
- Reboot ``target-vm``, debugging should start:
.. code:: bash
Waiting to reconnect...
Connected to target 10.137.0.19 on port 50000 on local IP 10.137.0.20.
You can get the target MAC address by running .kdtargetmac command.
Connected to Windows 10 19041 x64 target at (Thu Aug 3 14:05:48.069 2023 (UTC + 2:00)), ptr64 TRUE
- `bcdedit /debug on`
- `bcdedit /dbgsettings net hostip:<host-vm-ip> port:50000 key:1.1.1.1` (you can customize the key)
- `bcdedit /set "{dbgsettings}" busparams x.y.z` (use the busparams `kdnet` has shown earlier)
- In `host-vm` start WinDbg: `windbg -k net:port=50000,key=1.1.1.1`. It will listen for target's connection.
- Reboot `target-vm`, debugging should start:
~~~
Waiting to reconnect...
Connected to target 10.137.0.19 on port 50000 on local IP 10.137.0.20.
You can get the target MAC address by running .kdtargetmac command.
Connected to Windows 10 19041 x64 target at (Thu Aug 3 14:05:48.069 2023 (UTC + 2:00)), ptr64 TRUE
~~~
Happy debugging!

47
developer/general/continuous-integration.rst
View file

@ -1,39 +1,22 @@
---
lang: en
layout: doc
permalink: /doc/continuous-integration/
title: Continuous integration (CI)
---
===========================
Continuous integration (CI)
===========================
This page explains the [continuous integration
(CI)](https://en.wikipedia.org/wiki/Continuous_integration) infrastructure that
the Qubes OS Project uses.
## Website and documentation
This page explains the `continuous integration (CI) <https://en.wikipedia.org/wiki/Continuous_integration>`__ infrastructure that the Qubes OS Project uses.
The following commands may be useful as a way to interact with our CI
infrastructure on website
([qubesos.github.io](https://github.com/QubesOS/qubesos.github.io)) and
documentation ([qubes-doc](https://github.com/QubesOS/qubes-doc)) pull requests
(PRs). Note that special permissions may be required to use some of these
commands. These commands are generally issued by adding a comment to a PR
containing only the command.
Website and documentation
-------------------------
- `PipelineRetry`: Attempts to run the entire build pipeline over again. This
can be useful if CI incorrectly uses a stale branch instead of testing the PR
as if it were merged into `master`.
- `PipelineRetryFailed`: Retry just failed CI jobs. Do not update the branch.
The following commands may be useful as a way to interact with our CI infrastructure on website (`qubesos.github.io <https://github.com/QubesOS/qubesos.github.io>`__) and documentation (`qubes-doc <https://github.com/QubesOS/qubes-doc>`__) pull requests (PRs). Note that special permissions may be required to use some of these commands. These commands are generally issued by adding a comment to a PR containing only the command.
- ``PipelineRetry``: Attempts to run the entire build pipeline over again. This can be useful if CI incorrectly uses a stale branch instead of testing the PR as if it were merged into ``master``.
- ``PipelineRetryFailed``: Retry just failed CI jobs. Do not update the branch.
- ``PipelineRefresh``: Like ``PipelineRetry``, except it only fetches the job status from GitLab. It doesn’t schedule a new build.
- ``TestDeploy``: Deploys a test website, which is a live version of the Qubes website as if this PR had been merged. This can be useful for previewing a PR on a live public website. **Note:** You must wait for the site to finish building before issuing this command, or else it will deploy an empty website. To find the URL of the test website, look for text similar to “This branch was successfully deployed” and a button named something like “View deployment.” Note that there are two different testing sites: ``wwwtest`` is manually updated, whereas ``wwwpreview`` is managed by the ``TestDeploy`` command.
- `PipelineRefresh`: Like `PipelineRetry`, except it only fetches the job status
from GitLab. It doesn't schedule a new build.
- `TestDeploy`: Deploys a test website, which is a live version of the Qubes
website as if this PR had been merged. This can be useful for previewing a PR
on a live public website. **Note:** You must wait for the site to finish
building before issuing this command, or else it will deploy an empty
website. To find the URL of the test website, look for text similar to "This
branch was successfully deployed" and a button named something like "View
deployment." Note that there are two different testing sites: `wwwtest` is
manually updated, whereas `wwwpreview` is managed by the `TestDeploy`
command.

49
developer/general/devel-books.rst
View file

@ -1,31 +1,42 @@
---
lang: en
layout: doc
permalink: /doc/devel-books/
redirect_from:
- /en/doc/devel-books/
- /doc/DevelBooks/
- /wiki/DevelBooks/
ref: 32
title: Developer books
---
===============
Developer books
===============
Below is a list of various books that might be useful in learning some basics needed for Qubes development.
- A must-read about Xen internals:
- _[The Definitive Guide to the Xen Hypervisor](https://www.amazon.com/Definitive-Guide-Xen-Hypervisor/dp/013234971X)_, by David Chisnall
- `The Definitive Guide to the Xen Hypervisor <https://www.amazon.com/Definitive-Guide-Xen-Hypervisor/dp/013234971X>`__, by David Chisnall
- Some good books about the Linux kernel:
- _[Linux Kernel Development](https://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468)_, by Robert Love
- _[Linux Device Drivers](https://www.amazon.com/Linux-Device-Drivers-Jonathan-Corbet/dp/0596005903)_, by Jonathan Corbet
- `Linux Kernel Development <https://www.amazon.com/Linux-Kernel-Development-Robert-Love/dp/0672329468>`__, by Robert Love
- `Linux Device Drivers <https://www.amazon.com/Linux-Device-Drivers-Jonathan-Corbet/dp/0596005903>`__, by Jonathan Corbet
- Solid intro into Trusted Computing:
- _[Dynamics of a Trusted Platform](https://www.amazon.com/Dynamics-Trusted-Platform-Buildin-Grawrock/dp/1934053082)_, by David Grawrock (original Intel architect for TXT)
- `Dynamics of a Trusted Platform <https://www.amazon.com/Dynamics-Trusted-Platform-Buildin-Grawrock/dp/1934053082>`__, by David Grawrock (original Intel architect for TXT)
- Good book about GIT:
- _[Pro Git](https://git-scm.com/book/en/v2)_, by Scott Chacon and Ben Straub (complete book available free online)
- `Pro Git <https://git-scm.com/book/en/v2>`__, by Scott Chacon and Ben Straub (complete book available free online)
- Useful books about Python:
- _[Programming in Python 3](https://www.qtrac.eu/py3book.html)_, by Mark Summerfield
- _[Rapid GUI Programming with Python and Qt](https://www.qtrac.eu/pyqtbook.html)_, by Mark Summerfield
(Although note that [Qt is being replaced by GTK](/doc/usability-ux/#gnome-kde-and-xfce) in Qubes code.)
- `Programming in Python 3 <https://www.qtrac.eu/py3book.html>`__, by Mark Summerfield
- `Rapid GUI Programming with Python and Qt <https://www.qtrac.eu/pyqtbook.html>`__, by Mark Summerfield (Although note that :ref:`Qt is being replaced by GTK <developer/general/usability-ux:gnome, kde, and xfce>` in Qubes code.)

133
developer/general/developing-gui-applications.rst
View file

@ -1,102 +1,117 @@
---
lang: en
layout: doc
permalink: /doc/developing-gui-applications/
ref: 333
title: Developing Qubes OS GUI tools
---
=============================
Developing Qubes OS GUI tools
=============================
In order to avoid installing Qubes OS frontend tools you are working on in your own `dom0` or just to test them with less problems, you can use the mock Qubes object from the `qubesadmin` package.
## Running programs using mock Qubes object
In order to avoid installing Qubes OS frontend tools you are working on in your own ``dom0`` or just to test them with less problems, you can use the mock Qubes object from the ``qubesadmin`` package.
Where you would normally provide the Qubes object, use the `qubesadmin.tests.mock_app` package and one of the mock Qubes objects from it.
Running programs using mock Qubes object
----------------------------------------
For example, the following code can be used to run the `qui-domains` tool using the mock Qubes object (this code would replace the initial part of the main function):
```python
def main():
''' main function '''
# qapp = qubesadmin.Qubes()
# dispatcher = qubesadmin.events.EventsDispatcher(qapp)
# stats_dispatcher = qubesadmin.events.EventsDispatcher(qapp, api_method='admin.vm.Stats')
Where you would normally provide the Qubes object, use the ``qubesadmin.tests.mock_app`` package and one of the mock Qubes objects from it.
import qubesadmin.tests.mock_app as mock_app
qapp = mock_app.MockQubesComplete()
dispatcher = mock_app.MockDispatcher(qapp)
stats_dispatcher = mock_app.MockDispatcher(
qapp, api_method='admin.vm.Stats')
For example, the following code can be used to run the ``qui-domains`` tool using the mock Qubes object (this code would replace the initial part of the main function):
.. code:: python
def main():
''' main function '''
# qapp = qubesadmin.Qubes()
# dispatcher = qubesadmin.events.EventsDispatcher(qapp)
# stats_dispatcher = qubesadmin.events.EventsDispatcher(qapp, api_method='admin.vm.Stats')
import qubesadmin.tests.mock_app as mock_app
qapp = mock_app.MockQubesComplete()
dispatcher = mock_app.MockDispatcher(qapp)
stats_dispatcher = mock_app.MockDispatcher(
qapp, api_method='admin.vm.Stats')
# continue as normal
# continue as normal
```
To run a mocked program without installing it in a qube, remember to extend PYTHONPATH appropriately, for example:
```bash
~/qubes-sources/manager $ PYTHONPATH=../core-admin-client:. python3 qui/tray/domains.py
```
.. code:: bash
~/qubes-sources/manager $ PYTHONPATH=../core-admin-client:. python3 qui/tray/domains.py
The mock object does not provide events (yet).
Note: in order to see all qubes-relevant icons (like VM icons), install the `qubes-artwork` package.
**Note:** in order to see all qubes-relevant icons (like VM icons), install the ``qubes-artwork`` package.
How does it actually work
-------------------------
## How does it actually work
The mock Qubes object has a collection of expected Qubes RPC calls and the responses that a real system would provide. Writing these calls manually is a bit tedious, given that most frontend tools query a lot of qube properties. For example, on a medium-sized system, initializing Qube Manager involves about 300 separate RPC calls.
If you need more calls, you can add them to the mock object using the following syntax (the following example adds listing available vm kernels):
```python
mock_app.expected_calls[('dom0', 'admin.pool.volume.List', 'linux-kernel', None)] = \
b'0\x006.1.57-1.fc37\n6.1.43-1.fc37\ncustom_kernel\n'
.. code:: python
mock_app.expected_calls[('dom0', 'admin.pool.volume.List', 'linux-kernel', None)] = \
b'0\x006.1.57-1.fc37\n6.1.43-1.fc37\ncustom_kernel\n'
```
If error should be thrown, you need to provide the error code and name, for example:
```python
mock_app.expected_calls[("vmname", "admin.vm.property.Get", "property_name", None)] = \
b'2\x00QubesNoSuchPropertyError\x00\x00No such property\x00'
```
.. code:: python
For details of particular calls, you can use [Extending the mock Qubes object](#extending-the-mock-qubes-object).
mock_app.expected_calls[("vmname", "admin.vm.property.Get", "property_name", None)] = \
b'2\x00QubesNoSuchPropertyError\x00\x00No such property\x00'
## Available mocks
For details of particular calls, you can use `Extending the mock Qubes object <#extending-the-mock-qubes-object>`__.
Three mocks are available in the `mock_app` file:
* MockQubes, an extremely bare-bones Qubes testing instance, with just dom0, sys-net, and one template (fedora-36).
* MockQubesComplete, a more complex setup [![Qubes Manager running MockQubesComplete](/attachment/doc/doc-mock-app-ex1.png)](/attachment/doc/doc-mock-app-ex1.png)
* MockQubesWhonix, the setup above extended with several Whonix-related qubes
Available mocks
---------------
## Extending the mock Qubes object
Three mocks are available in the ``mock_app`` file:
- MockQubes, an extremely bare-bones Qubes testing instance, with just dom0, sys-net, and one template (fedora-36).
- MockQubesComplete, a more complex setup |Qubes Manager running MockQubesComplete|
- MockQubesWhonix, the setup above extended with several Whonix-related qubes
Extending the mock Qubes object
-------------------------------
To collect information to modify this script, you can use the wrapper function to wrap and output all qubesd calls used by a program running on a live qubes instance.
```python
qapp = qubesadmin.Qubes()
import qubesadmin.tests.mock_app as mock_app
qapp.qubesd_call = mock_app.wrapper(qapp.qubesd_call)
qapp._parse_qubesd_response = mock_app.wrapper(qapp._parse_qubesd_response)
```
.. code:: python
qapp = qubesadmin.Qubes()
import qubesadmin.tests.mock_app as mock_app
qapp.qubesd_call = mock_app.wrapper(qapp.qubesd_call)
qapp._parse_qubesd_response = mock_app.wrapper(qapp._parse_qubesd_response)
Writing tests
-------------
## Writing tests
The same mock Qubes can also be used to write tests. You can use the wrappers above to check which calls are made when certain actions are performed, and add them to the mock objects in the following way:
```python
# this is an excerpt from tests for Qubes Global Config tool
clockvm_combo.set_active_id('test-blue')
.. code:: python
mock_qapp.expected_calls[('dom0', 'admin.property.Set',
'clockvm', b'test-blue')] = b'0\x00'
basics_handler.save()
# this is an excerpt from tests for Qubes Global Config tool
clockvm_combo.set_active_id('test-blue')
mock_qapp.expected_calls[('dom0', 'admin.property.Set',
'clockvm', b'test-blue')] = b'0\x00'
basics_handler.save()
```
If the call is made correctly, the test will continue successfully; if an unexpected call is made, the test will fail.
Caution: the mock Qubes object does not react to changes like a normal Qubes object does. Further queries to the test object will continue to return initial values.
.. |Qubes Manager running MockQubesComplete| image:: /attachment/doc/doc-mock-app-ex1.png

355
developer/general/documentation-style-guide.md
View file

@ -1,355 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/documentation-style-guide/
redirect_from:
- /doc/doc-guidelines/
- /en/doc/doc-guidelines/
- /wiki/DocStyle/
- /doc/DocStyle/
ref: 30
title: Documentation style guide
---
_Also see [how to edit the documentation](/doc/how-to-edit-the-documentation/)._
Qubes OS documentation pages are stored as plain text Markdown files in the [qubes-doc](https://github.com/QubesOS/qubes-doc) repository. By cloning and regularly pulling from this repo, users can maintain their own up-to-date offline copy of all Qubes documentation rather than relying solely on the web.
The documentation is a volunteer community effort. People like you are constantly working to make it better. If you notice something that can be fixed or improved, please [edit the documentation](/doc/how-to-edit-the-documentation/)!
This page explains the standards we follow for writing, formatting, and organizing the documentation. Please follow these guidelines and conventions when editing the documentation. For the standards governing the website as a whole, please see the [website style guide](/doc/website-style-guide).
## Markdown conventions
All the documentation is written in Markdown for maximum accessibility. When making contributions, please observe the following style conventions. If you're not familiar with Markdown syntax, [this](https://daringfireball.net/projects/markdown/) is a great resource.
### Hyperlink syntax
Use non-reference-style links like `[website](https://example.com/)`. Do *not* use reference-style links like `[website][example]`, `[website][]` or `[website]`. This facilitates the localization process.
### Relative vs. absolute links
Always use relative rather than absolute paths for internal website links. For example, use `/doc/documentation-style-guide/` instead of `https://www.qubes-os.org/doc/documentation-style-guide/`.
You may use absolute URLs in the following cases:
- External links
- URLs that appear inside code blocks (e.g., in comments and document templates, and the plain text reproductions of [QSBs](/security/qsb/) and [Canaries](/security/canary/)), since they're not hyperlinks
- Git repo files like `README.md` and `CONTRIBUTING.md`, since they're not part of the website itself but rather of the auxiliary infrastructure supporting the website
This rule is important because using absolute URLs for internal website links breaks:
- Serving the website offline
- Website localization
- Generating offline documentation
- Automatically redirecting Tor Browser visitors to the correct page on the onion service mirror
### Image linking
See [how to add images](/doc/how-to-edit-the-documentation/#how-to-add-images) for the required syntax. This will make the image a hyperlink to the image file, allowing the reader to click on the image in order to view the full image by itself. This is important. Following best practices, our website has a responsive design, which allows the website to render appropriately across all screen sizes. When viewing this page on a smaller screen, such as on a mobile device, the image will automatically shrink down to fit the screen. If visitors cannot click on the image to view it in full size, then, depending on their device, they may have no way see the details in the image clearly.
In addition, make sure to link only to images in the [qubes-attachment](https://github.com/QubesOS/qubes-attachment) repository. Do not attempt to link to images hosted on other websites.
### HTML and CSS
Do not write HTML inside Markdown documents (except in rare, unavoidable cases, such as [alerts](#alerts)). In particular, never include HTML or CSS for styling, formatting, or white space control. That belongs in the (S)CSS files instead.
### Headings
Do not use `h1` headings (single `#` or `======` underline). These are automatically generated from the `title:` line in the YAML front matter.
Use Atx-style syntax for headings: `##h2`, `### h3`, etc. Do not use underlining syntax (`-----`).
### Indentation
Use spaces instead of tabs. Use hanging indentations where appropriate.
### Lists
If appropriate, make numerals in numbered lists match between Markdown source and HTML output. Some users read the Markdown source directly, and this makes numbered lists easier to follow.
### Code blocks
When writing code blocks, use [syntax highlighting](https://github.github.com/gfm/#info-string) where possible (see [here](https://github.com/jneen/rouge/wiki/List-of-supported-languages-and-lexers) for a list of supported languages). Use `[...]` for anything omitted.
### Line wrapping
Do not hard wrap text, except where necessary (e.g., inside code blocks).
### Do not use Markdown syntax for styling
For example, there is a common temptation to use block quotations (created by beginning lines with the `>` character) in order to stylistically distinguish some portion of text from the rest of the document, e.g.:
```
> Note: This is an important note!
```
This renders as:
> Note: This is an important note!
There are two problems with this:
1. It is a violation of the [separation of content and presentation](https://en.wikipedia.org/wiki/Separation_of_content_and_presentation), since it abuses markup syntax in order to achieve unintended stylistic results. The Markdown (and HTML, if any) should embody the *content* of the documentation, while the *presentation* is handled by (S)CSS.
2. It is an abuse of quotation syntax for text that is not actually a quotation. (You are not quoting anyone here. You're just telling the reader to note something and trying to draw their attention to your note visually.)
Instead, an example of an appropriate way to stylistically distinguish a portion of text is by using [alerts](#alerts). Consider also that extra styling and visual distinction may not even be necessary. In most cases, traditional writing methods are perfectly sufficient, e.g.,:
```
**Note:** This is an important note.
```
This renders as:
**Note:** This is an important note.
### Alerts
Alerts are sections of HTML used to draw the reader's attention to important information, such as warnings, and for stylistic purposes. They are typically styled as colored text boxes, usually accompanied by icons. Alerts should generally be used somewhat sparingly, so as not to cause [alert fatigue](https://en.wikipedia.org/wiki/Alarm_fatigue) and since they must be written in HTML instead of Markdown, which makes the source less readable and more difficult to work with for localization and automation purposes. Here are examples of several types of alerts and their recommended icons:
```
<div class="alert alert-success" role="alert">
<i class="fa fa-check-circle"></i>
<b>Did you know?</b> The Qubes OS installer is completely offline. It doesn't
even load any networking drivers, so there is no possibility of
internet-based data leaks or attacks during the installation process.
</div>
<div class="alert alert-info" role="alert">
<i class="fa fa-info-circle"></i>
<b>Note:</b> Using Rufus to create the installation medium means that you <a
href="https://github.com/QubesOS/qubes-issues/issues/2051">won't be able</a>
to choose the "Test this media and install Qubes OS" option mentioned in the
example below. Instead, choose the "Install Qubes OS" option.
</div>
<div class="alert alert-warning" role="alert">
<i class="fa fa-exclamation-circle"></i>
<b>Note:</b> Qubes OS is not meant to be installed inside a virtual machine
as a guest hypervisor. In other words, <b>nested virtualization</b> is not
supported. In order for a strict compartmentalization to be enforced, Qubes
OS needs to be able to manage the hardware directly.
</div>
<div class="alert alert-danger" role="alert">
<i class="fa fa-exclamation-triangle"></i>
<b>Warning:</b> Qubes has no control over what happens on your computer
before you install it. No software can provide security if it is installed on
compromised hardware. Do not install Qubes on a computer you don't trust. See
<a href="/doc/install-security/">installation security</a> for more
information.
</div>
```
These render as:
<div class="alert alert-success" role="alert">
<i class="fa fa-check-circle"></i>
<b>Did you know?</b> The Qubes OS installer is completely offline. It doesn't
even load any networking drivers, so there is no possibility of
internet-based data leaks or attacks during the installation process.
</div>
<div class="alert alert-info" role="alert">
<i class="fa fa-info-circle"></i>
<b>Note:</b> Using Rufus to create the installation medium means that you <a
href="https://github.com/QubesOS/qubes-issues/issues/2051">won't be able</a>
to choose the "Test this media and install Qubes OS" option mentioned in the
example below. Instead, choose the "Install Qubes OS" option.
</div>
<div class="alert alert-warning" role="alert">
<i class="fa fa-exclamation-circle"></i>
<b>Note:</b> Qubes OS is not meant to be installed inside a virtual machine
as a guest hypervisor. In other words, <b>nested virtualization</b> is not
supported. In order for a strict compartmentalization to be enforced, Qubes
OS needs to be able to manage the hardware directly.
</div>
<div class="alert alert-danger" role="alert">
<i class="fa fa-exclamation-triangle"></i>
<b>Warning:</b> Qubes has no control over what happens on your computer
before you install it. No software can provide security if it is installed on
compromised hardware. Do not install Qubes on a computer you don't trust. See
<a href="/doc/install-security/">installation security</a> for more
information.
</div>
## Writing guidelines
### Correct use of terminology
Familiarize yourself with the terms defined in the [glossary](/doc/glossary/). Use these terms consistently and accurately throughout your writing.
### Sentence case in headings
Use sentence case (rather than title case) in headings for the reasons explained [here](https://www.sallybagshaw.com.au/articles/sentence-case-v-title-case/). In particular, since the authorship of the Qubes documentation is decentralized and widely distributed among users from around the world, many contributors come from regions with different conventions for implementing title case, not to mention that there are often differing style guide recommendations even within a single region. It is much easier for all of us to implement sentence case consistently across our growing body of pages, which is very important for managing the ongoing maintenance burden and sustainability of the documentation.
### Writing command-line examples
When providing command-line examples:
- Tell the reader where to open a terminal (dom0 or a specific domU), and show the command along with its output (if any) in a code block, e.g.:
~~~markdown
Open a terminal in dom0 and run:
```shell_session
$ cd test
$ echo Hello
Hello
```
~~~
- Precede each command with the appropriate command prompt: At a minimum, the prompt should contain a trailing `#` (for the user `root`) or `$` (for other users) on Linux systems and `>` on Windows systems, respectively.
- Don't try to add comments inside the code block. For example, *don't* do this:
~~~markdown
Open a terminal in dom0 and run:
```shell_session
# Navigate to the new directory
$ cd test
# Generate a greeting
$ echo Hello
Hello
```
~~~
The `#` symbol preceding each comment is ambiguous with a root command prompt. Instead, put your comments *outside* of the code block in normal prose.
### Variable names in commands
Syntactically distinguish variables in commands. For example, this is ambiguous:
$ qvm-run --dispvm=disposable-template --service qubes.StartApp+xterm
It should instead be:
$ qvm-run --dispvm=<DISPOSABLE_TEMPLATE> --service qubes.StartApp+xterm
Note that we syntactically distinguish variables in three ways:
1. Surrounding them in angled brackets (`< >`)
2. Using underscores (`_`) instead of spaces between words
3. Using all capital letters
We have observed that many novices make the mistake of typing the surrounding angled brackets (`< >`) on the command line, even after substituting the desired real value between them. Therefore, in documentation aimed at novices, we also recommend clarifying that the angled brackets should not be typed. This can be accomplished in one of several ways:
- Explicitly say something like "without the angled brackets."
- Provide an example command using real values that excludes the angled brackets.
- If you know that almost all users will want to use (or should use) a specific command containing all real values and no variables, you might consider providing exactly that command and forgoing the version with variables. Novices may not realize which parts of the command they can substitute with different values, but if you've correctly judged that they should use the command you've provided as is, then this shouldn't matter.
### Capitalization of "qube"
We introduced the term ["qube"](/doc/glossary/#qube) as a user-friendly alternative to the term ["virtual machine" ("VM")](/doc/glossary/#vm) in the context of Qubes OS. Nonetheless, "qube" is a common noun like the words "compartment" and "container." Therefore, in English, "qube" follows the standard capitalization rules for common nouns. For example, "I have three qubes" is correct, while "I have three Qubes" is incorrect. Like other common nouns, "qube" should still be capitalized at the beginnings of sentences, the beginnings of sentence-case headings, and in title-case headings. Note, however, that starting a sentence with the plural of "qube" (e.g., "Qubes can be shut down...") can be ambiguous, since it may not be clear whether the referent is a plurality of qubes, [Qubes OS](/doc/glossary/#qubes-os), or even the Qubes OS Project itself. Hence, it is generally a good idea to rephrase such sentences in order to avoid this ambiguity.
Many people feel a strong temptation to capitalize the word "qube" all the time, like a proper noun, perhaps because it's a new and unfamiliar term that's closely associated with a particular piece of software (namely, Qubes OS). However, these factors are not relevant to the capitalization rules of English. In fact, it's not unusual for new common nouns to be introduced into English, especially in the context of technology. For example, "blockchain" is a relatively recent technical term that's a common noun. Why is it a common noun rather than a proper noun? Because proper nouns refer to *particular* people, places, things, and ideas. There are many different blockchains. However, even when there was just one, the word still denoted a collection of things rather than a particular thing. It happened to be the case that there was only one member in that collection at the time. For example, if there happened to be only one tree in the world, that wouldn't change the way we capitalize sentences like, "John sat under a tree." Intuitively, it makes sense that the addition and removal of objects from the world shouldn't cause published books to become orthographicallly incorrect while sitting on their shelves.
Accordingly, the reason "qube" is a common noun rather than a proper noun is because it doesn't refer to any one specific thing (in this case, any one specific virtual machine). Rather, it's the term for any virtual machine in a Qubes OS installation. (Technically, while qubes are currently implemented as virtual machines, Qubes OS is independent of its underlying compartmentalization technology. Virtual machines could be replaced with a different technology, and qubes would still be called "qubes.")
I have several qubes in my Qubes OS installation, and you have several in yours. Every Qubes OS user has their own set of qubes, just as each of us lives in some neighborhood on some street. Yet we aren't tempted to treat words like "neighborhood" or "street" as proper nouns (unless, of course, they're part of a name, like "Acorn Street"). Again, while this might seem odd because "qube" is a new word that we invented, that doesn't change how English works. After all, *every* word was a new word that someone invented at some point (otherwise we wouldn't have any words at all). We treat "telephone," "computer," "network," "program," and so on as common nouns, even though those were all new technological inventions in the not-too-distant past (on a historical scale, at least). So, we shouldn't allow ourselves to be confused by irrelevant factors, like the fact that the inventors happened to be *us* or that the invention was *recent* or is not in widespread use among humanity.
### English language conventions
For the sake of consistency and uniformity, the Qubes documentation aims to follow the conventions of American English, where applicable. (Please note that this is an arbitrary convention for the sake consistency and not a value judgment about the relative merits of British versus American English.)
## Organizational guidelines
### Do not duplicate documentation
Duplicating documentation is almost always a bad idea. There are many reasons for this. The main one is that almost all documentation has to be updated as some point. When similar documentation appears in more than one place, it is very easy for it to get updated in one place but not the others (perhaps because the person updating it doesn't realize it's in more than once place). When this happens, the documentation as a whole is now inconsistent, and the outdated documentation becomes a trap, especially for novice users. Such traps are often more harmful than if the documentation never existed in the first place. The solution is to **link** to existing documentation rather than duplicating it. There are some exceptions to this policy (e.g., information that is certain not to change for a very long time), but they are rare.
### Core vs. external documentation
Core documentation resides in the [Qubes OS Project's official repositories](https://github.com/QubesOS/), mainly in [qubes-doc](https://github.com/QubesOS/qubes-doc). External documentation can be anywhere else (such as forums, community websites, and blogs), but there is an especially large collection in the [Qubes Forum](https://forum.qubes-os.org/docs). External documentation should not be submitted to [qubes-doc](https://github.com/QubesOS/qubes-doc). If you've written a piece of documentation that is not appropriate for [qubes-doc](https://github.com/QubesOS/qubes-doc), we encourage you to submit it to the [Qubes Forum](https://forum.qubes-os.org/docs) instead. However, *linking* to external documentation from [qubes-doc](https://github.com/QubesOS/qubes-doc) is perfectly fine. Indeed, the maintainers of the [Qubes Forum](https://forum.qubes-os.org/) should regularly submit PRs against the documentation index (see [How to edit the documentation index](/doc/how-to-edit-the-documentation/#how-to-edit-the-documentation-index)) to add and update Qubes Forum links in the ["External documentation"](/doc/#external-documentation) section of the documentation table of contents.
The main difference between **core** (or **official**) and **external** (or **community** or **unofficial**) documentation is whether it documents software that is officially written and maintained by the Qubes OS Project. The purpose of this distinction is to keep the core docs maintainable and high-quality by limiting them to the software output by the Qubes OS Project. In other words, we take responsibility for documenting all of the software we put out into the world, but it doesn't make sense for us to take on the responsibility of documenting or maintaining documentation for anything else. For example, Qubes OS may use a popular Linux distribution for an official [TemplateVM](/doc/templates/). However, it would not make sense for a comparatively small project like ours, with modest funding and a lean workforce, to attempt to document software belonging to a large, richly-funded project with an army of paid and volunteer contributors, especially when they probably already have documentation of their own. This is particularly true when it comes to Linux in general. Although many users who are new to Qubes are also new to Linux, it makes absolutely no sense for our comparatively tiny project to try to document Linux in general when there is already a plethora of documentation out there.
Many contributors do not realize that there is a significant amount of work involved in *maintaining* documentation after it has been written. They may wish to write documentation and submit it to the core docs, but they see only their own writing process and fail to consider that it will have to be kept up-to-date and consistent with the rest of the docs for years afterward. Submissions to the core docs also have to [undergo a review process](/doc/how-to-edit-the-documentation/#security) to ensure accuracy before being merged, which takes up valuable time from the team. We aim to maintain high quality standards for the core docs (style and mechanics, formatting), which also takes up a lot of time. If the documentation involves anything external to the Qubes OS Project (such as a website, platform, program, protocol, framework, practice, or even a reference to a version number), the documentation is likely to become outdated when that external thing changes. It's also important to periodically review and update this documentation, especially when a new Qubes release comes out. Periodically, there may be technical or policy changes that affect all the core documentation. The more documentation there is relative to maintainers, the harder all of this will be. Since there are many more people who are willing to write documentation than to maintain it, these individually small incremental additions amount to a significant maintenance burden for the project.
On the positive side, we consider the existence of community documentation to be a sign of a healthy ecosystem, and this is quite common in the software world. The community is better positioned to write and maintain documentation that applies, combines, and simplifies the official documentation, e.g., tutorials that explain how to install and use various programs in Qubes, how to create custom VM setups, and introductory tutorials that teach basic Linux concepts and commands in the context of Qubes. In addition, just because the Qubes OS Project has officially written and maintains some flexible framework, such as `qrexec`, it does not make sense to include every tutorial that says "here's how to do something cool with `qrexec`" in the core docs. Such tutorials generally also belong in the community documentation.
See [#4693](https://github.com/QubesOS/qubes-issues/issues/4693) for more background information.
### Release-specific documentation
*See [#5308](https://github.com/QubesOS/qubes-issues/issues/5308) for pending changes to this policy.*
We maintain only one set of documentation for Qubes OS. We do not maintain a different set of documentation for each release of Qubes. Our single set of Qubes OS documentation is updated on a continual, rolling basis. Our first priority is to document all **current, stable releases** of Qubes. Our second priority is to document the next, upcoming release (if any) that is currently in the beta or release candidate stage.
In cases where a documentation page covers functionality that differs considerably between Qubes OS releases, the page should be subdivided into clearly-labeled sections that cover the different functionality in different releases (examples below).
In general, avoid mentioning specific Qubes versions in the body text of documentation, as these references rapidly go out of date and become misleading to readers.
#### Incorrect Example
```
## How to Foo
Fooing is the process by which one foos. There are both general and specific
versions of fooing, which vary in usefulness depending on your goals, but for
the most part, all fooing is fooing.
To foo in Qubes 3.2:
$ qvm-foo <foo-bar>
Note that this does not work in Qubes 4.0, where there is a special widget
for fooing, which you can find in the lower-right corner of the screen in
the Foo Manager. Alternatively, you can use the more general `qubes-baz`
command introduced in 4.0:
$ qubes-baz --foo <bar>
Once you foo, make sure to close the baz before fooing the next bar.
```
#### Correct Example
```
## Qubes 3.2
### How to Foo
Fooing is the process by which one foos. There are both general and specific
versions of fooing, which vary in usefulness depending on your goals, but for
the most part, all fooing is fooing.
To foo:
$ qvm-foo <foo-bar>
Once you foo, make sure to close the baz before fooing the next bar.
## Qubes 4.0
### How to Foo
Fooing is the process by which one foos. There are both general and specific
versions of fooing, which vary in usefulness depending on your goals, but for
the most part, all fooing is fooing.
There is a special widget for fooing, which you can find in the lower-right
corner of the screen in the Foo Manager. Alternatively, you can use the
general `qubes-baz` command:
$ qubes-baz --foo <bar>
Once you foo, make sure to close the baz before fooing the next bar.
```
Subdividing the page into clearly-labeled sections for each release has several benefits:
- It preserves good content for older (but still supported) releases. Many documentation contributors are also people who prefer to use the latest release. Many of them are tempted to *replace* existing content that applies to an older, supported release with content that applies only to the latest release. This is somewhat understandable. Since they only use the latest release, they may be focused on their own experience, and they may even regard the older release as deprecated, even when it's actually still supported. However, allowing this replacement of content would do a great disservice to those who still rely on the older, supported release. In many cases, these users value the stability and reliability of the older, supported release. With the older, supported release, there has been more time to fix bugs and make improvements in both the software and the documentation. Consequently, much of the documentation content for this release may have gone through several rounds of editing, review, and revision. It would be a tragedy for this content to vanish while the very set of users who most prize stability and reliability are depending on it.
- It's easy for readers to quickly find the information they're looking for, since they can go directly to the section that applies to their release.
- It's hard for readers to miss information they need, since it's all in one place. In the incorrect example, information that the reader needs could be in any paragraph in the entire document, and there's no way to tell without reading the entire page. In the correct example, the reader can simply skim the headings in order to know which parts of the page need to be read and which can be safely ignored. The fact that some content is repeated in the two release-specific sections is not a problem, since no reader has to read the same thing twice. Moreover, as one release gets updated, it's likely that the documentation for that release will also be updated. Therefore, content that is initially duplicated between release-specific sections will not necessarily stay that way, and this is a good thing: We want the documentation for a release that *doesn't* change to stay the same, and we want the documentation for a release that *does* change to change along with the software.
- It's easy for documentation contributors and maintainers to know which file to edit and update, since there's only one page for all Qubes OS releases. Initially creating the new headings and duplicating content that applies to both is only a one-time cost for each page, and many pages don't even require this treatment, since they apply to all currently-supported Qubes OS releases.
By contrast, an alternative approach, such as segregating the documentation into two different branches, would mean that contributions that apply to both Qubes releases would only end up in one branch, unless someone remembered to manually submit the same thing to the other branch and actually made the effort to do so. Most of the time, this wouldn't happen. When it did, it would mean a second pull request that would have to be reviewed. Over time, the different branches would diverge in non-release-specific content. Good general content that was submitted only to one branch would effectively disappear once that release was deprecated. (Even if it were still on the website, no one would look at it, since it would explicitly be in the subdirectory of a deprecated release, and there would be a motivation to remove it from the website so that search results wouldn't be populated with out-of-date information.)
For further discussion about release-specific documentation in Qubes, see [here](https://groups.google.com/d/topic/qubes-users/H9BZX4K9Ptk/discussion).
## Git conventions
Please follow our [Git commit message guidelines](/doc/coding-style/#commit-message-guidelines).

592
developer/general/gsoc.rst
View file

@ -1,157 +1,203 @@
---
lang: en
layout: doc
permalink: /gsoc/
redirect_from:
- /GSoC/
ref: 33
title: Google Summer of Code (GSoC)
---
============================
Google Summer of Code (GSoC)
============================
## Information for Students
Thank you for your interest in participating in the [Google Summer of Code program](https://summerofcode.withgoogle.com/) with the [Qubes OS team](/team/). You can read more about the Google Summer of Code program at the [official website](https://summerofcode.withgoogle.com/) and the [official FAQ](https://developers.google.com/open-source/gsoc/faq).
Information for Students
------------------------
Thank you for your interest in participating in the `Google Summer of Code program <https://summerofcode.withgoogle.com/>`__ with the `Qubes OS team <https://www.qubes-os.org/team/>`__. You can read more about the Google Summer of Code program at the `official website <https://summerofcode.withgoogle.com/>`__ and the `official FAQ <https://developers.google.com/open-source/gsoc/faq>`__.
Being accepted as a Google Summer of Code contributor is quite competitive. If you are interested in participating in the Summer of Code please be aware that you must be able to produce code for Qubes OS for 3-5 months. Your mentors, Qubes developers, will dedicate a portion of their time towards mentoring you. Therefore, we seek candidates who are committed to helping Qubes long-term and are willing to do quality work and be proactive in communicating with your mentor.
You don't have to be a proven developer -- in fact, this whole program is meant to facilitate joining Qubes and other free and open source communities. The Qubes community maintains information about [contributing to Qubes development](/doc/contributing/#contributing-code) and [how to send patches](/doc/source-code/#how-to-send-patches). In order to contribute code to the Qubes project, you must be able to [sign your code](/doc/code-signing/).
You don’t have to be a proven developer – in fact, this whole program is meant to facilitate joining Qubes and other free and open source communities. The Qubes community maintains information about :ref:`contributing to Qubes development <introduction/contributing:contributing code>` and :ref:`how to send patches <developer/code/source-code:how to send patches>`. In order to contribute code to the Qubes project, you must be able to :doc:`sign your code </developer/code/code-signing>`.
You should start learning the components that you plan on working on before the start date. Qubes developers are available on the [mailing lists](/support/#qubes-devel) for help. The GSoC timeline reserves a lot of time for bonding with the project -- use that time wisely. Good communication is key, you should plan to communicate with your team daily and formally report progress and plans weekly. Students who neglect active communication will be failed.
You should start learning the components that you plan on working on before the start date. Qubes developers are available on the :ref:`mailing lists <introduction/support:qubes-devel>` for help. The GSoC timeline reserves a lot of time for bonding with the project – use that time wisely. Good communication is key, you should plan to communicate with your team daily and formally report progress and plans weekly. Students who neglect active communication will be failed.
### Overview of Steps
Overview of Steps
^^^^^^^^^^^^^^^^^
- Join the :ref:`qubes-devel list <introduction/support:qubes-devel>` and introduce yourself, and meet your fellow developers
- Read `Google’s instructions for participating <https://developers.google.com/open-source/gsoc/>`__ and the `GSoC Student Manual <https://google.github.io/gsocguides/student/>`__
- Join the [qubes-devel list](/support/#qubes-devel) and introduce yourself, and meet your fellow developers
- Read [Google's instructions for participating](https://developers.google.com/open-source/gsoc/) and the [GSoC Student Manual](https://google.github.io/gsocguides/student/)
- Take a look at the list of ideas below
- Come up with a project that you are interested in (and feel free to propose your own! Don't feel limited by the list below.)
- Come up with a project that you are interested in (and feel free to propose your own! Don’t feel limited by the list below.)
- Read the Contributor Proposal guidelines below
- Write a first draft proposal and send it to the qubes-devel mailing list for review
- Submit proposal using [Google's web interface](https://summerofcode.withgoogle.com/) ahead of the deadline (this requires a Google Account!)
- Submit proposal using `Google’s web interface <https://summerofcode.withgoogle.com/>`__ ahead of the deadline (this requires a Google Account!)
- Submit proof of enrollment well ahead of the deadline
Coming up with an interesting idea that you can realistically achieve in the time available to you (one summer) is probably the most difficult part. We strongly recommend getting involved in advance of the beginning of GSoC, and we will look favorably on applications from prospective contributors who have already started to act like free and open source developers.
Before the summer starts, there are some preparatory tasks which are highly encouraged. First, if you aren't already, definitely start using Qubes as your primary OS as soon as possible! Also, it is encouraged that you become familiar and comfortable with the Qubes development workflow sooner than later. A good way to do this (and also a great way to stand out as an awesome applicant and make us want to accept you!) might be to pick up some issues from [qubes-issues](https://github.com/QubesOS/qubes-issues/issues) (our issue-tracking repo) and submit some patches addressing them. Some suitable issues might be those with tags ["help wanted" and "P: minor"](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20label%3A%22P%3A%20minor%22%20label%3A%22help%20wanted%22) (although more significant things are also welcome, of course). Doing this will get you some practice with [qubes-builder](/doc/qubes-builder-v2/), our code-signing policies, and some familiarity with our code base in general so you are ready to hit the ground running come summer.
Before the summer starts, there are some preparatory tasks which are highly encouraged. First, if you aren’t already, definitely start using Qubes as your primary OS as soon as possible! Also, it is encouraged that you become familiar and comfortable with the Qubes development workflow sooner than later. A good way to do this (and also a great way to stand out as an awesome applicant and make us want to accept you!) might be to pick up some issues from `qubes-issues <https://github.com/QubesOS/qubes-issues/issues>`__ (our issue-tracking repo) and submit some patches addressing them. Some suitable issues might be those with tags `“help wanted” and “P: minor” <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue%20is%3Aopen%20label%3A%22P%3A%20minor%22%20label%3A%22help%20wanted%22>`__ (although more significant things are also welcome, of course). Doing this will get you some practice with :doc:`qubes-builder </developer/building/qubes-builder-v2>`, our code-signing policies, and some familiarity with our code base in general so you are ready to hit the ground running come summer.
### Contributor proposal guidelines
Contributor proposal guidelines
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A project proposal is what you will be judged upon. Write a clear proposal on what you plan to do, the scope of your project, and why we should choose you to do it. Proposals are the basis of the GSoC projects and therefore one of the most important things to do well.
A project proposal is what you will be judged upon. Write a clear proposal on what you plan to do, the scope of your project, and why we should choose you to do it. Proposals are the basis of the GSoC projects and therefore one of the most important things to do well.
Below is the application template:
```
# Introduction
.. code:: bash
Every software project should solve a problem. Before offering the solution (your Google Summer of Code project), you should first define the problem. What’s the current state of things? What’s the issue you wish to solve and why? Then you should conclude with a sentence or two about your solution. Include links to discussions, features, or bugs that describe the problem further if necessary.
# Introduction
Every software project should solve a problem. Before offering the solution (your Google Summer of Code project), you should first define the problem. What’s the current state of things? What’s the issue you wish to solve and why? Then you should conclude with a sentence or two about your solution. Include links to discussions, features, or bugs that describe the problem further if necessary.
# Project goals
Be short and to the point, and perhaps format it as a list. Propose a clear list of deliverables, explaining exactly what you promise to do and what you do not plan to do. “Future developments” can be mentioned, but your promise for the Google Summer of Code term is what counts.
# Implementation
Be detailed. Describe what you plan to do as a solution for the problem you defined above. Include technical details, showing that you understand the technology. Illustrate key technical elements of your proposed solution in reasonable detail.
# Timeline
Show that you understand the problem, have a solution, have also broken it down into manageable parts, and that you have a realistic plan on how to accomplish your goal. Here you set expectations, so don’t make promises you can’t keep. A modest, realistic and detailed timeline is better than promising the impossible.
If you have other commitments during GSoC, such as a job, vacation, exams, internship, seminars, or papers to write, disclose them here. GSoC should be treated like a full-time job, and we will expect approximately 40 hours of work per week. If you have conflicts, explain how you will work around them. If you are found to have conflicts which you did not disclose, you may be failed.
Open and clear communication is of utmost importance. Include your plans for communication in your proposal; daily if possible. You will need to initiate weekly formal communications such as a detailed email to the qubes-devel mailing list. Lack of communication will result in you being failed.
# About me
Provide your contact information and write a few sentences about you and why you think you are the best for this job. Prior contributions to Qubes are helpful; list your commits. Name people (other developers, students, professors) who can act as a reference for you. Mention your field of study if necessary. Now is the time to join the relevant mailing lists. We want you to be a part of our community, not just contribute your code.
Tell us if you are submitting proposals to other organizations, and whether or not you would choose Qubes if given the choice.
Other things to think about:
* Are you comfortable working independently under a supervisor or mentor who is several thousand miles away, and perhaps 12 time zones away? How will you work with your mentor to track your work? Have you worked in this style before?
* If your native language is not English, are you comfortable working closely with a supervisor whose native language is English? What is your native language, as that may help us find a mentor who has the same native language?
* After you have written your proposal, you should get it reviewed. Do not rely on the Qubes mentors to do it for you via the web interface, although we will try to comment on every proposal. It is wise to ask a colleague or a developer to critique your proposal. Clarity and completeness are important.
# Project goals
Be short and to the point, and perhaps format it as a list. Propose a clear list of deliverables, explaining exactly what you promise to do and what you do not plan to do. “Future developments” can be mentioned, but your promise for the Google Summer of Code term is what counts.
# Implementation
Project Ideas
-------------
Be detailed. Describe what you plan to do as a solution for the problem you defined above. Include technical details, showing that you understand the technology. Illustrate key technical elements of your proposed solution in reasonable detail.
# Timeline
These project ideas were contributed by our developers and may be incomplete. If you are interested in submitting a proposal based on these ideas, you should contact the :ref:`qubes-devel mailing list <introduction/support:qubes-devel>` and associated GitHub issue to learn more about the idea.
Show that you understand the problem, have a solution, have also broken it down into manageable parts, and that you have a realistic plan on how to accomplish your goal. Here you set expectations, so don’t make promises you can’t keep. A modest, realistic and detailed timeline is better than promising the impossible.
.. code:: bash
If you have other commitments during GSoC, such as a job, vacation, exams, internship, seminars, or papers to write, disclose them here. GSoC should be treated like a full-time job, and we will expect approximately 40 hours of work per week. If you have conflicts, explain how you will work around them. If you are found to have conflicts which you did not disclose, you may be failed.
### Adding a Proposal
**Project**: Something that you're totally excited about
**Brief explanation**: What is the project, where does the code live?
**Expected results**: What is the expected result in the timeframe given
**Difficulty**: easy / medium / hard
**Knowledge prerequisite**: Pre-requisites for working on the project. What coding language and knowledge is needed?
If applicable, links to more information or discussions
**Size of the project**: either 175 hours (medium) or 350 hours (large)
**Mentor**: Name and email address.
Open and clear communication is of utmost importance. Include your plans for communication in your proposal; daily if possible. You will need to initiate weekly formal communications such as a detailed email to the qubes-devel mailing list. Lack of communication will result in you being failed.
# About me
Provide your contact information and write a few sentences about you and why you think you are the best for this job. Prior contributions to Qubes are helpful; list your commits. Name people (other developers, students, professors) who can act as a reference for you. Mention your field of study if necessary. Now is the time to join the relevant mailing lists. We want you to be a part of our community, not just contribute your code.
Qubes as a Vagrant provider
^^^^^^^^^^^^^^^^^^^^^^^^^^^
Tell us if you are submitting proposals to other organizations, and whether or not you would choose Qubes if given the choice.
Other things to think about:
* Are you comfortable working independently under a supervisor or mentor who is several thousand miles away, and perhaps 12 time zones away? How will you work with your mentor to track your work? Have you worked in this style before?
* If your native language is not English, are you comfortable working closely with a supervisor whose native language is English? What is your native language, as that may help us find a mentor who has the same native language?
* After you have written your proposal, you should get it reviewed. Do not rely on the Qubes mentors to do it for you via the web interface, although we will try to comment on every proposal. It is wise to ask a colleague or a developer to critique your proposal. Clarity and completeness are important.
```
## Project Ideas
These project ideas were contributed by our developers and may be incomplete. If you are interested in submitting a proposal based on these ideas, you should contact the [qubes-devel mailing list](/support/#qubes-devel) and associated GitHub issue to learn more about the idea.
```
### Adding a Proposal
**Project**: Something that you're totally excited about
**Brief explanation**: What is the project, where does the code live?
**Expected results**: What is the expected result in the timeframe given
**Difficulty**: easy / medium / hard
**Knowledge prerequisite**: Pre-requisites for working on the project. What coding language and knowledge is needed?
If applicable, links to more information or discussions
**Size of the project**: either 175 hours (medium) or 350 hours (large)
**Mentor**: Name and email address.
```
### Qubes as a Vagrant provider
**Project**: Qubes as a Vagrant provider
**Brief explanation**: Currently using Vagrant on Qubes requires finding an image that uses Docker as isolation provider and running Docker in a qube, or downloading the Vagrantfile and manually setting up a qube according to the Vagrantfile. This project aims at simplifying this workflow. Since introduction of Admin API, it's possible for a qube to provision another qube - which is exactly what is needed for Vagrant. [Related discussion](https://groups.google.com/d/msgid/qubes-devel/535299ca-d16a-4a70-8223-a4ac6be4be41%40googlegroups.com)
**Brief explanation**: Currently using Vagrant on Qubes requires finding an image that uses Docker as isolation provider and running Docker in a qube, or downloading the Vagrantfile and manually setting up a qube according to the Vagrantfile. This project aims at simplifying this workflow. Since introduction of Admin API, it’s possible for a qube to provision another qube - which is exactly what is needed for Vagrant. `Related discussion <https://groups.google.com/d/msgid/qubes-devel/535299ca-d16a-4a70-8223-a4ac6be4be41%40googlegroups.com>`__
**Expected results**:
- Design how Vagrant Qubes provider should look like, including:
- [box format](https://www.vagrantup.com/docs/plugins/providers.html#box-format)
- `box format <https://www.vagrantup.com/docs/plugins/providers.html#box-format>`__
- method for running commands inside (ssh vs qvm-run)
- Write a Vagrant provider able to create/start/stop/etc a VM
- Document how to configure and use the provider, including required qrexec policy changes and possibly firewall rules
- Write integration tests
**Difficulty**: medium
**Knowledge prerequisite**:
- Ruby
- Vagrant concepts
**Size of the project**: 350 hours
**Mentor**: [Wojtek Porczyk](/team/), [Marek Marczykowski-Górecki](/team/)
**Mentor**: `Wojtek Porczyk <https://www.qubes-os.org/team/>`__, `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__
System health monitor
^^^^^^^^^^^^^^^^^^^^^
### System health monitor
**Project**: System health monitor
**Brief explanation**: A tool that informs the user about common system and configuration issues. Some of this is already available, but scattered across different places. See related issues: [6663](https://github.com/QubesOS/qubes-issues/issues/6663), [2134](https://github.com/QubesOS/qubes-issues/issues/2134)
**Brief explanation**: A tool that informs the user about common system and configuration issues. Some of this is already available, but scattered across different places. See related issues: `6663 <https://github.com/QubesOS/qubes-issues/issues/6663>`__, `2134 <https://github.com/QubesOS/qubes-issues/issues/2134>`__
**Expected results**:
- a tool / service that checks for common issues and things needing user attention, for example:
- some updates to be applied (separate widget already exists)
- running out of disk space (separate widget already exists)
- insecure USB configuration (USB in dom0)
- some system VM crashed
- ...
- …
- a GUI that provides terse overview of the system state, and notifies the user if something bad happens
**Difficulty**: medium
**Knowledge prerequisite**:
- Python
- basic knowledge about systemd services
- PyGTK (optional)
**Size of the project**: 350 hours
**Mentor**: [Marta Marczykowska-Górecka](/team/)
**Mentor**: `Marta Marczykowska-Górecka <https://www.qubes-os.org/team/>`__
Mechanism for maintaining in-VM configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### Mechanism for maintaining in-VM configuration
**Project**: Mechanism for maintaining in-VM configuration
@ -159,127 +205,149 @@ If applicable, links to more information or discussions
**Expected results**:
- Design a mechanism how to _safely_ synchronize application configuration living in user home directory (`~/.config`, some other "dotfiles"). Mechanism should be resistant against malicious VM forcing its configuration on other VMs. Some approach could be a strict control which VM can send what changes (whitelist approach, not blacklist).
- Design a mechanism how to *safely* synchronize application configuration living in user home directory (``~/.config``, some other “dotfiles”). Mechanism should be resistant against malicious VM forcing its configuration on other VMs. Some approach could be a strict control which VM can send what changes (whitelist approach, not blacklist).
- Implementation of the above mechanism.
- Documentation how to configure it securely.
**Difficulty**: medium
**Knowledge prerequisite**:
- shell and/or python scripting
- Qubes OS qrexec services
**Size of the project**: 175 hours
**Mentor**: [Frédéric Pierret](/team/)
**Mentor**: `Frédéric Pierret <https://www.qubes-os.org/team/>`__
Qubes Live USB
^^^^^^^^^^^^^^
### Qubes Live USB
**Project**: Revive Qubes Live USB, integrate it with installer
**Brief explanation**: Qubes Live USB is based on Fedora tools to build live
distributions. But for Qubes we need some adjustments: starting Xen instead of
Linux kernel, smarter copy-on-write handling (we run there multiple VMs, so a
lot more data to save) and few more. Additionally in Qubes 3.2 we have
so many default VMs that default installation does not fit in 16GB image
(default value) - some subset of those VMs should be chosen. Ideally we'd like
to have just one image being both live system and installation image. More
details: [#1552](https://github.com/QubesOS/qubes-issues/issues/1552),
[#1965](https://github.com/QubesOS/qubes-issues/issues/1965).
**Brief explanation**: Qubes Live USB is based on Fedora tools to build live distributions. But for Qubes we need some adjustments: starting Xen instead of Linux kernel, smarter copy-on-write handling (we run there multiple VMs, so a lot more data to save) and few more. Additionally in Qubes 3.2 we have so many default VMs that default installation does not fit in 16GB image (default value) - some subset of those VMs should be chosen. Ideally we’d like to have just one image being both live system and installation image. More details: `#1552 <https://github.com/QubesOS/qubes-issues/issues/1552>`__, `#1965 <https://github.com/QubesOS/qubes-issues/issues/1965>`__.
**Expected results**:
- Adjust set of VMs and templates included in live edition.
- Update and fix build scripts for recent Qubes OS version.
- Update startup script to mount appropriate directories as either
copy-on-write (device-mapper snapshot), or tmpfs.
- Optimize memory usage: should be possible to run sys-net, sys-firewall, and
at least two more VMs on 4GB machine. This include minimizing writes to
copy-on-write layer and tmpfs (disable logging etc).
- Research option to install the system from live image. If feasible add
this option.
- Update startup script to mount appropriate directories as either copy-on-write (device-mapper snapshot), or tmpfs.
- Optimize memory usage: should be possible to run sys-net, sys-firewall, and at least two more VMs on 4GB machine. This include minimizing writes to copy-on-write layer and tmpfs (disable logging etc).
- Research option to install the system from live image. If feasible add this option.
**Difficulty**: hard
**Knowledge prerequisite**:
- System startup sequence: bootloaders (isolinux, syslinux, grub, UEFI), initramfs, systemd.
- Python and Bash scripting
- Filesystems and block devices: loop devices, device-mapper, tmpfs, overlayfs, sparse files.
**Size of the project**: 350 hours
**Mentor**: [Frédéric Pierret](/team/)
**Mentor**: `Frédéric Pierret <https://www.qubes-os.org/team/>`__
LogVM(s)
^^^^^^^^
### LogVM(s)
**Project**: LogVM(s)
**Brief explanation**: Qubes AppVMs do not have persistent /var (on purpose).
It would be useful to send logs generated by various VMs to a dedicated
log-collecting VM. This way logs will not only survive VM shutdown, but also be
immune to altering past entries. See
[#830](https://github.com/QubesOS/qubes-issues/issues/830) for details.
**Brief explanation**: Qubes AppVMs do not have persistent /var (on purpose). It would be useful to send logs generated by various VMs to a dedicated log-collecting VM. This way logs will not only survive VM shutdown, but also be immune to altering past entries. See `#830 <https://github.com/QubesOS/qubes-issues/issues/830>`__ for details.
**Expected results**:
- Design a _simple_ protocol for transferring logs. The less metadata (parsed
in log-collecting VM) the better.
- Implement log collecting service. Besides logs itself, should save
information about logs origin (VM name) and timestamp. The service should
_not_ trust sending VM in any of those.
- Implement log forwarder compatible with systemd-journald and rsyslog. A
mechanism (service/plugin) fetching logs in real time from those and sending
to log-collecting VM over qrexec service.
- Design a *simple* protocol for transferring logs. The less metadata (parsed in log-collecting VM) the better.
- Implement log collecting service. Besides logs itself, should save information about logs origin (VM name) and timestamp. The service should *not* trust sending VM in any of those.
- Implement log forwarder compatible with systemd-journald and rsyslog. A mechanism (service/plugin) fetching logs in real time from those and sending to log-collecting VM over qrexec service.
- Document the protocol.
- Write unit tests and integration tests.
**Difficulty**: easy
**Knowledge prerequisite**:
- syslog
- systemd
- Python/Bash scripting
**Size of the project**: 175 hours
**Mentor**: [Frédéric Pierret](/team/)
**Mentor**: `Frédéric Pierret <https://www.qubes-os.org/team/>`__
Whonix IPv6 and nftables support
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### Whonix IPv6 and nftables support
**Project**: Whonix IPv6 and nftables support
**Brief explanation**: [T509](https://phabricator.whonix.org/T509)
**Brief explanation**: `T509 <https://phabricator.whonix.org/T509>`__
**Expected results**:
- Work at upstream Tor: An older version of [TransparentProxy](https://trac.torproject.org/projects/tor/wiki/doc/TransparentProxy) page was the origin of Whonix. Update that page for nftables / IPv6 support without mentioning Whonix. Then discuss that on the tor-talk mailing list for wider input. [here](https://trac.torproject.org/projects/tor/ticket/21397)
- implement corridor feature request add IPv6 support / port to nftables - [issue](https://github.com/rustybird/corridor/issues/39)
- port [whonix-firewall](https://github.com/Whonix/whonix-firewall) to nftables
- Work at upstream Tor: An older version of `TransparentProxy <https://trac.torproject.org/projects/tor/wiki/doc/TransparentProxy>`__ page was the origin of Whonix. Update that page for nftables / IPv6 support without mentioning Whonix. Then discuss that on the tor-talk mailing list for wider input. `here <https://trac.torproject.org/projects/tor/ticket/21397>`__
- implement corridor feature request add IPv6 support / port to nftables - `issue <https://github.com/rustybird/corridor/issues/39>`__
- port `whonix-firewall <https://github.com/Whonix/whonix-firewall>`__ to nftables
- make connections to IPv6 Tor relays work
- make connections to IPv6 destinations work
**Difficulty**: medium
**Knowledge prerequisite**:
- nftables
- iptables
- IPv6
**Size of the project**: 175 hours
**Mentor**: [Patrick Schleizer](/team/)
**Mentor**: `Patrick Schleizer <https://www.qubes-os.org/team/>`__
GUI agent for Windows 8/10
^^^^^^^^^^^^^^^^^^^^^^^^^^
### GUI agent for Windows 8/10
**Project**: GUI agent for Windows 8/10
**Brief explanation**: Add support for Windows 8+ to the Qubes GUI agent and video driver. Starting from Windows 8, Microsoft requires all video drivers to conform to the WDDM display driver model which is incompatible with the current Qubes video driver. Unfortunately the WDDM model is much more complex than the old XPDM one and officially *requires* a physical GPU device (which may be emulated). Some progress has been made to create a full WDDM driver that *doesn't* require a GPU device, but the driver isn't working correctly yet. Alternatively, WDDM model supports display-only drivers which are much simpler but don't have access to system video memory and rendering surfaces (a key feature that would simplify seamless GUI mode). [#1861](https://github.com/QubesOS/qubes-issues/issues/1861)
**Brief explanation**: Add support for Windows 8+ to the Qubes GUI agent and video driver. Starting from Windows 8, Microsoft requires all video drivers to conform to the WDDM display driver model which is incompatible with the current Qubes video driver. Unfortunately the WDDM model is much more complex than the old XPDM one and officially *requires* a physical GPU device (which may be emulated). Some progress has been made to create a full WDDM driver that *doesn’t* require a GPU device, but the driver isn’t working correctly yet. Alternatively, WDDM model supports display-only drivers which are much simpler but don’t have access to system video memory and rendering surfaces (a key feature that would simplify seamless GUI mode). `#1861 <https://github.com/QubesOS/qubes-issues/issues/1861>`__
**Expected results**: Working display-only WDDM video driver or significant progress towards making the full WDDM driver work correctly.
@ -289,73 +357,93 @@ immune to altering past entries. See
**Size of the project**: 175 hours
**Mentor**: [Rafał Wojdyła](/team/)
**Mentor**: `Rafał Wojdyła <https://www.qubes-os.org/team/>`__
GNOME support in dom0 / GUI VM
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### GNOME support in dom0 / GUI VM
**Project**: GNOME support in dom0
**Brief explanation**: Integrating GNOME into Qubes dom0. This include:
- patching window manager to add colorful borders
- removing stuff not needed in dom0 (file manager(s), indexing services etc)
- adjusting menu for easy navigation (same applications in different VMs and such problems, dom0-related entries in one place)
- More info: [#1806](https://github.com/QubesOS/qubes-issues/issues/1806)
- More info: `#1806 <https://github.com/QubesOS/qubes-issues/issues/1806>`__
**Expected results**:
- Review existing support for other desktop environments (KDE, Xfce4, i3, awesome).
- Patch window manager to draw colorful borders (we use only server-side
decorations), there is already very similar patch in
[Cappsule project](https://github.com/cappsule/cappsule-gui).
- Configure GNOME to not make use of dom0 user home in visible way (no search
in files there, no file manager, etc).
- Configure GNOME to not look into external devices plugged in (no auto
mounting, device notifications etc).
- Package above modifications as RPMs, preferably as extra configuration files
and/or plugins than overwriting existing files. Exceptions to this rule may
apply if no other option.
- Adjust comps.xml (in installer-qubes-os repo) to define package group with
all required packages.
- Patch window manager to draw colorful borders (we use only server-side decorations), there is already very similar patch in `Cappsule project <https://github.com/cappsule/cappsule-gui>`__.
- Configure GNOME to not make use of dom0 user home in visible way (no search in files there, no file manager, etc).
- Configure GNOME to not look into external devices plugged in (no auto mounting, device notifications etc).
- Package above modifications as RPMs, preferably as extra configuration files and/or plugins than overwriting existing files. Exceptions to this rule may apply if no other option.
- Adjust comps.xml (in installer-qubes-os repo) to define package group with all required packages.
- Document installation procedure.
**Difficulty**: hard
**Knowledge prerequisite**:
- GNOME architecture
- C language (patching metacity)
- Probably also javascript - for modifying GNOME shell extensions
**Size of the project**: 175 hours
**Mentor**: [Frédéric Pierret](/team/), [Marek Marczykowski-Górecki](/team/)
**Mentor**: `Frédéric Pierret <https://www.qubes-os.org/team/>`__, `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__
Generalize the Qubes PDF Converter to other types of files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### Generalize the Qubes PDF Converter to other types of files
**Project**: Qubes Converters
**Brief explanation**: One of the pioneering ideas of Qubes is to use disposable virtual machines to convert untrustworthy files (such as documents given to journalists by unknown and potentially malicious whistleblowers) into trustworthy files. See [Joanna's blog on the Qubes PDF Convert](https://theinvisiblethings.blogspot.co.uk/2013/02/converting-untrusted-pdfs-into-trusted.html) for details of the idea. Joanna has implemented a prototype for PDF documents. The goal of this project would be to generalize beyond the simple prototype to accommodate a wide variety of file formats, including Word documents, audio files, video files, spreadsheets, and so on. The converters should prioritise safety over faithful conversion. For example the Qubes PDF converter typically leads to lower quality PDFs (e.g. cut and paste is no longer possible), because this makes the conversion process safer.
**Brief explanation**: One of the pioneering ideas of Qubes is to use disposable virtual machines to convert untrustworthy files (such as documents given to journalists by unknown and potentially malicious whistleblowers) into trustworthy files. See `Joanna’s blog on the Qubes PDF Convert <https://theinvisiblethings.blogspot.co.uk/2013/02/converting-untrusted-pdfs-into-trusted.html>`__ for details of the idea. Joanna has implemented a prototype for PDF documents. The goal of this project would be to generalize beyond the simple prototype to accommodate a wide variety of file formats, including Word documents, audio files, video files, spreadsheets, and so on. The converters should prioritise safety over faithful conversion. For example the Qubes PDF converter typically leads to lower quality PDFs (e.g. cut and paste is no longer possible), because this makes the conversion process safer.
**Expected results**: We expect that in the timeframe, it will be possible to implement many converters for many file formats. However, if any unexpected difficulties arise, we would prioritise a small number of safe and high quality converters over a large number of unsafe or unuseful converters.
**Expected results**: We expect that in the timeframe, it will be possible to implement many converters for many file formats. However, if any unexpected difficulties arise, we would prioritise a small number of safe and high quality converters over a large number of unsafe or unuseful converters.
**Difficulty**: easy
**Knowledge prerequisite**: Most of the coding will probably be implemented as shell scripts to interface with pre-existing converters (such as ImageMagick in the Qubes PDF converter). However, shell scripts are not safe for processing untrusted data, so any extra processing will need to be implemented in another language -- probably Python.
**Knowledge prerequisite**: Most of the coding will probably be implemented as shell scripts to interface with pre-existing converters (such as ImageMagick in the Qubes PDF converter). However, shell scripts are not safe for processing untrusted data, so any extra processing will need to be implemented in another language – probably Python.
**Size of the project**: 175 hours
**Mentors**: Andrew Clausen and Jean-Philippe Ouellet
### Progress towards reproducible builds
Progress towards reproducible builds
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**Project**: Progress towards reproducible builds
**Brief explanation**: A long-term goal is to be able to build the entire OS and installation media in a completely bit-wise deterministic manner, but there are many baby steps to be taken along that path. See:
- "[Security challenges for the Qubes build process](/news/2016/05/30/build-security/)"
- [This mailing list post](https://groups.google.com/d/msg/qubes-devel/gq-wb9wTQV8/mdliS4P2BQAJ)
- and [reproducible-builds.org](https://reproducible-builds.org/)
- “`Security challenges for the Qubes build process <https://www.qubes-os.org/news/2016/05/30/build-security/>`__”
- `This mailing list post <https://groups.google.com/d/msg/qubes-devel/gq-wb9wTQV8/mdliS4P2BQAJ>`__
- and `reproducible-builds.org <https://reproducible-builds.org/>`__
for more information and qubes-specific background.
@ -363,13 +451,15 @@ for more information and qubes-specific background.
**Difficulty**: medium
**Knowledge prerequisite**: qubes-builder [[1]](/doc/qubes-builder-v2/) [[2]](https://github.com/QubesOS/qubes-builderv2), and efficient at introspecting complex systems: comfortable with tracing and debugging tools, ability to quickly identify and locate issues within a large codebase (upstream build tools), etc.
**Knowledge prerequisite**: qubes-builder :doc:`[1] </developer/building/qubes-builder-v2>` `[2] <https://github.com/QubesOS/qubes-builderv2>`__, and efficient at introspecting complex systems: comfortable with tracing and debugging tools, ability to quickly identify and locate issues within a large codebase (upstream build tools), etc.
**Size of the project**: 350 hours
**Mentor**: [Marek Marczykowski-Górecki](/team/)
**Mentor**: `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__
Porting Qubes to ARM/aarch64
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### Porting Qubes to ARM/aarch64
**Project**: Porting Qubes to ARM/aarch64
@ -379,114 +469,139 @@ Qubes currently only supports the x86_64 CPU architecture. Xen currently has add
Some related discussion:
- [#4318](https://github.com/QubesOS/qubes-issues/issues/4318) on porting to ppc64.
- [#3894](https://github.com/QubesOS/qubes-issues/issues/3894) on porting to L4 microkernel.
- `#4318 <https://github.com/QubesOS/qubes-issues/issues/4318>`__ on porting to ppc64.
- `#3894 <https://github.com/QubesOS/qubes-issues/issues/3894>`__ on porting to L4 microkernel.
**Expected results**:
- Add cross-compilation support to qubes-builder and related components.
- Make aarch64 specific adjustments to Qubes toolstacks/manager (including passthrough of devices from device tree to guest domains).
- Aarch64 specific integration and unit tests.
- Production of generic u-boot or uefi capable image/iso for target hardware.
**Difficulty**: hard
**Knowledge prerequisite**:
- Libvirt and Qubes toolstacks (C and python languages).
- Xen debugging.
- General ARM architecture knowledge.
**Size of the project**: 350 hours
**Mentor**: [Marek Marczykowski-Górecki](/team/)
**Mentor**: `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__
Android development in Qubes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### Android development in Qubes
**Project**: Research running Android in Qubes VM (probably HVM) and connecting it to Android Studio
**Brief explanation**: The goal is to enable Android development (and testing!)
on Qubes OS. Currently it's only possible using qemu-emulated Android for ARM.
Since it's software emulation it's rather slow.
Details, reference: [#2233](https://github.com/QubesOS/qubes-issues/issues/2233)
**Brief explanation**: The goal is to enable Android development (and testing!) on Qubes OS. Currently it’s only possible using qemu-emulated Android for ARM. Since it’s software emulation it’s rather slow. Details, reference: `#2233 <https://github.com/QubesOS/qubes-issues/issues/2233>`__
**Expected results**:
- a simple way of setting up Android qubes with hardware emulation
(distributed as a template or as a salt, handling various modern Android versions)
- figuring out and implementing an easy and secure way to connect an Android
qube to a development qube with Android studio
- a simple way of setting up Android qubes with hardware emulation (distributed as a template or as a salt, handling various modern Android versions)
- figuring out and implementing an easy and secure way to connect an Android qube to a development qube with Android studio
- documentation and tests
**Difficulty**: hard
**Knowledge prerequisite**:
**Size of the project**: 350 hours
**Mentor**: Inquire on [qubes-devel](/support/#qubes-devel).
**Mentor**: Inquire on :ref:`qubes-devel <introduction/support:qubes-devel>`.
### Admin API Fuzzer
Admin API Fuzzer
^^^^^^^^^^^^^^^^
**Project**: Develop a [Fuzzer](https://en.wikipedia.org/wiki/Fuzzing) for the
[Qubes OS Admin API](/doc/admin-api/).
**Brief explanation**: The [Qubes OS Admin API](/doc/admin-api/)
enables VMs to execute privileged actions on other VMs or dom0 - if allowed by the Qubes OS RPC policy.
Programming errors in the Admin API however may cause these access rights to be more permissive
than anticipated by the programmer.
**Project**: Develop a `Fuzzer <https://en.wikipedia.org/wiki/Fuzzing>`__ for the :doc:`Qubes OS Admin API </developer/services/admin-api>`.
Since the Admin API is continuously growing and changing, continuous security assessments are required.
A [Fuzzer](https://en.wikipedia.org/wiki/Fuzzing) would help to automate part of these assessments.
**Brief explanation**: The :doc:`Qubes OS Admin API </developer/services/admin-api>` enables VMs to execute privileged actions on other VMs or dom0 - if allowed by the Qubes OS RPC policy. Programming errors in the Admin API however may cause these access rights to be more permissive than anticipated by the programmer.
Since the Admin API is continuously growing and changing, continuous security assessments are required. A `Fuzzer <https://en.wikipedia.org/wiki/Fuzzing>`__ would help to automate part of these assessments.
**Expected results**:
- fully automated & extensible Fuzzer for parts of the Admin API
- user & developer documentation
- fully automated & extensible Fuzzer for parts of the Admin API
- user & developer documentation
**Difficulty**: medium
**Prerequisites**:
- basic Python understanding
- some knowledge about fuzzing & existing fuzzing frameworks (e.g. [oss-fuzz](https://github.com/google/oss-fuzz/tree/master/projects/qubes-os))
- a hacker's curiosity
- basic Python understanding
- some knowledge about fuzzing & existing fuzzing frameworks (e.g. `oss-fuzz <https://github.com/google/oss-fuzz/tree/master/projects/qubes-os>`__)
- a hacker’s curiosity
**Size of the project**: 175 hours
**Mentor**: Inquire on [qubes-devel](/support/#qubes-devel).
**Mentor**: Inquire on :ref:`qubes-devel <introduction/support:qubes-devel>`.
Secure Boot support
^^^^^^^^^^^^^^^^^^^
### Secure Boot support
**Project**: Add support for protecting boot binaries with Secure Boot technology, using user-generated keys.
**Brief explanation**: Since recently, Xen supports "unified EFI boot" which allows to sign not only Xen binary itself, but also dom0 kernel and their parameters. While the base technology is there, enabling it is a painful and complex process. The goal of this project is to integrate configuration of this feature into Qubes, automating as much as possible. See discussion in [issue #4371](https://github.com/QubesOS/qubes-issues/issues/4371)
**Brief explanation**: Since recently, Xen supports “unified EFI boot” which allows to sign not only Xen binary itself, but also dom0 kernel and their parameters. While the base technology is there, enabling it is a painful and complex process. The goal of this project is to integrate configuration of this feature into Qubes, automating as much as possible. See discussion in `issue #4371 <https://github.com/QubesOS/qubes-issues/issues/4371>`__
**Expected results**:
- a tool to prepare relevant boot files for unified Xen EFI boot - this includes collecting Xen, dom0 kernel, initramfs, config file, and possibly few more (ucode update?); the tool should then sign the file with user provided key (preferably propose to generate it too)
- integrate it with updates mechanism, so new Xen or dom0 kernel will be picked up automatically
- include a fallback configuration that can be used for troubleshooting (main unified Xen EFI intentionally does not allow to manipulate parameters at boot time)
- a tool to prepare relevant boot files for unified Xen EFI boot - this includes collecting Xen, dom0 kernel, initramfs, config file, and possibly few more (ucode update?); the tool should then sign the file with user provided key (preferably propose to generate it too)
- integrate it with updates mechanism, so new Xen or dom0 kernel will be picked up automatically
- include a fallback configuration that can be used for troubleshooting (main unified Xen EFI intentionally does not allow to manipulate parameters at boot time)
**Difficulty**: hard
**Knowledge prerequisite**:
- basic understanding of Secure Boot
- Bash and Python scripting
- basic understanding of Secure Boot
- Bash and Python scripting
**Size of the project**: 175 hours
**Mentor**: [Marek Marczykowski-Górecki](/team/)
**Mentor**: `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__
Reduce logging of Disposable VMs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### Reduce logging of Disposable VMs
**Project**: Reduce logging of Disposable VMs
**Brief explanation**: Partial metadata of a DisposableVM is stored in the dom0 filesystem. This applies to various logs, GUI status files etc. There should be an option to hide as much of that as possible - including bypassing some logging, and removing various state files, or at the very least obfuscating any hints what is running inside DisposableVM. More details at [issue #4972](https://github.com/QubesOS/qubes-issues/issues/4972)
**Brief explanation**: Partial metadata of a DisposableVM is stored in the dom0 filesystem. This applies to various logs, GUI status files etc. There should be an option to hide as much of that as possible - including bypassing some logging, and removing various state files, or at the very least obfuscating any hints what is running inside DisposableVM. More details at `issue #4972 <https://github.com/QubesOS/qubes-issues/issues/4972>`__
**Expected results**: A DisposableVM should not leave logs hinting what was running inside.
@ -494,75 +609,84 @@ A [Fuzzer](https://en.wikipedia.org/wiki/Fuzzing) would help to automate part of
**Knowledge prerequisite**:
- Python scripting
- Basic knowledge of Linux system services management (systemd, syslog etc)
- Python scripting
- Basic knowledge of Linux system services management (systemd, syslog etc)
**Size of the project**: 350 hours
**Mentor**: [Marek Marczykowski-Górecki](/team/)
**Mentor**: `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__
Past Projects
-------------
## Past Projects
You can view the projects we had in 2017 in the [GSoC 2017 archive](https://summerofcode.withgoogle.com/archive/2017/organizations/5074771758809088/). We also participated in GSoC 2020 and GSoC 2021, and you can see the project in the [GSoC 2020 archive](https://summerofcode.withgoogle.com/archive/2020/organizations/4924517870206976/) and [GSoC 2021 archive](https://summerofcode.withgoogle.com/archive/2021/organizations/5682513023860736).
You can view the projects we had in 2017 in the `GSoC 2017 archive <https://summerofcode.withgoogle.com/archive/2017/organizations/5074771758809088/>`__. We also participated in GSoC 2020 and GSoC 2021, and you can see the project in the `GSoC 2020 archive <https://summerofcode.withgoogle.com/archive/2020/organizations/4924517870206976/>`__ and `GSoC 2021 archive <https://summerofcode.withgoogle.com/archive/2021/organizations/5682513023860736>`__.
Here are some successful projects which have been implemented in the past by Google Summer of Code participants.
### Template manager, new template distribution mechanism
Template manager, new template distribution mechanism
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**Project**: Template manager, new template distribution mechanism
**Brief explanation**: Template VMs currently are distributed using RPM
packages. There are multiple problems with that, mostly related to static
nature of RPM package (what files belong to the package). This means such
Template VM cannot be renamed, migrated to another storage (like LVM), etc.
Also we don't want RPM to automatically update template package itself (which
would override all the user changes there). More details:
[#2064](https://github.com/QubesOS/qubes-issues/issues/2064),
[#2534](https://github.com/QubesOS/qubes-issues/issues/2534),
[#3573](https://github.com/QubesOS/qubes-issues/issues/3573).
**Brief explanation**: Template VMs currently are distributed using RPM packages. There are multiple problems with that, mostly related to static nature of RPM package (what files belong to the package). This means such Template VM cannot be renamed, migrated to another storage (like LVM), etc. Also we don’t want RPM to automatically update template package itself (which would override all the user changes there). More details: `#2064 <https://github.com/QubesOS/qubes-issues/issues/2064>`__, `#2534 <https://github.com/QubesOS/qubes-issues/issues/2534>`__, `#3573 <https://github.com/QubesOS/qubes-issues/issues/3573>`__.
**Expected results**:
- Design new mechanism for distributing templates (possibly including some
package format - either reuse something already existing, or design
new one). The mechanism needs to handle:
- integrity protection (digital signatures), not parsing any data in dom0
prior to signature verification
- efficient handling of large sparse files
- ability to deploy the template into various storage mechanisms (sparse
files, LVM thin volumes etc).
- template metadata, templates repository - enable the user to browse
available templates (probably should be done in dedicated VM, or DisposableVM)
- manual template removal by users (without it, see problems such
as [#5509](https://github.com/QubesOS/qubes-issues/issues/5509)
- Implement the above mechanism:
- tool to download named template - should perform download operation in
some VM (as dom0 have no network access), then transfer the data to dom0,
verify its integrity and then create Template VM and feed it's root
filesystem image with downloaded data.
- tool to browse templates repository - both CLI and GUI (preferably integrated
with existing Template Manager tool)
- integrate both tools - user should be able to choose some template to be
installed from repository browsing tool - see
[#1705](https://github.com/QubesOS/qubes-issues/issues/1705) for some idea
(this one lacks integrity verification, but a similar service could
be developed with that added)
- If new "package" format is developed, add support for it into
[linux-template-builder](https://github.com/QubesOS/qubes-linux-template-builder).
- Document the mechanism.
- Write unit tests and integration tests.
- Design new mechanism for distributing templates (possibly including some package format - either reuse something already existing, or design new one). The mechanism needs to handle:
- integrity protection (digital signatures), not parsing any data in dom0 prior to signature verification
- efficient handling of large sparse files
- ability to deploy the template into various storage mechanisms (sparse files, LVM thin volumes etc).
- template metadata, templates repository - enable the user to browse available templates (probably should be done in dedicated VM, or DisposableVM)
- manual template removal by users (without it, see problems such as `#5509 <https://github.com/QubesOS/qubes-issues/issues/5509>`__
- Implement the above mechanism:
- tool to download named template - should perform download operation in some VM (as dom0 have no network access), then transfer the data to dom0, verify its integrity and then create Template VM and feed it’s root filesystem image with downloaded data.
- tool to browse templates repository - both CLI and GUI (preferably integrated with existing Template Manager tool)
- integrate both tools - user should be able to choose some template to be installed from repository browsing tool - see `#1705 <https://github.com/QubesOS/qubes-issues/issues/1705>`__ for some idea (this one lacks integrity verification, but a similar service could be developed with that added)
- If new “package” format is developed, add support for it into `linux-template-builder <https://github.com/QubesOS/qubes-linux-template-builder>`__.
- Document the mechanism.
- Write unit tests and integration tests.
**Knowledge prerequisite**:
- Large files (disk images) handling (sparse files, archive formats)
- Bash and Python scripting
- Data integrity handling - digital signatures (gpg2, gpgv2)
- PyGTK
- RPM package format, (yum) repository basics
- Large files (disk images) handling (sparse files, archive formats)
- Bash and Python scripting
- Data integrity handling - digital signatures (gpg2, gpgv2)
- PyGTK
- RPM package format, (yum) repository basics
**Mentor**: `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__
**Mentor**: [Marek Marczykowski-Górecki](/team/)
----
We adapted some of the language here about GSoC from the [KDE GSoC page](https://community.kde.org/GSoC).
We adapted some of the language here about GSoC from the `KDE GSoC page <https://community.kde.org/GSoC>`__.

284
developer/general/gsod.rst
View file

@ -1,187 +1,317 @@
---
lang: en
layout: doc
permalink: /gsod/
ref: 242
title: Google Season of Docs (GSoD)
---
============================
Google Season of Docs (GSoD)
============================
Thank you for your interest in participating in the [2024 Google Season of Docs](https://developers.google.com/season-of-docs/) program with the [Qubes OS team](/team/). This page details our 2024 project idea as well as completed past projects. You can read more about the Google Season of Docs in the official [guides](https://developers.google.com/season-of-docs/docs/) and [FAQ](https://developers.google.com/season-of-docs/docs/faq).
## Update and Expand How-To Guides -- Qubes OS
Thank you for your interest in participating in the `2024 Google Season of Docs <https://developers.google.com/season-of-docs/>`__ program with the `Qubes OS team <https://www.qubes-os.org/team/>`__. This page details our 2024 project idea as well as completed past projects. You can read more about the Google Season of Docs in the official `guides <https://developers.google.com/season-of-docs/docs/>`__ and `FAQ <https://developers.google.com/season-of-docs/docs/faq>`__.
### About the Qubes OS Project
Update and Expand How-To Guides – Qubes OS
------------------------------------------
Qubes OS is a security-focused operating system that allows you to organize your digital life into compartments called "qubes." If one qube is compromised, the others remain safe, so a single cyberattack can no longer take down your entire digital life in one fell swoop. You can think of using Qubes OS as having many different computers on your desk for different activities but with the convenience of a single physical machine, a single unified desktop environment, and a set of tools for using qubes together securely as parts of a unified system.
Qubes OS was launched in 2011 and has [received praise from security experts and organizations](/endorsements/) like Edward Snowden, the Freedom of the Press Foundation, Micah Lee, and Let's Encrypt. Qubes has [over 40,000 active users](/statistics/). From network-level to software-level protections, as well as protections against firmware and hardware attacks, Qubes OS is trying to protect the user from the most significant attacks they encounter so that they can get their work done safely.
About the Qubes OS Project
^^^^^^^^^^^^^^^^^^^^^^^^^^
Qubes OS is a security-focused operating system that allows you to organize your digital life into compartments called “qubes.” If one qube is compromised, the others remain safe, so a single cyberattack can no longer take down your entire digital life in one fell swoop. You can think of using Qubes OS as having many different computers on your desk for different activities but with the convenience of a single physical machine, a single unified desktop environment, and a set of tools for using qubes together securely as parts of a unified system.
Qubes OS was launched in 2011 and has `received praise from security experts and organizations <https://www.qubes-os.org/endorsements/>`__ like Edward Snowden, the Freedom of the Press Foundation, Micah Lee, and Let’s Encrypt. Qubes has :doc:`over 40,000 active users </introduction/statistics>`. From network-level to software-level protections, as well as protections against firmware and hardware attacks, Qubes OS is trying to protect the user from the most significant attacks they encounter so that they can get their work done safely.
The project's problem
^^^^^^^^^^^^^^^^^^^^^
### The project's problem
- Some of the existing content is stale. It refers to previous Qubes releases and has not yet been updated to the cover the latest stable release.
- There are important topic areas that the current guides do not completely cover, such as "understanding Qubes OS," "using Qubes OS in practice," "Qubes OS workflow for coding," "printing," and "audio."
- There are important topic areas that the current guides do not completely cover, such as “understanding Qubes OS,” “using Qubes OS in practice,” “Qubes OS workflow for coding,” “printing,” and “audio.”
- The guides do not consistently address users of all skill and experience levels (beginner, intermediate, and expert).
- Some of the guides lack relevant illustrations and screenshots.
- When the developers update a software tool, they should update all the guides that mentions this tool. However, there is currently no way for the developers to know which guides mention which tools.
### The project's scope
The project's scope
^^^^^^^^^^^^^^^^^^^
The project is estimated to need around six months to complete (see the timeline below). Qubes team members, including Michael Carbone, Andrew Wong, Marta Marczykowska-Górecka, and Marek Marczykowski-Górecki, will supervise and support the writer.
### Measuring the project's success
Measuring the project's success
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We will consider the project successful if:
- Existing guides are updated to describe currently supported Qubes OS version
- Missing common guides are identified
- 2-3 new guides are written
### Timeline
| Dates | Action items |
| --------------- | --------------------------------------- |
| May | Orientation |
| June - November | Update & extend how-to guides |
| December | Final project evaluation and case study |
### Project budget
Timeline
^^^^^^^^
| Expense | Amount |
| --------------------------------------- | ------- |
| writer (10 hours/week, 6 months) | $12,000 |
| TOTAL | $12,000 |
### Additional information
.. list-table::
:widths: 15 15
:align: center
:header-rows: 1
Qubes OS regularly participates in Google Summer of Code and Google Season of Docs. This is our third time participating in Google Season of Docs. Our mentorships for GSoD 2019, 2020, and 2023 were successes, and the projects were completed within the times allotted. The past Google Season of Docs projects have given us experience in working with technical writers and have helped us to understand the benefits that technical writers can bring to our project.
* - Dates
- Action items
* - May
- Orientation
* - June - November
- Update & extend how-to guides
* - December
- Final project evaluation and case study
## Past Projects
You can view the project we had in 2019 in the [2019 GSoD archive](https://developers.google.com/season-of-docs/docs/2019/participants/project-qubes) and the [2019 writer's report](https://web.archive.org/web/20200928002746/https://refre.ch/report-qubesos/).
Project budget
^^^^^^^^^^^^^^
You can view the project we had in 2020 in the [2020 GSoD archive](https://developers.google.com/season-of-docs/docs/2020/participants/project-qubesos-c1e0) and the [2020 writer's report](https://web.archive.org/web/20210723170547/https://gist.github.com/PROTechThor/bfe9b8b28295d88c438b6f6c754ae733).
You can view the results of the project we had in 2023 [here](https://www.youtube.com/playlist?list=PLjwSYc73nX6aHcpqub-6lzJbL0vhLleTB).
.. list-table::
:widths: 32 32
:align: center
:header-rows: 1
* - Expense
- Amount
* - writer (10 hours/week, 6 months)
- $12,000
* - TOTAL
- $12,000
Additional information
^^^^^^^^^^^^^^^^^^^^^^
Qubes OS regularly participates in Google Summer of Code and Google Season of Docs. This is our third time participating in Google Season of Docs. Our mentorships for GSoD 2019, 2020, and 2023 were successes, and the projects were completed within the times allotted. The past Google Season of Docs projects have given us experience in working with technical writers and have helped us to understand the benefits that technical writers can bring to our project.
Past Projects
-------------
You can view the project we had in 2019 in the `2019 GSoD archive <https://developers.google.com/season-of-docs/docs/2019/participants/project-qubes>`__ and the `2019 writer’s report <https://web.archive.org/web/20200928002746/https://refre.ch/report-qubesos/>`__.
You can view the project we had in 2020 in the `2020 GSoD archive <https://developers.google.com/season-of-docs/docs/2020/participants/project-qubesos-c1e0>`__ and the `2020 writer’s report <https://web.archive.org/web/20210723170547/https://gist.github.com/PROTechThor/bfe9b8b28295d88c438b6f6c754ae733>`__.
You can view the results of the project we had in 2023 `here <https://www.youtube.com/playlist?list=PLjwSYc73nX6aHcpqub-6lzJbL0vhLleTB>`__.
Here are some successful projects which have been implemented in the past by Google Season of Docs participants.
Instructional video series
^^^^^^^^^^^^^^^^^^^^^^^^^^
.. _the-projects-problem-1:
The project's problem
^^^^^^^^^^^^^^^^^^^^^
Here are some successful projects which have been implemented in the past by Google Season of Docs participants.
### Instructional video series
#### The project's problem
There is user demand for high-quality, up-to-date video guides that take users from zero Linux knowledge to using Qubes as a daily driver and performing specific tasks inside of Qubes, but almost no such videos exist. Although most of the required knowledge is documented, many users report that they would prefer to watch videos rather than read text or that they would find videos easier to understand and follow along with.
#### The project's scope
.. _the-projects-scope-1:
The project's scope
^^^^^^^^^^^^^^^^^^^
This project consists of creating a series of instructional videos that satisfy the following criteria:
- Prospective users who are not yet familiar with Linux or Qubes OS can easily understand and follow the videos.
- The videos make a good effort to catch and keep the attention of their target audience.
- Users can follow the videos step-by-step to install Qubes OS and accomplish various tasks.
- The videos show the actual software being used (i.e., Qubes OS and any relevant software running inside of it).
- The videos are technically accurate, include security warnings where appropriate, and use terminology in a way that is consistent with the rest of the documentation (also see the [glossary](/doc/glossary/)).
- The videos are technically accurate, include security warnings where appropriate, and use terminology in a way that is consistent with the rest of the documentation (also see the :doc:`glossary </user/reference/glossary>`).
- The video series is comprehensive enough that users do not need to consult the documentation or ask questions (e.g., on the forum) in order to accomplish the most popular tasks and activities.
- The videos include voice narration. (Showing the speaker is optional.)
- The quality of the videos is consistent with current standards regarding things like editing, transitions, animations, lighting, and audio quality.
- The videos are in high definition (minimum 1080p, preferably 4k).
- The videos are separated into a series, where each video is an appropriate length and is appropriately connected to the other videos in the series.
- The videos are suitable for upload and sharing on popular video-sharing and social-media platforms, such as YouTube and Twitter. (The account or channel under which the videos are uploaded is open to discussion on platforms where the Qubes OS Project does not already have a significant established presence, such as YouTube.)
- The videos are suitable for embedding in appropriate places in the Qubes documentation. (E.g., a video on how to update Qubes OS should be appropriate for appearing on the [how to update](/doc/how-to-update/) page.)
- The videos are suitable for embedding in appropriate places in the Qubes documentation. (E.g., a video on how to update Qubes OS should be appropriate for appearing on the :doc:`how to update </user/how-to-guides/how-to-update>` page.)
- Where possible, the videos should strive to be version-independent. (For example, a video explaining the template system should still be relevant many releases from now if the template system has not changed.)
Below is an example of the content (which is already [documented](/doc/)) that the video series is likely to cover. The precise scope of content is to be determined in consultation with the video creator.
Below is an example of the content (which is already :doc:`documented </index>`) that the video series is likely to cover. The precise scope of content is to be determined in consultation with the video creator.
- Introduction to Qubes
- Selecting appropriate hardware
- How to install Qubes OS
- First steps after installing
- How to organize your qubes
- How to update
- How to back up, restore, and migrate
- How to copy and paste text (including dom0)
- How to copy and move files (including dom0)
- How to install software
- How to use and customize disposables
- How to enter fullscreen mode
- How to use devices (including block storage, USB, PCI, and optical)
- Templates: understanding, installing, uninstalling, reinstalling, etc.
- Common troubleshooting (preferably included in previous videos at appropriate points)
- The Qubes firewall
- Passwordless root
- Anti Evil Maid
- Split GPG
- CTAP proxy
- YubiKey
- Whonix
- How to install and use a VPN in Qubes
- How to install and use Windows in Qubes
- Other popular topics, as time permits
The project is estimated to need around six months to complete (see the timeline below). Qubes team members, including Michael Carbone, Andrew Wong, and Marek Marczykowski-Górecki, will supervise and support the creator.
#### Measuring the project's success
.. _measuring-the-projects-success-1:
Measuring the project's success
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We will consider the project successful if, after publication of the video series:
- Actual prospective users with no prior familiarity with Linux or Qubes OS are able to successfully install and use Qubes OS as intended by following along with the videos.
- The reception to the videos is generally positive and complaints about quality and accuracy are minimal.
- Appropriate analytics (e.g., YouTube metrics) are average or better for videos of this type (to be determined in consultation with the creator).
### Consolidate troubleshooting guides
Consolidate troubleshooting guides
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**Project**: Consolidate troubleshooting guides
**Brief explanation**: Troubleshooting guides are scattered across many pages and sometimes incomplete, leading to repeatedly posting the same instruction over and over when helping users to diagnose problems.
This could be helped by writing a consolidated guide with a clear list of symptom-action layout.
**Brief explanation**: Troubleshooting guides are scattered across many pages and sometimes incomplete, leading to repeatedly posting the same instruction over and over when helping users to diagnose problems. This could be helped by writing a consolidated guide with a clear list of symptom-action layout.
**Expected results**:
- Review existing [troubleshooting guides](/doc/#troubleshooting)
- Review [issues](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+label%3A%22C%3A+doc%22) containing common troubleshooting steps (checking specific logs etc)
- Review existing :ref:`troubleshooting guides <troubleshooting>`
- Review `issues <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+label%3A%22C%3A+doc%22>`__ containing common troubleshooting steps (checking specific logs etc)
- Propose updated, consolidated troubleshooting documentation, including its layout
**Knowledge prerequisite**:
- [Markdown](https://daringfireball.net/projects/markdown/)
- `Markdown <https://daringfireball.net/projects/markdown/>`__
**Mentor**: [Marek Marczykowski-Górecki](/team/)
### Improve Getting Started page
**Mentor**: `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__
Improve Getting Started page
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**Project**: Improve Getting Started page
**Brief explanation**: The [Getting Started page](/doc/getting-started/) is the place a new user would go to understand better how to use Qubes. It is currently has old screenshots not using the default desktop environment and could have much better flow. In addition, this improved page content may end up being served more directly to the user via the [offline documentation](https://github.com/QubesOS/qubes-issues/issues/1019) or the firstboot guide.
**Expected results**:
- Review the existing page and website, similar pages for other OSes
- Provide visual mock-ups and proposed text
**Knowledge prerequisite**:
- basic Qubes OS knowledge
- [Markdown](https://daringfireball.net/projects/markdown/)
**Mentor**: [Michael Carbone](/team/)
### Rewrite qrexec documentation
**Project**: Rewrite qrexec documentation
**Brief explanation**: Current qrexec (qubes remote exec) documentation is hard to follow, important information is hidden within a wall of text.
Some parts are split into multiple sections, for example version specific to avoid duplication, but it doesn't help reading it.
Additionally, protocol documentation describes only few specific use cases, instead of being clear and precise protocol specification.
Fixing this last point may require very close cooperation with developers, as the current documentation doesn't multiple corner cases (that's one of the issue with its current shape).
**Brief explanation**: The :doc:`Getting Started page </introduction/getting-started>` is the place a new user would go to understand better how to use Qubes. It is currently has old screenshots not using the default desktop environment and could have much better flow. In addition, this improved page content may end up being served more directly to the user via the `offline documentation <https://github.com/QubesOS/qubes-issues/issues/1019>`__ or the firstboot guide.
**Expected results**:
- Review existing [qrexec documentation](/doc/qrexec3/) and an [issue about it](https://github.com/QubesOS/qubes-issues/issues/1392)
- Propose updated, consolidated admin documentation (policy writing, adding services)
- Propose consolidated protocol specification, based on the current documentation, and cooperation with developers
- Review the existing page and website, similar pages for other OSes
- Provide visual mock-ups and proposed text
**Knowledge prerequisite**:
- [Markdown](https://daringfireball.net/projects/markdown/)
- basic Qubes OS knowledge
**Mentor**: [Marek Marczykowski-Górecki](/team/)
- `Markdown <https://daringfireball.net/projects/markdown/>`__
**Mentor**: `Michael Carbone <https://www.qubes-os.org/team/>`__
Rewrite qrexec documentation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
**Project**: Rewrite qrexec documentation
**Brief explanation**: Current qrexec (qubes remote exec) documentation is hard to follow, important information is hidden within a wall of text. Some parts are split into multiple sections, for example version specific to avoid duplication, but it doesn’t help reading it. Additionally, protocol documentation describes only few specific use cases, instead of being clear and precise protocol specification. Fixing this last point may require very close cooperation with developers, as the current documentation doesn’t multiple corner cases (that’s one of the issue with its current shape).
**Expected results**:
- Review existing :doc:`qrexec documentation </developer/services/qrexec>` and an `issue about it <https://github.com/QubesOS/qubes-issues/issues/1392>`__
- Propose updated, consolidated admin documentation (policy writing, adding services)
- Propose consolidated protocol specification, based on the current documentation, and cooperation with developers
**Knowledge prerequisite**:
- `Markdown <https://daringfireball.net/projects/markdown/>`__
**Mentor**: `Marek Marczykowski-Górecki <https://www.qubes-os.org/team/>`__

198
developer/general/how-to-edit-the-documentation.md
View file

@ -1,198 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/how-to-edit-the-documentation/
title: How to edit the documentation
---
_Also see the [documentation style guide](/doc/documentation-style-guide/)._
Qubes OS documentation pages are stored as plain text Markdown files in the
[qubes-doc](https://github.com/QubesOS/qubes-doc) repository. By cloning and
regularly pulling from this repo, users can maintain their own up-to-date
offline copy of all Qubes documentation rather than relying solely on the web.
The documentation is a volunteer community effort. People like you are
constantly working to make it better. If you notice something that can be fixed
or improved, please follow the steps below to open a pull request!
## How to submit a pull request
We keep all the documentation in
[qubes-doc](https://github.com/QubesOS/qubes-doc), a dedicated Git repository
hosted on [GitHub](https://github.com/). Thanks to GitHub's easy web interface,
you can edit the documentation even if you're not familiar with Git or the
command line! All you need is a free GitHub account.
A few notes before we get started:
- Since Qubes is a security-oriented project, every documentation change will
be [reviewed](#security) before it's accepted. This allows us to maintain
quality control and protect our users.
- To give your contribution a better chance of being accepted, please follow
our [documentation style guide](/doc/documentation-style-guide/).
- We don't want you to spend time and effort on a contribution that we can't
accept. If your contribution would take a lot of time, please [file an
issue](/doc/issue-tracking/) for it first so that we can make sure we're on
the same page before significant works begins.
- Alternatively, you may already have written content that doesn't conform to
these guidelines, but you'd be willing to modify it so that it does. In this
case, you can still submit it by following the instructions below. Just make
a note in your pull request (PR) that you're aware of the changes that need
to be made and that you're just asking for the content to be reviewed before
you spend time making those changes.
- Finally, if you've written something that doesn't belong in qubes-doc but
that would be beneficial to the Qubes community, please consider adding it to
the [external
documentation](/doc/documentation-style-guide/#core-vs-external-documentation).
(**Advanced users:** If you're already familiar with GitHub or wish to work
from the command line, you can skip the rest of this section. All you need to
do to contribute is to [fork and
clone](https://guides.github.com/activities/forking/) the
[qubes-doc](https://github.com/QubesOS/qubes-doc) repo, make your changes, then
[submit a pull
request](https://help.github.com/articles/using-pull-requests/).)
Ok, let's begin. Every documentation page has a "Page Source on GitHub" button.
Depending on the size of your screen, it may either be on the side (larger
screens) or on the bottom (smaller screens).
[![page-source-button](/attachment/doc/doc-pr_01_page-source-button.png)](/attachment/doc/doc-pr_01_page-source-button.png)
When you click on it, you'll be taken to the source file --- usually a Markdown
(`.md`) file --- on GitHub. On this page, there will be a button to edit the
file.
[![github-edit](/attachment/doc/doc-pr_02_github-edit.png)](/attachment/doc/doc-pr_02_github-edit.png)
You'll be prompted to sign in with your GitHub username and password
(if you aren't already logged in). You can also create a free account from here.
[![github-sign-in](/attachment/doc/doc-pr_03_sign-in.png)](/attachment/doc/doc-pr_03_sign-in.png)
If this is your first contribution to the documentation, you need to "fork" the
repository (make your own copy). It's easy --- just click the big green button
on the next page. This step is only needed the first time you make a
contribution.
[![fork](/attachment/doc/doc-pr_04_fork.png)](/attachment/doc/doc-pr_04_fork.png)
Now you can make your modifications. You can also preview the changes to see
how they'll be formatted by clicking the "Preview changes" tab. If you want to
add images, please see [How to add images](#how-to-add-images). If you're
making formatting changes, please [render the site
locally](https://github.com/QubesOS/qubesos.github.io#instructions) to verify
that everything looks correct before submitting any changes.
[![edit](/attachment/doc/doc-pr_05_edit.png)](/attachment/doc/doc-pr_05_edit.png)
Once you're finished, describe your changes at the bottom and click "Propose
file change".
[![commit](/attachment/doc/doc-pr_06_commit-msg.png)](/attachment/doc/doc-pr_06_commit-msg.png)
After that, you'll see exactly what modifications you've made. At this stage,
those changes are still in your own copy of the documentation ("fork"). If
everything looks good, send those changes to us by pressing the "Create pull
request" button.
[![pull-request](/attachment/doc/doc-pr_07_review-changes.png)](/attachment/doc/doc-pr_07_review-changes.png)
You will be able to adjust the pull request message and title there. In most
cases, the defaults are ok, so you can just confirm by pressing the "Create
pull request" button again. However, if you're not ready for your PR to be
reviewed or merged yet, please [make a draft PR
instead](https://github.blog/2019-02-14-introducing-draft-pull-requests/).
[![pull-request-confirm](/attachment/doc/doc-pr_08_create-pull-request.png)](/attachment/doc/doc-pr_08_create-pull-request.png)
If any of your changes should be reflected in the [documentation index (a.k.a.
table of contents)](/doc/) --- for example, if you're adding a new page,
changing the title of an existing page, or removing a page --- please see [How
to edit the documentation index](#how-to-edit-the-documentation-index).
That's all! We will review your changes. If everything looks good, we'll pull
them into the official documentation. Otherwise, we may have some questions for
you, which we'll post in a comment on your pull request. (GitHub will
automatically notify you if we do.) If, for some reason, we can't accept your
pull request, we'll post a comment explaining why we can't.
[![done](/attachment/doc/doc-pr_09_done.png)](/attachment/doc/doc-pr_09_done.png)
## How to edit the documentation index
The source file for the [documentation index (a.k.a. table of contents)](/doc/)
is
[doc-index.yml](https://github.com/QubesOS/qubesos.github.io/blob/master/_data/doc-index.yml).
Editing this file will change what appears on the documentation index. If your
pull request (PR) adds, removes, or edits anything that should be reflected in
the documentation index, please make sure you also submit an associated pull
request against this file.
## How to add images
To add an image to a page, use the following syntax in the main document (see
[here](/doc/documentation-style-guide/#image-linking) for why this syntax is
important).
```
[![Image Title](/attachment/doc/image.png)](/attachment/doc/image.png)
```
Then, submit your image(s) in a separate pull request to the
[qubes-attachment](https://github.com/QubesOS/qubes-attachment) repository
using the same path and filename. This is the only permitted way to include
images. Do not link to images on other websites.
## Serving the website locally
You can serve the website offline on your local machine by following [these
instructions](https://github.com/QubesOS/qubesos.github.io#instructions). This
can be useful for making sure that your changes render the way you expect,
especially when your changes affect formatting, images, tables, styling, etc.
## Security
*Also see: [Should I trust this website?](/faq/#should-i-trust-this-website)*
All pull requests (PRs) against
[qubes-doc](https://github.com/QubesOS/qubes-doc) must pass review prior to be
merged, except in the case of [external
documentation](/doc/#external-documentation) (see
[#4693](https://github.com/QubesOS/qubes-issues/issues/4693)). This process is
designed to ensure that contributed text is accurate and non-malicious. This
process is a best effort that should provide a reasonable degree of assurance,
but it is not foolproof. For example, all text characters are checked for ANSI
escape sequences. However, binaries, such as images, are simply checked to
ensure they appear or function the way they should when the website is
rendered. They are not further analyzed in an attempt to determine whether they
are malicious.
Once a pull request passes review, the reviewer should add a signed comment
stating, "Passed review as of `<LATEST_COMMIT>`" (or similar). The
documentation maintainer then verifies that the pull request is mechanically
sound (no merge conflicts, broken links, ANSI escapes, etc.). If so, the
documentation maintainer then merges the pull request, adds a PGP-signed tag to
the latest commit (usually the merge commit), then pushes to the remote. In
cases in which another reviewer is not required, the documentation maintainer
may review the pull request (in which case no signed comment is necessary,
since it would be redundant with the signed tag).
## Questions, problems, and improvements
If you have a question about something you read in the documentation or about
how to edit the documentation, please post it on the
[forum](https://forum.qubes-os.org/) or send it to the appropriate [mailing
list](/support/). If you see that something in the documentation should be
fixed or improved, please [contribute](#how-to-submit-a-pull-request) the
change yourself. To report an issue with the documentation, please follow our
standard [issue reporting guidelines](/doc/issue-tracking/). (If you report an
issue with the documentation, you will likely be asked to submit a pull request
for it, unless there is a clear indication in your report that you are not
willing or able to do so.)

134
developer/general/package-contributions.rst
View file

@ -1,102 +1,106 @@
---
lang: en
layout: doc
permalink: /doc/package-contributions/
ref: 29
title: Package contributions
---
=====================
Package contributions
=====================
_This page is for developers who wish to contribute packages.
If you want to install contributed packages, please see [installing contributed packages](/doc/installing-contributed-packages/)._
We're very grateful to the talented and hard-working community members who contribute software packages to Qubes OS.
This page explains the inclusion criteria and procedures for such packages, as well as the roles and responsibilities of those involved.
*This page is for developers who wish to contribute packages. If you want to install contributed packages, please see* :doc:`installing contributed packages </user/advanced-topics/installing-contributed-packages>` *.*
We’re very grateful to the talented and hard-working community members who contribute software packages to Qubes OS. This page explains the inclusion criteria and procedures for such packages, as well as the roles and responsibilities of those involved.
Inclusion Criteria
------------------
In order to be accepted, packages must:
* In no way weaken the security of Qubes OS.
* Be published under an open-source license (read about the [Qubes OS License](/doc/license/)).
* Follow our [coding guidelines](/doc/coding-style/).
* Be thoroughly tested.
* Have a clearly-defined use case for Qubes users.
* Not be unduly burdensome to review.
- In no way weaken the security of Qubes OS.
- Be published under an open-source license (read about the :doc:`Qubes OS License </developer/code/license>`).
- Follow our :doc:`coding guidelines </developer/code/coding-style>`.
- Be thoroughly tested.
- Have a clearly-defined use case for Qubes users.
- Not be unduly burdensome to review.
(Please note that we always reserve the right to add criteria to this list.)
Contribution Procedure
----------------------
Before you start putting serious work into a package, we recommend that you discuss your idea with the Qubes developers and the broader community on the [qubes-devel mailing list](/support/#qubes-devel).
Once you have a package that's ready to become part of Qubes OS, please follow this procedure:
1. Ensure that your package satisfies the [Inclusion Criteria](#inclusion-criteria).
2. If your code isn't already on GitHub, create a GitHub repo that contains your code. You can have a look to an example package called [qubes-skeleton](https://github.com/QubesOS-contrib/qubes-skeleton).
3. If you haven't already, [sign your code](/doc/code-signing/).
4. Create an issue in [qubes-issues](https://github.com/QubesOS/qubes-issues/issues/) with the title `[Contribution] your-package-name`.
Include a link to your repo, a brief description of your package, and a brief explanation of why you think it should be included in Qubes.
Please note that the Qubes core developers are very busy.
If they are under heavy load when you submit your contribution, it may be a very long time before they have time to review your package.
If this happens, please do not be discouraged.
If you think they may have forgotten about your pending contribution, you may "bump" your request by commenting on your issue, but please do this *very* sparingly (i.e., no more than once a month).
We appreciate your understanding!
5. You may be asked followup questions.
If we decide to accept your contribution, you will be invited to join the [QubesOS-contrib](https://github.com/QubesOS-contrib) organization on GitHub as public recognition of your contribution (but without push access; see [Review Procedure](#review-procedure)), and [QubesOS-contrib](https://github.com/QubesOS-contrib) will fork your repo.
If we decide not to accept your contribution, we will state the reason and close the issue.
Before you start putting serious work into a package, we recommend that you discuss your idea with the Qubes developers and the broader community on the :ref:`qubes-devel mailing list <introduction/support:qubes-devel>`. Once you have a package that’s ready to become part of Qubes OS, please follow this procedure:
1. Ensure that your package satisfies the `Inclusion Criteria <#inclusion-criteria>`__.
2. If your code isn’t already on GitHub, create a GitHub repo that contains your code. You can have a look to an example package called `qubes-skeleton <https://github.com/QubesOS-contrib/qubes-skeleton>`__.
3. If you haven’t already, :doc:`sign your code </developer/code/code-signing>`.
4. Create an issue in `qubes-issues <https://github.com/QubesOS/qubes-issues/issues/>`__ with the title ``[Contribution] your-package-name``. Include a link to your repo, a brief description of your package, and a brief explanation of why you think it should be included in Qubes. Please note that the Qubes core developers are very busy. If they are under heavy load when you submit your contribution, it may be a very long time before they have time to review your package. If this happens, please do not be discouraged. If you think they may have forgotten about your pending contribution, you may “bump” your request by commenting on your issue, but please do this *very* sparingly (i.e., no more than once a month). We appreciate your understanding!
5. You may be asked followup questions. If we decide to accept your contribution, you will be invited to join the `QubesOS-contrib <https://github.com/QubesOS-contrib>`__ organization on GitHub as public recognition of your contribution (but without push access; see `Review Procedure <#review-procedure>`__), and `QubesOS-contrib <https://github.com/QubesOS-contrib>`__ will fork your repo. If we decide not to accept your contribution, we will state the reason and close the issue.
Update Procedure
----------------
*Anyone* can provide an update (patch) to a contributed package, not just the person who contributed that package!
The update procedure is the same for everyone, including the original package contributor.
If you would like to update an already-contributed package (specifically, a fork owned by [QubesOS-contrib](https://github.com/QubesOS-contrib)), please submit a [signed](/doc/code-signing/), fast-forwardable pull request to that repo with your changes.
Please note that your pull request **must** be both [signed](/doc/code-signing/) and fast-forwardable, or else it will be closed without further review.
One or more reviewers may post comments on your pull request.
Please be prepared to read and respond to these comments.
*Anyone* can provide an update (patch) to a contributed package, not just the person who contributed that package! The update procedure is the same for everyone, including the original package contributor.
If you would like to update an already-contributed package (specifically, a fork owned by `QubesOS-contrib <https://github.com/QubesOS-contrib>`__), please submit a :doc:`signed </developer/code/code-signing>`, fast-forwardable pull request to that repo with your changes. Please note that your pull request **must** be both :doc:`signed </developer/code/code-signing>` and fast-forwardable, or else it will be closed without further review. One or more reviewers may post comments on your pull request. Please be prepared to read and respond to these comments.
Review Procedure
----------------
This review procedure covers both original package contributions (see [Contribution Procedure](#contribution-procedure)) and all subsequent updates to those packages, including updates from the original package contributor (see [Update Procedure](#update-procedure)).
All changes will be reviewed by a Qubes Core Reviewer (QCR) and the [Package Maintainer](#package-maintainers) (PM).
In all cases, the QCR will be a core Qubes developer.
In some cases, the QCR and the PM will be the same person.
For example, if someone contributes a package, then disappears, and no suitable replacement has been found, then it is likely that a core Qubes developer will play both the QCR and PM roles for that package, at least until another suitable candidate volunteers to become the PM for that package.
This review procedure covers both original package contributions (see `Contribution Procedure <#contribution-procedure>`__) and all subsequent updates to those packages, including updates from the original package contributor (see `Update Procedure <#update-procedure>`__). All changes will be reviewed by a Qubes Core Reviewer (QCR) and the `Package Maintainer <#package-maintainers>`__ (PM). In all cases, the QCR will be a core Qubes developer. In some cases, the QCR and the PM will be the same person. For example, if someone contributes a package, then disappears, and no suitable replacement has been found, then it is likely that a core Qubes developer will play both the QCR and PM roles for that package, at least until another suitable candidate volunteers to become the PM for that package.
The review procedure is as follows:
1. Someone, S, wishes to make a change to a package, P.
2. S submits a fast-forwardable pull request against the fork of P's repo owned by [QubesOS-contrib](https://github.com/QubesOS-contrib).
3. The PM reviews the pull request.
If the pull request passes the PM's review, the PM adds a [signed](/doc/code-signing/) *comment* on the pull request stating that it has passed review.
(In cases in which S = PM, the PM can simply add a [signed](/doc/code-signing/) *tag* to the HEAD commit prior to submitting the pull request.)
If the pull request does not pass the PM's review, the PM leaves a comment on the pull request explaining why not.
4. The QCR reviews the pull request.
If the pull request passes the QCR's review, the QCR pushes a [signed](/doc/code-signing/) tag to the HEAD commit stating that it has passed review and fast-forward merges the pull request.
If the pull request does not pass the QCR's review, the QCR leaves a comment on the pull request explaining why not, and the QCR may decide to close the pull request.
In all the cases, the first condition to be validated by the QCR's review is to ensure that the contribution **will not** hijack any core packages of [QubesOS](https://github.com/QubesOS) and of course, none of the [QubesOS-contrib](https://github.com/QubesOS-contrib) packages too. More precisely, particular attention to the whole build pipeline will be made with a specific review of:
2. S submits a fast-forwardable pull request against the fork of P’s repo owned by `QubesOS-contrib <https://github.com/QubesOS-contrib>`__.
* Package dependencies,
* Build scripts (including downloaded ones),
* All downloaded components should be verified against static hash,
* RPM/DEB installation scripts (e.g. looking at constraints who would hijack other packages),
* Makefiles,
* Package build [reproducible](https://reproducible-builds.org/)
3. The PM reviews the pull request. If the pull request passes the PM’s review, the PM adds a :doc:`signed </developer/code/code-signing>` *comment* on the pull request stating that it has passed review. (In cases in which S = PM, the PM can simply add a :doc:`signed </developer/code/code-signing>` *tag* to the HEAD commit prior to submitting the pull request.) If the pull request does not pass the PM’s review, the PM leaves a comment on the pull request explaining why not.
and any steps which would result in partial/total compromise of legitimate components. For this part, you can have a look to an example package called [qubes-skeleton](https://github.com/QubesOS-contrib/qubes-skeleton).
4. The QCR reviews the pull request. If the pull request passes the QCR’s review, the QCR pushes a :doc:`signed </developer/code/code-signing>` tag to the HEAD commit stating that it has passed review and fast-forward merges the pull request. If the pull request does not pass the QCR’s review, the QCR leaves a comment on the pull request explaining why not, and the QCR may decide to close the pull request.
In all the cases, the first condition to be validated by the QCR’s review is to ensure that the contribution **will not** hijack any core packages of `QubesOS <https://github.com/QubesOS>`__ and of course, none of the `QubesOS-contrib <https://github.com/QubesOS-contrib>`__ packages too. More precisely, particular attention to the whole build pipeline will be made with a specific review of:
- Package dependencies,
- Build scripts (including downloaded ones),
- All downloaded components should be verified against static hash,
- RPM/DEB installation scripts (e.g. looking at constraints who would hijack other packages),
- Makefiles,
- Package build `reproducible <https://reproducible-builds.org/>`__
and any steps which would result in partial/total compromise of legitimate components. For this part, you can have a look to an example package called `qubes-skeleton <https://github.com/QubesOS-contrib/qubes-skeleton>`__.
Package Maintainers
-------------------
If you contribute a package, we assume that you will be the maintainer of that package, unless you tell us otherwise.
As the maintainer of the package, it is your privilege and responsibility to:
* [Review](#review-procedure) each pull request made against the package.
* Decide when the package has reached a new version, and notify the Qubes core developers when this occurs.
If you contribute a package, we assume that you will be the maintainer of that package, unless you tell us otherwise. As the maintainer of the package, it is your privilege and responsibility to:
If you do not wish to be the maintainer of your package, please let us know.
If you do not act on your maintainer duties for a given package for an extended period of time and after at least one reminder, we will assume that you no longer wish to be the maintainer for that package.
- `Review <#review-procedure>`__ each pull request made against the package.
- Decide when the package has reached a new version, and notify the Qubes core developers when this occurs.
If you do not wish to be the maintainer of your package, please let us know. If you do not act on your maintainer duties for a given package for an extended period of time and after at least one reminder, we will assume that you no longer wish to be the maintainer for that package.

79
developer/general/research.rst
View file

@ -1,17 +1,66 @@
---
lang: en
layout: site
permalink: /research/
redirect_from:
- /doc/qubes-research/
- /en/doc/qubes-research/
- /doc/QubesResearch/
- /wiki/QubesResearch/
ref: 139
title: Research
---
========
Research
========
Here are links to various research papers, projects, videos, and blog
posts related to Qubes OS.
Secure Software Development
===========================
- `Security challenges for the Qubes build process <https://blog.invisiblethings.org/2016/05/30/build-security.html>`__ by Joanna Rutkowska, May 2016
Towards Trusted Hardware
========================
- `Thoughts on the "physically secure" ORWL computer <https://blog.invisiblethings.org/2016/09/03/thoughts-about-orwl.html>`__ by Joanna Rutkowska, September 2016
- `State considered harmful <https://blog.invisiblethings.org/papers/2015/state_harmful.pdf>`__ by Joanna Rutkowska, December 2015
- `Intel x86 considered harmful <https://blog.invisiblethings.org/papers/2015/x86_harmful.pdf>`__ by Joanna Rutkowska, October 2015
Intel's Safeguard Extensions
============================
- `Thoughts on Intel's upcoming SoftwareGuard Extensions (Part 2) <https://blog.invisiblethings.org/2013/09/23/thoughts-on-intels-upcoming-software.html>`__ by Joanna Rutkowska, September 2013
- `Thoughts on Intel's upcoming SoftwareGuard Extensions (Part 1) <https://blog.invisiblethings.org/2013/08/30/thoughts-on-intels-upcoming-software.html>`__ by Joanna Rutkowska, August 2013
Attacks on Intel TXT
====================
- `Attacking Intel TXT® via SINIT code execution hijacking <https://invisiblethingslab.com/resources/2011/Attacking_Intel_TXT_via_SINIT_hijacking.pdf>`__ by Rafal Wojtczuk and Joanna Rutkowska, November 2011
- `Another Way to Circumvent Intel® Trusted Execution Technology <https://invisiblethingslab.com/resources/misc09/Another%20TXT%20Attack.pdf>`__ by Rafal Wojtczuk, Joanna Rutkowska, Alex Tereshkin, December 2009
- `ACPI: Design Principles and Concerns <https://cyber.gouv.fr/sites/default/files/IMG/pdf/article_acpi.pdf>`__ by Loic Duflot, Olivier Levillain, and Benjamin Morin, 2009
- `Attacking Intel® Trusted Execution Technology <https://invisiblethingslab.com/resources/bh09dc/Attacking%20Intel%20TXT%20-%20paper.pdf>`__ by Rafal Wojtczuk and Joanna Rutkowska
Software Attacks Coming Through Devices
=======================================
- `Following the White Rabbit: Software Attacks against Intel® VT-d <https://invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf>`__ by Rafal Wojtczuk and Joanna Rutkowska, April 2011
- `On Formally Verified Microkernels (and on attacking them) <https://blog.invisiblethings.org/2010/05/03/on-formally-verified-microkernels-and.html>`__ by Joanna Rutkowska, May 2010
- `Remotely Attacking Network Cards (or why we do need VT-d and TXT) <https://blog.invisiblethings.org/2010/04/30/remotely-attacking-network-cards-or-why.html>`__ by Joanna Rutkowska, April 2010
- `Can you still trust your network card? <https://cyber.gouv.fr/sites/default/files/IMG/pdf/csw-trustnetworkcard.pdf>`__ by Loïc Duflot, Yves-Alexis Perez and others
Application-level Security
==========================
- `Virtics: A System for Privilege Separation of Legacy Desktop Applications <https://www2.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-70.pdf>`__ by Matt Piotrowski, May 2010
Compartmentalization, Isolation, and Separation
===============================================
- `Software compartmentalization vs. physical separation <https://invisiblethingslab.com/resources/2014/Software_compartmentalization_vs_physical_separation.pdf>`__ by Joanna Rutkowska, August 2014
- `Breaking Up is Hard to Do: Security and Functionality in a Commodity Hypervisor <https://tjd.phlegethon.org/words/sosp11-xoar.pdf>`__ by Patrick Colp at el., October 2011
The Qubes Architecture
======================
- `Qubes virtual mini-summit 2021 <https://www.qubes-os.org/news/2021/07/30/minisummit-agenda/>`__ by 3mdeb and the Qubes team, August 2021
- `Qubes Air: Generalizing the Qubes Architecture <https://www.qubes-os.org/news/2018/01/22/qubes-air/>`__ by Joanna Rutkowska, January 2018
- `Introducing the Next Generation Qubes Core Stack <https://www.qubes-os.org/news/2017/10/03/core3/>`__ by Joanna Rutkowska, October 2017
- `Introducing the Qubes Admin API <https://www.qubes-os.org/news/2017/06/27/qubes-admin-api/>`__ by Joanna Rutkowska, June 2017
- `Qubes OS Architecture Spec v0.3 <https://www.qubes-os.org/attachment/doc/arch-spec-0.3.pdf>`__ by Joanna Rutkowska and Rafal Wojtczuk, January 2010
Here are links to various research papers, projects, videos, and blog posts
related to Qubes OS.
{% include research.html %}

325
developer/general/usability-ux.rst
View file

@ -1,230 +1,307 @@
---
lang: en
layout: doc
permalink: /doc/usability-ux/
ref: 31
title: Usability & UX
---
==============
Usability & UX
==============
Software that is too complicated to use, is often unused. Because we want as many people as possible to benefit from its unique security properties, the usability and user experience of Qubes OS is an utmost priority!
We ask anyone developing for Qubes OS to please read through this guide to better understand the user experience we strive to achieve. We also ask them to review [our visual style guide](/doc/visual-style-guide/) for other design related information.
We ask anyone developing for Qubes OS to please read through this guide to better understand the user experience we strive to achieve. We also ask them to review `our visual style guide <https://www.qubes-os.org/doc/visual-style-guide/>`__ for other design related information.
---
## Easy To Use
----
Easy To Use
-----------
An ideal user experience is friendly, and it beckons a new user to explore the interface. In this process, they can naturally discover how to use the software. Below are some guidelines that will help you design a user interface that accomplishes this goal.
<div class="focus">
<i class="fa fa-times"></i> <strong>Interfaces Should Not</strong>
</div>
|redx| **Interfaces Should Not**
- Require extensive configuration before a user can *begin* doing things
- Make it possible to break provided features or actions in unrecoverable ways
- Perform actions which compromise security and data
- Overwhelm the user with too much information and cognitive load
Perhaps the most common cause of mistakes is complexity. If there is a configuration setting that will significantly affect the user's experience, choose a safe and smart default then tuck this setting in an `Advanced Settings` panel.
<div class="focus">
<i class="fa fa-check"></i> <strong>Interfaces Should</strong>
</div>
Perhaps the most common cause of mistakes is complexity. If there is a configuration setting that will significantly affect the user’s experience, choose a safe and smart default then tuck this setting in an ``Advanced Settings`` panel.
|checkmark| **Interfaces Should**
- Make it easy to discover features and available actions
- Provide some understanding of what discovered features do
- Offer the ability to easily undo mistakes
- Choose intelligent defaults for settings
In making software easy to use, it is crucial to be mindful of [cognitive load](https://en.wikipedia.org/wiki/Cognitive_load) which dictates that *"humans are generally able to hold only seven +/- two units of information in short-term memory."* Making sure your interfaces don't pass this short-term memory limit is perhaps the most important factor in helping a user feel comfortable instead of overwhelmed.
---
## Easy to Understand
In making software easy to use, it is crucial to be mindful of `cognitive load <https://en.wikipedia.org/wiki/Cognitive_load>`__ which dictates that *“humans are generally able to hold only seven +/- two units of information in short-term memory.”* Making sure your interfaces don’t pass this short-term memory limit is perhaps the most important factor in helping a user feel comfortable instead of overwhelmed.
----
Easy to Understand
------------------
There will always be the need to communicate things to users. In these cases, an interface should aim to make this information easy to understand. The following are simple guides to help achieve this - none of these are absolute maxims!
<div class="focus">
<i class="fa fa-times"></i> <strong>Avoid Acronyms</strong>
</div>
|redx| **Avoid Acronyms**
Acronyms are compact and make good names for command line tools. They do not make graphical user interfaces more intuitive for non-technical users. Until one learns an acronym's meaning, it is gibberish. Avoid acronyms in your interfaces whenever possible!
Acronyms are compact and make good names for command line tools. They do not make graphical user interfaces more intuitive for non-technical users. Until one learns an acronym’s meaning, it is gibberish. Avoid acronyms in your interfaces whenever possible!
- `DVM` - Disposable Virtual Machine
- `GUID` - Global Unique Identifier
- `PID` - Process Identification Number
- `NetVM` - Networking Virtual Machine
- ``DVM`` - Disposable Virtual Machine
Despite this rule, some acronyms like `USB` are widely used and understood due to being in common use for over a decade. It is good to use these acronyms when the full words like `Universal Serial Bus` are more likely to confuse users.
- ``GUID`` - Global Unique Identifier
<div class="focus">
<i class="fa fa-check"></i> <strong> Use Simple Words</strong>
</div>
- ``PID`` - Process Identification Number
Use the minimum amount of words needed to be descriptive, but also informative. Go with common words that are as widely understood. Sometimes, inventing a word such as `Qube` to describe a `virtual machine` makes the life of the user much easier.
- ``NetVM`` - Networking Virtual Machine
- Use `Disposable Qube` instead of `DVM` or `Disposable Virtual Machine`
- Use `interface` instead of `GUI` or `Graphical User Interface`
- Use `application number` instead of `PID` or `Process Identification Number`
- Use `Networking` or `Networking Qube` instead of `NetVM` given context
---
<div class="focus">
<i class="fa fa-times"></i> <strong> Avoid Technical Words</strong>
</div>
Despite this rule, some acronyms like ``USB`` are widely used and understood due to being in common use for over a decade. It is good to use these acronyms when the full words like ``Universal Serial Bus`` are more likely to confuse users.
|checkmark| **Use Simple Words**
Use the minimum amount of words needed to be descriptive, but also informative. Go with common words that are as widely understood. Sometimes, inventing a word such as ``Qube`` to describe a ``virtual machine`` makes the life of the user much easier.
- Use ``Disposable Qube`` instead of ``DVM`` or ``Disposable Virtual Machine``
- Use ``interface`` instead of ``GUI`` or ``Graphical User Interface``
- Use ``application number`` instead of ``PID`` or ``Process Identification Number``
- Use ``Networking`` or ``Networking Qube`` instead of ``NetVM`` given context
----
|redx| **Avoid Technical Words**
Technical words are usually more accurate, but they often *only* make sense to technical users and are confusing and unhelpful to non-technical users. Examples of technical words that might show up in Qubes OS are:
- `root.img`
- `savefile`
- `qrexec-daemon`
- ``root.img``
These are all terms that have at some point showed up in users' notification messages. Each term is very specific, but requires the user to understand virtualization to interpret.
- ``savefile``
<div class="focus">
<i class="fa fa-check"></i> <strong> Use Common Concepts</strong>
</div>
- ``qrexec-daemon``
These are all terms that have at some point showed up in users’ notification messages. Each term is very specific, but requires the user to understand virtualization to interpret.
|checkmark| **Use Common Concepts**
Large amounts of the global population have been using computers for one or two decades and have formed some mental models of how things work. Leveraging these mental models are a huge gain.
- Use `disk space` instead of `root.img`, since while not quite accurate, it makes contextual sense
- Use `saving` instead of `savefile` as the former is the action trying to be completed
- Use `Qubes` instead of `qrexec-daemon` as it gives better context on what is happening
- Use ``disk space`` instead of ``root.img``, since while not quite accurate, it makes contextual sense
- Use ``saving`` instead of ``savefile`` as the former is the action trying to be completed
- Use ``Qubes`` instead of ``qrexec-daemon`` as it gives better context on what is happening
These words are more abstract and user relevant- they help a user understand what is happening based on already known concepts (disk space) or start to form a mental model of something new (Qubes).
---
<div class="focus">
<i class="fa fa-times"></i> <strong>Avoid Inconsistencies</strong>
</div>
----
|redx| **Avoid Inconsistencies**
It is easy to start abbreviating (or making acronyms) of long terms like ``Disposable Virtual Machine`` depending on where the term shows up in an interface.
- ``DVM``
- ``DispVM``
- ``DisposableVM``
It is easy to start abbreviating (or making acronyms) of long terms like `Disposable Virtual Machine` depending on where the term shows up in an interface.
- `DVM`
- `DispVM`
- `DisposableVM`
This variation in terms can cause new users to question or second guess what the three different variations mean, which can lead to inaction or mistakes.
<div class="focus">
<i class="fa fa-check"></i> <strong> Make Things Consistent</strong>
</div>
|checkmark| **Make Things Consistent**
Always strive to keep things consistent in the interfaces as well as documentation and other materials.
- Use `Disposable Qube` at all times as it meets other criteria as well.
- Use ``Disposable Qube`` at all times as it meets other criteria as well.
By using the same term throughout an interface, a user can create a mental model and relationship with that term allowing them to feel empowered.
---
<div class="focus">
<i class="fa fa-times"></i> <strong>Avoid Duplicate Words</strong>
</div>
----
It is easy to add words like `Domain` before items in a list or menu in an attempt to be descriptive, such as:
~~~
Menu
- Domain: work
- Domain: banking
- Domain: personal
~~~
|redx| **Avoid Duplicate Words**
The repeated use of the word `Domain` requires a user to read it for each item in the list, which makes extra work for the eye in parsing out the relevant word like `work, banking, or personal`. This also affects horizontal space on fixed width lines.
It is easy to add words like ``Domain`` before items in a list or menu in an attempt to be descriptive, such as:
<div class="focus">
<i class="fa fa-check"></i> <strong> Create Groups & Categories</strong>
</div>
.. code:: bash
Menu
- Domain: work
- Domain: banking
- Domain: personal
The repeated use of the word ``Domain`` requires a user to read it for each item in the list, which makes extra work for the eye in parsing out the relevant word like ``work, banking, or personal``. This also affects horizontal space on fixed width lines.
|checkmark| **Create Groups & Categories**
It is more efficient to group things under headings instead as this allows the eye to easily scan the uniqueness of the items. (As per our previous example:)
~~~
Domains
- Work
- Banking
- Personal
~~~
.. code:: bash
---
Domains
- Work
- Banking
- Personal
----
Easy To Complete
----------------
## Easy To Complete
Lastly, expected (and unexpected) situations often require user actions or input. Make resolving these occurences as easy as possible to complete the action.
<div class="focus">
<i class="fa fa-times"></i><strong>Don't Leave Users Stranded</strong>
</div>
|redx| **Don’t Leave Users Stranded**
Consider the following notifications:
- `The disk space of your Qube "Work" is full`
- `There was an error saving Qube "Personal"`
- ``The disk space of your Qube "Work" is full``
- ``There was an error saving Qube "Personal"``
Instead of displaying solvable errors like these and neglecting to provide a fix:
<div class="focus">
<i class="fa fa-check"></i><strong>Offer Actionable Solutions</strong>
</div>
|checkmark| **Offer Actionable Solutions**
Error messages and limits such as those in the previous example can be greatly improved by adding buttons or links to helpful information.
- Add a button to `Increase Disk Space`
- Add a link to a documentation page called `Troubleshoot saving data`
- Add a button to ``Increase Disk Space``
In adhering to these principles, you'll make undesirable situations more manageable for users instead of feeling stranded.
- Add a link to a documentation page called ``Troubleshoot saving data``
---
<div class="focus">
<i class="fa fa-check"></i><strong>Minimize Repetitive Steps</strong>
</div>
In adhering to these principles, you’ll make undesirable situations more manageable for users instead of feeling stranded.
----
|checkmark| **Minimize Repetitive Steps**
There are many cases where a user wants to perform an action on more than one file or folder. However in order to do the action, the user must repeat certain steps such as:
1. Click on `Open File` from a menu or button
1. Click on ``Open File`` from a menu or button
2. Navigate through file system
- Click Folder One
- Click Folder Two
- Click Folder Three
- Click Folder Four
- Click Folder One
- Click Folder Two
- Click Folder Three
- Click Folder Four
3. Select proper file
4. Complete task on file
That subtle act of clicking through a file system can prove to be significant if a user needs to open more than a couple files in the same directory. We can alleviate some of the work by changing the process:
1. Click on `Open File` from a menu or button
1. Click on ``Open File`` from a menu or button
2. Remember last open folder/file system
3. Select proper file
4. Complete task
Clearly, cutting out something as simple as navigating through the file system can save a user quite a bit of time. Alternatively, adding a button or menu item like `Open Multiple Files` might be even better, because remembering and using relevant hotkeys is often something only power users know how to do!
---
## GNOME, KDE, and Xfce
Clearly, cutting out something as simple as navigating through the file system can save a user quite a bit of time. Alternatively, adding a button or menu item like ``Open Multiple Files`` might be even better, because remembering and using relevant hotkeys is often something only power users know how to do!
The desktop GUIs that QubesOS versions 1 - 4.1 offer are [KDE](https://kde.org) and [Xfce](https://xfce.org). We are currently migrating towards using [GNOME](https://www.gnome.org). We know some people prefer KDE, but we believe Gnome is easier to use for average non-technical users. Xfce will always be supported, and technical users will always have the choice to use KDE or other desktop environments.
This change means you should use [GTK](https://gtk.org/) rather than Qt for new GUIs.
----
All three of these mentioned desktop environments have their own [human interface guidelines](https://en.wikipedia.org/wiki/Human_interface_guidelines), and we suggest you familiarize yourself with the platform you developing for.
- [GNOME Human Interface Guidelines](https://developer.gnome.org/hig/)
- [KDE HIG](https://hig.kde.org/)
- [Xfce UI Guidlines](https://wiki.xfce.org/dev/hig/general)
GNOME, KDE, and Xfce
--------------------
---
## Further Learning & Inspiration
The desktop GUIs that QubesOS versions 1 - 4.1 offer are `KDE <https://kde.org>`__ and `Xfce <https://xfce.org>`__. We are currently migrating towards using `GNOME <https://www.gnome.org>`__. We know some people prefer KDE, but we believe Gnome is easier to use for average non-technical users. Xfce will always be supported, and technical users will always have the choice to use KDE or other desktop environments.
This change means you should use `GTK <https://gtk.org/>`__ rather than Qt for new GUIs.
All three of these mentioned desktop environments have their own `human interface guidelines <https://en.wikipedia.org/wiki/Human_interface_guidelines>`__, and we suggest you familiarize yourself with the platform you developing for.
- `GNOME Human Interface Guidelines <https://developer.gnome.org/hig/>`__
- `KDE HIG <https://hig.kde.org/>`__
- `Xfce UI Guidlines <https://wiki.xfce.org/dev/hig/general>`__
----
Further Learning & Inspiration
------------------------------
Learning to make well designing intuitive interfaces and software is specialized skillset that can take years to cultivate, but if you are interested in furthering your understanding, we suggest the following resources:
- [Learn Design Principles](https://web.archive.org/web/20180101172357/http://learndesignprinciples.com/) by Melissa Mandelbaum
- [Usability in Free Software](https://jancborchardt.net/usability-in-free-software) by Jan C. Borchardt
- [Superheroes & Villains in Design](https://vimeo.com/70030549) by Aral Balkan
- [First Rule of Usability? Don’t Listen to Users](https://www.nngroup.com/articles/first-rule-of-usability-dont-listen-to-users/) by Jakob Nielsen
- [10 Usability Heuristics for User Interface Design](https://www.nngroup.com/articles/ten-usability-heuristics/) by Jakob Nielsen
- [Hack Design](https://hackdesign.org/) - online learning program
- `Learn Design Principles <https://web.archive.org/web/20180101172357/http://learndesignprinciples.com/>`__ by Melissa Mandelbaum
- `Usability in Free Software <https://jancborchardt.net/usability-in-free-software>`__ by Jan C. Borchardt
- `Superheroes & Villains in Design <https://vimeo.com/70030549>`__ by Aral Balkan
- `First Rule of Usability? Don’t Listen to Users <https://www.nngroup.com/articles/first-rule-of-usability-dont-listen-to-users/>`__ by Jakob Nielsen
- `10 Usability Heuristics for User Interface Design <https://www.nngroup.com/articles/ten-usability-heuristics/>`__ by Jakob Nielsen
- `Hack Design <https://hackdesign.org/>`__ - online learning program
.. |checkmark| image:: /attachment/doc/checkmark.png
.. |redx| image:: /attachment/doc/red_x.png

9
developer/general/visual-style-guide.md
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/visual-style-guide/
redirect_from:
- /doc/style-guide/
ref: 27
title: Visual style guide
---

78
developer/general/website-style-guide.md
View file

@ -1,78 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/website-style-guide/
title: Website style guide
---
This page explains the standards we follow for building and maintaining the
website. Please follow these guidelines and conventions when modifying the
website. For the standards governing the documentation in particular, please
see the [documentation style guide](/doc/documentation-style-guide/).
## Coding conventions
The following conventions apply to the website as a whole, including everything
written in HTML, CSS, YAML, and Liquid. These conventions are intended to keep
the codebase consistent when multiple collaborators are working on it. They
should be understood as a practical set of rules for maintaining order in this
specific codebase rather than as a statement of what is objectively right or
good.
### General practices
- Use comments to indicate the purposes of different blocks of code. This makes
the file easier to understand and navigate.
- Use descriptive variable names. Never use one or two letter variable names.
Avoid obscure abbreviations and made-up words.
- In general, make it easy for others to read your code. Your future self will
thank you, and so will your collaborators!
- [Don't Repeat Yourself
(DRY)!](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself) Instead of
repeating the same block of code multiple times, abstract it out into a
separate file and `include` that file where you need it.
### Whitespace
- Always use spaces. Never use tabs.
- Each indentation step should be exactly two (2) spaces.
- Whenever you add an opening tag, indent the following line. (Exception: If
you open and close the tag on the same line, do not indent the following
line.)
- Indent Liquid the same way as HTML.
- In general, the starting columns of every adjacent pair of lines should be no
more than two spaces apart (example below).
- No blank or empty lines. (Hint: When you feel you need one, add a comment on
that line instead.)
#### Indentation example
Here's an example that follows the indentation rules:
{% raw %}
```html
<table>
<tr>
<th title="Anchor Link"><span class="fa fa-link"></span></th>
{% for item in secs.htmlsections[0].columns %}
<th>{{ item.title }}</th>
{% endfor %}
</tr>
{% for canary in site.data.sec-canary reversed %}
<tr id="{{ canary.canary }}">
<td><a href="#{{ canary.canary }}" class="fa fa-link black-icon" title="Anchor link to Qubes Canary row: Qubes Canary #{{ canary.canary }}"></a></td>
<td>{{ canary.date }}</td>
<td><a href="https://github.com/QubesOS/qubes-secpack/blob/master/canaries/canary-{{ canary.canary }}-{{ canary.date | date: '%Y' }}.txt">Qubes Canary #{{ canary.canary }}</a></td>
</tr>
{% endfor %}
</table>
```
{% endraw %}

75
developer/releases/1_0/release-notes.rst
View file

@ -1,50 +1,63 @@
---
lang: en
layout: doc
permalink: /doc/releases/1.0/release-notes/
redirect_from:
- /en/doc/releases/1.0/release-notes/
ref: 18
title: Qubes R1.0 release notes
---
========================
Qubes R1.0 release notes
========================
Detailed release notes in [this blog post](https://blog.invisiblethings.org/2012/09/03/introducing-qubes-10.html).
## Known issues
Detailed release notes in `this blog post <https://blog.invisiblethings.org/2012/09/03/introducing-qubes-10.html>`__.
- Installer might not support some USB keyboards (\#230). This seems to include all the Mac Book keyboards (most PC laptops have PS2 keyboards and are not affected).
Known issues
------------
- If you don't enable Composition (System Setting -\> Desktop -\> Enable desktop effects), which you really should do, then the KDE task bar might get ugly (e.g. half of it might be black). This is some KDE bug that we don't plan to fix.
- Some keyboard layout set by KDE System Settings can cause [keyboard not working at all](https://groups.google.com/group/qubes-devel/browse_thread/thread/77d076b65dda7226). If you hit this issue, you can switch to console (by console login option) and manually edit `/etc/X11/xorg.conf.d/00-system-setup-keyboard.conf` (and `/etc/sysconfig/keyboard`) and place correct keyboard layout settings (details in linked thread). You can check if specific keyboard layout settings are proper using `setxkbmap` tool.
- Installer might not support some USB keyboards (#230). This seems to include all the Mac Book keyboards (most PC laptops have PS2 keyboards and are not affected).
- If you don’t enable Composition (System Setting -> Desktop -> Enable desktop effects), which you really should do, then the KDE task bar might get ugly (e.g. half of it might be black). This is some KDE bug that we don’t plan to fix.
- Some keyboard layout set by KDE System Settings can cause `keyboard not working at all <https://groups.google.com/group/qubes-devel/browse_thread/thread/77d076b65dda7226>`__. If you hit this issue, you can switch to console (by console login option) and manually edit ``/etc/X11/xorg.conf.d/00-system-setup-keyboard.conf`` (and ``/etc/sysconfig/keyboard``) and place correct keyboard layout settings (details in linked thread). You can check if specific keyboard layout settings are proper using ``setxkbmap`` tool.
- On systems with more than 8GB of RAM there is problem with DisposableVM. To fix it, limit maximum memory allocation for DispVM to 3GB
~~~
qvm-prefs -s fedora-17-x64-dvm maxmem 3072
qvm-create-default-dvm --default-template --default-script
~~~
.. code:: bash
- On some systems the KDE Window Manager might freeze upon resuming from S3 sleep when compositing is enabled (and the only method to log in to the system if this happens is to switch to a text console, enter your user's password, kill the kwin process, go back to the Xorg console, log in, and start a new instance of kwin using Konsole application :) If you experience such problems, make sure to disable compositing before putting the system into sleep by pressing Alt-Ctrl-F12 (and then enabling it back once you log in after resume) -- this way you should never see this problem again.
qvm-prefs -s fedora-17-x64-dvm maxmem 3072
qvm-create-default-dvm --default-template --default-script
## Downloads
See [Qubes Downloads](/doc/QubesDownloads/).
## Installation instructions
- On some systems the KDE Window Manager might freeze upon resuming from S3 sleep when compositing is enabled (and the only method to log in to the system if this happens is to switch to a text console, enter your user’s password, kill the kwin process, go back to the Xorg console, log in, and start a new instance of kwin using Konsole application :) If you experience such problems, make sure to disable compositing before putting the system into sleep by pressing Alt-Ctrl-F12 (and then enabling it back once you log in after resume) – this way you should never see this problem again.
See [Installation Guide](/doc/installation-guide/).
## Upgrading
### From Qubes 1.0-rc1
Downloads
---------
If you're already running Qubes 1.0-rc1, you don't need to reinstall, it's just enough to update the packages in your Dom0 and the template VM(s). The easiest way for doing this is to click on the Update Button in the Qubes Manger -- one click when you selected Dom0, and one click for each of your template VM (by default there is just one template).
### From Qubes 1.0 Beta 3
See `Qubes Downloads <https://www.qubes-os.org/downloads/>`__.
Installation instructions
-------------------------
See :doc:`Installation Guide </user/downloading-installing-upgrading/installation-guide>`.
Upgrading
---------
From Qubes 1.0-rc1
^^^^^^^^^^^^^^^^^^
If you’re already running Qubes 1.0-rc1, you don’t need to reinstall, it’s just enough to update the packages in your Dom0 and the template VM(s). The easiest way for doing this is to click on the Update Button in the Qubes Manger – one click when you selected Dom0, and one click for each of your template VM (by default there is just one template).
From Qubes 1.0 Beta 3
^^^^^^^^^^^^^^^^^^^^^
If you have Qubes Beta 3 currently installed on your system, you must reinstall from scratch, as we offer no direct upgrade option in the installer (sorry). However, we do offer tools for smooth migration of your AppVMs. In order to do that, please backup your AppVMs using the ``qvm-backup`` tool :doc:`as usual </user/how-to-guides/how-to-back-up-restore-and-migrate>`. Then, after you install Qubes 1.0 rc1, you can restore them using ``qvm-backup-restore`` tool. However, because we have changed the default template in RC1, you should tell qvm-back-restore about that by passing ``--replace-template`` option:
.. code:: bash
qvm-backup-restore <backup_dir> --replace-template=fedora-15-x64:fedora-17-x64
If you have Qubes Beta 3 currently installed on your system, you must reinstall from scratch, as we offer no direct upgrade option in the installer (sorry). However, we do offer tools for smooth migration of your AppVMs. In order to do that, please backup your AppVMs using the `qvm-backup` tool [as usual](/doc/backup-restore/). Then, after you install Qubes 1.0 rc1, you can restore them using `qvm-backup-restore` tool. However, because we have changed the default template in RC1, you should tell qvm-back-restore about that by passing `--replace-template` option:
~~~
qvm-backup-restore <backup_dir> --replace-template=fedora-15-x64:fedora-17-x64
~~~

137
developer/releases/2_0/release-notes.rst
View file

@ -1,85 +1,118 @@
---
lang: en
layout: doc
permalink: /doc/releases/2.0/release-notes/
redirect_from:
- /en/doc/releases/2.0/release-notes/
ref: 25
title: Qubes R2.0 release notes
---
========================
Qubes R2.0 release notes
========================
Detailed release notes in [this blog post](https://blog.invisiblethings.org/2014/09/26/announcing-qubes-os-release-2.html)
## New features since 1.0
Detailed release notes in `this blog post <https://blog.invisiblethings.org/2014/09/26/announcing-qubes-os-release-2.html>`__
* Support for generic fully virtualized VMs (without qemu in the TCB!)
* Support for Windows-based AppVMs integration (clipboard, file exchange, qrexec, pv drivers)
* Secure audio input to select AppVMs (Hello Skype users!)
* Clipboard is now also controlled by central policies, unified with other qrexec policies.
* Out of the box TorVM support
* Experimental support for PVUSB
* Updated Xorg packages in Dom0 to support new GPUs
* DisposableVM customization support
* Introduced Xfce 4.10 environment for Dom0 as an alternative to KDE
* Advanced infrastructure for system backups
* Ability to autostart selected VM at system startup
* Support for dynamic screen resolution change
* Dom0 distribution upgraded to Fedora 20
New features since 1.0
----------------------
## Known issues
* On some graphics cards the Xfce4 Window Manager (one of the two supported Dom0 Windows Managers in Qubes R2, the other being KDE) might behave "strangely", e.g. decorations might not be drawn sometimes. Also the accompanying lightdm login manager might incorrectly display the wallpaper. If you're facing those problems, it's advisable to use the KDE Window Manager and kdm instead of Xfce4 and lightdm (this is default if one chooses the KDE only installation option in the installer).
- Support for generic fully virtualized VMs (without qemu in the TCB!)
* Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead.
- Support for Windows-based AppVMs integration (clipboard, file exchange, qrexec, pv drivers)
* If your GPU is not correctly supported by the Dom0 kernel (e.g. the 3D desktop effects do not run smoothly) then you might experience "heaviness" with Windows 7-based AppVMs. In that case, please solve the problem with your GPU support in Dom0 in the first place (by using a different kernel), or install Qubes OS on a different system.
- Secure audio input to select AppVMs (Hello Skype users!)
* Under some circumstances, Qubes backup can create broken backup, without any visible message (\#902). It is advisable to verify a backup to spot the problem. If you encounter this problem, backup VM directory manually.
- Clipboard is now also controlled by central policies, unified with other qrexec policies.
- Out of the box TorVM support
- Experimental support for PVUSB
- Updated Xorg packages in Dom0 to support new GPUs
- DisposableVM customization support
- Introduced Xfce 4.10 environment for Dom0 as an alternative to KDE
- Advanced infrastructure for system backups
- Ability to autostart selected VM at system startup
- Support for dynamic screen resolution change
- Dom0 distribution upgraded to Fedora 20
Known issues
------------
- On some graphics cards the Xfce4 Window Manager (one of the two supported Dom0 Windows Managers in Qubes R2, the other being KDE) might behave “strangely”, e.g. decorations might not be drawn sometimes. Also the accompanying lightdm login manager might incorrectly display the wallpaper. If you’re facing those problems, it’s advisable to use the KDE Window Manager and kdm instead of Xfce4 and lightdm (this is default if one chooses the KDE only installation option in the installer).
- Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead.
- If your GPU is not correctly supported by the Dom0 kernel (e.g. the 3D desktop effects do not run smoothly) then you might experience “heaviness” with Windows 7-based AppVMs. In that case, please solve the problem with your GPU support in Dom0 in the first place (by using a different kernel), or install Qubes OS on a different system.
- Under some circumstances, Qubes backup can create broken backup, without any visible message (#902). It is advisable to verify a backup to spot the problem. If you encounter this problem, backup VM directory manually.
- System shutdown sometimes is very slow (#903). To mitigate the problem, shutdown all the VMs first.
- For other known issues take a look at `our trac tickets <https://wiki.qubes-os.org/query?status=accepted&status=assigned&status=new&status=reopened&type=defect&milestone=Release+2.1+(post+R2)&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&order=priority>`__
* System shutdown sometimes is very slow (\#903). To mitigate the problem, shutdown all the VMs first.
* For other known issues take a look at [our trac tickets](https://wiki.qubes-os.org/query?status=accepted&status=assigned&status=new&status=reopened&type=defect&milestone=Release+2.1+(post+R2)&col=id&col=summary&col=status&col=type&col=priority&col=milestone&col=component&order=priority)
It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems.
## Downloads
Downloads
---------
See [Qubes Downloads](/doc/QubesDownloads/).
## Installation instructions
See `Qubes Downloads <https://www.qubes-os.org/downloads/>`__.
See [Installation Guide](/doc/installation-guide/).
Installation instructions
-------------------------
## Upgrading
### From Qubes R2 rc1
See :doc:`Installation Guide </user/downloading-installing-upgrading/installation-guide>`.
Upgrading from Qubes R2 rc1 should be a simple matter of installing updates for [dom0](/doc/how-to-install-software-in-dom0/) and [VMs](/doc/software-update-vm/).
Upgrading
---------
### From Qubes R2 beta 3 and older
The easiest and safest way to upgrade to Qubes R2 (especially from older releases) is to install it from scratch and use [qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs.
From Qubes R2 rc1
^^^^^^^^^^^^^^^^^
Users of R2 beta 3 can upgrade using procedure that has been described [here](/doc/upgrade-to-r2/).
Note: if the user has custom Template VMs (i.e. other than the default template, e.g. created from it by cloning), or Standalone VMs, then the user should perform manual upgrade from R2B3 to R2rc1, as described under the link given above.
Upgrading from Qubes R2 rc1 should be a simple matter of installing updates for :doc:`dom0 </user/advanced-topics/how-to-install-software-in-dom0>` and :doc:`VMs </user/how-to-guides/how-to-install-software>`.
### Migrating between beta releases
From Qubes R2 beta 3 and older
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#### From Qubes R1 to R2 beta1
If you're already running Qubes Release 1, you don't need to reinstall, it's just enough to update the packages in your Dom0 and the template VM(s). This procedure is described [here](/doc/upgrade-to-r2/).
The easiest and safest way to upgrade to Qubes R2 (especially from older releases) is to install it from scratch and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs.
#### From Qubes R1 or R2 Beta 1 to R2 beta2
Users of R2 beta 3 can upgrade using procedure that has been described :doc:`here </user/downloading-installing-upgrading/upgrade/2>`.
Because of the distribution change in R2B2 (from fc13 to fc18) it's preferred that users reinstall Qubes R2B2 from scratch, and use [qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs.
**Note:** if the user has custom Template VMs (i.e. other than the default template, e.g. created from it by cloning), or Standalone VMs, then the user should perform manual upgrade from R2B3 to R2rc1, as described under the link given above.
Advanced users (and advanced users only) can also try a manual upgrade procedure that has been described [here](/doc/upgrade-to-r2b2/). It's advisable to backup your VMs before proceeding anyway!
Migrating between beta releases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#### Upgrading from Qubes R1 or R2 Beta 2 to R2 beta 3
The easiest and safest way to upgrade to Qubes R2B3 is to install it from scratch and use [qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs.
From Qubes R1 to R2 beta1
^^^^^^^^^^^^^^^^^^^^^^^^^
Users can also try a manual upgrade procedure that has been described [here](/doc/upgrade-to-r2b3/).
Note: if the user has custom Template VMs (i.e. other than the default template, e.g. created from it by cloning), or Standalone VMs, then the user should perform manual upgrade from R2B2 to R2B3, as described under the link given above.
If you’re already running Qubes Release 1, you don’t need to reinstall, it’s just enough to update the packages in your Dom0 and the template VM(s). This procedure is described :doc:`here </user/downloading-installing-upgrading/upgrade/2>`.
From Qubes R1 or R2 Beta 1 to R2 beta2
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Because of the distribution change in R2B2 (from fc13 to fc18) it’s preferred that users reinstall Qubes R2B2 from scratch, and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs.
Advanced users (and advanced users only) can also try a manual upgrade procedure that has been described :doc:`here </user/downloading-installing-upgrading/upgrade/2b2>`. It’s advisable to backup your VMs before proceeding anyway!
Upgrading from Qubes R1 or R2 Beta 2 to R2 beta 3
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The easiest and safest way to upgrade to Qubes R2B3 is to install it from scratch and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs.
Users can also try a manual upgrade procedure that has been described :doc:`here </user/downloading-installing-upgrading/upgrade/2b3>`.
**Note:** if the user has custom Template VMs (i.e. other than the default template, e.g. created from it by cloning), or Standalone VMs, then the user should perform manual upgrade from R2B2 to R2B3, as described under the link given above.

94
developer/releases/3_0/release-notes.rst
View file

@ -1,62 +1,80 @@
---
lang: en
layout: doc
permalink: /doc/releases/3.0/release-notes/
redirect_from:
- /en/doc/releases/3.0/release-notes/
ref: 19
title: Qubes R3.0 release notes
---
========================
Qubes R3.0 release notes
========================
Qubes R3.0 Release Notes
------------------------
### Qubes R3.0 Release Notes
This Qubes OS release is dedicated to the memory of Caspar Bowden.
## New features since 2.0
New features since 2.0
^^^^^^^^^^^^^^^^^^^^^^
* HAL (Hypervisor Abstraction Layer) - based on libvirt, opens a whole new
possibilities of using different hypervisors. Currently Qubes OS uses Xen.
* Xen 4.4 - many new features, but for us the most important is much more
mature libxl toolstack.
* Qrexec 3 - greatly improved performance by using direct VM-VM connections and
bigger buffers.
* Debian templates gets official support.
* Whonix templates
* Build system improvements - especially support for distribution-specific
plugins (makes supporting multiple distributions much easier) and building
templates using DispVM.
* Automated tests - makes much easier to find bugs, before its even shipped to users
## Known issues
- HAL (Hypervisor Abstraction Layer) - based on libvirt, opens a whole new possibilities of using different hypervisors. Currently Qubes OS uses Xen.
* Windows Tools: `qvm-block` does not work
- Xen 4.4 - many new features, but for us the most important is much more mature libxl toolstack.
* UEFI is not supported, you need to enable "legacy boot" in BIOS before installing Qubes OS
- Qrexec 3 - greatly improved performance by using direct VM-VM connections and bigger buffers.
* Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead.
- Debian templates gets official support.
- Whonix templates
- Build system improvements - especially support for distribution-specific plugins (makes supporting multiple distributions much easier) and building templates using DispVM.
- Automated tests - makes much easier to find bugs, before its even shipped to users
Known issues
^^^^^^^^^^^^
- Windows Tools: ``qvm-block`` does not work
- UEFI is not supported, you need to enable “legacy boot” in BIOS before installing Qubes OS
- Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead.
- If your GPU is not correctly supported by the Dom0 kernel (e.g. the 3D desktop effects do not run smoothly) then you might experience “heaviness” with Windows 7-based AppVMs. In that case, please solve the problem with your GPU support in Dom0 in the first place (by using a different kernel), or install Qubes OS on a different system.
- For other known issues take a look at `our tickets <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.0%22+label%3Abug>`__
* If your GPU is not correctly supported by the Dom0 kernel (e.g. the 3D desktop effects do not run smoothly) then you might experience "heaviness" with Windows 7-based AppVMs. In that case, please solve the problem with your GPU support in Dom0 in the first place (by using a different kernel), or install Qubes OS on a different system.
* For other known issues take a look at [our tickets](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.0%22+label%3Abug)
It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems.
## Downloads
Downloads
^^^^^^^^^
See [Qubes Downloads](/doc/QubesDownloads/).
## Installation instructions
See `Qubes Downloads <https://www.qubes-os.org/downloads/>`__.
See [Installation Guide](/doc/installation-guide/).
Installation instructions
^^^^^^^^^^^^^^^^^^^^^^^^^
## Upgrading
### From R3.0 release candidate
See :doc:`Installation Guide </user/downloading-installing-upgrading/installation-guide>`.
Upgrading
^^^^^^^^^
From R3.0 release candidate
---------------------------
If you are using Qubes R3.0rc1, R3.0rc2 or R3.0rc3, just install system updates, there is no special steps required.
### From R2.0 or earlier
From R2.0 or earlier
--------------------
The easiest and safest way to upgrade to Qubes R3.0 is to install it from scratch and use [qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs.
Users of Qubes R2 can upgrade using [experimental procedure](/doc/upgrade-to-r3.0/).
The easiest and safest way to upgrade to Qubes R3.0 is to install it from scratch and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs.
Users of Qubes R2 can upgrade using :doc:`experimental procedure </user/downloading-installing-upgrading/upgrade/3_0>`.

33
developer/releases/3_0/schedule.rst
View file

@ -1,15 +1,20 @@
---
lang: en
layout: doc
permalink: /doc/releases/3.0/schedule/
redirect_from:
- /en/doc/releases/3.0/schedule/
ref: 20
title: Qubes R3.0 release schedule
---
===========================
Qubes R3.0 release schedule
===========================
.. list-table::
:widths: 11 11
:align: center
:header-rows: 1
* - Date
- Stage
* - 5 Sep 2015
- current-testing freeze before 3.0-rc3
* - 15 Sep 2015
- 3.0-rc3 release
* - 1 Oct 2015
- 3.0 release
| Date | Stage |
| -----------:| ------------------------------------- |
| 5 Sep 2015 | current-testing freeze before 3.0-rc3 |
| 15 Sep 2015 | 3.0-rc3 release |
| 1 Oct 2015 | 3.0 release |

103
developer/releases/3_1/release-notes.rst
View file

@ -1,65 +1,82 @@
---
lang: en
layout: doc
permalink: /doc/releases/3.1/release-notes/
ref: 16
title: Qubes R3.1 release notes
---
========================
Qubes R3.1 release notes
========================
## New features since 3.0
* Management Stack based of Salt Stack in dom0 - [documentation](/doc/salt/)
* Out of the box Whonix setup
* UEFI support
* LIVE edition (still alpha, not part of R3.1-rc1)
* Updated GPU drivers in dom0
* Colorful window application icons (instead of just colorful lock icon)
* PV Grub support ([documentation](/doc/managing-vm-kernels/))
* Out of the box USB VM setup, including [handling USB mouse](https://github.com/QubesOS/qubes-app-linux-input-proxy/blob/master/README.md)
* Xen upgraded to 4.6, for better hardware support (especially Skylake platform)
* Improve updates proxy flexibility - especially repositories served over HTTPS
New features since 3.0
----------------------
You can get detailed description in [completed github issues](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+3.1%22+label%3Arelease-notes+is%3Aclosed)
## Known issues
- Management Stack based of Salt Stack in dom0 - :doc:`documentation </user/advanced-topics/salt>`
* Installation image does not fit on DVD, requires either DVD DL, or USB stick (5GB or more)
- Out of the box Whonix setup
* Windows Tools: `qvm-block` does not work
- UEFI support
* Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead.
- LIVE edition (still alpha, not part of R3.1-rc1)
- Updated GPU drivers in dom0
- Colorful window application icons (instead of just colorful lock icon)
- PV Grub support (:doc:`documentation </user/advanced-topics/managing-vm-kernels>`)
- Out of the box USB VM setup, including `handling USB mouse <https://github.com/QubesOS/qubes-app-linux-input-proxy/blob/master/README.md>`__
- Xen upgraded to 4.6, for better hardware support (especially Skylake platform)
- Improve updates proxy flexibility - especially repositories served over HTTPS
You can get detailed description in `completed github issues <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+3.1%22+label%3Arelease-notes+is%3Aclosed>`__
Known issues
------------
- Installation image does not fit on DVD, requires either DVD DL, or USB stick (5GB or more)
- Windows Tools: ``qvm-block`` does not work
- Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead.
- USB mouse (in the case of USB VM) does not work at first system startup (just after completing firstboot). Workaround: restart the system.
- For other known issues take a look at `our tickets <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.1%22+label%3Abug>`__
* USB mouse (in the case of USB VM) does not work at first system startup (just after completing firstboot). Workaround: restart the system.
* For other known issues take a look at [our tickets](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.1%22+label%3Abug)
It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems.
## Downloads
Downloads
---------
See [Qubes Downloads](/downloads/).
## Installation instructions
See `Qubes Downloads <https://www.qubes-os.org/downloads/>`__.
See [Installation Guide](/doc/installation-guide/).
Installation instructions
-------------------------
## Upgrading
### From R3.0
See :doc:`Installation Guide </user/downloading-installing-upgrading/installation-guide>`.
The easiest and safest way to upgrade to Qubes R3.1 is to install it from
scratch and use [qubes backup and restore tools](/doc/backup-restore/) for
migrating of all of the user VMs.
Upgrading
---------
Users of Qubes R3.0 can upgrade using [experimental
procedure](/doc/upgrade-to-r3.1/).
### From R2 or earlier
From R3.0
^^^^^^^^^
When upgrading from earlier versions the easiest and safest way is to install
it from scratch and use [qubes backup and restore tools](/doc/backup-restore/)
for migrating of all of the user VMs.
Alternatively you can [upgrade to R3.0
using](/doc/releases/3.0/release-notes/#upgrading) first, then follow the
instructions above. This will be time consuming process.
The easiest and safest way to upgrade to Qubes R3.1 is to install it from scratch and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs.
Users of Qubes R3.0 can upgrade using :doc:`experimental procedure </user/downloading-installing-upgrading/upgrade/3_1>`.
From R2 or earlier
^^^^^^^^^^^^^^^^^^
When upgrading from earlier versions the easiest and safest way is to install it from scratch and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs.
Alternatively you can :ref:`upgrade to R3.0 using <developer/releases/3_0/release-notes:upgrading>` first, then follow the instructions above. This will be time consuming process.

44
developer/releases/3_1/schedule.rst
View file

@ -1,20 +1,28 @@
---
lang: en
layout: doc
permalink: /doc/releases/3.1/schedule/
redirect_from:
- /en/doc/releases/3.1/schedule/
ref: 17
title: Qubes R3.1 release schedule
---
===========================
Qubes R3.1 release schedule
===========================
This schedule is based on [Version Scheme](/doc/version-scheme/#release-schedule).
| Date | Stage |
| -----------:| --------------------------------------- |
| 8 Dec 2015 | 3.1-rc1 release |
| 5 Jan 2016 | current-testing freeze before 3.1-rc2 |
| 12 Jan 2016 | 3.1-rc2 release |
| 26 Jan 2016 | decide whether 3.1-rc2 is the final 3.1 |
| 9 Feb 2016 | current-testing freeze before 3.1-rc3 |
| ~~16 Feb 2016~~ <br/> 23 Feb 2016 | 3.1-rc3 release |
This schedule is based on :ref:`Version Scheme <developer/releases/version-scheme:release schedule>`.
.. list-table::
:widths: 38 38
:align: center
:header-rows: 1
* - Date
- Stage
* - 8 Dec 2015
- 3.1-rc1 release
* - 5 Jan 2016
- current-testing freeze before 3.1-rc2
* - 12 Jan 2016
- 3.1-rc2 release
* - 26 Jan 2016
- decide whether 3.1-rc2 is the final 3.1
* - 9 Feb 2016
- current-testing freeze before 3.1-rc3
* - :strike:`16 Feb 2016` 23 Feb 2016
- 3.1-rc3 release

95
developer/releases/3_2/release-notes.rst
View file

@ -1,61 +1,76 @@
---
lang: en
layout: doc
permalink: /doc/releases/3.2/release-notes/
ref: 21
title: Qubes R3.2 release notes
---
========================
Qubes R3.2 release notes
========================
## New features since 3.1
* Management Stack extended to support in-VM configuration - [documentation](/doc/salt/)
* PV USB - [documentation](/doc/usb/)
* Dom0 update to Fedora 23 for better hardware support
* Kernel 4.4.x
* Default desktop environment switched to Xfce4
* KDE 5 support (but it is no longer the default one)
* Tiling window managers support: awesome, [i3](/doc/i3/)
* More flexible Qubes RPC services - [related ticket](https://github.com/QubesOS/qubes-issues/issues/1876), [documentation](/doc/qrexec/#service-policies-with-arguments)
New features since 3.1
----------------------
You can get detailed description in [completed github issues](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+3.2%22+label%3Arelease-notes+is%3Aclosed)
## Known issues
- Management Stack extended to support in-VM configuration - :doc:`documentation </user/advanced-topics/salt>`
* [Fedora 23 reached EOL in December 2016](https://fedoraproject.org/wiki/End_of_life). There is a [manual procedure to upgrade your VMs](/news/2018/01/06/fedora-26-upgrade/).
- PV USB - :doc:`documentation </user/how-to-guides/how-to-use-usb-devices>`
* Windows Tools: `qvm-block` does not work
- Dom0 update to Fedora 23 for better hardware support
- Kernel 4.4.x
- Default desktop environment switched to Xfce4
- KDE 5 support (but it is no longer the default one)
- Tiling window managers support: awesome, :doc:`i3 </user/advanced-topics/i3>`
- More flexible Qubes RPC services - `related ticket <https://github.com/QubesOS/qubes-issues/issues/1876>`__, :ref:`documentation <developer/services/qrexec:service policies with arguments>`
You can get detailed description in `completed github issues <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+3.2%22+label%3Arelease-notes+is%3Aclosed>`__
Known issues
------------
- `Fedora 23 reached EOL in December 2016 <https://fedoraproject.org/wiki/End_of_life>`__. There is a `manual procedure to upgrade your VMs <https://www.qubes-os.org/news/2018/01/06/fedora-26-upgrade/>`__.
- Windows Tools: ``qvm-block`` does not work
- Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead.
- For other known issues take a look at `our tickets <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.2%22+label%3Abug>`__
* Some icons in the Qubes Manager application might not be drawn correctly when using the Xfce4 environment in Dom0. If this bothers you, please use the KDE environment instead.
* For other known issues take a look at [our tickets](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+3.2%22+label%3Abug)
It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems.
## Downloads
Downloads
---------
See [Qubes Downloads](/downloads/).
## Installation instructions
See `Qubes Downloads <https://www.qubes-os.org/downloads/>`__.
See [Installation Guide](/doc/installation-guide/).
After installation, [manually upgrade to Fedora 26](/news/2018/01/06/fedora-26-upgrade/).
Installation instructions
-------------------------
## Upgrading
### From R3.1
See :doc:`Installation Guide </user/downloading-installing-upgrading/installation-guide>`. After installation, `manually upgrade to Fedora 26 <https://www.qubes-os.org/news/2018/01/06/fedora-26-upgrade/>`__.
The easiest and safest way to upgrade to Qubes R3.2 is to install it from
scratch and use [qubes backup and restore tools](/doc/backup-restore/) for
migrating of all of the user VMs.
Upgrading
---------
Users of Qubes R3.1 can also upgrade using [this
procedure](/doc/upgrade-to-r3.2/).
### From R3.0 or earlier
From R3.1
^^^^^^^^^
When upgrading from earlier versions the easiest and safest way is to install
it from scratch and use [qubes backup and restore tools](/doc/backup-restore/)
for migrating of all of the user VMs.
Alternatively you can [upgrade to R3.1 using](/doc/releases/3.1/release-notes/#upgrading) first, then follow
the instructions above. This will be time consuming process.
The easiest and safest way to upgrade to Qubes R3.2 is to install it from scratch and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs.
Users of Qubes R3.1 can also upgrade using :doc:`this procedure </user/downloading-installing-upgrading/upgrade/3_2>`.
From R3.0 or earlier
^^^^^^^^^^^^^^^^^^^^
When upgrading from earlier versions the easiest and safest way is to install it from scratch and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs.
Alternatively you can :ref:`upgrade to R3.1 using <developer/releases/3_1/release-notes:upgrading>` first, then follow the instructions above. This will be time consuming process.

50
developer/releases/3_2/schedule.rst
View file

@ -1,22 +1,32 @@
---
lang: en
layout: doc
permalink: /doc/releases/3.2/schedule/
redirect_from:
- /en/doc/releases/3.2/schedule/
ref: 22
title: Qubes R3.2 release schedule
---
===========================
Qubes R3.2 release schedule
===========================
This schedule is based on [Version Scheme](/doc/version-scheme/#release-schedule).
| Date | Stage |
| -----------:| --------------------------------------- |
| 18 Jun 2016 | 3.2-rc1 release |
| 2 Jul 2016 | decide whether 3.2-rc1 is the final 3.2 |
| ~~16 Jul 2016~~ <br/> 20 Jul 2016 | current-testing freeze before 3.2-rc2 |
| ~~23 Jul 2016~~ <br/> 27 Jul 2016 | 3.2-rc2 release |
| ~~5 Aug 2016~~ <br/> 9 Aug 2016 | decide whether 3.2-rc2 is the final 3.2 |
| 24 Aug 2016 | current-testing freeze before 3.2-rc3 |
| 31 Aug 2016 | 3.2-rc3 release |
| 29 Sep 2016 | 3.2 release |
This schedule is based on :ref:`Version Scheme <developer/releases/version-scheme:release schedule>`.
.. list-table::
:widths: 38 38
:align: center
:header-rows: 1
* - Date
- Stage
* - 18 Jun 2016
- 3.2-rc1 release
* - 2 Jul 2016
- decide whether 3.2-rc1 is the final 3.2
* - :strike:`16 Jul 2016` 20 Jul 2016
- current-testing freeze before 3.2-rc2
* - :strike:`23 Jul 2016` 27 Jul 2016
- 3.2-rc2 release
* - :strike:`5 Aug 2016` 9 Aug 2016
- decide whether 3.2-rc2 is the final 3.2
* - 24 Aug 2016
- current-testing freeze before 3.2-rc3
* - 31 Aug 2016
- 3.2-rc3 release
* - 29 Sep 2016
- 3.2 release

154
developer/releases/4_0/release-notes.rst
View file

@ -1,107 +1,145 @@
---
lang: en
layout: doc
permalink: /doc/releases/4.0/release-notes/
ref: 23
title: Qubes R4.0 release notes
---
========================
Qubes R4.0 release notes
========================
New features since 3.2
----------------------
* Core management scripts rewrite with better structure and extensibility, [current API documentation](https://dev.qubes-os.org/projects/core-admin/en/latest/) and the documentation API index as a [webarchive](https://web.archive.org/web/20230128102821/https://dev.qubes-os.org/projects/qubes-core-admin/en/latest/)
* [Admin API](/news/2017/06/27/qubes-admin-api/) allowing strictly controlled managing from non-dom0
* All `qvm-*` command-line tools rewritten, some options have changed
* Renaming VM directly is prohibited, there is GUI to clone under new name and remove old VM
* Use [PVH](https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt) and [HVM](https://github.com/QubesOS/qubes-issues/issues/2185) by default to [mitigate Meltdown & Spectre](https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt) and lower the [attack surface on Xen](https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-024-2016.txt)
* Create USB VM by default
* [Multiple DisposableVMs templates support](https://github.com/QubesOS/qubes-issues/issues/2253)
* New [backup format](/doc/backup-emergency-restore-v4/) using scrypt key-derivation function
* Non-encrypted backups no longer supported
* [split VM packages](https://github.com/QubesOS/qubes-issues/issues/2771), for better support minimal, specialized templates
* [Qubes Manager decomposition](https://github.com/QubesOS/qubes-issues/issues/2132) - domains and devices widgets instead of full Qubes Manager; devices widget support also USB
* [More flexible firewall interface](/doc/vm-interface/) for ease unikernel integration
* Template VMs do not have network interface by default, [qrexec-based updates proxy](https://github.com/QubesOS/qubes-issues/issues/1854) is used instead
* More flexible IP addressing for VMs - [custom IP](https://github.com/QubesOS/qubes-issues/issues/1477), [hidden from the IP](https://github.com/QubesOS/qubes-issues/issues/1143)
* More flexible Qubes RPC policy - [related ticket](https://github.com/QubesOS/qubes-issues/issues/865), [documentation](/doc/qrexec/#specifying-vms-tags-types-targets-etc)
* [New Qubes RPC confirmation window](https://github.com/QubesOS/qubes-issues/issues/910), including option to specify destination VM
* [New storage subsystem design](https://github.com/QubesOS/qubes-issues/issues/1842)
* Dom0 update to Fedora 25 for better hardware support
* Kernel 4.9.x
You can get detailed description in [completed github issues](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+4.0%22+label%3Arelease-notes+is%3Aclosed)
- Core management scripts rewrite with better structure and extensibility, `current API documentation <https://dev.qubes-os.org/projects/core-admin/en/latest/>`__ and the documentation API index as a `webarchive <https://web.archive.org/web/20230128102821/https://dev.qubes-os.org/projects/qubes-core-admin/en/latest/>`__
- `Admin API <https://www.qubes-os.org/news/2017/06/27/qubes-admin-api/>`__ allowing strictly controlled managing from non-dom0
- All ``qvm-*`` command-line tools rewritten, some options have changed
- Renaming VM directly is prohibited, there is GUI to clone under new name and remove old VM
- Use `PVH <https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt>`__ and `HVM <https://github.com/QubesOS/qubes-issues/issues/2185>`__ by default to `mitigate Meltdown & Spectre <https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt>`__ and lower the `attack surface on Xen <https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-024-2016.txt>`__
- Create USB VM by default
- `Multiple DisposableVMs templates support <https://github.com/QubesOS/qubes-issues/issues/2253>`__
- New :doc:`backup format </user/how-to-guides/backup-emergency-restore-v4>` using scrypt key-derivation function
- Non-encrypted backups no longer supported
- `split VM packages <https://github.com/QubesOS/qubes-issues/issues/2771>`__, for better support minimal, specialized templates
- `Qubes Manager decomposition <https://github.com/QubesOS/qubes-issues/issues/2132>`__ - domains and devices widgets instead of full Qubes Manager; devices widget support also USB
- :doc:`More flexible firewall interface </developer/debugging/vm-interface>` for ease unikernel integration
- Template VMs do not have network interface by default, `qrexec-based updates proxy <https://github.com/QubesOS/qubes-issues/issues/1854>`__ is used instead
- More flexible IP addressing for VMs - `custom IP <https://github.com/QubesOS/qubes-issues/issues/1477>`__, `hidden from the IP <https://github.com/QubesOS/qubes-issues/issues/1143>`__
- More flexible Qubes RPC policy - `related ticket <https://github.com/QubesOS/qubes-issues/issues/865>`__, :ref:`documentation <developer/services/qrexec:specifying vms: tags, types, targets, etc.>`
- `New Qubes RPC confirmation window <https://github.com/QubesOS/qubes-issues/issues/910>`__, including option to specify destination VM
- `New storage subsystem design <https://github.com/QubesOS/qubes-issues/issues/1842>`__
- Dom0 update to Fedora 25 for better hardware support
- Kernel 4.9.x
You can get detailed description in `completed github issues <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+4.0%22+label%3Arelease-notes+is%3Aclosed>`__
Security Notes
--------------
* PV VMs migrated from 3.2 to 4.0-rc4 or later are automatically set to PVH mode in order to protect against Meltdown (see [QSB #37](https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt)).
However, PV VMs migrated from any earlier 4.0 release candidate (RC1, RC2, or RC3) are not automatically set to PVH mode.
These must be set manually.
* The following steps may need to be applied in dom0 and Fedora 26 TemplateVMs in order to receive updates (see [#3737](https://github.com/QubesOS/qubes-issues/issues/3737)).
- PV VMs migrated from 3.2 to 4.0-rc4 or later are automatically set to PVH mode in order to protect against Meltdown (see `QSB #37 <https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-037-2018.txt>`__). However, PV VMs migrated from any earlier 4.0 release candidate (RC1, RC2, or RC3) are not automatically set to PVH mode. These must be set manually.
- The following steps may need to be applied in dom0 and Fedora 26 TemplateVMs in order to receive updates (see `#3737 <https://github.com/QubesOS/qubes-issues/issues/3737>`__).
Steps for dom0 updates:
1. Open the Qubes Menu by clicking on the "Q" icon in the top-left corner of the screen.
2. Select `Terminal Emulator`.
1. Open the Qubes Menu by clicking on the “Q” icon in the top-left corner of the screen.
2. Select ``Terminal Emulator``.
3. In the window that opens, enter this command:
```
sudo nano /etc/yum.repos.d/qubes-dom0.repo
```
.. code:: bash
sudo nano /etc/yum.repos.d/qubes-dom0.repo
4. This opens the nano text editor. Change all four instances of ``http`` to ``https``.
5. Press ``CTRL+X``, then ``Y``, then ``ENTER`` to save changes and exit.
4. This opens the nano text editor. Change all four instances of `http` to `https`.
5. Press `CTRL+X`, then `Y`, then `ENTER` to save changes and exit.
6. Check for updates normally.
Steps for Fedora 26 TemplateVM updates:
1. Open the Qubes Menu by clicking on the "Q" icon in the top-left corner of the screen.
2. Select `Template: fedora-26`, then `fedora-26: Terminal`.
1. Open the Qubes Menu by clicking on the “Q” icon in the top-left corner of the screen.
2. Select ``Template: fedora-26``, then ``fedora-26: Terminal``.
3. In the window that opens, enter the command for your version:
```
[Qubes 3.2] sudo gedit /etc/yum.repos.d/qubes-r3.repo
[Qubes 4.0] sudo gedit /etc/yum.repos.d/qubes-r4.repo
```
.. code:: bash
[Qubes 3.2] sudo gedit /etc/yum.repos.d/qubes-r3.repo
[Qubes 4.0] sudo gedit /etc/yum.repos.d/qubes-r4.repo
4. This opens the gedit text editor in a window. Change all four instances of ``http`` to ``https``.
5. Click the “Save” button in the top-right corner of the window.
4. This opens the gedit text editor in a window. Change all four instances of `http` to `https`.
5. Click the "Save" button in the top-right corner of the window.
6. Close the window.
7. Check for updates normally.
8. Shut down the TemplateVM.
Known issues
------------
* Locale using coma as decimal separator [crashes qubesd](https://github.com/QubesOS/qubes-issues/issues/3753). Either install with different locale (English (United States) for example), or manually apply fix explained in that issue.
* In the middle of installation, [keyboard layout reset to US](https://github.com/QubesOS/qubes-issues/issues/3352). Be careful what is the current layout while setting default user password (see upper right screen corner).
- Locale using coma as decimal separator `crashes qubesd <https://github.com/QubesOS/qubes-issues/issues/3753>`__. Either install with different locale (English (United States) for example), or manually apply fix explained in that issue.
* On some laptops (for example Librem 15v2), touchpad do not work directly after installation. Reboot the system to fix the issue.
- In the middle of installation, `keyboard layout reset to US <https://github.com/QubesOS/qubes-issues/issues/3352>`__. Be careful what is the current layout while setting default user password (see upper right screen corner).
* List of USB devices may contain device identifiers instead of name
- On some laptops (for example Librem 15v2), touchpad do not work directly after installation. Reboot the system to fix the issue.
- List of USB devices may contain device identifiers instead of name
- With R4.0.1, which ships kernel-4.19, you may never reach the anaconda startup and be block on an idle black screen with blinking cursor. You can try to add ``plymouth.ignore-serial-consoles`` in the grub installer boot menu right after ``quiet rhgb``. With legacy mode, you can do it directly when booting the DVD or USB key. In UEFI mode, follow the same procedure described for :ref:`disabling <user/troubleshooting/uefi-troubleshooting:installation freezes before displaying installer>` ``nouveau`` module (related `solved issue <https://github.com/QubesOS/qubes-issues/issues/3849>`__ in further version of Qubes).
- For other known issues take a look at `our tickets <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+4.0%22+label%3Abug>`__
* With R4.0.1, which ships kernel-4.19, you may never reach the anaconda startup and be block on an idle black screen with blinking cursor. You can try to add `plymouth.ignore-serial-consoles` in the grub installer boot menu right after `quiet rhgb`. With legacy mode, you can do it directly when booting the DVD or USB key. In UEFI mode, follow the same procedure described for [disabling](/doc/uefi-troubleshooting/#installation-freezes-before-displaying-installer) `nouveau` module (related [solved issue](https://github.com/QubesOS/qubes-issues/issues/3849) in further version of Qubes).
* For other known issues take a look at [our tickets](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+4.0%22+label%3Abug)
It is advised to install updates just after system installation to apply bug fixes for (some of) the above problems.
Downloads
---------
See [Qubes Downloads](/downloads/).
See `Qubes Downloads <https://www.qubes-os.org/downloads/>`__.
Installation instructions
-------------------------
See [Installation Guide](/doc/installation-guide/).
See :doc:`Installation Guide </user/downloading-installing-upgrading/installation-guide>`.
Upgrading
---------
There is no in-place upgrade path from earlier Qubes versions. The only
supported option to upgrade to Qubes R4.0 is to install it from scratch and use
[qubes backup and restore tools](/doc/backup-restore/) for migrating of all of the user VMs.
We also provide [detailed instruction](/doc/upgrade-to-r4.0/) for this procedure.
There is no in-place upgrade path from earlier Qubes versions. The only supported option to upgrade to Qubes R4.0 is to install it from scratch and use :doc:`qubes backup and restore tools </user/how-to-guides/how-to-back-up-restore-and-migrate>` for migrating of all of the user VMs. We also provide :doc:`detailed instruction </user/downloading-installing-upgrading/upgrade/4_0>` for this procedure.

68
developer/releases/4_0/schedule.rst
View file

@ -1,28 +1,44 @@
---
lang: en
layout: doc
permalink: /doc/releases/4.0/schedule/
redirect_from:
- /en/doc/releases/4.0/schedule/
ref: 24
title: Qubes R4.0 release schedule
---
===========================
Qubes R4.0 release schedule
===========================
This schedule is based on [Version Scheme](/doc/version-scheme/#release-schedule).
| Date | Stage |
| -----------:| --------------------------------------- |
| 31 Jul 2017 | 4.0-rc1 release |
| ~~28 Aug 2017~~ <br/>~~11 Sep 2017~~ <br/>~~9 Oct 2017~~ <br/>18 Oct 2017 | current-testing freeze before 4.0-rc2 |
| ~~4 Sep 2017~~ <br/> ~~18 Sep 2017~~ <br/>~~16 Oct 2017~~ <br/>23 Oct 2017 | 4.0-rc2 release |
| 6 Nov 2017 | decide whether 4.0-rc2 is the final 4
| 20 Nov 2017 | current-testing freeze before 4.0-rc3 |
| 27 Nov 2017 | 4.0-rc3 release |
| 11 Dec 2017 | decide whether 4.0-rc3 is the final 4.0 |
| 1 Jan 2018 | current-testing freeze before 4.0-rc4 |
| ~~8 Jan 2018~~ <br/>31 Jan 2018 | 4.0-rc4 release |
| ~~22 Jan 2018~~ <br/>14 Feb 2018 | decide whether 4.0-rc4 is the final 4.0 |
| 27 Feb 2018 | current-testing freeze before 4.0-rc5 |
| 6 Mar 2018 | 4.0-rc5 release |
| 20 Mar 2018 | decide whether 4.0-rc5 is the final 4.0 |
| 28 Mar 2018 | final 4.0 release |
This schedule is based on :ref:`Version Scheme <developer/releases/version-scheme:release schedule>`.
.. list-table::
:widths: 88 88
:align: center
:header-rows: 1
* - Date
- Stage
* - 31 Jul 2017
- 4.0-rc1 release
* - :strike:`28 Aug 2017` :strike:`11 Sep 2017` :strike:`9 Oct 2017` 18 Oct 2017
- current-testing freeze before 4.0-rc2
* - :strike:`4 Sep 2017` :strike:`18 Sep 2017` :strike:`16 Oct 2017` 23 Oct 2017
- 4.0-rc2 release
* - 6 Nov 2017
- decide whether 4.0-rc2 is the final 4
* - 20 Nov 2017
- current-testing freeze before 4.0-rc3
* - 27 Nov 2017
- 4.0-rc3 release
* - 11 Dec 2017
- decide whether 4.0-rc3 is the final 4.0
* - 1 Jan 2018
- current-testing freeze before 4.0-rc4
* - :strike:`8 Jan 2018` 31 Jan 2018
- 4.0-rc4 release
* - :strike:`22 Jan 2018` 14 Feb 2018
- decide whether 4.0-rc4 is the final 4.0
* - 27 Feb 2018
- current-testing freeze before 4.0-rc5
* - 6 Mar 2018
- 4.0-rc5 release
* - 20 Mar 2018
- decide whether 4.0-rc5 is the final 4.0
* - 28 Mar 2018
- final 4.0 release

147
developer/releases/4_1/release-notes.rst
View file

@ -1,93 +1,116 @@
---
layout: doc
title: Qubes OS 4.1 release notes
permalink: /doc/releases/4.1/release-notes/
---
==========================
Qubes OS 4.1 release notes
==========================
## New features and improvements since Qubes 4.0
- Optional qubes-remote-support package now available from repositories
(strictly opt-in, no package installed by default; no new ports or network
connections open by default; requires explicit connection initiation by the
user, then requires sharing a code word with the remote party before a
connection can be established; see
[#6364](https://github.com/QubesOS/qubes-issues/issues/6364) for more
information)
- Qubes firewall reworked to be more defensive (see
[#5540](https://github.com/QubesOS/qubes-issues/issues/5540) for details)
New features and improvements since Qubes 4.0
---------------------------------------------
- Optional qubes-remote-support package now available from repositories (strictly opt-in, no package installed by default; no new ports or network connections open by default; requires explicit connection initiation by the user, then requires sharing a code word with the remote party before a connection can be established; see `#6364 <https://github.com/QubesOS/qubes-issues/issues/6364>`__ for more information)
- Qubes firewall reworked to be more defensive (see `#5540 <https://github.com/QubesOS/qubes-issues/issues/5540>`__ for details)
- Xen upgraded to version 4.14
- Dom0 operating system upgraded to Fedora 32
- Default desktop environment upgraded to Xfce 4.14
- Upgraded default template releases
- Experimental support for GUI running outside of dom0 (hybrid mode GUI domain
without real GPU passthrough; see
[#5662](https://github.com/QubesOS/qubes-issues/issues/5662) for details)
- Experimental support for audio server running outside of dom0 ("Audio domain")
- Experimental support for GUI running outside of dom0 (hybrid mode GUI domain without real GPU passthrough; see `#5662 <https://github.com/QubesOS/qubes-issues/issues/5662>`__ for details)
- Experimental support for audio server running outside of dom0 (“Audio domain”)
- sys-firewall and sys-usb are now disposables by default
- UEFI boot now loads GRUB, which in turn loads Xen, making the boot path
similar to legacy boot and allowing the user to modify boot parameters or
choose an alternate boot menu entry
- New qrexec policy format (see
[#4370](https://github.com/QubesOS/qubes-issues/issues/4370) for details)
- qrexec protocol improvements (see
[#4909](https://github.com/QubesOS/qubes-issues/issues/4909) for details)
- UEFI boot now loads GRUB, which in turn loads Xen, making the boot path similar to legacy boot and allowing the user to modify boot parameters or choose an alternate boot menu entry
- New qrexec policy format (see `#4370 <https://github.com/QubesOS/qubes-issues/issues/4370>`__ for details)
- qrexec protocol improvements (see `#4909 <https://github.com/QubesOS/qubes-issues/issues/4909>`__ for details)
- New qrexec-policy daemon
- Simplified using in-qube kernels
- Windows USB and audio support courtesy of
[tabit-pro](https://github.com/tabit-pro) (see
[#5802](https://github.com/QubesOS/qubes-issues/issues/5802) and
[#2624](https://github.com/QubesOS/qubes-issues/issues/2624))
- Windows USB and audio support courtesy of `tabit-pro <https://github.com/tabit-pro>`__ (see `#5802 <https://github.com/QubesOS/qubes-issues/issues/5802>`__ and `#2624 <https://github.com/QubesOS/qubes-issues/issues/2624>`__)
- Clarified disposable-related terminology and properties
- Default kernelopts can now be specified by a kernel package
- Improved support for high-resolution displays
- Improved notifications when a system drive runs out of free space
- Support for different cursor shapes
- "Paranoid mode" backup restore option now properly supported using
disposables
- “Paranoid mode” backup restore option now properly supported using disposables
- Users can now choose between Debian and Fedora in the installer
- Certain files and applications are now opened in disposables, e.g.,
Thunderbird email attachments
- Certain files and applications are now opened in disposables, e.g., Thunderbird email attachments
- New graphical interface for managing testing repository updates
- New "Cute Qube" icon family (replaces padlock icons)
- New “Cute Qube” icon family (replaces padlock icons)
- Disposable qube types now use the disposable icon
- New Template Manager tool for installing, removing, and updating templates
(meanwhile, the tool previously known as the "Template Manager," which was
for mass template switching, has been integrated into the Qube Manager)
- The "file" storage driver has been deprecated in Qubes 4.1 and will be
removed in Qubes 4.2
- `property-del` event renamed to `property-reset` to avoid confusion
- qrexec no longer supports non-executable files in `/etc/qubes-rpc`
- New Template Manager tool for installing, removing, and updating templates (meanwhile, the tool previously known as the “Template Manager,” which was for mass template switching, has been integrated into the Qube Manager)
- The “file” storage driver has been deprecated in Qubes 4.1 and will be removed in Qubes 4.2
- ``property-del`` event renamed to ``property-reset`` to avoid confusion
- qrexec no longer supports non-executable files in ``/etc/qubes-rpc``
- qrexec components have been reorganized into the core-qrexec repository
- The `qvm-pool` argument parser has been rewritten and improved
- The ``qvm-pool`` argument parser has been rewritten and improved
- Removed the need for the out-of-tree u2mfn kernel module
- Qrexec services can now run as a socket server
- Improved template distribution mechanism
- Now possible to restart qrexec-agent
- The term "VM" has largely been replaced by "qube"
- GUI daemon is now configured using `qvm-features` tool,
`/etc/qubes/guid.conf` file is no longer used
- `qvm-run` tool got `--no-shell` option to run a single command without using
a shell inside the qube
- MAC Randomization for iwlwifi (see [#938](https://github.com/QubesOS/qubes-issues/issues/938))
For a full list, including more detailed descriptions, please see
[here](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+4.1%22+label%3A%22release+notes%22+is%3Aclosed).
- The term “VM” has largely been replaced by “qube”
## Known issues
- GUI daemon is now configured using ``qvm-features`` tool, ``/etc/qubes/guid.conf`` file is no longer used
For a full list of known 4.1 issues with open bug reports, please see
[here](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+4.1%22+label%3A%22T%3A+bug%22).
We strongly recommend [updating Qubes OS](/doc/how-to-update/) immediately
after installation in order to apply any and all available bug fixes.
- ``qvm-run`` tool got ``--no-shell`` option to run a single command without using a shell inside the qube
## Download
- MAC Randomization for iwlwifi (see `#938 <https://github.com/QubesOS/qubes-issues/issues/938>`__)
See [downloads](/downloads/).
## Installation instructions
See the [installation guide](/doc/installation-guide/).
For a full list, including more detailed descriptions, please see `here <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+4.1%22+label%3A%22release+notes%22+is%3Aclosed>`__.
## Upgrading
Known issues
------------
Please see [how to upgrade to Qubes 4.1](/doc/upgrade/4.1/).
For a full list of known 4.1 issues with open bug reports, please see `here <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A%22Release+4.1%22+label%3A%22T%3A+bug%22>`__. We strongly recommend :doc:`updating Qubes OS </user/how-to-guides/how-to-update>` immediately after installation in order to apply any and all available bug fixes.
Download
--------
See `downloads <https://www.qubes-os.org/downloads/>`__.
Installation instructions
-------------------------
See the :doc:`installation guide </user/downloading-installing-upgrading/installation-guide>`.
Upgrading
---------
Please see :doc:`how to upgrade to Qubes 4.1 </user/downloading-installing-upgrading/upgrade/4_1>`.

59
developer/releases/4_1/schedule.rst
View file

@ -1,25 +1,38 @@
---
lang: en
layout: doc
permalink: /doc/releases/4.1/schedule/
redirect_from:
- /en/doc/releases/4.1/schedule/
title: Qubes R4.1 release schedule
---
===========================
Qubes R4.1 release schedule
===========================
The table below is based on our [release schedule
policy](/doc/version-scheme/#release-schedule).
| Date | Stage |
| ----------:| ----------------------------------------- |
| 2021-10-11 | 4.1.0-rc1 release |
| 2021-11-08 | current-testing freeze before 4.1.0-rc2 |
| 2021-11-15 | 4.1.0-rc2 release |
| 2021-11-29 | decide whether 4.1.0-rc2 is the final 4.1 |
| 2021-12-13 | current-testing freeze before 4.1.0-rc3 |
| 2021-12-20 | 4.1.0-rc3 release |
| 2022-01-03 | decide whether 4.1.0-rc3 is the final 4.1 |
| 2022-01-11 | current-testing freeze before 4.1.0-rc4 |
| 2022-01-18 | 4.1.0-rc4 release |
| 2022-01-31 | decide whether 4.1.0-rc4 is the final 4.1 |
| 2022-02-04 | final 4.1.0 release |
The table below is based on our :ref:`release schedule policy <developer/releases/version-scheme:release schedule>`.
.. list-table::
:widths: 10 10
:align: center
:header-rows: 1
* - Date
- Stage
* - 2021-10-11
- 4.1.0-rc1 release
* - 2021-11-08
- current-testing freeze before 4.1.0-rc2
* - 2021-11-15
- 4.1.0-rc2 release
* - 2021-11-29
- decide whether 4.1.0-rc2 is the final 4.1
* - 2021-12-13
- current-testing freeze before 4.1.0-rc3
* - 2021-12-20
- 4.1.0-rc3 release
* - 2022-01-03
- decide whether 4.1.0-rc3 is the final 4.1
* - 2022-01-11
- current-testing freeze before 4.1.0-rc4
* - 2022-01-18
- 4.1.0-rc4 release
* - 2022-01-31
- decide whether 4.1.0-rc4 is the final 4.1
* - 2022-02-04
- final 4.1.0 release

152
developer/releases/4_2/release-notes.rst
View file

@ -1,84 +1,136 @@
---
layout: doc
title: Qubes OS 4.2 release notes
permalink: /doc/releases/4.2/release-notes/
---
==========================
Qubes OS 4.2 release notes
==========================
## New features and improvements since Qubes 4.1
- Dom0 upgraded to Fedora 37 ([#6982](https://github.com/QubesOS/qubes-issues/issues/6982))
New features and improvements since Qubes 4.1
---------------------------------------------
- Dom0 upgraded to Fedora 37 (`#6982 <https://github.com/QubesOS/qubes-issues/issues/6982>`__)
- Xen upgraded to version 4.17
- Default Debian template upgraded to Debian 12
- Default Fedora and Debian templates use Xfce instead of GNOME ([#7784](https://github.com/QubesOS/qubes-issues/issues/7784))
- SELinux support in Fedora templates ([#4239](https://github.com/QubesOS/qubes-issues/issues/4239))
- Default Fedora and Debian templates use Xfce instead of GNOME (`#7784 <https://github.com/QubesOS/qubes-issues/issues/7784>`__)
- SELinux support in Fedora templates (`#4239 <https://github.com/QubesOS/qubes-issues/issues/4239>`__)
- Several GUI applications rewritten (screenshots below), including:
- Applications Menu (also available as preview in R4.1) ([#6665](https://github.com/QubesOS/qubes-issues/issues/6665)), ([#5677](https://github.com/QubesOS/qubes-issues/issues/5677))
- Qubes Global Settings ([#6898](https://github.com/QubesOS/qubes-issues/issues/6898))
- Applications Menu (also available as preview in R4.1) (`#6665 <https://github.com/QubesOS/qubes-issues/issues/6665>`__), (`#5677 <https://github.com/QubesOS/qubes-issues/issues/5677>`__)
- Qubes Global Settings (`#6898 <https://github.com/QubesOS/qubes-issues/issues/6898>`__)
- Create New Qube
- Qubes Update ([#7443](https://github.com/QubesOS/qubes-issues/issues/7443))
- New `qubes-vm-update` tool ([#7443](https://github.com/QubesOS/qubes-issues/issues/7443))
- Unified `grub.cfg` location for both UEFI and legacy boot ([#7985](https://github.com/QubesOS/qubes-issues/issues/7985))
- PipeWire support ([#6358](https://github.com/QubesOS/qubes-issues/issues/6358))
- fwupd integration for firmware updates ([#4855](https://github.com/QubesOS/qubes-issues/issues/4855))
- Optional automatic clipboard clearing ([#3415](https://github.com/QubesOS/qubes-issues/issues/3415))
- Official packages built using Qubes Builder v2 ([#6486](https://github.com/QubesOS/qubes-issues/issues/6486))
- Qubes Update (`#7443 <https://github.com/QubesOS/qubes-issues/issues/7443>`__)
- New ``qubes-vm-update`` tool (`#7443 <https://github.com/QubesOS/qubes-issues/issues/7443>`__)
- Unified ``grub.cfg`` location for both UEFI and legacy boot (`#7985 <https://github.com/QubesOS/qubes-issues/issues/7985>`__)
- PipeWire support (`#6358 <https://github.com/QubesOS/qubes-issues/issues/6358>`__)
- fwupd integration for firmware updates (`#4855 <https://github.com/QubesOS/qubes-issues/issues/4855>`__)
- Optional automatic clipboard clearing (`#3415 <https://github.com/QubesOS/qubes-issues/issues/3415>`__)
- Official packages built using Qubes Builder v2 (`#6486 <https://github.com/QubesOS/qubes-issues/issues/6486>`__)
- Split GPG management in Qubes Global Settings
- Qrexec services use new qrexec policy format by default (but old format is still supported) ([#8000](https://github.com/QubesOS/qubes-issues/issues/8000))
- Qrexec services use new qrexec policy format by default (but old format is still supported) (`#8000 <https://github.com/QubesOS/qubes-issues/issues/8000>`__)
- Improved keyboard layout switching
For a full list, including more detailed descriptions, please see [here](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+4.2%22+label%3A%22release+notes%22+is%3Aclosed). Below are some screenshots of the new and improved Qubes GUI tools.
For a full list, including more detailed descriptions, please see `here <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+sort%3Aupdated-desc+milestone%3A%22Release+4.2%22+label%3A%22release+notes%22+is%3Aclosed>`__. Below are some screenshots of the new and improved Qubes GUI tools.
The new Qubes OS Update tool:
[![Screenshot of the Qubes OS Update tool](/attachment/site/4-2_update.png)](/attachment/site/4-2_update.png)
|Screenshot of the Qubes OS Update tool|
The new Qubes OS Global Config tool:
[![Screenshot of the Qubes OS Global Config tool](/attachment/site/4-2_global-config_1.png)](/attachment/site/4-2_global-config_1.png)
[![Screenshot of the Qubes OS Global Config tool](/attachment/site/4-2_global-config_2.png)](/attachment/site/4-2_global-config_2.png)
|Screenshot of the Qubes OS Global Config tool| |image1|
The new Qubes OS Policy Editor tool:
[![Screenshot of the Qubes OS Policy Editor tool](/attachment/site/4-2_policy-editor.png)](/attachment/site/4-2_policy-editor.png)
|Screenshot of the Qubes OS Policy Editor tool|
## Known issues
Known issues
------------
- DomU firewalls have completely switched to nftables. Users should add their custom rules to the `custom-input` and `custom-forward` chains. (For more information, see issues [#5031](https://github.com/QubesOS/qubes-issues/issues/5031) and [#6062](https://github.com/QubesOS/qubes-issues/issues/6062).)
- Templates restored in 4.2 from a pre-4.2 backup continue to target their original Qubes OS release repos. If you are using fresh templates on a clean 4.2 installation, or if you performed an [in-place upgrade from 4.1 to 4.2](/doc/upgrade/4.2/#in-place-upgrade), then this does not affect you. (For more information, see issue [#8701](https://github.com/QubesOS/qubes-issues/issues/8701).)
- DomU firewalls have completely switched to nftables. Users should add their custom rules to the ``custom-input`` and ``custom-forward`` chains. (For more information, see issues `#5031 <https://github.com/QubesOS/qubes-issues/issues/5031>`__ and `#6062 <https://github.com/QubesOS/qubes-issues/issues/6062>`__.)
Also see the [full list of open bug reports affecting Qubes 4.2](https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+label%3Aaffects-4.2+label%3A%22T%3A+bug%22+is%3Aopen).
- Templates restored in 4.2 from a pre-4.2 backup continue to target their original Qubes OS release repos. If you are using fresh templates on a clean 4.2 installation, or if you performed an :ref:`in-place upgrade from 4.1 to 4.2 <user/downloading-installing-upgrading/upgrade/4_2:in-place upgrade>`, then this does not affect you. (For more information, see issue `#8701 <https://github.com/QubesOS/qubes-issues/issues/8701>`__.)
We strongly recommend [updating Qubes OS](/doc/how-to-update/) immediately after installation in order to apply all available bug fixes.
## Notes
- Qubes 4.2 does not support Debian 11 templates (see [supported template releases](/doc/supported-releases/#templates)). Please [upgrade your Debian templates](/doc/templates/debian/#upgrading) to Debian 12.
Also see the `full list of open bug reports affecting Qubes 4.2 <https://github.com/QubesOS/qubes-issues/issues?q=is%3Aissue+label%3Aaffects-4.2+label%3A%22T%3A+bug%22+is%3Aopen>`__.
We strongly recommend :doc:`updating Qubes OS </user/how-to-guides/how-to-update>` immediately after installation in order to apply all available bug fixes.
Notes
-----
- Qubes 4.2 does not support Debian 11 templates (see :ref:`supported template releases <user/downloading-installing-upgrading/supported-releases:templates>`). Please :ref:`upgrade your Debian templates <user/templates/debian/debian:upgrading>` to Debian 12.
- Qubes 4.2.2 includes a fix for `#8332: File-copy qrexec service is overly restrictive <https://github.com/QubesOS/qubes-issues/issues/8332>`__. As explained in the issue comments, we introduced a change in Qubes 4.2.0 that caused inter-qube file-copy/move actions to reject filenames containing, e.g., non-Latin characters and certain symbols. The rationale for this change was to mitigate the security risks associated with unusual unicode characters and invalid encoding in filenames, which some software might handle in an unsafe manner and which might cause confusion for users. Such a change represents a trade-off between security and usability.
- Qubes 4.2.2 includes a fix for [#8332: File-copy qrexec service is overly restrictive](https://github.com/QubesOS/qubes-issues/issues/8332). As explained in the issue comments, we introduced a change in Qubes 4.2.0 that caused inter-qube file-copy/move actions to reject filenames containing, e.g., non-Latin characters and certain symbols. The rationale for this change was to mitigate the security risks associated with unusual unicode characters and invalid encoding in filenames, which some software might handle in an unsafe manner and which might cause confusion for users. Such a change represents a trade-off between security and usability.
- After the change went live, we received several user reports indicating more severe usability problems than we had anticipated. Moreover, these problems were prompting users to resort to dangerous workarounds (such as packing files into an archive format prior to copying) that carry far more risk than the original risk posed by the unrestricted filenames. In addition, we realized that this was a backward-incompatible change that should not have been introduced in a minor release in the first place.
- Therefore, we have decided, for the time being, to restore the original (pre-4.2) behavior by introducing a new `allow-all-names` argument for the `qubes.Filecopy` service. By default, `qvm-copy` and similar tools will use this less restrictive service (`qubes.Filecopy +allow-all-names`) whenever they detect any files that would be have been blocked by the more restrictive service (`qubes.Filecopy +`). If no such files are detected, they will use the more restrictive service.
- Users who wish to opt for the more restrictive 4.2.0 and 4.2.1 behavior can do so by modifying their RPC policy rules. To switch a single rule to the more restrictive behavior, change `*` in the argument column to `+` (i.e., change "any argument" to "only empty"). To use the more restrictive behavior globally, add the following "deny" rule before all other relevant rules:
```
qubes.Filecopy +allow-all-names @anyvm @anyvm deny
```
- For more information, see [RPC policies](/doc/rpc-policy/) and [Qube configuration interface](/doc/vm-interface/#qubes-rpc).
- Beginning with Qubes 4.2, the recommended way to update Qubes OS via the command line has changed. Salt is no longer the preferred method, though it is still supported. Instead, `qubes-dom0-update` is recommended for updating dom0, and `qubes-vm-update` is recommended for updating templates and standalones. (The recommended way to update via the GUI has not changed. The Qubes Update tool is still the preferred method.) For more information, see [How to update](/doc/how-to-update/).
- Therefore, we have decided, for the time being, to restore the original (pre-4.2) behavior by introducing a new ``allow-all-names`` argument for the ``qubes.Filecopy`` service. By default, ``qvm-copy`` and similar tools will use this less restrictive service (``qubes.Filecopy +allow-all-names``) whenever they detect any files that would be have been blocked by the more restrictive service (``qubes.Filecopy +``). If no such files are detected, they will use the more restrictive service.
## Download
- Users who wish to opt for the more restrictive 4.2.0 and 4.2.1 behavior can do so by modifying their RPC policy rules. To switch a single rule to the more restrictive behavior, change ``*`` in the argument column to ``+`` (i.e., change “any argument” to “only empty”). To use the more restrictive behavior globally, add the following “deny” rule before all other relevant rules:
All Qubes ISOs and associated [verification files](/security/verifying-signatures/) are available on the [downloads](/downloads/) page.
.. code:: bash
## Installation instructions
qubes.Filecopy +allow-all-names @anyvm @anyvm deny
See the [installation guide](/doc/installation-guide/).
## Upgrading
Please see [how to upgrade to Qubes 4.2](/doc/upgrade/4.2/).
- For more information, see :doc:`RPC policies </user/advanced-topics/rpc-policy>` and :ref:`Qube configuration interface <developer/debugging/vm-interface:qubes rpc>`.
- Beginning with Qubes 4.2, the recommended way to update Qubes OS via the command line has changed. Salt is no longer the preferred method, though it is still supported. Instead, ``qubes-dom0-update`` is recommended for updating dom0, and ``qubes-vm-update`` is recommended for updating templates and standalones. (The recommended way to update via the GUI has not changed. The Qubes Update tool is still the preferred method.) For more information, see :doc:`How to update </user/how-to-guides/how-to-update>`.
Download
--------
All Qubes ISOs and associated :doc:`verification files </project-security/verifying-signatures>` are available on the `downloads <https://www.qubes-os.org/downloads/>`__ page.
Installation instructions
-------------------------
See the :doc:`installation guide </user/downloading-installing-upgrading/installation-guide>`.
Upgrading
---------
Please see :doc:`how to upgrade to Qubes 4.2 </user/downloading-installing-upgrading/upgrade/4_2>`.
.. |Screenshot of the Qubes OS Update tool| image:: /attachment/site/4-2_update.png
.. |Screenshot of the Qubes OS Global Config tool| image:: /attachment/site/4-2_global-config_1.png
.. |image1| image:: /attachment/site/4-2_global-config_2.png
.. |Screenshot of the Qubes OS Policy Editor tool| image:: /attachment/site/4-2_policy-editor.png

40
developer/releases/4_2/schedule.rst
View file

@ -1,20 +1,26 @@
---
lang: en
layout: doc
permalink: /doc/releases/4.2/schedule/
redirect_from:
- /en/doc/releases/4.2/schedule/
title: Qubes R4.2 release schedule
---
===========================
Qubes R4.2 release schedule
===========================
_**Please note:** This page is still an unfinished draft in progress. It is being updated as Qubes 4.2 development and testing continues._
The table below is based on our [release schedule
policy](/doc/version-scheme/#release-schedule).
**Please note:** *This page is still an unfinished draft in progress. It is being updated as Qubes 4.2 development and testing continues.*
The table below is based on our :ref:`release schedule policy <developer/releases/version-scheme:release schedule>`.
.. list-table::
:widths: 10 10
:align: center
:header-rows: 1
* - Date
- Stage
* - 2023-06-02
- 4.2.0-rc1 release
* - 2023-08-28
- 4.2.0-rc2 release
* - 2023-09-03
- 4.2.0-rc3 release
* - 2023-10-13
- 4.2.0-rc4 release
| Date | Stage |
| ----------:| ----------------------------------------- |
| 2023-06-02 | 4.2.0-rc1 release |
| 2023-08-28 | 4.2.0-rc2 release |
| 2023-09-03 | 4.2.0-rc3 release |
| 2023-10-13 | 4.2.0-rc4 release |

39
developer/releases/notes.rst
View file

@ -1,16 +1,25 @@
---
lang: en
layout: doc
permalink: /doc/releases/notes/
ref: 13
title: Release notes
---
=============
Release notes
=============
.. toctree::
:maxdepth: 1
Qubes R1.0 release notes </developer/releases/1_0/release-notes>
Qubes R2.0 release notes </developer/releases/2_0/release-notes>
Qubes R3.0 release notes </developer/releases/3_0/release-notes>
Qubes R3.1 release notes </developer/releases/3_1/release-notes>
Qubes R3.2 release notes </developer/releases/3_2/release-notes>
Qubes R4.0 release notes </developer/releases/4_0/release-notes>
Qubes R4.1 release notes </developer/releases/4_1/release-notes>
Qubes R4.2 release notes </developer/releases/4_2/release-notes>
- [Qubes R1.0 release notes](/doc/releases/1.0/release-notes/)
- [Qubes R2.0 release notes](/doc/releases/2.0/release-notes/)
- [Qubes R3.0 release notes](/doc/releases/3.0/release-notes/)
- [Qubes R3.1 release notes](/doc/releases/3.1/release-notes/)
- [Qubes R3.2 release notes](/doc/releases/3.2/release-notes/)
- [Qubes R4.0 release notes](/doc/releases/4.0/release-notes/)
- [Qubes R4.1 release notes](/doc/releases/4.1/release-notes/)
- [Qubes R4.2 release notes](/doc/releases/4.2/release-notes/)

33
developer/releases/schedules.rst
View file

@ -1,14 +1,21 @@
---
lang: en
layout: doc
permalink: /doc/releases/schedules/
ref: 15
title: Release schedules
---
=================
Release schedules
=================
.. toctree::
:maxdepth: 1
Qubes R3.0 release schedule </developer/releases/3_0/schedule>
Qubes R3.1 release schedule </developer/releases/3_1/schedule>
Qubes R3.2 release schedule </developer/releases/3_2/schedule>
Qubes R4.0 release schedule </developer/releases/4_0/schedule>
Qubes R4.1 release schedule </developer/releases/4_1/schedule>
Qubes R4.2 release schedule </developer/releases/4_2/schedule>
- [Qubes R3.0 release schedule](/doc/releases/3.0/schedule/)
- [Qubes R3.1 release schedule](/doc/releases/3.1/schedule/)
- [Qubes R3.2 release schedule](/doc/releases/3.2/schedule/)
- [Qubes R4.0 release schedule](/doc/releases/4.0/schedule/)
- [Qubes R4.1 release schedule](/doc/releases/4.1/schedule/)
- [Qubes R4.2 release schedule](/doc/releases/4.2/schedule/)

79
developer/releases/todo.rst
View file

@ -1,43 +1,64 @@
---
lang: en
layout: doc
permalink: /doc/releases/todo/
redirect_from:
- /en/doc/releases/todo/
ref: 14
title: Release checklist
---
=================
Release checklist
=================
*the checklist is probably unfinished*
On -rc1
-------
* write schedule
* create package repositories (linux-yum, linux-deb)
* update repository definition (core-agent-linux, installer-qubes-os/qubes-release)
* push all packages to `current-testing`
* draft release notes, one note per feature
* create upgrade package in previous release branch (r2->r3.0, r3.0->r3.1, etc) - core-agent-linux
* make sure that keys for the current release are included in previous release's qubes-release package (for upgrade)
* build ISO and push to mirrors
- write schedule
- create package repositories (linux-yum, linux-deb)
- update repository definition (core-agent-linux, installer-qubes-os/qubes-release)
- push all packages to ``current-testing``
- draft release notes, one note per feature
- create upgrade package in previous release branch (r2->r3.0, r3.0->r3.1, etc) - core-agent-linux
- make sure that keys for the current release are included in previous release’s qubes-release package (for upgrade)
- build ISO and push to mirrors
On subsequent -rc
-----------------
* push packages to `current`
* update release notes
* build ISO and push to mirrors
* notify @Rudd-O about the new ISO for new torrent hosting
- push packages to ``current``
- update release notes
- build ISO and push to mirrors
- notify @Rudd-O about the new ISO for new torrent hosting
On final release
----------------
* push packages to `current`
* finish release notes
* update InstallationInstructions
* build ISO and push to mirrors
* push `qubes-release` package to `current`
* notify @Rudd-O about the new ISO for new torrent hosting
* write news post
* announce
- push packages to ``current``
- finish release notes
- update InstallationInstructions
- build ISO and push to mirrors
- push ``qubes-release`` package to ``current``
- notify @Rudd-O about the new ISO for new torrent hosting
- write news post
- announce

226
developer/releases/version-scheme.rst
View file

@ -1,189 +1,107 @@
---
lang: en
layout: doc
permalink: /doc/version-scheme/
redirect_from:
- /en/doc/version-scheme/
- /doc/VersionScheme/
- /wiki/VersionScheme/
ref: 151
title: Version scheme
---
==============
Version scheme
==============
The Qubes OS Project uses the [semantic versioning](https://semver.org/)
standard. Version numbers are written as `<major>.<minor>.<patch>`. When
`<patch>` is omitted (e.g., `4.1`), it is usually either because `<patch>` is
zero (as in `4.1.0`) or because we are referring to a specific minor release
irrespective of any particular patch release within it. Similarly, the major
release number alone (e.g., `R4`) is sometimes used to refer to an entire
release series inclusive of all minor and patch releases within it.
In general, patch releases are for backward-compatible bug fixes, minor
releases are for backward-compatible enhancements and new features, and major
release are for any backward-incompatible changes. This means that, in general,
one should *not* try to introduce features or enhancements in patch releases or
any backward-incompatible changes in patch or minor releases. (Templates are a
notable exception, as upstream OSes almost always have their own release
schedules.) Bug fixes are allowed in all releases, and backward-compatible
changes are allowed in all major and minor releases.
The Qubes OS Project uses the `semantic versioning <https://semver.org/>`__ standard. Version numbers are written as ``<major>.<minor>.<patch>``. When ``<patch>`` is omitted (e.g., ``4.1``), it is usually either because ``<patch>`` is zero (as in ``4.1.0``) or because we are referring to a specific minor release irrespective of any particular patch release within it. Similarly, the major release number alone (e.g., ``R4``) is sometimes used to refer to an entire release series inclusive of all minor and patch releases within it.
Qubes OS minor releases generally include new features, new templates, and
occasionally new defaults, but they are still backward-compatible in the sense
that qubes and features that worked in the previous release still function,
though the UI may be different in some cases. In general, deprecated features
are removed only in major releases, and in-place upgrades between major versions
are not guaranteed.
In general, patch releases are for backward-compatible bug fixes, minor releases are for backward-compatible enhancements and new features, and major release are for any backward-incompatible changes. This means that, in general, one should *not* try to introduce features or enhancements in patch releases or any backward-incompatible changes in patch or minor releases. (Templates are a notable exception, as upstream OSes almost always have their own release schedules.) Bug fixes are allowed in all releases, and backward-compatible changes are allowed in all major and minor releases.
Following standard practice, **version** refers to any build that has been
assigned a version name or number, e.g., `3.2-rc2`, `4.0.4`, `4.1-beta1`. By
contrast, **release** refers to any version that is intended for consumption by
the general userbase. For example, `4.0.4` was both a **version** and a
**release**, since it was stable and intended for general public use, while
`4.1-beta1` was a **version** but *not* a **release**, since it was not stable
and was intended only for [testing](/doc/testing/). All releases are
versions, but not all versions are releases.
Qubes OS minor releases generally include new features, new templates, and occasionally new defaults, but they are still backward-compatible in the sense that qubes and features that worked in the previous release still function, though the UI may be different in some cases. In general, deprecated features are removed only in major releases, and in-place upgrades between major versions are not guaranteed.
The letter **R**, as in `R4.1`, stands for **release**. The abbreviation **RC**,
as in `3.2-rc2`, stands for **release candidate**.
Following standard practice, **version** refers to any build that has been assigned a version name or number, e.g., ``3.2-rc2``, ``4.0.4``, ``4.1-beta1``. By contrast, **release** refers to any version that is intended for consumption by the general userbase. For example, ``4.0.4`` was both a **version** and a **release**, since it was stable and intended for general public use, while ``4.1-beta1`` was a **version** but *not* a **release**, since it was not stable and was intended only for :doc:`testing </user/downloading-installing-upgrading/testing>`. All releases are versions, but not all versions are releases.
## Qubes distributions and products
The letter **R**, as in ``R4.1``, stands for **release**. The abbreviation **RC**, as in ``3.2-rc2``, stands for **release candidate**.
We intend to make it easy to make a remix of Qubes, targeting another
hypervisor or isolation provider. We may also create commercial products
intended for specific circumstances. There is one distinguished distribution
called **Qubes OS**. All source code for it is available for download under a
[free and open-source license](/doc/license/) and is openly developed on
[GitHub](https://github.com/QubesOS) and our [mailing
lists](https://www.qubes-os.org/support/). The rest of this document discusses
Qubes OS. Another remix may have its own version series.
Qubes distributions and products
--------------------------------
## Release versioning
Qubes OS as a whole is released from time to time. When preparing a new release,
we decide on the `<major>.<minor>` numbers (e.g., `3.0`, which is short for
`3.0.0`). We then publish the first release candidate, e.g., `3.0.0-rc1`. When
we feel that enough progress has been made, we'll release `3.0.0-rc2` and so on.
All these versions (which are not yet releases) are considered unstable and are
not intended for production use. You are welcome to [help us
test](/doc/testing/) these versions.
We intend to make it easy to make a remix of Qubes, targeting another hypervisor or isolation provider. We may also create commercial products intended for specific circumstances. There is one distinguished distribution called **Qubes OS**. All source code for it is available for download under a :doc:`free and open-source license </developer/code/license>` and is openly developed on `GitHub <https://github.com/QubesOS>`__ and our :doc:`mailing lists </introduction/support>`. The rest of this document discusses Qubes OS. Another remix may have its own version series.
When enough progress has been made, we announce the first stable release, e.g.
`3.0.0`. This is not only a version but an actual release. It is considered
stable, and we commit to supporting it according to our [support
schedule](/doc/supported-releases/). Core components are branched at this
moment, and bug fixes are backported from the main branch. Please see [help,
support, mailing lists, and forum](/support/) for places to ask questions about
stable releases. No major features or interface incompatibilities are to be
included in this release. We release bug fixes as patch releases (`3.0.1`,
`3.0.2`, and so on), while backward-compatible enhancements and new features
are introduced in the next minor release (e.g., `3.1`). Any
backward-incompatible changes are introduced in the next major release (e.g.,
`4.0`).
Release versioning
------------------
Please see [issue tracking](/doc/issue-tracking/) for information about how
releases are handled in the issue tracker.
## Release schedule
Qubes OS as a whole is released from time to time. When preparing a new release, we decide on the ``<major>.<minor>`` numbers (e.g., ``3.0``, which is short for ``3.0.0``). We then publish the first release candidate, e.g., ``3.0.0-rc1``. When we feel that enough progress has been made, we’ll release ``3.0.0-rc2`` and so on. All these versions (which are not yet releases) are considered unstable and are not intended for production use. You are welcome to :doc:`help us test </user/downloading-installing-upgrading/testing>` these versions.
There is no specific schedule for releases other than a general roadmap.
When the time comes, we declare a feature freeze, tag `-rc1`, and
release an ISO. From this point on, no new features are accepted, and our
schedule begins.
When enough progress has been made, we announce the first stable release, e.g. ``3.0.0``. This is not only a version but an actual release. It is considered stable, and we commit to supporting it according to our :doc:`support schedule </user/downloading-installing-upgrading/supported-releases>`. Core components are branched at this moment, and bug fixes are backported from the main branch. Please see :doc:`help, support, mailing lists, and forum </introduction/support>` for places to ask questions about stable releases. No major features or interface incompatibilities are to be included in this release. We release bug fixes as patch releases (``3.0.1``, ``3.0.2``, and so on), while backward-compatible enhancements and new features are introduced in the next minor release (e.g., ``3.1``). Any backward-incompatible changes are introduced in the next major release (e.g., ``4.0``).
Each release candidate period is as follows: For the first two weeks, we accept
and assign bug reports to be fixed before the next release candidate. For the
next two weeks, we generally focus on fixing assigned bug reports, so issues
discovered during this period may be postponed until a later RC. Finally,
there is a one week current-testing freeze, during which time no new packages
are released, in the hope that they will be installed and tested by wider user
base.
Please see :doc:`issue tracking </introduction/issue-tracking>` for information about how releases are handled in the issue tracker.
The next RC is released five weeks after the former. All packages are published
in the `current` repository, and the cycle starts over. There should always be
at least one release candidate before the final release.
Release schedule
----------------
| Stage | Duration |
| ------------------------ | --------- |
| initial testing | two weeks |
| bug fixing | two weeks |
| `current-testing` freeze | one week |
Starting with the second cycle (that is, after `-rc1`), two weeks into the cycle
(after the primary bug-reporting period), we decide whether there should be
another RC. If, based on the bugs that have been reported, we decide that the
latest RC will be designated as the stable release, then we decide on its
release date, which should be no more than one week later.
There is no specific schedule for releases other than a general roadmap. When the time comes, we declare a feature freeze, tag ``-rc1``, and release an ISO. From this point on, no new features are accepted, and our schedule begins.
[![Release cycle](/attachment/doc/release-cycle.svg)](/attachment/doc/release-cycle.svg)
Each release candidate period is as follows: For the first two weeks, we accept and assign bug reports to be fixed before the next release candidate. For the next two weeks, we generally focus on fixing assigned bug reports, so issues discovered during this period may be postponed until a later RC. Finally, there is a one week current-testing freeze, during which time no new packages are released, in the hope that they will be installed and tested by wider user base.
## Bug priorities
The next RC is released five weeks after the former. All packages are published in the ``current`` repository, and the cycle starts over. There should always be at least one release candidate before the final release.
When deciding whether the current release candidate is the final one, the
Committee takes bug [priorities](/doc/issue-tracking/#priority) into
consideration. The meaning of them is as follows:
.. list-table::
:widths: 26 26
:align: center
:header-rows: 1
- `blocker` --- when any such bug is present in the current release candidate,
it can't be considered final release. Bugs with this priority must be fixed
before the next release candidate, even if that means delaying its release
(which should be considered only last resort option).
* - Stage
- Duration
* - initial testing
- two weeks
* - bug fixing
- two weeks
* - ``current-testing`` freeze
- one week
- `critical` --- when any such bug is present in the current release candidate,
it can't be considered final release. But such bugs are not qualified to
delay next release candidate release.
- `major` --- existence of such bugs do not strictly prevent the current
release candidate be considered final (but of course we should try hard to
not have them there). Fixing bugs of this priority can be delayed and
qualified as updates to the final stable release.
Starting with the second cycle (that is, after ``-rc1``), two weeks into the cycle (after the primary bug-reporting period), we decide whether there should be another RC. If, based on the bugs that have been reported, we decide that the latest RC will be designated as the stable release, then we decide on its release date, which should be no more than one week later.
- `default` and `minor` --- existence of such bugs do not prevent the current
release candidate be considered final. Fixing such bugs can be delayed to the
next Qubes OS release. Eventually such fixes might be backported as an update
to the stable release(s). (`default` should really be assigned a more
specific priority, but in practice there are too many issues and not enough
time, so `default` ends up staying on many issues.)
|Release cycle|
All above is about bugs, no features should be assigned to the current release
after first `-rc`. Supreme Committee is free to adjust priorities
appropriately.
Bug priorities
--------------
## Component version
Qubes release is defined as specific versions of components, which are
developed more or less separately. Their versions are composed of major and
minor version of target Qubes OS release followed by third component which is
just incremented. There is no apparent indication that given version is stable
or not.
When deciding whether the current release candidate is the final one, the Committee takes bug :ref:`priorities <introduction/issue-tracking:priority>` into consideration. The meaning of them is as follows:
There are some non-essential components like `qubes-apps-*` that are shared
between releases. Their versions indicate oldest qubes-release that is
supported. We try hard to support multiple releases by one branch to ease code
maintenance.
- ``blocker`` — when any such bug is present in the current release candidate, it can’t be considered final release. Bugs with this priority must be fixed before the next release candidate, even if that means delaying its release (which should be considered only last resort option).
Different Qubes releases remixes may comprise of different components and
version are not guaranteed to be monotonic between releases. We may decide that
for newer release some component should be downgraded. There is no guarantee
that arbitrary combination of different versions of random components will
yield usable (or even install-able) compilation.
- ``critical`` — when any such bug is present in the current release candidate, it can’t be considered final release. But such bugs are not qualified to delay next release candidate release.
## Git tags and branches
- ``major`` — existence of such bugs do not strictly prevent the current release candidate be considered final (but of course we should try hard to not have them there). Fixing bugs of this priority can be delayed and qualified as updates to the final stable release.
We mark each component version in the repository by tag containing
`v<version>`. Likewise, each Qubes OS release is marked by `R<release>` tag.
- ``default`` and ``minor`` — existence of such bugs do not prevent the current release candidate be considered final. Fixing such bugs can be delayed to the next Qubes OS release. Eventually such fixes might be backported as an update to the stable release(s). (``default`` should really be assigned a more specific priority, but in practice there are too many issues and not enough time, so ``default`` ends up staying on many issues.)
At the release of some release we create branches named like `release2`. Only
bug fixes and compatible improvements are backported to these branches. These
branches should compile. All new development is done in `main` branch. This
branch is totally unsupported and may not even compile depending on maintainer
of repository.
All version and release tags should be made and signed by someone from ITL
staff. Public keys are included in `qubes-builder` and available at
<https://keys.qubes-os.org/keys/>.
## Check installed version
All above is about bugs, no features should be assigned to the current release after first ``-rc``. Supreme Committee is free to adjust priorities appropriately.
If you want to know which version you are running, for example to report an
issue, you can either check in the Qubes Manager menu under `About > Qubes OS`
or in the file `/etc/qubes-release` in dom0. For the latter you can use a
command like `cat /etc/qubes-release` in a dom0 terminal.
Component version
-----------------
Qubes release is defined as specific versions of components, which are developed more or less separately. Their versions are composed of major and minor version of target Qubes OS release followed by third component which is just incremented. There is no apparent indication that given version is stable or not.
There are some non-essential components like ``qubes-apps-*`` that are shared between releases. Their versions indicate oldest qubes-release that is supported. We try hard to support multiple releases by one branch to ease code maintenance.
Different Qubes releases remixes may comprise of different components and version are not guaranteed to be monotonic between releases. We may decide that for newer release some component should be downgraded. There is no guarantee that arbitrary combination of different versions of random components will yield usable (or even install-able) compilation.
Git tags and branches
---------------------
We mark each component version in the repository by tag containing ``v<version>``. Likewise, each Qubes OS release is marked by ``R<release>`` tag.
At the release of some release we create branches named like ``release2``. Only bug fixes and compatible improvements are backported to these branches. These branches should compile. All new development is done in ``main`` branch. This branch is totally unsupported and may not even compile depending on maintainer of repository.
All version and release tags should be made and signed by someone from ITL staff. Public keys are included in ``qubes-builder`` and available at https://keys.qubes-os.org/keys/.
Check installed version
-----------------------
If you want to know which version you are running, for example to report an issue, you can either check in the Qubes Manager menu under ``About > Qubes OS`` or in the file ``/etc/qubes-release`` in dom0. For the latter you can use a command like ``cat /etc/qubes-release`` in a dom0 terminal.
.. |Release cycle| image:: /attachment/doc/release-cycle.png

1162
developer/services/admin-api.rst
View file

File diff suppressed because it is too large Load diff

82
developer/services/disposablevm-implementation.rst
View file

@ -1,58 +1,74 @@
---
lang: en
layout: doc
permalink: /doc/disposablevm-implementation/
redirect_from:
- /doc/dvm-impl/
- /en/doc/dvm-impl/
- /doc/DVMimpl/
- /wiki/DVMimpl/
ref: 34
title: Disposable implementation
---
=========================
Disposable implementation
=========================
**Note: The content below applies to Qubes R3.2.**
DisposableVM image preparation
------------------------------
DisposableVM is not started like other VMs, by executing equivalent of `xl create` - it would be too slow. Instead, DisposableVM are started by restore from a savefile.
Preparing a savefile is done by `/usr/lib/qubes/qubes_prepare_saved_domain.sh` script. It takes two mandatory arguments, appvm name (APPVM) and the savefile name, and optional path to "prerun" script. The script executes the following steps:
DisposableVM is not started like other VMs, by executing equivalent of ``xl create`` - it would be too slow. Instead, DisposableVM are started by restore from a savefile.
1. APPVM is started by `qvm-start`
2. xenstore key `/local/domain/appvm_domain_id/qubes_save_request` is created
3. if prerun script was specified, copy it to `qubes_save_script` xenstore key
4. wait for the `qubes_used_mem` key to appear
5. (in APPVM) APPVM boots normally, up to the point in `/etc/init.d/qubes_core` script when the presence of `qubes_save_request` key is tested. If it exists, then
1. (in APPVM) if exists, prerun script is retrieved from the respective xenstore key and executed. This preloads filesystem cache with useful applications, so that they will start faster.
2. (in APPVM) the amount of used memory is stored to `qubes_used_mem` xenstore key
3. (in APPVM) busy-waiting for `qubes_restore_complete` xenstore key to appear
Preparing a savefile is done by ``/usr/lib/qubes/qubes_prepare_saved_domain.sh`` script. It takes two mandatory arguments, appvm name (APPVM) and the savefile name, and optional path to “prerun” script. The script executes the following steps:
1. APPVM is started by ``qvm-start``
2. xenstore key ``/local/domain/appvm_domain_id/qubes_save_request`` is created
3. if prerun script was specified, copy it to ``qubes_save_script`` xenstore key
4. wait for the ``qubes_used_mem`` key to appear
5. (in APPVM) APPVM boots normally, up to the point in ``/etc/init.d/qubes_core`` script when the presence of ``qubes_save_request`` key is tested. If it exists, then
1. (in APPVM) if exists, prerun script is retrieved from the respective xenstore key and executed. This preloads filesystem cache with useful applications, so that they will start faster.
2. (in APPVM) the amount of used memory is stored to ``qubes_used_mem`` xenstore key
3. (in APPVM) busy-waiting for ``qubes_restore_complete`` xenstore key to appear
6. when ``qubes_used_mem`` key appears, the domain memory is reduced to this amount, to make the savefile smaller.
6. when `qubes_used_mem` key appears, the domain memory is reduced to this amount, to make the savefile smaller.
7. APPVM private image is detached
8. the domain is saved via `xl save`
9. the COW file volatile.img (cow for root fs and swap) is packed to `saved_cows.tar` archive
The `qubes_prepare_saved_domain.sh` script is lowlevel. It is usually called by `qvm-create-default-dvm` script, that takes care of creating a special AppVM (named template\_name-dvm) to be passed to `qubes_prepare_saved_domain.sh`, as well as copying the savefile to /dev/shm (the latter action is not done if the `/var/lib/qubes/dvmdata/dont_use_shm` file exists).
8. the domain is saved via ``xl save``
9. the COW file volatile.img (cow for root fs and swap) is packed to ``saved_cows.tar`` archive
The ``qubes_prepare_saved_domain.sh`` script is lowlevel. It is usually called by ``qvm-create-default-dvm`` script, that takes care of creating a special AppVM (named template_name-dvm) to be passed to ``qubes_prepare_saved_domain.sh``, as well as copying the savefile to /dev/shm (the latter action is not done if the ``/var/lib/qubes/dvmdata/dont_use_shm`` file exists).
Restoring a DisposableVM from the savefile
------------------------------------------
Normally, disposable VM is created when qubes rpc request with target *\$dispvm* is received. Then, as a part of rpc connection setup, the `qfile-daemon-dvm` program is executed; it executes `/usr/lib/qubes/qubes_restore` program. It is crucial that this program executes quickly, to make DisposableVM creation overhead bearable for the user. Its main steps are:
Normally, disposable VM is created when qubes rpc request with target *$dispvm* is received. Then, as a part of rpc connection setup, the ``qfile-daemon-dvm`` program is executed; it executes ``/usr/lib/qubes/qubes_restore`` program. It is crucial that this program executes quickly, to make DisposableVM creation overhead bearable for the user. Its main steps are:
1. modify the savefile so that the VM name, VM UUID, MAC address and IP address are unique
2. restore the COW files from the `saved_cows.tar`
3. create the `/var/run/qubes/fast_block_attach` file, whose presence tells the `/etc/xen/scripts/block` script to bypass some redundant checks and execute as fast as possible.
4. execute `xl restore` in order to restore a domain.
5. create the same xenstore keys as normally created when AppVM boots (e.g. `qubes_ip`)
6. create the `qubes_restore_complete` xenstore key. This allows the boot process in DisposableVM to continue.
2. restore the COW files from the ``saved_cows.tar``
3. create the ``/var/run/qubes/fast_block_attach`` file, whose presence tells the ``/etc/xen/scripts/block`` script to bypass some redundant checks and execute as fast as possible.
4. execute ``xl restore`` in order to restore a domain.
5. create the same xenstore keys as normally created when AppVM boots (e.g. ``qubes_ip``)
6. create the ``qubes_restore_complete`` xenstore key. This allows the boot process in DisposableVM to continue.
The actual passing of files between AppVM and a DisposableVM is implemented via qubes rpc.
Validating the DisposableVM savefile
------------------------------------
DisposableVM savefile contains references to template rootfs and to COW files. The COW files are restored before each DisposableVM start, so they cannot change. On the other hand, if templateVM is started, the template rootfs will change, and it may not be coherent with the COW files.
Therefore, the check for template rootfs modification time being older than DisposableVM savefile modification time is required. It is done in `qfilexchgd` daemon, just before restoring DisposableVM. If necessary, an attempt is made to recreate the DisposableVM savefile, using the last template used (or default template, if run for the first time) and the default prerun script, residing at `/var/lib/qubes/vm-templates/templatename/dispvm_prerun.sh`. Unfortunately, the prerun script takes a lot of time to execute - therefore, after template rootfs modification, the next DisposableVM creation can be longer by about 2.5 minutes.
Therefore, the check for template rootfs modification time being older than DisposableVM savefile modification time is required. It is done in ``qfilexchgd`` daemon, just before restoring DisposableVM. If necessary, an attempt is made to recreate the DisposableVM savefile, using the last template used (or default template, if run for the first time) and the default prerun script, residing at ``/var/lib/qubes/vm-templates/templatename/dispvm_prerun.sh``. Unfortunately, the prerun script takes a lot of time to execute - therefore, after template rootfs modification, the next DisposableVM creation can be longer by about 2.5 minutes.

45
developer/services/dom0-secure-updates.rst
View file

@ -1,49 +1,50 @@
---
lang: en
layout: doc
permalink: /doc/dom0-secure-updates/
redirect_from:
- /en/doc/dom0-secure-updates/
- /doc/Dom0SecureUpdates/
- /wiki/Dom0SecureUpdates/
ref: 43
title: Dom0 secure updates
---
===================
Dom0 secure updates
===================
Reasons for Dom0 updates
------------------------
Normally there should be few reasons for updating software in Dom0. This is because there is no networking in Dom0, which means that even if some bugs will be discovered e.g. in the Dom0 Desktop Manager, this really is not a problem for Qubes, because all the third-party software running in Dom0 is not accessible from VMs or network in any way. Some exceptions to the above include: Qubes GUI daemon, Xen store daemon, and disk back-ends (we plan to move the disk backends to untrusted domain in Qubes 2.0). Of course we believe this software is reasonably secure and we hope it will not need patching.
Normally there should be few reasons for updating software in Dom0. This is because there is no networking in Dom0, which means that even if some bugs will be discovered e.g. in the Dom0 Desktop Manager, this really is not a problem for Qubes, because all the third-party software running in Dom0 is not accessible from VMs or network in any way. Some exceptions to the above include: Qubes GUI daemon, Xen store daemon, and disk back-ends (we plan to move the disk backends to untrusted domain in Qubes 2.0). Of course we believe this software is reasonably secure and we hope it will not need patching.
However, we anticipate some other situations when updating Dom0 software might be required:
- Updating drivers/libs for new hardware support
- Correcting non-security related bugs (e.g. new buttons for qubes manager)
- Adding new features (e.g. GUI backup tool)
- Correcting non-security related bugs (e.g. new buttons for qubes manager)
- Adding new features (e.g. GUI backup tool)
Problems with traditional network-based update mechanisms
---------------------------------------------------------
Dom0 is the most trusted domain on Qubes OS, and for this reason we decided to design Qubes in such a way that Dom0 is not connected to any network. In fact only certain domains can be connected to a network via so-called network domains. There can also be more than one network domain, e.g. in case the user is connected to more than one physically or logically separated networks.
Dom0 is the most trusted domain on Qubes OS, and for this reason we decided to design Qubes in such a way that Dom0 is not connected to any network. In fact only certain domains can be connected to a network via so-called network domains. There can also be more than one network domain, e.g. in case the user is connected to more than one physically or logically separated networks.
Secure update mechanism we use in Qubes (starting from Beta 2)
-------------------------------------------------------------
--------------------------------------------------------------
Keeping Dom0 not connected to any network makes it hard, however, to provide updates for software in Dom0. For this reason we have come up with the following mechanism for Dom0 updates, which minimizes the amount of untrusted input processed by Dom0 software:
The update process is initiated by [qubes-dom0-update script](https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qubes-dom0-update), running in Dom0.
The update process is initiated by `qubes-dom0-update script <https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qubes-dom0-update>`__, running in Dom0.
Updates (`*.rpm` files) are checked and downloaded by UpdateVM, which by default is the same as the firewall VM, but can be configured to be any other, network-connected VM. This is done by [qubes-download-dom0-updates.sh script](https://github.com/QubesOS/qubes-core-agent-linux/blob/release2/misc/qubes-download-dom0-updates.sh) (this script is executed using qrexec by the previously mentioned qubes-dom0-update). Note that we assume that this script might get compromised and fetch maliciously compromised downloads -- this is not a problem as Dom0 verifies digital signatures on updates later. The downloaded rpm files are placed in a `/var/lib/qubes/dom0-updates` directory on UpdateVM filesystem (again, they might get compromised while being kept there, still this isn't a problem). This directory is passed to yum using the `--installroot=` option.
Updates (``*.rpm`` files) are checked and downloaded by UpdateVM, which by default is the same as the firewall VM, but can be configured to be any other, network-connected VM. This is done by `qubes-download-dom0-updates.sh script <https://github.com/QubesOS/qubes-core-agent-linux/blob/release2/misc/qubes-download-dom0-updates.sh>`__ (this script is executed using qrexec by the previously mentioned qubes-dom0-update). Note that we assume that this script might get compromised and fetch maliciously compromised downloads – this is not a problem as Dom0 verifies digital signatures on updates later. The downloaded rpm files are placed in a ``/var/lib/qubes/dom0-updates`` directory on UpdateVM filesystem (again, they might get compromised while being kept there, still this isn’t a problem). This directory is passed to yum using the ``--installroot=`` option.
Once updates are downloaded, the update script that runs in UpdateVM requests an RPM service [qubes.ReceiveUpdates](https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qubes.ReceiveUpdates) to be executed in Dom0. This service is implemented by [qubes-receive-updates script](https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qubes-receive-updates) running in Dom0. The Dom0's qubes-dom0-update script (which originally initiated the whole update process) waits until qubes-receive-updates finished.
Once updates are downloaded, the update script that runs in UpdateVM requests an RPM service `qubes.ReceiveUpdates <https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qubes.ReceiveUpdates>`__ to be executed in Dom0. This service is implemented by `qubes-receive-updates script <https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qubes-receive-updates>`__ running in Dom0. The Dom0’s qubes-dom0-update script (which originally initiated the whole update process) waits until qubes-receive-updates finished.
The qubes-receive-updates script processes the untrusted input from Update VM: it first extracts the received `*.rpm` files (that are sent over qrexec data connection) and then verifies digital signature on each file. The qubes-receive-updates script is a security-critical component of the Dom0 update process (as is the [qfile-dom0-unpacker.c](https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qfile-dom0-unpacker.c) and the rpm utility, both used by the qubes-receive-updates for processing the untrusted input).
The qubes-receive-updates script processes the untrusted input from Update VM: it first extracts the received ``*.rpm`` files (that are sent over qrexec data connection) and then verifies digital signature on each file. The qubes-receive-updates script is a security-critical component of the Dom0 update process (as is the `qfile-dom0-unpacker.c <https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qfile-dom0-unpacker.c>`__ and the rpm utility, both used by the qubes-receive-updates for processing the untrusted input).
Once qubes-receive-updates finished unpacking and verifying the updates, the updates are placed in ``qubes-receive-updates`` directory in Dom0 filesystem. Those updates are now trusted. Dom0 is configured (see /etc/yum.conf in Dom0) to use this directory as a default (and only) [yum repository](https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qubes-cached.repo).
Once qubes-receive-updates finished unpacking and verifying the updates, the updates are placed in ``qubes-receive-updates`` directory in Dom0 filesystem. Those updates are now trusted. Dom0 is configured (see /etc/yum.conf in Dom0) to use this directory as a default (and only) `yum repository <https://github.com/QubesOS/qubes-core-admin-linux/blob/release2/dom0-updates/qubes-cached.repo>`__.
Finally, qubes-dom0-update runs ``yum update`` that fetches the RPMs from qubes-cached repo and installs them as usual.
Security benefit of our update mechanism
----------------------------------------
The benefit of the above scheme is that one doesn't need to trust the TCP/IP stack, the HTTP stack, and wget implementation in order to safely deliver and install updates in Dom0. One only needs to trust a few hundred lines of code as discussed above, as well as the rpm utility for properly checking digital signature (but this is required in any update scheme).
The benefit of the above scheme is that one doesn’t need to trust the TCP/IP stack, the HTTP stack, and wget implementation in order to safely deliver and install updates in Dom0. One only needs to trust a few hundred lines of code as discussed above, as well as the rpm utility for properly checking digital signature (but this is required in any update scheme).

27
developer/services/qfilecopy.rst
View file

@ -1,29 +1,28 @@
---
lang: en
layout: doc
permalink: /doc/qfilecopy/
redirect_from:
- /en/doc/qfilecopy/
- /doc/Qfilecopy/
- /wiki/Qfilecopy/
ref: 35
title: Inter-qube file copying (qfilecopy)
---
===================================
Inter-qube file copying (qfilecopy)
===================================
There are two cases when we need a mechanism to copy files between VMs:
- "regular" file copy - when user instructs file manager to copy a given files/directories to a different VM
- DispVM copy - user selects "open in DispVM" on a file; this file must be copied to a DisposableVM, edited by user, and possibly a modified file copied back from DispVM to VM.
- “regular” file copy - when user instructs file manager to copy a given files/directories to a different VM
- DispVM copy - user selects “open in DispVM” on a file; this file must be copied to a DisposableVM, edited by user, and possibly a modified file copied back from DispVM to VM.
In the early days of Qubes OS, for both cases, a block device (backed by a file in dom0 with a vfat filesystem on it) was attached to VM, file(s) copied there, and then the device was detached and attached to target VM. In the DispVM case, if a edited file has been modified, another block device is passed to requester VM in order to update the source file.
This has the following disadvantages:
- performance - dom0 has to prepare and attach/detach block devices, which is slow because of hotplug scripts involvement.
- security - VM kernel parses partition table and filesystem metadata from the block device; they are controlled by (potentially untrusted) sender VM.
In modern Qubes OS releases, we have reimplemented interVM file copy using qrexec, which addresses the above mentioned disadvantages. Nowadays, even more generic solution (qubes rpc) is used. See the developer docs on qrexec and qubes rpc. In a nutshell, the file sender and the file receiver just read/write from stdin/stdout, and the qubes rpc layer passes data properly - so, no block devices are used.
The rpc action for regular file copy is *qubes.Filecopy*, the rpc client is named *qfile-agent*, the rpc server is named *qfile-unpacker*. For DispVM copy, the rpc action is *qubes.OpenInVM*, the rpc client is named *qopen-in-vm*, rpc server is named *vm-file-editor*. Note that the *qubes.OpenInVM* action can be done on a normal AppVM, too.
Being a rpc server, *qfile-unpacker* must be coded securely, as it processes potentially untrusted data format. Particularly, we do not want to use external tar or cpio and be prone to all vulnerabilities in them; we want a simplified, small utility, that handles only directory/file/symlink file type, permissions, mtime/atime, and assume user/user ownership. In the current implementation, the code that actually parses the data from srcVM has ca 100 lines of code and executes chrooted to the destination directory. The latter is hardcoded to `~user/QubesIncoming/srcVM`; because of chroot, there is no possibility to alter files outside of this directory.
Being a rpc server, *qfile-unpacker* must be coded securely, as it processes potentially untrusted data format. Particularly, we do not want to use external tar or cpio and be prone to all vulnerabilities in them; we want a simplified, small utility, that handles only directory/file/symlink file type, permissions, mtime/atime, and assume user/user ownership. In the current implementation, the code that actually parses the data from srcVM has ca 100 lines of code and executes chrooted to the destination directory. The latter is hardcoded to ``~user/QubesIncoming/srcVM``; because of chroot, there is no possibility to alter files outside of this directory.

79
developer/services/qfileexchgd.rst
View file

@ -1,55 +1,72 @@
---
lang: en
layout: doc
permalink: /doc/qfileexchgd/
redirect_from:
- /en/doc/qfileexchgd/
- /doc/Qfileexchgd/
- /wiki/Qfileexchgd/
ref: 40
title: Qfileexchgd (deprecated)
---
========================
Qfileexchgd (deprecated)
========================
**This mechanism is obsolete as of Qubes Beta 1!**
Please see this [page](/doc/qfilecopy/) instead.
Please see this :doc:`page </developer/services/qfilecopy>` instead.
Overview
--------
*qfilexchgd* is a dom0 daemon responsible for managing exchange of block devices ("virtual pendrives") between VMs. It is used for
*qfilexchgd* is a dom0 daemon responsible for managing exchange of block devices (“virtual pendrives”) between VMs. It is used for
- copying files between AppVMs
- copying a single file between an AppVM and a DisposableVM
*qfilexchgd* is started after first *qubes\_guid* has been started, so that it has access to X display in dom0 to present dialog messages.
*qfilexchgd* is started after first *qubes_guid* has been started, so that it has access to X display in dom0 to present dialog messages.
*qfilexchgd* is event driven. The sources of events are:
- trigger of xenstore watch for the changes in `/local/domain` xenstore hierarchy - to detect start/stop of VMs, and maintain vmname-\>vm\_xid dictionary
- trigger of xenstore watch for a change in `/local/domain/domid/device/qpen` key - VMs write to this key to request service from *qfilexchgd*
- trigger of xenstore watch for the changes in ``/local/domain`` xenstore hierarchy - to detect start/stop of VMs, and maintain vmname->vm_xid dictionary
- trigger of xenstore watch for a change in ``/local/domain/domid/device/qpen`` key - VMs write to this key to request service from *qfilexchgd*
Copying files between AppVMs
----------------------------
1. AppVM1 user runs *qvm-copy-to-vm* script (accessible from Dolphin file manager by right click on a "file(s)-\>Actions-\>Send to VM" menu). It calls */usr/lib/qubes/qubes\_penctl new*, and it writes "new" request to its `device/qpen` xenstore key. *qfilexchgd* creates a new 1G file, makes vfat fs on it, and does block-attach so that this file is attached as `/dev/xvdg` in AppVM1.
2. AppVM1 mounts `/dev/xvdg` on `/mnt/outgoing` and copies requested files there, then unmounts it.
3. AppVM1 writes "send DestVM" request to its `device/qpen` xenstore key (calling */usr/lib/qubes/qubes\_penctl send DestVM*). After getting confirmation by displaying a dialog box in dom0 display, *qfilexchgd* detaches `/dev/xvdg` from AppVM1, attaches it as `/dev/xvdh` to DestVM.
4. In DestVM, udev script for `/dev/xvdh` named *qubes\_add\_pendrive\_script* (see `/etc/udev/rules.d/qubes.rules`) mounts `/dev/xvdh` on `/mnt/incoming`, and then waits for `/mnt/incoming` to become unmounted. A file manager running in DestVM shows a new volume, and user in DestVM may copy files from it. When user in DestVM is done, then user unmounts `/mnt/incoming`. *qubes\_add\_pendrive*\_script then tells *qfilexchgd* to detach `/dev/xvdh` and terminates.
1. AppVM1 user runs *qvm-copy-to-vm* script (accessible from Dolphin file manager by right click on a “file(s)->Actions->Send to VM” menu). It calls */usr/lib/qubes/qubes_penctl new*, and it writes “new” request to its ``device/qpen`` xenstore key. *qfilexchgd* creates a new 1G file, makes vfat fs on it, and does block-attach so that this file is attached as ``/dev/xvdg`` in AppVM1.
2. AppVM1 mounts ``/dev/xvdg`` on ``/mnt/outgoing`` and copies requested files there, then unmounts it.
3. AppVM1 writes “send DestVM” request to its ``device/qpen`` xenstore key (calling */usr/lib/qubes/qubes_penctl send DestVM*). After getting confirmation by displaying a dialog box in dom0 display, *qfilexchgd* detaches ``/dev/xvdg`` from AppVM1, attaches it as ``/dev/xvdh`` to DestVM.
4. In DestVM, udev script for ``/dev/xvdh`` named *qubes_add_pendrive_script* (see ``/etc/udev/rules.d/qubes.rules``) mounts ``/dev/xvdh`` on ``/mnt/incoming``, and then waits for ``/mnt/incoming`` to become unmounted. A file manager running in DestVM shows a new volume, and user in DestVM may copy files from it. When user in DestVM is done, then user unmounts ``/mnt/incoming``. *qubes_add_pendrive*_script then tells *qfilexchgd* to detach ``/dev/xvdh`` and terminates.
Copying a single file between AppVM and a DisposableVM
------------------------------------------------------
In order to minimize attack surface presented by necessity to process virtual pendrive metadata sent by (potentially compromised and malicious) DisposableVM, AppVM\<-\>DisposableVM file exchange protocol does not use any filesystem.
1. User in AppVM1 runs *qvm-open-in-dvm* (accessible from Dolphin file manager by right click on a "file-\>Actions-\>Open in DisposableVM" menu). *qvm-open-in-dvm*
1. gets a new `/dev/xvdg` (just as described in previous paragraph)
2. computes a new unique transaction seq SEQ by incrementing `/home/user/.dvm/seq` contents,
3. writes the requested file name (say, /home/user/document.txt) to `/home/user/.dvm/SEQ`
4. creates a dvm\_header (see core.git/appvm/dvm.h) on `/dev/xvdg`, followed by file contents
5. writes the "send disposable SEQ" command to its `device/qpen` xenstore key.
In order to minimize attack surface presented by necessity to process virtual pendrive metadata sent by (potentially compromised and malicious) DisposableVM, AppVM<->DisposableVM file exchange protocol does not use any filesystem.
1. User in AppVM1 runs *qvm-open-in-dvm* (accessible from Dolphin file manager by right click on a “file->Actions->Open in DisposableVM” menu). *qvm-open-in-dvm*
1. gets a new ``/dev/xvdg`` (just as described in previous paragraph)
2. computes a new unique transaction seq SEQ by incrementing ``/home/user/.dvm/seq`` contents,
3. writes the requested file name (say, /home/user/document.txt) to ``/home/user/.dvm/SEQ``
4. creates a dvm_header (see core.git/appvm/dvm.h) on ``/dev/xvdg``, followed by file contents
5. writes the “send disposable SEQ” command to its ``device/qpen`` xenstore key.
2. *qfilexchgd* sees that “send” argument==“disposable”, and creates a new DisposableVM by calling */usr/lib/qubes/qubes_restore*. It adds the new DisposableVM to qubesDB via qvm_collection.add_new_disposablevm. Then it attaches the virtual pendrive (previously attached as ``/dev/xvdg`` at AppVM1) as ``/dev/xvdh`` in DisposableVM.
3. In DisposableVM, *qubes_add_pendrive_script* sees non-zero ``qubes_transaction_seq`` key in xenstore, and instead processing the virtual pendrive as in the case of normal copy, treats it as DVM transaction (a request, because we run in DisposableVM). It retrieves the body of the file passed in ``/dev/xvdh``, copies to /tmp, and runs *mime-open* utility to open appropriate executable to edit it. When *mime-open* returns, if the file was modified, it is sent back to AppVM1 (by writing “send AppVM1 SEQ” to ``device/qpen`` xenstore key). Then DisposableVM destroys itself.
4. In AppVM1, a new ``/dev/xvdh`` appears (because DisposableVM has sent it). *qubes_add_pendrive_script* sees non-zero ``qubes_transaction_seq`` key, and treats it as DVM transaction (a response, because we run in AppVM, not DisposableVM). It retrieves the filename from ``/home/user/.dvm/SEQ``, and copies data from ``/dev/xvdh`` to it.
2. *qfilexchgd* sees that "send" argument=="disposable", and creates a new DisposableVM by calling */usr/lib/qubes/qubes\_restore*. It adds the new DisposableVM to qubesDB via qvm\_collection.add\_new\_disposablevm. Then it attaches the virtual pendrive (previously attached as `/dev/xvdg` at AppVM1) as `/dev/xvdh` in DisposableVM.
3. In DisposableVM, *qubes\_add\_pendrive\_script* sees non-zero `qubes_transaction_seq` key in xenstore, and instead processing the virtual pendrive as in the case of normal copy, treats it as DVM transaction (a request, because we run in DisposableVM). It retrieves the body of the file passed in `/dev/xvdh`, copies to /tmp, and runs *mime-open* utility to open appropriate executable to edit it. When *mime-open* returns, if the file was modified, it is sent back to AppVM1 (by writing "send AppVM1 SEQ" to `device/qpen` xenstore key). Then DisposableVM destroys itself.
4. In AppVM1, a new `/dev/xvdh` appears (because DisposableVM has sent it). *qubes\_add\_pendrive\_script* sees non-zero `qubes_transaction_seq` key, and treats it as DVM transaction (a response, because we run in AppVM, not DisposableVM). It retrieves the filename from `/home/user/.dvm/SEQ`, and copies data from `/dev/xvdh` to it.

109
developer/services/qmemman.rst
View file

@ -1,86 +1,115 @@
---
lang: en
layout: doc
permalink: /doc/qmemman/
redirect_from:
- /en/doc/qmemman/
- /doc/Qmemman/
- /wiki/Qmemman/
ref: 41
title: Qubes memory manager (qmemman)
---
==============================
Qubes memory manager (qmemman)
==============================
Rationale
---------
Traditionally, Xen VMs are assigned a fixed amount of memory. It is not the optimal solution, as some VMs may require more memory than assigned initially, while others underutilize memory. Thus, there is a need for solution capable of shifting free memory from VM to another VM.
The [tmem](https://web.archive.org/web/20210712161104/https://oss.oracle.com/projects/tmem/) project provides a "pseudo-RAM" that is assigned on per-need basis. However this solution has some disadvantages:
The `tmem <https://web.archive.org/web/20210712161104/https://oss.oracle.com/projects/tmem/>`__ project provides a “pseudo-RAM” that is assigned on per-need basis. However this solution has some disadvantages:
- It does not provide real RAM, just an interface to copy memory to/from fast, RAM-based storage. It is perfect for swap, good for file cache, but not ideal for many tasks.
- It is deeply integrated with the Linux kernel. When Qubes will support Windows guests natively, we would have to port *tmem* to Windows, which may be challenging.
Therefore, in Qubes another solution is used. There is the *qmemman* dom0 daemon. All VMs report their memory usage (via xenstore) to *qmemman*, and it makes decisions on whether to balance memory across domains. The actual mechanism to add/remove memory from a domain (*xc.domain\_set\_target\_mem*) is already supported by both PV Linux guests and Windows guests (the latter via PV drivers).
Therefore, in Qubes another solution is used. There is the *qmemman* dom0 daemon. All VMs report their memory usage (via xenstore) to *qmemman*, and it makes decisions on whether to balance memory across domains. The actual mechanism to add/remove memory from a domain (*xc.domain_set_target_mem*) is already supported by both PV Linux guests and Windows guests (the latter via PV drivers).
Similarly, when there is need for Xen free memory (for instance, in order to create a new VM), traditionally the memory is obtained from dom0 only. When *qmemman* is running, it offers an interface to obtain memory from all domains.
To sum up, *qmemman* pros and cons.
<div class="focus">
<i class="fa fa-check"></i> <strong>Pros</strong>
</div>
|checkmark| **Pros**
- provides automatic balancing of memory across participating PV and HVM domains, based on their memory demand
- works well in practice, with less than 1% CPU consumption in the idle case
- simple, concise implementation
<div class="focus">
<i class="fa fa-times"></i> <strong>Cons</strong>
</div>
|redx| **Cons**
- the algorithm to calculate the memory requirement for a domain is necessarily simple, and may not closely reflect reality
- *qmemman* is notified by a VM about memory usage change not more often than 10 times per second (to limit CPU overhead in VM). Thus, there can be up to 0.1s delay until qmemman starts to react to the new memory requirements
- it takes more time to obtain free Xen memory, as all participating domains need to instructed to yield memory
Interface
---------
*qmemman* listens for the following events:
- writes to `/local/domain/domid/memory/meminfo` xenstore keys by *meminfo-writer* process in VM. The content of this key is taken from the VM's `/proc/meminfo` pseudofile ; *meminfo-writer* just strips some unused lines from it. Note that *meminfo-writer* writes its xenstore key only if the VM memory usage has changed significantly enough since the last update (by default 30MB), to prevent flooding with almost identical data
- commands issued over Unix socket `/var/run/qubes/qmemman.sock`. Currently, the only command recognized is to free the specified amount of memory. The QMemmanClient class implements the protocol.
- if the `/var/run/qubes/do-not-membalance` file exists, *qmemman* suspends memory balancing. It is primarily used when allocating memory for a to-be-created domain, to prevent using up the free Xen memory by the balancing algorithm before the domain creation is completed.
- writes to ``/local/domain/domid/memory/meminfo`` xenstore keys by *meminfo-writer* process in VM. The content of this key is taken from the VM’s ``/proc/meminfo`` pseudofile ; *meminfo-writer* just strips some unused lines from it. Note that *meminfo-writer* writes its xenstore key only if the VM memory usage has changed significantly enough since the last update (by default 30MB), to prevent flooding with almost identical data
- commands issued over Unix socket ``/var/run/qubes/qmemman.sock``. Currently, the only command recognized is to free the specified amount of memory. The QMemmanClient class implements the protocol.
- if the ``/var/run/qubes/do-not-membalance`` file exists, *qmemman* suspends memory balancing. It is primarily used when allocating memory for a to-be-created domain, to prevent using up the free Xen memory by the balancing algorithm before the domain creation is completed.
Algorithms basics
-----------------
The core VM property is `prefmem`. It denotes the amount of memory that should be enough for a domain to run efficiently in the nearest future. All *qmemman* algorithms will never shrink domain memory below `prefmem`. Currently, `prefmem` is simply 130% of current memory usage in a domain (without buffers and cache, but including swap). Naturally, `prefmem` is calculated by *qmemman* based on the information passed by *meminfo-writer*.
Whenever *meminfo-writer* running in domain A provides new data on memory usage to *qmemman*, the `prefmem` value for A is updated and the following balance algorithm (*qmemman\_algo.balance*) is triggered. Its output is the list of (domain\_id, new\_memory\_target\_to\_be\_set) pairs:
The core VM property is ``prefmem``. It denotes the amount of memory that should be enough for a domain to run efficiently in the nearest future. All *qmemman* algorithms will never shrink domain memory below ``prefmem``. Currently, ``prefmem`` is simply 130% of current memory usage in a domain (without buffers and cache, but including swap). Naturally, ``prefmem`` is calculated by *qmemman* based on the information passed by *meminfo-writer*.
Whenever *meminfo-writer* running in domain A provides new data on memory usage to *qmemman*, the ``prefmem`` value for A is updated and the following balance algorithm (*qmemman_algo.balance*) is triggered. Its output is the list of (domain_id, new_memory_target_to_be_set) pairs:
1. TOTAL_PREFMEM = sum of ``prefmem`` of all participating domains
2. TOTAL_MEMORY = sum of all memory assigned to participating domains plus Xen free memory
3. if TOTAL_MEMORY > TOTAL_PREFMEM, then redistribute TOTAL_MEMORY across all domains proportionally to their ``prefmem``
4. if TOTAL_MEMORY < TOTAL_PREFMEM, then
1. for all domains whose ``prefmem`` is less than actual memory, shrink them to their ``prefmem``
2. redistribute memory reclaimed in the previous step between the rest of domains, proportionally to their ``prefmem``
1. TOTAL\_PREFMEM = sum of `prefmem` of all participating domains
2. TOTAL\_MEMORY = sum of all memory assigned to participating domains plus Xen free memory
3. if TOTAL\_MEMORY \> TOTAL\_PREFMEM, then redistribute TOTAL\_MEMORY across all domains proportionally to their `prefmem`
4. if TOTAL\_MEMORY \< TOTAL\_PREFMEM, then
1. for all domains whose `prefmem` is less than actual memory, shrink them to their `prefmem`
2. redistribute memory reclaimed in the previous step between the rest of domains, proportionally to their `prefmem`
In order to avoid too frequent memory redistribution, it is actually executed only if one of the below conditions hold:
- the sum of memory size changes for all domains is more than MIN\_TOTAL\_MEMORY\_TRANSFER (150MB)
- one of the domains is below its `prefmem`, and more than MIN\_MEM\_CHANGE\_WHEN\_UNDER\_PREF (15MB) would be added to it
- the sum of memory size changes for all domains is more than MIN_TOTAL_MEMORY_TRANSFER (150MB)
Additionally, the balance algorithm is tuned so that XEN\_FREE\_MEM\_LEFT (50MB) is always left as Xen free memory, to make coherent memory allocations in driver domains work.
- one of the domains is below its ``prefmem``, and more than MIN_MEM_CHANGE_WHEN_UNDER_PREF (15MB) would be added to it
Additionally, the balance algorithm is tuned so that XEN_FREE_MEM_LEFT (50MB) is always left as Xen free memory, to make coherent memory allocations in driver domains work.
Whenever *qmemman* is asked to return X megabytes of memory to Xen free pool, the following algorithm (*qmemman_algo.balloon*) is executed:
1. find all domains (“donors”) whose actual memory is greater than its ``prefmem``
2. calculate how much memory can be reclaimed by shrinking donors to their ``prefmem``. If it is less than X, return error.
3. shrink donors, proportionally to their ``prefmem``, so that X MB should become free
4. wait BALOON_DELAY (0.1s)
5. if some domain have not given back any memory, remove it from the donors list, and go to step 2, unless we already did MAX_TRIES (20) iterations (then return error).
Whenever *qmemman* is asked to return X megabytes of memory to Xen free pool, the following algorithm (*qmemman\_algo.balloon*) is executed:
1. find all domains ("donors") whose actual memory is greater than its `prefmem`
2. calculate how much memory can be reclaimed by shrinking donors to their `prefmem`. If it is less than X, return error.
3. shrink donors, proportionally to their `prefmem`, so that X MB should become free
4. wait BALOON\_DELAY (0.1s)
5. if some domain have not given back any memory, remove it from the donors list, and go to step 2, unless we already did MAX\_TRIES (20) iterations (then return error).
Notes
-----
Conventional means of viewing the memory available to Qubes will give incorrect values for `dom0` since commands such as `free` will only show the memory allocated for `dom0`. Run the `xl info` command in `dom0` and read the `total_memory` field to see the total memory available to Qubes.
Conventional means of viewing the memory available to Qubes will give incorrect values for ``dom0`` since commands such as ``free`` will only show the memory allocated for ``dom0``. Run the ``xl info`` command in ``dom0`` and read the ``total_memory`` field to see the total memory available to Qubes.
.. |checkmark| image:: /attachment/doc/checkmark.png
.. |redx| image:: /attachment/doc/red_x.png

375
developer/services/qrexec-internals.rst
View file

@ -1,256 +1,349 @@
---
lang: en
layout: doc
permalink: /doc/qrexec-internals/
redirect_from:
- /doc/qrexec3-implementation/
- /en/doc/qrexec3-implementation/
- /doc/Qrexec3Implementation/
- /wiki/Qrexec3Implementation/
ref: 39
title: 'Qrexec: Qubes RPC internals'
---
===========================
Qrexec: Qubes RPC internals
===========================
(*This page details the current implementation of qrexec (qrexec3).
A [general introduction](/doc/qrexec/) to qrexec is also available.
For the implementation of qrexec2, see [here](/doc/qrexec2/#qubes-rpc-internals).*)
(*This page details the current implementation of qrexec (qrexec3). A* :doc:`general introduction </developer/services/qrexec>` *to qrexec is also available. For the implementation of qrexec2, see* :ref:`here <developer/services/qrexec2:qubes rpc internals>` *.*)
The qrexec framework consists of a number of processes communicating with each other using a common IPC protocol, described in detail below.
Components residing in the same domain (`qrexec-client-vm` to `qrexec-agent`, `qrexec-client` to `qrexec-daemon`) use local sockets as the underlying transport medium.
Components in separate domains (`qrexec-daemon` to `qrexec-agent`, data channel between `qrexec-agent`s) use vchan links.
Because of [vchan limitation](https://github.com/qubesos/qubes-issues/issues/951), it is not possible to establish qrexec connection back to the source domain.
Components residing in the same domain (``qrexec-client-vm`` to ``qrexec-agent``, ``qrexec-client`` to ``qrexec-daemon``) use local sockets as the underlying transport medium. Components in separate domains (``qrexec-daemon`` to ``qrexec-agent``, data channel between ``qrexec-agent(s)``) use vchan links. Because of `vchan limitation <https://github.com/qubesos/qubes-issues/issues/951>`__, it is not possible to establish qrexec connection back to the source domain.
## Dom0 tools implementation
Dom0 tools implementation
-------------------------
The following programs handle parts of the framework: sending and receiving requests, verifying permissions, and administering connections.
These tools are not designed to be used by users directly.
### qrexec-daemon
The following programs handle parts of the framework: sending and receiving requests, verifying permissions, and administering connections. These tools are not designed to be used by users directly.
`/usr/sbin/qrexec-daemon`
qrexec-daemon
^^^^^^^^^^^^^
``/usr/sbin/qrexec-daemon``
One instance is required for every active domain. ``qrexec-daemon`` is responsible for both:
- handling execution and service requests from **dom0** (source: ``qrexec-client``); and
- handling service requests from the associated domain (source: ``qrexec-client-vm``, then ``qrexec-agent``).
One instance is required for every active domain.
`qrexec-daemon` is responsible for both:
- handling execution and service requests from **dom0** (source: `qrexec-client`); and
- handling service requests from the associated domain (source: `qrexec-client-vm`, then `qrexec-agent`).
Command line usage:
`qrexec-daemon domain-id domain-name [default user]`
``qrexec-daemon domain-id domain-name [default user]``
- `domain-id`: Numeric Qubes ID assigned to the associated domain.
- `domain-name`: Associated domain name.
- `default user`: Optional. If passed, `qrexec-daemon` uses this user as default for all execution requests that don't specify one.
- ``domain-id``: Numeric Qubes ID assigned to the associated domain.
### qrexec-client
- ``domain-name``: Associated domain name.
`/usr/bin/qrexec-client`
- ``default user``: Optional. If passed, ``qrexec-daemon`` uses this user as default for all execution requests that don’t specify one.
Used to pass execution and service requests to `qrexec-daemon`.
qrexec-client
^^^^^^^^^^^^^
``/usr/bin/qrexec-client``
Used to pass execution and service requests to ``qrexec-daemon``.
Command line usage:
- `-d target-domain-name`: Specifies the target for the execution/service request.
- `-l local-program`: Optional. If present, `local-program` is executed and its stdout/stdin are used when sending/receiving data to/from the remote peer.
- `-e`: Optional. If present, stdout/stdin are not connected to the remote peer. Only process creation status code is received.
- `-c <request-id,src-domain-name,src-domain-id>`: used for connecting a VM-VM service request by `qrexec-policy`. Details described below in the service example.
- `cmdline`: Command line to pass to `qrexec-daemon` as the execution/service request. Service request format is described below in the service example.
- ``-d target-domain-name``: Specifies the target for the execution/service request.
## VM tools implementation
- ``-l local-program``: Optional. If present, ``local-program`` is executed and its stdout/stdin are used when sending/receiving data to/from the remote peer.
### qrexec-agent
- ``-e``: Optional. If present, stdout/stdin are not connected to the remote peer. Only process creation status code is received.
`/usr/lib/qubes/qrexec-agent`
- ``-c <request-id,src-domain-name,src-domain-id>``: used for connecting a VM-VM service request by ``qrexec-policy``. Details described below in the service example.
One instance runs in each active domain.
Responsible for:
- ``cmdline``: Command line to pass to ``qrexec-daemon`` as the execution/service request. Service request format is described below in the service example.
- Handling service requests from `qrexec-client-vm` and passing them to connected `qrexec-daemon` in dom0.
- Executing associated `qrexec-daemon` execution/service requests.
The `qrexec-agent` command takes no parameters.
### qrexec-client-vm
VM tools implementation
-----------------------
`/usr/bin/qrexec-client-vm`
Runs in an active domain.
Used to pass service requests to `qrexec-agent`.
qrexec-agent
^^^^^^^^^^^^
``/usr/lib/qubes/qrexec-agent``
One instance runs in each active domain. Responsible for:
- Handling service requests from ``qrexec-client-vm`` and passing them to connected ``qrexec-daemon`` in dom0.
- Executing associated ``qrexec-daemon`` execution/service requests.
The ``qrexec-agent`` command takes no parameters.
qrexec-client-vm
^^^^^^^^^^^^^^^^
``/usr/bin/qrexec-client-vm``
Runs in an active domain. Used to pass service requests to ``qrexec-agent``.
Command line usage:
`qrexec-client-vm target-domain-name service-name local-program [local program arguments]`
``qrexec-client-vm target-domain-name service-name local-program [local program arguments]``
- `target-domain-name`: Target domain for the service request. Source is the current domain.
- `service-name`: Requested service name.
- `local-program`: `local-program` is executed locally and its stdin/stdout are connected to the remote service endpoint.
- ``target-domain-name``: Target domain for the service request. Source is the current domain.
## Qrexec protocol details
- ``service-name``: Requested service name.
The qrexec protocol is message-based.
All messages share a common header followed by an optional data packet.
- ``local-program``: ``local-program`` is executed locally and its stdin/stdout are connected to the remote service endpoint.
```c
/* uniform for all peers, data type depends on message type */
struct msg_header {
uint32_t type; /* message type */
uint32_t len; /* data length */
};
```
When two peers establish connection, the server sends `MSG_HELLO` followed by `peer_info` struct:
```c
struct peer_info {
uint32_t version; /* qrexec protocol version */
};
```
Qrexec protocol details
-----------------------
The client then should reply with its own `MSG_HELLO` and `peer_info`.
The lower of two versions define protocol used for this connection.
If either side does not support this version, the connection is closed.
The qrexec protocol is message-based. All messages share a common header followed by an optional data packet.
.. code:: c
/* uniform for all peers, data type depends on message type */
struct msg_header {
uint32_t type; /* message type */
uint32_t len; /* data length */
};
When two peers establish connection, the server sends ``MSG_HELLO`` followed by ``peer_info`` struct:
.. code:: c
struct peer_info {
uint32_t version; /* qrexec protocol version */
};
The client then should reply with its own ``MSG_HELLO`` and ``peer_info``. The lower of two versions define protocol used for this connection. If either side does not support this version, the connection is closed.
Details of all possible use cases and the messages involved are described below.
### dom0: request execution of `cmd` in domX
dom0: request execution of ``cmd`` in domX
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
![qrexec internals diagram dom0-vm](/attachment/doc/qrexec-dom0-vm.png)
- **dom0**: `qrexec-client` is invoked in **dom0** as follows:
.. figure:: /attachment/doc/qrexec-dom0-vm.png
:alt: qrexec internals diagram dom0-vm
qrexec-client -d domX [-l local_program] user:cmd
- (If `local_program` is set, `qrexec-client` executes it and uses that child's stdin/stdout in place of its own when exchanging data with `qrexec-agent` later.)
- `qrexec-client` translates that request into a `MSG_EXEC_CMDLINE` message sent to `qrexec-daemon`, with `connect_domain` set to 0 (connect to **dom0**) and `connect_port` also set to 0 (allocate a port).
- **dom0**: ``qrexec-client`` is invoked in **dom0** as follows:
- **dom0**: `qrexec-daemon` allocates a free port (in this case 513), and sends a `MSG_EXEC_CMDLINE` back to the client with connection parameters (**domX** and 513) and with command field empty.
.. code:: bash
- `qrexec-client` disconnects from the daemon, starts a vchan server on port 513 and awaits connection.
qrexec-client -d domX [-l local_program] user:cmd
- Then, `qrexec-daemon` passes on the request as `MSG_EXEC_CMDLINE` message to the `qrexec-agent` running in **domX**. In this case, the connection parameters are **dom0** and 513.
- **domX**: `qrexec-agent` receives `MSG_EXEC_CMDLINE`, and starts the command (`user:cmd`, or `cmd` as user `user`). If possible, this is actually delegated to a separate server (`qrexec-fork-server`) also running on domX.
- After starting the command, `qrexec-fork-server` connects to `qrexec-client` in **dom0** over the provided vchan port 513.
- (If ``local_program`` is set, ``qrexec-client`` executes it and uses that child’s stdin/stdout in place of its own when exchanging data with ``qrexec-agent`` later.)
- Data is forwarded between the `qrexec-client` in **dom0** and the command executed in **domX** using `MSG_DATA_STDIN`, `MSG_DATA_STDOUT` and `MSG_DATA_STDERR`.
- ``qrexec-client`` translates that request into a ``MSG_EXEC_CMDLINE`` message sent to ``qrexec-daemon``, with ``connect_domain`` set to 0 (connect to **dom0**) and ``connect_port`` also set to 0 (allocate a port).
- Empty messages (with data `len` field set to 0 in `msg_header`) are an EOF marker. Peer receiving such message should close the associated input/output pipe.
- When `cmd` terminates, **domX**'s `qrexec-fork-server` sends `MSG_DATA_EXIT_CODE` header to `qrexec-client` followed by the exit code (**int**).
### domX: request execution of service `admin.Service` in dom0
- **dom0**: ``qrexec-daemon`` allocates a free port (in this case 513), and sends a ``MSG_EXEC_CMDLINE`` back to the client with connection parameters (**domX** and 513) and with command field empty.
![qrexec internals diagram vm-dom0](/attachment/doc/qrexec-vm-dom0.png)
- ``qrexec-client`` disconnects from the daemon, starts a vchan server on port 513 and awaits connection.
- **domX**: `qrexec-client-vm` is invoked as follows:
- Then, ``qrexec-daemon`` passes on the request as ``MSG_EXEC_CMDLINE`` message to the ``qrexec-agent`` running in **domX**. In this case, the connection parameters are **dom0** and 513.
qrexec-client-vm dom0 admin.Service [local_program] [params]
- (If `local_program` is set, it will be executed in **domX** and connected to the remote command's stdin/stdout).
- `qrexec-client-vm` connects to `qrexec-agent` and requests service execution (`admin.Service`) in **dom0**.
- **domX**: ``qrexec-agent`` receives ``MSG_EXEC_CMDLINE``, and starts the command (``user:cmd``, or ``cmd`` as user ``user``). If possible, this is actually delegated to a separate server (``qrexec-fork-server``) also running on domX.
- `qrexec-agent` assigns an internal identifier to the request. It's based on a file descriptor of the connected `qrexec-client-vm`: in this case, `SOCKET11`.
- After starting the command, ``qrexec-fork-server`` connects to ``qrexec-client`` in **dom0** over the provided vchan port 513.
- `qrexec-agent` forwards the request (`MSG_TRIGGER_SERVICE3`) to its corresponding `qrexec-daemon` running in dom0.
- **dom0**: `qrexec-daemon` receives the request and triggers `qrexec-policy` program, passing all necessary parameters: source domain **domX**, target domain **dom0**, service `admin.Service` and identifier `SOCKET11`.
- `qrexec-policy` evaluates if the RPC should be allowed or denied, possibly also launching a GUI confirmation prompt.
- Data is forwarded between the ``qrexec-client`` in **dom0** and the command executed in **domX** using ``MSG_DATA_STDIN``, ``MSG_DATA_STDOUT`` and ``MSG_DATA_STDERR``.
- (If the RPC is denied, it returns with exit code 1, in which case `qrexec-daemon` sends a `MSG_SERVICE_REFUSED` back).
- Empty messages (with data ``len`` field set to 0 in ``msg_header``) are an EOF marker. Peer receiving such message should close the associated input/output pipe.
- **dom0**: If the RPC is allowed, `qrexec-policy` will launch a `qrexec-client` with the right command:
- When ``cmd`` terminates, **domX**’s ``qrexec-fork-server`` sends ``MSG_DATA_EXIT_CODE`` header to ``qrexec-client`` followed by the exit code (**int**).
qrexec-client -d dom0 -c domX,X,SOCKET11 "QUBESRPC admin.Service domX name dom0"
- The `-c domX,X,SOCKET11` are parameters indicating how connect back to **domX** and pass its input/output.
- The command parameter describes the RPC call: it contains service name (`admin.Service`), source domain (`domX`) and target description (`name dom0`, could also be e.g. `keyword @dispvm`). The target description is important in case the original target wasn't dom0, but the service is executing in dom0.
- `qrexec-client` connects to a `qrexec-daemon` for **domX** and sends a `MSG_SERVICE_CONNECT` with connection parameters (**dom0**, and port 0, indicating a port should be allocated) and request identifier (`SOCKET11`).
- `qrexec-daemon` allocates a free port (513) and sends back connection parameters to `qrexec-client` (**domX** port 513).
domX: request execution of service ``admin.Service`` in dom0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
- `qrexec-client` starts the command, and tries to connect to **domX** over the provided port 513.
- Then, `qrexec-daemon` forwards the connection request (`MSG_SERVICE_CONNECT`) to `qrexec-agent` running in **domX**, with the right parameters (**dom0** port 513, request `SOCKET11`).
.. figure:: /attachment/doc/qrexec-vm-dom0.png
:alt: qrexec internals diagram vm-dom0
- **dom0**: Because the command has the form `QUBESRPC: ...`, it is started through the `qubes-rpc-multiplexer` program with the provided parameters (`admin.Service domX name dom0`). That program finds and executes the necessary script in `/etc/qubes-rpc/`.
- **domX**: `qrexec-agent` receives the `MSG_SERVICE_CONNECT` and passes the connection parameters back to the connected `qrexec-client-vm`. It identifies the `qrexec-client-vm` by the request identifier (`SOCKET11` means file descriptor 11).
- `qrexec-client-vm` starts a vchan server on 513 and receives a connection from `qrexec-client`.
- **domX**: ``qrexec-client-vm`` is invoked as follows:
.. code:: bash
qrexec-client-vm dom0 admin.Service [local_program] [params]
- (If ``local_program`` is set, it will be executed in **domX** and connected to the remote command’s stdin/stdout).
- ``qrexec-client-vm`` connects to ``qrexec-agent`` and requests service execution (``admin.Service``) in **dom0**.
- ``qrexec-agent`` assigns an internal identifier to the request. It’s based on a file descriptor of the connected ``qrexec-client-vm``: in this case, ``SOCKET11``.
- ``qrexec-agent`` forwards the request (``MSG_TRIGGER_SERVICE3``) to its corresponding ``qrexec-daemon`` running in dom0.
- **dom0**: ``qrexec-daemon`` receives the request and triggers ``qrexec-policy`` program, passing all necessary parameters: source domain **domX**, target domain **dom0**, service ``admin.Service`` and identifier ``SOCKET11``.
- ``qrexec-policy`` evaluates if the RPC should be allowed or denied, possibly also launching a GUI confirmation prompt.
- (If the RPC is denied, it returns with exit code 1, in which case ``qrexec-daemon`` sends a ``MSG_SERVICE_REFUSED`` back).
- **dom0**: If the RPC is allowed, ``qrexec-policy`` will launch a ``qrexec-client`` with the right command:
.. code:: bash
qrexec-client -d dom0 -c domX,X,SOCKET11 "QUBESRPC admin.Service domX name dom0"
- The ``-c domX,X,SOCKET11`` are parameters indicating how connect back to **domX** and pass its input/output.
- The command parameter describes the RPC call: it contains service name (``admin.Service``), source domain (``domX``) and target description (``name dom0``, could also be e.g. ``keyword @dispvm``). The target description is important in case the original target wasn’t dom0, but the service is executing in dom0.
- ``qrexec-client`` connects to a ``qrexec-daemon`` for **domX** and sends a ``MSG_SERVICE_CONNECT`` with connection parameters (**dom0**, and port 0, indicating a port should be allocated) and request identifier (``SOCKET11``).
- ``qrexec-daemon`` allocates a free port (513) and sends back connection parameters to ``qrexec-client`` (**domX** port 513).
- ``qrexec-client`` starts the command, and tries to connect to **domX** over the provided port 513.
- Then, ``qrexec-daemon`` forwards the connection request (``MSG_SERVICE_CONNECT``) to ``qrexec-agent`` running in **domX**, with the right parameters (**dom0** port 513, request ``SOCKET11``).
- **dom0**: Because the command has the form ``QUBESRPC: ...``, it is started through the ``qubes-rpc-multiplexer`` program with the provided parameters (``admin.Service domX name dom0``). That program finds and executes the necessary script in ``/etc/qubes-rpc/``.
- **domX**: ``qrexec-agent`` receives the ``MSG_SERVICE_CONNECT`` and passes the connection parameters back to the connected ``qrexec-client-vm``. It identifies the ``qrexec-client-vm`` by the request identifier (``SOCKET11`` means file descriptor 11).
- ``qrexec-client-vm`` starts a vchan server on 513 and receives a connection from ``qrexec-client``.
- Data is forwarded between **dom0** and **domX** as in the previous example (dom0-VM).
### domX: invoke execution of qubes service `qubes.Service` in domY
![qrexec internals diagram vm-vm](/attachment/doc/qrexec-vm-vm.png)
- **domX**: `qrexec-client-vm` is invoked as follows:
domX: invoke execution of qubes service ``qubes.Service`` in domY
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
qrexec-client-vm domY qubes.Service [local_program] [params]
- (If `local_program` is set, it will be executed in **domX** and connected to the remote command's stdin/stdout).
.. figure:: /attachment/doc/qrexec-vm-vm.png
:alt: qrexec internals diagram vm-vm
- The request is forwarded as `MSG_TRIGGER_SERVICE3` to `qrexec-daemon` running in **dom0**, then to `qrexec-policy`, then (if allowed) to `qrexec-client`.
- **domX**: ``qrexec-client-vm`` is invoked as follows:
.. code:: bash
qrexec-client-vm domY qubes.Service [local_program] [params]
- (If ``local_program`` is set, it will be executed in **domX** and connected to the remote command’s stdin/stdout).
- The request is forwarded as ``MSG_TRIGGER_SERVICE3`` to ``qrexec-daemon`` running in **dom0**, then to ``qrexec-policy``, then (if allowed) to ``qrexec-client``.
- This is the same as in the previous example (VM-dom0).
- **dom0**: If the RPC is allowed, `qrexec-policy` will launch a `qrexec-client` with the right command:
qrexec-client -d domY -c domX,X,SOCKET11 user:cmd "DEFAULT:QUBESRPC qubes.Service domX"
- The `-c domX,X,SOCKET11` are parameters indicating how connect back to **domX** and pass its input/output.
- **dom0**: If the RPC is allowed, ``qrexec-policy`` will launch a ``qrexec-client`` with the right command:
- The command parameter describes the service call: it contains the username (or `DEFAULT`), service name (`qubes.Service`) and source domain (`domX`).
.. code:: bash
- `qrexec-client` will then send a `MSG_EXEC_CMDLINE` message to `qrexec-daemon` for **domY**. The message will be with port number 0, requesting port allocation.
qrexec-client -d domY -c domX,X,SOCKET11 user:cmd "DEFAULT:QUBESRPC qubes.Service domX"
- `qrexec-daemon` for **domY** will allocate a port (513) and send it back. It will also send a `MSG_EXEC_CMDLINE` to its corresponding agent. (It will also translate `DEFAULT` to the configured default username).
- Then, `qrexec-client` will also send `MSG_SERVICE_CONNECT` message to **domX**'s agent, indicating that it should connect to **domY** over port 513.
- Having notified both domains about a connection, `qrexec-client` now exits.
- The ``-c domX,X,SOCKET11`` are parameters indicating how connect back to **domX** and pass its input/output.
- **domX**: `qrexec-agent` receives a `MSG_SERVICE_CONNECT` with connection parameters (**domY** port 513) and request identifier (`SOCKET11`). It sends the connection parameters back to the right `qrexec-client-vm`.
- The command parameter describes the service call: it contains the username (or ``DEFAULT``), service name (``qubes.Service``) and source domain (``domX``).
- `qrexec-client-vm` starts a vchan server on port 513. note that this is different than in the other examples: `MSG_SERVICE_CONNECT` means you should start a server, `MSG_EXEC_CMDLINE` means you should start a client.
- ``qrexec-client`` will then send a ``MSG_EXEC_CMDLINE`` message to ``qrexec-daemon`` for **domY**. The message will be with port number 0, requesting port allocation.
- **domY**: `qrexec-agent` receives a `MSG_EXEC_CMDLINE` with the command to execute (`user:QUBESRPC...`) and connection parameters (**domX** port 513).
- ``qrexec-daemon`` for **domY** will allocate a port (513) and send it back. It will also send a ``MSG_EXEC_CMDLINE`` to its corresponding agent. (It will also translate ``DEFAULT`` to the configured default username).
- Then, ``qrexec-client`` will also send ``MSG_SERVICE_CONNECT`` message to **domX**’s agent, indicating that it should connect to **domY** over port 513.
- Having notified both domains about a connection, ``qrexec-client`` now exits.
- **domX**: ``qrexec-agent`` receives a ``MSG_SERVICE_CONNECT`` with connection parameters (**domY** port 513) and request identifier (``SOCKET11``). It sends the connection parameters back to the right ``qrexec-client-vm``.
- ``qrexec-client-vm`` starts a vchan server on port 513. note that this is different than in the other examples: ``MSG_SERVICE_CONNECT`` means you should start a server, ``MSG_EXEC_CMDLINE`` means you should start a client.
- **domY**: ``qrexec-agent`` receives a ``MSG_EXEC_CMDLINE`` with the command to execute (``user:QUBESRPC...``) and connection parameters (**domX** port 513).
- It forwards the request to ``qrexec-fork-server``, which handles the command and connects to **domX** over the provided port.
- Because the command is of the form ``QUBESRPC ...``, ``qrexec-fork-server`` starts it using ``qubes-rpc-multiplexer`` program, which finds and executes the necessary script in ``/etc/qubes-rpc/``.
- It forwards the request to `qrexec-fork-server`, which handles the command and connects to **domX** over the provided port.
- Because the command is of the form `QUBESRPC ...`, `qrexec-fork-server` starts it using `qubes-rpc-multiplexer` program, which finds and executes the necessary script in `/etc/qubes-rpc/`.
- After that, the data is passed between **domX** and **domY** as in the previous examples (dom0-VM, VM-dom0).
## `qrexec-policy` implementation
`qrexec-policy` is a mechanism for evaluating whether an RPC call should be allowed. For introduction, see [Qubes RPC administration](/doc/qrexec/#qubes-rpc-administration).
### `qrexec-policy-daemon`
``qrexec-policy`` implementation
--------------------------------
This is a service running in dom0. It is called by `qrexec-daemon` and is responsible for evaluating the request and possibly launching an action.
The daemon listens on a socket (`/var/run/qubes/policy.sock`). It accepts requests in the format described in [qrexec-policy-daemon.rst](https://github.com/QubesOS/qubes-core-qrexec/blob/master/doc/qrexec-policy-daemon.rst) and replies with `result=allow/deny`.
``qrexec-policy`` is a mechanism for evaluating whether an RPC call should be allowed. For introduction, see :ref:`Qubes RPC administration <developer/services/qrexec:qubes rpc administration>`.
A standalone version is called `qrexec-policy-exec` and is available as a fallback.
``qrexec-policy-daemon``
^^^^^^^^^^^^^^^^^^^^^^^^
### `qrexec-policy-agent`
This is a service running in the GuiVM. It is called by `qrexec-policy-daemon` in order to display prompts and notifications to the user.
This is a service running in dom0. It is called by ``qrexec-daemon`` and is responsible for evaluating the request and possibly launching an action.
It is a [socket-based Qubes RPC service](/doc/qrexec-socket-services/). Requests are in JSON format, and response is simple ASCII.
The daemon listens on a socket (``/var/run/qubes/policy.sock``). It accepts requests in the format described in `qrexec-policy-daemon.rst <https://github.com/QubesOS/qubes-core-qrexec/blob/master/doc/qrexec-policy-daemon.rst>`__ and replies with ``result=allow/deny``.
A standalone version is called ``qrexec-policy-exec`` and is available as a fallback.
``qrexec-policy-agent``
^^^^^^^^^^^^^^^^^^^^^^^
This is a service running in the GuiVM. It is called by ``qrexec-policy-daemon`` in order to display prompts and notifications to the user.
It is a :doc:`socket-based Qubes RPC service </developer/services/qrexec-socket-services>`. Requests are in JSON format, and response is simple ASCII.
There are two endpoints:
- `policy.Ask` - ask the user about whether to execute a given action
- `policy.Notify` - notify the user about an action.
- ``policy.Ask`` - ask the user about whether to execute a given action
See [qrexec-policy-agent.md](https://github.com/QubesOS/qubes-core-qrexec/blob/master/doc/qrexec-policy-agent.md) for protocol details.
- ``policy.Notify`` - notify the user about an action.
See `qrexec-policy-agent.md <https://github.com/QubesOS/qubes-core-qrexec/blob/master/doc/qrexec-policy-agent.md>`__ for protocol details.

374
developer/services/qrexec-socket-services.rst
View file

@ -1,235 +1,269 @@
---
lang: en
layout: doc
permalink: /doc/qrexec-socket-services/
ref: 42
title: 'Qrexec: socket-based services'
---
=============================
Qrexec: socket-based services
=============================
*This page describes how to implement and use new socket-backed services for qrexec. See [qrexec](/doc/qrexec/) for general overview of the qrexec framework.*
As of Qubes 4.1, qrexec allows implementing services not only as executable files, but also as Unix sockets.
This allows Qubes RPC requests to be handled by a server running in a VM and listening for connections.
*This page describes how to implement and use new socket-backed services for qrexec. See* :doc:`qrexec </developer/services/qrexec>` *for general overview of the qrexec framework.*
## How it works
As of Qubes 4.1, qrexec allows implementing services not only as executable files, but also as Unix sockets. This allows Qubes RPC requests to be handled by a server running in a VM and listening for connections.
When a Qubes RPC service is invoked,
qrexec searches for a file that handles it in the qubes-rpc directories (`/etc/qubes-rpc` or `/usr/local/etc/qubes-rpc`).
If the file is a Unix socket, qrexec will try to connect to it.
How it works
------------
When a Qubes RPC service is invoked, qrexec searches for a file that handles it in the qubes-rpc directories (``/etc/qubes-rpc`` or ``/usr/local/etc/qubes-rpc``). If the file is a Unix socket, qrexec will try to connect to it.
Before passing user input, the socket service will receive a null-terminated service descriptor, i.e. the part after ``QUBESRPC``. When running in a VM, this is:
.. code:: bash
<service_name> <source>\0
Before passing user input, the socket service will receive a null-terminated service descriptor, i.e. the part after `QUBESRPC`.
When running in a VM, this is:
```
<service_name> <source>\0
```
When running in dom0, it is:
```
<service_name> <source> <target_type> <target>\0
```
.. code:: bash
(The target type can be `name`, in which case target is a domain name, or `keyword`, in which the target is a keyword like `@dispvm`).
<service_name> <source> <target_type> <target>\0
Afterwards, data provided by the service's user (as stdin) is sent into the socket, and data received from the socket is sent back to the user (as stdout).
When the service closes the socket, an exit code of 0 is sent back to the user.
### Differences from executable-based services
From the user point of view, the socket-based service behaves almost like an executable-based one.
Here are the differences:
(The target type can be ``name``, in which case target is a domain name, or ``keyword``, in which the target is a keyword like ``@dispvm``).
* There is no stderr (the socket provides only one output stream).
Currently, that means stderr will also never be closed on user's end.
* There is no exit code.
When the socket connection is closed, exit code 0 is sent to the user.
Afterwards, data provided by the service’s user (as stdin) is sent into the socket, and data received from the socket is sent back to the user (as stdout). When the service closes the socket, an exit code of 0 is sent back to the user.
## Recommended use
Differences from executable-based services
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Create a program that binds to path *outside* `/etc/qubes-rpc`, such as `/var/run/my-daemon.sock`.
Put a symlink in `/etc/qubes-rpc`, e.g. `ln -s /var/run/my-daemon.sock /etc/qubes-rpc/qubes.Service`.
If your program handles multiple services, create multiple symlinks.
You can dispatch based on the service descriptor.
From the user point of view, the socket-based service behaves almost like an executable-based one. Here are the differences:
- There is no stderr (the socket provides only one output stream). Currently, that means stderr will also never be closed on user’s end.
- There is no exit code. When the socket connection is closed, exit code 0 is sent to the user.
Recommended use
---------------
Create a program that binds to path *outside* ``/etc/qubes-rpc``, such as ``/var/run/my-daemon.sock``. Put a symlink in ``/etc/qubes-rpc``, e.g. ``ln -s /var/run/my-daemon.sock /etc/qubes-rpc/qubes.Service``.
If your program handles multiple services, create multiple symlinks. You can dispatch based on the service descriptor.
Do not run the program as root.
You can use systemd and socket activation so that the program is started only when the service is invoked.
See the below example.
You can use systemd and socket activation so that the program is started only when the service is invoked. See the below example.
## Example: `qrexec-policy-agent`
Example: ``qrexec-policy-agent``
--------------------------------
`qrexec-policy-agent` is the program that handles "ask" prompts for Qubes RPC calls.
It is a good example of an application that:
* Uses Python and asyncio.
* Runs as a daemon, to save some overhead on starting process.
* Runs as a normal user.
This is achieved using user's instance of systemd.
* Uses systemd socket activation.
This way it can be installed in all VMs, but started only if it's ever needed.
``qrexec-policy-agent`` is the program that handles “ask” prompts for Qubes RPC calls. It is a good example of an application that:
See the [qubes-core-qrexec](https://github.com/QubesOS/qubes-core-qrexec/) repository for details.
- Uses Python and asyncio.
- Runs as a daemon, to save some overhead on starting process.
- Runs as a normal user. This is achieved using user’s instance of systemd.
- Uses systemd socket activation. This way it can be installed in all VMs, but started only if it’s ever needed.
See the `qubes-core-qrexec <https://github.com/QubesOS/qubes-core-qrexec/>`__ repository for details.
Systemd unit files
^^^^^^^^^^^^^^^^^^
### Systemd unit files
**/lib/systemd/user/qubes-qrexec-policy-agent.service**: This is the service configuration.
```
[Unit]
Description=Qubes remote exec policy agent
ConditionUser=!root
ConditionGroup=qubes
Requires=qubes-qrexec-policy-agent.socket
.. code:: bash
[Unit]
Description=Qubes remote exec policy agent
ConditionUser=!root
ConditionGroup=qubes
Requires=qubes-qrexec-policy-agent.socket
[Service]
Type=simple
ExecStart=/usr/bin/qrexec-policy-agent
[Install]
WantedBy=default.target
[Service]
Type=simple
ExecStart=/usr/bin/qrexec-policy-agent
[Install]
WantedBy=default.target
```
**/lib/systemd/user/qubes-qrexec-policy-agent.socket**: This is the socket file that will activate the service.
```
[Unit]
Description=Qubes remote exec policy agent socket
ConditionUser=!root
ConditionGroup=qubes
PartOf=qubes-qrexec-policy-agent.service
.. code:: bash
[Socket]
ListenStream=/var/run/qubes/policy-agent.sock
[Unit]
Description=Qubes remote exec policy agent socket
ConditionUser=!root
ConditionGroup=qubes
PartOf=qubes-qrexec-policy-agent.service
[Socket]
ListenStream=/var/run/qubes/policy-agent.sock
[Install]
WantedBy=sockets.target
[Install]
WantedBy=sockets.target
```
Note the `ConditionUser` and `ConditionGroup` that ensure that the socket and service is started only as the right user
Start the socket using `systemctl --user start`.
Enable it using `systemctl --user enable`, so that it starts automatically.
Note the ``ConditionUser`` and ``ConditionGroup`` that ensure that the socket and service is started only as the right user
Start the socket using ``systemctl --user start``. Enable it using ``systemctl --user enable``, so that it starts automatically.
.. code:: bash
systemctl --user start qubes-qrexec-policy-agent.socket
systemctl --user enable qubes-qrexec-policy-agent.socket
```
systemctl --user start qubes-qrexec-policy-agent.socket
systemctl --user enable qubes-qrexec-policy-agent.socket
```
Alternatively, you can enable the service by creating a symlink:
```
sudo ln -s /lib/systemd/user/qubes-qrexec-policy-agent.socket /lib/systemd/user/sockets.target.wants/
```
.. code:: bash
### Link in qubes-rpc
sudo ln -s /lib/systemd/user/qubes-qrexec-policy-agent.socket /lib/systemd/user/sockets.target.wants/
`qrexec-policy-agent` will handle a Qubes RPC service called `policy.Ask`, so we add a link:
```
sudo ln -s /var/run/qubes/policy-agent.sock /etc/qubes-rpc/policy.Ask
```
### Python server with socket activation
Link in qubes-rpc
^^^^^^^^^^^^^^^^^
Socket activation in systemd works by starting our program with the socket file already bound at a specific file descriptor.
It's a simple mechanism based on a few environment variables, but the canonical way is to use the `sd_listen_fds()` function from systemd library (or, in our case, its Python version).
``qrexec-policy-agent`` will handle a Qubes RPC service called ``policy.Ask``, so we add a link:
.. code:: bash
sudo ln -s /var/run/qubes/policy-agent.sock /etc/qubes-rpc/policy.Ask
Python server with socket activation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Socket activation in systemd works by starting our program with the socket file already bound at a specific file descriptor. It’s a simple mechanism based on a few environment variables, but the canonical way is to use the ``sd_listen_fds()`` function from systemd library (or, in our case, its Python version).
Install the Python systemd library:
```
sudo dnf install python3-systemd
```
.. code:: bash
sudo dnf install python3-systemd
Here is the server code:
```python
import os
import asyncio
import socket
.. code:: python
from systemd.daemon import listen_fds
import os
import asyncio
import socket
from systemd.daemon import listen_fds
class SocketService:
def __init__(self, socket_path, socket_activated=False):
self._socket_path = socket_path
self._socket_activated = socket_activated
async def run(self):
server = await self.start()
async with server:
await server.serve_forever()
async def start(self):
if self._socket_activated:
fds = listen_fds()
if fds:
assert len(fds) == 1, 'too many listen_fds: {}'.format(
listen_fds)
sock = socket.socket(fileno=fds[0])
return await asyncio.start_unix_server(self._client_connected,
sock=sock)
if os.path.exists(self._socket_path):
os.unlink(self._socket_path)
return await asyncio.start_unix_server(self._client_connected,
path=self._socket_path)
async def _client_connected(self, reader, writer):
try:
data = await reader.read()
assert b'\0' in data, data
service_descriptor, data = data.split(b'\0', 1)
response = await self.handle_request(service_descriptor, data)
writer.write(response)
await writer.drain()
finally:
writer.close()
await writer.wait_closed()
async def handle_request(self, service_descriptor, data):
# process params, return response
return response
def main():
socket_path = '/var/run/qubes/policy-agent.sock'
service = SocketService(socket_path)
loop = asyncio.get_event_loop()
loop.run_until_complete(service.run())
if __name__ == '__main__':
main()
class SocketService:
def __init__(self, socket_path, socket_activated=False):
self._socket_path = socket_path
self._socket_activated = socket_activated
You can also use ``qrexec/server.py`` from `qubes-core-qrexec <https://github.com/QubesOS/qubes-core-qrexec/>`__ repository, which is a variant of the above code - but note that currently it’s somewhat more specific (JSON requests and ASCII responses; no target handling in service descriptors).
async def run(self):
server = await self.start()
async with server:
await server.serve_forever()
Using the service
^^^^^^^^^^^^^^^^^
async def start(self):
if self._socket_activated:
fds = listen_fds()
if fds:
assert len(fds) == 1, 'too many listen_fds: {}'.format(
listen_fds)
sock = socket.socket(fileno=fds[0])
return await asyncio.start_unix_server(self._client_connected,
sock=sock)
if os.path.exists(self._socket_path):
os.unlink(self._socket_path)
return await asyncio.start_unix_server(self._client_connected,
path=self._socket_path)
async def _client_connected(self, reader, writer):
try:
data = await reader.read()
assert b'\0' in data, data
service_descriptor, data = data.split(b'\0', 1)
response = await self.handle_request(service_descriptor, data)
writer.write(response)
await writer.drain()
finally:
writer.close()
await writer.wait_closed()
async def handle_request(self, service_descriptor, data):
# process params, return response
return response
def main():
socket_path = '/var/run/qubes/policy-agent.sock'
service = SocketService(socket_path)
loop = asyncio.get_event_loop()
loop.run_until_complete(service.run())
if __name__ == '__main__':
main()
```
You can also use `qrexec/server.py` from [qubes-core-qrexec](https://github.com/QubesOS/qubes-core-qrexec/) repository, which is a variant of the above code - but note that currently it's somewhat more specific (JSON requests and ASCII responses; no target handling in service descriptors).
### Using the service
The service is invoked in the same way as a standard Qubes RPC service:
```
echo <input_data> | qrexec-client -d domX 'DEFAULT:QUBESRPC policy.Ask'
```
.. code:: bash
echo <input_data> | qrexec-client -d domX 'DEFAULT:QUBESRPC policy.Ask'
You can also connect to it locally, but remember to include the service descriptor:
```
echo -e 'policy.Ask dom0\0<input data>' | nc -U /etc/qubes-rpc/policy.Ask
```
.. code:: bash
echo -e 'policy.Ask dom0\0<input data>' | nc -U /etc/qubes-rpc/policy.Ask
Further reading
---------------
- :doc:`Qrexec overview </developer/services/qrexec>`
- :doc:`Qrexec internals </developer/services/qrexec-internals>`
- `qubes-core-qrexec <https://github.com/QubesOS/qubes-core-qrexec/>`__ repository - contains the above example
- `systemd.socket <https://www.freedesktop.org/software/systemd/man/systemd.socket.html>`__ - socket unit configuration
- `Streams in Python asyncio <https://docs.python.org/3/library/asyncio-stream.html>`__
## Further reading
* [Qrexec overview](/doc/qrexec/)
* [Qrexec internals](/doc/qrexec-internals/)
* [qubes-core-qrexec](https://github.com/QubesOS/qubes-core-qrexec/) repository - contains the above example
* [systemd.socket](https://www.freedesktop.org/software/systemd/man/systemd.socket.html) - socket unit configuration
* [Streams in Python asyncio](https://docs.python.org/3/library/asyncio-stream.html)

505
developer/services/qrexec.rst
View file

@ -1,361 +1,328 @@
---
lang: en
layout: doc
permalink: /doc/qrexec/
redirect_from:
- /en/doc/qrexec3/
- /doc/Qrexec3/
- /doc/qrexec3/
- /wiki/Qrexec3/
- /en/doc/qrexec/
- /doc/Qrexec/
- /wiki/Qrexec/
ref: 37
title: 'Qrexec: secure communication across domains'
---
===========================================
Qrexec: secure communication across domains
===========================================
(*This page is about qrexec v3. For qrexec v2, see [here](/doc/qrexec2/).*)
The **qrexec framework** is used by core Qubes components to implement communication between domains.
Qubes domains are strictly isolated by design.
However, the OS needs a mechanism to allow the administrative domain (dom0) to force command execution in another domain (VM).
For instance, when a user selects an application from the KDE menu, it should start in the selected VM.
Also, it is often useful to be able to pass stdin/stdout/stderr from an application running in a VM to dom0 (and the other way around).
(For example, so that a VM can notify dom0 that there are updates available for it).
By default, Qubes allows VMs to initiate such communications in specific circumstances.
The qrexec framework generalizes this process by providing a remote procedure call (RPC) for the Qubes architecture.
It allows users and developers to use and design secure inter-VM tools.
(*This page is about qrexec v3. For qrexec v2, see* :doc:`here </developer/services/qrexec2>` *.*)
## Qrexec basics: architecture and examples
The **qrexec framework** is used by core Qubes components to implement communication between domains. Qubes domains are strictly isolated by design. However, the OS needs a mechanism to allow the administrative domain (dom0) to force command execution in another domain (VM). For instance, when a user selects an application from the KDE menu, it should start in the selected VM. Also, it is often useful to be able to pass stdin/stdout/stderr from an application running in a VM to dom0 (and the other way around). (For example, so that a VM can notify dom0 that there are updates available for it). By default, Qubes allows VMs to initiate such communications in specific circumstances. The qrexec framework generalizes this process by providing a remote procedure call (RPC) for the Qubes architecture. It allows users and developers to use and design secure inter-VM tools.
Qrexec is built on top of *vchan*, a Xen library providing data links between VMs.
During domain startup, a process named `qrexec-daemon` is started in dom0, and a process named `qrexec-agent` is started in the VM.
They are connected over a **vchan** channel.
`qrexec-daemon` listens for connections from a dom0 utility named `qrexec-client`.
Let's say we want to start a process (call it `VMprocess`) in a VM (`someVM`).
Typically, the first thing that a `qrexec-client` instance does is to send a request to the `qrexec-daemon`, which in turn relays it to `qrexec-agent` running in `someVM`.
`qrexec-daemon` assigns unique vchan connection details and sends them to both `qrexec-client` (in dom0) and `qrexec-agent` (in `someVM`).
`qrexec-client` starts a vchan server, which `qrexec-agent` then connects to.
Once this channel is established, stdin/stdout/stderr from the VMprocess is passed between `qrexec-agent` and the `qrexec-client` process.
Qrexec basics: architecture and examples
----------------------------------------
![qrexec basics diagram](/attachment/doc/qrexec3-basics.png)
The `qrexec-client` command is used to make connections to VMs from dom0.
For example, the following command creates an empty file called `hello-world.txt` in the home folder of `someVM`:
Qrexec is built on top of *vchan*, a Xen library providing data links between VMs. During domain startup, a process named ``qrexec-daemon`` is started in dom0, and a process named ``qrexec-agent`` is started in the VM. They are connected over a **vchan** channel. ``qrexec-daemon`` listens for connections from a dom0 utility named ``qrexec-client``. Let’s say we want to start a process (call it ``VMprocess``) in a VM (``someVM``). Typically, the first thing that a ``qrexec-client`` instance does is to send a request to the ``qrexec-daemon``, which in turn relays it to ``qrexec-agent`` running in ``someVM``. ``qrexec-daemon`` assigns unique vchan connection details and sends them to both ``qrexec-client`` (in dom0) and ``qrexec-agent`` (in ``someVM``). ``qrexec-client`` starts a vchan server, which ``qrexec-agent`` then connects to. Once this channel is established, stdin/stdout/stderr from the VMprocess is passed between ``qrexec-agent`` and the ``qrexec-client`` process.
```
$ qrexec-client -e -d someVM user:'touch hello-world.txt'
```
.. figure:: /attachment/doc/qrexec3-basics.png
:alt: qrexec basics diagram
The string before the colon specifies which user will run the command.
The `-e` flag tells `qrexec-client` to exit immediately after sending the execution request and receiving a status code from `qrexec-agent` (if the process creation succeeded).
With this option, no further data is passed between the domains.
The following command demonstrates an open channel between dom0 and someVM (in this case, a remote shell):
qrexec basics diagram
```
$ qrexec-client -d someVM user:bash
```
The ``qrexec-client`` command is used to make connections to VMs from dom0. For example, the following command creates an empty file called ``hello-world.txt`` in the home folder of ``someVM``:
The `qvm-run` command is heavily based on `qrexec-client`.
It also handles additional activities, e.g. starting the domain if the domain is not up yet and starting the GUI daemon.
It is usually more convenient to use `qvm-run`.
.. code:: bash
There can be an almost arbitrary number of `qrexec-client` processes for a given domain.
The limiting factor is the number of available vchan channels, which depends on the underlying hypervisor, as well the domain's OS.
$ qrexec-client -e -d someVM user:'touch hello-world.txt'
For more details on the qrexec framework and protocol, see "[Qubes RPC internals](/doc/qrexec-internals)."
## Qubes RPC services
Some common tasks (like copying files between VMs) have an RPC-like structure: a process in one VM (say, the file sender) needs to invoke and send/receive data to some process in other VM (say, the file receiver).
The Qubes RPC framework was created to securely facilitate a range of such actions.
The string before the colon specifies which user will run the command. The ``-e`` flag tells ``qrexec-client`` to exit immediately after sending the execution request and receiving a status code from ``qrexec-agent`` (if the process creation succeeded). With this option, no further data is passed between the domains. The following command demonstrates an open channel between dom0 and someVM (in this case, a remote shell):
Inter-VM communication must be tightly controlled to prevent one VM from taking control of another, possibly more privileged, VM.
The design decision was made to pass all control communication via dom0 which can enforce proper authorization.
It is therefore natural to reuse the already-existing qrexec framework.
.. code:: bash
Note that bare qrexec provides `VM <-> dom0` connectivity, but the command execution is always initiated by dom0.
There are cases when a VM needs to invoke and send data to a command in dom0 (e.g. to pass information on newly installed `.desktop` files).
This framework allows dom0 to be the RPC target as well.
$ qrexec-client -d someVM user:bash
Thanks to the framework, RPC programs are very simple -- both RPC client and server just use their stdin/stdout to pass data.
The framework does all the inner work to connect the processes to eachother via `qrexec-daemon` and `qrexec-agent`.
Disposable VMs are tightly integrated -- RPC to a DisposableVM is identical to RPC to an AppVM or StandaloneVM: all one needs is to pass `@dispvm` as the remote domain name.
## Qubes RPC administration
### Policy files
The ``qvm-run`` command is heavily based on ``qrexec-client``. It also handles additional activities, e.g. starting the domain if the domain is not up yet and starting the GUI daemon. It is usually more convenient to use ``qvm-run``.
There can be an almost arbitrary number of ``qrexec-client`` processes for a given domain. The limiting factor is the number of available vchan channels, which depends on the underlying hypervisor, as well the domain’s OS.
For more details on the qrexec framework and protocol, see “:doc:`Qubes RPC internals </developer/services/qrexec-internals>`.”
Qubes RPC services
------------------
Some common tasks (like copying files between VMs) have an RPC-like structure: a process in one VM (say, the file sender) needs to invoke and send/receive data to some process in other VM (say, the file receiver). The Qubes RPC framework was created to securely facilitate a range of such actions.
Inter-VM communication must be tightly controlled to prevent one VM from taking control of another, possibly more privileged, VM. The design decision was made to pass all control communication via dom0 which can enforce proper authorization. It is therefore natural to reuse the already-existing qrexec framework.
Note that bare qrexec provides ``VM <-> dom0`` connectivity, but the command execution is always initiated by dom0. There are cases when a VM needs to invoke and send data to a command in dom0 (e.g. to pass information on newly installed ``.desktop`` files). This framework allows dom0 to be the RPC target as well.
Thanks to the framework, RPC programs are very simple – both RPC client and server just use their stdin/stdout to pass data. The framework does all the inner work to connect the processes to eachother via ``qrexec-daemon`` and ``qrexec-agent``. Disposable VMs are tightly integrated – RPC to a DisposableVM is identical to RPC to an AppVM or StandaloneVM: all one needs is to pass ``@dispvm`` as the remote domain name.
Qubes RPC administration
------------------------
Policy files
^^^^^^^^^^^^
| The dom0 directories ``/etc/qubes/policy.d/`` and ``/run/qubes/policy.d/`` contain files that set policy for each available RPC action that a VM might call. For example, ``/etc/qubes/policy.d/90-default.policy`` contains the default policy settings.
| When making changes to existing policies it is recommended that you create a *new* policy file starting with a lower number, like ``/etc/qubes/policy.d/30-user.policy``.
| You may keep your custom policies in one file like ``/etc/qubes/policy.d/30-user.policy``, or you may choose to have multiple files, like ``/etc/qubes/policy.d/10-copy.policy``, ``/etc/qubes/policy.d/10-open.policy``.
| Together the contents of these files make up the RPC access policy database: the files are merged, with policies in lower number files overriding policies in higher numbered files. If there are entries in both ``/run/qubes/policy.d/`` and ``/etc/qubes/policy.d/`` with the same name, it isn’t specified which takes precedence, so you should avoid this situation.
The dom0 directories `/etc/qubes/policy.d/` and `/run/qubes/policy.d/` contain files that set policy for each available RPC action that a VM might call.
For example, `/etc/qubes/policy.d/90-default.policy` contains the default policy settings.
When making changes to existing policies it is recommended that you create a *new* policy file starting with a lower number, like `/etc/qubes/policy.d/30-user.policy`.
You may keep your custom policies in one file like `/etc/qubes/policy.d/30-user.policy`, or you may choose to have multiple files, like `/etc/qubes/policy.d/10-copy.policy`, `/etc/qubes/policy.d/10-open.policy`.
Together the contents of these files make up the RPC access policy database: the files are merged, with policies in lower number files overriding policies in higher numbered files.
If there are entries in both `/run/qubes/policy.d/` and `/etc/qubes/policy.d/` with the same name, it isn't specified which takes precedence, so you should avoid this situation.
Policies are defined in lines with the following format:
```
service-name|* +argument|* source destination action [options]
```
.. code:: bash
You can specify the source and destination by name or by one of the reserved keywords such as `*`, `@dispvm`, or `dom0`.
(Of these three, only `*` keyword makes sense in the source field.
Service calls from dom0 are currently always allowed, and `@dispvm` means "new VM created for this particular request," so it is never a source of request.)
Other methods using *tags* and *types* are also available (and discussed below).
service-name|* +argument|* source destination action [options]
Whenever a RPC request for an action is received, the domain checks the first matching line of the files in `/etc/qubes/policy.d/` and `/run/qubes/policy.d/` to determine access:
whether to allow the request, what VM to redirect the execution to, and what user account the program should run under.
Note that if the request is redirected (`target=` parameter), policy action remains the same -- even if there is another rule which would otherwise deny such request.
If no policy rule is matched, the action is denied.
Files in `/run/qubes/policy.d/` are deleted when the system is rebooted.
This is useful for temporary policy that contains the name or UUID of a disposable VM, which will not be meaningful after the system has rebooted.
Such policy files can be created manually, but they are usually created automatically by a Qrexec call to dom0.
### Making an RPC call
You can specify the source and destination by name or by one of the reserved keywords such as ``*``, ``@dispvm``, or ``dom0``. (Of these three, only ``*`` keyword makes sense in the source field. Service calls from dom0 are currently always allowed, and ``@dispvm`` means “new VM created for this particular request,” so it is never a source of request.) Other methods using *tags* and *types* are also available (and discussed below).
Whenever a RPC request for an action is received, the domain checks the first matching line of the files in ``/etc/qubes/policy.d/`` and ``/run/qubes/policy.d/`` to determine access: whether to allow the request, what VM to redirect the execution to, and what user account the program should run under. Note that if the request is redirected (``target=`` parameter), policy action remains the same – even if there is another rule which would otherwise deny such request. If no policy rule is matched, the action is denied.
Files in ``/run/qubes/policy.d/`` are deleted when the system is rebooted. This is useful for temporary policy that contains the name or UUID of a disposable VM, which will not be meaningful after the system has rebooted. Such policy files can be created manually, but they are usually created automatically by a Qrexec call to dom0.
Making an RPC call
^^^^^^^^^^^^^^^^^^
From outside of dom0, RPC calls take the following form:
```
$ qrexec-client-vm target_vm_name RPC_ACTION_NAME rpc_client_path client arguments
```
.. code:: bash
$ qrexec-client-vm target_vm_name RPC_ACTION_NAME rpc_client_path client arguments
For example:
```
$ qrexec-client-vm work qubes.StartApp+firefox
```
.. code:: bash
Note that only stdin/stdout is passed between RPC server and client -- notably, no command line arguments are passed.
By default, stderr of client and server is logged in the syslog/journald of the VM where the process is running.
$ qrexec-client-vm work qubes.StartApp+firefox
It is also possible to call service without specific client program -- in which case server stdin/out will be connected with the terminal:
```
$ qrexec-client-vm target_vm_name RPC_ACTION_NAME
```
### Answering an RPC call
Note that only stdin/stdout is passed between RPC server and client – notably, no command line arguments are passed. By default, stderr of client and server is logged in the syslog/journald of the VM where the process is running.
In other for a RPC call to be answered in the target VM, a file in either of the following locations must exist, containing the file name of the program that will be invoked, or being that program itself -- in which case it must have executable permission set (`chmod +x`):
- `/etc/qubes-rpc/RPC_ACTION_NAME` when you make it in the template qube;
- `/usr/local/etc/qubes-rpc/RPC_ACTION_NAME` for making it only in an app qube.
It is also possible to call service without specific client program – in which case server stdin/out will be connected with the terminal:
The source VM name can then be accessed in the server process via
`QREXEC_REMOTE_DOMAIN` environment variable. (Note the source VM has *no*
control over the name provided in this variable--the name of the VM is
provided by dom0, and so is trusted.)
.. code:: bash
$ qrexec-client-vm target_vm_name RPC_ACTION_NAME
Answering an RPC call
^^^^^^^^^^^^^^^^^^^^^
In other for a RPC call to be answered in the target VM, a file in either of the following locations must exist, containing the file name of the program that will be invoked, or being that program itself – in which case it must have executable permission set (``chmod +x``): - ``/etc/qubes-rpc/RPC_ACTION_NAME`` when you make it in the template qube; - ``/usr/local/etc/qubes-rpc/RPC_ACTION_NAME`` for making it only in an app qube.
The source VM name can then be accessed in the server process via ``QREXEC_REMOTE_DOMAIN`` environment variable. (Note the source VM has *no* control over the name provided in this variable–the name of the VM is provided by dom0, and so is trusted.)
Specifying VMs: tags, types, targets, etc.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
### Specifying VMs: tags, types, targets, etc.
There are severals methods for specifying source/target VMs in RPC policies.
- `@tag:some-tag` - meaning a VM with tag `some-tag`
- `@type:type` - meaning a VM of `type` (like `AppVM`, `TemplateVM` etc)
- ``@tag:some-tag`` - meaning a VM with tag ``some-tag``
Target VM can be also specified as `@default`, which matches the case when calling VM didn't specified any particular target (either by using `@default` target, or empty target).
For DisposableVMs, `@dispvm:DISP_VM` is very similar to `@dispvm` but forces using a particular VM (`DISP_VM`) as a base VM to be started as DisposableVM.
For example:
- ``@type:type`` - meaning a VM of ``type`` (like ``AppVM``, ``TemplateVM`` etc)
```
* * anon-whonix @dispvm:anon-whonix-dvm allow
```
Adding such policy itself will not force usage of this particular `DISP_VM` - it will only allow it when specified by the caller.
But `@dispvm:DISP_VM` can also be used as target in request redirection, so _it is possible_ to force particular `DISP_VM` usage, when caller didn't specify it:
```
* * anon-whonix @dispvm allow target=@dispvm:anon-whonix-dvm
```
Target VM can be also specified as ``@default``, which matches the case when calling VM didn’t specified any particular target (either by using ``@default`` target, or empty target). For DisposableVMs, ``@dispvm:DISP_VM`` is very similar to ``@dispvm`` but forces using a particular VM (``DISP_VM``) as a base VM to be started as DisposableVM. For example:
Note that without redirection, this rule would allow using default Disposable VM (`default_dispvm` VM property, which itself defaults to global `default_dispvm` property).
Also note that the request will be allowed (`allow` action) even if there is no second rule allowing calls to `@dispvm:anon-whonix-dvm`, or even if there is a rule explicitly denying it.
This is because the redirection happens _after_ considering the action.
.. code:: bash
The policy confirmation dialog (`ask` action) allows the user to specify target VM.
User can choose from VMs that, according to policy, would lead to `ask` or `allow` actions.
It is not possible to select VM that policy would deny.
By default no VM is selected, even if the caller provided some, but policy can specify default value using `default_target=` parameter.
For example:
* * anon-whonix @dispvm:anon-whonix-dvm allow
```
* * work-mail work-archive allow
* * work-mail @tag:work ask default_target=work-files
* * work-mail @default ask default_target=work-files
```
The first rule allows calls from `work-mail` to `work-archive`, without any confirmation.
The second rule will ask the user about calls from `work-mail` VM to any VM with tag `work`.
And the confirmation dialog will have `work-files` VM chosen by default, regardless of the VM specified by the caller (`work-mail` VM).
The third rule allows the caller to not specify target VM at all and let the user choose, still - from VMs with tag `work` (and `work-archive`, regardless of tag), and with `work-files` as default.
### RPC services and security
Adding such policy itself will not force usage of this particular ``DISP_VM`` - it will only allow it when specified by the caller. But ``@dispvm:DISP_VM`` can also be used as target in request redirection, so *it is possible* to force particular ``DISP_VM`` usage, when caller didn’t specify it:
Be very careful when coding and adding a new RPC service.
Unless the offered functionality equals full control over the target (it is the case with e.g. `qubes.VMShell` action), any vulnerability in an RPC server can be fatal to Qubes security.
On the other hand, this mechanism allows to delegate processing of untrusted input to less privileged (or disposable) AppVMs, thus wise usage of it increases security.
.. code:: bash
* * anon-whonix @dispvm allow target=@dispvm:anon-whonix-dvm
Note that without redirection, this rule would allow using default Disposable VM (``default_dispvm`` VM property, which itself defaults to global ``default_dispvm`` property). Also note that the request will be allowed (``allow`` action) even if there is no second rule allowing calls to ``@dispvm:anon-whonix-dvm``, or even if there is a rule explicitly denying it. This is because the redirection happens *after* considering the action.
The policy confirmation dialog (``ask`` action) allows the user to specify target VM. User can choose from VMs that, according to policy, would lead to ``ask`` or ``allow`` actions. It is not possible to select VM that policy would deny. By default no VM is selected, even if the caller provided some, but policy can specify default value using ``default_target=`` parameter. For example:
.. code:: bash
* * work-mail work-archive allow
* * work-mail @tag:work ask default_target=work-files
* * work-mail @default ask default_target=work-files
The first rule allows calls from ``work-mail`` to ``work-archive``, without any confirmation. The second rule will ask the user about calls from ``work-mail`` VM to any VM with tag ``work``. And the confirmation dialog will have ``work-files`` VM chosen by default, regardless of the VM specified by the caller (``work-mail`` VM). The third rule allows the caller to not specify target VM at all and let the user choose, still - from VMs with tag ``work`` (and ``work-archive``, regardless of tag), and with ``work-files`` as default.
RPC services and security
^^^^^^^^^^^^^^^^^^^^^^^^^
Be very careful when coding and adding a new RPC service. Unless the offered functionality equals full control over the target (it is the case with e.g. ``qubes.VMShell`` action), any vulnerability in an RPC server can be fatal to Qubes security. On the other hand, this mechanism allows to delegate processing of untrusted input to less privileged (or disposable) AppVMs, thus wise usage of it increases security.
For example, this command will run the ``firefox`` command in a DisposableVM based on ``work``:
.. code:: bash
$ qvm-run --dispvm=work firefox
For example, this command will run the `firefox` command in a DisposableVM based on `work`:
```
$ qvm-run --dispvm=work firefox
```
By contrast, consider this command:
```
$ qvm-run --dispvm=work --service qubes.StartApp+firefox
```
.. code:: bash
This will look for a `firefox.desktop` file in a standard location in a DisposableVM based on `work`, then launch the application described by that file.
The practical difference is that the bare `qvm-run` command uses the `qubes.VMShell` service, which allows you to run an arbitrary command with arbitrary arguments, essentially providing full control over the target VM.
By contrast, the `qubes.StartApp` service allows you to run only applications that are advertised in `/usr/share/applications` (or other standard locations) *without* control over the arguments, so giving a VM access to `qubes.StartApp` is much safer.
While there isn't much practical difference between the two commands above when starting an application from dom0 in Qubes 4.0, there is a significant security risk when launching applications from a domU (e.g., from a separate GUI domain).
This is why `qubes.StartApp` uses our standard `qrexec` argument grammar to strictly filter the permissible grammar of the `Exec=` lines in `.desktop` files that are passed from untrusted domUs to dom0, thereby protecting dom0 from command injection by maliciously-crafted `.desktop` files.
### Service policies with arguments
Sometimes a service name alone isn't enough to make reasonable qrexec policy.
One example of such a situation is [qrexec-based USB passthrough](/doc/how-to-use-usb-devices/).
Using just a service name would make it difficult to express the policy "allow access to devices X and Y, but deny to all others."
It isn't feasible to create a separate service for every device: we would need to change the code in multiple files any time we wanted to update the service.
For this reason it is possible to specify a service argument, which will be subject to a policy.
A service argument can make service policies more fine-grained.
With arguments, it is easier to write more precise policies using the "allow" and "deny" actions, instead of relying on the "ask" method.
(Writing too many "ask" policies offloads additional decisions to the user.
Generally, the fewer choices the user must make, the lower the chance to make a mistake.)
The argument is specified in the second column of the policy line, as +ARGUMENT.
If the policy uses "\*" as an argument, then it will match any argument (including no argument).
As rules are processed in order, any lines with a specific argument below the line with the wildcard argument will be ignored.
So for instance, we might have policies which are different depending on the argument:
```
Device +device1 * * allow
Device +device2 * * deny
Device * * * ask
```
When calling a service that takes an argument, just add the argument to the service name separated with `+`.
```
$ qrexec-client-vm target_vm_name RPC_ACTION_NAME+ARGUMENT
```
The script will receive `ARGUMENT` as its argument.
The argument will also become available as the `QREXEC_SERVICE_ARGUMENT` environment variable.
This means it is possible to install a different script for a particular service argument.
See [below](#rpc-service-with-argument-file-reader) for an example of an RPC service using an argument.
$ qvm-run --dispvm=work --service qubes.StartApp+firefox
## Qubes RPC examples
This will look for a ``firefox.desktop`` file in a standard location in a DisposableVM based on ``work``, then launch the application described by that file. The practical difference is that the bare ``qvm-run`` command uses the ``qubes.VMShell`` service, which allows you to run an arbitrary command with arbitrary arguments, essentially providing full control over the target VM. By contrast, the ``qubes.StartApp`` service allows you to run only applications that are advertised in ``/usr/share/applications`` (or other standard locations) *without* control over the arguments, so giving a VM access to ``qubes.StartApp`` is much safer. While there isn’t much practical difference between the two commands above when starting an application from dom0 in Qubes 4.0, there is a significant security risk when launching applications from a domU (e.g., from a separate GUI domain). This is why ``qubes.StartApp`` uses our standard ``qrexec`` argument grammar to strictly filter the permissible grammar of the ``Exec=`` lines in ``.desktop`` files that are passed from untrusted domUs to dom0, thereby protecting dom0 from command injection by maliciously-crafted ``.desktop`` files.
Service policies with arguments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sometimes a service name alone isn’t enough to make reasonable qrexec policy. One example of such a situation is :doc:`qrexec-based USB passthrough </user/how-to-guides/how-to-use-usb-devices>`. Using just a service name would make it difficult to express the policy “allow access to devices X and Y, but deny to all others.” It isn’t feasible to create a separate service for every device: we would need to change the code in multiple files any time we wanted to update the service.
For this reason it is possible to specify a service argument, which will be subject to a policy. A service argument can make service policies more fine-grained. With arguments, it is easier to write more precise policies using the “allow” and “deny” actions, instead of relying on the “ask” method. (Writing too many “ask” policies offloads additional decisions to the user. Generally, the fewer choices the user must make, the lower the chance to make a mistake.)
The argument is specified in the second column of the policy line, as +ARGUMENT. If the policy uses “*” as an argument, then it will match any argument (including no argument). As rules are processed in order, any lines with a specific argument below the line with the wildcard argument will be ignored. So for instance, we might have policies which are different depending on the argument:
.. code:: bash
Device +device1 * * allow
Device +device2 * * deny
Device * * * ask
When calling a service that takes an argument, just add the argument to the service name separated with ``+``.
.. code:: bash
$ qrexec-client-vm target_vm_name RPC_ACTION_NAME+ARGUMENT
The script will receive ``ARGUMENT`` as its argument. The argument will also become available as the ``QREXEC_SERVICE_ARGUMENT`` environment variable. This means it is possible to install a different script for a particular service argument.
See `below <#rpc-service-with-argument-file-reader>`__ for an example of an RPC service using an argument.
Qubes RPC examples
------------------
To demonstrate some of the possibilities afforded by the qrexec framework, here are two examples of custom RPC services.
### Simple RPC service (addition)
Simple RPC service (addition)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We can create an RPC service that adds two integers in a target domain (the server, call it "anotherVM") and returns back the result to the invoker (the client, "someVM").
In someVM, create a file with the following contents and save it with the path `/usr/bin/our_test_add_client`:
```
#!/bin/sh
echo $1 $2 # pass data to RPC server
exec cat >&$SAVED_FD_1 # print result to the original stdout, not to the other RPC endpoint
```
We can create an RPC service that adds two integers in a target domain (the server, call it “anotherVM”) and returns back the result to the invoker (the client, “someVM”). In someVM, create a file with the following contents and save it with the path ``/usr/bin/our_test_add_client``:
Our server will be anotherVM at `/usr/bin/our_test_add_server`.
The code for this file is:
.. code:: bash
```
#!/bin/sh
read arg1 arg2 # read from stdin, which is received from the RPC client
echo $(($arg1+$arg2)) # print to stdout, which is passed to the RPC client
```
#!/bin/sh
echo $1 $2 # pass data to RPC server
exec cat >&$SAVED_FD_1 # print result to the original stdout, not to the other RPC endpoint
We'll need to create a service called `test.Add` with its own definition and policy file in dom0.
Now we need to define what the service does.
In this case, it should call our addition script.
We define the service with a symlink at `/etc/qubes-rpc/test.Add` pointing to our server script (the script can be also placed directly in `/etc/qubes-rpc/test.Add` - make sure the file has executable bit set!):
```
ln -s /usr/bin/our_test_add_server /etc/qubes-rpc/test.Add
```
The administrative domain will direct traffic based on the current RPC policies.
In dom0, create a file at `/etc/qubes/policy.d/30-test.policy` containing the following:
Our server will be anotherVM at ``/usr/bin/our_test_add_server``. The code for this file is:
.. code:: bash
#!/bin/sh
read arg1 arg2 # read from stdin, which is received from the RPC client
echo $(($arg1+$arg2)) # print to stdout, which is passed to the RPC client
We’ll need to create a service called ``test.Add`` with its own definition and policy file in dom0. Now we need to define what the service does. In this case, it should call our addition script. We define the service with a symlink at ``/etc/qubes-rpc/test.Add`` pointing to our server script (the script can be also placed directly in ``/etc/qubes-rpc/test.Add`` - make sure the file has executable bit set!):
.. code:: bash
ln -s /usr/bin/our_test_add_server /etc/qubes-rpc/test.Add
The administrative domain will direct traffic based on the current RPC policies. In dom0, create a file at ``/etc/qubes/policy.d/30-test.policy`` containing the following:
.. code:: bash
test.Add * * * ask
```
test.Add * * * ask
```
This will allow our client and server to communicate.
Before we make the call, ensure that the client and server scripts have executable permissions.
Finally, invoke the RPC service.
Before we make the call, ensure that the client and server scripts have executable permissions. Finally, invoke the RPC service.
```
$ qrexec-client-vm anotherVM test.Add /usr/bin/our_test_add_client 1 2
```
.. code:: bash
We should get "3" as answer.
(dom0 will ask for confirmation first.)
$ qrexec-client-vm anotherVM test.Add /usr/bin/our_test_add_client 1 2
**Note:** For a real world example of writing a qrexec service, see this [blog post](https://blog.invisiblethings.org/2013/02/21/converting-untrusted-pdfs-into-trusted.html).
### RPC service with argument (file reader)
Here we create an RPC call that reads a specific file from a predefined directory on the target.
This example uses an [argument](#service-policies-with-arguments) to the policy.
In this example a simplified workflow will be used. The service code is placed directly in the service definition file on the target VM.
No separate client script will be needed.
We should get “3” as answer. (dom0 will ask for confirmation first.)
First, on your target VM, create two files in the home directory: `testfile1` and `testfile2`.
Have them contain two different "Hello world!" lines.
**Note:** For a real world example of writing a qrexec service, see this `blog post <https://blog.invisiblethings.org/2013/02/21/converting-untrusted-pdfs-into-trusted.html>`__.
Next, we define the RPC service.
On the target VM, place the code below at `/etc/qubes-rpc/test.File`:
RPC service with argument (file reader)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```
#!/bin/sh
argument="$1" # service argument, also available as $QREXEC_SERVICE_ARGUMENT
if [ -z "$argument" ]; then
echo "ERROR: No argument given!"
exit 1
fi
cat "/home/user/$argument"
```
Make sure the file is executable!
(The service argument is already sanitized by qrexec framework.
It is guaranteed to not contain any spaces or slashes, so there should be no need for additional path sanitization.)
Here we create an RPC call that reads a specific file from a predefined directory on the target. This example uses an `argument <#service-policies-with-arguments>`__ to the policy. In this example a simplified workflow will be used. The service code is placed directly in the service definition file on the target VM. No separate client script will be needed.
Now we create the policy file in dom0, at `/etc/qubes/policy.d/30-test.policy`.
The contents of the file are below.
Replace "source_vm1" and others with the names of your own chosen domains.
First, on your target VM, create two files in the home directory: ``testfile1`` and ``testfile2``. Have them contain two different “Hello world!” lines.
```
test.File +testfile1 source_vm1 target_vm allow
test.File +testfile2 source_vm2 target_vm allow
test.File * * * deny
```
Next, we define the RPC service. On the target VM, place the code below at ``/etc/qubes-rpc/test.File``:
With this done, we can run some tests.
Invoke RPC from `source_vm1` via
.. code:: bash
```
[user@source_vm1] $ qrexec-client-vm target_vm test.File+testfile1
```
#!/bin/sh
argument="$1" # service argument, also available as $QREXEC_SERVICE_ARGUMENT
if [ -z "$argument" ]; then
echo "ERROR: No argument given!"
exit 1
fi
cat "/home/user/$argument"
Make sure the file is executable! (The service argument is already sanitized by qrexec framework. It is guaranteed to not contain any spaces or slashes, so there should be no need for additional path sanitization.)
Now we create the policy file in dom0, at ``/etc/qubes/policy.d/30-test.policy``. The contents of the file are below. Replace “source_vm1” and others with the names of your own chosen domains.
.. code:: bash
test.File +testfile1 source_vm1 target_vm allow
test.File +testfile2 source_vm2 target_vm allow
test.File * * * deny
With this done, we can run some tests. Invoke RPC from ``source_vm1`` via
.. code:: bash
[user@source_vm1] $ qrexec-client-vm target_vm test.File+testfile1
We should get the contents of ``/home/user/testfile1`` printed to the terminal. Invoking the service from ``source_vm2`` should result in a denial, but ``testfile2`` should work.
.. code:: bash
[user@source_vm2] $ qrexec-client-vm target_vm test.File+testfile1
Request refused
[user@source_vm2] $ qrexec-client-vm target_vm test.File+testfile2
We should get the contents of `/home/user/testfile1` printed to the terminal.
Invoking the service from `source_vm2` should result in a denial, but `testfile2` should work.
```
[user@source_vm2] $ qrexec-client-vm target_vm test.File+testfile1
Request refused
[user@source_vm2] $ qrexec-client-vm target_vm test.File+testfile2
```
And when invoked with other arguments or from a different VM, it should also be denied.

442
developer/services/qrexec2.rst
View file

@ -1,338 +1,276 @@
---
lang: en
layout: doc
permalink: /doc/qrexec2/
redirect_from:
- /doc/qrexec2-implementation/
- /en/doc/qrexec2-implementation/
- /doc/Qrexec2Implementation/
- /wiki/Qrexec2Implementation/
ref: 38
title: Qrexec v2 (deprecated)
---
======================
Qrexec v2 (deprecated)
======================
(*This page is about qrexec v2. For qrexec v3, see [here](/doc/qrexec/).*)
Qubes **qrexec** is a framework for implementing inter-VM (incl. Dom0-VM)
services. It offers a mechanism to start programs in VMs, redirect their
stdin/stdout, and a policy framework to control this all.
(*This page is about qrexec v2. For qrexec v3, see* :doc:`here </developer/services/qrexec>` *.*)
## Qrexec basics
Qubes **qrexec** is a framework for implementing inter-VM (incl. Dom0-VM) services. It offers a mechanism to start programs in VMs, redirect their stdin/stdout, and a policy framework to control this all.
During each domain creation a process named `qrexec-daemon` is started in
dom0, and a process named `qrexec-agent` is started in the VM. They are
connected over `vchan` channel.
Qrexec basics
-------------
Typically, the first thing that a `qrexec-client` instance does is to send
a request to `qrexec-agent` to start a process in the VM. From then on,
the stdin/stdout/stderr from this remote process will be passed to the
`qrexec-client` process.
During each domain creation a process named ``qrexec-daemon`` is started in dom0, and a process named ``qrexec-agent`` is started in the VM. They are connected over ``vchan`` channel.
Typically, the first thing that a ``qrexec-client`` instance does is to send a request to ``qrexec-agent`` to start a process in the VM. From then on, the stdin/stdout/stderr from this remote process will be passed to the ``qrexec-client`` process.
E.g., to start a primitive shell in a VM type the following in Dom0 console:
```shell_session
[user@dom0 ~]$ /usr/lib/qubes/qrexec-client -d <vm name> user:bash
```
.. code:: bash
[user@dom0 ~]$ /usr/lib/qubes/qrexec-client -d <vm name> user:bash
The string before first semicolon specifies what user to run the command as.
Adding `-e` on the `qrexec-client` command line results in mere command
execution (no data passing), and `qrexec-client` exits immediately after
sending the execution request.
Adding ``-e`` on the ``qrexec-client`` command line results in mere command execution (no data passing), and ``qrexec-client`` exits immediately after sending the execution request.
There is also the `-l <local program>` flag, which directs `qrexec-client`
to pass stdin/stdout of the remote program not to its stdin/stdout, but to
the (spawned for this purpose) `<local program>`.
There is also the ``-l <local program>`` flag, which directs ``qrexec-client`` to pass stdin/stdout of the remote program not to its stdin/stdout, but to the (spawned for this purpose) ``<local program>``.
The `qvm-run` command is heavily based on `qrexec-client`. It also takes care
of additional activities (e.g., starting the domain, if it is not up yet, and
starting the GUI daemon). Thus, it is usually more convenient to use `qvm-run`.
The ``qvm-run`` command is heavily based on ``qrexec-client``. It also takes care of additional activities (e.g., starting the domain, if it is not up yet, and starting the GUI daemon). Thus, it is usually more convenient to use ``qvm-run``.
There can be almost arbitrary number of `qrexec-client` processes for a domain
(i.e., `qrexec-client` processes connected to the same `qrexec-daemon`);
their data is multiplexed independently.
There can be almost arbitrary number of ``qrexec-client`` processes for a domain (i.e., ``qrexec-client`` processes connected to the same ``qrexec-daemon``); their data is multiplexed independently.
There is a similar command line utility available inside Linux AppVMs (note
the `-vm` suffix): `qrexec-client-vm` that will be described in subsequent
sections.
There is a similar command line utility available inside Linux AppVMs (note the ``-vm`` suffix): ``qrexec-client-vm`` that will be described in subsequent sections.
## Qubes RPC services
Qubes RPC services
------------------
Apart from simple Dom0-\>VM command executions, as discussed above, it is
also useful to have more advanced infrastructure for controlled inter-VM
RPC/services. This might be used for simple things like inter-VM file
copy operations, as well as more complex tasks like starting a DispVM,
and requesting it to do certain operations on a handed file(s).
Instead of implementing complex RPC-like mechanisms for inter-VM communication,
Qubes takes a much simpler and pragmatic approach and aims to only provide
simple *pipes* between the VMs, plus ability to request *pre-defined* programs
(servers) to be started on the other end of such pipes, and a centralized
policy (enforced by the `qrexec-policy` process running in dom0) which says
which VMs can request what services from what VMs.
Apart from simple Dom0->VM command executions, as discussed above, it is also useful to have more advanced infrastructure for controlled inter-VM RPC/services. This might be used for simple things like inter-VM file copy operations, as well as more complex tasks like starting a DispVM, and requesting it to do certain operations on a handed file(s).
Thanks to the framework and automatic stdin/stdout redirection, RPC programs
are very simple; both the client and server just use their stdin/stdout to pass
data. The framework does all the inner work to connect these file descriptors
to each other via `qrexec-daemon` and `qrexec-agent`. Additionally, DispVMs
are tightly integrated; RPC to a DispVM is a simple matter of using a magic
`$dispvm` keyword as the target VM name.
Instead of implementing complex RPC-like mechanisms for inter-VM communication, Qubes takes a much simpler and pragmatic approach and aims to only provide simple *pipes* between the VMs, plus ability to request *pre-defined* programs (servers) to be started on the other end of such pipes, and a centralized policy (enforced by the ``qrexec-policy`` process running in dom0) which says which VMs can request what services from what VMs.
All services in Qubes are identified by a single string, which by convention
takes a form of `qubes.ServiceName`. Each VM can provide handlers for each of
the known services by providing a file in `/etc/qubes-rpc/` directory with
the same name as the service it is supposed to handle. This file will then
be executed by the qrexec service, if the dom0 policy allowed the service to
be requested (see below). Typically, the files in `/etc/qubes-rpc/` contain
just one line, which is a path to the specific binary that acts as a server
for the incoming request, however they might also be the actual executable
themselves. Qrexec framework is careful about connecting the stdin/stdout
of the server process with the corresponding stdin/stdout of the requesting
process in the requesting VM (see example Hello World service described below).
Thanks to the framework and automatic stdin/stdout redirection, RPC programs are very simple; both the client and server just use their stdin/stdout to pass data. The framework does all the inner work to connect these file descriptors to each other via ``qrexec-daemon`` and ``qrexec-agent``. Additionally, DispVMs are tightly integrated; RPC to a DispVM is a simple matter of using a magic ``$dispvm`` keyword as the target VM name.
## Qubes RPC administration
All services in Qubes are identified by a single string, which by convention takes a form of ``qubes.ServiceName``. Each VM can provide handlers for each of the known services by providing a file in ``/etc/qubes-rpc/`` directory with the same name as the service it is supposed to handle. This file will then be executed by the qrexec service, if the dom0 policy allowed the service to be requested (see below). Typically, the files in ``/etc/qubes-rpc/`` contain just one line, which is a path to the specific binary that acts as a server for the incoming request, however they might also be the actual executable themselves. Qrexec framework is careful about connecting the stdin/stdout of the server process with the corresponding stdin/stdout of the requesting process in the requesting VM (see example Hello World service described below).
Besides each VM needing to provide explicit programs to serve each supported
service, the inter-VM service RPC is also governed by a central policy in Dom0.
Qubes RPC administration
------------------------
Besides each VM needing to provide explicit programs to serve each supported service, the inter-VM service RPC is also governed by a central policy in Dom0.
In dom0, there is a bunch of files in ``/etc/qubes-rpc/policy/`` directory, whose names describe the available RPC actions; their content is the RPC access policy database. Some example of the default services in Qubes are:
.. code:: bash
qubes.Filecopy
qubes.OpenInVM
qubes.ReceiveUpdates
qubes.SyncAppMenus
qubes.VMShell
qubes.ClipboardPaste
qubes.Gpg
qubes.NotifyUpdates
qubes.PdfConvert
In dom0, there is a bunch of files in `/etc/qubes-rpc/policy/` directory,
whose names describe the available RPC actions; their content is the RPC
access policy database. Some example of the default services in Qubes are:
```
qubes.Filecopy
qubes.OpenInVM
qubes.ReceiveUpdates
qubes.SyncAppMenus
qubes.VMShell
qubes.ClipboardPaste
qubes.Gpg
qubes.NotifyUpdates
qubes.PdfConvert
```
These files contain lines with the following format:
```
srcvm destvm (allow|deny|ask)[,user=user_to_run_as][,target=VM_to_redirect_to]
```
.. code:: bash
You can specify `srcvm` and `destvm` by name, or by one of `$anyvm`,
`$dispvm`, `dom0` reserved keywords (note string `dom0` does not match the
`$anyvm` pattern; all other names do). Only `$anyvm` keyword makes sense
in the `srcvm` field (service calls from dom0 are currently always allowed,
`$dispvm` means "new VM created for this particular request" - so it is never
a source of request). Currently, there is no way to specify source VM by type,
but this is planned for Qubes R3.
srcvm destvm (allow|deny|ask)[,user=user_to_run_as][,target=VM_to_redirect_to]
Whenever a RPC request for service named "XYZ" is received, the first line
in `/etc/qubes-rpc/policy/XYZ` that matches the actual `srcvm`/`destvm` is
consulted to determine whether to allow RPC, what user account the program
should run in target VM under, and what VM to redirect the execution to. If
the policy file does not exist, user is prompted to create one *manually*;
if still there is no policy file after prompting, the action is denied.
On the target VM, the `/etc/qubes-rpc/XYZ` must exist, containing the file
name of the program that will be invoked.
### Requesting VM-VM (and VM-Dom0) services execution
You can specify ``srcvm`` and ``destvm`` by name, or by one of ``$anyvm``, ``$dispvm``, ``dom0`` reserved keywords (note string ``dom0`` does not match the ``$anyvm`` pattern; all other names do). Only ``$anyvm`` keyword makes sense in the ``srcvm`` field (service calls from dom0 are currently always allowed, ``$dispvm`` means “new VM created for this particular request” - so it is never a source of request). Currently, there is no way to specify source VM by type, but this is planned for Qubes R3.
Whenever a RPC request for service named “XYZ” is received, the first line in ``/etc/qubes-rpc/policy/XYZ`` that matches the actual ``srcvm``/``destvm`` is consulted to determine whether to allow RPC, what user account the program should run in target VM under, and what VM to redirect the execution to. If the policy file does not exist, user is prompted to create one *manually*; if still there is no policy file after prompting, the action is denied.
On the target VM, the ``/etc/qubes-rpc/XYZ`` must exist, containing the file name of the program that will be invoked.
Requesting VM-VM (and VM-Dom0) services execution
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In a src VM, one should invoke the qrexec client via the following command:
```
/usr/lib/qubes/qrexec-client-vm <target vm name> <service name> <local program path> [local program arguments]
```
.. code:: bash
Note that only stdin/stdout is passed between RPC server and client --
notably, no cmdline argument are passed.
/usr/lib/qubes/qrexec-client-vm <target vm name> <service name> <local program path> [local program arguments]
The source VM name can be accessed in the server process via
`QREXEC_REMOTE_DOMAIN` environment variable. (Note the source VM has *no*
control over the name provided in this variable--the name of the VM is
provided by dom0, and so is trusted.)
By default, stderr of client and server is logged to respective
`/var/log/qubes/qrexec.XID` files, in each of the VM.
Be very careful when coding and adding a new RPC service! Any vulnerability
in a RPC server can be fatal to security of the target VM!
Note that only stdin/stdout is passed between RPC server and client – notably, no cmdline argument are passed.
If requesting VM-VM (and VM-Dom0) services execution *without cmdline helper*,
connect directly to `/var/run/qubes/qrexec-agent-fdpass` socket as described
[below](#all-the-pieces-together-at-work).
The source VM name can be accessed in the server process via ``QREXEC_REMOTE_DOMAIN`` environment variable. (Note the source VM has *no* control over the name provided in this variable–the name of the VM is provided by dom0, and so is trusted.)
### Revoking "Yes to All" authorization
By default, stderr of client and server is logged to respective ``/var/log/qubes/qrexec.XID`` files, in each of the VM.
Qubes RPC policy supports an "ask" action, that will prompt the user whether
a given RPC call should be allowed. It is set as default for services such
as inter-VM file copy. A prompt window launches in dom0, that gives the user
option to click "Yes to All", which allows the action and adds a new entry
to the policy file, which will unconditionally allow further calls for given
(service, srcVM, dstVM) tuple.
Be very careful when coding and adding a new RPC service! Any vulnerability in a RPC server can be fatal to security of the target VM!
In order to remove such authorization, issue this command from a Dom0 terminal
(example below for `qubes.Filecopy` service):
If requesting VM-VM (and VM-Dom0) services execution *without cmdline helper*, connect directly to ``/var/run/qubes/qrexec-agent-fdpass`` socket as described `below <#all-the-pieces-together-at-work>`__.
```shell_session
sudo nano /etc/qubes-rpc/policy/qubes.Filecopy
```
Revoking "Yes to All" authorization
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
and then remove any line(s) ending in "allow" (before the first `##` comment)
which are the "Yes to All" results.
A user might also want to set their own policies in this section. This may
mostly serve to prevent the user from mistakenly copying files or text from
a trusted to untrusted domain, or vice-versa.
Qubes RPC policy supports an “ask” action, that will prompt the user whether a given RPC call should be allowed. It is set as default for services such as inter-VM file copy. A prompt window launches in dom0, that gives the user option to click “Yes to All”, which allows the action and adds a new entry to the policy file, which will unconditionally allow further calls for given (service, srcVM, dstVM) tuple.
### Qubes RPC "Hello World" service
In order to remove such authorization, issue this command from a Dom0 terminal (example below for ``qubes.Filecopy`` service):
We will show the necessary files to create a simple RPC call that adds two
integers on the target VM and returns back the result to the invoking VM.
.. code:: bash
* Client code on source VM (`/usr/bin/our_test_add_client`)
sudo nano /etc/qubes-rpc/policy/qubes.Filecopy
```bash
#!/bin/sh
echo $1 $2 # pass data to rpc server
exec cat >&$SAVED_FD_1 # print result to the original stdout, not to the other rpc endpoint
```
* Server code on target VM (`/usr/bin/our_test_add_server`)
and then remove any line(s) ending in “allow” (before the first ``##`` comment) which are the “Yes to All” results.
```bash
#!/bin/sh
read arg1 arg2 # read from stdin, which is received from the rpc client
echo $(($arg1+$arg2)) # print to stdout - so, pass to the rpc client
```
A user might also want to set their own policies in this section. This may mostly serve to prevent the user from mistakenly copying files or text from a trusted to untrusted domain, or vice-versa.
* Policy file in dom0 (`/etc/qubes-rpc/policy/test.Add`)
Qubes RPC "Hello World" service
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```shell_session
$anyvm $anyvm ask
```
* Server path definition on target VM (`/etc/qubes-rpc/test.Add`)
We will show the necessary files to create a simple RPC call that adds two integers on the target VM and returns back the result to the invoking VM.
```
/usr/bin/our_test_add_server
```
- Client code on source VM (``/usr/bin/our_test_add_client``)
* To test this service, run the following in the source VM:
.. code:: bash
```
/usr/lib/qubes/qrexec-client-vm <target VM> test.Add /usr/bin/our_test_add_client 1 2
```
#!/bin/sh
echo $1 $2 # pass data to rpc server
exec cat >&$SAVED_FD_1 # print result to the original stdout, not to the other rpc endpoint
and we should get "3" as answer, provided dom0 policy allows the call to pass
through, which would happen after we click "Yes" in the popup that should
appear after the invocation of this command. If we changed the policy from
"ask" to "allow", then no popup should be presented, and the call will always
be allowed.
**Note:** For a real world example of writing a qrexec service, see this
[blog post](https://blog.invisiblethings.org/2013/02/21/converting-untrusted-pdfs-into-trusted.html).
- Server code on target VM (``/usr/bin/our_test_add_server``)
### More high-level RPCs?
.. code:: bash
As previously noted, Qubes aims to provide mechanisms that are very simple
and thus with very small attack surface. This is the reason why the inter-VM
RPC framework is very primitive and doesn't include any serialization or
other function arguments passing, etc. We should remember, however, that
users/app developers are always free to run more high-level RPC protocols on
top of qrexec. Care should be taken, however, to consider potential attack
surfaces that are exposed to untrusted or less trusted VMs in that case.
#!/bin/sh
read arg1 arg2 # read from stdin, which is received from the rpc client
echo $(($arg1+$arg2)) # print to stdout - so, pass to the rpc client
## Qubes RPC internals
(*This is about the implementation of qrexec v2. For the implementation of
qrexec v3, see [here](/doc/qrexec-internals/). Note that the user
API in v3 is backward compatible: qrexec apps written for Qubes R2 should
run without modification on Qubes R3.*)
- Policy file in dom0 (``/etc/qubes-rpc/policy/test.Add``)
.. code:: bash
$anyvm $anyvm ask
- Server path definition on target VM (``/etc/qubes-rpc/test.Add``)
.. code:: bash
/usr/bin/our_test_add_server
- To test this service, run the following in the source VM:
.. code:: bash
/usr/lib/qubes/qrexec-client-vm <target VM> test.Add /usr/bin/our_test_add_client 1 2
and we should get “3” as answer, provided dom0 policy allows the call to pass through, which would happen after we click “Yes” in the popup that should appear after the invocation of this command. If we changed the policy from “ask” to “allow”, then no popup should be presented, and the call will always be allowed.
**Note:** For a real world example of writing a qrexec service, see this `blog post <https://blog.invisiblethings.org/2013/02/21/converting-untrusted-pdfs-into-trusted.html>`__.
More high-level RPCs?
^^^^^^^^^^^^^^^^^^^^^
As previously noted, Qubes aims to provide mechanisms that are very simple and thus with very small attack surface. This is the reason why the inter-VM RPC framework is very primitive and doesn’t include any serialization or other function arguments passing, etc. We should remember, however, that users/app developers are always free to run more high-level RPC protocols on top of qrexec. Care should be taken, however, to consider potential attack surfaces that are exposed to untrusted or less trusted VMs in that case.
Qubes RPC internals
-------------------
(*This is about the implementation of qrexec v2. For the implementation of qrexec v3, see* :doc:`here </developer/services/qrexec-internals>` *. Note that the user API in v3 is backward compatible: qrexec apps written for Qubes R2 should run without modification on Qubes R3.*)
Dom0 tools implementation
-------------------------
## Dom0 tools implementation
Players:
* `/usr/lib/qubes/qrexec-daemon`: started by mgmt stack (qubes.py) when a
VM is started.
* `/usr/lib/qubes/qrexec-policy`: internal program used to evaluate the
policy file and making the 2nd half of the connection.
* `/usr/lib/qubes/qrexec-client`: raw command line tool that talks to the
daemon via unix socket (`/var/run/qubes/qrexec.XID`)
- ``/usr/lib/qubes/qrexec-daemon``: started by mgmt stack (qubes.py) when a VM is started.
- ``/usr/lib/qubes/qrexec-policy``: internal program used to evaluate the policy file and making the 2nd half of the connection.
- ``/usr/lib/qubes/qrexec-client``: raw command line tool that talks to the daemon via unix socket (``/var/run/qubes/qrexec.XID``)
**Note:** None of the above tools are designed to be used by users.
## Linux VMs implementation
Linux VMs implementation
------------------------
Players:
* `/usr/lib/qubes/qrexec-agent`: started by VM bootup scripts, a daemon.
* `/usr/lib/qubes/qubes-rpc-multiplexer`: executes the actual service program,
as specified in VM's `/etc/qubes-rpc/qubes.XYZ`.
* `/usr/lib/qubes/qrexec-client-vm`: raw command line tool that talks to
the agent.
- ``/usr/lib/qubes/qrexec-agent``: started by VM bootup scripts, a daemon.
**Note:** None of the above tools are designed to be used by
users. `qrexec-client-vm` is designed to be wrapped up by Qubes apps.
- ``/usr/lib/qubes/qubes-rpc-multiplexer``: executes the actual service program, as specified in VM’s ``/etc/qubes-rpc/qubes.XYZ``.
## Windows VMs implementation
- ``/usr/lib/qubes/qrexec-client-vm``: raw command line tool that talks to the agent.
`%QUBES_DIR%` is the installation path (`c:\Program Files\Invisible Things
Lab\Qubes OS Windows Tools` by default).
* `%QUBES_DIR%\bin\qrexec-agent.exe`: runs as a system service. Responsible
both for raw command execution and interpreting RPC service requests.
* `%QUBES_DIR%\qubes-rpc`: directory with `qubes.XYZ` files that contain
commands for executing RPC services. Binaries for the services are contained
in `%QUBES_DIR%\qubes-rpc-services`.
* `%QUBES_DIR%\bin\qrexec-client-vm`: raw command line tool that talks to
the agent.
**Note:** None of the above tools are designed to be used by
users. `qrexec-client-vm` is designed to be wrapped up by Qubes apps.
**Note:** None of the above tools are designed to be used by users. ``qrexec-client-vm`` is designed to be wrapped up by Qubes apps.
## All the pieces together at work
Windows VMs implementation
--------------------------
**Note:** This section is not needed to use qrexec for writing Qubes
apps. Also note the [qrexec framework implemention in Qubes R3](/doc/qrexec3/)
significantly differs from what is described in this section.
The VM-VM channels in Qubes R2 are made via "gluing" two VM-Dom0 and Dom0-VM
vchan connections:
``%QUBES_DIR%`` is the installation path (``c:\Program Files\Invisible Things Lab\Qubes OS Windows Tools`` by default).
![qrexec2-internals.png](/attachment/doc/qrexec2-internals.png)
- ``%QUBES_DIR%\bin\qrexec-agent.exe``: runs as a system service. Responsible both for raw command execution and interpreting RPC service requests.
Note that Dom0 never examines the actual data flowing in neither of the two
vchan connections.
- ``%QUBES_DIR%\qubes-rpc``: directory with ``qubes.XYZ`` files that contain commands for executing RPC services. Binaries for the services are contained in ``%QUBES_DIR%\qubes-rpc-services``.
When a user in a source VM executes `qrexec-client-vm` utility, the following
steps are taken:
- ``%QUBES_DIR%\bin\qrexec-client-vm``: raw command line tool that talks to the agent.
* `qrexec-client-vm` connects to `qrexec-agent`'s
`/var/run/qubes/qrexec-agent-fdpass` unix socket 3 times. Reads 4 bytes from
each of them, which is the fd number of the accepted socket in agent. These
3 integers, in text, concatenated, form "connection identifier" (CID)
* `qrexec-client-vm` writes to `/var/run/qubes/qrexec-agent` fifo a blob,
consisting of target vmname, rpc action, and CID
* `qrexec-client-vm` executes the rpc client, passing the above mentioned
unix sockets as process stdin/stdout, and optionally stderr (if the
`PASS_LOCAL_STDERR` env variable is set)
* `qrexec-agent` passes the blob to `qrexec-daemon`, via
`MSG_AGENT_TO_SERVER_TRIGGER_CONNECT_EXISTING` message over vchan
* `qrexec-daemon` executes `qrexec-policy`, passing source vmname, target
vmname, rpc action, and CID as cmdline arguments
* `qrexec-policy` evaluates the policy file. If successful, creates a pair of
`qrexec-client` processes, whose stdin/stdout are cross-connected.
* The first `qrexec-client` connects to the src VM, using the `-c ClientID`
parameter, which results in not creating a new process, but connecting to
the existing process file descriptors (these are the fds of unix socket
created in step 1).
* The second `qrexec-client` connects to the target VM, and executes
`qubes-rpc-multiplexer` command there with the rpc action as the cmdline
argument. Finally, `qubes-rpc-multiplexer` executes the correct rpc server
on the target.
* In the above step, if the target VM is `$dispvm`, the DispVM is created
via the `qfile-daemon-dvm` program. The latter waits for the `qrexec-client`
process to exit, and then destroys the DispVM.
*TODO: Protocol description ("wire-level" spec)*
**Note:** None of the above tools are designed to be used by users. ``qrexec-client-vm`` is designed to be wrapped up by Qubes apps.
All the pieces together at work
-------------------------------
**Note:** This section is not needed to use qrexec for writing Qubes apps. Also note the :doc:`qrexec framework implemention in Qubes R3 </developer/services/qrexec>` significantly differs from what is described in this section.
The VM-VM channels in Qubes R2 are made via “gluing” two VM-Dom0 and Dom0-VM vchan connections:
.. figure:: /attachment/doc/qrexec2-internals.png
:alt: qrexec2-internals.png
Note that Dom0 never examines the actual data flowing in neither of the two vchan connections.
When a user in a source VM executes ``qrexec-client-vm`` utility, the following steps are taken:
- ``qrexec-client-vm`` connects to ``qrexec-agent``’s ``/var/run/qubes/qrexec-agent-fdpass`` unix socket 3 times. Reads 4 bytes from each of them, which is the fd number of the accepted socket in agent. These 3 integers, in text, concatenated, form “connection identifier” (CID)
- ``qrexec-client-vm`` writes to ``/var/run/qubes/qrexec-agent`` fifo a blob, consisting of target vmname, rpc action, and CID
- ``qrexec-client-vm`` executes the rpc client, passing the above mentioned unix sockets as process stdin/stdout, and optionally stderr (if the ``PASS_LOCAL_STDERR`` env variable is set)
- ``qrexec-agent`` passes the blob to ``qrexec-daemon``, via ``MSG_AGENT_TO_SERVER_TRIGGER_CONNECT_EXISTING`` message over vchan
- ``qrexec-daemon`` executes ``qrexec-policy``, passing source vmname, target vmname, rpc action, and CID as cmdline arguments
- ``qrexec-policy`` evaluates the policy file. If successful, creates a pair of ``qrexec-client`` processes, whose stdin/stdout are cross-connected.
- The first ``qrexec-client`` connects to the src VM, using the ``-c ClientID`` parameter, which results in not creating a new process, but connecting to the existing process file descriptors (these are the fds of unix socket created in step 1).
- The second ``qrexec-client`` connects to the target VM, and executes ``qubes-rpc-multiplexer`` command there with the rpc action as the cmdline argument. Finally, ``qubes-rpc-multiplexer`` executes the correct rpc server on the target.
- In the above step, if the target VM is ``$dispvm``, the DispVM is created via the ``qfile-daemon-dvm`` program. The latter waits for the ``qrexec-client`` process to exit, and then destroys the DispVM.
*TODO: Protocol description (“wire-level” spec)*

118
developer/system/architecture.rst
View file

@ -1,84 +1,74 @@
---
lang: en
layout: doc
permalink: /doc/architecture/
redirect_from:
- /doc/qubes-architecture/
- /en/doc/qubes-architecture/
- /doc/QubesArchitecture/
- /wiki/QubesArchitecture/
ref: 56
title: Architecture
---
============
Architecture
============
Qubes implements a security-by-compartmentalization approach. To do this, Qubes
utilizes virtualization technology in order to isolate various programs from
each other and even to sandbox many system-level components, such as networking
and storage subsystems, so that the compromise of any of these programs or
components does not affect the integrity of the rest of the system.
[![qubes-schema-v2.png](/attachment/doc/qubes-schema-v2.png)](/attachment/doc/qubes-schema-v2.png)
Qubes implements a security-by-compartmentalization approach. To do this, Qubes utilizes virtualization technology in order to isolate various programs from each other and even to sandbox many system-level components, such as networking and storage subsystems, so that the compromise of any of these programs or components does not affect the integrity of the rest of the system.
Qubes lets the user define many secure compartments known as
[qubes](/doc/glossary/#qube), which are implemented as lightweight [virtual
machines (VMs)](/doc/glossary/#vm). For example, the user can have “personal,”
“work,” “shopping,” “bank,” and “random” app qubes and can use the applications
within those qubes just as if they were executing on the local machine. At the
same time, however, these applications are well isolated from each other. Qubes
also supports secure copy-and-paste and file sharing between qubes, of course.
|qubes-schema-v2.png|
Qubes lets the user define many secure compartments known as :ref:`qubes <user/reference/glossary:qube>`, which are implemented as lightweight :ref:`virtual machines (VMs) <user/reference/glossary:vm>`. For example, the user can have “personal,” “work,” “shopping,” “bank,” and “random” app qubes and can use the applications within those qubes just as if they were executing on the local machine. At the same time, however, these applications are well isolated from each other. Qubes also supports secure copy-and-paste and file sharing between qubes, of course.
Key architecture features
-------------------------
## Key architecture features
- Based on a secure bare-metal hypervisor (Xen)
- Networking code sand-boxed in an unprivileged VM (using IOMMU/VT-d)
- USB stacks and drivers sand-boxed in an unprivileged VM (currently
experimental feature)
- USB stacks and drivers sand-boxed in an unprivileged VM (currently experimental feature)
- No networking code in the privileged domain (dom0)
- All user applications run in “app qubes,” lightweight VMs based on Linux
- Centralized updates of all app qubes based on the same template
- Qubes GUI virtualization presents applications as if they were running
locally
- Qubes GUI virtualization presents applications as if they were running locally
- Qubes GUI provides isolation between apps sharing the same desktop
- Secure system boot based (optional)
(For those interested in the history of the project, [Architecture Spec v0.3
[PDF]](/attachment/doc/arch-spec-0.3.pdf) is the original 2009 document that
started this all. Please note that this document is for historical interest
only. For the latest information, please see the rest of the [System
Documentation](/doc/#system).)
## Qubes Core Stack
Qubes Core Stack is, as the name implies, the core component of Qubes OS. It's
the glue that connects all the other components together, and which allows
users and admins to interact with and configure the system. The other
components of the Qubes system include:
(For those interested in the history of the project, :download:`Architecture Spec v0.3 [PDF] </attachment/doc/arch-spec-0.3.pdf>` is the original 2009 document that started this all. Please note that this document is for historical interest only. For the latest information, please see the rest of the :ref:`System Documentation <system>`.)
Qubes Core Stack
----------------
Qubes Core Stack is, as the name implies, the core component of Qubes OS. It’s the glue that connects all the other components together, and which allows users and admins to interact with and configure the system. The other components of the Qubes system include:
- VM-located core agents (implementing e.g. qrexec endpoints used by various Qubes services)
- VM-customizations (making the VMs lightweight and working well with seamless GUI virtualization)
- Qubes GUI virtualization (the protocol, VM-located agents, and daemons located in the GUI domain which, for now, happens to be the same as dom0),
- GUI domain customizations (Desktop Environment customizations, decoration coloring plugin, etc)
- The admin qube distribution (various customizations, special services, such as for receiving and verifying updates, in the future: custom distro)
- The Xen hypervisor (with a bunch of customization patches, occasional hardening) or - in the future - some other virtualising or containerizing software or technology
- Multiple “Qubes Apps” (various services built on top of Qubes qrexec infrastructure, such as: trusted PDF and Image converters, Split GPG, safe USB proxies for HID devices, USB proxy for offering USB devices (exposed via qvm-usb), Yubikey support, USB Armory support, etc)
- Various ready-to-use templates (e.g. Debian-, Whonix-based), which are used to create actual VMs, i.e. provide the root filesystem to the VMs
- VM-located core agents (implementing e.g. qrexec endpoints used by various
Qubes services)
- VM-customizations (making the VMs lightweight and working well with seamless
GUI virtualization)
- Qubes GUI virtualization (the protocol, VM-located agents, and daemons
located in the GUI domain which, for now, happens to be the same as dom0),
- GUI domain customizations (Desktop Environment customizations, decoration
coloring plugin, etc)
- The admin qube distribution (various customizations, special services, such
as for receiving and verifying updates, in the future: custom distro)
- The Xen hypervisor (with a bunch of customization patches, occasional
hardening) or - in the future - some other virtualising or containerizing
software or technology
- Multiple "Qubes Apps" (various services built on top of Qubes qrexec
infrastructure, such as: trusted PDF and Image converters, Split GPG, safe
USB proxies for HID devices, USB proxy for offering USB devices (exposed via
qvm-usb), Yubikey support, USB Armory support, etc)
- Various ready-to-use templates (e.g. Debian-, Whonix-based), which are used
to create actual VMs, i.e. provide the root filesystem to the VMs
- Salt Stack integration
And all these components are "glued together" by the Qubes Core Stack.
[![Qubes system components](/attachment/doc/qubes-components.png)](/attachment/doc/qubes-components.png)
This diagram illustrates the location of all these components in the overall
system architecture. Unlike the other Qubes architecture diagram above, this
one takes an app-qube-centric approach.
And all these components are “glued together” by the Qubes Core Stack.
|Qubes system components|
This diagram illustrates the location of all these components in the overall system architecture. Unlike the other Qubes architecture diagram above, this one takes an app-qube-centric approach.
.. |qubes-schema-v2.png| image:: /attachment/doc/qubes-schema-v2.png
.. |Qubes system components| image:: /attachment/doc/qubes-components.png

93
developer/system/audio.rst
View file

@ -1,69 +1,70 @@
---
lang: en
layout: doc
permalink: /doc/audio-virtualization/
ref: 60
title: Audio virtualization
---
====================
Audio virtualization
====================
VMs on Qubes OS have access to virtualized audio through the PulseAudio module. It consists of two parts:
- ``pacat-simple-vchan`` running in a dom0/Audio VM (standalone application, one per VM, connected to the PulseAudio daemon)
- ``module-vchan-sink`` running in a VM (loaded into the PulseAudio process)
VMs on Qubes OS have access to virtualized audio through the PulseAudio module.
It consists of two parts:
- `pacat-simple-vchan` running in a dom0/Audio VM (standalone application, one per VM, connected to the PulseAudio daemon)
- `module-vchan-sink` running in a VM (loaded into the PulseAudio process)
Protocol
--------
The protocol between these two parts is designed to be as simple as possible.
Specifically, there is no audio format negotiation, no compression, etc.
All the audio data is transferred as raw 44.1kHz samples in S16LE format, 2 channels.
The protocol between these two parts is designed to be as simple as possible. Specifically, there is no audio format negotiation, no compression, etc. All the audio data is transferred as raw 44.1kHz samples in S16LE format, 2 channels.
These two parts are connected with two vchan links:
1. Connection on vchan port 4713 used to transfer audio from VM to dom0/Audio VM.
There is no negotiation, no headers, etc.
Only raw samples are sent over this channel.
2. Connection on vchan port 4714 used to transfer audio from dom0/Audio VM to VM.
Like the previous one, audio data is sent as raw samples.
1. Connection on vchan port 4713 used to transfer audio from VM to dom0/Audio VM. There is no negotiation, no headers, etc. Only raw samples are sent over this channel.
Additionally, the second vchan connection (on port 4714) is used in the opposite direction (VM to dom0) to send simple notifications/commands about the audio state.
Each such notification is a 4-byte number in little-endian format.
2. Connection on vchan port 4714 used to transfer audio from dom0/Audio VM to VM. Like the previous one, audio data is sent as raw samples.
Additionally, the second vchan connection (on port 4714) is used in the opposite direction (VM to dom0) to send simple notifications/commands about the audio state. Each such notification is a 4-byte number in little-endian format.
List of defined codes:
- `0x00010001` -- VM wants to receive audio input (some process is listening); prior to this message, `pacat-simple-vchan` will not send any audio samples to the VM.
- `0x00010000` -- VM does not want to receive audio input (no process is listening anymore); after this message, `pacat-simple-vchan` will not send any audio samples to the VM.
- `0x00020000` -- VM does not want to send audio output; informational for dom0, to avoid buffer under runs (may affect PulseAudio calculated delays).
- `0x00020001` -- VM does want to send audio output.
- ``0x00010001`` – VM wants to receive audio input (some process is listening); prior to this message, ``pacat-simple-vchan`` will not send any audio samples to the VM.
- ``0x00010000`` – VM does not want to receive audio input (no process is listening anymore); after this message, ``pacat-simple-vchan`` will not send any audio samples to the VM.
- ``0x00020000`` – VM does not want to send audio output; informational for dom0, to avoid buffer under runs (may affect PulseAudio calculated delays).
- ``0x00020001`` – VM does want to send audio output.
pacat-simple-vchan
------------------
This is the dom0 (or Audio VM) part of the audio virtualization.
It is responsible for transferring audio samples between the PulseAudio daemon in dom0/Audio VM (which has access to the actual audio hardware) and a VM playing/recording sounds.
A separate `pacat-simple-vchan` process runs in dom0 for each VM with audio.
Each of them opens separate input and output streams to their local PulseAudio, which allows for controlling the volume of each VM separately (using the `pavucontrol` tool, for example).
In order to (re)create a stream for a VM in dom0, `pacat-simple-vchan` can be used. In order to find the needed parameters, domid and domain name, the command `xl list` can be used.
Audio input to the VM is not sent by default.
It needs to be both requested by the VM part and explicitly enabled in `pacat-simple-vchan`.
The mechanism to do this differs between Qubes versions.
In Qubes before R4.1, `pacat-simple-vchan` is controlled over system D-Bus:
This is the dom0 (or Audio VM) part of the audio virtualization. It is responsible for transferring audio samples between the PulseAudio daemon in dom0/Audio VM (which has access to the actual audio hardware) and a VM playing/recording sounds. A separate ``pacat-simple-vchan`` process runs in dom0 for each VM with audio. Each of them opens separate input and output streams to their local PulseAudio, which allows for controlling the volume of each VM separately (using the ``pavucontrol`` tool, for example). In order to (re)create a stream for a VM in dom0, ``pacat-simple-vchan`` can be used. In order to find the needed parameters, domid and domain name, the command ``xl list`` can be used.
- destination: `org.qubesos.Audio.VMNAME` (where `VMNAME` is the VM's name)
- object path: `/org/qubesos/audio`
- interface: `org.qubesos.Audio`
- property: `RecAllowed` (which can be set using the `org.freedesktop.DBus.Properties` interface)
Audio input to the VM is not sent by default. It needs to be both requested by the VM part and explicitly enabled in ``pacat-simple-vchan``. The mechanism to do this differs between Qubes versions. In Qubes before R4.1, ``pacat-simple-vchan`` is controlled over system D-Bus:
In Qubes R4.1 and later, `pacat-simple-vchan` is controlled over a UNIX socket at `/var/run/qubes/audio-control.VMNAME` (where `VMNAME` is the VM's name).
Supported commands:
- destination: ``org.qubesos.Audio.VMNAME`` (where ``VMNAME`` is the VM’s name)
- `audio-input 1\n` - enable audio input
- `audio-input 0\n` - disable audio input
- object path: ``/org/qubesos/audio``
These commands can be sent using the `qubes.AudioInputEnable+VMNAME` and `qubes.AudioInputDisable+VMNAME` qrexec services, respectively.
The current status is written into QubesDB at `/audio-input/VMNAME` (where `VMNAME` is the VM's name) with either `1` or `0` values.
The lack of a key means that the `pacat-simple-vchan` for a given VM is not running.
- interface: ``org.qubesos.Audio``
In either version, it is exposed to the user as device of class `mic`, which can be attached to a VM (for example, using the `qvm-device mic` command).
- property: ``RecAllowed`` (which can be set using the ``org.freedesktop.DBus.Properties`` interface)
In Qubes R4.1 and later, ``pacat-simple-vchan`` is controlled over a UNIX socket at ``/var/run/qubes/audio-control.VMNAME`` (where ``VMNAME`` is the VM’s name). Supported commands:
- ``audio-input 1\n`` - enable audio input
- ``audio-input 0\n`` - disable audio input
These commands can be sent using the ``qubes.AudioInputEnable+VMNAME`` and ``qubes.AudioInputDisable+VMNAME`` qrexec services, respectively. The current status is written into QubesDB at ``/audio-input/VMNAME`` (where ``VMNAME`` is the VM’s name) with either ``1`` or ``0`` values. The lack of a key means that the ``pacat-simple-vchan`` for a given VM is not running.
In either version, it is exposed to the user as device of class ``mic``, which can be attached to a VM (for example, using the ``qvm-device mic`` command).

1865
developer/system/gui.rst
View file

File diff suppressed because it is too large Load diff

181
developer/system/networking.rst
View file

@ -1,73 +1,158 @@
---
lang: en
layout: doc
permalink: /doc/networking/
redirect_from:
- /doc/qubes-net/
- /en/doc/qubes-net/
- /doc/QubesNet/
- /wiki/QubesNet/
ref: 59
title: Networking
---
==========
Networking
==========
## Overall description
In Qubes, the standard Xen networking is used, based on backend driver in the driver domain and frontend drivers in VMs. In order to eliminate layer 2 attacks originating from a compromised VM, routed networking is used instead of the default bridging of `vif` devices and NAT is applied at each network hop. The default *vif-route* script had some deficiencies (requires `eth0` device to be up, and sets some redundant iptables rules), therefore the custom *vif-route-qubes* script is used.
Overall description
-------------------
The IP address of `eth0` interface in AppVM, as well as two IP addresses to be used as nameservers (`DNS1` and `DNS2`), are passed via QubesDB to AppVM during its boot (thus, there is no need for DHCP daemon in the network driver domain). `DNS1` and `DNS2` are private addresses; whenever an interface is brought up in the network driver domain, the */usr/lib/qubes/qubes\_setup\_dnat\_to\_ns* script sets up the DNAT iptables rules translating `DNS1` and `DNS2` to the newly learned real dns servers. This way AppVM networking configuration does not need to be changed when configuration in the network driver domain changes (e.g. user switches to a different WLAN). Moreover, in the network driver domain, there is no DNS server either, and consequently there are no ports open to the VMs.
## Routing tables examples
In Qubes, the standard Xen networking is used, based on backend driver in the driver domain and frontend drivers in VMs. In order to eliminate layer 2 attacks originating from a compromised VM, routed networking is used instead of the default bridging of ``vif`` devices and NAT is applied at each network hop. The default *vif-route* script had some deficiencies (requires ``eth0`` device to be up, and sets some redundant iptables rules), therefore the custom *vif-route-qubes* script is used.
The IP address of ``eth0`` interface in AppVM, as well as two IP addresses to be used as nameservers (``DNS1`` and ``DNS2``), are passed via QubesDB to AppVM during its boot (thus, there is no need for DHCP daemon in the network driver domain). ``DNS1`` and ``DNS2`` are private addresses; whenever an interface is brought up in the network driver domain, the */usr/lib/qubes/qubes_setup_dnat_to_ns* script sets up the DNAT iptables rules translating ``DNS1`` and ``DNS2`` to the newly learned real dns servers. This way AppVM networking configuration does not need to be changed when configuration in the network driver domain changes (e.g. user switches to a different WLAN). Moreover, in the network driver domain, there is no DNS server either, and consequently there are no ports open to the VMs.
Routing tables examples
-----------------------
VM routing table is simple:
||
|Destination|Gateway|Genmask|Flags|Metric|Ref|Use|Iface|
|0.0.0.0|0.0.0.0|0.0.0.0|U|0|0|0|eth0|
.. list-table::
:widths: 4 4 4 4 4 4 4 4
:align: center
:header-rows: 1
* - Destination
- Gateway
- Genmask
- Flags
- Metric
- Ref
- Use
- Iface
* - 0.0.0.0
- 0.0.0.0
- 0.0.0.0
- U
- 0
- 0
- 0
- eth0
Network driver domain routing table is a bit longer:
||
|Destination|Gateway|Genmask|Flags|Metric|Ref|Use|Iface|
|10.137.0.16|0.0.0.0|255.255.255.255|UH|0|0|0|vif4.0|
|10.137.0.7|0.0.0.0|255.255.255.255|UH|0|0|0|vif10.0|
|10.137.0.9|0.0.0.0|255.255.255.255|UH|0|0|0|vif9.0|
|10.137.0.8|0.0.0.0|255.255.255.255|UH|0|0|0|vif8.0|
|10.137.0.12|0.0.0.0|255.255.255.255|UH|0|0|0|vif3.0|
|192.168.0.0|0.0.0.0|255.255.255.0|U|1|0|0|eth0|
|0.0.0.0|192.168.0.1|0.0.0.0|UG|0|0|0|eth0|
.. list-table::
:widths: 1 1 1 1 1 1 1 1
:align: center
:header-rows: 1
## IPv6
* - Destination
- Gateway
- Genmask
- Flags
- Metric
- Ref
- Use
- Iface
* - 10.137.0.16
- 0.0.0.0
- 255.255.255.255
- UH
- 0
- 0
- 0
- vif4.0
* - 10.137.0.7
- 0.0.0.0
- 255.255.255.255
- UH
- 0
- 0
- 0
- vif10.0
* - 10.137.0.9
- 0.0.0.0
- 255.255.255.255
- UH
- 0
- 0
- 0
- vif9.0
* - 10.137.0.8
- 0.0.0.0
- 255.255.255.255
- UH
- 0
- 0
- 0
- vif8.0
* - 10.137.0.12
- 0.0.0.0
- 255.255.255.255
- UH
- 0
- 0
- 0
- vif3.0
* - 192.168.0.0
- 0.0.0.0
- 255.255.255.0
- U
- 1
- 0
- 0
- eth0
* - 0.0.0.0
- 192.168.0.1
- 0.0.0.0
- UG
- 0
- 0
- 0
- eth0
Starting with Qubes 4.0, there is opt-in support for IPv6 forwarding. Similar to the IPv4, traffic is routed and NAT is applied at each network gateway. This way we avoid reconfiguring every connected qube whenever uplink connection is changed, and even telling the qube what that uplink is - which may be complex when VPN or other tunneling services are employed.
The feature can be enabled on any network-providing qube, and will be propagated down the network tree, so every qube connected to it will also have IPv6 enabled.
To enable the `ipv6` feature use `qvm-features` tool and set the value to `1`. For example to enable it on `sys-net`, execute in dom0:
IPv6
----
```
qvm-features sys-net ipv6 1
```
It is also possible to explicitly disable IPv6 support for some qubes, even if it is connected to IPv6-providing one. This can be done by setting `ipv6` feature to empty value:
Starting with Qubes 4.0, there is opt-in support for IPv6 forwarding. Similar to the IPv4, traffic is routed and NAT is applied at each network gateway. This way we avoid reconfiguring every connected qube whenever uplink connection is changed, and even telling the qube what that uplink is - which may be complex when VPN or other tunneling services are employed. The feature can be enabled on any network-providing qube, and will be propagated down the network tree, so every qube connected to it will also have IPv6 enabled. To enable the ``ipv6`` feature use ``qvm-features`` tool and set the value to ``1``. For example to enable it on ``sys-net``, execute in dom0:
.. code:: bash
qvm-features sys-net ipv6 1
It is also possible to explicitly disable IPv6 support for some qubes, even if it is connected to IPv6-providing one. This can be done by setting ``ipv6`` feature to empty value:
.. code:: bash
qvm-features ipv4-only-qube ipv6 ''
```
qvm-features ipv4-only-qube ipv6 ''
```
This configuration is presented below - green qubes have IPv6 access, red one does not.
![ipv6-1](/attachment/doc/ipv6-1.png)
.. figure:: /attachment/doc/ipv6-1.png
:alt: ipv6-1
In that case, system uplink connection have native IPv6. But in some cases it may not be true. Then some tunneling solution can be used (for example teredo). The same will apply when the user is connected to VPN service providing IPv6 support, regardless of user's internet connection.
Such configuration can be expressed by enabling `ipv6` feature only on some subset of Qubes networking, for example by creating separate qube to encapsulate IPv6 traffic and setting `ipv6` to `1` only there. See diagram below
![ipv6-2](/attachment/doc/ipv6-2.png)
Besides enabling IPv6 forwarding, the standard Qubes firewall can be used to limit what network resources are available to each qube. Currently only the `qvm-firewall` command supports adding IPv6 rules, the GUI firewall editor will have this ability later.
In that case, system uplink connection have native IPv6. But in some cases it may not be true. Then some tunneling solution can be used (for example teredo). The same will apply when the user is connected to VPN service providing IPv6 support, regardless of user’s internet connection. Such configuration can be expressed by enabling ``ipv6`` feature only on some subset of Qubes networking, for example by creating separate qube to encapsulate IPv6 traffic and setting ``ipv6`` to ``1`` only there. See diagram below
**Note:** Setting or unsetting the `ipv6` feature only affects qubes-configured networking. It does not affect e.g. external interfaces. If you want to restrict IPv6 on these interfaces change the settings in Network Manager. Alternatively, disable IPv6 support using methods appropriate to the underlying template.
.. figure:: /attachment/doc/ipv6-2.png
:alt: ipv6-2
### Limitations
Currently only IPv4 DNS servers are configured, regardless of `ipv6` feature state. It is done this way to avoid reconfiguring all connected qubes whenever IPv6 DNS becomes available or not. Configuring qubes to always use IPv6 DNS and only fallback to IPv4 may result in relatively long timeouts and poor usability.
But note that DNS using IPv4 does not prevent to return IPv6 addresses. In practice this is only a problem for IPv6-only networks.
Besides enabling IPv6 forwarding, the standard Qubes firewall can be used to limit what network resources are available to each qube. Currently only the ``qvm-firewall`` command supports adding IPv6 rules, the GUI firewall editor will have this ability later.
**Note:** Setting or unsetting the ``ipv6`` feature only affects qubes-configured networking. It does not affect e.g. external interfaces. If you want to restrict IPv6 on these interfaces change the settings in Network Manager. Alternatively, disable IPv6 support using methods appropriate to the underlying template.
Limitations
^^^^^^^^^^^
Currently only IPv4 DNS servers are configured, regardless of ``ipv6`` feature state. It is done this way to avoid reconfiguring all connected qubes whenever IPv6 DNS becomes available or not. Configuring qubes to always use IPv6 DNS and only fallback to IPv4 may result in relatively long timeouts and poor usability. But note that DNS using IPv4 does not prevent to return IPv6 addresses. In practice this is only a problem for IPv6-only networks.

8
developer/system/qubes-core-admin-client.md
View file

@ -1,8 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/qubes-core-admin-client/
redirect_to: https://dev.qubes-os.org/projects/core-admin-client/en/latest/
ref: 245
title: Qubes core admin client
---

8
developer/system/qubes-core-admin.md
View file

@ -1,8 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/qubes-core-admin/
redirect_to: https://dev.qubes-os.org/projects/core-admin/en/latest/
ref: 246
title: Qubes core admin
---

8
developer/system/qubes-core-stack.md
View file

@ -1,8 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/qubes-core-stack/
redirect_to: /news/2017/10/03/core3/
ref: 247
title: Qubes core stack
---

82
developer/system/security-critical-code.rst
View file

@ -1,70 +1,70 @@
---
lang: en
layout: doc
permalink: /doc/security-critical-code/
redirect_from:
- /en/doc/security-critical-code/
- /doc/SecurityCriticalCode/
- /wiki/SecurityCriticalCode/
- /trac/wiki/SecurityCriticalCode/
ref: 55
title: Security-critical code
---
======================
Security-critical code
======================
Below is a list of security-critical (i.e., trusted) code components in Qubes OS.
A successful attack against any of these components could compromise the system's security.
This code can be thought of as the Trusted Computing Base (TCB) of Qubes OS.
One of the main goals of the project is to keep the TCB to an absolute minimum.
The size of the current TCB is on the order of hundreds of thousands of lines of C code, which is several orders of magnitude less than other OSes.
(In Windows, Linux, and Mac OSes, the amount of trusted code is typically on the order of tens of *millions* of lines of C code.)
For more information, see [Qubes Security Goals](/security/goals/).
Below is a list of security-critical (i.e., trusted) code components in Qubes OS. A successful attack against any of these components could compromise the system’s security. This code can be thought of as the Trusted Computing Base (TCB) of Qubes OS. One of the main goals of the project is to keep the TCB to an absolute minimum. The size of the current TCB is on the order of hundreds of thousands of lines of C code, which is several orders of magnitude less than other OSes. (In Windows, Linux, and Mac OSes, the amount of trusted code is typically on the order of tens of *millions* of lines of C code.)
For more information, see :doc:`Qubes Security Goals </developer/system/security-design-goals>`.
Security-critical Qubes-specific Components
-------------------------------------------
The following code components are security-critical in Qubes OS:
- Dom0-side of the libvchan library
- Dom0-side of the GUI virtualization code (`qubes-guid`)
- Dom0-side of the sound virtualization code (`pacat-simple-vchan`)
- Dom0-side in qrexec-related code (`qrexec_daemon`)
- VM memory manager (`qmemman`) that runs in Dom0
- Select Qubes RPC servers that run in Dom0: `qubes.ReceiveUpdates` and `qubes.SyncAppMenus`
- The `qubes.Filecopy` RPC server that runs in a VM (critical because it could allow one VM to compromise another if the user allows a file copy operation to be performed between them)
- Dom0-side of the GUI virtualization code (``qubes-guid``)
- Dom0-side of the sound virtualization code (``pacat-simple-vchan``)
- Dom0-side in qrexec-related code (``qrexec_daemon``)
- VM memory manager (``qmemman``) that runs in Dom0
- Select Qubes RPC servers that run in Dom0: ``qubes.ReceiveUpdates`` and ``qubes.SyncAppMenus``
- The ``qubes.Filecopy`` RPC server that runs in a VM (critical because it could allow one VM to compromise another if the user allows a file copy operation to be performed between them)
Security-critical Third-party Components
----------------------------------------
We did not create these components, but Qubes OS relies on them.
At the current stage of the project, we cannot afford to spend the time to thoroughly review and audit them, so we more or less "blindly" trust that they are secure.
We did not create these components, but Qubes OS relies on them. At the current stage of the project, we cannot afford to spend the time to thoroughly review and audit them, so we more or less “blindly” trust that they are secure.
- The Xen hypervisor
- Xen's xenstore backend running in Dom0
- Xen's block backend running in Dom0's kernel
- Xen’s xenstore backend running in Dom0
- Xen’s block backend running in Dom0’s kernel
- The RPM program used in Dom0 for verifying signatures on dom0 updates
- Somewhat trusted: log viewing software in dom0 that parses VM-influenced logs
Attacks on Networking Components
--------------------------------
Here are two examples of networking components that an adversary might seek to attack (or in which to exploit a vulnerability as part of an attack):
- Xen network PV frontends
- VMs' core networking stacks (core TCP/IP code)
Hypothetically, an adversary could compromise a NetVM, `sys-net-1`, and try to use it to attack the VMs connected to that NetVM.
However, Qubes allows for the existence of more than one NetVM, so the adversary would not be able to use `sys-net-1` in order to attack VMs connected to a *different* NetVM, `sys-net-2` without also compromising `sys-net-2`.
In addition, the adversary would not be able to use `sys-net-1` (or, for that matter, `sys-net-2`) to attack VMs that have networking disabled (i.e., VMs that are not connected to any NetVM).
- VMs’ core networking stacks (core TCP/IP code)
Buggy Code vs. Backdoored Code
Hypothetically, an adversary could compromise a NetVM, ``sys-net-1``, and try to use it to attack the VMs connected to that NetVM. However, Qubes allows for the existence of more than one NetVM, so the adversary would not be able to use ``sys-net-1`` in order to attack VMs connected to a *different* NetVM, ``sys-net-2`` without also compromising ``sys-net-2``. In addition, the adversary would not be able to use ``sys-net-1`` (or, for that matter, ``sys-net-2``) to attack VMs that have networking disabled (i.e., VMs that are not connected to any NetVM).
Buggy Code vs. Backdoored Code
------------------------------
There is an important distinction between buggy code and maliciously backdoored code.
We could have the most secure architecture and the most bulletproof TCB that perfectly isolates all domains from each other, but it would all be pretty useless if all the code we ran inside our domains (e.g. the code in our email clients, word processors, and web browsers) were backdoored.
In that case, only network-isolated domains would be somewhat trustworthy.
This means that we must trust at least some of the vendors that supply the code we run inside our domains.
(We don't have to trust *all* of them, but we at least have to trust the few that provide the apps we use in the most critical domains.)
In practice, we trust the software provided by the [Fedora Project](https://getfedora.org/).
This software is signed by Fedora distribution keys, so it is also critical that the tools used in domains for software updates (`dnf` and `rpm`) are trustworthy.
There is an important distinction between buggy code and maliciously backdoored code. We could have the most secure architecture and the most bulletproof TCB that perfectly isolates all domains from each other, but it would all be pretty useless if all the code we ran inside our domains (e.g. the code in our email clients, word processors, and web browsers) were backdoored. In that case, only network-isolated domains would be somewhat trustworthy.
This means that we must trust at least some of the vendors that supply the code we run inside our domains. (We don’t have to trust *all* of them, but we at least have to trust the few that provide the apps we use in the most critical domains.) In practice, we trust the software provided by the `Fedora Project <https://getfedora.org/>`__. This software is signed by Fedora distribution keys, so it is also critical that the tools used in domains for software updates (``dnf`` and ``rpm``) are trustworthy.

19
developer/system/security-design-goals.rst
View file

@ -1,17 +1,8 @@
---
lang: en
layout: doc
permalink: /doc/security-design-goals/
redirect_from:
- /security/goals/
- /doc/security-goals/
- /en/doc/security-goals/
- /doc/SecurityGoals/
- /wiki/SecurityGoals/
ref: 210
title: Security design goals
---
=====================
Security design goals
=====================
Qubes OS implements a security-by-isolation (or security-by-compartmentalization) approach by providing the ability to easily create many security domains. These domains are implemented as lightweight Virtual Machines (VMs) running under the Xen hypervisor. Qubes' main objective is to provide strong isolation between these domains, so that even if an attacker compromises one of the domains, the others are still safe. Qubes, however, does not attempt to provide any security isolation for applications running within the same domain. For example, a buggy web browser running in a Qubes domain could still be compromised just as easily as on a regular Linux distribution. The difference that Qubes makes is that now the attacker doesn't have access to all the software running in the other domains.
Qubes OS implements a security-by-isolation (or security-by-compartmentalization) approach by providing the ability to easily create many security domains. These domains are implemented as lightweight Virtual Machines (VMs) running under the Xen hypervisor. Qubes’ main objective is to provide strong isolation between these domains, so that even if an attacker compromises one of the domains, the others are still safe. Qubes, however, does not attempt to provide any security isolation for applications running within the same domain. For example, a buggy web browser running in a Qubes domain could still be compromised just as easily as on a regular Linux distribution. The difference that Qubes makes is that now the attacker doesn’t have access to all the software running in the other domains.
Qubes also provides features that make it easy and convenient to run these multiple domains, such as seamless GUI integration into one common desktop, secure clipboard copy and paste between domains, secure file transfer between domains, disposable VMs, and much more. Qubes also provides an advanced networking infrastructure that allows for the creation of multiple network VMs which isolate all the world-facing networking stacks and proxy VMs which can be used for advanced VPN configurations and tunneling over untrusted connections.

13
developer/system/system-doc.md
View file

@ -1,13 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/system-doc/
redirect_from:
- /en/doc/system-doc/
- /doc/SystemDoc/
- /wiki/SystemDoc/
redirect_to:
- /doc/#developer-documentation
ref: 62
title: System documentation
---

97
developer/system/template-implementation.rst
View file

@ -1,98 +1,137 @@
---
lang: en
layout: doc
permalink: /doc/template-implementation/
redirect_from:
- /en/doc/template-implementation/
- /doc/TemplateImplementation/
- /wiki/TemplateImplementation/
ref: 58
title: Template implementation
---
=======================
Template implementation
=======================
Block devices of a VM
---------------------
## Block devices of a VM
Every VM has 4 block devices connected:
- **xvda** – base root device (/) – details described below
- **xvdb** – private.img – place where VM always can write.
- **xvdc** – volatile.img, discarded at each VM restart – here is placed swap and temporal "/" modifications (see below)
- **xvdc** – volatile.img, discarded at each VM restart – here is placed swap and temporal “/” modifications (see below)
- **xvdd** – modules.img – kernel modules and firmware
### private.img (xvdb)
private.img (xvdb)
^^^^^^^^^^^^^^^^^^
This is mounted as /rw and here is placed all VM private data. This includes:
- */home* – which is bind mounted to /rw/home
- */usr/local* – which is symlink to /rw/usrlocal
- some config files (/rw/config) called by qubes core scripts (ex /rw/config/rc.local)
**Note:** Whenever a TemplateBasedVM is created, the contents of the `/home` directory of its parent TemplateVM are *not* copied to the [child TemplateBasedVM's](/doc/templates/#inheritance-and-persistence) `/home`. The child TemplateBasedVM's `/home` is independent from its parent TemplateVM's `/home`, which means that any changes to the parent TemplateVM's `/home` will not affect the child TemplateBasedVM's `/home`. Once a TemplateBasedVM has been created, any changes in its `/home`, `/usr/local`, or `/rw/config` directories will be persistent across reboots, which means that any files stored there will still be available after restarting the TemplateBasedVM. No changes in any other directories in TemplateBasedVMs persist in this manner. If you would like to make changes in other directories which *do* persist in this manner, you must make those changes in the parent TemplateVM.
### modules.img (xvdd)
**Note:** Whenever a TemplateBasedVM is created, the contents of the ``/home`` directory of its parent TemplateVM are *not* copied to the :ref:`child TemplateBasedVM’s <user/templates/templates:inheritance and persistence>` ``/home``. The child TemplateBasedVM’s ``/home`` is independent from its parent TemplateVM’s ``/home``, which means that any changes to the parent TemplateVM’s ``/home`` will not affect the child TemplateBasedVM’s ``/home``. Once a TemplateBasedVM has been created, any changes in its ``/home``, ``/usr/local``, or ``/rw/config`` directories will be persistent across reboots, which means that any files stored there will still be available after restarting the TemplateBasedVM. No changes in any other directories in TemplateBasedVMs persist in this manner. If you would like to make changes in other directories which *do* persist in this manner, you must make those changes in the parent TemplateVM.
modules.img (xvdd)
^^^^^^^^^^^^^^^^^^
As the kernel is chosen in dom0, there must be some way to provide matching kernel modules to VM. Qubes kernel directory consists of 3 files:
- *vmlinuz* – actual kernel
- *initramfs* – initial ramdisk containing script to setup snapshot devices (see below) and mount /lib/modules
- *modules.img* – filesystem image of /lib/modules with matching kernel modules and firmware (/lib/firmware/updates is symlinked to /lib/modules/firmware)
Normally kernel "package" is common for many VMs (can be set using qvm-prefs). One of them can be set as default (qvm-set-default-kernel) to simplify kernel updates (by default all VMs use the default kernel). All installed kernels are placed in /var/lib/qubes/vm-kernels as separate subdirs. In this case, modules.img is attached to the VM as R/O device.
There is a special case when the VM can have a custom kernel – when it is updateable (StandaloneVM or TemplateVM) and the kernel is set to "none" (by qvm-prefs). In this case the VM uses the kernel from the "kernels" VM subdir and modules.img is attached as R/W device.
## Qubes TemplateVM implementation
Normally kernel “package” is common for many VMs (can be set using qvm-prefs). One of them can be set as default (qvm-set-default-kernel) to simplify kernel updates (by default all VMs use the default kernel). All installed kernels are placed in /var/lib/qubes/vm-kernels as separate subdirs. In this case, modules.img is attached to the VM as R/O device.
There is a special case when the VM can have a custom kernel – when it is updateable (StandaloneVM or TemplateVM) and the kernel is set to “none” (by qvm-prefs). In this case the VM uses the kernel from the “kernels” VM subdir and modules.img is attached as R/W device.
Qubes TemplateVM implementation
-------------------------------
TemplateVM has a shared root.img across all AppVMs that are based on it. This mechanism has some advantages over a simple common device connected to multiple VMs:
- root.img can be modified while there are AppVMs running – without corrupting the filesystem
- multiple AppVMs that are using different versions of root.img (from various points in time) can be running concurrently
There are two layers of the device-mapper snapshot device; the first one enables modifying root.img without stopping the AppVMs and the second one, which is contained in the AppVM, enables temporal modifications to its filesystem. These modifications will be discarded after a restart of the AppVM.
![TemplateSharing2.png](/attachment/doc/TemplateSharing2.png)
.. figure:: /attachment/doc/TemplateSharing2.png
:alt: TemplateSharing2.png
Snapshot device in Dom0
^^^^^^^^^^^^^^^^^^^^^^^
### Snapshot device in Dom0
This device consists of:
- root.img – real template filesystem
- root-cow.img – differences between the device as seen by AppVM and the current root.img
The above is achieved through creating device-mapper snapshots for each version of root.img. When an AppVM is started, a xen hotplug script (/etc/xen/scripts/block-snapshot) reads the inode numbers of root.img and root-cow.img; these numbers are used as the snapshot device's name. When a device with the same name exists the new AppVM will use it – therefore, AppVMs based on the same version of root.img will use the same device. Of course, the device-mapper cannot use the files directly – it must be connected through /dev/loop\*. The same mechanism detects if there is a loop device associated with a file determined by the device and inode numbers – or if creating a new loop device is necessary.
The above is achieved through creating device-mapper snapshots for each version of root.img. When an AppVM is started, a xen hotplug script (/etc/xen/scripts/block-snapshot) reads the inode numbers of root.img and root-cow.img; these numbers are used as the snapshot device’s name. When a device with the same name exists the new AppVM will use it – therefore, AppVMs based on the same version of root.img will use the same device. Of course, the device-mapper cannot use the files directly – it must be connected through /dev/loop*. The same mechanism detects if there is a loop device associated with a file determined by the device and inode numbers – or if creating a new loop device is necessary.
When an AppVM is stopped the xen hotplug script checks whether the device is still in use – if it is not, the script removes the snapshot and frees the loop device.
#### Changes to template filesystem
Changes to template filesystem
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order for the full potential of the snapshot device to be realized, every change in root.img must save the original version of the modified block in root-cow.img. This is achieved by a snapshot-origin device.
When TemplateVM is started, it receives the snapshot-origin device connected as a root device (in read-write mode). Therefore, every change to this device is immediately saved in root.img – but remains invisible to the AppVM, which uses the snapshot.
When TemplateVM is stopped, the xen script moves root-cow.img to root-cow.img.old and creates a new one (using the `qvm-template-commit` tool). The snapshot device will remain untouched due to the loop device, which uses an actual file on the disk (by inode, not by name). Linux kernel frees the old root-cow.img files as soon as they are unused by all snapshot devices (to be exact, loop devices). The new root-cow.img file will get a new inode number, and so new AppVMs will get new snapshot devices (with different names).
When TemplateVM is stopped, the xen script moves root-cow.img to root-cow.img.old and creates a new one (using the ``qvm-template-commit`` tool). The snapshot device will remain untouched due to the loop device, which uses an actual file on the disk (by inode, not by name). Linux kernel frees the old root-cow.img files as soon as they are unused by all snapshot devices (to be exact, loop devices). The new root-cow.img file will get a new inode number, and so new AppVMs will get new snapshot devices (with different names).
Rollback template changes
^^^^^^^^^^^^^^^^^^^^^^^^^
#### Rollback template changes
There is possibility to rollback last template changes. Using the automatic snapshot that is normally saved every time the template is shutdown.
Refer to volume backup and revert [documentation](/doc/volume-backup-revert) for more information.
Refer to volume backup and revert :doc:`documentation </user/advanced-topics/volume-backup-revert>` for more information.
Snapshot device in AppVM
^^^^^^^^^^^^^^^^^^^^^^^^
### Snapshot device in AppVM
Root device is exposed to AppVM in read-only mode. AppVM can write only in:
- private.img – persistent storage (mounted in /rw) used for /home, /usr/local – in future versions, its use may be extended
- volatile.img – temporary storage, which is discarded after an AppVM restart
volatile.img is divided into two partitions:
1. changes to root device
2. swap partition
Inside of an AppVM, the root device is wrapped by the snapshot in the first partition of volatile.img. Therefore, the AppVM can write anything to its filesystem – however, such changes will be discarded after a restart.
### StandaloneVM
StandaloneVM
^^^^^^^^^^^^
Standalone VM enables user to modify root filesystem persistently. It can be created using *--standalone* switch to *qvm-create*.
Standalone VM enables user to modify root filesystem persistently. It can be created using *–standalone* switch to *qvm-create*.
It is implemented just like TemplateVM (has own root.img connected as R/W device), but no other VMs can be based on it.

470
developer/system/template-manager.rst
View file

@ -1,280 +1,342 @@
---
layout: doc
title: Template manager
permalink: /doc/template-manager/
---
================
Template manager
================
This document discusses the designs and technical details of `qvm-template`, a
template manager application. The goal of the project is to design a new
mechanism for template distribution and a unified tool for template management.
## Motivation
This document discusses the designs and technical details of ``qvm-template``, a template manager application. The goal of the project is to design a new mechanism for template distribution and a unified tool for template management.
Motivation
----------
This project was originally proposed in the 2020 Google Summer of Code program.
Previously, templates were distributed by RPM packages and managed by
`yum`/`dnf`. However, tracking inherently dynamic VM images with a package
manager suited for static files creates some challenges. For example, users may
accidentally update the images, overriding local changes
([#996](https://github.com/QubesOS/qubes-issues/issues/996),
[#1647](https://github.com/QubesOS/qubes-issues/issues/1647)). (Or in the case
of [#2061](https://github.com/QubesOS/qubes-issues/issues/2061), want to
specifically override the changes.) Other operations that work well on normal
VMs are also somewhat inconsistent on RPM-managed templates. This includes
actions such as renaming
([#839](https://github.com/QubesOS/qubes-issues/issues/839)), removal
([#5509](https://web.archive.org/web/20210526123932/https://github.com/QubesOS/qubes-issues/issues/5509)) and
backup/restore ([#1385](https://github.com/QubesOS/qubes-issues/issues/1385),
[#1453](https://github.com/QubesOS/qubes-issues/issues/1453), [discussion
thread
1](https://groups.google.com/forum/#!topic/qubes-devel/rwc2_miCNNE/discussion),
[discussion thread
2](https://groups.google.com/forum/#!topic/qubes-users/uQEUpv4THsY/discussion)).
In turn, this creates inconveniences and confusion for users
([#1403](https://github.com/QubesOS/qubes-issues/issues/1403),
[#4518](https://github.com/QubesOS/qubes-issues/issues/4518)).
Previously, templates were distributed by RPM packages and managed by ``yum``/``dnf``. However, tracking inherently dynamic VM images with a package manager suited for static files creates some challenges. For example, users may accidentally update the images, overriding local changes (`#996 <https://github.com/QubesOS/qubes-issues/issues/996>`__, `#1647 <https://github.com/QubesOS/qubes-issues/issues/1647>`__). (Or in the case of `#2061 <https://github.com/QubesOS/qubes-issues/issues/2061>`__, want to specifically override the changes.) Other operations that work well on normal VMs are also somewhat inconsistent on RPM-managed templates. This includes actions such as renaming (`#839 <https://github.com/QubesOS/qubes-issues/issues/839>`__), removal (`#5509 <https://web.archive.org/web/20210526123932/https://github.com/QubesOS/qubes-issues/issues/5509>`__) and backup/restore (`#1385 <https://github.com/QubesOS/qubes-issues/issues/1385>`__, `#1453 <https://github.com/QubesOS/qubes-issues/issues/1453>`__, `discussion thread 1 <https://groups.google.com/forum/#!topic/qubes-devel/rwc2_miCNNE/discussion>`__, `discussion thread 2 <https://groups.google.com/forum/#!topic/qubes-users/uQEUpv4THsY/discussion>`__). In turn, this creates inconveniences and confusion for users (`#1403 <https://github.com/QubesOS/qubes-issues/issues/1403>`__, `#4518 <https://github.com/QubesOS/qubes-issues/issues/4518>`__).
Also, the usage of RPM packages meant that installing a template results in
arbitrary code execution, which is not ideal.
Also, the usage of RPM packages meant that installing a template results in arbitrary code execution, which is not ideal.
Besides distribution, users may also wish to have an integrated template
management application
([#2062](https://github.com/QubesOS/qubes-issues/issues/2062),
[#2064](https://github.com/QubesOS/qubes-issues/issues/2064),
[#2534](https://github.com/QubesOS/qubes-issues/issues/2534),
[#3040](https://github.com/QubesOS/qubes-issues/issues/3040)), as opposed to
the situation where multiple programs are required for different purposes,
e.g., `qubes-dom0-update`, `dnf`, `qvm-remove`, `qubes-manager`.
Besides distribution, users may also wish to have an integrated template management application (`#2062 <https://github.com/QubesOS/qubes-issues/issues/2062>`__, `#2064 <https://github.com/QubesOS/qubes-issues/issues/2064>`__, `#2534 <https://github.com/QubesOS/qubes-issues/issues/2534>`__, `#3040 <https://github.com/QubesOS/qubes-issues/issues/3040>`__), as opposed to the situation where multiple programs are required for different purposes, e.g., ``qubes-dom0-update``, ``dnf``, ``qvm-remove``, ``qubes-manager``.
To tackle these issues, `qvm-template` was created. It strives to provide not
only a better mechanism for handling template installation but also a
consistent user-facing interface to deal with template management.
To tackle these issues, ``qvm-template`` was created. It strives to provide not only a better mechanism for handling template installation but also a consistent user-facing interface to deal with template management.
## Features
Features
--------
- Install/reinstall/downgrade/upgrade templates, either from local packages or remote repositories
- Install/reinstall/downgrade/upgrade templates, either from local packages or
remote repositories
- Ability to install templates in alternative pools
- Possibility for the template package to specify options such as the kernel
or `virt_mode` used by the resulting template
- Possibility for the template package to specify options such as the kernel or ``virt_mode`` used by the resulting template
- List and show information about local and available templates
- Machine-readable output for easy extension
- Search for templates
- Remove templates
- Optionally, VMs based on the template to be removed can be either removed
or "disassociated" -- namely, have their templates changed to a "dummy"
one.
- Optionally, VMs based on the template to be removed can be either removed or “disassociated” – namely, have their templates changed to a “dummy” one.
- Show available repositories
- Works in both dom0 and management VMs by utilizing the Admin API
- Works well with existing tools
- Command-line interface with DNF-like usage
- A graphical interface also available
## Package format
The RPM package format is still used. However, the contents are manually
extracted instead of installing the whole package. This allows us to take
advantage of existing tools for things like repository management. We can also
avoid the burden of dealing with verification, reducing the risk of issues like
[QSB-028](/news/2016/12/19/qsb-28/).
The package name should be in the form `qubes-template-<TEMPLATE_NAME>`.
Package format
--------------
The package metadata (summary, description, etc.) should not contain the `|`
character to avoid possibly cryptic errors. This is because of its use as an
internal separator. Note that as we already consider the repository metadata
untrusted. This should not result in security issues.
The file structure should be quite similar to previous template RPMs. Namely,
there should be the following files in the package:
The RPM package format is still used. However, the contents are manually extracted instead of installing the whole package. This allows us to take advantage of existing tools for things like repository management. We can also avoid the burden of dealing with verification, reducing the risk of issues like `QSB-028 <https://www.qubes-os.org/news/2016/12/19/qsb-28/>`__.
The package name should be in the form ``qubes-template-<TEMPLATE_NAME>``.
The package metadata (summary, description, etc.) should not contain the ``|`` character to avoid possibly cryptic errors. This is because of its use as an internal separator. Note that as we already consider the repository metadata untrusted. This should not result in security issues.
The file structure should be quite similar to previous template RPMs. Namely, there should be the following files in the package:
- ``var/lib/qubes/vm-templates/<TEMPLATE_NAME>/root.img.part.[00,01,...]``
- Split tarball of template ``root.img``
- Note that the file is still split due to tools such as ``rpm2cpio`` not supporting large files. (Notably, the cpio format does not support files over 4GiB.)
- ``var/lib/qubes/vm-templates/<TEMPLATE_NAME>/template.conf``
- `var/lib/qubes/vm-templates/<TEMPLATE_NAME>/root.img.part.[00,01,...]`
- Split tarball of template `root.img`
- Note that the file is still split due to tools such as `rpm2cpio` not
supporting large files. (Notably, the cpio format does not support files
over 4GiB.)
- `var/lib/qubes/vm-templates/<TEMPLATE_NAME>/template.conf`
- Stores custom package metadata (as RPM does not support custom attributes).
- Uses `KEY=VALUE` format.
- Fields (corresponding to
[qvm-prefs](https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-prefs.html#common-properties)
and
[qvm-features](https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-features.html#list-of-known-features)
tags with the same names)
- `virt_mode`
- Setting this to `pv` requires user confirmation.
- Permitted values: `pv`, `pvh`, `hvm`.
- `kernel`
- Only allowed to be set to "" (without quotes), i.e., "none", for
PVGrub.
- Network-related flags: (Must be set to IPv4 addresses in the form of
`x.x.x.x`.)
- `net.fake-ip`
- `net.fake-gateway`
- `net.fake-netmask`
- Boolean flags: (Permitted values are "1" and "0", denoting "true" and
"false" respectively.)
- `no-monitor-layout`
- `pci-e820-host`
- `linux-stubdom`
- `gui`
- `gui-emulated`
- `qrexec`
- `var/lib/qubes/vm-templates/<TEMPLATE_NAME>/whitelisted-appmenus.list`
- Uses ``KEY=VALUE`` format.
- Fields (corresponding to `qvm-prefs <https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-prefs.html#common-properties>`__ and `qvm-features <https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-features.html#list-of-known-features>`__ tags with the same names)
- ``virt_mode``
- Setting this to ``pv`` requires user confirmation.
- Permitted values: ``pv``, ``pvh``, ``hvm``.
- ``kernel``
- Only allowed to be set to “” (without quotes), i.e., “none”, for PVGrub.
- Network-related flags: (Must be set to IPv4 addresses in the form of ``x.x.x.x``.)
- ``net.fake-ip``
- ``net.fake-gateway``
- ``net.fake-netmask``
- Boolean flags: (Permitted values are “1” and “0”, denoting “true” and “false” respectively.)
- ``no-monitor-layout``
- ``pci-e820-host``
- ``linux-stubdom``
- ``gui``
- ``gui-emulated``
- ``qrexec``
- ``var/lib/qubes/vm-templates/<TEMPLATE_NAME>/whitelisted-appmenus.list``
- Contains default app menu entries of the template itself.
- `var/lib/qubes/vm-templates/<TEMPLATE_NAME>/vm-whitelisted-appmenus.list`
- ``var/lib/qubes/vm-templates/<TEMPLATE_NAME>/vm-whitelisted-appmenus.list``
- Contains default app menu entries of VMs based on the template.
- `var/lib/qubes/vm-templates/<TEMPLATE_NAME>/netvm-whitelisted-appmenus.list`
- ``var/lib/qubes/vm-templates/<TEMPLATE_NAME>/netvm-whitelisted-appmenus.list``
- Contains default app menu entries of NetVMs based on the template.
- These three files are the same as the current format.
- Note that the contents of these files are stored in `qvm-features` upon
installation. See the section below for details.
## Metadata storage
- Note that the contents of these files are stored in ``qvm-features`` upon installation. See the section below for details.
The template manager needs to keep metadata of installed templates such as
versions and origin. This data can be stored via `qvm-features` to keep things
consistent when, e.g., `qvm-remove` is used. Besides, backups are also more
easily handled this way.
Also, the fields can serve as an indicator of whether a template is installed
by `qvm-template`.
### Fields
Metadata storage
----------------
The template manager needs to keep metadata of installed templates such as versions and origin. This data can be stored via ``qvm-features`` to keep things consistent when, e.g., ``qvm-remove`` is used. Besides, backups are also more easily handled this way.
Also, the fields can serve as an indicator of whether a template is installed by ``qvm-template``.
Fields
^^^^^^
Most of the fields should be fairly self-explanatory.
- `template-name`
- Note that this field needs to be consistent with the template name to be
considered valid.
- `template-epoch`
- `template-version`
- `template-release`
- `template-reponame`
- `template-buildtime`
- `template-installtime`
- The times are in UTC, and are of the format `YYYY-MM-DD HH:MM:SS`.
- `template-license`
- `template-url`
- `template-summary`
- `template-description`
- Note that the newlines in this field are converted to `|` to work better
with existing tools like `qvm-features`.
- `menu-items`
- `default-menu-items`
- `netvm-menu-items`
- The `*menu-items` entries store the contents of
`var/lib/qubes/vm-templates/<TEMPLATE_NAME>/whitelisted-appmenus.list`,
`var/lib/qubes/vm-templates/<TEMPLATE_NAME>/vm-whitelisted-appmenus.list`,
`var/lib/qubes/vm-templates/<TEMPLATE_NAME>/netvm-whitelisted-appmenus.list`
respectively.
- Note that newlines are converted to spaces, again for it to work better
with existing tools. This should not cause ambiguity as [the FreeDesktop
specifications](https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html)
forbid spaces in .desktop file names.
- ``template-name``
## Repository management
- Note that this field needs to be consistent with the template name to be considered valid.
For UpdateVMs to access the repository configuration, the package
[qubes-repo-templates](https://github.com/WillyPillow/qubes-repo-templates) is
created with the following contents:
- `/etc/qubes/repo-templates/*.repo`: repository configuration
- `/etc/qubes/repo-templates/keys`: PGP keys
As template keys may be less trusted, they are *not* added to the system RPM
keychain but instead managed separately.
- ``template-epoch``
## Qrexec protocol
- ``template-version``
Dom0 and management VMs without network access also need to interact with
template repositories. The following qrexec calls that list and download
templates are thus proposed.
- ``template-release``
- `qubes.TemplateSearch`: wraps `dnf repoquery`
- `qubes.TemplateDownload`: wraps `dnf download`
- ``template-reponame``
- ``template-buildtime``
- ``template-installtime``
- The times are in UTC, and are of the format ``YYYY-MM-DD HH:MM:SS``.
- ``template-license``
- ``template-url``
- ``template-summary``
- ``template-description``
- Note that the newlines in this field are converted to ``|`` to work better with existing tools like ``qvm-features``.
- ``menu-items``
- ``default-menu-items``
- ``netvm-menu-items``
- The ``*menu-items`` entries store the contents of ``var/lib/qubes/vm-templates/<TEMPLATE_NAME>/whitelisted-appmenus.list``, ``var/lib/qubes/vm-templates/<TEMPLATE_NAME>/vm-whitelisted-appmenus.list``, ``var/lib/qubes/vm-templates/<TEMPLATE_NAME>/netvm-whitelisted-appmenus.list`` respectively.
- Note that newlines are converted to spaces, again for it to work better with existing tools. This should not cause ambiguity as `the FreeDesktop specifications <https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html>`__ forbid spaces in .desktop file names.
Repository management
---------------------
For UpdateVMs to access the repository configuration, the package `qubes-repo-templates <https://github.com/WillyPillow/qubes-repo-templates>`__ is created with the following contents:
- ``/etc/qubes/repo-templates/*.repo``: repository configuration
- ``/etc/qubes/repo-templates/keys``: PGP keys
As template keys may be less trusted, they are *not* added to the system RPM keychain but instead managed separately.
Qrexec protocol
---------------
Dom0 and management VMs without network access also need to interact with template repositories. The following qrexec calls that list and download templates are thus proposed.
- ``qubes.TemplateSearch``: wraps ``dnf repoquery``
- ``qubes.TemplateDownload``: wraps ``dnf download``
Input
^^^^^
### Input
Both calls accept the following format from standard input:
```text
arg1
arg2
...
argN
package-file-spec
---
repo config
```
.. code:: bash
arg1
arg2
...
argN
package-file-spec
---
repo config
In other words, the input consists of two parts separated by the line ``---``. The first part contains some arguments and ``package-file-spec`` that indicates the pattern to be queried or downloaded. The following arguments are allowed:
- ``--enablerepo=<repoid>``
- ``--disablerepo=<repoid>``
- ``--repoid=<repoid>``
- ``--releasever=<release>``
- ``--refresh``
In other words, the input consists of two parts separated by the line `---`.
The first part contains some arguments and `package-file-spec` that indicates
the pattern to be queried or downloaded. The following arguments are allowed:
- `--enablerepo=<repoid>`
- `--disablerepo=<repoid>`
- `--repoid=<repoid>`
- `--releasever=<release>`
- `--refresh`
where the usage is identical to that of DNF.
For the exact definition of `package-file-spec`, refer to the DNF
documentation.
For the exact definition of ``package-file-spec``, refer to the DNF documentation.
The second part contains the repository configurations in `yum.repos.d` format.
The second part contains the repository configurations in ``yum.repos.d`` format.
### Output
Output
^^^^^^
`qubes.TemplateSearch` prints each package in
`%{name}|%{epoch}|%{version}|%{release}|%{reponame}|%{downloadsize}|%{buildtime}|%{license}|%{url}|%{summary}|%{description}|`
format to standard output, separated by newlines. Note that there is a `|` at
the end of the line. This is because `%{description}` may contain newlines, and
doing so allows us to split the entries by `|\n`. (As we are using `dnf
repoquery --qf`, we are unable to escape the newlines in advance.)
`qubes.TemplateDownload`, on the other hand, directly outputs the downloaded
content to standard output.
``qubes.TemplateSearch`` prints each package in ``%{name}|%{epoch}|%{version}|%{release}|%{reponame}|%{downloadsize}|%{buildtime}|%{license}|%{url}|%{summary}|%{description}|`` format to standard output, separated by newlines. Note that there is a ``|`` at the end of the line. This is because ``%{description}`` may contain newlines, and doing so allows us to split the entries by ``|\n``. (As we are using ``dnf repoquery --qf``, we are unable to escape the newlines in advance.)
## Machine-readable output
``qubes.TemplateDownload``, on the other hand, directly outputs the downloaded content to standard output.
The commands `qvm-template list` and `qvm-template info` provide
machine-readable output in both pipe(`|`)-separated and JSON format. See the
`qvm-template` man page for details.
Machine-readable output
-----------------------
## Interactions with existing tools
### `qvm-remove`
The commands ``qvm-template list`` and ``qvm-template info`` provide machine-readable output in both pipe(``|``)-separated and JSON format. See the ``qvm-template`` man page for details.
The existing `qvm-remove` tool should behave identically to `qvm-template
remove` -- albeit without fancy features like disassociation. This is unlike
the previous situation where `qvm-remove` cannot remove RPM-installed
templates.
Interactions with existing tools
--------------------------------
Notably, the metadata needs no special handling as it is stored in VM features
and thus automatically consistent.
### Renaming and cloning
``qvm-remove``
^^^^^^^^^^^^^^
A template is treated as non-manager-installed once renamed or cloned. However,
relevant metadata in the VM features is still retained for future extension and
to serve as a hint for the user.
## Further reading
The existing ``qvm-remove`` tool should behave identically to ``qvm-template remove`` – albeit without fancy features like disassociation. This is unlike the previous situation where ``qvm-remove`` cannot remove RPM-installed templates.
Notably, the metadata needs no special handling as it is stored in VM features and thus automatically consistent.
Renaming and cloning
^^^^^^^^^^^^^^^^^^^^
A template is treated as non-manager-installed once renamed or cloned. However, relevant metadata in the VM features is still retained for future extension and to serve as a hint for the user.
Further reading
---------------
Initial Google Summer of Code (2020) project proposal:
- <https://hackmd.io/aYauztkGR0iOIoh8fJLecw>
- https://hackmd.io/aYauztkGR0iOIoh8fJLecw
Previous design document:
- <https://gist.github.com/WillyPillow/b8a643ddbd9235a97bc187e6e44b16e4>
- https://gist.github.com/WillyPillow/b8a643ddbd9235a97bc187e6e44b16e4
Discussion threads:
- <https://groups.google.com/forum/#!topic/qubes-devel/6Zb_WLy3GY4>
- <https://groups.google.com/forum/#!topic/qubes-devel/PyJogqT1TUg>
- <https://groups.google.com/forum/#!topic/qubes-devel/2XaMP4Us3kg>
- <https://groups.google.com/forum/#!topic/qubes-devel/wF_84b1BR0A>
- <https://groups.google.com/forum/#!topic/qubes-devel/pYHnihVCBM0>
- https://groups.google.com/forum/#!topic/qubes-devel/6Zb_WLy3GY4
- https://groups.google.com/forum/#!topic/qubes-devel/PyJogqT1TUg
- https://groups.google.com/forum/#!topic/qubes-devel/2XaMP4Us3kg
- https://groups.google.com/forum/#!topic/qubes-devel/wF_84b1BR0A
- https://groups.google.com/forum/#!topic/qubes-devel/pYHnihVCBM0

73
developer/system/vm-sudo.rst
View file

@ -1,50 +1,53 @@
---
lang: en
layout: doc
permalink: /doc/vm-sudo-implementation/
redirect_from:
- /en/doc/vm-sudo-implementation/
- /doc/VMSudo-implementation/
ref: 341
title: Passwordless root access in qubes
---
=================================
Passwordless root access in qubes
=================================
The rationale behind passwordless root in qubes is set out [here](/doc/vm-sudo). Implementation is by the qubes-core-agent-passwordless-root package.
The rationale behind passwordless root in qubes is set out :doc:`here </user/security-in-qubes/vm-sudo>`. Implementation is by the qubes-core-agent-passwordless-root package.
This page sets out the configuration changes made, with (not necessary complete) list of mechanisms depending on each of them:
1. sudo (`/etc/sudoers.d/qubes`):
1. sudo (``/etc/sudoers.d/qubes``):
```
Defaults !requiretty
%qubes ALL=(ALL) ROLE=unconfined_r TYPE=unconfined_t NOPASSWD: ALL
.. code:: bash
(...)
```
Defaults !requiretty
%qubes ALL=(ALL) ROLE=unconfined_r TYPE=unconfined_t NOPASSWD: ALL
(...)
- Easy user -> root access (main option for the user).
- `qvm-usb` (not really working, as of R2).
2. PolicyKit (`/etc/polkit-1/rules.d/00-qubes-allow-all.rules`):
```
//allow any action, detailed reasoning in sudoers.d/qubes
polkit.addRule(function(action,subject) { if (subject.isInGroup("qubes")) return polkit.Result.YES; });
- Easy user -> root access (main option for the user).
```
- ``qvm-usb`` (not really working, as of R2).
2. PolicyKit (``/etc/polkit-1/rules.d/00-qubes-allow-all.rules``):
.. code:: bash
//allow any action, detailed reasoning in sudoers.d/qubes
polkit.addRule(function(action,subject) { if (subject.isInGroup("qubes")) return polkit.Result.YES; });
PAM (``/etc/pam.d/su.qubes`` or ``/usr/share/pam-configs/su.qubes``) ``auth sufficient pam_succeed_if.so use_uid user ingroup qubes``
- NetworkManager configuration from normal user (``nm-applet``).
- Updates installation (``gpk-update-viewer``).
- User can use pkexec just like sudo **Note:** above is needed mostly because Qubes user GUI session isn’t treated by PolicyKit/logind as “local” session because of the way in which X server and session is started. Perhaps we will address this issue in the future, but this is really low priority. Patches welcomed anyway.
PAM (`/etc/pam.d/su.qubes` or `/usr/share/pam-configs/su.qubes`)
```
auth sufficient pam_succeed_if.so use_uid user ingroup qubes
```
- NetworkManager configuration from normal user (`nm-applet`).
- Updates installation (`gpk-update-viewer`).
- User can use pkexec just like sudo Note: above is needed mostly because Qubes user GUI session isn't treated by PolicyKit/logind as "local" session because of the way in which X server and session is started.
Perhaps we will address this issue in the future, but this is really low priority.
Patches welcomed anyway.
3. Empty root password:
- Used for access to 'root' account from text console (`qvm-console-dispvm`) - the only way to access the VM when GUI isn't working.
- Can be used for easy 'su -' from user to root.
- Used for access to ‘root’ account from text console (``qvm-console-dispvm``) - the only way to access the VM when GUI isn’t working.
- Can be used for easy ‘su -’ from user to root.

17
doc.md
View file

@ -1,17 +0,0 @@
---
lang: en
layout: doc
permalink: /doc/
redirect_from:
- /en/doc/
- /doc/UserDoc/
- /wiki/UserDoc/
- /doc/QubesDocs/
- /wiki/QubesDocs/
- /help/
- /en/help/
- /en/community/
- /community/
ref: 12
title: Documentation
---

12
external/building-guides/building-archlinux-template.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/building-archlinux-template/
- /en/doc/building-archlinux-template/
- /doc/BuildingArchlinuxTemplate/
- /wiki/BuildingArchlinuxTemplate/
redirect_to: https://forum.qubes-os.org/t/19052
ref: 116
title: Building Arch Linux template
---

12
external/building-guides/building-non-fedora-template.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/building-non-fedora-template/
- /en/doc/building-non-fedora-template/
- /doc/BuildingNonFedoraTemplate/
- /wiki/BuildingNonFedoraTemplate/
redirect_to: https://forum.qubes-os.org/t/18972
ref: 117
title: Building non-Fedora template
---

10
external/building-guides/building-whonix-template.md vendored
View file

@ -1,10 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/building-whonix-template/
- /en/doc/building-whonix-template/
redirect_to: https://forum.qubes-os.org/t/18981
ref: 115
title: Building Whonix templates
---

9
external/configuration-guides/change-time-zone.md vendored
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/change-time-zone/
redirect_to: https://forum.qubes-os.org/t/18983
ref: 109
title: Changing your time zone
---

12
external/configuration-guides/disk-trim.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/disk-trim/
- /en/doc/disk-trim/
- /doc/DiskTRIM/
- /wiki/DiskTRIM/
redirect_to: https://forum.qubes-os.org/t/19054
ref: 104
title: Disk TRIM
---

12
external/configuration-guides/external-audio.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/external-audio/
- /en/doc/external-audio/
- /doc/ExternalAudio/
- /wiki/ExternalAudio/
redirect_to: https://forum.qubes-os.org/t/18984
ref: 100
title: External audio
---

12
external/configuration-guides/fetchmail.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/fetchmail/
- /en/doc/fetchmail/
- /doc/Fetchmail/
- /wiki/Fetchmail/
redirect_to: https://forum.qubes-os.org/t/18985
ref: 114
title: Fetchmail
---

12
external/configuration-guides/install-nvidia-driver.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/install-nvidia-driver/
- /en/doc/install-nvidia-driver/
- /doc/InstallNvidiaDriver/
- /wiki/InstallNvidiaDriver/
redirect_to: https://forum.qubes-os.org/t/18987
ref: 96
title: How to install an Nvidia driver
---

7
external/configuration-guides/multiboot.md vendored
View file

@ -1,7 +0,0 @@
---
lang: en
layout: doc
redirect_to: https://forum.qubes-os.org/t/18988
ref: 112
title: Multibooting
---

12
external/configuration-guides/multimedia.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/multimedia/
- /en/doc/multimedia/
- /doc/Multimedia/
- /wiki/Multimedia/
redirect_to: https://forum.qubes-os.org/t/19055
ref: 105
title: How to make a multimedia template
---

12
external/configuration-guides/mutt.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/mutt/
- /en/doc/mutt/
- /doc/Mutt/
- /wiki/Mutt/
redirect_to: https://forum.qubes-os.org/t/18989
ref: 106
title: Mutt
---

12
external/configuration-guides/network-bridge-support.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/network-bridge-support/
- /en/doc/network-bridge-support/
- /doc/NetworkBridgeSupport/
- /wiki/NetworkBridgeSupport/
redirect_to: https://forum.qubes-os.org/t/18990
ref: 113
title: Network bridge support
---

12
external/configuration-guides/network-printer.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/network-printer/
- /en/doc/network-printer/
- /doc/NetworkPrinter/
- /wiki/NetworkPrinter/
redirect_to: https://forum.qubes-os.org/t/19056
ref: 108
title: Network printer
---

12
external/configuration-guides/postfix.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/postfix/
- /en/doc/postfix/
- /doc/Postfix/
- /wiki/Postfix/
redirect_to: https://forum.qubes-os.org/t/18991
ref: 107
title: Postfix
---

12
external/configuration-guides/rxvt.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/rxvt/
- /en/doc/rxvt/
- /doc/Rxvt/
- /wiki/Rxvt/
redirect_to: https://forum.qubes-os.org/t/18992
ref: 103
title: Rxvt
---

13
external/configuration-guides/vpn.md vendored
View file

@ -1,13 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/vpn/
- /doc/privacy/vpn/
- /en/doc/vpn/
- /doc/VPN/
- /wiki/VPN/
redirect_to: https://forum.qubes-os.org/t/19061
ref: 102
title: VPN
---

12
external/configuration-guides/w3m.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/w3m/
- /en/doc/mutt/
- /doc/W3m/
- /wiki/W3m/
redirect_to: https://forum.qubes-os.org/t/18993
ref: 101
title: Reducing the fingerprint of the text-based web browser w3m
---

12
external/configuration-guides/zfs.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/zfs/
- /en/doc/zfs/
- /doc/ZFS/
- /wiki/ZFS/
redirect_to: https://forum.qubes-os.org/t/18994
ref: 111
title: ZFS
---

9
external/customization-guides/dark-theme.md vendored
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/dark-theme/
redirect_to: https://forum.qubes-os.org/t/18997
ref: 74
title: Dark theme
---

9
external/customization-guides/fedora-minimal-template-customization.md vendored
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /en/doc/fedora-minimal-template-customization/
redirect_to: https://forum.qubes-os.org/t/18999
ref: 76
title: Fedora minimal template customization
---

12
external/customization-guides/language-localization.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/language-localization/
- /en/doc/language-localization/
- /doc/LanguageLocalization/
- /wiki/LanguageLocalization/
redirect_to: https://forum.qubes-os.org/t/19001
ref: 73
title: Language localization
---

9
external/customization-guides/removing-templatevm-packages.md vendored
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/removing-templatevm-packages/
redirect_to: https://forum.qubes-os.org/t/19002
ref: 75
title: Removing template packages
---

9
external/customization-guides/windows-template-customization.md vendored
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/windows-template-customization/
redirect_to: https://forum.qubes-os.org/t/19005
ref: 72
title: Windows template customization
---

9
external/os-guides/centos.md vendored
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/templates/centos/
redirect_to: https://forum.qubes-os.org/t/19006
ref: 81
title: CentOS template
---

9
external/os-guides/gentoo.md vendored
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/templates/gentoo/
redirect_to: https://forum.qubes-os.org/t/19007
ref: 221
title: Gentoo template
---

12
external/os-guides/linux-hvm-tips.md vendored
View file

@ -1,12 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/linux-hvm-tips/
- /en/doc/linux-hvm-tips/
- /doc/LinuxHVMTips/
- /wiki/LinuxHVMTips/
redirect_to: https://forum.qubes-os.org/t/19008
ref: 82
title: Linux HVM tips
---

9
external/os-guides/netbsd.md vendored
View file

@ -1,9 +0,0 @@
---
lang: en
layout: doc
redirect_from:
- /doc/netbsd/
redirect_to: https://forum.qubes-os.org/t/19009
ref: 84
title: How to create a NetBSD qube
---

Some files were not shown because too many files have changed in this diff Show more