mirror of
https://github.com/QubesOS/qubes-doc.git
synced 2025-07-27 08:45:58 -04:00
Convert to RST
This is done using tools at https://github.com/maiska/qubes-translation-utilz, commit 4c8e2a7f559fd37e29b51769ed1ab1c6cf92e00d.
This commit is contained in:
Marek Marczykowski-Górecki
parent
e3db139fe3
commit
7e464d0f40
428 changed files with 32833 additions and 29703 deletions
|
|
@ -1,524 +0,0 @@
|
|||
---
|
||||
lang: en
|
||||
layout: doc
|
||||
permalink: /doc/development-workflow/
|
||||
redirect_from:
|
||||
- /en/doc/development-workflow/
|
||||
- /doc/DevelopmentWorkflow/
|
||||
- /wiki/DevelopmentWorkflow/
|
||||
ref: 66
|
||||
title: Development workflow
|
||||
---
|
||||
|
||||
A workflow for developing Qubes OS+
|
||||
|
||||
To begin, setup [QubesBuilder](/doc/qubes-builder-v2/). This guide
|
||||
assumes you're using qubes-builder v2 to build Qubes.
|
||||
|
||||
## Repositories and committing Code
|
||||
|
||||
Qubes source code is split into many git repos. These are all contained in the
|
||||
`artifacts/sources` directory under qubes-builder. Subdirectories there are
|
||||
separate components, stored in separate git repositories.
|
||||
|
||||
The best way to write and contribute code is to create a git repo somewhere
|
||||
(e.g., GitHub) for the repo you are interested in editing (e.g.,
|
||||
`qubes-manager`, `core-agent-linux`, etc). To integrate your repo with the rest
|
||||
of Qubes, cd to the repo directory and add your repository as a remote in git
|
||||
|
||||
**Example:**
|
||||
|
||||
~~~
|
||||
$ cd qubes-builder/artifacts/sources/qubes-manager
|
||||
$ git remote add abel git@GitHub.com:abeluck/qubes-manager.git
|
||||
~~~
|
||||
|
||||
You can then proceed to easily develop in your own branches, pull in new
|
||||
commits from the dev branches, merge them, and eventually push to your own repo.
|
||||
|
||||
When you are ready to submit your changes to Qubes to be merged, push your
|
||||
changes, then create a signed git tag (using `git tag -s`). Finally, send a
|
||||
letter to the Qubes listserv describing the changes, and including a link to
|
||||
your repository. If you are using GitHub you can instead create a pull request.
|
||||
Don't forget to include the public PGP key you use to sign your tags.
|
||||
|
||||
### Kernel-specific notes
|
||||
|
||||
#### Prepare fresh version of kernel sources, with Qubes-specific patches applied
|
||||
|
||||
In `qubes-builder/artifacts/sources/linux-kernel`:
|
||||
|
||||
~~~
|
||||
make prep
|
||||
~~~
|
||||
|
||||
The resulting tree will be in kernel-\<VERSION\>/linux-\<VERSION\>:
|
||||
|
||||
~~~
|
||||
ls -ltrd kernel*/linux*
|
||||
~~~
|
||||
|
||||
~~~
|
||||
drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18
|
||||
drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj
|
||||
~~~
|
||||
|
||||
#### Go to the kernel tree and update the version
|
||||
|
||||
In `qubes-builder/artifacts/sources/linux-kernel`:
|
||||
|
||||
~~~
|
||||
cd kernel-3.4.18/linux-3.4.18
|
||||
~~~
|
||||
|
||||
#### Changing the config
|
||||
|
||||
In `kernel-3.4.18/linux-3.4.18`:
|
||||
|
||||
~~~
|
||||
cp ../../config .config
|
||||
make oldconfig
|
||||
~~~
|
||||
|
||||
Now change the configuration. For example, in `kernel-3.4.18/linux-3.4.18`:
|
||||
|
||||
~~~
|
||||
make menuconfig
|
||||
~~~
|
||||
|
||||
Copy the modified config back into the kernel tree:
|
||||
|
||||
~~~
|
||||
cp .config ../../../config
|
||||
~~~
|
||||
|
||||
#### Patching the code
|
||||
|
||||
TODO: describe the workflow for patching the code, below are some random notes, not working well
|
||||
|
||||
~~~
|
||||
ln -s ../../patches.xen
|
||||
export QUILT_PATCHES=patches.xen
|
||||
export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
|
||||
export QUILT_SERIES=../../series-pvops.conf
|
||||
|
||||
quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch
|
||||
quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \
|
||||
drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \
|
||||
include/xen/interface/io/usbif.h
|
||||
|
||||
*edit something*
|
||||
|
||||
quilt refresh
|
||||
cd ../..
|
||||
vi series.conf
|
||||
~~~
|
||||
|
||||
#### Building RPMs
|
||||
|
||||
Now is a good moment to make sure you have changed the kernel release name in
|
||||
rel file. For example, if you change it to '1debug201211116c' the
|
||||
resulting RPMs will be named
|
||||
'kernel-3.4.18-1debug20121116c.pvops.qubes.x86\_64.rpm'. This will help
|
||||
distinguish between different versions of the same package.
|
||||
|
||||
You might want to take a moment here to review (git diff, git status), commit
|
||||
your changes locally.
|
||||
|
||||
To actually build RPMs, in qubes-builder:
|
||||
|
||||
~~~
|
||||
./qb -c linux-kernel package fetch prep build
|
||||
~~~
|
||||
|
||||
RPMs will appear in
|
||||
`artifacts/repository/destination_name/package_name`
|
||||
(for example `artifacts/repository/host-fc37/linux-kernel-6.6.31-1.1/`
|
||||
|
||||
### Useful [QubesBuilder](/doc/qubes-builder-v2/) commands
|
||||
|
||||
1. `./qb package diff` - show uncommitted changes
|
||||
2. ` ./qb repository check-release-status-for-component` and
|
||||
`./qb repository check-release-status-for-template` - show version of each
|
||||
component/template (based on git tags)
|
||||
3. `./qb package sign` - sign built packages
|
||||
4. `./qb package publish` and `./qb package upload` - publish signed packages
|
||||
and upload published
|
||||
repository
|
||||
|
||||
## Copying Code to dom0
|
||||
|
||||
When developing it is convenient to be able to rapidly test changes. Assuming
|
||||
you're developing Qubes on Qubes, you should be working in a special VM for
|
||||
Qubes and occasionally you will want to transfer code or RPMs back to dom0 for
|
||||
testing.
|
||||
|
||||
Here are some handy scripts Marek has shared to facilitate this.
|
||||
|
||||
You may also like to run your [test environment on separate
|
||||
machine](/doc/test-bench/).
|
||||
|
||||
### Syncing dom0 files
|
||||
|
||||
TODO: edit this script to be more generic
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
set -x
|
||||
set -e
|
||||
|
||||
QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes
|
||||
QUBES_PY=$QUBES_PY_DIR/qubes.py
|
||||
QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py
|
||||
|
||||
qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv
|
||||
cp $QUBES_PY qubes.py.bak$$
|
||||
cp $QUBESUTILS_PY qubesutils.py.bak$$
|
||||
cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$
|
||||
sudo cp qvm-core/qubes.py $QUBES_PY
|
||||
sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY
|
||||
sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/
|
||||
sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/
|
||||
sudo cp misc/vm-template-hvm.conf /usr/share/qubes/
|
||||
sudo cp misc/qubes-start.desktop /usr/share/qubes/
|
||||
sudo cp misc/block-snapshot /etc/xen/scripts/
|
||||
sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/
|
||||
# FIXME(Abel Luck): I hope to
|
||||
~~~
|
||||
|
||||
### Apply qvm-tools
|
||||
|
||||
TODO: make it more generic
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
BAK=qvm-tools.bak$$
|
||||
mkdir -p $BAK
|
||||
cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/
|
||||
sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/
|
||||
~~~
|
||||
|
||||
### Copy from dom0 to an appvm
|
||||
|
||||
~~~
|
||||
#/bin/sh
|
||||
#
|
||||
# usage ./cp-domain <vm_name> <file_to_copy>
|
||||
#
|
||||
domain=$1
|
||||
file=$2
|
||||
fname=`basename $file`
|
||||
|
||||
qvm-run $domain 'mkdir /home/user/incoming/dom0 -p'
|
||||
cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname"
|
||||
~~~
|
||||
|
||||
## Git connection between VMs
|
||||
|
||||
Sometimes it's useful to transfer git commits between VMs. You can use `git
|
||||
format-patch` for that and simply copy the files. But you can also setup
|
||||
custom qrexec service for it.
|
||||
|
||||
Below example assumes that you use `builder-RX` directory in target VM to
|
||||
store sources in qubes-builder layout (where `X` is some number). Make sure that
|
||||
all the scripts are executable.
|
||||
|
||||
Service file (save in `/usr/local/etc/qubes-rpc/local.Git` in target VM):
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
exec 2>/tmp/log2
|
||||
|
||||
read service rel repo
|
||||
echo "Params: $service $rel $repo" >&2
|
||||
# Adjust regexps if needed
|
||||
echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1
|
||||
echo "$rel" | grep -q '^[0-9.]\+$' || exit 1
|
||||
path="/home/user/builder-R$rel/qubes-src/$repo"
|
||||
if [ "$repo" = "builder" ]; then
|
||||
path="/home/user/builder-R$rel"
|
||||
fi
|
||||
case $service in
|
||||
git-receive-pack|git-upload-pack)
|
||||
echo "starting $service $path" >&2
|
||||
exec $service $path
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported service: $service" >&2
|
||||
;;
|
||||
esac
|
||||
~~~
|
||||
|
||||
Client script (save in `~/bin/git-qrexec` in source VM):
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
VMNAME=$1
|
||||
|
||||
(echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git
|
||||
~~~
|
||||
|
||||
You will also need to setup qrexec policy in dom0 (`/etc/qubes-rpc/policy/local.Git`).
|
||||
|
||||
Usage:
|
||||
|
||||
~~~
|
||||
[user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux"
|
||||
[user@source core-agent-linux]$ git push testbuilder master
|
||||
~~~
|
||||
|
||||
You can create `~/bin/add-remote` script to ease adding remotes:
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
[ -n "$1" ] || exit 1
|
||||
|
||||
if [ "$1" = "tb" ]; then
|
||||
git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`"
|
||||
exit $?
|
||||
fi
|
||||
|
||||
git remote add $1 git@GitHub.com:$1/qubes-`basename $PWD`
|
||||
~~~
|
||||
|
||||
It should be executed from component top level directory. This script takes one
|
||||
argument - remote name. If it is `tb`, then it creates qrexec-based git remote
|
||||
to `testbuilder` VM. Otherwise it creates remote pointing at GitHub account of
|
||||
the same name. In any case it points at repository matching current directory
|
||||
name.
|
||||
|
||||
## Sending packages to different VM
|
||||
|
||||
Other useful script(s) can be used to setup local package repository hosted in
|
||||
some VM. This way you can keep your development VM behind firewall, while
|
||||
having an option to expose some yum/apt repository to the local network (to
|
||||
have them installed on test machine).
|
||||
|
||||
To achieve this goal, a dummy repository can be created, which instead of
|
||||
populating metadata locally, will upload the packages to some other VM and
|
||||
trigger repository update there (using qrexec). You can use `unstable`
|
||||
repository flavor, because there is no release managing rules bundled (unlike
|
||||
current and current-testing).
|
||||
|
||||
### RPM packages - yum repo
|
||||
|
||||
In source VM, grab [linux-yum](https://GitHub.com/QubesOS/qubes-linux-yum) repository (below is assumed you've made it in
|
||||
`~/repo-yum-upload` directory) and replace `update_repo.sh` script with:
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
VMNAME=repo-vm
|
||||
|
||||
set -e
|
||||
qvm-copy-to-vm $VMNAME $1
|
||||
# remove only files, leave directory structure
|
||||
find -type f -name '*.rpm' -delete
|
||||
# trigger repo update
|
||||
qrexec-client-vm $VMNAME local.UpdateYum
|
||||
~~~
|
||||
|
||||
In target VM, setup actual yum repository (also based on [linux-yum](https://GitHub.com/QubesOS/qubes-linux-yum), this time
|
||||
without modifications). You will also need to setup some gpg key for signing
|
||||
packages (it is possible to force yum to install unsigned packages, but it
|
||||
isn't possible for `qubes-dom0-update` tool). Fill `~/.rpmmacros` with
|
||||
key description:
|
||||
|
||||
~~~
|
||||
%_gpg_name Test packages signing key
|
||||
~~~
|
||||
|
||||
Then setup `local.UpdateYum` qrexec service (`/usr/local/etc/qubes-rpc/local.UpdateYum`):
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
real_repository=/home/user/linux-yum
|
||||
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
|
||||
|
||||
find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1
|
||||
|
||||
rsync -lr --remove-source-files $incoming/ $real_repository
|
||||
cd $real_repository
|
||||
export SKIP_REPO_CHECK=1
|
||||
if [ -d $incoming/r3.1 ]; then
|
||||
./update_repo-unstable.sh r3.1
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r3.0 ]; then
|
||||
./update_repo-unstable.sh r3.0
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r2 ]; then
|
||||
./update_repo-unstable.sh r2
|
||||
fi
|
||||
find $incoming -type d -empty -delete
|
||||
exit 0
|
||||
~~~
|
||||
|
||||
Of course you will also need to setup qrexec policy in dom0
|
||||
`/etc/qubes-rpc/policy/local.UpdateYum`.
|
||||
|
||||
If you want to access the repository from network, you need to setup HTTP
|
||||
server serving it, and configure the system to let other machines actually
|
||||
reach this HTTP server. You can use for example using [port
|
||||
forwarding](/doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world) or setting up Tor hidden service. Configuration
|
||||
details of those services are outside of the scope of this page.
|
||||
|
||||
Usage: setup `builder.conf` in source VM to use your dummy-uploader repository:
|
||||
|
||||
~~~
|
||||
LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1
|
||||
~~~
|
||||
|
||||
Then use `make update-repo-unstable` to upload the packages. You can also
|
||||
specify selected components on command line, then build them and upload to the
|
||||
repository:
|
||||
|
||||
~~~
|
||||
make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable
|
||||
~~~
|
||||
|
||||
On the test machine, add yum repository (`/etc/yum.repos.d`) pointing at just
|
||||
configured HTTP server. For example:
|
||||
|
||||
~~~
|
||||
[local-test]
|
||||
name=Test
|
||||
baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20
|
||||
~~~
|
||||
|
||||
Remember to also import gpg public key using `rpm --import`.
|
||||
|
||||
### Deb packages - Apt repo
|
||||
|
||||
Steps are mostly the same as in the case of yum repo. The only details that differ:
|
||||
|
||||
- use [linux-deb](https://GitHub.com/QubesOS/qubes-linux-deb) instead of [linux-yum](https://GitHub.com/QubesOS/qubes-linux-yum) as a base - both in source and target VM
|
||||
- use different `update_repo.sh` script in source VM (below)
|
||||
- use `local.UpdateApt` qrexec service in target VM (code below)
|
||||
- in target VM additionally place `update-local-repo.sh` script in repository dir (code below)
|
||||
|
||||
`update_repo.sh` script:
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
set -e
|
||||
|
||||
current_release=$1
|
||||
VMNAME=repo-vm
|
||||
|
||||
qvm-copy-to-vm $VMNAME $1
|
||||
find $current_release -type f -name '*.deb' -delete
|
||||
rm -f $current_release/vm/db/*
|
||||
qrexec-client-vm $VMNAME local.UpdateApt
|
||||
~~~
|
||||
|
||||
`local.UpdateApt` service code (`/usr/local/etc/qubes-rpc/local.UpdateApt` in repo-serving VM):
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
|
||||
|
||||
rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/
|
||||
cd /home/user/linux-deb
|
||||
export SKIP_REPO_CHECK=1
|
||||
if [ -d $incoming/r3.1 ]; then
|
||||
for dist in `ls r3.1/vm/dists`; do
|
||||
./update-local-repo.sh r3.1/vm $dist
|
||||
done
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r3.0 ]; then
|
||||
for dist in `ls r3.0/vm/dists`; do
|
||||
./update-local-repo.sh r3.0/vm $dist
|
||||
done
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r2 ]; then
|
||||
for dist in `ls r2/vm/dists`; do
|
||||
./update-local-repo.sh r2/vm $dist
|
||||
done
|
||||
fi
|
||||
find $incoming -type d -empty -delete
|
||||
exit 0
|
||||
~~~
|
||||
|
||||
`update-local-repo.sh`:
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
set -e
|
||||
|
||||
# Set this to your local repository signing key
|
||||
SIGN_KEY=01ABCDEF
|
||||
|
||||
[ -z "$1" ] && { echo "Usage: $0 <repo> <dist>"; exit 1; }
|
||||
|
||||
REPO_DIR=$1
|
||||
DIST=$2
|
||||
|
||||
if [ "$DIST" = "wheezy-unstable" ]; then
|
||||
DIST_TAG=deb7
|
||||
elif [ "$DIST" = "jessie-unstable" ]; then
|
||||
DIST_TAG=deb8
|
||||
elif [ "$DIST" = "stretch-unstable" ]; then
|
||||
DIST_TAG=deb9
|
||||
fi
|
||||
|
||||
pushd $REPO_DIR
|
||||
mkdir -p dists/$DIST/main/binary-amd64
|
||||
dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages
|
||||
gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz
|
||||
cat > dists/$DIST/Release <<EOF
|
||||
Label: Test repo
|
||||
Suite: $DIST
|
||||
Codename: $DIST
|
||||
Date: `date -R`
|
||||
Architectures: amd64
|
||||
Components: main
|
||||
SHA1:
|
||||
EOF
|
||||
function calc_sha1() {
|
||||
f=dists/$DIST/$1
|
||||
echo -n " "
|
||||
echo -n `sha1sum $f|cut -d' ' -f 1` ""
|
||||
echo -n `stat -c %s $f` ""
|
||||
echo $1
|
||||
}
|
||||
calc_sha1 main/binary-amd64/Packages >> dists/$DIST/Release
|
||||
|
||||
rm -f $DIST/Release.gpg
|
||||
rm -f $DIST/InRelease
|
||||
gpg -abs -u "$SIGN_KEY" \
|
||||
< dists/$DIST/Release > dists/$DIST/Release.gpg
|
||||
gpg -a -s --clearsign -u "$SIGN_KEY" \
|
||||
< dists/$DIST/Release > dists/$DIST/InRelease
|
||||
popd
|
||||
|
||||
if [ `id -u` -eq 0 ]; then
|
||||
chown -R --reference=$REPO_DIR $REPO_DIR
|
||||
fi
|
||||
~~~
|
||||
|
||||
Usage: add this line to `/etc/apt/sources.list` on test machine (adjust host and path):
|
||||
|
||||
~~~
|
||||
deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main
|
||||
~~~
|
||||
559
developer/building/development-workflow.rst
Normal file
559
developer/building/development-workflow.rst
Normal file
|
|
@ -0,0 +1,559 @@
|
|||
====================
|
||||
Development workflow
|
||||
====================
|
||||
|
||||
|
||||
A workflow for developing Qubes OS+
|
||||
|
||||
To begin, setup :doc:`QubesBuilder </developer/building/qubes-builder-v2>`. This guide assumes you’re using qubes-builder v2 to build Qubes.
|
||||
|
||||
Repositories and committing Code
|
||||
--------------------------------
|
||||
|
||||
|
||||
Qubes source code is split into many git repos. These are all contained in the ``artifacts/sources`` directory under qubes-builder. Subdirectories there are separate components, stored in separate git repositories.
|
||||
|
||||
The best way to write and contribute code is to create a git repo somewhere (e.g., GitHub) for the repo you are interested in editing (e.g., ``qubes-manager``, ``core-agent-linux``, etc). To integrate your repo with the rest of Qubes, cd to the repo directory and add your repository as a remote in git
|
||||
|
||||
**Example:**
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ cd qubes-builder/artifacts/sources/qubes-manager
|
||||
$ git remote add abel git@GitHub.com:abeluck/qubes-manager.git
|
||||
|
||||
|
||||
|
||||
You can then proceed to easily develop in your own branches, pull in new commits from the dev branches, merge them, and eventually push to your own repo.
|
||||
|
||||
When you are ready to submit your changes to Qubes to be merged, push your changes, then create a signed git tag (using ``git tag -s``). Finally, send a letter to the Qubes listserv describing the changes, and including a link to your repository. If you are using GitHub you can instead create a pull request. Don’t forget to include the public PGP key you use to sign your tags.
|
||||
|
||||
Kernel-specific notes
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
Prepare fresh version of kernel sources, with Qubes-specific patches applied
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
In ``qubes-builder/artifacts/sources/linux-kernel``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make prep
|
||||
|
||||
|
||||
|
||||
The resulting tree will be in kernel-<VERSION>/linux-<VERSION>:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
ls -ltrd kernel*/linux*
|
||||
|
||||
|
||||
|
||||
.. code:: bash
|
||||
|
||||
drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18
|
||||
drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj
|
||||
|
||||
|
||||
|
||||
Go to the kernel tree and update the version
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
In ``qubes-builder/artifacts/sources/linux-kernel``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
cd kernel-3.4.18/linux-3.4.18
|
||||
|
||||
|
||||
|
||||
Changing the config
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
In ``kernel-3.4.18/linux-3.4.18``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
cp ../../config .config
|
||||
make oldconfig
|
||||
|
||||
|
||||
|
||||
Now change the configuration. For example, in ``kernel-3.4.18/linux-3.4.18``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make menuconfig
|
||||
|
||||
|
||||
|
||||
Copy the modified config back into the kernel tree:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
cp .config ../../../config
|
||||
|
||||
|
||||
|
||||
Patching the code
|
||||
^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
TODO: describe the workflow for patching the code, below are some random notes, not working well
|
||||
|
||||
.. code:: bash
|
||||
|
||||
ln -s ../../patches.xen
|
||||
export QUILT_PATCHES=patches.xen
|
||||
export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
|
||||
export QUILT_SERIES=../../series-pvops.conf
|
||||
|
||||
quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch
|
||||
quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \
|
||||
drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \
|
||||
include/xen/interface/io/usbif.h
|
||||
|
||||
*edit something*
|
||||
|
||||
quilt refresh
|
||||
cd ../..
|
||||
vi series.conf
|
||||
|
||||
|
||||
|
||||
Building RPMs
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
|
||||
Now is a good moment to make sure you have changed the kernel release name in rel file. For example, if you change it to ‘1debug201211116c’ the resulting RPMs will be named ‘kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm’. This will help distinguish between different versions of the same package.
|
||||
|
||||
You might want to take a moment here to review (git diff, git status), commit your changes locally.
|
||||
|
||||
To actually build RPMs, in qubes-builder:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
./qb -c linux-kernel package fetch prep build
|
||||
|
||||
|
||||
|
||||
RPMs will appear in ``artifacts/repository/destination_name/package_name`` (for example ``artifacts/repository/host-fc37/linux-kernel-6.6.31-1.1/``
|
||||
|
||||
Useful :doc:`QubesBuilder </developer/building/qubes-builder-v2>` commands
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
1. ``./qb package diff`` - show uncommitted changes
|
||||
|
||||
2. ``./qb repository check-release-status-for-component`` and ``./qb repository check-release-status-for-template`` - show version of each component/template (based on git tags)
|
||||
|
||||
3. ``./qb package sign`` - sign built packages
|
||||
|
||||
4. ``./qb package publish`` and ``./qb package upload`` - publish signed packages and upload published repository
|
||||
|
||||
|
||||
|
||||
Copying Code to dom0
|
||||
--------------------
|
||||
|
||||
|
||||
When developing it is convenient to be able to rapidly test changes. Assuming you’re developing Qubes on Qubes, you should be working in a special VM for Qubes and occasionally you will want to transfer code or RPMs back to dom0 for testing.
|
||||
|
||||
Here are some handy scripts Marek has shared to facilitate this.
|
||||
|
||||
You may also like to run your :doc:`test environment on separate machine </developer/debugging/test-bench>`.
|
||||
|
||||
Syncing dom0 files
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
TODO: edit this script to be more generic
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
set -x
|
||||
set -e
|
||||
|
||||
QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes
|
||||
QUBES_PY=$QUBES_PY_DIR/qubes.py
|
||||
QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py
|
||||
|
||||
qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv
|
||||
cp $QUBES_PY qubes.py.bak$$
|
||||
cp $QUBESUTILS_PY qubesutils.py.bak$$
|
||||
cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$
|
||||
sudo cp qvm-core/qubes.py $QUBES_PY
|
||||
sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY
|
||||
sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/
|
||||
sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/
|
||||
sudo cp misc/vm-template-hvm.conf /usr/share/qubes/
|
||||
sudo cp misc/qubes-start.desktop /usr/share/qubes/
|
||||
sudo cp misc/block-snapshot /etc/xen/scripts/
|
||||
sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/
|
||||
# FIXME(Abel Luck): I hope to
|
||||
|
||||
|
||||
|
||||
Apply qvm-tools
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
TODO: make it more generic
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
BAK=qvm-tools.bak$$
|
||||
mkdir -p $BAK
|
||||
cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/
|
||||
sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/
|
||||
|
||||
|
||||
|
||||
Copy from dom0 to an appvm
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#/bin/sh
|
||||
#
|
||||
# usage ./cp-domain <vm_name> <file_to_copy>
|
||||
#
|
||||
domain=$1
|
||||
file=$2
|
||||
fname=`basename $file`
|
||||
|
||||
qvm-run $domain 'mkdir /home/user/incoming/dom0 -p'
|
||||
cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname"
|
||||
|
||||
|
||||
|
||||
Git connection between VMs
|
||||
--------------------------
|
||||
|
||||
|
||||
Sometimes it’s useful to transfer git commits between VMs. You can use ``git format-patch`` for that and simply copy the files. But you can also setup custom qrexec service for it.
|
||||
|
||||
Below example assumes that you use ``builder-RX`` directory in target VM to store sources in qubes-builder layout (where ``X`` is some number). Make sure that all the scripts are executable.
|
||||
|
||||
Service file (save in ``/usr/local/etc/qubes-rpc/local.Git`` in target VM):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
exec 2>/tmp/log2
|
||||
|
||||
read service rel repo
|
||||
echo "Params: $service $rel $repo" >&2
|
||||
# Adjust regexps if needed
|
||||
echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1
|
||||
echo "$rel" | grep -q '^[0-9.]\+$' || exit 1
|
||||
path="/home/user/builder-R$rel/qubes-src/$repo"
|
||||
if [ "$repo" = "builder" ]; then
|
||||
path="/home/user/builder-R$rel"
|
||||
fi
|
||||
case $service in
|
||||
git-receive-pack|git-upload-pack)
|
||||
echo "starting $service $path" >&2
|
||||
exec $service $path
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported service: $service" >&2
|
||||
;;
|
||||
esac
|
||||
|
||||
|
||||
|
||||
Client script (save in ``~/bin/git-qrexec`` in source VM):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
VMNAME=$1
|
||||
|
||||
(echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git
|
||||
|
||||
|
||||
|
||||
You will also need to setup qrexec policy in dom0 (``/etc/qubes-rpc/policy/local.Git``).
|
||||
|
||||
Usage:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
[user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux"
|
||||
[user@source core-agent-linux]$ git push testbuilder master
|
||||
|
||||
|
||||
|
||||
You can create ``~/bin/add-remote`` script to ease adding remotes:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
[ -n "$1" ] || exit 1
|
||||
|
||||
if [ "$1" = "tb" ]; then
|
||||
git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`"
|
||||
exit $?
|
||||
fi
|
||||
|
||||
git remote add $1 git@GitHub.com:$1/qubes-`basename $PWD`
|
||||
|
||||
|
||||
|
||||
It should be executed from component top level directory. This script takes one argument - remote name. If it is ``tb``, then it creates qrexec-based git remote to ``testbuilder`` VM. Otherwise it creates remote pointing at GitHub account of the same name. In any case it points at repository matching current directory name.
|
||||
|
||||
Sending packages to different VM
|
||||
--------------------------------
|
||||
|
||||
|
||||
Other useful script(s) can be used to setup local package repository hosted in some VM. This way you can keep your development VM behind firewall, while having an option to expose some yum/apt repository to the local network (to have them installed on test machine).
|
||||
|
||||
To achieve this goal, a dummy repository can be created, which instead of populating metadata locally, will upload the packages to some other VM and trigger repository update there (using qrexec). You can use ``unstable`` repository flavor, because there is no release managing rules bundled (unlike current and current-testing).
|
||||
|
||||
RPM packages - yum repo
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
In source VM, grab `linux-yum <https://GitHub.com/QubesOS/qubes-linux-yum>`__ repository (below is assumed you’ve made it in ``~/repo-yum-upload`` directory) and replace ``update_repo.sh`` script with:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
VMNAME=repo-vm
|
||||
|
||||
set -e
|
||||
qvm-copy-to-vm $VMNAME $1
|
||||
# remove only files, leave directory structure
|
||||
find -type f -name '*.rpm' -delete
|
||||
# trigger repo update
|
||||
qrexec-client-vm $VMNAME local.UpdateYum
|
||||
|
||||
|
||||
|
||||
In target VM, setup actual yum repository (also based on `linux-yum <https://GitHub.com/QubesOS/qubes-linux-yum>`__, this time without modifications). You will also need to setup some gpg key for signing packages (it is possible to force yum to install unsigned packages, but it isn’t possible for ``qubes-dom0-update`` tool). Fill ``~/.rpmmacros`` with key description:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
%_gpg_name Test packages signing key
|
||||
|
||||
|
||||
|
||||
Then setup ``local.UpdateYum`` qrexec service (``/usr/local/etc/qubes-rpc/local.UpdateYum``):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
real_repository=/home/user/linux-yum
|
||||
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
|
||||
|
||||
find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1
|
||||
|
||||
rsync -lr --remove-source-files $incoming/ $real_repository
|
||||
cd $real_repository
|
||||
export SKIP_REPO_CHECK=1
|
||||
if [ -d $incoming/r3.1 ]; then
|
||||
./update_repo-unstable.sh r3.1
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r3.0 ]; then
|
||||
./update_repo-unstable.sh r3.0
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r2 ]; then
|
||||
./update_repo-unstable.sh r2
|
||||
fi
|
||||
find $incoming -type d -empty -delete
|
||||
exit 0
|
||||
|
||||
|
||||
|
||||
Of course you will also need to setup qrexec policy in dom0 ``/etc/qubes-rpc/policy/local.UpdateYum``.
|
||||
|
||||
If you want to access the repository from network, you need to setup HTTP server serving it, and configure the system to let other machines actually reach this HTTP server. You can use for example using :ref:`port forwarding <user/security-in-qubes/firewall:port forwarding to a qube from the outside world>` or setting up Tor hidden service. Configuration details of those services are outside of the scope of this page.
|
||||
|
||||
Usage: setup ``builder.conf`` in source VM to use your dummy-uploader repository:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1
|
||||
|
||||
|
||||
|
||||
Then use ``make update-repo-unstable`` to upload the packages. You can also specify selected components on command line, then build them and upload to the repository:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable
|
||||
|
||||
|
||||
|
||||
On the test machine, add yum repository (``/etc/yum.repos.d``) pointing at just configured HTTP server. For example:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
[local-test]
|
||||
name=Test
|
||||
baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20
|
||||
|
||||
|
||||
|
||||
Remember to also import gpg public key using ``rpm --import``.
|
||||
|
||||
Deb packages - Apt repo
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
Steps are mostly the same as in the case of yum repo. The only details that differ:
|
||||
|
||||
- use `linux-deb <https://GitHub.com/QubesOS/qubes-linux-deb>`__ instead of `linux-yum <https://GitHub.com/QubesOS/qubes-linux-yum>`__ as a base - both in source and target VM
|
||||
|
||||
- use different ``update_repo.sh`` script in source VM (below)
|
||||
|
||||
- use ``local.UpdateApt`` qrexec service in target VM (code below)
|
||||
|
||||
- in target VM additionally place ``update-local-repo.sh`` script in repository dir (code below)
|
||||
|
||||
|
||||
|
||||
``update_repo.sh`` script:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
set -e
|
||||
|
||||
current_release=$1
|
||||
VMNAME=repo-vm
|
||||
|
||||
qvm-copy-to-vm $VMNAME $1
|
||||
find $current_release -type f -name '*.deb' -delete
|
||||
rm -f $current_release/vm/db/*
|
||||
qrexec-client-vm $VMNAME local.UpdateApt
|
||||
|
||||
|
||||
|
||||
``local.UpdateApt`` service code (``/usr/local/etc/qubes-rpc/local.UpdateApt`` in repo-serving VM):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
|
||||
|
||||
rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/
|
||||
cd /home/user/linux-deb
|
||||
export SKIP_REPO_CHECK=1
|
||||
if [ -d $incoming/r3.1 ]; then
|
||||
for dist in `ls r3.1/vm/dists`; do
|
||||
./update-local-repo.sh r3.1/vm $dist
|
||||
done
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r3.0 ]; then
|
||||
for dist in `ls r3.0/vm/dists`; do
|
||||
./update-local-repo.sh r3.0/vm $dist
|
||||
done
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r2 ]; then
|
||||
for dist in `ls r2/vm/dists`; do
|
||||
./update-local-repo.sh r2/vm $dist
|
||||
done
|
||||
fi
|
||||
find $incoming -type d -empty -delete
|
||||
exit 0
|
||||
|
||||
|
||||
|
||||
``update-local-repo.sh``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
#!/bin/sh
|
||||
|
||||
set -e
|
||||
|
||||
# Set this to your local repository signing key
|
||||
SIGN_KEY=01ABCDEF
|
||||
|
||||
[ -z "$1" ] && { echo "Usage: $0 <repo> <dist>"; exit 1; }
|
||||
|
||||
REPO_DIR=$1
|
||||
DIST=$2
|
||||
|
||||
if [ "$DIST" = "wheezy-unstable" ]; then
|
||||
DIST_TAG=deb7
|
||||
elif [ "$DIST" = "jessie-unstable" ]; then
|
||||
DIST_TAG=deb8
|
||||
elif [ "$DIST" = "stretch-unstable" ]; then
|
||||
DIST_TAG=deb9
|
||||
fi
|
||||
|
||||
pushd $REPO_DIR
|
||||
mkdir -p dists/$DIST/main/binary-amd64
|
||||
dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages
|
||||
gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz
|
||||
cat > dists/$DIST/Release <<EOF
|
||||
Label: Test repo
|
||||
Suite: $DIST
|
||||
Codename: $DIST
|
||||
Date: `date -R`
|
||||
Architectures: amd64
|
||||
Components: main
|
||||
SHA1:
|
||||
EOF
|
||||
function calc_sha1() {
|
||||
f=dists/$DIST/$1
|
||||
echo -n " "
|
||||
echo -n `sha1sum $f|cut -d' ' -f 1` ""
|
||||
echo -n `stat -c %s $f` ""
|
||||
echo $1
|
||||
}
|
||||
calc_sha1 main/binary-amd64/Packages >> dists/$DIST/Release
|
||||
|
||||
rm -f $DIST/Release.gpg
|
||||
rm -f $DIST/InRelease
|
||||
gpg -abs -u "$SIGN_KEY" \
|
||||
< dists/$DIST/Release > dists/$DIST/Release.gpg
|
||||
gpg -a -s --clearsign -u "$SIGN_KEY" \
|
||||
< dists/$DIST/Release > dists/$DIST/InRelease
|
||||
popd
|
||||
|
||||
if [ `id -u` -eq 0 ]; then
|
||||
chown -R --reference=$REPO_DIR $REPO_DIR
|
||||
fi
|
||||
|
||||
|
||||
|
||||
Usage: add this line to ``/etc/apt/sources.list`` on test machine (adjust host and path):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main
|
||||
|
||||
|
||||
|
|
@ -1,59 +0,0 @@
|
|||
---
|
||||
lang: en
|
||||
layout: doc
|
||||
permalink: /doc/qubes-builder-details/
|
||||
redirect_from:
|
||||
- /en/doc/qubes-builder-details/
|
||||
- /doc/QubesBuilderDetails/
|
||||
- /wiki/QubesBuilderDetails/
|
||||
ref: 65
|
||||
title: Qubes builder details
|
||||
---
|
||||
|
||||
|
||||
<div class="alert alert-warning" role="alert">
|
||||
<i class="fa fa-exclamation-circle"></i>
|
||||
<b>Note:</b> This information concerns the old Qubes builder (v1). It supports
|
||||
only building Qubes 4.1 or earlier.<br>The build process has been completely rewritten in <a href="https://github.com/QubesOS/qubes-builderv2/">qubes-builder v2</a>. This can be used for building Qubes R4.1 and later, and all related components.</div>
|
||||
|
||||
Components Makefile.builder file
|
||||
--------------------------------
|
||||
|
||||
[QubesBuilder](/doc/qubes-builder/) expects that each component have *Makefile.builder* file in its root directory. This file specifies what should be done to build the package. As name suggests, this is normal makefile, which is included by builder as its configuration. Its main purpose is to set some variables. Generally all available variables/settings are described as comments at the beginning of Makefile.\* in [QubesBuilder](/doc/qubes-builder/).
|
||||
|
||||
Variables for Linux build:
|
||||
|
||||
- `RPM_SPEC_FILES` List (space separated) of spec files for RPM package build. Path should be relative to component root directory. [QubesBuilder](/doc/qubes-builder/) will install all BuildRequires (in chroot environment) before the build. In most Qubes components all spec files are kept in *rpm\_spec* directory. This is mainly used for Fedora packages build.
|
||||
- `ARCH_BUILD_DIRS` List (space separated) of directories with PKGBUILD files for Archlinux package build. Similar to RPM build, [QubesBuilder](/doc/qubes-builder/) will install all makedepends, then build the package.
|
||||
|
||||
Most components uses *archlinux* directory for this purpose, so its good to keep this style.
|
||||
|
||||
Variables for Windows build:
|
||||
|
||||
- `WIN_COMPILER` Choose which compiler should be used for this component, thus which build scripts. Currently two options available:
|
||||
- `WDK` - Windows Driver Kit (default). Command used to build: *build -cZg*.
|
||||
- `mingw` - MinGW (Windows gcc port). Command used to build: *make all*
|
||||
- `WIN_SOURCE_SUBDIRS` List of directories in which above command should be run. In most cases it will be only one entry: current directory (*.*).
|
||||
- `WIN_PREBUILD_CMD` Command to run before build, mostly useful for WDK build (in mingw case, you can use makefile for this purpose). Can be used to set some variables, preprocess some files etc.
|
||||
- `WIN_SIGN_CMD` Command used to sign resulting binaries. Note that default value is *sign.bat*. If you don't want to sign binaries, specify some placeholder here (eg. *true*). Check existing components (eg. vmm-xen-windows-pvdrivers) for example scripts. This command will be run with certain environment variables:
|
||||
- `CERT_FILENAME` Path to key file (pfx format)
|
||||
- `CERT_PASSWORD` Key password
|
||||
- `CERT_PUBLIC_FILENAME` Certificate path in the case of self-signed cert
|
||||
- `CERT_CROSS_CERT_FILENAME` Certificate path in the case of correct autheticode cert
|
||||
- `SIGNTOOL` Path to signtool
|
||||
- `WIN_PACKAGE_CMD` Command used to produce installation package (msi or msm). Default value is *wix.bat*, similar to above - use *true* if you don't want this command.
|
||||
- `WIN_OUTPUT_HEADERS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with public headers of the package - for use in other components.
|
||||
- `WIN_OUTPUT_LIBS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with libraries (both DLL and implib) of the package - for use in other components. Note that [QubesBuilder](/doc/qubes-builder/) will copy files specified as *\$(WIN\_OUTPUT\_LIBS)/\*/\** to match WDK directory layout (*\<specified directory\>/\<arch directory\>/\<actual libraries\>*), so you in mingw build you need to place libraries in some additional subdirectory.
|
||||
- `WIN_BUILD_DEPS` List of components required to build this one. [QubesBuilder](/doc/qubes-builder/) will copy files specified with `WIN_OUTPUT_HEADERS` and `WIN_OUTPUT_LIBS` of those components to some directory and provide its path with `QUBES_INCLUDES` and `QUBES_LIBS` variables. Use those variables in your build scripts (*sources* or *Makefile* - depending on selected compiler). You can assume that the variables are always set and directories always exists, even if empty.
|
||||
|
||||
builder.conf settings
|
||||
---------------------
|
||||
|
||||
Most settings are documented in *builder.conf.default* file, which can be used as template the actual configuration.
|
||||
|
||||
**TODO**
|
||||
|
||||
Notes
|
||||
-----
|
||||
|
||||
* For a list of custom TemplateVMs available in QubesBuilder look at [Supported Versions page](/doc/supported-releases/).
|
||||
78
developer/building/qubes-builder-details.rst
Normal file
78
developer/building/qubes-builder-details.rst
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
=====================
|
||||
Qubes builder details
|
||||
=====================
|
||||
|
||||
|
||||
.. warning::
|
||||
|
||||
**Note:** This information concerns the old Qubes builder (v1). It supports only building Qubes 4.1 or earlier.The build process has been completely rewritten in `qubes-builder v2 <https://github.com/QubesOS/qubes-builderv2/>`__ . This can be used for building Qubes R4.1 and later, and all related components.
|
||||
|
||||
Components Makefile.builder file
|
||||
--------------------------------
|
||||
|
||||
|
||||
:doc:`QubesBuilder </developer/building/qubes-builder>` expects that each component have *Makefile.builder* file in its root directory. This file specifies what should be done to build the package. As name suggests, this is normal makefile, which is included by builder as its configuration. Its main purpose is to set some variables. Generally all available variables/settings are described as comments at the beginning of Makefile.* in :doc:`QubesBuilder </developer/building/qubes-builder>`.
|
||||
|
||||
Variables for Linux build:
|
||||
|
||||
- ``RPM_SPEC_FILES`` List (space separated) of spec files for RPM package build. Path should be relative to component root directory. :doc:`QubesBuilder </developer/building/qubes-builder>` will install all BuildRequires (in chroot environment) before the build. In most Qubes components all spec files are kept in *rpm_spec* directory. This is mainly used for Fedora packages build.
|
||||
|
||||
- ``ARCH_BUILD_DIRS`` List (space separated) of directories with PKGBUILD files for Archlinux package build. Similar to RPM build, :doc:`QubesBuilder </developer/building/qubes-builder>` will install all makedepends, then build the package.
|
||||
|
||||
|
||||
|
||||
Most components uses *archlinux* directory for this purpose, so its good to keep this style.
|
||||
|
||||
Variables for Windows build:
|
||||
|
||||
- ``WIN_COMPILER`` Choose which compiler should be used for this component, thus which build scripts. Currently two options available:
|
||||
|
||||
- ``WDK`` - Windows Driver Kit (default). Command used to build: *build -cZg*.
|
||||
|
||||
- ``mingw`` - MinGW (Windows gcc port). Command used to build: *make all*
|
||||
|
||||
|
||||
|
||||
- ``WIN_SOURCE_SUBDIRS`` List of directories in which above command should be run. In most cases it will be only one entry: current directory (*.*).
|
||||
|
||||
- ``WIN_PREBUILD_CMD`` Command to run before build, mostly useful for WDK build (in mingw case, you can use makefile for this purpose). Can be used to set some variables, preprocess some files etc.
|
||||
|
||||
- ``WIN_SIGN_CMD`` Command used to sign resulting binaries. Note that default value is *sign.bat*. If you don’t want to sign binaries, specify some placeholder here (eg. *true*). Check existing components (eg. vmm-xen-windows-pvdrivers) for example scripts. This command will be run with certain environment variables:
|
||||
|
||||
- ``CERT_FILENAME`` Path to key file (pfx format)
|
||||
|
||||
- ``CERT_PASSWORD`` Key password
|
||||
|
||||
- ``CERT_PUBLIC_FILENAME`` Certificate path in the case of self-signed cert
|
||||
|
||||
- ``CERT_CROSS_CERT_FILENAME`` Certificate path in the case of correct autheticode cert
|
||||
|
||||
- ``SIGNTOOL`` Path to signtool
|
||||
|
||||
|
||||
|
||||
- ``WIN_PACKAGE_CMD`` Command used to produce installation package (msi or msm). Default value is *wix.bat*, similar to above - use *true* if you don’t want this command.
|
||||
|
||||
- ``WIN_OUTPUT_HEADERS`` Directory (relative to ``WIN_SOURCE_SUBDIRS`` element) with public headers of the package - for use in other components.
|
||||
|
||||
- ``WIN_OUTPUT_LIBS`` Directory (relative to ``WIN_SOURCE_SUBDIRS`` element) with libraries (both DLL and implib) of the package - for use in other components. Note that :doc:`QubesBuilder </developer/building/qubes-builder>` will copy files specified as *$(WIN_OUTPUT_LIBS)/\*/\** to match WDK directory layout (*<specified directory>/<arch directory>/<actual libraries>*), so you in mingw build you need to place libraries in some additional subdirectory.
|
||||
|
||||
- ``WIN_BUILD_DEPS`` List of components required to build this one. :doc:`QubesBuilder </developer/building/qubes-builder>` will copy files specified with ``WIN_OUTPUT_HEADERS`` and ``WIN_OUTPUT_LIBS`` of those components to some directory and provide its path with ``QUBES_INCLUDES`` and ``QUBES_LIBS`` variables. Use those variables in your build scripts (*sources* or *Makefile* - depending on selected compiler). You can assume that the variables are always set and directories always exists, even if empty.
|
||||
|
||||
|
||||
|
||||
builder.conf settings
|
||||
---------------------
|
||||
|
||||
|
||||
Most settings are documented in *builder.conf.default* file, which can be used as template the actual configuration.
|
||||
|
||||
**TODO**
|
||||
|
||||
Notes
|
||||
-----
|
||||
|
||||
|
||||
- For a list of custom TemplateVMs available in QubesBuilder look at :doc:`Supported Versions page </user/downloading-installing-upgrading/supported-releases>`.
|
||||
|
||||
|
||||
|
|
@ -1,166 +0,0 @@
|
|||
---
|
||||
lang: en
|
||||
layout: doc
|
||||
permalink: /doc/qubes-builder-v2/
|
||||
redirect_from:
|
||||
- /en/doc/qubes-builder-v2/
|
||||
- /doc/QubesBuilder2/
|
||||
- /wiki/QubesBuilder2/
|
||||
ref: 311
|
||||
title: Qubes builder v2
|
||||
---
|
||||
|
||||
This is a brief introduction to using Qubes Builder v2 to work with Qubes OS
|
||||
sources. It will walk you through installing and configuring Builder v2, and
|
||||
using it to fetch and build Qubes OS packages.
|
||||
|
||||
For details and customization, use [Qubes OS v2 builder documentation](https://github.com/QubesOS/qubes-builderv2/).
|
||||
|
||||
# Overview
|
||||
|
||||
In the second generation of Qubes OS builder, container or disposable qube
|
||||
isolation is used to perform every stage of the build and release process.
|
||||
From fetching sources to building, everything is executed inside an isolated
|
||||
*cage* (either a disposable or a container) using an *executor*. For every
|
||||
command that needs to perform an action on sources, like cloning and
|
||||
verifying Git repos, rendering a SPEC file, generating SRPM or Debian
|
||||
source packages, a new cage is used. Only the signing, publishing, and
|
||||
uploading stages are executed locally outside a cage.
|
||||
|
||||
|
||||
# Setup
|
||||
|
||||
This is a simple setup using a docker executor. This is a good default choice;
|
||||
if you don't know which executor to use, use docker.
|
||||
|
||||
1. First, decide what qube you are going to use when working with Qubes
|
||||
Builder v2. It can be an AppVM or a Standalone qube, with some steps
|
||||
different between the two.
|
||||
|
||||
2. Installing dependencies
|
||||
|
||||
- If you want to use an app qube for developing, install dependencies in the template.
|
||||
If you are using a standalone, install them in the qube itself.
|
||||
Dependencies are specified in `dependencies-*.
|
||||
txt` files in the main builder directory, and you can install them easily
|
||||
in the following ways:
|
||||
1. for Fedora, use:
|
||||
|
||||
```shell
|
||||
$ sudo dnf install $(cat dependencies-fedora.txt)
|
||||
$ test -f /usr/share/qubes/marker-vm && sudo dnf install qubes-gpg-split
|
||||
```
|
||||
2. for Debian (note: some Debian packages require Debian version 13 or
|
||||
later), use:
|
||||
|
||||
```shell
|
||||
$ sudo apt install $(cat dependencies-debian.txt)
|
||||
$ test -f /usr/share/qubes/marker-vm && sudo apt install qubes-gpg-split
|
||||
```
|
||||
|
||||
If you have installed dependencies in the template, close it, and
|
||||
(re)start the development qube.
|
||||
|
||||
3. Clone the qubes-builder v2 repository into a location of your choice:
|
||||
|
||||
```shell
|
||||
git clone https://github.com/QubesOS/qubes-builderv2
|
||||
cd qubes-builderv2/
|
||||
```
|
||||
|
||||
4. If you haven't previously used docker in the current qube, you need to set up
|
||||
some permissions. In particular, the user has to be added to the `docker`
|
||||
group:
|
||||
|
||||
```shell
|
||||
$ sudo usermod -aG docker user
|
||||
```
|
||||
Next, **restart the qube**.
|
||||
|
||||
5. Finally, you need to generate a docker image:
|
||||
|
||||
```shell
|
||||
$ tools/generate-container-image.sh docker
|
||||
```
|
||||
|
||||
In an app qube, as `/var/lib/docker` is not persistent by default, you also
|
||||
need to use [bind-dirs](/doc/bind-dirs/) to avoid repeating this step after reboot, adding
|
||||
the following to the `/rw/config/qubes-bind-dirs.d/docker.conf` file in
|
||||
this qube:
|
||||
|
||||
```
|
||||
binds+=( '/var/lib/docker' )
|
||||
```
|
||||
|
||||
# Configuration
|
||||
|
||||
To use Qubes OS Builder v2, you need to have a `builder.yml` configuration file.
|
||||
You can use one of the sample files from the `example-configs/` directory; for a
|
||||
more readable `builder.yml`, you can also include one of the files from that
|
||||
directory in your `builder.yml`. An example `builder.yml` is:
|
||||
|
||||
```
|
||||
# include configuration relevant for the current release
|
||||
include:
|
||||
- example-configs/qubes-os-r4.2.yml
|
||||
|
||||
# which repository to use to fetch sources
|
||||
use-qubes-repo:
|
||||
version: 4.2
|
||||
testing: true
|
||||
|
||||
# each package built will have local build number appended to package release
|
||||
# number. It makes it easier to update in testing environment
|
||||
increment-devel-versions: true
|
||||
|
||||
# reduce output
|
||||
debug: false
|
||||
|
||||
# this can be set to true if you do not want sources to be automatically
|
||||
# fetched from git
|
||||
skip-git-fetch: false
|
||||
|
||||
# executor configuration
|
||||
executor:
|
||||
type: docker
|
||||
options:
|
||||
image: "qubes-builder-fedora:latest"
|
||||
```
|
||||
|
||||
|
||||
# Using Builder v2
|
||||
|
||||
To fetch sources - in this example, for the `core-admin-client` package, you
|
||||
can use the following command:
|
||||
|
||||
```shell
|
||||
$ ./qb -c core-admin-client package fetch
|
||||
```
|
||||
|
||||
This will fetch the sources for the listed package and place them in
|
||||
`artifacts/sources` directory.
|
||||
|
||||
To build a package (from sources in the `artifacts/sources` directory), use:
|
||||
|
||||
```shell
|
||||
$ ./qb -c core-admin-client package fetch prep build
|
||||
```
|
||||
|
||||
or, if you want to build for a specific target (`host-fc37` is a `dom0`
|
||||
using Fedora 37, `vm-fc40` would be a qube using Fedora 40 etc.), use:
|
||||
|
||||
```shell
|
||||
$ ./qb -c core-admin-client -d host-fc37 package fetch prep build
|
||||
```
|
||||
|
||||
If you want to fetch the entire Qubes OS source use the following:
|
||||
|
||||
```shell
|
||||
$ ./qb package fetch
|
||||
```
|
||||
|
||||
**caution**: some repositories might have additional requirements. You can
|
||||
disable repositories that are not needed in the `example-configs/*.yml`
|
||||
file you are using by commenting them out. In particular, `python-fido2`,
|
||||
`lvm` and `windows`-related repositories have special requirements.
|
||||
|
||||
155
developer/building/qubes-builder-v2.rst
Normal file
155
developer/building/qubes-builder-v2.rst
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
================
|
||||
Qubes builder v2
|
||||
================
|
||||
|
||||
|
||||
This is a brief introduction to using Qubes Builder v2 to work with Qubes OS sources. It will walk you through installing and configuring Builder v2, and using it to fetch and build Qubes OS packages.
|
||||
|
||||
For details and customization, use `Qubes OS v2 builder documentation <https://github.com/QubesOS/qubes-builderv2/>`__.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
|
||||
In the second generation of Qubes OS builder, container or disposable qube isolation is used to perform every stage of the build and release process. From fetching sources to building, everything is executed inside an isolated *cage* (either a disposable or a container) using an *executor*. For every command that needs to perform an action on sources, like cloning and verifying Git repos, rendering a SPEC file, generating SRPM or Debian source packages, a new cage is used. Only the signing, publishing, and uploading stages are executed locally outside a cage.
|
||||
|
||||
Setup
|
||||
-----
|
||||
|
||||
|
||||
This is a simple setup using a docker executor. This is a good default choice; if you don’t know which executor to use, use docker.
|
||||
|
||||
1. First, decide what qube you are going to use when working with Qubes Builder v2. It can be an AppVM or a Standalone qube, with some steps different between the two.
|
||||
|
||||
2. Installing dependencies
|
||||
|
||||
- If you want to use an app qube for developing, install dependencies in the template. If you are using a standalone, install them in the qube itself. Dependencies are specified in ``dependencies-*. txt`` files in the main builder directory, and you can install them easily in the following ways:
|
||||
|
||||
|
||||
|
||||
1. for Fedora, use:
|
||||
|
||||
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo dnf install $(cat dependencies-fedora.txt)
|
||||
$ test -f /usr/share/qubes/marker-vm && sudo dnf install qubes-gpg-split
|
||||
|
||||
|
||||
2. for Debian (note: some Debian packages require Debian version 13 or later), use:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo apt install $(cat dependencies-debian.txt)
|
||||
$ test -f /usr/share/qubes/marker-vm && sudo apt install qubes-gpg-split
|
||||
|
||||
If you have installed dependencies in the template, close it, and (re)start the development qube.
|
||||
|
||||
3. Clone the qubes-builder v2 repository into a location of your choice:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
git clone https://github.com/QubesOS/qubes-builderv2
|
||||
cd qubes-builderv2/
|
||||
|
||||
|
||||
4. If you haven’t previously used docker in the current qube, you need to set up some permissions. In particular, the user has to be added to the ``docker`` group:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo usermod -aG docker user
|
||||
|
||||
Next, **restart the qube**.
|
||||
|
||||
5. Finally, you need to generate a docker image:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ tools/generate-container-image.sh docker
|
||||
|
||||
In an app qube, as ``/var/lib/docker`` is not persistent by default, you also need to use :doc:`bind-dirs </user/advanced-topics/bind-dirs>` to avoid repeating this step after reboot, adding the following to the ``/rw/config/qubes-bind-dirs.d/docker.conf`` file in this qube:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
binds+=( '/var/lib/docker' )
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Configuration
|
||||
-------------
|
||||
|
||||
|
||||
To use Qubes OS Builder v2, you need to have a ``builder.yml`` configuration file. You can use one of the sample files from the ``example-configs/`` directory; for a more readable ``builder.yml``, you can also include one of the files from that directory in your ``builder.yml``. An example ``builder.yml`` is:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
# include configuration relevant for the current release
|
||||
include:
|
||||
- example-configs/qubes-os-r4.2.yml
|
||||
|
||||
# which repository to use to fetch sources
|
||||
use-qubes-repo:
|
||||
version: 4.2
|
||||
testing: true
|
||||
|
||||
# each package built will have local build number appended to package release
|
||||
# number. It makes it easier to update in testing environment
|
||||
increment-devel-versions: true
|
||||
|
||||
# reduce output
|
||||
debug: false
|
||||
|
||||
# this can be set to true if you do not want sources to be automatically
|
||||
# fetched from git
|
||||
skip-git-fetch: false
|
||||
|
||||
# executor configuration
|
||||
executor:
|
||||
type: docker
|
||||
options:
|
||||
image: "qubes-builder-fedora:latest"
|
||||
|
||||
|
||||
|
||||
Using Builder v2
|
||||
----------------
|
||||
|
||||
|
||||
To fetch sources - in this example, for the ``core-admin-client`` package, you can use the following command:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ ./qb -c core-admin-client package fetch
|
||||
|
||||
|
||||
This will fetch the sources for the listed package and place them in ``artifacts/sources`` directory.
|
||||
|
||||
To build a package (from sources in the ``artifacts/sources`` directory), use:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ ./qb -c core-admin-client package fetch prep build
|
||||
|
||||
|
||||
or, if you want to build for a specific target (``host-fc37`` is a ``dom0`` using Fedora 37, ``vm-fc40`` would be a qube using Fedora 40 etc.), use:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ ./qb -c core-admin-client -d host-fc37 package fetch prep build
|
||||
|
||||
|
||||
If you want to fetch the entire Qubes OS source use the following:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ ./qb package fetch
|
||||
|
||||
|
||||
**caution**: some repositories might have additional requirements. You can disable repositories that are not needed in the ``example-configs/*.yml`` file you are using by commenting them out. In particular, ``python-fido2``, ``lvm`` and ``windows``-related repositories have special requirements.
|
||||
|
|
@ -1,218 +0,0 @@
|
|||
---
|
||||
lang: en
|
||||
layout: doc
|
||||
permalink: /doc/qubes-builder/
|
||||
redirect_from:
|
||||
- /en/doc/qubes-builder/
|
||||
- /doc/QubesBuilder/
|
||||
- /wiki/QubesBuilder/
|
||||
ref: 64
|
||||
title: Qubes builder
|
||||
---
|
||||
|
||||
<div class="alert alert-warning" role="alert">
|
||||
<i class="fa fa-exclamation-circle"></i>
|
||||
<b>Note:</b> These instructions concern the older Qubes builder (v1). It supports
|
||||
only building Qubes 4.2 or earlier.<br>The build process has been completely rewritten in <a href="https://github.com/QubesOS/qubes-builderv2/">qubes-builder v2</a>. This can be used for building Qubes R4.2 and later, and all related components.
|
||||
</div>
|
||||
|
||||
**Note: See [ISO building instructions](/doc/qubes-iso-building/) for a streamlined overview on how to use the build system.**
|
||||
|
||||
|
||||
We have a fully automated build system for Qubes, that downloads, builds and
|
||||
packages all the Qubes components, and finally should spit out a ready-to-use
|
||||
installation ISO, all in a [secure](/news/2016/05/30/build-security/) way.
|
||||
|
||||
In order to use it, you should use an rpm-based distro, like Fedora :), and should ensure the following packages are installed:
|
||||
|
||||
- sudo
|
||||
- gnupg
|
||||
- git
|
||||
- createrepo
|
||||
- rpm-build
|
||||
- dnf-plugins-core
|
||||
- make
|
||||
- wget
|
||||
- rpmdevtools
|
||||
- python3-sh
|
||||
- dialog
|
||||
- rpm-sign
|
||||
- dpkg-dev
|
||||
- debootstrap
|
||||
- python3-pyyaml
|
||||
- devscripts
|
||||
- perl-Digest-MD5
|
||||
- perl-Digest-SHA
|
||||
|
||||
Usually you can install those packages by just issuing:
|
||||
|
||||
```shell
|
||||
sudo dnf install gnupg git createrepo rpm-build make wget rpmdevtools python3-sh dialog rpm-sign dpkg-dev debootstrap python3-pyyaml devscripts perl-Digest-MD5 perl-Digest-SHA
|
||||
```
|
||||
|
||||
The build system creates build environments in chroots and so no other packages are needed on the host.
|
||||
All files created by the build system are contained within the qubes-builder directory.
|
||||
The full build requires some 25GB of free space, so keep that in mind when deciding where to place this directory.
|
||||
|
||||
The build system is configured via builder.conf file.
|
||||
You can use the setup.sh script to create and modify this file.
|
||||
Alternatively, you can copy the provided default builder.conf, and modify it as needed, e.g.:
|
||||
|
||||
```shell
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
# edit the builder.conf file and set the following variables:
|
||||
NO_SIGN=1
|
||||
```
|
||||
|
||||
One additional useful requirement is that 'sudo root' must work without any prompt, which is default on most distros (e.g. 'sudo bash' brings you the root shell without asking for any password).
|
||||
This is important as the builder needs to switch to root and then back to user several times during the build process.
|
||||
|
||||
Additionally, if building with signing enabled (NO\_SIGN is not set), you must adjust `\~/.rpmmacros` file so that it points to the GPG key used for package signing, e.g.:
|
||||
|
||||
```bash
|
||||
%_signature gpg
|
||||
%_gpg_path /home/user/.gnupg
|
||||
%_gpg_name AC1BF9B3 # <-- Key ID used for signing
|
||||
```
|
||||
|
||||
It is also recommended that you use an empty passphrase for the private key used for signing.
|
||||
Contrary to a popular belief, this doesn't affect your key or sources security -- if somebody compromised your system, then the game is over anyway, whether you have used an additional passphrase for the key or not.
|
||||
|
||||
So, to build Qubes you would do:
|
||||
|
||||
```shell
|
||||
# Import the Qubes master key
|
||||
gpg --recv-keys 0xDDFA1A3E36879494
|
||||
|
||||
# Verify its fingerprint, set as 'trusted'.
|
||||
# This is described here:
|
||||
# https://www.qubes-os.org/doc/VerifyingSignatures
|
||||
|
||||
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
|
||||
gpg --import qubes-developers-keys.asc
|
||||
|
||||
git clone https://github.com/QubesOS/qubes-builder.git qubes-builder
|
||||
cd qubes-builder
|
||||
|
||||
# Verify its integrity:
|
||||
git tag -v `git describe`
|
||||
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
# edit the builder.conf file and set the following variables:
|
||||
# NO_SIGN="1"
|
||||
|
||||
# Download all components:
|
||||
|
||||
make get-sources
|
||||
|
||||
# And now to build all Qubes RPMs (this will take a few hours):
|
||||
|
||||
make qubes
|
||||
|
||||
# ... and then to build the ISO
|
||||
|
||||
make iso
|
||||
```
|
||||
|
||||
And this should produce a shiny new ISO.
|
||||
|
||||
You can also build selected component separately. Eg. to compile only gui virtualization agent/daemon:
|
||||
|
||||
```shell
|
||||
make gui-daemon
|
||||
```
|
||||
|
||||
You can get a full list from make help.
|
||||
|
||||
## Making customized build
|
||||
|
||||
### Manual source modification
|
||||
|
||||
You can also modify sources somehow if you wish.
|
||||
Here are some basic steps:
|
||||
|
||||
1. Download qubes-builder as described above (if you want to use marmarek's branches, you should also download qubes-builder from his repo - replace 'QubesOS' with 'marmarek' in above git clone command)
|
||||
2. Edit builder.conf (still the same as above), some useful additions:
|
||||
- You can also set GIT\_PREFIX="marmarek/qubes-" to use marmarek's repo instead of "mainstream" - it contains newer (but less tested) versions
|
||||
3. Download unmodified sources
|
||||
|
||||
```shell
|
||||
make get-sources
|
||||
```
|
||||
|
||||
4. **Make your modifications here**
|
||||
|
||||
5. Build the Qubes
|
||||
`make qubes` actually is just meta target which builds all required
|
||||
components in correct order. The list of components is configured in
|
||||
builder.conf. You can also check the current value at the end of `make
|
||||
help`, or using `make build-info`.
|
||||
|
||||
6. `get-sources` is already done, so continue with the next one. You can skip `sign-all` if you've disabled signing
|
||||
|
||||
```shell
|
||||
make vmm-xen core-admin linux-kernel gui-daemon template desktop-linux-kde installer-qubes-os manager linux-dom0-updates
|
||||
```
|
||||
|
||||
7. build iso installation image
|
||||
|
||||
```shell
|
||||
make iso
|
||||
```
|
||||
|
||||
### Use pre-built Qubes packages
|
||||
|
||||
For building just a few selected packages, it's very useful to download pre-built qubes-specific dependencies from `{yum,deb}.qubes-os.org`.
|
||||
This is especially true for `gcc`, which takes several hours to build.
|
||||
|
||||
Before creating the `chroot`, add this to your `builder.conf`:
|
||||
|
||||
```
|
||||
USE_QUBES_REPO_VERSION = $(RELEASE)
|
||||
```
|
||||
|
||||
It will add the 'current' Qubes repository to your `chroot` environment.
|
||||
Next, specify which components (`gcc`, for example) you want to download instead of compiling:
|
||||
|
||||
```
|
||||
COMPONENTS := $(filter-out gcc,$(COMPONENTS))
|
||||
```
|
||||
|
||||
Alternatively, edit the actual COMPONENTS list which is defined in the included version-dependent config from example-configs (see series of include directives near the beginning of `builder.conf`).
|
||||
This way, you can build only the packages in which you are interested.
|
||||
|
||||
If you also want to use the 'current-testing' repository, add this to your configuration:
|
||||
|
||||
```
|
||||
USE_QUBES_REPO_TESTING = 1
|
||||
```
|
||||
|
||||
In the case of an existing `chroot`, for mock-enabled builds, this works immediately because `chroot` is constructed each time separately.
|
||||
For legacy builds, it will not add the necessary configuration into the build environment unless a specific builder change or configuration would force rebuilding chroot.
|
||||
|
||||
Also, once enabled, disabling this setting will not disable repositories in relevant chroots.
|
||||
And even if it did, there could be some leftover packages installed from those repos (which may or may not be desirable).
|
||||
|
||||
**Note**
|
||||
If you are building Ubuntu templates, you cannot use this option.
|
||||
This is because Qubes does not provide official packages for Ubuntu templates.
|
||||
|
||||
## Code verification keys management
|
||||
|
||||
[QubesBuilder](/doc/qubes-builder/) by default verifies signed tags on every downloaded code.
|
||||
Public keys used for that are stored in `keyrings/git`.
|
||||
By default Qubes developers' keys are imported automatically, but if you need some additional keys (for example your own), you can add them using:
|
||||
|
||||
```shell
|
||||
GNUPGHOME=$PWD/keyrings/git gpg --import /path/to/key.asc
|
||||
GNUPGHOME=$PWD/keyrings/git gpg --edit-key ID_OF_JUST_IMPORTED_KEY
|
||||
# here use "trust" command to set key fully or ultimately trusted - only those keys are accepted by QubesBuilder
|
||||
```
|
||||
|
||||
All Qubes developers' keys are signed by the Qubes Master Signing Key (which is set as ultimately trusted key), so are trusted automatically.
|
||||
|
||||
If you are the owner of Master key and want to revoke such signature, use the `revsig` gpg key edit command and update the key in qubes-developers-keys.asc - now the key will be no longer trusted (unless manually set as such).
|
||||
|
||||
## Further information
|
||||
|
||||
For advanced [QubesBuilder](/doc/qubes-builder/) use, and preparing sources, take a look at [QubesBuilderDetails](/doc/qubes-builder-details/) page, or [QubesBuilder's doc directory](https://github.com/marmarek/qubes-builder/tree/master/doc).
|
||||
241
developer/building/qubes-builder.rst
Normal file
241
developer/building/qubes-builder.rst
Normal file
|
|
@ -0,0 +1,241 @@
|
|||
=============
|
||||
Qubes builder
|
||||
=============
|
||||
|
||||
|
||||
.. warning::
|
||||
|
||||
**Note:** These instructions concern the older Qubes builder (v1). It supports only building Qubes 4.2 or earlier.The build process has been completely rewritten in `qubes-builder v2 <https://github.com/QubesOS/qubes-builderv2/>`__ . This can be used for building Qubes R4.2 and later, and all related components.
|
||||
|
||||
**Note: See** :doc:`ISO building instructions </developer/building/qubes-iso-building>` **for a streamlined overview on how to use the build system.**
|
||||
|
||||
We have a fully automated build system for Qubes, that downloads, builds and packages all the Qubes components, and finally should spit out a ready-to-use installation ISO, all in a `secure <https://www.qubes-os.org/news/2016/05/30/build-security/>`__ way.
|
||||
|
||||
In order to use it, you should use an rpm-based distro, like Fedora :), and should ensure the following packages are installed:
|
||||
|
||||
- sudo
|
||||
|
||||
- gnupg
|
||||
|
||||
- git
|
||||
|
||||
- createrepo
|
||||
|
||||
- rpm-build
|
||||
|
||||
- dnf-plugins-core
|
||||
|
||||
- make
|
||||
|
||||
- wget
|
||||
|
||||
- rpmdevtools
|
||||
|
||||
- python3-sh
|
||||
|
||||
- dialog
|
||||
|
||||
- rpm-sign
|
||||
|
||||
- dpkg-dev
|
||||
|
||||
- debootstrap
|
||||
|
||||
- python3-pyyaml
|
||||
|
||||
- devscripts
|
||||
|
||||
- perl-Digest-MD5
|
||||
|
||||
- perl-Digest-SHA
|
||||
|
||||
|
||||
|
||||
Usually you can install those packages by just issuing:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo dnf install gnupg git createrepo rpm-build make wget rpmdevtools python3-sh dialog rpm-sign dpkg-dev debootstrap python3-pyyaml devscripts perl-Digest-MD5 perl-Digest-SHA
|
||||
|
||||
|
||||
The build system creates build environments in chroots and so no other packages are needed on the host. All files created by the build system are contained within the qubes-builder directory. The full build requires some 25GB of free space, so keep that in mind when deciding where to place this directory.
|
||||
|
||||
The build system is configured via builder.conf file. You can use the setup.sh script to create and modify this file. Alternatively, you can copy the provided default builder.conf, and modify it as needed, e.g.:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
# edit the builder.conf file and set the following variables:
|
||||
NO_SIGN=1
|
||||
|
||||
|
||||
One additional useful requirement is that ‘sudo root’ must work without any prompt, which is default on most distros (e.g. ‘sudo bash’ brings you the root shell without asking for any password). This is important as the builder needs to switch to root and then back to user several times during the build process.
|
||||
|
||||
Additionally, if building with signing enabled (NO_SIGN is not set), you must adjust ``\~/.rpmmacros`` file so that it points to the GPG key used for package signing, e.g.:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
%_signature gpg
|
||||
%_gpg_path /home/user/.gnupg
|
||||
%_gpg_name AC1BF9B3 # <-- Key ID used for signing
|
||||
|
||||
|
||||
It is also recommended that you use an empty passphrase for the private key used for signing. Contrary to a popular belief, this doesn’t affect your key or sources security – if somebody compromised your system, then the game is over anyway, whether you have used an additional passphrase for the key or not.
|
||||
|
||||
So, to build Qubes you would do:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
# Import the Qubes master key
|
||||
gpg --recv-keys 0xDDFA1A3E36879494
|
||||
|
||||
# Verify its fingerprint, set as 'trusted'.
|
||||
# This is described here:
|
||||
# https://www.qubes-os.org/doc/VerifyingSignatures
|
||||
|
||||
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
|
||||
gpg --import qubes-developers-keys.asc
|
||||
|
||||
git clone https://github.com/QubesOS/qubes-builder.git qubes-builder
|
||||
cd qubes-builder
|
||||
|
||||
# Verify its integrity:
|
||||
git tag -v `git describe`
|
||||
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
# edit the builder.conf file and set the following variables:
|
||||
# NO_SIGN="1"
|
||||
|
||||
# Download all components:
|
||||
|
||||
make get-sources
|
||||
|
||||
# And now to build all Qubes RPMs (this will take a few hours):
|
||||
|
||||
make qubes
|
||||
|
||||
# ... and then to build the ISO
|
||||
|
||||
make iso
|
||||
|
||||
|
||||
And this should produce a shiny new ISO.
|
||||
|
||||
You can also build selected component separately. Eg. to compile only gui virtualization agent/daemon:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make gui-daemon
|
||||
|
||||
|
||||
You can get a full list from make help.
|
||||
|
||||
Making customized build
|
||||
-----------------------
|
||||
|
||||
|
||||
Manual source modification
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
You can also modify sources somehow if you wish. Here are some basic steps:
|
||||
|
||||
1. Download qubes-builder as described above (if you want to use marmarek’s branches, you should also download qubes-builder from his repo - replace ‘QubesOS’ with ‘marmarek’ in above git clone command)
|
||||
|
||||
2. Edit builder.conf (still the same as above), some useful additions:
|
||||
|
||||
|
||||
|
||||
- You can also set GIT_PREFIX=“marmarek/qubes-” to use marmarek’s repo instead of “mainstream” - it contains newer (but less tested) versions
|
||||
|
||||
|
||||
|
||||
3. Download unmodified sources
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make get-sources
|
||||
|
||||
|
||||
4. **Make your modifications here**
|
||||
|
||||
5. Build the Qubes ``make qubes`` actually is just meta target which builds all required components in correct order. The list of components is configured in builder.conf. You can also check the current value at the end of ``make help``, or using ``make build-info``.
|
||||
|
||||
6. ``get-sources`` is already done, so continue with the next one. You can skip ``sign-all`` if you’ve disabled signing
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make vmm-xen core-admin linux-kernel gui-daemon template desktop-linux-kde installer-qubes-os manager linux-dom0-updates
|
||||
|
||||
|
||||
7. build iso installation image
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make iso
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Use pre-built Qubes packages
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
|
||||
For building just a few selected packages, it’s very useful to download pre-built qubes-specific dependencies from ``{yum,deb}.qubes-os.org``. This is especially true for ``gcc``, which takes several hours to build.
|
||||
|
||||
Before creating the ``chroot``, add this to your ``builder.conf``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
USE_QUBES_REPO_VERSION = $(RELEASE)
|
||||
|
||||
|
||||
|
||||
It will add the ‘current’ Qubes repository to your ``chroot`` environment. Next, specify which components (``gcc``, for example) you want to download instead of compiling:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
COMPONENTS := $(filter-out gcc,$(COMPONENTS))
|
||||
|
||||
|
||||
|
||||
Alternatively, edit the actual COMPONENTS list which is defined in the included version-dependent config from example-configs (see series of include directives near the beginning of ``builder.conf``). This way, you can build only the packages in which you are interested.
|
||||
|
||||
If you also want to use the ‘current-testing’ repository, add this to your configuration:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
USE_QUBES_REPO_TESTING = 1
|
||||
|
||||
|
||||
|
||||
In the case of an existing ``chroot``, for mock-enabled builds, this works immediately because ``chroot`` is constructed each time separately. For legacy builds, it will not add the necessary configuration into the build environment unless a specific builder change or configuration would force rebuilding chroot.
|
||||
|
||||
Also, once enabled, disabling this setting will not disable repositories in relevant chroots. And even if it did, there could be some leftover packages installed from those repos (which may or may not be desirable).
|
||||
|
||||
**Note** If you are building Ubuntu templates, you cannot use this option. This is because Qubes does not provide official packages for Ubuntu templates.
|
||||
|
||||
Code verification keys management
|
||||
---------------------------------
|
||||
|
||||
|
||||
:doc:`QubesBuilder </developer/building/qubes-builder>` by default verifies signed tags on every downloaded code. Public keys used for that are stored in ``keyrings/git``. By default Qubes developers’ keys are imported automatically, but if you need some additional keys (for example your own), you can add them using:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
GNUPGHOME=$PWD/keyrings/git gpg --import /path/to/key.asc
|
||||
GNUPGHOME=$PWD/keyrings/git gpg --edit-key ID_OF_JUST_IMPORTED_KEY
|
||||
# here use "trust" command to set key fully or ultimately trusted - only those keys are accepted by QubesBuilder
|
||||
|
||||
|
||||
All Qubes developers’ keys are signed by the Qubes Master Signing Key (which is set as ultimately trusted key), so are trusted automatically.
|
||||
|
||||
If you are the owner of Master key and want to revoke such signature, use the ``revsig`` gpg key edit command and update the key in qubes-developers-keys.asc - now the key will be no longer trusted (unless manually set as such).
|
||||
|
||||
Further information
|
||||
-------------------
|
||||
|
||||
|
||||
For advanced :doc:`QubesBuilder </developer/building/qubes-builder>` use, and preparing sources, take a look at :doc:`QubesBuilderDetails </developer/building/qubes-builder-details>` page, or `QubesBuilder’s doc directory <https://github.com/marmarek/qubes-builder/tree/master/doc>`__.
|
||||
|
|
@ -1,219 +0,0 @@
|
|||
---
|
||||
lang: en
|
||||
layout: doc
|
||||
permalink: /doc/qubes-iso-building/
|
||||
redirect_from:
|
||||
- /doc/qubes-r3-building/
|
||||
- /en/doc/qubes-r3-building/
|
||||
- /en/doc/qubes-iso-building/
|
||||
- /doc/QubesR3Building/
|
||||
- /wiki/QubesR3Building/
|
||||
ref: 63
|
||||
title: Qubes ISO building
|
||||
---
|
||||
|
||||
<div class="alert alert-warning" role="alert">
|
||||
<i class="fa fa-exclamation-circle"></i>
|
||||
<b>Note:</b> These instructions concern the older Qubes builder (v1). It supports
|
||||
only building Qubes 4.2 or earlier.<br>The build process has been completely rewritten in <a href="https://github.com/QubesOS/qubes-builderv2/">qubes-builder v2</a>. This can be used for building Qubes R4.2 and later versions, and all related components.
|
||||
</div>
|
||||
|
||||
Build Environment
|
||||
-----------------
|
||||
|
||||
Fedora 36 (and 37) has been successfully used to build Qubes R4.1 with the below steps.
|
||||
Other rpm-based operating systems may also work.
|
||||
Travis-CI uses Ubuntu 18.04 to perform test builds, except it can not test the `./setup` script.
|
||||
|
||||
**Notes:** On modern Fedora system (like Fedora 37) SeLinux is enforced by
|
||||
default and is blocking the build system. You would get error like
|
||||
"can't create transaction lock on /.../rpm/.rpm.lock (Permission denied)".
|
||||
You can set SeLinux to permissive mode with
|
||||
|
||||
~~~bash
|
||||
sudo setenforce 0
|
||||
~~~
|
||||
|
||||
|
||||
In `dom0`, install the Fedora 36 (or 37) template if you don't already have it.
|
||||
|
||||
~~~
|
||||
sudo qubes-dom0-update qubes-template-fedora-36
|
||||
~~~
|
||||
|
||||
Create a standalone AppVM from the Fedora template.
|
||||
Set private storage to at least 60 GB if you will be building only the default templates; 100 GB or more if you plan on additional.
|
||||
It's not required, but if you allocate additional CPU cores, the build process can utilize them at some steps such as the kernel build.
|
||||
Likewise, more memory (up to 16 GB) can help.
|
||||
Last, you may want to disable memory balancing, but keep in mind the impact on your other qubes.
|
||||
|
||||
Once you've built the development AppVM, open a Terminal window to it and install the necessary dependencies (see [QubesBuilder](/doc/qubes-builder/) for more info):
|
||||
|
||||
~~~
|
||||
$ sudo dnf install git createrepo rpm-build rpm-sign make python3-sh rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5 perl-Digest-SHA
|
||||
~~~
|
||||
|
||||
Get the necessary keys to verify the sources (run these and other commands below as a regular user, not root):
|
||||
|
||||
~~~
|
||||
wget https://keys.qubes-os.org/keys/qubes-master-signing-key.asc
|
||||
gpg --import qubes-master-signing-key.asc
|
||||
gpg --edit-key 36879494
|
||||
fpr
|
||||
# Verify fingerprint! See Note below!
|
||||
# Once verified, set trust to *ultimate*
|
||||
# (Typical sequence is trust, 5, Y, q)
|
||||
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
|
||||
gpg --import qubes-developers-keys.asc
|
||||
~~~
|
||||
|
||||
**Note** In the above process, we do *not* rely on the security of our server (keys.qubes-os.org) nor the connection (ssl, cert) -- we only rely on you getting the Qubes Master Signing Key fingerprint *somehow* and ensuring they match!
|
||||
See [verifying signatures](/security/verifying-signatures/#how-to-import-and-authenticate-the-qubes-master-signing-key) for verification sources.
|
||||
|
||||
Now let's bootstrap the builder. Unfortunately, the builder cannot verify itself (the classic Chicken and Egg problem), so we need to verify the signature manually:
|
||||
|
||||
~~~
|
||||
git clone https://github.com/QubesOS/qubes-builder.git
|
||||
cd qubes-builder
|
||||
git tag -v `git describe`
|
||||
~~~
|
||||
|
||||
**Note** It's very important to check if the verification message contains "Good signature from ..." and does not contain "WARNING: This key is not certified with a trusted signature!".
|
||||
|
||||
Assuming the verification went fine, we're good to go with all the rest without ever thinking more about verifying digital signatures on all the rest of the components, apart from an additional step if doing a non-scripted build.
|
||||
The builder will do that for us for each component, every time we build, even for all auxiliary files (e.g. Xen or Linux kernel sources).
|
||||
|
||||
Build using setup script
|
||||
-----------------
|
||||
|
||||
Let's configure the builder first (see [procedure](/doc/qubes-iso-building/#build-using-manual-steps) at bottom if you would prefer to manually configure):
|
||||
|
||||
~~~
|
||||
cd ~/qubes-builder
|
||||
./setup
|
||||
# Select Yes to add Qubes Master Signing Key
|
||||
# Select Yes to add Qubes OS Signing Key
|
||||
# Select 4.1 for version
|
||||
# Stable
|
||||
# Select Yes for fast Git cloning
|
||||
# Select Current (if you want the option to use pre-built packages)
|
||||
# Select No (we want a full build)
|
||||
# Select fc36 and bullseye (for the currently shipping templates)
|
||||
# Select builder-rpm, builder-debian, template-whonix, mgmt-salt
|
||||
# Select Yes to add adrelanos's third party key
|
||||
# Select Yes (to download)
|
||||
~~~
|
||||
|
||||
Once it completes downloading, re-run `setup` to add the Whonix templates:
|
||||
|
||||
~~~
|
||||
./setup
|
||||
# Choose the same options as above, except at templates select:
|
||||
# fc36, bullseye, whonix-gateway-16, whonix-workstation-16
|
||||
~~~
|
||||
|
||||
Continue the build process with:
|
||||
|
||||
~~~
|
||||
make install-deps
|
||||
make get-sources
|
||||
~~~
|
||||
|
||||
When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable in `builder.conf` at this stage to specify the currently shipping Tor Browser version.
|
||||
See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/).
|
||||
|
||||
You may also want to add `COMPONENTS := $(filter-out gcc,$(COMPONENTS))` to bypass a multiple hour compile step.
|
||||
See [QubesBuilder](/doc/qubes-builder/#use-pre-built-qubes-packages) for more detail.
|
||||
|
||||
Finally, if you are making a test build, use:
|
||||
|
||||
~~~
|
||||
make qubes
|
||||
make iso
|
||||
~~~
|
||||
|
||||
Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`):
|
||||
|
||||
~~~
|
||||
make qubes
|
||||
make sign-all
|
||||
make iso
|
||||
~~~
|
||||
|
||||
Enjoy your new ISO!
|
||||
|
||||
Build using manual steps
|
||||
-----------------
|
||||
|
||||
Instead of using `./setup`, you can manually configure the build.
|
||||
The script takes care of a lot of the keyring preparation for us, so we first need to set that up.
|
||||
|
||||
If you will be building Whonix templates:
|
||||
|
||||
~~~
|
||||
cd ~
|
||||
gpg --keyserver pgp.mit.edu --recv-keys 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
|
||||
gpg --fingerprint 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
|
||||
~~~
|
||||
|
||||
**Note:** It's very important to check the fingerprint displayed against multiple sources such as the [Whonix web site](https://www.whonix.org/wiki/Whonix_Signing_Key), etc.
|
||||
It should look something like this:
|
||||
|
||||
~~~
|
||||
pub rsa4096 2014-01-16 [SC] [expires: 2026-01-23]
|
||||
Key fingerprint = 916B 8D99 C38E AF5E 8ADC 7A2A 8D66 066A 2EEA CCDA
|
||||
uid [ unknown] Patrick Schleizer <adrelanos@kicksecure.com>
|
||||
uid [ unknown] Patrick Schleizer <adrelanos@riseup.net>
|
||||
uid [ unknown] Patrick Schleizer <adrelanos@whonix.org>
|
||||
sub rsa4096 2014-01-16 [E] [expires: 2026-01-23]
|
||||
sub rsa4096 2014-01-16 [A] [expires: 2026-01-23]
|
||||
sub rsa4096 2014-01-16 [S] [expires: 2026-01-23]
|
||||
|
||||
~~~
|
||||
|
||||
Next, prepare the Git keyring directory and copy them in:
|
||||
|
||||
~~~
|
||||
export GNUPGHOME=~/qubes-builder/keyrings/git
|
||||
mkdir --parents "$GNUPGHOME"
|
||||
cp ~/.gnupg/pubring.gpg "$GNUPGHOME"
|
||||
cp ~/.gnupg/trustdb.gpg "$GNUPGHOME"
|
||||
chmod --recursive 700 "$GNUPGHOME"
|
||||
~~~
|
||||
|
||||
Copy one of the example configurations:
|
||||
|
||||
~~~
|
||||
cd ~/qubes-builder
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
~~~
|
||||
|
||||
Edit `builder.conf`, referring to `doc/Configuration.md` for a description of all available options.
|
||||
|
||||
Continue the build process with:
|
||||
|
||||
~~~
|
||||
make install-deps
|
||||
make get-sources
|
||||
unset GNUPGHOME
|
||||
~~~
|
||||
|
||||
When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable at this stage to specify the currently shipping Tor Browser version.
|
||||
See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/).
|
||||
|
||||
Finally, if you are making a test build, use:
|
||||
|
||||
~~~
|
||||
make qubes
|
||||
make iso
|
||||
~~~
|
||||
|
||||
Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`):
|
||||
|
||||
~~~
|
||||
make qubes
|
||||
make sign-all
|
||||
make iso
|
||||
~~~
|
||||
|
||||
Enjoy your new ISO!
|
||||
226
developer/building/qubes-iso-building.rst
Normal file
226
developer/building/qubes-iso-building.rst
Normal file
|
|
@ -0,0 +1,226 @@
|
|||
==================
|
||||
Qubes ISO building
|
||||
==================
|
||||
|
||||
|
||||
.. warning::
|
||||
|
||||
**Note:** These instructions concern the older Qubes builder (v1). It supports only building Qubes 4.2 or earlier.The build process has been completely rewritten in `qubes-builder v2 <https://github.com/QubesOS/qubes-builderv2/>`__ . This can be used for building Qubes R4.2 and later versions, and all related components.
|
||||
|
||||
Build Environment
|
||||
-----------------
|
||||
|
||||
|
||||
Fedora 36 (and 37) has been successfully used to build Qubes R4.1 with the below steps. Other rpm-based operating systems may also work. Travis-CI uses Ubuntu 18.04 to perform test builds, except it can not test the ``./setup`` script.
|
||||
|
||||
**Notes:** On modern Fedora system (like Fedora 37) SeLinux is enforced by default and is blocking the build system. You would get error like “can’t create transaction lock on /…/rpm/.rpm.lock (Permission denied)”. You can set SeLinux to permissive mode with
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo setenforce 0
|
||||
|
||||
|
||||
In ``dom0``, install the Fedora 36 (or 37) template if you don’t already have it.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo qubes-dom0-update qubes-template-fedora-36
|
||||
|
||||
|
||||
|
||||
Create a standalone AppVM from the Fedora template. Set private storage to at least 60 GB if you will be building only the default templates; 100 GB or more if you plan on additional. It’s not required, but if you allocate additional CPU cores, the build process can utilize them at some steps such as the kernel build. Likewise, more memory (up to 16 GB) can help. Last, you may want to disable memory balancing, but keep in mind the impact on your other qubes.
|
||||
|
||||
Once you’ve built the development AppVM, open a Terminal window to it and install the necessary dependencies (see :doc:`QubesBuilder </developer/building/qubes-builder>` for more info):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
$ sudo dnf install git createrepo rpm-build rpm-sign make python3-sh rpmdevtools rpm-sign dialog perl-open python3-pyyaml perl-Digest-MD5 perl-Digest-SHA
|
||||
|
||||
|
||||
|
||||
Get the necessary keys to verify the sources (run these and other commands below as a regular user, not root):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
wget https://keys.qubes-os.org/keys/qubes-master-signing-key.asc
|
||||
gpg --import qubes-master-signing-key.asc
|
||||
gpg --edit-key 36879494
|
||||
fpr
|
||||
# Verify fingerprint! See Note below!
|
||||
# Once verified, set trust to *ultimate*
|
||||
# (Typical sequence is trust, 5, Y, q)
|
||||
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
|
||||
gpg --import qubes-developers-keys.asc
|
||||
|
||||
|
||||
|
||||
**Note** In the above process, we do *not* rely on the security of our server (keys.qubes-os.org) nor the connection (ssl, cert) – we only rely on you getting the Qubes Master Signing Key fingerprint *somehow* and ensuring they match! See :ref:`verifying signatures <project-security/verifying-signatures:how to import and authenticate the qubes master signing key>` for verification sources.
|
||||
|
||||
Now let’s bootstrap the builder. Unfortunately, the builder cannot verify itself (the classic Chicken and Egg problem), so we need to verify the signature manually:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
git clone https://github.com/QubesOS/qubes-builder.git
|
||||
cd qubes-builder
|
||||
git tag -v `git describe`
|
||||
|
||||
|
||||
|
||||
**Note** It’s very important to check if the verification message contains “Good signature from …” and does not contain “WARNING: This key is not certified with a trusted signature!”.
|
||||
|
||||
Assuming the verification went fine, we’re good to go with all the rest without ever thinking more about verifying digital signatures on all the rest of the components, apart from an additional step if doing a non-scripted build. The builder will do that for us for each component, every time we build, even for all auxiliary files (e.g. Xen or Linux kernel sources).
|
||||
|
||||
Build using setup script
|
||||
------------------------
|
||||
|
||||
|
||||
Let’s configure the builder first (see :ref:`procedure <developer/building/qubes-iso-building:build using manual steps>` at bottom if you would prefer to manually configure):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
cd ~/qubes-builder
|
||||
./setup
|
||||
# Select Yes to add Qubes Master Signing Key
|
||||
# Select Yes to add Qubes OS Signing Key
|
||||
# Select 4.1 for version
|
||||
# Stable
|
||||
# Select Yes for fast Git cloning
|
||||
# Select Current (if you want the option to use pre-built packages)
|
||||
# Select No (we want a full build)
|
||||
# Select fc36 and bullseye (for the currently shipping templates)
|
||||
# Select builder-rpm, builder-debian, template-whonix, mgmt-salt
|
||||
# Select Yes to add adrelanos's third party key
|
||||
# Select Yes (to download)
|
||||
|
||||
|
||||
|
||||
Once it completes downloading, re-run ``setup`` to add the Whonix templates:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
./setup
|
||||
# Choose the same options as above, except at templates select:
|
||||
# fc36, bullseye, whonix-gateway-16, whonix-workstation-16
|
||||
|
||||
|
||||
|
||||
Continue the build process with:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make install-deps
|
||||
make get-sources
|
||||
|
||||
|
||||
|
||||
When building the Whonix templates, you will often need to add/update the ``WHONIX_TBB_VERSION`` variable in ``builder.conf`` at this stage to specify the currently shipping Tor Browser version. See the related note under `Extra Whonix Build Options <https://forum.qubes-os.org/t/18981>`__.
|
||||
|
||||
You may also want to add ``COMPONENTS := $(filter-out gcc,$(COMPONENTS))`` to bypass a multiple hour compile step. See :ref:`QubesBuilder <developer/building/qubes-builder:use pre-built qubes packages>` for more detail.
|
||||
|
||||
Finally, if you are making a test build, use:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make qubes
|
||||
make iso
|
||||
|
||||
|
||||
|
||||
Or for a fully signed build (this requires setting ``SIGN_KEY`` in ``builder.conf``):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make qubes
|
||||
make sign-all
|
||||
make iso
|
||||
|
||||
|
||||
|
||||
Enjoy your new ISO!
|
||||
|
||||
Build using manual steps
|
||||
------------------------
|
||||
|
||||
|
||||
Instead of using ``./setup``, you can manually configure the build. The script takes care of a lot of the keyring preparation for us, so we first need to set that up.
|
||||
|
||||
If you will be building Whonix templates:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
cd ~
|
||||
gpg --keyserver pgp.mit.edu --recv-keys 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
|
||||
gpg --fingerprint 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
|
||||
|
||||
|
||||
|
||||
**Note:** It’s very important to check the fingerprint displayed against multiple sources such as the `Whonix web site <https://www.whonix.org/wiki/Whonix_Signing_Key>`__, etc. It should look something like this:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
pub rsa4096 2014-01-16 [SC] [expires: 2026-01-23]
|
||||
Key fingerprint = 916B 8D99 C38E AF5E 8ADC 7A2A 8D66 066A 2EEA CCDA
|
||||
uid [ unknown] Patrick Schleizer <adrelanos@kicksecure.com>
|
||||
uid [ unknown] Patrick Schleizer <adrelanos@riseup.net>
|
||||
uid [ unknown] Patrick Schleizer <adrelanos@whonix.org>
|
||||
sub rsa4096 2014-01-16 [E] [expires: 2026-01-23]
|
||||
sub rsa4096 2014-01-16 [A] [expires: 2026-01-23]
|
||||
sub rsa4096 2014-01-16 [S] [expires: 2026-01-23]
|
||||
|
||||
|
||||
|
||||
Next, prepare the Git keyring directory and copy them in:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
export GNUPGHOME=~/qubes-builder/keyrings/git
|
||||
mkdir --parents "$GNUPGHOME"
|
||||
cp ~/.gnupg/pubring.gpg "$GNUPGHOME"
|
||||
cp ~/.gnupg/trustdb.gpg "$GNUPGHOME"
|
||||
chmod --recursive 700 "$GNUPGHOME"
|
||||
|
||||
|
||||
|
||||
Copy one of the example configurations:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
cd ~/qubes-builder
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
|
||||
|
||||
|
||||
Edit ``builder.conf``, referring to ``doc/Configuration.md`` for a description of all available options.
|
||||
|
||||
Continue the build process with:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make install-deps
|
||||
make get-sources
|
||||
unset GNUPGHOME
|
||||
|
||||
|
||||
|
||||
When building the Whonix templates, you will often need to add/update the ``WHONIX_TBB_VERSION`` variable at this stage to specify the currently shipping Tor Browser version. See the related note under `Extra Whonix Build Options <https://forum.qubes-os.org/t/18981>`__.
|
||||
|
||||
Finally, if you are making a test build, use:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make qubes
|
||||
make iso
|
||||
|
||||
|
||||
|
||||
Or for a fully signed build (this requires setting ``SIGN_KEY`` in ``builder.conf``):
|
||||
|
||||
.. code:: bash
|
||||
|
||||
make qubes
|
||||
make sign-all
|
||||
make iso
|
||||
|
||||
|
||||
|
||||
Enjoy your new ISO!
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
---
|
||||
lang: en
|
||||
layout: doc
|
||||
permalink: /doc/qubes-template-configs/
|
||||
redirect_to: https://github.com/QubesOS/qubes-template-configs
|
||||
ref: 248
|
||||
title: Qubes template configs
|
||||
---
|
||||
Loading…
Add table
Add a link
Reference in a new issue