mirror of
https://github.com/QubesOS/qubes-doc.git
synced 2025-12-15 15:59:23 -05:00
Reorganize files to account for new "External" section
QubesOS/qubes-issues#4693
This commit is contained in:
Andrew David Wong
parent
5cc99a23d1
commit
d31c786942
203 changed files with 0 additions and 0 deletions
546
developer/building/development-workflow.md
Normal file
546
developer/building/development-workflow.md
Normal file
|
|
@ -0,0 +1,546 @@
|
|||
---
|
||||
layout: doc
|
||||
title: Development Workflow
|
||||
permalink: /doc/development-workflow/
|
||||
redirect_from:
|
||||
- /en/doc/development-workflow/
|
||||
- /doc/DevelopmentWorkflow/
|
||||
- /wiki/DevelopmentWorkflow/
|
||||
---
|
||||
|
||||
Development Workflow
|
||||
====================
|
||||
|
||||
A workflow for developing Qubes OS+
|
||||
|
||||
First things first, setup [QubesBuilder](/doc/qubes-builder/). This guide
|
||||
assumes you're using qubes-builder to build Qubes.
|
||||
|
||||
Repositories and committing Code
|
||||
--------------------------------
|
||||
|
||||
Qubes is split into a bunch of git repos. This are all contained in the
|
||||
`qubes-src` directory under qubes-builder. Subdirectories there are separate
|
||||
components, stored in separate git repositories.
|
||||
|
||||
The best way to write and contribute code is to create a git repo somewhere
|
||||
(e.g., github) for the repo you are interested in editing (e.g.,
|
||||
`qubes-manager`, `core-agent-linux`, etc). To integrate your repo with the rest
|
||||
of Qubes, cd to the repo directory and add your repository as a remote in git
|
||||
|
||||
**Example:**
|
||||
|
||||
~~~
|
||||
$ cd qubes-builder/qubes-src/qubes-manager
|
||||
$ git remote add abel git@github.com:abeluck/qubes-manager.git
|
||||
~~~
|
||||
|
||||
You can then proceed to easily develop in your own branches, pull in new
|
||||
commits from the dev branches, merge them, and eventually push to your own repo
|
||||
on github.
|
||||
|
||||
When you are ready to submit your changes to Qubes to be merged, push your
|
||||
changes, then create a signed git tag (using `git tag -s`). Finally, send a
|
||||
letter to the Qubes listserv describing the changes and including the link to
|
||||
your repository. You can also create pull request on github. Don't forget to
|
||||
include your public PGP key you use to sign your tags.
|
||||
|
||||
### Kernel-specific notes
|
||||
|
||||
#### Prepare fresh version of kernel sources, with Qubes-specific patches applied
|
||||
|
||||
In qubes-builder/qubes-src/linux-kernel:
|
||||
|
||||
~~~
|
||||
make prep
|
||||
~~~
|
||||
|
||||
The resulting tree will be in kernel-\<VERSION\>/linux-\<VERSION\>:
|
||||
|
||||
~~~
|
||||
ls -ltrd kernel*/linux*
|
||||
~~~
|
||||
|
||||
~~~
|
||||
drwxr-xr-x 23 user user 4096 Nov 5 09:50 kernel-3.4.18/linux-3.4.18
|
||||
drwxr-xr-x 6 user user 4096 Nov 21 20:48 kernel-3.4.18/linux-obj
|
||||
~~~
|
||||
|
||||
#### Go to the kernel tree and update the version
|
||||
|
||||
In qubes-builder/qubes-src/linux-kernel:
|
||||
|
||||
~~~
|
||||
cd kernel-3.4.18/linux-3.4.18
|
||||
~~~
|
||||
|
||||
#### Changing the config
|
||||
|
||||
In kernel-3.4.18/linux-3.4.18:
|
||||
|
||||
~~~
|
||||
cp ../../config .config
|
||||
make oldconfig
|
||||
~~~
|
||||
|
||||
Now change the configuration. For example, in kernel-3.4.18/linux-3.4.18:
|
||||
|
||||
~~~
|
||||
make menuconfig
|
||||
~~~
|
||||
|
||||
Copy the modified config back into the kernel tree:
|
||||
|
||||
~~~
|
||||
cp .config ../../../config
|
||||
~~~
|
||||
|
||||
#### Patching the code
|
||||
|
||||
TODO: describe the workflow for patching the code, below are some random notes, not working well
|
||||
|
||||
~~~
|
||||
ln -s ../../patches.xen
|
||||
export QUILT_PATCHES=patches.xen
|
||||
export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
|
||||
export QUILT_SERIES=../../series-pvops.conf
|
||||
|
||||
quilt new patches.xen/pvops-3.4-0101-usb-xen-pvusb-driver-bugfix.patch
|
||||
quilt add drivers/usb/host/Kconfig drivers/usb/host/Makefile \
|
||||
drivers/usb/host/xen-usbback/* drivers/usb/host/xen-usbfront.c \
|
||||
include/xen/interface/io/usbif.h
|
||||
|
||||
*edit something*
|
||||
|
||||
quilt refresh
|
||||
cd ../..
|
||||
vi series.conf
|
||||
~~~
|
||||
|
||||
#### Building RPMS
|
||||
|
||||
TODO: Is this step generic for all subsystems?
|
||||
|
||||
Now it is a good moment to make sure you have changed kernel release name in
|
||||
rel file. For example, if you change it to '1debug201211116c' the
|
||||
resulting RPMs will be named
|
||||
'kernel-3.4.18-1debug20121116c.pvops.qubes.x86\_64.rpm'. This will help
|
||||
distinguish between different versions of the same package.
|
||||
|
||||
You might want to take a moment here to review (git diff, git status), commit
|
||||
your changes locally.
|
||||
|
||||
To actually build RPMS, in qubes-builder:
|
||||
|
||||
~~~
|
||||
make linux-kernel
|
||||
~~~
|
||||
|
||||
RPMS will appear in qubes-src/linux-kernel/pkgs/fc20/x86\_64:
|
||||
|
||||
~~~
|
||||
-rw-rw-r-- 1 user user 42996126 Nov 17 04:08 kernel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm
|
||||
-rw-rw-r-- 1 user user 43001450 Nov 17 05:36 kernel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm
|
||||
-rw-rw-r-- 1 user user 8940138 Nov 17 04:08 kernel-devel-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm
|
||||
-rw-rw-r-- 1 user user 8937818 Nov 17 05:36 kernel-devel-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm
|
||||
-rw-rw-r-- 1 user user 54490741 Nov 17 04:08 kernel-qubes-vm-3.4.18-1debug20121116c.pvops.qubes.x86_64.rpm
|
||||
-rw-rw-r-- 1 user user 54502117 Nov 17 05:37 kernel-qubes-vm-3.4.18-1debug20121117a.pvops.qubes.x86_64.rpm
|
||||
~~~
|
||||
|
||||
### Useful [QubesBuilder](/doc/qubes-builder/) commands
|
||||
|
||||
1. `make check` - will check if all the code was committed into repository and
|
||||
if all repository are tagged with signed tag.
|
||||
2. `make show-vtags` - show version of each component (based on git tags) -
|
||||
mostly useful just before building ISO. **Note:** this will not show version
|
||||
for components containing changes since last version tag
|
||||
3. `make push` - push change from **all** repositories to git server. You must
|
||||
set proper remotes (see above) for all repositories first.
|
||||
4. `make prepare-merge` - fetch changes from remote repositories (can be
|
||||
specified on commandline via GIT\_SUBDIR or GIT\_REMOTE vars), (optionally)
|
||||
verify tags and show the changes. This do not merge the changes - there are
|
||||
left for review as FETCH\_HEAD ref. You can merge them using `git merge
|
||||
FETCH_HEAD` (in each repo directory). Or `make do-merge` to merge all of them.
|
||||
|
||||
Copying Code to dom0
|
||||
--------------------
|
||||
|
||||
When developing it is convenient to be able to rapidly test changes. Assuming
|
||||
you're developing Qubes on Qubes, you should be working in a special VM for
|
||||
Qubes and occasionally you will want to transfer code or rpms back to dom0 for
|
||||
testing.
|
||||
|
||||
Here are some handy scripts Marek has shared to facilitate this.
|
||||
|
||||
You may also like to run your [test environment on separate
|
||||
machine](/doc/test-bench/).
|
||||
|
||||
### Syncing dom0 files
|
||||
|
||||
TODO: edit this script to be more generic
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
set -x
|
||||
set -e
|
||||
|
||||
QUBES_PY_DIR=/usr/lib64/python2.6/site-packages/qubes
|
||||
QUBES_PY=$QUBES_PY_DIR/qubes.py
|
||||
QUBESUTILS_PY=$QUBES_PY_DIR/qubesutils.py
|
||||
|
||||
qvm-run -p qubes-devel 'cd qubes-builder/qubes-src/core/dom0; tar c qmemman/qmemman*.py qvm-core/*.py qvm-tools/* misc/vm-template-hvm.conf misc/qubes-start.desktop ../misc/block-snapshot aux-tools ../qrexec' |tar xv
|
||||
cp $QUBES_PY qubes.py.bak$$
|
||||
cp $QUBESUTILS_PY qubesutils.py.bak$$
|
||||
cp /etc/xen/scripts/block-snapshot block-snapshot.bak$$
|
||||
sudo cp qvm-core/qubes.py $QUBES_PY
|
||||
sudo cp qvm-core/qubesutils.py $QUBESUTILS_PY
|
||||
sudo cp qvm-core/guihelpers.py $QUBES_PY_DIR/
|
||||
sudo cp qmemman/qmemman*.py $QUBES_PY_DIR/
|
||||
sudo cp misc/vm-template-hvm.conf /usr/share/qubes/
|
||||
sudo cp misc/qubes-start.desktop /usr/share/qubes/
|
||||
sudo cp misc/block-snapshot /etc/xen/scripts/
|
||||
sudo cp aux-tools/qubes-dom0-updates.cron /etc/cron.daily/
|
||||
# FIXME(Abel Luck): I hope to
|
||||
~~~
|
||||
|
||||
### Apply qvm-tools
|
||||
|
||||
TODO: make it more generic
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
BAK=qvm-tools.bak$$
|
||||
mkdir -p $BAK
|
||||
cp -a /usr/bin/qvm-* /usr/bin/qubes-* $BAK/
|
||||
sudo cp qvm-tools/qvm-* qvm-tools/qubes-* /usr/bin/
|
||||
~~~
|
||||
|
||||
### Copy from dom0 to an appvm
|
||||
|
||||
~~~
|
||||
#/bin/sh
|
||||
#
|
||||
# usage ./cp-domain <vm_name> <file_to_copy>
|
||||
#
|
||||
domain=$1
|
||||
file=$2
|
||||
fname=`basename $file`
|
||||
|
||||
qvm-run $domain 'mkdir /home/user/incoming/dom0 -p'
|
||||
cat $file| qvm-run --pass-io $domain "cat > /home/user/incoming/dom0/$fname"
|
||||
~~~
|
||||
|
||||
## Git connection between VMs
|
||||
|
||||
Sometimes it's useful to transfer git commits between VMs. You can use `git
|
||||
format-patch` for that and simply copy the files. But you can also setup
|
||||
custom qrexec service for it.
|
||||
|
||||
Below example assumes that you use `builder-RX` directory in target VM to
|
||||
store sources in qubes-builder layout (where `X` is some number). Make sure that
|
||||
all the scripts are executable.
|
||||
|
||||
Service file (save in `/usr/local/etc/qubes-rpc/local.Git` in target VM):
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
exec 2>/tmp/log2
|
||||
|
||||
read service rel repo
|
||||
echo "Params: $service $rel $repo" >&2
|
||||
# Adjust regexps if needed
|
||||
echo "$repo" | grep -q '^[A-Za-z0-9-]\+$' || exit 1
|
||||
echo "$rel" | grep -q '^[0-9.]\+$' || exit 1
|
||||
path="/home/user/builder-R$rel/qubes-src/$repo"
|
||||
if [ "$repo" = "builder" ]; then
|
||||
path="/home/user/builder-R$rel"
|
||||
fi
|
||||
case $service in
|
||||
git-receive-pack|git-upload-pack)
|
||||
echo "starting $service $path" >&2
|
||||
exec $service $path
|
||||
;;
|
||||
*)
|
||||
echo "Unsupported service: $service" >&2
|
||||
;;
|
||||
esac
|
||||
~~~
|
||||
|
||||
Client script (save in `~/bin/git-qrexec` in source VM):
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
VMNAME=$1
|
||||
|
||||
(echo $GIT_EXT_SERVICE $2 $3; exec cat) | qrexec-client-vm $VMNAME local.Git
|
||||
~~~
|
||||
|
||||
You will also need to setup qrexec policy in dom0 (`/etc/qubes-rpc/policy/local.Git`).
|
||||
|
||||
Usage:
|
||||
|
||||
~~~
|
||||
[user@source core-agent-linux]$ git remote add testbuilder "ext::git-qrexec testbuilder 3 core-agent-linux"
|
||||
[user@source core-agent-linux]$ git push testbuilder master
|
||||
~~~
|
||||
|
||||
You can create `~/bin/add-remote` script to ease adding remotes:
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
[ -n "$1" ] || exit 1
|
||||
|
||||
if [ "$1" = "tb" ]; then
|
||||
git remote add $1 "ext::git-qrexec testbuilder 3 `basename $PWD`"
|
||||
exit $?
|
||||
fi
|
||||
|
||||
git remote add $1 git@github.com:$1/qubes-`basename $PWD`
|
||||
~~~
|
||||
|
||||
It should be executed from component top level directory. This script takes one
|
||||
argument - remote name. If it is `tb`, then it creates qrexec-based git remote
|
||||
to `testbuilder` VM. Otherwise it creates remote pointing at github account of
|
||||
the same name. In any case it points at repository matching current directory
|
||||
name.
|
||||
|
||||
|
||||
## Sending packages to different VM
|
||||
|
||||
Other useful script(s) can be used to setup local package repository hosted in
|
||||
some VM. This way you can keep your development VM behind firewall, while
|
||||
having an option to expose some yum/apt repository to the local network (to
|
||||
have them installed on test machine).
|
||||
|
||||
To achieve this goal, a dummy repository can be created, which instead of
|
||||
populating metadata locally, will upload the packages to some other VM and
|
||||
trigger repository update there (using qrexec). You can use `unstable`
|
||||
repository flavor, because there is no release managing rules bundled (unlike
|
||||
current and current-testing).
|
||||
|
||||
### RPM packages - yum repo
|
||||
|
||||
In source VM, grab [linux-yum] repository (below is assumed you've made it in
|
||||
`~/repo-yum-upload` directory) and replace `update_repo.sh` script with:
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
VMNAME=repo-vm
|
||||
|
||||
set -e
|
||||
qvm-copy-to-vm $VMNAME $1
|
||||
# remove only files, leave directory structure
|
||||
find -type f -name '*.rpm' -delete
|
||||
# trigger repo update
|
||||
qrexec-client-vm $VMNAME local.UpdateYum
|
||||
~~~
|
||||
|
||||
In target VM, setup actual yum repository (also based on [linux-yum], this time
|
||||
without modifications). You will also need to setup some gpg key for signing
|
||||
packages (it is possible to force yum to install unsigned packages, but it
|
||||
isn't possible for `qubes-dom0-update` tool). Fill `~/.rpmmacros` with
|
||||
key description:
|
||||
|
||||
~~~
|
||||
%_gpg_name Test packages signing key
|
||||
~~~
|
||||
|
||||
Then setup `local.UpdateYum` qrexec service (`/usr/local/etc/qubes-rpc/local.UpdateYum`):
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
real_repository=/home/user/linux-yum
|
||||
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
|
||||
|
||||
find $incoming -name '*.rpm' |xargs rpm -K |grep -iv pgp |cut -f1 -d: |xargs -r setsid -w rpm --addsign 2>&1
|
||||
|
||||
rsync -lr --remove-source-files $incoming/ $real_repository
|
||||
cd $real_repository
|
||||
export SKIP_REPO_CHECK=1
|
||||
if [ -d $incoming/r3.1 ]; then
|
||||
./update_repo-unstable.sh r3.1
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r3.0 ]; then
|
||||
./update_repo-unstable.sh r3.0
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r2 ]; then
|
||||
./update_repo-unstable.sh r2
|
||||
fi
|
||||
find $incoming -type d -empty -delete
|
||||
exit 0
|
||||
~~~
|
||||
|
||||
Of course you will also need to setup qrexec policy in dom0
|
||||
`/etc/qubes-rpc/policy/local.UpdateYum`.
|
||||
|
||||
If you want to access the repository from network, you need to setup HTTP
|
||||
server serving it, and configure the system to let other machines actually
|
||||
reach this HTTP server. You can use for example using [port
|
||||
forwarding][port-forwarding] or setting up Tor hidden service. Configuration
|
||||
details of those services are outside of the scope of this page.
|
||||
|
||||
Usage: setup `builder.conf` in source VM to use your dummy-uploader repository:
|
||||
|
||||
~~~
|
||||
LINUX_REPO_BASEDIR = ../../repo-yum-upload/r3.1
|
||||
~~~
|
||||
|
||||
Then use `make update-repo-unstable` to upload the packages. You can also
|
||||
specify selected components on command line, then build them and upload to the
|
||||
repository:
|
||||
|
||||
~~~
|
||||
make COMPONENTS="core-agent-linux gui-agent-linux linux-utils" qubes update-repo-unstable
|
||||
~~~
|
||||
|
||||
On the test machine, add yum repository (`/etc/yum.repos.d`) pointing at just
|
||||
configured HTTP server. For example:
|
||||
|
||||
~~~
|
||||
[local-test]
|
||||
name=Test
|
||||
baseurl=http://local-test.lan/linux-yum/r$releasever/unstable/dom0/fc20
|
||||
~~~
|
||||
|
||||
Remember to also import gpg public key using `rpm --import`.
|
||||
|
||||
### Deb packages - Apt repo
|
||||
|
||||
Steps are mostly the same as in the case of yum repo. The only details that differ:
|
||||
|
||||
- use [linux-deb] instead of [linux-yum] as a base - both in source and target VM
|
||||
- use different `update_repo.sh` script in source VM (below)
|
||||
- use `local.UpdateApt` qrexec service in target VM (code below)
|
||||
- in target VM additionally place `update-local-repo.sh` script in repository dir (code below)
|
||||
|
||||
`update_repo.sh` script:
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
set -e
|
||||
|
||||
current_release=$1
|
||||
VMNAME=repo-vm
|
||||
|
||||
qvm-copy-to-vm $VMNAME $1
|
||||
find $current_release -type f -name '*.deb' -delete
|
||||
rm -f $current_release/vm/db/*
|
||||
qrexec-client-vm $VMNAME local.UpdateApt
|
||||
~~~
|
||||
|
||||
`local.UpdateApt` service code (`/usr/local/etc/qubes-rpc/local.UpdateApt` in repo-serving VM):
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
if [ -z "$QREXEC_REMOTE_DOMAIN" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
incoming=/home/user/QubesIncoming/$QREXEC_REMOTE_DOMAIN
|
||||
|
||||
rsync -lr --remove-source-files $incoming/ /home/user/linux-deb/
|
||||
cd /home/user/linux-deb
|
||||
export SKIP_REPO_CHECK=1
|
||||
if [ -d $incoming/r3.1 ]; then
|
||||
for dist in `ls r3.1/vm/dists`; do
|
||||
./update-local-repo.sh r3.1/vm $dist
|
||||
done
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r3.0 ]; then
|
||||
for dist in `ls r3.0/vm/dists`; do
|
||||
./update-local-repo.sh r3.0/vm $dist
|
||||
done
|
||||
fi
|
||||
|
||||
if [ -d $incoming/r2 ]; then
|
||||
for dist in `ls r2/vm/dists`; do
|
||||
./update-local-repo.sh r2/vm $dist
|
||||
done
|
||||
fi
|
||||
find $incoming -type d -empty -delete
|
||||
exit 0
|
||||
~~~
|
||||
|
||||
`update-local-repo.sh`:
|
||||
|
||||
~~~
|
||||
#!/bin/sh
|
||||
|
||||
set -e
|
||||
|
||||
# Set this to your local repository signing key
|
||||
SIGN_KEY=01ABCDEF
|
||||
|
||||
[ -z "$1" ] && { echo "Usage: $0 <repo> <dist>"; exit 1; }
|
||||
|
||||
REPO_DIR=$1
|
||||
DIST=$2
|
||||
|
||||
if [ "$DIST" = "wheezy-unstable" ]; then
|
||||
DIST_TAG=deb7
|
||||
elif [ "$DIST" = "jessie-unstable" ]; then
|
||||
DIST_TAG=deb8
|
||||
elif [ "$DIST" = "stretch-unstable" ]; then
|
||||
DIST_TAG=deb9
|
||||
fi
|
||||
|
||||
pushd $REPO_DIR
|
||||
mkdir -p dists/$DIST/main/binary-amd64
|
||||
dpkg-scanpackages --multiversion --arch "*$DIST_TAG*" . > dists/$DIST/main/binary-amd64/Packages
|
||||
gzip -9c dists/$DIST/main/binary-amd64/Packages > dists/$DIST/main/binary-amd64/Packages.gz
|
||||
cat > dists/$DIST/Release <<EOF
|
||||
Label: Test repo
|
||||
Suite: $DIST
|
||||
Codename: $DIST
|
||||
Date: `date -R`
|
||||
Architectures: amd64
|
||||
Components: main
|
||||
SHA1:
|
||||
EOF
|
||||
function calc_sha1() {
|
||||
f=dists/$DIST/$1
|
||||
echo -n " "
|
||||
echo -n `sha1sum $f|cut -d' ' -f 1` ""
|
||||
echo -n `stat -c %s $f` ""
|
||||
echo $1
|
||||
}
|
||||
calc_sha1 main/binary-amd64/Packages >> dists/$DIST/Release
|
||||
|
||||
rm -f $DIST/Release.gpg
|
||||
rm -f $DIST/InRelease
|
||||
gpg -abs -u "$SIGN_KEY" \
|
||||
< dists/$DIST/Release > dists/$DIST/Release.gpg
|
||||
gpg -a -s --clearsign -u "$SIGN_KEY" \
|
||||
< dists/$DIST/Release > dists/$DIST/InRelease
|
||||
popd
|
||||
|
||||
if [ `id -u` -eq 0 ]; then
|
||||
chown -R --reference=$REPO_DIR $REPO_DIR
|
||||
fi
|
||||
~~~
|
||||
|
||||
Usage: add this line to `/etc/apt/sources.list` on test machine (adjust host and path):
|
||||
|
||||
~~~
|
||||
deb http://local-test.lan/linux-deb/r3.1 jessie-unstable main
|
||||
~~~
|
||||
|
||||
[port-forwarding]: /doc/firewall/#port-forwarding-to-a-qube-from-the-outside-world
|
||||
[linux-yum]: https://github.com/QubesOS/qubes-linux-yum
|
||||
[linux-deb]: https://github.com/QubesOS/qubes-linux-deb
|
||||
49
developer/building/qubes-builder-details.md
Normal file
49
developer/building/qubes-builder-details.md
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
---
|
||||
layout: doc
|
||||
title: Qubes Builder Details
|
||||
permalink: /doc/qubes-builder-details/
|
||||
redirect_from:
|
||||
- /en/doc/qubes-builder-details/
|
||||
- /doc/QubesBuilderDetails/
|
||||
- /wiki/QubesBuilderDetails/
|
||||
---
|
||||
|
||||
[QubesBuilder](/doc/qubes-builder/) "API"
|
||||
========================================
|
||||
|
||||
Components Makefile.builder file
|
||||
--------------------------------
|
||||
|
||||
[QubesBuilder](/doc/qubes-builder/) expects that each component have *Makefile.builder* file in its root directory. This file specifies what should be done to build the package. As name suggests, this is normal makefile, which is included by builder as its configuration. Its main purpose is to set some variables. Generally all available variables/settings are described as comments at the beginning of Makefile.\* in [QubesBuilder](/doc/qubes-builder/).
|
||||
|
||||
Variables for Linux build:
|
||||
|
||||
- `RPM_SPEC_FILES` List (space separated) of spec files for RPM package build. Path should be relative to component root directory. [QubesBuilder](/doc/qubes-builder/) will install all BuildRequires (in chroot environment) before the build. In most Qubes components all spec files are kept in *rpm\_spec* directory. This is mainly used for Fedora packages build.
|
||||
- `ARCH_BUILD_DIRS` List (space separated) of directories with PKGBUILD files for Archlinux package build. Similar to RPM build, [QubesBuilder](/doc/qubes-builder/) will install all makedepends, then build the package.
|
||||
|
||||
Most components uses *archlinux* directory for this purpose, so its good to keep this style.
|
||||
|
||||
Variables for Windows build:
|
||||
|
||||
- `WIN_COMPILER` Choose which compiler should be used for this component, thus which build scripts. Currently two options available:
|
||||
- `WDK` - Windows Driver Kit (default). Command used to build: *build -cZg*.
|
||||
- `mingw` - MinGW (Windows gcc port). Command used to build: *make all*
|
||||
- `WIN_SOURCE_SUBDIRS` List of directories in which above command should be run. In most cases it will be only one entry: current directory (*.*).
|
||||
- `WIN_PREBUILD_CMD` Command to run before build, mostly useful for WDK build (in mingw case, you can use makefile for this purpose). Can be used to set some variables, preprocess some files etc.
|
||||
- `WIN_SIGN_CMD` Command used to sign resulting binaries. Note that default value is *sign.bat*. If you don't want to sign binaries, specify some placeholder here (eg. *true*). Check existing components (eg. vmm-xen-windows-pvdrivers) for example scripts. This command will be run with certain environment variables:
|
||||
- `CERT_FILENAME` Path to key file (pfx format)
|
||||
- `CERT_PASSWORD` Key password
|
||||
- `CERT_PUBLIC_FILENAME` Certificate path in the case of self-signed cert
|
||||
- `CERT_CROSS_CERT_FILENAME` Certificate path in the case of correct autheticode cert
|
||||
- `SIGNTOOL` Path to signtool
|
||||
- `WIN_PACKAGE_CMD` Command used to produce installation package (msi or msm). Default value is *wix.bat*, similar to above - use *true* if you don't want this command.
|
||||
- `WIN_OUTPUT_HEADERS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with public headers of the package - for use in other components.
|
||||
- `WIN_OUTPUT_LIBS` Directory (relative to `WIN_SOURCE_SUBDIRS` element) with libraries (both DLL and implib) of the package - for use in other components. Note that [QubesBuilder](/doc/qubes-builder/) will copy files specified as *\$(WIN\_OUTPUT\_LIBS)/\*/\** to match WDK directory layout (*\<specified directory\>/\<arch directory\>/\<actual libraries\>*), so you in mingw build you need to place libraries in some additional subdirectory.
|
||||
- `WIN_BUILD_DEPS` List of components required to build this one. [QubesBuilder](/doc/qubes-builder/) will copy files specified with `WIN_OUTPUT_HEADERS` and `WIN_OUTPUT_LIBS` of those components to some directory and provide its path with `QUBES_INCLUDES` and `QUBES_LIBS` variables. Use those variables in your build scripts (*sources* or *Makefile* - depending on selected compiler). You can assume that the variables are always set and directories always exists, even if empty.
|
||||
|
||||
builder.conf settings
|
||||
---------------------
|
||||
|
||||
Most settings are documented in *builder.conf.default* file, which can be used as template the actual configuration.
|
||||
|
||||
**TODO**
|
||||
176
developer/building/qubes-builder.md
Normal file
176
developer/building/qubes-builder.md
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
---
|
||||
layout: doc
|
||||
title: Qubes Builder
|
||||
permalink: /doc/qubes-builder/
|
||||
redirect_from:
|
||||
- /en/doc/qubes-builder/
|
||||
- /doc/QubesBuilder/
|
||||
- /wiki/QubesBuilder/
|
||||
---
|
||||
|
||||
**Note: See [ISO building instructions](/doc/qubes-iso-building/) for a streamlined overview on how to use the build system.**
|
||||
|
||||
Building Qubes from scratch
|
||||
===========================
|
||||
|
||||
We have a fully automated build system for Qubes, that downloads, builds and
|
||||
packages all the Qubes components, and finally should spit out a ready-to-use
|
||||
installation ISO.
|
||||
|
||||
In order to use it, one should use an rpm-based distro, like Fedora :), and should ensure the following packages are installed:
|
||||
|
||||
- sudo
|
||||
- gnupg
|
||||
- git
|
||||
- createrepo
|
||||
- rpm-build
|
||||
- make
|
||||
- wget
|
||||
- rpmdevtools
|
||||
- python2-sh
|
||||
- dialog
|
||||
- rpm-sign
|
||||
- dpkg-dev
|
||||
- debootstrap
|
||||
- PyYAML
|
||||
- devscripts
|
||||
- perl-Digest-MD5
|
||||
- perl-Digest-SHA
|
||||
|
||||
Usually one can install those packages by just issuing:
|
||||
|
||||
sudo dnf install gnupg git createrepo rpm-build make wget rpmdevtools python2-sh dialog rpm-sign dpkg-dev debootstrap PyYAML devscripts perl-Digest-MD5 perl-Digest-SHA
|
||||
|
||||
The build system creates build environments in chroots and so no other packages are needed on the host. All files created by the build system are contained within the qubes-builder directory. The full build requires some 25GB of free space, so keep that in mind when deciding where to place this directory.
|
||||
|
||||
The build system is configured via builder.conf file -- one should copy the provided default builder.conf, and modify it as needed, e.g.:
|
||||
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
# edit the builder.conf file and set the following variables:
|
||||
NO_SIGN=1
|
||||
|
||||
One additional useful requirement is that 'sudo root' must work without any prompt, which is default on most distros (e.g. 'sudo bash' brings you the root shell without asking for any password). This is important as the builder needs to switch to root and then back to user several times during the build process.
|
||||
|
||||
Additionally, if building with signing enabled (NO\_SIGN is not set), one must adjust \~/.rpmmacro file so that it points to the GPG key used for package signing, e.g.:
|
||||
|
||||
%_signature gpg
|
||||
%_gpg_path /home/user/.gnupg
|
||||
%_gpg_name AC1BF9B3 # <-- Key ID used for signing
|
||||
|
||||
It is also recommended to use an empty passphrase for the private key used for signing. Contrary to a popular belief, this doesn't affect your key or sources security -- if somebody compromised your system, then the game is over anyway, whether you have used an additional passphrase for the key or not.
|
||||
|
||||
So, to build Qubes one would do:
|
||||
|
||||
# Import the Qubes master key
|
||||
gpg --recv-keys 0xDDFA1A3E36879494
|
||||
|
||||
# Verify its fingerprint, set as 'trusted'.
|
||||
# This is described here:
|
||||
# https://www.qubes-os.org/doc/VerifyingSignatures
|
||||
|
||||
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
|
||||
gpg --import qubes-developers-keys.asc
|
||||
|
||||
git clone git://github.com/QubesOS/qubes-builder.git qubes-builder
|
||||
cd qubes-builder
|
||||
|
||||
# Verify its integrity:
|
||||
git tag -v `git describe`
|
||||
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
# edit the builder.conf file and set the following variables:
|
||||
# NO_SIGN="1"
|
||||
|
||||
# Download all components:
|
||||
|
||||
make get-sources
|
||||
|
||||
# And now to build all Qubes rpms (this will take a few hours):
|
||||
|
||||
make qubes
|
||||
|
||||
# ... and then to build the ISO
|
||||
|
||||
make iso
|
||||
|
||||
And this should produce a shiny new ISO.
|
||||
|
||||
You can also build selected component separately. Eg. to compile only gui virtualization agent/daemon:
|
||||
|
||||
make gui-daemon
|
||||
|
||||
You can get a full list from make help. For advanced use and preparing sources
|
||||
for use with [QubesBuilder](/doc/qubes-builder/) take a look at [doc directory
|
||||
in QubesBuilder](https://github.com/marmarek/qubes-builder/tree/master/doc) or
|
||||
[QubesBuilderDetails](/doc/qubes-builder-details/) page.
|
||||
|
||||
Making customized build
|
||||
-----------------------
|
||||
|
||||
### Manual source modification
|
||||
|
||||
If you want to somehow modify sources, you can also do it, here are some basic steps:
|
||||
|
||||
1. Download qubes-builder as described above (if you want to use marmarek's branches, you should also download qubes-builder from his repo - replace 'QubesOS' with 'marmarek' in above git clone command)
|
||||
2. Edit builder.conf (still the same as above), some useful additions:
|
||||
- You can also set GIT\_PREFIX="marmarek/qubes-" to use my repo instead of "mainstream" - it contains newer (but less tested) versions
|
||||
|
||||
3. Download unmodified sources
|
||||
|
||||
make get-sources
|
||||
|
||||
4. **Make your modifications here**
|
||||
|
||||
5. Build the Qubes
|
||||
`make qubes` actually is just meta target which build all required
|
||||
components in correct order. List of components is configured in
|
||||
builder.conf. You can also check the current value at the end of `make
|
||||
help`, or using `make build-info`.
|
||||
|
||||
6. `get-sources` is already done, so continue with the next one. You can skip `sign-all` if you've disabled signing
|
||||
|
||||
make vmm-xen core-admin linux-kernel gui-daemon template desktop-linux-kde installer-qubes-os manager linux-dom0-updates
|
||||
|
||||
1. build iso installation image
|
||||
|
||||
make iso
|
||||
|
||||
### Use pre-built Qubes packages
|
||||
|
||||
For building just few selected packages, it's very useful to download pre-built qubes-specific dependencies from `{yum,deb}.qubes-os.org`.
|
||||
This is especially true for `gcc`, which takes several hours to build.
|
||||
|
||||
Before creating the `chroot`, add this to your `builder.conf`:
|
||||
|
||||
USE_QUBES_REPO_VERSION = $(RELEASE)
|
||||
|
||||
It will add the 'current' Qubes repository to your `chroot` environment.
|
||||
Next, specify which components (`gcc`, for example) you want to download instead of compiling:
|
||||
|
||||
COMPONENTS := $(filter-out gcc,$(COMPONENTS))
|
||||
|
||||
Alternatively, edit the actual COMPONENTS list which is defined in the included version-dependent config from example-configs (see series of include directives near the beginning of `builder.conf`).
|
||||
This way, you can build only the packages in which you are interested.
|
||||
|
||||
If you also want to use the 'current-testing' repository, add this to your configuration:
|
||||
|
||||
USE_QUBES_REPO_TESTING = 1
|
||||
|
||||
In the case of an existing `chroot`, for mock-enabled builds, it works immediately because `chroot` is constructed each time separately.
|
||||
For legacy builds, it will not add the necessary configuration into the build environment unless a specific builder change or configuration would force rebuilding chroot.
|
||||
|
||||
Also, once enabled, disabling this setting will not disable repositories in relevant chroots.
|
||||
And even if it did, there could be leftover packages installed from those repos (which may or may not be desirable).
|
||||
|
||||
Code verification keys management
|
||||
---------------------------------
|
||||
|
||||
[QubesBuilder](/doc/qubes-builder/) by default verifies signed tags on every downloaded code. Public keys used for that are stored in `keyrings/git`. By default Qubes developers' keys are imported automatically, but if you need some additional keys (for example your own), you can add them using:
|
||||
|
||||
GNUPGHOME=$PWD/keyrings/git gpg --import /path/to/key.asc
|
||||
GNUPGHOME=$PWD/keyrings/git gpg --edit-key ID_OF_JUST_IMPORTED_KEY
|
||||
# here use "trust" command to set key fully or ultimately trusted - only those keys are accepted by QubesBuilder
|
||||
|
||||
All Qubes developers' keys are signed by the Qubes Master Signing Key (which is set as ultimately trusted key), so are trusted automatically.
|
||||
|
||||
If you are the owner of Master key and want to revoke such signature, use the `revsig` gpg key edit command and update the key in qubes-developers-keys.asc - now the key will be no longer trusted (unless manually set as such).
|
||||
202
developer/building/qubes-iso-building.md
Normal file
202
developer/building/qubes-iso-building.md
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
---
|
||||
layout: doc
|
||||
title: Qubes ISO Building
|
||||
permalink: /doc/qubes-iso-building/
|
||||
redirect_from:
|
||||
- /doc/qubes-r3-building/
|
||||
- /en/doc/qubes-r3-building/
|
||||
- /en/doc/qubes-iso-building/
|
||||
- /doc/QubesR3Building/
|
||||
- /wiki/QubesR3Building/
|
||||
---
|
||||
|
||||
Building Qubes OS ISO
|
||||
=========================
|
||||
|
||||
Build Environment
|
||||
-----------------
|
||||
|
||||
Fedora 29 has been successfully used to build Qubes R4.0 with the below steps.
|
||||
Other rpm-based operating systems may also work.
|
||||
Travis-CI uses Ubuntu 14.04 to perform test builds, except it can not test the `./setup` script.
|
||||
|
||||
In `dom0`, install the Fedora 29 template if you don't already have it.
|
||||
|
||||
~~~
|
||||
sudo qubes-dom0-update qubes-template-fedora-29
|
||||
~~~
|
||||
|
||||
Create a standalone AppVM from the Fedora 29 template.
|
||||
Set private storage to at least 60 GB if you will be building only the default templates; 100 GB or more if you plan on additional.
|
||||
It's not required, but if you allocate additional CPU cores, the build process can utilize them at some steps such as the kernel build.
|
||||
Likewise, more memory (up to 16 GB) can help.
|
||||
Last, you may want to disable memory balancing, but keep in mind the impact on your other qubes.
|
||||
|
||||
Once you've built the development AppVM, open a Terminal window to it and install the necessary dependencies (see [QubesBuilder](/doc/qubes-builder/) for more info):
|
||||
|
||||
~~~
|
||||
$ sudo dnf install perl-Digest-SHA rpmdevtools rpm-build dialog rpm-sign python2-sh dpkg-dev debootstrap devscripts
|
||||
~~~
|
||||
|
||||
Get the necessary keys to verify the sources (run these and other commands below as a regular user, not root):
|
||||
|
||||
~~~
|
||||
wget https://keys.qubes-os.org/keys/qubes-master-signing-key.asc
|
||||
gpg --import qubes-master-signing-key.asc
|
||||
gpg --edit-key 36879494
|
||||
fpr
|
||||
# Verify fingerprint! See Note below!
|
||||
# Once verified, set trust to *ultimate*
|
||||
# (Typical sequence is trust, 5, q)
|
||||
wget https://keys.qubes-os.org/keys/qubes-developers-keys.asc
|
||||
gpg --import qubes-developers-keys.asc
|
||||
~~~
|
||||
|
||||
**Note** In the above process, we do *not* rely on the security of our server (keys.qubes-os.org) nor the connection (ssl, cert) -- we only rely on you getting the Qubes Master Signing Key fingerprint *somehow* and ensuring they match!
|
||||
See [Verifying Signatures](/security/verifying-signatures/#1-get-the-qubes-master-signing-key-and-verify-its-authenticity) for verification sources.
|
||||
|
||||
Now let's bootstrap the builder. Unfortunately, the builder cannot verify itself (the classic Chicken and Egg problem), so we need to verify the signature manually:
|
||||
|
||||
~~~
|
||||
git clone git://github.com/QubesOS/qubes-builder.git
|
||||
cd qubes-builder
|
||||
git tag -v `git describe`
|
||||
~~~
|
||||
|
||||
**Note** It's very important to check if the verification message contains "Good signature from ..." and does not contain "WARNING: This key is not certified with a trusted signature!".
|
||||
|
||||
Assuming the verification went fine, we're good to go with all the rest without ever thinking more about verifying digital signatures on all the rest of the components, apart from an additional step if doing a non-scripted build.
|
||||
The builder will do that for us for each component, every time we build, even for all auxiliary files (e.g. Xen or Linux kernel sources).
|
||||
|
||||
|
||||
Build using setup script
|
||||
-----------------
|
||||
|
||||
Let's configure the builder first (see [procedure](/doc/qubes-iso-building/#build-using-manual-steps) at bottom if you would prefer to manually configure):
|
||||
|
||||
~~~
|
||||
cd ~/qubes-builder
|
||||
./setup
|
||||
# Select Yes to add Qubes Master Signing Key
|
||||
# Select Yes to add Qubes OS Signing Key
|
||||
# Select 4.0 for version
|
||||
# Stable
|
||||
# Select Current (if you want the option to use pre-built packages)
|
||||
# No (we want a full build)
|
||||
# Select fc29 and stretch (for the currently shipping templates)
|
||||
# Select builder-rpm, builder-debian, template-whonix, mgmt-salt
|
||||
# Yes (to download)
|
||||
~~~
|
||||
|
||||
Once it completes downloading, re-run `setup` to add the Whonix templates:
|
||||
|
||||
~~~
|
||||
./setup
|
||||
# Choose the same options as above, except at templates select:
|
||||
# fc29, stretch, whonix-gateway-14, whonix-workstation-14
|
||||
# If prompted, choose Yes to add adrelanos's third party key
|
||||
~~~
|
||||
|
||||
Continue the build process with:
|
||||
|
||||
~~~
|
||||
make install-deps
|
||||
make get-sources
|
||||
~~~
|
||||
|
||||
When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable in `builder.conf` at this stage to specify the currently shipping Tor Browser version.
|
||||
See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/).
|
||||
|
||||
You may also want to add `COMPONENTS := $(filter-out gcc,$(COMPONENTS))` to bypass a multiple hour compile step.
|
||||
See [QubesBuilder](/doc/qubes-builder/#use-pre-built-qubes-packages) for more detail.
|
||||
|
||||
Finally, if you are making a test build, use:
|
||||
|
||||
~~~
|
||||
make qubes
|
||||
make iso
|
||||
~~~
|
||||
|
||||
Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`):
|
||||
|
||||
~~~
|
||||
make qubes
|
||||
make sign-all
|
||||
make iso
|
||||
~~~
|
||||
|
||||
Enjoy your new ISO!
|
||||
|
||||
|
||||
Build using manual steps
|
||||
-----------------
|
||||
|
||||
Instead of using `./setup`, you can manually configure the build.
|
||||
The script takes care of a lot of the keyring preparation for us, so we first need to set that up.
|
||||
|
||||
If you will be building Whonix templates:
|
||||
|
||||
~~~
|
||||
cd ~
|
||||
gpg --keyserver pgp.mit.edu --recv-keys 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
|
||||
gpg --fingerprint 916B8D99C38EAF5E8ADC7A2A8D66066A2EEACCDA
|
||||
~~~
|
||||
|
||||
**Note:** It's very important to check the fingerprint displayed against multiple sources such as the [Whonix web site](https://www.whonix.org/wiki/Whonix_Signing_Key), etc.
|
||||
It should look something like this:
|
||||
|
||||
~~~
|
||||
pub 4096R/2EEACCDA 2014-01-16 [expires: 2021-04-17]
|
||||
Key fingerprint = 916B 8D99 C38E AF5E 8ADC 7A2A 8D66 066A 2EEA CCDA
|
||||
uid Patrick Schleizer <adrelanos@riseup.net>
|
||||
sub 4096R/CE998547 2014-01-16 [expires: 2021-04-17]
|
||||
sub 4096R/119B3FD6 2014-01-16 [expires: 2021-04-17]
|
||||
sub 4096R/77BB3C48 2014-01-16 [expires: 2021-04-17]
|
||||
~~~
|
||||
|
||||
Next, prepare the Git keyring directory and copy them in:
|
||||
|
||||
~~~
|
||||
export GNUPGHOME=~/qubes-builder/keyrings/git
|
||||
mkdir --parents "$GNUPGHOME"
|
||||
cp ~/.gnupg/pubring.gpg "$GNUPGHOME"
|
||||
cp ~/.gnupg/trustdb.gpg "$GNUPGHOME"
|
||||
chmod --recursive 700 "$GNUPGHOME"
|
||||
~~~
|
||||
|
||||
Copy one of the example configurations:
|
||||
|
||||
~~~
|
||||
cd ~/qubes-builder
|
||||
cp example-configs/qubes-os-master.conf builder.conf
|
||||
~~~
|
||||
|
||||
Edit `builder.conf`, referring to `doc/Configuration.md` for a description of all available options.
|
||||
|
||||
Continue the build process with:
|
||||
|
||||
~~~
|
||||
make install-deps
|
||||
make get-sources
|
||||
unset GNUPGHOME
|
||||
~~~
|
||||
|
||||
When building the Whonix templates, you will often need to add/update the `WHONIX_TBB_VERSION` variable at this stage to specify the currently shipping Tor Browser version.
|
||||
See the related note under [Extra Whonix Build Options](/doc/building-whonix-template/).
|
||||
|
||||
Finally, if you are making a test build, use:
|
||||
|
||||
~~~
|
||||
make qubes
|
||||
make iso
|
||||
~~~
|
||||
|
||||
Or for a fully signed build (this requires setting `SIGN_KEY` in `builder.conf`):
|
||||
|
||||
~~~
|
||||
make qubes
|
||||
make sign-all
|
||||
make iso
|
||||
~~~
|
||||
|
||||
Enjoy your new ISO!
|
||||
Loading…
Add table
Add a link
Reference in a new issue