From 44ea629d9ff247c3d25e97aae397a15659929e30 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Fri, 28 Aug 2020 02:52:01 +0800 Subject: [PATCH 01/20] Include documentation for template manager design --- developer/system/template-manager.md | 253 +++++++++++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 developer/system/template-manager.md diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md new file mode 100644 index 00000000..cf4b470c --- /dev/null +++ b/developer/system/template-manager.md @@ -0,0 +1,253 @@ +Template Manager Design +======================= + +Motivation +---------- + +Previously, template VMs were distributed by RPM packages and managed by `yum`/`dnf`. +However, tracking inherently dynamic VM images with a package manager suited for static files creates some challenges. +For example, users may accidentally update the images, overriding local changes ([#996], [#1647]). +(Or in the case of [#2061], want to specifically override the changes.) +Other operations that work well on normal VMs are also somewhat inconsistent on RPM-managed templates. +This includes actions such as renaming ([#839]), removal ([#5509]) and backup/restore ([#1385], [#1453], [discussion thread 1], [discussion thread 2]). +In turn, this creates inconveniences and confusion for users ([#1403], [#4518]). + +Also, the usage of RPM packages meant that installing a template results in arbitrary code execution, which is not ideal. + +Besides distribution, users may also wish to have an integrated template management application ([#2062], [#2064], [#2534], [#3040]), as opposed to the situation where multiple programs are required for different purposes, e.g., `qubes-dom0-update`, `dnf`, `qvm-remove`, `qubes-manager`. + +To tackle these issues, `qvm-template` is created. +It strives to provide not only a better mechanism for handling template installation but also a consistent user-facing interface to deal with template management. + +[#996]: https://github.com/QubesOS/qubes-issues/issues/996 + +[#1647]: https://github.com/QubesOS/qubes-issues/issues/1647 + +[#2061]: https://github.com/QubesOS/qubes-issues/issues/2061 + +[#839]: https://github.com/QubesOS/qubes-issues/issues/839 + +[#5509]: https://github.com/QubesOS/qubes-issues/issues/5509 + +[#1385]: https://github.com/QubesOS/qubes-issues/issues/1385 + +[#1453]: https://github.com/QubesOS/qubes-issues/issues/1453 + +[#1403]: https://github.com/QubesOS/qubes-issues/issues/1403 + +[#4518]: https://github.com/QubesOS/qubes-issues/issues/4518 + +[#2062]: https://github.com/QubesOS/qubes-issues/issues/2062 + +[#2064]: https://github.com/QubesOS/qubes-issues/issues/2064 + +[#2534]: https://github.com/QubesOS/qubes-issues/issues/2534 + +[#3040]: https://github.com/QubesOS/qubes-issues/issues/3040 + +[discussion thread 1]: https://groups.google.com/forum/#!topic/qubes-devel/rwc2_miCNNE/discussion + +[discussion thread 2]: https://groups.google.com/forum/#!topic/qubes-users/uQEUpv4THsY/discussion + +Features +-------- + +- Install/reinstall/downgrade/upgrade templates, either from local packages or remote repositories + - Ability to install templates in alternative pools + - Possibility for the template package to specify options such as the kernel or `virt_mode` used +- List and show information about local and available templates + - Machine-readable output for easy extension +- Search for templates +- Remove templates + - Ability to disassociate or remove VMs based on the template to be removed +- Show available repositories +- Works in both dom0 and management VMs by utilizing the Admin API +- Works well with existing tools +- Command-line interface with DNF-like usage +- A graphical interface also available + +Package Format +-------------- + +The RPM package format is still used. +However, the contents are manually extracted instead of installing the whole package. +This allows us to take advantage of existing tools for things like repository management. +We can also avoid the burden of dealing with verification, reducing the risk of issues like [QSB-28]. + +The package name should be in the form `qubes-template-${template_name}`. + +For the package metadata (e.g., summary, description), one needs to ensure that the character `|` does not appear. +This is because we use the character as a separator internally. +Failing to do so may result in `qvm-template` to fail with cryptic error messages. +Note that as we already consider the repository metadata untrusted, this should not result in security issues. + +The file structure should be quite similar to previous template RPMs. +Namely, there should be the following files in the package: + +- `var/lib/qubes/vm-templates/${template_name}/root.img.part.{00,01,...}` + - Split tarball of template `root.img` + - Note that the file is still split due to tools such as `rpm2cpio` not supporting large files. + (Notably, the cpio format does not support files over 4GiB.) +- `var/lib/qubes/vm-templates/${template_name}/template.conf` + - Stores custom package metadata (as RPM does not support custom attributes). + - Uses `KEY=VALUE` format. + - Fields (corresponding to [qvm-prefs]/[qvm-features] tags with the same names) + - `virt_mode` + - Setting this to `pv` requires user confirmation. + - `kernel` + - Only allowed to be set to "", i.e., "none", for PVGrub. + - `no-monitor-layout` + - `pci-e820-host` + - `linux-stubdom` + - `gui` + - `gui-emulated` + - `qrexec` + - These six flags are only allowed to be set to "1" or "0", denoting "true" and "false" respectively. + - `net.fake-ip` + - `net.fake-gateway` + - `net.fake-netmask` +- `var/lib/qubes/vm-templates/${template_name}/whitelisted-appmenus.list` + - Contains default app menu entries of the template itself. +- `var/lib/qubes/vm-templates/${template_name}/vm-whitelisted-appmenus.list` + - Contains default app menu entries of VMs based on the template. +- `var/lib/qubes/vm-templates/${template_name}/netvm-whitelisted-appmenus.list` + - Contains default app menu entries of NetVMs based on the template. + - These three files are the same as the current format. + - Note that the contents of these files are stored in `qvm-features` upon installation. + See the section below for details. + +[QSB-28]: https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-028-2016.txt + +[qvm-prefs]: https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-prefs.html#common-properties + +[qvm-features]: https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-features.html#list-of-known-features + +Metadata Storage +---------------- + +The template manager needs to keep metadata of installed templates such as versions and origin. +This data can be stored via `qvm-features` to keep things consistent when, e.g., `qvm-remove` is used. +Besides, backups are also more easily handled this way. + +Also, the fields can serve as an indicator of whether a template is installed by `qvm-template`. + +### Fields ### + +Most of the fields should be fairly self-explanatory. + +- `template-name` + - Note that this field needs to be consistent with the template name to be considered valid. +- `template-epoch` +- `template-version` +- `template-release` +- `template-reponame` +- `template-buildtime` +- `template-install-date` + - The times are in ISO 8601 format, and can be parsed in Python with `datetime.datetime.fromisoformat()`. +- `template-licence` +- `template-url` +- `template-summary` +- `template-description` + - Note that the newlines in this field are converted to `|` to work better with existing tools like `qvm-features`. +- `menu-items` +- `default-menu-items` +- `netvm-menu-items` + - The `*menu-items` entries store the contents of `var/lib/qubes/vm-templates/${template_name}/whitelisted-appmenus.list`, `var/lib/qubes/vm-templates/${template_name}/vm-whitelisted-appmenus.list`, `var/lib/qubes/vm-templates/${template_name}/netvm-whitelisted-appmenus.list` respectively. + - Note that newlines are converted to spaces, again for it to work better with existing tools. + This should not cause ambiguity as [the FreeDesktop specifications](freedesktop-spec) forbid spaces from the names of desktop files. + +[freedesktop-spec]: https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html + +Repository Management +--------------------- + +For UpdateVMs to access repository configuration, the package [qubes-repo-templates] is created with the following contents: + +- `/usr/share/qubes/repo-templates/*.repo`: repository configuration +- `/usr/share/qubes/repo-templates/keys`: PGP keys + +As template keys may be less trusted, they are *not* added to the system RPM keychain but instead managed separately. + +[qubes-repo-templates]: https://github.com/WillyPillow/qubes-repo-templates + +Qrexec Protocol +--------------- + +Dom0 and management VMs without network access also need to interact with template repositories. +The following qrexec calls that list and download templates are thus proposed. + +- `qubes.TemplateSearch`: wraps `dnf repoquery` +- `qubes.TemplateDownload`: wraps `dnf download` + +### Input ### + +Both calls accept the following format from standard input: + +``` +arg1 +arg2 +... +argN +package-file-spec +--- +repo config +``` + +In other words, the input consists of two parts separated by the line `---`. +The first part contains some arguments and `package-file-spec` that indicates the pattern to be queried or downloaded. +The following arguments are allowed: + +- `--enablerepo=` +- `--disablerepo=` +- `--repoid=` +- `--releasever=` +- `--refresh` + +where the usage is identical to that of DNF. + +For the exact definition of `package-file-spec`, refer to the DNF documentation. + +The second part contains the repository configurations in `yum.repos.d` format. + +### Output ### + +`qubes.TemplateSearch` prints each package in `%{name}|%{epoch}|%{version}|%{release}|%{reponame}|%{downloadsize}|%{buildtime}|%{license}|%{url}|%{summary}|%{description}|` format to standard output, separated by newlines. +Note that there is a `|` at the end of the line. +This is because `%{description}` may contain newlines, and doing so allows us to split the entries by `|\n`. +(As we are using `dnf repoquery --qf`, we are unable to escape the newlines in advance.) + +`qubes.TemplateDownload`, on the other hand, directly outputs the downloaded content to standard output. + +Machine-readable Output +----------------------- + +The commands `qvm-template list` and `qvm-template info` provide machine-readable output in both pipe(`|`)-separated and JSON format. +See the `qvm-template` man page for details. + +Interactions with Existing Tools +-------------------------------- + +### `qvm-remove` ### + +Should work straightforwardly. + +### Renaming / Cloning ### + +A template is treated as non-manager-installed once renamed or cloned. +However, relevant metadata in the VM features is still retained for future extension and to serve as a hint for the user. + +Further Reading +--------------- + +Initial Google Summer of Code (2020) project proposal: +- + +Previous design document: +- + +Discussion threads: +- +- +- +- +- From 3b314c84f700163c1443f38155145842ead6fee9 Mon Sep 17 00:00:00 2001 From: Andrew David Wong Date: Fri, 28 Aug 2020 11:13:04 -0500 Subject: [PATCH 02/20] Add YAML header --- developer/system/template-manager.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index cf4b470c..d0a8a81f 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -1,3 +1,9 @@ +--- +layout: doc +title: Template Manager +permalink: /doc/template-manager/ +--- + Template Manager Design ======================= From 9249f074b35466f39f168865eb3449d524a1f982 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Sat, 29 Aug 2020 17:38:01 +0800 Subject: [PATCH 03/20] Fix whitespace. --- developer/system/template-manager.md | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index d0a8a81f..2b65b16b 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -26,29 +26,17 @@ To tackle these issues, `qvm-template` is created. It strives to provide not only a better mechanism for handling template installation but also a consistent user-facing interface to deal with template management. [#996]: https://github.com/QubesOS/qubes-issues/issues/996 - [#1647]: https://github.com/QubesOS/qubes-issues/issues/1647 - [#2061]: https://github.com/QubesOS/qubes-issues/issues/2061 - [#839]: https://github.com/QubesOS/qubes-issues/issues/839 - [#5509]: https://github.com/QubesOS/qubes-issues/issues/5509 - [#1385]: https://github.com/QubesOS/qubes-issues/issues/1385 - [#1453]: https://github.com/QubesOS/qubes-issues/issues/1453 - [#1403]: https://github.com/QubesOS/qubes-issues/issues/1403 - [#4518]: https://github.com/QubesOS/qubes-issues/issues/4518 - [#2062]: https://github.com/QubesOS/qubes-issues/issues/2062 - [#2064]: https://github.com/QubesOS/qubes-issues/issues/2064 - [#2534]: https://github.com/QubesOS/qubes-issues/issues/2534 - [#3040]: https://github.com/QubesOS/qubes-issues/issues/3040 [discussion thread 1]: https://groups.google.com/forum/#!topic/qubes-devel/rwc2_miCNNE/discussion @@ -246,12 +234,15 @@ Further Reading --------------- Initial Google Summer of Code (2020) project proposal: + - Previous design document: + - Discussion threads: + - - - From 54629daa32c5d42d83670014a8538dada2a76160 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Sat, 29 Aug 2020 17:39:09 +0800 Subject: [PATCH 04/20] Change title to "Template Manager". --- developer/system/template-manager.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 2b65b16b..218a2f52 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -4,8 +4,8 @@ title: Template Manager permalink: /doc/template-manager/ --- -Template Manager Design -======================= +Template Manager +================ Motivation ---------- From 3ebde47b2332615d625280bbe8280d36079a4822 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Sat, 29 Aug 2020 20:49:10 +0800 Subject: [PATCH 05/20] Replace "template VM" with "TemplateVM". --- developer/system/template-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 218a2f52..884c5098 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -10,7 +10,7 @@ Template Manager Motivation ---------- -Previously, template VMs were distributed by RPM packages and managed by `yum`/`dnf`. +Previously, TemplateVMs were distributed by RPM packages and managed by `yum`/`dnf`. However, tracking inherently dynamic VM images with a package manager suited for static files creates some challenges. For example, users may accidentally update the images, overriding local changes ([#996], [#1647]). (Or in the case of [#2061], want to specifically override the changes.) From ec37cc6984079e1c9e821f9fbbcde903d2ece278 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Sat, 29 Aug 2020 21:04:36 +0800 Subject: [PATCH 06/20] Add "Introduction" section. --- developer/system/template-manager.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 884c5098..476d1fab 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -7,6 +7,12 @@ permalink: /doc/template-manager/ Template Manager ================ +Introduction +------------ + +This document discusses the designs and technical details of `qvm-template`, a template manager application proposed in the 2020 Google Summer of Code program. +The goals of the project is to design a new mechanism for TemplateVM distribution and a unified tool for template management. + Motivation ---------- From 56cf2b768e6bfeede78f34afbac4f91d649a5c65 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Sat, 29 Aug 2020 21:07:49 +0800 Subject: [PATCH 07/20] Fix whitespace. --- developer/system/template-manager.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 476d1fab..2400cad0 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -44,9 +44,7 @@ It strives to provide not only a better mechanism for handling template installa [#2064]: https://github.com/QubesOS/qubes-issues/issues/2064 [#2534]: https://github.com/QubesOS/qubes-issues/issues/2534 [#3040]: https://github.com/QubesOS/qubes-issues/issues/3040 - [discussion thread 1]: https://groups.google.com/forum/#!topic/qubes-devel/rwc2_miCNNE/discussion - [discussion thread 2]: https://groups.google.com/forum/#!topic/qubes-users/uQEUpv4THsY/discussion Features @@ -117,9 +115,7 @@ Namely, there should be the following files in the package: See the section below for details. [QSB-28]: https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-028-2016.txt - [qvm-prefs]: https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-prefs.html#common-properties - [qvm-features]: https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-features.html#list-of-known-features Metadata Storage @@ -183,7 +179,7 @@ The following qrexec calls that list and download templates are thus proposed. Both calls accept the following format from standard input: -``` +``` text arg1 arg2 ... From dd42704add8e5db483c802352f80ff825f733afa Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Sat, 29 Aug 2020 21:14:37 +0800 Subject: [PATCH 08/20] Fix link syntax. --- developer/system/template-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 2400cad0..d812fb08 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -150,7 +150,7 @@ Most of the fields should be fairly self-explanatory. - `netvm-menu-items` - The `*menu-items` entries store the contents of `var/lib/qubes/vm-templates/${template_name}/whitelisted-appmenus.list`, `var/lib/qubes/vm-templates/${template_name}/vm-whitelisted-appmenus.list`, `var/lib/qubes/vm-templates/${template_name}/netvm-whitelisted-appmenus.list` respectively. - Note that newlines are converted to spaces, again for it to work better with existing tools. - This should not cause ambiguity as [the FreeDesktop specifications](freedesktop-spec) forbid spaces from the names of desktop files. + This should not cause ambiguity as [the FreeDesktop specifications][freedesktop-spec] forbid spaces from the names of desktop files. [freedesktop-spec]: https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html From 95ee4559ab057130390f249102847af86ff85341 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Sun, 30 Aug 2020 02:04:35 +0800 Subject: [PATCH 09/20] Fix incorrect feature name --- developer/system/template-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index d812fb08..13f590cf 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -138,7 +138,7 @@ Most of the fields should be fairly self-explanatory. - `template-release` - `template-reponame` - `template-buildtime` -- `template-install-date` +- `template-installtime` - The times are in ISO 8601 format, and can be parsed in Python with `datetime.datetime.fromisoformat()`. - `template-licence` - `template-url` From a99b53ec11af55e73bf46719758886fb6d1fcbab Mon Sep 17 00:00:00 2001 From: Andrew David Wong Date: Sat, 29 Aug 2020 15:34:07 -0500 Subject: [PATCH 10/20] Remove superfluous "Introduction" heading --- developer/system/template-manager.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 13f590cf..ded0a7c5 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -7,9 +7,6 @@ permalink: /doc/template-manager/ Template Manager ================ -Introduction ------------- - This document discusses the designs and technical details of `qvm-template`, a template manager application proposed in the 2020 Google Summer of Code program. The goals of the project is to design a new mechanism for TemplateVM distribution and a unified tool for template management. From 14737126181cb73c78b75d9c859916ea3d046416 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Wed, 2 Sep 2020 01:06:20 +0800 Subject: [PATCH 11/20] Replace with atx headers --- developer/system/template-manager.md | 40 +++++++++++----------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index ded0a7c5..506f6231 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -4,14 +4,12 @@ title: Template Manager permalink: /doc/template-manager/ --- -Template Manager -================ +# Template Manager This document discusses the designs and technical details of `qvm-template`, a template manager application proposed in the 2020 Google Summer of Code program. The goals of the project is to design a new mechanism for TemplateVM distribution and a unified tool for template management. -Motivation ----------- +## Motivation Previously, TemplateVMs were distributed by RPM packages and managed by `yum`/`dnf`. However, tracking inherently dynamic VM images with a package manager suited for static files creates some challenges. @@ -44,8 +42,7 @@ It strives to provide not only a better mechanism for handling template installa [discussion thread 1]: https://groups.google.com/forum/#!topic/qubes-devel/rwc2_miCNNE/discussion [discussion thread 2]: https://groups.google.com/forum/#!topic/qubes-users/uQEUpv4THsY/discussion -Features --------- +## Features - Install/reinstall/downgrade/upgrade templates, either from local packages or remote repositories - Ability to install templates in alternative pools @@ -61,8 +58,7 @@ Features - Command-line interface with DNF-like usage - A graphical interface also available -Package Format --------------- +## Package Format The RPM package format is still used. However, the contents are manually extracted instead of installing the whole package. @@ -115,8 +111,7 @@ Namely, there should be the following files in the package: [qvm-prefs]: https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-prefs.html#common-properties [qvm-features]: https://dev.qubes-os.org/projects/core-admin-client/en/stable/manpages/qvm-features.html#list-of-known-features -Metadata Storage ----------------- +## Metadata Storage The template manager needs to keep metadata of installed templates such as versions and origin. This data can be stored via `qvm-features` to keep things consistent when, e.g., `qvm-remove` is used. @@ -124,7 +119,7 @@ Besides, backups are also more easily handled this way. Also, the fields can serve as an indicator of whether a template is installed by `qvm-template`. -### Fields ### +### Fields Most of the fields should be fairly self-explanatory. @@ -151,8 +146,7 @@ Most of the fields should be fairly self-explanatory. [freedesktop-spec]: https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html -Repository Management ---------------------- +## Repository Management For UpdateVMs to access repository configuration, the package [qubes-repo-templates] is created with the following contents: @@ -163,8 +157,7 @@ As template keys may be less trusted, they are *not* added to the system RPM key [qubes-repo-templates]: https://github.com/WillyPillow/qubes-repo-templates -Qrexec Protocol ---------------- +## Qrexec Protocol Dom0 and management VMs without network access also need to interact with template repositories. The following qrexec calls that list and download templates are thus proposed. @@ -172,7 +165,7 @@ The following qrexec calls that list and download templates are thus proposed. - `qubes.TemplateSearch`: wraps `dnf repoquery` - `qubes.TemplateDownload`: wraps `dnf download` -### Input ### +### Input Both calls accept the following format from standard input: @@ -202,7 +195,7 @@ For the exact definition of `package-file-spec`, refer to the DNF documentation. The second part contains the repository configurations in `yum.repos.d` format. -### Output ### +### Output `qubes.TemplateSearch` prints each package in `%{name}|%{epoch}|%{version}|%{release}|%{reponame}|%{downloadsize}|%{buildtime}|%{license}|%{url}|%{summary}|%{description}|` format to standard output, separated by newlines. Note that there is a `|` at the end of the line. @@ -211,26 +204,23 @@ This is because `%{description}` may contain newlines, and doing so allows us to `qubes.TemplateDownload`, on the other hand, directly outputs the downloaded content to standard output. -Machine-readable Output ------------------------ +## Machine-readable Output The commands `qvm-template list` and `qvm-template info` provide machine-readable output in both pipe(`|`)-separated and JSON format. See the `qvm-template` man page for details. -Interactions with Existing Tools --------------------------------- +## Interactions with Existing Tools -### `qvm-remove` ### +### `qvm-remove` Should work straightforwardly. -### Renaming / Cloning ### +### Renaming / Cloning A template is treated as non-manager-installed once renamed or cloned. However, relevant metadata in the VM features is still retained for future extension and to serve as a hint for the user. -Further Reading ---------------- +## Further Reading Initial Google Summer of Code (2020) project proposal: From 61fb01d6730b2e2d60e9b5d997a0516c5b1a390c Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Wed, 2 Sep 2020 01:10:49 +0800 Subject: [PATCH 12/20] Move mention of GSoC to Motivations section --- developer/system/template-manager.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 506f6231..98b56a8c 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -6,11 +6,13 @@ permalink: /doc/template-manager/ # Template Manager -This document discusses the designs and technical details of `qvm-template`, a template manager application proposed in the 2020 Google Summer of Code program. +This document discusses the designs and technical details of `qvm-template`, the template manager application in Qubes OS. The goals of the project is to design a new mechanism for TemplateVM distribution and a unified tool for template management. ## Motivation +This project was originally proposed in the 2020 Google Summer of Code program. + Previously, TemplateVMs were distributed by RPM packages and managed by `yum`/`dnf`. However, tracking inherently dynamic VM images with a package manager suited for static files creates some challenges. For example, users may accidentally update the images, overriding local changes ([#996], [#1647]). From 96dce2dada8ff3b2dc1cb439c45be9bd5b19a095 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Fri, 4 Sep 2020 02:04:42 +0800 Subject: [PATCH 13/20] Add indication that qvm-template is not yet merged --- developer/system/template-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 98b56a8c..ccd92370 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -6,7 +6,7 @@ permalink: /doc/template-manager/ # Template Manager -This document discusses the designs and technical details of `qvm-template`, the template manager application in Qubes OS. +This document discusses the designs and technical details of `qvm-template`, a soon-to-be template manager application currently in development for Qubes OS. The goals of the project is to design a new mechanism for TemplateVM distribution and a unified tool for template management. ## Motivation From 01110f8253564159d39f6bd228c2520cc9bb1699 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Fri, 4 Sep 2020 02:21:38 +0800 Subject: [PATCH 14/20] Fix Google groups link --- developer/system/template-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index ccd92370..ecb78d7c 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -238,4 +238,4 @@ Discussion threads: - - - -- +- From 2de76980aeec629b37d20b5d6f83b34224f5e406 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Tue, 15 Sep 2020 00:31:11 +0800 Subject: [PATCH 15/20] Update info regarding time format --- developer/system/template-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index ecb78d7c..1f55d79c 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -133,7 +133,7 @@ Most of the fields should be fairly self-explanatory. - `template-reponame` - `template-buildtime` - `template-installtime` - - The times are in ISO 8601 format, and can be parsed in Python with `datetime.datetime.fromisoformat()`. + - The times are in UTC, and are of the format `YYYY-MM-DD HH:MM:SS`. - `template-licence` - `template-url` - `template-summary` From 2bc69fc46637d1c4552d98804a8ad0d288b687f7 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Tue, 15 Sep 2020 11:56:04 +0800 Subject: [PATCH 16/20] Tweak slightly for clarity --- developer/system/template-manager.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 1f55d79c..1300f711 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -53,7 +53,7 @@ It strives to provide not only a better mechanism for handling template installa - Machine-readable output for easy extension - Search for templates - Remove templates - - Ability to disassociate or remove VMs based on the template to be removed + - Optionally, VMs based on the template to be removed can be either removed or "disassociated" -- namely, have their templates changed to a "dummy" one. - Show available repositories - Works in both dom0 and management VMs by utilizing the Admin API - Works well with existing tools @@ -69,9 +69,8 @@ We can also avoid the burden of dealing with verification, reducing the risk of The package name should be in the form `qubes-template-${template_name}`. -For the package metadata (e.g., summary, description), one needs to ensure that the character `|` does not appear. -This is because we use the character as a separator internally. -Failing to do so may result in `qvm-template` to fail with cryptic error messages. +The package metadata (summary, description, etc.) should not contain the `|` character to avoid possibly cryptic errors. +This is because of its use as an internal separator. Note that as we already consider the repository metadata untrusted, this should not result in security issues. The file structure should be quite similar to previous template RPMs. @@ -89,16 +88,16 @@ Namely, there should be the following files in the package: - Setting this to `pv` requires user confirmation. - `kernel` - Only allowed to be set to "", i.e., "none", for PVGrub. - - `no-monitor-layout` - - `pci-e820-host` - - `linux-stubdom` - - `gui` - - `gui-emulated` - - `qrexec` - - These six flags are only allowed to be set to "1" or "0", denoting "true" and "false" respectively. - `net.fake-ip` - `net.fake-gateway` - `net.fake-netmask` + - Boolean flags: (Permitted values are "1" and "0", denoting "true" and "false" respectively.) + - `no-monitor-layout` + - `pci-e820-host` + - `linux-stubdom` + - `gui` + - `gui-emulated` + - `qrexec` - `var/lib/qubes/vm-templates/${template_name}/whitelisted-appmenus.list` - Contains default app menu entries of the template itself. - `var/lib/qubes/vm-templates/${template_name}/vm-whitelisted-appmenus.list` @@ -144,7 +143,7 @@ Most of the fields should be fairly self-explanatory. - `netvm-menu-items` - The `*menu-items` entries store the contents of `var/lib/qubes/vm-templates/${template_name}/whitelisted-appmenus.list`, `var/lib/qubes/vm-templates/${template_name}/vm-whitelisted-appmenus.list`, `var/lib/qubes/vm-templates/${template_name}/netvm-whitelisted-appmenus.list` respectively. - Note that newlines are converted to spaces, again for it to work better with existing tools. - This should not cause ambiguity as [the FreeDesktop specifications][freedesktop-spec] forbid spaces from the names of desktop files. + This should not cause ambiguity as [the FreeDesktop specifications][freedesktop-spec] forbid spaces in .desktop file names. [freedesktop-spec]: https://specifications.freedesktop.org/desktop-entry-spec/desktop-entry-spec-latest.html @@ -215,7 +214,10 @@ See the `qvm-template` man page for details. ### `qvm-remove` -Should work straightforwardly. +The existing `qvm-remove` tool should behave identically to `qvm-template remove` -- albeit without fancy features like disassociation. +This is unlike the previous situation where `qvm-remove` cannot remove RPM-installed templates. + +Notably, the metadata needs no special handling as it is stored in VM features and thus automatically consistent. ### Renaming / Cloning From 8716d23f48242d1ec9047b0b23bd18175738c86c Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Tue, 8 Jun 2021 13:39:02 +0800 Subject: [PATCH 17/20] qvm-template: Reflect path changes in qubes-repo-templates See https://github.com/QubesOS/qubes-repo-templates/commit/337efb1d3c8e518e9adc688b590fa617992648a4. --- developer/system/template-manager.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 1300f711..b83f0856 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -151,8 +151,8 @@ Most of the fields should be fairly self-explanatory. For UpdateVMs to access repository configuration, the package [qubes-repo-templates] is created with the following contents: -- `/usr/share/qubes/repo-templates/*.repo`: repository configuration -- `/usr/share/qubes/repo-templates/keys`: PGP keys +- `/etc/qubes/repo-templates/*.repo`: repository configuration +- `/etc/qubes/repo-templates/keys`: PGP keys As template keys may be less trusted, they are *not* added to the system RPM keychain but instead managed separately. @@ -170,7 +170,7 @@ The following qrexec calls that list and download templates are thus proposed. Both calls accept the following format from standard input: -``` text +```text arg1 arg2 ... From 76232f9a1fb4566b5a2a0ba9eb928911228bb1d2 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Tue, 8 Jun 2021 13:40:25 +0800 Subject: [PATCH 18/20] qvm-template: Change `template-licence` to `template-license` --- developer/system/template-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index b83f0856..4120965b 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -133,7 +133,7 @@ Most of the fields should be fairly self-explanatory. - `template-buildtime` - `template-installtime` - The times are in UTC, and are of the format `YYYY-MM-DD HH:MM:SS`. -- `template-licence` +- `template-license` - `template-url` - `template-summary` - `template-description` From a77ff318bd58ec00058de4b33d967c8a800d7978 Mon Sep 17 00:00:00 2001 From: WillyPillow Date: Tue, 8 Jun 2021 13:49:07 +0800 Subject: [PATCH 19/20] qvm-template: Update constraints regarding template RPM metadata --- developer/system/template-manager.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 4120965b..730512cc 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -86,11 +86,13 @@ Namely, there should be the following files in the package: - Fields (corresponding to [qvm-prefs]/[qvm-features] tags with the same names) - `virt_mode` - Setting this to `pv` requires user confirmation. + - Permitted values: `pv`, `pvh`, `hvm`. - `kernel` - - Only allowed to be set to "", i.e., "none", for PVGrub. - - `net.fake-ip` - - `net.fake-gateway` - - `net.fake-netmask` + - Only allowed to be set to "" (without quotes), i.e., "none", for PVGrub. + - Network-related flags: (Must be set to IPv4 addresses in the form of `x.x.x.x`.) + - `net.fake-ip` + - `net.fake-gateway` + - `net.fake-netmask` - Boolean flags: (Permitted values are "1" and "0", denoting "true" and "false" respectively.) - `no-monitor-layout` - `pci-e820-host` From b7dcfb64bbbcfd78f3522c3f69a592635b82f938 Mon Sep 17 00:00:00 2001 From: Andrew David Wong Date: Wed, 4 Aug 2021 21:41:08 -0700 Subject: [PATCH 20/20] Update developer/system/template-manager.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Marta Marczykowska-Górecka --- developer/system/template-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developer/system/template-manager.md b/developer/system/template-manager.md index 730512cc..465e067f 100644 --- a/developer/system/template-manager.md +++ b/developer/system/template-manager.md @@ -48,7 +48,7 @@ It strives to provide not only a better mechanism for handling template installa - Install/reinstall/downgrade/upgrade templates, either from local packages or remote repositories - Ability to install templates in alternative pools - - Possibility for the template package to specify options such as the kernel or `virt_mode` used + - Possibility for the template package to specify options such as kernel or `virt_mode` used by the resulting template VM - List and show information about local and available templates - Machine-readable output for easy extension - Search for templates