mirror of
https://github.com/QubesOS/qubes-doc.git
synced 2025-05-04 15:55:16 -04:00
Markdown formatting fixes
- mark all code blocks with ``` - unify empty lines between sections - adjust list syntax (no space before dash) - adjust headers to use Atx-style syntax - remove trailing spaces
This commit is contained in:
Maya
Marek Marczykowski-Górecki
parent
2bde7d07e0
commit
67a92614aa
148 changed files with 4025 additions and 3639 deletions
|
|
@ -7,14 +7,12 @@ redirect_from:
|
|||
- /doc/AutomatedTests/
|
||||
---
|
||||
|
||||
Automated Tests
|
||||
===============
|
||||
# Automated Tests
|
||||
|
||||
Unit and Integration Tests
|
||||
--------------------------
|
||||
## Unit and Integration Tests
|
||||
|
||||
Starting with Qubes R3 we use [python unittest][unittest] to perform automatic tests of Qubes OS.
|
||||
Despite the name, we use it for both [unit tests](https://en.wikipedia.org/wiki/Unit_tests) and [integration tests](https://en.wikipedia.org/wiki/Integration_tests).
|
||||
Starting with Qubes R3 we use [python unittest][unittest] to perform automatic tests of Qubes OS.
|
||||
Despite the name, we use it for both [unit tests](https://en.wikipedia.org/wiki/Unit_tests) and [integration tests](https://en.wikipedia.org/wiki/Integration_tests).
|
||||
The main purpose is, of course, to deliver much more stable releases.
|
||||
|
||||
The integration tests must be run in dom0, but some unit tests can run inside a VM as well.
|
||||
|
|
@ -46,6 +44,7 @@ Our test runner runs mostly the same as the standard one, but it has some nice a
|
|||
|
||||
You can use `python3 -m qubes.tests.run -h` to get usage information:
|
||||
|
||||
```
|
||||
[user@dom0 ~]$ python3 -m qubes.tests.run -h
|
||||
usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast]
|
||||
[--do-not-clean] [--do-clean] [--loglevel LEVEL]
|
||||
|
|
@ -83,9 +82,11 @@ You can use `python3 -m qubes.tests.run -h` to get usage information:
|
|||
When running only specific tests, write their names like in log, in format:
|
||||
MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.".
|
||||
Example: basic/TC_00_Basic/test_000_create
|
||||
```
|
||||
|
||||
For instance, to run only the tests for the fedora-21 template, you can use the `-l` option, then filter the list:
|
||||
|
||||
```
|
||||
[user@dom0 ~]$ python3 -m qubes.tests.run -l | grep fedora-21
|
||||
network/VmNetworking_fedora-21/test_000_simple_networking
|
||||
network/VmNetworking_fedora-21/test_010_simple_proxyvm
|
||||
|
|
@ -108,6 +109,7 @@ For instance, to run only the tests for the fedora-21 template, you can use the
|
|||
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app
|
||||
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file
|
||||
[user@dom0 ~]$ sudo -E python3 -m qubes.tests.run -v `python3 -m qubes.tests.run -l | grep fedora-21`
|
||||
```
|
||||
|
||||
Example test run:
|
||||
|
||||
|
|
@ -115,11 +117,12 @@ Example test run:
|
|||
|
||||
Tests are also compatible with nose2 test runner, so you can use this instead:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop qubesd; sudo -E nose2 -v --plugin nose2.plugins.loader.loadtests qubes.tests; sudo systemctl start qubesd
|
||||
```
|
||||
|
||||
This may be especially useful together with various nose2 plugins to store tests results (for example `nose2.plugins.junitxml`), to ease presenting results. This is what we use on [OpenQA].
|
||||
|
||||
|
||||
### Unit testing inside a VM
|
||||
|
||||
Many unit tests will also work inside a VM. However all of the tests requiring a dedicated VM to be run (mostly the integration tests) will be skipped.
|
||||
|
|
@ -132,6 +135,7 @@ its dependency [qubes-core-qrexec](https://github.com/QubesOS/qubes-core-qrexec)
|
|||
The below example however will assume that you set up a build environment as described in the [Qubes Builder documentation](/doc/qubes-builder/).
|
||||
|
||||
Assuming you cloned the `qubes-builder` repository to your home directory inside a fedora VM, you can use the following commands to run the unit tests:
|
||||
|
||||
```{.bash}
|
||||
cd ~
|
||||
sudo dnf install python3-pip lvm2 python35 python3-virtualenv
|
||||
|
|
@ -158,16 +162,17 @@ the current stable branch.
|
|||
|
||||
Test runs can be altered using environment variables:
|
||||
|
||||
- `DEFAULT_LVM_POOL` - LVM thin pool to use for tests, in `VolumeGroup/ThinPool` format
|
||||
- `QUBES_TEST_PCIDEV` - PCI device to be used in PCI passthrough tests (for example sound card)
|
||||
- `QUBES_TEST_TEMPLATES` - space separated list of templates to run tests on; if not set, all installed templates are tested
|
||||
- `QUBES_TEST_LOAD_ALL` - load all tests (including tests for all templates) when relevant test modules are imported; this needs to be set for test runners not supporting [load_tests protocol](https://docs.python.org/3/library/unittest.html#load-tests-protocol)
|
||||
- `DEFAULT_LVM_POOL` - LVM thin pool to use for tests, in `VolumeGroup/ThinPool` format
|
||||
- `QUBES_TEST_PCIDEV` - PCI device to be used in PCI passthrough tests (for example sound card)
|
||||
- `QUBES_TEST_TEMPLATES` - space separated list of templates to run tests on; if not set, all installed templates are tested
|
||||
- `QUBES_TEST_LOAD_ALL` - load all tests (including tests for all templates) when relevant test modules are imported; this needs to be set for test runners not supporting [load_tests protocol](https://docs.python.org/3/library/unittest.html#load-tests-protocol)
|
||||
|
||||
### Adding a new test to core-admin
|
||||
|
||||
After adding a new unit test to [core-admin/qubes/tests](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) you'll have to include it in [core-admin/qubes/tests/\_\_init\_\_.py](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests/__init__.py)
|
||||
|
||||
|
||||
#### Editing `__init__.py`
|
||||
|
||||
You'll also need to add your test at the bottom of the `__init__.py` file, in the method `def load_tests`, in the for loop with `modname`.
|
||||
Again, given the hypothetical `example.py` test:
|
||||
|
||||
|
|
@ -213,7 +218,7 @@ class SomeTestCase(unittest.TestCase):
|
|||
# first test that actually use event loop will try to dereference (already
|
||||
# destroyed) objects, resulting in SEGV
|
||||
self.loop = quamash.QEventLoop(self.qtapp)
|
||||
|
||||
|
||||
def tearDown(self):
|
||||
[...]
|
||||
# process any pending events before destroying the object
|
||||
|
|
@ -237,11 +242,10 @@ class SomeTestCase(unittest.TestCase):
|
|||
gc.collect()
|
||||
~~~
|
||||
|
||||
## Installation Tests with openQA
|
||||
|
||||
Installation Tests with openQA
|
||||
------------------------------
|
||||
**URL:** <https://openqa.qubes-os.org/>
|
||||
|
||||
**URL:** <https://openqa.qubes-os.org/>
|
||||
**Tests:** <https://github.com/marmarek/openqa-tests-qubesos>
|
||||
|
||||
Manually testing the installation of Qubes OS is a time-consuming process.
|
||||
|
|
@ -258,4 +262,3 @@ Thanks to an anonymous donor, our openQA system is hosted in a datacenter on har
|
|||
|
||||
[unittest]: https://docs.python.org/3/library/unittest.html
|
||||
[OpenQA]: http://open.qa/
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue