Update automated tests doc

- python3
- test runner help
- drop some of outdated info
This commit is contained in:
Marek Marczykowski-Górecki 2019-07-30 18:42:44 +02:00
parent cc0fdc714f
commit 9009c514b9
No known key found for this signature in database
GPG Key ID: F32894BE9684938A

View File

@ -21,18 +21,18 @@ Integration tests are written with the assumption that they will be called on de
**Do not run these tests on installations with important data, because you might lose it.** **Do not run these tests on installations with important data, because you might lose it.**
Since these tests were written with this expectation, all the VMs with a name starting with `test-` on the installation are removed during the process, and all the tests are recklessly started from dom0, even when testing VM components. Since these tests were written with this expectation, all the VMs with a name starting with `test-` on the installation are removed during the process, and all the tests are recklessly started from dom0, even when testing VM components.
Most of the tests are stored in the [core-admin repository](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) in the `tests` directory. Most of the tests are stored in the [core-admin repository](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) in the `qubes/tests` directory.
To start them you can use standard python unittest runner: To start them you can use standard python unittest runner:
python -m unittest -v qubes.tests python3 -m unittest -v qubes.tests
Or our custom one: Or our custom one:
python -m qubes.tests.run -v python3 -m qubes.tests.run -v
Our test runner runs mostly the same as the standard one, but it has some nice additional features like color output and not needing the "qubes.test" prefix. Our test runner runs mostly the same as the standard one, but it has some nice additional features like color output and not needing the "qubes.test" prefix.
It also has the ability to run lone selected template tests. It also has the ability to run lone selected template tests.
You can use `python -m qubes.tests.run -h` to get usage information: You can use `python3 -m qubes.tests.run -h` to get usage information:
[user@dom0 ~]$ python -m qubes.tests.run -h [user@dom0 ~]$ python3 -m qubes.tests.run -h
usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast] usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast]
[--do-not-clean] [--do-clean] [--loglevel LEVEL] [--do-not-clean] [--do-clean] [--loglevel LEVEL]
[--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg] [--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg]
@ -49,10 +49,6 @@ You can use `python -m qubes.tests.run -h` to get usage information:
--list, -l list all available tests and exit --list, -l list all available tests and exit
--failfast, -f stop on the first fail, error or unexpected success --failfast, -f stop on the first fail, error or unexpected success
--no-failfast disable --failfast --no-failfast disable --failfast
--do-not-clean, --dnc, -D
do not execute tearDown on failed tests. Implies
--failfast.
--do-clean, -C do execute tearDown even on failed tests.
--loglevel LEVEL, -L LEVEL --loglevel LEVEL, -L LEVEL
logging level for file and syslog forwarding (one of: logging level for file and syslog forwarding (one of:
NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL; NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL;
@ -65,6 +61,10 @@ You can use `python -m qubes.tests.run -h` to get usage information:
log most important things to kernel ring-buffer log most important things to kernel ring-buffer
--no-kmsg, --i-am-smarter-than-kay-sievers --no-kmsg, --i-am-smarter-than-kay-sievers
do not abuse kernel ring-buffer do not abuse kernel ring-buffer
--allow-running-along-qubesd
allow running in parallel with qubesd; this is
DANGEROUS and WILL RESULT IN INCONSISTENT SYSTEM STATE
--break-to-repl break to REPL after tests
When running only specific tests, write their names like in log, in format: When running only specific tests, write their names like in log, in format:
MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.". MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.".
@ -72,7 +72,7 @@ You can use `python -m qubes.tests.run -h` to get usage information:
For instance, to run only the tests for the fedora-21 template, you can use the `-l` option, then filter the list: For instance, to run only the tests for the fedora-21 template, you can use the `-l` option, then filter the list:
[user@dom0 ~]$ python -m qubes.tests.run -l | grep fedora-21 [user@dom0 ~]$ python3 -m qubes.tests.run -l | grep fedora-21
network/VmNetworking_fedora-21/test_000_simple_networking network/VmNetworking_fedora-21/test_000_simple_networking
network/VmNetworking_fedora-21/test_010_simple_proxyvm network/VmNetworking_fedora-21/test_010_simple_proxyvm
network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm
@ -93,7 +93,7 @@ For instance, to run only the tests for the fedora-21 template, you can use the
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app
vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file
[user@dom0 ~]$ python -m qubes.tests.run -v `python -m qubes.tests.run -l | grep fedora-21` [user@dom0 ~]$ python3 -m qubes.tests.run -v `python3 -m qubes.tests.run -l | grep fedora-21`
Example test run: Example test run:
@ -122,17 +122,7 @@ Test run can be altered using environment variables:
- `QUBES_TEST_LOAD_ALL` - load all tests (including tests for all templates) when relevant test modules are imported; this needs to be set for test runners not supporting [load_tests protocol](https://docs.python.org/3/library/unittest.html#load-tests-protocol) - `QUBES_TEST_LOAD_ALL` - load all tests (including tests for all templates) when relevant test modules are imported; this needs to be set for test runners not supporting [load_tests protocol](https://docs.python.org/3/library/unittest.html#load-tests-protocol)
### Adding a new test to core-admin ### Adding a new test to core-admin
After adding a new unit test to [core-admin/tests](https://github.com/QubesOS/qubes-core-admin/tree/master/tests) you'll have to make sure of two things: After adding a new unit test to [core-admin/qubes/tests](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests) you'll have to include it in [core-admin/qubes/tests/\_\_init\_\_.py](https://github.com/QubesOS/qubes-core-admin/tree/master/qubes/tests/__init__.py)
1. That the test will be added to the RPM file created by [QubesBuilder](/doc/qubes-builder/). For this you need to edit the [core-admin/tests/Makefile](https://github.com/QubesOS/qubes-core-admin/tree/master/tests/Makefile)
2. That the test will be loaded by [core-admin/tests/\_\_init\_\_.py](https://github.com/QubesOS/qubes-core-admin/tree/master/tests/__init__.py)
#### Editing the Makefile
To add your tests, you must append these two lines to the end of the makefile, which will copy your test and its compiled version to the right directory in the RPM file.
If your test is `example.py`, the appended lines would be:
cp example.py $(DESTDIR)$(PYTHON_TESTSPATH)
cp example.py[co] $(DESTDIR)$(PYTHON_TESTSPATH)
#### Editing `__init__.py` #### Editing `__init__.py`