diff --git a/AutomatedTests.md b/AutomatedTests.md new file mode 100644 index 00000000..4581d6e6 --- /dev/null +++ b/AutomatedTests.md @@ -0,0 +1,96 @@ +--- +layout: doc +title: Automated tests +permalink: /doc/AutomatedTests/ +--- + +Automatic tests +=============== + +Starting with Qubes R3 we use [python unittest](TODO) to perform automatic +tests of Qubes OS. Regardless of the name, we use it for both [unit +tests](https://en.wikipedia.org/wiki/Unit_tests) and [integration +tests](https://en.wikipedia.org/wiki/Integration_tests). The main purpose is of +course deliver much more stable releases. + +Integration tests are written with assumption to be called on dedicated +hardware. **Do not run those test on machine where you have important data, you +can loose it**. Especially all the VMs with name starting with `test-` are +removed. All the tests are started from dom0, even when testing some VM +component. Those tests will create new VM(s), run the test, then remove the VM(s). + +Most of the tests are stored in [core-admin +repository](https://github.com/QubesOS/qubes-core-admin/tree/master/tests) in +`tests` directory. To start them you can use standard python unittest runner: + python -m unittest -v qubes.tests +Or our custom one: + python -m qubes.tests.run -v + +Our test runner can be used mostly the same as the standard one, with some nice +additional features like no need to prefix all the tests with "qubes.tests", or +color output. It is also the only way to run only selected template tests. + +You can use `python -m qubes.tests.run -h` to get usage information: + + [user@dom0 ~]$ python -m qubes.tests.run -h + usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast] + [--do-not-clean] [--do-clean] [--loglevel LEVEL] + [--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg] + [TESTNAME [TESTNAME ...]] + + positional arguments: + TESTNAME list of tests to run named like in description + (default: run all tests) + + optional arguments: + -h, --help show this help message and exit + --verbose, -v increase console verbosity level + --quiet, -q decrease console verbosity level + --list, -l list all available tests and exit + --failfast, -f stop on the first fail, error or unexpected success + --no-failfast disable --failfast + --do-not-clean, --dnc, -D + do not execute tearDown on failed tests. Implies + --failfast. + --do-clean, -C do execute tearDown even on failed tests. + --loglevel LEVEL, -L LEVEL + logging level for file and syslog forwarding (one of: + NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL; + default: DEBUG) + --logfile FILE, -o FILE + if set, test run will be also logged to file + --syslog reenable logging to syslog + --no-syslog disable logging to syslog + --kmsg, --very-brave-or-very-stupid + log most important things to kernel ring-buffer + --no-kmsg, --i-am-smarter-than-kay-sievers + do not abuse kernel ring-buffer + + When running only specific tests, write their names like in log, in format: + MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.". + Example: basic/TC_00_Basic/test_000_create + +For example to run only tests for fedora-21 template, you can use `-l` option, then filter the list: + + [user@dom0 ~]$ python -m qubes.tests.run -l | grep fedora-21 + network/VmNetworking_fedora-21/test_000_simple_networking + network/VmNetworking_fedora-21/test_010_simple_proxyvm + network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm + network/VmNetworking_fedora-21/test_030_firewallvm_firewall + network/VmNetworking_fedora-21/test_040_inter_vm + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_000_start_shutdown + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_010_run_gui_app + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_050_qrexec_simple_eof + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_051_qrexec_simple_eof_reverse + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_052_qrexec_vm_service_eof + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_053_qrexec_vm_service_eof_reverse + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_060_qrexec_exit_code_dom0 + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_065_qrexec_exit_code_vm + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_100_qrexec_filecopy + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_110_qrexec_filecopy_deny + vm_qrexec_gui/TC_00_AppVM_fedora-21/test_120_qrexec_filecopy_self + vm_qrexec_gui/TC_20_DispVM_fedora-21/test_000_prepare_dvm + vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run + vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app + vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file + [user@dom0 ~]$ python -m qubes.tests.run -v `python -m qubes.tests.run -l | grep fedora-21` diff --git a/SystemDoc.md b/SystemDoc.md index 0af5c2fe..895ea38f 100644 --- a/SystemDoc.md +++ b/SystemDoc.md @@ -32,6 +32,7 @@ System Documentation for Developers 3. Debugging: 1. [Profiling python code](/wiki/Profiling) 2. [Test environment in separate machine for automatic tests](/wiki/TestBench) + 3. [Automated tests](/doc/AutomatedTests/) 3. [VM-dom0 internal configuration interface](/wiki/SystemDoc/VMInterface) 4. [Debugging Windows VMs](/wiki/WindowsDebugging)