Tests documentation

AutomatedTests.md

@@ -0,0 +1,96 @@
+---
+layout: doc
+title: Automated tests
+permalink: /doc/AutomatedTests/
+---
+
+Automatic tests
+===============
+
+Starting with Qubes R3 we use [python unittest](TODO) to perform automatic
+tests of Qubes OS. Regardless of the name, we use it for both [unit
+tests](https://en.wikipedia.org/wiki/Unit_tests) and [integration
+tests](https://en.wikipedia.org/wiki/Integration_tests). The main purpose is of
+course deliver much more stable releases.
+
+Integration tests are written with assumption to be called on dedicated
+hardware. **Do not run those test on machine where you have important data, you
+can loose it**. Especially all the VMs with name starting with `test-` are
+removed. All the tests are started from dom0, even when testing some VM
+component. Those tests will create new VM(s), run the test, then remove the VM(s).
+
+Most of the tests are stored in [core-admin
+repository](https://github.com/QubesOS/qubes-core-admin/tree/master/tests) in
+`tests` directory. To start them you can use standard python unittest runner:
+    python -m unittest -v qubes.tests
+Or our custom one:
+    python -m qubes.tests.run -v
+
+Our test runner can be used mostly the same as the standard one, with some nice
+additional features like no need to prefix all the tests with "qubes.tests", or
+color output. It is also the only way to run only selected template tests.
+
+You can use `python -m qubes.tests.run -h` to get usage information:
+
+    [user@dom0 ~]$ python -m qubes.tests.run -h
+    usage: run.py [-h] [--verbose] [--quiet] [--list] [--failfast] [--no-failfast]
+                  [--do-not-clean] [--do-clean] [--loglevel LEVEL]
+                  [--logfile FILE] [--syslog] [--no-syslog] [--kmsg] [--no-kmsg]
+                  [TESTNAME [TESTNAME ...]]
+
+    positional arguments:
+      TESTNAME              list of tests to run named like in description
+                            (default: run all tests)
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      --verbose, -v         increase console verbosity level
+      --quiet, -q           decrease console verbosity level
+      --list, -l            list all available tests and exit
+      --failfast, -f        stop on the first fail, error or unexpected success
+      --no-failfast         disable --failfast
+      --do-not-clean, --dnc, -D
+                            do not execute tearDown on failed tests. Implies
+                            --failfast.
+      --do-clean, -C        do execute tearDown even on failed tests.
+      --loglevel LEVEL, -L LEVEL
+                            logging level for file and syslog forwarding (one of:
+                            NOTSET, DEBUG, INFO, WARN, WARNING, ERROR, CRITICAL;
+                            default: DEBUG)
+      --logfile FILE, -o FILE
+                            if set, test run will be also logged to file
+      --syslog              reenable logging to syslog
+      --no-syslog           disable logging to syslog
+      --kmsg, --very-brave-or-very-stupid
+                            log most important things to kernel ring-buffer
+      --no-kmsg, --i-am-smarter-than-kay-sievers
+                            do not abuse kernel ring-buffer
+
+    When running only specific tests, write their names like in log, in format:
+    MODULE+"/"+CLASS+"/"+FUNCTION. MODULE should omit initial "qubes.tests.".
+    Example: basic/TC_00_Basic/test_000_create
+
+For example to run only tests for fedora-21 template, you can use `-l` option, then filter the list:
+
+    [user@dom0 ~]$ python -m qubes.tests.run -l | grep fedora-21
+    network/VmNetworking_fedora-21/test_000_simple_networking
+    network/VmNetworking_fedora-21/test_010_simple_proxyvm
+    network/VmNetworking_fedora-21/test_020_simple_proxyvm_nm
+    network/VmNetworking_fedora-21/test_030_firewallvm_firewall
+    network/VmNetworking_fedora-21/test_040_inter_vm
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_000_start_shutdown
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_010_run_gui_app
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_050_qrexec_simple_eof
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_051_qrexec_simple_eof_reverse
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_052_qrexec_vm_service_eof
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_053_qrexec_vm_service_eof_reverse
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_060_qrexec_exit_code_dom0
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_065_qrexec_exit_code_vm
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_100_qrexec_filecopy
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_110_qrexec_filecopy_deny
+    vm_qrexec_gui/TC_00_AppVM_fedora-21/test_120_qrexec_filecopy_self
+    vm_qrexec_gui/TC_20_DispVM_fedora-21/test_000_prepare_dvm
+    vm_qrexec_gui/TC_20_DispVM_fedora-21/test_010_simple_dvm_run
+    vm_qrexec_gui/TC_20_DispVM_fedora-21/test_020_gui_app
+    vm_qrexec_gui/TC_20_DispVM_fedora-21/test_030_edit_file
+    [user@dom0 ~]$ python -m qubes.tests.run -v `python -m qubes.tests.run -l | grep fedora-21`

SystemDoc.md

@@ -32,6 +32,7 @@ System Documentation for Developers
 3.  Debugging:
     1.  [Profiling python code](/wiki/Profiling)
     2.  [Test environment in separate machine for automatic tests](/wiki/TestBench)
+    3.  [Automated tests](/doc/AutomatedTests/)
     3.  [VM-dom0 internal configuration interface](/wiki/SystemDoc/VMInterface)
     4.  [Debugging Windows VMs](/wiki/WindowsDebugging)