From 7e8ea7b0c3e8696cd56832bfde80caa1bb523da2 Mon Sep 17 00:00:00 2001 From: deeplow Date: Sun, 2 Jan 2022 03:21:38 -0500 Subject: [PATCH] add documentation for openqa_investigator script --- developer/debugging/automated-tests.md | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/developer/debugging/automated-tests.md b/developer/debugging/automated-tests.md index c57df136..031bffa9 100644 --- a/developer/debugging/automated-tests.md +++ b/developer/debugging/automated-tests.md @@ -242,13 +242,12 @@ class SomeTestCase(unittest.TestCase): gc.collect() ~~~ -## Installation Tests with openQA +## Automated tests with openQA **URL:** - **Tests:** -Manually testing the installation of Qubes OS is a time-consuming process. +Manually testing Qubes OS and its installation is a time-consuming process. We use [OpenQA](http://open.qa/) to automate this process. It works by installing Qubes in KVM and interacting with it as a user would, including simulating mouse clicks and keyboard presses. Then, it checks the output to see whether various tests were passed, e.g. by comparing the virtual screen output to screenshots of a successful installation. @@ -258,4 +257,23 @@ KVM also supports nested virtualization, so HVM should theoretically work. In practice, however, either Xen or QEMU crashes when this is attempted. Nonetheless, PV works well, which is sufficient for automated installation testing. -Thanks to an anonymous donor, our openQA system is hosted in a datacenter on hardware that meets these requirements. +Thanks to present and past donors who have provided the infrastructure for Qubes' openQA system with hardware that meets these requirements. + +### Looking for patterns in tests + +In order to better visualize patterns in tests the [`openqa_investigator`](https://github.com/QubesOS/openqa-tests-qubesos/blob/master/utils/openqa_investigator.py) script can be used. +It feeds off of the openQA test data to make graph plots. Here is an example: + +![openqa-investigator-splitgpg-example.png](/attachment/doc/openqa-investigator-splitgpg-example.png) + +Some outputs: + - plot by tests + - plot by errors + - markdown + +Some filters: + - filter by error + - filter by test name + +Chekout the script's help with `python3 openqa_investigator.py --help` +to see all available options.