oST_Projects » openQA improvement

Sprint Report

=============

{{toc}}

Justification of the project

----------------------------

The last 12.3 release was important for us for different of reasons. One of then is that we tried to include the quality assurance (QA) protocol a bit early in the release process. One of the reason for that decision was that we include for first time Secure Boot, and this technology was preceded by some reports of broken machines.

The focus of the QA was put initially in the boot process during 12.3 release, but a set of user space tools was tested also, like for example the network manager and the WiFi connectivity (for a full list of item tested during 12.3 see [1])

In parallel, we try to include the openQA tool in the release process, delegating a subset of those tests that can be done in an automatic way using a virtual machine. But soon we discovered a set of problems with the original version of openQA in relation with the requirement of our process:

- Screen resolution: openQA heavily relies on defined screen resolutions. If a milestone change the resolution, all the tests can be useless, creating untestable builds.

- With openQA is difficult to know in witch state of the test is in. When a major failure appeared, openQA lost the control over the test.

- Complicated maintenance: some image matching is MD5 based (deprecated in openQA, but with working code in some parts and tests), some other is screen-shot based. There is a test that use a median color to make an assert.

- Disk Usage: the screen-shot file format (PPM) is inefficient and wastes disk space. This is a limiting factor.

For a more comprehensive background of the motivations, please, refer to the article in the openSUSE blog [2]

Goals achieved in the project

-----------------------------

These are the main goals achieved during this project, and the tasks associated with these goals.

- Integration with openCV. We use a single graphic library for all the basic graphic operations.

    - action \#293: Replace the SWIG by XS

    - action \#294: Replace the basic API that use Perl calls to openCV call in C++

- Improve graphic matching

    - action \#295: Adapt the openCV fuzzy matching

    - action \#374: Improve openCV matching

- Remove of the PPM image format to save disk space

    - action \#258: Convert images from PPM to PNG and resize the images to a fixed size.

    - action \#306: Create a video from PNG files

- Remove deprecated MD5 matching

    - action \#263: Replace the image matching based on md5 sums by openCV.

-   Create the concept of Needle. A needle is an screen-shot with a metadata in JSON format. The needle is used to make asserts inside a test. Every needle define one or more areas that represent an image snippet that can be preprocessed with an OCR.

    -   action \#259: Adapt the test definitions to the new matching

    -   action \#283: add function to verify a needle is no longer present

    -   action \#297: Test Tesseract as a better OCR

    -   action \#298: Cretate goandclick API function

    -   action \#299: Refactor needels to remove the good label for tag

    -   action \#300: Add checkneedle method to the API

    -   action \#301: Write needle DB query API

    -   action \#302: Create needles for DVD installation

    -   action \#303: Create needles for NET installation

    -   action \#304: Create needles for KDE installation

    -   action \#305: Create needles for GNOME installation

    -   action \#310: Create a way to filter the needles that are used in the test

    -   action \#313: Evaluate a composite approach for the needles

-   The needle concept resolve two different problems indirectly:

    -   Avoid the definitions of global timeouts, because the assert can fail as sooner as possible

    -   Better control of the status of the test. The test can check that what is the current status and make decisions about that. For example, the test can check if one command ask for the root password, and if this is the case, insert the it as requested.

-   Improve the control for the QEMU instance, allowing multiple connections at the same time to the same QEMU virtual machine in parallel.

    -   action \#257: rewrite qemu monitor backend to use json

    -   action \#285: port qemu backend to QMP

-   Add snapshots to the virtual machine, so the user can decide from where to re-launch the test, and implement some rollback policies that assure a-known-state for the next test when a previous one fail.

    -   action \#383: Add QEMU snapshots

    -   action \#396: add “passed with defects” state

    -   action \#478: Merge test drivers in a single function

    -   action \#497: Implement SKIPTO environment variable

-   Change the architecture of the project, adding distributed workers and a queue for job dispatching.

    -   action \#377: get rid of checklog

    -   action \#474: Regular expressions to analyze ISO names

    -   action \#496: implement the json dispatcher interface

    -   action \#498: write worker around os-autoinst

    -   action \#499: Create a RPC function to register new ISO images and create testing jobs for them

    -   action \#515: write systemd generator for workers

-   Create a proper package to install openQA en openSUSE and SLE machines

    -   action \#232: Create openCV 2.4.4 package for SLE

-   Improve the user interface, creating a new needle editor in javascript and adding more development tools

    -   action \#361: Update WebUI to allow new openQA features

    -   action \#286: add a needle editor

    -   action \#379: Add debug information in the user interface

    -   action \#380: Exted fail JSON document to put needles candidates

    -   action \#384: Create clone-needle from a diff image

    -   action \#385: Improve the visualization of the unmatched needle

    -   action \#386: Enhance navigation in test results: an easy one to go to the next failing testmodule

    -   action \#388: Keep webUI up to date to handle new os-autoinst output

    -   action \#389: Use mean square error to detect the best candidate

    -   action \#466: Create a basic signaling architecture for os-autoinst

    -   action \#467: Update the webIU to control os-autoinst

But the main goal of the project is test every build of -Factory (especially milestones and beta) using a more reliable tool, where the actual bugs of the distribution are well an clearly spotted to the release manager and the openSUSE community and team.

Description of the project

--------------------------

The project has used five developers (non full time) during 10 weeks. The full description of the tasks and the time distribution can be found on the Gantt diagram [3] of the project.

The project has been split in 10 development milestones where new features was added, 1 integration milestone where the current version of openQA was integrated with upstream, 2 maintenance milestones, where end of sprint and future tasks where managed, and 2 management milestones where the project manager can put tasks related with the management of the project and the general marketing actions.

The source code of the project is now divided in three repositories:

* Development version of os-autoins:

  https://github.com/openSUSE-Team/os-autoinst

* Needles database:

  https://github.com/openSUSE-Team/os-autoinst-needles

* Development version of the user interface application:

  https://github.com/openSUSE-Team/openQA

The integration between the development version and the upstream version

is done here: https://github.com/bmwiedemann/os-autoinst/tree/v2

Conclusion

----------

-   As we said previously, all the technical goals can be resumed in one single goal: have a better tool where the release manager and the team can discover bugs in the distribution. The success or fail of

    the project need to be devised using a simple metric: How many bugs we can see with the help of openQA during the 13.1 release process? Today is a bit too soon to measure this, we need to wait to deploy the tool and use it during 13.1 release process.

-   In the technical point of view, the team managed to improve three key aspects:

    -   We have a better way to describe tests and run them in a controllable environment

    -   We improve the architecture of openQA, opening the possibility to run it in a distributed environment

    -   We have better tools to diagnose when a test fail, and fix it for the usual scenarios

-   The project is now in maintenance mode. We defined a set of tasks that we can work on in future sprints [4]

-   We will use the githup bug tracker to register the new bugs that we need to work on during the maintenance period [5]

-   SUSE and openSUSE Team invest in this sprint 500 hours of effort between design/development and dissemination. The total investment, including hardware, indirect cost, etc. are about 23.000 EUR (30.000 USD). But there is some extra cost related with the deployment and exploitation of the tool that are not considered yet.

References

----------

[1] http://board.opensuse.org/projects/openqa-improvement/wiki/TestedIn123

[2] http://lizards.opensuse.org/2013/06/06/openqa-in-opensuse/

[3] https://board.opensuse.org/projects/openqa-improvement/issues/gantt

[4] https://progress.opensuse.org/versions/50

[5] https://github.com/bmwiedemann/os-autoinst/issues