oST_Projects » openQA improvement

Sprint Report

=============

{{toc}}

Justification of the project

----------------------------

The last 12.3 release was important for us for different of reasons. One

of then is that we tried to include the quality assurance (QA) protocol

a bit early in the release process. One of the reason for that decision

was that we include for first time Secure Boot, and this technology was

preceded by some reports of broken machines.

The focus of the QA was put initially in the boot process during 12.3

release, but a set of user space tools was tested also, like for example

the network manager and the WiFi connectivity (for a full list of item

tested during 12.3 see [1])

In parallel, we try to include the openQA tool in the release process,

delegating a subset of those tests that can be done in an automatic way

using a virtual machine. But soon we discovered a set of problems with

the original version of openQA in relation with the requirement of our

process:

-   Screen resolution: openQA heavily relies on defined screen

    resolutions. If a milestone change the resolution, all the tests can

    be useless, creating untestable builds.

-   With openQA is difficult to know in witch state of the test is in.

    When a major failure appeared, openQA lost the control over the

    test.

-   Complicated maintenance: some image matching is MD5 based

    (deprecated in openQA, but with working code in some parts and

    tests), some other is screen-shot based. There is a test that use a

    median color to make an assert.

-   Disk Usage: the screen-shot file format (PPM) is inefficient and

    wastes disk space. This is a limiting factor.

For a more comprehensive background of the motivations, please, refer to

the article in the openSUSE blog [2]

Goals achieved in the project

-----------------------------

These are the main goals achieved during this project, and the tasks

associated with these goals.

-   Integration with openCV. We use a single graphic library for all the

    basic graphic operations.

    -   action \#293: Replace the SWIG by XS

    -   action \#294: Replace the basic API that use Perl calls to

        openCV call in C++

-   Improve graphic matching

    -   action \#295: Adapt the openCV fuzzy matching

    -   action \#374: Improve openCV matching

-   Remove of the PPM image format to save disk space

    -   action \#258: Convert images from PPM to PNG and resize the

        images to a fixed size.

    -   action \#306: Create a video from PNG files

-   Remove deprecated MD5 matching

    -   action \#263: Replace the image matching based on md5 sums by

        openCV.

-   Create the concept of Needle. A needle is an screen-shot with a

    metadata in JSON format. The needle is used to make asserts inside a

    test. Every needle define one or more areas that represent an image

    snippet that can be preprocessed with an OCR.

    -   action \#259: Adapt the test definitions to the new matching

    -   action \#283: add function to verify a needle is no longer

        present

    -   action \#297: Test Tesseract as a better OCR

    -   action \#298: Cretate goandclick API function

    -   action \#299: Refactor needels to remove the good label for tag

    -   action \#300: Add checkneedle method to the API

    -   action \#301: Write needle DB query API

    -   action \#302: Create needles for DVD installation

    -   action \#303: Create needles for NET installation

    -   action \#304: Create needles for KDE installation

    -   action \#305: Create needles for GNOME installation

    -   action \#310: Create a way to filter the needles that are used

        in the test

    -   action \#313: Evaluate a composite approach for the needles

-   The needle concept resolve two different problems indirectly:

    -   Avoid the definitions of global timeouts, because the assert can

        fail as sooner as possible

    -   Better control of the status of the test. The test can check

        that what is the current status and make decisions about that.

        For example, the test can check if one command ask for the root

        password, and if this is the case, insert the it as requested.

-   Improve the control for the QEMU instance, allowing multiple

    connections at the same time to the same QEMU virtual machine in

    parallel.

    -   action \#257: rewrite qemu monitor backend to use json

    -   action \#285: port qemu backend to QMP

-   Add snapshots to the virtual machine, so the user can decide from

    where to re-launch the test, and implement some rollback policies

    that assure a-known-state for the next test when a previous one

    fail.

    -   action \#383: Add QEMU snapshots

    -   action \#396: add “passed with defects” state

    -   action \#478: Merge test drivers in a single function

    -   action \#497: Implement SKIPTO environment variable

-   Change the architecture of the project, adding distributed workers

    and a queue for job dispatching.

    -   action \#377: get rid of checklog

    -   action \#474: Regular expressions to analyze ISO names

    -   action \#496: implement the json dispatcher interface

    -   action \#498: write worker around os-autoinst

    -   action \#499: Create a RPC function to register new ISO images

        and create testing jobs for them

    -   action \#515: write systemd generator for workers

-   Create a proper package to install openQA en openSUSE and SLE

    machines

    -   action \#232: Create openCV 2.4.4 package for SLE

-   Improve the user interface, creating a new needle editor in

    javascript and adding more development tools

    -   action \#361: Update WebUI to allow new openQA features

    -   action \#286: add a needle editor

    -   action \#379: Add debug information in the user interface

    -   action \#380: Exted fail JSON document to put needles candidates

    -   action \#384: Create clone-needle from a diff image

    -   action \#385: Improve the visualization of the unmatched needle

    -   action \#386: Enhance navigation in test results: an easy one to

        go to the next failing testmodule

    -   action \#388: Keep webUI up to date to handle new os-autoinst

        output

    -   action \#389: Use mean square error to detect the best candidate

    -   action \#466: Create a basic signaling architecture for

        os-autoinst

    -   action \#467: Update the webIU to control os-autoinst

But the main goal of the project is test every build of -Factory

(especially milestones and beta) using a more reliable tool, where the

actual bugs of the distribution are well an clearly spotted to the

release manager and the openSUSE community and team.

Description of the project

--------------------------

The project has used five developers (non full time) during 10 weeks.

The full description of the tasks and the time distribution can be found

on the Gantt diagram [3] of the project.

The project has been split in 10 development milestones where new

features was added, 1 integration milestone where the current version of

openQA was integrated with upstream, 2 maintenance milestones, where end

of sprint and future tasks where managed, and 2 management milestones

where the project manager can put tasks related with the management of

the project and the general marketing actions.

The source code of the project is now divided in three repositories:\

\* Development version of os-autoins:

https://github.com/openSUSE-Team/os-autoinst\

\* Needles database:

https://github.com/openSUSE-Team/os-autoinst-needles\

\* Development version of the user interface application:

https://github.com/openSUSE-Team/openQA

The integration between the development version and the upstream version

is done here: https://github.com/bmwiedemann/os-autoinst/tree/v2

Conclusion

----------

-   As we said previously, all the technical goals can be resumed in one

    single goal: have a better tool where the release manager and the

    team can discover bugs in the distribution. The success or fail of

    the project need to be devised using a simple metric: How many bugs

    we can see with the help of openQA during the 13.1 release process?

    Today is a bit too soon to measure this, we need to wait to deploy

    the tool and use it during 13.1 release process.

-   In the technical point of view, the team managed to improve three

    key aspects:

    -   We have a better way to describe tests and run them in a

        controllable environment

    -   We improve the architecture of openQA, opening the possibility

        to run it in a distributed environment

    -   We have better tools to diagnose when a test fail, and fix it

        for the usual scenarios

-   The project is now in maintenance mode. We defined a set of tasks

    that we can work on in future sprints [4]

-   We will use the githup bug tracker to register the new bugs that we

    need to work on during the maintenance period [5]

-   SUSE and openSUSE Team invest in this sprint 500 hours of effort

    between design/development and dissemination. The total investment,

    including hardware, indirect cost, etc. are about 23.000 EUR (30.000

    USD). But there is some extra cost related with the deployment and

    exploitation of the tool that are not considered yet.

References

----------

[1]

http://board.opensuse.org/projects/openqa-improvement/wiki/TestedIn123\

[2] http://lizards.opensuse.org/2013/06/06/openqa-in-opensuse/\

[3] https://board.opensuse.org/projects/openqa-improvement/issues/gantt\

[4] https://board.opensuse.org/versions/show/50\

[5] https://github.com/bmwiedemann/os-autoinst/issues