Sprint Report¶

• Table of contents
• Sprint Report
  • Justification of the project
  • Goals achieved in the project
  • Description of the project
  • Conclusion
  • References

Justification of the project¶

The last 12.3 release was important for us for different of reasons. One
of then is that we tried to include the quality assurance (QA) protocol
a bit early in the release process. One of the reason for that decision
was that we include for first time Secure Boot, and this technology was
preceded by some reports of broken machines.

The focus of the QA was put initially in the boot process during 12.3
release, but a set of user space tools was tested also, like for example
the network manager and the WiFi connectivity (for a full list of item
tested during 12.3 see [1])

In parallel, we try to include the openQA tool in the release process,
delegating a subset of those tests that can be done in an automatic way
using a virtual machine. But soon we discovered a set of problems with
the original version of openQA in relation with the requirement of our
process:

• Screen resolution: openQA heavily relies on defined screen
  resolutions. If a milestone change the resolution, all the tests can
  be useless, creating untestable builds.

• With openQA is difficult to know in witch state of the test is in.
  When a major failure appeared, openQA lost the control over the
  test.

• Complicated maintenance: some image matching is MD5 based
  (deprecated in openQA, but with working code in some parts and
  tests), some other is screen-shot based. There is a test that use a
  median color to make an assert.

• Disk Usage: the screen-shot file format (PPM) is inefficient and
  wastes disk space. This is a limiting factor.

For a more comprehensive background of the motivations, please, refer to
the article in the openSUSE blog [2]

Goals achieved in the project¶

These are the main goals achieved during this project, and the tasks
associated with these goals.

• Integration with openCV. We use a single graphic library for all the
  basic graphic operations.

  • action #293: Replace the SWIG by XS
  • action #294: Replace the basic API that use Perl calls to openCV call in C++

• Improve graphic matching

  • action #295: Adapt the openCV fuzzy matching
  • action #374: Improve openCV matching

• Remove of the PPM image format to save disk space

  • action #258: Convert images from PPM to PNG and resize the images to a fixed size.
  • action #306: Create a video from PNG files

• Remove deprecated MD5 matching

  • action #263: Replace the image matching based on md5 sums by openCV.

• Create the concept of Needle. A needle is an screen-shot with a
  metadata in JSON format. The needle is used to make asserts inside a
  test. Every needle define one or more areas that represent an image
  snippet that can be preprocessed with an OCR.

  • action #259: Adapt the test definitions to the new matching
  • action #283: add function to verify a needle is no longer present
  • action #297: Test Tesseract as a better OCR
  • action #298: Cretate goandclick API function
  • action #299: Refactor needels to remove the good label for tag
  • action #300: Add checkneedle method to the API
  • action #301: Write needle DB query API
  • action #302: Create needles for DVD installation
  • action #303: Create needles for NET installation
  • action #304: Create needles for KDE installation
  • action #305: Create needles for GNOME installation
  • action #310: Create a way to filter the needles that are used in the test
  • action #313: Evaluate a composite approach for the needles

• The needle concept resolve two different problems indirectly:

  • Avoid the definitions of global timeouts, because the assert can fail as sooner as possible
  • Better control of the status of the test. The test can check that what is the current status and make decisions about that. For example, the test can check if one command ask for the root password, and if this is the case, insert the it as requested.

• Improve the control for the QEMU instance, allowing multiple
  connections at the same time to the same QEMU virtual machine in
  parallel.

  • action #257: rewrite qemu monitor backend to use json
  • action #285: port qemu backend to QMP

• Add snapshots to the virtual machine, so the user can decide from
  where to re-launch the test, and implement some rollback policies
  that assure a-known-state for the next test when a previous one
  fail.

  • action #383: Add QEMU snapshots
  • action #396: add “passed with defects” state
  • action #478: Merge test drivers in a single function
  • action #497: Implement SKIPTO environment variable

• Change the architecture of the project, adding distributed workers
  and a queue for job dispatching.

  • action #377: get rid of checklog
  • action #474: Regular expressions to analyze ISO names
  • action #496: implement the json dispatcher interface
  • action #498: write worker around os-autoinst
  • action #499: Create a RPC function to register new ISO images and create testing jobs for them
  • action #515: write systemd generator for workers

• Create a proper package to install openQA en openSUSE and SLE
  machines

  • action #232: Create openCV 2.4.4 package for SLE

• Improve the user interface, creating a new needle editor in
  javascript and adding more development tools

  • action #361: Update WebUI to allow new openQA features
  • action #286: add a needle editor
  • action #379: Add debug information in the user interface
  • action #380: Exted fail JSON document to put needles candidates
  • action #384: Create clone-needle from a diff image
  • action #385: Improve the visualization of the unmatched needle
  • action #386: Enhance navigation in test results: an easy one to go to the next failing testmodule
  • action #388: Keep webUI up to date to handle new os-autoinst output
  • action #389: Use mean square error to detect the best candidate
  • action #466: Create a basic signaling architecture for os-autoinst
  • action #467: Update the webIU to control os-autoinst

But the main goal of the project is test every build of -Factory
(especially milestones and beta) using a more reliable tool, where the
actual bugs of the distribution are well an clearly spotted to the
release manager and the openSUSE community and team.

Description of the project¶

The project has used five developers (non full time) during 10 weeks.
The full description of the tasks and the time distribution can be found
on the Gantt diagram [3] of the project.

The project has been split in 10 development milestones where new
features was added, 1 integration milestone where the current version of
openQA was integrated with upstream, 2 maintenance milestones, where end
of sprint and future tasks where managed, and 2 management milestones
where the project manager can put tasks related with the management of
the project and the general marketing actions.

The source code of the project is now divided in three repositories:\
* Development version of os-autoins:
https://github.com/openSUSE-Team/os-autoinst\
* Needles database:
https://github.com/openSUSE-Team/os-autoinst-needles\
* Development version of the user interface application:
https://github.com/openSUSE-Team/openQA

The integration between the development version and the upstream version
is done here: https://github.com/bmwiedemann/os-autoinst/tree/v2

Conclusion¶

• As we said previously, all the technical goals can be resumed in one
  single goal: have a better tool where the release manager and the
  team can discover bugs in the distribution. The success or fail of
  the project need to be devised using a simple metric: How many bugs
  we can see with the help of openQA during the 13.1 release process?
  Today is a bit too soon to measure this, we need to wait to deploy
  the tool and use it during 13.1 release process.

• In the technical point of view, the team managed to improve three
  key aspects:

  • We have a better way to describe tests and run them in a controllable environment
  • We improve the architecture of openQA, opening the possibility to run it in a distributed environment
  • We have better tools to diagnose when a test fail, and fix it for the usual scenarios

• The project is now in maintenance mode. We defined a set of tasks
  that we can work on in future sprints [4]

• We will use the githup bug tracker to register the new bugs that we
  need to work on during the maintenance period [5]

• SUSE and openSUSE Team invest in this sprint 500 hours of effort
  between design/development and dissemination. The total investment,
  including hardware, indirect cost, etc. are about 23.000 EUR (30.000
  USD). But there is some extra cost related with the deployment and
  exploitation of the tool that are not considered yet.

References¶

[1]
http://board.opensuse.org/projects/openqa-improvement/wiki/TestedIn123\
[2] http://lizards.opensuse.org/2013/06/06/openqa-in-opensuse/\
[3] https://board.opensuse.org/projects/openqa-improvement/issues/gantt\
[4] https://board.opensuse.org/versions/show/50\
[5] https://github.com/bmwiedemann/os-autoinst/issues