Project

General

Profile

Sprint Report » History » Version 22

ancorgs, 2013-10-28 14:04

1 20
Sprint Report
2
=============
3
4
{{toc}}
5
6
Justification of the project
7
----------------------------
8
9 21 aplanas
The last 12.3 release was important for us for different of reasons. One of then is that we tried to include the quality assurance (QA) protocol a bit early in the release process. One of the reason for that decision was that we include for first time Secure Boot, and this technology was preceded by some reports of broken machines.
10 20
11 21 aplanas
The focus of the QA was put initially in the boot process during 12.3 release, but a set of user space tools was tested also, like for example the network manager and the WiFi connectivity (for a full list of item tested during 12.3 see [1])
12 20
13 21 aplanas
In parallel, we try to include the openQA tool in the release process, delegating a subset of those tests that can be done in an automatic way using a virtual machine. But soon we discovered a set of problems with the original version of openQA in relation with the requirement of our process:
14 20
15 21 aplanas
- Screen resolution: openQA heavily relies on defined screen resolutions. If a milestone change the resolution, all the tests can be useless, creating untestable builds.
16 20
17 21 aplanas
- With openQA is difficult to know in witch state of the test is in. When a major failure appeared, openQA lost the control over the test.
18 20
19 21 aplanas
- Complicated maintenance: some image matching is MD5 based (deprecated in openQA, but with working code in some parts and tests), some other is screen-shot based. There is a test that use a median color to make an assert.
20 20
21 21 aplanas
- Disk Usage: the screen-shot file format (PPM) is inefficient and wastes disk space. This is a limiting factor.
22 20
23 21 aplanas
For a more comprehensive background of the motivations, please, refer to the article in the openSUSE blog [2]
24 20
25
Goals achieved in the project
26
-----------------------------
27
28 21 aplanas
These are the main goals achieved during this project, and the tasks associated with these goals.
29 20
30 21 aplanas
- Integration with openCV. We use a single graphic library for all the basic graphic operations.
31
    - action \#293: Replace the SWIG by XS
32
    - action \#294: Replace the basic API that use Perl calls to openCV call in C++
33 20
34 21 aplanas
- Improve graphic matching
35
    - action \#295: Adapt the openCV fuzzy matching
36
    - action \#374: Improve openCV matching
37 20
38 21 aplanas
- Remove of the PPM image format to save disk space
39
    - action \#258: Convert images from PPM to PNG and resize the images to a fixed size.
40
    - action \#306: Create a video from PNG files
41 20
42 21 aplanas
- Remove deprecated MD5 matching
43
    - action \#263: Replace the image matching based on md5 sums by openCV.
44 20
45 21 aplanas
-   Create the concept of Needle. A needle is an screen-shot with a metadata in JSON format. The needle is used to make asserts inside a test. Every needle define one or more areas that represent an image snippet that can be preprocessed with an OCR.
46 20
    -   action \#259: Adapt the test definitions to the new matching
47 21 aplanas
    -   action \#283: add function to verify a needle is no longer present
48 20
    -   action \#297: Test Tesseract as a better OCR
49
    -   action \#298: Cretate goandclick API function
50
    -   action \#299: Refactor needels to remove the good label for tag
51
    -   action \#300: Add checkneedle method to the API
52
    -   action \#301: Write needle DB query API
53
    -   action \#302: Create needles for DVD installation
54
    -   action \#303: Create needles for NET installation
55
    -   action \#304: Create needles for KDE installation
56
    -   action \#305: Create needles for GNOME installation
57 21 aplanas
    -   action \#310: Create a way to filter the needles that are used in the test
58 20
    -   action \#313: Evaluate a composite approach for the needles
59
60
-   The needle concept resolve two different problems indirectly:
61 21 aplanas
    -   Avoid the definitions of global timeouts, because the assert can fail as sooner as possible
62
    -   Better control of the status of the test. The test can check that what is the current status and make decisions about that. For example, the test can check if one command ask for the root password, and if this is the case, insert the it as requested.
63 20
64 21 aplanas
-   Improve the control for the QEMU instance, allowing multiple connections at the same time to the same QEMU virtual machine in parallel.
65 20
    -   action \#257: rewrite qemu monitor backend to use json
66
    -   action \#285: port qemu backend to QMP
67
68 21 aplanas
-   Add snapshots to the virtual machine, so the user can decide from where to re-launch the test, and implement some rollback policies that assure a-known-state for the next test when a previous one fail.
69 20
    -   action \#383: Add QEMU snapshots
70
    -   action \#396: add “passed with defects” state
71
    -   action \#478: Merge test drivers in a single function
72
    -   action \#497: Implement SKIPTO environment variable
73
74 21 aplanas
-   Change the architecture of the project, adding distributed workers and a queue for job dispatching.
75 20
    -   action \#377: get rid of checklog
76
    -   action \#474: Regular expressions to analyze ISO names
77
    -   action \#496: implement the json dispatcher interface
78
    -   action \#498: write worker around os-autoinst
79 21 aplanas
    -   action \#499: Create a RPC function to register new ISO images and create testing jobs for them
80 20
    -   action \#515: write systemd generator for workers
81
82 21 aplanas
-   Create a proper package to install openQA en openSUSE and SLE machines
83 20
    -   action \#232: Create openCV 2.4.4 package for SLE
84
85 21 aplanas
-   Improve the user interface, creating a new needle editor in javascript and adding more development tools
86 20
    -   action \#361: Update WebUI to allow new openQA features
87
    -   action \#286: add a needle editor
88
    -   action \#379: Add debug information in the user interface
89
    -   action \#380: Exted fail JSON document to put needles candidates
90
    -   action \#384: Create clone-needle from a diff image
91
    -   action \#385: Improve the visualization of the unmatched needle
92 21 aplanas
    -   action \#386: Enhance navigation in test results: an easy one to go to the next failing testmodule
93
    -   action \#388: Keep webUI up to date to handle new os-autoinst output
94 20
    -   action \#389: Use mean square error to detect the best candidate
95 21 aplanas
    -   action \#466: Create a basic signaling architecture for os-autoinst
96 20
    -   action \#467: Update the webIU to control os-autoinst
97
98 21 aplanas
But the main goal of the project is test every build of -Factory (especially milestones and beta) using a more reliable tool, where the actual bugs of the distribution are well an clearly spotted to the release manager and the openSUSE community and team.
99 20
100
Description of the project
101
--------------------------
102
103 21 aplanas
The project has used five developers (non full time) during 10 weeks. The full description of the tasks and the time distribution can be found on the Gantt diagram [3] of the project.
104 20
105 21 aplanas
The project has been split in 10 development milestones where new features was added, 1 integration milestone where the current version of openQA was integrated with upstream, 2 maintenance milestones, where end of sprint and future tasks where managed, and 2 management milestones where the project manager can put tasks related with the management of the project and the general marketing actions.
106 20
107 21 aplanas
The source code of the project is now divided in three repositories:
108 20
109 21 aplanas
* Development version of os-autoins:
110
  https://github.com/openSUSE-Team/os-autoinst
111
* Needles database:
112
  https://github.com/openSUSE-Team/os-autoinst-needles
113
* Development version of the user interface application:
114
  https://github.com/openSUSE-Team/openQA
115
116 20
The integration between the development version and the upstream version
117
is done here: https://github.com/bmwiedemann/os-autoinst/tree/v2
118
119
Conclusion
120
----------
121
122 21 aplanas
-   As we said previously, all the technical goals can be resumed in one single goal: have a better tool where the release manager and the team can discover bugs in the distribution. The success or fail of
123
    the project need to be devised using a simple metric: How many bugs we can see with the help of openQA during the 13.1 release process? Today is a bit too soon to measure this, we need to wait to deploy the tool and use it during 13.1 release process.
124 20
125 21 aplanas
-   In the technical point of view, the team managed to improve three key aspects:
126
    -   We have a better way to describe tests and run them in a controllable environment
127
    -   We improve the architecture of openQA, opening the possibility to run it in a distributed environment
128
    -   We have better tools to diagnose when a test fail, and fix it for the usual scenarios
129 20
130 21 aplanas
-   The project is now in maintenance mode. We defined a set of tasks that we can work on in future sprints [4]
131 20
132 21 aplanas
-   We will use the githup bug tracker to register the new bugs that we need to work on during the maintenance period [5]
133 20
134 21 aplanas
-   SUSE and openSUSE Team invest in this sprint 500 hours of effort between design/development and dissemination. The total investment, including hardware, indirect cost, etc. are about 23.000 EUR (30.000 USD). But there is some extra cost related with the deployment and exploitation of the tool that are not considered yet.
135 20
136
References
137
----------
138
139 21 aplanas
[1] http://board.opensuse.org/projects/openqa-improvement/wiki/TestedIn123
140
[2] http://lizards.opensuse.org/2013/06/06/openqa-in-opensuse/
141
[3] https://board.opensuse.org/projects/openqa-improvement/issues/gantt
142 22 ancorgs
[4] https://progress.opensuse.org/versions/50
143 20
[5] https://github.com/bmwiedemann/os-autoinst/issues