Project

General

Profile

Wiki » History » Version 26

riafarov, 2020-01-14 15:54

1 18 okurz
# test results overview
2
* Latest report based on openQA test results http://s.qa.suse.de/test-status , SLE12: http://s.qa.suse.de/test-status-sle12 , SLE15: http://s.qa.suse.de/test-status-sle15
3
4
5 1 mgriessmeier
# QA SLE Functional - Team description
6
7 10 okurz
{{toc}}
8
9 15 okurz
**QSF (QA SLE Functional)** is a virtual team focusing on QA of the "functional" domain of the SUSE SLE products. The virtual team is mainly comprised of members of [SUSE QA SLE Nbg](https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Organization/Members_and_Responsibilities#QA_SLE_NBG_Team) including members from [SUSE QA SLE Prg](https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Organization/Members_and_Responsibilities#QA_SLE_PRG_Team). The [SLE Departement](https://wiki.suse.net/index.php/SUSE-Quality_Assurance/SLE_Department#QSF_.28QA_SLE_Functional.29) page describes our QA responsibilities. We focus on our automatic tests running in [openQA](https://openqa.suse.de) under the job groups "Functional" as well as "Autoyast" for the respective products, for example [SLE 15 / Functional](https://openqa.suse.de/group_overview/110) and [SLE 15 / Autoyast](https://openqa.suse.de/group_overview/129). We back our automatic tests with exploratory manual tests, especially for the product milestone builds. Additionally we care about corresponding openSUSE openQA tests (see as well https://openqa.opensuse.org).
10 1 mgriessmeier
11 8 okurz
* long-term roadmap: http://s.qa.suse.de/qa-long-term
12
* overview of current openQA SLE12SP5 tests with progress ticket references: https://openqa.suse.de/tests/overview?distri=sle&version=12-SP5&groupid=139&groupid=142
13 11 okurz
* fate tickets for SLE12SP5 feature testing: based on http://s.qa.suse.de/qa_sle_functional_feature_tests_sle12sp5 new report based on all tickets with milestone before SLE12SP5 GM, http://s.qa.suse.de/qa_sle_functional_feature_tests_sle15sp1 for SLE15SP1
14 1 mgriessmeier
* only "blocker" or "shipstopper" bugs on "interesting products" for SLE15 http://s.qa.suse.de/qa_sle_functional_bug_query_sle15_2, http://s.qa/qa_sle_bugs_sle12_2 for SLE12
15 7 szarate
* Better organization of planned work can be seen at the [SUSE QA](https://progress.opensuse.org/projects/suseqa) project (which is not public).
16 1 mgriessmeier
17
# Test plan
18
19 3 szarate
When looking for coverage of certain components or use cases keep the [openQA glossary](http://open.qa/docs/#concept) in mind. It is important to understand that "tests in openQA" could be a scenario, for example a "textmode installation run", a combined multi-machine scenario, for example "a remote ssh based installation using X-forwarding", or a test module, for example "vim", which checks if the vim editor is correctly installed, provides correct rendering and basic functionality. You are welcome to contact any member of the team to ask for more clarification about this.
20 1 mgriessmeier
21
In detail the following areas are tested as part of "SLE functional":
22
23
* different hardware setups (UEFI, acpi)
24
* support for localization
25 19 okurz
* openSUSE: virtualization - some "virtualization" tests are active on o3 with reduced set compared to SLE coverage (on behalf of QA SLE virtualization due to team capacity constraints, clarified in QA SLE coordination meeting 2018-03-28)
26
* openSUSE: migration - comparable to "virtualization", a reduced set compared to SLE coverage is active on o3 (on behalf of QA SLE migration due to team capacity constraints, clarified in QA SLE coordination meeting 2018-04)
27 1 mgriessmeier
28 19 okurz
29 1 mgriessmeier
## QSF-y
30
31 26 riafarov
Virtual team focuses on testing YaST components, including installer and snapper.
32
33
Detailed test plan for SLES can be found here: [SLES_Integration_Level_Testplan.md](https://gitlab.suse.de/qsf-y/qa-sle-functional-y/blob/master/SLES_Integration_Level_Testplan.md)
34 1 mgriessmeier
35 18 okurz
* Latest report based on openQA test results SLE12: http://s.qa.suse.de/test-status-sle12-yast , SLE15: http://s.qa.suse.de/test-status-sle15-yast
36
37
38 1 mgriessmeier
## QSF-u
39
40 2 mgriessmeier
"Testing is the future, and the future starts with you"
41
42 1 mgriessmeier
* basic operations (firefox, zypper, logout/reboot/shutdown)
43
* boot_to_snapshot
44
* functional application tests (kdump, gpg, ipv6, java, git, openssl, openvswitch, VNC)
45
* NIS (server, client)
46
* toolchain (development module)
47
* systemd
48
* "transactional-updates" as part of the corresponding SLE server role, not CaaSP
49 18 okurz
50
* Latest report based on openQA test results SLE12: http://s.qa.suse.de/test-status-sle12-functional , SLE15: http://s.qa.suse.de/test-status-sle15-functional
51 1 mgriessmeier
52
53 6 okurz
## Explicitly not covered by QSF
54
55
* quarterly updated media: Expected to be covered by Maintenance + QAM
56
57
58 1 mgriessmeier
### What we do
59
60
We collected opinions, personal experiences and preferences starting with the following four topics: What are fun-tasks ("new tests", "collaborate", "do it right"), what parts are annoying ("old & sporadic issues"), what do we think is expected from qsf-u ("be quick", "keep stuff running", "assess quality") and what we should definitely keep doing to prevent stakeholders becoming disappointed ("build validation", "communication & support").
61
62
#### How we work on our backlog
63
64 12 okurz
* no "due date"
65
* we pick up tickets that have not been previously discussed
66
* more flexible choice
67
* WIP-limits:
68
 * global limit of 10 tickets "In Progress"
69 1 mgriessmeier
70 14 okurz
* target numbers or "guideline", "should be", in priorities:
71
 1. New, untriaged: 0
72
 2. Workable: 40
73
 3. New, assigned to [u]: ideally less than 200 (should not stop you from triaging)
74 12 okurz
75
* SLAs for priority tickets - how to ensure to work on tickets which are more urgent?
76 1 mgriessmeier
 * "taken": <1d: immediate -> looking daily
77
 * 2-3d: urgent
78
 * first goal is "urgency removal": <1d: immediate, 1w: urgent
79
80 12 okurz
* our current "cycle time" is 1h - 1y (maximum, with interruptions)
81
82 1 mgriessmeier
* everybody should set priority + milestone in obvious cases, e.g. new reproducible test failures in multiple critical scenarios, in general case the PO decides
83
84
#### How we like to choose our battles
85
86
We self-assessed our tasks on a scale from "administrative" to "creative" and found in the following descending order: daily test review (very "administrative"), ticket triaging, milestone validation, code review, create needles, infrastructure issues, fix and cleanup tests, find bugs while fixing failing tests, find bugs while designing new tests, new automated tests (very "creative"). Then we found we appreciate if our work has a fair share of both sides. Probably a good ratio is 60% creative plus 40% administrative tasks. Both types have their advantages and we should try to keep the healthy balance.
87
88
89
#### What "product(s)" do we (really) *care* about?
90
91
Brainstorming results:
92
93
* openSUSE Krypton -> good example of something that we only remotely care about or not at all even though we see the connection point, e.g. test plasma changes early before they reach TW or Leap as operating systems we rely on or SLE+packagehub which SUSE does not receive direct revenue from but indirect benefit. Should be "community only", that includes members from QSF though
94
* openQA -> (like OBS), helps to provide ROI for SUSE
95
* SLE(S) (in development versions)
96
* Tumbleweed
97
* Leap, because we use it
98
* SLES HA
99
* SLE migration
100
* os-autoinst-distri-opensuse+backend+needles
101
102
From this list strictly no "product" gives us direct revenue however most likely SLE(S) (as well as SLES HA and SLE migration) are good examples of direct connection to revenue (based on SLE subscriptions). Conducting a poll in the team has revealed that 3 persons see "SLE(S)" as our main product and 3 see "os-autoinst-distri-opensuse+backend+needles" as the main product. We mainly agreed that however we can not *own* a product like "SLE" because that product is mainly not under our control.
103
104
Visualizing "cost of testing" vs. "risk of business impact" showed that both metrics have an inverse dependency, e.g. on a range from "upstream source code" over "package self-tests", "openSUSE Factory staging", "Tumbleweed", "SLE" we consider SLE to have the highest business risk attached and therefore defines our priority however testing at upstream source level is considered most effective to prevent higher cost of bugs or issues. Our conclusion is that we must ensure that the high-risk SLE base has its quality assured while supporting a quality assurance process as early as possible in the development process. package self-tests as well as the openQA staging tests are seen as a useful approach in that direction as well as "domain specfic specialist QA engineers" working closely together with according in-house development parties.
105
106
# Documentation
107
108
This documentation should only be interesting for the team QA SLE functional. If you find that some of the following topics are interesting for other people, please extract those topics to another wiki section.
109
110
### QA SLE functional Dashboards
111
112
In room 3.2.15 from Nuremberg office are two dedicated laptops each with a monitor attached showing a selected overview of openQA test resuls with important builds from SLE and openSUSE.
113 4 szarate
Such laptops are configured with a root account with the default password for production machines. First point of contact: [slindomansilla.suse.com](mailto:slindomansilla@suse.com), (okurz@suse.de)[mailto:okurz@suse.de]
114 1 mgriessmeier
115
* ''dashboard-osd-3215.suse.de'': Showing current view of openqa.suse.de filtered for some job group results, e.g. "Functional"
116
* ''dashboard-o3-3215.suse.de'': Showing current view of openqa.opensuse.org filtered for some job group results which we took responsibility to review and are mostly interested in
117
118 24 dheidler
### dashboard-osd-3215
119 1 mgriessmeier
120
* OS: openSUSE Tumbleweed
121
* Services: ssh, mosh, vnc, x2x
122
* Users:
123
** root
124
** dashboard
125
* VNC: `vncviewer dashboard-osd-3215`
126
* X2X: `ssh -XC dashboard@dashboard-osd-3215 x2x -west -to :0.0`
127
** (attaches the dashboard monitor as an extra display to the left of your screens. Then move the mouse over and the attached X11 server will capture mouse and keyboard)
128
129
#### Content of /home/dashboard/.xinitrc
130
131 3 szarate
```
132 1 mgriessmeier
#
133
# Source common code shared between the
134
# X session and X init scripts
135
#
136
. /etc/X11/xinit/xinitrc.common
137
138
xset -dpms
139
xset s off
140
xset s noblank
141
[...]
142
#
143
# Add your own lines here...
144
#
145
$HOME/bin/osd_dashboard &
146 3 szarate
```
147 1 mgriessmeier
148
#### Content of /home/dashboard/bin/osd_dashboard
149
150 3 szarate
```
151 1 mgriessmeier
#!/bin/bash
152
153
DISPLAY=:0 unclutter &
154
155
DISPLAY=:0 xset -dpms
156
DISPLAY=:0 xset s off
157
DISPLAY=:0 xset s noblank
158
159
url="${url:-"https://openqa.suse.de/?group=SLE+15+%2F+%28Functional%7CAutoyast%29&default_expanded=1&limit_builds=3&time_limit_days=14&show_tags=1&fullscreen=1#"}"
160 20 dheidler
DISPLAY=:0 chromium --kiosk "$url"
161 3 szarate
```
162 1 mgriessmeier
163
#### Cron job:
164
165 3 szarate
```
166 1 mgriessmeier
Min     H       DoM     Mo      DoW     Command
167 23 dheidler
*	*	*	*	*	/home/dashboard/bin/reload_chromium
168 3 szarate
```
169 1 mgriessmeier
170 21 dheidler
#### Content of /home/dashboard/bin/reload_chromium
171 1 mgriessmeier
172 3 szarate
```
173 1 mgriessmeier
#!/bin/bash
174
175
DISPLAY=:0 xset -dpms
176
DISPLAY=:0 xset s off
177
DISPLAY=:0 xset s noblank
178
179 22 dheidler
DISPLAY=:0 xdotool windowactivate $(DISPLAY=:0 xdotool search --class Chromium)
180 21 dheidler
DISPLAY=:0 xdotool key F5
181
DISPLAY=:0 xdotool windowactivate $(DISPLAY=:0 xdotool getactivewindow)
182 3 szarate
```
183 1 mgriessmeier
184
#### Issues:
185
186
* ''When the screen shows a different part of the web page''
187
** a simple mouse scroll through vnc or x2x may suffice.
188
* ''When the builds displayed are freeze without showing a new build, it usually means that midori, the browser displaying the info on the screen, crashed.''
189
** you can try to restart midori this way:
190
*** ps aux | grep midori
191
*** kill $pid
192
*** /home/dashboard/bin/osd_dashboard
193
** If this also doesn't work, restart the machine.
194 25 dheidler
195
196
### dashboard-o3
197
198
* Raspberry Pi 3B+
199
* IP: `10.160.65.207`
200
201
#### Content of /home/tux/.xinitrc
202
```
203
#!/bin/bash
204
205
unclutter &
206
openbox &
207
xset s off
208
xset -dpms
209
sleep 5
210
url="https://openqa.opensuse.org?group=openSUSE Tumbleweed\$|openSUSE Leap [0-9]{2}.?[0-9]*\$|openSUSE Leap.\*JeOS\$|openSUSE Krypton|openQA|GNOME Next&limit_builds=2&time_limit_days=14&&show_tags=1&fullscreen=1#build-results"
211
chromium --kiosk "$url" &
212
213
while sleep 300 ; do
214
        xdotool windowactivate $(xdotool search --class Chromium)
215
        xdotool key F5
216
        xdotool windowactivate $(xdotool getactivewindow)
217
done
218
```
219
220
#### Content of /usr/share/lightdm/lightdm.conf.d/50-suse-defaults.conf
221
```
222
[Seat:*]
223
pam-service = lightdm
224
pam-autologin-service = lightdm-autologin
225
pam-greeter-service = lightdm-greeter
226
xserver-command=/usr/bin/X
227
session-wrapper=/etc/X11/xdm/Xsession
228
greeter-setup-script=/etc/X11/xdm/Xsetup
229
session-setup-script=/etc/X11/xdm/Xstartup
230
session-cleanup-script=/etc/X11/xdm/Xreset
231
autologin-user=tux
232
autologin-timeout=0
233
```