Project

General

Profile

Wiki » History » Version 81

okurz, 2019-06-21 12:45
glossary is upstream in http://open.qa/docs/#concepts

1 3 okurz
# Introduction
2 1 alarrosa
3 3 okurz
This is the organisation wiki for the **openQA Project**.
4 49 okurz
The source code is hosted in the [os-autoinst github project](http://github.com/os-autoinst/), especially [openQA itself](http://github.com/os-autoinst/openQA) and the main backend [os-autoinst](http://github.com/os-autoinst/os-autoinst)
5 1 alarrosa
6 48 okurz
If you are interested in the tests for SUSE/openSUSE products take a look into the [openqatests](https://progress.opensuse.org/projects/openqatests) project.
7
8 70 szarate
If you are looking for entry level issues to contribute to the backend, take a look at [this search query](https://progress.opensuse.org/projects/openqav3/search?utf8=%E2%9C%93&issues=1&q=entrance+level+issue)
9
10 14 okurz
{{toc}}
11
12 3 okurz
# Organisational
13 1 alarrosa
14 51 okurz
## ticket workflow
15
16 65 SLindoMansilla
Picture: http://imagebin.suse.de/2127/img
17 64 SLindoMansilla
18 51 okurz
The following ticket statuses are used together and their meaning is explained:
19
20 63 okurz
* *New*: No one has worked on the ticket (e.g. the ticket has not been properly refined) or no one is feeling responsible for the work on this ticket.
21 73 riafarov
* *Workable*: The ticket has been refined and is ready to be picked.
22
* *In Progress*: Assignee is actively working on the ticket.
23 1 alarrosa
* *Resolved*: The complete work on this issue is done and the according issue is supposed to be fixed as observed (Should be updated together with a link to a merged pull request or also a link to an production openQA showing the effect)
24 73 riafarov
* *Feedback*: Further work on the ticket is blocked by open points or is awaiting for the feedback to proceed. Sometimes also used to ask Assignee about progress on inactivity.
25 74 okurz
* *Blocked*: Further work on the ticket is blocked by some external dependency (e.g. bugs, not implemented features). There should be a link to another ticket, bug, trello card, etc. where it can be seen what the ticket is blocked by.
26 51 okurz
* *Rejected*: The issue is considered invalid, should not be done, is considered out of scope.
27
* *Closed*: As this can be set only by administrators it is suggested to not use this status.
28
29
It is good practice to update the status together with a comment about it, e.g. a link to a pull request or a reason for reject.
30
31 80 okurz
## ticket categories
32
33
* *Concrete Bugs*: Regressions, crashes, error messages
34
* *Feature requests*: Ideas or wishes for extension, enhancement, improvement
35
* *Organisational*: Organisational tasks within the project(s), not directly code related
36
* *Support*: Support of users, usage problems, questions
37
38
Please avoid the use of other, deprecated categories
39
40 13 okurz
## ticket templates
41
You can use these templates to fill in tickets and further improve them with more detail over time. Copy the code block, paste it into a new issue, replace every block marked with "<…>" with your content or delete if not appropriate.
42
43 71 nicksinger
### Defects
44 13 okurz
45
Subject: `<Short description, example: "openQA dies when triggering any Windows ME tests">`
46
47 1 alarrosa
48 13 okurz
```
49 71 nicksinger
## Observation
50 13 okurz
<description of what can be observed and what the symptoms are, provide links to failing test results and/or put short blocks from the log output here to visualize what is happening>
51
52 71 nicksinger
## Steps to reproduce
53 1 alarrosa
* <do this>
54 13 okurz
* <do that>
55 1 alarrosa
* <observe result>
56 13 okurz
57 71 nicksinger
## Problem
58 13 okurz
<problem investigation, can also include different hypotheses, should be labeled as "H1" for first hypothesis, etc.>
59
60 71 nicksinger
## Suggestion
61 13 okurz
<what to do as a first step>
62
63 71 nicksinger
## Workaround
64 13 okurz
<example: retrigger job>
65
```
66
67
example ticket: #10526
68
69 72 nicksinger
### Feature requests
70 13 okurz
71
Subject: `<Short description, example: "grub3 btrfs support" (feature)>`
72
73
74
```
75
## User story
76
<As a <role>, I want to <do an action>, to <achieve which goal> >
77
78 72 nicksinger
## Acceptance criteria
79 13 okurz
* <**AC1:** the first acceptance criterion that needs to be fulfilled to do this, example: Clicking "restart button" causes restart of the job>
80
* <**AC2:** also think about the "not-actions", example: other jobs are not affected>
81
82 72 nicksinger
## Tasks
83 13 okurz
* <first task to do as an easy starting point>
84 69 okurz
* <what do do next, all tasks optionally with an effort estimation in hours, e.g. "(0.5-2h)">
85 13 okurz
* <optional: mark "optional" tasks>
86
87 72 nicksinger
## Further details
88 17 okurz
<everything that does not fit into above sections>
89 13 okurz
```
90
91
example ticket: #10212
92
93 62 SLindoMansilla
## Further decision steps working on test issues
94 61 SLindoMansilla
95 62 SLindoMansilla
Test issues could be one of the following sources. Feel free to use the following template in tickets as well
96 1 alarrosa
97 62 SLindoMansilla
```
98
## Problem
99
* **H1** The product has changed
100
 * **H1.1** product changed slightly but in an acceptable way without the need for communication with DEV+RM --> adapt test
101
 * **H1.2** product changed slightly but in an acceptable way found after feedback from RM --> adapt test
102
 * **H1.3** product changed significantly --> after approval by RM adapt test
103 61 SLindoMansilla
104 62 SLindoMansilla
* **H2** Fails because of changes in test setup
105
 * **H2.1** Our test hardware equipment behaves different
106
 * **H2.2** The network behaves different
107
108
* **H3** Fails because of changes in test infrastructure software, e.g. os-autoinst, openQA
109
* **H4** Fails because of changes in test management configuration, e.g. openQA database settings
110
* **H5** Fails because of changes in the test software itself (the test plan in source code as well as needles)
111
* **H6** Sporadic issue, i.e. the root problem is already hidden in the system for a long time but does not show symptoms every time
112
```
113 25 okurz
114
## pull request handling on github
115
116
As a reviewer of pull requests on github for all related repositories, e.g. https://github.com/os-autoinst/os-autoinst-distri-opensuse/pulls, apply labels in case PRs are open for a longer time and can not be merged so that we keep our backlog clean and know why PRs are blocked.
117
118
* **notready**: Triaged as not ready yet for merging, no (immediate) reaction by the reviewee, e.g. when tests are missing, other scenarios break, only tested for one of SLE/TW
119
* **wip**: Marked by the reviewee itself as "[WIP]" or "[DO-NOT-MERGE]" or similar
120
* **question**: Questions to the reviewee, not answered yet
121 54 okurz
122
123
## Where to contribute?
124
125
If you want to help openQA development you can take a look into the existing [issues](https://progress.opensuse.org/projects/openqav3/issues). There are also some "always valid" tasks to be working on:
126
127
* *improve test coverage*:
128
 * *user story*: As openqa backend as well as test developer I want better test coverage of our projects to reduce technical debt
129
 * *acceptance criteria*: test coverage is significantly higher than before
130
 * *suggestions*: check current coverage in each individual project (os-autoinst/openQA/os-autoinst-distri-opensuse) and add tests as necessary
131
132 28 okurz
133 1 alarrosa
# Use cases
134 40 okurz
135 28 okurz
The following use cases 1-6 have been defined within a SUSE workshop (others have been defined later) to clarify how different actors work with openQA. Some of them are covered already within openQA quite well, some others are stated as motivation for further feature development.
136
137 6 okurz
## Use case 1
138 4 okurz
**User:** QA-Project Managment
139 1 alarrosa
**primary actor:** QA Project Manager, QA Team Leads
140
**stakeholder:** Directors, VP
141 7 okurz
**trigger:** product milestones, providing a daily status
142 1 alarrosa
**user story:** „As a QA project manager I want to check on a daily basis the „openQA Dashboard“ to get a summary/an overall status of the „reviewers results“ in order to take the right actions and prioritize tasks in QA accordingly.“
143 28 okurz
	
144 4 okurz
## Use case 2
145 1 alarrosa
**User:** openQA-Admin
146
**primary actor:** Backend-Team
147 4 okurz
**stakeholder:** Qa-Prjmgr, QA-TL, openQA Tech-Lead
148 7 okurz
**trigger:** Bugs, features, new testcases
149 5 okurz
**user story:** „As an openQA admin I constantly check in the web-UI the system health and I manage its configuration to ensure smooth operation of the tool.“
150 28 okurz
151 1 alarrosa
## Use case 3
152
**User:** QA-Reviewer
153
**primary actor:** QA-Team
154 4 okurz
**stakeholder:** QA-Prjmgr, Release-Mgmt, openQA-Admin
155 7 okurz
**trigger:** every new build
156
**user story:** „As an openQA-Reviewer at any point in time I review on the webpage of openQA the overall status of a build in order to track and find bugs, because I want to find bugs as early as possible and report them.“
157 28 okurz
158 1 alarrosa
## Use case 4
159
**User:** Testcase-Contributor
160 4 okurz
**primary actor:** All development teams, Maintenance QA
161 5 okurz
**stakeholder:** QA-Reviewer, openQA-Admin, openQA Tech-Lead
162 40 okurz
**trigger:** features, new functionality, bugs, new product/package
163 7 okurz
**user story:** „As developer when there are new features, new functionality, bugs, new product/package in git I contribute my testcases because I want to ensure good quality submissions and smooth product integration.“
164 28 okurz
165 4 okurz
## Use case 5
166
**User:** Release-Mgmt
167
**primary actor:** Release Manager
168 1 alarrosa
**stakeholder:** Directors, VP, PM, TAMs, Partners
169 7 okurz
**trigger:** Milestones
170
**user story:** „As a Release-Manager on a daily basis I check on a dashboard for the product health/build status in order to act early in case of failures and have concrete and current reports.“
171 28 okurz
172 4 okurz
## Use case 6
173
**User:** Staging-Admin
174
**primary actor:** Staging-Manager for the products
175 1 alarrosa
**stakeholder:** Release-Mgmt, Build-Team
176
**trigger:** every single submission to projects
177 40 okurz
**user story:** „As a Staging-Manager I review the build status of packages with every staged submission to the „staging projects“ in the „staging dashboard“ and the test-status of the pre-integrated fixes, because I want to identify major breakage before integration to the products and provide fast feedback back to the development.“
178
179
## Use case 7
180
**User:** Bug investigator
181
**primary actor:** Any bug assignee for openQA observed bugs
182
**stakeholder:** Developer
183
**trigger:** bugs
184 8 okurz
**user story:** „As a developer that has been assigned a bug which has been observed in openQA I can review referenced tests, find a newer and the most recent job in the same scenario, understand what changed since the last successful job, what other jobs show same symptoms to investigate the root cause fast and use openQA for verification of a bug fix.“
185 15 okurz
186 8 okurz
# Thoughts about categorizing test results, issues, states within openQA
187
by okurz
188
189
When reviewing test results it is important to distinguish between different causes of "failed tests"
190
191
## Nomenclature
192
193 58 okurz
### Test status categories
194 1 alarrosa
A common definition about the status of a test regarding the product it tests: "false|true positive|negative" as described on https://en.wikipedia.org/wiki/False_positives_and_false_negatives. "positive|negative" describes the outcome of a test ("positive": test signals presence of issue; "negative": no signal) whereas "false|true" describes the conclusion of the test regarding the presence of issues in the SUT or product in our case ("true": correct reporting; "false": incorrect reporting), e.g. "true negative", test successful, no issues detected and there are no issues, product is working as expected by customer. Another example: Think of testing as of a fire alarm. An alarm (event detector) should only go off (be "positive") *if* there is a fire (event to detect) --> "true positive" whereas *if* there is *no* fire there should be *no* alarm --> "true negative".
195 10 okurz
196 1 alarrosa
Another common but potentially ambiguous categorization:
197 10 okurz
198
* *broken*: the test is not behaving as expected (Ambiguity: "as expected" by whom?) --> commonly a "false positive", can also be "false negative" but hard to detect
199
* *failing*: the test is behaving as expected, but the test output is a fail --> "true positive"
200
* *working*: the test is behaving as expected (with no comment regarding the result, though some might ambiguously imply 'result is negative')
201
* *passing*: the test is behaving as expected, but the result is a success --> "true negative"
202 8 okurz
203 9 okurz
If in doubt declare a test as "broken". We should review the test and examine if it is behaving as expected.
204 10 okurz
205 8 okurz
Be careful about "positive/negative" as some might also use "positive" to incorrectly denote a passing test (and "negative" for failing test) as an indicator of "working product" not an indicator about "issue present". If you argue what is "used in common speech" think about how "false positive" is used as in "false alarm" --> "positive" == "alarm raised", also see https://narainko.wordpress.com/2012/08/26/understanding-false-positive-and-false-negative/
206
207 10 okurz
### Priorization of work regarding categories
208 3 okurz
In this sense development+QA want to accomplish a "true negative" state whenever possible (no issues present, therefore none detected). As QA and test developers we want to prevent "false positives" ("false alarms" declaring a product as broken when it is not but the test failed for other reasons), also known as "type I error" and "false negatives" (a product issue is not catched by tests and might "slip through" QA and at worst is only found by an external outside customer) also known as "type II error". Also see https://en.wikipedia.org/wiki/Type_I_and_type_II_errors. In the context of openQA and system testing paired with screen matching a "false positive" is much more likely as the tests are very susceptible to subtle variations and changes even if they should be accepted. So when in doubt, create an issue in progress, look at it again, and find that it was a false alarm, rather than wasting more peoples time with INVALID bug reports by believing the product to be broken when it isn't. To quote Richard Brown: "I […] believe this is the route to ongoing improvement - if we have tests which produce such false alarms, then that is a clear indicator that the test needs to be reworked to be less ambiguous, and that IS our job as openQA developers to deal with".
209 11 okurz
210
## Further categorization of statuses, issues and such in testing, especially automatic tests
211
By okurz
212
213
This categorization scheme is meant to help in communication in either written or spoken discussions being simple, concise, easy to remember while unambiguous in every case.
214
While used for naming it should also be used as a decision tree and can be followed from the top following each branch.
215
216
### Categorization scheme
217
218
To keep it simple I will try to go in steps of deciding if a potential issue is of one of two categories in every step (maybe three) and go further down from there. The degree of further detailing is not limited, i.e. it can be further extended. Naming scheme should follow arabic number (for two levels just 1 and 2) counting schemes added from the right for every additional level of decision step and detail without any separation between the digits, e.g. "1111" for the first type in every level of detail up to level four. Also, I am thinking of giving the fully written form phonetic name to unambiguously identify each on every level as long as not more individual levels are necessary. The alphabet should be reserved for higher levels and higher priority types.
219
Every leaf of the tree must have an action assigned to it.
220 12 okurz
221 11 okurz
1 **failed** (ZULU)
222
11 new (passed->failed) (YANKEE)
223
111 product issue ("true positive") (WHISKEY)
224 44 okurz
1111 unfiled issue (SIERRA)
225 11 okurz
11111 hard issue (openqa *fail*) (KILO)
226
111121 critical / potential ship stopper (INDIA) --> immediately file bug report with "ship_stopper?" flag; opt. inform RM directly
227 44 okurz
111122 non-critical hard issue (HOTEL) --> file bug report
228 11 okurz
11112 soft issue (openqa *softfail* on job level, not on module level) (JULIETT) --> file bug report on failing test module
229
1112 bugzilla bug exists (ROMEO)
230
11121 bug was known to openqa / openqa developer --> cross-reference (bug->test, test->bug) AND raise review process issue, improve openqa process
231
11122 bug was filed by other sources (e.g. beta-tester) --> cross-reference (bug->test, test->bug)
232
112 test issue ("false positive") (VICTOR)
233
1121 progress issue exists (QUEBEC) --> cross-reference (issue->test, test->issue)
234
1122 unfiled test issue (PAPA)
235
11221 easy to do w/o progress issue
236
112211 need needles update --> re-needle if sure, TODO how to notify?
237
112212 pot. flaky, timeout
238
1122121 retrigger yields PASS --> comment in progress about flaky issue fixed
239
1122122 reproducible on retrigger --> file progress issue
240
11222 needs progress issue filed --> file progress issue
241
12 existing / still failing (failed->failed) (XRAY)
242
121 product issue (UNIFORM)
243
1211 unfiled issue (OSCAR) --> file bug report AND raise review process issue (why has it not been found and filed?)
244
1212 bugzilla bug exists (NOVEMBER) --> ensure cross-reference, also see rules for 1112 ROMEO
245
122 test issue (TANGO)
246
1221 progress issue exists (MIKE) --> monitor, if persisting reprioritize test development work
247
1222 needs progress issue filed (LIMA) --> file progress issue AND raise review process issue, see 1211 OSCAR
248
2 **passed** (ALFA)
249
21 stable (passed->passed) (BRAVO)
250
211 existing "true negative" (DELTA) --> monitor, maybe can be made stricter
251
212 existing "false negative" (ECHO) --> needs test improvement
252
22 fixed (failed->passed) (CHARLIE)
253
222 fixed "true negative" (FOXTROTT) --> TODO split monitor, see 211 DELTA
254
2221 was test issue --> close progress issue
255
2222 was product issue
256
22221 no bug report exists --> raise review process issue (why was it not filed?)
257
22222 bug report exists
258
222221 was marked as RESOLVED FIXED
259
221 fixed but "false negative" (GOLF) --> potentially revert test fix, also see 212 ECHO
260 41 okurz
261
262 11 okurz
Priority from high to low: INDIA->OSCAR->HOTEL->JULIETT->…
263 35 okurz
264 16 okurz
# Advanced features in openQA
265
266
There are some features in openQA for reviewing test results and common practices. Some of these features are presented here based on the pull requests from github.
267
268
## Show previous results in test results page [gh#538](https://github.com/os-autoinst/openQA/pull/538)
269
270
On a tests result page there is a tab for "previous results" showing the result of test runs in the same scenario. This shows previous builds as well as test runs in the same build. This way you can easily check and compare results from before including any comments, labels, bug references (see next section). This helps to answer questions like "Is this a new issue", "Is it reproducable", "has it been seen in before", "how does the history look like".
271
272
Querying the database for former test runs of the same scenario is a rather
273
costly operation which we do not want to do for multiple test results at once
274
but only for each individual test result (1:1 relation). This is why this is done in each individual test result and not for a complete build.
275
276
The evaluation of previous jobs is limited but can be adjusted with the query parameter `limit_previous=<nr>` in the test URL, e.g. to provide a link to the tab in the results page showing the previous 30 results of test 1234 on openqa.opensuse.org you would write
277
`http://openqa.opensuse.org/tests/1234?limit_previous=30#previous`
278
279
Remember that the higher the limit, the more complex the database queries will be increasing the lookup time as well as the load on openQA to generate the result.
280
281
Related issue: #10212
282
283
Screenshot of feature:
284
![screenshot_20160210_142024](https://cloud.githubusercontent.com/assets/1693432/12948308/7e915a3c-d001-11e5-840b-2f070c3cb8a5.png)
285 36 okurz
286
## Link to latest in scenario name [gh#836](https://github.com/os-autoinst/openQA/pull/836)
287
288
Find the always latest job in a scenario with the link after the scenario name in the tab "Previous results"
289
Screenshot:
290
![openqa_link_to_latest_in_previous](https://cloud.githubusercontent.com/assets/1693432/18145393/5b5fb544-6fcb-11e6-967b-f24ffc6a498c.png)
291
292
293 34 okurz
294
## Add 'latest' query route [gh#815](https://github.com/os-autoinst/openQA/pull/815)
295
296
Should always refer to most recent job for the specified scenario.
297
298
* have the same link for test development, i.e. if one retriggers tests, the
299
person has to always update the URL. If there would be a static URL even the
300
browser can be instructed to reload the page automatically
301
302
* for linking to the always current execution of the last job within one
303
scenario, e.g. to respond faster to the standard question in bug reports "does
304
this bug still happen?"
305
306
Examples:
307
308
* `tests/latest?distri=opensuse&version=13.1&flavor=DVD&arch=x86_64&test=kde&machine=64bit`
309
* `tests/latest?flavor=DVD&arch=x86_64&test=kde`
310
* `tests/latest?test=foobar` - this searches for the most recent job using test_suite 'foobar' covering all distri, version, flavor, arch, machines. To be more specific, add the other query entries.
311 33 okurz
312 16 okurz
## Show certificate next to builds on overview if all failures are labeled [gh#560](https://github.com/os-autoinst/openQA/pull/560), [gh#1052](https://github.com/os-autoinst/openQA/pull/1052)
313 55 mkittler
314 16 okurz
See [online documentation about review badges](http://open.qa/docs/#_review_badges).
315
316
## Allow group overview query by result [gh#531](https://github.com/os-autoinst/openQA/pull/531)
317
318
This allows e.g. to show only failed builds. Could be included like in http://lists.opensuse.org/opensuse-factory/2016-02/msg00018.html for "known defects".
319
320 1 alarrosa
Example: Add query parameters like `…&result=failed&arch=x86_64` to show only failed for the single architecture selected.
321 31 okurz
322
## Add web UI controls to select more builds in group_overview [gh#804](https://github.com/os-autoinst/openQA/pull/804)
323
324
The query parameter 'limit_builds' allows to show more than the default 10
325
builds on demand. Just like we have for configuring previous results, the
326
current commit adds web UI selections to reload the same page with
327
higher number of builds on demand. For this, the limit of days is increased
328
to show more builds but still limited by the selected number.
329
330
Example screenshot:
331
![openqa_limit_builds_current_bold](https://cloud.githubusercontent.com/assets/1693432/17462279/59e344e6-5ca8-11e6-8350-42a0fbb5267d.png)
332
333 18 okurz
334
## Add more query parameters for configuring last builds [gh#575](https://github.com/os-autoinst/openQA/pull/575)
335
336
By using advanced query parameters in the URLs you can configure the search for builds.
337
Higher numbers would yield more complex database queries but can be selected
338
for special investigation use cases with the advanced query parameters, e.g. if one wants to get an overview of a longer history.
339
This applies to both the index dashboard and group overview page.
340
341
Example to show up to three week old builds instead of the default two weeks
342
with up to 20 builds instead of up to 10 being the default for the group
343
overview page:
344
345 16 okurz
    http://openqa/group_overview/1?time_limit_days=21&limit_builds=20
346 20 okurz
347 19 okurz
## Build tagging and keeping important builds [gh#591](https://github.com/os-autoinst/openQA/pull/591)
348 56 mkittler
349 1 alarrosa
See [online documentation about build tagging](http://open.qa/docs/#_build_tagging).
350 32 okurz
351
## Add web UI controls to filter only tagged or all builds [gh#807](https://github.com/os-autoinst/openQA/pull/807)
352
353
Using a new query parameter 'only_tagged=[0|1]' the list can be filtered, e.g. show only tagged (important) builds.
354
355
Example screenshot:
356
![openqa_limit_builds_current_bold_and_only_tagged](https://cloud.githubusercontent.com/assets/1693432/17464792/49bb6b18-5ce7-11e6-8053-7b74faf193a7.png)
357
358
Related issue: #11052
359 53 okurz
360 21 okurz
## Carry over bugrefs from previous jobs in same scenario if still failing [gh#564](https://github.com/os-autoinst/openQA/pull/564)
361
362
It is possible to label all failing tests but tedious to do by a human user
363
as many failures are just having the same issue until it gets fixed.
364
It helps if a label is preserved for a build that is still failing. This
365 1 alarrosa
idea is inspired by
366 53 okurz
https://wiki.jenkins-ci.org/display/JENKINS/Claim+plugin
367 21 okurz
and has been activated for bugrefs.
368 53 okurz
369 21 okurz
Does not carry over bugrefs over passes: After a job passed a new issue in a subsequent fail is assumed to be failed
370 1 alarrosa
for a different reason.
371
372 23 okurz
Related issue: #10212
373
374 27 okurz
375 26 okurz
## Distinguish product and test issues bugref [gh#708](https://github.com/os-autoinst/openQA/pull/708)
376
377
"progress" is used to track test issues, bugzilla for product issues, at least for SUSE/openSUSE. openQA bugrefs distinguish this and show corresponding icons
378
379
![different_bug_icons](https://cloud.githubusercontent.com/assets/1693432/15814910/e4e74bf6-2bc9-11e6-83de-20f18a7494de.png)
380 37 mkittler
381
## Pinning comments as group description
382
This is possible by adding the keyword `pinned-description` anywhere in a comment on the group overview page. Then the comment will be shown at the top of the group overview page. However, it only works as operator or admin.
383 38 mkittler
384 57 mkittler
## Filtering test results in test result overview
385
386
See [online documentation](http://open.qa/docs/#_filtering_test_results_and_builds).
387 38 mkittler
388 23 okurz
389 52 okurz
## Proposals for uses of labels
390 23 okurz
With [Show bug or label icon on overview if labeled (gh#550)](https://github.com/os-autoinst/openQA/pull/550) it is possible to add custom labels just by writing them. Nevertheless, a convention should be found for a common benefit. <del>Beware that labels are also automatically carried over with (Carry over labels from previous jobs in same scenario if still failing [gh#564])(https://github.com/os-autoinst/openQA/pull/564) which might make consistent test failures less visible when reviewers only look for test results without labels or bugrefs.</del> Labels are not anymore automatically carried over ([gh#1071](https://github.com/os-autoinst/openQA/pull/1071)).
391
392
List of proposed labels with their meaning and where they could be applied.
393
394
* ***`fixed_<build_ref>`***: If a test failure is already fixed in a more recent build and no bug reference is known, use this label together with a reference to a more recent passed test run in the same scenario. Useful for reviewing older builds. Example (https://openqa.suse.de/tests/382518#comments):
395
396
```
397
label:fixed_Build1501
398
399
t#382919
400
```
401 24 okurz
402
* ***`needles_added`***: In case needles were missing for test changes or expected product changes caused needle matching to fail, use this label with a reference to the test PR or a proper reasoning why the needles were missing and how you added them. Example (https://openqa.suse.de/tests/388521#comments):
403
404
```
405
label:needles_added
406
407
needles for https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/1353 were missing, added by jpupava in the meantime.
408 60 mgriessmeier
```
409
410 67 okurz
# s390x Test Organisation
411 1 alarrosa
412 67 okurz
See the following picture for a graphical overview of the current s390x test infrastructure at SUSE:
413
414
![SUSE s390x test infrastructure](qa_sle_openqa_s390x_test_infrastructure.jpg)
415
416 75 okurz
## Upgrades
417 60 mgriessmeier
418
### on z/VM 
419
#### special Requirements
420
421
Due to the lack of proper use of hdd-images on zVM, we need to workaround this with having a dedicated worker_class aka a dedicated Host where we run two jobs with START_AFTER_TEST,
422
the first one which installs the basesystem we want to have upgraded and a second one which is doing the actually upgrade (e.g migration_offline_sle12sp2_zVM_preparation and migration_offline_sle12sp2_zVM)
423
424
Since we encountered issues with randomly other preparation jobs are started in between there, we need to ensure that we have one complete chain for all migration jobs running on one worker, that means for example:
425
426 75 okurz
1. migration_offline_sle12sp2_zVM_preparation 
427
1. migration_offline_sle12sp2_zVM (START_AFTER_TEST=#1) 
428
1. migration_offline_sle12sp2_allpatterns_zVM_preparation (START_AFTER_TEST=#2) 
429
1. migration_offline_sle12sp2_allpatterns_zVM 
430
1. ...
431 66 okurz
432
This scheme ensures that all actual Upgrade jobs are finding the prepared system and are able to upgrade it
433
434
### on z/KVM
435
436 67 okurz
No special requirements anymore, see details in #18016
437 77 nicksinger
438
## Automated z/VM LPAR installation with openQA using qnipl
439
440 78 nicksinger
There is an ongoing effort to automate the LPAR creation and installation on z/VM. A first idea resulted in the creation of [qnipl](https://github.com/openSUSE/dracut-qnipl). `qnipl` enables one to boot a very slim initramfs from a shared medium (e.g. shared SCSI-disks) and supply it with the needed parameters to chainload a "normal SLES installation" using kexec.
441 77 nicksinger
This method is required for z/VM because snipl (Simple network initial program loader) can only load/boot LPARs from specific disks, not network resources.
442
443
### Setup
444
445
1. Get a shared disk for all your LPARs
446
  * Normally this can easily done by infra/gschlotter
447
  * Disks needs to be connected to all guests which should be able to network-boot
448
1. Boot a fully installed SLES on one of the LPARs to start preparing the shared-disk
449
1. Put a DOS partition table on the disk and create one single, large partition on there
450
1. Put a FS on there. Our first test was on ext2 and it worked flawlessly in our attempts
451
1. Install `zipl` (The s390x bootloader from IBM) on this partition
452
  * A simple and sufficient config can be found in [poo#33682](https://progress.opensuse.org/issues/33682)
453
1. clone [`qnipl`](https://github.com/nicksinger/dracut-qnipl) to your dracut modules (e.g. /usr/lib/dracut/modules.d/95qnipl)
454
1. Include the module named `qnipl` to your dracut modules for initramfs generation
455
  * e.g. in /etc/dracut.conf.d/99-qnipl.conf add: `add_dracutmodules+=qnipl`
456
1. Generate your initramfs (e.g. `dracut -f -a "url-lib qnipl" --no-hostonly-cmdline /tmp/custom_initramfs`)
457
  * Put the initramfs next to your kernel binary on the partition you want to prepare
458
1. From now on you can use `snipl` to boot any LPAR connected with this shared disk from network
459
  * example: `snipl -f ./snipl.conf -s P0069A27-LP3 -A fa00 --wwpn_scsiload 500507630713d3b3 --lun_scsiload 4001401100000000 --ossparms_scsiload "install=http://openqa.suse.de/assets/repo/SLE-15-Installer-DVD-s390x-Build533.2-Media1 hostip=10.161.159.3/20 gateway=10.161.159.254 Nameserver=10.160.0.1 Domain=suse.de ssh=1 regurl=http://all-533.2.proxy.scc.suse.de"`
460
  * `--ossparms_scsiload` is then evaluated and used by `qnipl` to kexec into the installer with the (for the installer) needed parameters
461
462
### Further details
463
464 78 nicksinger
Further details can also be found in the [github repo](https://github.com/openSUSE/dracut-qnipl/blob/master/README.md). Pull requests, questions and ideas always welcome!