Project

General

Profile

Wiki » History » Version 124

dheidler, 2021-08-18 12:38

1 3 okurz
# Introduction
2 1 alarrosa
3 3 okurz
This is the organisation wiki for the **openQA Project**.
4 49 okurz
The source code is hosted in the [os-autoinst github project](http://github.com/os-autoinst/), especially [openQA itself](http://github.com/os-autoinst/openQA) and the main backend [os-autoinst](http://github.com/os-autoinst/os-autoinst)
5 1 alarrosa
6 48 okurz
If you are interested in the tests for SUSE/openSUSE products take a look into the [openqatests](https://progress.opensuse.org/projects/openqatests) project.
7
8 70 szarate
If you are looking for entry level issues to contribute to the backend, take a look at [this search query](https://progress.opensuse.org/projects/openqav3/search?utf8=%E2%9C%93&issues=1&q=entrance+level+issue)
9
10 14 okurz
{{toc}}
11
12 3 okurz
# Organisational
13 1 alarrosa
14 51 okurz
## ticket workflow
15
16 65 SLindoMansilla
Picture: http://imagebin.suse.de/2127/img
17 64 SLindoMansilla
18 51 okurz
The following ticket statuses are used together and their meaning is explained:
19
20 63 okurz
* *New*: No one has worked on the ticket (e.g. the ticket has not been properly refined) or no one is feeling responsible for the work on this ticket.
21 73 riafarov
* *Workable*: The ticket has been refined and is ready to be picked.
22
* *In Progress*: Assignee is actively working on the ticket.
23 1 alarrosa
* *Resolved*: The complete work on this issue is done and the according issue is supposed to be fixed as observed (Should be updated together with a link to a merged pull request or also a link to an production openQA showing the effect)
24 73 riafarov
* *Feedback*: Further work on the ticket is blocked by open points or is awaiting for the feedback to proceed. Sometimes also used to ask Assignee about progress on inactivity.
25 74 okurz
* *Blocked*: Further work on the ticket is blocked by some external dependency (e.g. bugs, not implemented features). There should be a link to another ticket, bug, trello card, etc. where it can be seen what the ticket is blocked by.
26 51 okurz
* *Rejected*: The issue is considered invalid, should not be done, is considered out of scope.
27
* *Closed*: As this can be set only by administrators it is suggested to not use this status.
28
29
It is good practice to update the status together with a comment about it, e.g. a link to a pull request or a reason for reject.
30
31 80 okurz
## ticket categories
32
33
* *Concrete Bugs*: Regressions, crashes, error messages
34
* *Feature requests*: Ideas or wishes for extension, enhancement, improvement
35
* *Organisational*: Organisational tasks within the project(s), not directly code related
36
* *Support*: Support of users, usage problems, questions
37
38 1 alarrosa
Please avoid the use of other, deprecated categories
39
40
Suggestion by *okurz*: I recommend to avoid the word "bug" in our categories because of the usual "is it a bug or a feature" struggle. Instead I suggest to strictly define "Regressions & Crashes" to clearly separate "it used to work in before" from "this was never part of requirements" for Features. Any ticket of this category also means that our project processes missed something so we have points for improvements, e.g. extend things to look out for in code review.
41 100 okurz
42
## Epics and Sagas
43
44
[epic]s and [saga]s belong to the "coordination" tracker, project contributors are not required to follow this convention but the tracker may be changed automagically in the future: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html 
45 83 okurz
46 13 okurz
## ticket templates
47
You can use these templates to fill in tickets and further improve them with more detail over time. Copy the code block, paste it into a new issue, replace every block marked with "<…>" with your content or delete if not appropriate.
48
49 71 nicksinger
### Defects
50 13 okurz
51
Subject: `<Short description, example: "openQA dies when triggering any Windows ME tests">`
52
53 1 alarrosa
54 13 okurz
```
55 71 nicksinger
## Observation
56 13 okurz
<description of what can be observed and what the symptoms are, provide links to failing test results and/or put short blocks from the log output here to visualize what is happening>
57
58 71 nicksinger
## Steps to reproduce
59 1 alarrosa
* <do this>
60 13 okurz
* <do that>
61 1 alarrosa
* <observe result>
62 13 okurz
63 71 nicksinger
## Problem
64 13 okurz
<problem investigation, can also include different hypotheses, should be labeled as "H1" for first hypothesis, etc.>
65
66 71 nicksinger
## Suggestion
67 123 okurz
* <what to do as a first step>
68
* <Fix the actual problem>
69
* <Consider fixing the design>
70
* <Consider fixing the team's process>
71
* <Consider to explore further>
72 13 okurz
73 71 nicksinger
## Workaround
74 13 okurz
<example: retrigger job>
75
```
76
77
example ticket: #10526
78
79 104 okurz
For tickets referencing "auto_review" see
80
https://github.com/os-autoinst/scripts/blob/master/README.md#auto-review---automatically-detect-known-issues-in-openqa-jobs-label-openqa-jobs-with-ticket-references-and-optionally-retrigger
81
for a suggested template snippet.
82
83 72 nicksinger
### Feature requests
84 13 okurz
85
Subject: `<Short description, example: "grub3 btrfs support" (feature)>`
86
87
88
```
89
## User story
90
<As a <role>, I want to <do an action>, to <achieve which goal> >
91
92 72 nicksinger
## Acceptance criteria
93 13 okurz
* <**AC1:** the first acceptance criterion that needs to be fulfilled to do this, example: Clicking "restart button" causes restart of the job>
94
* <**AC2:** also think about the "not-actions", example: other jobs are not affected>
95
96 72 nicksinger
## Tasks
97 13 okurz
* <first task to do as an easy starting point>
98 69 okurz
* <what do do next, all tasks optionally with an effort estimation in hours, e.g. "(0.5-2h)">
99 13 okurz
* <optional: mark "optional" tasks>
100
101 72 nicksinger
## Further details
102 17 okurz
<everything that does not fit into above sections>
103 13 okurz
```
104
105
example ticket: #10212
106
107 62 SLindoMansilla
## Further decision steps working on test issues
108 61 SLindoMansilla
109 62 SLindoMansilla
Test issues could be one of the following sources. Feel free to use the following template in tickets as well
110 1 alarrosa
111 62 SLindoMansilla
```
112
## Problem
113
* **H1** The product has changed
114
 * **H1.1** product changed slightly but in an acceptable way without the need for communication with DEV+RM --> adapt test
115
 * **H1.2** product changed slightly but in an acceptable way found after feedback from RM --> adapt test
116
 * **H1.3** product changed significantly --> after approval by RM adapt test
117 61 SLindoMansilla
118 62 SLindoMansilla
* **H2** Fails because of changes in test setup
119
 * **H2.1** Our test hardware equipment behaves different
120
 * **H2.2** The network behaves different
121
122
* **H3** Fails because of changes in test infrastructure software, e.g. os-autoinst, openQA
123
* **H4** Fails because of changes in test management configuration, e.g. openQA database settings
124
* **H5** Fails because of changes in the test software itself (the test plan in source code as well as needles)
125
* **H6** Sporadic issue, i.e. the root problem is already hidden in the system for a long time but does not show symptoms every time
126
```
127 25 okurz
128
## pull request handling on github
129
130
As a reviewer of pull requests on github for all related repositories, e.g. https://github.com/os-autoinst/os-autoinst-distri-opensuse/pulls, apply labels in case PRs are open for a longer time and can not be merged so that we keep our backlog clean and know why PRs are blocked.
131
132
* **notready**: Triaged as not ready yet for merging, no (immediate) reaction by the reviewee, e.g. when tests are missing, other scenarios break, only tested for one of SLE/TW
133
* **wip**: Marked by the reviewee itself as "[WIP]" or "[DO-NOT-MERGE]" or similar
134
* **question**: Questions to the reviewee, not answered yet
135 54 okurz
136
137
## Where to contribute?
138
139
If you want to help openQA development you can take a look into the existing [issues](https://progress.opensuse.org/projects/openqav3/issues). There are also some "always valid" tasks to be working on:
140
141
* *improve test coverage*:
142
 * *user story*: As openqa backend as well as test developer I want better test coverage of our projects to reduce technical debt
143
 * *acceptance criteria*: test coverage is significantly higher than before
144
 * *suggestions*: check current coverage in each individual project (os-autoinst/openQA/os-autoinst-distri-opensuse) and add tests as necessary
145
146 28 okurz
147 1 alarrosa
# Use cases
148 40 okurz
149 28 okurz
The following use cases 1-6 have been defined within a SUSE workshop (others have been defined later) to clarify how different actors work with openQA. Some of them are covered already within openQA quite well, some others are stated as motivation for further feature development.
150
151 6 okurz
## Use case 1
152 4 okurz
**User:** QA-Project Managment
153 1 alarrosa
**primary actor:** QA Project Manager, QA Team Leads
154
**stakeholder:** Directors, VP
155 7 okurz
**trigger:** product milestones, providing a daily status
156 1 alarrosa
**user story:** „As a QA project manager I want to check on a daily basis the „openQA Dashboard“ to get a summary/an overall status of the „reviewers results“ in order to take the right actions and prioritize tasks in QA accordingly.“
157 28 okurz
	
158 4 okurz
## Use case 2
159 1 alarrosa
**User:** openQA-Admin
160
**primary actor:** Backend-Team
161 4 okurz
**stakeholder:** Qa-Prjmgr, QA-TL, openQA Tech-Lead
162 7 okurz
**trigger:** Bugs, features, new testcases
163 5 okurz
**user story:** „As an openQA admin I constantly check in the web-UI the system health and I manage its configuration to ensure smooth operation of the tool.“
164 28 okurz
165 1 alarrosa
## Use case 3
166
**User:** QA-Reviewer
167
**primary actor:** QA-Team
168 4 okurz
**stakeholder:** QA-Prjmgr, Release-Mgmt, openQA-Admin
169 7 okurz
**trigger:** every new build
170
**user story:** „As an openQA-Reviewer at any point in time I review on the webpage of openQA the overall status of a build in order to track and find bugs, because I want to find bugs as early as possible and report them.“
171 28 okurz
172 1 alarrosa
## Use case 4
173
**User:** Testcase-Contributor
174 4 okurz
**primary actor:** All development teams, Maintenance QA
175 5 okurz
**stakeholder:** QA-Reviewer, openQA-Admin, openQA Tech-Lead
176 40 okurz
**trigger:** features, new functionality, bugs, new product/package
177 7 okurz
**user story:** „As developer when there are new features, new functionality, bugs, new product/package in git I contribute my testcases because I want to ensure good quality submissions and smooth product integration.“
178 28 okurz
179 4 okurz
## Use case 5
180
**User:** Release-Mgmt
181
**primary actor:** Release Manager
182 1 alarrosa
**stakeholder:** Directors, VP, PM, TAMs, Partners
183 7 okurz
**trigger:** Milestones
184
**user story:** „As a Release-Manager on a daily basis I check on a dashboard for the product health/build status in order to act early in case of failures and have concrete and current reports.“
185 28 okurz
186 4 okurz
## Use case 6
187
**User:** Staging-Admin
188
**primary actor:** Staging-Manager for the products
189 1 alarrosa
**stakeholder:** Release-Mgmt, Build-Team
190
**trigger:** every single submission to projects
191 40 okurz
**user story:** „As a Staging-Manager I review the build status of packages with every staged submission to the „staging projects“ in the „staging dashboard“ and the test-status of the pre-integrated fixes, because I want to identify major breakage before integration to the products and provide fast feedback back to the development.“
192
193
## Use case 7
194
**User:** Bug investigator
195
**primary actor:** Any bug assignee for openQA observed bugs
196
**stakeholder:** Developer
197
**trigger:** bugs
198 8 okurz
**user story:** „As a developer that has been assigned a bug which has been observed in openQA I can review referenced tests, find a newer and the most recent job in the same scenario, understand what changed since the last successful job, what other jobs show same symptoms to investigate the root cause fast and use openQA for verification of a bug fix.“
199 15 okurz
200 8 okurz
# Thoughts about categorizing test results, issues, states within openQA
201
by okurz
202
203
When reviewing test results it is important to distinguish between different causes of "failed tests"
204
205
## Nomenclature
206
207 58 okurz
### Test status categories
208 1 alarrosa
A common definition about the status of a test regarding the product it tests: "false|true positive|negative" as described on https://en.wikipedia.org/wiki/False_positives_and_false_negatives. "positive|negative" describes the outcome of a test ("positive": test signals presence of issue; "negative": no signal) whereas "false|true" describes the conclusion of the test regarding the presence of issues in the SUT or product in our case ("true": correct reporting; "false": incorrect reporting), e.g. "true negative", test successful, no issues detected and there are no issues, product is working as expected by customer. Another example: Think of testing as of a fire alarm. An alarm (event detector) should only go off (be "positive") *if* there is a fire (event to detect) --> "true positive" whereas *if* there is *no* fire there should be *no* alarm --> "true negative".
209 10 okurz
210 1 alarrosa
Another common but potentially ambiguous categorization:
211 10 okurz
212
* *broken*: the test is not behaving as expected (Ambiguity: "as expected" by whom?) --> commonly a "false positive", can also be "false negative" but hard to detect
213
* *failing*: the test is behaving as expected, but the test output is a fail --> "true positive"
214
* *working*: the test is behaving as expected (with no comment regarding the result, though some might ambiguously imply 'result is negative')
215
* *passing*: the test is behaving as expected, but the result is a success --> "true negative"
216 8 okurz
217 9 okurz
If in doubt declare a test as "broken". We should review the test and examine if it is behaving as expected.
218 10 okurz
219 8 okurz
Be careful about "positive/negative" as some might also use "positive" to incorrectly denote a passing test (and "negative" for failing test) as an indicator of "working product" not an indicator about "issue present". If you argue what is "used in common speech" think about how "false positive" is used as in "false alarm" --> "positive" == "alarm raised", also see https://narainko.wordpress.com/2012/08/26/understanding-false-positive-and-false-negative/
220
221 10 okurz
### Priorization of work regarding categories
222 3 okurz
In this sense development+QA want to accomplish a "true negative" state whenever possible (no issues present, therefore none detected). As QA and test developers we want to prevent "false positives" ("false alarms" declaring a product as broken when it is not but the test failed for other reasons), also known as "type I error" and "false negatives" (a product issue is not catched by tests and might "slip through" QA and at worst is only found by an external outside customer) also known as "type II error". Also see https://en.wikipedia.org/wiki/Type_I_and_type_II_errors. In the context of openQA and system testing paired with screen matching a "false positive" is much more likely as the tests are very susceptible to subtle variations and changes even if they should be accepted. So when in doubt, create an issue in progress, look at it again, and find that it was a false alarm, rather than wasting more peoples time with INVALID bug reports by believing the product to be broken when it isn't. To quote Richard Brown: "I […] believe this is the route to ongoing improvement - if we have tests which produce such false alarms, then that is a clear indicator that the test needs to be reworked to be less ambiguous, and that IS our job as openQA developers to deal with".
223 11 okurz
224
## Further categorization of statuses, issues and such in testing, especially automatic tests
225
By okurz
226
227
This categorization scheme is meant to help in communication in either written or spoken discussions being simple, concise, easy to remember while unambiguous in every case.
228
While used for naming it should also be used as a decision tree and can be followed from the top following each branch.
229
230
### Categorization scheme
231
232
To keep it simple I will try to go in steps of deciding if a potential issue is of one of two categories in every step (maybe three) and go further down from there. The degree of further detailing is not limited, i.e. it can be further extended. Naming scheme should follow arabic number (for two levels just 1 and 2) counting schemes added from the right for every additional level of decision step and detail without any separation between the digits, e.g. "1111" for the first type in every level of detail up to level four. Also, I am thinking of giving the fully written form phonetic name to unambiguously identify each on every level as long as not more individual levels are necessary. The alphabet should be reserved for higher levels and higher priority types.
233
Every leaf of the tree must have an action assigned to it.
234 12 okurz
235 11 okurz
1 **failed** (ZULU)
236
11 new (passed->failed) (YANKEE)
237
111 product issue ("true positive") (WHISKEY)
238 44 okurz
1111 unfiled issue (SIERRA)
239 11 okurz
11111 hard issue (openqa *fail*) (KILO)
240
111121 critical / potential ship stopper (INDIA) --> immediately file bug report with "ship_stopper?" flag; opt. inform RM directly
241 44 okurz
111122 non-critical hard issue (HOTEL) --> file bug report
242 11 okurz
11112 soft issue (openqa *softfail* on job level, not on module level) (JULIETT) --> file bug report on failing test module
243
1112 bugzilla bug exists (ROMEO)
244
11121 bug was known to openqa / openqa developer --> cross-reference (bug->test, test->bug) AND raise review process issue, improve openqa process
245
11122 bug was filed by other sources (e.g. beta-tester) --> cross-reference (bug->test, test->bug)
246
112 test issue ("false positive") (VICTOR)
247
1121 progress issue exists (QUEBEC) --> cross-reference (issue->test, test->issue)
248
1122 unfiled test issue (PAPA)
249
11221 easy to do w/o progress issue
250
112211 need needles update --> re-needle if sure, TODO how to notify?
251
112212 pot. flaky, timeout
252
1122121 retrigger yields PASS --> comment in progress about flaky issue fixed
253
1122122 reproducible on retrigger --> file progress issue
254
11222 needs progress issue filed --> file progress issue
255
12 existing / still failing (failed->failed) (XRAY)
256
121 product issue (UNIFORM)
257
1211 unfiled issue (OSCAR) --> file bug report AND raise review process issue (why has it not been found and filed?)
258
1212 bugzilla bug exists (NOVEMBER) --> ensure cross-reference, also see rules for 1112 ROMEO
259
122 test issue (TANGO)
260
1221 progress issue exists (MIKE) --> monitor, if persisting reprioritize test development work
261
1222 needs progress issue filed (LIMA) --> file progress issue AND raise review process issue, see 1211 OSCAR
262
2 **passed** (ALFA)
263
21 stable (passed->passed) (BRAVO)
264
211 existing "true negative" (DELTA) --> monitor, maybe can be made stricter
265
212 existing "false negative" (ECHO) --> needs test improvement
266
22 fixed (failed->passed) (CHARLIE)
267
222 fixed "true negative" (FOXTROTT) --> TODO split monitor, see 211 DELTA
268
2221 was test issue --> close progress issue
269
2222 was product issue
270
22221 no bug report exists --> raise review process issue (why was it not filed?)
271
22222 bug report exists
272
222221 was marked as RESOLVED FIXED
273
221 fixed but "false negative" (GOLF) --> potentially revert test fix, also see 212 ECHO
274 41 okurz
275
276 11 okurz
Priority from high to low: INDIA->OSCAR->HOTEL->JULIETT->…
277 35 okurz
278 82 okurz
# Proposals for uses of labels
279 23 okurz
With [Show bug or label icon on overview if labeled (gh#550)](https://github.com/os-autoinst/openQA/pull/550) it is possible to add custom labels just by writing them. Nevertheless, a convention should be found for a common benefit. <del>Beware that labels are also automatically carried over with (Carry over labels from previous jobs in same scenario if still failing [gh#564])(https://github.com/os-autoinst/openQA/pull/564) which might make consistent test failures less visible when reviewers only look for test results without labels or bugrefs.</del> Labels are not anymore automatically carried over ([gh#1071](https://github.com/os-autoinst/openQA/pull/1071)).
280
281
List of proposed labels with their meaning and where they could be applied.
282
283
* ***`fixed_<build_ref>`***: If a test failure is already fixed in a more recent build and no bug reference is known, use this label together with a reference to a more recent passed test run in the same scenario. Useful for reviewing older builds. Example (https://openqa.suse.de/tests/382518#comments):
284
285
```
286
label:fixed_Build1501
287
288
t#382919
289
```
290 24 okurz
291
* ***`needles_added`***: In case needles were missing for test changes or expected product changes caused needle matching to fail, use this label with a reference to the test PR or a proper reasoning why the needles were missing and how you added them. Example (https://openqa.suse.de/tests/388521#comments):
292
293
```
294
label:needles_added
295
296
needles for https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/1353 were missing, added by jpupava in the meantime.
297 60 mgriessmeier
```
298
299 67 okurz
# s390x Test Organisation
300 1 alarrosa
301 67 okurz
See the following picture for a graphical overview of the current s390x test infrastructure at SUSE:
302
303
![SUSE s390x test infrastructure](qa_sle_openqa_s390x_test_infrastructure.jpg)
304
305 75 okurz
## Upgrades
306 60 mgriessmeier
307
### on z/VM 
308
#### special Requirements
309
310
Due to the lack of proper use of hdd-images on zVM, we need to workaround this with having a dedicated worker_class aka a dedicated Host where we run two jobs with START_AFTER_TEST,
311
the first one which installs the basesystem we want to have upgraded and a second one which is doing the actually upgrade (e.g migration_offline_sle12sp2_zVM_preparation and migration_offline_sle12sp2_zVM)
312
313
Since we encountered issues with randomly other preparation jobs are started in between there, we need to ensure that we have one complete chain for all migration jobs running on one worker, that means for example:
314
315 75 okurz
1. migration_offline_sle12sp2_zVM_preparation 
316
1. migration_offline_sle12sp2_zVM (START_AFTER_TEST=#1) 
317
1. migration_offline_sle12sp2_allpatterns_zVM_preparation (START_AFTER_TEST=#2) 
318
1. migration_offline_sle12sp2_allpatterns_zVM 
319
1. ...
320 66 okurz
321
This scheme ensures that all actual Upgrade jobs are finding the prepared system and are able to upgrade it
322
323
### on z/KVM
324
325 67 okurz
No special requirements anymore, see details in #18016
326 77 nicksinger
327
## Automated z/VM LPAR installation with openQA using qnipl
328
329 78 nicksinger
There is an ongoing effort to automate the LPAR creation and installation on z/VM. A first idea resulted in the creation of [qnipl](https://github.com/openSUSE/dracut-qnipl). `qnipl` enables one to boot a very slim initramfs from a shared medium (e.g. shared SCSI-disks) and supply it with the needed parameters to chainload a "normal SLES installation" using kexec.
330 77 nicksinger
This method is required for z/VM because snipl (Simple network initial program loader) can only load/boot LPARs from specific disks, not network resources.
331
332
### Setup
333
334
1. Get a shared disk for all your LPARs
335
  * Normally this can easily done by infra/gschlotter
336
  * Disks needs to be connected to all guests which should be able to network-boot
337
1. Boot a fully installed SLES on one of the LPARs to start preparing the shared-disk
338
1. Put a DOS partition table on the disk and create one single, large partition on there
339
1. Put a FS on there. Our first test was on ext2 and it worked flawlessly in our attempts
340
1. Install `zipl` (The s390x bootloader from IBM) on this partition
341
  * A simple and sufficient config can be found in [poo#33682](https://progress.opensuse.org/issues/33682)
342
1. clone [`qnipl`](https://github.com/nicksinger/dracut-qnipl) to your dracut modules (e.g. /usr/lib/dracut/modules.d/95qnipl)
343
1. Include the module named `qnipl` to your dracut modules for initramfs generation
344
  * e.g. in /etc/dracut.conf.d/99-qnipl.conf add: `add_dracutmodules+=qnipl`
345
1. Generate your initramfs (e.g. `dracut -f -a "url-lib qnipl" --no-hostonly-cmdline /tmp/custom_initramfs`)
346
  * Put the initramfs next to your kernel binary on the partition you want to prepare
347
1. From now on you can use `snipl` to boot any LPAR connected with this shared disk from network
348
  * example: `snipl -f ./snipl.conf -s P0069A27-LP3 -A fa00 --wwpn_scsiload 500507630713d3b3 --lun_scsiload 4001401100000000 --ossparms_scsiload "install=http://openqa.suse.de/assets/repo/SLE-15-Installer-DVD-s390x-Build533.2-Media1 hostip=10.161.159.3/20 gateway=10.161.159.254 Nameserver=10.160.0.1 Domain=suse.de ssh=1 regurl=http://all-533.2.proxy.scc.suse.de"`
349
  * `--ossparms_scsiload` is then evaluated and used by `qnipl` to kexec into the installer with the (for the installer) needed parameters
350
351
### Further details
352
353 78 nicksinger
Further details can also be found in the [github repo](https://github.com/openSUSE/dracut-qnipl/blob/master/README.md). Pull requests, questions and ideas always welcome!
354 84 okurz
355 109 okurz
# Infrastructure setup for o3 (openqa.opensuse.org) and osd (openqa.suse.de)
356 1 alarrosa
357 109 okurz
## o3 (openqa.opensuse.org)
358
359 113 okurz
o3 consists of a VM running the web UI and physical worker machines. The VM for o3 has netapp backed storage on rotating disk so less performant than SSD but cheaper. So eventually we might have the possibility to use SSD based storage. Currently there are four virtual storage devices provided to o3 totalling to 10 TB.
360 88 okurz
361 109 okurz
### Automatic update of o3
362 1 alarrosa
363 87 okurz
o3 is automatically deployed on a daily base, that includes both the webUI host as well as the workers.
364 90 okurz
365 109 okurz
#### Automatic update of o3 webUI host
366 1 alarrosa
367 90 okurz
Done with cron job in `/etc/cron.d/auto-update`
368
369 109 okurz
#### Recurring automatic update of openQA workers
370 87 okurz
371 94 okurz
All o3 workers (except power8) apply a daily automatic update and are "Transactional Servers" running openSUSE Leap. power8 is non-transactional with a weekly update every Sunday.
372 87 okurz
373
This was for a number of reasons including:
374
375
* Getting all the machines consistent after a few years of drift
376
* Making it easier to keep them consistent by leveraging a read only root filesystem
377
* Guaranteeing rollbackability by using transactional updates
378
379 1 alarrosa
This was done by rbrown also to fulfill the prerequisite to getting them viable for multi-machine testing
380 87 okurz
381 90 okurz
These systems currently patch themselves and reboot automatically in the default maintenance window of 0330-0500 CET/CEST.
382 87 okurz
383
On problems this could be changed in the following way:
384
385
* Edit the maintenance window in /etc/rebootmgr.conf
386
* Disable the automatic reboot by "systemctl disable rebootmgr.service"
387
* Disable the automatic patching by "systemctl disable transactional-update.timer"
388
389
SUSE employees have access to the bootmenu for the openQA worker machines, e.g. openqaworker1 and openqaworker4 via openqaworker1- ipmi.suse.de and openqaworker4-ipmi.suse.de which are both connected to the r&d network. For imagetester one would need to go through SUSE-IT in an unlikely event of a boot-preventing update. "snapper rollback" can be executed from a booted, functionally operative machine which one can ssh into.
390 84 okurz
391 91 okurz
To execute commands manually on all workers one can do for example the following:
392
393
```
394
for i in aarch64 openqaworker1 openqaworker4 openqaworker7 power8 imagetester rebel; do echo $i && ssh root@$i "(transactional-update -n dup || zypper -n dup) && reboot" ; done
395
```
396 1 alarrosa
397 91 okurz
mind the correct list of machines.
398
399 92 okurz
For manual investigation https://github.com/kubic-project/microos-toolbox can be helpful
400
401 111 okurz
#### Rollback of updates
402
403
Updates on workers can be rolled back using `transactional-update` affecting the transactional workers (others are likely not updated that often):
404
405
```
406
for i in aarch64 openqaworker1 openqaworker4 openqaworker7 power8 imagetester rebel; do echo $i && ssh root@$i "transactional-update rollback last && reboot"; done
407
```
408
409
Updates on the central webUI host openqa.opensuse.org can be rolled back by using either older variants of packages that receive maintenance updates or using the locally cached packages in e.g. /var/cache/zypp/packages/devel_openQA/noarch using `zypper in --oldpackage`, similar to https://github.com/os-autoinst/openQA/blob/master/script/openqa-rollback#L39
410
411 109 okurz
### Accessing o3 infrastructure
412 96 okurz
413
The o3 webui host as well as the workers within the o3 infrastructure can be accessed over ssh by using `ssh -p 2213 gate.opensuse.org`. Ask one of the existing admins to put your ssh key there to be able to login.
414
415 102 okurz
To give access for a new user an existing admin can do the following:
416 1 alarrosa
417 102 okurz
```
418
sudo useradd -G users,trusted --create-home $user
419 112 okurz
echo "$ssh_key_from_user" | sudo tee -a /home/$user/.ssh/authorized_keys
420 102 okurz
```
421
422 109 okurz
#### SSH configuration
423 105 nicksinger
424
To easily access all hosts behind the jump host you can use the following config for your ssh client (`~.ssh/config`):
425
426
```
427
Host ariel
428
  HostName gate.opensuse.org
429
  Port 2213
430
431
Host *.opensuse.org
432
  ProxyCommand ssh -q -A -x ariel -W %h:%p
433
```
434
435
**A word of warning**: be aware that this enables agent-forwarding to at least the jumphost. Please read up for yourself if and how bad you consider the security implications of doing so.
436
437 108 SLindoMansilla
438
#### Debugging qemu SUTs in openqa.opensuse.org
439
440
SUT: System Under Test
441
442
os-autoinst starts qemu with network type that doesn't allow access from the outside, so ssh is not possible. But, qemu is started with a VNC channel available from the host (the openQA-worker).
443
Running vncviewer inside a headless server is useless, but it is possible to use gate.opensuse.org as a jump host and SSH port forwarding to start vncviewer client from your desktop environment and connect to the VNC channel of the qemu SUT.
444
445
```
446
ssh -p 2213 -L LOCAL_PORT:WORKER_HOSTNAME:QEMU_VNC_PORT USERNAME@gate.opensuse.org
447
```
448
449
For example, if user **bernhard**, wants to connect to openqaworker7:11, and wants to use local port **43043**
450
Being the IP of openqaworker7 **192.168.112.12**
451
And the VNC channel port of openqa-worker@11 **6001** (5990 + 11)
452
453
##### 1. Create SSH tunnel with port forwarding
454
* on laptop shell 1: ssh -p 2213 -L 43043:192.168.112.12:6001 bernhard@gate.opensuse.org
455 1 alarrosa
* Keep shell open to keep the tunnel open and the port forwarding
456 108 SLindoMansilla
457 1 alarrosa
##### 2. Open vncviewer
458
* on laptop shell 2: vncviewer -Shared localhost:43043
459
* `-shared` is needed to not kick the VNC connection of os-autoinst. If it is kicked, the job will terminate and the qemu process will be killed.
460
461 109 okurz
### AArch64 specific configurations on o3
462 1 alarrosa
463 109 okurz
On o3, the aarch64 workers need additional configuration.
464
465
#### Setup HugePages
466
467
You need to setup HugePages support to improve performances with qemu VM and to match current aarch64 `MACHINE` configuration.
468
For the D05 machine, the configuration is: `40` pages with a size of `1G`.
469
If there are some permissions issues on `/dev/hugepages/`, check https://progress.opensuse.org/issues/53234
470
471 121 okurz
### Monitoring
472
473
There is an internal munin instance on o3. Anyone wanting to look at the HTML pages, do this:
474
```
475
rsync -a o3:/srv/www/htdocs/munin ~/o3-munin/ 
476
```
477
(where "o3" is configured in your ssh config of course)
478
479 89 ggardet_arm
## Mitigation of boot failure or disk issues
480
481
### Worker stuck in recovery
482
483
Check disk health and consider manual fixup of mount points, e.g.:
484
485
```
486
test -e /dev/md/openqa || lsblk -n | grep -v nvme | grep "/$" && mdadm --create /dev/md/openqa --level=0 --force --raid-devices=$(ls /dev/nvme?n1 | wc -l) --run /dev/nvme?n1 || mdadm --create /dev/md/openqa --level=0 --force --raid-devices=1 --run /dev/nvme0n1p3
487
```
488
489 106 okurz
## PPC specific configurations
490
491
In one case it was necessary to disable snapshots for petitboot with `nvram -p default --update-config "petitboot,snapshots?=false"` to prevent a race condition between dm_raid and btrfs trying to discover bootable devices (https://progress.opensuse.org/issues/68053#note-25). In another case https://bugzilla.opensuse.org/show_bug.cgi?id=1174166 caused the boot entries to be not properly discovered and it was necessary to prevent grub from trying to update the according sections (https://progress.opensuse.org/issues/68053#note-31).
492 89 ggardet_arm
493 84 okurz
## Moving worker from osd to o3
494
495
* Ensure system management, e.g. over IPMI works. This is untouched by the following steps and can be used during the process for recovery and setup
496
* Ensure network is configured for DHCP
497
* Instruct SUSE-IT to change VLAN for machine from 2 to 662 (example: https://infra.nue.suse.com/SelfService/Display.html?id=16458)
498
* Remove from osd:
499
500
```
501
salt-key -y -d openqaworker7.suse.de
502
```
503
504
* Add entry on o3 to `/etc/dnsmasq.d/openqa.conf` with MAC address, e.g.
505
506
```
507
dhcp-host=54:ab:3a:24:34:b8,openqaworker7
508
```
509
510
* Add entry to `/etc/hosts` which dnsmasq picks up to give out a DHCP lease, e.g.
511
512
```
513
192.168.112.12   openqaworker7.openqanet.opensuse.org openqaworker7
514
```
515
516 85 okurz
* Adapt NFS mount point
517
518
```
519
sed -i '/openqa\.suse\.de/d' /etc/fstab && echo 'openqa1-opensuse:/ /var/lib/openqa/share nfs4 ro,fsc 0 0' >> /etc/fstab
520
```
521
522 84 okurz
* Reload dnsmasq with `systemctl restart dnsmasq`
523
* Restart network on machine (over IMPI) using `systemctl restart network` and monitor in o3:`journalctl -f -u dnsmasq` until address is assigned, e.g.:
524
525
```
526
Feb 29 10:48:30 ariel dnsmasq[28105]: read /etc/hosts - 30 addresses
527
Feb 29 10:48:54 ariel dnsmasq-dhcp[28105]: DHCPREQUEST(eth1) 10.160.1.101 54:ab:3a:24:34:b8
528
Feb 29 10:48:54 ariel dnsmasq-dhcp[28105]: DHCPNAK(eth1) 10.160.1.101 54:ab:3a:24:34:b8 wrong network
529
Feb 29 10:49:10 ariel dnsmasq-dhcp[28105]: DHCPDISCOVER(eth1) 54:ab:3a:24:34:b8
530
Feb 29 10:49:10 ariel dnsmasq-dhcp[28105]: DHCPOFFER(eth1) 192.168.112.12 54:ab:3a:24:34:b8
531
Feb 29 10:49:10 ariel dnsmasq-dhcp[28105]: DHCPREQUEST(eth1) 192.168.112.12 54:ab:3a:24:34:b8
532
Feb 29 10:49:10 ariel dnsmasq-dhcp[28105]: DHCPACK(eth1) 192.168.112.12 54:ab:3a:24:34:b8 openqaworker7
533 85 okurz
```
534
535
* Ensure all mountpoints up
536
537
```
538
mount -a
539 84 okurz
```
540
541
* Change root password to o3 one
542 86 okurz
* Allow ssh password authentication: `sed -i 's/^PasswordAuthentication/#&/' /etc/ssh/sshd_config && systemctl restart sshd`
543 84 okurz
* Add personal ssh key to machine, e.g. openqaworker7:/root/.ssh/authorized_keys
544
* Update /etc/openqa/client.conf with the same key as used on other workers for "openqa1-opensuse"
545
* Update /etc/openqa/workers.ini with similar config as used on other workers, e.g. based on openqaworker4, example:
546
547
```
548
# diff -Naur /etc/openqa/workers.ini{.osd,}
549
--- /etc/openqa/workers.ini.osd 2020-02-29 15:21:47.737998821 +0100
550
+++ /etc/openqa/workers.ini     2020-02-29 15:22:53.334464958 +0100
551
@@ -1,17 +1,10 @@
552
-# This file is generated by salt - don't touch
553
-# Hosted on https://gitlab.suse.de/openqa/salt-pillars-openqa
554
-# numofworkers: 10
555
-
556
 [global]
557
-HOST=openqa.suse.de
558
-CACHEDIRECTORY=/var/lib/openqa/cache
559
-LOG_LEVEL=debug
560
-WORKER_CLASS=qemu_x86_64,qemu_x86_64_staging,tap,openqaworker7
561
-WORKER_HOSTNAME=10.160.1.101
562
-
563
-[1]
564
-WORKER_CLASS=qemu_x86_64,qemu_x86_64_staging,tap,qemu_x86_64_ibft,openqaworker7
565
+HOST=http://openqa1-opensuse
566
+WORKER_HOSTNAME=192.168.112.12
567
+CACHEDIRECTORY = /var/lib/openqa/cache
568
+CACHELIMIT = 50
569
+WORKER_CLASS = openqaworker7,qemu_x86_64
570
571
-[openqa.suse.de]
572
-TESTPOOLSERVER = rsync://openqa.suse.de/tests
573
+[http://openqa1-opensuse]
574
+TESTPOOLSERVER = rsync://openqa1-opensuse/tests
575
```
576
577
* Remove OSD specifics
578
579
```
580
systemctl disable --now auto-update.timer salt-minion telegraf
581
for i in  NPI SUSE_CA telegraf-monitoring; do zypper rr $i; done
582
zypper -n dup --force-resolution --allow-vendor-change
583
```
584
585
* If the machine is not a transactional-server one has the following options: Keep as is and handle like power8 (also not transactional), enable transactional updates w/o root being r/o, change to root being r/o on-the-fly, reinstall as transactional. At least option 2 is suggested, enable transactional updates:
586
587
```
588
zypper -n in transactional-update
589
systemctl enable --now transactional-update.timer rebootmgr
590
```
591
592
* Enable apparmor
593
594
```
595
zypper -n in apparmor-utils
596
systemctl unmask apparmor
597
systemctl enable --now apparmor
598
```
599
600
* Switch firewall from SuSEfirewall2 to firewalld
601
602
```
603
zypper -n in firewalld && zypper -n rm SuSEfirewall2
604
systemctl enable --now firewalld
605
firewall-cmd --zone=trusted --add-interface=br1
606
firewall-cmd --set-default-zone trusted
607
firewall-cmd --zone=trusted --add-masquerade
608
```
609
610
* Copy over special openSUSE UEFI staging images, see #63382
611
* Check operation with a single openQA worker instance:
612
613
```
614
systemctl enable --now openqa-worker.target openqa-worker@1
615
```
616
617
* Test with an openQA job cloned from a production job, e.g. for openqaworker7
618
619
```
620
openqa-clone-job --within-instance https://openqa.opensuse.org/t${id} WORKER_CLASS=openqaworker7
621
```
622
623
* After the latest openQA job could successfully finish enable more worker instances
624
625
```
626
systemctl unmask openqa-worker@{2..14} && systemctl enable --now openqa-worker@{2..14}
627
```
628
629
* Monitor if nightly update works, e.g. look for journal entry:
630
631
```
632
Mar 01 00:08:26 openqaworker7 transactional-update[10933]: Calling zypper up
633
634
Mar 01 00:08:51 openqaworker7 transactional-update[10933]: transactional-update finished - informed rebootmgr
635
Mar 01 00:08:51 openqaworker7 systemd[1]: Started Update the system.
636
637
Mar 01 03:30:00 openqaworker7 rebootmgrd[40760]: rebootmgr: reboot triggered now!
638
639
Mar 01 03:36:32 openqaworker7 systemd[1]: Reached target openQA Worker.
640
```
641 93 okurz
642 95 okurz
## Distribution upgrades
643
644 101 okurz
Consider using https://github.com/okurz/auto-upgrade/blob/master/auto-upgrade or manual:
645
646 95 okurz
```
647 98 livdywan
new_version=15.2 # Specify the target release
648 1 alarrosa
649 98 livdywan
# Change the release via the special $releasever
650 1 alarrosa
. /etc/os-release
651
sed -i -e "s/${VERSION_ID}/\$releasever/g" /etc/zypp/repos.d/*
652
zypper --releasever=$new_version ref
653
test -f /etc/openqa/openqa.ini && sudo -u geekotest /opt/openqa-scripts/dump-psql
654
zypper -n --releasever=$new_version dup --auto-agree-with-licenses --replacefiles --download-in-advance
655
656
# Check config files for relevant changes
657 95 okurz
rpmconfigcheck
658
for i in $(cat /var/adm/rpmconfigcheck) ; do vimdiff ${i%.rpm*} $i ; done
659
rm $(cat /var/adm/rpmconfigcheck)
660
661 98 livdywan
reboot
662
systemctl --failed
663 1 alarrosa
```
664 97 okurz
665 109 okurz
## Remote management with IPMI
666 95 okurz
667 119 livdywan
o3 and osd worker machines are controllable over IPMI from within the SUSE network, see [openqa/workerconf.sls](https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls) for the commands.
668
It is recommended to use [shell aliases](https://gitlab.suse.de/openqa/salt-pillars-openqa#get-ipmi-definition-aliases) for convenience.
669 109 okurz
670
`ipmitool` can sometimes behave unreliably. It seems (to okurz) as if ipmitool version ipmitool-1.8.18+git20200916.1245aaa387dc from openSUSE Tumbleweed or Factory or the "systemsmanagement" OBS repo is more reliable than the version supplied with openSUSE Leap 15.2 (See #80544#note-14) and given a stable internet connection it is certainly possible to have a consistent serial console experience.
671
672 110 okurz
To ensure that remotely controlled machines power on automatically after a power loss ensure to set the power restory policy to "previous", especially for new machines. Using https://gitlab.suse.de/openqa/salt-pillars-openqa/#get-ipmi-definition-aliases :
673
674
```
675
IFS=$'\n'; for i in $(sed 's/^alias .*="\(.*\)"/\1/' ~/.openqa_ipmi_aliases); do eval "$i" chassis policy previous; done
676
```
677
678 109 okurz
## openQA infrastructure needs (o3 + osd)
679
680 115 okurz
TL;DR: new OSD ARM workers needed, missing redundancy for o3-ppc, rest is needing replacement as nearly all current hardware is out of vendor provided maintenance (as of 2021-05), SSD storage for o3 would be good
681 93 okurz
682
2020-03: SUSE IT (EngInfra) provided us more space for O3 but we have only slow rotating-disk storage. Performance could be improved by providing SSD storage.
683
684
The most time and effort we currently struggle with storage space for OSD (openqa.suse.de) ~~both OSD (openqa.suse.de) as well as O3 (openqa.opensuse.org) (2020-03: Situation on o3 resolved with more storage provided by SUSE IT)~~. Both instances (OSD + O3) are using precious netapp-storage but there is currently no better approach to use different, external storage. An increase of the available space would be appreciated, ~~o3 being more important right now than osd,~~ see https://progress.opensuse.org/issues/57494 for details. Graphs like 
685
https://stats.openqa-monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=1578343509900&to=1578653794173&fullscreen&panelId=12 show how usual test backlogs are worked on within OSD by architecture. It can be seen that both the ppc64le and aarch64 backlogs are reduced fast so we do not need more ppc64le or aarch64 machines. However, we have a stability problem with all three aarch64 workers. Potentially new machine(s) could help, see https://progress.opensuse.org/issues/41882 for details.
686 107 okurz
687 117 okurz
## Setup guide for new machines
688
689
* Make sure to set /etc/salt/minion_id to the FQDN (see #90875#note-2 for reference)
690
* Add to salt using https://gitlab.suse.de/openqa/salt-states-openqa
691
692 120 okurz
## Take machines out of salt-controlled production
693
694
E.g. for investigation or development or manual maintenance work
695 118 okurz
696
```
697
ssh osd "sudo salt-key -y -d $hostname"
698
ssh $hostname "sudo systemctl disable --now telegraf openqa-worker-auto-restart@\*"
699
```
700
701
## Bring back machines into production
702
703
```
704 124 dheidler
ssh osd "sudo salt-key -a $hostname && sudo salt --state-output=changes $hostname state.apply"
705 118 okurz
```
706
707
Depending on your actions further manual cleanup might be necessary, e.g. `ssh $hostname "sudo systemctl unmask telegraf salt-minion"`
708 117 okurz
709 122 okurz
## Backup
710
711
Automatic backup for the o3 webui host introduced with https://gitlab.suse.de/okurz/backup-server-salt/tree/master/rsnapshot covering so far /etc and the SQL database dumps.
712
713 109 okurz
## Best practices for infrastructure work
714 107 okurz
715
* Same as in OSD deployment we should look for failed grafana alerts if users report something suspicious
716
* Collect all the information between "last good" and "first bad" and then also find the git diff in openqa/salt-states-openqa
717
* Apply proper "scientific method" with written down hypotheses, experiments and conclusions in tickets, follow https://progress.opensuse.org/projects/openqav3/wiki#Further-decision-steps-working-on-test-issues
718
* Keep salt states to describe what should *not* be there
719
* Try out older btrfs snapshots in systems for crosschecking and boot with disabled salt. In the kernel cmdline append `systemd.mask=salt-minion.service`
720
* Team should conduct a work backlog check on a daily base, e.g. look for urgent tickets related to infrastructure problems
721 116 okurz
* Test reboot stability of machines with commands like in https://progress.opensuse.org/issues/78010#note-31 e.g.
722
723
```
724
for run in {01..30}; do for host in $host; do echo -n "run: $run, $host: ping .. " && timeout -k 5 600 sh -c "until ping -c30 $host >/dev/null; do :; done" && echo -n "ok, ssh .. " && timeout -k 5 600 sh -c "until nc -z -w 1 $host 22; do :; done" && echo -n "ok, uptime/reboot: " && ssh $host "uptime && sudo reboot" && sleep 120 || break; done || break; done
725
```