action #103542
open[qe-core] Kiwi tests for SLE 15+ failing
Added by vpelcak over 2 years ago. Updated about 2 months ago.
0%
Description
I noticed that SLE 15 and newer tests of Kiwi started to fail or are already failing for some time now.
https://openqa.suse.de/parent_group_overview/24#grouped_by_build
Maybe it should be considered to reorganize them in a way that we do not lose visibility on them.
Updated by tjyrinki_suse over 2 years ago
- Start date deleted (
2021-12-06)
Since they are old tests with no current maintainer, this ticket will need refining for background starting from what are kiwi images, and if the purpose of testing those in the way they used to be tested is still relevant in 2022.
For some references I can find, the tests were developed 3 years ago, pushed to os-autoinst-distri-opensuse with PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6673, closing ticket https://progress.opensuse.org/issues/43625 which did not explain much more than "kiwi" images are downloaded and run. There is also a kiwi team that already does some automated testing, test results of which are fetched by these tests.
What the current tests were doing when they were working (and still doing for older SLE iamges) was download image, login, and fetch the supposed test results from OBS. If the automated testing run by kiwi team already boot the built image, then this would not add much more than report the OBS status in openQA.
Updated by szarate over 2 years ago
I'm inclined to propose to drop the test, it literally is not doing anything that is not done already in OBS, other than checking status of a build?. In this case I sugest:
- Drop the test
- Reassign test so that mantainer can fix it
- If there's actual business value/business case for us to invest in testing Kiwi (building appliances from scratch) on a functional level, for every product, then ask virtualization team's PO if it is something they would consider testing, or do some kind of image validation, which would basically duplicate https://github.com/OSInside/kiwi-functional-tests
- Rewrite a simpler test that builds a released product, boots it and shuts it down (see above)
Updated by vpelcak over 2 years ago
This is kinda special test.
It is testing kiwi in a way, that the team developing the test uses that version of kiwi to build those images and we just test its basic functionality thus testing if kiwi works properly. If it passes, that concludes kiwi regression tests.
I reached out Jordi Massaguer Pla for info about the stuff if he is still the maintainer of it.
I do not insist on any particular solution as long as we do not lose the test coverage.
Updated by szarate over 2 years ago
vpelcak wrote:
This is kinda special test.
It is testing kiwi in a way, that the team developing the test uses that version of kiwi to build those images and we just test its basic functionality thus testing if kiwi works properly. If it passes, that concludes kiwi regression tests.
I reached out Jordi Massaguer Pla for info about the stuff if he is still the maintainer of it.
I do not insist on any particular solution as long as we do not lose the test coverage.
What makes it special? It looks from what I can understand, this job is being used as some sort of CI check.
So unless there's an actual reason to keep such Check and add resources to fix/mantain it, I still suggest to drop it, or that the team that uses it for kiwi regression tests, takes care of these.
If the failures block updates, please add a ticket for the test to be unscheduled.
Updated by vpelcak over 2 years ago
It is special because we are not testing kiwi itself directly but we're testing images built by kiwi instead.
Manual test of kiwi is rather tedious and time consuming. Having it automated makes the testing a breeze.
If you come up with a better way, I'm all for it. But dropping it without replacement is just offloading burden on Update Validation squad which then at least needs to be consulted.
Updated by favogt over 2 years ago
In a discussion about a kiwi maintenance update this topic came up, and I think this mail (or thread) explain what the tests are for: http://mailman.suse.de/pipermail/jeos-internal/2021-December/003618.html
Updated by slo-gin over 1 year ago
This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.
Updated by slo-gin about 2 months ago
This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.