QA » openQA Project » openQA Tests

I'm going to try to collect what we are doing in openQA group for YaST to establish some dependencies (you probably have done this exercise before but I haven't had the opportunity yet):

• Check installation media
• Add additional add-on product via ftp and http:
  • HA module: It has a first boot but does not check that the module is added. Nevertheless, what about not finishing installation on those scenarios and combine them? We could focus in the rest of options besides http and ftp and not care about the installation as the other products already care about it, for example.
  • Development Tools module (dud?): via ftp, it looks kind of the same thing that we are doing we HA in the previous paragraph.
• Patterns:
  • Add all modules (not all but most important?) and add all patterns.
  • Minimal base, for un-selecting all the patterns we could have used same approach than https://openqa.suse.de/tests/2426569#step/select_patterns_and_packages/8 instead of iterate. We are doing in this test a lot of stuff that is not reflected in the description, is really needed?
• Expert Partitioner:
  • Setup different RAID configurations but we don't verify them after installation.
  • Warnings for btrfs.
  • LVM + RAID
• Guided Setup:
  • Explicit btrfs for each arch.
  • Explicit ext4 for each arch.
  • Explicit xfs for several archs (I think both previous are most used, perhaps we could just have a x64_86 autoyast test for xfs)
  • Enabling crypt partition.
  • Enabling crypt partition in LVM.
  • Select the right disk
• Usb install. Does it make sense to install with btrfs in this test?
• Test yast2_clone_system module: here we have a dependency with create_hdd_gnome. Besides that, are we really using to reinstall from that profile?
• Detect YaST2 failures in a simple installation.
• gnome installation (http, self-signed https and smb)
• gpt: gpt it is default so I guess we could include this one in some other default that we have.
• installer extended: multiple extra checks.
• icsi client and server: here we have a dependency with create_hdd_gnome
• role minimal
• multipath hw
• nvme hw
• nis client & server: here we have a dependency with create_hdd_gnome
• release notes: perhaps we could put all together in extended_tests.
• remote controllers (ssh, vnc)
• repo_inst: Are not we covering this in gnome_http?
• rt product
• Skip registration: it also contains after installation some tests that I don't think are related?
• Switch keyboard: probably another candidate to unify in extended_tests or not even that, after changing back the language to EN-US we could stop execution.
• Yast2 gui: here we have a dependency with create_hdd_gnome
• Yast2 ncurses: here we have a dependency with create_hdd_gnome
• Yast2 ui devel: I don't get the description but here we have a dependency with create_hdd_minimal_base+sdk -> but I think we could avoid it in some way.
• Hostname checks
• Check disable self-udpate
• ZFCP storage.
• AutoYaST tests, multiple tests.

According to previous analysis I would try to think about question rised in this ticket, taking into account that we should "cook for ourself" and reduce this dependencies in a way that we should not be bottle-neck for anybody or vice versa.

Well, on one hand it might be good for each job group with the backing team to have their own "image creation job" to provide each team more flexibility. However in reality it is quite likely that either a regression introduced by QSF-y or a YaST regression is causing the installation to fail. And every team in parallel investigating what is wrong is quite an effort. Why the ticket is not in the backlog of QSF-u? Well, because it is the QSF-y team that wanted "split out", see parent ticket.

What hasn't been mentioned yet as an alternative but might reduce more dependency on the image creating jobs within openQA would be to create "appliance images" with kiwi inside OBS directly.

Just to clarify, it doesn't matter where these test suites currently are as it depends on what is their purpose. I agree with statements @okurz to use those to publish images, which is the purpose of the test suite rather than testing installation, so no one is blocked from their testing. This will also allow us to completely change the approach there, e.g. by not using installer at all, but kiwi instead.

I agree, I could see that we are already doing a lot of testing of installation in yast subgroup so using images with kiwi sounds good to me. So if I am understanding, kiwi is able to create all those different images with different modules selected, etc. using customization.

https://build.opensuse.org/project/show/devel:kanku:images# is an example where Frank Schreiner (M0ses) is creating quite some images already. We could either use these images or collaborate with him to move them to a more standard place or do it similar to that approach on our own or ask RMs if they are interested to ship as products or at least build in sync with their products and we use it for testing.