Project

General

Profile

action #18956

[sle][virtualization][x11regressions] unused tests 'tests/virtualization' -> load_virtualization_tests() in products/sle/main.pm is never called

Added by okurz about 3 years ago. Updated over 1 year ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
New test
Target version:
-
Start date:
2017-05-04
Due date:
% Done:

0%

Estimated time:
Difficulty:
Duration:

Description

Observation

I found

sub load_virtualization_tests() {
    # standalone suite to fit needed installation
    if (get_var("STANDALONEVT")) {
        loadtest "virtualization/boot";
        loadtest "virtualization/installation";
        loadtest "virtualization/prepare_sle12";
    }
    loadtest "virtualization/yast_virtualization";
    loadtest "virtualization/virt_install";
    loadtest "virtualization/virt_top";
    loadtest "virtualization/virtman_install";
    loadtest "virtualization/virtman_view";
    loadtest "virtualization/virtman_storage";
    loadtest "virtualization/virtman_virtualnet";
    loadtest "virtualization/virtman_networkinterface";
    loadtest "virtualization/virtman_create_guest";
}

in products/sle/main.pm. Apparently this function is never called. I suggest to either check these modules and make use of them or just delete the corresponding code.


Related issues

Related to openQA Tests - action #15132: [saga][epic] Better structure of test plans in main.pmBlocked2018-11-202020-11-27

History

#1 Updated by okurz about 3 years ago

  • Subject changed from [sle][virtualization] unused tests -> load_virtualization_tests() in products/sle/main.pm is never called to [sle][virtualization][x11regressions] unused tests 'tests/virtualization' -> load_virtualization_tests() in products/sle/main.pm is never called

If I read it right there is a whole tree tests/virtualization which is unused except for

        loadtest "virtualization/yast_virtualization";
        loadtest "virtualization/virtman_view";

which are called as part of load_x11regression_other. Results available e.g. sle-12-SP3-Desktop-DVD-x86_64-Build0212-regression-other@64bit or latest

#2 Updated by xlai about 3 years ago

This is not written or used by us. You can use git blame to see who wrote that.

#3 Updated by okurz about 3 years ago

Yeah, sure, I know :-) The code has effectively not been changed since 1 year at least. The last real change was

commit 31acc529
Author: Chingkai <qkzhu@suse.com>
Date:   Fri May 20 12:50:59 2016 +0800

    modify&add yast_virtualization virtman_view to run in regression_others group

but the other changes in before were by "Antoine Giniès (aginies)".

xlai: IMHO we should make sure the test code we have is either used or deleted to not give a false impression. Do you agree?

aginies: It looks like the code contributions you did were never used. Do you have a proposal how to proceed?

#4 Updated by aginies about 3 years ago

Yeah that's really sad that this test suite was never used since it was developed and functional 2 years ago.... It was usable under SLE12GA and SP1. It covers a lot of virt-manager testing and some double check with virsh command. I had spent a lot of time writing this, and i was not able to get this merge for a while due to the continuous moving target of openQA. I was using it under my own openQA test suite. I am not a QA engineer. My initial work was to provide more testing for virtualization using the incoming openQA platform. So i do not know what should be done now, the code is there, it's a working code (or it was: see https://w3.suse.de/~aginies/videos/sle12_virt-manager.ogv), it is not perfect (like other testing suite there is some workaround). Due to the poor interest of external contribuation, i now consider any participation or any kind of development for openQA as something in very low priority, to not say /dev/null, for me an my teams, and i do not recommends any contribution to openQA until a clear and acceptable process to contribute is in place. People outside QA don't have days or weeks to work on QA testing, they have spare time for doing this job, which means this is not possible to be uptodate on everything in openQA.
regards.

#5 Updated by xlai about 3 years ago

okurz wrote:

Yeah, sure, I know :-) The code has effectively not been changed since 1 year at least. The last real change was

commit 31acc529
Author: Chingkai <qkzhu@suse.com>
Date:   Fri May 20 12:50:59 2016 +0800

    modify&add yast_virtualization virtman_view to run in regression_others group

but the other changes in before were by "Antoine Giniès (aginies)".

xlai: IMHO we should make sure the test code we have is either used or deleted to not give a false impression. Do you agree?

Sure, it is best to be used since we already spend a lot of efforts on it.

Besides 'used' and 'deleted', maybe there can be a third option to temporarily keep it when the writer can not decide whether it should be abandoned or used finally. Of course after some time, finally, the writer should make a hard decision. If not useful, then delete it, or just start to use it, by themselves. Your monitor script for unused code can be a quite useful reminder tool to push the writer to a decision. My humble opinion though ~

aginies: It looks like the code contributions you did were never used. Do you have a proposal how to proceed?

#6 Updated by okurz about 3 years ago

aginies wrote:

Due to the poor interest of external contribuation

Thank you for your honest opinion although I am not sure why you are saying this. I myself appreciate very much any contribution.

People outside QA don't have days or weeks to work on QA testing…

My opinion about this is that openQA tests are system tests which should be shared responsibility among whole SUSE R&D and openSUSE community and part of product development. They are not (only) "QA testing".

What can I do to make you feel different about it?

#7 Updated by aginies about 3 years ago

okurz wrote:

aginies wrote:

Due to the poor interest of external contribuation
Thank you for your honest opinion although I am not sure why you are saying this. I myself appreciate very much any contribution.

It's just based on fact, a working testsuite submitted 2 years ago not integrated in any test plan, this is not what we can call interest if external contribution. And this test suite is covering most of virt-manager option, is not just launching virt-manager. This is what we call a "cold shower".

People outside QA don't have days or weeks to work on QA testing…

My opinion about this is that openQA tests are system tests which should be shared responsibility among whole SUSE R&D and openSUSE community and part of product development. They are not (only) "QA testing".

Based on my past experience and the poor reliability of the openQA's API and openQA platform 2 years ago, it was very hard to get something working, ie: the "famous" main.pm file code was changed every day. External people can not spent time checking any changes on the openQA platform and adjust their code or whatever else. This is too much time consuming. People in charge if openQA should adjust the code of running test suite or at least inform of change contributors. The slogan was more "we break everything, find where and adapt", not really a SUSE slogan.

What can I do to make you feel different about it?

Currently there is no rules, no process, too much hidden stuff (quite complex to understand where to grab needles for SLE, then testsuite on github, you must have your own platform to develop your test suite, etc....). Welcome to the jungle. Even internally seems this is not so easy for QA to communicate and collaborate, ie: all my features testing for HA SP2 were lost in space for SP3, and still not merge in the official QA platform.

#9 Updated by okurz almost 3 years ago

Took me a very long time to come up with answers to your previous comment as you are expressing quite some heavy statements.

aginies wrote:

okurz wrote:

aginies wrote:

Due to the poor interest of external contribuation
Thank you for your honest opinion although I am not sure why you are saying this. I myself appreciate very much any contribution.

It's just based on fact, a working testsuite submitted 2 years ago not integrated in any test plan, this is not what we can call interest if external contribution. And this test suite is covering most of virt-manager option, is not just launching virt-manager. This is what we call a "cold shower".

2 years ago I was not yet with the company and also not yet a contributor to the openSUSE/SLE openQA tests. But I know for sure a lot has changed since then. I can only guess that the missing enablement of your test modules has just gone unnoticed but I know people assumed virtualization tests to be in place. Just providing the code and then not following on with making sure the tests are triggered (or even enabled in the test schedule) just does not work out because we have such a broad test coverage already.

[…]

What can I do to make you feel different about it?

Currently there is no rules, no process, too much hidden stuff (quite complex to understand where to grab needles for SLE, then testsuite on github, you must have your own platform to develop your test suite, etc....). Welcome to the jungle. Even internally seems this is not so easy for QA to communicate and collaborate

What you are saying is 1) not true, we do have rules and a process. Maybe it was very much based on personal communication 2 years ago but still present. Nowadays that is more written down, e.g. see https://github.com/os-autoinst/os-autoinst-distri-opensuse/#how-to-contribute and https://progress.opensuse.org/projects/openqatests/wiki/#Organisational . Our Definition of DONE also includes one important point: "At least one successful test run has been observed on osd or o3 and referenced in the corresponding progress item or bugzilla bug report if one exists" which should have prevented test modules going in but then never used.
And 2) your statement is quite negative and not helping to improve the situation.

Besides, the openSUSE distribution release managers and release engineers (therefore not only QA) are both happy users and contributors to both openQA as well as according tests. For SLE apparently openQA is a major part of the development workflow actively involving the RMs. The SLE setup is of course one step harder than openSUSE tests but still written down: https://wiki.microfocus.net/index.php/OpenQA#Installation . Also I know of many helpful people within the SUSE internal IRC channel [#testing](irc://irc.suse.de/testing) as well as people in person offering help.

[…], ie: all my features testing for HA SP2 were lost in space for SP3, and still not merge in the official QA platform.

I don't know about these changes. But if you point me to them I will make sure they are handled accordingly or provide helpful feedback. You might have mentioned this just as an example but my offer is a generic one. Btw, HA testing is now fully in responsibility of the QA CSS departement.

#10 Updated by aginies over 2 years ago

This testsuite will have been useful for SLE15. currently we are facing multiple bug with virt-manager, any progress on the integration? Is it something we plan to have? As a reminder virt-manager is the only graphical tools we have under SLE, so this is an important tool. How we can move forward on that front?

#11 Updated by okurz over 2 years ago

  • Category changed from Enhancement to existing tests to New test
  • Status changed from New to In Progress
  • Assignee set to okurz

See https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4037 for my weekend project to get some parts of it already running for openSUSE :-)

#12 Updated by okurz over 2 years ago

https://w3.suse.de/~aginies/openQA/sle12_virt-manager.ogv for reference for the old test contribution.

#13 Updated by okurz over 2 years ago

  • Description updated (diff)

https://openqa.opensuse.org/tests/558348 is the test passed in the current state for openSUSE Tumbleweed. Added a new test suite "virtualization" and added to "openSUSE Tumbleweed" job group. Next step: Test for Leap.

#14 Updated by okurz over 2 years ago

  • Assignee changed from okurz to xlai

xlai, Calen: regarding "virtualization" tests I don't know if you have followed the discussion on research@suse.de and this ticket. I now have successfully enabled part of the tests for openSUSE Tumbleweed as a "weekend fun project". Maybe you want to take over now?

#15 Updated by okurz over 1 year ago

  • Related to action #15132: [saga][epic] Better structure of test plans in main.pm added

#16 Updated by xlai over 1 year ago

Currently we do not have enough time to take it, will look into it when time available.

#17 Updated by cachen over 1 year ago

First all, Tumbleweed isn't SLE-Virtualization test target, we may not able to contribute there unless in our free time,

Secondly, openQA Virtualization module isn't owned by SLE-Virtualization, which is using for GUI tool virt-manager, which is perhaps Desktop team is still interested(or they were involved in this module with Antoine, I add Yifan in the loop), since for SLES such as libvirt xl qemu hypervisor kernel areas are in our highly concern, virt-manager is in very low priority. However, we can take a look whether anything can be quickly fixed and be reused if time available.

Also available in: Atom PDF