QA » openQA Project » openQA Tests

Take a HPC test as example:

use base 'hpcbase';
use base 'hpc::cluster';
use strict;
use warnings;
use testapi;
use utils;
use lockapi;

sub run {
my ($self) = @_;
$self->select_serial_terminal;

 # disable packagekitd
 quit_packagekit();

 # Stop firewall
 systemctl 'stop ' . $self->firewall;

 $self->provision_cluster();

 set_hostname(get_var('HOSTNAME', 'susetest'));

 if (get_var('HPC_REPO')) {
     my $repo = get_var('HPC_REPO');
     my $reponame = get_required_var('HPC_REPONAME');
     zypper_call("ar -f $repo $reponame");
     assert_script_run "zypper lr | grep $reponame";

     zypper_call("--gpg-auto-import-keys ref");
     zypper_call 'up';
 }

}

It is clear this is a function test and easy to use serial console, while migration test is a process so need transfer all modules in the process to be on serial console.

I will use the https://openqa.nue.suse.com/tests/8011693 as an example to switch register_system to serial console as the first step.

Let me illustrate the point I was trying to make in the quoted email:
Have a look at https://openqa.nue.suse.com/tests/8011687. It indicates a failure somewhere in 'install_service' which is part of the migration test workflow - it is still unrelated to the migration test itself, but part of preparation of the system which will later be used for migration.
To help with pinpointing the issue and to determine whether we indeed have a problem with a package or there is an issue with the test case, I need to determine what steps have been performed. Apparently, a set of packages gets installed. This is a CLI operation. I can see parts of the installation process in the captured output (see https://openqa.nue.suse.com/tests/8011687#step/install_service/11) but the command line hasn't been captured.
If I'm able to see all commands that have been issued, together with their stdout and strerr output, I would be able to determine the root cause much more quickly.
Alternatively, I will look into the link to the test script in the left column: (https://openqa.nue.suse.com/tests/8011687/modules/install_service/steps/1/src) - this however doesn't reveal the commands issued either.
I agree that not all of migration testing can be done that way: an YaST driven offline migration and a call to 'yast migration' will bring up UI elements that will have to be screen shotted.
A zypper migration is CLI, it prompts the user for input, but for pure migration testing this can be bypassed.

eeich wrote:

Let me illustrate the point I was trying to make in the quoted email:
Have a look at https://openqa.nue.suse.com/tests/8011687. It indicates a failure somewhere in 'install_service' which is part of the migration test workflow - it is still unrelated to the migration test itself, but part of preparation of the system which will later be used for migration.
To help with pinpointing the issue and to determine whether we indeed have a problem with a package or there is an issue with the test case, I need to determine what steps have been performed. Apparently, a set of packages gets installed. This is a CLI operation. I can see parts of the installation process in the captured output (see https://openqa.nue.suse.com/tests/8011687#step/install_service/11) but the command line hasn't been captured.
If I'm able to see all commands that have been issued, together with their stdout and strerr output, I would be able to determine the root cause much more quickly.

I will try to change the install_service to serial console for you to have a look, wait https://openqa.nue.suse.com/tests/8033195#step/install_service/74.

This may need more work since if the service check module each select own console will cause more code change.

Alternatively, I will look into the link to the test script in the left column: (https://openqa.nue.suse.com/tests/8011687/modules/install_service/steps/1/src) - this however doesn't reveal the commands issued either.

For service check, the real code located in each service module and service_check.pm is just the entrance and controller so you can't see the related code directly.

I agree that not all of migration testing can be done that way: an YaST driven offline migration and a call to 'yast migration' will bring up UI elements that will have to be screen shotted.
A zypper migration is CLI, it prompts the user for input, but for pure migration testing this can be bypassed.

Yes, I think such CLI test should be able to transfer to serial console.

@eeich Hi, Egbert, could you have a look on the install_service on serial console? Is this what you expected?
https://openqa.nue.suse.com/tests/8033195#step/install_service/1

The previous log already removed, I run a new log for the ticket, https://openqa.nue.suse.com/tests/8445129#step/install_service/1

@Egbert, please have a look on the latest test results, if you think such run on serial console as you expected, I can set a setting for migration on SLEHPC to make these modules such as zypper migration to run on serial console.

leli wrote:

@Egbert, please have a look on the latest test results, if you think such run on serial console as you expected, I can set a setting for migration on SLEHPC to make these modules such as zypper migration to run on serial console.

Sorry for missing this!
I had to look for a while what has changed and compare it to the original issue. I can see the first steps - up to https://openqa.nue.suse.com/tests/8445129#step/install_service/47 converted to pure text output. This should have the added benefit of allowing the text to be used for result checking, no need to update the needles when the console font has changed.
Does this also capture output to stderr?

eeich wrote:

leli wrote:

@Egbert, please have a look on the latest test results, if you think such run on serial console as you expected, I can set a setting for migration on SLEHPC to make these modules such as zypper migration to run on serial console.

Sorry for missing this!
I had to look for a while what has changed and compare it to the original issue. I can see the first steps - up to https://openqa.nue.suse.com/tests/8445129#step/install_service/47 converted to pure text output. This should have the added benefit of allowing the text to be used for result checking, no need to update the needles when the console font has changed.
Ok, so I think we need arrange special migration test for HPC, zypper migration via cli cmd and no needle needed so will be a good choice. I will set a setting to control such action to select serial console for the test on such scenario for SLE HPC.
Does this also capture output to stderr?
Yes, it will but I haven't capture such failure on serial console since each module may change the console and the failure is random. Anyway, we will see it later.
https://openqa.nue.suse.com/tests/8479486#step/install_service/373

Thank you! This looks a lot better already. What I'm missing, though is what the command that fails has printed to stdout and stderr. Usually zypper is rather verbose about the reason of a failure. So this output - together with the command line - are important to understand what's going on. Of course, the last lines of the zypper logs are relevant as well, however, in this case, they do not seem to be related to the issue at hand.
If you could also capture the command output to stdout and stderr (ideally flagged what is what), it would be great!
We should have this for all command executions. A wrapper that takes care of collecting this output would be a great feature.

Wait http://openqa.nue.suse.com/tests/8727887#live to get some debug info for
my @pkginstall = split('\n', script_output q[zypper search -r SLE-Module-HPC] . $version . q[-Pool -r SLE-Module-HPC] . $version . q[-Updates | cut -d '|' -f 2 | sed -e 's/ //g' | grep -E '.-hpc.' | grep -vE 'system|module|suse' | grep -vE '.:digit:+:digit:+.gnu|.:digit:+:digit:+.-hpc' | grep -vE '.-static$' | grep -vE '.hpc-macros.'], proceed_on_failure => 1, timeout => 180);

@egbert, I have updated the code to make cmd and output shown, please have a look on http://openqa.nue.suse.com/tests/8750202#step/install_service/38 and http://openqa.nue.suse.com/tests/8750202#step/install_service/42

leli wrote:

@egbert, I have updated the code to make cmd and output shown, please have a look on http://openqa.nue.suse.com/tests/8750202#step/install_service/38 and http://openqa.nue.suse.com/tests/8750202#step/install_service/42

Great, this provides a lot of useful information which had been missing before, thank you!