QA » openQA Project » openQA Tests

https://openqa.suse.de/tests/995815#step/addon_products_sle/11 failed because the job is within in the test development group with a more restricted quota and the worker is very busy so the job started late, the sdk was already deleted by gru cleanup because quota was exceeded. It is correct that the asset got deleted because when we have a scheduled job we never know if it will ever be triggered. So the image creation job should rather go into the sle functional group where I moved it now.

To have a good template for SLE 12SP4 as well as SLE15 I removed the scheduling from the test dev group and added it to SLE 12 SP3 functional: https://openqa.suse.de/admin/job_templates/55 . We won't have a new build triggered there but it serves as a template to mark the jobs that are supposed to work.

The jobs you referenced were obsoleted (don't know why, maybe sle15?) so I triggered new ones: https://openqa.suse.de/tests/1080197 , https://openqa.suse.de/tests/1080198, https://openqa.suse.de/tests/1080199 that might complete but will take some time. I will keep this ticket on feedback for now to monitor.

I will create another ticket to rename some scenarios: I don't think the "sles12_qa" prefix makes sense for test suites.

passed fs: https://openqa.suse.de/tests/1080198
passed sched: https://openqa.suse.de/tests/1080199

missing process: https://openqa.suse.de/t1080236, retriggered as https://openqa.suse.de/tests/1080236#step/acceptance_process_stress/24, failing to connect to the root terminal it seems.

@mgriessmeier any idea?

as agreed with riafarov we suggest to run these tests only on zkvm-images workerclass
It might work on others too if we recreate the image, but this is untested

we should improve the error handling in cases of select_console failing on zkvm, e.g. execute commands over the svirt connection and check for the existance of corresponding ssh connections for each console (and potentially xterm-console processes as well on the worker)

SLE15 scenario create_hdd_minimal_base+sdk is fine so we can use it. I added "create_hdd_minimal_base+sdk@s390x-kvm" to SLE 15 Functional and sched_stress@s390x-kvm, process_stress@s390x-kvm, fs_stress@s390x-kvm to the test development job group.

Triggered one test manually with

openqa_clone_job_osd --skip-deps 1186961 PUBLISH_HDD_1= INSTALLONLY= _GROUP="Test Development: SLE 15" BOOT_HDD_IMAGE=1 HDD_1=SLES-15-s390x-278.1-minimal_with_sdk278.1_installed.qcow2 MAX_JOB_TIME=7200 QA_HEAD_REPO=http://149.44.176.2/ibs/QA:/Head/SLE-15/ QA_TESTSET=acceptance_fs_stress VIRTIO_CONSOLE=0 TEST=fs_stress

-> https://openqa.suse.de/tests/1190743#live

failed because couldn't find a pty device?!?

@mgriessmeier and me have fixed initial issue: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3676 problem was that domain xml was not defined when we tried to grep for pty device.
Now fails to switch to root console: http://opeth.suse.de/tests/5709#step/acceptance_fs_stress/30

It looks booting the pre-created hdd works fine, but test failed in select_console('root-console') step. When test running in s390 zvm with svirt, what kind of console should be used?

qa_run.pm:
sub system_login {
my $self = shift;
$self->wait_boot;
if (get_var('VIRTIO_CONSOLE')) {
select_console('root-virtio-terminal');
}
else {
select_console('root-console');
}
}

09:04:51.5209 15757 activate_console, console: root-console, type: console
09:04:51.5210 15757 backend s390x || zkvm
09:04:51.5213 15757 <<< testapi::assert_screen(mustmatch='password-prompt', timeout=30)
09:04:51.5849 15759 MATCH(password-prompt-20170927:0.00)
09:04:51.5984 15759 MATCH(password-prompt-20150730:0.00)
09:04:51.6143 15759 MATCH(password-prompt-20141126:0.00)
09:04:51.9707 15759 MATCH(password-prompt-20160405:0.00)
09:04:51.9857 15759 MATCH(password-prompt-20160414:0.00)
09:04:52.0012 15759 MATCH(password-prompt-bsc965787-20160216:0.00)
09:04:52.0210 15759 MATCH(password-prompt-ipmi-20170619:0.00)
09:04:52.0367 15759 MATCH(password-prompt-ipmi-20170627:0.00)
09:04:52.0629 15759 MATCH(password-prompt-xterm-20150818:0.00)
09:04:52.0635 15759 WARNING: check_asserted_screen took 0.52 seconds - make your needles more specific
09:04:52.0639 15759 DEBUG_IO:

cachen wrote:

It looks booting the pre-created hdd works fine, but test failed in select_console('root-console') step. When test running in s390 zvm with svirt, what kind of console should be used?
[…]

Nothing is wrong about the console but there is a preparatory step missing. This is why I stated in #18936#note-27 that #19362 should come first by splitting qa_run and using the standard test modules to boot to a complete system in the proper way. Trying to patch the custom stress test code is IMHO not efficient.

okurz wrote:

cachen wrote:

It looks booting the pre-created hdd works fine, but test failed in select_console('root-console') step. When test running in s390 zvm with svirt, what kind of console should be used?
[…]

Nothing is wrong about the console but there is a preparatory step missing. This is why I stated in #18936#note-27 that #19362 should come first by splitting qa_run and using the standard test modules to boot to a complete system in the proper way. Trying to patch the custom stress test code is IMHO not efficient.

clearly, system_login is the first step of this testing from qa_run.pm, it looks need some specify setup for s390 zvm host connection in this first step, which can be defined in qa_run.pm.
Although #19362 to splitting workload is the task we would like to enhance, but I don't think that is the root cause. Now the key point is, we need to know how can connect and reuse the s390_created_hdd as testing host in openQA, let's say, what step missing before "select_console('root-console')", does it need reconnect_s390?

[...]
sub run {
my $self = shift;
$self->system_login();
$self->prepare_repos();
[...]

cachen wrote:

Now the key point is, we need to know how can connect and reuse the s390_created_hdd as testing host in openQA, let's say, what step missing before "select_console('root-console')", does it need reconnect_s390?

and to solve that question I proposed to work on #19362 first

okurz wrote:

cachen wrote:

Now the key point is, we need to know how can connect and reuse the s390_created_hdd as testing host in openQA, let's say, what step missing before "select_console('root-console')", does it need reconnect_s390?

and to solve that question I proposed to work on #19362 first

My understanding is #19362 splitting workload is an enhancement of the test structure, better for debugging/reviewing, but not fix this main issue. The urgent thing is to learn which step is missing, as for putting this step in qa_run.pm structure or in a single *.pm for calling, I would leave to tester to decide. #19362 was created many month ago, I want to enhance the structure, however Sero and Nathan were taking many other high priority on project testing, testsuite fixing, so, for those 'enhancement' task we have to put in lower priority.

If you don't mind, I will ask Nathan help to look at this issue with you together, since our other testsuite testing in 'Kernel''Userspace' groups will see the same problem when they be enabled on s390 arch. On the other hand, when Sero finish his high priority tasks, he will keep taking the splitting task. Is it fine to you? :)

https://openqa.suse.de/tests/1204159# fs_stress test run on s390x

that looks very promising.

wow, how can I express my big thanks to your and raifarov great activity and supporting in openQA, appreciated so much! We will follow up and learn more of openQA usage.

So https://openqa.suse.de/tests/1204159# for build 278.1 was fine but build 300.3 failed to boot: https://openqa.suse.de/tests/1210591#step/boot_to_desktop/20
ideas?

Enabled stress tests in functional job group for zkvm, as works fine there: https://openqa.suse.de/tests/1227017#previous

For x-kvm we need to add %MACHINE% to the created qcow2 image name. Workaround for shutdown is implemented.

With workaround fix for x-kvm, now we are able to create image using create_hdd_minimal_base+sdk job and trigger stress tests there. But before that we also need to add %MACHINE% variable into published image name, and in all chained jobs, so we use relevant qcow image. Which should be done here: https://progress.opensuse.org/issues/26022

Enabled qcow2 creation for zkvm and x-kvm, stress for x-kvm are enabled in dev group only atm to prove that they work.
Here is the runs on x-kvm:
https://openqa.suse.de/tests/1233848
https://openqa.suse.de/tests/1234671
https://openqa.suse.de/tests/1235017

wow, good one :-)