QA » openQA Project » openQA Tests

I tried to add some wait time:

diff --git a/tests/installation/bootloader_zkvm.pm b/tests/installation/bootloader_zkvm.pm
index 76c5a5c8b..b4ce9446f 100644
--- a/tests/installation/bootloader_zkvm.pm
+++ b/tests/installation/bootloader_zkvm.pm
@@ -50,7 +50,7 @@ sub set_svirt_domain_elements {

         # show this on screen and make sure that kernel and initrd are actually saved
         enter_cmd "wget $repo/boot/s390x/initrd -O $zkvm_img_path/$name.initrd";
-        assert_screen "initrd-saved";
+        assert_screen "initrd-saved", 90;
         enter_cmd "wget $repo/boot/s390x/linux -O $zkvm_img_path/$name.kernel";
         assert_screen "kernel-saved";
     }

However, it seems later tests were still slow https://openqa.suse.de/tests/9121775#step/bootloader_zkvm/37

fwiw: manual installation on s390zp18 works just fine and without any noticeable performance impacts

so the issue looks to me like that the SUT is either not sending data over serial to openQA, or that openQA cannot receive this data.
manually the serial output openQA is waiting for is present.

It also affects SLE Micro 5.3 Alpha 2 release, as we can't do validation on this arch.

From my side, after playing around it looks like there is significant delay to get the serial output, the most success I have had was with this job, latest successful result is this one.
It does look like something goes through sometimes, but I really cannot determine any factor that affected the success rate.
Anyway I ll check a bit more tomorrow, hoping that it fixes itself.

https://openqa.suse.de/tests/9125700/logfile?filename=autoinst-log.txt

The issue is still there.

[2022-07-14T03:26:50.910940+02:00] [debug] no match: 6.9s, best candidate: bootloader_zkvm-initrd-saved-20160607 (0.00)
[2022-07-14T03:26:51.981234+02:00] [debug] no match: 5.9s, best candidate: bootloader_zkvm-initrd-saved-20160607 (0.00)
[2022-07-14T03:26:52.913105+02:00] [debug] no match: 4.9s, best candidate: bootloader_zkvm-initrd-saved-20160607 (0.00)
[2022-07-14T03:26:53.914215+02:00] [debug] no match: 3.9s, best candidate: bootloader_zkvm-initrd-saved-20160607 (0.00)
[2022-07-14T03:26:54.915226+02:00] [debug] no match: 2.9s, best candidate: bootloader_zkvm-initrd-saved-20160607 (0.00)
[2022-07-14T03:26:55.916294+02:00] [debug] no match: 1.9s, best candidate: bootloader_zkvm-initrd-saved-20160607 (0.00)
[2022-07-14T03:26:56.986691+02:00] [debug] no match: 0.9s, best candidate: bootloader_zkvm-initrd-saved-20160607 (0.00)
[2022-07-14T03:26:57.987640+02:00] [debug] no match: -0.1s, best candidate: bootloader_zkvm-initrd-saved-20160607 (0.00)
[2022-07-14T03:26:58.247167+02:00] [debug] >>> testapi::_check_backend_response: match=initrd-saved timed out after 30 (assert_screen)
[2022-07-14T03:26:58.314293+02:00] [info] ::: basetest::runtest: # Test died: no candidate needle with tag(s) 'initrd-saved' matched
[2022-07-14T03:26:58.315794+02:00] [debug] tests/installation/bootloader_zkvm.pm:96 called testapi::select_console
[2022-07-14T03:26:58.315920+02:00] [debug] <<< testapi::select_console(testapi_console="svirt")
[2022-07-14T03:26:58.321021+02:00] [debug] Establishing VNC connection to localhost:47135
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":35331"
      after 28809 requests (28652 known processed) with 0 events remaining.
xterm: fatal IO error 11 (Resource temporarily unavailable) or KillClient on X server ":35331"
[2022-07-14T03:26:58.333696+02:00] [info] ::: backend::driver::__ANON__: Driver backend collected unknown process with pid 491857 and exit status: 1
[2022-07-14T03:26:58.334038+02:00] [info] ::: backend::driver::__ANON__: Driver backend collected unknown process with pid 491866 and exit status: 0
[2022-07-14T03:26:58.334248+02:00] [info] ::: backend::driver::__ANON__: Driver backend collected unknown process with pid 491859 and exit status: 84

https://openqa.suse.de/tests/9126241/logfile?filename=autoinst-log.txt

somebody changed the machine definition of: s390x-kvm-sle12

Settings key '_now_in_workerconf_NETDEV' is invalid at /usr/lib/os-autoinst/bmwqemu.pm line 90.
    bmwqemu::load_vars() called at /usr/bin/isotovideo line 395
 at /usr/lib/os-autoinst/bmwqemu.pm line 313.
    bmwqemu::tiedvars::STORE(bmwqemu::tiedvars=HASH(0x1003d7bbe78), "_now_in_workerconf_NETDEV", "eth2") called at /usr/lib/os-autoinst/bmwqemu.pm line 90
    bmwqemu::load_vars() called at /usr/bin/isotovideo line 395

but that wasnt the issue

to add my findings here about the apparently lost serial output which I can reproduce locally

following the approach openQA does to grab the serial output I ran ssh -t root@s390zp18 virsh console openQA-SUT-7-113528
When I send data to the SUT via echo "foobar" > /dev/pts/<pty> it will appear on the ssh'ed serial output.
This works fine when I do it from my notebook or a worker outside the .qa.suse.de subnet. However, from grenache-1 and other machines in the qa subnet, I observed that the output doesn't get through or is delayed.

furthermore I noticed that my ssh connection to grenache-1 is hanging from time to time so I did some investigation for network performance.

mtr showed that there seems to be another issue in network performance between 10.162.x.x and 10.161.x.x (which is the s390 network) and is most likely is related to the serial output redirection (cause it is done over ssh)

for grenache-1 to s390zp[18,19] I got package losses of about 50%
for grenache-1 to z13router it was about 20%
for openqaworker2.suse.de or my notebook to the 10.161.x.x subnet, everything was fine
furthermore I noticed that qanet.qa.suse.de showed 100% packageloss to 10.161.x.x

I pinged Gerhard and he is also looking into this and I will create an enginfra ticket later on if we cannot solve it in a timely manner

jira-sd tracking ticket: https://sd.suse.com/servicedesk/customer/portal/1/SD-92689

@mgriessmeier do you think something in https://monitor.qa.suse.de/d/WDgrenache-1/worker-dashboard-grenache-1?orgId=1&refresh=1m&viewPanel=65099 should show this packet loss as well?

okurz wrote:

@mgriessmeier do you think something in https://monitor.qa.suse.de/d/WDgrenache-1/worker-dashboard-grenache-1?orgId=1&refresh=1m&viewPanel=65099 should show this packet loss as well?

@okurz, actually it does I just realized, it's barely seeable when all graphs are selected, but when you select only zp18 and zp19 you can see it pretty clearly - very interesting though that s390zp17 and s390zp11 are not showing it....

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/422 has been merged, let's see how everything responds: http://stats.openqa-monitor.qa.suse.de/goto/WwiTxWg4z?orgId=1

[2022-07-15T12:27:30.327108+02:00] [debug] [run_ssh_cmd(qemu-img create '/var/lib/libvirt/images/openQA-SUT-7a.img' -f qcow2 -b '/var/lib/libvirt/images//SLES-15-SP4-s390x-minimal_installed_for_LTP.qcow2' 42949672960)] stderr:
  qemu-img: /var/lib/libvirt/images/openQA-SUT-7a.img: Failed to get "write" lock
  Is another process using the image [/var/lib/libvirt/images/openQA-SUT-7a.img]?

https://openqa.suse.de/tests/9132833/logfile?filename=autoinst-log.txt now let's see where is there another worker using the same VIRSH_INSTANCE number

szarate wrote:

Matthias was using the instance 7 on the machine that failed... we can call this done :), will leave in feedback until Monday

Please allow some time for people to review merge requests in salt-pillars. https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/422 was created by you, approved by you, merged by you. Can be done in case of emergencies but then I suggest to not call this ticket done without waiting for feedback from others for a reasonable time. I suggest to keep this ticket open until workers are back on grenache-1 or a different solution has been found. But right now we have multiple worker instances on grenache-1 with no useful worker class. Of course we need to understand and fix first the underlying issue, that is https://sd.suse.com/servicedesk/customer/portal/1/SD-92689, right?

okurz wrote:

szarate wrote:

Matthias was using the instance 7 on the machine that failed... we can call this done :), will leave in feedback until Monday

Please allow some time for people to review merge requests in salt-pillars. https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/422 was created by you, approved by you, merged by you. Can be done in case of emergencies but then I suggest to not call this ticket done without waiting for feedback from others for a reasonable time.

I'll keep it open until the end of the sprint.

I suggest to keep this ticket open until workers are back on grenache-1 or a different solution has been found.

I take moving the workers to openQA worker 2 is a good solution, if you disagree, create a ticket to move the workers back to grenache :)

But right now we have multiple worker instances on grenache-1 with no useful worker class.

When creating the ticket above, mention that the openQA worker distribution has to be improved and rebalanced in the worker config too.

Of course we need to understand and fix first the underlying issue, that is https://sd.suse.com/servicedesk/customer/portal/1/SD-92689, right?

I guess that's what #107062 is for

Please allow some time for people to review merge requests in salt-pillars. https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/422 was created by you, approved by you, merged by you. Can be done in case of emergencies but then I suggest to not call this ticket done without waiting for feedback from others for a reasonable time.

Sure, people had plenty of time to review in such urgent situation :), another story would be if I merged right after creating the PR.

szarate wrote:

okurz wrote:

szarate wrote:

Matthias was using the instance 7 on the machine that failed... we can call this done :), will leave in feedback until Monday

Please allow some time for people to review merge requests in salt-pillars. https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/422 was created by you, approved by you, merged by you. Can be done in case of emergencies but then I suggest to not call this ticket done without waiting for feedback from others for a reasonable time.

I'll keep it open until the end of the sprint.

Well, I meant the merge request. I assume with "it" you mean this ticket. Anyway, see further below regarding any followup work.

I suggest to keep this ticket open until workers are back on grenache-1 or a different solution has been found.

I take moving the workers to openQA worker 2 is a good solution, if you disagree, create a ticket to move the workers back to grenache :)

Sure. I think it's not a good idea to spread out the rollback actions to separate tickets as it's still the same original issue and moving to another worker is a temporary workaround but I created a ticket if it helps your planning: #113701

But right now we have multiple worker instances on grenache-1 with no useful worker class.

When creating the ticket above, mention that the openQA worker distribution has to be improved and rebalanced in the worker config too.

Done, included with #113701#note-2

Of course we need to understand and fix first the underlying issue, that is https://sd.suse.com/servicedesk/customer/portal/1/SD-92689, right?

I guess that's what #107062 is for

I don't mind if you want to use #107062 for tracking or this ticket or the newly created #113701, please just make sure that https://sd.suse.com/servicedesk/customer/portal/1/SD-92689 is tracked and as soon as resolved #113701 is followed up with.