QA » openQA Project

I can reproduce this issue 90 % of times on openqaw6-kvm.qa.suse.de which runs SLES 12 SP2 with updates. However I can't reproduce it on my local machine which runs Leap 42.2 with updates. That's one difference, the other is remote v. local host -- I tried to mitigate impact of timing but of not avail. Clearly the behavior differs as SLES presents 720x400 px resolution in BIOS, but Leap does 800x600 px.

I suggest to migrate the machine, as well as openqaw5-xen.qa.suse.de, to what we use for openQA workers, i.e. Leap, without the openqa-worker started there.

Then we would have on svirt backend the same configuration we have for qemu backend, as there are SUTs started on Leap as well, if I understand it correctly.

Actually it's reproducible on Leap 42.2. as well. Even if I update Qemu to 2.8.0 (the latest). I believe it's a Qemu bug, reported as bsc#1016968. Strangely I am unable to reproduce it with Leap 42.1 JeOS as a guest VM.

IMHO the "virtualization" team should test virtualization from both perspective of hypervisor host as well as guest. The current "virtualization" job group concentrates more on the hypervisor, the svirt backend is much more suitable for testing the guest as SUT under the control of different hypervisors. As this ticket here refers to the latter while taking the backend as a testing prerequisite I agree with you that the infrastructure should be more harmonized and the worker host should have the same installation as the other worker hosts. But aren't the tests running on openqaworker2 which is at least as of today Leap 42.2?

okurz wrote:

But aren't the tests running on openqaworker2 which is at least as of today Leap 42.2?

I don't speak about openQA worker, but the virtualization hosts openqaw5-xen.qa.suse.de and openqaw6-kvm.qa.suse.de we use as intermediaries. They are the VM hosts where VMs via svirt backend are actually started, i.e. they are not started on the openQA worker itself as it is with qemu backend.

I see, can you document the purpose and setup of these hosts on https://wiki.microfocus.net/index.php/OpenQA#Hardware accordingly? or is it documented already somewhere else? then please reference the according documentation.

I don't really know what we can do here. The socket is blocking, so a read of 0 means EOF. All we can try is reconnecting

coolo wrote:

I don't really know what we can do here. The socket is blocking, so a read of 0 means EOF. All we can try is reconnecting

Yes, that would mitigate the bug in Qemu (https://bugzilla.suse.com/show_bug.cgi?id=1016968).

Be careful though as for zKVM raw is broken and zlre works :(

michalnowak wrote:

I can reproduce this issue 90 % of times on openqaw6-kvm.qa.suse.de which runs SLES 12 SP2 with updates. However I can't reproduce it on my local machine which runs Leap 42.2 with updates. That's one difference, the other is remote v. local host -- I tried to mitigate impact of timing but of not avail. Clearly the behavior differs as SLES presents 720x400 px resolution in BIOS, but Leap does 800x600 px.

I suggest to migrate the machine, as well as openqaw5-xen.qa.suse.de, to what we use for openQA workers, i.e. Leap, without the openqa-worker started there.

Then we would have on svirt backend the same configuration we have for qemu backend, as there are SUTs started on Leap as well, if I understand it correctly.

If the bug is with SLES 12 SP2, what is the bug ID for the SLES 12 SP2 bug?

I am okay considering the migration to Leap but that is a MAJOR request with huge impacts, we should at least consider you know, fixing the issue in SLE 12 SP2 as people who pay us actually use it..

recent example on sle-12-SP3-JeOS-for-kvm-and-xen-x86_64-Build5.5-textmode+scc_sles@svirt-xen-hvm

Disabled ZRLE on svirt but it did not help either (not to mention other problems), it's just triggered elsewhere:

DIE unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.pm line 918.

 at /usr/lib/os-autoinst/backend/baseclass.pm line 73.
    backend::baseclass::die_handler('unexpected end of data at /usr/lib/os-autoinst/consoles/VNC.p...') called at /usr/lib/os-autoinst/consoles/VNC.pm line 918
    consoles::VNC::_receive_update('consoles::VNC=HASH(0x621cdf0)') called at /usr/lib/os-autoinst/consoles/VNC.pm line 880
    consoles::VNC::_receive_message('consoles::VNC=HASH(0x621cdf0)') called at /usr/lib/os-autoinst/consoles/VNC.pm line 782
    consoles::VNC::update_framebuffer('consoles::VNC=HASH(0x621cdf0)') called at /usr/lib/os-autoinst/consoles/vnc_base.pm line 68
    consoles::vnc_base::request_screen_update('consoles::vnc_base=HASH(0x3b29350)', undef) called at /usr/lib/os-autoinst/backend/baseclass.pm line 521
    backend::baseclass::bouncer('backend::svirt=HASH(0x4830760)', 'request_screen_update', undef) called at /usr/lib/os-autoinst/backend/baseclass.pm line 504
    backend::baseclass::request_screen_update('backend::svirt=HASH(0x4830760)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 167
    eval {...} called at /usr/lib/os-autoinst/backend/baseclass.pm line 151
    backend::baseclass::run_capture_loop('backend::svirt=HASH(0x4830760)') called at /usr/lib/os-autoinst/backend/baseclass.pm line 122
    backend::baseclass::run('backend::svirt=HASH(0x4830760)', 6, 9) called at /usr/lib/os-autoinst/backend/driver.pm line 85
    backend::driver::start('backend::driver=HASH(0x48af0f8)') called at /usr/lib/os-autoinst/backend/driver.pm line 48
    backend::driver::new('backend::driver', 'svirt') called at /usr/bin/isotovideo line 198
    main::init_backend() called at /usr/bin/isotovideo line 263

http://assam.suse.cz/tests/5560/file/autoinst-log.txt

coolo wrote:

How would you recommend reproducing this easiest?

Clone any JeOS on Xen HVM e.g. https://openqa.suse.de/tests/858635, make sure all VIRSH_* variables are set, set VIRSH_INSTANCE to higher number than 1 not to collude with OSD jobs. On worker file "HDD_1" has to exist, e.g. as a non-zero sized file, but a image actually used is downloaded on the VM host automatically.

It should fail in GRUB, either in bootloader_uefi, redefine_svirt_domain or console_reboot test module.