QA » openQA Project

All s390x tests (both zKVM and zVM) fail as incomplete.

I think the impact isn't that high. E.g. I recently restarted the s390x-zVM test https://openqa.suse.de/tests/2747431 to test whether another issue is fixed now and it passed.

But maybe it is actually the same issue as https://progress.opensuse.org/issues/49676 - even though the error message is different. So I restarted https://openqa.suse.de/tests/2748923 to see whether the fix we deployed yesterday afternoon fixes this as well.

Ok, it still fails. And it is happening on grenache-1 where the change which caused https://progress.opensuse.org/issues/49676 isn't even deployed. So I guess it is really a different issue.

Note that no deployment of os-autoinst happened between the last good and the first incomplete (or even the last incomplete). It appears the last deployment announced on Tuesday didn't include the workers which are relevant here. The os-autoinst version was always 4.5.1551283945.1b03daca according to the log.

Like I already mentioned, changes in os-autoinst can not make a difference here because the version hasn't changed. I also saw no relevant changes in the test distribution. So it must be something else.

The error basically says the VNC protocol handshake succeeded but the security handshake failed. When I read the code correctly, the password is not used at this point. So password being incorrectly is likely not the cause, too. Instead it seems that the server returns no "security types". The VNC standards has the following to say about this:

If number-of-security-types is zero,then for some reason the connection failed (e.g. the server can not support the desired protocol version). This is followed by a string describing the reason (where a string is specified as a length followed by that many ASCII characters):

Unfortunately os-autoinst does not read the reason so we don't know why it fails. We should likely implement reading the reason for the failure. Nevertheless the question is what might have changed on the server-side?

Interesting, maybe this? https://bugzilla.suse.com/show_bug.cgi?id=1129412
Wrong SLE version. Also the VNC is happening on the worker, not the SUT.

SLindoMansilla wrote:

Interesting, maybe this? https://bugzilla.suse.com/show_bug.cgi?id=1129412

Not sure if that one is duplicate of https://bugzilla.suse.com/show_bug.cgi?id=1129073 but that's for SLE 15, so should not affect SLE 12.

I don't find any zVM job failing on this. Could you provide a link?
I think only zKVM is affected.
I also think that is is sporadic, because not all zKVM failed.

(I think IRC doesn't work for me over VPN)

zVM affected job found: https://openqa.suse.de/tests/2745246

Let's see reproducibility:

• https://openqa.suse.de/tests/overview?build=poo49793_zkvm_round1
• https://openqa.suse.de/tests/overview?build=poo49793_zvm_round1

I tried to test my PR https://github.com/os-autoinst/os-autoinst/pull/1137 locally to see what error message the server gives us. However, it didn't work. In the first place because I hadn't set WORKER_HOSTNAME and then because grenache-1:13 was apparently re-enabled and interfered with my testing (I'm using its virsh config for my local tests).

@mgriessmeier tested a previous build and it worked. So it is likely really just the product bug which has already been mentioned: https://bugzilla.suse.com/show_bug.cgi?id=1129412

Of course the error handling on the os-autoinst-side should be better:

1. The error from the VNC server should be returned. That is what I'm attempting here: https://github.com/os-autoinst/os-autoinst/pull/1137
2. It should be a test failure and not an incomplete - at least if we actually can connect to the VNC server but it behaves unexpectedly (like here).

PR for getting rid of Can't locate object method "blessed" via package " in the error message: https://github.com/os-autoinst/os-autoinst/pull/1139

So, as we have suspected, it's a bug on s390x https://bugzilla.suse.com/show_bug.cgi?id=1131569 which breaks VNC connection.
@mkittler: Thanks for improving handling of such cases!