QA » openQA Project » openQA Tests » qe-yam

There is a check if the SUT is shutdown

os-autoinst/backend/svirt.pm:        $rsp = $self->run_ssh_cmd("! virsh $libvirt_connector dominfo $vmname | grep -w 'shut off'");

This check can fail when the XEN hypervisor starts a new VM from a unrelated openQA test with the same name (ex. openQA-SUT-5) immediately after the one we are testing is shutdown.

A possible solutions

1. Check it not by $vmname but by the xen ID from the virsh dominfo. The ID is different after the VM is started again.
2. Check the VM uptime if it is less then ~30 seconds if it is not shutdown with xl uptime $vmname

I think 1. is better, currently trying to implement it.

My theory is that "shutdown" check for VMs can fail because we are checking if a VM with name "X" is shutdown. Before the check is done a different openQA test can start on the hypervisor and run a VM with the same name.

I am trying to get the VM by ID and check if it is shutdown by id.
But this logic is in os-autoinst.git and I am a little lost, and it seems that this change is better suited for tools-team.

Two files of interest are
os-autoinst/backend/svirt.pm sub is_shutdown
and
os-autoinst/consoles/sshVirtsh.pm #define the new domain

My theory is that "shutdown" check for VMs can fail because we are checking if a VM with name "X" is shutdown. Before the check is done a different openQA test can start on the hypervisor and run a VM with the same name.

Can such a name reuse actually happen? The name should be machine and worker number specific.

I am not sure how VIRSH_INSTANCE is define, but when I login to a hypervisor it is usually a single digit number (openQA-SUT-*) and diffrent from the VM ID

unreal6:~ # virsh list
 Id   Name           State
------------------------------
 0    Domain-0       running
 3    openQA-SUT-8   running
 4    openQA-SUT-5   running

When openQA is busy with a lot of tests vm openQA-SUT-8 can be shut down, and a different test on the same hypervisor can start a VM with the same name openQA-SUT-8

It happens rarely but this is failing the shutdown check, what I have notice manually monitoring is that the new VM can have the same name openQA-SUT-8, but will have a different ID

The os-autoinst doc says:

If you add multiple instances, be sure to assign a different `VIRSH_INSTANCE`
and `VIRSH_MAC`. For more details about the variables, checkout
[backend_vars.asciidoc](backend_vars.asciidoc).