QA » openQA Project » openQA Tests » Containers

Created MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1716

MERGED, see new template https://openqa.suse.de/admin/job_templates/566

Last build https://openqa.suse.de/tests/overview?distri=sle-micro&version=6.0&build=X.21.8&groupid=566
all scheduled tests pass.

https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1758

Created MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1763,
for all products/flavors in checklist.

MR 1763 Merged.

Some logic-errors fixed in new MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1766,
also MERGED.

See tests in last build of https://openqa.suse.de/group_overview/572

Today a new error affected Base-VMware-Updates tests SL Micro 6.0 Product Increments - Containers, failed in boot phase, due to SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk image located in hdd/fixed.

See analysis here below:

Findings on VMware boot error, by matching the autoinst logs of A Vs B casaes:

(A) boot PASS:

https://openqa.suse.de/tests/15036824/logfile?filename=autoinst-log.txt

run_ssh_cmd(if test -e /vmfs/volumes/datastore1/openQA/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk; then while lsof | grep 'cp.*SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk'; do echo File SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk is being copied by other process, sleeping for 60 seconds; sleep 60;done;else cp /vmfs/volumes/openqa/hdd/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk /vmfs/volumes/datastore1/openQA/;fi;)] exit-code: 0

and

(B) boot FAIL:

https://openqa.suse.de/tests/15040545/logfile?filename=autoinst-log.txt

run_ssh_cmd(if test -e /vmfs/volumes/Datastore2/openQA/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk; then while lsof | grep 'cp.*SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk'; do echo File SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk is being copied by other process, sleeping for 60 seconds; sleep 60;done;else cp /vmfs/volumes/openqa/hdd/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk /vmfs/volumes/Datastore2/openQA/;fi;)] stderr:
  cp: can't stat '/vmfs/volumes/openqa/hdd/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk': No such file or directory
... exit-code: 1

The .vmdk file exists in ./hdd/fixed, for both A and B case,
but

• for B, SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk not found in VMWARE_DATASTORE by the script, else triggered cp from ./hdd/, but no file here and fails;

• for A instead, that script seems to find already .vmdk in VMWARE_DATASTORE, probably for previous works, so no (wrong) cp executed and errror skipped.

Now, the previous cleanup should have to remove image .vmdk for A, but expression doesn't match it, expecting some more string like "openQA-SUT" appended, therefore basename SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk file not removed from VMWARE_DATASTORE. This cleanup runs before above if.

Cleanup for B also ran, but some files resulted locked, so did not delete those. This could be the logic behind above A .vmdk already present.

Proposed solution, by priority:

1) in _copy_image_vmware, cp should fallback to hdd/fixed when no file found in hdd.

2) Cleanup should remove also ${file_basename}:

   ...
   rm -f ${vmware_openqa_datastore}*${name}* \\
      ${vmware_openqa_datastore}*${file_basename}

3) Clarify/normalize images management in test code, always for both hdd and hdd/fixed.

For item n.1 above, created PR2524, last Aug 4.

Activities temporary paused will be resumed soon.

Activity restarted:

noted in the os_autoinst repo code flow, the add_disk($self, $args) routine, calling the _copy_image_to_vm_host($args,...), provides in the @args also original image full-path-file (eventually placed in subfolder like fixed/), coming from bootloader_svirt.pm call (see my $hddpath); but then that full-path is never used and even lost.

In fact, when internally calling _copy_image_vmware(...,$file_basename,...), the only basename is extracted and passed as parameter, but there the original image path is hard-coded and partially re-calculated, without any subfolder management.
Therefore images in hdd/ (or iso/) subolders are not correctly managed in copy commands.

In last PR 2524 update, replaced in _copy_image_to_vm_host() and inner-called _copy_image_vmware() the file_basename input parameter with the full path source file, passed by add_disk $args, coming (only) from bootloader_svirt.pm settings at runtime (or similar bootloader_zkvm.pm): this way we allow management of images also in subfolders like fixed/, avoiding the unneeded original folder recalculation.

All tests in SL Micro 6.0 Product Increments - Containers, and other groups too, are actually all failing, affected by a IBS repo renaming issue, causing install_updates to fail.

Poo opened: https://progress.opensuse.org/issues/165536, but issue managed in the named Jira ticket.

Moreover in last builds, a not-yet-clear behavior in such tests let the bootloader_svirt.pm step pass ok, even being the original image placed in the not-managed hdd/fixed/ subdirectory, that caused the error discussed in https://progress.opensuse.org/issues/162941#note-14. I.e. it could it be the image is also present into the destination folder already.

Recently resolved issue about IBS repo renaming, the tests in group 572 pass almost all, but 2 vmware tests still fail in podman netawark/skopeo/remote: for those issues I created poo https://progress.opensuse.org/issues/165884.

About hdd/fixed/ issue in os-autoinst, PR 2524 has been updated, all code fix reverted and simply introduced on-demand debugging in nfs datastore script, to verify the image file status.