QA (public) » openQA Project (public) » openQA Tests (public) » Containers and images

Created MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1716

MERGED, see new template https://openqa.suse.de/admin/job_templates/566

Last build https://openqa.suse.de/tests/overview?distri=sle-micro&version=6.0&build=X.21.8&groupid=566
all scheduled tests pass.

https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1758

Created MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1763,
for all products/flavors in checklist.

MR 1763 Merged.

Some logic-errors fixed in new MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1766,
also MERGED.

See tests in last build of https://openqa.suse.de/group_overview/572

Today a new error affected Base-VMware-Updates tests SL Micro 6.0 Product Increments - Containers, failed in boot phase, due to SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk image located in hdd/fixed.

See analysis here below:

Findings on VMware boot error, by matching the autoinst logs of A Vs B casaes:

(A) boot PASS:

https://openqa.suse.de/tests/15036824/logfile?filename=autoinst-log.txt

run_ssh_cmd(if test -e /vmfs/volumes/datastore1/openQA/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk; then while lsof | grep 'cp.*SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk'; do echo File SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk is being copied by other process, sleeping for 60 seconds; sleep 60;done;else cp /vmfs/volumes/openqa/hdd/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk /vmfs/volumes/datastore1/openQA/;fi;)] exit-code: 0

and

(B) boot FAIL:

https://openqa.suse.de/tests/15040545/logfile?filename=autoinst-log.txt

run_ssh_cmd(if test -e /vmfs/volumes/Datastore2/openQA/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk; then while lsof | grep 'cp.*SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk'; do echo File SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk is being copied by other process, sleeping for 60 seconds; sleep 60;done;else cp /vmfs/volumes/openqa/hdd/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk /vmfs/volumes/Datastore2/openQA/;fi;)] stderr:
  cp: can't stat '/vmfs/volumes/openqa/hdd/SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk': No such file or directory
... exit-code: 1

The .vmdk file exists in ./hdd/fixed, for both A and B case,
but

• for B, SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk not found in VMWARE_DATASTORE by the script, else triggered cp from ./hdd/, but no file here and fails;

• for A instead, that script seems to find already .vmdk in VMWARE_DATASTORE, probably for previous works, so no (wrong) cp executed and errror skipped.

Now, the previous cleanup should have to remove image .vmdk for A, but expression doesn't match it, expecting some more string like "openQA-SUT" appended, therefore basename SL-Micro.x86_64-6.0-Base-VMware-GM.vmdk file not removed from VMWARE_DATASTORE. This cleanup runs before above if.

Cleanup for B also ran, but some files resulted locked, so did not delete those. This could be the logic behind above A .vmdk already present.

Proposed solution, by priority:

1) in _copy_image_vmware, either cp should fallback to hdd/fixed when no file found in hdd OR $file_basename should contain the path including hdd/fixed.

2) Cleanup should remove also file_basename, if none using it:

   ...
   rm -f ${vmware_openqa_datastore}*${name}* \
      ${vmware_openqa_datastore}*${file_basename}

3) Clarify/normalize images management in test code, always for both hdd and hdd/fixed.

For item n.1 above, created PR2524, last Aug 4.

Activities temporary paused will be resumed soon.

Activity restarted:

noted in the os_autoinst repo code flow, the add_disk($self, $args) routine, calling the _copy_image_to_vm_host($args,...), provides in the @args also original image full-path-file (eventually placed in subfolder like fixed/), coming from bootloader_svirt.pm call (see my $hddpath); but then that full-path is never used and even lost.

In fact, when internally calling _copy_image_vmware(...,$file_basename,...), the only basename is extracted and passed as parameter, but there the original image path is hard-coded and partially re-calculated, without any subfolder management.
Therefore images in hdd/ (or iso/) subolders are not correctly managed in copy commands.

In last PR 2524 update, replaced in _copy_image_to_vm_host() and inner-called _copy_image_vmware() the file_basename input parameter with the full path source file, passed by add_disk $args, coming (only) from bootloader_svirt.pm settings at runtime (or similar bootloader_zkvm.pm): this way we allow management of images also in subfolders like fixed/, avoiding the unneeded original folder recalculation.

All tests in SL Micro 6.0 Product Increments - Containers, and other groups too, are actually all failing, affected by a IBS repo renaming issue, causing install_updates to fail.

Poo opened: https://progress.opensuse.org/issues/165536, but issue managed in the named Jira ticket.

Moreover in last builds, a not-yet-clear behavior in such tests let the bootloader_svirt.pm step pass ok, even being the original image placed in the not-managed hdd/fixed/ subdirectory, that caused the error discussed in https://progress.opensuse.org/issues/162941#note-14. I.e. it could it be the image is also present into the destination folder already.

Recently resolved issue about IBS repo renaming, the tests in group 572 pass almost all, but 2 vmware tests still fail in podman netawark/skopeo/remote: for those issues I created poo https://progress.opensuse.org/issues/165884.

About hdd/fixed/ issue in os-autoinst, PR 2524 has been updated, all code fix reverted and simply introduced on-demand debugging in nfs datastore script,adding VMWARE_NFS_DATASTORE_DEBUG=1, to verify the image file status.

Today all VMware tests in grp/572 PASS.

In particular, using WORKER_CLASS unreal7 all pass ok; see slack.

But still the tests having WORKER qesapworker assigned, fail because of issues on the used server esxi7: a poo ticket for this issue has been opened by eng.team: https://progress.opensuse.org/issues/166529.

Suggested, as W.A. until fixed, to run VMware tests on workers using unreal7:
WORKER_CLASS=sapworker1,svirt-vmware70 or WORKER_CLASS=unreal7,svirt-vmware70 or WORKER_CLASS=unreal7.
I.e. https://openqa.suse.de/tests/15390100 pass.

At the moment all VMware tests in https://openqa.suse.de/group_overview/572 pass ok.

Please note that in actual VMware tests run the issue in note-14/B is no more present, because the image file results already present in the expected folder (here transferred by some unknown or manual operation), as also revealed cloning the test with VMWARE_NFS_DATASTORE_DEBUG=1, from PR 2524. See bash snippet in i.e. job 15400838

But this could mean that the existing local image is always used, because never cleaned , so that eventual new image update from builds are never tested.

A possible correction could be, in sequential changes:

1. ensure that the right full-path image is provided as origin in _copy_image_vmware, as proposed in PR https://github.com/os-autoinst/os-autoinst/pull/2542
2. Define a lock-file policy for these VMware images (in place of the lsof check), to prevent clean up when running test are using it.
3. cleanup in item n.2 of note-14 implemented.

Unschedule VMware for SLEM product increments, MR https://gitlab.suse.de/qac/qac-openqa-yaml/-/merge_requests/1842 MERGED