Project

General

Profile

action #109319

[qe-core] aarch64 tests failing in qemu-img due to broken image (was: "with cache error") size:S

Added by martinsmac 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2022-03-31
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test show cache error in aarch64 tests.

https://openqa.suse.de/tests/8445867

Acceptance criteria

  • AC1: behavior of conflicting assets is documented

Test suite description

Reproducible

Fails since (at least) Build 20220331-1

Further details

Tests failing when finish and try copy results/disk:
https://openqa.suse.de/tests/8445867

Likely error from autoinst-log.txt:
[2022-03-31T13:31:38.644100+02:00] [info] [pid:76743] +++ setup notes +++
[2022-03-31T13:31:38.644688+02:00] [info] [pid:76743] Running on openqaworker-arm-3:16 (Linux 5.3.18-150300.59.60-default #1 SMP Fri Mar 18 18:37:08 UTC 2022 (79e1683) aarch64)
[2022-03-31T13:31:38.661507+02:00] [debug] [pid:76743] Found HDD_1, caching SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2
[2022-03-31T13:31:38.672284+02:00] [info] [pid:76743] Downloading SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2, request #211 sent to Cache Service
[2022-03-31T13:31:48.914663+02:00] [info] [pid:76743] Download of SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2 processed:
[info] [#211]
Cache size of "/var/lib/openqa/cache" is 46 GiB, with limit 50 GiB
[info] [#211]
Downloading "SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2" from "http://openqa.suse.de/tests/8445867/asset/hdd/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2"
[info] [#211]
Content of "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2" has not changed, updating last use

[2022-03-31T13:31:48.930541+02:00] [debug] [pid:76743] Found ISO, caching SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso
[2022-03-31T13:31:48.942152+02:00] [info] [pid:76743] Downloading SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso, request #212 sent to Cache Service
[2022-03-31T13:32:29.661289+02:00] [info] [pid:76743] Download of SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso processed:
[info] [#207]
Cache size of "/var/lib/openqa/cache" is 45 GiB, with limit 50 GiB
[info] [#207]
Downloading "SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso" from "http://openqa.suse.de/tests/8445832/asset/iso/SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso"
[info] [#207]
Cache size 46 GiB + needed 7.8 GiB exceeds limit of 50 GiB, purging least used assets
[info] [#207]
Purging "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP1-aarch64-GM-gnome-allpatterns.qcow2" because we need space for new assets, reclaiming 3.4 GiB
[info] [#207]
Purging "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP1-aarch64-GM-SDK-gnome-allpatterns.qcow2" because we need space for new assets, reclaiming 2.8 GiB
[info] [#207]
Size of "/var/lib/openqa/cache/openqa.suse.de/SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso" is 7.8 GiB, with ETag ""1f0fbf000-5a7d39678f200""
[info] [#207]
Download of "/var/lib/openqa/cache/openqa.suse.de/SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso" successful (55 MiB/s), new cache size is 47 GiB

[2022-03-31T13:32:29.678158+02:00] [debug] [pid:76743] Found UEFI_PFLASH_VARS, caching SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2
[2022-03-31T13:32:29.695262+02:00] [info] [pid:76743] Downloading SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2, request #213 sent to Cache Service
[2022-03-31T13:32:34.862055+02:00] [info] [pid:76743] Download of SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2 processed:
[info] [#213]
Cache size of "/var/lib/openqa/cache" is 47 GiB, with limit 50 GiB
[info] [#213]
Downloading "SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2" from "http://openqa.suse.de/tests/8445867/asset/hdd/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2"
[info] [#213]
Size of "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2" is 323 KiB, with ETag ""50c00-5db7fbe069dde""
[info] [#213]
Download of "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2" successful (2.3 MiB/s), new cache size is 47 GiB

[2022-03-31T13:32:34.875713+02:00] [info] [pid:76743] Rsync from 'rsync://openqa.suse.de/tests' to '/var/lib/openqa/cache/openqa.suse.de', request #215 sent to Cache Service
[2022-03-31T13:32:55.245782+02:00] [info] [pid:76743] Output of rsync:
[info] [#215] Calling: rsync -avHP --timeout 1800 rsync://openqa.suse.de/tests/ --delete /var/lib/openqa/cache/openqa.suse.de/tests/
Try 1:
receiving incremental file list

Suggestions

  • put into our documentation that multiple jobs that try to publish the same image are not supported
    • use a more specific PUBLISH_HDD variable
  • have openQA detect that two assets are conflicting
    • guestimating to be a task of 2-3 days, involving a new database column

Related issues

Related to openQA Project - action #106898: Protection against asset clobberingResolved2022-02-16

Related to openQA Project - action #112742: [tools] aarch64 - qemu-img: /var/lib/openqa/pool/14/raid/hd0-overlay0: Image is not in qcow2 formatNew2022-06-20

History

#1 Updated by okurz 3 months ago

  • Related to action #106898: Protection against asset clobbering added

#2 Updated by okurz 3 months ago

  • Subject changed from [qe-core] aarch64 tests failing with cache error to [qe-core] aarch64 tests failing in qemu-img due to broken image (was: "with cache error")
  • Category set to Feature requests
  • Target version set to Ready

error message:

runcmd '/usr/bin/qemu-img create -f qcow2 -F qcow2 -b /var/lib/openqa/pool/16/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2 /var/lib/openqa/pool/16/raid/hd0-overlay0 949551104' failed with exit code 1: 'qemu-img: /var/lib/openqa/pool/16/raid/hd0-overlay0: Image is not in qcow2 format

Likely what we need to do is explicitly put into our documentation that multiple jobs that try to publish the same image are not supported. Also see #106898 about that

#3 Updated by cdywan 3 months ago

  • Subject changed from [qe-core] aarch64 tests failing in qemu-img due to broken image (was: "with cache error") to [qe-core] aarch64 tests failing in qemu-img due to broken image (was: "with cache error") size:S
  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by mkittler 3 months ago

  • Assignee set to mkittler

#5 Updated by mkittler 3 months ago

  • Status changed from Workable to Feedback

#6 Updated by mkittler 3 months ago

PR to actually avoid the corruption: https://github.com/os-autoinst/openQA/pull/4597

#7 Updated by mkittler 3 months ago

  • Status changed from Feedback to Resolved

Both PR have been merged and deployed so I'm resolving the issue.

Note that the last PR can only help to prevent broken images if moving the file is atomic which is something we cannot actually guarantee in general. So it makes still sense to warn about it in the documentation. (I suppose the filesystems we use in our production instances should make it atomic and we also avoid cross-filesystem moves.)

#8 Updated by okurz 7 days ago

  • Related to action #112742: [tools] aarch64 - qemu-img: /var/lib/openqa/pool/14/raid/hd0-overlay0: Image is not in qcow2 format added

Also available in: Atom PDF