Actions
action #109319
closed[qe-core] aarch64 tests failing in qemu-img due to broken image (was: "with cache error") size:S
Start date:
2022-03-31
Due date:
% Done:
0%
Estimated time:
Tags:
Description
Observation¶
openQA test show cache error in aarch64 tests.
https://openqa.suse.de/tests/8445867
Acceptance criteria¶
- AC1: behavior of conflicting assets is documented
Test suite description¶
Reproducible¶
Fails since (at least) Build 20220331-1
Further details¶
Tests failing when finish and try copy results/disk:
https://openqa.suse.de/tests/8445867
Likely error from autoinst-log.txt:
[2022-03-31T13:31:38.644100+02:00] [info] [pid:76743] +++ setup notes +++
[2022-03-31T13:31:38.644688+02:00] [info] [pid:76743] Running on openqaworker-arm-3:16 (Linux 5.3.18-150300.59.60-default #1 SMP Fri Mar 18 18:37:08 UTC 2022 (79e1683) aarch64)
[2022-03-31T13:31:38.661507+02:00] [debug] [pid:76743] Found HDD_1, caching SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2
[2022-03-31T13:31:38.672284+02:00] [info] [pid:76743] Downloading SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2, request #211 sent to Cache Service
[2022-03-31T13:31:48.914663+02:00] [info] [pid:76743] Download of SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2 processed:
[info] [#211]
Cache size of "/var/lib/openqa/cache" is 46 GiB, with limit 50 GiB
[info] [#211]
Downloading "SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2" from "http://openqa.suse.de/tests/8445867/asset/hdd/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2"
[info] [#211]
Content of "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2" has not changed, updating last use
[2022-03-31T13:31:48.930541+02:00] [debug] [pid:76743] Found ISO, caching SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso
[2022-03-31T13:31:48.942152+02:00] [info] [pid:76743] Downloading SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso, request #212 sent to Cache Service
[2022-03-31T13:32:29.661289+02:00] [info] [pid:76743] Download of SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso processed:
[info] [#207]
Cache size of "/var/lib/openqa/cache" is 45 GiB, with limit 50 GiB
[info] [#207]
Downloading "SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso" from "http://openqa.suse.de/tests/8445832/asset/iso/SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso"
[info] [#207]
Cache size 46 GiB + needed 7.8 GiB exceeds limit of 50 GiB, purging least used assets
[info] [#207]
Purging "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP1-aarch64-GM-gnome-allpatterns.qcow2" because we need space for new assets, reclaiming 3.4 GiB
[info] [#207]
Purging "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP1-aarch64-GM-SDK-gnome-allpatterns.qcow2" because we need space for new assets, reclaiming 2.8 GiB
[info] [#207]
Size of "/var/lib/openqa/cache/openqa.suse.de/SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso" is 7.8 GiB, with ETag ""1f0fbf000-5a7d39678f200""
[info] [#207]
Download of "/var/lib/openqa/cache/openqa.suse.de/SLE-15-SP2-Installer-DVD-aarch64-GM-DVD1.iso" successful (55 MiB/s), new cache size is 47 GiB
[2022-03-31T13:32:29.678158+02:00] [debug] [pid:76743] Found UEFI_PFLASH_VARS, caching SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2
[2022-03-31T13:32:29.695262+02:00] [info] [pid:76743] Downloading SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2, request #213 sent to Cache Service
[2022-03-31T13:32:34.862055+02:00] [info] [pid:76743] Download of SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2 processed:
[info] [#213]
Cache size of "/var/lib/openqa/cache" is 47 GiB, with limit 50 GiB
[info] [#213]
Downloading "SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2" from "http://openqa.suse.de/tests/8445867/asset/hdd/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2"
[info] [#213]
Size of "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2" is 323 KiB, with ETag ""50c00-5db7fbe069dde""
[info] [#213]
Download of "/var/lib/openqa/cache/openqa.suse.de/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio-uefi-vars.qcow2" successful (2.3 MiB/s), new cache size is 47 GiB
[2022-03-31T13:32:34.875713+02:00] [info] [pid:76743] Rsync from 'rsync://openqa.suse.de/tests' to '/var/lib/openqa/cache/openqa.suse.de', request #215 sent to Cache Service
[2022-03-31T13:32:55.245782+02:00] [info] [pid:76743] Output of rsync:
[info] [#215] Calling: rsync -avHP --timeout 1800 rsync://openqa.suse.de/tests/ --delete /var/lib/openqa/cache/openqa.suse.de/tests/
Try 1:
receiving incremental file list
Suggestions¶
- put into our documentation that multiple jobs that try to publish the same image are not supported
- use a more specific PUBLISH_HDD variable
- have openQA detect that two assets are conflicting
- guestimating to be a task of 2-3 days, involving a new database column
Updated by okurz about 2 years ago
- Related to action #106898: Protection against asset clobbering added
Updated by okurz about 2 years ago
- Subject changed from [qe-core] aarch64 tests failing with cache error to [qe-core] aarch64 tests failing in qemu-img due to broken image (was: "with cache error")
- Category set to Feature requests
- Target version set to Ready
error message:
runcmd '/usr/bin/qemu-img create -f qcow2 -F qcow2 -b /var/lib/openqa/pool/16/SLES-15-SP2-aarch64-mru-install-minimal-with-addons-Build20220331-1-Server-DVD-Updates-aarch64-virtio.qcow2 /var/lib/openqa/pool/16/raid/hd0-overlay0 949551104' failed with exit code 1: 'qemu-img: /var/lib/openqa/pool/16/raid/hd0-overlay0: Image is not in qcow2 format
Likely what we need to do is explicitly put into our documentation that multiple jobs that try to publish the same image are not supported. Also see #106898 about that
Updated by livdywan about 2 years ago
- Subject changed from [qe-core] aarch64 tests failing in qemu-img due to broken image (was: "with cache error") to [qe-core] aarch64 tests failing in qemu-img due to broken image (was: "with cache error") size:S
- Description updated (diff)
- Status changed from New to Workable
Updated by mkittler about 2 years ago
- Status changed from Workable to Feedback
Updated by mkittler about 2 years ago
PR to actually avoid the corruption: https://github.com/os-autoinst/openQA/pull/4597
Updated by mkittler about 2 years ago
- Status changed from Feedback to Resolved
Both PR have been merged and deployed so I'm resolving the issue.
Note that the last PR can only help to prevent broken images if moving the file is atomic which is something we cannot actually guarantee in general. So it makes still sense to warn about it in the documentation. (I suppose the filesystems we use in our production instances should make it atomic and we also avoid cross-filesystem moves.)
Updated by okurz almost 2 years ago
- Related to action #112742: [tools] aarch64 - qemu-img: /var/lib/openqa/pool/14/raid/hd0-overlay0: Image is not in qcow2 format added
Actions