Project

General

Profile

action #94997

Downloading compressed disk images is broken on o3

Added by dancermak 5 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2021-07-01
Due date:
2021-08-13
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

I have started running kiwi image tests on o3 and started to run into issues with HDD_1_DECOMPRESS_URL: when this variable is set, then the test will never succeed, because one of the following cases happens:

  1. The download just fails, see: https://openqa.opensuse.org/tests/1814637 or https://openqa.opensuse.org/tests/1814639
  2. The download "succeeds", but the resulting disk image is a zero byte fail and openQA can not boot from that. See for instance: https://openqa.opensuse.org/tests/1814632, https://openqa.opensuse.org/tests/1814631, https://openqa.opensuse.org/tests/1814633, https://openqa.opensuse.org/tests/1814634

This must be related to some settings on o3, because this works perfectly on my local machine.

Steps to reproduce

  • create a new job and set HDD_1_DECOMPRESS_URL=http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/kiwi-test-image-luks.x86_64.raw.xz
  • watch it fail to download the image

Problem

absolutely no clue, it works on my local openQA instance

Suggestion

maybe the background downloader is overloaded?

Workaround

none that I know off

History

#1 Updated by okurz 5 months ago

  • Project changed from openQA Infrastructure to openQA Project
  • Category set to Concrete Bugs
  • Target version set to Ready

might be specific to o3 but still something to improve in openQA in general to make it clearer to the user what's going on.

Can you please update the ticket according to https://progress.opensuse.org/projects/openqav3/wiki/#Defects ?

#2 Updated by okurz 5 months ago

can you please update the ticket description instead of adding the new description as comment?

Also, https://openqa.opensuse.org/minion/jobs?offset=0&task=download_asset might provide some insight

#3 Updated by dancermak 5 months ago

  • Description updated (diff)

#4 Updated by dancermak 5 months ago

okurz wrote:

can you please update the ticket description instead of adding the new description as comment?

Sorry, I thought that the Edit button edits the description…

Also, https://openqa.opensuse.org/minion/jobs?offset=0&task=download_asset might provide some insight

Not really. This one here downloaded a zero bytes file: https://openqa.opensuse.org/minion/jobs?id=656141 and this one https://openqa.opensuse.org/minion/jobs?id=656140 failed due to "too many download errors" but I really don't see any additional information on that page.

#5 Updated by okurz 5 months ago

From o3 grep 'Extracting.*failed' /var/log/openqa_gru

[2021-07-01T09:10:37.0930 UTC] [error] [#655446] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-custom-partitions.x86_64-1.15.1-Build9.25.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:39.0374 UTC] [error] [#655448] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk.x86_64-1.42.1-Build9.25.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:41.0366 UTC] [error] [#655452] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-simple.x86_64-1.42.1-Build9.25.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:47.0370 UTC] [error] [#655450] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build11.3.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:53.0960 UTC] [error] [#655456] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-lvm.x86_64-1.42.1-Build9.25.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:56.0834 UTC] [error] [#655460] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-overlayroot.x86_64-1.42.1-Build9.25.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:56.0901 UTC] [error] [#655462] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-suse-on-dnf.x86_64-1.42.1-Build9.25.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:11:03.0550 UTC] [error] [#655464] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-ramdisk.x86_64-1.42.1-Build9.25.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:11:06.0027 UTC] [error] [#655458] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-orthos.x86_64-1.13.1-Build10.2.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:11:31.0623 UTC] [error] [#655454] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-luks.x86_64-1.15.1-Build9.25.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:08.0957 UTC] [error] [#655995] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-simple.x86_64-1.42.1-Build9.26.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:09.0747 UTC] [error] [#655991] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-custom-partitions.x86_64-1.15.1-Build9.26.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:10.0917 UTC] [error] [#655993] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk.x86_64-1.42.1-Build9.26.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:13.0050 UTC] [error] [#656011] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-ramdisk.x86_64-1.42.1-Build9.26.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:14.0522 UTC] [error] [#656008] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build11.4.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:45.0582 UTC] [error] [#656142] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-overlayroot.x86_64-1.42.1-Build9.26.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:47.0314 UTC] [error] [#656138] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-lvm.x86_64-1.42.1-Build9.26.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:49.0611 UTC] [error] [#656144] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-suse-on-dnf.x86_64-1.42.1-Build9.26.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:53.0115 UTC] [error] [#656140] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-orthos.x86_64-1.13.1-Build10.3.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:56.0766 UTC] [error] [#656136] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-luks.x86_64-1.15.1-Build9.26.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:16.0944 UTC] [error] [#660689] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-custom-partitions.x86_64-1.15.3-Build9.7.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:19.0054 UTC] [error] [#660691] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk.x86_64-1.15.3-Build9.7.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:20.0007 UTC] [error] [#660693] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-simple.x86_64-1.15.3-Build9.7.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:23.0652 UTC] [error] [#660700] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-lvm.x86_64-1.15.3-Build9.7.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:28.0885 UTC] [error] [#660705] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-overlayroot.x86_64-1.15.3-Build9.7.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:29.0208 UTC] [error] [#660707] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-ramdisk.x86_64-1.15.3-Build9.7.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:35.0801 UTC] [error] [#660697] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-luks.x86_64-1.15.3-Build9.7.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.

so

  1. Why is the same error detail not accessible over the webUI?
  2. Can we look into monitoring logs for errors again?
  3. Should IO::Uncompress::UnXz be a general dependency of the openQA package? According to https://metacpan.org/dist/IO-Compress IO::Uncompress is included in there but https://build.opensuse.org/package/show/devel:languages:perl/perl-IO-Compress is not currently part of openSUSE Factory . zypper in -C 'perl(IO::Uncompress::UnXz)' installs perl-IO-Compress-Lzma, but not on Leap 15.2 it seems

#6 Updated by dancermak 5 months ago

  1. Why is the same error detail not accessible over the webUI?

The error is not visible in the webUI, because the error message is generated in lib/OpenQA/Task/Asset/Download.pm:104:

    unless ($downloader->download($url, $assetpath, $options)) {
        $ctx->error(my $msg = qq{Downloading "$url" failed because of too many download errors});
        return $job->fail($msg);
    }

and the download function swallows the actual cause of the error (from lib/OpenQA/Downloader.pm:106):

            try {
                die "Could not determine archive type\n"
                  unless my $ae = Archive::Extract->new(archive => $tempfile);
                die $ae->error unless $ae->extract(to => $target);
            }
            catch {
                $log->error(qq{Extracting "$tempfile" failed: $_});
                $ret = $code;
            };

I would suggest that we refactor this function and make the error message visible in the webUI as well. Especially as atm every error, be it download, decompression or /tmp/ is full will result in the error message Downloading "$url" failed because of too many download errors (which is not overly helpful anyway).

  1. Can we look into monitoring logs for errors again?

  2. Should IO::Uncompress::UnXz be a general dependency of the openQA package?
    According to https://metacpan.org/dist/IO-Compress IO::Uncompress is included in there but https://build.opensuse.org/package/show/devel:languages:perl/perl-IO-Compress is not currently part of openSUSE Factory . zypper in -C 'perl(IO::Uncompress::UnXz)' installs perl-IO-Compress-Lzma, but not on Leap 15.2 it seems

It should become a dependency imho.

#7 Updated by tinita 5 months ago

We discussed this in the team.
The problem is that the dual life module IO::Compress is not in Factory, but IO::Compress::Lzma needs the newer version from CPAN, the one in the perl core is too old.
There are comments trying to explain why we can't put it into Factory:
https://build.opensuse.org/package/show/devel:languages:perl/perl-IO-Compress

We now decided to link the necessary modules from devel:languages:perl in https://build.opensuse.org/project/show/devel:openQA:Leap:15.2 et al

I did 15.2 and the OBS build for the PR https://github.com/os-autoinst/openQA/pull/4011 doesn't say "unresolvable" any more, so this worked.
I linked the following modules:

  • perl-IO-Compress
  • perl-Compress-Raw-Lzma
  • perl-Compress-Raw-Bzip2
  • perl-Compress-Raw-Zlib
  • perl-IO-Compress-Lzma

e.g.

osc linkpac openSUSE:Factory perl-IO-Compress-Lzma  devel:openQA:Leap:15.3
osc linkpac devel:languages:perl perl-Compress-Raw-Zlib  devel:openQA:Leap:15.2
...

Will do that for 15.1 and 15.3 too

Edit: done for 15.1 and 15.3 too

#8 Updated by mkittler 5 months ago

  • Status changed from New to In Progress
  • Assignee set to dancermak

I've seen that a PR for showing the actual error message has been merged. This leaves dealing with the dependency which is handled in https://github.com/os-autoinst/openQA/pull/4011.

#9 Updated by openqa_review 5 months ago

  • Due date set to 2021-07-21

Setting due date based on mean cycle time of SUSE QE Tools

#10 Updated by okurz 4 months ago

  • Due date changed from 2021-07-21 to 2021-08-13
  • Target version changed from Ready to future

To be followed up with later. For now I plan that dancermak can work on the topic on their own so we can move out of the team's backlog.

#11 Updated by dancermak 4 months ago

  • Status changed from In Progress to Resolved

It works now on o3

Also available in: Atom PDF