Project

General

Profile

Actions

action #94997

closed

Downloading compressed disk images is broken on o3

Added by dancermak over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2021-07-01
Due date:
2021-08-13
% Done:

0%

Estimated time:

Description

Observation

I have started running kiwi image tests on o3 and started to run into issues with HDD_1_DECOMPRESS_URL: when this variable is set, then the test will never succeed, because one of the following cases happens:

  1. The download just fails, see: https://openqa.opensuse.org/tests/1814637 or https://openqa.opensuse.org/tests/1814639
  2. The download "succeeds", but the resulting disk image is a zero byte fail and openQA can not boot from that. See for instance: https://openqa.opensuse.org/tests/1814632, https://openqa.opensuse.org/tests/1814631, https://openqa.opensuse.org/tests/1814633, https://openqa.opensuse.org/tests/1814634

This must be related to some settings on o3, because this works perfectly on my local machine.

Steps to reproduce

  • create a new job and set HDD_1_DECOMPRESS_URL=http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/kiwi-test-image-luks.x86_64.raw.xz
  • watch it fail to download the image

Problem

absolutely no clue, it works on my local openQA instance

Suggestion

maybe the background downloader is overloaded?

Workaround

none that I know off


Related issues 1 (0 open1 closed)

Related to openQA Project - action #128651: [spike][timeboxed:20h] Current openQA+os-autoinst+dependencies updated in SLE+packagehub size:MResolvedjbaier_cz2023-04-26

Actions
Actions #1

Updated by okurz over 2 years ago

  • Project changed from openQA Infrastructure to openQA Project
  • Category set to Regressions/Crashes
  • Target version set to Ready

might be specific to o3 but still something to improve in openQA in general to make it clearer to the user what's going on.

Can you please update the ticket according to https://progress.opensuse.org/projects/openqav3/wiki/#Defects ?

Actions #2

Updated by okurz over 2 years ago

can you please update the ticket description instead of adding the new description as comment?

Also, https://openqa.opensuse.org/minion/jobs?offset=0&task=download_asset might provide some insight

Actions #3

Updated by dancermak over 2 years ago

  • Description updated (diff)
Actions #4

Updated by dancermak over 2 years ago

okurz wrote:

can you please update the ticket description instead of adding the new description as comment?

Sorry, I thought that the Edit button edits the description…

Also, https://openqa.opensuse.org/minion/jobs?offset=0&task=download_asset might provide some insight

Not really. This one here downloaded a zero bytes file: https://openqa.opensuse.org/minion/jobs?id=656141 and this one https://openqa.opensuse.org/minion/jobs?id=656140 failed due to "too many download errors" but I really don't see any additional information on that page.

Actions #5

Updated by okurz over 2 years ago

From o3 grep 'Extracting.*failed' /var/log/openqa_gru

[2021-07-01T09:10:37.0930 UTC] [error] [#655446] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-custom-partitions.x86_64-1.15.1-Build9.25.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:39.0374 UTC] [error] [#655448] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk.x86_64-1.42.1-Build9.25.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:41.0366 UTC] [error] [#655452] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-simple.x86_64-1.42.1-Build9.25.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:47.0370 UTC] [error] [#655450] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build11.3.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:53.0960 UTC] [error] [#655456] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-lvm.x86_64-1.42.1-Build9.25.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:56.0834 UTC] [error] [#655460] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-overlayroot.x86_64-1.42.1-Build9.25.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:10:56.0901 UTC] [error] [#655462] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-suse-on-dnf.x86_64-1.42.1-Build9.25.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:11:03.0550 UTC] [error] [#655464] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-ramdisk.x86_64-1.42.1-Build9.25.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:11:06.0027 UTC] [error] [#655458] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-orthos.x86_64-1.13.1-Build10.2.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T09:11:31.0623 UTC] [error] [#655454] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-luks.x86_64-1.15.1-Build9.25.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:08.0957 UTC] [error] [#655995] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-simple.x86_64-1.42.1-Build9.26.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:09.0747 UTC] [error] [#655991] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-custom-partitions.x86_64-1.15.1-Build9.26.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:10.0917 UTC] [error] [#655993] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk.x86_64-1.42.1-Build9.26.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:13.0050 UTC] [error] [#656011] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-ramdisk.x86_64-1.42.1-Build9.26.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T12:22:14.0522 UTC] [error] [#656008] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build11.4.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:45.0582 UTC] [error] [#656142] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-overlayroot.x86_64-1.42.1-Build9.26.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:47.0314 UTC] [error] [#656138] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-lvm.x86_64-1.42.1-Build9.26.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:49.0611 UTC] [error] [#656144] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-suse-on-dnf.x86_64-1.42.1-Build9.26.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:53.0115 UTC] [error] [#656140] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-orthos.x86_64-1.13.1-Build10.3.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-01T13:04:56.0766 UTC] [error] [#656136] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-luks.x86_64-1.15.1-Build9.26.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:16.0944 UTC] [error] [#660689] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-custom-partitions.x86_64-1.15.3-Build9.7.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:19.0054 UTC] [error] [#660691] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk.x86_64-1.15.3-Build9.7.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:20.0007 UTC] [error] [#660693] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-simple.x86_64-1.15.3-Build9.7.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:23.0652 UTC] [error] [#660700] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-lvm.x86_64-1.15.3-Build9.7.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:28.0885 UTC] [error] [#660705] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-overlayroot.x86_64-1.15.3-Build9.7.vmdk.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:29.0208 UTC] [error] [#660707] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-disk-ramdisk.x86_64-1.15.3-Build9.7.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.
[2021-07-02T11:36:35.0801 UTC] [error] [#660697] Extracting "/var/lib/openqa/share/factory/tmp/kiwi-test-image-luks.x86_64-1.15.3-Build9.7.raw.xz" failed: You do not have 'IO::Uncompress::UnXz' installed - Please install it as soon as possible.

so

  1. Why is the same error detail not accessible over the webUI?
  2. Can we look into monitoring logs for errors again?
  3. Should IO::Uncompress::UnXz be a general dependency of the openQA package? According to https://metacpan.org/dist/IO-Compress IO::Uncompress is included in there but https://build.opensuse.org/package/show/devel:languages:perl/perl-IO-Compress is not currently part of openSUSE Factory . zypper in -C 'perl(IO::Uncompress::UnXz)' installs perl-IO-Compress-Lzma, but not on Leap 15.2 it seems
Actions #6

Updated by dancermak over 2 years ago

  1. Why is the same error detail not accessible over the webUI?

The error is not visible in the webUI, because the error message is generated in lib/OpenQA/Task/Asset/Download.pm:104:

    unless ($downloader->download($url, $assetpath, $options)) {
        $ctx->error(my $msg = qq{Downloading "$url" failed because of too many download errors});
        return $job->fail($msg);
    }

and the download function swallows the actual cause of the error (from lib/OpenQA/Downloader.pm:106):

            try {
                die "Could not determine archive type\n"
                  unless my $ae = Archive::Extract->new(archive => $tempfile);
                die $ae->error unless $ae->extract(to => $target);
            }
            catch {
                $log->error(qq{Extracting "$tempfile" failed: $_});
                $ret = $code;
            };

I would suggest that we refactor this function and make the error message visible in the webUI as well. Especially as atm every error, be it download, decompression or /tmp/ is full will result in the error message Downloading "$url" failed because of too many download errors (which is not overly helpful anyway).

  1. Can we look into monitoring logs for errors again?

  2. Should IO::Uncompress::UnXz be a general dependency of the openQA package?
    According to https://metacpan.org/dist/IO-Compress IO::Uncompress is included in there but https://build.opensuse.org/package/show/devel:languages:perl/perl-IO-Compress is not currently part of openSUSE Factory . zypper in -C 'perl(IO::Uncompress::UnXz)' installs perl-IO-Compress-Lzma, but not on Leap 15.2 it seems

It should become a dependency imho.

Actions #7

Updated by tinita over 2 years ago

We discussed this in the team.
The problem is that the dual life module IO::Compress is not in Factory, but IO::Compress::Lzma needs the newer version from CPAN, the one in the perl core is too old.
There are comments trying to explain why we can't put it into Factory:
https://build.opensuse.org/package/show/devel:languages:perl/perl-IO-Compress

We now decided to link the necessary modules from devel:languages:perl in https://build.opensuse.org/project/show/devel:openQA:Leap:15.2 et al

I did 15.2 and the OBS build for the PR https://github.com/os-autoinst/openQA/pull/4011 doesn't say "unresolvable" any more, so this worked.
I linked the following modules:

  • perl-IO-Compress
  • perl-Compress-Raw-Lzma
  • perl-Compress-Raw-Bzip2
  • perl-Compress-Raw-Zlib
  • perl-IO-Compress-Lzma

e.g.

osc linkpac openSUSE:Factory perl-IO-Compress-Lzma  devel:openQA:Leap:15.3
osc linkpac devel:languages:perl perl-Compress-Raw-Zlib  devel:openQA:Leap:15.2
...

Will do that for 15.1 and 15.3 too

Edit: done for 15.1 and 15.3 too

Actions #8

Updated by mkittler over 2 years ago

  • Status changed from New to In Progress
  • Assignee set to dancermak

I've seen that a PR for showing the actual error message has been merged. This leaves dealing with the dependency which is handled in https://github.com/os-autoinst/openQA/pull/4011.

Actions #9

Updated by openqa_review over 2 years ago

  • Due date set to 2021-07-21

Setting due date based on mean cycle time of SUSE QE Tools

Actions #10

Updated by okurz over 2 years ago

  • Due date changed from 2021-07-21 to 2021-08-13
  • Target version changed from Ready to future

To be followed up with later. For now I plan that dancermak can work on the topic on their own so we can move out of the team's backlog.

Actions #11

Updated by dancermak over 2 years ago

  • Status changed from In Progress to Resolved

It works now on o3

Actions #12

Updated by jbaier_cz 10 months ago

  • Related to action #128651: [spike][timeboxed:20h] Current openQA+os-autoinst+dependencies updated in SLE+packagehub size:M added
Actions

Also available in: Atom PDF