QA » openQA Project

Could this be related to #120067?

I had a look yesterday but unfortunately I'm not sure how to get further information, e.g. xzgrep -i -P 'Downloading.*kiwi' /var/log/openqa_gru* on o3 returns no results. (There are generally logs for these downloads, e.g. just xzgrep -i -P 'Downloading.*' /var/log/openqa_gru* returns many results.)

It isn't clear whether the problem is within openQA's downloader itself or on the remote end. If the problem it on the remote end, maybe would downloader could at least do some sanity checks, e.g. whether the size is zero. If the sanity jobs would fail the jobs would be incompleted with a meaningful reason and maybe we could even retry the download a few times before it comes to that.

mkittler wrote:

It isn't clear whether the problem is within openQA's downloader itself or on the remote end. If the problem it on the remote end, maybe would downloader could at least do some sanity checks, e.g. whether the size is zero. If the sanity jobs would fail the jobs would be incompleted with a meaningful reason and maybe we could even retry the download a few times before it comes to that.

I find it unlikely to be on the remote end, as I never see any of these issues on my local machine. I am also quite close to o3 (geographically speaking), so I'd guess that we would use the same mirrors.

Two ideas what one could do to check:

1. Try to reproduce manually with wget/curl from o3 webui host and/or workers
2. Try to reproduce with manually spawned minion controlled download jobs

I've already got a 404 response for the first best job (https://openqa.opensuse.org/tests/3050041, http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/kiwi-test-image-luks.x86_64-1.15.1-Build44.6.raw) I found.

The same counts for the passing jobs on https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20230118&groupid=85 (the latest build as of now). They have been passing so presumably openQA could even download the assets at the time but now they're all gone. That let's me wonder whether the problem is simply that openQA is often just too late.

Due to the new build there are now also logs present when searching via xzgrep -i -P 'Downloading.*kiwi' /var/log/openqa_gru*. However, there's not much interesting being logged as the latest build actually didn't run into download errors anymore¹. I'll try to start such a download job locally to check the behavior in case of a 404 error. I wouldn't be surprised if that would leave a zero-sized asset.

¹ The log just contains messages like

[2023-01-18T07:36:38.019960Z] [debug] Process 5710 is performing job "2182254" with task "download_asset"
[2023-01-18T07:36:38.113624Z] [debug] [#2182254] Downloading "http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/kiwi-test-image-MicroOS.x86_64-16.0.0-Build46.6.qcow2" to "/var/lib/openqa/share/factory/hdd/kiwi-test-image-MicroOS.x86_64-16.0.0-Build46.6.qcow2"
[2023-01-18T07:36:38.114237Z] [info] [#2182254] Downloading "kiwi-test-image-MicroOS.x86_64-16.0.0-Build46.6.qcow2" from "http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/kiwi-test-image-MicroOS.x86_64-16.0.0-Build46.6.qcow2"

In case of a 404 error the job's reason is set to Reason: preparation failed: … and we would see a clear log message about it:

[debug] Process 16123 is performing job "1820" with task "download_asset"
[debug] [#1820] Downloading "http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/iso/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build49.5.install.iso" to "/hdd/openqa-devel/openqa/share/factory/iso/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build49.5.install.iso"
[info] [#1820] Downloading "kiwi-test-image-disk-legacy.x86_64-1.42.1-Build49.5.install.iso" from "http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/iso/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build49.5.install.iso"
[debug] Process 16125 is performing job "1821" with task "download_asset"
[info] [#1820] Download of "/hdd/openqa-devel/openqa/share/factory/iso/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build49.5.install.iso" failed: 404 Not Found
[error] [#1820] Downloading "http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/iso/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build49.5.install.iso" failed with: Download of "/hdd/openqa-devel/openqa/share/factory/iso/kiwi-test-image-disk-legacy.x86_64-1.42.1-Build49.5.install.iso" failed: 404 Not Found
[debug] [pid:16123] _carry_over_candidate(3488): _failure_reason=GRU:failed
[debug] Process 16135 is performing job "1822" with task "finalize_job_results"

This also leaves no empty file in the file system. The same counts if the asset needs to be extracted.

That means:

• The issue we've seen on o3 was definitely not caused by assets simply gone on the remote end (even though the quick cleanup could be problematic as well).
• The error handling in our HTTP downloading/extraction code seems to work generally (and does not just leave us with zero-sized files and than continuing the test execution). Judging by the code this is also true for other errors than 404.

To follow the suggestions from @okurz I can download some other assets on http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images from o3 (because the URLs referenced by current jobs are already 404).

Looks like some files on http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images also give a 404 error despite being still listed, e.g. for http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/kiwi-test-image-custom-partitions.x86_64.raw.xz I get a 404 page after being redirected to the mirror. Also the 404 errors that happen only after a redirection are handled well by openQA's downloader. So that cannot be our case as well.

Due to the mirroring error I couldn't do that many successful download. However, the downloads that didn't run into the problem actually worked without some other problems. So I guess at least downloading via wget from o3 (as suggested in #123175#note-8) works normally.

I have also checked the downloader code more closely. It even has a check to see whether the size of the actually downloaded file matches the size specified in the HTTP header. I couldn't spot any mistake except that it would likely run into the error Size of "$target" differs, expected $header_size but downloaded $actual_size if the Content-Length header is not present in the HTTP response. However, that kind of error would be handled and not lead to the behavior we saw.

A new build has just been scheduled and so far all download jobs shown on the Minion dashboard are either finished (which means nothing went totally wrong but download errors might have been occurred¹) or still waiting/active. There are no failures so far (except old/unrelated jobs) so nothing went completely wrong (e.g. an unhandled exception in download jobs). I've checked the size of some downloaded assets and couldn't find any suspiciously small files. So it doesn't look like we'd be able to reproduce the problem mentioned in the ticket description but I'll keep tracking what's going on. (None of the openQA jobs in https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20230124&groupid=85 have already been started at this point.)

I'm also following the gru logs via tail --follow /var/log/openqa and so far everything looks good (except some unrelated problems saving needles).

¹ Download errors will not result in failing Minion jobs anymore but should lead to incomplete jobs with the according reason - something that did not happen with the jobs mentioned in the ticket description.

One workaround for restarting affected job scenarios including a re-download of assets is to go to https://openqa.opensuse.org/admin/assets, copy-paste the asset name from job details into that page and click the "delete asset" button. Then retrigger tests which should trigger a re-download.

Following ideas:

• One can add a test module in the test distribution similar to https://openqa.opensuse.org/tests/3065960#step/isosize/1 but checking for validity of the asset size, i.e. >1
• Can we provide a hint to users where the asset comes from, e.g. link to the minion gru job from the openQA webUI
• Provide a link to the asset page https://openqa.opensuse.org/admin/assets searching for the asset

@mkittler consider the above points for this ticket with quick implementations, otherwise I suggest to put into #65271

I have been creating a PR for the last suggestion as it is very easy to implement: https://github.com/os-autoinst/openQA/pull/4986

I don't think we can provide a link to the GRU job because when I've initially investigated this ticket I couldn't find any related GRU jobs anymore. Likely those jobs have already been cleaned up (as they are usually cleaned up quite fast unless they'd be failures). Note that this should normally also not required because if such a job fails the information should be present as reason in the job's infobox (regardless of whether the job has been deleted meanwhile). Why this didn't work in this case I don't know. Maybe the jobs couldn't even be created in the first place. Until we can pin down the problem more closely I don't think this idea is feasible/useful.

About adding a specific test module: @dancermak do you think it is useful? From my perspective it shouldn't be too hard to figure out that the test failed due to a zero-sized asset so I don't see much benefit in adding an extra check for that. However, maybe depending on your experience this might not be generally true.

mkittler wrote:

About adding a specific test module: @dancermak do you think it is useful? From my perspective it shouldn't be too hard to figure out that the test failed due to a zero-sized asset so I don't see much benefit in adding an extra check for that. However, maybe depending on your experience this might not be generally true.

This looks very useful!

I am not sure if we need a general check for zero sized disk images. Once you know that this is a potential issue, you tend to spot that immediately. Before I was made aware that this could happen, this was quite the puzzle though.

The overview https://openqa.opensuse.org/tests/overview?build=20230124 shows actually many more kiwi jobs. So some jobs of the last build have actually failed. A few jobs even show a similar symptom (e.g. https://openqa.opensuse.org/tests/3063931) but I haven't seen any zero-sized assets. (I have checked all jobs in that build that haven't passed/softfailed.)

Note that #123649 is supposedly really related considering my findings in #123649#note-11.

The PR for UI improvements in openQA has been merged.

The ticket #123649 was only related in the sense that the kiwi jobs mentioned here triggered the error message mentioned in that ticket. However, this was another problem and not the reason for zero-sized downloads. At least I'd say so because when reproducing the Could not chdir back to start dir '' error the files themselves have actually been downloaded and extracted (and had a size, I have also verified it by computing the sha256 checksum).

Since all cases mentioned in the ticket description use the decompression feature I'll have a close look at. I learned from #123649 that the Archive::Extract module we're using doesn't return a negative return code in all error cases. Maybe the error handling of that module is broken in other ways, too. That would explain why the error handling on our side doesn't do its job (as from our side it looks like Archive::Extract succeeded).

About adding a test module: @dancermak Since you find it useful, do you want to implement that as maintainer of those tests? Otherwise I can also have a look. Maybe I can also add a sanity check in openQA itself at least for the extraction part (e.g. an uncompressed file should be at least as big as the compressed version¹).

¹ Although, I'm not sure whether that makes generally sense. Supposedly, in unfortunate cases, compression can make things worse.

To answer the footnote from my previous comment: Compression can make it worse, e.g. running xz on a file containing a sha256sum (base64 encoded + newline) the sizes goes up from 65 byte to 124 byte. So a sanity check for compressed size < uncompressed size doesn't make generally sense.

mkittler wrote:

Maybe it is best to abandon Archive::Extract [...]

I quickly looked into what I could find regarding the dependency chain. It looks like we would need to start with https://build.opensuse.org/package/live_build_log/devel:languages:perl:CPAN-F/perl-FFI-Platypus/openSUSE_Tumbleweed/x86_64 failing with

[   13s] +/usr/bin/perl -Iinc -MAlien::Base::Wrapper=Alien::FFI,Alien::psapi -e cc -- -D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -Wall -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -c -o blib/lib/auto/share/dist/FFI-Platypus/probe/src/dlrun.o blib/lib/auto/share/dist/FFI-Platypus/probe/src/dlrun.c
[   13s] Can't locate Alien/FFI.pm in @INC (you may need to install the Alien::FFI module) (@INC contains: inc /usr/lib/perl5/site_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.36.0 /usr/lib/perl5/vendor_perl/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.36.0 /usr/lib/perl5/5.36.0/x86_64-linux-thread-multi /usr/lib/perl5/5.36.0 /usr/lib/perl5/site_perl) at inc/Alien/Base/Wrapper.pm line 68.
[   13s] BEGIN failed--compilation aborted.

which looks like something that we should be able to fix by ensuring that https://metacpan.org/pod/Alien::FFI is packaged. Maybe https://build.opensuse.org/package/show/devel%3Alanguages%3Aperl%3ACPAN-A/perl-Alien already provides it and I couldn't see that. Then the dependency chain might resolve and we can submit the dependency which otherwise doesn't look too bad.

Note that it is also just a theory that the extraction code is the problem as I could not pin-down the problem exactly. [...]

Neither I do have another lead. I suggest if the above approach does not look feasible within a reasonable time we will not follow on with a full problem resolution until we learned more.

We discussed it in the unblock. Why not use bsdtar and keep it simple.

PR for using bsdtar: https://github.com/os-autoinst/openQA/pull/5004

The PR has been merged and deployed. Let's see how well it behaves in production.

The last kiwi images build is from 24 days ago so meanwhile nothing has happened in production that would give us further clues. So when the switch to bsdtar generally works I'd leave it at that and resolve the ticket.

mkittler wrote:

The PR has been merged and deployed. Let's see how well it behaves in production.

The last kiwi images build is from 24 days ago so meanwhile nothing has happened in production that would give us further clues. So when the switch to bsdtar generally works I'd leave it at that and resolve the ticket.

I have triggered a new round of builds today and got a ton of new failures with either zero sized image files (e.g. https://openqa.opensuse.org/tests/3135908) or workers that fail to start (e.g. https://openqa.opensuse.org/tests/3135926). So unfortunately it looks like switching to bsdtar did not solve the issue.

The workers that failed to start are likely a different issue. The jobs have been automatically restarted, right?

I tried to reproduce this again locally via

openqa-clone-job … https://openqa.opensuse.org/tests/3135908 HDD_1_DECOMPRESS_URL=http://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/kiwi-test-image-custom-partitions.x86_64-1.15.1-Build45.35.raw.xz

but the extraction works just fine (and the test also passed). I had to specify a different URL because the one from the original job is a 404 page now. When keeping the URL as-is I'm getting an incomplete with the Reason: preparation failed: Downloading "http://download.opensuse.org/…/kiwi-test-image-custom-partitions.x86_64-1.15.1-Build45.28.raw.xz" failed with: … and there are no zero-sized leftovers on disk. So the error handling works in this case.

I've already been investigating the error handling on our side before. I couldn't find anything so I concluded the problem must be in the archiving code. Apparently I must have missed something. I guess I can go though the code and tests again. Maybe I can spot a mistake this time.

mkittler wrote:

I've added a "Workaround" section although I suppose @dancermak is already aware of it.

Unfortunately, the workarounds are not really an option for my intended use-case:

• manually removing the files from the webui host has a few issues. First this doesn't scale at all, given that I've often seen as many as half of the tests of a full run fail. Additionally it requires someone with very high permissions to comb through o3, find the correct files for the failed tests, remove them, restart all the tests, report back on github…
• not using compression is not an option either. Last time we turned of compression of the disk images, we filled up OBS and killed it.

Wouldn't it be possible for openQA to check after a test failed, whether one of the disk/iso assets has zero size and if yes, delete the asset and re-try? This would have the advantage that this issue gets worked around in one place and not by every single user over and over again.