QA » openQA Project

Note, we're currently running git commit 4861e34 , from 2020-02-05. I checked lib/OpenQA/Task/Asset/Limit.pm and lib/OpenQA/Schema/ResultSet/Assets.pm and neither has really changed since then (just a license notice change in the latter).

I can not open the Minion dashboard without login. However, it looks like the asset in question is HDD_1=disk_f32_minimal_3_x86_64.img. I can only find this setting in the parent job. The child job only has ISO=Fedora-Workstation-Live-x86_64-FEDORA-2020-9d65d662e2.iso. So unless the parent job is supposed to create the ISO file it seems that there's simply the HDD_1 setting missing in the child job.

(We have usually e.g. PUBLISH_HDD_1=opensuse-Tumbleweed-x86_64-20200317-kde@64bit.qcow2 in the parent and HDD_1=opensuse-Tumbleweed-x86_64-20200317-kde@64bit.qcow2 in the child job.)

No, that's not the asset in question. That's a fixed asset. The asset that got wiped was Fedora-Workstation-Live-x86_64-FEDORA-2020-9d65d662e2.iso . That is created and uploaded by 764286 (live_build).

I'd agree those are the choices, but if 1) is the current case, then things changed in some rewrite along the line, because 2) used to be intentionally and explicitly the case (or rather, there was not even any configuration choice, we just always kept assets associated with pending jobs). I know because I wrote it :) (you also commented on it, though after coolo merged it)

https://github.com/os-autoinst/openQA/pull/1518

I know about the feature to retain "pending" assets. There is also a testcase for that feature in t/14-grutasks.t. Of course the test case might be broken as well as the feature but we haven't changed a lot in that area recently. I'll try to break the "pending" job handling locally to check whether the test in t/14-grutasks.t would catch that. Depending on the outcome I can investigate the problem further. Of course without access to the openQA instance and general lack of reproducibility I might get stuck. That also raises the questions: Is this happening frequently on your instance (@AdamWill)? Is this happening in our instances (@tools-team)? I haven't came across it so far on our instances.

@mkittler, yup, that's why I wrote "if 1) is the current case" - I figured it might well not be an intentional change.

Unfortunately no this isn't happening very often or anything - I just happened to catch a few cases. I believe that, at the time this happened, I was re-running a lot of update tests simultaneously, and the asset size limit for our 'Fedora Updates' group is only 100GB, so it's possible we ran into some edge case in the logic here or something.

Later today I'll try and gin up a search to find any other cases that failed similarly.

• PR for improving the test: https://github.com/os-autoinst/openQA/pull/2910
• I have also already a test for the case when the pending job is not the most recent one. In this case the implementation is broken, indeed.

Apparently it is not like I suspected. The test I've created in fact fails in a different way than expected. Recognizing that an asset is pending does not only rely on the most recent job associated to that asset and therefore the asset would actually be always correctly preserved. (So the condition case when j.id is not null and j.result='none' then 1 else 0 end in the query is actually evaluated for every associated job.)

But in my test a different and seemingly unrelated asset is additionally deleted when the most recent associated job is not pending but only a previous one. The setup of the unit tests we have is really hard to understand and surprising things happen when you adjust a detail. But maybe I actually found another limitation within the accounting which might be the reason for the issue.

I mean, I don't know what details you want. I reported a bug: I saw that a test had failed incomplete because an asset it needed had been garbage collected while it was scheduled. That's not supposed to happen, so I reported it. That's really all I've got.