QA » openQA Project » openQA Infrastructure

More details:

The targets are in the https://openqa.suse.de/assets/repo/fixed:

https://openqa.suse.de/assets/repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1
https://openqa.suse.de/assets/repo/fixed/SLE-15-SP3-Online-x86_64-GM-Media1

But, when/if symlinks from .../fixed to .../assets are created, they are gopng to be cleaned up somwhere in time.

Possible causue: Isn't there some WHITE-list of things which isn't supposed to be auto cleaned up from the assests?

I was updating the ticket as you were typing, I think I have answered your question.

(Antonios Pappas) Having the installation media hosted on OSD is not a hard requirement. The requirement is that the ipmi workers with nvdimm memory are able to reach the installation media
Using REPO_0=https://download.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/ allows the test to run problem free
Would you think that switching all jobs to this solution is better? all NVDIMM jobs not every single job
(Oliver Kurz) Use 2., definitely. openQA assets are to be used to ensure content does not change, like builds of SLE15-SP6 but something like SLE-15-SP2-Full-GM must never change on download.suse.de so, yeah, use that

okurz wrote in #note-14:

jkohoutek wrote in #note-12:

No, it's not solved until we find the root cause and what happening there.

• Why there are all other images linekd in Assests?

What do you mean by that? Do you mean why does e.g. https://openqa.suse.de/tests/13198459#downloads mention https://openqa.suse.de/tests/13198459/asset/iso/SLE-15-SP2-Installer-DVD-x86_64-GM-DVD1.iso ? That would be because this file is simply linked in the job settings with "ISO=SLE-15-SP2-Installer-DVD-x86_64-GM-DVD1.iso"

We are not talking abou ISOs here, but about the install targets.

More specific question: Why those persist; beside a huge list of others; https://openqa.suse.de/assets/repo/SLE-15-SP2-Full-ppc64le-GM-Media1/ and https://openqa.suse.de/assets/repo/SLE-15-SP2-Full-s390x-GM-Media1/ but when x86_64 is linked there same way as https://openqa.suse.de/assets/repo/SLE-15-SP2-Online-x86_64-GM-Media1/ it gets purged?

• Why this 2 still got purged from there?

repositories are never listed on assets but only individual files

I don't follow you here, I see a really huge load of the install targets there: https://openqa.suse.de/assets/repo/

• Where is documented the process of the purge?

As explained in #152377-5 that's in http://open.qa/docs/#_asset_handling

jkohoutek wrote in #note-15:

okurz wrote in #note-14:

jkohoutek wrote in #note-12:

No, it's not solved until we find the root cause and what happening there.

• Why there are all other images linekd in Assests?

What do you mean by that? Do you mean why does e.g. https://openqa.suse.de/tests/13198459#downloads mention https://openqa.suse.de/tests/13198459/asset/iso/SLE-15-SP2-Installer-DVD-x86_64-GM-DVD1.iso ? That would be because this file is simply linked in the job settings with "ISO=SLE-15-SP2-Installer-DVD-x86_64-GM-DVD1.iso"

We are not talking abou ISOs here, but about the install targets.

More specific question: Why those persist; beside a huge list of others; https://openqa.suse.de/assets/repo/SLE-15-SP2-Full-ppc64le-GM-Media1/ and https://openqa.suse.de/assets/repo/SLE-15-SP2-Full-s390x-GM-Media1/ but when x86_64 is linked there same way as https://openqa.suse.de/assets/repo/SLE-15-SP2-Online-x86_64-GM-Media1/ it gets purged?

To answer that I looked into logs on OSD and found

openqa:/var/log # zgrep 'SLE-15-SP2-Full-s390x-GM-Media1' openqa_gru*
openqa_gru:[2024-01-16T16:00:02.598510+01:00] [info] [pid:21626] Registering asset repo/SLE-15-SP2-Full-s390x-GM-Media1
openqa_gru:                 name        => "repo/SLE-15-SP2-Full-s390x-GM-Media1",
openqa_gru:[2024-01-16T16:00:58.627253+01:00] [info] [pid:21626] Removing asset repo/SLE-15-SP2-Full-s390x-GM-Media1 (not in any group, age (1219 days) exceeds limit (7 days)
openqa_gru:[2024-01-16T16:00:58.630012+01:00] [info] [pid:21626] GRU: removed /var/lib/openqa/share/factory/repo/SLE-15-SP2-Full-s390x-GM-Media1
…
openqa_gru.9.xz:[2024-01-16T07:00:02.026605+01:00] [info] [pid:14452] Registering asset repo/SLE-15-SP2-Full-s390x-GM-Media1
openqa_gru.9.xz:                 name        => "repo/SLE-15-SP2-Full-s390x-GM-Media1",
openqa_gru.9.xz:[2024-01-16T07:00:44.303850+01:00] [info] [pid:14452] Removing asset repo/SLE-15-SP2-Full-s390x-GM-Media1 (not in any group, age (1219 days) exceeds limit (7 days)
openqa_gru.9.xz:[2024-01-16T07:00:44.306424+01:00] [info] [pid:14452] GRU: removed /var/lib/openqa/share/factory/repo/SLE-15-SP2-Full-s390x-GM-Media1

so at least openQA does state that it would remove SLE-15-SP2-Full-s390x-GM-Media1 although the symlink seems to persist:

openqa:/var/log # ls -l /var/lib/openqa/share/factory/repo/ | grep SLE-15-SP2-Full
lrwxrwxrwx  1 geekotest nogroup        47 Aug 19  2020 SLE-15-SP2-Full-ppc64le-GM-Media1 -> fixed/SLE-15-SP2-Full-ppc64le-Build209.2-Media1
lrwxrwxrwx  1 geekotest nogroup        38 Jan 16 16:45 SLE-15-SP2-Full-s390x-GM-Media1 -> fixed/SLE-15-SP2-Full-s390x-GM-Media1/

For ppc64le it looks different though and the logfiles do not mention that symlink at all, possibly because openQA jobs only silently reference it so that symlink might never get known by openQA hence never removed?

For x86_64 I see no reference to removing any repo links:

openqa:/var/log # zgrep 'SLE-15-SP2-Online-x86_64-GM-Media1' openqa_gru* | grep -v '\<iso\>'
openqa_gru:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru-20240110.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru-20240111.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru-20240112.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru-20240113.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru-20240114.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru-20240115.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru-20240116.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.1.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.10.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.11.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.12.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.13.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.14.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.15.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.16.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.17.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.18.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.19.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.2.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.2.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.20.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.3.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.4.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.5.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.6.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.7.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.8.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",
openqa_gru.9.xz:                 name        => "repo/fixed/SLE-15-SP2-Online-x86_64-GM-Media1",

jkohoutek do you have a further idea how to proceed?

• Why this 2 still got purged from there?

repositories are never listed on assets but only individual files

I don't follow you here, I see a really huge load of the install targets there: https://openqa.suse.de/assets/repo/

Sorry, I meant they are never listed as assets on the job details page but openQA properly tracks them as assets.

• Where is documented the process of the purge?

As explained in #152377-5 that's in http://open.qa/docs/#_asset_handling

okurz wrote in #note-16:

so at least openQA does state that it would remove SLE-15-SP2-Full-s390x-GM-Media1 although the symlink seems to persist:

openqa:/var/log # ls -l /var/lib/openqa/share/factory/repo/ | grep SLE-15-SP2-Full
lrwxrwxrwx  1 geekotest nogroup        47 Aug 19  2020 SLE-15-SP2-Full-ppc64le-GM-Media1 -> fixed/SLE-15-SP2-Full-ppc64le-Build209.2-Media1
lrwxrwxrwx  1 geekotest nogroup        38 Jan 16 16:45 SLE-15-SP2-Full-s390x-GM-Media1 -> fixed/SLE-15-SP2-Full-s390x-GM-Media1/

For ppc64le it looks different though and the logfiles do not mention that symlink at all, possibly because openQA jobs only silently reference it so that symlink might never get known by openQA hence never removed?

For x86_64 I see no reference to removing any repo links:

Because we stop futile creating of them

jkohoutek do you have a further idea how to proceed?

From the behavior you find in the logs I even more think that there is white list of what shouldn't be removed somewhere and x86_64 for 15-SP2 and SP3 should be added to it.

Also, what concerns me from your finds is behavior around the s390 and the ppc64 ones:
s390 is not deleted, even when it should be and log even state it was removed - maybe the white list I mentioned before?
ppc64 is even stranger, it just lie there

Yes, I also do not know what's going on. Asking more team members for help.

I've just briefly read over comments so maybe I'm missing something.

s390 is not deleted, even when it should be and log even state it was removed - maybe the white list I mentioned before?

What white-list? I don't think we have a white-list for preserving certain assets. Note that the deletion of assets can fail due to missing permissions/ownership or someone might have put it back again. Additionally, the conditions under which assets are removed are not totally trivial. So maybe the asset just hasn't expired yet. Or the cleanup task hasn't run (or hasn't reached the point where it would delete that asset). If you want me to investigate this nevertheless, please state the exact name of the asset and why you think it should have been removed.

When symlinking assets from the fixed directory (or basically from anywhere) to the regular assets directory the symlinked asset is subject to the normal cleanup which is not a bug. In case of directories symlinks are apparently followed. Maybe that's something we should not do. It would likely be trivial to implement not following symlinks. Of course that would only fix half of the problem. At least when I understand your use case correctly you want even the symlink to stay. So I guess we needed to preserve the symlink if it points to the fixed directory and it is not dangling.

By the way, on OSD the following assets are affected:

martchus@openqa:/assets/factory> find repo/ -maxdepth 1 -type l -exec readlink {} \; | grep fixed/
fixed/SLE-12-SP3-Server-DVD-s390x-GM-DVD1/
fixed/SLE-12-SP4-Server-DVD-s390x-GM-DVD1/
fixed/SLE-12-SP?-SDK-POOL-aarch64-BuildGM-Media1/
fixed/SLE-12-SP?-SDK-POOL-ppc64le-BuildGM-Media1/
fixed/SLE-12-SP?-SDK-POOL-s390x-BuildGM-Media1/
fixed/SLE-12-SP?-SDK-POOL-x86_64-BuildGM-Media1/
fixed/SLE-12-Server-DVD-s390x-GM-DVD1/
fixed/SLE-15-Installer-DVD-s390x-GM-DVD1/
fixed/SLE-15-SP1-Installer-DVD-s390x-GM-DVD1/
fixed/SLE-15-SP1-Installer-DVD-s390x-QU4-Media1
fixed/SLE-15-SP2-Full-ppc64le-Build209.2-Media1
fixed/SLE-15-SP2-Installer-DVD-s390x-GM-DVD1/
fixed/SLE-15-SP2-Online-ppc64le-GM-Media1/
fixed/SLE-15-SP3-Full-s390x-GM-Media1/
fixed/SLE-15-SP3-Full-s390x-GM-Media1/
fixed/SLE-15-SP4-Full-s390x-GM-Media1/
fixed/SLE-15-SP4-Online-x86_64-GM-Media1/
fixed/SLE-15-SP5-Online-x86_64-GM-Media1/
fixed/SLE-15-SP3-Online-aarch64-GM-Media1/
fixed/SLE-15-SP5-Full-ppc64le-GM-Media1/
fixed/SLE-15-SP4-Online-aarch64-GM-Media1/
fixed/SLE-12-SP5-Server-DVD-s390x-GM-DVD1/
fixed/SLE-15-SP4-Online-ppc64le-GM-Media1/
fixed/SLE-15-SP3-Online-x86_64-GM-Media1/
fixed/SLE-15-SP4-Online-s390x-GM-Media1/
fixed/SLE-15-SP2-Full-s390x-GM-Media1/
fixed/SLE-15-SP2-Online-s390x-GM-Media1/
fixed/SLE-15-SP3-Online-s390x-GM-Media1/
fixed/SLE-15-SP5-Online-s390x-GM-Media1/

Would something like that help? https://github.com/os-autoinst/openQA/pull/5428

(Of course that change would only work for relative symlinks as-is and still needed testing.)

It also looks like our cleanup code would preserve dangling symlinks forever (as it skips assets which don't exist). I haven't tested that but it might also explain why some assets are not deleted.

I've just updated https://github.com/os-autoinst/openQA/pull/5428 so have a look and give feedback whether it'll help in your case.

apappas wrote in #note-25:

This is still failing https://openqa.suse.de/tests/13311194#step/boot_from_pxe/7

The output is a bit clobbered but AFAICT from the video, everything was entered correctly before the SUT responded.

The problem here is that "SLE-15-SP1-Installer-DVD-x86_64-GM-Media1" does not exist on OSD.
openqa:/var/log # zgrep 'SLE-15-SP1-Installer-DVD-x86_64-GM-Media1' openqa_gru* shows no match so I assume no one ever recreated it there in place. And to remind what was discussed in https://suse.slack.com/archives/C02CANHLANP/p1704812941555379

(Oliver Kurz) who has an idea about https://openqa.suse.de/tests/13198459#step/boot_from_pxe/6 https://progress.opensuse.org/issues/152377 trying to boot assets in the "fixed/" directory over iPXE?
(Antonios Pappas) Since this is not officially supported, I tried using the download.suse.de sources and it works. 1. Having the installation media hosted on OSD is not a hard requirement. The requirement is that the ipmi workers with nvdimm memory are able to reach the installation media

1. Using REPO_0=https://download.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/ allows the test to run problem free Would you think that switching all jobs to this solution is better? all NVDIMM jobs not every single job (Oliver Kurz) Use 2., definitely. openQA assets are to be used to ensure content does not change, like builds of SLE15-SP6 but something like SLE-15-SP2-Full-GM must never change on download.suse.de so, yeah, use that

So your step 2. needs to be followed. I suggest you create a separate ticket within the scope of qe-sap for that and follow-up there. Here we can then focus on the inconsistencies in asset handling and cleanup.