QA » openQA Project » openQA Infrastructure

All important results should be visible from https://openqa.suse.de/admin/obs_rsync . Following that link I find https://openqa.suse.de/admin/obs_rsync/SUSE:SLE-15-SP5:GA:TEST%7Cbase/runs/.run_221019_103656_29.1 for build 29.1. https://openqa.suse.de/admin/obs_rsync/SUSE:SLE-15-SP5:GA:TEST%7Cbase/runs/.run_221019_103656_29.1/download/files_iso.lst lists the full medium for all four architectures. The full "repository" is extracted on the openQA host itself from the ISO file, visible in:
https://openqa.suse.de/admin/obs_rsync/SUSE:SLE-15-SP5:GA:TEST%7Cbase/runs/.run_221019_103656_29.1/download/rsync_iso.cmd

rsync --timeout=3600 -tlp4 --specials dist.suse.de::repos/SUSE:/SLE-15-SP5:/GA:/TEST/images/iso/*SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso /var/lib/openqa/factory/iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso
rsync --timeout=3600 -tlp4 --specials dist.suse.de::repos/SUSE:/SLE-15-SP5:/GA:/TEST/images/iso/*SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 /var/lib/openqa/factory/other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256
[ -d /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 ] || {
    mkdir /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
    bsdtar xf /var/lib/openqa/factory/iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso -C /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
}

with the logs in https://openqa.suse.de/admin/obs_rsync/SUSE:SLE-15-SP5:GA:TEST%7Cbase/runs/.run_221019_103656_29.1/download/rsync_iso.cmd.log

+ rsync --timeout=3600 -tlp4 --specials 'dist.suse.de::repos/SUSE:/SLE-15-SP5:/GA:/TEST/images/iso/*SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso' /var/lib/openqa/factory/iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso
+ rsync --timeout=3600 -tlp4 --specials 'dist.suse.de::repos/SUSE:/SLE-15-SP5:/GA:/TEST/images/iso/*SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256' /var/lib/openqa/factory/other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256
+ '[' -d /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 ']'
+ mkdir /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
+ bsdtar xf /var/lib/openqa/factory/iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso -C /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1

so it looks like the repo was created. Who can remind me where we can find log messages regarding the cleanup of assets?

I opened https://sd.suse.com/servicedesk/customer/portal/1/SD-102307

Kind request to extend vdc assigned to VM openqa.suse.de from 7.0T to 10.0T Waiting for support

   This is a kind request to extend vdc assigned to VM openqa.suse.de from 7.0T to 10.0T which should reside on “cheap+slow” storage, i.e. rotating disks.

   Reasoning: More and more products are tested by more and more development groups hence demanding more space.

   Related internal issue: https://progress.opensuse.org/issues/119215

okurz wrote:

I found on OSD

openqa:/var/log # xzgrep -i SLE-15-SP5-Full-aarch64-Build29 openqa_gru.8.xz
[2022-10-20T04:08:31.562809+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 110 (20620640697)
[2022-10-20T04:08:31.562875+02:00] [debug] [pid:31541] Asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) picked into group 110
[2022-10-20T04:08:31.562928+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 129 (11005337889)
[2022-10-20T04:08:31.563014+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 143 (163476307)
[2022-10-20T04:08:31.563077+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 251 (3067777832)
[2022-10-20T04:08:31.563148+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 265 (400818176)
[2022-10-20T04:08:31.563241+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 267 (5771211776)
[2022-10-20T04:08:31.563302+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 301 (3321888768)
[2022-10-20T04:08:31.563357+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 452 (46858240)
[2022-10-20T04:08:31.563710+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 110 (10068957625)
[2022-10-20T04:08:31.563781+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 129 (11005337889)
[2022-10-20T04:08:31.563840+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 143 (163476307)
[2022-10-20T04:08:31.563910+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 267 (5771211776)
[2022-10-20T04:08:31.563980+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 452 (46858240)
[2022-10-20T04:08:33.396420+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 110 (1590165945)
[2022-10-20T04:08:33.396470+02:00] [debug] [pid:31541] Asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) picked into group 110
[2022-10-20T04:08:33.396522+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 129 (445919993)
[2022-10-20T04:08:33.396573+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 143 (17576275)
[2022-10-20T04:08:33.396623+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 452 (46196224)
                 name        => "iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso",
                 name        => "repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1",
                 name        => "other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256",
[2022-10-20T04:18:45.478011+02:00] [info] [pid:31541] Removing asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (belonging to job groups: 129, 143, 452, 267, 110 within parent job groups 15)
[2022-10-20T04:19:33.121798+02:00] [info] [pid:31541] GRU: removed /var/lib/openqa/share/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
[2022-10-20T04:28:59.262163+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 110 (20609593380)
[2022-10-20T04:28:59.262233+02:00] [debug] [pid:31541] Asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) picked into group 110
[2022-10-20T04:28:59.262310+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 129 (11005337889)
[2022-10-20T04:28:59.262407+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 143 (163476307)
[2022-10-20T04:28:59.262494+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 251 (3067777832)
[2022-10-20T04:28:59.262572+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 265 (400818176)
[2022-10-20T04:28:59.262643+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 267 (5771211776)
[2022-10-20T04:28:59.263746+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 301 (3321888768)
[2022-10-20T04:28:59.263840+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 452 (46858240)
[2022-10-20T04:29:00.041376+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 110 (1579118628)
[2022-10-20T04:29:00.041454+02:00] [debug] [pid:31541] Asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) picked into group 110
[2022-10-20T04:29:00.041539+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 129 (445919993)
[2022-10-20T04:29:00.041624+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 143 (17576275)
[2022-10-20T04:29:00.041709+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 452 (46196224)

in particular:

[2022-10-20T04:18:45.478011+02:00] [info] [pid:31541] Removing asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (belonging to job groups: 129, 143, 452, 267, 110 within parent job groups 15)
[2022-10-20T04:19:33.121798+02:00] [info] [pid:31541] GRU: removed /var/lib/openqa/share/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1

so the asset was simply removed because it does not fit anymore.

@waynechen55 Please understand how the asset management on openQA works:
http://open.qa/docs/#_asset_handling

Based on available space I could extend the size limit for assets in the functional job group (id 110) from 260GB to 400GB which should allow to save all four Full repos though still taking a lot of space.

The next build should again have a full aarch64 repo.

Further information: https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&viewPanel=74&from=now-2y&to=now shows the evolution of asset space usage over the course of the past two years. Within https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&viewPanel=74&from=now-7d&to=now we see how the space usage goes down to 74% corresponding to (0.74 * 5.0T = 5.18TB) until we reach the limit of 80% (0.80 * 5.0T = 5.60TB) when we trigger the space-aware cleanup. So effectively we have a buffer of 5.60TB - 5.18TB = 0.420 ~ 420GB which we can still assign to groups.

If you need much more space for assets on openqa.suse.de please organize with all relevant user groups to formulate such request to SUSE-IT.

I can not have no 15-SP5 aarch64 Builds for testing. It is not something can be sacrificed.

okurz wrote:

I opened https://sd.suse.com/servicedesk/customer/portal/1/SD-102307

Kind request to extend vdc assigned to VM openqa.suse.de from 7.0T to 10.0T Waiting for support

   This is a kind request to extend vdc assigned to VM openqa.suse.de from 7.0T to 10.0T which should reside on “cheap+slow” storage, i.e. rotating disks.

   Reasoning: More and more products are tested by more and more development groups hence demanding more space.

   Related internal issue: https://progress.opensuse.org/issues/119215

Thanks for your support @okurz. I hope the next build which will be released on 2022-10-27 has full media for aarch64. Otherwise it impacts 15-SP5 final alpha QA full run.

As alternative you could trigger a resync and trigger openQA tests but this would be multiple hundreds of jobs. What could also work is to manually extract the repo folder and retrigger only the relevant tests

okurz wrote:

https://sd.suse.com/servicedesk/customer/portal/1/SD-102307 resolved. I increased the partition and filesystem

okurz@openqa:~> sudo parted -s -a opt /dev/vdc resizepart 1 100%
Warning: Partition /dev/vdc is being used, continuing anyway.
okurz@openqa:~> sudo xfs_growfs /dev/vdc
meta-data=/dev/vdc isize=512 agcount=7, agsize=268435455 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=1879048185, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 1879048185 to 2684354550
okurz@openqa:~> df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 8.0K 4.0M 1% /dev
tmpfs 12G 3.6M 12G 1% /dev/shm
tmpfs 4.7G 1.2M 4.7G 1% /run
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
/dev/vda1 20G 6.8G 12G 37% /
/dev/vdb 200G 58G 143G 29% /srv
/dev/vde 5.5T 3.2T 2.4T 58% /space-slow
/dev/vdc 10T 5.6T 4.5T 56% /assets
/dev/loop0 4.9G 1.3G 3.4G 27% /home
/dev/vdd 5.5T 3.9T 1.6T 71% /results
tmpfs 2.0G 0 2.0G 0% /run/user/1001
tmpfs 2.0G 0 2.0G 0% /run/user/0
tmpfs 2.0G 0 2.0G 0% /run/user/17254

I bumped up some asset quota limits in job groups. That should be more than enough space for now.

Thanks for your timely support.