Project

General

Profile

Actions

action #119215

closed

[openQA][repos][aarch64][15-SP5[Full Media] 15-SP5 Full media is missing from http://openqa.suse.de/assets/repo/

Added by waynechen55 about 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
Start date:
2022-10-24
Due date:
% Done:

0%

Estimated time:

Description

Observation

There is no 15-SP5 full media for aarch64 on website http://openqa.suse.de/assets/repo/. I do not think there is something still under building at the moment on build.suse.de for Build29.1.
Only aarch64 full media is missing from openQA website:

Steps to reproduce

Impact

Any test suite using aarch64 full media will fail.

Problem

It seems there is some thing wrong with openQA infrastructure

Suggestion

  • Check why full media build for aarch64 is missing

Workaround

n/a


Files

Selection_042.png (50.6 KB) Selection_042.png waynechen55, 2022-10-24 01:09

Related issues 2 (0 open2 closed)

Related to openQA Infrastructure - action #73183: Extend vdc (/assets)Resolvedokurz2020-10-092020-10-30

Actions
Related to openQA Infrastructure - action #71575: [osd][alert] limited /assets - idea: ask EngInfra for slow+cheap storage from central server for /assets/fixed onlyResolvedmkittler2020-09-02

Actions
Actions #1

Updated by okurz about 2 years ago

  • Target version set to Ready
Actions #2

Updated by okurz about 2 years ago

  • Status changed from New to In Progress
  • Assignee set to okurz
Actions #3

Updated by okurz about 2 years ago

All important results should be visible from https://openqa.suse.de/admin/obs_rsync . Following that link I find https://openqa.suse.de/admin/obs_rsync/SUSE:SLE-15-SP5:GA:TEST%7Cbase/runs/.run_221019_103656_29.1 for build 29.1. https://openqa.suse.de/admin/obs_rsync/SUSE:SLE-15-SP5:GA:TEST%7Cbase/runs/.run_221019_103656_29.1/download/files_iso.lst lists the full medium for all four architectures. The full "repository" is extracted on the openQA host itself from the ISO file, visible in:
https://openqa.suse.de/admin/obs_rsync/SUSE:SLE-15-SP5:GA:TEST%7Cbase/runs/.run_221019_103656_29.1/download/rsync_iso.cmd

rsync --timeout=3600 -tlp4 --specials dist.suse.de::repos/SUSE:/SLE-15-SP5:/GA:/TEST/images/iso/*SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso /var/lib/openqa/factory/iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso
rsync --timeout=3600 -tlp4 --specials dist.suse.de::repos/SUSE:/SLE-15-SP5:/GA:/TEST/images/iso/*SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 /var/lib/openqa/factory/other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256
[ -d /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 ] || {
    mkdir /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
    bsdtar xf /var/lib/openqa/factory/iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso -C /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
}

with the logs in https://openqa.suse.de/admin/obs_rsync/SUSE:SLE-15-SP5:GA:TEST%7Cbase/runs/.run_221019_103656_29.1/download/rsync_iso.cmd.log

+ rsync --timeout=3600 -tlp4 --specials 'dist.suse.de::repos/SUSE:/SLE-15-SP5:/GA:/TEST/images/iso/*SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso' /var/lib/openqa/factory/iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso
+ rsync --timeout=3600 -tlp4 --specials 'dist.suse.de::repos/SUSE:/SLE-15-SP5:/GA:/TEST/images/iso/*SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256' /var/lib/openqa/factory/other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256
+ '[' -d /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 ']'
+ mkdir /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
+ bsdtar xf /var/lib/openqa/factory/iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso -C /var/lib/openqa/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1

so it looks like the repo was created. Who can remind me where we can find log messages regarding the cleanup of assets?

Actions #4

Updated by okurz about 2 years ago

  • Due date set to 2022-11-11
  • Status changed from In Progress to Feedback

I found on OSD

openqa:/var/log # xzgrep -i SLE-15-SP5-Full-aarch64-Build29 openqa_gru.8.xz
[2022-10-20T04:08:31.562809+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 110 (20620640697)
[2022-10-20T04:08:31.562875+02:00] [debug] [pid:31541] Asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) picked into group 110
[2022-10-20T04:08:31.562928+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 129 (11005337889)
[2022-10-20T04:08:31.563014+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 143 (163476307)
[2022-10-20T04:08:31.563077+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 251 (3067777832)
[2022-10-20T04:08:31.563148+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 265 (400818176)
[2022-10-20T04:08:31.563241+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 267 (5771211776)
[2022-10-20T04:08:31.563302+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 301 (3321888768)
[2022-10-20T04:08:31.563357+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 452 (46858240)
[2022-10-20T04:08:31.563710+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 110 (10068957625)
[2022-10-20T04:08:31.563781+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 129 (11005337889)
[2022-10-20T04:08:31.563840+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 143 (163476307)
[2022-10-20T04:08:31.563910+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 267 (5771211776)
[2022-10-20T04:08:31.563980+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 452 (46858240)
[2022-10-20T04:08:33.396420+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 110 (1590165945)
[2022-10-20T04:08:33.396470+02:00] [debug] [pid:31541] Asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) picked into group 110
[2022-10-20T04:08:33.396522+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 129 (445919993)
[2022-10-20T04:08:33.396573+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 143 (17576275)
[2022-10-20T04:08:33.396623+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 452 (46196224)
                 name        => "iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso",
                 name        => "repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1",
                 name        => "other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256",
[2022-10-20T04:18:45.478011+02:00] [info] [pid:31541] Removing asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (belonging to job groups: 129, 143, 452, 267, 110 within parent job groups 15)
[2022-10-20T04:19:33.121798+02:00] [info] [pid:31541] GRU: removed /var/lib/openqa/share/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
[2022-10-20T04:28:59.262163+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 110 (20609593380)
[2022-10-20T04:28:59.262233+02:00] [debug] [pid:31541] Asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) picked into group 110
[2022-10-20T04:28:59.262310+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 129 (11005337889)
[2022-10-20T04:28:59.262407+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 143 (163476307)
[2022-10-20T04:28:59.262494+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 251 (3067777832)
[2022-10-20T04:28:59.262572+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 265 (400818176)
[2022-10-20T04:28:59.262643+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 267 (5771211776)
[2022-10-20T04:28:59.263746+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 301 (3321888768)
[2022-10-20T04:28:59.263840+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 452 (46858240)
[2022-10-20T04:29:00.041376+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 110 (1579118628)
[2022-10-20T04:29:00.041454+02:00] [debug] [pid:31541] Asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) picked into group 110
[2022-10-20T04:29:00.041539+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 129 (445919993)
[2022-10-20T04:29:00.041624+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 143 (17576275)
[2022-10-20T04:29:00.041709+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 452 (46196224)

in particular:

[2022-10-20T04:18:45.478011+02:00] [info] [pid:31541] Removing asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (belonging to job groups: 129, 143, 452, 267, 110 within parent job groups 15)
[2022-10-20T04:19:33.121798+02:00] [info] [pid:31541] GRU: removed /var/lib/openqa/share/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1

so the asset was simply removed because it does not fit anymore.

@waynechen55 Please understand how the asset management on openQA works:
http://open.qa/docs/#_asset_handling

Based on available space I could extend the size limit for assets in the functional job group (id 110) from 260GB to 400GB which should allow to save all four Full repos though still taking a lot of space.

The next build should again have a full aarch64 repo.

Further information: https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&viewPanel=74&from=now-2y&to=now shows the evolution of asset space usage over the course of the past two years. Within https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&viewPanel=74&from=now-7d&to=now we see how the space usage goes down to 74% corresponding to (0.74 * 5.0T = 5.18TB) until we reach the limit of 80% (0.80 * 5.0T = 5.60TB) when we trigger the space-aware cleanup. So effectively we have a buffer of 5.60TB - 5.18TB = 0.420 ~ 420GB which we can still assign to groups.

If you need much more space for assets on openqa.suse.de please organize with all relevant user groups to formulate such request to SUSE-IT.

Actions #5

Updated by okurz about 2 years ago

Actions #6

Updated by okurz about 2 years ago

  • Related to action #71575: [osd][alert] limited /assets - idea: ask EngInfra for slow+cheap storage from central server for /assets/fixed only added
Actions #7

Updated by okurz about 2 years ago

I opened https://sd.suse.com/servicedesk/customer/portal/1/SD-102307

Kind request to extend vdc assigned to VM openqa.suse.de from 7.0T to 10.0T Waiting for support

   This is a kind request to extend vdc assigned to VM openqa.suse.de from 7.0T to 10.0T which should reside on “cheap+slow” storage, i.e. rotating disks.

   Reasoning: More and more products are tested by more and more development groups hence demanding more space.

   Related internal issue: https://progress.opensuse.org/issues/119215
Actions #8

Updated by waynechen55 about 2 years ago

okurz wrote:

I found on OSD

openqa:/var/log # xzgrep -i SLE-15-SP5-Full-aarch64-Build29 openqa_gru.8.xz
[2022-10-20T04:08:31.562809+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 110 (20620640697)
[2022-10-20T04:08:31.562875+02:00] [debug] [pid:31541] Asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) picked into group 110
[2022-10-20T04:08:31.562928+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 129 (11005337889)
[2022-10-20T04:08:31.563014+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 143 (163476307)
[2022-10-20T04:08:31.563077+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 251 (3067777832)
[2022-10-20T04:08:31.563148+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 265 (400818176)
[2022-10-20T04:08:31.563241+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 267 (5771211776)
[2022-10-20T04:08:31.563302+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 301 (3321888768)
[2022-10-20T04:08:31.563357+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 452 (46858240)
[2022-10-20T04:08:31.563710+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 110 (10068957625)
[2022-10-20T04:08:31.563781+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 129 (11005337889)
[2022-10-20T04:08:31.563840+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 143 (163476307)
[2022-10-20T04:08:31.563910+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 267 (5771211776)
[2022-10-20T04:08:31.563980+02:00] [debug] [pid:31541] Checking whether asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (12108819405) fits into group 452 (46858240)
[2022-10-20T04:08:33.396420+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 110 (1590165945)
[2022-10-20T04:08:33.396470+02:00] [debug] [pid:31541] Asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) picked into group 110
[2022-10-20T04:08:33.396522+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 129 (445919993)
[2022-10-20T04:08:33.396573+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 143 (17576275)
[2022-10-20T04:08:33.396623+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 452 (46196224)
                 name        => "iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso",
                 name        => "repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1",
                 name        => "other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256",
[2022-10-20T04:18:45.478011+02:00] [info] [pid:31541] Removing asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (belonging to job groups: 129, 143, 452, 267, 110 within parent job groups 15)
[2022-10-20T04:19:33.121798+02:00] [info] [pid:31541] GRU: removed /var/lib/openqa/share/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1
[2022-10-20T04:28:59.262163+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 110 (20609593380)
[2022-10-20T04:28:59.262233+02:00] [debug] [pid:31541] Asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) picked into group 110
[2022-10-20T04:28:59.262310+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 129 (11005337889)
[2022-10-20T04:28:59.262407+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 143 (163476307)
[2022-10-20T04:28:59.262494+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 251 (3067777832)
[2022-10-20T04:28:59.262572+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 265 (400818176)
[2022-10-20T04:28:59.262643+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 267 (5771211776)
[2022-10-20T04:28:59.263746+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 301 (3321888768)
[2022-10-20T04:28:59.263840+02:00] [debug] [pid:31541] Checking whether asset iso/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso (10551683072) fits into group 452 (46858240)
[2022-10-20T04:29:00.041376+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 110 (1579118628)
[2022-10-20T04:29:00.041454+02:00] [debug] [pid:31541] Asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) picked into group 110
[2022-10-20T04:29:00.041539+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 129 (445919993)
[2022-10-20T04:29:00.041624+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 143 (17576275)
[2022-10-20T04:29:00.041709+02:00] [debug] [pid:31541] Checking whether asset other/SLE-15-SP5-Full-aarch64-Build29.1-Media1.iso.sha256 (111) fits into group 452 (46196224)

in particular:

[2022-10-20T04:18:45.478011+02:00] [info] [pid:31541] Removing asset repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1 (belonging to job groups: 129, 143, 452, 267, 110 within parent job groups 15)
[2022-10-20T04:19:33.121798+02:00] [info] [pid:31541] GRU: removed /var/lib/openqa/share/factory/repo/SLE-15-SP5-Full-aarch64-Build29.1-Media1

so the asset was simply removed because it does not fit anymore.

@waynechen55 Please understand how the asset management on openQA works:
http://open.qa/docs/#_asset_handling

Based on available space I could extend the size limit for assets in the functional job group (id 110) from 260GB to 400GB which should allow to save all four Full repos though still taking a lot of space.

The next build should again have a full aarch64 repo.

Further information: https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&viewPanel=74&from=now-2y&to=now shows the evolution of asset space usage over the course of the past two years. Within https://monitor.qa.suse.de/d/WebuiDb/webui-summary?orgId=1&viewPanel=74&from=now-7d&to=now we see how the space usage goes down to 74% corresponding to (0.74 * 5.0T = 5.18TB) until we reach the limit of 80% (0.80 * 5.0T = 5.60TB) when we trigger the space-aware cleanup. So effectively we have a buffer of 5.60TB - 5.18TB = 0.420 ~ 420GB which we can still assign to groups.

If you need much more space for assets on openqa.suse.de please organize with all relevant user groups to formulate such request to SUSE-IT.

I can not have no 15-SP5 aarch64 Builds for testing. It is not something can be sacrificed.

Actions #9

Updated by waynechen55 about 2 years ago

okurz wrote:

I opened https://sd.suse.com/servicedesk/customer/portal/1/SD-102307

Kind request to extend vdc assigned to VM openqa.suse.de from 7.0T to 10.0T Waiting for support

   This is a kind request to extend vdc assigned to VM openqa.suse.de from 7.0T to 10.0T which should reside on “cheap+slow” storage, i.e. rotating disks.

   Reasoning: More and more products are tested by more and more development groups hence demanding more space.

   Related internal issue: https://progress.opensuse.org/issues/119215

Thanks for your support @okurz. I hope the next build which will be released on 2022-10-27 has full media for aarch64. Otherwise it impacts 15-SP5 final alpha QA full run.

Actions #10

Updated by okurz about 2 years ago

As alternative you could trigger a resync and trigger openQA tests but this would be multiple hundreds of jobs. What could also work is to manually extract the repo folder and retrigger only the relevant tests

Actions #11

Updated by okurz about 2 years ago

  • Due date deleted (2022-11-11)
  • Status changed from Feedback to Resolved

https://sd.suse.com/servicedesk/customer/portal/1/SD-102307 resolved. I increased the partition and filesystem

okurz@openqa:~> sudo parted -s -a opt /dev/vdc resizepart 1 100%
Warning: Partition /dev/vdc is being used, continuing anyway.
okurz@openqa:~> sudo xfs_growfs /dev/vdc
meta-data=/dev/vdc isize=512 agcount=7, agsize=268435455 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=1879048185, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 1879048185 to 2684354550
okurz@openqa:~> df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 8.0K 4.0M 1% /dev
tmpfs 12G 3.6M 12G 1% /dev/shm
tmpfs 4.7G 1.2M 4.7G 1% /run
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
/dev/vda1 20G 6.8G 12G 37% /
/dev/vdb 200G 58G 143G 29% /srv
/dev/vde 5.5T 3.2T 2.4T 58% /space-slow
/dev/vdc 10T 5.6T 4.5T 56% /assets
/dev/loop0 4.9G 1.3G 3.4G 27% /home
/dev/vdd 5.5T 3.9T 1.6T 71% /results
tmpfs 2.0G 0 2.0G 0% /run/user/1001
tmpfs 2.0G 0 2.0G 0% /run/user/0
tmpfs 2.0G 0 2.0G 0% /run/user/17254

I bumped up some asset quota limits in job groups. That should be more than enough space for now.

Actions #12

Updated by waynechen55 about 2 years ago

okurz wrote:

https://sd.suse.com/servicedesk/customer/portal/1/SD-102307 resolved. I increased the partition and filesystem

okurz@openqa:~> sudo parted -s -a opt /dev/vdc resizepart 1 100%
Warning: Partition /dev/vdc is being used, continuing anyway.
okurz@openqa:~> sudo xfs_growfs /dev/vdc
meta-data=/dev/vdc isize=512 agcount=7, agsize=268435455 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0
= reflink=1 bigtime=0 inobtcount=0
data = bsize=4096 blocks=1879048185, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 1879048185 to 2684354550
okurz@openqa:~> df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.0M 8.0K 4.0M 1% /dev
tmpfs 12G 3.6M 12G 1% /dev/shm
tmpfs 4.7G 1.2M 4.7G 1% /run
tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup
/dev/vda1 20G 6.8G 12G 37% /
/dev/vdb 200G 58G 143G 29% /srv
/dev/vde 5.5T 3.2T 2.4T 58% /space-slow
/dev/vdc 10T 5.6T 4.5T 56% /assets
/dev/loop0 4.9G 1.3G 3.4G 27% /home
/dev/vdd 5.5T 3.9T 1.6T 71% /results
tmpfs 2.0G 0 2.0G 0% /run/user/1001
tmpfs 2.0G 0 2.0G 0% /run/user/0
tmpfs 2.0G 0 2.0G 0% /run/user/17254

I bumped up some asset quota limits in job groups. That should be more than enough space for now.

Thanks for your timely support.

Actions

Also available in: Atom PDF