Project

General

Profile

Actions
Copy link

action #131147

closed

Reduce /assets usage on o3

Added by okurz 11 months ago. Updated 11 months ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-06-20
Due date:
% Done:

0%

Estimated time:
Tags:
infra

Description

Motivation

From https://mailman.suse.de/mlarch/SuSE/o3-admins/2023/o3-admins.2023.06/msg00042.html : We have received an alert message by munin about /assets on o3 being 92% full. I think 92% (right now even increased to 93% on o3) is alarming and we should have been noticed about from zabbix where likely the old alarm thresholds were not migrated. We should look into the overly full /assets and reduce the usage as well as ensure that there is sufficient alerting. We could go with munin but I guess for something as low-level as disk usage zabbix should be easy enough to use.

Acceptance criteria

  • AC1: /assets is used up to 80%

Related issues 4 (1 open — 3 closed)

Related to openQA Infrastructure - action #131021: [O3 repo]Missing openSUSE-Tumbleweed-oss-x86_64-CURRENT directory in /var/lib/openqa/share/factory/repo size:MResolvedmkittler2023-06-162023-07-05

Actions
Related to openQA Infrastructure - action #121594: Extend OSD storage space for "results" to make bug investigation and failure archeology easier - 2022Resolvedokurz

Actions
Related to openQA Infrastructure - action #120064: SSD SAN storage (NetApp) for O3 would improve user experience (can not just buy drives on our own)New2022-11-08

Actions
Copied to openQA Infrastructure - action #131150: Add alarms for partition usage on o3 size:MResolvedlivdywan2023-06-20

Actions
Actions
Copy link
 #1

Updated by okurz 11 months ago

  • Subject changed from Add alarms for partition usage on o3 to Reduce /assets usage on o3 + add alarms for partition usage on o3
  • Priority changed from Normal to High
Actions
Copy link
 #2

Updated by okurz 11 months ago

  • Related to action #131021: [O3 repo]Missing openSUSE-Tumbleweed-oss-x86_64-CURRENT directory in /var/lib/openqa/share/factory/repo size:M added
Actions
Copy link
 #3

Updated by okurz 11 months ago

  • Related to action #121594: Extend OSD storage space for "results" to make bug investigation and failure archeology easier - 2022 added
Actions
Copy link
 #4

Updated by okurz 11 months ago

  • Copied to action #131150: Add alarms for partition usage on o3 size:M added
Actions
Copy link
 #5

Updated by okurz 11 months ago

  • Related to action #120064: SSD SAN storage (NetApp) for O3 would improve user experience (can not just buy drives on our own) added
Actions
Copy link
 #6

Updated by okurz 11 months ago

  • Subject changed from Reduce /assets usage on o3 + add alarms for partition usage on o3 to Reduce /assets usage on o3
  • Status changed from New to In Progress
  • Assignee set to okurz

I split out the "alarm improvement ticket" in #131150, focussing on cleanup.

I reduced asset quotas:

On o3 in /etc/fstab I added comments explaining where the storage resides.

I found in #120064 that vdd on o3 was increased in size 5.0->6.6TB but the filesystem never adjusted.

Actions
Copy link
 #7

Updated by openqa_review 11 months ago

  • Due date set to 2023-07-05

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #8

Updated by okurz 11 months ago

  • Status changed from In Progress to Feedback
pvresize /dev/vdd
lvresize --resizefs -l +100%FREE /dev/vg0/assets

so df -h shows

/dev/vda1                 20G   12G  7.7G  60% /
tmpfs                    5.0G  488K  5.0G   1% /tmp
/dev/vdc                 100G   20G   81G  20% /var/lib/pgsql
/dev/vdb1                5.0T  3.4T  1.7T  68% /space
/dev/mapper/vg0-archive  1.0T  3.7G 1020G   1% /archive
/dev/loop0               4.9G  3.6G  1.1G  77% /home
/dev/mapper/vg0-assets   5.6T  3.8T  1.9T  68% /assets

will monitor over 24h to see the trend of usage.

Actions
Copy link
 #9

Updated by okurz 11 months ago

  • Description updated (diff)
Actions
Copy link
 #10

Updated by okurz 11 months ago

  • Due date deleted (2023-07-05)
  • Status changed from Feedback to Resolved
/dev/mapper/vg0-assets   5.6T  4.5T  1.2T  80% /assets

so exactly at the configured limit for the space aware cleanup.

Actions
Copy link

Also available in: Atom PDF