Project

General

Profile

Actions

action #131147

closed

Reduce /assets usage on o3

Added by okurz 12 months ago. Updated 12 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2023-06-20
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Motivation

From https://mailman.suse.de/mlarch/SuSE/o3-admins/2023/o3-admins.2023.06/msg00042.html : We have received an alert message by munin about /assets on o3 being 92% full. I think 92% (right now even increased to 93% on o3) is alarming and we should have been noticed about from zabbix where likely the old alarm thresholds were not migrated. We should look into the overly full /assets and reduce the usage as well as ensure that there is sufficient alerting. We could go with munin but I guess for something as low-level as disk usage zabbix should be easy enough to use.

Acceptance criteria

  • AC1: /assets is used up to 80%

Related issues 4 (1 open3 closed)

Related to openQA Infrastructure - action #131021: [O3 repo]Missing openSUSE-Tumbleweed-oss-x86_64-CURRENT directory in /var/lib/openqa/share/factory/repo size:MResolvedmkittler2023-06-162023-07-05

Actions
Related to openQA Infrastructure - action #121594: Extend OSD storage space for "results" to make bug investigation and failure archeology easier - 2022Resolvedokurz

Actions
Related to openQA Infrastructure - action #120064: SSD SAN storage (NetApp) for O3 would improve user experience (can not just buy drives on our own)New2022-11-08

Actions
Copied to openQA Infrastructure - action #131150: Add alarms for partition usage on o3 size:MResolvedlivdywan2023-06-20

Actions
Actions #1

Updated by okurz 12 months ago

  • Subject changed from Add alarms for partition usage on o3 to Reduce /assets usage on o3 + add alarms for partition usage on o3
  • Priority changed from Normal to High
Actions #2

Updated by okurz 12 months ago

  • Related to action #131021: [O3 repo]Missing openSUSE-Tumbleweed-oss-x86_64-CURRENT directory in /var/lib/openqa/share/factory/repo size:M added
Actions #3

Updated by okurz 12 months ago

  • Related to action #121594: Extend OSD storage space for "results" to make bug investigation and failure archeology easier - 2022 added
Actions #4

Updated by okurz 12 months ago

  • Copied to action #131150: Add alarms for partition usage on o3 size:M added
Actions #5

Updated by okurz 12 months ago

  • Related to action #120064: SSD SAN storage (NetApp) for O3 would improve user experience (can not just buy drives on our own) added
Actions #6

Updated by okurz 12 months ago

  • Subject changed from Reduce /assets usage on o3 + add alarms for partition usage on o3 to Reduce /assets usage on o3
  • Status changed from New to In Progress
  • Assignee set to okurz

I split out the "alarm improvement ticket" in #131150, focussing on cleanup.

I reduced asset quotas:

On o3 in /etc/fstab I added comments explaining where the storage resides.

I found in #120064 that vdd on o3 was increased in size 5.0->6.6TB but the filesystem never adjusted.

Actions #7

Updated by openqa_review 12 months ago

  • Due date set to 2023-07-05

Setting due date based on mean cycle time of SUSE QE Tools

Actions #8

Updated by okurz 12 months ago

  • Status changed from In Progress to Feedback
pvresize /dev/vdd
lvresize --resizefs -l +100%FREE /dev/vg0/assets

so df -h shows

/dev/vda1                 20G   12G  7.7G  60% /
tmpfs                    5.0G  488K  5.0G   1% /tmp
/dev/vdc                 100G   20G   81G  20% /var/lib/pgsql
/dev/vdb1                5.0T  3.4T  1.7T  68% /space
/dev/mapper/vg0-archive  1.0T  3.7G 1020G   1% /archive
/dev/loop0               4.9G  3.6G  1.1G  77% /home
/dev/mapper/vg0-assets   5.6T  3.8T  1.9T  68% /assets

will monitor over 24h to see the trend of usage.

Actions #9

Updated by okurz 12 months ago

  • Description updated (diff)
Actions #10

Updated by okurz 12 months ago

  • Due date deleted (2023-07-05)
  • Status changed from Feedback to Resolved
/dev/mapper/vg0-assets   5.6T  4.5T  1.2T  80% /assets

so exactly at the configured limit for the space aware cleanup.

Actions

Also available in: Atom PDF