action #131147
closed
Reduce /assets usage on o3
Added by okurz over 1 year ago.
Updated over 1 year ago.
Description
Motivation¶
From https://mailman.suse.de/mlarch/SuSE/o3-admins/2023/o3-admins.2023.06/msg00042.html : We have received an alert message by munin about /assets on o3 being 92% full. I think 92% (right now even increased to 93% on o3) is alarming and we should have been noticed about from zabbix where likely the old alarm thresholds were not migrated. We should look into the overly full /assets and reduce the usage as well as ensure that there is sufficient alerting. We could go with munin but I guess for something as low-level as disk usage zabbix should be easy enough to use.
Acceptance criteria¶
- AC1: /assets is used up to 80%
- Subject changed from Add alarms for partition usage on o3 to Reduce /assets usage on o3 + add alarms for partition usage on o3
- Priority changed from Normal to High
- Related to action #131021: [O3 repo]Missing openSUSE-Tumbleweed-oss-x86_64-CURRENT directory in /var/lib/openqa/share/factory/repo size:M added
- Related to action #121594: Extend OSD storage space for "results" to make bug investigation and failure archeology easier - 2022 added
- Copied to action #131150: Add alarms for partition usage on o3 size:M added
- Related to action #120064: SSD SAN storage (NetApp) for O3 would improve user experience (can not just buy drives on our own) added
- Subject changed from Reduce /assets usage on o3 + add alarms for partition usage on o3 to Reduce /assets usage on o3
- Status changed from New to In Progress
- Assignee set to okurz
I split out the "alarm improvement ticket" in #131150, focussing on cleanup.
I reduced asset quotas:
On o3 in /etc/fstab I added comments explaining where the storage resides.
I found in #120064 that vdd on o3 was increased in size 5.0->6.6TB but the filesystem never adjusted.
- Due date set to 2023-07-05
Setting due date based on mean cycle time of SUSE QE Tools
- Status changed from In Progress to Feedback
pvresize /dev/vdd
lvresize --resizefs -l +100%FREE /dev/vg0/assets
so df -h shows
/dev/vda1 20G 12G 7.7G 60% /
tmpfs 5.0G 488K 5.0G 1% /tmp
/dev/vdc 100G 20G 81G 20% /var/lib/pgsql
/dev/vdb1 5.0T 3.4T 1.7T 68% /space
/dev/mapper/vg0-archive 1.0T 3.7G 1020G 1% /archive
/dev/loop0 4.9G 3.6G 1.1G 77% /home
/dev/mapper/vg0-assets 5.6T 3.8T 1.9T 68% /assets
will monitor over 24h to see the trend of usage.
- Description updated (diff)
- Due date deleted (
2023-07-05)
- Status changed from Feedback to Resolved
/dev/mapper/vg0-assets 5.6T 4.5T 1.2T 80% /assets
so exactly at the configured limit for the space aware cleanup.
Also available in: Atom
PDF