Project

General

Profile

Actions
Copy link

action #174313

open

[o3][zabbix][alert] / and /var/tmp: "Disk space is low and might be full in 7d (used > 85%)" since 2024-12-11 06:50 size:S

Added by okurz 7 days ago. Updated about 3 hours ago.

Status:
In Progress
Priority:
High
Assignee:
mkittler
Category:
Regressions/Crashes
Target version:
openQA Project (public) - Ready
Start date:
Due date:
2025-01-02 (Due in 14 days)
% Done:

0%

Estimated time:
Tags:
infra, reactive work

Description

Observation

From https://zabbix.nue.suse.com/zabbix.php?show=1&name=&inventory%5B0%5D%5Bfield%5D=type&inventory%5B0%5D%5Bvalue%5D=&evaltype=0&tags%5B0%5D%5Btag%5D=&tags%5B0%5D%5Boperator%5D=0&tags%5B0%5D%5Bvalue%5D=&show_tags=3&tag_name_format=0&tag_priority=&show_opdata=0&show_timeline=1&filter_name=&filter_show_counter=0&filter_custom_time=0&sort=clock&sortorder=DESC&age_state=0&show_suppressed=0&unacknowledged=0&compact_view=0&details=0&highlight_row=0&action=problem.view

2024-12-11 06:50:26                                Warning                PROBLEM                ariel.dmz-prg2.suse.org        /var/tmp: Disk space is low and might be full in 7d (used > 85%)        1d 9h 40m        No                Application: Filesystem /var/tmp
2024-12-11 06:50:23                                Warning                PROBLEM                ariel.dmz-prg2.suse.org        /: Disk space is low and might be full in 7d (used > 85%)        1d 9h 40m        No                Application: Filesystem /

Suggestions

  • we're keeping a long list of old packages in /var/cache/zypp/packages/. It goes back to february 2023
  • Research if zypper can provide such options, otherwise add a custom systemd service or extend openqa-auto-update to remove older cached packages based on number and/or age
  • Ensure that this frees up enough space and crosscheck the alert on zabbix again

Related issues 2 (1 open — 1 closed)

Related to openQA Infrastructure (public) - action #40196: [monitoring] monitor internal port 9526, port 80, external port 443 accessibility of o3 and response times size:MResolvedokurz2018-08-23

Actions
Copied to openQA Infrastructure (public) - action #174316: [o3][zabbix][alert] warning about depleting storage space but no email? size:SWorkable2024-12-12

Actions
Actions
Copy link
 #1

Updated by okurz 7 days ago

  • Copied to action #174316: [o3][zabbix][alert] warning about depleting storage space but no email? size:S added
Actions
Copy link
 #2

Updated by okurz 6 days ago

  • Related to action #40196: [monitoring] monitor internal port 9526, port 80, external port 443 accessibility of o3 and response times size:M added
Actions
Copy link
 #3

Updated by gpathak 3 days ago

  • Assignee set to gpathak

The /var directory is taking up 11GiB.

gpathak@ariel:~> sudo du -ahcx /var/ | sort -hr | head
11G /var/
11G total
8.5G    /var/cache
8.4G    /var/cache/zypp
8.3G    /var/cache/zypp/packages
7.7G    /var/cache/zypp/packages/devel_openQA
5.3G    /var/cache/zypp/packages/devel_openQA/x86_64
2.4G    /var/cache/zypp/packages/devel_openQA/noarch
1.3G    /var/log
910M    /var/log/journal/06446c641307496183dfdf8dccebdceb
gpathak@ariel:~>

The /var/log is 1.3GiB

Actions
Copy link
 #4

Updated by gpathak 3 days ago

Actions
Copy link
 #5

Updated by gpathak 3 days ago

  • Assignee deleted (gpathak)
Actions
Copy link
 #6

Updated by tinita 2 days ago

It seems we're keeping a long list of old packages in /var/cache/zypp/packages/. It goes back to february 2023:

ls -lrth /var/cache/zypp/packages/devel_openQA/x86_64/openQA-common-*                                                                                                                             
-rw-r--r-- 1 root root 459K Feb 15  2023 /var/cache/zypp/packages/devel_openQA/x86_64/openQA-common-4.6.1676474487.945e502-lp154.5577.1.x86_64.rpm

Not sure how to configure this to a lower duration.

Actions
Copy link
 #7

Updated by okurz 1 day ago

  • Subject changed from [o3][zabbix][alert] / and /var/tmp: "Disk space is low and might be full in 7d (used > 85%)" since 2024-12-11 06:50 to [o3][zabbix][alert] / and /var/tmp: "Disk space is low and might be full in 7d (used > 85%)" since 2024-12-11 06:50 size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #8

Updated by mkittler about 15 hours ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions
Copy link
 #9

Updated by mkittler about 15 hours ago · Edited

We're keeping the packages of the following repos indefinitely:

grep -iR keeppackages=1 /etc/zypp/repos.d
/etc/zypp/repos.d/devel_openQA.repo:keeppackages=1
/etc/zypp/repos.d/devel_openQA_Leap.repo:keeppackages=1

Not sure whether zypper has a way of specifying the number of packages to keep. For now I just used `find /var/cache/zypp/packages -ipath 'devel_openqa' -mtime +365 -delete´ to delete everything older than a year.

I can setup a systemd service/timer to invoke a command like that periodically. I can also set keeppackages=0 but we probably enabled this for the sake of easier downgrades. So this is probably not a good solution.

One could also add the following to openqa-auto-update:

if [[ $OPENQA_PACKAGE_CACHE_RETENTION ]]; then
    find /var/cache/zypp/packages -type f -ipath '*devel*openQA*' -mtime "+$OPENQA_PACKAGE_CACHE_RETENTION" -delete
fi

Of course this breaks if one uses a different repository name or a different packagesdir. So it is probably not the best idea to add it to the generic openqa-auto-update script.

Actions
Copy link
 #10

Updated by openqa_review about 3 hours ago

  • Due date set to 2025-01-02

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link

Also available in: Atom PDF