action #45260
closed
Added by okurz over 5 years ago.
Updated over 5 years ago.
Description
Observation¶
Received a monitoring alert "** PROBLEM Service Alert: openqa.suse.de/fs_/ is CRITICAL **" at 20:16
Info: CRIT - 100.0% used (9.57 of 9.57 GB), trend: +2.52 GB / 24 hours
meaning that something caused the root drive to run rapidly out of space during today. journalctl --since=today reveals the first obvious entry at 19:30
Dec 17 19:30:01 openqa postfix/cleanup[29785]: C682126CD1: message-id=<20181217183001.C682126CD1@linux.site>
okurz@openqa:/> sudo du --one-file-system --max-depth=1 --block-size=M | sort -n
1M ./lost+found
1M ./mnt
1M ./selinux
5M ./bin
7M ./sbin
17M ./lib64
23M ./opt
26M ./etc
74M ./boot
163M ./root
417M ./var
669M ./lib
679M ./tmp
2965M ./usr
4722M ./home
9760M .
Pretty sure I will find something in home that went mad ;)
- Status changed from New to Resolved
- Assignee set to okurz
okurz@openqa:/home/acarvajal> ls -ltrah
total 3.7G
…
-rw-r--r-- 1 root root 1.2G Dec 17 19:14 sle-12-SP4-ppc64le-ha-alpha-alpha-node01.qcow2
-rw-r--r-- 1 root root 1.2G Dec 17 19:14 sle-12-SP4-ppc64le-ha-alpha-alpha-node02.qcow2
-rw-r--r-- 1 root root 1.1G Dec 17 19:28 ha_supportserver_upgrade_sle_12-SP4_ppc64le.qcow2
-rw-r--r-- 1 root root 228M Dec 17 19:29 ha_supportserver_upgrade_sle_12-SP4_ppc64le_luns.qcow2
I moved these with sudo mv *.qcow2 /var/lib/openqa/share/factory/hdd/
and I think by this resolved the urgent part of the issue :)
- Copied to action #45263: osd 100% disk usage on / -> moved files to /var/lib/openqa/share/factory added
Also available in: Atom
PDF