action #71383
closed[u] test fails in bootloader_zkvm - s390pb.suse.de ran out of inodes
0%
Description
Observation¶
openQA test in scenario sle-15-SP3-Online-s390x-create_hdd_gnome@foursixnine/os-autoinst-distri-opensuse#oopsitsbrokenagain@zkvm fails in
bootloader_zkvm
We seem to have hit the max number of files/inodes in the / partition of that machine
touch /tmp/test; df -ah /tmp && df -i /tmp
touch: cannot touch ‘/tmp/test’: No space left on device
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/zkvm1-root 6.2G 4.6G 1.3G 79% /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/zkvm1-root 417792 417792 0 100% /
So openQA jobs can't run on it for now...
Updated by szarate over 3 years ago
So I have moved /var/lib/libvirt/* to the partition where /var/lib/libvirt/images were before, so we end up with few inodes left and system can perform operations like yum clean etc... however, still some files need to be deleted or moved over, to regain more reliability Looks at kernel sources
[root@s390pb data]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/zkvm1-root 417792 417767 25 100% /
Updated by szarate over 3 years ago
- Status changed from In Progress to Resolved
Found the actual culprit: tons of messages waiting in /var/spool/clientmqueue deleted few, started sendmail, and now works like charm
https://openqa.suse.de/tests/4691122
Also enabled it, just in case.
Updated by szarate over 3 years ago
- Precedes action #70936: [SLE][Migration][SLE15SP3] test fails in reconnect_mgmt_console - Test died: Error connecting to VNC server <s390qa101.qa.suse.de:5901>: IO::Socket::INET: connect: Connection timed out added