[Alerting] File systems alert, / on osd
[Alerting] File systems alert One of the file systems is too full Metric name Value /: Used Percentage 90.049
du -x --max-depth=1 -BM | sort -n
1M ./lost+found 1M ./mnt 1M ./selinux 1M ./storage 2M ./bin 6M ./sbin 12M ./lib64 26M ./etc 46M ./root 99M ./boot 147M ./opt 1085M ./lib 1284M ./var 4128M ./usr 10083M ./tmp 16915M .
seems like /tmp has a very big contribution now. A lot of temporary directories like
6TyfduRNJ6, oldest one since 2021-03-24 03:48 . Unfortunately there are hardly any logs going back in time, like because / is that full that also the systemd journal does not save more. 2021-03-24 is not a date where we commonly automatically reboot the system so not sure if non-openQA package upgrades caused a change.
I found some files like tmp.vvDweS5srZ which look like autoinst-log.txt or worker-log.txt . I assume that these are temporary files from openqa-investigate
- Assignee set to okurz
- Priority changed from Urgent to High
I deleted some directories and files on osd. Likely a similar problem can exist on osd. Maybe https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues#L134 combined with an unexpected exit of the script could be the problem. the tempfile is deleted but only if the function exits successfully. I assume we should ensure deleting that file in an
- Status changed from Feedback to Blocked
1616561148_719579.png autoinst-log-live.txt last.png serial-terminal-live.txt
this looks more like a regression in openQA or some dependency. Created #92344
And deleted many temporary files and directories on osd so that we are back to 38% usage now, 12G available.
- Status changed from Blocked to Feedback