Added by mkittler over 2 years ago. Updated over 2 years ago.
0%
Description
rm -rf /var/cache/zypp/packages helped as a temporary measure. Logrotation for /var/log/openqa_scheduler and /var/log/openqa_gru seems broken.
Files
/space./space/logs because there was still not enough disk space left to perform the log rotation.We should have enough free disk space again:
/dev/vda1 20G 6,7G 13G 36% /
This should do it for now. Other areas don't look problematic (checked via ncdu -x /). If necessary we can still reduce the log storage duration or store the openQA logs under /space/logs.
It is still not clear why the logrotate config was changed to remove handling of scheduler and gru logs. There actually were rotated scheduler/gru logs from September and the logrotate config was modified on 14. Oktober.
Let's conduct 5 WHYs tomorrow 14.00 CE(S)T, at least Oli and I would find it useful to understand why this wasn't working before, and what changed.
Attached screenshot of munin disk space graph.
rm -rf /var/cache/zypp/packages and restarted openqa-livehandler.service1,9G /var/log/journal
2,7G /var/log/openqa_scheduler
6,9G /var/log/openqa_gru
Improvements
Looks like the changed logrotate config works. Since we've created a follow-up ticket for further improvements I'm resolving this one.