action #115208
Updated by livdywan almost 2 years ago
## Motivation
For the third time this week [failed-systemd-services](https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services) has been alerting in the middle of the European night and recovering:
```
2022-08-11 02:59:52
openqa
logrotate-openqa
1
2022-08-10 17:43:01 QA-Power8-4-kvm openqa-worker-cacheservice-minion 1
```
Previously [only logrotate-openqa was failing](https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=1659985969945&to=1660023851901).
```
sudo journalctl -fu logrotate-openqa
Aug 11 09:00:02 openqa logrotate[26982]: Last rotated at 2022-08-11 04:00
Aug 11 09:00:02 openqa logrotate[26982]: log does not need rotating (log size is below the 'size' threshold)
Aug 11 09:00:02 openqa logrotate[26982]: rotating pattern: /var/log/apache2/openqa.access_log 307200000 bytes (20 rotations)
Aug 11 09:00:02 openqa logrotate[26982]: empty log files are not rotated, old logs are removed
Aug 11 09:00:02 openqa logrotate[26982]: considering log /var/log/apache2/openqa.access_log
Aug 11 09:00:02 openqa logrotate[26982]: Now: 2022-08-11 09:00
Aug 11 09:00:02 openqa logrotate[26982]: Last rotated at 2022-08-11 05:00
Aug 11 09:00:02 openqa logrotate[26982]: log does not need rotating (log size is below the 'size' threshold)
Aug 11 09:00:02 openqa systemd[1]: logrotate-openqa.service: Deactivated successfully.
Aug 11 09:00:02 openqa systemd[1]: Finished Rotate openQA log files.
```
## Acceptance criteria
* **AC1:** logrotate-openqa is known to work reliably
* **AC2:** failed-systemd-services is not alerting on a regular basis
## Suggestions
* Look into what's causing logrotate-openqa to fail
* Check if there's enough disk space / if the logs are deleted before rotation