Actions
action #93195
closed[Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1
Start date:
2021-05-28
Due date:
2021-06-11
% Done:
0%
Estimated time:
Description
Observation¶
Cheked the systemctl --failed on openqaworker-arm-1:
xiaojing@openqaworker-arm-1:~> systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● logrotate.service loaded failed failed Rotate log files
The status of logrotate showed:
● logrotate.service - Rotate log files
Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2021-05-28 00:00:00 UTC; 5h 37min ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Main PID: 33959 (code=exited, status=1/FAILURE)
May 28 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
May 28 00:00:00 openqaworker-arm-1 logrotate[33959]: error: failed to rename /var/log/openvswitch/ovs-vswitchd.log to /var/log/openvswitch/ovs-vswitchd.log-2 ... : Permission denied
May 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
May 28 00:00:00 openqaworker-arm-1 systemd[1]: Failed to start Rotate log files.
May 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Unit entered failed state.
May 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Workaround¶
After run systemctl start logrotate.service, got the [OK] email.
Updated by VANASTASIADIS over 3 years ago
- Priority changed from Normal to High
- Target version set to Ready
Updated by mkittler over 3 years ago
Looks like logrotate runs as root. Strange that it can not rename a file due to a permission error than. It is also strange that simply restarting the service (I assume without adjust changing any permissions) helped.
Maybe No, it actually already depends on it (indirectly)./var/log hasn't been fully mounted at the point logratoate ran? As /var/log is on a distinct btrfs subvolume it actually has its own fstab entry. Maybe logrotate.service should depend on var-log.mount?
The full log looks like this, so there's not more information except that it happened 2 days in a row:
martchus@openqaworker-arm-1:~> sudo journalctl --since '7 day ago' -fu logrotate.service
-- Logs begin at Fri 2020-05-15 20:42:16 UTC. --
Mai 22 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 22 00:00:11 openqaworker-arm-1 systemd[1]: Started Rotate log files.
Mai 23 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 23 00:00:06 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 24 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 24 00:00:08 openqaworker-arm-1 systemd[1]: Started Rotate log files.
Mai 25 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 25 00:00:11 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 26 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 26 00:00:00 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 27 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 27 00:00:00 openqaworker-arm-1 logrotate[4765]: error: failed to rename /var/log/openvswitch/ovs-vswitchd.log to /var/log/openvswitch/ovs-vswitchd.log-20210527: Permission denied
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: Failed to start Rotate log files.
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Unit entered failed state.
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
-- Reboot --
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 28 00:00:00 openqaworker-arm-1 logrotate[33959]: error: failed to rename /var/log/openvswitch/ovs-vswitchd.log to /var/log/openvswitch/ovs-vswitchd.log-20210528: Permission denied
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: Failed to start Rotate log files.
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Unit entered failed state.
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Mai 28 05:39:55 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 28 05:40:03 openqaworker-arm-1 systemd[1]: Started Rotate log files.
Updated by okurz over 3 years ago
- Related to action #62306: osd logrotate fails sporadically on "error opening /var/log/salt/master: Permission denied", only at 00:00, i.e. midnight every day. added
Updated by okurz over 3 years ago
- Related to action #57476: Recurring partitions full and logrotate fails, possibly due to disabling /var/log/openqa as log target added
Updated by okurz over 3 years ago
- Related to action #53570: Failed service "logrotate" on aarch64.o.o and openqaworker1.o.o added
Updated by okurz over 3 years ago
- Related to action #34510: Add logrotate for logs in OSD added
Updated by okurz over 3 years ago
- Related to action #78064: failing logrotate on monitor.qa.suse.de due to mariadb/mysql? added
Updated by okurz over 3 years ago
- Subject changed from [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28 to [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1
- Status changed from New to In Progress
- Assignee set to okurz
Please also be aware about what we already have in salt, e.g. https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch_boo1181418.sls referencing https://bugzilla.suse.com/show_bug.cgi?id=1181418
Updated by okurz over 3 years ago
- Due date set to 2021-06-11
- Status changed from In Progress to Feedback
Updated by okurz over 3 years ago
- Status changed from Feedback to Resolved
https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=now-30d&to=now shows the last occurence of "logrotate" on 2021-05-28, good since then
Actions