Project

General

Profile

action #93195

[Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1

Added by Xiaojing_liu 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2021-05-28
Due date:
2021-06-11
% Done:

0%

Estimated time:

Description

Observation

Cheked the systemctl --failed on openqaworker-arm-1:

xiaojing@openqaworker-arm-1:~> systemctl --failed
  UNIT              LOAD   ACTIVE SUB    DESCRIPTION                                                                                                         
● logrotate.service loaded failed failed Rotate log files

The status of logrotate showed:

● logrotate.service - Rotate log files
   Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2021-05-28 00:00:00 UTC; 5h 37min ago
     Docs: man:logrotate(8)
           man:logrotate.conf(5)
 Main PID: 33959 (code=exited, status=1/FAILURE)

May 28 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
May 28 00:00:00 openqaworker-arm-1 logrotate[33959]: error: failed to rename /var/log/openvswitch/ovs-vswitchd.log to /var/log/openvswitch/ovs-vswitchd.log-2 ... : Permission denied
May 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
May 28 00:00:00 openqaworker-arm-1 systemd[1]: Failed to start Rotate log files.
May 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Unit entered failed state.
May 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.

Workaround

After run systemctl start logrotate.service, got the [OK] email.


Related issues

Related to openQA Infrastructure - action #62306: osd logrotate fails sporadically on "error opening /var/log/salt/master: Permission denied", only at 00:00, i.e. midnight every day.Resolved2020-01-192020-04-14

Related to openQA Infrastructure - action #57476: Recurring partitions full and logrotate fails, possibly due to disabling /var/log/openqa as log targetResolved2019-09-28

Related to openQA Infrastructure - action #53570: Failed service "logrotate" on aarch64.o.o and openqaworker1.o.oResolved2019-06-30

Related to openQA Project - action #34510: Add logrotate for logs in OSDResolved2018-04-09

Related to openQA Infrastructure - action #78064: failing logrotate on monitor.qa.suse.de due to mariadb/mysql?Resolved2020-11-162020-11-24

History

#1 Updated by VANASTASIADIS 2 months ago

  • Priority changed from Normal to High
  • Target version set to Ready

#2 Updated by mkittler 2 months ago

Looks like logrotate runs as root. Strange that it can not rename a file due to a permission error than. It is also strange that simply restarting the service (I assume without adjust changing any permissions) helped.

Maybe /var/log hasn't been fully mounted at the point logratoate ran? As /var/log is on a distinct btrfs subvolume it actually has its own fstab entry. Maybe logrotate.service should depend on var-log.mount? No, it actually already depends on it (indirectly).

The full log looks like this, so there's not more information except that it happened 2 days in a row:

martchus@openqaworker-arm-1:~> sudo journalctl --since '7 day ago' -fu logrotate.service 
-- Logs begin at Fri 2020-05-15 20:42:16 UTC. --
Mai 22 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 22 00:00:11 openqaworker-arm-1 systemd[1]: Started Rotate log files.
Mai 23 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 23 00:00:06 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 24 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 24 00:00:08 openqaworker-arm-1 systemd[1]: Started Rotate log files.
Mai 25 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 25 00:00:11 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 26 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 26 00:00:00 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 27 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 27 00:00:00 openqaworker-arm-1 logrotate[4765]: error: failed to rename /var/log/openvswitch/ovs-vswitchd.log to /var/log/openvswitch/ovs-vswitchd.log-20210527: Permission denied
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: Failed to start Rotate log files.
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Unit entered failed state.
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
-- Reboot --
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 28 00:00:00 openqaworker-arm-1 logrotate[33959]: error: failed to rename /var/log/openvswitch/ovs-vswitchd.log to /var/log/openvswitch/ovs-vswitchd.log-20210528: Permission denied
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: Failed to start Rotate log files.
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Unit entered failed state.
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Mai 28 05:39:55 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 28 05:40:03 openqaworker-arm-1 systemd[1]: Started Rotate log files.

#3 Updated by okurz 2 months ago

  • Related to action #62306: osd logrotate fails sporadically on "error opening /var/log/salt/master: Permission denied", only at 00:00, i.e. midnight every day. added

#4 Updated by okurz 2 months ago

  • Related to action #57476: Recurring partitions full and logrotate fails, possibly due to disabling /var/log/openqa as log target added

#5 Updated by okurz 2 months ago

  • Related to action #53570: Failed service "logrotate" on aarch64.o.o and openqaworker1.o.o added

#6 Updated by okurz 2 months ago

#7 Updated by okurz 2 months ago

  • Related to action #78064: failing logrotate on monitor.qa.suse.de due to mariadb/mysql? added

#8 Updated by okurz 2 months ago

  • Subject changed from [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28 to [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1
  • Status changed from New to In Progress
  • Assignee set to okurz

#9 Updated by okurz 2 months ago

  • Due date set to 2021-06-11
  • Status changed from In Progress to Feedback

#10 Updated by okurz about 2 months ago

  • Status changed from Feedback to Resolved

https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=now-30d&to=now shows the last occurence of "logrotate" on 2021-05-28, good since then

Also available in: Atom PDF