https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-05-28T08:50:39ZopenSUSE Project Management ToolopenQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4112822021-05-28T08:50:39ZVANASTASIADISvasilios.anastasiadis@suse.com
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li><li><strong>Target version</strong> set to <i>Ready</i></li></ul> openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4114082021-05-28T15:36:33Zmkittlermarius.kittler@suse.com
<ul></ul><p>Looks like logrotate runs as root. Strange that it can not rename a file due to a permission error than. It is also strange that simply restarting the service (I assume without adjust changing any permissions) helped.</p>
<p><del>Maybe <code>/var/log</code> hasn't been fully mounted at the point logratoate ran? As <code>/var/log</code> is on a distinct btrfs subvolume it actually has its own fstab entry. Maybe <code>logrotate.service</code> should depend on <code>var-log.mount</code>?</del> No, it actually already depends on it (indirectly).</p>
<p>The full log looks like this, so there's not more information except that it happened 2 days in a row:</p>
<pre><code>martchus@openqaworker-arm-1:~> sudo journalctl --since '7 day ago' -fu logrotate.service
-- Logs begin at Fri 2020-05-15 20:42:16 UTC. --
Mai 22 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 22 00:00:11 openqaworker-arm-1 systemd[1]: Started Rotate log files.
Mai 23 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 23 00:00:06 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 24 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 24 00:00:08 openqaworker-arm-1 systemd[1]: Started Rotate log files.
Mai 25 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 25 00:00:11 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 26 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 26 00:00:00 openqaworker-arm-1 systemd[1]: Started Rotate log files.
-- Reboot --
Mai 27 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 27 00:00:00 openqaworker-arm-1 logrotate[4765]: error: failed to rename /var/log/openvswitch/ovs-vswitchd.log to /var/log/openvswitch/ovs-vswitchd.log-20210527: Permission denied
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: Failed to start Rotate log files.
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Unit entered failed state.
Mai 27 00:00:17 openqaworker-arm-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
-- Reboot --
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 28 00:00:00 openqaworker-arm-1 logrotate[33959]: error: failed to rename /var/log/openvswitch/ovs-vswitchd.log to /var/log/openvswitch/ovs-vswitchd.log-20210528: Permission denied
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: Failed to start Rotate log files.
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Unit entered failed state.
Mai 28 00:00:00 openqaworker-arm-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Mai 28 05:39:55 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mai 28 05:40:03 openqaworker-arm-1 systemd[1]: Started Rotate log files.
</code></pre> openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4114202021-05-28T17:12:37Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed behind-schedule" href="/issues/62306">action #62306</a>: osd logrotate fails sporadically on "error opening /var/log/salt/master: Permission denied", only at 00:00, i.e. midnight every day.</i> added</li></ul> openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4114262021-05-28T17:12:44Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-6 priority-high2 closed" href="/issues/57476">action #57476</a>: Recurring partitions full and logrotate fails, possibly due to disabling /var/log/openqa as log target</i> added</li></ul> openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4114322021-05-28T17:12:49Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-3 priority-lowest closed" href="/issues/53570">action #53570</a>: Failed service "logrotate" on aarch64.o.o and openqaworker1.o.o</i> added</li></ul> openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4114382021-05-28T17:12:56Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/34510">action #34510</a>: Add logrotate for logs in OSD</i> added</li></ul> openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4114442021-05-28T17:13:06Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed behind-schedule" href="/issues/78064">action #78064</a>: failing logrotate on monitor.qa.suse.de due to mariadb/mysql?</i> added</li></ul> openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4114502021-05-28T18:04:33Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>[Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28</i> to <i>[Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1</i></li><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li></ul><p>Please also be aware about what we already have in salt, e.g. <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch_boo1181418.sls" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch_boo1181418.sls</a> referencing <a href="https://bugzilla.suse.com/show_bug.cgi?id=1181418" class="external">https://bugzilla.suse.com/show_bug.cgi?id=1181418</a></p>
openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4114532021-05-28T20:37:05Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> set to <i>2021-06-11</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/497" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/497</a> , merged</p>
openQA Infrastructure - action #93195: [Alerting] Failed systemd services alert (except openqa.suse.de) on 2021-05-28, logrotate.service on openqaworker-arm-1https://progress.opensuse.org/issues/93195?journal_id=4147812021-06-09T20:56:19Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p><a href="https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=now-30d&to=now" class="external">https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=now-30d&to=now</a> shows the last occurence of "logrotate" on 2021-05-28, good since then</p>