https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-02-28T10:06:29ZopenSUSE Project Management ToolopenQA Infrastructure - action #125132: [alert] logrotate failed on OSDhttps://progress.opensuse.org/issues/125132?journal_id=6070012023-02-28T10:06:29Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed" href="/issues/124412">action #124412</a>: [alert] logrotate services failed on openqa-piworker.qa.suse.de and OSD size:M</i> added</li></ul> openQA Infrastructure - action #125132: [alert] logrotate failed on OSDhttps://progress.opensuse.org/issues/125132?journal_id=6070042023-02-28T10:07:22Zokurzokurz@suse.com
<ul><li><strong>Tags</strong> set to <i>infra, osd, logrotate</i></li><li><strong>Assignee</strong> set to <i>mkittler</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/22072">@mkittler</a> as you are working on that within <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: [alert] logrotate services failed on openqa-piworker.qa.suse.de and OSD size:M (Resolved)" href="https://progress.opensuse.org/issues/124412">#124412</a> please take this into account</p>
openQA Infrastructure - action #125132: [alert] logrotate failed on OSDhttps://progress.opensuse.org/issues/125132?journal_id=6070432023-02-28T10:24:41Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li></ul><p>I haven't received a mail about failing systemd services today anymore. I suppose this ticket has been created for a mail from yesterday.</p>
<p>This mentioned log is even from 15 Feb. So it had happened before my changes for <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: [alert] logrotate services failed on openqa-piworker.qa.suse.de and OSD size:M (Resolved)" href="https://progress.opensuse.org/issues/124412">#124412</a> were deployed. The most recent encounter is:</p>
<pre><code>sudo journalctl --since '1 day ago' -fu logrotate.service -u logrotate-openqa.service
…
Feb 27 15:00:00 openqa logrotate[10366]: uncompress_prog is now /usr/bin/xzdec
Feb 27 15:00:00 openqa logrotate[10366]: error: state file /var/lib/misc/logrotate.status is already locked
Feb 27 15:00:00 openqa logrotate[10366]: logrotate does not support parallel execution on the same set of logfiles.
Feb 27 15:00:00 openqa systemd[1]: logrotate-openqa.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Feb 27 15:00:00 openqa systemd[1]: logrotate-openqa.service: Failed with result 'exit-code'.
Feb 27 15:00:00 openqa systemd[1]: Failed to start Rotate openQA log files.
Feb 27 15:48:25 openqa logrotate[29411]: error: destination /var/log/rsyncd.log-20230227.xz already exists, skipping rotation
Feb 27 15:48:25 openqa logrotate[29411]: error: destination /var/log/zypper.log-20230227.xz already exists, skipping rotation
Feb 27 15:48:25 openqa systemd[1]: logrotate.service: Deactivated successfully.
Feb 27 15:48:25 openqa systemd[1]: Finished Rotate log files.
…
</code></pre>
<p>and also that was (2 hours) before the MR has been merged.</p>
<p>I suppose we can keep this ticket open (in addition to <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: [alert] logrotate services failed on openqa-piworker.qa.suse.de and OSD size:M (Resolved)" href="https://progress.opensuse.org/issues/124412">#124412</a>) as it is specific to the OSD problem only (although it is basically duplicating <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: [alert] logrotate services failed on openqa-piworker.qa.suse.de and OSD size:M (Resolved)" href="https://progress.opensuse.org/issues/124412">#124412</a>).</p>
<p>Looks like my MR has still one mistake in it:</p>
<pre><code>Feb 27 18:00:00 openqa bash[19337]: ++ /usr/bin/systemctl --user is-active logrotate.service
Feb 27 18:00:01 openqa bash[19340]: Failed to connect to bus: $DBUS_SESSION_BUS_ADDRESS and $XDG_RUNTIME_DIR not defined (consider using --machine=<user>@.host --user to connect to bus of other user)
Feb 27 18:00:01 openqa bash[19322]: + is_active=
Feb 27 18:00:01 openqa bash[19322]: + [[ '' == active ]]
Feb 27 18:00:01 openqa bash[19322]: + [[ '' == activating ]]
Feb 27 18:00:01 openqa bash[19322]: + exit 0
</code></pre>
<p>The <code>--user</code> flag shouldn't have been used. I've created a MR to fix this (<a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/800">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/800</a>) and have also applied the change manually on OSD.</p>
openQA Infrastructure - action #125132: [alert] logrotate failed on OSDhttps://progress.opensuse.org/issues/125132?journal_id=6076132023-03-01T10:37:39Zokurzokurz@suse.com
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Normal</i></li></ul><p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/800" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/800</a> merged and effective, back to "Normal".</p>
openQA Infrastructure - action #125132: [alert] logrotate failed on OSDhttps://progress.opensuse.org/issues/125132?journal_id=6076162023-03-01T10:38:45Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>We will know if the problem reappears from alerting.</p>