Project

General

Profile

Actions

action #137813

closed

[alert] Failed systemd services - qamaster - logrotate fails on /var/log/messages with "/usr/bin/xz: (stdin): Read error: Input/output error" size:S

Added by livdywan 8 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2023-10-07
Due date:
% Done:

0%

Estimated time:

Description

[alert] Failed systemd services - qamaster - logrotate fails on /var/log/messages with "/usr/bin/xz: (stdin): Read error: Input/output error"

Observation

From https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1

2023-10-07 14:08:20

qamaster

logrotate

okurz@qamaster:~> sudo systemctl status logrotate
x logrotate.service - Rotate log files
     Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static)
     Active: failed (Result: exit-code) since Sat 2023-10-07 00:00:09 CEST; 14h ago
TriggeredBy: . logrotate.timer
       Docs: man:logrotate(8)
             man:logrotate.conf(5)
   Main PID: 2037 (code=exited, status=1/FAILURE)

Oct 07 00:00:00 qamaster systemd[1]: Starting Rotate log files...
Oct 07 00:00:04 qamaster logrotate[2037]: error: Compressing program wrote following message to stderr when compressing log /var/log/messages-20231007:
Oct 07 00:00:04 qamaster logrotate[2037]: /usr/bin/xz: (stdin): Read error: Input/output error
Oct 07 00:00:04 qamaster logrotate[2037]: error: failed to compress log /var/log/messages-20231007
Oct 07 00:00:09 qamaster systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Oct 07 00:00:09 qamaster systemd[1]: logrotate.service: Failed with result 'exit-code'.
Oct 07 00:00:09 qamaster systemd[1]: Failed to start Rotate log files.

/var/log/messages sounds like syslog, possibly from rsyslog. I suggest we just make sure this is not used at all anymore as systemd journald should suffice.

Suggestions

  • can we just fully switch to systemd journal? what isn't in the journal yet? Likely we can. This is just an effect of the system being an old installation migrated over multiple versions. Just remove "rsyslog" and whatever and ensure everything writes into the system journal

Workaround

  • systemctl restart logrotate

Related issues 1 (0 open1 closed)

Related to openQA Project - action #135482: Move to systemd journal only on o3+osd (was: Missing openqa_websockets log file on OSD for websocket server) size:MRejectedokurz2023-09-11

Actions
Actions #2

Updated by jbaier_cz 8 months ago

  • Assignee set to jbaier_cz
Actions #3

Updated by jbaier_cz 8 months ago

  • Status changed from Workable to In Progress
Actions #4

Updated by jbaier_cz 8 months ago

  • Status changed from In Progress to Resolved

I did zypper remove --clean-deps rsyslog and mkdir /var/log/journal; there is no mention about rsyslog installation in our salt, so I guess we should be good.

Actions #5

Updated by okurz 8 months ago

  • Related to action #135482: Move to systemd journal only on o3+osd (was: Missing openqa_websockets log file on OSD for websocket server) size:M added
Actions

Also available in: Atom PDF