Actions
action #62309
closedlogrotate fails on QA-Power8-4-kvm (and powerqaworker-qam-1) with "error: destination /var/log/messages-20200118.xz already exists, skipping rotation"
Added by okurz over 4 years ago. Updated about 4 years ago.
Start date:
2020-01-19
Due date:
2020-04-14
% Done:
0%
Estimated time:
Description
> sudo systemctl status logrotate
● logrotate.service - Rotate log files
Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2020-01-19 00:00:00 CET; 22h ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 48560 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 48560 (code=exited, status=1/FAILURE)
Jan 19 00:00:00 QA-Power8-4-kvm systemd[1]: Starting Rotate log files...
Jan 19 00:00:00 QA-Power8-4-kvm logrotate[48560]: error: destination /var/log/messages-20200118.xz already exists, skipping rotation
Jan 19 00:00:00 QA-Power8-4-kvm systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Jan 19 00:00:00 QA-Power8-4-kvm systemd[1]: Failed to start Rotate log files.
Jan 19 00:00:00 QA-Power8-4-kvm systemd[1]: logrotate.service: Unit entered failed state.
Jan 19 00:00:00 QA-Power8-4-kvm systemd[1]: logrotate.service: Failed with result 'exit-code'.
Updated by okurz over 4 years ago
- Copied from action #62306: osd logrotate fails sporadically on "error opening /var/log/salt/master: Permission denied", only at 00:00, i.e. midnight every day. added
Updated by okurz about 4 years ago
- Status changed from New to Rejected
- Assignee set to okurz
logrotate is very stable for a month on that host:
QA-Power8-4-kvm:/home/okurz # journalctl -u logrotate
Jan 29 00:00:00 QA-Power8-4-kvm systemd[1]: Starting Rotate log files...
…
Mar 01 00:00:34 QA-Power8-4-kvm systemd[1]: Started Rotate log files.
I assume the problem happened because by accident, maybe manually, logrotate was triggered a second time during the same day which fails in the way as described above. I don't think we need to do anything further about it.
Updated by okurz about 4 years ago
- Status changed from Rejected to Workable
I think the same thing happened again now on powerqaworker-qam-1:
okurz@powerqaworker-qam-1:~> sudo systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● logrotate.service loaded failed failed Rotate log files
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
okurz@powerqaworker-qam-1:~> sudo systemctl status logrotate
● logrotate.service - Rotate log files
Loaded: loaded (/usr/lib/systemd/system/logrotate.service; static; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2020-03-07 00:00:00 CET; 11h ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 43196 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 43196 (code=exited, status=1/FAILURE)
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: Starting Rotate log files...
Mar 07 00:00:00 powerqaworker-qam-1 logrotate[43196]: error: destination /var/log/messages-20200306.xz already exists, skipping rotation
Mar 07 00:00:00 powerqaworker-qam-1 logrotate[43196]: error: destination /var/log/zypper.log-20200306.xz already exists, skipping rotation
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: Failed to start Rotate log files.
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: logrotate.service: Unit entered failed state.
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
okurz@powerqaworker-qam-1:~> sudo journalctl --since=yesterday -u logrotate
-- Logs begin at Fri 2020-01-31 19:22:28 CET, end at Sat 2020-03-07 11:37:10 CET. --
Mar 06 00:00:00 powerqaworker-qam-1 systemd[1]: Starting Rotate log files...
Mar 06 00:00:22 powerqaworker-qam-1 systemd[1]: Started Rotate log files.
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: Starting Rotate log files...
Mar 07 00:00:00 powerqaworker-qam-1 logrotate[43196]: error: destination /var/log/messages-20200306.xz already exists, skipping rotation
Mar 07 00:00:00 powerqaworker-qam-1 logrotate[43196]: error: destination /var/log/zypper.log-20200306.xz already exists, skipping rotation
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: Failed to start Rotate log files.
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: logrotate.service: Unit entered failed state.
Mar 07 00:00:00 powerqaworker-qam-1 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Normally it seems logrotate should have used 20200307 as timestamp, not the previous day. This is also what happened on all other hosts:
$ ssh osd 'sudo salt -C G@roles:worker cmd.run "ls -ltra /var/log/zypper.log-202003*"'
QA-Power8-5-kvm.qa.suse.de:
-rw-r----- 1 root root 491672 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 297324 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 275048 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 834036 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 553060 Mar 6 23:36 /var/log/zypper.log-20200307.xz
powerqaworker-qam-1:
-rw-r----- 1 root root 502428 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 305112 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 267896 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 839212 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 252548 Mar 5 23:07 /var/log/zypper.log-20200306.xz
QA-Power8-4-kvm.qa.suse.de:
-rw-r----- 1 root root 505444 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 304500 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 270648 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 830240 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 546260 Mar 6 23:36 /var/log/zypper.log-20200307.xz
malbec.arch.suse.de:
-rw-r----- 1 root root 498072 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 306368 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 1083864 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 550884 Mar 6 23:36 /var/log/zypper.log-20200307.xz
grenache-1.qa.suse.de:
-rw-r----- 1 root root 517372 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 305372 Mar 2 23:08 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 266464 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 831740 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 544836 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker6.suse.de:
-rw-r----- 1 root root 519520 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 317444 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 282172 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 843860 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 555188 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker9.suse.de:
-rw-r----- 1 root root 513000 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 314220 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 273420 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 839824 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 563696 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker5.suse.de:
-rw-r----- 1 root root 523112 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 320096 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 281220 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 846016 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 566624 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker3.suse.de:
-rw-r----- 1 root root 528432 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 325784 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 281320 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 856288 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 564260 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker8.suse.de:
-rw-r----- 1 root root 261048 Feb 29 23:32 /var/log/zypper.log-20200301.xz
-rw-r----- 1 root root 271804 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 320844 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 275896 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 859244 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 269160 Mar 5 23:07 /var/log/zypper.log-20200306.xz
-rw-r----- 1 root root 307260 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker2.suse.de:
-rw-r----- 1 root root 483668 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 544864 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 818552 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 526228 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker10.suse.de:
-rw-r----- 1 root root 510000 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 318432 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 273408 Mar 3 23:07 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 841204 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 556536 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker13.suse.de:
-rw-r----- 1 root root 486856 Mar 1 23:32 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 306572 Mar 2 23:07 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 1101104 Mar 4 23:07 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 531576 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker-arm-1.suse.de:
-rw-r----- 1 root root 515504 Mar 1 23:34 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 307764 Mar 2 23:09 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 2227856 Mar 4 23:28 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 535372 Mar 6 23:36 /var/log/zypper.log-20200307.xz
openqaworker-arm-2.suse.de:
-rw-r----- 1 root root 471068 Mar 1 23:49 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 302652 Mar 2 23:10 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 891148 Mar 4 23:48 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 475816 Mar 6 23:16 /var/log/zypper.log-20200307.xz
openqaworker-arm-3.suse.de:
-rw-r----- 1 root root 481840 Feb 29 23:16 /var/log/zypper.log-20200301.xz
-rw-r----- 1 root root 251688 Mar 1 23:16 /var/log/zypper.log-20200302.xz
-rw-r----- 1 root root 293092 Mar 2 23:08 /var/log/zypper.log-20200303.xz
-rw-r----- 1 root root 270428 Mar 3 23:08 /var/log/zypper.log-20200304.xz
-rw-r----- 1 root root 1966164 Mar 4 23:08 /var/log/zypper.log-20200305.xz
-rw-r----- 1 root root 547748 Mar 6 23:37 /var/log/zypper.log-20200307.xz
But I see a different inconsistency:
$ ssh osd 'sudo salt \* cmd.run "date +\"%Z %z\""'
QA-Power8-5-kvm.qa.suse.de:
CET +0100
QA-Power8-4-kvm.qa.suse.de:
CET +0100
powerqaworker-qam-1:
CET +0100
malbec.arch.suse.de:
CET +0100
grenache-1.qa.suse.de:
CET +0100
openqaworker8.suse.de:
CET +0100
openqaworker2.suse.de:
CET +0100
openqaworker3.suse.de:
CET +0100
openqaworker9.suse.de:
CET +0100
openqaworker5.suse.de:
CET +0100
openqa-monitor.qa.suse.de:
CET +0100
openqaworker6.suse.de:
CET +0100
openqa.suse.de:
CET +0100
openqaworker13.suse.de:
CET +0100
openqaworker10.suse.de:
UTC +0000
openqaworker-arm-1.suse.de:
UTC +0000
openqaworker-arm-3.suse.de:
CET +0100
openqaworker-arm-2.suse.de:
UTC +0000
IMHO we should switch all hosts to the same timezone. I personally would prefer UTC but CET seems more common, should be easy to adjust openqaworker10, arm1 and arm3. o3 is similar, CET more common, power8 and imagetester UTC. However powerqaworker-qam-1 is not any different than the common. It seems like logrotate triggered at exactly 00:00:00 took by accident still the date from the previous day, as if it was started at 23:59:59.
Another point I found:
$ ssh osd 'sudo salt \* cmd.run "grep DAILY_TIME /etc/sysconfig/cron"'
QA-Power8-4-kvm.qa.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
QA-Power8-5-kvm.qa.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
powerqaworker-qam-1:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
malbec.arch.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
grenache-1.qa.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker8.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker3.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker5.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker6.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker2.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker9.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqa-monitor.qa.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker10.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker13.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker-arm-1.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqa.suse.de:
DAILY_TIME="19:30"
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker-arm-2.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
openqaworker-arm-3.suse.de:
DAILY_TIME=""
# Maximum days not running when using a fixed time set in DAILY_TIME.
so all machines trigger daily jobs at 00:00 local time which is 23:00 UTC.
Updated by okurz about 4 years ago
- Subject changed from logrotate fails on QA-Power8-4-kvm with "error: destination /var/log/messages-20200118.xz already exists, skipping rotation" to logrotate fails on QA-Power8-4-kvm (and powerqaworker-qam-1) with "error: destination /var/log/messages-20200118.xz already exists, skipping rotation"
Updated by okurz about 4 years ago
- Status changed from Workable to Feedback
As the problem happened for rotating /var/log/messages which we shouldn't need when we rather rely on persistent systemd journal I suggest we simply remove rsyslog completely as we do not have it on many other machines anyway:
# salt -l error --no-color -C '*' cmd.run 'rpm -qa | grep syslog'
openqaworker2.suse.de:
openqaworker5.suse.de:
openqaworker3.suse.de:
openqaworker6.suse.de:
openqaworker8.suse.de:
openqa-monitor.qa.suse.de:
malbec.arch.suse.de:
syslog-service-2.0-lp151.3.4.noarch
rsyslog-8.33.1-lp151.6.10.1.ppc64le
QA-Power8-5-kvm.qa.suse.de:
rsyslog-8.33.1-lp151.6.10.1.ppc64le
syslog-service-2.0-lp151.3.4.noarch
openqaworker10.suse.de:
openqaworker13.suse.de:
QA-Power8-4-kvm.qa.suse.de:
syslog-service-2.0-lp151.3.4.noarch
rsyslog-8.33.1-lp151.6.10.1.ppc64le
powerqaworker-qam-1:
syslog-service-2.0-lp151.3.4.noarch
rsyslog-8.33.1-lp151.6.10.1.ppc64le
grenache-1.qa.suse.de:
rsyslog-8.33.1-lp151.6.10.1.ppc64le
syslog-service-2.0-lp151.3.4.noarch
openqaworker-arm-1.suse.de:
syslog-service-2.0-lp151.3.3.noarch
rsyslog-8.33.1-lp151.6.10.1.aarch64
openqaworker-arm-2.suse.de:
rsyslog-8.33.1-lp151.6.10.1.aarch64
syslog-service-2.0-lp151.3.3.noarch
openqa.suse.de:
https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/journal.sls is where we ensure a persistent journal so we can just make sure the package is absent.
Created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/295 to fix by removing syslog.
Updated by okurz about 4 years ago
- Status changed from Feedback to Resolved
syslog has been removed from all workers, logrotate shows no problems on no worker so far:
# salt -l error --no-color -C '*' cmd.run 'journalctl --since=yesterday -e -u logrotate'
QA-Power8-4-kvm.qa.suse.de:
-- Logs begin at Sun 2020-03-01 16:44:16 CET, end at Tue 2020-03-31 10:07:11 CEST. --
Mar 30 00:00:00 QA-Power8-4-kvm systemd[1]: Starting Rotate log files...
Mar 30 00:00:13 QA-Power8-4-kvm systemd[1]: Started Rotate log files.
Mar 31 00:00:00 QA-Power8-4-kvm systemd[1]: Starting Rotate log files...
Mar 31 00:00:03 QA-Power8-4-kvm systemd[1]: Started Rotate log files.
powerqaworker-qam-1:
-- Logs begin at Thu 2020-03-05 07:58:23 CET, end at Tue 2020-03-31 10:07:11 CEST. --
Mar 30 00:00:00 powerqaworker-qam-1 systemd[1]: Starting Rotate log files...
Mar 30 00:00:19 powerqaworker-qam-1 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 powerqaworker-qam-1 systemd[1]: Starting Rotate log files...
Mar 31 00:00:02 powerqaworker-qam-1 systemd[1]: Started Rotate log files.
openqaworker2.suse.de:
-- Logs begin at Mon 2020-03-16 21:45:09 CET, end at Tue 2020-03-31 10:07:10 CEST. --
Mar 30 00:00:00 openqaworker2 systemd[1]: Starting Rotate log files...
Mar 30 00:00:03 openqaworker2 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 openqaworker2 systemd[1]: Starting Rotate log files...
Mar 31 00:00:02 openqaworker2 systemd[1]: Started Rotate log files.
openqaworker5.suse.de:
-- Logs begin at Tue 2020-03-24 00:38:19 CET, end at Tue 2020-03-31 10:07:10 CEST. --
Mar 30 00:00:00 openqaworker5 systemd[1]: Starting Rotate log files...
Mar 30 00:00:04 openqaworker5 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 openqaworker5 systemd[1]: Starting Rotate log files...
Mar 31 00:00:02 openqaworker5 systemd[1]: Started Rotate log files.
openqaworker-arm-1.suse.de:
-- Logs begin at Wed 2020-01-08 12:17:31 UTC, end at Tue 2020-03-31 08:07:10 UTC. --
Mar 30 00:00:04 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mar 30 00:00:16 openqaworker-arm-1 systemd[1]: Started Rotate log files.
Mar 31 00:00:04 openqaworker-arm-1 systemd[1]: Starting Rotate log files...
Mar 31 00:00:11 openqaworker-arm-1 systemd[1]: Started Rotate log files.
openqaworker6.suse.de:
-- Logs begin at Mon 2019-02-18 15:10:35 CET, end at Tue 2020-03-31 10:07:10 CEST. --
Mar 30 00:00:00 openqaworker6 systemd[1]: Starting Rotate log files...
Mar 30 00:00:02 openqaworker6 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 openqaworker6 systemd[1]: Starting Rotate log files...
Mar 31 00:00:02 openqaworker6 systemd[1]: Started Rotate log files.
QA-Power8-5-kvm.qa.suse.de:
-- Logs begin at Thu 2020-02-27 00:28:46 CET, end at Tue 2020-03-31 10:07:11 CEST. --
Mar 30 00:00:00 QA-Power8-5-kvm systemd[1]: Starting Rotate log files...
Mar 30 00:00:11 QA-Power8-5-kvm systemd[1]: Started Rotate log files.
Mar 31 00:00:00 QA-Power8-5-kvm systemd[1]: Starting Rotate log files...
Mar 31 00:00:04 QA-Power8-5-kvm systemd[1]: Started Rotate log files.
openqa-monitor.qa.suse.de:
-- Logs begin at Mon 2020-03-30 21:15:36 CEST, end at Tue 2020-03-31 10:07:10 CEST. --
Mar 31 00:00:00 openqa-monitor systemd[1]: Starting Rotate log files...
Mar 31 00:00:09 openqa-monitor systemd[1]: Started Rotate log files.
openqaworker10.suse.de:
-- Logs begin at Mon 2018-06-18 12:58:07 UTC, end at Tue 2020-03-31 08:07:11 UTC. --
Mar 30 00:00:00 openqaworker10 systemd[1]: Starting Rotate log files...
Mar 30 00:00:04 openqaworker10 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 openqaworker10 systemd[1]: Starting Rotate log files...
Mar 31 00:00:02 openqaworker10 systemd[1]: Started Rotate log files.
openqaworker8.suse.de:
-- Logs begin at Thu 2018-01-11 09:56:40 CET, end at Tue 2020-03-31 10:07:11 CEST. --
Mar 30 00:00:00 openqaworker8 systemd[1]: Starting Rotate log files...
Mar 30 00:00:00 openqaworker8 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 openqaworker8 systemd[1]: Starting Rotate log files...
Mar 31 00:00:03 openqaworker8 systemd[1]: Started Rotate log files.
openqaworker3.suse.de:
-- Logs begin at Mon 2020-03-23 08:26:37 CET, end at Tue 2020-03-31 10:07:11 CEST. --
Mar 30 00:00:00 openqaworker3 systemd[1]: Starting Rotate log files...
Mar 30 00:00:04 openqaworker3 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 openqaworker3 systemd[1]: Starting Rotate log files...
Mar 31 00:00:02 openqaworker3 systemd[1]: Started Rotate log files.
openqa.suse.de:
-- Logs begin at Mon 2020-03-30 11:06:59 CEST, end at Tue 2020-03-31 10:07:08 CEST. --
Mar 31 00:00:00 openqa systemd[1]: Starting Rotate log files...
Mar 31 00:42:24 openqa systemd[1]: Started Rotate log files.
malbec.arch.suse.de:
-- Logs begin at Wed 2020-01-15 09:58:59 CET, end at Tue 2020-03-31 10:07:10 CEST. --
Mar 30 00:00:00 malbec systemd[1]: Starting Rotate log files...
Mar 30 00:00:04 malbec systemd[1]: Started Rotate log files.
Mar 31 00:00:00 malbec systemd[1]: Starting Rotate log files...
Mar 31 00:00:05 malbec systemd[1]: Started Rotate log files.
grenache-1.qa.suse.de:
-- Logs begin at Wed 2020-03-11 04:26:26 CET, end at Tue 2020-03-31 10:07:11 CEST. --
Mar 30 00:00:00 grenache-1 systemd[1]: Starting Rotate log files...
Mar 30 00:00:23 grenache-1 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 grenache-1 systemd[1]: Starting Rotate log files...
Mar 31 00:00:03 grenache-1 systemd[1]: Started Rotate log files.
openqaworker13.suse.de:
-- Logs begin at Thu 2018-10-04 14:09:46 CEST, end at Tue 2020-03-31 10:07:15 CEST. --
Mar 30 00:00:00 openqaworker13 systemd[1]: Starting Rotate log files...
Mar 30 00:00:01 openqaworker13 systemd[1]: Started Rotate log files.
Mar 31 00:00:00 openqaworker13 systemd[1]: Starting Rotate log files...
Mar 31 00:00:06 openqaworker13 systemd[1]: Started Rotate log files.
Actions