action #136370
closed
systemd service rsnapshot@beta on backup-vm.qe.nue2.suse.org failed due to process conflict
Added by okurz 8 months ago.
Updated 8 months ago.
Description
Observation¶
From https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=now-6h&to=now
2023-09-23 08:22:00
backup-vm
rsnapshot@beta
From ssh backup-vm.qe.nue2.suse.org "sudo systemctl status rsnapshot@beta"
Sep 23 03:30:00 backup-vm systemd[1]: Starting rsnapshot (beta) backup...
Sep 23 03:30:00 backup-vm rsnapshot[21512]: ----------------------------------------------------------------------------
Sep 23 03:30:00 backup-vm rsnapshot[21512]: rsnapshot encountered an error! The program was invoked with these options:
Sep 23 03:30:00 backup-vm rsnapshot[21512]: /usr/bin/rsnapshot beta
Sep 23 03:30:00 backup-vm rsnapshot[21512]: ----------------------------------------------------------------------------
Sep 23 03:30:00 backup-vm rsnapshot[21512]: ERROR: Lockfile /var/run/rsnapshot.pid exists and so does its process, can not continue
Sep 23 03:30:00 backup-vm rsnapshot[21532]: /usr/bin/rsnapshot beta: ERROR: Lockfile /var/run/rsnapshot.pid exists and so does its process, can not continue
Sep 23 03:30:00 backup-vm systemd[1]: rsnapshot@beta.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 03:30:00 backup-vm systemd[1]: rsnapshot@beta.service: Failed with result 'exit-code'.
Sep 23 03:30:00 backup-vm systemd[1]: Failed to start rsnapshot (beta) backup.
Suggestions¶
- Look which other jobs ran at the time, likely another level conflicting with this one (alpha, gamma, etc.)
Rollback actions¶
- Remove silence for failed systemd services
- Priority changed from High to Low
- Target version changed from Ready to future
Alert disappeared so I suspect the next day the problem didn't show again. Let's see if this happens again
- Status changed from New to In Progress
- Assignee set to okurz
- Priority changed from Low to High
- Target version changed from future to Ready
Now rsnapshot@gamma makes a problem
Sep 30 03:30:00 backup-vm rsnapshot[8046]: ------------------------------------------------------------>
Sep 30 03:30:00 backup-vm rsnapshot[8046]: rsnapshot encountered an error! The program was invoked with>
Sep 30 03:30:00 backup-vm rsnapshot[8046]: /usr/bin/rsnapshot gamma
Sep 30 03:30:00 backup-vm rsnapshot[8046]: ------------------------------------------------------------>
Sep 30 03:30:00 backup-vm rsnapshot[8046]: ERROR: Lockfile /var/run/rsnapshot.pid exists and so does it>
Sep 30 03:30:00 backup-vm rsnapshot[8065]: /usr/bin/rsnapshot gamma: ERROR: Lockfile /var/run/rsnapshot>
Sep 30 03:30:00 backup-vm systemd[1]: rsnapshot@gamma.service: Main process exited, code=exited, status>
Sep 30 03:30:00 backup-vm systemd[1]: rsnapshot@gamma.service: Failed with result 'exit-code'.
Sep 30 03:30:00 backup-vm systemd[1]: Failed to start rsnapshot (gamma) backup.
- Description updated (diff)
- Related to action #134519: We were not notified that backup.qa.suse.de did not create backups size:M added
- Status changed from In Progress to Resolved
merged and deployed.
# systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Mon 2023-10-02 12:00:00 CEST 56min left Mon 2023-10-02 08:00:00 CEST 3h 3min ago rsnapshot-alpha.timer rsnapshot@alpha.service
Mon 2023-10-02 12:00:00 CEST 56min left Mon 2023-10-02 11:00:00 CEST 3min 40s ago snapper-timeline.timer snapper-timeline.service
Tue 2023-10-03 00:00:00 CEST 12h left Mon 2023-10-02 00:00:00 CEST 11h ago logrotate.timer logrotate.service
Tue 2023-10-03 00:21:36 CEST 13h left Mon 2023-10-02 00:46:40 CEST 10h ago backup-sysconfig.timer backup-sysconfig.service
Tue 2023-10-03 00:54:21 CEST 13h left Mon 2023-10-02 00:43:14 CEST 10h ago check-battery.timer check-battery.service
Tue 2023-10-03 01:39:03 CEST 14h left Mon 2023-10-02 01:35:40 CEST 9h ago backup-rpmdb.timer backup-rpmdb.service
Tue 2023-10-03 03:14:35 CEST 16h left Mon 2023-10-02 03:14:40 CEST 7h ago auto-upgrade.timer auto-upgrade.service
Tue 2023-10-03 03:30:00 CEST 16h left Mon 2023-10-02 03:30:00 CEST 7h ago rsnapshot-beta.timer rsnapshot@beta.service
Tue 2023-10-03 03:47:20 CEST 16h left Mon 2023-10-02 03:47:20 CEST 7h ago snapper-cleanup.timer snapper-cleanup.service
Tue 2023-10-03 03:51:47 CEST 16h left Mon 2023-10-02 03:51:47 CEST 7h ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sat 2023-10-07 03:00:00 CEST 4 days left Sat 2023-09-30 03:30:00 CEST 2 days ago rsnapshot-gamma.timer rsnapshot@gamma.service
Mon 2023-10-09 00:00:00 CEST 6 days left Mon 2023-10-02 00:00:00 CEST 11h ago btrfs-balance.timer btrfs-balance.service
Mon 2023-10-09 00:00:00 CEST 6 days left Mon 2023-10-02 00:00:00 CEST 11h ago btrfs-trim.timer btrfs-trim.service
Mon 2023-10-09 01:32:17 CEST 6 days left Mon 2023-10-02 00:56:30 CEST 10h ago fstrim.timer fstrim.service
Wed 2023-11-01 00:00:00 CET 4 weeks 1 day left Sun 2023-10-01 00:00:00 CEST 1 day 11h ago btrfs-scrub.timer btrfs-scrub.service
Wed 2023-11-01 02:00:00 CET 4 weeks 1 day left Sun 2023-10-01 02:00:00 CEST 1 day 9h ago rsnapshot-delta.timer rsnapshot@delta.service
looks good. I called systemctl reset-failed and removed the silence.
Also available in: Atom
PDF