Project

General

Profile

Actions

action #136370

closed

systemd service rsnapshot@beta on backup-vm.qe.nue2.suse.org failed due to process conflict

Added by okurz 9 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2023-09-23
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

From https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services?orgId=1&from=now-6h&to=now
2023-09-23 08:22:00
backup-vm

rsnapshot@beta

From ssh backup-vm.qe.nue2.suse.org "sudo systemctl status rsnapshot@beta"

Sep 23 03:30:00 backup-vm systemd[1]: Starting rsnapshot (beta) backup...
Sep 23 03:30:00 backup-vm rsnapshot[21512]: ----------------------------------------------------------------------------
Sep 23 03:30:00 backup-vm rsnapshot[21512]: rsnapshot encountered an error! The program was invoked with these options:
Sep 23 03:30:00 backup-vm rsnapshot[21512]: /usr/bin/rsnapshot beta
Sep 23 03:30:00 backup-vm rsnapshot[21512]: ----------------------------------------------------------------------------
Sep 23 03:30:00 backup-vm rsnapshot[21512]: ERROR: Lockfile /var/run/rsnapshot.pid exists and so does its process, can not continue
Sep 23 03:30:00 backup-vm rsnapshot[21532]: /usr/bin/rsnapshot beta: ERROR: Lockfile /var/run/rsnapshot.pid exists and so does its process, can not continue
Sep 23 03:30:00 backup-vm systemd[1]: rsnapshot@beta.service: Main process exited, code=exited, status=1/FAILURE
Sep 23 03:30:00 backup-vm systemd[1]: rsnapshot@beta.service: Failed with result 'exit-code'.
Sep 23 03:30:00 backup-vm systemd[1]: Failed to start rsnapshot (beta) backup.

Suggestions

  • Look which other jobs ran at the time, likely another level conflicting with this one (alpha, gamma, etc.)

Rollback actions

  • Remove silence for failed systemd services

Related issues 1 (0 open1 closed)

Related to openQA Infrastructure - action #134519: We were not notified that backup.qa.suse.de did not create backups size:MResolvedlivdywan2023-08-23

Actions
Actions #1

Updated by okurz 9 months ago

  • Priority changed from High to Low
  • Target version changed from Ready to future

Alert disappeared so I suspect the next day the problem didn't show again. Let's see if this happens again

Actions #2

Updated by okurz 9 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz
  • Priority changed from Low to High
  • Target version changed from future to Ready

Now rsnapshot@gamma makes a problem

Sep 30 03:30:00 backup-vm rsnapshot[8046]: ------------------------------------------------------------>
Sep 30 03:30:00 backup-vm rsnapshot[8046]: rsnapshot encountered an error! The program was invoked with>
Sep 30 03:30:00 backup-vm rsnapshot[8046]: /usr/bin/rsnapshot gamma
Sep 30 03:30:00 backup-vm rsnapshot[8046]: ------------------------------------------------------------>
Sep 30 03:30:00 backup-vm rsnapshot[8046]: ERROR: Lockfile /var/run/rsnapshot.pid exists and so does it>
Sep 30 03:30:00 backup-vm rsnapshot[8065]: /usr/bin/rsnapshot gamma: ERROR: Lockfile /var/run/rsnapshot>
Sep 30 03:30:00 backup-vm systemd[1]: rsnapshot@gamma.service: Main process exited, code=exited, status>
Sep 30 03:30:00 backup-vm systemd[1]: rsnapshot@gamma.service: Failed with result 'exit-code'.
Sep 30 03:30:00 backup-vm systemd[1]: Failed to start rsnapshot (gamma) backup.
Actions #3

Updated by okurz 9 months ago

  • Description updated (diff)
Actions #4

Updated by okurz 9 months ago

  • Related to action #134519: We were not notified that backup.qa.suse.de did not create backups size:M added
Actions #6

Updated by okurz 9 months ago

  • Status changed from In Progress to Resolved

merged and deployed.

# systemctl list-timers
NEXT                         LEFT               LAST                         PASSED        UNIT                         ACTIVATES                     
Mon 2023-10-02 12:00:00 CEST 56min left         Mon 2023-10-02 08:00:00 CEST 3h 3min ago   rsnapshot-alpha.timer        rsnapshot@alpha.service
Mon 2023-10-02 12:00:00 CEST 56min left         Mon 2023-10-02 11:00:00 CEST 3min 40s ago  snapper-timeline.timer       snapper-timeline.service
Tue 2023-10-03 00:00:00 CEST 12h left           Mon 2023-10-02 00:00:00 CEST 11h ago       logrotate.timer              logrotate.service
Tue 2023-10-03 00:21:36 CEST 13h left           Mon 2023-10-02 00:46:40 CEST 10h ago       backup-sysconfig.timer       backup-sysconfig.service
Tue 2023-10-03 00:54:21 CEST 13h left           Mon 2023-10-02 00:43:14 CEST 10h ago       check-battery.timer          check-battery.service
Tue 2023-10-03 01:39:03 CEST 14h left           Mon 2023-10-02 01:35:40 CEST 9h ago        backup-rpmdb.timer           backup-rpmdb.service
Tue 2023-10-03 03:14:35 CEST 16h left           Mon 2023-10-02 03:14:40 CEST 7h ago        auto-upgrade.timer           auto-upgrade.service
Tue 2023-10-03 03:30:00 CEST 16h left           Mon 2023-10-02 03:30:00 CEST 7h ago        rsnapshot-beta.timer         rsnapshot@beta.service
Tue 2023-10-03 03:47:20 CEST 16h left           Mon 2023-10-02 03:47:20 CEST 7h ago        snapper-cleanup.timer        snapper-cleanup.service
Tue 2023-10-03 03:51:47 CEST 16h left           Mon 2023-10-02 03:51:47 CEST 7h ago        systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sat 2023-10-07 03:00:00 CEST 4 days left        Sat 2023-09-30 03:30:00 CEST 2 days ago    rsnapshot-gamma.timer        rsnapshot@gamma.service
Mon 2023-10-09 00:00:00 CEST 6 days left        Mon 2023-10-02 00:00:00 CEST 11h ago       btrfs-balance.timer          btrfs-balance.service
Mon 2023-10-09 00:00:00 CEST 6 days left        Mon 2023-10-02 00:00:00 CEST 11h ago       btrfs-trim.timer             btrfs-trim.service
Mon 2023-10-09 01:32:17 CEST 6 days left        Mon 2023-10-02 00:56:30 CEST 10h ago       fstrim.timer                 fstrim.service
Wed 2023-11-01 00:00:00 CET  4 weeks 1 day left Sun 2023-10-01 00:00:00 CEST 1 day 11h ago btrfs-scrub.timer            btrfs-scrub.service
Wed 2023-11-01 02:00:00 CET  4 weeks 1 day left Sun 2023-10-01 02:00:00 CEST 1 day 9h ago  rsnapshot-delta.timer        rsnapshot@delta.service

looks good. I called systemctl reset-failed and removed the silence.

Actions

Also available in: Atom PDF