action #152599
closed
[alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de`
Added by mkittler 5 months ago.
Updated 5 months ago.
Description
Observation¶
martchus@backup-vm:~> sudo systemctl status rsnapshot@alpha.service
× rsnapshot@alpha.service - rsnapshot (alpha) backup
Loaded: loaded (/etc/systemd/system/rsnapshot@.service; static)
Active: failed (Result: exit-code) since Wed 2023-12-13 16:20:27 CET; 49min ago
TriggeredBy: ● rsnapshot-alpha.timer
Main PID: 14765 (code=exited, status=1/FAILURE)
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15411]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15412]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:19:08 backup-vm rsnapshot[14765]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:19:08 backup-vm rsnapshot[18048]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:20:27 backup-vm rsnapshot[18290]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 16:20:27 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.
- Status changed from In Progress to Feedback
Full log with the error:
Dec 13 16:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 16:00:08 backup-vm rsnapshot[14831]: Welcome to ariel.infra.opensuse.org!
Dec 13 16:00:08 backup-vm rsnapshot[14831]: ************************
Dec 13 16:00:08 backup-vm rsnapshot[14831]: * Managed by SaltStack *
Dec 13 16:00:08 backup-vm rsnapshot[14831]: ************************
Dec 13 16:00:08 backup-vm rsnapshot[14831]: Make sure changes get pushed into the state repo!
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ----------------------------------------------------------------------------
Dec 13 16:03:36 backup-vm rsnapshot[14765]: rsnapshot encountered an error! The program was invoked with these options:
Dec 13 16:03:36 backup-vm rsnapshot[14765]: /usr/bin/rsnapshot alpha
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ----------------------------------------------------------------------------
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/lib/openqa/SQL-DUMPS/
Dec 13 16:03:36 backup-vm rsnapshot[15410]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/lib/openqa/SQL-DUMPS/
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15411]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15412]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:19:08 backup-vm rsnapshot[14765]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:19:08 backup-vm rsnapshot[18048]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:20:27 backup-vm rsnapshot[18290]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 16:20:27 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.
That's not very detailed but likely means that the underlying SSH connection broke. So maybe just a single incident. I restarted the service to see whether it works now.
It happened again:
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[26587]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[25914]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:03:43 backup-vm rsnapshot[26588]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:20:57 backup-vm rsnapshot[25914]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:20:58 backup-vm rsnapshot[29512]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:21:50 backup-vm rsnapshot[29683]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 20:21:50 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.
I restarted it.
- Status changed from Feedback to In Progress
If it happened again then we should do more than just restarting though
It failed again but then further runs succeeded:
Dec 13 20:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 20:00:03 backup-vm rsnapshot[25981]: Welcome to ariel.infra.opensuse.org!
Dec 13 20:00:03 backup-vm rsnapshot[25981]: ************************
Dec 13 20:00:03 backup-vm rsnapshot[25981]: * Managed by SaltStack *
Dec 13 20:00:03 backup-vm rsnapshot[25981]: ************************
Dec 13 20:00:03 backup-vm rsnapshot[25981]: Make sure changes get pushed into the state repo!
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ----------------------------------------------------------------------------
Dec 13 20:03:43 backup-vm rsnapshot[25914]: rsnapshot encountered an error! The program was invoked with these options:
Dec 13 20:03:43 backup-vm rsnapshot[25914]: /usr/bin/rsnapshot alpha
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ----------------------------------------------------------------------------
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[26587]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[25914]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:03:43 backup-vm rsnapshot[26588]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:20:57 backup-vm rsnapshot[25914]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:20:58 backup-vm rsnapshot[29512]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:21:50 backup-vm rsnapshot[29683]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 20:21:50 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.
Dec 13 22:53:01 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 22:53:06 backup-vm rsnapshot[27954]: Welcome to ariel.infra.opensuse.org!
Dec 13 22:53:06 backup-vm rsnapshot[27954]: ************************
Dec 13 22:53:06 backup-vm rsnapshot[27954]: * Managed by SaltStack *
Dec 13 22:53:06 backup-vm rsnapshot[27954]: ************************
Dec 13 22:53:06 backup-vm rsnapshot[27954]: Make sure changes get pushed into the state repo!
Dec 13 23:15:11 backup-vm rsnapshot[31884]: /usr/bin/rsnapshot alpha: completed successfully
Dec 13 23:15:11 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 13 23:15:11 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 00:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 00:00:07 backup-vm rsnapshot[7501]: Welcome to ariel.infra.opensuse.org!
Dec 14 00:00:07 backup-vm rsnapshot[7501]: ************************
Dec 14 00:00:07 backup-vm rsnapshot[7501]: * Managed by SaltStack *
Dec 14 00:00:07 backup-vm rsnapshot[7501]: ************************
Dec 14 00:00:07 backup-vm rsnapshot[7501]: Make sure changes get pushed into the state repo!
Dec 14 00:03:17 backup-vm rsnapshot[7501]: removed '/var/lib/openqa/SQL-DUMPS/2023-12-05.dump'
Dec 14 00:19:17 backup-vm rsnapshot[10744]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 00:19:17 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 00:19:17 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 04:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 04:00:01 backup-vm rsnapshot[31642]: Welcome to ariel.infra.opensuse.org!
Dec 14 04:00:01 backup-vm rsnapshot[31642]: ************************
Dec 14 04:00:01 backup-vm rsnapshot[31642]: * Managed by SaltStack *
Dec 14 04:00:01 backup-vm rsnapshot[31642]: ************************
Dec 14 04:00:01 backup-vm rsnapshot[31642]: Make sure changes get pushed into the state repo!
Dec 14 04:16:54 backup-vm rsnapshot[2179]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 04:16:54 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 04:16:54 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 08:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 08:00:10 backup-vm rsnapshot[9483]: Welcome to ariel.infra.opensuse.org!
Dec 14 08:00:10 backup-vm rsnapshot[9483]: ************************
Dec 14 08:00:10 backup-vm rsnapshot[9483]: * Managed by SaltStack *
Dec 14 08:00:10 backup-vm rsnapshot[9483]: ************************
Dec 14 08:00:10 backup-vm rsnapshot[9483]: Make sure changes get pushed into the state repo!
Dec 14 08:21:54 backup-vm rsnapshot[13174]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 08:21:54 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 08:21:54 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
So maybe just temporary networking issues. I'll keep having a look at the situation.
- Copied to action #152649: [alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de` size:M added
- Status changed from In Progress to Rejected
Also available in: Atom
PDF