action #152599: [alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de` - openQA Infrastructure - openSUSE Project Management Tool

Actions

Copy link

action #152599

closed

[alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de`

Added by mkittler 11 months ago. Updated 11 months ago.

Status:

Rejected

Priority:

High

Assignee:

mkittler

Category:

Target version:

openQA Project - Ready

Start date:

2023-12-13

Due date:

% Done:

Estimated time:

Tags:

alert

Description

Observation¶

martchus@backup-vm:~> sudo systemctl status rsnapshot@alpha.service
× rsnapshot@alpha.service - rsnapshot (alpha) backup
     Loaded: loaded (/etc/systemd/system/rsnapshot@.service; static)
     Active: failed (Result: exit-code) since Wed 2023-12-13 16:20:27 CET; 49min ago
TriggeredBy: ● rsnapshot-alpha.timer
   Main PID: 14765 (code=exited, status=1/FAILURE)

Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15411]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15412]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:19:08 backup-vm rsnapshot[14765]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:19:08 backup-vm rsnapshot[18048]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:20:27 backup-vm rsnapshot[18290]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 16:20:27 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by mkittler 11 months ago

Status changed from In Progress to Feedback

Full log with the error:

Dec 13 16:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 16:00:08 backup-vm rsnapshot[14831]: Welcome to ariel.infra.opensuse.org!
Dec 13 16:00:08 backup-vm rsnapshot[14831]: ************************
Dec 13 16:00:08 backup-vm rsnapshot[14831]: * Managed by SaltStack *
Dec 13 16:00:08 backup-vm rsnapshot[14831]: ************************
Dec 13 16:00:08 backup-vm rsnapshot[14831]: Make sure changes get pushed into the state repo!
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ----------------------------------------------------------------------------
Dec 13 16:03:36 backup-vm rsnapshot[14765]: rsnapshot encountered an error! The program was invoked with these options:
Dec 13 16:03:36 backup-vm rsnapshot[14765]: /usr/bin/rsnapshot alpha
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ----------------------------------------------------------------------------
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/lib/openqa/SQL-DUMPS/
Dec 13 16:03:36 backup-vm rsnapshot[15410]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/lib/openqa/SQL-DUMPS/
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15411]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15412]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:19:08 backup-vm rsnapshot[14765]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:19:08 backup-vm rsnapshot[18048]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:20:27 backup-vm rsnapshot[18290]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 16:20:27 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.

That's not very detailed but likely means that the underlying SSH connection broke. So maybe just a single incident. I restarted the service to see whether it works now.

Actions

Copy link

Updated by tinita 11 months ago

It happened again:

Dec 13 20:03:43 backup-vm rsnapshot[25914]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[26587]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[25914]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:03:43 backup-vm rsnapshot[26588]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:20:57 backup-vm rsnapshot[25914]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:20:58 backup-vm rsnapshot[29512]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:21:50 backup-vm rsnapshot[29683]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 20:21:50 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.

I restarted it.

Actions

Copy link

Updated by okurz 11 months ago

Status changed from Feedback to In Progress

If it happened again then we should do more than just restarting though

Actions

Copy link

Updated by mkittler 11 months ago

It failed again but then further runs succeeded:

Dec 13 20:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 20:00:03 backup-vm rsnapshot[25981]: Welcome to ariel.infra.opensuse.org!
Dec 13 20:00:03 backup-vm rsnapshot[25981]: ************************
Dec 13 20:00:03 backup-vm rsnapshot[25981]: * Managed by SaltStack *
Dec 13 20:00:03 backup-vm rsnapshot[25981]: ************************
Dec 13 20:00:03 backup-vm rsnapshot[25981]: Make sure changes get pushed into the state repo!
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ----------------------------------------------------------------------------
Dec 13 20:03:43 backup-vm rsnapshot[25914]: rsnapshot encountered an error! The program was invoked with these options:
Dec 13 20:03:43 backup-vm rsnapshot[25914]: /usr/bin/rsnapshot alpha
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ----------------------------------------------------------------------------
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[26587]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[25914]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:03:43 backup-vm rsnapshot[26588]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:20:57 backup-vm rsnapshot[25914]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:20:58 backup-vm rsnapshot[29512]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:21:50 backup-vm rsnapshot[29683]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 20:21:50 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.
Dec 13 22:53:01 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 22:53:06 backup-vm rsnapshot[27954]: Welcome to ariel.infra.opensuse.org!
Dec 13 22:53:06 backup-vm rsnapshot[27954]: ************************
Dec 13 22:53:06 backup-vm rsnapshot[27954]: * Managed by SaltStack *
Dec 13 22:53:06 backup-vm rsnapshot[27954]: ************************
Dec 13 22:53:06 backup-vm rsnapshot[27954]: Make sure changes get pushed into the state repo!
Dec 13 23:15:11 backup-vm rsnapshot[31884]: /usr/bin/rsnapshot alpha: completed successfully
Dec 13 23:15:11 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 13 23:15:11 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 00:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 00:00:07 backup-vm rsnapshot[7501]: Welcome to ariel.infra.opensuse.org!
Dec 14 00:00:07 backup-vm rsnapshot[7501]: ************************
Dec 14 00:00:07 backup-vm rsnapshot[7501]: * Managed by SaltStack *
Dec 14 00:00:07 backup-vm rsnapshot[7501]: ************************
Dec 14 00:00:07 backup-vm rsnapshot[7501]: Make sure changes get pushed into the state repo!
Dec 14 00:03:17 backup-vm rsnapshot[7501]: removed '/var/lib/openqa/SQL-DUMPS/2023-12-05.dump'
Dec 14 00:19:17 backup-vm rsnapshot[10744]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 00:19:17 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 00:19:17 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 04:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 04:00:01 backup-vm rsnapshot[31642]: Welcome to ariel.infra.opensuse.org!
Dec 14 04:00:01 backup-vm rsnapshot[31642]: ************************
Dec 14 04:00:01 backup-vm rsnapshot[31642]: * Managed by SaltStack *
Dec 14 04:00:01 backup-vm rsnapshot[31642]: ************************
Dec 14 04:00:01 backup-vm rsnapshot[31642]: Make sure changes get pushed into the state repo!
Dec 14 04:16:54 backup-vm rsnapshot[2179]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 04:16:54 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 04:16:54 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 08:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 08:00:10 backup-vm rsnapshot[9483]: Welcome to ariel.infra.opensuse.org!
Dec 14 08:00:10 backup-vm rsnapshot[9483]: ************************
Dec 14 08:00:10 backup-vm rsnapshot[9483]: * Managed by SaltStack *
Dec 14 08:00:10 backup-vm rsnapshot[9483]: ************************
Dec 14 08:00:10 backup-vm rsnapshot[9483]: Make sure changes get pushed into the state repo!
Dec 14 08:21:54 backup-vm rsnapshot[13174]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 08:21:54 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 08:21:54 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.

So maybe just temporary networking issues. I'll keep having a look at the situation.

Actions

Copy link