action #152599: [alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de` - openQA Infrastructure - openSUSE Project Management Tool

Actions

Copy link

action #152599

closed

[alert] `rsnapshot@alpha.service` failed on `backup.qa.suse.de`

Added by mkittler 5 months ago. Updated 5 months ago.

Status:

Rejected

Priority:

High

Assignee:

mkittler

Category:

Target version:

openQA Project - Ready

Start date:

2023-12-13

Due date:

% Done:

Estimated time:

Tags:

alert

Description

Observation¶

martchus@backup-vm:~> sudo systemctl status rsnapshot@alpha.service
× rsnapshot@alpha.service - rsnapshot (alpha) backup
     Loaded: loaded (/etc/systemd/system/rsnapshot@.service; static)
     Active: failed (Result: exit-code) since Wed 2023-12-13 16:20:27 CET; 49min ago
TriggeredBy: ● rsnapshot-alpha.timer
   Main PID: 14765 (code=exited, status=1/FAILURE)

Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15411]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15412]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:19:08 backup-vm rsnapshot[14765]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:19:08 backup-vm rsnapshot[18048]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:20:27 backup-vm rsnapshot[18290]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 16:20:27 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by mkittler 5 months ago

Status changed from In Progress to Feedback

Full log with the error:

Dec 13 16:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 16:00:08 backup-vm rsnapshot[14831]: Welcome to ariel.infra.opensuse.org!
Dec 13 16:00:08 backup-vm rsnapshot[14831]: ************************
Dec 13 16:00:08 backup-vm rsnapshot[14831]: * Managed by SaltStack *
Dec 13 16:00:08 backup-vm rsnapshot[14831]: ************************
Dec 13 16:00:08 backup-vm rsnapshot[14831]: Make sure changes get pushed into the state repo!
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ----------------------------------------------------------------------------
Dec 13 16:03:36 backup-vm rsnapshot[14765]: rsnapshot encountered an error! The program was invoked with these options:
Dec 13 16:03:36 backup-vm rsnapshot[14765]: /usr/bin/rsnapshot alpha
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ----------------------------------------------------------------------------
Dec 13 16:03:36 backup-vm rsnapshot[14765]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/lib/openqa/SQL-DUMPS/
Dec 13 16:03:36 backup-vm rsnapshot[15410]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/lib/openqa/SQL-DUMPS/
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15411]: WARNING: root@o3:/var/log/zypp/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[14765]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:03:36 backup-vm rsnapshot[15412]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 16:19:08 backup-vm rsnapshot[14765]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:19:08 backup-vm rsnapshot[18048]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 16:20:27 backup-vm rsnapshot[18290]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 16:20:27 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 16:20:27 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.

That's not very detailed but likely means that the underlying SSH connection broke. So maybe just a single incident. I restarted the service to see whether it works now.

Actions

Copy link

Updated by tinita 5 months ago

It happened again:

Dec 13 20:03:43 backup-vm rsnapshot[25914]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[26587]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[25914]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:03:43 backup-vm rsnapshot[26588]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:20:57 backup-vm rsnapshot[25914]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:20:58 backup-vm rsnapshot[29512]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:21:50 backup-vm rsnapshot[29683]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 20:21:50 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.

I restarted it.

Actions

Copy link

Updated by okurz 5 months ago

Status changed from Feedback to In Progress

If it happened again then we should do more than just restarting though

Actions

Copy link

Updated by mkittler 5 months ago

It failed again but then further runs succeeded:

Dec 13 20:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 20:00:03 backup-vm rsnapshot[25981]: Welcome to ariel.infra.opensuse.org!
Dec 13 20:00:03 backup-vm rsnapshot[25981]: ************************
Dec 13 20:00:03 backup-vm rsnapshot[25981]: * Managed by SaltStack *
Dec 13 20:00:03 backup-vm rsnapshot[25981]: ************************
Dec 13 20:00:03 backup-vm rsnapshot[25981]: Make sure changes get pushed into the state repo!
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ----------------------------------------------------------------------------
Dec 13 20:03:43 backup-vm rsnapshot[25914]: rsnapshot encountered an error! The program was invoked with these options:
Dec 13 20:03:43 backup-vm rsnapshot[25914]: /usr/bin/rsnapshot alpha
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ----------------------------------------------------------------------------
Dec 13 20:03:43 backup-vm rsnapshot[25914]: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[26587]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsync returned 255 while processing root@o3:/var/log/zypp/
Dec 13 20:03:43 backup-vm rsnapshot[25914]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:03:43 backup-vm rsnapshot[26588]: WARNING: root@o3:/srv/tftpboot/ skipped due to rollback plan
Dec 13 20:20:57 backup-vm rsnapshot[25914]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:20:58 backup-vm rsnapshot[29512]: WARNING: Rolling back "openqa.opensuse.org/"
Dec 13 20:21:50 backup-vm rsnapshot[29683]: /usr/bin/rsnapshot alpha: ERROR: /usr/bin/rsnapshot alpha: completed, but with some errors
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Main process exited, code=exited, status=1/FAILURE
Dec 13 20:21:50 backup-vm systemd[1]: rsnapshot@alpha.service: Failed with result 'exit-code'.
Dec 13 20:21:50 backup-vm systemd[1]: Failed to start rsnapshot (alpha) backup.
Dec 13 22:53:01 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 13 22:53:06 backup-vm rsnapshot[27954]: Welcome to ariel.infra.opensuse.org!
Dec 13 22:53:06 backup-vm rsnapshot[27954]: ************************
Dec 13 22:53:06 backup-vm rsnapshot[27954]: * Managed by SaltStack *
Dec 13 22:53:06 backup-vm rsnapshot[27954]: ************************
Dec 13 22:53:06 backup-vm rsnapshot[27954]: Make sure changes get pushed into the state repo!
Dec 13 23:15:11 backup-vm rsnapshot[31884]: /usr/bin/rsnapshot alpha: completed successfully
Dec 13 23:15:11 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 13 23:15:11 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 00:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 00:00:07 backup-vm rsnapshot[7501]: Welcome to ariel.infra.opensuse.org!
Dec 14 00:00:07 backup-vm rsnapshot[7501]: ************************
Dec 14 00:00:07 backup-vm rsnapshot[7501]: * Managed by SaltStack *
Dec 14 00:00:07 backup-vm rsnapshot[7501]: ************************
Dec 14 00:00:07 backup-vm rsnapshot[7501]: Make sure changes get pushed into the state repo!
Dec 14 00:03:17 backup-vm rsnapshot[7501]: removed '/var/lib/openqa/SQL-DUMPS/2023-12-05.dump'
Dec 14 00:19:17 backup-vm rsnapshot[10744]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 00:19:17 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 00:19:17 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 04:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 04:00:01 backup-vm rsnapshot[31642]: Welcome to ariel.infra.opensuse.org!
Dec 14 04:00:01 backup-vm rsnapshot[31642]: ************************
Dec 14 04:00:01 backup-vm rsnapshot[31642]: * Managed by SaltStack *
Dec 14 04:00:01 backup-vm rsnapshot[31642]: ************************
Dec 14 04:00:01 backup-vm rsnapshot[31642]: Make sure changes get pushed into the state repo!
Dec 14 04:16:54 backup-vm rsnapshot[2179]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 04:16:54 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 04:16:54 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.
Dec 14 08:00:00 backup-vm systemd[1]: Starting rsnapshot (alpha) backup...
Dec 14 08:00:10 backup-vm rsnapshot[9483]: Welcome to ariel.infra.opensuse.org!
Dec 14 08:00:10 backup-vm rsnapshot[9483]: ************************
Dec 14 08:00:10 backup-vm rsnapshot[9483]: * Managed by SaltStack *
Dec 14 08:00:10 backup-vm rsnapshot[9483]: ************************
Dec 14 08:00:10 backup-vm rsnapshot[9483]: Make sure changes get pushed into the state repo!
Dec 14 08:21:54 backup-vm rsnapshot[13174]: /usr/bin/rsnapshot alpha: completed successfully
Dec 14 08:21:54 backup-vm systemd[1]: rsnapshot@alpha.service: Deactivated successfully.
Dec 14 08:21:54 backup-vm systemd[1]: Finished rsnapshot (alpha) backup.

So maybe just temporary networking issues. I'll keep having a look at the situation.

Actions

Copy link