openQA Project (public) » openQA Infrastructure (public)

I've just tried whether I can connect to the backup VM via SSH. I wanted to see what services are running and noticed that all the time I use systemctl status … the SSH connection is terminated, e.g.

backup-vm:~ # systemctl status dbus.service
Connection to backup.qa.suse.de closed by remote host.
Connection to backup.qa.suse.de closed.

The automatic backup for o3 was initially introduced with #44078 and that never covered osd so putting that into a separate ticket. Consider OSD out-of-scope and covered in the new ticket #94015

The host was stuck as it ran out-of-memory. It isn't clear what caused this condition. The automatic backups are actually stored under /home/rsnapshot/ and the files under /home/backup/ which are mentioned in the ticket description have been created manually (so it is no surprise that they're not updated). The actual automatic backups seem to work, e.g. triggering rsnapshot alpha manually worked and the cron configuration is actually in place.

the host is up since more than 176h …

I've now enabled our usual salt setup for the host so it should now reboot weekly.

The host has almost 4 GiB memory which should actually be more than enough considering only a small number of services runs on the host. Maybe it makes sense to add some basic graphs/alerts in Grafana for the backup host so we can keep an eye on it.

SR for adding generic the monitoring for the host: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/507

With https://gitlab.suse.de/openqa/salt-states-openqa/-/commit/da6e34e0121f1f4f4042ef3e4687873311e6e228 the systemd services monitoring/alert should now cover the backup host as well.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/507 merged. If you can ensure that our generic alerts like "failed systemd services" are covered for the backup host as well I think that should be enough. Having a check for free space on the backup host would be nice on top.

SR for fixing the memory alert: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/510

MR merged but deployment failed in https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/468624 , don't want to check manually if the relevant part was deployed as we should fix the problems in deployment anyway.

With https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/510 the memory alert works and I enabled it again.

And yes, the pipeline still fails. The remaining errors are about user creation. e.g.:

          ID: ldevulder
    Function: user.present
      Result: False
     Comment: These values could not be changed: {'home': '/home/ldevulder'}
     Started: 11:14:22.376449
    Duration: 27.562 ms

I still don't know why that's the case.

The memory alert is fixed and with https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/513#note_327858 also the ping-based alert. What remains are the strange user creation errors.

The user problem is fixed, /etc/passwd was broken again. I hope it won't break again.

I'm keeping this ticket open until https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/514 has been merged.

mkittler wrote:

The user problem is fixed, /etc/passwd was broken again. I hope it won't break again.

What do you mean with "broken again"? When has that happened in before?

The first time I ran salt on the machine the last entry in the file was broken causing many errors. Removing the line helped. At some point there were only these user-related errors remaining (but I haven't checked /etc/passwd again immediately because I haven't expected it to break again).

hm, ok. We create users properly with salt so I don't think we should try to do something on top. Well, good enough :) Can you verify AC1+AC3?

I've read the comment but currently the VPN is down so no, I cannot verify it at the moment.

• AC1: It looks good:

martchus@backup-vm:~> ls -ltra /home/rsnapshot
insgesamt 8
drwxr-xr-x  4 root root   50 25. Jun 2020  _delete.14889
drwxr-xr-x  4 root root   50 26. Jun 2020  _delete.15189
drwxr-xr-x  6 root root  109 29. Jan 04:04 delta.2
drwxr-xr-x  6 root root  109 26. Feb 04:03 delta.1
drwxr-xr-x  6 root root  109 26. Mär 04:04 delta.0
drwxr-xr-x  6 root root  109 14. Mai 04:05 gamma.3
drwxr-xr-x  6 root root  109 20. Mai 08:05 _delete.651
drwxr-xr-x  6 root root  109 21. Mai 04:12 gamma.2
drwxr-xr-x  6 root root  109 22. Mai 00:05 _delete.15401
drwxr-xr-x  6 root root  109 22. Mai 12:04 _delete.20003
drwxr-xr-x  4 root root   65 26. Mai 12:04 _delete.14362
drwxr-xr-x  6 root root  109 26. Mai 20:04 _delete.26284
drwxr-xr-x  6 root root  109 29. Mai 04:04 gamma.1
drwxr-xr-x  4 root root   65 29. Mai 12:04 _delete.8886
drwxr-xr-x  6 root root  109 29. Mai 16:04 _delete.16796
drwxr-xr-x  6 root root  109 29. Mai 20:04 _delete.22377
drwxr-xr-x  4 root root   65 29. Mai 20:04 _delete.12833
drwxr-xr-x  6 root root  109 30. Mai 00:04 _delete.2735
drwxr-xr-x  4 root root   50 30. Mai 08:00 _delete.14087
drwxr-xr-x  4 root root   50 31. Mai 16:00 _delete.23694
drwxr-xr-x  4 root root   50  2. Jun 00:00 _delete.13239
drwxr-xr-x  4 root root   50  2. Jun 20:00 _delete.16276
drwxr-xr-x  4 root root   50  3. Jun 00:00 _delete.22317
drwxr-xr-x  4 root root   50  3. Jun 08:00 _delete.27611
drwxr-xr-x  4 root root   50  3. Jun 12:00 _delete.1851
drwxr-xr-x  4 root root   50  4. Jun 04:00 _delete.8595
drwxr-xr-x  4 root root   50  4. Jun 08:00 _delete.12420
drwxr-xr-x  4 root root   50  4. Jun 16:00 _delete.16685
drwxr-xr-x  4 root root   50  4. Jun 20:00 _delete.19139
drwxr-xr-x 56 root root 4096 15. Jun 14:56 ..
drwxr-xr-x  6 root root  109 18. Jun 04:04 gamma.0
drwxr-xr-x  6 root root  109 23. Jun 04:03 beta.6
drwxr-xr-x  6 root root  109 24. Jun 04:03 beta.5
drwxr-xr-x  6 root root  109 25. Jun 04:03 beta.4
drwxr-xr-x  6 root root  109 26. Jun 04:03 beta.3
drwxr-xr-x  6 root root  109 27. Jun 04:03 beta.2
drwxr-xr-x  6 root root  109 28. Jun 04:03 beta.1
drwxr-xr-x  6 root root  109 29. Jun 04:03 beta.0
drwxr-xr-x  6 root root  109 29. Jun 12:04 alpha.5
drwxr-xr-x  6 root root  109 29. Jun 16:03 alpha.4
drwxr-xr-x  6 root root  109 29. Jun 20:04 alpha.3
drwxr-xr-x  6 root root  109 30. Jun 00:04 alpha.2
drwxr-xr-x  6 root root  109 30. Jun 04:04 alpha.1
drwxr-xr-x  6 root root  109 30. Jun 08:03 alpha.0
drwxr-xr-x 45 root root 4096 30. Jun 08:04 .

• AC3: The alert is configured in exactly the same way as for the worker dashboards so I expect it to work (see https://stats.openqa-monitor.qa.suse.de/d/GDbackup-vm/dashboard-for-backup-vm?tab=alert&editPanel=65090&viewPanel=65090&orgId=1&refresh=1m vs. https://stats.openqa-monitor.qa.suse.de/d/WDgrenache-1/worker-dashboard-grenache-1?tab=alert&editPanel=65090&viewPanel=65090&orgId=1&refresh=1m)

alright. So, resolve?