QA (public) » openQA Project (public) » openQA Infrastructure (public)

we might want to block on #181301, otherwise the AC allow a completely unmanaged script on OSD which is hardly an improvement.

@okurz @livdywan Do we want email alerts of failed attempts or failure case? cron is capable of sending email on job/script failure.

gpathak wrote in #note-7:

@okurz @livdywan Do we want email alerts of failed attempts or failure case? cron is capable of sending email on job/script failure.

Yes but we already have alerts for failed systemd services so I want that, not any custom solution :)

I did a quick test as suggested by @okurz, on a fresh TW VM installed openQA from OBS repository https://download.opensuse.org/repositories/devel:/openQA:/GitHub:/os-autoinst:/openQA:/PR-6437/openSUSE_Tumbleweed/devel:openQA:GitHub:os-autoinst:openQA:PR-6437.repo of the Pull Request.

The timers/service unit didn't get activated automatically as can be seen the screenshot.

The service unit files are dependent on openqa-setup-db.service and postgresql.service, since we instruct the user to enable and start postgresql service after a fresh installation, the other units dependent on this service gets activated/started automatically on every boot and timers also kicks in as soon as the service units are activated.

Continuing with suggestions on the PR: https://github.com/os-autoinst/openQA/pull/6437

gpathak wrote in #note-10:

• systemd timer addition: https://github.com/os-autoinst/openQA/pull/6437
• cronjob removal: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1457

These two merge requests should be merged one after another.

livdywan wrote in #note-15:

I saw two emails again this morning. If it takes longer to implement a clean solution it should be mitigated. Raising priority accordingly.

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1458

FYI merged this one, too.

gpathak wrote in #note-17:

gpathak wrote in #note-10:

• systemd timer addition: https://github.com/os-autoinst/openQA/pull/6437
• cronjob removal: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1457

These two merge requests should be merged one after another.

One more merge request in backup-server-salt repository: https://gitlab.suse.de/qa-sle/backup-server-salt/-/merge_requests/25

The update was deployed to o3. I enabled the timer and executed the service once. I moved /var/lib/openqa/SQL-DUMPS/* to /var/lib/openqa/backup/. Merged https://gitlab.suse.de/qa-sle/backup-server-salt/-/merge_requests/25. Deployed changes to backup.qa.suse.de.

okurz wrote in #note-21:

The update was deployed to o3. I enabled the timer and executed the service once. I moved /var/lib/openqa/SQL-DUMPS/* to /var/lib/openqa/backup/. Merged https://gitlab.suse.de/qa-sle/backup-server-salt/-/merge_requests/25. Deployed changes to backup.qa.suse.de.

On o3 the timer is enabled but inactive

gpathak@ariel:~> sudo systemctl status openqa-dump-db.timer 
○ openqa-dump-db.timer - Daily openQA database dump and cleanup task
     Loaded: loaded (/usr/lib/systemd/system/openqa-dump-db.timer; enabled; preset: disabled)
     Active: inactive (dead)
    Trigger: n/a
   Triggers: ● openqa-dump-db.service
gpathak@ariel:~> sudo systemctl status op^Cqa-dump-db.timer 
gpathak@ariel:~>

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1457 deployed. I removed the file OSD:/etc/cron.d/dump-openqa with content

40 23 * * * postgres openqa-dump-db 1>/dev/null

manually. Called `systemctl list-timers which shows that openqa-dump-timer is next triggered in 11h which looks good.

As the SQL database backup and sync to backup hosts is critical I suggest to monitor this over the next days, i.e. check again after the weekend.

I saw the file /etc/cron.d/dump-openqa on OSD again, removed it.