Salt automation: auto-deploy and monitoring
Our general Salt goals are
- all core services to be in salt
- automating our automation
- better monitoring/reporting
This time we will focus on auto-deploy and Monitoring
The following solutions were proposed :
trigger the state.apply via the CI and send the output on monitoring
The auto-deploy via the CI can be done in the similar way as we are using for openqa.
Oliver Kurz offer us help in this are.
Marin Caj and Ricardo Klein will investigate monitoring options for salt so
we store changes and status in Monitoring tool.
https://gitlab.suse.de/openqa/salt-states-openqa/blob/master/.gitlab-ci.yml has the important pieces that we use already. I can propose something for a salt repo on gitlab.i.o.o
- Private changed from Yes to No
I can't access gitlab.suse.de, therefore I'm looking forward to your MR to gitlab.infra.o.o ;-)
A little warning: Some servers have a "salt timebomb" (aka manually edited config files which are not synced back to salt), therefore please send out a warning before actually enabling this everywhere. IIRC this affects anna/elsa, daffy* (see the open MRs for keepalived) and mirrordb* (postgresql config files). This shouldn't stop us from auto-deploying changes in salt, but obviously we need to get these servers re-salted before ;-)
Update on the "salt timebombs" mentioned in my previous comment:
- the keepalived config for anna/elsa and daffy* in salt is in sync with reality since yesterday
- for the postgresql pg_hba.conf on mirrordb1, I just submitted https://gitlab.infra.opensuse.org/infra/salt/merge_requests/302
As soon as MR 302 gets accepted, the known (to me) salt timebombs are gone, and we can apply a highstate everywhere again.