action #134837
Updated by livdywan 9 months ago
## Observation
From https://suse.slack.com/archives/C02CANHLANP/p1693394887125729 Paolo Stivanin made us aware that the test repository checkout for os-autoinst-distri-opensuse on OSD was not up-to-date. I found that the repository checkout had commits from 2023-08-29 but not the expected ones for 2023-08-30. `journalctl -e -u cron` showed as last entries
```
Aug 29 08:05:01 openqa CRON[11204]: (root) CMD (touch /var/lib/openqa/factory/repo/cvd/*)
Aug 29 08:05:02 openqa CRON[11199]: (root) CMDEND (touch /var/lib/openqa/factory/repo/cvd/*)
Aug 29 09:05:01 openqa CRON[19801]: (root) CMD (touch /var/lib/openqa/factory/repo/cvd/*)
Aug 29 10:05:01 openqa CRON[24320]: (root) CMD (touch /var/lib/openqa/factory/repo/cvd/*)
Aug 29 10:05:02 openqa CRON[24317]: (root) CMDEND (touch /var/lib/openqa/factory/repo/cvd/*)
```
and the service
```
# systemctl status cron
. cron.service - Command Scheduler
Loaded: loaded (/usr/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
Active: inactive (dead)
```
I called fetchneedles manually and started the cron service and it was fine again. But we did not receive any alert and no other notice that the service was enabled but not running. Can we find a way to be alerted about systemd services enabled but not running?
## Acceptance Criteria
* **AC1:** We are alerted about enabled services not running (or active?)
## Suggestions
* Research about how to find enabled systemd services not running and how to monitor for that
* We already have a check triggered by telegraf for failed services as well as masked services. Look into that and possible extend or be inspired by those solutions
* Ensure to have monitoring panels and alerts
## Workaround
Restart the service manually