Monitoring on log alerts, also on grafana log alerts, maybe grafana-loki?
cdywan and me realized that a gitlab webhook trigger sent from grafana got lost due to an issue with the EngInfra maintained gitlab issue. The error was mentioned in the logfile of grafana but nobody realized. Would be nice to have better monitoring of log files. We have "openqa_logwarn" but maybe nowadays other solutions are not that hard anymore :)
- AC1: Single entry point for aggregated logs is present for OSD infrastructure
- AC2: Alerts based on error messages can be defined in grafana