QA » openQA Project » openQA Infrastructure

why did we not catch this issue in our tests [...]?

We didn't catch it in our tests because the tests didn't have a bugref comments without a corresponding entry in the Bugs table.

tinita wrote:

[…]
The other is that it isn't really possible to mark single conditions as uncoverable.

What do you mean by that? https://metacpan.org/pod/Devel::Cover#Branches and https://metacpan.org/pod/Devel::Cover#Conditions explains how branches and conditions can be marked as uncoverable as well. Just that if we would really aim for 100% branch+condition coverage we would like go mad trying to mark all "uncoverable" conditions/branches as such.

ilausuch wrote:

• Check if we can filter the logs to cherry pick only the interesting logs to not overload the DB

but that's what openqa-logwarn is doing. Why not just call openqa-logwarn from telegraf and record the exit status, create a grafana dashboard for that with instructions how to check logs when the alert triggers

I think could be interesting to test this plugin (processor/parser)
https://github.com/influxdata/telegraf/blob/release-1.19/plugins/processors/parser/README.md

But also next article although old, is interesting because shows how to create a parser for a log file, and also use patterns. This could be really interesting because we could use this to analyze "[error] " pattern
https://www.influxdata.com/blog/telegraf-correlate-log-metrics-data-performance-bottlenecks/

okurz wrote:

ilausuch wrote:

• Check if we can filter the logs to cherry pick only the interesting logs to not overload the DB

but that's what openqa-logwarn is doing. Why not just call openqa-logwarn from telegraf and record the exit status, create a grafana dashboard for that with instructions how to check logs when the alert triggers

Ok, an other interesting approach

Following https://github.com/influxdata/telegraf/blob/master/plugins/inputs/exec/README.md the way to have metrics about logs should be creating a configuration

[[inputs.exec]]
command = "/root/openqa-logwarn/logwarn_openqa"
data_format = "influx"
interval = "120s"

Using the example in the repo logwarn_openqa give us

[Mon Sep 26 21:16:43 2016] [19382:error] File iso/bar.iso is not a registered asset
[Mon Sep 26 21:16:46 2016] [19382:error] Directory repo/baz.license is not a registered asset
[Mon Sep 26 21:17:27 2016] [30003:error] MISSING S:scheduled R:none
[Wed Mar 29 11:25:38 2017] [14008:warn] FOOBAR=RAID0:ppc64le not found - check for typos and dependency cycles
[2018-02-06T04:02:24.0435 CET] [error] The world is coming to an end

I created a proposal PR. It is not tested yet
https://github.com/os-autoinst/salt-states-openqa/pull/1
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/530

I tested the configuration creating a simple test with the lines that the MR introduces, and works in the moment we have logwarn installed and openqa_logwarn in the /opt directory

telegraf --test --debug --config telegraf-webui.conf  
2021-08-03T12:55:38Z I! Starting Telegraf 
2021-08-03T12:55:38Z D! [agent] Initializing plugins
2021-08-03T12:55:38Z D! [agent] Starting service inputs
2021-08-03T12:55:38Z D! [agent] Stopping service inputs
2021-08-03T12:55:38Z D! [agent] Input channel closed
2021-08-03T12:55:38Z D! [agent] Stopped Successfully

I tested also what happens if logwarn is not installed, openqa_logwarn is not in the /opt directory or /var/log/openqa is not present.

ilausuch wrote:

I tested also what happens if logwarn is not installed, openqa_logwarn is not in the /opt directory or /var/log/openqa is not present.

!530 uses pkg.installed and git.cloned so I would assume salt fails if either one is missing.

In other words, why does it matter what happens if logwarn is not installed? Maybe you can clarify what you expect here?

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/530 merged. I doubt we need to care more what happens if logwarn is not installed. The important part is about logwarn doing its job of warning about important new errors and not reporting about known stuff. Does this work?

The SR has been merged but I don't find all questions I've asked in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/530#note_335055 addressed in the comments here. It would have been nice to check whether the data gathered now can actually be used within Grafana. Accessing the data from Grafana could also help to answer the question raised in @okurz's last comment.

Notes from our conversation:

• The config should use commands = ["foobar"] like we also do here https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/monitoring/telegraf/telegraf-common.conf#L34
• https://github.com/os-autoinst/openqa-logwarn needs to be modified or forked to produce logwarn,tag1=a,tag2=b i=42i,j=43i,k=44i style output
• https://github.com/martchus/openqa-helper#monitoring-with-telegrafinfluxdbgrafana has suggestions on how to test this

Note that this kind of alerting would also be useful to have for worker logs (see #96710#note-23).