QA » openQA Project » openQA Infrastructure

@osukup could you please update us after Monday if mcaj did update the monitoring repository so the grafana 9 package can be build for leap15.4?

after short talk with @mcaj, meta of project was updated to build 15.4 and 15.5 repositories against Update .. so with Requires: go >= 1.19 grafana-9 now should be buildable in project

Discussed in daily 2023-03-15 and we found that https://monitor.qa.suse.de was automatically upgraded to grafana 9.3.6 and we rolled back the alerting for now, see #125303. So we can experiment with the new version, e.g. either locally in a container/VM, on an openQA staging instance like openqa-staging-1.qa.suse.de or in production

Looks like we currently provision dashboards via files on disk. I suppose therefore it makes sense to do the same for alerts which means following option 1 from https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources (https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning).

Setting due date based on mean cycle time of SUSE QE Tools

I suggest we could proceed with that ticket in a pair-programming session, e.g. after the infra meeting. I suppose I could benefit a lot from doing it together with Nick or Oliver. Maybe you have better ideas for importing production data locally or we just enable unified alerting in production again and continue with the already mentioned steps from there.

Ok, so copying the DB and enabling ldap on my local instance made it work. I could export everything. The file is over 30000 lines long but Kate and Vim handle it well. I couldn't find a way to delete all alerts so I've just kept all alerts there, moved the exported YAML to the provisioning directory and restarted the service. It seems that not deleting the alerts before is not a big deal. In fact, I've ended up with the same number of alerts than before (no duplicates) and all have the "provisioned" label. So we can just skip the deletion part.

Maybe it makes still sense to enable unified alerting in production first and export stuff from there - just in case my local version misses something. Before that, I'll prepare how the salt states change would look like.

We're back using unified alerting and my MR for the provisioning has been merged.

Just to recap, it had been created using the following logic to split the exported file and discard certain alerts:

my %new_groups_by_dashboard;
my $groups = $yaml->{groups};
for my $group (@$groups) {
  # get rid of all alerts that haven't been provisioned so far
  next unless $group->{folder} eq 'Salt';

  # create a separate set of groups per dashboard
  die "$group->{name} has no rules\n" unless my $rules = $group->{rules};
  die "$group->{name} has not exactly one rule\n" unless @$rules == 1;
  die "$group->{name} has not dashboardUid\n" unless my $dashboard_uid = $rules->[0]->{dashboardUid};
  my $groups = $new_groups_by_dashboard{$dashboard_uid} //= [];
  push @$groups, $group;
}

my $output_dir = path("$ENV{OPENQA_BASEDIR}/repos/salt-states-openqa/monitoring/grafana/alerting");
$output_dir->make_path->remove_tree({keep_root => 1});
for my $key (keys %new_groups_by_dashboard) {
  my $new_yaml = {apiVersion => 1, groups => $new_groups_by_dashboard{$key}};
  my $output_file = path($output_dir, "dashboard-$key.yaml");
  $output_file->spurt($dump_yaml->($new_yaml));
}

Today I've restarted Grafana and unlike what I've experienced with Grafana 9.4 locally the conflict with existing alerts was a problem:

Mar 23 10:51:09 openqa-monitor grafana-server[7603]: Failed to start grafana. error: alert rules: a conflicting alert rule is found: rule title under the same organisation and folder should be unique
Mar 23 10:51:09 openqa-monitor grafana-server[7603]: alert rules: a conflicting alert rule is found: rule title under the same organisation and folder should be unique

So I've deleted the relevant 320 alerts manually via the web UI, restarted the service and now all alerts in the "Salt" show up as "provisioned".

So the first step was successful. Now we need to re-templatize the alerts:

The following files should be generated from a worker-specific template:

find -iname '*WD*' | sort
./dashboard-WDgrenache-1.yaml
./dashboard-WDmalbec.yaml
./dashboard-WDopenqaworker14.yaml
./dashboard-WDopenqaworker16.yaml
./dashboard-WDopenqaworker17.yaml
./dashboard-WDopenqaworker18.yaml
./dashboard-WDopenqaworker-arm-1.yaml
./dashboard-WDopenqaworker-arm-2.yaml
./dashboard-WDopenqaworker-arm-3.yaml
./dashboard-WDopenqaworker-arm-4.yaml
./dashboard-WDopenqaworker-arm-5.yaml
./dashboard-WDpowerqaworker-qam-1.yaml
./dashboard-WDQA-Power8-4-kvm.yaml
./dashboard-WDQA-Power8-5-kvm.yaml
./dashboard-WDworker10.yaml
./dashboard-WDworker11.yaml
./dashboard-WDworker12.yaml
./dashboard-WDworker13.yaml
./dashboard-WDworker2.yaml
./dashboard-WDworker3.yaml
./dashboard-WDworker5.yaml
./dashboard-WDworker6.yaml
./dashboard-WDworker8.yaml
./dashboard-WDworker9.yaml

The following files should be generated from a template for generic hosts:

find -iname '*GD*' | sort
./dashboard-GDbackup-vm.yaml
./dashboard-GDbaremetal-support.yaml
./dashboard-GDjenkins.yaml
./dashboard-GDopenqa-piworker.yaml
./dashboard-GDopenqaw5-xen.yaml
./dashboard-GDqamasternue.yaml
./dashboard-GDschort-server.yaml
./dashboard-GDstorage.yaml
./dashboard-GDtumblesle.yaml

The remaining files can stay as they are.

This corresponds to the split/templating we have for the legacy alert data.

MR for making generic/worker alerts templates: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/816

I haven't tested it yet but maybe we just merge it and find out how well it works. I'm also still not sure how to deal with UIDs, see https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/816#note_499823.

Looks like I'll also have to take care of alerts that originate from webui.services.json and certificates.json but that would be the next step.

I've noticed a problem with the file provisioning and unified alerting: One cannot edit alerts anymore via the web UI at all. The dashboard provisioning and legacy alerting allows to edit everything in the web UI and upon saving one gets a prompt to C&P the JSON. That's apparently not how it works for the new alerting. On the upside we can now easier make changed directly within YAML which is especially for the templated alerts very useful. However, sometimes it would be nice to play around in the web UI. Now one can only play around with the query in the query explorer anymore. I'm not sure how one would export a query from the query explorer as JSON (and not raw string), e.g. to update that query in one of our alert YAML files.

If we'd used the REST-API to update alerts we could use the x-disable-provenance header. Supposedly the alerts wouldn't then not be considered "Provisioned" anymore. Then the corresponding label in the web UI would disappear and the edit button reappear. I don't think there'd be a prompt containing the JSON/YAML upon saving. So we'd somehow needed to export the changes back manually. Hence I'm not sure whether using the REST-API would be a big win.

Note that currently we either needed to restart the Grfana service manually or call a special API route manually in order to make changes to provisioned alerts effective (see note in the official documentation). I suppose we should automate this. However, the other problems I've mentioned are likely something to discuss before.

Overall, this is not very nice. We will likely have to re-setup silences after merging the templating MR. We will likely loose the ability to edit alerts as conveniently via the web UI as it was possible before. Unfortunately, the legacy alerting is going to be removed in Grafana 10 so there's likely not way around this (except staying on Grafana < 10 forever).

After rebooting openqaworker13 we have received only received notifications for the "Disk I/O time alert". This is a manageable amount of mails. So although I believe that this alert does not fall under #122842 I have changed the notification channel back to normal. I will leave a message in the team chat about it. I have also just created #126962 for using templating consistently.