QA » openQA Project » openQA Infrastructure

Observation¶

[Alerting] QA-Power8-4-kvm: Disk I/O time alert

Metric name sdj
Value 26600.000

http://stats.openqa-monitor.qa.suse.de/d/WDQA-Power8-4-kvm/worker-dashboard-qa-power8-4-kvm?tab=alert&viewPanel=56720&orgId=1

Problem¶

We just recently in #110269 worked on Disk I/O time alerts, also on QA-Power8-4-kvm. Either we need to relax values even more, or there is a real hardware problem or we need to find different solutions, e.g. longer pending time.

Acceptance criteria¶

• AC1: No more alerts

Suggestions¶

• Check that there are no actual hardware issues e.g. using smartctl, do what https://progress.opensuse.org/issues/110269#note-12 says
• Bump the values again

• Why do we monitor the disk sdj? the machine seems to have only two real physical devices, sda and sdb. journalctl | grep sdj reports:

  May 22 03:33:08 QA-Power8-4-kvm kernel: sd 7:0:0:3: [sdj] Attached SCSI removable disk
  May 29 03:33:08 QA-Power8-4-kvm kernel: sd 7:0:0:3: [sdj] Attached SCSI removable disk
  Jun 05 03:33:07 QA-Power8-4-kvm kernel: sd 8:0:0:3: [sdj] Attached SCSI removable disk
  

We should make sure we do not care about such devices or do not even have these. The devices always show up during boot.

• Take a look what was done in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/682 to ignore certain devices, also see https://github.com/influxdata/telegraf/tree/master/plugins/inputs/diskio for options