action #112196
Updated by okurz over 2 years ago
## Observation
[Alerting] QA-Power8-4-kvm: Disk I/O time alert
Metric name sdj
Value 26600.000
http://stats.openqa-monitor.qa.suse.de/d/WDQA-Power8-4-kvm/worker-dashboard-qa-power8-4-kvm?tab=alert&viewPanel=56720&orgId=1
## Problem
We just recently in #110269 worked on Disk I/O time alerts, also on QA-Power8-4-kvm. Either we need to relax values even more, or there is a real hardware problem or we need to find different solutions, e.g. longer pending time.
## Acceptance criteria
* **AC1:** No more alerts
## Suggestions
* Check that there are no actual hardware issues e.g. using smartctl, do what https://progress.opensuse.org/issues/110269#note-12 says
* Bump the values again
* Why do we monitor the disk sdj? the machine seems to have only two real physical devices, sda and sdb. `journalctl | grep sdj` reports:
```
May 22 03:33:08 QA-Power8-4-kvm kernel: sd 7:0:0:3: [sdj] Attached SCSI removable disk
May 29 03:33:08 QA-Power8-4-kvm kernel: sd 7:0:0:3: [sdj] Attached SCSI removable disk
Jun 05 03:33:07 QA-Power8-4-kvm kernel: sd 8:0:0:3: [sdj] Attached SCSI removable disk
```
We should make sure we do not care about such devices or do not even have these. The devices always show up during boot.
* Take a look what was done in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/682 to ignore certain devices, also see https://github.com/influxdata/telegraf/tree/master/plugins/inputs/diskio for options