QA » openQA Project » openQA Infrastructure

I added information about the kernel versions that have been used in the SUSE IT ticket and also asked for help in https://chat.suse.de/channel/suse-it-ama?msg=hweMdK2ttE26nyxTc

I also updated https://status.opensuse.org/admin to reflect the current status.

I will try an older kernel myself:

zypper install --oldpackage kernel-default-4.12.14-lp151.28.36.1
grub2-set-default 'openSUSE, with Linux 4.12.14-lp151.28.36-default'
reboot

did not work, still booted most recent kernel. Trying first to preserve older version:

sed -i -n 's/latest-1/&,latest-2/p' /etc/zypp/zypp.conf
zypper install --oldpackage kernel-default-4.12.14-lp151.28.36.1
grub2-set-default 'openSUSE, with Linux 4.12.14-lp151.28.36-default'
grub2-set-default 'openSUSE, with Linux 4.12.14-lp151.28.40-default'
reboot

but the machine still reboots 4.12.14-lp151.28.44-default.

As it seems the biggest I/O load comes from workers trying to update their cache I try to reduce the pressure by temporarily stopping openqa worker instances on openqaworker4 plus the complete openqa-worker.target on openqaworker7:

ssh root@openqaworker4 "systemctl stop openqa-worker@{5..20}"
ssh root@openqaworker7 "systemctl stop openqa-worker.target"

this is not boot-persistent so will automatically enable again after next reboot the latest.

Installed hdparm and conducted hdparm -tT /dev/vdb:

 Timing cached reads:   6122 MB in  2.00 seconds = 3063.64 MB/sec
 Timing buffered disk reads:  14 MB in  3.28 seconds =   4.26 MB/sec

In comparison osd:

$ for i in a c e d ; do sudo hdparm -tT /dev/vd$i; done

/dev/vda:
 Timing cached reads:   3968 MB in  2.00 seconds = 1986.07 MB/sec
 Timing buffered disk reads: 368 MB in  3.03 seconds = 121.46 MB/sec

/dev/vdc:
 Timing cached reads:   3486 MB in  2.00 seconds = 1744.09 MB/sec
 Timing buffered disk reads: 214 MB in  3.02 seconds =  70.76 MB/sec

/dev/vde:
 Timing cached reads:   4238 MB in  2.00 seconds = 2121.77 MB/sec
 Timing buffered disk reads: 802 MB in  3.00 seconds = 267.22 MB/sec

/dev/vdd:
 Timing cached reads:   4800 MB in  2.00 seconds = 2401.98 MB/sec
 Timing buffered disk reads: 272 MB in  3.03 seconds =  89.69 MB/sec

No problems in cached reads but buffered disk reads suffer a lot on o3 in comparison.

Initially the machine was rebooted by gschlotter to bump up the storage from 4TB to 5TB which is certainly appreciated. gschlotter also kindly booted into older kernel
4.12.14-lp151.28.36-default but still same symptoms, so again most likely we need someone with access to the hypervisor and storage to find out the I/O regression.

Just for the sake of completeness, the openQA changes since the time of "last good", as found out from o3 /var/log/zypp/history:

$ git log1 --no-merges b26b51beb..2e73b2a4a
d509e69f4 (okurz/enhance/simplify_systemd_enque) Provide enqueue convenience scripts rather than systemd
c4827b6a1 (okurz/enhance/doc_clock) docs: Mention the importance of synchronized clocks for remote worker machines
3804e375d docs: Update "Run workers" to not only mention "virtual machines" as backend
2f5d14b7f The job result is a string and not a number
b0370449f (Martchus/result-size-2) Account freed disk space when deleting logs
721d73a2c (Martchus/restart-error) Turn warning about missing assets when restarting jobs into error
3f161b61b (okurz/enhance/license) Harmonize License copyright format
4495ec42e (kalikiana/job_groups_validation_docs) Validate parameters passed to job group API routes
03b69edf0 (kalikiana/audit-log-unify) Unify single/multi audit event rendering

the only thing that I could see remotely relevant for "I/O impact" would be "b0370449f (Martchus/result-size-2) Account freed disk space when deleting logs".

[03/04/2020 15:58:27] <Martchus> okurz: You could revert it hot-patch-style but it is so unlikely that I wouldn't revert the change in the repo. And it is unlikely because the gru process didn't even show up in iotop while everything was slow yesterday. (And the results cleanup task was not running anyways.)
[03/04/2020 15:59:26] <Martchus> And the function this PR changes is to my knowlege only executed within the cleanup task (I could check that again).
[03/04/2020 16:18:01] <okurz> Martchus: ok. Wasn't sure if there is anything e.g. running on inserting results or so. Of course, if it's only in the non-permanent minion jobs even less likely. Also, the problem today was very prominent when effectively no jobs were yet running, only requests to assets from cache services. And most of the according processes are just I/O stuck. We just don't know what the underlying storage is doing. Maybe it is even 
[03/04/2020 16:18:01] <okurz> doing something useful, i.e. full data integrity check after the volume was increased from 4TB to 5TB