QA » openQA Project » openQA Infrastructure

https://openqa.suse.de/tests/3513596#step/DOR000/7 as given in chat is 6 days old and the last deployment of os-autoinst on openqaworker13 is from Oct 16th (as of today 13 days old). The uptime is 74 days and there was no qemu update since Sep 8th.

As such I have serious doubt this is caused by infrastructure. Especially as https://openqa.suse.de/tests/3529082#next_previous shows it also 26 days ago: https://openqa.suse.de/tests/3426802#step/DOR000/7

coolo wrote:

https://openqa.suse.de/tests/3513596#step/DOR000/7 as given in chat is 6 days old and the last deployment of os-autoinst on openqaworker13 is from Oct 16th (as of today 13 days old). The uptime is 74 days and there was no qemu update since Sep 8th.

As such I have serious doubt this is caused by infrastructure. Especially as https://openqa.suse.de/tests/3529082#next_previous shows it also 26 days ago: https://openqa.suse.de/tests/3426802#step/DOR000/7

https://openqa.suse.de/tests/3426802#step/DOR000/7 is literally the only recorded case where DOR000 failed before last week (in QAM Single Incidents jobs). As such, I'm inclined to write that one off as a random hickup which would normally be ignored. Here's a breakdown of DOR000 failures in the entire recorded history of QAM Single Incidents:

12SP1@x86_64: 0 since 2019-08-21
12SP2@x86_64: 1 since 2019-10-23, 0 more since 2019-08-23
12SP2@ppc64le: 6 since 2019-10-23, 0 more since 2019-08-23
12SP3@x86_64: 3 since 2019-10-23, 0 more since 2019-08-21
12SP3@ppc64le: 11 since 2019-10-23, 0 more since 2019-08-21
12SP4@x86_64: 4 since 2019-10-23, 0 more since 2019-08-01
12SP4@ppc64le: 12 since 2019-10-23, 1 more since 2019-08-01
15GA@x86_64: 2 since 2019-10-23, 0 more since 2019-10-01
15GA@ppc64le: 9 since 2019-10-23, 0 more since 2019-10-01
15SP1@x86_64: 5 since 2019-10-23, 0 more since 2019-08-21
15SP1@ppc64le: 6 since 2019-10-23, 0 more since 2019-08-21

The 13 extra failures on 3 weeks old 15GA live patch 9 reproduced today are not included. As you can see, the problem was almost non-existent before 2019-10-23. Now it reproduces regularly even on kernels originally testes on 2019-10-04 when just 1 failure was recorded out of 96 DOR000 tests.

Some more relevant data:

• all failing DOR000@PPC64LE jobs in the last 6 days ran on QA-Power8-4 or QA-Power8-5.
• all passing DOR000@PPC64LE jobs in the last 6 days ran on powerqaworker-qam-1 and malbec
• all failing DOR000@x86_64 jobs in the last 6 days ran on openqaworker3, openqaworker10 or openqaworker13
• all passing DOR000@x86_64 jobs in the last 6 days ran on openqaworker2, openqaworker3, openqaworker5, openqaworker6, openqaworker7, openqaworker8, openqaworker9 or openqaworker10

openqaworker3 has 1 failure, otherwise passes
openqaworker10 has roughly half passes, half failures

We will have to see how the performance problem caused by 58823 impacts this. So once we deploy a fix for this, the workers should be better off.

This is a 'regression' caused by https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/203 - we basically prefer with this change reproducible failures over random timeouts. And this is what you see.

There is no good setting for this - so fix is to avoid these slow disks for running IO tests. tmpfs is no solution either due to out of disk errors, so we need to buy ssds/nvme :(

I deployed a partial revert

coolo wrote:

I deployed a partial revert

If you're trying to prevent random timeouts during VM shutdown due to delayed I/O buffer writeback, wouldn't it be better to simply flush VM buffers between test modules using the commit QEMU command?
https://qemu.weilnetz.de/doc/qemu-doc.html#index-commit

might reduce the propability but cant not prevent other VMs interfering or also within the "await_install" module where a lot of I/O happens and we need to abort the timeout just after that – before being able to sync to disk.