QA » openQA Project » openQA Tests

@yosun do you have an idea how we can improve the test feedback again? In before we had a more helpful test module name "acceptance_fs_stress" failing, now it is "execute_test_run".

okurz wrote:

yosun wrote:

In this fail case, some time you can't get useful log in snapshot, but by looking into console you may found some Call trace happened.
https://openqa.suse.de/tests/1561334/file/serial0.txt

Well, sorry that does not help because I am mainly asking because the label carry over, the test overview page as well as openqa-review mainly rely on the name of the first test module failing in a scenario.

Most kernel test fails need to be check via console and tarbal(if has), when you see a fail.

Well, actually there is a better way. I implemented a simple y2log parser for installation and yast failures already and slindomansilla has adopted it in a cool way for the systemd-testsuite in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4795/files so I am thinking that a similar way could be applied for the kernel test failures as well.

Nice suggestion, it will reduce the workforce to analysis log. In that case, this ticket need to split into many ticket for particular tests in different set of script. Kernel fails are little bit complicated than userspace's, you need to check test log to find minor issues, to check /var/log/messages or journal log for other issue, and to check kdump files for crash bugs. I suggest to create subtask to follow.
For kernel tests part split to:

• ctcs2 based testcase
• xfstests
• ltp
• network ...

BTW, those three stress test, most likely a kernel acceptance test, if you think it fails most likely in kernel way and better to debug in kernel-qa team. Then feel free to transfer them into kernel job group for better classification.

That's actually a good idea. I will discuss with marita as PM and sebchlad as kernel&network PO.

Any update in this part?

no response, remove assignee.

I just need to talk to sebchlad and mawerner about this but I keep forgetting :(

@sebchlad as discussed. IIUC QSK is fine to take over the "…_stress" test suites and schedule according scenarios within SLE12 and SLE15 tests as well as improve them over time and hopefully also schedule them for relevant openSUSE tests. So I propose that you remove the according scenarios from the SLE15 and SLE12 schedule and add the according ones in the Kernel job group. OK? If you expect QSF to do something feel free to reassign to me, otherwise remove the "[functional]" tag and add "[kernel]" after moving the test scenarios.

sched_stress, fs_stress, process_stress are not run in kernel job group any more.

but they are! E.g. https://openqa.suse.de/tests/3527726 in Kernel for SLE15SP2. Also the ticket is not about "tests should not be run in kernel group" but about a too generic test failure in "execute_test_run". Please see the initial ticket description for the actual issue. Btw, mau-qa_acceptance_fs_stress seems to handle this better already with QA_TESTSET=acceptance_fs_stress instead of QA_TESTSUITE=fs_stress. Maybe that's the easy fix.

okurz wrote:

but they are! E.g. https://openqa.suse.de/tests/3527726 in Kernel for SLE15SP2.

Our bad. We decided to not run stress tests in kernel group and removed it for SLE12-SP5 but forgot about SLE15 group. Now they are removed from that group as well.

Also the ticket is not about "tests should not be run in kernel group" but about a too generic test failure in "execute_test_run". Please see the initial ticket description for the actual issue. Btw, mau-qa_acceptance_fs_stress seems to handle this better already with QA_TESTSET=acceptance_fs_stress instead of QA_TESTSUITE=fs_stress. Maybe that's the easy fix.

I know this ticket is not about that. Anyway, I have removed QA_TESTSUITE=fs_stress and added QA_TESTSET=acceptance_fs_stress in the test settings in case someone wants to enable that test again for whatever purpose.