QA » openQA Project » openQA Tests

Hello,
@okurz I'm fairly new to openQA, so first thing first sorry for this and the past issues caused in prod by my limited knowledge about the topic :(

1) about https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6651 : yes, I didn't think about testing it also on Leap and TW. Next time I'll test the code really on every platform :)

2) about http://d502.qam.suse.de/tests/274. I didn't pay attention to the other tests because I thought they were a separate thing. I thought that openQA would treat each test individually and that they weren't related with each other. I didn't know that a test could cause a failure on another unrelated test.

3) I'm looking into it!

pstivanin wrote:

1) about https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6651 : yes, I didn't think about testing it also on Leap and TW. Next time I'll test the code really on every platform :)

Well, I guess testing on "every platform" is a bit too much. A more diligent PR review and also inviting reviewers outside QAM might be helpful. This goes much more to the existing reviewers than to you as new contributor and reviewee.

2) about http://d502.qam.suse.de/tests/274. I didn't pay attention to the other tests because I thought they were a separate thing. I thought that openQA would treat each test individually and that they weren't related with each other. I didn't know that a test could cause a failure on another unrelated test.

In principle, if a test module passes, the next modules just continue within the same session. Think of it as just checkpoints within the same shell session. These checkpoints are only used to fall back to in case of failures but because your test module did not fail it did continue. There is also the option to just run less tests within one scenario. You simply added your test module to the "catch-it-all" scenario :)

Ok, thanks for the explanation :)

• by scenario you mean the parent node? (eg "x11/evolution", "sysauth", "console") ?

• if I have correctly understood, the problem is that 8 tests are executed before zbar and dracut is the only one rebooting the instance (therefore resetting the console). And because the zbar test is expecting the console to be logged in, then the tests fails. Am I correct?

the zbar test is executing 'select_console' as first thing (line 21). From my understanding (http://open.qa/api/testapi/#_select_console) this method should do also the login if not yet done.
So how can a reboot affect this test?

As you can see from http://d502.qam.suse.de/tests/387 (oS 15.1) and http://d502.qam.suse.de/tests/388 (oS TW), disabling the reboot code in the dracut test made the zbar test green.

@okurz was right, rebooting the instance (therefore resetting the console) is the issue because zbar is expecting a different state.

Now I'll look deeper into zbar (but it will take a bit because I'm not yet really familiar with how to deal with needles), maybe it's enough to update the needle there.

pstivanin wrote:

• by scenario you mean the parent node? (eg "x11/evolution", "sysauth", "console") ?

Actually no. I am using the same wording as in http://open.qa/docs/#concepts

I think all the other open points you have found out yourself already or we have discussed in jangouts :)

stripped down test suite: http://d502.qam.suse.de/tests/416

• all green besides rail (which was already red).

Now I'm really puzzled:

zbar green:

• dracut (fixed version) and zbar: http://d502.qam.suse.de/tests/414
• stripped down test suite v1 (with fixed dracut): http://d502.qam.suse.de/tests/416
• stripped down test suite v2 (with fixed dracut): http://d502.qam.suse.de/tests/418
• complete test suite without dracut: http://d502.qam.suse.de/tests/427

zbar red:

• complete test suite (with fixed dracut): http://d502.qam.suse.de/tests/421

@okurz what do you think about idea to create extratest for dracut? as it does reboot during its tests, basically in similar way like it is for kdump_and_crash (to isolate it from others)

Petr's idea would work: http://d502.qam.suse.de/tests/428

• consoltest_finish was read already before (see 402.2 for ooo): https://openqa.opensuse.org/tests/overview?distri=opensuse&version=15.1&build=402.2&groupid=50

To summarize, I think we have 3 possibilities here:

1. create an extratest for dracut, as suggested by Petr
2. move dracut with kdump
3. decrease dracut's coverage by removing the reboot part

pstivanin wrote:

To summarize, I think we have 3 possibilities here:

1. create an extratest for dracut, as suggested by Petr
2. move dracut with kdump
3. decrease dracut's coverage by removing the reboot part

All of these can work, I prefer point 1.

latest TW build is green for zbar https://openqa.opensuse.org/tests/848123 because the needle was updated

Both oS 15.1 and TW are now green thanks to the needle update :)

What shall we do now? Should we still split dracut or not?

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6745

Test runs are still running (SLE 12.1, 12.4, 15.0 and oS Leap and TW)

Copying & pasting the email exchange Oliver and I had.

Me:

Hello Oliver,
let's take this link as an example: https://openqa.suse.de/tests/overvi
ew?distri=sle&version=15-SP1&build=161.1&groupid=110

What we want to achieve is to make dracut callable only when
EXTRATEST=dracut is set (like I did here https://github.com/os-autoinst
/os-autoinst-distri-opensuse/pull/6745), but we also want to "add
another row" on the table you see in the first link, right? So we would
have a new row in that table called "dracut" where only the dracut test
is executed, right?
Or do we want to keep dracut in mau-extratests and
extra_tests_in_textmode?

Oliver:

Given that the daily test reviewers, e.g. openSUSE release managers, are
frustated by the instability of the existing extra_tests_in_textmode I
think a dedicated scenario is the right approach at least for the build
validation job groups.
For this you need to go to
http://openqa.opensuse.org/admin/test_suites
add another test suite based on the example of extra_tests_in_textmode
which we already have and then add the according scenario everywhere
where we have extra_tests_in_textmode already scheduled, e.g.
go to https://openqa.opensuse.org/admin/groups
open at best all the groups one by one, take a look for
"extra_tests_in_textmode" and make sure we have the extra_tests_dracut
testsuite scheduled for all combinations where previously
extra_tests_in_textmode is in. For "mau-extratests" I suggest to decide
in coordination with the daily test result reviewers of openSUSE/SLE
tests. Sorry, I do not know who that would be at least not for openSUSE.
I have seen mainly astieger (Andreas Stieger) do the reviewer and he
promised me that QAM would pick up the test review according to a proper
schedule but I am not aware if that happened – bluntly, seems like not :)

I am also following the tickets so feel free to ask the same questions
in tickets instead of personal emails which I struggle with to answer in
time. This has the added benefit of more information documented which is
also easily accessible for others then. Feel free to copy the
information from this email in your ticket as well.

@okurz: ok, thanks!
Before editing stuff in ooo I guess we need to review and merge the aforementioned PR so we don't break ooo, right?
To recap:

• in ooo I can edit myself info for textmode in order to add dracut
• in ooo I need to find the responsible for mau before being able to add dracut
• in osd I need to find the responsible for textmode and mau before being able to add dracut

The PR is ready to be merged: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6745

After that we can proceed with editing the openqa instances.

great news! Please make sure we have both o3 and osd covered. Of course it would be fine to delegate the specific test suite and scenario adjustements to other people.

@pstivanin any progress?

In
https://openqa.opensuse.org/tests/latest?machine=64bit&version=Tumbleweed&flavor=DVD&arch=x86_64&distri=opensuse&test=extra_tests_in_textmode#dependencies

I can see that there is no scenario "dracut" yet so can you enable the according scenarios?

@okurz I don't have permissions to edit job definition. We have to wait for @pcervinka I guess

I see, I have no problem to make you "admin" on both openQA instances to do the change. However I would ask you to join during normal working ours on the IRC channels #opensuse-factory and internal #testing

@okurz done!

@okurz any news about giving me permissions to edit o3?

ping me on #opensuse-factory IRC and I will add you :)

For now I have enabled it only for TW 64bit and Leap 64bit.
If everything goes well I will proceed to add the test suite to all other job groups.