QA » openQA Project

@geor since when is this happening? Please include a "last good" as well as a link to latest. You should be able to find those links if you click the "report issue" buttons in openQA from failing test cases

The issue seems to appear with the introduction of SLE 15 SP4. bsc#1191112 is first bug opened on the issue and according to the bug entry, the glitch has been happening since build 31.2. However, due to the sporadic nature of the glitch, it could be that it was introduced before that.
The oldest archived 15-SP4 yast2_gui test run available is for build 36.1 where the issue is also present

On SLE 15 SP5 it has been happening since day 1.

Let's mob this one tomorrow

Seems like the work-around has changed in the meantime: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/YaST/workarounds.pm

cdywan wrote:

Seems like the work-around has changed in the meantime: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/YaST/workarounds.pm

yes, it is renamed with this ticket and and not shown as soft-failure because it is not a bug, the rest is the same.

JERiveraMoya wrote:

cdywan wrote:

Seems like the work-around has changed in the meantime: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/YaST/workarounds.pm

yes, it is renamed with this ticket and and not shown as soft-failure because it is not a bug, the rest is the same.

I'm not sure I understand what you mean by that. Do you mean this shouldn't show as a soft failure?

We tried to debug the issue collaboratively just now but we couldn't find the same issue anymore. Would you have a "latest" link that shows the concrete case? Otherwise we have no way of reproducing the issue right now.

Hi, thanks for taking a look, I meant that we don't represent it with a soft-failure anymore (orange job status) as it was investigated by developers and it is not a bug, it is only reproducible in our infrastructure affecting multiple tests. Now we represent it as a record_info like here https://openqa.suse.de/tests/10692804#step/yast2_security/15

This is still an occurrence:
https://openqa.suse.de/tests/10720029#step/iscsi_server/52
It should show: https://openqa.suse.de/tests/10698100#step/iscsi_server/51

Another one is https://openqa.suse.de/tests/10724854#step/yast2_security/25 vs https://openqa.suse.de/tests/10692804#step/yast2_security/18

JERiveraMoya wrote:

This is still an occurrence:
https://openqa.suse.de/tests/10720029#step/iscsi_server/52
It should show: https://openqa.suse.de/tests/10698100#step/iscsi_server/51

This looks to me like it fails in my·$module_name·=·y2_module_guitest::launch_yast2_module_x11('iscsi-lio-server',·target_match·=>·'iscsi-lio-server'); without ever applying the work-around which is only used in the config_2way_authentication case.

Another one is https://openqa.suse.de/tests/10724854#step/yast2_security/25 vs https://openqa.suse.de/tests/10692804#step/yast2_security/18

In tests/yast2_gui/yast2_security.pm it seems the work-around is missing/needed for assert_screen "yast2_security-login-attempts".

So I guess this is up to test maintainers since it doesn't seem to be an "openQA issue" as such, but given that I've just debugged this and the files already open in helix I went ahead and prepared a PR. The other case isn't as straight-forward, but maybe something like this could work here.

Not sure who is best to take over from here, but feel free to grab and adjust my proposed fixes as needed.

cdywan wrote:

JERiveraMoya wrote:

This is still an occurrence:
https://openqa.suse.de/tests/10720029#step/iscsi_server/52
It should show: https://openqa.suse.de/tests/10698100#step/iscsi_server/51

This looks to me like it fails in my·$module_name·=·y2_module_guitest::launch_yast2_module_x11('iscsi-lio-server',·target_match·=>·'iscsi-lio-server'); without ever applying the work-around which is only used in the config_2way_authentication case.

yes, the problem appears randomly in different places, here it looks like we haven't applied and 50% of time we hit the issue and the content is not loaded, so it is a good example.

Another one is https://openqa.suse.de/tests/10724854#step/yast2_security/25 vs https://openqa.suse.de/tests/10692804#step/yast2_security/18

In tests/yast2_gui/yast2_security.pm it seems the work-around is missing/needed for assert_screen "yast2_security-login-attempts".

So I guess this is up to test maintainers since it doesn't seem to be an "openQA issue" as such, but given that I've just debugged this and the files already open in helix I went ahead and prepared a PR. The other case isn't as straight-forward, but maybe something like this could work here.

Not sure who is best to take over from here, but feel free to grab and adjust my proposed fixes as needed.

It is not a needle issue, when you click that item Login Setting only the title of the screen is refreshed "Login Settings", but the content is the remaining from Security Overview.
We verified that manually a time ago, of course with openQA, outside openQA not even the developer could do reproduce it.
Thanks for trying out, too many YaST modules are hacked with this workaround stopping having proper automation and propagating to testing in maintenance products.
Let me know if you somehow cannot catch it just stopping the automation with the openQA debugger, this Login setting could be a good example I believe as it is not MM.

JERiveraMoya wrote:

It is not a needle issue, when you click that item Login Setting only the title of the screen is refreshed "Login Settings", but the content is the remaining from Security Overview.

Not sure what you mean by "needle issue" so let me be more explicit on what I so far understand:

• The issue here is YaST windows not being redrawn/ refreshed/ updated. Hence the needle won't match.
• What the work-around does is it presses keys when the needle didn't match and tries once more check the screen.
• Said work-around is not being used in the cases I mentioned.

yes, you are getting all right, YaST windows not being redrawn/ refreshed/ updated only in openQA infrastructure, that is the issue itself.

So to see what device may be more reliable I'm currently re-running the same jobs in different configurations, namely virtio-vga, qxl-vga, virtio-gpu-pci and bochs-display as per docs, see:

openqa-clone-job --within-instance http://openqa.suse.de/tests/10720029 QEMU_VIDEO_DEVICE=virtio-vga
Cloning parents of sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build81.1-autoyast_create_hdd_gnome@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit
Cloning parents of sle-15-SP5-Online-x86_64-Build81.1-iscsi_client_normal_auth_backstore_hdd@64bit
3 jobs have been created:
 - sle-15-SP5-Online-x86_64-Build81.1-autoyast_create_hdd_gnome@64bit -> http://openqa.suse.de/tests/10985379
 - sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit -> http://openqa.suse.de/tests/10985378
 - sle-15-SP5-Online-x86_64-Build81.1-iscsi_client_normal_auth_backstore_hdd@64bit -> http://openqa.suse.de/tests/10985377
qa.suse.de/tests/10720029 QEMU_VIDEO_DEVICE=qxl-vga   job --within-instance http://openq
Cloning parents of sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build81.1-autoyast_create_hdd_gnome@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit
Cloning parents of sle-15-SP5-Online-x86_64-Build81.1-iscsi_client_normal_auth_backstore_hdd@64bit
3 jobs have been created:
 - sle-15-SP5-Online-x86_64-Build81.1-autoyast_create_hdd_gnome@64bit -> http://openqa.suse.de/tests/10985396
 - sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit -> http://openqa.suse.de/tests/10985394
 - sle-15-SP5-Online-x86_64-Build81.1-iscsi_client_normal_auth_backstore_hdd@64bit -> http://openqa.suse.de/tests/10985395
qa.suse.de/tests/10720029 QEMU_VIDEO_DEVICE=virtio-gpu-pci--within-instance http://openq
Cloning parents of sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build81.1-autoyast_create_hdd_gnome@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit
Cloning parents of sle-15-SP5-Online-x86_64-Build81.1-iscsi_client_normal_auth_backstore_hdd@64bit
3 jobs have been created:
 - sle-15-SP5-Online-x86_64-Build81.1-autoyast_create_hdd_gnome@64bit -> http://openqa.suse.de/tests/10985401
 - sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit -> http://openqa.suse.de/tests/10985400
 - sle-15-SP5-Online-x86_64-Build81.1-iscsi_client_normal_auth_backstore_hdd@64bit -> http://openqa.suse.de/tests/10985399
qa.suse.de/tests/10720029 QEMU_VIDEO_DEVICE=bochs-display --within-instance http://openq
Cloning parents of sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build81.1-autoyast_create_hdd_gnome@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit
Cloning parents of sle-15-SP5-Online-x86_64-Build81.1-iscsi_client_normal_auth_backstore_hdd@64bit
3 jobs have been created:
 - sle-15-SP5-Online-x86_64-Build81.1-autoyast_create_hdd_gnome@64bit -> http://openqa.suse.de/tests/10985405
 - sle-15-SP5-Online-x86_64-Build81.1-iscsi_server_normal_auth_backstore_hdd@64bit -> http://openqa.suse.de/tests/10985407
 - sle-15-SP5-Online-x86_64-Build81.1-iscsi_client_normal_auth_backstore_hdd@64bit -> http://openqa.suse.de/tests/10985406

cdywan wrote:

So to see what device may be more reliable I'm currently re-running the same jobs in different configurations, namely virtio-vga, qxl-vga, virtio-gpu-pci and bochs-display as per docs, see:

Apparently all devices eventually reproduce the issue, so unfortunately that was a bit of a red herring.

(@geor is not anymore in the project)
In the first comment of the bug that we opened for sle 15 sp5 you can see the reference to the bug we open for sle 15 sp4:
https://bugzilla.suse.com/show_bug.cgi?id=1204176#c0
https://bugzilla.suse.com/show_bug.cgi?id=1191112 (ignored the verified fixed, that bug was too confusing at the beginning)
According to first comment from ticket above, the problem appeared in SLE15-SP4 (e.g. 39.1)
The problem is in qemu afik, see other archs than x86_64 (I didn't find it for s390x):
ppc64le: https://openqa.suse.de/tests/10941690#step/yast2_lan_restart_bridge/42
aarch64: https://openqa.suse.de/tests/10941688#step/yast2_lan_restart_bridge/34
Product sle 15 sp4 and sle 15 sp5 (not seen in O3 or other places that I'm aware...).

JERiveraMoya wrote:

Product sle 15 sp4 and sle 15 sp5 (not seen in O3 or other places that I'm aware...).

so in Tumbleweed and and also Leap the problem does not reproduce? That would be interesting. Could you share openQA jobs that can be compared for SLE15SP4/5 and Tumbleweed+Leap?

okurz wrote:

JERiveraMoya wrote:

Product sle 15 sp4 and sle 15 sp5 (not seen in O3 or other places that I'm aware...).

so in Tumbleweed and and also Leap the problem does not reproduce? That would be interesting. Could you share openQA jobs that can be compared for SLE15SP4/5 and Tumbleweed+Leap?

there is the same test suite in TW, but not exactly the same is tested because not all the yast modules are available among products/archs
https://openqa.opensuse.org/tests/3247358#step/yast2_control_center/71
same for Leap: https://openqa.opensuse.org/tests/3239149#

Then I suggest you

JERiveraMoya wrote:

okurz wrote:

JERiveraMoya wrote:

Product sle 15 sp4 and sle 15 sp5 (not seen in O3 or other places that I'm aware...).

so in Tumbleweed and and also Leap the problem does not reproduce? That would be interesting. Could you share openQA jobs that can be compared for SLE15SP4/5 and Tumbleweed+Leap?

there is the same test suite in TW, but not exactly the same is tested because not all the yast modules are available among products/archs
https://openqa.opensuse.org/tests/3247358#step/yast2_control_center/71
same for Leap: https://openqa.opensuse.org/tests/3239149#

Ok, good. Then I suggest you gather in a https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation if the problem happens with a same error rate if at all for the different OS/version combinations.

okurz wrote:

Then I suggest you

JERiveraMoya wrote:

okurz wrote:

JERiveraMoya wrote:

Product sle 15 sp4 and sle 15 sp5 (not seen in O3 or other places that I'm aware...).

so in Tumbleweed and and also Leap the problem does not reproduce? That would be interesting. Could you share openQA jobs that can be compared for SLE15SP4/5 and Tumbleweed+Leap?

there is the same test suite in TW, but not exactly the same is tested because not all the yast modules are available among products/archs
https://openqa.opensuse.org/tests/3247358#step/yast2_control_center/71
same for Leap: https://openqa.opensuse.org/tests/3239149#

Ok, good. Then I suggest you gather in a https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation if the problem happens with a same error rate if at all for the different OS/version combinations.

Statistically, the failure occurs

• for SLE15-SP4 by rate 97% : https://openqa.suse.de/tests/overview?distri=sle&build=poo124652_investigation_SP4&version=15-SP4
• for SLE15-SP5 by rate 99% : https://openqa.suse.de/tests/overview?build=poo124652_investigation&version=15-SP5&distri=sle

syrianidou_sofia wrote:

Statistically, the failure occurs

• for SLE15-SP4 by rate 97% : https://openqa.suse.de/tests/overview?distri=sle&build=poo124652_investigation_SP4&version=15-SP4
• for SLE15-SP5 by rate 99% : https://openqa.suse.de/tests/overview?build=poo124652_investigation&version=15-SP5&distri=sle

Nice! Can you do the same for Tumbleweed and SLE15-SP5? The "last good" from the initial ticket description is https://openqa.suse.de/tests/5900202#step/yast2_control_center/94 on SLE15-SP3 with state of maintenance update from 2 years ago. Can you try to clone or replicate that so that we can distinguish between test environment and SUT OS state?

For SLE 15 SP3 you have already 100 runs from Maintenance update job group for yam:
https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=mru-iscsi_server_normal_auth_backstore_fileio&version=15-SP3#next_previous
where the workaround is not triggered and there are no failures.

For tumbleweed that exact scenario iscsi is not tested, @ssyrianidou, could you somehow hack the module in yast2_gui a bit to try to run something similar as Oliver suggested?. It this glitch would have been seen in openSUSE we would have had plenty of tickets in our backlog so I really doubt it happens, we probably could assume there is not issue there.

JERiveraMoya wrote:

For SLE 15 SP3 you have already 100 runs from Maintenance update job group for yam:
https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=mru-iscsi_server_normal_auth_backstore_fileio&version=15-SP3#next_previous
where the workaround is not triggered and there are no failures.

For tumbleweed that exact scenario iscsi is not tested, @ssyrianidou, could you somehow hack the module in yast2_gui a bit to try to run something similar as Oliver suggested?. It this glitch would have been seen in openSUSE we would have had plenty of tickets in our backlog so I really doubt it happens, we probably could assume there is not issue there.

Indeed, I have checked already the previous runs on O3 for yast2_gui. I didn't see the glitch in any of the modules. On osd the same modules expedite the glitch very frequently so I would assume that it's not reproducible for opensuse.

Added task for Yam squad to try out suggested comparable setup.

okurz wrote:

I updated the description of the text following https://progress.opensuse.org/projects/openqav3/wiki#Further-decision-steps-working-on-test-issues after I reviewed all comments with all measurements.

There is one experiment which is still open which I suggest to run

• E2.1-2 From #124652#note-24 Run openQA test on SLE15-SP3 with state of maintenance update from 2 years ago ("last good" https://openqa.suse.de/tests/5900202#step/yast2_control_center/94 from #Motivation)

• @syrianidou_sofia can you do that?

For the yast2_control_center module 100 tests were ran for SLES 15 SP3 GMC installation without updates and 100 for updated SP2. The glitch does not seem to be present in any of the tests:
SP2: https://openqa.suse.de/tests/overview?build=glitch_investigation_15_SP2&version=15-SP2&distri=sle
SP3: https://openqa.suse.de/tests/overview?version=15-SP3&build=glitch_investigation_15_SP3&distri=sle

okurz wrote:

Nice. So as expected the problem was introduced with a maintenance update. I updated the ticket description accordingly. From your measurements further bisection could be done to identify the exact problematic patch.

@geor @syrianidou_sofia As we have clearly not found an easy way to work around this in openQA and because it's really product specific I suggest you update https://bugzilla.suse.com/show_bug.cgi?id=1204176 with the information that a SLE maintenance update clearly introduced the issue but also that not all versions of openSUSE are affected. This might be interesting for the YaST team. If they are not interested then you need to look into workarounds from test perspective.

OK, the bug is updated here: https://bugzilla.suse.com/show_bug.cgi?id=1204176#c30

that's great. Thank you. Now, I don't think there is anything more that we can from the side of openQA or os-autoinst. Do you still need this ticket open to handle any work regarding workarounds in tests?