QA » openQA Project » openQA Tests

YaST2 UI tests are sometimes complicated because there are 3 different UI: SLE 12 gtk, openSUSE TW: QT, openSUSE Leap gtk
It means we need to provide 3 different needles for each test.

I will take time to check this and make a list at first. Before make changes for extratests (yast2 gui tests) in lib/main_common.pm, we need to talk about this in detail.

yast2_lan_hostname and yast2_nis failed for openSUSE Leap

For console/yast2_cmdline, console/yast2_dns_server, console/yast2_nfs_client I did test run against SLE:
http://e13.suse.de/tests/3150#

dns_server has an issue: https://bugzilla.opensuse.org/show_bug.cgi?id=1036630
cmdcline, nfs_client have status of failure and I need to check why.

It doesn't make sense atm to put these 3 test modules into SLES extratest before the failure got resolved.

yast2_nfs_client should be added into SLE extratest because it has worked after I added missing needles for it. And it has softfail however.

Reference test from SLE:
https://openqa.suse.de/tests/1019970
sle-12-SP3-Server-DVD-x86_64-Build0440-extra_tests_on_gnome@64bit

Try to find out why following 2 modules are excluded for openSUSE
x11/yast2_lan_restart
x11/user_defined_snapshot

http://e13.suse.de/tests/3169#step/yast2_lan_restart/16

shows for now that needle is missing (if it has existed, it could be removed already for a long time)
So create missing needles should be the first bug fix.

yast2_lan_restart:
For openSUSE TW I created all missing needles, finally!

http://e13.suse.de/tests/3227#

PR for yast2_lan_restart:

https://github.com/os-autoinst/os-autoinst-needles-opensuse/pull/223
and
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3236

working on user_defined_snapshot.pm

At moment I have problem with rebooting of default system from user_defined_snapshot.
http://e13.suse.de/tests/3308#

PR for yast2_lan_hostname got merged this morning:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3260#event-1156450437

And yast2_hostname fails now for openSUSE Leap 42.3: https://openqa.opensuse.org/tests/445396#step/yast2_lan_hostname/9

Please take extra care about failing openSUSE Leap 42.3 tests

--
@okurz checking now...
No production issue, checked against openSUSE-42.3-x86_64-Build0315, however "Activating network services" takes time.
clone the test from openqa.opensuse.org and it works without any issue. But there is failure for checking server status later.
The issue ís about: grep DHCLIENT_SET_HOSTNAME /etc/sysconfig/network/dhcp|grep yes
DHCLIENT_SET_HOSTNAME="yes" is not showing up at moment. So it means that updating network configuration is somehow not working.

This is not happening for manual testing.

@okurz
the latest job looks good, even checking configuration result is correct:

https://openqa.opensuse.org/tests/latest?arch=x86_64&test=yast2_ui&version=42.3&machine=64bit&flavor=DVD&distri=opensuse#step/yast2_lan_hostname/29

PR for yast2_apparmor submitted:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3306

Talked to zluo, tests on TW look quite stable (softfail), see https://openqa.opensuse.org/tests?match=yast2_ui

To crosscheck statistics I triggered 100 job runs:

for i in {1..100}; do openqa_clone_job_o3 --skip-download --skip-chained-deps 459028 TEST=yast_ui_stability_check_$i _GROUP="Development Tumbleweed" ; done

-> https://openqa.opensuse.org/tests/overview?build=20170730&distri=opensuse&groupid=38&version=Tumbleweed

Those are the acceptance criteria:
AC1: Exceptions have an explanation why they can not be run in certain scenarios
AC2: We have the same scenario enabled for all SLE, openSUSE Tumbleweed and openSUSE Leap

@zluo: what exactly have to be done in order to meet those criteria?
Do we have all exceptions explained?
Do we have the same scenario enabled for all SLE, openSUSE Tumbleweed and openSUSE Leap?

@sebchlad

• what exactly have to be done in order to meet those criteria?

fix bugs in existing tests or product issue incl. issues on openQA is P1.
This has been done almost, we can see status of subtasks of this ticket. These subtasks have been added later to this ticket. This ticket got changed a lot in last months.

• Do we have all exceptions explained?
  Anton has disabled some test modules because they didn't work for some reasons. I've checked the related main_common.pm. I've commented for some tests, but not all of it. Because of changed situation and fixing bugs are P1, so harmonize extratests/yast2_ui is not P1 for now.

• Do we have the same scenario enabled for all SLE, openSUSE Tumbleweed and openSUSE Leap?

Yes, but only couple of tests are not enabled due to various reasons: product difference, bugs existing only on specific products.
yast2_ui tests have been removed from openQA (Leap only). The condition has been changed as well, however we can still work on it's development and we do hope to resolve all bugs related to test code. After this we can ask for taking backing to daily testing on openQA if they are working stable.

I hope this is clear to you :)

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3607#pullrequestreview-64242438

PR got merged.