action #19922
closed[sle][functional][opensuse]Harmonize extratests/yast2_ui tests for openSUSE and SLE
0%
Description
within lib/main_common.pm there are some test modules which are only called for openSUSE, some only for SLE without a good comment and explanation why the distributions need to differ. This should be improved. At best we want the same modules be called in all cases.
acceptance criteria¶
- AC1: Exceptions have an explanation why they can not be run in certain scenarios
- AC2: We have the same scenario enabled for all SLE, openSUSE Tumbleweed and openSUSE Leap
tasks¶
- Check for "TODO" in lib/main_common.pm (0.3 - 2h)
- Crosscheck other "if sle/opensuse" branches in this file (1-6h)
- Re-enable yast2_ui tests on openSUSE Leap (1-6h)
Updated by zluo almost 7 years ago
YaST2 UI tests are sometimes complicated because there are 3 different UI: SLE 12 gtk, openSUSE TW: QT, openSUSE Leap gtk
It means we need to provide 3 different needles for each test.
I will take time to check this and make a list at first. Before make changes for extratests (yast2 gui tests) in lib/main_common.pm, we need to talk about this in detail.
Updated by zluo almost 7 years ago
yast2_lan_hostname and yast2_nis failed for openSUSE Leap
Updated by zluo almost 7 years ago
For console/yast2_cmdline, console/yast2_dns_server, console/yast2_nfs_client I did test run against SLE:
http://e13.suse.de/tests/3150#
dns_server has an issue: https://bugzilla.opensuse.org/show_bug.cgi?id=1036630
cmdcline, nfs_client have status of failure and I need to check why.
It doesn't make sense atm to put these 3 test modules into SLES extratest before the failure got resolved.
Updated by zluo almost 7 years ago
yast2_nfs_client should be added into SLE extratest because it has worked after I added missing needles for it. And it has softfail however.
Updated by zluo almost 7 years ago
Reference test from SLE:
https://openqa.suse.de/tests/1019970
sle-12-SP3-Server-DVD-x86_64-Build0440-extra_tests_on_gnome@64bit
Try to find out why following 2 modules are excluded for openSUSE
x11/yast2_lan_restart
x11/user_defined_snapshot
Updated by zluo almost 7 years ago
http://e13.suse.de/tests/3169#step/yast2_lan_restart/16
shows for now that needle is missing (if it has existed, it could be removed already for a long time)
So create missing needles should be the first bug fix.
Updated by zluo almost 7 years ago
yast2_lan_restart:
For openSUSE TW I created all missing needles, finally!
Updated by zluo almost 7 years ago
Updated by okurz almost 7 years ago
- Status changed from New to In Progress
So the above mentioned PRs were merged and executed on openqa.opensuse.org but that could not work because yast2_lan_restart hardcodes SUSE internal DNS servers, e.g. https://openqa.opensuse.org/tests/442033#step/yast2_lan_restart/245 showing 10.160.2.88 . Hence I had to revert the test plan with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3247 for now.
@lnussel, @dimstar, @coolo What do you think, which DNS server should we use for testing? E.g. the one the worker has configured themselves, external google DNS 8.8.8.8?
The test hardcodes 10.160.2.88 which is SUSE internal and this one is not reachable from within the o3 infrastructure. Dimstar and lnussel prefer relying solely on internal ressources. But also the test should rather not change anything in the end. So probably better to
1) not change this DNS setting at all (preferred if this works)
2) read out previous entry from /etc/resolv.conf from within the test and insert that (preferred if 1 does not work)
3) use an openSUSE DNS server in case we test on distri opensuse
--
@okurz
PR submitted: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3264
Updated by zluo almost 7 years ago
working on user_defined_snapshot.pm
At moment I have problem with rebooting of default system from user_defined_snapshot.
http://e13.suse.de/tests/3308#
Updated by zluo almost 7 years ago
PR for yast2_lan_hostname got merged this morning:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3260#event-1156450437
Updated by okurz almost 7 years ago
And yast2_hostname fails now for openSUSE Leap 42.3: https://openqa.opensuse.org/tests/445396#step/yast2_lan_hostname/9
Please take extra care about failing openSUSE Leap 42.3 tests
--
@okurz checking now...
No production issue, checked against openSUSE-42.3-x86_64-Build0315, however "Activating network services" takes time.
clone the test from openqa.opensuse.org and it works without any issue. But there is failure for checking server status later.
The issue ís about: grep DHCLIENT_SET_HOSTNAME /etc/sysconfig/network/dhcp|grep yes
DHCLIENT_SET_HOSTNAME="yes" is not showing up at moment. So it means that updating network configuration is somehow not working.
This is not happening for manual testing.
@okurz
the latest job looks good, even checking configuration result is correct:
Updated by okurz almost 7 years ago
- Blocked by action #20202: test fails in yast2_apparmor added
Updated by okurz almost 7 years ago
- Blocks action #19022: [opensuse][functional] test fails in yast2_control_center added
Updated by okurz almost 7 years ago
- Blocks action #20204: [opensuse][functional] test fails in yast2_ftp added
Updated by okurz almost 7 years ago
- Blocks deleted (action #19022: [opensuse][functional] test fails in yast2_control_center)
Updated by okurz almost 7 years ago
- Blocks deleted (action #20204: [opensuse][functional] test fails in yast2_ftp)
Updated by okurz almost 7 years ago
- Blocked by action #20204: [opensuse][functional] test fails in yast2_ftp added
Updated by okurz almost 7 years ago
- Blocks action #19022: [opensuse][functional] test fails in yast2_control_center added
Updated by okurz almost 7 years ago
- Blocked by action #20332: [opensuse][functional] test fails in yast2_datetime added
Updated by okurz almost 7 years ago
- Blocks deleted (action #19022: [opensuse][functional] test fails in yast2_control_center)
Updated by okurz almost 7 years ago
- Blocked by action #19022: [opensuse][functional] test fails in yast2_control_center added
Updated by okurz almost 7 years ago
- Blocked by action #20200: [opensuse]yast2_samba random test failure added
Updated by okurz almost 7 years ago
- Blocked by action #19902: [opensuse]yast2 gui test yast2 bootloader failed recently added
Updated by okurz almost 7 years ago
- Description updated (diff)
- Priority changed from Normal to High
lnussel disabled yast2_ui for openSUSE Leap for now. I extended the ACs and tasks in the description.
Updated by zluo almost 7 years ago
PR for yast2_apparmor submitted:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3306
Updated by riafarov almost 7 years ago
- Blocked by action #20564: [opensuse][functional] Make yast2_proxy stable added
Updated by riafarov almost 7 years ago
- Blocked by action #20640: [opensuse][functional] Improve yast2_http tests added
Updated by riafarov almost 7 years ago
- Blocked by action #20668: [opensuse][functional] Make yast2_ui stable added
Updated by okurz over 6 years ago
Talked to zluo, tests on TW look quite stable (softfail), see https://openqa.opensuse.org/tests?match=yast2_ui
To crosscheck statistics I triggered 100 job runs:
for i in {1..100}; do openqa_clone_job_o3 --skip-download --skip-chained-deps 459028 TEST=yast_ui_stability_check_$i _GROUP="Development Tumbleweed" ; done
Updated by zluo over 6 years ago
- Related to action #20970: yast2_dns_server failed added
Updated by okurz over 6 years ago
- Blocks action #20968: [functional][yast][y] extend yast2_users added
Updated by okurz over 6 years ago
- Blocks action #20966: Extend yast2_software_management added
Updated by zluo over 6 years ago
- Related to action #23616: [sle][functional][opensuse]yast2_xinetd failed recently in most cases added
Updated by zluo over 6 years ago
- Related to action #23730: yast2_ntclient failed recently in yast2_ui added
Updated by sebchlad over 6 years ago
Those are the acceptance criteria:
AC1: Exceptions have an explanation why they can not be run in certain scenarios
AC2: We have the same scenario enabled for all SLE, openSUSE Tumbleweed and openSUSE Leap
@zluo: what exactly have to be done in order to meet those criteria?
Do we have all exceptions explained?
Do we have the same scenario enabled for all SLE, openSUSE Tumbleweed and openSUSE Leap?
Updated by zluo over 6 years ago
- what exactly have to be done in order to meet those criteria?
fix bugs in existing tests or product issue incl. issues on openQA is P1.
This has been done almost, we can see status of subtasks of this ticket. These subtasks have been added later to this ticket. This ticket got changed a lot in last months.
Do we have all exceptions explained?
Anton has disabled some test modules because they didn't work for some reasons. I've checked the related main_common.pm. I've commented for some tests, but not all of it. Because of changed situation and fixing bugs are P1, so harmonize extratests/yast2_ui is not P1 for now.Do we have the same scenario enabled for all SLE, openSUSE Tumbleweed and openSUSE Leap?
Yes, but only couple of tests are not enabled due to various reasons: product difference, bugs existing only on specific products.
yast2_ui tests have been removed from openQA (Leap only). The condition has been changed as well, however we can still work on it's development and we do hope to resolve all bugs related to test code. After this we can ask for taking backing to daily testing on openQA if they are working stable.
I hope this is clear to you :)
Updated by zluo over 6 years ago
- Blocks deleted (action #20968: [functional][yast][y] extend yast2_users)
Updated by zluo over 6 years ago
- Blocks deleted (action #20966: Extend yast2_software_management )
Updated by zluo over 6 years ago
- Blocked by deleted (action #20200: [opensuse]yast2_samba random test failure)
Updated by zluo over 6 years ago
- Related to action #23788: [opensuse]yast2_proxy failed recently added
Updated by zluo over 6 years ago
- Related to deleted (action #23616: [sle][functional][opensuse]yast2_xinetd failed recently in most cases)
Updated by zluo over 6 years ago
- Related to deleted (action #23730: yast2_ntclient failed recently in yast2_ui)
Updated by zluo over 6 years ago
- Related to deleted (action #23788: [opensuse]yast2_proxy failed recently)
Updated by zluo over 6 years ago
- Priority changed from High to Normal
set priority to normal since only we have at moment open issues related to product, not to test code.
Updated by okurz over 6 years ago
- Priority changed from Normal to High
that's a wrong conclusion. The harmonisation of the tests should not be hindered by any product bug. Actually it is the other way around. If we would have the same tests running on both openSUSE and SLE the work on product bugs would be even simpler because we can compare results.
Please state what steps are necessary to be conducted and what is missing for this ticket to be done according to the acceptance criteria mentioned in the original description.
Btw, I checked your experiment with "problem_detection()" in yast2_xinetd and it works for me with the following code:
my ($self) = @_;
…
assert_screen([qw(yast2_xinetd_startup yast2_xinetd_core-dumped)], 90);
…
$self->problem_detection();
Updated by okurz over 6 years ago
- Related to coordination #23820: [sle][functional][sle15][epic]update yast2_ui for sle15 added
Updated by okurz over 6 years ago
- Related to action #23788: [opensuse]yast2_proxy failed recently added
Updated by okurz over 6 years ago
- Subject changed from Harmonize extratests/yast2_ui tests for openSUSE and SLE to [sle][functional][opensuse]Harmonize extratests/yast2_ui tests for openSUSE and SLE
Updated by okurz over 6 years ago
- Subject changed from [sle][functional][opensuse]Harmonize extratests/yast2_ui tests for openSUSE and SLE to [sle][functional][opensuse][epic]Harmonize extratests/yast2_ui tests for openSUSE and SLE
Updated by zluo over 6 years ago
- Related to action #25640: [sle][functional][sle15] - add workround for susefirewall2 and create new needles for yast2_network_settings added
Updated by okurz over 6 years ago
- Subject changed from [sle][functional][opensuse][epic]Harmonize extratests/yast2_ui tests for openSUSE and SLE to [sle][functional][opensuse]Harmonize extratests/yast2_ui tests for openSUSE and SLE
- Description updated (diff)
Updated by okurz over 6 years ago
- Status changed from In Progress to Resolved
added "yast2_ncurses" and "yast2_gui" to the test plan for openSUSE Leap 15.0, maxlin and lnussel agreed.
There is still a TODO about #20200 which was a rejected ticket but because yast2_samba still fails consistently on openSUSE Tumbleweed I reopened the ticket
There is only one exception of a test module which differs between openSUSE and SLE which is "yast2_nis" relying on SUSE's internal NIS server and this one is properly commented
We are done here