action #101475
closed[sle][migration][SLE15SP4][regression] test fails in check_upgraded_service - rpcinfo | grep nfs' failed
100%
Description
Observation¶
openQA test in scenario sle-15-SP4-Regression-on-Migration-from-SLE15-SPx-s390x-offline_sles15sp1_ltss_pscc_basesys-srv-desk-dev-contm-lgm-py2-tsm-wsm_all_full@s390x-kvm-sle12 fails in
check_upgraded_service
Test suite description¶
Reproducible¶
Fails since (at least) Build 52.1
Expected result¶
Last good: 50.1 (or more recent)
Further details¶
Always latest result in this scenario: latest
Updated by hjluo over 2 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 10
hit it again in 55.1 https://openqa.nue.suse.com/tests/7555430#step/check_upgraded_service/239
Updated by hjluo over 2 years ago
./hj-tools/hj-clone.sh -j 7555430 -s "_GROUP=0"
https://openqa.nue.suse.com/t7580828 FAILED https://openqa.nue.suse.com/tests/7580828#step/check_upgraded_service/232
https://openqa.nue.suse.com/t7580829 PASSED
so this is a sporadic issue and maybe a timing issue. we may wait a while before check the nfs
Updated by hjluo over 2 years ago
passed in 55.1 https://openqa.nue.suse.com/tests/7580900
Updated by hjluo over 2 years ago
Updated by hjluo over 2 years ago
- Status changed from In Progress to Resolved
- % Done changed from 10 to 100
so close it now since we've passed in 3 consecutive builds.
Updated by leli over 2 years ago
- Status changed from Resolved to Workable
Still found the same issue, https://openqa.nue.suse.com/tests/7620268#step/check_upgraded_service/234
Updated by hjluo over 2 years ago
PASSED on 61.1 https://openqa.nue.suse.com/tests/7620269
Updated by hjluo over 2 years ago
new run passed https://openqa.nue.suse.com/tests/7642994
Updated by hjluo over 2 years ago
- Status changed from Workable to In Progress
- % Done changed from 100 to 70
Updated by hjluo over 2 years ago
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7620269 -s "_GROUP=0"
https://openqa.nue.suse.com/t7693984
https://openqa.nue.suse.com/t7693985
ALL PASSED
Updated by hjluo over 2 years ago
you can see we've tried 2 times before the nfs started and can be shown in rpcinfo.
https://openqa.nue.suse.com/tests/7693984/logfile?filename=autoinst-log.txt
[2021-11-17T10:25:00.011650+01:00] [debug] tests/installation/install_service.pm:21 called service_check::install_services -> lib/service_check.pm:252 called (eval) -> lib/service_check.pm:263 called services::rpcbind::full_rpcbind_check -> lib/services/rpcbind.pm:91 called services::rpcbind::check_function -> lib/services/rpcbind.pm:66 called testapi::script_run
[2021-11-17T10:25:00.011814+01:00] [debug] <<< testapi::script_run(cmd="rpcinfo | grep nfs", output="", quiet=undef, timeout=undef)
[2021-11-17T10:25:00.011992+01:00] [debug] tests/installation/install_service.pm:21 called service_check::install_services -> lib/service_check.pm:252 called (eval) -> lib/service_check.pm:263 called services::rpcbind::full_rpcbind_check -> lib/services/rpcbind.pm:91 called services::rpcbind::check_function -> lib/services/rpcbind.pm:66 called testapi::script_run
[2021-11-17T10:25:00.012160+01:00] [debug] <<< testapi::type_string(string="rpcinfo | grep nfs", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
Updated by hjluo over 2 years ago
Updated by hjluo over 2 years ago
after discussion with richard and lemon, we'd check the nfs server before check the rpcinfo|grep nfs in the check_funciton
changed code and rerun:
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7620269 -s "_GROUP=0"
https://openqa.nue.suse.com/t7714308
https://openqa.nue.suse.com/t7714309
https://openqa.nue.suse.com/t7714310
you can see it's still 2 times before we get nfs info from rpcinfo.
https://openqa.nue.suse.com/tests/7714309/logfile?filename=autoinst-log.txt
[2021-11-22T04:54:11.947954+01:00] [debug] <<< testapi::script_run(cmd="rpcinfo | grep nfs", output="", timeout=undef, quiet=undef)
[2021-11-22T04:54:11.948122+01:00] [debug] tests/installation/install_service.pm:21 called service_check::install_services -> lib/service_check.pm:252 called (eval) -> lib/service_check.pm:263 called services::rpcbind::full_rpcbind_check -> lib/services/rpcbind.pm:93 called services::rpcbind::check_function -> lib/services/rpcbind.pm:68 called testapi::script_run
[2021-11-22T04:54:11.948290+01:00] [debug] <<< testapi::type_string(string="rpcinfo | grep nfs", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
Updated by hjluo over 2 years ago
run for test log to file a bug after discussion with lemon and richard
https://openqa.nue.suse.com/t7714591
https://openqa.nue.suse.com/t7714592
https://openqa.nue.suse.com/t7714593
Updated by hjluo over 2 years ago
filed bug https://bugzilla.suse.com/show_bug.cgi?id=1193028 and changed the code.
Updated by hjluo over 2 years ago
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7620269 -s "_GROUP=0"
https://openqa.nue.suse.com/t7733741
https://openqa.nue.suse.com/t7733742
https://openqa.nue.suse.com/t7733743
Updated by hjluo over 2 years ago
this case tried 10 times and failed to get nfs info
https://openqa.nue.suse.com/tests/7733742#step/check_upgraded_service/261
Updated by hjluo over 2 years ago
Updated by vsvecova over 2 years ago
Hello, as already pointed out by jpupava in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13706, these changes seem to be breaking the maintenance tests: https://openqa.suse.de/tests/7755629 - failing on all products (from SLE12-SP5 to SLE15-SP2).
Updated by hjluo over 2 years ago
Hello, you can see that our changes do include t the line 'grep working /tmp/nfs/test', it just timeout, our fix finds the NFS from rpcinfo command and it works. so you can file a ticket for this timeout issue.
Thanks.
Updated by coolo over 2 years ago
Can you please revert the change now instead? The problem is blocking updates for 4 days already.
Updated by dzedro over 2 years ago
PR is reverted, no there will be no new ticket for timeout or typo or whatever is causing failures.
This PR broke other tests and it is either fixed ASAP or reverted.
This is nothing new, don't break other tests and don't make ridiculous demands.
Updated by hjluo over 2 years ago
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7766072 -s "_GROUP=0"
https://openqa.nue.suse.com/t7790902
https://openqa.nue.suse.com/t7790903
Updated by leli over 2 years ago
- Estimated time changed from 8.00 h to 12.00 h
Add estimated time since this issue need more time to analyze.
Updated by hjluo over 2 years ago
change the code and rerun:
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7842009 -s "_GROUP=0"
https://openqa.nue.suse.com/t7842341
tools ./hj-tools/hj-clone.sh -j 7779966 -s "_GROUP=0"
https://openqa.nue.suse.com/t7842336
https://openqa.nue.suse.com/t7842337
Updated by hjluo over 2 years ago
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7842013 -s "_GROUP=0"
https://openqa.nue.suse.com/t7843157 PASSED on sle-15-SP3-Server-DVD-Updates-aarch64-Build20211213-1-mau-extratests2
another one:
https://openqa.nue.suse.com/t7849805 PASSED
https://openqa.nue.suse.com/t7849841 -> w/o cat PASSED
Updated by hjluo over 2 years ago
Updated by hjluo over 2 years ago
lates verify run:
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7842013 -s "_GROUP=0"
https://openqa.nue.suse.com/t7850238
https://openqa.nue.suse.com/t7850497
Updated by hjluo over 2 years ago
verify run on sle-15-SP3-Server-DVD-Updates-aarch64-Build20211217-1-mau-extratests2@aarch64-virtio
https://openqa.suse.de/group_overview/366
https://openqa.suse.de/tests/7870345
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7870345 -s "_GROUP=0"
Updated by hjluo over 2 years ago
regressoin on s390x
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7519039 -s "_GROUP=0"
https://openqa.nue.suse.com/t7872464
Updated by hjluo over 2 years ago
all tests for https://openqa.suse.de/tests/overview?distri=sle&version=15-SP3&build=20211217-1&groupid=366
for i in 7870345 7870329 7870297 ; do
https://openqa.nue.suse.com/t7882525
https://openqa.nue.suse.com/t7882526
https://openqa.nue.suse.com/t7882527
ALL blocked by wget_ipv6
Updated by hjluo over 2 years ago
Maintenance 15sp3
for i in 7893077 7893061 7893029; do
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done
https://openqa.nue.suse.com/tests/7894069
https://openqa.nue.suse.com/t7893742
https://openqa.nue.suse.com/t7893743
ALL PASSED
Updated by hjluo over 2 years ago
maintenance 12sp5
for i in 7892368 7892490 7892423; do
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done
https://openqa.nue.suse.com/t7897933
https://openqa.nue.suse.com/t7897934
https://openqa.nue.suse.com/t7897935
Updated by hjluo over 2 years ago
rerun with Maintenance: SLE 15 SP3 Updates build 20211229-1
for i in 7918442 7919577 7918335; do
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done
https://openqa.nue.suse.com/t7921866
https://openqa.nue.suse.com/t7921867
https://openqa.nue.suse.com/t7921868
ALL PASSED
Updated by hjluo over 2 years ago
for i in 7960457 7960443 7960404;do
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done
https://openqa.nue.suse.com/t7960955 -> https://openqa.nue.suse.com/tests/7961302
https://openqa.nue.suse.com/t7960956
https://openqa.nue.suse.com/t7960957
ALL PASSED
Updated by hjluo over 2 years ago
for i in 7964121 7963886 7963846; do
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done
https://openqa.nue.suse.com/t7964697
https://openqa.nue.suse.com/t7964698
https://openqa.nue.suse.com/t7964699
Updated by hjluo over 2 years ago
- Status changed from In Progress to Resolved
- % Done changed from 70 to 100