Project

General

Profile

Actions

action #101475

closed

[sle][migration][SLE15SP4][regression] test fails in check_upgraded_service - rpcinfo | grep nfs' failed

Added by leli over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
-
Start date:
2021-10-26
Due date:
% Done:

100%

Estimated time:
12.00 h
Difficulty:

Description

Observation

openQA test in scenario sle-15-SP4-Regression-on-Migration-from-SLE15-SPx-s390x-offline_sles15sp1_ltss_pscc_basesys-srv-desk-dev-contm-lgm-py2-tsm-wsm_all_full@s390x-kvm-sle12 fails in
check_upgraded_service

Test suite description

Reproducible

Fails since (at least) Build 52.1

Expected result

Last good: 50.1 (or more recent)

Further details

Always latest result in this scenario: latest

Actions #1

Updated by hjluo over 2 years ago

  • Assignee set to hjluo
Actions #2

Updated by hjluo over 2 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10
Actions #3

Updated by hjluo over 2 years ago

./hj-tools/hj-clone.sh -j 7555430 -s "_GROUP=0"
https://openqa.nue.suse.com/t7580828 FAILED https://openqa.nue.suse.com/tests/7580828#step/check_upgraded_service/232
https://openqa.nue.suse.com/t7580829 PASSED

so this is a sporadic issue and maybe a timing issue. we may wait a while before check the nfs

Actions #6

Updated by hjluo over 2 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 10 to 100

so close it now since we've passed in 3 consecutive builds.

Actions #7

Updated by leli over 2 years ago

  • Status changed from Resolved to Workable
Actions #10

Updated by hjluo over 2 years ago

now blocked by bug bsc#1192683 in 63.1

Actions #11

Updated by hjluo over 2 years ago

  • Status changed from Workable to In Progress
  • % Done changed from 100 to 70
Actions #12

Updated by hjluo over 2 years ago

./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7620269 -s "_GROUP=0"
https://openqa.nue.suse.com/t7693984
https://openqa.nue.suse.com/t7693985
ALL PASSED

Actions #13

Updated by hjluo over 2 years ago

you can see we've tried 2 times before the nfs started and can be shown in rpcinfo.

https://openqa.nue.suse.com/tests/7693984/logfile?filename=autoinst-log.txt

[2021-11-17T10:25:00.011650+01:00] [debug] tests/installation/install_service.pm:21 called service_check::install_services -> lib/service_check.pm:252 called (eval) -> lib/service_check.pm:263 called services::rpcbind::full_rpcbind_check -> lib/services/rpcbind.pm:91 called services::rpcbind::check_function -> lib/services/rpcbind.pm:66 called testapi::script_run
[2021-11-17T10:25:00.011814+01:00] [debug] <<< testapi::script_run(cmd="rpcinfo | grep nfs", output="", quiet=undef, timeout=undef)
[2021-11-17T10:25:00.011992+01:00] [debug] tests/installation/install_service.pm:21 called service_check::install_services -> lib/service_check.pm:252 called (eval) -> lib/service_check.pm:263 called services::rpcbind::full_rpcbind_check -> lib/services/rpcbind.pm:91 called services::rpcbind::check_function -> lib/services/rpcbind.pm:66 called testapi::script_run
[2021-11-17T10:25:00.012160+01:00] [debug] <<< testapi::type_string(string="rpcinfo | grep nfs", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)

Actions #15

Updated by hjluo over 2 years ago

after discussion with richard and lemon, we'd check the nfs server before check the rpcinfo|grep nfs in the check_funciton
changed code and rerun:
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7620269 -s "_GROUP=0"
https://openqa.nue.suse.com/t7714308
https://openqa.nue.suse.com/t7714309
https://openqa.nue.suse.com/t7714310

you can see it's still 2 times before we get nfs info from rpcinfo.
https://openqa.nue.suse.com/tests/7714309/logfile?filename=autoinst-log.txt
[2021-11-22T04:54:11.947954+01:00] [debug] <<< testapi::script_run(cmd="rpcinfo | grep nfs", output="", timeout=undef, quiet=undef)
[2021-11-22T04:54:11.948122+01:00] [debug] tests/installation/install_service.pm:21 called service_check::install_services -> lib/service_check.pm:252 called (eval) -> lib/service_check.pm:263 called services::rpcbind::full_rpcbind_check -> lib/services/rpcbind.pm:93 called services::rpcbind::check_function -> lib/services/rpcbind.pm:68 called testapi::script_run
[2021-11-22T04:54:11.948290+01:00] [debug] <<< testapi::type_string(string="rpcinfo | grep nfs", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)

Actions #16

Updated by hjluo over 2 years ago

run for test log to file a bug after discussion with lemon and richard
https://openqa.nue.suse.com/t7714591
https://openqa.nue.suse.com/t7714592
https://openqa.nue.suse.com/t7714593

Actions #17

Updated by hjluo over 2 years ago

filed bug https://bugzilla.suse.com/show_bug.cgi?id=1193028 and changed the code.

Actions #18

Updated by hjluo over 2 years ago

./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7620269 -s "_GROUP=0"

https://openqa.nue.suse.com/t7733741
https://openqa.nue.suse.com/t7733742
https://openqa.nue.suse.com/t7733743

Actions #19

Updated by hjluo over 2 years ago

this case tried 10 times and failed to get nfs info
https://openqa.nue.suse.com/tests/7733742#step/check_upgraded_service/261

Actions #22

Updated by vsvecova over 2 years ago

Hello, as already pointed out by jpupava in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13706, these changes seem to be breaking the maintenance tests: https://openqa.suse.de/tests/7755629 - failing on all products (from SLE12-SP5 to SLE15-SP2).

Actions #23

Updated by hjluo over 2 years ago

Hello, you can see that our changes do include t the line 'grep working /tmp/nfs/test', it just timeout, our fix finds the NFS from rpcinfo command and it works. so you can file a ticket for this timeout issue.

Thanks.

Actions #24

Updated by coolo over 2 years ago

Can you please revert the change now instead? The problem is blocking updates for 4 days already.

Actions #25

Updated by dzedro over 2 years ago

PR is reverted, no there will be no new ticket for timeout or typo or whatever is causing failures.
This PR broke other tests and it is either fixed ASAP or reverted.
This is nothing new, don't break other tests and don't make ridiculous demands.

Actions #26

Updated by hjluo over 2 years ago

./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7766072 -s "_GROUP=0"
https://openqa.nue.suse.com/t7790902
https://openqa.nue.suse.com/t7790903

Actions #27

Updated by leli over 2 years ago

  • Estimated time changed from 8.00 h to 12.00 h

Add estimated time since this issue need more time to analyze.

Actions #28

Updated by hjluo over 2 years ago

change the code and rerun:
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7842009 -s "_GROUP=0"
https://openqa.nue.suse.com/t7842341

tools ./hj-tools/hj-clone.sh -j 7779966 -s "_GROUP=0"

https://openqa.nue.suse.com/t7842336
https://openqa.nue.suse.com/t7842337

Actions #29

Updated by hjluo over 2 years ago

./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7842013 -s "_GROUP=0"
https://openqa.nue.suse.com/t7843157 PASSED on sle-15-SP3-Server-DVD-Updates-aarch64-Build20211213-1-mau-extratests2

another one:
https://openqa.nue.suse.com/t7849805 PASSED
https://openqa.nue.suse.com/t7849841 -> w/o cat PASSED

Actions #31

Updated by hjluo over 2 years ago

lates verify run:
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7842013 -s "_GROUP=0"
https://openqa.nue.suse.com/t7850238
https://openqa.nue.suse.com/t7850497

Actions #32

Updated by hjluo over 2 years ago

verify run on sle-15-SP3-Server-DVD-Updates-aarch64-Build20211217-1-mau-extratests2@aarch64-virtio
https://openqa.suse.de/group_overview/366
https://openqa.suse.de/tests/7870345
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7870345 -s "_GROUP=0"

https://openqa.nue.suse.com/t7871255 PASSED

Actions #33

Updated by hjluo over 2 years ago

regressoin on s390x
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j 7519039 -s "_GROUP=0"
https://openqa.nue.suse.com/t7872464

Actions #35

Updated by hjluo over 2 years ago

Maintenance 15sp3
for i in 7893077 7893061 7893029; do

./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done
https://openqa.nue.suse.com/tests/7894069
https://openqa.nue.suse.com/t7893742
https://openqa.nue.suse.com/t7893743
ALL PASSED

Actions #36

Updated by hjluo over 2 years ago

maintenance 12sp5
for i in 7892368 7892490 7892423; do

./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done

https://openqa.nue.suse.com/t7897933
https://openqa.nue.suse.com/t7897934
https://openqa.nue.suse.com/t7897935

Actions #37

Updated by hjluo over 2 years ago

rerun with Maintenance: SLE 15 SP3 Updates build 20211229-1
for i in 7918442 7919577 7918335; do

./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done
https://openqa.nue.suse.com/t7921866
https://openqa.nue.suse.com/t7921867
https://openqa.nue.suse.com/t7921868

ALL PASSED

Actions #38

Updated by hjluo over 2 years ago

for i in 7960457 7960443 7960404;do
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done

https://openqa.nue.suse.com/t7960955 -> https://openqa.nue.suse.com/tests/7961302
https://openqa.nue.suse.com/t7960956
https://openqa.nue.suse.com/t7960957

ALL PASSED

Actions #39

Updated by hjluo over 2 years ago

for i in 7964121 7963886 7963846; do
./hj-tools/hj-branch.sh -a hjluo -b rpcinfo -j $i -s "_GROUP=0"
done
https://openqa.nue.suse.com/t7964697
https://openqa.nue.suse.com/t7964698
https://openqa.nue.suse.com/t7964699

Actions #40

Updated by hjluo over 2 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 70 to 100
Actions

Also available in: Atom PDF