action #60662

[SLE][Migration][SLE15SP2][Regression] Some regression tests fails on ppc64le - /dev/hvc0: permission denied

Added by tinawang123 4 months ago. Updated 7 days ago.

Status:In ProgressStart date:05/12/2019
Priority:UrgentDue date:
Assignee:hjluo% Done:

80%

Category:Bugs in existing testsEstimated time:16.00 hours
Target version:-
Difficulty:
Duration:

Description

Observation

openQA test in scenario sle-15-SP2-Regression-on-Migration-from-SLE15-SPX-to-SLE15-SP2-ppc64le-online_sles15_pscc_Addons-basesys-srv-desk-dev-contm-lgm-pcm-wsm_all_full@ppc64le fails in
curl_https

Test suite description

Reproducible

Fails since (at least) Build 80.5

Further details

Always latest result in this scenario: latest


Related issues

Related to openQA Tests - action #60935: [SLE][Migration][SLE15SP2][Regression]test fails in salt ... Resolved 12/12/2019

History

#1 Updated by coolgw 4 months ago

dev/hvc0 not permission

#2 Updated by hjluo 4 months ago

  • Assignee set to hjluo

#3 Updated by leli 4 months ago

  • Estimated time set to 16.00

#4 Updated by hjluo 4 months ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 20

rerun it in 108.1 to check it it still hit this issue.
https://openqa.suse.de/tests/3739548 this case failed at zypper migration
reason: proxy scc is not working during zypper migration

#5 Updated by hjluo 4 months ago

  • Status changed from In Progress to Blocked
  • % Done changed from 20 to 30

migration was not completed and can't reach curl_https module.

#7 Updated by hjluo 3 months ago

skip module install_service and check_upgraded_service and rerun https://openqa.suse.de/tests/375091

#8 Updated by hjluo 3 months ago

  • % Done changed from 30 to 40

it looks like the connection that curl to https://eu.httpbin.org was timedout sometimes.

curl -f -v https://eu.httpbin.org/get failed: script timeout: curl -f -v https://httpbin.org/get 2>&1 at /usr/lib/os-autoinst/testapi.pm line 1091.

[2019-12-04T08:45:27.131 CET] [debug] Maybe the network is busy. Retry: 7 of 7
[2019-12-04T08:45:27.131 CET] [debug] curl failed (with retries) at /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/console/curl_https.pm line 36.

curl_https-dmesg.log:[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinux-5.3.13-1-default root=UUID=567b5484-b2b4-40b4-85a1-b19d7550fc9a plymouth.ignore-serial-consoles console=hvc0 console=tty splash=silent showopts crashkernel=640M

#9 Updated by hjluo 3 months ago

#10 Updated by hjluo 3 months ago

try console=tty console=hvc0 in kernel boot parameters

#12 Updated by hjluo 3 months ago

  • Status changed from Blocked to In Progress

#14 Updated by hjluo 3 months ago

  • Status changed from In Progress to Blocked

#15 Updated by leli 3 months ago

  • Subject changed from [SLE][Migration][SLE15SP2]test fails in curl_https - permission denied to [SLE][Migration][SLE15SP2][Regression]test fails in curl_https - permission denied

#16 Updated by hjluo 2 months ago

the test case name is online_sles15_pscc_lp-basesys-srv-desk-dev-contm-lgm-wsm_all_full on ppc64le. On latest build 132.1 I saw this case passed.
and in 132.1 blocked at zypper migration for poo#63184

#17 Updated by hjluo about 1 month ago

  • Status changed from Blocked to Resolved
  • % Done changed from 50 to 100

till 142.1 didn't hit this issue so we'll close this by now and will reopen it once the test case hit this case again.

#19 Updated by hjluo 30 days ago

  • % Done changed from 100 to 60

actually it's not the permission issue, just because the shell output is not fully returned.

#20 Updated by hjluo 30 days ago

verify run with enlarging timeout https://openqa.nue.suse.com/tests/3975555 passed.
and the previous failed one is: http://openqa.suse.de/tests/3971265

#21 Updated by leli 29 days ago

  • Priority changed from Normal to Urgent

#22 Updated by hjluo 28 days ago

  • Subject changed from [SLE][Migration][SLE15SP2][Regression]test fails in curl_https - permission denied to [SLE][Migration][SLE15SP2][Regression] Some regression tests fails on ppc64le - /dev/hvc0: permission denied

This kind of issue includes curl_https,salt,mtab, zypper_lifecycle,keymap_or_locale_x11,sshxterm

#23 Updated by coolgw 23 days ago

  • Related to action #60935: [SLE][Migration][SLE15SP2][Regression]test fails in salt - salt-run timeout added

#24 Updated by okurz 12 days ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: offline_sles15sp1_pscc_lp-basesys-srv-desk-dev-contm-lgm-py2-tsm-wsm_all_full
https://openqa.suse.de/tests/4042033

To prevent further reminder comments one of the following options should be followed:
1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released"
3. The label in the openQA scenario is removed

#26 Updated by hjluo 9 days ago

in the logfile we can see the console wait time of 3o is not enough
https://openqa.nue.suse.com/tests/4042033/file/autoinst-log.txt

[2020-03-26T16:02:31.936 CET] [debug] tests/console/curl_https.pm:24 called testapi::select_console -> lib/susedistribution.pm:884 called testapi::assert_screen
[2020-03-26T16:02:31.936 CET] [debug] <<< testapi::assert_screen(mustmatch="user-console", no_wait=1, timeout=30)
[2020-03-26T16:02:32.050 CET] [debug] >>> testapi::_handle_found_needle: found user-console-20191207, similarity 1.00 @ 145/210
[2020-03-26T16:02:32.050 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:02:32.050 CET] [debug] <<< testapi::type_string(string="cat - > /tmp/script4V9zF.sh;\n", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2020-03-26T16:02:35.132 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:02:35.133 CET] [debug] <<< testapi::type_string(string="curl -f -v https://httpbin.org/get 2>&1\n", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2020-03-26T16:02:39.327 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:02:39.328 CET] [debug] <<< testapi::send_key(key="ctrl-d", wait_screen_change=0, do_wait=0)
[2020-03-26T16:02:39.929 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:02:39.929 CET] [debug] <<< testapi::type_string(string="(echo 4V9zF; bash -eox pipefail /tmp/script4V9zF.sh ; echo SCRIPT_FINISHED4V9zF-\$?-) | tee /dev/hvc0\n", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2020-03-26T16:02:51.402 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:02:51.403 CET] [debug] <<< testapi::wait_serial(quiet=undef, regexp="SCRIPT_FINISHED4V9zF-\d+-", timeout=30, expect_not_found=0, no_regex=0, buffer_size=undef, record_output=1)
[2020-03-26T16:03:22.536 CET] [debug] >>> testapi::wait_serial: SCRIPT_FINISHED4V9zF-\d+-: fail
[2020-03-26T16:03:22.537 CET] [debug] curl -f -v https://eu.httpbin.org/get failed: script timeout: curl -f -v https://httpbin.org/get 2>&1 at /usr/lib/os-autoinst/testapi.pm line 1104.

[2020-03-26T16:03:22.537 CET] [debug] Maybe the network is busy. Retry: 1 of 7
[2020-03-26T16:03:22.537 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:03:22.537 CET] [debug] <<< testapi::type_string(string="cat - > /tmp/script4V9zF.sh;\n", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2020-03-26T16:03:25.606 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:03:25.607 CET] [debug] <<< testapi::type_string(string="curl -f -v https://httpbin.org/get 2>&1\n", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2020-03-26T16:03:29.782 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:03:29.782 CET] [debug] <<< testapi::send_key(key="ctrl-d", wait_screen_change=0, do_wait=0)
[2020-03-26T16:03:30.383 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:03:30.383 CET] [debug] <<< testapi::type_string(string="(echo 4V9zF; bash -eox pipefail /tmp/script4V9zF.sh ; echo SCRIPT_FINISHED4V9zF-\$?-) | tee /dev/hvc0\n", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2020-03-26T16:03:41.870 CET] [debug] tests/console/curl_https.pm:29 called (eval) -> tests/console/curl_https.pm:30 called testapi::validate_script_output
[2020-03-26T16:03:41.870 CET] [debug] <<< testapi::wait_serial(quiet=undef, timeout=30, regexp="SCRIPT_FINISHED4V9zF-\d+-", record_output=1, buffer_size=undef, expect_not_found=0, no_regex=0)
[2020-03-26T16:04:12.999 CET] [debug] >>> testapi::wait_serial: SCRIPT_FINISHED4V9zF-\d+-: fail
[2020-03-26T16:04:13.000 CET] [debug] curl -f -v https://eu.httpbin.org/get failed: script timeout: curl -f -v https://httpbin.org/get 2>&1 at /usr/lib/os-autoinst/testapi.pm line 1104.

#27 Updated by hjluo 7 days ago

  • % Done changed from 70 to 80

now we use the script_run to avoid the console output which may be caused by the valide_script_output. with new fix it will works even
we hit the /dev/hvc0 permission denied.
http://149.44.176.58/tests/4072395#step/curl_https/8 ==>permission denied
http://149.44.176.58/tests/4072613#step/curl_https/2 ==>no permission denied

Also available in: Atom PDF