QA » openQA Project » openQA Tests

After manual test of 456.1, a VNC server was running on :1 on the installed system. This ticket could be deprecated.

slindomansilla really wants to keep it and work on it during the time. The scenario "gnome@ipmi" is not there anymore for SLE15 as of now. Could be I have scheduled "default@ipmi" instead. If not please add gnome@ipmi back.

blocked by "Test does not reach that step in 15-SP1" -> TODO: create ticket for https://openqa.suse.de/tests/latest?test=default&distri=sle&version=15-SP1&machine=64bit-ipmi&arch=x86_64&flavor=Installer-DVD# and update link to latest

workaround locally to get through welcome test module, but now I have problem with grub_test:

http://e13.suse.de/tests/7945#step/grub_test

http://e13.suse.de/tests/7946#step/first_boot/1

shows successful test run for first_boot of sles 15 sp1

http://e13.suse.de/tests/7947#step/first_boot/1

looks good now as well after I create new needle: grub2 (login_console-grub2-for-sles12-ipmi-20180831)

needle PR to fix grub_test:

https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/923

http://e13.suse.de/tests/latest?version=12-SP4&machine=64bit-ipmi&distri=sle&flavor=Server-DVD&test=default&arch=x86_64#next_previous

shows now stable tests results, I use workaround for boot option(characters limitation) in bootloader_setup.pm:

# Enable linuxrc core dumps https://en.opensuse.org/SDB:Linuxrc#p_linuxrccore
type_string_very_slow "linuxrc.core=/dev/$serialdev " unless $args{pxe};
type_string_very_slow "linuxrc.debug=4,trace " unless $args{pxe};

https://openqa.suse.de/tests/latest?distri=sle&flavor=Installer-DVD&test=gnome&arch=x86_64&version=15-SP1&machine=64bit-ipmi# doesn't show any problem with pxe.

https://openqa.suse.de/tests/2039165# shows grub_test failed.

https://openqa.suse.de/tests/2039165# shows grub_test failed.

but on my our ipmi machine it just works fine:
http://e13.suse.de/tests/8091#step/first_boot/14

So I assume strongly that we have problem with hardware performance on osd.

Will run more test runs for this for now to be sure that is related to hardware...

openqaworker13:3 on osd has problem with boot from disk, just for record.

zluo wrote:

So I assume strongly that we have problem with hardware performance on osd.

as always, this could be true but then we need to gather more data to point in the direction of what is at fault or what would need to be changed.

https://openqa.suse.de/tests/2039347#step/first_boot/4

shows this time grub menu and entries, but still failed for linux_login.

Compared with my cloned tests:

http://e13.suse.de/tests/8093#next_previous

no issue at all.

Need to collect more examples for now.

created gnome-first_boot_on-ipmi@64bit-ipmi on osd:

https://openqa.suse.de/tests/2039612#next_previous

https://openqa.suse.de/tests/2039612#next_previous shows issues for boot_from_pxe, grub_test.

first_boot reached, but got connection issue:

https://openqa.suse.de/tests/2039613#step/first_boot/4

or could not boot up:
https://openqa.suse.de/tests/2039616#step/first_boot/4

The test runs on osd above mentioned shows issues with ipmi worker and connection issue.

zluo wrote:

https://openqa.suse.de/tests/2039347#step/first_boot/4

shows this time grub menu and entries, but still failed for linux_login.

Compared with my cloned tests:

http://e13.suse.de/tests/8093#next_previous

no issue at all.

Need to collect more examples for now.

I had a look in the morning with @mgriessmeier at this. We figured out that grub_test should actually send return to confirm booting from HDD. first_boot should only wait for the login manager and should not handle the bootloader at all. Why it worked on your machine is unclear to me but I guess you maybe just created another needle.
I'll open a poo for this.

@nsinger needle PR got merged a week ago: https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/923

So it should be available on osd as well. And some tests on osd reached firs_boot as well.

https://openqa.suse.de/tests/2039631
https://openqa.suse.de/tests/2039619

openqaw1:1 seems to be okay

started 20 times test runs: https://openqa.suse.de/tests/2073161#next_previous

https://openqa.suse.de/tests/2073160 shows successful test run.
but we still have other issue: connection refused, so this is setup or network issue which doesn't have to do with initial problem here.

openqaworker2 makes trouble as I can see.

gonzo.qa.suse.de

2018-09-19T14:47:17.0319 CEST] [debug] <<< testapi::select_console(testapi_console='x11')
/usr/lib/os-autoinst/consoles/vnc_base.pm:71:{
'password' => 'nots3cr3t',
'port' => 5901,
'hostname' => 'gonzo.qa.suse.de'
}
[2018-09-19T14:47:19.0327 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused
[2018-09-19T14:47:20.0328 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused
[2018-09-19T14:47:21.0329 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused
[2018-09-19T14:47:22.0331 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused
[2018-09-19T14:47:23.0332 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused
[2018-09-19T14:47:24.0334 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused
[2018-09-19T14:47:25.0335 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused
[2018-09-19T14:47:26.0336 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused
[2018-09-19T14:47:27.0461 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused at /usr/lib/os-autoinst/testapi.pm line 1416.

https://openqa.suse.de/tests/2073157#step/grub_test/3 shows a different issue fro grub_test: openqaworker13:3

https://openqa.suse.de/tests/2079399#step/boot_from_pxe/27 shows boot_from_pxe has problem with repo url.

https://openqa.suse.de/tests/2079406#step/scc_registration/7

and this is a product issue which reported by me: http://bugzilla.suse.com/show_bug.cgi?id=1108975

Please see https://progress.opensuse.org/issues/41480

http://e13.suse.de/tests/8632#step/first_boot/3

shows problem with first_boot. This is sporadic however.

my $boot_timeout = (get_var('SES5_DEPLOY') || check_var('VIRSH_VMM_FAMILY', 'hyperv')) ? 450 : 200;
# SLE >= 15 s390x does not offer auto-started VNC server in SUT, only login prompt as in textmode
return if check_var('ARCH', 's390x') && sle_version_at_least('15');
if (check_var('WORKER_CLASS', 'hornet')) {
# hornet does not show the console output
diag "waiting $boot_timeout seconds to let hornet boot and finish initial script";
sleep $boot_timeout;
reset_consoles;
select_console 'root-ssh';
return;
}

boot_timeout 200 seems to be not enough here. I also the same issue on osd:
https://openqa.suse.de/tests/2104272#step/first_boot/3

Pretty sure no timeout will be enough when you are stuck in the grub menu and something like return key is not sent :)

http://e13.suse.de/tests/8705#step/await_install/4 shows a new issue but this is an other product issue. Will file a bug report.

PR for increase boot_timeout:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5851

bug report for the issue with await_install(mount issue):

https://bugzilla.suse.com/show_bug.cgi?id=1110283

we have rare problem for boot_from_pxe:

http://e13.suse.de/tests/8700#step/boot_from_pxe/30

screen is overflooded with SUSE Linux Enterprise 15.
I see string: Unexpected response: >>w<<

[2018-09-29T12:53:39.0582 CEST] [debug] <<< testapi::send_key(key='alt-sysrq-w', do_wait=0)
BYTES {"json_cmd_token":"QmlwrkXY","send_key":{"key":"alt-sysrq-w","json_cmd_token":"Dxyiclri"}}
[2018-09-29T12:53:39.0788 CEST] [debug] <<< testapi::wait_serial(regexp='SysRq : Show Blocked State', timeout=1)

So for this case I suspect ipmi machine got problem with loading files, it is probably a network issue at this moment.

We can see after fail:

Loading basic drivers...exec: insmod /modules/af_paclet.ko = 0

So it means ssh_vnc_wait_time = 300 is still not enough...

zluo wrote:

I see string: Unexpected response: >>w<<
[…]
So for this case I suspect ipmi machine got problem with loading files, it is probably a network issue at this moment.

What files are you talking about? This key press is happening in the post_fail_hook so can not be the real issue. https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5769 introduced this magic-sysrq-keypress. dheidler worked on that. AFAIR he planned to exclude it from triggering on IPMI.

"Loading basic drivers...exec: insmod /modules/af_paclet.ko = 0" shows up, this is what I mean

yes, magic-sysrq-keypress should not confuse in this case/on ipmi, thanks.

compared with successful test run http://e13.suse.de/tests/8728:

After loading basic drivers...exec: insmod /modues/af_packet.ko = 0

the whole modules cannot be loaded. I found 2 errors but they don't seem to have impact on booting up to VNC stage

It looks already much better, but it failed still sporadic:

http://e13.suse.de/tests/8846#step/boot_from_pxe/30

will try with 450 for ipmi now:

my $ssh_vnc_wait_time = (check_var('BACKEND', 'ipmi')) ? 450 : 300;

http://e13.suse.de/tests/8855#next_previous shows green status except boot_from_pxe.

It means that first_boot doesn't fail.

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5851/files

changes for ipmi only:

my $boot_timeout = (get_var('SES5_DEPLOY') || check_var('VIRSH_VMM_FAMILY', 'hyperv') || check_var('BACKEND', 'ipmi')) ? 450 : 200;

PR merged, changes deployed to osd. Could you follow https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation and trigger some more tests on osd to check statistics?

mark this test run on osd: https://openqa.suse.de/tests/2122654#

Will check this later.

it failed because of https://progress.opensuse.org/issues/41762

check on my local machine: http://e13.suse.de/tests/9147

and restart on osd: https://openqa.suse.de/tests/2123010

run tests on osd:

https://openqa.suse.de/tests/2123037#next_previous

trigger 20 times:

http://e13.suse.de/tests/9150#next_previous

https://openqa.suse.de/tests/2122251#step/first_boot/5 shows a another issue again:

[2018-10-04T15:05:33.0518 CEST] [debug] Error connecting to host : IO::Socket::INET: connect: Connection refused

https://progress.opensuse.org/issues/31375#note-57

remote_slave_ready-ipmi-20171023.json is mtaching for needle tag vnc-server-started at: http://e13.suse.de/tests/9170#step/boot_from_pxe/7

but now it failed:

https://openqa.suse.de/tests/2123037#step/boot_from_pxe/30

https://openqa.suse.de/tests/2122457 shows that it works fine.

https://openqa.suse.de/tests/2127922#step/partitioning_filesystem/6 shows partitioning_filesystem has failed in build 62.6, but this is an another issue.

I think the original issue with first_boot got resolved. And the issue with boot_from_pxe still remained, but we continue on it: https://progress.opensuse.org/issues/41693

Trying to make this ticket a subticket of: #23650

We have timeout problem on first_boot on LTP on ipmi backend:
https://openqa.suse.de/tests/2262596#step/first_boot/2
Any idea what that caused?
Problem was with GRUB_TIMEOUT setting first to 300 (good for IPMI) and even with 900.

@pvorel I think this is neither #42863, this ticket, nor #31375, rather its own, new thing. The 'ret' key in the grub screen does not have an effect. Better track as its own issue.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/2522783

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/2745480

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/2799086

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/2826516

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/2890123

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/2915078

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/2974778

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/2992439

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/3058673

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/3159955

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: default@64bit-ipmi
https://openqa.suse.de/tests/3227997