QA » openQA Project » openQA Tests

The connection to the client via ssh couldn't be established. I'm investigating the reason for that.

Olli, you mentioned that you labeled the client scenario with the same progress issue and updated the subject line. Which should be the client scenario?

zluo has a running multimachine test environment so please try to clone and run these tests locally

cloned the job: http://e13.suse.de/tests/4205#

the issue is on support site, first_boot failed

support server status
http://e13.suse.de/tests/4204

looks like target cannot be installed because there was some code-change apparently and it looks for the wrong needles now

https://openqa.suse.de/tests/1180149# compared to SP3 https://openqa.suse.de/tests/1058714#

Till Build260.4 the test remote_ssh_controller can finish boot_to_desktop and reach remote_controller test module:
http://openqa.suse.de/tests/1173562

However the cloned test of this job also fails at boot_to_desktop:
http://f146.suse.de/tests/1212

The needle 'linux-login' was removed from the matching list, thus boot_to_desktop failed. I'll test with the needle added in the list.

I created PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3628
Maybe someone else can continue test with it.

With PR adapted and needles adjusted/created, the test can proceed on zluo's test machine:
http://e13.suse.de/tests/4266
http://e13.suse.de/tests/4265

Both have passed.

needles PR: https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/503
Please merge, thanks!

Please merge PR so we can close the ticket. Thanks.

The needles MR is merged but I don't want to merge the test PR. Maybe just a variable like "TEXTMODE=1" is missing on the scenario or something but it's not the right way to just accept the text login prompt.

Please evaluate if the problem is SLE15 specifc or if it also does not work anymore for SLE12

There is a variable VIDEOMODE=text in the setttings for this scenario. I compared the settings with SLE12SP3, there' no suspicious changes.

Olli, please take a look of the testrun for SLE12SP3: http://openqa.suse.de/tests/1058714#step/boot_to_desktop/2

At this stage, it looks for exactly these three needles: emergency-mode, emergency-shell, linux-login. And for SLE15 the test failed, because someone removed linux-login, instead put display-manager in the queue. What I did in my PR is to let it be back in the previous status. I don't see a reason that it shouldn't be merged.

yi wrote:

The needle 'linux-login' was removed from the matching list, thus boot_to_desktop failed. I'll test with the needle added in the list.

So I think we agreed that the needle was not removed from the matching list but the test code and/or test settings changed.

I triggered tests with explicitly set DESKTOP=textmode

$ openqa_clone_job_osd 1187284 DESKTOP=textmode TEST=okurz_poo#23814_triggered_with_videomode_text
Cloning dependencies of sle-15-Leanos-DVD-x86_64-Build278.1-remote_ssh_controller@64bit
Created job #1190375: sle-15-Leanos-DVD-x86_64-Build278.1-remote_ssh_target_ftp@64bit -> https://openqa.suse.de/t1190375
Created job #1190376: sle-15-Leanos-DVD-x86_64-Build278.1-remote_ssh_controller@64bit -> https://openqa.suse.de/t1190376

The test now succeeds to login to the text session so I conclude that we need the variable DESKTOP=textmode on the testsuite. I added this now to the testsuite remote_ssh_controller.

Now we are back to the old problem in https://openqa.suse.de/tests/1190376#step/remote_controller/32 , the ssh connection does not succeed. So back to my suggestion in #23814#note-11

Recent testruns(after removing of a workaround needle):
https://openqa.suse.de/tests/1193261
https://openqa.suse.de/tests/1193261

We tried a couple time to verify the issue. Fixed locally the needle inst-textselected which was not matched, and re-run the test., but without success.

http://e13.suse.de/tests/4323#step/remote_target/1

send_key 'ret' seems doesn't to work here, the support-server fails at stage remote-target.

@okurz

installation_ready is not available to remote_controller even after I added following changes (your suggestion) in remote_controller.pm:

--
type_string "ssh root\@$target_ip\n";
assert_screen "remote-ssh-login";
if (!check_screen('remote-ssh-login')) {
type_string "ssh -vvvv root\@$target_ip\n";
assert_screen('remote-ssh-login', 600);
}

type_string "yes\n";¶

same issue at stage remote-target:

http://e13.suse.de/tests/4327#step/remote_target/1

I use following command to clone the job:

sudo /usr/share/openqa/script/clone_job.pl --from http://openqa.suse.de 1193332 VNC_TYPING_LIMIT=40 DESKTOP=textmode NETBOOT=1 REMOTE_CONTROLLER=ssh

check the needle tag again, add inst-textselected-with_colormenu to inst-textselected because esc is needed after textmode selected and send_key 'ret' works then after this change.

However it stops at stage installation after the installation got started:

http://e13.suse.de/tests/4331#step/remote_target/5

// found ftp path. it seems that ftp path is wrong, otherwise URL page for installation won't be pop up.

well strange thing is // is working in browser however this is wrong.

the problem on my local machine is ftp port closed, so open ftp port 21, it looks then good:

http://e13.suse.de/tests/4335#step/remote_target/1

fixed now:

http://e13.suse.de/tests/4352
PR https://gitlab.suse.de/openqa/os-autoinst-needles-sles/merge_requests/514

@okurz
okay, will check this then

https://openqa.suse.de/tests/1196678#step/remote_controller/32

it seem that login itself has problem for remote-controller.

possible network issue on osd.

compare with my local server:

http://e13.suse.de/tests/4363#step/remote_controller/31

remote-ssh-login-20160511.json is there...

https://openqa.suse.de/tests/1195278#step/first_boot/6

shows that first_boot progress to prompt login.
this is actually already okay. the question why displaymanager (a gui needle) is needed?

the issue above is a different issue and needs to be solved at first.

compare with http://e13.suse.de/tests/4265#step/first_boot/1 (9 days ago)

it made steps successfully to first_boot in gui.

change first_boot.pm for the case when it boots up in textmode (text-login needle is required).

assert_screen [qw(displaymanager emergency-shell emergency-mode text-login)], $boot_timeout;

fixed and it looks at: http://e13.suse.de/tests/4370#step/first_boot/1

PR:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3672

that again failed in the controller assuming the desktop to be gnome on initial bootup. So we need to ensure that "change_desktop" is not called but the desktop of the controller is still textmode, hm.

The problem was introduced by adding more multimachine workers and connecting them together by using GRE tunnels.

When we were using only one multimachine worker the test didn't use any VLAN in ovs-switch but it worked because it was within one worker and the same ovs-switch.

Now I've set NETWORKS=fixed to remote_ssh* and remote_vnc* tests in osd which should assign VMs to the same VLAN on ovs-switch.

In between we had SLE15 ssh tests that worked but now we have an incomplete: https://openqa.suse.de/tests/1440078

Stuck in

[2018-02-01T00:01:46.0202 CET] [debug] ||| starting remote_controller tests/remote/remote_controller.pm
[2018-02-01T00:01:46.0228 CET] [debug] mutex lock 'installation_ready'
[2018-02-01T00:01:46.0258 CET] [debug] mutex lock 'installation_ready' unavailable, sleeping 5s

I recommend you check this again :) Could be a race condition?

Parent job timed out after 2 hours waiting for child.
Child job says: "State: cancelled finished about 9 hours ago ( 0 )" - seems it never started because there was no free worker (I'm guessing here)

Both were restarted and passed fine. Snippet from autoinst-log.txt is normal behavior of locked mutex.
I think this issue is resolved, wdyt?