action #50168

[qam][ipmi] unable to login openQA SUT via ssh with ipmi backend

Added by Julie_CAO 11 months ago. Updated 7 months ago.

Status:ResolvedStart date:09/04/2019
Priority:HighDue date:
Assignee:-% Done:

0%

Category:Bugs in existing tests
Target version:-
Difficulty:
Duration:

Description

The failure happened quite often in recent days. It resulted tens of failing tests in login ssh console steps.

Here is the failure in one of the SUT which I loged in with ipmi console.

10.162.3.50
gonzo-1:~ # systemctl status sshd
● sshd.service - OpenSSH Daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset:
disabled)
Active: active (running) since Mon 2019-04-08 22:52:58 EDT; 32min
ago
Main PID: 1568 (sshd)
Tasks: 1
CGroup: /system.slice/sshd.service
└─1568 /usr/sbin/sshd -D

Apr 08 22:52:58 gonzo-1 systemd[1]: Starting OpenSSH Daemon...
Apr 08 22:52:58 gonzo-1 sshd-gen-keys-start[1546]: Checking for missing server k
eys in /etc/ssh
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on 0.0.0.0 port 22.
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on :: port 22.
Apr 08 22:52:58 gonzo-1 systemd[1]: Started OpenSSH Daemon.
Apr 08 22:53:07 gonzo-1 sshd[1843]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33952 ssh2
Apr 08 22:53:11 gonzo-1 sshd[1845]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33958 ssh2
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ecdsa_key
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ed25519_key

gonzo-1:~ # ll /etc/ssh/
total 168
drwxr-xr-x 1 root root 380 Apr 8 22:55 bak
-rw-r--r-- 1 root root 2375 Apr 7 00:35 ldap.conf
-rw------- 1 root root 132839 Apr 8 22:55 moduli
-rw-r--r-- 1 root root 2365 Apr 8 22:55 ssh_config
-rw------- 1 root root 668 Apr 8 22:55 ssh_host_dsa_key
-rw-r--r-- 1 root root 602 Apr 8 22:55 ssh_host_dsa_key.pub
-rw------- 1 root root 527 Apr 8 22:55 ssh_host_key
-rw-r--r-- 1 root root 331 Apr 8 22:55 ssh_host_key.pub
-rw------- 1 root root 887 Apr 8 22:55 ssh_host_rsa_key
-rw-r--r-- 1 root root 222 Apr 8 22:55 ssh_host_rsa_key.pub
-rw-r----- 1 root root 3420 Apr 7 00:35 sshd_config

History

#1 Updated by Julie_CAO 11 months ago

Sorry, pasted an incorrect IP of SUT. the correct IP is 10.162.2.98.

#2 Updated by Julie_CAO 11 months ago

  • Subject changed from sshd failed to be start on openQA worker with ipmi backend to sshd failed to be start on openQA SUT with ipmi backend

#4 Updated by SLindoMansilla 11 months ago

  • Subject changed from sshd failed to be start on openQA SUT with ipmi backend to [qam][ipmi] sshd failed to be start on openQA SUT with ipmi backend

As a result of backlog triaging (see https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging for more information).

Please, feel free to adjust the category or the "[label]" if you think different.

#5 Updated by SLindoMansilla 11 months ago

I am not able to change the description.

Observation

The failure happened quite often in recent days. It resulted tens of failing tests in login ssh console steps.

Here is the failure in one of the SUT which I loged in with ipmi console.

10.162.2.98 //correct the SUT IP

gonzo-1:~ # systemctl status sshd
● sshd.service - OpenSSH Daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset:
 disabled)
   Active: active (running) since Mon 2019-04-08 22:52:58 EDT; 32min 
ago
 Main PID: 1568 (sshd)
    Tasks: 1
   CGroup: /system.slice/sshd.service
           └─1568 /usr/sbin/sshd -D

Apr 08 22:52:58 gonzo-1 systemd[1]: Starting OpenSSH Daemon...
Apr 08 22:52:58 gonzo-1 sshd-gen-keys-start[1546]: Checking for missing server k
eys in /etc/ssh
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on 0.0.0.0 port 22.
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on :: port 22.
Apr 08 22:52:58 gonzo-1 systemd[1]: Started OpenSSH Daemon.
Apr 08 22:53:07 gonzo-1 sshd[1843]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33952 ssh2
Apr 08 22:53:11 gonzo-1 sshd[1845]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33958 ssh2
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ecdsa_key
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ed25519_key


gonzo-1:~ # ll /etc/ssh/
total 168
drwxr-xr-x 1 root root    380 Apr  8 22:55 bak
-rw-r--r-- 1 root root   2375 Apr  7 00:35 ldap.conf
-rw------- 1 root root 132839 Apr  8 22:55 moduli
-rw-r--r-- 1 root root   2365 Apr  8 22:55 ssh_config
-rw------- 1 root root    668 Apr  8 22:55 ssh_host_dsa_key
-rw-r--r-- 1 root root    602 Apr  8 22:55 ssh_host_dsa_key.pub
-rw------- 1 root root    527 Apr  8 22:55 ssh_host_key
-rw-r--r-- 1 root root    331 Apr  8 22:55 ssh_host_key.pub
-rw------- 1 root root    887 Apr  8 22:55 ssh_host_rsa_key
-rw-r--r-- 1 root root    222 Apr  8 22:55 ssh_host_rsa_key.pub
-rw-r----- 1 root root   3420 Apr  7 00:35 sshd_config

Reproducible

Not fail for every run, about 1/3 of logins failed. And happened on different SUT. //julie added.
Test fails since 210.1
in scenario sle-15-SP1-Installer-DVD-x86_64-gi-guest_sles12sp4-on-host-developing-kvm@64bit-ipmi

Expected result

Last good 208.1

Further information

Latest in sle-15-SP1-Installer-DVD-x86_64-gi-guest_sles12sp4-on-host-developing-kvm@64bit-ipmi

#6 Updated by nicksinger 11 months ago

I fail to see why sshd is blamed here. The login-prompt clearly appears, openQA just misses to enter the credentials at the right time. This race might be influenced by several factors but in the end we should get rid of the race condition here. E.g. on s390 there was some work to switch over to ssh keys.

#7 Updated by Julie_CAO 11 months ago

The ssh started with errors and we are unable to ssh login these SUT. After we access the SUT via ipmi console and restart sshd, the error disappeared and we are able to login the SUT via ssh. So the issue should the sshd start with error.

#8 Updated by Julie_CAO 11 months ago

  • Subject changed from [qam][ipmi] sshd failed to be start on openQA SUT with ipmi backend to [qam][ipmi] unable to login openQA SUT via ssh with ipmi backend

#10 Updated by Julie_CAO 11 months ago

found it should be a sles15sp1 bug rather than an openQA infra issue. Just open a bug against sles15sp1 rc2:
Bug 1132187 - [ssh][build212.1]Unable to login via ssh due to "error: Could not load host key" https://bugzilla.suse.com/show_bug.cgi?id=1132187

#11 Updated by tjyrinki_suse 7 months ago

  • Status changed from New to Resolved

This was deemed to be a problem in SLE instead of openQA infra. Please reopen if it would be found to be otherwise. The bugzilla bug seems untouched since April.

Also available in: Atom PDF