action #50168
closed
[qam][ipmi] unable to login openQA SUT via ssh with ipmi backend
Added by Julie_CAO about 5 years ago.
Updated over 4 years ago.
Category:
Bugs in existing tests
Description
The failure happened quite often in recent days. It resulted tens of failing tests in login ssh console steps.
Here is the failure in one of the SUT which I loged in with ipmi console.
10.162.3.50
gonzo-1:~ # systemctl status sshd
● sshd.service - OpenSSH Daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset:
disabled)
Active: active (running) since Mon 2019-04-08 22:52:58 EDT; 32min
ago
Main PID: 1568 (sshd)
Tasks: 1
CGroup: /system.slice/sshd.service
└─1568 /usr/sbin/sshd -D
Apr 08 22:52:58 gonzo-1 systemd[1]: Starting OpenSSH Daemon...
Apr 08 22:52:58 gonzo-1 sshd-gen-keys-start[1546]: Checking for missing server k
eys in /etc/ssh
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on 0.0.0.0 port 22.
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on :: port 22.
Apr 08 22:52:58 gonzo-1 systemd[1]: Started OpenSSH Daemon.
Apr 08 22:53:07 gonzo-1 sshd[1843]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33952 ssh2
Apr 08 22:53:11 gonzo-1 sshd[1845]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33958 ssh2
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ecdsa_key
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ed25519_key
gonzo-1:~ # ll /etc/ssh/
total 168
drwxr-xr-x 1 root root 380 Apr 8 22:55 bak
-rw-r--r-- 1 root root 2375 Apr 7 00:35 ldap.conf
-rw------- 1 root root 132839 Apr 8 22:55 moduli
-rw-r--r-- 1 root root 2365 Apr 8 22:55 ssh_config
-rw------- 1 root root 668 Apr 8 22:55 ssh_host_dsa_key
-rw-r--r-- 1 root root 602 Apr 8 22:55 ssh_host_dsa_key.pub
-rw------- 1 root root 527 Apr 8 22:55 ssh_host_key
-rw-r--r-- 1 root root 331 Apr 8 22:55 ssh_host_key.pub
-rw------- 1 root root 887 Apr 8 22:55 ssh_host_rsa_key
-rw-r--r-- 1 root root 222 Apr 8 22:55 ssh_host_rsa_key.pub
-rw-r----- 1 root root 3420 Apr 7 00:35 sshd_config
Sorry, pasted an incorrect IP of SUT. the correct IP is 10.162.2.98.
- Subject changed from sshd failed to be start on openQA worker with ipmi backend to sshd failed to be start on openQA SUT with ipmi backend
- Subject changed from sshd failed to be start on openQA SUT with ipmi backend to [qam][ipmi] sshd failed to be start on openQA SUT with ipmi backend
I am not able to change the description.
Observation¶
The failure happened quite often in recent days. It resulted tens of failing tests in login ssh console steps.
Here is the failure in one of the SUT which I loged in with ipmi console.
10.162.2.98 //correct the SUT IP¶
gonzo-1:~ # systemctl status sshd
● sshd.service - OpenSSH Daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset:
disabled)
Active: active (running) since Mon 2019-04-08 22:52:58 EDT; 32min
ago
Main PID: 1568 (sshd)
Tasks: 1
CGroup: /system.slice/sshd.service
└─1568 /usr/sbin/sshd -D
Apr 08 22:52:58 gonzo-1 systemd[1]: Starting OpenSSH Daemon...
Apr 08 22:52:58 gonzo-1 sshd-gen-keys-start[1546]: Checking for missing server k
eys in /etc/ssh
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on 0.0.0.0 port 22.
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on :: port 22.
Apr 08 22:52:58 gonzo-1 systemd[1]: Started OpenSSH Daemon.
Apr 08 22:53:07 gonzo-1 sshd[1843]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33952 ssh2
Apr 08 22:53:11 gonzo-1 sshd[1845]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33958 ssh2
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ecdsa_key
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ed25519_key
gonzo-1:~ # ll /etc/ssh/
total 168
drwxr-xr-x 1 root root 380 Apr 8 22:55 bak
-rw-r--r-- 1 root root 2375 Apr 7 00:35 ldap.conf
-rw------- 1 root root 132839 Apr 8 22:55 moduli
-rw-r--r-- 1 root root 2365 Apr 8 22:55 ssh_config
-rw------- 1 root root 668 Apr 8 22:55 ssh_host_dsa_key
-rw-r--r-- 1 root root 602 Apr 8 22:55 ssh_host_dsa_key.pub
-rw------- 1 root root 527 Apr 8 22:55 ssh_host_key
-rw-r--r-- 1 root root 331 Apr 8 22:55 ssh_host_key.pub
-rw------- 1 root root 887 Apr 8 22:55 ssh_host_rsa_key
-rw-r--r-- 1 root root 222 Apr 8 22:55 ssh_host_rsa_key.pub
-rw-r----- 1 root root 3420 Apr 7 00:35 sshd_config
Reproducible¶
Not fail for every run, about 1/3 of logins failed. And happened on different SUT. //julie added.
Test fails since 210.1
in scenario sle-15-SP1-Installer-DVD-x86_64-gi-guest_sles12sp4-on-host-developing-kvm@64bit-ipmi
Expected result¶
Last good 208.1
Further information¶
Latest in sle-15-SP1-Installer-DVD-x86_64-gi-guest_sles12sp4-on-host-developing-kvm@64bit-ipmi
I fail to see why sshd is blamed here. The login-prompt clearly appears, openQA just misses to enter the credentials at the right time. This race might be influenced by several factors but in the end we should get rid of the race condition here. E.g. on s390 there was some work to switch over to ssh keys.
The ssh started with errors and we are unable to ssh login these SUT. After we access the SUT via ipmi console and restart sshd, the error disappeared and we are able to login the SUT via ssh. So the issue should the sshd start with error.
- Subject changed from [qam][ipmi] sshd failed to be start on openQA SUT with ipmi backend to [qam][ipmi] unable to login openQA SUT via ssh with ipmi backend
found it should be a sles15sp1 bug rather than an openQA infra issue. Just open a bug against sles15sp1 rc2:
Bug 1132187 - [ssh][build212.1]Unable to login via ssh due to "error: Could not load host key" https://bugzilla.suse.com/show_bug.cgi?id=1132187
- Status changed from New to Resolved
This was deemed to be a problem in SLE instead of openQA infra. Please reopen if it would be found to be otherwise. The bugzilla bug seems untouched since April.
Also available in: Atom
PDF