Project

General

Profile

Actions
Copy link

action #50168

closed

[qam][ipmi] unable to login openQA SUT via ssh with ipmi backend

Added by Julie_CAO about 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
Bugs in existing tests
Target version:
-
Start date:
2019-04-09
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

The failure happened quite often in recent days. It resulted tens of failing tests in login ssh console steps.

Here is the failure in one of the SUT which I loged in with ipmi console.

10.162.3.50
gonzo-1:~ # systemctl status sshd
● sshd.service - OpenSSH Daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset:
disabled)
Active: active (running) since Mon 2019-04-08 22:52:58 EDT; 32min
ago
Main PID: 1568 (sshd)
Tasks: 1
CGroup: /system.slice/sshd.service
└─1568 /usr/sbin/sshd -D

Apr 08 22:52:58 gonzo-1 systemd[1]: Starting OpenSSH Daemon...
Apr 08 22:52:58 gonzo-1 sshd-gen-keys-start[1546]: Checking for missing server k
eys in /etc/ssh
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on 0.0.0.0 port 22.
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on :: port 22.
Apr 08 22:52:58 gonzo-1 systemd[1]: Started OpenSSH Daemon.
Apr 08 22:53:07 gonzo-1 sshd[1843]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33952 ssh2
Apr 08 22:53:11 gonzo-1 sshd[1845]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33958 ssh2
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ecdsa_key
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ed25519_key

gonzo-1:~ # ll /etc/ssh/
total 168
drwxr-xr-x 1 root root 380 Apr 8 22:55 bak
-rw-r--r-- 1 root root 2375 Apr 7 00:35 ldap.conf
-rw------- 1 root root 132839 Apr 8 22:55 moduli
-rw-r--r-- 1 root root 2365 Apr 8 22:55 ssh_config
-rw------- 1 root root 668 Apr 8 22:55 ssh_host_dsa_key
-rw-r--r-- 1 root root 602 Apr 8 22:55 ssh_host_dsa_key.pub
-rw------- 1 root root 527 Apr 8 22:55 ssh_host_key
-rw-r--r-- 1 root root 331 Apr 8 22:55 ssh_host_key.pub
-rw------- 1 root root 887 Apr 8 22:55 ssh_host_rsa_key
-rw-r--r-- 1 root root 222 Apr 8 22:55 ssh_host_rsa_key.pub
-rw-r----- 1 root root 3420 Apr 7 00:35 sshd_config

Actions
Copy link
 #1

Updated by Julie_CAO about 5 years ago

Sorry, pasted an incorrect IP of SUT. the correct IP is 10.162.2.98.

Actions
Copy link
 #2

Updated by Julie_CAO about 5 years ago

  • Subject changed from sshd failed to be start on openQA worker with ipmi backend to sshd failed to be start on openQA SUT with ipmi backend
Actions
Copy link
 #4

Updated by SLindoMansilla about 5 years ago

  • Subject changed from sshd failed to be start on openQA SUT with ipmi backend to [qam][ipmi] sshd failed to be start on openQA SUT with ipmi backend

As a result of backlog triaging (see https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging for more information).

Please, feel free to adjust the category or the "[label]" if you think different.

Actions
Copy link
 #5

Updated by SLindoMansilla about 5 years ago

I am not able to change the description.

Observation

The failure happened quite often in recent days. It resulted tens of failing tests in login ssh console steps.

Here is the failure in one of the SUT which I loged in with ipmi console.

10.162.2.98 //correct the SUT IP

gonzo-1:~ # systemctl status sshd
● sshd.service - OpenSSH Daemon
   Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; vendor preset:
 disabled)
   Active: active (running) since Mon 2019-04-08 22:52:58 EDT; 32min 
ago
 Main PID: 1568 (sshd)
    Tasks: 1
   CGroup: /system.slice/sshd.service
           └─1568 /usr/sbin/sshd -D

Apr 08 22:52:58 gonzo-1 systemd[1]: Starting OpenSSH Daemon...
Apr 08 22:52:58 gonzo-1 sshd-gen-keys-start[1546]: Checking for missing server k
eys in /etc/ssh
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on 0.0.0.0 port 22.
Apr 08 22:52:58 gonzo-1 sshd[1568]: Server listening on :: port 22.
Apr 08 22:52:58 gonzo-1 systemd[1]: Started OpenSSH Daemon.
Apr 08 22:53:07 gonzo-1 sshd[1843]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33952 ssh2
Apr 08 22:53:11 gonzo-1 sshd[1845]: Accepted keyboard-interactive/pam for root f
rom 10.160.0.147 port 33958 ssh2
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ecdsa_key
Apr 08 23:22:19 gonzo-1 sshd[4251]: error: Could not load host key: /etc
/ssh/ssh_host_ed25519_key


gonzo-1:~ # ll /etc/ssh/
total 168
drwxr-xr-x 1 root root    380 Apr  8 22:55 bak
-rw-r--r-- 1 root root   2375 Apr  7 00:35 ldap.conf
-rw------- 1 root root 132839 Apr  8 22:55 moduli
-rw-r--r-- 1 root root   2365 Apr  8 22:55 ssh_config
-rw------- 1 root root    668 Apr  8 22:55 ssh_host_dsa_key
-rw-r--r-- 1 root root    602 Apr  8 22:55 ssh_host_dsa_key.pub
-rw------- 1 root root    527 Apr  8 22:55 ssh_host_key
-rw-r--r-- 1 root root    331 Apr  8 22:55 ssh_host_key.pub
-rw------- 1 root root    887 Apr  8 22:55 ssh_host_rsa_key
-rw-r--r-- 1 root root    222 Apr  8 22:55 ssh_host_rsa_key.pub
-rw-r----- 1 root root   3420 Apr  7 00:35 sshd_config

Reproducible

Not fail for every run, about 1/3 of logins failed. And happened on different SUT. //julie added.
Test fails since 210.1
in scenario sle-15-SP1-Installer-DVD-x86_64-gi-guest_sles12sp4-on-host-developing-kvm@64bit-ipmi

Expected result

Last good 208.1

Further information

Latest in sle-15-SP1-Installer-DVD-x86_64-gi-guest_sles12sp4-on-host-developing-kvm@64bit-ipmi

Actions
Copy link
 #6

Updated by nicksinger about 5 years ago

I fail to see why sshd is blamed here. The login-prompt clearly appears, openQA just misses to enter the credentials at the right time. This race might be influenced by several factors but in the end we should get rid of the race condition here. E.g. on s390 there was some work to switch over to ssh keys.

Actions
Copy link
 #7

Updated by Julie_CAO about 5 years ago

The ssh started with errors and we are unable to ssh login these SUT. After we access the SUT via ipmi console and restart sshd, the error disappeared and we are able to login the SUT via ssh. So the issue should the sshd start with error.

Actions
Copy link
 #8

Updated by Julie_CAO about 5 years ago

  • Subject changed from [qam][ipmi] sshd failed to be start on openQA SUT with ipmi backend to [qam][ipmi] unable to login openQA SUT via ssh with ipmi backend
Actions
Copy link
 #10

Updated by Julie_CAO about 5 years ago

found it should be a sles15sp1 bug rather than an openQA infra issue. Just open a bug against sles15sp1 rc2:
Bug 1132187 - [ssh][build212.1]Unable to login via ssh due to "error: Could not load host key" https://bugzilla.suse.com/show_bug.cgi?id=1132187

Actions
Copy link
 #11

Updated by tjyrinki_suse over 4 years ago

  • Status changed from New to Resolved

This was deemed to be a problem in SLE instead of openQA infra. Please reopen if it would be found to be otherwise. The bugzilla bug seems untouched since April.

Actions
Copy link

Also available in: Atom PDF