action #19742
closed[tools][virtualization][new ipmi backend] The root-ssh console can not work when the host kernel upgraded.
0%
Description
When the host is upgraded via a command line executed on root-ssh console, after this command finish(or even during this command), the root-ssh console window gets black and can not get expected serial output.
Job link:
prj2_host_upgrade_sles12sp1_to_sles12sp3_kvm: https://openqa.suse.de/tests/991682 and
prj2_host_upgrade_sles12sp1_to_sles12sp3_xen: https://openqa.suse.de/tests/992135.
Is there any way to let this test step work, except doing it on sut console?
Updated by coolo almost 7 years ago
is your test doing a reboot of the SUT? If so, you will need to reconnect the ssh
Updated by xlai almost 7 years ago
No, no reboot happened.
The problem maybe that after upgrade, the kernel changed. So the fifo device -sshserial may not work any more. See in kvm job https://openqa.suse.de/tests/991682/file/autoinst-log.txt - error: file /boot/System.map-3.12.74-60.64.40-default: No such file or directory. In xen fail job, connection just broke in the middle of the upgrade script running.
If this guess is true, then using root-ssh console is not suitable for host upgrade test. But if switching back to ipmi console, the serial console is very very unstable for the step with so much serial output test. Which to prefer then?
kvm job key error log:¶
20:48:03.7944 29971 <<< testapi::type_string(string='(/usr/share/qa/tools/test-VH-Upgrade-std-kvm-sles12sp1-sles12sp3-run 02; echo CMD_FINISHED-34507) 2>&1 | tee -a /dev/sshserial
', max_interval=250, wait_screen_changes=0)
20:48:29.7747 Debug: /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/virt_autotest/host_upgrade_step2_run.pm:29 called virt_autotest_base::run_test
20:48:29.7749 29971 <<< testapi::wait_serial(timeout=12600, regexp='CMD_FINISHED-34507')
Initializing test run for control file test-VH-Upgrade-std-kvm-sles12sp1-sles12sp3-02.tcf...
Current time: Fri Jun 9 16:48:26 EDT 2017
*[37m[1m Test in progress [0m*
vhPrepAndUpdate ... ... PASSED (42m31s)
*[37m[1m Test run complete [0m*
Current time: Fri Jun 9 17:30:57 EDT 2017
Exiting test run..
error: file /boot/System.map-3.12.74-60.64.40-default: No such file or directory
00:18:31.4532 29971 >>> testapi::wait_serial: CMD_FINISHED-34507: fail
00:18:31.8712 29971 # Test died: Timeout due to cmd run :[/usr/share/qa/tools/test-VH-Upgrade-std-kvm-sles12sp1-sles12sp3-run 02]¶
xen job key error log: No error message given, connection just broke
13:06:04.9851 9053 <<< testapi::type_string(string='(/usr/share/qa/tools/test-VH-Upgrade-std-xen-sles12sp1-sles12sp3-run 02; echo CMD_FINISHED-821923) 2>&1 | tee -a /dev/sshserial
', max_interval=250, wait_screen_changes=0)
13:06:31.1628 Debug: /var/lib/openqa/cache/openqa.suse.de/tests/sle/tests/virt_autotest/host_upgrade_step2_run.pm:29 called virt_autotest_base::run_test
13:06:31.1630 9053 <<< testapi::wait_serial(timeout=12600, regexp='CMD_FINISHED-821923')
Initializing test run for control file test-VH-Upgrade-std-xen-sles12sp1-sles12sp3-02.tcf...
Current time: Sat Jun 10 09:06:29 EDT 2017
*[37m[1m Test in progress [0m*
16:36:32.7603 9053 >>> testapi::wait_serial: CMD_FINISHED-821923: fail
16:36:33.1550 9053 # Test died: Timeout due to cmd run :[/usr/share/qa/tools/test-VH-Upgrade-std-xen-sles12sp1-sles12sp3-run 02]
Updated by coolo almost 7 years ago
- Assignee changed from coolo to xlai
There is nothing the backend can do about it. The next best thing you can do is running the ctcs in background and have a loop awaiting its end. And if this loop detects /dev/sshserial to be gone, do something about it.
Updated by xlai almost 7 years ago
The issue disappears in sles12sp3 gmc, see https://openqa.suse.de/tests/1059105 and https://openqa.suse.de/tests/1058389.
Will follow the issue if it happens again in future.
Updated by okurz over 6 years ago
- Subject changed from [tools][new ipmi backend] The root-ssh console can not work when the host kernel upgraded. to [tools][virtualization][new ipmi backend] The root-ssh console can not work when the host kernel upgraded.
- Category set to Bugs in existing tests