Project

General

Profile

action #54275

[kernel]kotd][ltp][s390x] test fails in install_ltp - failure after install_kotd, console issue

Added by pcervinka about 2 years ago. Updated 11 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Target version:
QE Kernel - QE Kernel Done
Start date:
2019-07-15
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

openQA test in scenario sle-12-SP5-Server-DVD-s390x-install_ltp+sle+Server-DVD+KOTD@s390x-kvm-sle12 fails in
install_ltp

Test suite description

maintainer: pcervinka, pvorel

Reproducible

Fails since (at least) Build 0222

Expected result

Last good: (unknown) (or more recent)

Further details

Always latest result in this scenario: latest

Test probably needs to schedule again bootloader_zkvm after reboot in install_kotd to properly setup test scenario.


Related issues

Related to openQA Tests - action #54260: [kernel][s390x] Occasional failures due timeoutRejected2019-07-15

History

#1 Updated by pcervinka about 2 years ago

  • Subject changed from [kernel]kotd][ltp][s390x] test fails in install_ltp - failure after install_kotd, bootloader_zkvm needed to [kernel]kotd][ltp][s390x] test fails in install_ltp - failure after install_kotd, console issue

Experimented with reconnect management console, it helps standard console, but not much to serial terminal.
When is SUT rebooted, it is not possible reconnect by virsh console as session already exist.

[2019-07-15T12:02:53.080 CEST] [debug] Command executed: '[ -f '/tmp/serial_terminal.txt.HSwG7zmFqv2QmkFK' ] && rm -v /tmp/serial_terminal.txt.HSwG7zmFqv2QmkFK', ret=0
[2019-07-15T12:02:53.080 CEST] [debug] Backend process died, backend errors are reported below in the following lines:
problem with virsh: cmd: 'virsh console openQA-SUT-4 console1', output of script wrapper: 'Script started on Mon Jul 15 12:02:51 2019
Connected to domain openQA-SUT-4
Escape character is ^]
error: operation failed: Active console session exists for this domain

#2 Updated by pcervinka about 2 years ago

I will try --force to virsh console but not sure about side effects.

#3 Updated by pvorel about 2 years ago

I guess you're talking about https://openqa.suse.de/tests/3062903

Can't we add a hook into power_action() to disable consoles::sshVirtshSUT on reboot?
I'd try avoid --force. It should be safe, but it can lead to really cryptic errors.

#4 Updated by pcervinka about 2 years ago

Yes, I'm afraid --force could lead something, which we can't predict at the moment. Will try to look on your hint about disabling console.

#5 Updated by pcervinka about 2 years ago

  • Status changed from New to In Progress

#6 Updated by pcervinka about 2 years ago

The problem is more complex than it looks. It fails on attaching console1 in command virsh console openQA-SUT-4 console1. This console should be connected all the time and it doesn't count with system reboot at all, because should be initialized only once per test.

Console init is done in function open_serial_console_via_ssh in `svirt.pm'. Description of the function says:

Opens SSH connection to grab serial terminal log
(using consoles::serial_screen, saved into serial_terminal.txt).

This method is not supposed to be called twice for test run due logging
into file.

If I understand, we don't have any option to cleanly stop and reconnect console1 in the same job. Adding --force to $cmd = "virsh console $name $devname$port"; didn't work.

#7 Updated by pcervinka about 2 years ago

Workaround for KOTD installation could be in use of root-console for svirt backend. pvorel what do you think? I think this is the quickest solution we can have now.

#8 Updated by pcervinka about 2 years ago

PR created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7974, verification on OSD on all backends in progress.

#9 Updated by pcervinka about 2 years ago

I realized that normal LTP installation in install_ltp+sle+Server-DVD is done with SERIAL_CONSOLE=0, so probably ma workaround is not needed.
Restarted KOTD install job on osd https://openqa.suse.de/tests/3109961 and added SERIAl_CONSOLE=0.

#10 Updated by pvorel about 2 years ago

SERIAl_CONSOLE=0 is just a temporary workaround, we plan to remove it once we fix svirt #54260.

#11 Updated by pcervinka about 2 years ago

  • Status changed from In Progress to Feedback

Added SERIAL_CONSOLE=0 to install_ltp+sle+Server-DVD+KOTD to have LTP with KOTD working. As we agreed, I will create new poo for better fix, not just related to KOTD.

#13 Updated by pcervinka about 2 years ago

  • Related to action #54260: [kernel][s390x] Occasional failures due timeout added

#14 Updated by pcervinka about 2 years ago

KOTD job on svirt is OK now. Let's wait for next build.

#15 Updated by pcervinka about 2 years ago

  • Status changed from Feedback to Resolved

#16 Updated by jlausuch about 2 years ago

  • Status changed from Resolved to Closed
  • Target version changed from 445 to 457

#17 Updated by pcervinka 11 months ago

  • Target version changed from 457 to QE Kernel Done

Also available in: Atom PDF