action #153111
open[openQA][console][ipmi][sol] xterm process quits and ipmi sol console crashes
0%
Description
Observation¶
According to http://open.qa/api/testapi/, reset_consoles can be used to ensure next select_console really activates a console. But it seems that this is not true for sol console:
select_console 'sol', await_console => 0;
reset_consoles;
select_console 'sol', await_console => 0;
The sol console can not be activated anymore after operations shown in above code block. Screenshot captured by openQA looks like as below:
The machine on which this openQA instance run has been given a fresh reinstall. It runs new Leap 15.5 system. And I installed openQA on this machine by using:
curl -s https://raw.githubusercontent.com/os-autoinst/openQA/master/script/openqa-bootstrap | bash -x
Steps to reproduce¶
- select_console 'sol', await_console => 0;
- reset_consoles;
- select_console 'sol', await_console => 0;
Impact¶
Can not use reset_consoles with select_console.
Problem¶
Maybe the sol console is not cleaned up completely by reset_consoles.
Suggestions¶
- Clarify whether reset_consoles work well with sol console
- What reset_consoles does to sol console
Workaround¶
n/a
Files
| inactive_ipmi_sol_console.png (16 KB) inactive_ipmi_sol_console.png | |||
| inactive_ipmi_sol_console.png (16 KB) inactive_ipmi_sol_console.png | |||
| autoints-log.txt (402 KB) autoints-log.txt |
Updated by okurz 11 months ago
- Due date set to 2024-01-18
- Category set to Support
- Status changed from New to Feedback
- Assignee set to okurz
- Target version set to Ready
reset_consoles only resets the transient consoles, i.e. the ones that need to be reinitialized after resetting/rebooting the SUT. The "sol" is persistent over reboots hence reset_consoles shouldn't have an effect. We might be able to help you if you can provide more details about the specific openQA job you want to run. Can you provide the autoinst-log.txt file and/or direct access to the openQA instance so that we could take a look?
Updated by waynechen55 11 months ago · Edited
- File inactive_ipmi_sol_console.png inactive_ipmi_sol_console.png added
- File autoints-log.txt autoints-log.txt added
okurz wrote in #note-3:
reset_consolesonly resets the transient consoles, i.e. the ones that need to be reinitialized after resetting/rebooting the SUT. The "sol" is persistent over reboots hencereset_consolesshouldn't have an effect. We might be able to help you if you can provide more details about the specific openQA job you want to run. Can you provide the autoinst-log.txt file and/or direct access to the openQA instance so that we could take a look?
* The test run failed at select_console('sol') after reset_consoles. It called process_reboot in lib/transactional.pm, then failed at:¶
if (is_ipmi) {
reset_consoles;
select_console 'sol', await_console => 0;
send_key 'ret';
}
The process_reboot is as below:
sub process_reboot {
my (%args) = @_;
$args{trigger} //= 0;
$args{automated_rollback} //= 0;
$args{expected_grub} //= 1;
$args{expected_passphrase} //= 0;
if (is_public_cloud) {
my $instance = publiccloud::instances::get_instance();
$instance->softreboot(); # Handled re-establishing of the required ssh tunnel and consoles
return;
}
# Switch to root-console as we need VNC to check for grub and for login prompt
my $prev_console = current_console();
select_console 'root-console', await_console => 0;
handle_first_grub if ($args{automated_rollback});
if (!is_s390x && (is_microos || is_sle_micro('<6.0'))) {
microos_reboot $args{trigger};
record_kernel_audit_messages();
} elsif (is_backend_s390x) {
prepare_system_shutdown;
enter_cmd "reboot";
opensusebasetest::wait_boot(opensusebasetest->new(), bootloader_time => 300);
record_kernel_audit_messages();
} else {
power_action('reboot', observe => !$args{trigger}, keepconsole => 1, ipmisol => 1);
if (is_s390x || is_pvm) {
reconnect_mgmt_console(timeout => 500) unless $args{automated_rollback};
}
if (!is_s390x && $args{expected_grub}) {
if (is_aarch64 && check_screen('tianocore-mainmenu', 30)) {
# Use firmware boot manager of aarch64 to boot HDD, when needed
opensusebasetest::handle_uefi_boot_disk_workaround();
}
if ($args{expected_passphrase}) {
unlock_if_encrypted();
}
# Replace by wait_boot if possible
if (is_ipmi) {
reset_consoles;
select_console 'sol', await_console => 0;
send_key 'ret';
}
assert_screen 'grub2', 150;
wait_screen_change { send_key 'ret' };
}
assert_screen 'linux-login', 200;
# Login & clear login needle
select_console 'root-console';
record_kernel_audit_messages();
assert_script_run 'clear';
}
# Switch to the previous console
select_console $prev_console;
}
* select_console('sol') in this case becomes:¶
* Attached autoinst-log.txt for this case.¶
Updated by okurz 11 months ago
I asked in a broader scope in https://suse.slack.com/archives/C02CANHLANP/p1704813309287429
And who can help for https://progress.opensuse.org/issues/153111 ?
@waynechen55 can you state what is your reference? Do you see this test code running fine in other environments? Or do you have a reference job that is not using the reset_consoles?
Updated by waynechen55 10 months ago
okurz wrote in #note-5:
I asked in a broader scope in https://suse.slack.com/archives/C02CANHLANP/p1704813309287429
And who can help for https://progress.opensuse.org/issues/153111 ?
@waynechen55 can you state what is your reference? Do you see this test code running fine in other environments? Or do you have a reference job that is not using the
reset_consoles?
I will have a deeper look later.
Updated by okurz 10 months ago
- Due date deleted (
2024-01-18) - Status changed from Feedback to New
- Assignee deleted (
okurz) - Priority changed from Normal to Low
- Target version changed from Ready to future
@waynechen55 if you have more information to make the issue easier to reproduce we might have a chance to help otherwise I see no good point to improve so I will unassign for now and keep this in "future".
Updated by waynechen55 10 months ago · Edited
okurz wrote in #note-7:
@waynechen55 if you have more information to make the issue easier to reproduce we might have a chance to help otherwise I see no good point to improve so I will unassign for now and keep this in "future".
I encountered this issue again. The only thing I know for sure is that there is no process "xterm" judging by ps axu when this issue occurs. It seems that the console crashes somehow. In normal case, there should be a process as below:
localhost:/var/lib/libvirt/images/temp # ps axu | grep -i xterm
_openqa+ 23195 0.3 0.0 54620 11612 ? S 12:34 0:00 xterm -fn eurlatgr -fullscreen +sb -b 0 -bc -uc -bcf 200 -bcn 200 +bdc -xrm xterm*boldMode: false -xrm xterm*foreground: rgb:aa/aa/aa -xrm xterm*background: rgb:00/00/00 -xrm xterm*color0: rgb:00/00/00 -xrm xterm*color1: rgb:aa/00/00 -xrm xterm*color2: rgb:00/aa/00 -xrm xterm*color3: rgb:aa/55/00 -xrm xterm*color4: rgb:00/00/aa -xrm xterm*color5: rgb:aa/00/aa -xrm xterm*color6: rgb:00/aa/aa -xrm xterm*color7: rgb:aa/aa/aa -xrm xterm*color8: rgb:55/55/55 -xrm xterm*color9: rgb:ff/55/55 -xrm xterm*color10: rgb:55/ff/55 -xrm xterm*color11: rgb:ff/ff/55 -xrm xterm*color12: rgb:55/55/ff -xrm xterm*color13: rgb:ff/55/ff -xrm xterm*color14: rgb:55/ff/ff -xrm xterm*color15: rgb:ff/ff/ff -title ipmitool:sol -e bash -c TERM=xterm ipmitool -I lanplus -H vh017-ipmi.qa2.suse.asia -U waynechen -P xxxxxxxxxx sol activate
The os-autoinst and worker runs on:
localhost:/var/lib/libvirt/images/temp # cat /etc/os-release
NAME="openSUSE Leap"
VERSION="15.5"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.5"
PRETTY_NAME="openSUSE Leap 15.5"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.5"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
LOGO="distributor-logo-Leap"
And openQA software:
localhost:/var/lib/libvirt/images/temp # zypper info openQA*
Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata ..............................................................................................[done]
Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ...................................................................................................[done]
Loading repository data...
Reading installed packages...
Information for package openQA:
-------------------------------
Repository : devel:openQA
Name : openQA
Version : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch : x86_64
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 14.2 MiB
Installed : Yes
Status : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL : http://os-autoinst.github.io/openQA/
Summary : The openQA web-frontend, scheduler and tools
Description :
openQA is a testing framework that allows you to test GUI applications on one
hand and bootloader and kernel on the other. In both cases, it is difficult to
script tests and verify the output. Output can be a popup window or it can be
an error in early boot even before init is executed.
openQA is an automated test tool that makes it possible to test the whole
installation process of an operating system. It uses virtual machines to
reproduce the process, check the output (both serial console and screen) in
every step and send the necessary keystrokes and commands to proceed to the
next. openQA can check whether the system can be installed, whether it works
properly in 'live' mode, whether applications work or whether the system
responds as expected to different installation options and commands.
Even more importantly, openQA can run several combinations of tests for every
revision of the operating system, reporting the errors detected for each
combination of hardware configuration, installation options and variant of the
operating system.
Information for package openQA-bootstrap:
-----------------------------------------
Repository : devel:openQA
Name : openQA-bootstrap
Version : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch : x86_64
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 9.8 KiB
Installed : Yes
Status : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL : http://os-autoinst.github.io/openQA/
Summary : Automated openQA setup
Description :
This can automatically setup openQA - either directly on your system
or within a systemd-nspawn container.
Information for package openQA-client:
--------------------------------------
Repository : devel:openQA
Name : openQA-client
Version : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch : x86_64
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 53.8 KiB
Installed : Yes
Status : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL : http://os-autoinst.github.io/openQA/
Summary : Client tools for remote openQA management
Description :
Tools and support files for openQA client script. Client script is
a convenient helper for interacting with openQA webui REST API.
Information for package openQA-common:
--------------------------------------
Repository : devel:openQA
Name : openQA-common
Version : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch : x86_64
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 379.6 KiB
Installed : Yes
Status : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL : http://os-autoinst.github.io/openQA/
Summary : The openQA common tools for web-frontend and workers
Description :
This package contain shared resources for openQA web-frontend and
openQA workers.
Information for package openQA-local-db:
----------------------------------------
Repository : devel:openQA
Name : openQA-local-db
Version : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch : x86_64
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 617 B
Installed : Yes
Status : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL : http://os-autoinst.github.io/openQA/
Summary : Helper package to ease setup of postgresql DB
Description :
You only need this package if you have a local postgresql server
next to the webui.
Information for package openQA-single-instance:
-----------------------------------------------
Repository : devel:openQA
Name : openQA-single-instance
Version : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch : x86_64
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 0 B
Installed : Yes
Status : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL : http://os-autoinst.github.io/openQA/
Summary : Convenience package for a single-instance setup using apache proxy
Description :
Use this package to setup a local instance with all services provided together.
Information for package openQA-worker:
--------------------------------------
Repository : devel:openQA
Name : openQA-worker
Version : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch : x86_64
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 154.3 KiB
Installed : Yes
Status : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL : http://os-autoinst.github.io/openQA/
Summary : The openQA worker
Description :
The openQA worker manages test engine (provided by os-autoinst package).
Updated by waynechen55 10 months ago
Information for package xterm:¶
Repository : Update repository with updates from SUSE Linux Enterprise 15
Name : xterm
Version : 330-150200.11.12.1
Arch : x86_64
Vendor : SUSE LLC https://www.suse.com/
Installed Size : 38.0 KiB
Installed : Yes (automatically)
Status : up-to-date
Source package : xterm-330-150200.11.12.1.src
Upstream URL : http://invisible-island.net/xterm/
Summary : The basic X terminal program
Description :
This package contains the basic X.Org terminal program desktop launcher.
waynechen:/var/lib/openqa/share/tests/opensuse # zypper info xterm-console
Loading repository data...
Reading installed packages...
Information for package xterm-console:¶
Repository : devel:openQA
Name : xterm-console
Version : 1.1-lp155.17.1
Arch : noarch
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 911.9 KiB
Installed : Yes
Status : up-to-date
Source package : xterm-console-1.1-lp155.17.1.src
Upstream URL : https://github.com/os-autoinst/xterm-console
Summary : A Linux vt console look-alike xterm wrapper
Description :
This package contains the basic X.Org terminal program.
Updated by waynechen55 10 months ago
- Subject changed from [openQA][console][ipmi][sol] Can not select_console('sol') after reset_consoles to [openQA][console][ipmi][sol] xterm process quits and ipmi sol console crashes
There is a similar failure on OSD, please refer to https://openqa.suse.de/tests/13402294#step/ipxe_install/5. @okurz
Updated by okurz 3 months ago
This ticket was mentioned in https://sd.suse.com/servicedesk/customer/portal/1/SD-165090 so let me provide an update here
okurz wrote in #note-7:
@waynechen55 if you have more information to make the issue easier to reproduce we might have a chance to help otherwise I see no good point to improve so I will unassign for now and keep this in "future".
This is still the current state. Without a better way to reproduce the easier it is very unlikely we will be able to provide any more help.
Updated by okurz 2 months ago
- Related to action #165611: [openQA][infra][sut][aarch64] Power supply failure on squiddlydiddly arm64 machine size:M added