Project

General

Profile

Actions

action #153111

open

[openQA][console][ipmi][sol] xterm process quits and ipmi sol console crashes

Added by waynechen55 11 months ago. Updated 4 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Support
Target version:
QA (public, currently private due to #173521) - future
Start date:
2024-01-04
Due date:
% Done:

0%

Estimated time:

Description

Observation

According to http://open.qa/api/testapi/, reset_consoles can be used to ensure next select_console really activates a console. But it seems that this is not true for sol console:

select_console 'sol', await_console => 0;
reset_consoles;
select_console 'sol', await_console => 0;

The sol console can not be activated anymore after operations shown in above code block. Screenshot captured by openQA looks like as below:

The machine on which this openQA instance run has been given a fresh reinstall. It runs new Leap 15.5 system. And I installed openQA on this machine by using:

curl -s https://raw.githubusercontent.com/os-autoinst/openQA/master/script/openqa-bootstrap | bash -x

Steps to reproduce

  • select_console 'sol', await_console => 0;
  • reset_consoles;
  • select_console 'sol', await_console => 0;

Impact

Can not use reset_consoles with select_console.

Problem

Maybe the sol console is not cleaned up completely by reset_consoles.

Suggestions

  • Clarify whether reset_consoles work well with sol console
  • What reset_consoles does to sol console

Workaround

n/a


Files

inactive_ipmi_sol_console.png (16 KB) inactive_ipmi_sol_console.png waynechen55, 2024-01-04 13:07
inactive_ipmi_sol_console.png (16 KB) inactive_ipmi_sol_console.png waynechen55, 2024-01-05 01:22
autoints-log.txt (402 KB) autoints-log.txt waynechen55, 2024-01-05 01:22

Related issues 1 (0 open1 closed)

Related to openQA Infrastructure (public) - action #165611: [openQA][infra][sut][aarch64] Power supply failure on squiddlydiddly arm64 machine size:MResolvedokurz2024-08-22

Actions
Actions #1

Updated by waynechen55 11 months ago

  • Description updated (diff)
Actions #2

Updated by waynechen55 11 months ago

  • Description updated (diff)
Actions #3

Updated by okurz 11 months ago

  • Due date set to 2024-01-18
  • Category set to Support
  • Status changed from New to Feedback
  • Assignee set to okurz
  • Target version set to Ready

reset_consoles only resets the transient consoles, i.e. the ones that need to be reinitialized after resetting/rebooting the SUT. The "sol" is persistent over reboots hence reset_consoles shouldn't have an effect. We might be able to help you if you can provide more details about the specific openQA job you want to run. Can you provide the autoinst-log.txt file and/or direct access to the openQA instance so that we could take a look?

Updated by waynechen55 11 months ago · Edited

okurz wrote in #note-3:

reset_consoles only resets the transient consoles, i.e. the ones that need to be reinitialized after resetting/rebooting the SUT. The "sol" is persistent over reboots hence reset_consoles shouldn't have an effect. We might be able to help you if you can provide more details about the specific openQA job you want to run. Can you provide the autoinst-log.txt file and/or direct access to the openQA instance so that we could take a look?

* The test run failed at select_console('sol') after reset_consoles. It called process_reboot in lib/transactional.pm, then failed at:

            if (is_ipmi) {
                reset_consoles;
                select_console 'sol', await_console => 0;
                send_key 'ret';
            }

The process_reboot is as below:

sub process_reboot {
    my (%args) = @_;
    $args{trigger} //= 0;
    $args{automated_rollback} //= 0;
    $args{expected_grub} //= 1;
    $args{expected_passphrase} //= 0;

    if (is_public_cloud) {
        my $instance = publiccloud::instances::get_instance();
        $instance->softreboot();    # Handled re-establishing of the required ssh tunnel and consoles
        return;
    }

    # Switch to root-console as we need VNC to check for grub and for login prompt
    my $prev_console = current_console();
    select_console 'root-console', await_console => 0;

    handle_first_grub if ($args{automated_rollback});

    if (!is_s390x && (is_microos || is_sle_micro('<6.0'))) {
        microos_reboot $args{trigger};
        record_kernel_audit_messages();
    } elsif (is_backend_s390x) {
        prepare_system_shutdown;
        enter_cmd "reboot";
        opensusebasetest::wait_boot(opensusebasetest->new(), bootloader_time => 300);
        record_kernel_audit_messages();
    } else {
        power_action('reboot', observe => !$args{trigger}, keepconsole => 1, ipmisol => 1);
        if (is_s390x || is_pvm) {
            reconnect_mgmt_console(timeout => 500) unless $args{automated_rollback};
        }
        if (!is_s390x && $args{expected_grub}) {
            if (is_aarch64 && check_screen('tianocore-mainmenu', 30)) {
                # Use firmware boot manager of aarch64 to boot HDD, when needed
                opensusebasetest::handle_uefi_boot_disk_workaround();
            }
            if ($args{expected_passphrase}) {
                unlock_if_encrypted();
            }
            # Replace by wait_boot if possible
            if (is_ipmi) {
                reset_consoles;
                select_console 'sol', await_console => 0;
                send_key 'ret';
            }
            assert_screen 'grub2', 150;
            wait_screen_change { send_key 'ret' };
        }
        assert_screen 'linux-login', 200;

        # Login & clear login needle
        select_console 'root-console';
        record_kernel_audit_messages();
        assert_script_run 'clear';
    }

    # Switch to the previous console
    select_console $prev_console;
}

* select_console('sol') in this case becomes:

* Attached autoinst-log.txt for this case.

Actions #5

Updated by okurz 11 months ago

I asked in a broader scope in https://suse.slack.com/archives/C02CANHLANP/p1704813309287429

And who can help for https://progress.opensuse.org/issues/153111 ?

@waynechen55 can you state what is your reference? Do you see this test code running fine in other environments? Or do you have a reference job that is not using the reset_consoles?

Actions #6

Updated by waynechen55 11 months ago

okurz wrote in #note-5:

I asked in a broader scope in https://suse.slack.com/archives/C02CANHLANP/p1704813309287429

And who can help for https://progress.opensuse.org/issues/153111 ?

@waynechen55 can you state what is your reference? Do you see this test code running fine in other environments? Or do you have a reference job that is not using the reset_consoles?

I will have a deeper look later.

Actions #7

Updated by okurz 11 months ago

  • Due date deleted (2024-01-18)
  • Status changed from Feedback to New
  • Assignee deleted (okurz)
  • Priority changed from Normal to Low
  • Target version changed from Ready to future

@waynechen55 if you have more information to make the issue easier to reproduce we might have a chance to help otherwise I see no good point to improve so I will unassign for now and keep this in "future".

Actions #8

Updated by waynechen55 11 months ago · Edited

okurz wrote in #note-7:

@waynechen55 if you have more information to make the issue easier to reproduce we might have a chance to help otherwise I see no good point to improve so I will unassign for now and keep this in "future".

I encountered this issue again. The only thing I know for sure is that there is no process "xterm" judging by ps axu when this issue occurs. It seems that the console crashes somehow. In normal case, there should be a process as below:

localhost:/var/lib/libvirt/images/temp # ps axu | grep -i xterm
_openqa+ 23195  0.3  0.0  54620 11612 ?        S    12:34   0:00 xterm -fn eurlatgr -fullscreen +sb -b 0 -bc -uc -bcf 200 -bcn 200 +bdc -xrm xterm*boldMode: false -xrm xterm*foreground: rgb:aa/aa/aa -xrm xterm*background: rgb:00/00/00 -xrm xterm*color0: rgb:00/00/00 -xrm xterm*color1: rgb:aa/00/00 -xrm xterm*color2: rgb:00/aa/00 -xrm xterm*color3: rgb:aa/55/00 -xrm xterm*color4: rgb:00/00/aa -xrm xterm*color5: rgb:aa/00/aa -xrm xterm*color6: rgb:00/aa/aa -xrm xterm*color7: rgb:aa/aa/aa -xrm xterm*color8: rgb:55/55/55 -xrm xterm*color9: rgb:ff/55/55 -xrm xterm*color10: rgb:55/ff/55 -xrm xterm*color11: rgb:ff/ff/55 -xrm xterm*color12: rgb:55/55/ff -xrm xterm*color13: rgb:ff/55/ff -xrm xterm*color14: rgb:55/ff/ff -xrm xterm*color15: rgb:ff/ff/ff -title ipmitool:sol -e bash -c TERM=xterm ipmitool -I lanplus -H vh017-ipmi.qa2.suse.asia -U waynechen -P xxxxxxxxxx sol activate

The os-autoinst and worker runs on:

localhost:/var/lib/libvirt/images/temp # cat /etc/os-release 
NAME="openSUSE Leap"
VERSION="15.5"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.5"
PRETTY_NAME="openSUSE Leap 15.5"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.5"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap"
LOGO="distributor-logo-Leap"

And openQA software:

localhost:/var/lib/libvirt/images/temp # zypper info openQA*
Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata ..............................................................................................[done]
Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ...................................................................................................[done]
Loading repository data...
Reading installed packages...


Information for package openQA:
-------------------------------
Repository     : devel:openQA
Name           : openQA
Version        : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:openQA
Installed Size : 14.2 MiB
Installed      : Yes
Status         : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL   : http://os-autoinst.github.io/openQA/
Summary        : The openQA web-frontend, scheduler and tools
Description    : 
    openQA is a testing framework that allows you to test GUI applications on one
    hand and bootloader and kernel on the other. In both cases, it is difficult to
    script tests and verify the output. Output can be a popup window or it can be
    an error in early boot even before init is executed.

    openQA is an automated test tool that makes it possible to test the whole
    installation process of an operating system. It uses virtual machines to
    reproduce the process, check the output (both serial console and screen) in
    every step and send the necessary keystrokes and commands to proceed to the
    next. openQA can check whether the system can be installed, whether it works
    properly in 'live' mode, whether applications work or whether the system
    responds as expected to different installation options and commands.

    Even more importantly, openQA can run several combinations of tests for every
    revision of the operating system, reporting the errors detected for each
    combination of hardware configuration, installation options and variant of the
    operating system.


Information for package openQA-bootstrap:
-----------------------------------------
Repository     : devel:openQA
Name           : openQA-bootstrap
Version        : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:openQA
Installed Size : 9.8 KiB
Installed      : Yes
Status         : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL   : http://os-autoinst.github.io/openQA/
Summary        : Automated openQA setup
Description    : 
    This can automatically setup openQA - either directly on your system
    or within a systemd-nspawn container.


Information for package openQA-client:
--------------------------------------
Repository     : devel:openQA
Name           : openQA-client
Version        : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:openQA
Installed Size : 53.8 KiB
Installed      : Yes
Status         : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL   : http://os-autoinst.github.io/openQA/
Summary        : Client tools for remote openQA management
Description    : 
    Tools and support files for openQA client script. Client script is
    a convenient helper for interacting with openQA webui REST API.


Information for package openQA-common:
--------------------------------------
Repository     : devel:openQA
Name           : openQA-common
Version        : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:openQA
Installed Size : 379.6 KiB
Installed      : Yes
Status         : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL   : http://os-autoinst.github.io/openQA/
Summary        : The openQA common tools for web-frontend and workers
Description    : 
    This package contain shared resources for openQA web-frontend and
    openQA workers.


Information for package openQA-local-db:
----------------------------------------
Repository     : devel:openQA
Name           : openQA-local-db
Version        : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:openQA
Installed Size : 617 B
Installed      : Yes
Status         : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL   : http://os-autoinst.github.io/openQA/
Summary        : Helper package to ease setup of postgresql DB
Description    : 
    You only need this package if you have a local postgresql server
    next to the webui.


Information for package openQA-single-instance:
-----------------------------------------------
Repository     : devel:openQA
Name           : openQA-single-instance
Version        : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:openQA
Installed Size : 0 B
Installed      : Yes
Status         : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL   : http://os-autoinst.github.io/openQA/
Summary        : Convenience package for a single-instance setup using apache proxy
Description    : 
    Use this package to setup a local instance with all services provided together.


Information for package openQA-worker:
--------------------------------------
Repository     : devel:openQA
Name           : openQA-worker
Version        : 4.6.1704466891.4d4e5b7-lp155.6262.1
Arch           : x86_64
Vendor         : obs://build.opensuse.org/devel:openQA
Installed Size : 154.3 KiB
Installed      : Yes
Status         : out-of-date (version 4.6.1704307071.9c8a390-lp155.6258.1 installed)
Source package : openQA-4.6.1704466891.4d4e5b7-lp155.6262.1.src
Upstream URL   : http://os-autoinst.github.io/openQA/
Summary        : The openQA worker
Description    : 
    The openQA worker manages test engine (provided by os-autoinst package).

@okurz

Actions #9

Updated by waynechen55 10 months ago

Information for package xterm:

Repository : Update repository with updates from SUSE Linux Enterprise 15
Name : xterm
Version : 330-150200.11.12.1
Arch : x86_64
Vendor : SUSE LLC https://www.suse.com/
Installed Size : 38.0 KiB
Installed : Yes (automatically)
Status : up-to-date
Source package : xterm-330-150200.11.12.1.src
Upstream URL : http://invisible-island.net/xterm/
Summary : The basic X terminal program
Description :

This package contains the basic X.Org terminal program desktop launcher.

waynechen:/var/lib/openqa/share/tests/opensuse # zypper info xterm-console
Loading repository data...
Reading installed packages...

Information for package xterm-console:

Repository : devel:openQA
Name : xterm-console
Version : 1.1-lp155.17.1
Arch : noarch
Vendor : obs://build.opensuse.org/devel:openQA
Installed Size : 911.9 KiB
Installed : Yes
Status : up-to-date
Source package : xterm-console-1.1-lp155.17.1.src
Upstream URL : https://github.com/os-autoinst/xterm-console
Summary : A Linux vt console look-alike xterm wrapper
Description :
This package contains the basic X.Org terminal program.

Actions #10

Updated by waynechen55 10 months ago

  • Subject changed from [openQA][console][ipmi][sol] Can not select_console('sol') after reset_consoles to [openQA][console][ipmi][sol] xterm process quits and ipmi sol console crashes

There is a similar failure on OSD, please refer to https://openqa.suse.de/tests/13402294#step/ipxe_install/5. @okurz

Actions #11

Updated by okurz 4 months ago

This ticket was mentioned in https://sd.suse.com/servicedesk/customer/portal/1/SD-165090 so let me provide an update here

okurz wrote in #note-7:

@waynechen55 if you have more information to make the issue easier to reproduce we might have a chance to help otherwise I see no good point to improve so I will unassign for now and keep this in "future".

This is still the current state. Without a better way to reproduce the easier it is very unlikely we will be able to provide any more help.

Actions #12

Updated by okurz 3 months ago

  • Related to action #165611: [openQA][infra][sut][aarch64] Power supply failure on squiddlydiddly arm64 machine size:M added
Actions

Also available in: Atom PDF