Project

General

Profile

action #50456

[functional][u] gather logs and information from SUT in failed state

Added by jorauch over 2 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Enhancement to existing tests
Target version:
SUSE QA - Milestone 25
Start date:
2019-04-16
Due date:
% Done:

0%

Estimated time:
Difficulty:
hard

Description

Motivation

As discussed in https://progress.opensuse.org/issues/49763 a way to gather logs from a SUT in failed state (e.g. according to bsc#1130701) would be very useful
Some good entry points are provided in the related ticket

Acceptance criteria

  • AC1 Kernel messages do not cause pot_fail_hook to fail
  • AC2 When bsc#1130701 happens, the job provides logs from post_fail_hook.
  • AC3 A proper console is used for kernel messages (e.g. log-console on tty5)

Suggestions

  • Find out why on aarch64 the behavior of the serial and root console is different (merged output)
  • Implement a solution to avoid that kernel messages interrupt the uploading of logs.

Checklist

  • Investigate why kernel messages are sent to root-console only on aarch64
  • Don't send kernel messages to root-console or don't use root-console

Related issues

Related to openQA Tests - action #49763: [functional][u] test fails in force_scheduled_tasks - enable problem detection post fail hookResolved2019-03-27

Related to openQA Tests - action #51770: [functional][u] 'sysrq: Show Blocked State' pollutes console and assert_screen failsResolved2019-05-21

History

#1 Updated by jorauch over 2 years ago

  • Related to action #49763: [functional][u] test fails in force_scheduled_tasks - enable problem detection post fail hook added

#2 Updated by jorauch over 2 years ago

  • Description updated (diff)

#3 Updated by jorauch over 2 years ago

As suggested by szarate:
If you ask me, add a new console to the tests, called logging, or log as an user and become root, extract the logs, end of story.

#4 Updated by SLindoMansilla over 2 years ago

  • Checklist set to [ ] Investigate why kernel messages are sent to root-console only on aarch64, [ ] Don't send kernel messages to root-console or don't use root-console
  • Description updated (diff)
  • Priority changed from Normal to High

#5 Updated by mgriessmeier over 2 years ago

  • Description updated (diff)
  • Status changed from New to Workable

#6 Updated by zluo over 2 years ago

  • Assignee set to zluo
  • Target version changed from Milestone 24 to Milestone 25

take over and check.

#7 Updated by zluo over 2 years ago

post_fail_hook already uses select_console('log-console').

#8 Updated by zluo over 2 years ago

bootloader_setup.pm

sub select_bootmenu_more {
my ($tag, $more) = @_;
my @params;
# do not waste time waiting when we already matched
assert_screen 'inst-bootmenu', 15 unless match_has_tag 'inst-bootmenu';
stop_grub_timeout;
# after installation-images 14.210 added a submenu
if ($more && check_screen 'inst-submenu-more', 0) {
send_key_until_needlematch('inst-onmore', get_var('OFW') ? 'up' : 'down', 10, 5);
send_key "ret";
}
send_key_until_needlematch($tag, get_var('OFW') ? 'up' : 'down', 10, 3);
if (get_var('UEFI')) {
send_key 'e';
send_key 'down' for (1 .. 4);
send_key 'end';
# newer versions of qemu on arch automatically add 'console=ttyS0' so
# we would end up nowhere. Setting console parameter explicitly
# See https://bugzilla.suse.com/show_bug.cgi?id=1032335 for details
push @params, 'console=tty1' if get_var('MACHINE') =~ /aarch64/;
# Hyper-V defaults to 1280x1024, we need to fix it here
push @params, get_hyperv_fb_video_resolution if check_var('VIRSH_VMM_FAMILY', 'hyperv');
type_string_very_slow(" @params ");
send_key 'f10';
}

tty1 is set. but https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/susedistribution.pm

defines at LINE 401:
$self->add_console('installation', 'tty-console', {tty => check_var('VIDEOMODE', 'text') ? 1 : 7});
It means tty7 shoud be used.

But I cannot find setting VIDEOMODE in https://openqa.suse.de/tests/2738495#settings, so I think the installation stays still in tty1.

#9 Updated by zluo over 2 years ago

  • Status changed from Workable to In Progress

#10 Updated by zluo over 2 years ago

in /lib/console.pm

sub post_fail_hook {
my ($self) = shift;
select_console('log-console');
$self->SUPER::post_fail_hook;
$self->remount_tmp_if_ro;
$self->export_logs_basic;
# Export extra log after failure for further check gdm issue 1127317, also poo#45236 used for tracking action on Openqa
$self->export_logs_desktop;
}

#12 Updated by zluo over 2 years ago

changes:

sub post_fail_hook {
my $self = shift;
select_console('log-console');
$self->export_logs();
$self->export_logs_locale();
}

log-console tty5 is used now.

#13 Updated by zluo over 2 years ago

  • Checklist set to [x] Investigate why kernel messages are sent to root-console only on aarch64

#15 Updated by zluo over 2 years ago

  • Checklist set to [x] Don't send kernel messages to root-console or don't use root-console

#16 Updated by SLindoMansilla over 2 years ago

  • Status changed from In Progress to Resolved

PR merged: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/7479

Verification on OSD: https://openqa.suse.de/tests/2902908#step/consoletest_setup/43

Notice no kernel messages in the console. The post fail hook fails for other reasons. Logs are uploaded. AC's fulfilled.

#17 Updated by szarate over 2 years ago

  • Related to action #51770: [functional][u] 'sysrq: Show Blocked State' pollutes console and assert_screen fails added

Also available in: Atom PDF