QA » openQA Project » openQA Tests

http://f40.suse.de/tests/884 shows problem with clone_job atm.

trying directly on osd:
http://openqa.suse.de/tests/2408946

trying to get test result: http://openqa.suse.de/tests/2409260#settings

if shutdown is triggered and if it failed, there is no way to execute post_fail_hook.

http://f40.suse.de/tests/899#step/shutdown/

of course we can collect some logs before power_action, but is this what we want to do really?

The idea is to call the post_fail_hook in any case as the system might still be stuck before the actual power down. On a high level you can describe the states "shutdown" as such:

• before shutdown was triggered
• system is shutting down
• system is powered off

The post_fail_hook should be able to identify the states and provide information suitable for every state.

use DEBUG_SHUTDOWN to change log level so that we can call later /usr/lib/systemd/system-shutdown/debug.sh in post_fail_hook:

http://f40.suse.de/tests/904

http://f40.suse.de/tests/927#step/shutdown/7 shows problem with

assert_script_run("journalctl --no-pager -b 0 > /tmp/full_journal.log");

typing issue happens again and again.

http://f40.suse.de/tests/928#step/shutdown/9

with a empty space before journalctl it seems better but upload_logs failed because of curl server connection issue.

http://f40.suse.de/tests/930#step/shutdown/16 resolved the issue with typing, it requires select_console('log-console');

but connection issue for uploading logs is still there...

I found the reason why upload is not working, because cleanup_before_shutdown stop network.service:

  # Proceed with dhcp cleanup on qemu backend only.
    # Cleanup is made, because if same hdd image used in multimachine scenario
    # on several nodes, the dhcp clients use same id and cause conflicts on dhcpd server.
    if (check_var('BACKEND', 'qemu')) {
    my $network_status = script_output('systemctl status network');
    # Do dhcp cleanup for wicked
    if ($network_status =~ /wicked/) {
    systemctl 'stop network.service';
    systemctl 'stop wickedd.service';
    assert_script_run('ls /var/lib/wicked/');
    save_screenshot;
    assert_script_run('rm -f /var/lib/wicked/*.xml');
    }
    }

changes in cleanup_before_shutdown.pm

# Proceed with dhcp cleanup on qemu backend only.
# Cleanup is made, because if same hdd image used in multimachine scenario
# on several nodes, the dhcp clients use same id and cause conflicts on dhcpd server.
# network.service is needed for uploading log for DEBUG_SHUTDOWN 
if (check_var('BACKEND', 'qemu') && !get_var('DEBUG_SHUTDOWN')){
my $network_status = script_output('systemctl status network');
# Do dhcp cleanup for wicked
if ($network_status =~ /wicked/) {
systemctl 'stop network.service';
systemctl 'stop wickedd.service';
assert_script_run('ls /var/lib/wicked/');
save_screenshot;
assert_script_run('rm -f /var/lib/wicked/*.xml');
}
}

http://f40.suse.de/tests/936#step/shutdown/21 shows successful uploading of logs.

Next step should be check this on s390 remote worker and think about what kind of additional logs are helpful.

http://f40.suse.de/tests/950#step/shutdown/15 shows expected results.

PR:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6702

cleanup_before_shutdown before shutdown collects already dmesg logs to serial0.txt, so add journal log extra to serial0.txt with comment "--- dmesg ---" and "journactl log".

http://f40.suse.de/tests/1076/file/serial0.txt shows now expected logs.

I like the idea of the PR but please keep in mind that the goal is to have a helpful post_fail_hook ready in all cases and not just on request when DEBUG_SHUTDOWN=1 is set

The Ticket is about to provide useful logs for shutdown, and if DEBUG_SHUTDONW is not set, then we cannot get full logs. This can be done before shutdown (when shutdown got triggered, the network is in most cases down for example.) So I think this is what we can achieve atm. The goal you mean is different then it here.

zluo wrote:

The Ticket is about to provide useful logs for shutdown, and if DEBUG_SHUTDONW is not set, then we cannot get full logs.

Please remember #42038#note-12 which we also discussed in person. The idea is if the system is actually not shut down so when we can still access a console. So I think in the post_fail_hook we should be able to try that.

Note:

before shutdown was triggered

• this is no problem, we can catch the logs in cleanup_before_shutdown.

system is shutting down

• this is complicated, we need to define what we need to check and what is possible to check during system's shutting down

system is powered off

• this is already covered in the test.