Project

General

Profile

Actions

action #71188

open

coordination #102906: [saga][epic] Increased stability of tests with less "known failures", known incompletes handled automatically within openQA

coordination #102909: [epic] Prevent more incompletes already within os-autoinst or openQA

job incomplete with auto_review:"backend died: QEMU exited unexpectedly, see log for details" and no other obvious information in the logfile what went wrong

Added by ilausuch over 3 years ago. Updated over 2 years ago.

Status:
Workable
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2020-09-10
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://openqa.suse.de/tests/4667727 incomplete reason "backend died: QEMU exited unexpectedly, see log for details"

https://openqa.suse.de/tests/4667727/file/autoinst-log.txt shows just:


[2020-09-09T19:53:48.385 CEST] [debug] <<< testapi::wait_serial(quiet=undef, timeout=1000, expect_not_found=0, record_output=undef, buffer_size=undef, regexp=qr/cmy9h-\d+-/, no_regex=0)
[2020-09-09T19:53:49.454 CEST] [debug] >>> testapi::wait_serial: (?^:cmy9h-\d+-): ok
[2020-09-09T19:53:49.454 CEST] [debug] tests/console/force_scheduled_tasks.pm:54 called testapi::assert_script_run
[2020-09-09T19:53:49.454 CEST] [debug] <<< testapi::assert_script_run(cmd="for i in \$(systemctl list-units --type=timer --state=active --no-legend | sed -e 's/\\(\\S\\+\\)\\.timer\\s.*/\\1/'); do echo \"Triggering systemd timed service \$i\" && systemctl start \$i && systemctl mask \$i.{service,timer}; done", quiet=undef, fail_message="", timeout=1000)
[2020-09-09T19:53:49.454 CEST] [debug] tests/console/force_scheduled_tasks.pm:54 called testapi::assert_script_run
[2020-09-09T19:53:49.454 CEST] [debug] <<< testapi::type_string(string="for i in \$(systemctl list-units --type=timer --state=active --no-legend | sed -e 's/\\(\\S\\+\\)\\.timer\\s.*/\\1/'); do echo \"Triggering systemd timed service \$i\" && systemctl start \$i && systemctl mask \$i.{service,timer}; done", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2020-09-09T19:53:52.124 CEST] [info] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
[2020-09-09T19:53:52.127 CEST] [debug] sending magic and exit
[2020-09-09T19:53:52.127 CEST] [debug] received magic close
[2020-09-09T19:53:52.128 CEST] [debug] THERE IS NOTHING TO READ 16 5 4
[2020-09-09T19:53:52.128 CEST] [debug] stopping command server 28437 because test execution ended
[2020-09-09T19:53:52.128 CEST] [debug] isotovideo: informing websocket clients before stopping command server: http://127.0.0.1:20053/Bx3fOTDMBl3VvYnl/broadcast
[2020-09-09T19:53:52.142 CEST] [debug] commands process exited: 0
[2020-09-09T19:53:52.151 CEST] [debug] backend process exited: 0
[2020-09-09T19:53:52.177 CEST] [debug] Driver backend collected unknown process with pid 28559 and exit status: 0
[2020-09-09T19:53:52.177 CEST] [debug] done with command server
[2020-09-09T19:53:52.177 CEST] [debug] stopping autotest process 28441
[2020-09-09T19:53:52.177 CEST] [debug] autotest received signal TERM, saving results of current test before exiting
[2020-09-09T19:53:52.187 CEST] [debug] [autotest] process exited: 1
[2020-09-09T19:53:53.288 CEST] [debug] done with autotest process
[2020-09-09T19:53:53.288 CEST] [debug] isotovideo failed
[2020-09-09T19:53:53.288 CEST] [debug] stopping backend process 28500
[2020-09-09T19:53:53.288 CEST] [debug] done with backend process
28430: EXIT 1
[2020-09-09T19:53:53.0361 CEST] [info] [pid:1576] Isotovideo exit status: 1
[2020-09-09T19:53:53.0387 CEST] [info] [pid:1576] +++ worker notes +++
[2020-09-09T19:53:53.0387 CEST] [info] [pid:1576] End time: 2020-09-09 17:53:53
[2020-09-09T19:53:53.0387 CEST] [info] [pid:1576] Result: died
[2020-09-09T19:53:53.0393 CEST] [info] [pid:37797] Uploading logs_from_installation_system-y2logs.tar.bz2
[2020-09-09T19:54:22.0706 CEST] [info] [pid:37797] Uploading video.ogv
[2020-09-09T19:59:29.0220 CEST] [info] [pid:37797] Uploading vars.json
[2020-09-09T19:59:29.0786 CEST] [info] [pid:37797] Uploading autoinst-log.txt

Steps to reproduce

Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
for example to look for this ticket call openqa-query-for-job-label poo#71188

Acceptance criteria

  • AC1: A user better understands what could have gone wrong and the next step is clear what to do

Suggestions

  • Regardless of what happened maybe just restarting would have helped
  • Check if there is any information on the worker host for this time

Related issues 1 (1 open0 closed)

Related to openQA Project - action #75091: incomplete jobs with one of the isotovideo sub-processes receiving a signal and just terminating, no clue why or who/what triggered the terminationNew2020-10-22

Actions
Actions

Also available in: Atom PDF