openQA Project (public)

Observation¶

openqa_install+publish jobs in openQA in OpenQA tests started failing, which caused a wave of Unreviewed issue (Group 24 openQA) emails to be sent every half hour or so.

The emails contain this:

# --- 8< ---
# [2023-02-09T09:03:50.690315+01:00] [debug] [pid:21411] QEMU status is not 'shutdown', it is 'running'
# [2023-02-09T09:03:50.690400+01:00] [debug] [pid:21268] backend shutdown state: 
# [2023-02-09T09:03:50.690645+01:00] [info] [pid:21411] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
# [2023-02-09T09:03:51.741976+01:00] [debug] [pid:21411] Passing remaining frames to the video encoder
# frame056 fps=0 q=0 Lsize!41kB time:02:07.29 bitrate7.8kbits/s speed=083x    
# video:2122kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.936463%
# [2023-02-09T09:03:54.217738+01:00] [debug] [pid:21411] Waiting for video encoder to finalize the video
# [2023-02-09T09:03:54.217801+01:00] [debug] [pid:21411] The external video encoder (pid 21537) terminated
# [2023-02-09T09:03:54.217841+01:00] [debug] [pid:21411] The built-in video encoder (pid 21538) terminated
# [2023-02-09T09:03:54.218255+01:00] [debug] [pid:21411] QEMU: qemu-system-x86_64: terminating on signal 15 from pid 21411 (/usr/bin/isotovideo: backen)
# --- >8 ---

The SIGTERM is expected here. More relevant messages can also be found in the log:

[2023-02-09T10:36:43.583422+01:00] [debug] [pid:12662] no match: -0.9s, best candidate: openqa-dashboard-no_jobs-tumbleweed-20200106 (0.00)
[2023-02-09T10:36:43.892722+01:00] [debug] [pid:12502] >>> testapi::_check_backend_response: match=openqa-dashboard timed out after 360 (assert_screen)
[2023-02-09T10:36:43.986352+01:00] [info] [pid:12502] ::: basetest::runtest: # Test died: no candidate needle with tag(s) 'openqa-dashboard' matched

Acceptance criteria¶

• AC1: There is always a helpful hint explaining why the email was sent
• AC2: Jobs are not built so frequently that they cause email floods

Out of scope¶

• A failure in the same job doesn't cause repeated emails

Suggestions¶

• Verify that a comment like poo#124143 prevents an "unreview issue" from being detected
• Investigate what error message if any triggered the script
• Add a note on why the job is "unreviewed" i.e. unreviewed always means no bug ref
• Always include no candidate needle with tag(s) messages in the email
• Consider explicitly treating needle mismatches as "reviewed"
• There's [2023-02-09T10:37:10.650093+01:00] [warn] [pid:12502] !!! testapi::script_run: DEPRECATED call of script_run() in lib/openQAcoretest.pm:8 adddie_on_timeout => ?to the call or set $distri->{script_run_die_on_timeout} to avoid this warning which could be seen as an unreviewed error message but it's not seen in the snippet
• Check if this could be an unintended side-effect of #98862
• Check the openQA-in-openQA trigger-test-monitor pipeline in jenkins.qa.suse.de/ , maybe we trigger too many