QA » openQA Project » openQA Infrastructure

Observation¶

Jobs assigned to worker35:x can not really run actually. They all hang, no output from live view and live log, no loaded test modules, although they are in running state.

Job 14896205 assigned to worker35:50 in running state and hang
Job 14896209 assigned to worker35:51 in running state and hang
Job 14896235 assigned to worker35:49 in running state and hang

Failures with reason isotovideo died: Unable to clone Git repository 'https://github.com/waynechen55/os-autoinst-distri-opensuse.git#wayne/enable_kernel_log' specified via CASEDIR (see log for details) at /usr/lib/os-autoinst/OpenQA/Isotovideo/Utils.pm line 164. look like so:

[2024-07-11T16:35:17.821676+02:00] [info] [pid:58222] ::: OpenQA::Isotovideo::Utils::clone_git: Cloning git URL 'https://github.com/waynechen55/os-autoinst-distri-opensuse.git' into '/var/lib/openqa/pool/49'[2024-07-11T16:35:17.821776+02:00] [info] [pid:58222] ::: OpenQA::Isotovideo::Utils::clone_git: Checking out git refspec/branch 'wayne/enable_kernel_log'[2024-07-11T16:36:00.214318+02:00] [debug] [pid:58222] Cloning into 'os-autoinst-distri-opensuse'...
error: RPC failed; curl 18 HTTP/2 stream 5 was not closed cleanly before end of the underlying connection
error: 1732 bytes of body are still expected
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

Or they fail with reason Reason: backend died: ipmitool -I lanplus -H fibonacci-ipmi.qe.prg2.suse.org -U ADMIN -P [masked] mc guid: Error: Received an Unexpected Open Session Response look like so:

[2024-07-11T16:41:46.020309+02:00] [debug] [pid:63553] Launching external video encoder: ffmpeg -y -hide_banner -nostats -r 24 -f image2pipe -vcodec ppm -i - -pix_fmt yuv420p -c:v libvpx-vp9 -crf 35 -b:v 1500k -cpu-used 1 'video.webm'[2024-07-11T16:41:50.111055+02:00] [info] [pid:63553] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
ipmitool -I lanplus -H fibonacci-ipmi.qe.prg2.suse.org -U ADMIN -P [masked] mc guid: Error: Received an Unexpected Open Session Response
Error: Received an Unexpected Open Session Response
[...]
Error: Received an Unexpected Open Session Response
Error: Unable to establish IPMI v2 / RMCP+ session at /usr/lib/os-autoinst/backend/ipmi.pm line 45.[2024-07-11T16:41:50.111690+02:00] [debug] [pid:63553] Passing remaining frames to the video encoder
[image2pipe @ 0x5568ecfe1480] Could not find codec parameters for stream 0 (Video: ppm, none): unspecified size
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options

Steps to reproduce¶

• Trigger ipmi backend job
• Job assigned to worker35:x

Impact¶

Can not run job efficiently and effectively

Problem¶

Looks like problem related to worker process

Acceptance criteria¶

• AC1: Worker processes jobs successfully again

Suggestions¶

• Check relevant worker process
• Check worker settings
• Check other related processes
• Confirm if this is an issue with the job/worker setup, or there is an underlying bug e.g. assets being slow to download, git sources being slow
• Confirm if this is one issue or two separate issues and file follow-up tickets as needed

Workaround¶

n/a