Actions
action #167911
closedopenQA Project - coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes
openQA Project - coordination #96263: [epic] Exclude certain Minion tasks from "Too many Minion job failures alert" alert
openQA Project - coordination #99831: [epic] Better handle minion tasks failing with "Job terminated unexpectedly"
Scripts CI | Failed pipeline - openqa-schedule-mm-ping-test incompletes on o3
Status:
Rejected
Priority:
High
Assignee:
-
Category:
Regressions/Crashes
Target version:
Start date:
2024-10-08
Due date:
% Done:
0%
Estimated time:
Description
Observation¶
We got alerted about https://gitlab.suse.de/openqa/scripts-ci/-/jobs/3201123 with these two jobs:
the later showing in its logs:
[2024-10-08T08:56:49.527267Z] [debug] [pid:101306] QEMU: QEMU emulator version 7.1.0 (SUSE Linux Enterprise 15)
[2024-10-08T08:56:49.527323Z] [debug] [pid:101306] QEMU: Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
[2024-10-08T08:56:49.527371Z] [debug] [pid:101306] QEMU: qemu-system-x86_64: terminating on signal 15 from pid 66034 (/usr/bin/perl)
[2024-10-08T08:56:49.528267Z] [debug] [pid:101306] sending magic and exit
[2024-10-08T08:56:49.528475Z] [info] [pid:101306] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
myjsonrpc: remote end terminated connection, stopping at /usr/lib/os-autoinst/myjsonrpc.pm line 43.
[2024-10-08T08:56:49.528697Z] [debug] [pid:101306] sending magic and exit
[2024-10-08T08:56:49.528827Z] [info] [pid:101306] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
myjsonrpc: remote end terminated connection, stopping at /usr/lib/os-autoinst/myjsonrpc.pm line 43.
[2024-10-08T08:56:49.528897Z] [debug] [pid:101306] sending magic and exit
[2024-10-08T08:56:49.529055Z] [warn] [pid:101306] !!! backend::baseclass::run_capture_loop: capture loop failed myjsonrpc: remote end terminated connection, stopping at /usr/lib/os-autoinst/myjsonrpc.pm line 43.
[2024-10-08T08:56:49.529115Z] [debug] [pid:101306] sending magic and exit
[2024-10-08T08:56:49.529200Z] [info] [pid:101306] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
myjsonrpc: remote end terminated connection, stopping at /usr/lib/os-autoinst/myjsonrpc.pm line 43.
[2024-10-08T08:56:49.529293Z] [debug] [pid:101306] sending magic and exit
Use of uninitialized value $_[2] in substr at /usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/IO/Handle.pm line 475.
IO::Handle::write(IO::Pipe::End=GLOB(0x564e0b69ce48), undef, undef) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 338
Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x564e0b6af750), CODE(0x564e0b6b4b28)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 492
Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x564e0b6af750)) called at /usr/lib/os-autoinst/backend/driver.pm line 72
backend::driver::start(backend::driver=HASH(0x564e0c3dcb60)) called at /usr/lib/os-autoinst/backend/driver.pm line 37
backend::driver::new("backend::driver", "qemu") called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Backend.pm line 14
OpenQA::Isotovideo::Backend::new("OpenQA::Isotovideo::Backend") called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 109
OpenQA::Isotovideo::Runner::create_backend(OpenQA::Isotovideo::Runner=HASH(0x564e04033900)) called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 251
OpenQA::Isotovideo::Runner::init(OpenQA::Isotovideo::Runner=HASH(0x564e04033900)) called at /usr/bin/isotovideo line 182
eval {...} called at /usr/bin/isotovideo line 177
Use of uninitialized value $_[1] in substr at /usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/IO/Handle.pm line 475.
IO::Handle::write(IO::Pipe::End=GLOB(0x564e0b69ce48), undef, undef) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 338
Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x564e0b6af750), CODE(0x564e0b6b4b28)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 492
Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x564e0b6af750)) called at /usr/lib/os-autoinst/backend/driver.pm line 72
backend::driver::start(backend::driver=HASH(0x564e0c3dcb60)) called at /usr/lib/os-autoinst/backend/driver.pm line 37
backend::driver::new("backend::driver", "qemu") called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Backend.pm line 14
OpenQA::Isotovideo::Backend::new("OpenQA::Isotovideo::Backend") called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 109
OpenQA::Isotovideo::Runner::create_backend(OpenQA::Isotovideo::Runner=HASH(0x564e04033900)) called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 251
OpenQA::Isotovideo::Runner::init(OpenQA::Isotovideo::Runner=HASH(0x564e04033900)) called at /usr/bin/isotovideo line 182
eval {...} called at /usr/bin/isotovideo line 177
[2024-10-08T08:56:52.608756Z] [info] +++ worker notes +++
[2024-10-08T08:56:52.608952Z] [info] End time: 2024-10-08 08:56:52
[2024-10-08T08:56:52.609048Z] [info] Result: cancel
[2024-10-08T08:56:52.616907Z] [info] Uploading autoinst-log.txt
Updated by tinita about 1 month ago
- Related to action #167797: scripts-ci multimachine test CI job fails due to job incompleting with "minion failed" size:M added
Updated by tinita about 1 month ago
Unfortunately this time I can't find a corresponding minion job with that error message this time, like in the other ticket #167797:
select * from minion_jobs where result::text like '%Job terminated unexpectedly%' and created >= '2024-10-08' limit 10;
...
(0 rows)
Updated by okurz about 1 month ago
- Category set to Regressions/Crashes
- Target version set to Ready
- Parent task set to #99831
Updated by tinita about 1 month ago
- Related to deleted (action #167797: scripts-ci multimachine test CI job fails due to job incompleting with "minion failed" size:M)
Updated by tinita about 1 month ago
- Is duplicate of action #167797: scripts-ci multimachine test CI job fails due to job incompleting with "minion failed" size:M added
Actions