Project

General

Profile

Actions

action #167911

closed

openQA Project (public) - coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

openQA Project (public) - coordination #96263: [epic] Exclude certain Minion tasks from "Too many Minion job failures alert" alert

openQA Project (public) - coordination #99831: [epic] Better handle minion tasks failing with "Job terminated unexpectedly"

Scripts CI | Failed pipeline - openqa-schedule-mm-ping-test incompletes on o3

Added by nicksinger 2 months ago. Updated 2 months ago.

Status:
Rejected
Priority:
High
Assignee:
-
Category:
Regressions/Crashes
Start date:
2024-10-08
Due date:
% Done:

0%

Estimated time:

Description

Observation

We got alerted about https://gitlab.suse.de/openqa/scripts-ci/-/jobs/3201123 with these two jobs:

the later showing in its logs:

[2024-10-08T08:56:49.527267Z] [debug] [pid:101306] QEMU: QEMU emulator version 7.1.0 (SUSE Linux Enterprise 15)
[2024-10-08T08:56:49.527323Z] [debug] [pid:101306] QEMU: Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
[2024-10-08T08:56:49.527371Z] [debug] [pid:101306] QEMU: qemu-system-x86_64: terminating on signal 15 from pid 66034 (/usr/bin/perl)
[2024-10-08T08:56:49.528267Z] [debug] [pid:101306] sending magic and exit
[2024-10-08T08:56:49.528475Z] [info] [pid:101306] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
  myjsonrpc: remote end terminated connection, stopping at /usr/lib/os-autoinst/myjsonrpc.pm line 43.
[2024-10-08T08:56:49.528697Z] [debug] [pid:101306] sending magic and exit
[2024-10-08T08:56:49.528827Z] [info] [pid:101306] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
  myjsonrpc: remote end terminated connection, stopping at /usr/lib/os-autoinst/myjsonrpc.pm line 43.
[2024-10-08T08:56:49.528897Z] [debug] [pid:101306] sending magic and exit
[2024-10-08T08:56:49.529055Z] [warn] [pid:101306] !!! backend::baseclass::run_capture_loop: capture loop failed myjsonrpc: remote end terminated connection, stopping at /usr/lib/os-autoinst/myjsonrpc.pm line 43.

[2024-10-08T08:56:49.529115Z] [debug] [pid:101306] sending magic and exit
[2024-10-08T08:56:49.529200Z] [info] [pid:101306] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
  myjsonrpc: remote end terminated connection, stopping at /usr/lib/os-autoinst/myjsonrpc.pm line 43.
[2024-10-08T08:56:49.529293Z] [debug] [pid:101306] sending magic and exit
Use of uninitialized value $_[2] in substr at /usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/IO/Handle.pm line 475.
    IO::Handle::write(IO::Pipe::End=GLOB(0x564e0b69ce48), undef, undef) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 338
    Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x564e0b6af750), CODE(0x564e0b6b4b28)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 492
    Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x564e0b6af750)) called at /usr/lib/os-autoinst/backend/driver.pm line 72
    backend::driver::start(backend::driver=HASH(0x564e0c3dcb60)) called at /usr/lib/os-autoinst/backend/driver.pm line 37
    backend::driver::new("backend::driver", "qemu") called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Backend.pm line 14
    OpenQA::Isotovideo::Backend::new("OpenQA::Isotovideo::Backend") called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 109
    OpenQA::Isotovideo::Runner::create_backend(OpenQA::Isotovideo::Runner=HASH(0x564e04033900)) called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 251
    OpenQA::Isotovideo::Runner::init(OpenQA::Isotovideo::Runner=HASH(0x564e04033900)) called at /usr/bin/isotovideo line 182
    eval {...} called at /usr/bin/isotovideo line 177
Use of uninitialized value $_[1] in substr at /usr/lib/perl5/5.26.1/x86_64-linux-thread-multi/IO/Handle.pm line 475.
    IO::Handle::write(IO::Pipe::End=GLOB(0x564e0b69ce48), undef, undef) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 338
    Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x564e0b6af750), CODE(0x564e0b6b4b28)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 492
    Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x564e0b6af750)) called at /usr/lib/os-autoinst/backend/driver.pm line 72
    backend::driver::start(backend::driver=HASH(0x564e0c3dcb60)) called at /usr/lib/os-autoinst/backend/driver.pm line 37
    backend::driver::new("backend::driver", "qemu") called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Backend.pm line 14
    OpenQA::Isotovideo::Backend::new("OpenQA::Isotovideo::Backend") called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 109
    OpenQA::Isotovideo::Runner::create_backend(OpenQA::Isotovideo::Runner=HASH(0x564e04033900)) called at /usr/lib/os-autoinst/OpenQA/Isotovideo/Runner.pm line 251
    OpenQA::Isotovideo::Runner::init(OpenQA::Isotovideo::Runner=HASH(0x564e04033900)) called at /usr/bin/isotovideo line 182
    eval {...} called at /usr/bin/isotovideo line 177
[2024-10-08T08:56:52.608756Z] [info] +++ worker notes +++
[2024-10-08T08:56:52.608952Z] [info] End time: 2024-10-08 08:56:52
[2024-10-08T08:56:52.609048Z] [info] Result: cancel
[2024-10-08T08:56:52.616907Z] [info] Uploading autoinst-log.txt

Related issues 1 (0 open1 closed)

Is duplicate of openQA Project (public) - action #167797: scripts-ci multimachine test CI job fails due to job incompleting with "minion failed" size:MResolvedmkittler2024-10-04

Actions
Actions #1

Updated by tinita 2 months ago

  • Related to action #167797: scripts-ci multimachine test CI job fails due to job incompleting with "minion failed" size:M added
Actions #2

Updated by tinita 2 months ago

Unfortunately this time I can't find a corresponding minion job with that error message this time, like in the other ticket #167797:

select * from minion_jobs where result::text like '%Job terminated unexpectedly%' and created >= '2024-10-08' limit 10;
...
(0 rows)
Actions #3

Updated by okurz 2 months ago

  • Category set to Regressions/Crashes
  • Target version set to Ready
  • Parent task set to #99831
Actions #4

Updated by tinita 2 months ago

  • Related to deleted (action #167797: scripts-ci multimachine test CI job fails due to job incompleting with "minion failed" size:M)
Actions #5

Updated by tinita 2 months ago

  • Is duplicate of action #167797: scripts-ci multimachine test CI job fails due to job incompleting with "minion failed" size:M added
Actions #6

Updated by tinita 2 months ago

  • Status changed from New to Rejected
Actions

Also available in: Atom PDF