action #177775
closedci: fullstack-unstable test is more unstable than usual
Description
Observation¶
fullstack-unstable has been failing quite often recently.
https://app.circleci.com/pipelines/github/os-autoinst/openQA?branch=master
Latest failure: https://app.circleci.com/pipelines/github/os-autoinst/openQA/15835/workflows/04b2d0e0-d43c-4b80-a064-36ef5bcfcda7/jobs/152282
Retry 1 of 5 …
[14:12:23] t/05-scheduler-full.t .. 1/? unexpected OP_CUSTOM (catch) at /usr/lib/perl5/5.26.1/B/Deparse.pm line 1667.
unexpected OP_CUSTOM (catch) at /usr/lib/perl5/5.26.1/B/Deparse.pm line 1667.
[...]
[14:12:23] t/05-scheduler-full.t .. 3/?
# Failed test 'sub process openqa-webapi terminated with exit code 9'
[...]
unexpected OP_CUSTOM (catch) at /usr/lib/perl5/5.26.1/B/Deparse.pm line 1667.
[14:13:52] t/05-scheduler-full.t .. 3/?
# Failed test 'sub process openqa-worker-unstable terminated with exit code 9'
In some cases it can apparently fail like this as well e.g. https://github.com/os-autoinst/openQA/pull/6202 / https://app.circleci.com/pipelines/github/os-autoinst/openQA/15836/workflows/642aee2e-12fe-4155-bda5-74e17d33d996/jobs/152291 (but not seen it in the same CI run):
# Failed test 'no more jobs need to be allocated'
# at t/05-scheduler-full.t line 122.
# got: '1'
# expected: '0'
# Looks like you failed 1 test of 4.
[14:17:47] t/05-scheduler-full.t .. 2/?
# Failed test 'Scheduler worker job allocation'
# at t/05-scheduler-full.t line 126.
Suggestions¶
Updated by mkittler 5 days ago
Note that the worker process didn't terminate with exit code here but was actually terminated by that signal - which is SIGKILL. So maybe it was killed due to resource constraints in the CI env. Then https://github.com/os-autoinst/openQA/pull/6207 might fix the issue. I also created https://github.com/os-autoinst/openQA/pull/6208 to improve the error message.