25-cache-service.t fails repeatedly but circleCI receives the status as "success"
t/25-cache-service.t fails repeatedly but circleCI receives the status as "success", e.g. see https://app.circleci.com/pipelines/github/os-autoinst/openQA/4217/workflows/cbd431a2-6d05-4e95-a10c-747023efba23/jobs/40329/steps
- AC1: a job in circleci does not report success if tests fails for all retries
Could this be a regression from f3b5dd95c ?
- Try to reproduce locally or within circleci
Likely the problem is older. Looking back further into the old history from https://app.circleci.com/pipelines/github/os-autoinst/openQA?branch=master I find e.g. https://app.circleci.com/pipelines/github/os-autoinst/openQA/3950/workflows/ce67fc66-a48f-4f31-a750-f99c5f2ece0f/jobs/37623/steps from 22 days ago and it shows the same problem. Did someone merge a change that had a big coverage drop and did not realize the problem?
EDIT: Digging even further the oldest I found so far is https://app.circleci.com/pipelines/github/os-autoinst/openQA/3806/workflows/a002e2f8-c9f3-4229-bbfb-66eb8258c243/jobs/36169/steps from 2020-08-11 commit 40c3d2c . https://app.circleci.com/pipelines/github/os-autoinst/openQA/3471/workflows/d603f1cc-6b9c-4610-9f4f-a2429fb99649/jobs/33218 from 2020-07-03 commit 46240bf,a010576 is still fine.
I think I may have found the commit where it started to fail:
https://github.com/os-autoinst/openQA/pull/3242 deps: Update Mojo-IOLoop-ReadWriteProcess (0.25 -> 0.27)
Last good: https://app.circleci.com/pipelines/github/os-autoinst/openQA/3484/workflows/367605f2-05ae-4666-bf63-f77f146078d6/jobs/33336
First bad: https://app.circleci.com/pipelines/github/os-autoinst/openQA/3503/workflows/d59af86e-378d-4150-ba44-b85c92f10e17/jobs/33453
I just tested this locally and was able to confirm.
% make test-with-database PROVE_ARGS="-v t/25-cache-service.t" ... ok 22 - OpenQA::CacheService::Task::Sync ok 23 - no (unexpected) warnings (via done_testing) 1..23 # Worker cache service on port 9530 stopped # Looks like your test exited with -1 just after 23. Dubious, test returned 255 (wstat 65280, 0xff00) All 23 subtests passed Test Summary Report ------------------- t/25-cache-service.t (Wstat: 65280 Tests: 23 Failed: 0) Non-zero exit status: 255 Files=1, Tests=23, 132 wallclock secs ( 0.08 usr 0.00 sys + 17.74 cusr 1.52 csys = 19.34 CPU) Result: FAIL make: *** [Makefile:174: test-unit-and-integration] Error 1 make: Leaving directory '/home/tina/openqa-devel/repos/openQA' make: *** [Makefile:169: test-with-database] Error 2 % echo $? 2
However, with the older version of the module, everything is fine:
% cpanm -l local Mojo::IOLoop::ReadWriteProcess@0.25 % export PERL5LIB=$PWD/local/lib/perl5 % make test-with-database PROVE_ARGS="-v t/25-cache-service.t" ... ok 22 - OpenQA::CacheService::Task::Sync ok 23 - no (unexpected) warnings (via done_testing) 1..23 # Worker cache service on port 9530 stopped ok All tests successful. Files=1, Tests=23, 130 wallclock secs ( 0.07 usr 0.01 sys + 16.29 cusr 1.39 csys = 17.76 CPU) Result: PASS make: Leaving directory '/home/tina/openqa-devel/repos/openQA' [ 0 = 1 ] || pg_ctl -D /dev/shm/tpg stop waiting for server to shut down.... done server stopped % echo $? 0
for os-autoinst trying with https://github.com/os-autoinst/os-autoinst/pull/1536