Project

General

Profile

action #71449

25-cache-service.t fails repeatedly but circleCI receives the status as "success"

Added by okurz 10 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2020-09-16
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

t/25-cache-service.t fails repeatedly but circleCI receives the status as "success", e.g. see https://app.circleci.com/pipelines/github/os-autoinst/openQA/4217/workflows/cbd431a2-6d05-4e95-a10c-747023efba23/jobs/40329/steps

Acceptance criteria

  • AC1: a job in circleci does not report success if tests fails for all retries

Problem

Could this be a regression from f3b5dd95c ?

Suggestion

  • Try to reproduce locally or within circleci

Related issues

Copied to openQA Project - action #71863: [os-autoinst] Include tests in code coverage analysisWorkable2020-09-16

History

#1 Updated by okurz 10 months ago

Likely the problem is older. Looking back further into the old history from https://app.circleci.com/pipelines/github/os-autoinst/openQA?branch=master I find e.g. https://app.circleci.com/pipelines/github/os-autoinst/openQA/3950/workflows/ce67fc66-a48f-4f31-a750-f99c5f2ece0f/jobs/37623/steps from 22 days ago and it shows the same problem. Did someone merge a change that had a big coverage drop and did not realize the problem?

EDIT: Digging even further the oldest I found so far is https://app.circleci.com/pipelines/github/os-autoinst/openQA/3806/workflows/a002e2f8-c9f3-4229-bbfb-66eb8258c243/jobs/36169/steps from 2020-08-11 commit 40c3d2c . https://app.circleci.com/pipelines/github/os-autoinst/openQA/3471/workflows/d603f1cc-6b9c-4610-9f4f-a2429fb99649/jobs/33218 from 2020-07-03 commit 46240bf,a010576 is still fine.

#3 Updated by okurz 10 months ago

great that you found this. I prepared https://github.com/os-autoinst/openQA/pull/3399 to analyze the code coverage of the folder "t/" as well. This should tell us which code is never executed.

#4 Updated by okurz 10 months ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz

#5 Updated by tinita 10 months ago

I just tested this locally and was able to confirm.

% make test-with-database PROVE_ARGS="-v t/25-cache-service.t"
...
ok 22 - OpenQA::CacheService::Task::Sync
ok 23 - no (unexpected) warnings (via done_testing)
1..23
    # Worker cache service on port 9530 stopped
# Looks like your test exited with -1 just after 23.
Dubious, test returned 255 (wstat 65280, 0xff00)
All 23 subtests passed 

Test Summary Report
-------------------
t/25-cache-service.t (Wstat: 65280 Tests: 23 Failed: 0)
  Non-zero exit status: 255
Files=1, Tests=23, 132 wallclock secs ( 0.08 usr  0.00 sys + 17.74 cusr  1.52 csys = 19.34 CPU)
Result: FAIL
make[1]: *** [Makefile:174: test-unit-and-integration] Error 1
make[1]: Leaving directory '/home/tina/openqa-devel/repos/openQA'
make: *** [Makefile:169: test-with-database] Error 2
% echo $?
2

However, with the older version of the module, everything is fine:

% cpanm -l local Mojo::IOLoop::ReadWriteProcess@0.25
% export PERL5LIB=$PWD/local/lib/perl5
% make test-with-database PROVE_ARGS="-v t/25-cache-service.t"
...
ok 22 - OpenQA::CacheService::Task::Sync
ok 23 - no (unexpected) warnings (via done_testing)
1..23
    # Worker cache service on port 9530 stopped
ok
All tests successful.
Files=1, Tests=23, 130 wallclock secs ( 0.07 usr  0.01 sys + 16.29 cusr  1.39 csys = 17.76 CPU)
Result: PASS
make[1]: Leaving directory '/home/tina/openqa-devel/repos/openQA'
[ 0 = 1 ] || pg_ctl -D /dev/shm/tpg stop
waiting for server to shut down.... done
server stopped
% echo $?
0

#6 Updated by okurz 10 months ago

  • Status changed from In Progress to Feedback

https://github.com/os-autoinst/openQA/pull/3400 to fix the wrong evaluation in Makefile. An actual fix for the failing 25-cache-service.t would be https://github.com/os-autoinst/openQA/pull/3391

#7 Updated by okurz 10 months ago

merged https://github.com/os-autoinst/openQA/pull/3399 about "t/" code coverage. We could do the same for os-autoinst btw.

#8 Updated by okurz 10 months ago

#9 Updated by okurz 10 months ago

  • Copied to action #71863: [os-autoinst] Include tests in code coverage analysis added

#10 Updated by okurz 10 months ago

  • Status changed from Feedback to Resolved

We properly evaluate the status of tests and include them in code coverage analysis as well

Also available in: Atom PDF