Project

General

Profile

action #69148

regression: output of self-tests on circle CI is missing from log view, "tests" tab shows nothing (due to timeout)

Added by okurz 6 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2020-07-20
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

https://app.circleci.com/pipelines/github/os-autoinst/openQA/3546/workflows/28188eaa-92c1-41f7-b32f-739f7c1fe038/jobs/33798/steps shows

[10:34:59] t/ui/10-tests_overview.t ............... ok   130578 ms ( 0.21 usr  0.02 sys + 29.77 cusr 87.83 csys = 117.83 CPU)
[10:37:10] t/ui/12-needle-edit.t .................. 3/? [10:37:10] t/ui/12-needle-edit.t .................. 4/? [10:37:10] t/ui/12-needle-edit.t .................. 5/? [10:37:10] t/ui/12-needle-edit.t .................. 6/?                                                          [10:37:10] t/ui/12-needle-edit.t .................. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/6 subtests 
[10:41:45] t/ui/13-admin-no-login.t ............... ok   123534 ms ( 0.03 usr  0.01 sys + 27.36 cusr 89.84 csys = 117.24 CPU)

with no details which we had some time ago, probably around 2020-03 and earlier. The tests tab does not show any details either, probably because the job timed out

Acceptance criteria

  • AC1: The circleCI log not only shows the summary but details about the failure, i.e. every failure that was reported, though not the verbose test output

Problem

Clearly a regression as this worked in the past. Maybe due to our change to add the "--merge" parameter on prove?

Suggestions

  • Research in past logs to see if we can still find references of old jobs
  • Check how tests look locally
  • Play around with parameters, e.g. "--merge" which one we need and how they behave
  • Fix it :)

History

#1 Updated by okurz 6 months ago

  • Description updated (diff)
  • Status changed from New to Workable

#2 Updated by cdywan 6 months ago

  • Status changed from Workable to In Progress
  • Assignee set to cdywan

Nice to see I'm not the only one to see this anymore (I had been looking into this a while back).

A way to reproduce it easily: set the timeout to something small like 8 in the Makefile and run at least two short tests. Observe that there is no testplan regardless of results.

I'll attempt to fix it by making Prove aware of the fact that timeout kills it before it can record the results using --trap and -s SIGINT -k 5.

#4 Updated by okurz 6 months ago

PR merged, I want proof that it actually works though, so let some tests timeout on circle CI :)

#5 Updated by cdywan 5 months ago

  • Status changed from In Progress to Feedback

#6 Updated by okurz 4 months ago

  • Status changed from Feedback to Resolved

cdywan did you do something for an actual proof? I guess not but 2 months have passed and we did not come back to this ticket so let's assume it's actually fixed for good

Also available in: Atom PDF