Project

General

Profile

action #73162

t/01-test-utilities.t fails in "test would have failed"

Added by okurz 10 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2020-10-09
Due date:
2020-10-25
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

[ 1108s]     #   Failed test 'test would have failed'
[ 1108s]     #   at ./t/01-test-utilities.t line 47.
[ 1108s] 
[ 1108s]     #   Failed test 'manual termination via stop_service does not trigger _fail_and_exit'
[ 1108s]     #   at ./t/01-test-utilities.t line 58.
[ 1108s]     #          got: '1'
[ 1108s]     #     expected: '0'
[ 1108s]     # Looks like you failed 2 tests of 5.
[ 1108s] 
[ 1108s] #   Failed test 'warnings in sub processes are fatal test failures'
[ 1108s] #   at ./t/01-test-utilities.t line 59.
[ 1108s] # Looks like you failed 1 test of 1.
[ 1108s] ./t/01-test-utilities.t ..................................... 
[ 1108s] # Subtest: warnings in sub processes are fatal test failures
[ 1108s]     # PID of test-process: 5808
[ 1108s]     ok 1 - warning logged
[ 1108s]     not ok 2 - test would have failed
[ 1108s]     # PID of test-process: 5813
[ 1108s]     ok 3 - message of test failure
[ 1108s]     ok 4 - exit code of test failure is non-zero
[ 1108s]     not ok 5 - manual termination via stop_service does not trigger _fail_and_exit
[ 1108s]     1..5
[ 1108s] not ok 1 - warnings in sub processes are fatal test failures

in https://build.opensuse.org/build/devel:openQA:TestGithub:PR-3456/openSUSE_Factory/x86_64/openQA/_log from https://github.com/os-autoinst/openQA/pull/3456

Checking reproducibility…

EDIT: happened again, e.g. in https://build.opensuse.org/package/live_build_log/devel:openQA/openQA/openSUSE_Factory/x86_64 even though at time of writing it was fine again

Acceptance criteria

  • AC1: t/01-test-utilities.t does not fail in OBS checks reproducibly (either fixed or excluded)

Suggestions

  • Wait until this happens again and take a look in the log for details
  • Try to reproduce failure in test OBS project
  • Fix or exclude

History

#1 Updated by okurz 10 months ago

  • Due date set to 2020-10-18
  • Status changed from In Progress to Feedback
  • Priority changed from Normal to Low

I don't see it in build.opensuse.org/project/show/devel:openQA

#2 Updated by okurz 10 months ago

  • Status changed from Feedback to Rejected

haven't seen this again

#3 Updated by okurz 9 months ago

  • Description updated (diff)
  • Due date changed from 2020-10-18 to 2020-10-25
  • Status changed from Rejected to Feedback
  • Priority changed from Low to Normal

#4 Updated by okurz 9 months ago

  • Status changed from Feedback to Workable
  • Assignee changed from okurz to mkittler

Failed again and found details in https://build.opensuse.org/public/build/devel:openQA:tested/openSUSE_Tumbleweed/x86_64/openQA/_log today:

[  904s]     #   Failed test 'test would have failed'
[  904s]     #   at ./t/01-test-utilities.t line 47.
[  904s] 
[  904s]     #   Failed test 'manual termination via stop_service does not trigger _fail_and_exit'
[  904s]     #   at ./t/01-test-utilities.t line 58.
[  904s]     #          got: '1'
[  904s]     #     expected: '0'
[  904s]     # Looks like you failed 2 tests of 5.
[  904s] 
[  904s] #   Failed test 'warnings in sub processes are fatal test failures'
[  904s] #   at ./t/01-test-utilities.t line 59.
[  904s] # Looks like you failed 1 test of 1.
[  905s] ./t/01-test-utilities.t ..................................... 
[  905s] # Subtest: warnings in sub processes are fatal test failures
[  905s]     # PID of test-process: 5508
[  905s]     ok 1 - warning logged
[  905s]     not ok 2 - test would have failed
[  905s]     # PID of test-process: 5513
[  905s]     ok 3 - message of test failure
[  905s]     ok 4 - exit code of test failure is non-zero
[  905s]     not ok 5 - manual termination via stop_service does not trigger _fail_and_exit
[  905s]     1..5
[  905s] not ok 1 - warnings in sub processes are fatal test failures
[  905s] 1..1
[  905s] Dubious, test returned 1 (wstat 256, 0x100)
[  905s] Failed 1/1 subtests 

mkittler do you think you could take a look into this?

#5 Updated by mkittler 9 months ago

  • Status changed from Workable to In Progress

#6 Updated by mkittler 9 months ago

PR (also contains explanation): https://github.com/os-autoinst/openQA/pull/3477

#7 Updated by okurz 9 months ago

The PR is merged but unfortunately did not work, https://build.opensuse.org/public/build/devel:openQA/openSUSE_Tumbleweed/x86_64/openQA/_log running openQA-4.6.1603211536.4498c99a4 corresponding to https://github.com/os-autoinst/openQA/commit/4498c99a4376dd7a0e2a2a7ac0f7baf67112fc0c which is the commit of the above PR shows still:

[ 1128s] 
[ 1128s]     #   Failed test 'test would have failed'
[ 1128s]     #   at ./t/01-test-utilities.t line 49.
[ 1128s] 
[ 1128s]     #   Failed test 'manual termination via stop_service does not trigger _fail_and_exit'
[ 1128s]     #   at ./t/01-test-utilities.t line 61.
[ 1128s]     #          got: '1'
[ 1128s]     #     expected: '0'
[ 1128s]     # Looks like you failed 2 tests of 5.
[ 1128s] 
[ 1128s] #   Failed test 'warnings in sub processes are fatal test failures'
[ 1128s] #   at ./t/01-test-utilities.t line 62.
[ 1128s] # Looks like you failed 1 test of 1.
[ 1128s] ./t/01-test-utilities.t ..................................... 
[ 1128s] # Subtest: warnings in sub processes are fatal test failures
[ 1128s]     # PID of test-process-1: 5818
[ 1128s]     # waiting at most 70 seconds for SIGCHLD (sleep is supposed to be interrupted by SIGCHLD)
[ 1128s]     ok 1 - warning logged
[ 1128s]     not ok 2 - test would have failed
[ 1128s]     # PID of test-process-2: 5877
[ 1128s]     # waiting at most 70 seconds for SIGTERM (sleep is supposed to be interrupted by SIGTERM)
[ 1128s]     ok 3 - message of test failure
[ 1128s]     ok 4 - exit code of test failure is non-zero
[ 1128s]     not ok 5 - manual termination via stop_service does not trigger _fail_and_exit
[ 1128s]     1..5
[ 1128s] not ok 1 - warnings in sub processes are fatal test failures
[ 1128s] 1..1
[ 1128s] Dubious, test returned 1 (wstat 256, 0x100)
[ 1128s] Failed 1/1 subtests 

#9 Updated by mkittler 9 months ago

  • Status changed from In Progress to New
  • Assignee deleted (mkittler)

Then I'm out of ideas. Not sure how to test the code. If we would mock everything it would be quite useless because I want to test whether the code actually works in accordance to the OS signal handling. Maybe just keep the test for local development (where I've never run into the error). Since you've already removed the test from OBS it shouldn't prevent our builds anymore at least.

#10 Updated by okurz 9 months ago

  • Status changed from New to Resolved
  • Assignee set to okurz

I think it's ok like it is now. My PR is merged, the test is excluded from OBS checks so not failing there anymore but we still run the test fine within circleCI checks just fine.

Also available in: Atom PDF