action #73162
closed
t/01-test-utilities.t fails in "test would have failed"
Added by okurz over 3 years ago.
Updated over 3 years ago.
Category:
Regressions/Crashes
Description
Observation¶
[ 1108s] # Failed test 'test would have failed'
[ 1108s] # at ./t/01-test-utilities.t line 47.
[ 1108s]
[ 1108s] # Failed test 'manual termination via stop_service does not trigger _fail_and_exit'
[ 1108s] # at ./t/01-test-utilities.t line 58.
[ 1108s] # got: '1'
[ 1108s] # expected: '0'
[ 1108s] # Looks like you failed 2 tests of 5.
[ 1108s]
[ 1108s] # Failed test 'warnings in sub processes are fatal test failures'
[ 1108s] # at ./t/01-test-utilities.t line 59.
[ 1108s] # Looks like you failed 1 test of 1.
[ 1108s] ./t/01-test-utilities.t .....................................
[ 1108s] # Subtest: warnings in sub processes are fatal test failures
[ 1108s] # PID of test-process: 5808
[ 1108s] ok 1 - warning logged
[ 1108s] not ok 2 - test would have failed
[ 1108s] # PID of test-process: 5813
[ 1108s] ok 3 - message of test failure
[ 1108s] ok 4 - exit code of test failure is non-zero
[ 1108s] not ok 5 - manual termination via stop_service does not trigger _fail_and_exit
[ 1108s] 1..5
[ 1108s] not ok 1 - warnings in sub processes are fatal test failures
in https://build.opensuse.org/build/devel:openQA:TestGithub:PR-3456/openSUSE_Factory/x86_64/openQA/_log from https://github.com/os-autoinst/openQA/pull/3456
Checking reproducibility…
EDIT: happened again, e.g. in https://build.opensuse.org/package/live_build_log/devel:openQA/openQA/openSUSE_Factory/x86_64 even though at time of writing it was fine again
Acceptance criteria¶
- AC1: t/01-test-utilities.t does not fail in OBS checks reproducibly (either fixed or excluded)
Suggestions¶
- Wait until this happens again and take a look in the log for details
- Try to reproduce failure in test OBS project
- Fix or exclude
- Due date set to 2020-10-18
- Status changed from In Progress to Feedback
- Priority changed from Normal to Low
I don't see it in build.opensuse.org/project/show/devel:openQA
- Status changed from Feedback to Rejected
- Description updated (diff)
- Due date changed from 2020-10-18 to 2020-10-25
- Status changed from Rejected to Feedback
- Priority changed from Low to Normal
- Status changed from Feedback to Workable
- Assignee changed from okurz to mkittler
Failed again and found details in https://build.opensuse.org/public/build/devel:openQA:tested/openSUSE_Tumbleweed/x86_64/openQA/_log today:
[ 904s] # Failed test 'test would have failed'
[ 904s] # at ./t/01-test-utilities.t line 47.
[ 904s]
[ 904s] # Failed test 'manual termination via stop_service does not trigger _fail_and_exit'
[ 904s] # at ./t/01-test-utilities.t line 58.
[ 904s] # got: '1'
[ 904s] # expected: '0'
[ 904s] # Looks like you failed 2 tests of 5.
[ 904s]
[ 904s] # Failed test 'warnings in sub processes are fatal test failures'
[ 904s] # at ./t/01-test-utilities.t line 59.
[ 904s] # Looks like you failed 1 test of 1.
[ 905s] ./t/01-test-utilities.t .....................................
[ 905s] # Subtest: warnings in sub processes are fatal test failures
[ 905s] # PID of test-process: 5508
[ 905s] ok 1 - warning logged
[ 905s] not ok 2 - test would have failed
[ 905s] # PID of test-process: 5513
[ 905s] ok 3 - message of test failure
[ 905s] ok 4 - exit code of test failure is non-zero
[ 905s] not ok 5 - manual termination via stop_service does not trigger _fail_and_exit
[ 905s] 1..5
[ 905s] not ok 1 - warnings in sub processes are fatal test failures
[ 905s] 1..1
[ 905s] Dubious, test returned 1 (wstat 256, 0x100)
[ 905s] Failed 1/1 subtests
@mkittler do you think you could take a look into this?
- Status changed from Workable to In Progress
The PR is merged but unfortunately did not work, https://build.opensuse.org/public/build/devel:openQA/openSUSE_Tumbleweed/x86_64/openQA/_log running openQA-4.6.1603211536.4498c99a4 corresponding to https://github.com/os-autoinst/openQA/commit/4498c99a4376dd7a0e2a2a7ac0f7baf67112fc0c which is the commit of the above PR shows still:
[ 1128s]
[ 1128s] # Failed test 'test would have failed'
[ 1128s] # at ./t/01-test-utilities.t line 49.
[ 1128s]
[ 1128s] # Failed test 'manual termination via stop_service does not trigger _fail_and_exit'
[ 1128s] # at ./t/01-test-utilities.t line 61.
[ 1128s] # got: '1'
[ 1128s] # expected: '0'
[ 1128s] # Looks like you failed 2 tests of 5.
[ 1128s]
[ 1128s] # Failed test 'warnings in sub processes are fatal test failures'
[ 1128s] # at ./t/01-test-utilities.t line 62.
[ 1128s] # Looks like you failed 1 test of 1.
[ 1128s] ./t/01-test-utilities.t .....................................
[ 1128s] # Subtest: warnings in sub processes are fatal test failures
[ 1128s] # PID of test-process-1: 5818
[ 1128s] # waiting at most 70 seconds for SIGCHLD (sleep is supposed to be interrupted by SIGCHLD)
[ 1128s] ok 1 - warning logged
[ 1128s] not ok 2 - test would have failed
[ 1128s] # PID of test-process-2: 5877
[ 1128s] # waiting at most 70 seconds for SIGTERM (sleep is supposed to be interrupted by SIGTERM)
[ 1128s] ok 3 - message of test failure
[ 1128s] ok 4 - exit code of test failure is non-zero
[ 1128s] not ok 5 - manual termination via stop_service does not trigger _fail_and_exit
[ 1128s] 1..5
[ 1128s] not ok 1 - warnings in sub processes are fatal test failures
[ 1128s] 1..1
[ 1128s] Dubious, test returned 1 (wstat 256, 0x100)
[ 1128s] Failed 1/1 subtests
- Status changed from In Progress to New
- Assignee deleted (
mkittler)
Then I'm out of ideas. Not sure how to test the code. If we would mock everything it would be quite useless because I want to test whether the code actually works in accordance to the OS signal handling. Maybe just keep the test for local development (where I've never run into the error). Since you've already removed the test from OBS it shouldn't prevent our builds anymore at least.
- Status changed from New to Resolved
- Assignee set to okurz
I think it's ok like it is now. My PR is merged, the test is excluded from OBS checks so not failing there anymore but we still run the test fine within circleCI checks just fine.
Also available in: Atom
PDF