action #38621
closed
openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues
[functional][y] test fails in welcome - "Module is not signed with expected PKCS#7 message" (bsc#1093659) - Use serial exception catching feature from openQA to make sure the jobs reference the bug, e.g. as label
Added by okurz almost 6 years ago.
Updated almost 4 years ago.
Category:
Bugs in existing tests
Target version:
SUSE QA - Milestone 18
Description
Observation¶
openQA test in scenario sle-12-SP4-Server-DVD-Staging:B-x86_64-ext4@uefi-staging fails in
welcome
on https://bugzilla.suse.com/show_bug.cgi?id=1093659
Acceptance criteria¶
- AC1: All jobs hitting bsc#1093659 will not need any work from manual reviewers
Suggestions¶
- Work on #36448 first
- Abort the test as soon as this soft-fail has been hit (or find a way how a job label can be created)
- Update documentation about feature detecting serial error
Further details¶
Always latest result in this scenario: latest
The "automatic label" in the acceptance criteria could be either (1) just soft-failing jobs showing up with the bug reference on http://s.qa.suse.de/test-status or (2) an automatic comment in the openQA webui. Most likely (1) would be easier but means we have to abort the whole test run as soon as we found the issue and recorded the soft failure to prevent any subsequent test module failing the module as "fail" overrides "softfail".
- Copied from action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels added
- Copied from deleted (action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels)
- Related to action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels added
- Related to deleted (action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels)
- Blocked by action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels added
- Status changed from New to Workable
- Status changed from Workable to In Progress
- Due date changed from 2018-07-31 to 2018-08-14
- Estimated time set to 8.00 h
- Status changed from In Progress to Feedback
- Related to action #39059: [sle][functional][y] detect "openSUSE sucks bug" about btrfs balance and record_soft_fail (was: yast2_gui tests modules as application could not start up) added
I guess it's about bsc#1093659 bsc#1093659 is verified fixed, so no further action is required.
- Due date changed from 2018-08-14 to 2018-08-28
Missing part: if test fails before (e.g. cause of assert_screen failure or wait_serial), then test won't have serial parsed, and hence carry over won't work. Checking for a solution to that.
- Related to coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added
- Related to deleted (coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues)
- Parent task set to #39719
- Status changed from Feedback to In Progress
- Status changed from In Progress to Feedback
Not to rush with solution and get implementation which matches bigger vision, I will stop with changes in os-autoinst to make original idea work better, and then continue after having discussion with at least all stakeholders.
- Status changed from Feedback to Resolved
We've discussed with @szarate and he also has concerns about automated comment, so I will resolve this ticket, as carryover works now even test module failed before. On top bug is resolved.
We can continue with bigger vision in epic kindly created by @okurz: https://progress.opensuse.org/issues/39719
- Due date deleted (
2018-08-28)
Also available in: Atom
PDF