Project

General

Profile

action #38621

openQA Project - coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues

[functional][y] test fails in welcome - "Module is not signed with expected PKCS#7 message" (bsc#1093659) - Use serial exception catching feature from openQA to make sure the jobs reference the bug, e.g. as label

Added by okurz almost 3 years ago. Updated 12 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Target version:
SUSE QA tests - Milestone 18
Start date:
2018-05-23
Due date:
% Done:

0%

Estimated time:
8.00 h
Difficulty:

Description

Observation

openQA test in scenario sle-12-SP4-Server-DVD-Staging:B-x86_64-ext4@uefi-staging fails in
welcome
on https://bugzilla.suse.com/show_bug.cgi?id=1093659

Acceptance criteria

  • AC1: All jobs hitting bsc#1093659 will not need any work from manual reviewers

Suggestions

  • Work on #36448 first
  • Abort the test as soon as this soft-fail has been hit (or find a way how a job label can be created)
  • Update documentation about feature detecting serial error

Further details

Always latest result in this scenario: latest

The "automatic label" in the acceptance criteria could be either (1) just soft-failing jobs showing up with the bug reference on http://s.qa.suse.de/test-status or (2) an automatic comment in the openQA webui. Most likely (1) would be easier but means we have to abort the whole test run as soon as we found the issue and recorded the soft failure to prevent any subsequent test module failing the module as "fail" overrides "softfail".


Related issues

Related to openQA Tests - action #39059: [sle][functional][y] detect "openSUSE sucks bug" about btrfs balance and record_soft_fail (was: yast2_gui tests modules as application could not start up)Resolved2018-08-012018-10-09

Blocked by openQA Tests - action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labelsResolved2018-05-232018-08-14

History

#1 Updated by okurz almost 3 years ago

  • Copied from action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels added

#2 Updated by okurz almost 3 years ago

  • Copied from deleted (action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels)

#3 Updated by okurz almost 3 years ago

  • Related to action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels added

#4 Updated by okurz almost 3 years ago

  • Related to deleted (action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels)

#5 Updated by okurz almost 3 years ago

  • Blocked by action #36448: [functional][y] Fix serial exception catching feature from openQA to be able to use for job labels added

#6 Updated by okurz almost 3 years ago

  • Status changed from New to Workable

#7 Updated by riafarov almost 3 years ago

  • Assignee set to riafarov

#8 Updated by riafarov almost 3 years ago

  • Status changed from Workable to In Progress

#9 Updated by riafarov almost 3 years ago

  • Due date changed from 2018-07-31 to 2018-08-14

#10 Updated by riafarov almost 3 years ago

  • Estimated time set to 8.00 h

#11 Updated by riafarov almost 3 years ago

  • Status changed from In Progress to Feedback

#12 Updated by okurz almost 3 years ago

  • Related to action #39059: [sle][functional][y] detect "openSUSE sucks bug" about btrfs balance and record_soft_fail (was: yast2_gui tests modules as application could not start up) added

#14 Updated by riafarov almost 3 years ago

I guess it's about bsc#1093659 bsc#1093659 is verified fixed, so no further action is required.

#15 Updated by riafarov almost 3 years ago

  • Due date changed from 2018-08-14 to 2018-08-28

#16 Updated by riafarov almost 3 years ago

Missing part: if test fails before (e.g. cause of assert_screen failure or wait_serial), then test won't have serial parsed, and hence carry over won't work. Checking for a solution to that.

#17 Updated by okurz almost 3 years ago

  • Related to coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues added

#18 Updated by okurz almost 3 years ago

  • Related to deleted (coordination #39719: [saga][epic] Detection of "known failures" for stable tests, easy test results review and easy tracking of known issues)

#19 Updated by okurz almost 3 years ago

  • Parent task set to #39719

#20 Updated by riafarov almost 3 years ago

  • Status changed from Feedback to In Progress

#21 Updated by riafarov almost 3 years ago

https://github.com/os-autoinst/os-autoinst/pull/1014 changes in os-autoinst to detect serial failures even if test failed before

#22 Updated by riafarov almost 3 years ago

  • Status changed from In Progress to Feedback

Not to rush with solution and get implementation which matches bigger vision, I will stop with changes in os-autoinst to make original idea work better, and then continue after having discussion with at least all stakeholders.

#23 Updated by riafarov almost 3 years ago

  • Status changed from Feedback to Resolved

We've discussed with szarate and he also has concerns about automated comment, so I will resolve this ticket, as carryover works now even test module failed before. On top bug is resolved.
We can continue with bigger vision in epic kindly created by okurz: https://progress.opensuse.org/issues/39719

#24 Updated by okurz 12 months ago

  • Due date deleted (2018-08-28)

Also available in: Atom PDF