Project

General

Profile

Actions
Copy link

action #32977

closed

[functional][yast][y]Release notes content is not really checked

Added by mkravec about 6 years ago. Updated about 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Category:
Bugs in existing tests
Target version:
openQA Project - Milestone 15
Start date:
2018-03-09
Due date:
2018-03-27
% Done:

0%

Estimated time:
Difficulty:

Description

Release notes on sle15 have content from sles12-sp3 and test is passing.
Modify test to check for proper content, for example using assert_screen("release-notes-sles-".get_var('VERSION'));

Example of invalid test: http://openqa-apac1.suse.de/tests/372/modules/releasenotes/steps/5

Actions
Copy link
 #1

Updated by mkravec about 6 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by okurz about 6 years ago

  • Subject changed from Release notes content is not really checked to [functional]Release notes content is not really checked
  • Due date set to 2018-03-27
  • Target version set to Milestone 15

I recommend to not make ourselves rely on version specific needles. They are a lot of maintenance work and lead to other people distrusting openQA (or QA). However, what about simply creating workaround needles on top with a bugref?

Actions
Copy link
 #3

Updated by okurz about 6 years ago

  • Due date deleted (2018-03-27)
  • Assignee deleted (mloviska)
  • Target version changed from Milestone 15 to Milestone 17

sprint13 too full, just as well as M15 and M16

Actions
Copy link
 #4

Updated by okurz about 6 years ago

  • Subject changed from [functional]Release notes content is not really checked to [functional][yast][y]Release notes content is not really checked
Actions
Copy link
 #5

Updated by okurz about 6 years ago

  • Checklist item changed from [ ] Remove needles matching non-specific content, [ ] Add needles with specific SLES version in the tag, [ ] Modify release-notes test module to
  • Due date set to 2018-03-27
  • Status changed from New to Workable
  • Assignee set to okurz
  • Target version changed from Milestone 17 to Milestone 15

Actually I want to check if I can not just generate a error-detection needle with workaround property.

Actions
Copy link
 #6

Updated by okurz about 6 years ago

  • Status changed from Workable to Resolved

https://openqa.suse.de/tests/1551946#step/releasenotes/4 is showing the current, correct release notes. If there would be wrong release notes I would create an additional needle with workaround property to track it as soft-failure. This will also be monitored by openqa-review. I do not see the benefit in letting an additional scenario fail with a needle mismatch. We have enough of these and they don't help to build trust.

Actions
Copy link

Also available in: Atom PDF