Project

General

Profile

action #48392

action #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule

[functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule

Added by okurz over 1 year ago. Updated 14 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
SUSE QA tests - Milestone 23
Start date:
2019-02-25
Due date:
2019-03-12
% Done:

0%

Estimated time:
Difficulty:
Duration: 12

Description

Acceptance criteria

  • AC1: research on the ways how to prevent breaking staging tests using automated checks

Suggestions

  • Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
  • Add a call with isotovideo _exit_after_schedule=1 to only evaluate the schedule and compare against a reference
  • Add instructions to update the reference file whenever a change to the staging tests is actually intended

Further details

It might sound counter-intuitive to use a blacklist for excluding test modules from staging tests (loadtest "…" unless is_staging;) and then define a whitelist for the test to check against but it might be a start towards #15132


Related issues

Related to QA - action #47441: [functional][u] test code feedback - integrate bots to test distribution on githubResolved

History

#1 Updated by riafarov over 1 year ago

I suggest spike first

#2 Updated by okurz over 1 year ago

sure, I'm ok with that.

#3 Updated by riafarov over 1 year ago

  • Subject changed from [functional][y] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule to [functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
  • Description updated (diff)

#4 Updated by ybonatakis about 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to ybonatakis

#5 Updated by riafarov about 1 year ago

Summary of our discussion:
1) We need to research what is feasible to do
2) Problem we are trying to solve is that we break staging tests (e.g. https://openqa.suse.de/tests/overview?distri=sle&version=12-SP5&build=Y.5.19&groupid=50)
these happens for two major reasons: a) we unintentionally change schedule which affects staging b) we change test modules which are executed on staging and will break (e.g. missing needle, or patch for SLE 12 breaks SLE 15)
3) We should check if that's possible to avoid such scenarios using some automated checks

Ideas we got so far:
1) Move all those tests to declarative scheduling (See https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6993) and validate if those got changed, this still doesn't solve problem if test module gets changed, but we know exact list of those
2) Evaluate schedule using isotovideo and do same checks as in 1).

To research:
1) If travis allows to add comments to the PR (should be feasible with scripts), otherwise we should fail, so person takes a look and merge confirms that changes are tested
2) If travis can parse PR message to detect that VR for staging was provided

#6 Updated by okurz about 1 year ago

  • Related to action #47441: [functional][u] test code feedback - integrate bots to test distribution on github added

#7 Updated by riafarov about 1 year ago

  • Status changed from In Progress to Resolved

riafarov will create followup tickets.

Also available in: Atom PDF