Project

General

Profile

Actions

action #48392

closed

coordination #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule

[functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule

Added by okurz about 5 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
SUSE QA - Milestone 23
Start date:
2019-02-25
Due date:
2019-03-12
% Done:

0%

Estimated time:

Description

Acceptance criteria

  • AC1: research on the ways how to prevent breaking staging tests using automated checks

Suggestions

  • Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
  • Add a call with isotovideo _exit_after_schedule=1 to only evaluate the schedule and compare against a reference
  • Add instructions to update the reference file whenever a change to the staging tests is actually intended

Further details

It might sound counter-intuitive to use a blacklist for excluding test modules from staging tests (loadtest "…" unless is_staging;) and then define a whitelist for the test to check against but it might be a start towards #15132


Related issues 1 (0 open1 closed)

Related to QA - action #47441: [functional][u] test code feedback - integrate bots to test distribution on githubResolvedszarate

Actions
Actions #1

Updated by riafarov about 5 years ago

I suggest spike first

Actions #2

Updated by okurz about 5 years ago

sure, I'm ok with that.

Actions #3

Updated by riafarov about 5 years ago

  • Subject changed from [functional][y] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule to [functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
  • Description updated (diff)
Actions #4

Updated by ybonatakis almost 5 years ago

  • Status changed from Workable to In Progress
  • Assignee set to ybonatakis
Actions #5

Updated by riafarov almost 5 years ago

Summary of our discussion:
1) We need to research what is feasible to do
2) Problem we are trying to solve is that we break staging tests (e.g. https://openqa.suse.de/tests/overview?distri=sle&version=12-SP5&build=Y.5.19&groupid=50)
these happens for two major reasons: a) we unintentionally change schedule which affects staging b) we change test modules which are executed on staging and will break (e.g. missing needle, or patch for SLE 12 breaks SLE 15)
3) We should check if that's possible to avoid such scenarios using some automated checks

Ideas we got so far:
1) Move all those tests to declarative scheduling (See https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6993) and validate if those got changed, this still doesn't solve problem if test module gets changed, but we know exact list of those
2) Evaluate schedule using isotovideo and do same checks as in 1).

To research:
1) If travis allows to add comments to the PR (should be feasible with scripts), otherwise we should fail, so person takes a look and merge confirms that changes are tested
2) If travis can parse PR message to detect that VR for staging was provided

Actions #6

Updated by okurz almost 5 years ago

  • Related to action #47441: [functional][u] test code feedback - integrate bots to test distribution on github added
Actions #7

Updated by riafarov almost 5 years ago

  • Status changed from In Progress to Resolved

riafarov will create followup tickets.

Actions

Also available in: Atom PDF