action #48392
closed
coordination #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
[functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
Added by okurz almost 6 years ago.
Updated over 4 years ago.
Target version:
SUSE QA (private) - Milestone 23
Description
Acceptance criteria¶
- AC1: research on the ways how to prevent breaking staging tests using automated checks
Suggestions¶
- Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
- Add a call with
isotovideo _exit_after_schedule=1 to only evaluate the schedule and compare against a reference
- Add instructions to update the reference file whenever a change to the staging tests is actually intended
Further details¶
It might sound counter-intuitive to use a blacklist for excluding test modules from staging tests (loadtest "…" unless is_staging;) and then define a whitelist for the test to check against but it might be a start towards #15132
- Subject changed from [functional][y] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule to [functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
- Description updated (diff)
- Status changed from Workable to In Progress
- Assignee set to ybonatakis
Summary of our discussion:
1) We need to research what is feasible to do
2) Problem we are trying to solve is that we break staging tests (e.g. https://openqa.suse.de/tests/overview?distri=sle&version=12-SP5&build=Y.5.19&groupid=50)
these happens for two major reasons: a) we unintentionally change schedule which affects staging b) we change test modules which are executed on staging and will break (e.g. missing needle, or patch for SLE 12 breaks SLE 15)
3) We should check if that's possible to avoid such scenarios using some automated checks
Ideas we got so far:
1) Move all those tests to declarative scheduling (See https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6993) and validate if those got changed, this still doesn't solve problem if test module gets changed, but we know exact list of those
2) Evaluate schedule using isotovideo and do same checks as in 1).
To research:
1) If travis allows to add comments to the PR (should be feasible with scripts), otherwise we should fail, so person takes a look and merge confirms that changes are tested
2) If travis can parse PR message to detect that VR for staging was provided
- Related to action #47441: [functional][u] test code feedback - integrate bots to test distribution on github added
- Status changed from In Progress to Resolved
riafarov will create followup tickets.
Also available in: Atom
PDF