QA (public) » openQA Project (public)

Motivation¶

Sometimes when a new test module is introduced – for example by new contributors which are mainly starting with new test modules and not changing existing ones – we forget in code review to think about if this module should be excluded from staging tests causing staging test failures, some frustration on side of SLE release managers and subsequently requests like #37826 which we should try to prevent because they make our life miserable. We should think about a way to prevent staging test breakage by unintentional introduction of new test modules into staging tests, e.g. using self-tests in os-autoinst-distri-opensuse. This test can have the additional benefit that we actually evaluate the test code before we deploy it to production where previously we had "trivial" failures but they could only be catched in production.

Acceptance criteria¶

• AC1: travis CI tests on PRs fail if new test modules are introduced to staging tests unintentionally

Suggestions¶

• Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
• Add a call with isotovideo _exit_after_schedule=1 to only evaluate the schedule and compare against a reference
• Add instructions to update the reference file whenever a change to the staging tests is actually intended

Further details¶

It might sound counter-intuitive to use a blacklist for excluding test modules from staging tests (loadtest "…" unless is_staging;) and then define a whitelist for the test to check against but it might be a start towards #15132