action #37958

[functional][y][epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule

Added by okurz over 1 year ago. Updated 17 days ago.

Status:BlockedStart date:25/02/2019
Priority:NormalDue date:24/03/2020
Assignee:riafarov% Done:

67%

Category:Enhancement to existing testsEstimated time:18.00 hours
Target version:QA - future
Difficulty:
Duration: 282

Description

Motivation

Sometimes when a new test module is introduced – for example by new contributors which are mainly starting with new test modules and not changing existing ones – we forget in code review to think about if this module should be excluded from staging tests causing staging test failures, some frustration on side of SLE release managers and subsequently requests like #37826 which we should try to prevent because they make our life miserable. We should think about a way to prevent staging test breakage by unintentional introduction of new test modules into staging tests, e.g. using self-tests in os-autoinst-distri-opensuse. This test can have the additional benefit that we actually evaluate the test code before we deploy it to production where previously we had "trivial" failures but they could only be catched in production.

Acceptance criteria

  • AC1: travis CI tests on PRs fail if new test modules are introduced to staging tests unintentionally

Suggestions

  • Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
  • Add a call with isotovideo _exit_after_schedule=1 to only evaluate the schedule and compare against a reference
  • Add instructions to update the reference file whenever a change to the staging tests is actually intended

Further details

It might sound counter-intuitive to use a blacklist for excluding test modules from staging tests (loadtest "…" unless is_staging;) and then define a whitelist for the test to check against but it might be a start towards #15132


Subtasks

action #48389: [tools] self-tests in os-autoinst-distri-opensuse executi...Workable

action #48392: [functional][y][spike/research] self-tests in os-autoinst...Resolvedybonatakis

action #49064: [functional][y] Move SLES staging tests to use declarativ...Resolvedybonatakis

action #54311: [functional][y] Implement hint to trigger VR for staging ...Resolvedoorlov

action #59115: [y][functional][timeboxed: 16h] Investigate the best way ...Resolvedriafarov


Related issues

Related to openQA Tests - action #15132: [epic] Better structure of test plans in main.pm Blocked 20/11/2018 27/11/2020
Copied from openQA Tests - action #36523: [functional][y][medium] self-tests for https://github.com... Resolved 25/05/2018 17/07/2018

History

#1 Updated by okurz over 1 year ago

  • Copied from action #36523: [functional][y][medium] self-tests for https://github.com/os-autoinst/os-autoinst-needles-opensuse/ added

#2 Updated by okurz over 1 year ago

  • Related to action #15132: [epic] Better structure of test plans in main.pm added

#4 Updated by okurz over 1 year ago

  • Subject changed from [functional] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule to [functional][y] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
  • Due date set to 09/10/2018
  • Target version changed from Milestone 18 to Milestone 19

#5 Updated by okurz over 1 year ago

@y-team I would be ok to move to u-team if you prefer, up to you

#6 Updated by riafarov over 1 year ago

  • Assignee set to riafarov
  • Estimated time set to 13.00

I will clarify priority with okurz

#7 Updated by riafarov over 1 year ago

  • Due date deleted (09/10/2018)

Do not have capacity in the current sprint.

#8 Updated by okurz over 1 year ago

  • Due date set to 23/10/2018
  • Target version changed from Milestone 19 to Milestone 20

#9 Updated by okurz over 1 year ago

  • Due date deleted (23/10/2018)
  • Assignee changed from riafarov to okurz
  • Target version changed from Milestone 20 to future
  • Estimated time deleted (13.00)

Discussed in the y-team's refinement, need to rethink by now because we want to improve test schedule structure as a whole anyway. I will try to split out e.g. "simple isotovideo run" and "more explicit staging tests"

#10 Updated by okurz about 1 year ago

  • Subject changed from [functional][y] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule to [functional][y][epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
  • Status changed from Workable to Blocked
  • Difficulty deleted (medium)

#11 Updated by okurz 12 months ago

  • Assignee changed from okurz to riafarov

to track as stand-in as discussed

#12 Updated by riafarov 11 months ago

  • Due date changed from 09/04/2019 to 12/03/2019

due to changes in a related task

#13 Updated by riafarov 11 months ago

  • Due date changed from 12/03/2019 to 23/04/2019

due to changes in a related task

#14 Updated by riafarov 10 months ago

  • Due date changed from 23/04/2019 to 07/05/2019

due to changes in a related task

#15 Updated by JERiveraMoya 10 months ago

  • Due date changed from 07/05/2019 to 21/05/2019

due to changes in a related task

#16 Updated by JERiveraMoya 9 months ago

  • Due date changed from 21/05/2019 to 04/06/2019

due to changes in a related task

#17 Updated by riafarov 9 months ago

  • Due date changed from 04/06/2019 to 16/07/2019

due to changes in a related task

#18 Updated by JERiveraMoya 7 months ago

  • Due date changed from 13/08/2019 to 27/08/2019

due to changes in a related task

#19 Updated by riafarov 7 months ago

  • Due date changed from 27/08/2019 to 10/09/2019

due to changes in a related task

#20 Updated by oorlov 4 months ago

  • Due date changed from 05/11/2019 to 19/11/2019

due to changes in a related task

#21 Updated by riafarov 29 days ago

  • Due date changed from 28/01/2020 to 11/02/2020

due to changes in a related task

#22 Updated by riafarov 17 days ago

  • Due date changed from 11/02/2020 to 24/03/2020

due to changes in a related task

Also available in: Atom PDF