Project

General

Profile

Actions

coordination #37958

closed

[epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule

Added by okurz over 6 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2019-02-25
Due date:
2020-05-19
% Done:

100%

Estimated time:
(Total: 16.00 h)

Description

Motivation

Sometimes when a new test module is introduced – for example by new contributors which are mainly starting with new test modules and not changing existing ones – we forget in code review to think about if this module should be excluded from staging tests causing staging test failures, some frustration on side of SLE release managers and subsequently requests like #37826 which we should try to prevent because they make our life miserable. We should think about a way to prevent staging test breakage by unintentional introduction of new test modules into staging tests, e.g. using self-tests in os-autoinst-distri-opensuse. This test can have the additional benefit that we actually evaluate the test code before we deploy it to production where previously we had "trivial" failures but they could only be catched in production.

Acceptance criteria

  • AC1: travis CI tests on PRs fail if new test modules are introduced to staging tests unintentionally

Suggestions

  • Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
  • Add a call with isotovideo _exit_after_schedule=1 to only evaluate the schedule and compare against a reference
  • Add instructions to update the reference file whenever a change to the staging tests is actually intended

Further details

It might sound counter-intuitive to use a blacklist for excluding test modules from staging tests (loadtest "…" unless is_staging;) and then define a whitelist for the test to check against but it might be a start towards #15132


Subtasks 5 (0 open5 closed)

action #48389: self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideoResolvedokurz2019-02-25

Actions
action #48392: [functional][y][spike/research] self-tests in os-autoinst-distri-opensuse for impact on staging test scheduleResolvedybonatakis2019-02-252019-03-12

Actions
action #49064: [functional][y] Move SLES staging tests to use declarative scheduling mechanismResolvedybonatakis2019-03-122019-06-04

Actions
action #54311: [functional][y] Implement hint to trigger VR for staging in case modules used there are changed in the PRResolvedoorlov2019-07-162019-09-10

Actions
action #65432: [functional][y] test if schedules and test data files in the job group do exist in the os-autoinst/os-autoinst-distri-opensuse repoResolvedJERiveraMoya2020-04-082020-05-19

Actions

Related issues 2 (1 open1 closed)

Related to openQA Tests - coordination #15132: [saga][epic] Better structure of test plans in main.pmBlockedokurz2018-11-20

Actions
Copied from openQA Tests - action #36523: [functional][y][medium] self-tests for https://github.com/os-autoinst/os-autoinst-needles-opensuse/Resolvednicksinger2018-05-252018-07-17

Actions
Actions #1

Updated by okurz over 6 years ago

  • Copied from action #36523: [functional][y][medium] self-tests for https://github.com/os-autoinst/os-autoinst-needles-opensuse/ added
Actions #2

Updated by okurz over 6 years ago

Actions #4

Updated by okurz about 6 years ago

  • Subject changed from [functional] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule to [functional][y] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
  • Due date set to 2018-10-09
  • Target version changed from Milestone 18 to Milestone 19
Actions #5

Updated by okurz about 6 years ago

@y-team I would be ok to move to u-team if you prefer, up to you

Actions #6

Updated by riafarov about 6 years ago

  • Assignee set to riafarov
  • Estimated time set to 13.00 h

I will clarify priority with okurz

Actions #7

Updated by riafarov about 6 years ago

  • Due date deleted (2018-10-09)

Do not have capacity in the current sprint.

Actions #8

Updated by okurz about 6 years ago

  • Due date set to 2018-10-23
  • Target version changed from Milestone 19 to Milestone 20
Actions #9

Updated by okurz about 6 years ago

  • Due date deleted (2018-10-23)
  • Assignee changed from riafarov to okurz
  • Target version changed from Milestone 20 to future
  • Estimated time deleted (13.00 h)

Discussed in the y-team's refinement, need to rethink by now because we want to improve test schedule structure as a whole anyway. I will try to split out e.g. "simple isotovideo run" and "more explicit staging tests"

Actions #10

Updated by okurz almost 6 years ago

  • Subject changed from [functional][y] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule to [functional][y][epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
  • Status changed from Workable to Blocked
  • Difficulty deleted (medium)
Actions #11

Updated by okurz over 5 years ago

  • Assignee changed from okurz to riafarov

to track as stand-in as discussed

Actions #12

Updated by riafarov over 5 years ago

  • Due date changed from 2019-04-09 to 2019-03-12

due to changes in a related task

Actions #13

Updated by riafarov over 5 years ago

  • Due date changed from 2019-03-12 to 2019-04-23

due to changes in a related task

Actions #14

Updated by riafarov over 5 years ago

  • Due date changed from 2019-04-23 to 2019-05-07

due to changes in a related task

Actions #15

Updated by JERiveraMoya over 5 years ago

  • Due date changed from 2019-05-07 to 2019-05-21

due to changes in a related task

Actions #16

Updated by JERiveraMoya over 5 years ago

  • Due date changed from 2019-05-21 to 2019-06-04

due to changes in a related task

Actions #17

Updated by riafarov over 5 years ago

  • Due date changed from 2019-06-04 to 2019-07-16

due to changes in a related task

Actions #18

Updated by JERiveraMoya over 5 years ago

  • Due date changed from 2019-08-13 to 2019-08-27

due to changes in a related task

Actions #19

Updated by riafarov over 5 years ago

  • Due date changed from 2019-08-27 to 2019-09-10

due to changes in a related task

Actions #20

Updated by oorlov about 5 years ago

  • Due date changed from 2019-11-05 to 2019-11-19

due to changes in a related task

Actions #21

Updated by riafarov almost 5 years ago

  • Due date changed from 2020-01-28 to 2020-02-11

due to changes in a related task

Actions #22

Updated by riafarov almost 5 years ago

  • Due date changed from 2020-02-11 to 2020-03-24

due to changes in a related task

Actions #23

Updated by riafarov over 4 years ago

  • Due date changed from 2020-04-21 to 2020-05-05

due to changes in a related task

Actions #24

Updated by riafarov over 4 years ago

  • Due date changed from 2020-05-05 to 2020-05-19

due to changes in a related task: #65432

Actions #25

Updated by riafarov over 4 years ago

  • Project changed from openQA Tests to openQA Project
  • Subject changed from [functional][y][epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule to [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
  • Category deleted (Enhancement to existing tests)
  • Status changed from Blocked to New
  • Assignee deleted (riafarov)

Un-assigning myself as last ticket remaining is for the tools team.

Actions #26

Updated by okurz over 4 years ago

  • Category set to Feature requests
Actions #27

Updated by szarate about 4 years ago

  • Tracker changed from action to coordination
Actions #29

Updated by okurz about 3 years ago

  • Status changed from New to Resolved
  • Assignee set to okurz
  • Target version changed from future to Ready

All subtickets resolved. Latest one was #48389 with which we can fully cover AC1 and call this "Resolved"

Actions

Also available in: Atom PDF