Project

General

Profile

coordination #51362

[functional][u][epic] Ensure consistence and semantic of INSTALLONLY tests

Added by SLindoMansilla over 2 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Target version:
SUSE QA - Milestone 31
Start date:
2019-05-14
Due date:
% Done:

100%

Estimated time:
(Total: 42.00 h)
Difficulty:

Description

Motivation

At some point the setting INSTALLONLY=1 started to be used to avoid the schedule of tests module that tests the installed system.
Since the tests in os-autoinst-distri-opensuse are used by a lot of projects, the schedule in main.pm started to limit the schedule of different.
Some test suites intended to perform an installation, but scheduling a different set of modules for the installed system. The complexity of main.pm made them to look for an easy way of setting the schedule. Using the setting INSTALLONLY=1 to avoid the schedule of common post installation tests, and explicitly adding other modules.
This inconsistency has been taking for a long time. We started to have some problems to maintain and adapt those test.

Acceptance criteria

  • AC: All test suites that use the setting INSTALLONLY=1 should perform an install only scenario.

Suggestions

  • A Could we use the feature SCHEDULE of os-autoinst in each test suite to explicitly set the list of modules?
    • Pros: main.pm doesn't need to be used. Schedule is clear to test developer before starting a job.
    • Cons: Schedule cannot be set programmatically.
  • B Instead of using INSTALLONLY to avoid the schedule of post installation tests, using a setting POST_INSTALLATION which defines the set of modules to be schedule ["transactional_server", "extra_tests_textmode", ...]
    • Pros: We can adapt the schedule programmatically. We don't need to continually check for special cases in main.pm
    • Cons: To know the schedule, the test developer needs to follow the logic of main.pm.
  • C Adapt all existing tests that are abusing the setting INSTALLONLY to don't use that setting.
    • Pros: We can adapt the schedule programmatically.
    • Cons: We need to always check for special cases in main.pm. To know the schedule, the test developer needs to follow the logic of main.pm.
  • D Using YML schedule

Subtasks

action #51470: [functional][u] Research about usage of YML schedule and POST_INSTALLATION to get rid of misused INSTALLONLY settingResolvedmgriessmeier

History

#1 Updated by SLindoMansilla over 2 years ago

  • Subject changed from [functional][u] Ensure consistence and semantic of INSTALLONLY tests to [functional][u][epic] Ensure consistence and semantic of INSTALLONLY tests

#2 Updated by SLindoMansilla over 2 years ago

  • Description updated (diff)

Suggestion A not valid because it needs to be set for each test suite. YML schedule should be used instead. See #51470

#3 Updated by SLindoMansilla over 2 years ago

  • Description updated (diff)

Suggestion C was rejected by the team during retrospective meeting.

#4 Updated by okurz over 2 years ago

  • Status changed from New to Blocked
  • Assignee set to SLindoMansilla
  • Target version set to Milestone 26

#5 Updated by okurz over 2 years ago

  • Target version changed from Milestone 26 to Milestone 27

#6 Updated by mgriessmeier about 2 years ago

  • Assignee changed from SLindoMansilla to mgriessmeier

#7 Updated by mgriessmeier about 2 years ago

  • Target version changed from Milestone 27 to Milestone 28

#8 Updated by mgriessmeier almost 2 years ago

  • Target version changed from Milestone 28 to Milestone 31

#9 Updated by SLindoMansilla over 1 year ago

  • Status changed from Blocked to Resolved

QSF-U already used YAML schedule, which ignores the setting INSTALLONLY

#10 Updated by szarate about 1 year ago

  • Tracker changed from action to coordination
  • Difficulty deleted (hard)

Also available in: Atom PDF