[functional][u][epic] Ensure consistence and semantic of INSTALLONLY tests
At some point the setting INSTALLONLY=1 started to be used to avoid the schedule of tests module that tests the installed system.
Since the tests in os-autoinst-distri-opensuse are used by a lot of projects, the schedule in main.pm started to limit the schedule of different.
Some test suites intended to perform an installation, but scheduling a different set of modules for the installed system. The complexity of main.pm made them to look for an easy way of setting the schedule. Using the setting INSTALLONLY=1 to avoid the schedule of common post installation tests, and explicitly adding other modules.
This inconsistency has been taking for a long time. We started to have some problems to maintain and adapt those test.
- AC: All test suites that use the setting INSTALLONLY=1 should perform an install only scenario.
- A Could we use the feature SCHEDULE of os-autoinst in each test suite to explicitly set the list of modules?
- Pros: main.pm doesn't need to be used. Schedule is clear to test developer before starting a job.
- Cons: Schedule cannot be set programmatically.
- B Instead of using INSTALLONLY to avoid the schedule of post installation tests, using a setting POST_INSTALLATION which defines the set of modules to be schedule ["transactional_server", "extra_tests_textmode", ...]
- Pros: We can adapt the schedule programmatically. We don't need to continually check for special cases in main.pm
- Cons: To know the schedule, the test developer needs to follow the logic of main.pm.
- C Adapt all existing tests that are abusing the setting INSTALLONLY to don't use that setting.
- Pros: We can adapt the schedule programmatically.
- Cons: We need to always check for special cases in main.pm. To know the schedule, the test developer needs to follow the logic of main.pm.
- D Using YML schedule
#11 Updated by szarate about 1 year ago
See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html