action #10784
opentest framework for openQA test distributions
0%
Description
user story¶
As a test distribution developer using openQA I want to execute my tests in a safe and fast environment to catch early mistakes before executing anything on a productive server or real hardware
acceptance criteria¶
- "os-autoinst-distri-example" or "os-autoinst-distri-opensuse" can be executed locally without relying on any real worker
tasks¶
- maybe a simple first step could be to check importability of main.pm as is done by isotovideo
- mock the testapi to execute a "happy path" of a test, e.g. assert_screen always succeeds and such
- provide means to select the mock as either a "null backend" or a mode executing the test distribution locally, e.g. "--dry-run"
- provide a way to configure this mode to simulate different real backends or machine types
- optional: provide "failing assert_screen" and such to cover more execution paths
further details¶
benefit: By using this we can give a first hint to test distribution developers which code path they are touching, which variables are involved, etc. Another use case is the detection of unused test modules based on variable sets.
Updated by okurz about 8 years ago
As discussed with coolo and others, the "null backend" is the way to go, e.g. call "isotovideo" and also encode empty videos and such.
Updated by okurz over 7 years ago
- Category set to Feature requests
- Status changed from New to In Progress
some preliminary work on null backend has been done: https://github.com/okurz/os-autoinst/tree/feature/null_backend
Updated by coolo over 6 years ago
- Status changed from In Progress to New
- Target version set to future
Page not found
Updated by okurz over 6 years ago
In the aforementioned branch I tried to develop a "null backend" and actually succeeded to start tests. Then I realized that we can't really reach anywhere useful because the tests are too much tied in to the instrumented SUT behaviour. E.g. when mocking every testapi call to "succeed" the test would still fail very early because the next "check_screen" or "wait_serial" mock-succeeds and leading the test flow into a branch where we expect something else, e.g. with match_has_tag which in turn also mock-succeeds and then gets stuck or fails because the mocked SUT - which isn't really there for a null backend - simply does not behave like the real thing. I don't know how to continue from there.
Updated by coolo over 6 years ago
yeah, we would need to provide records of real tests to replay. Which is quite a challenge
Updated by okurz over 6 years ago
Correct. I am thinking of getting the actions and expected results from the logfile, store these without all the not necessary debug output in the tests framework as "recordings" and play back these checking for regressions (i.e. any change is considered a test failure by default).
Updated by okurz almost 6 years ago
- Target version changed from future to future
Updated by okurz almost 5 years ago
- Related to action #48389: self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo added
Updated by okurz almost 4 years ago
- Status changed from New to Workable
- Priority changed from Normal to Low
I added a "null" backend to os-autoinst with https://github.com/os-autoinst/os-autoinst/commit/added079de81e6d22080e470cc3c9fdd604c63ee which can be used as a first step.
Updated by okurz about 1 month ago
- Target version changed from Tools - Next to future