[userspace] split qa_run.pm
Now if the qa_run.pm related testcases fails not in what they tested, it'll hard to judge where it fails with. To split qa_run.pm into process is a solution for that.
Split it in to:
- Booting the OS
- Preparation test package
- Run test process
- Upload log and analysis process It's a basic idea, the real change will depends. But the target is fixed to solve the debug problem.
#2 Updated by yosun over 4 years ago
From this related issue you'll know what I mean: https://progress.opensuse.org/issues/17872
It really don't have any useful information in log, if it fails in boot part. What we need to do is to let reviewer more easy to tell if it's a fail in our testsuite. If not, the issue will keep bother us a lot in the future. Oliver and Richard also complained this in last week.
#8 Updated by riafarov over 4 years ago
yosun, please take a look on this PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3695
#10 Updated by yosun over 4 years ago
riafarov, reviewed, and copy some explain of this task here.
Actually the task of split qa_run is for review result more easy, and it's more than just move booting steps out. But you picked the best way to modify it in this PR. Fails before booting success is not failed by what it test, it fails because of OS install or system booting. --> what solved by this PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3695
And fails from booting success to testsuite install finish, is another period not failed by what it test, it fails because of network issue or module issue --> this part I'll do it inside of qa_run.pm
Then fails from testsuite install finish to testsuite start running is because of testsuite internal issue. It means qa_test_* need to be fixed. --> also for me todo in split qa_run.pm
And fails from testsuite run to run finish is what we expected product fail/bug.
#16 Updated by yosun almost 4 years ago
I think this task have been finished by Calen, https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4132/files