action #30775
closed[functional][y] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed
0%
Description
Currently load_snapshot is not available from test code , and only condition which trigger it is test failure .
But there are cases where we need to reset system to previous Milestone even if module pass.
We have limitations on bare metal, and to proceed here, we can ignore it.
After discussion with Anton, he is fine to start in a conservative way, so that test module should have flag set and we don't change any behavior of the test execution with this feature.
Suggestion¶
Implement additional tests flag "force_reset" which will force openQA to revert to previous snapshot no matter if module pass or fail
Acceptance criteria¶
- Test module actions can be cleaned up by reverting SUT state to the last milestone snapshot
Updated by okurz almost 7 years ago
- Category set to Feature requests
I wonder what is the use case. Is this another idea to speed up tests based on snapshots? But I thought you used btrfs snapshots for this?
Updated by coolo almost 7 years ago
- Target version set to future
I already mentioned when the issue came up last time: this would be specific to the few backends with snapshots available and as such I'm not supporting it.
Updated by asmorodskyi almost 7 years ago
- Currently in openqa.suse.de has 192069 jobs and query "select count(*) from jobs where machine in (select name from machines where backend='qemu');" gives us that 166332 are using qemu backend which is 86% of all jobs. So it is not just one of others it is most used one
- I am not asking to create some totally new functionality which will require a lot of effort to implement and support. No matter what you said about limited support among backends we already having functionality of creating snapshots and rolling back in case of failure , so this ticket is low hanging fruit which will not harm for 14% of our jobs but will give new abilities to other 86%
Updated by coolo almost 7 years ago
It fosters writing backend specific tests - this is harm
Updated by asmorodskyi almost 7 years ago
there is always some price which you should pay. Price for NOT having this feature for me that I lock down my tests with BTRFS so in case of any issues with BTRFS my tests will fail.
Can you state what kind of harm of using qemu specific feature with tests which running only in qemu ?
Also I didn't get how the fact that openQA will provide some new functionality is enforcing people to obligatory use it ? It would be still up to tests writer to use it or not
Updated by szarate almost 7 years ago
Updated by rpalethorpe almost 7 years ago
On backends which do not support snapshots, milestones could be handled in a different way. For LTP we will probably reset the machine when a test freezes it. Possibly we could introduce some callbacks for milestones which different backends, tests and architectures can override. On bare metal, getting the machine back into a previous state might require resetting the machine and rerunning some modules. Getting the block storage back into a previous state will depend on a lot of factors so it will be complicated. We could use BTRFS snapshots where they are available, but it requires dynamically running a module to handle it. The worst case will require a full reinstallation.
Updated by okurz about 6 years ago
What I see could be done even though I think it is a bit different to what asmorodskyi wanted: Where we revert to a snapshot after a failed module on backends that support snapshots we could extend that to revert on demand also based on a test variable or even a test module flag.
So what I see as necessary steps to implement that:
- Find the location of snapshot reverting, probably https://github.com/os-autoinst/os-autoinst/blob/master/autotest.pm#L339
- Extend tests we have for snapshot handling, e.g. t/18-qemu.t
- Extend it to also revert based on test variables, e.g. pseudo code change
- revert_to_last_good if ($module_failed && snapshots_supported());
+ revert_to_last_good if ($module_failed && (snapshots_supported() || get_var('ROLLBACK_AFTER_EACH_MODULE') || $test_module{force_reset}));
}
sub handle_failed_module {
- revert_to_last_good if snapshots_supported();
+ revert_to_last_good if snapshots_supported();
}
Updated by riafarov about 6 years ago
- Subject changed from Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed to [functional][y] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed
- Description updated (diff)
- Due date set to 2018-10-09
- Status changed from New to Workable
- Target version changed from future to Milestone 19
Updated by riafarov about 6 years ago
- Status changed from Workable to In Progress
Updated by riafarov about 6 years ago
- Status changed from In Progress to Feedback
Updated by riafarov about 6 years ago
- Status changed from Feedback to In Progress
Updated by riafarov about 6 years ago
- Status changed from In Progress to Feedback
Change required in tests after renaming norollback flag to no_rollback
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5870
Updated by riafarov about 6 years ago
- Subject changed from [functional][y] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed to [functional][y][epic] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed
Updated by riafarov about 6 years ago
- Subject changed from [functional][y][epic] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed to [functional][y] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed
Updated by riafarov about 6 years ago
- Due date changed from 2018-10-09 to 2018-10-23
Updated by okurz about 6 years ago
- Target version changed from Milestone 19 to Milestone 20
Updated by riafarov about 6 years ago
os-autoinst change merged and deployed to osd
Updated by riafarov about 6 years ago
- Due date changed from 2018-10-23 to 2018-11-06
Changes to openQA also merged. Not yet deployed to o3.
Updated by riafarov about 6 years ago
- Due date changed from 2018-11-06 to 2018-11-20
Updated by okurz about 6 years ago
I know it is a pain but for now we need to wait for o3 deployment. We might want to consider investing more time in an automated deployment infrastructure for os-autoinst changes which are less risky than openQA+database. Especiallly after o3 workers changed to transactional-update servers.
Updated by riafarov almost 6 years ago
- Status changed from Feedback to Resolved
As openQA change is just visualization and doesn't affect functional part, I will resolve that one, as os-autoinst change is merged and deployed to osd and o3.