Project

General

Profile

Actions

action #30775

closed

[functional][y] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed

Added by asmorodskyi almost 7 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
SUSE QA - Milestone 20
Start date:
2018-01-24
Due date:
2018-11-20
% Done:

0%

Estimated time:
8.00 h

Description

Currently load_snapshot is not available from test code , and only condition which trigger it is test failure .
But there are cases where we need to reset system to previous Milestone even if module pass.
We have limitations on bare metal, and to proceed here, we can ignore it.

After discussion with Anton, he is fine to start in a conservative way, so that test module should have flag set and we don't change any behavior of the test execution with this feature.

Suggestion

Implement additional tests flag "force_reset" which will force openQA to revert to previous snapshot no matter if module pass or fail

Acceptance criteria

  1. Test module actions can be cleaned up by reverting SUT state to the last milestone snapshot
Actions #1

Updated by okurz almost 7 years ago

  • Category set to Feature requests

I wonder what is the use case. Is this another idea to speed up tests based on snapshots? But I thought you used btrfs snapshots for this?

Actions #2

Updated by coolo almost 7 years ago

  • Target version set to future

I already mentioned when the issue came up last time: this would be specific to the few backends with snapshots available and as such I'm not supporting it.

Actions #3

Updated by asmorodskyi almost 7 years ago

  1. Currently in openqa.suse.de has 192069 jobs and query "select count(*) from jobs where machine in (select name from machines where backend='qemu');" gives us that 166332 are using qemu backend which is 86% of all jobs. So it is not just one of others it is most used one
  2. I am not asking to create some totally new functionality which will require a lot of effort to implement and support. No matter what you said about limited support among backends we already having functionality of creating snapshots and rolling back in case of failure , so this ticket is low hanging fruit which will not harm for 14% of our jobs but will give new abilities to other 86%
Actions #4

Updated by coolo almost 7 years ago

It fosters writing backend specific tests - this is harm

Actions #5

Updated by asmorodskyi almost 7 years ago

there is always some price which you should pay. Price for NOT having this feature for me that I lock down my tests with BTRFS so in case of any issues with BTRFS my tests will fail.
Can you state what kind of harm of using qemu specific feature with tests which running only in qemu ?
Also I didn't get how the fact that openQA will provide some new functionality is enforcing people to obligatory use it ? It would be still up to tests writer to use it or not

Actions #7

Updated by rpalethorpe almost 7 years ago

On backends which do not support snapshots, milestones could be handled in a different way. For LTP we will probably reset the machine when a test freezes it. Possibly we could introduce some callbacks for milestones which different backends, tests and architectures can override. On bare metal, getting the machine back into a previous state might require resetting the machine and rerunning some modules. Getting the block storage back into a previous state will depend on a lot of factors so it will be complicated. We could use BTRFS snapshots where they are available, but it requires dynamically running a module to handle it. The worst case will require a full reinstallation.

Actions #8

Updated by okurz over 6 years ago

  • Target version changed from future to future
Actions #9

Updated by okurz about 6 years ago

What I see could be done even though I think it is a bit different to what asmorodskyi wanted: Where we revert to a snapshot after a failed module on backends that support snapshots we could extend that to revert on demand also based on a test variable or even a test module flag.

So what I see as necessary steps to implement that:

- revert_to_last_good if ($module_failed && snapshots_supported());
+ revert_to_last_good if ($module_failed && (snapshots_supported() || get_var('ROLLBACK_AFTER_EACH_MODULE') || $test_module{force_reset}));
}

sub handle_failed_module {
-    revert_to_last_good if snapshots_supported();
+    revert_to_last_good if snapshots_supported();
}
Actions #10

Updated by riafarov about 6 years ago

  • Subject changed from Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed to [functional][y] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed
  • Description updated (diff)
  • Due date set to 2018-10-09
  • Status changed from New to Workable
  • Target version changed from future to Milestone 19
Actions #11

Updated by riafarov about 6 years ago

  • Estimated time set to 8.00 h
Actions #12

Updated by riafarov about 6 years ago

  • Assignee set to riafarov
Actions #13

Updated by riafarov about 6 years ago

  • Status changed from Workable to In Progress
Actions #15

Updated by riafarov about 6 years ago

  • Status changed from Feedback to In Progress
Actions #16

Updated by riafarov about 6 years ago

  • Status changed from In Progress to Feedback

Change required in tests after renaming norollback flag to no_rollback
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5870

Actions #17

Updated by riafarov about 6 years ago

  • Subject changed from [functional][y] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed to [functional][y][epic] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed
Actions #18

Updated by riafarov about 6 years ago

  • Subject changed from [functional][y][epic] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed to [functional][y] Add test flag which force openQA to fallback to latest Milestone snapshot even when test passed
Actions #19

Updated by riafarov about 6 years ago

  • Due date changed from 2018-10-09 to 2018-10-23
Actions #20

Updated by okurz about 6 years ago

  • Target version changed from Milestone 19 to Milestone 20
Actions #21

Updated by riafarov about 6 years ago

os-autoinst change merged and deployed to osd

Actions #22

Updated by riafarov about 6 years ago

  • Due date changed from 2018-10-23 to 2018-11-06

Changes to openQA also merged. Not yet deployed to o3.

Actions #23

Updated by riafarov about 6 years ago

  • Due date changed from 2018-11-06 to 2018-11-20
Actions #24

Updated by okurz about 6 years ago

I know it is a pain but for now we need to wait for o3 deployment. We might want to consider investing more time in an automated deployment infrastructure for os-autoinst changes which are less risky than openQA+database. Especiallly after o3 workers changed to transactional-update servers.

Actions #25

Updated by riafarov almost 6 years ago

  • Status changed from Feedback to Resolved

As openQA change is just visualization and doesn't affect functional part, I will resolve that one, as os-autoinst change is merged and deployed to osd and o3.

Actions

Also available in: Atom PDF