QA » openQA Project

please be aware of https://progress.opensuse.org/issues/60443 there might be some connection between them

asmorodskyi wrote:

please be aware of https://progress.opensuse.org/issues/60443 there might be some connection between them

I don't see how that ticket is related. This one is specifically about rolling back to the wrong snapshot. Aside from that the job ran all the way to the end and isotovideo exited gracefully.

My best guess is that QEMU just creates a new snapshot named something like lastgood.001, lastgood.002 and so on instead of overwriting lastgood as we want it to.

I also do not see the relation to #60443 . As you created this ticket in "openQA Infrastructure", why do you think this is specific to our infrastructure? Your description and comments makes it sound like a generic problem within os-autoinst. So should we move this to openQA Project instead?

Test suite that will reproduce the bug (4 test modules):

1. boot to console and save a VM snapshot (test_flags: milestone => 1)
2. create a test file and save another VM snapshot (test_flags: milestone => 1)
3. die()
4. check whether the test file created in step 2 exists

OpenQA is still rolling back to the wrong snapshot on test module timeout:
https://openqa.suse.de/tests/5653440#step/AD074/8

Not sure what you expected. No one worked on this ticket and we do not currently plan to do, so this should be expected, right?

okurz wrote:

Not sure what you expected. No one worked on this ticket and we do not currently plan to do, so this should be expected, right?

I expected someone to raise the priority of this ticket because it's breaking tests.

cdywan wrote:

As per the description, this should actually be considered a Feature Request, if this never worked 🤔

Fixing a broken existing feature to behave according to official documentation does not count as a "feature request" in my book.

MDoucha wrote:

okurz wrote:

Not sure what you expected. No one worked on this ticket and we do not currently plan to do, so this should be expected, right?

I expected someone to raise the priority of this ticket because it's breaking tests.

I think it'd be good to have a quick call to clarify what we're looking at. Because I see a known issue that rarely pops up but would be nice to look into, and you seem to see a bug that needs to be fixed.

Not sure if you get notifications, so also poking on RC.

So we also have cases like https://openqa.suse.de/tests/5670834 where everything "passes" despite the rollback being broken.

Right. Normally the rollback only happens on failures but this test uses the always_rollback flag so it relies on the rollback working.

Module AD043 failed and triggered VM rollback.

But this module is actually tests/kernel/run_ltp.pm, right? Maybe what makes your test special is that "AD043" is not a real test module? So autotest it actually still at "create_junkfile_ltp" which is the last module corresponding to a real Perl module (tests/kernel/create_junkfile_ltp.pm).

I've tried writing a set of reproducer modules and I can't reproduce the issue.
Reproducer code: https://github.com/mdoucha/os-autoinst-distri-opensuse/commit/d72eece4eda301a2c42c9348fd147d97d13d9267
12SP2@x86_64: https://openqa.suse.de/tests/5730930
12SP2@ppc64le: https://openqa.suse.de/tests/5730960

The failure in module bang is intentional. Successful reproduction would cause another failure either in check_post_rollback or check_post_crash. It doesn't matter whether the 3 check modules have always_rollback or not.

mkittler wrote:

Module AD043 failed and triggered VM rollback.

But this module is actually tests/kernel/run_ltp.pm, right? Maybe what makes your test special is that "AD043" is not a real test module? So autotest it actually still at "create_junkfile_ltp" which is the last module corresponding to a real Perl module (tests/kernel/create_junkfile_ltp.pm).

All modules after setup in the two jobs above are renamed but I still can't reproduce the issue. I guess that the Btrfs bug that causes those LTP failures is involved somehow, as if the disk got corrupted during create_junkfile_ltp and the corruption was saved into the snapshot. But in that case, I don't understand how so many tests pass on a corrupted filesystem. And if they pass only because the corruption is hidden by Btrfs structures cached in RAM, why restoring the snapshot doesn't hide the corruption again?

Thanks for the investigation. Too bad that the issue is not reproducible. Unfortunately I can not tell you whether your suspicion about btrfs is correct or not. It seems generally plausible but raises the question you've mentioned.

So unfortunately it seems like we still have no reproducer, although the issue still occurs in production, with and without always_rollback. To re-iterate what I think we can realistically do:

1. Re-evaluate workarounds i.e. not using snapshots here
2. Use unique identifiers and make logs more explicit
3. Validate snapshots explicitly

The first one would be up to @MDoucha, the other ones I'm thinking of straightforward approaches that don't change behavior for now but would help with identifying whatever the real problem is.