QA » openQA Project » openQA Tests

@xlai, have you set this var:
_CHKSEL_RATE_WAIT_TIME=120
It may not solve the issue completely, but append it to your test to see if it will reduce the frequency of isse

xlai wrote:

mitiao wrote:

@xlai, have you set this var:
_CHKSEL_RATE_WAIT_TIME=120
It may not solve the issue completely, but append it to your test to see if it will reduce the frequency of isse

Will try. Thanks for the advice.

With the parameter, the issue happen again on latest build 550.2,

Job list(total 3, happen ratio 3/34, nearly10%):
https://openqa.suse.de/tests/1599762#
https://openqa.suse.de/tests/1599951/file/autoinst-log.txt
https://openqa.suse.de/tests/1599952/file/autoinst-log.txt

xlai wrote:

xlai wrote:

mitiao wrote:

@xlai, have you set this var:
_CHKSEL_RATE_WAIT_TIME=120
It may not solve the issue completely, but append it to your test to see if it will reduce the frequency of isse

Will try. Thanks for the advice.

With the parameter, the issue happen again on latest build 550.2,

Job list(total 3, happen ratio 3/34, nearly10%):
https://openqa.suse.de/tests/1599762#
https://openqa.suse.de/tests/1599951/file/autoinst-log.txt
https://openqa.suse.de/tests/1599952/file/autoinst-log.txt

On build 555.1, the happen ratio douled -- 6 cases incomplete due to this issue, happen ratio is 20% now.

Really appreciate your efforts on it. Look forward to the fix! Thanks.

okurz wrote:

@mitiao do you have an idea what to do next?

No idea yet, currently i am working on other stuffs.
@dasantiago may give some help.

On SLE15 RC3 build 567.1 , four tests failed by this issue, happen ratio is 4/34=11%.

mgriessmeier wrote:

@mitiao: could you please give us an update on the state here?
Are you working on this actively? Do you need any help by the QSF team?

No update yet, i put this in my schedule later.
Any help welcome and if anyone have idea or able to fix it, please take it :)

mgriessmeier wrote:

We didn't see this for a longer time now on jobs which are covered by the QSF team.
so unassigning from our backlog for now.
If you need any further assistance here, feel free to ask

Virtualization job group still keeps meeting the issue, eg build 635.1, see https://openqa.suse.de/tests/1711157/file/autoinst-log.txt.

This issue has been marked a serious blocking openqa backend issue for virtualization job group, since we meet a lot and blocked a lot by this issue.

Hope this can be fixed ASAP.

I might have an idea here... let's see...

I'm not able to get an ipmi job running on my openQA instance...

if anyone wants to pick up the idea - My guess is that we are missing a disable_vnc_stall for the ipmi backend before we reboot, so I've modified prepare_system_shutdown in lib/utils:

@@ -239,7 +239,7 @@ sub prepare_system_shutdown {
     # kill the ssh connection before triggering reboot
     console('root-ssh')->kill_ssh if check_var('BACKEND', 'ipmi');

-    if (check_var('ARCH', 's390x')) {
+    if (check_var('ARCH', 's390x') || check_var('BACKEND', 'ipmi')) {
         if (check_var('BACKEND', 's390x')) {
             # kill serial ssh connection (if it exists)
             eval { console('iucvconn')->kill_ssh unless get_var('BOOT_EXISTING_S390', ''); };
         }
         console('installation')->disable_vnc_stalls;

mgriessmeier wrote:

I'm not able to get an ipmi job running on my openQA instance...

if anyone wants to pick up the idea - My guess is that we are missing a disable_vnc_stall for the ipmi backend before we reboot, so I've modified prepare_system_shutdown in lib/utils:

@@ -239,7 +239,7 @@ sub prepare_system_shutdown {
     # kill the ssh connection before triggering reboot
     console('root-ssh')->kill_ssh if check_var('BACKEND', 'ipmi');

-    if (check_var('ARCH', 's390x')) {
+    if (check_var('ARCH', 's390x') || check_var('BACKEND', 'ipmi')) {
         if (check_var('BACKEND', 's390x')) {
             # kill serial ssh connection (if it exists)
             eval { console('iucvconn')->kill_ssh unless get_var('BOOT_EXISTING_S390', ''); };
         }
         console('installation')->disable_vnc_stalls;

Nice, thank you for the solution idea!

@Alice, @mitiao, any idea? can we pick up this fix and try at least in Beijing ipmi server firstly? If the fix not harm testing, then maybe we can have the fix merged and try in openqa.suse.de, what do you think?

cachen wrote:

mgriessmeier wrote:

I'm not able to get an ipmi job running on my openQA instance...

if anyone wants to pick up the idea - My guess is that we are missing a disable_vnc_stall for the ipmi backend before we reboot, so I've modified prepare_system_shutdown in lib/utils:

@@ -239,7 +239,7 @@ sub prepare_system_shutdown {
     # kill the ssh connection before triggering reboot
     console('root-ssh')->kill_ssh if check_var('BACKEND', 'ipmi');

-    if (check_var('ARCH', 's390x')) {
+    if (check_var('ARCH', 's390x') || check_var('BACKEND', 'ipmi')) {
         if (check_var('BACKEND', 's390x')) {
             # kill serial ssh connection (if it exists)
             eval { console('iucvconn')->kill_ssh unless get_var('BOOT_EXISTING_S390', ''); };
         }
         console('installation')->disable_vnc_stalls;

Nice, thank you for the solution idea!

@Alice, @mitiao, any idea? can we pick up this fix and try at least in Beijing ipmi server firstly? If the fix not harm testing, then maybe we can have the fix merged and try in openqa.suse.de, what do you think?

Thanks to mattias for the suggestion!
I will try locally. It is just that our situation is more complex here. We not only has this step called but also other parts of code that handles reboot, so we may need to extend this idea. I will try locally first to see if it can work well. Then we push to openqa to see if it can make less incomplete jobs.

PR is proposed in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5236.

This change will not affect workflow. Let's see if it can kill incomplete job due to half open socket.

xlai wrote:

PR is proposed in https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5236.

This change will not affect workflow. Let's see if it can kill incomplete job due to half open socket.

From the PR's comment by zaoliang, it reproduced easily on his test with the changes, so it was proved to be not working.

Now it is open for new suggestions or bug fix on the backend side.

Please comment back if disagree with the conclusion about this suggestion.

This issue still reproduces a lot in sle12sp4 openqa testing.

Failed jobs examples:
http://openqa.suse.de/tests/1795850
http://openqa.suse.de/tests/1795947

Both failed at reboot_after_installation step which is the final common os installation step, rather than real virtualization specific code.

update to this issue:
broken ssh sock-connection by shutdown will trigger the half open check .

In virtualization test :
select_console root-ssh will create and add ssh sock to monitor code , reboot without disconnect will cause half-open issue.

so this can be fix by use prepare_system_shutdown before every reboot/shutdown step in the testcase .

I submit 2 PR for fixing .

It will be better if developer can review and give some comment about this

PR for test :
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5778

PR for backend :
https://github.com/os-autoinst/os-autoinst/pull/1026

Thanks

Jerry tang

jerrytang wrote:

update to this issue:
broken ssh sock-connection by shutdown will trigger the half open check .

In virtualization test :
select_console root-ssh will create and add ssh sock to monitor code , reboot without disconnect will cause half-open issue.

so this can be fix by use prepare_system_shutdown before every reboot/shutdown step in the testcase .

Jerry, I don't understand your argumentation here while following your code-change on the same time. If I understand you correctly, calling prepare_system_shutdown before every reboot fixes this. However, your change doesn't call prepare_system_shutdown but changes much much more in the backend and test code. So simple question:

1. Does calling prepare_system_shutdown before reboot fix this issue?

Jerry, I don't understand your argumentation here while following your code-change on the same time. If I understand you correctly, calling prepare_system_shutdown before every reboot fixes this. However, your change doesn't call prepare_system_shutdown but changes much much more in the backend and test code. So simple question:

1. Does calling prepare_system_shutdown before reboot fix this issue?

Theoretically ，calling prepare_system_shutdown before reboot fix this issue .

During fix testcase side i found reboot is special for this situation .
The currently scheme ：
1.reboot require ssh-connected(root-ssh console: xvnc+xterm+ssh)。

1. prepare_system_shutdown will disconnect ssh.

so you can see the problem.
call prepare_system_shutdown you never get chance to reboot .

my pr is just one way to handle this , better way is welcome .

Did you try disable_vnc_stalls on the active console before restart is triggered? I used to have problems with VNC stalls on the svirt backend, this helped. See: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/ecf740a0999cc8cb29c93880b7d43b080933be3c/lib/power_action_utils.pm#L47. Do you use power_action()? disable_vnc_stalls is used from there.

michalnowak wrote:

Did you try disable_vnc_stalls on the active console before restart is triggered? I used to have problems with VNC stalls on the svirt backend, this helped. See: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/ecf740a0999cc8cb29c93880b7d43b080933be3c/lib/power_action_utils.pm#L47. Do you use power_action()? disable_vnc_stalls is used from there.

could you please explain how this works?
because I'm not sure vnc sock is add to the monitor socks ($self->{select}).

And this issue is not reproduce 100%.
( I think it's because of the race condition between

1 send_key 'alt-o' means shutdown host kill ssh
2 power_action('reboot', observe => 1, keepconsole => 1, first_reboot => 1); means remove and kill ssh .
.
if 2 faster 1, then it's fine
)

https://openqa.suse.de/tests/2095884/file/autoinst-log.txt seems to be one of the last examples, just to reference a more recent job :)

I think you missed to mention the important step which IMHO can destroy everything:

https://github.com/os-autoinst/os-autoinst-distri-opensuse/commit/ae9cf2e2c51d24ae3505fa5fedcdb8b3528d1707 introduced a wait_screen_change around the alt-o keypress. This relies on the remote VNC connection which might be not there already at this time. I guess removing the wait_screen_change can actually fix the problem -> https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5915

Thanks all the above suggestions and solutions!

We tried disable_vnc_stalls by calling prepare_system_shutdown for ipmi backend, which is not 100% fix the issue.

Will try solution of removing wait_screen_change.

I feel we are very close to get this issue fix :)

to prove the race condition , i create a job intend to get half-open issue:¶

 #after ipmi host boot 
 use_ssh_serial_console;
 type_string("reboot\n");
 sleep 4;  #===========>this will make sure reboot kill ssh first
 save_screenshot;
 power_action('reboot', observe => 1, keepconsole => 1, first_reboot => 1);
 save_screenshot;

I run 4 times , and all of them get half-open problem .
http://10.67.132.86/tests/309#next_previous

Also i replace type_string("reboot\n"); ====> type_string("shutdown -r 1\n"); to avoid half-open.
and result is no half-open issue.

problem happened after send-key alt-o , your PR may not work as expect.

So let me try to phrase in my own words: Pressing 'alt-o' causes the SUT to reboot which will close the socket from one side. This is not a problem if the corresponding ssh connection is killed fast enough but not otherwise, hence the race condition. However, we can not simply call prepare_system_shutdown which would also be triggered by power_action because we still need a connection to the SUT to send the 'alt-o' key. https://github.com/os-autoinst/os-autoinst/pull/902 which was done to detect "half-open sockets" might have introduced exactly that problem. IMHO the backend needs to handle this gracefully, e.g. we press 'alt-o' and then just call power_action but I do not know in which way exactly this could work.

okurz wrote:

So let me try to phrase in my own words: Pressing 'alt-o' causes the SUT to reboot which will close the socket from one side. This is not a problem if the corresponding ssh connection is killed fast enough but not otherwise, hence the race condition. However, we can not simply call prepare_system_shutdown which would also be triggered by power_action because we still need a connection to the SUT to send the 'alt-o' key. https://github.com/os-autoinst/os-autoinst/pull/902 which was done to detect "half-open sockets" might have introduced exactly that problem. IMHO the backend needs to handle this gracefully, e.g. we press 'alt-o' and then just call power_action but I do not know in which way exactly this could work.

exactly.
As installation session has

• no io cache policy .
• no simple way to reboot in ssh session. so you can see my PR is use 2nd way to hard_reset.

The issue was still happened time to time during RC2 build 0421 acceptance testing, we have to staring the tests and prepare to retrigger the tests failed by this issue. Personally I don't like to pushing too much, but it affect the acceptance result deliver so much, and more ipmi relevant tests will be added :(

Let me try to understand the current situation from comments and PRs:
first all I think we are clear of why and when Half-open happens in installation->reboot_after_installation step,finally in same page :)
So far there are 2 options:
1)To just reduce the error hitting by Olive's PR#5915 or by prepare_system_shutdown
2)Fix it as Jerry's PRs by Roughly reboot the ipmi server as hard_reset (perhaps to have server reboot directly in power_action for ipmi specify is prefer?)

@coolo, you are PO of openQA and expert/author of ipmi backend, we need your suggestion and decision, or maybe you have better solution :)

cachen wrote:

So far there are 2 options:
1)To just reduce the error hitting by Olive's PR#5915 or by prepare_system_shutdown

I think this option can be applied regardless. It might just help with the symptoms but still help :)

2)Fix it as Jerry's PRs by Roughly reboot the ipmi server as hard_reset (perhaps to have server reboot directly in power_action for ipmi specify is prefer?)

@coolo, you are PO of openQA and expert/author of ipmi backend, we need your suggestion and decision, or maybe you have better solution :)

I would still go with #32746#note-53 which is a different approach: Handle the uni-directional socket termination gracefully in the backend.

Anyway I update my pr follow some point of coolo mentioned , move all action in the prepare_system_shutdown function;
https://github.com/os-autoinst/os-autoinst/pull/1026#issuecomment-429277855

But , still need backend api supported in my "NOT" graceful way.

I hope this issue can be fixed soon , waiting for graceful way;

jerrytang wrote:

I hope this issue can be fixed soon , waiting for graceful way;

Don't wait, better try to fix it yourself. I made a suggestion in
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5778#pullrequestreview-164196772

https://github.com/os-autoinst/os-autoinst/pull/902 merged, https://openqa.opensuse.org/tests/772175 is VR on normal qemu-x86_64, sle-15-SP1-Installer-DVD-x86_64-Build66.2-gi-guest_sles12sp2-on-host-developing-kvm@64bit-ipmi shows that at least the IPMI virtualization tests are also not broken now :)

Please crosscheck if this helps to mitigate the problem, e.g. with a good statistical analysis, monitor jobs triggered after that, etc.

Hi cachen,

cachen wrote:

So far the error hitting has reduced in virtualization tests by the merged PO#5915

I am happy to hear that my PR help a bit :)

since the rest fix should touch the deep in openQA backend, I agree with Jerry to hand over it back to openQA Tools group […]

Hm, I am not sure this will work. The current tools team does not really have experts in the aforementioned domain and I think Jerry was already on a good track to fix it for good.

okurz wrote:

Hi cachen,

cachen wrote:

So far the error hitting has reduced in virtualization tests by the merged PO#5915

I am happy to hear that my PR help a bit :)

I appreciated so much ;)

since the rest fix should touch the deep in openQA backend, I agree with Jerry to hand over it back to openQA Tools group […]

Hm, I am not sure this will work. The current tools team does not really have experts in the aforementioned domain and I think Jerry was already on a good track to fix it for good.

Unfortunately, I have to have Jerry back to Performance testing.

mitiao(Wei Jiang) know the whole background, he has been involved in discussion and testing a lot in Beijing office :)
I think he can take over the rest thing to optimize the codes, of course Jerry and Virtualization group will continue to support and verify for his code.

I understood from xlai that the issue is still present even though less likely. According to xlai mitiao is on it

New happens in VT reboot_after_installation step:

https://openqa.nue.suse.com/tests/2503084
https://openqa.nue.suse.com/tests/2504185

Does it caused by the rewrite in commit 023c4c09dca87d17b3cec325f3adb5288525a211 ?