QA » openQA Project

Issue was not seen for a long time after the first workaround, so the assumption is that this was a performance issue on our mainframe
-> closing for now

reopened due to recent fails...

--> adjusted the random wait time for establishing a ftp connection in ftpboot to 90seconds

As described in action #12276 a workaround in the tests has been applied to retry on errors on either download errors or s3270 related errors: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/1463

"QABOOT" has been simplified by removing the "random sleep time" to prevent download issues. The assumption for the sleep was that too many jobs started at the same time could cause some timeout issue. Regardless if this is true or not the issue is not fixed by that. That is why the retry is the better behavior.

It seems another issue (or maybe the real problem?) why we did not see any backtrace on "die" in the logfile is that the die-handler catches the exception but silently forwards it to the web UI showing the "wait_serial"-like failure windows including all details and the backtrace. https://github.com/os-autoinst/os-autoinst/pull/513 prevents this confusion by outputting any message, e.g. backtrace, to the logfile, first. Grepping logfiles of multiple jobs is also much easier than looking at the web UI.

Now no sporadic timeout on "#cp … cms" or downloading kernel+initrd should make the tests fail. However, the backend implementation of the s3270 console is still lacking. It could be we introduced a regression or it was not completely stable after all. mgriessmeier proposes to just replace "expect_s3270" calls by "assert_screen" on the x3270 window. okurz proposes to improve the backend console properly (needs better understanding how it really works by okurz+mgriessmeier).

The output for the previous test is

 '4145152 bytes transferred.                                                      ',
'INPUT                                                                           ',
'4153344 bytes transferred.                                                      ',
'4161536 bytes transferred.                                                      ',
'Server not responding, closing connection.                                      ',
'Abnormal inter-VM communication condition                                       ',
'Abnormal inter-VM communication condition                                       ',
'>>>ABOR                                                      

so the download aborts with a new warning from probably the FTP call but this should be catched now from our high-level retry in the test.

https://openqa.suse.de/tests/443270 is failing in the "Loading installation system" step.

Seen in both is that the line "INPUT" is typed too early before the xedit mode was encountered. Then the content that should go into the file edited is instead typed too early and lost. That is why the next step "Loading installation system" is never reached.

EDIT: This error was introduced with os-autoinst#d7b93259, fixed with next PR

After a complete code walkthrough with mgriessmeier, PR done with mgriessmeier: https://github.com/os-autoinst/os-autoinst/pull/514

The most important change is basically:

-                warn "status line matches neither buffer_ready nor buffer_full:\n" . Dumper($result) . $status_line;
+                confess "expect_3270: timed out waiting for 'expected_status'.\n" . "  waiting for ${\Dumper \%arg}\n" . "  last output:\n" . Dumper($result) . $status_line;

i.e. a fatal error as in before which can be catched from the test code with "eval". Previously we exchanged an "confess" by "warn" on the same location as we had the feeling that this line is part of expected behaviour which was right considering that this point is reached when test code is looking for certain text in the status line which does not show up in the specified timeout. We were wrong to change it to a non-fatal warn as then this situation could not be detected anymore from the test code and therefore not properly handled. The confess message was probably implemented a long time ago and not improved when the functionality to also search for custom status line entries was introduced, i.e. not only "RUNNING" or "MORE..."

The correct behaviour from the test point of view is as implemented with the latest change in the test, i.e. retry on network errors.

Therefore rejecting H3 and accepting H2.

new backend is deployed, let's wait some more tests to see this does not come back. Also review subtasks before closing

EDIT: New backend was not deployed on all workers until today (2016-06-20), it seems. It should work now.