QA » openQA Project

load of the system seems not too high: https://openqa.suse.de/tests/796211/file/xterm-loadavg.txt 0.50 0.79 0.42 1/238 8266

Build0260 also affected: https://openqa.suse.de/tests/795241

Another one in 0260: https://openqa.suse.de/tests/796258

another one: https://openqa.suse.de/tests/796012#step/change_password/10

https://openqa.suse.de/tests/797387

Discussed with Oliver, below is the current leading hypothesis

https://openqa.suse.de/tests/796211#step/xterm/4 and https://openqa.suse.de/tests/796211#step/xterm/4 both show that the issue is not X11 bound

The issue is that os-autoinst is typing before the bash shell is ready to accept input https://openqa.suse.de/tests/796211#step/xterm/4 shows that many of the Enter presses were committed before bash was accepting input, even though the prompt was on the screen (confirmed by the assert_screen)

With os-autoinsts fractional time, needling on assert screen and such is a lot faster, so os-autoinst is typing before the prompt is ready

We can rule out hardware overload, this problem does not happen when systems are overloaded - in fact overloading a system seems to AVOID this problem.

Hypothethis - There may be a performance issue around bash that increases the time between the bash prompt being visible on the screen and it being ready to accept input.
This performance issue might be a regression compared to SP2, or could be just detected now as a result of the performance improvements to os-autoinst
This performance issue might be a problem with the generation of the bash prompt - is there an issue finding the hostname? is there an issue finding the username?

Jozef's pull request in January shows that the problem was at least present in mid January.

As discussed with Oliver, we're going to need some manual testing. Simple tests would be an installation of SLES 12 SP3 and holding down the enterkey on a bash prompt in xterm or tty. A well working system should produce an output with an almost endless stream of prompts

rbrown@ibrokeit:~>
rbrown@ibrokeit:~>
rbrown@ibrokeit:~>
rbrown@ibrokeit:~>
rbrown@ibrokeit:~>

If we're able to reproduce this issue, we'd expect to see something like

rbrown@ibrokeit:~>

rbrown@ibrokeit:~>

Which would imply that bash is taking too long to generate it's prompt, just like we see in openQA

If we can reproduce this outside of openQA, then it's a clear bug that needs to be filed

If we can reproduce it outside of openQA but not on SP2, then we know it is an SP3 specific bug (but like I say above, this might not be a regression)

if we can't reproduce it out of openQA, there is still the chance of it being a bug. Areas of further investigation would include name resolution, username detection, and how they are done by bash when creating the prompt.

The different network configuration for the SUT in openQA could have a part to play - because it's installed by YaST our configuration should be valid, but are we sure it is? Is SELF name resolution working properly? the hostname.pm test only checks hostnamectl, is that the method used by bash?

If we manage to rule out all of the above as a problem with the product, then I would be prepared to accept that in this case openQA could be testing 'too fast', in which case I think we should investigate adding appropriate delays in either assert_screen or the affected tests so they give more time between checking the prompt is visible and then using it.

https://openqa.suse.de/tests/789944#

This typing too soon happened on my local openQA (desktop with 2 workers)

Update: without nfs, just local hdd

Typing too soon / too late happened a lot in build 0267:

All gedit failures:
https://openqa.suse.de/tests/804361#step/gedit/5
https://openqa.suse.de/tests/804365#step/gedit/5
https://openqa.suse.de/tests/804342#step/gedit/5
https://openqa.suse.de/tests/804413#step/gedit/5
https://openqa.suse.de/tests/804357#step/gedit/5

Gnome terminal:
https://openqa.suse.de/tests/804357#step/gnome_terminal/7

Firefox:
https://openqa.suse.de/tests/804345#step/firefox/4

GPK updates:
https://openqa.suse.de/tests/804350#step/updates_packagekit_gpk/5
https://openqa.suse.de/tests/804375#step/updates_packagekit_gpk/5

VNC:
https://openqa.suse.de/tests/802819#step/vnc_two_passwords/11

X11_start_program: (not typing program name in many tests)
https://openqa.suse.de/tests/804319#step/xterm/1

same behaviour in many Maintenace Test's , Test Repo + Incidents ( +-5 cases per run)

seems also related:
https://openqa.suse.de/tests/804730#step/yast2_snapper/11

typing echo0 - probably missing a space here

also related:
https://openqa.suse.de/tests/804703#step/wireshark/4

things getting typed twice compared to old jobs

Here is another failed test with system performance statistics
Job was started at 2017-03-08 07:24:58

http://10.100.12.155/tests/5565#step/ssh_cleanup/3

pdf
ftp://10.100.12.155/sar.pdf
text
ftp://10.100.12.155/sar.txt

http://argus.suse.cz/tests/160#step/x11regressions_setup/4 hit locally :)

https://openqa.suse.de/tests/806657

https://openqa.suse.de/tests/806822#step/zypper_lifecycle/29 is another fun example of missing keys from beginning of string in console but not immediately after console switching

https://openqa.suse.de/tests/807661#step/xterm/6 single 'ret' had an effect, nothing more is shown in xterm.
https://openqa.suse.de/tests/807661#step/sshxterm/5 same job, next module, missing characters from beginning.

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2525 is my proposal for a workaround but I will test something more thoroughly later on, e.g. some kind of send_key_until_needle_match 'next-line-in-xterm', 'ret' or a while-loop, maybe.

https://openqa.suse.de/tests/807824#step/updates_packagekit_gpk/6 in xterm, missing keys in beginning

https://openqa.suse.de/tests/807449#step/check_console_font/3 in check_console_font, beginning "echo" missing

Staging http://openqa.suse.de/tests/810619#step/xterm/6

As we chatted about in my office today - I think the console failures of this type like the one we saw today could/should be mitigated by a better needle that actually checks it has the correct console so it doesn't type into the user console before it's actually flipped to the root console

Have you or anyone else submitted a needle to fix that?

The X11 failures, like we saw in gedit, need more investigation and http://openqa.suse.de/tests/810619#step/xterm/6 alone isn't enough for me to get a feel for the problem:
More examples would help, so I can profile the system performance at the time and such.
I'm waiting and seeing before deciding what to do about https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2525/

If things are really getting in the way of staging, then I can accept gh#2525 as a 'make release managers happy' measure, but as we're just on the dark side of Alpha 2 I would prefer to have more time to investigate if you can help make that happen

What do you think?

RBrownSUSE wrote:

As we chatted about in my office today - I think the console failures of this type like the one we saw today could/should be mitigated by a better needle that actually checks it has the correct console so it doesn't type into the user console before it's actually flipped to the root console

Have you or anyone else submitted a needle to fix that?

I am on it in #14826

The X11 failures, like we saw in gedit, need more investigation and http://openqa.suse.de/tests/810619#step/xterm/6 alone isn't enough for me to get a feel for the problem:
More examples would help, so I can profile the system performance at the time and such.
I'm waiting and seeing before deciding what to do about https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2525/

If things are really getting in the way of staging, then I can accept gh#2525 as a 'make release managers happy' measure, but as we're just on the dark side of Alpha 2 I would prefer to have more time to investigate if you can help make that happen

What do you think?

I think not enough people trust the system so after two weeks with no solution in sight we should expedite to find a workaround and make some people hunt for product bugs while we can continue to investigate further on this. If we simply put in a record_soft_failure or maybe even a single but unambiguous "diag" message we can find occurences of these problems and gather more data. I updated my PR to record a soft failure and try again. This might help to gather even more data.

so we merged the test pull request which might help us to detect this (or break everything, as always, but I will monitor)

But now for something completely different. In
https://openqa.opensuse.org/tests/369987#step/inkscape/8
which is on o3, we see a mistyping of "xtrem" instead of "xterm". The test code is fine and so is the call to the testapi

testapi::type_string(string='xterm', max_interval=250, wait_screen_changes=0)

same issue?

https://openqa.suse.de/tests/818019 - another example of missing keys job

recent example of failing job: https://openqa.suse.de/tests/820752#step/xterm/9

In build 0288 review:
https://openqa.suse.de/tests/826663#step/zypper_lifecycle/59
https://openqa.suse.de/tests/826673#step/zypper_lifecycle/55
https://openqa.suse.de/tests/826653#step/zypper_lifecycle/29
https://openqa.suse.de/tests/826680#step/zypper_lifecycle/29
https://openqa.suse.de/tests/826680#step/updates_packagekit_gpk/11
https://openqa.suse.de/tests/826696#step/sshd/15
https://openqa.suse.de/tests/826679#step/sshd/17
https://openqa.suse.de/tests/826683#step/mtab/1

Careful not to mix different issues. the zypper_lifecycle cases are 'not waiting for select-console to finish'.

Same with sshd

same with mtab

o3 examples of mistyped:
https://openqa.opensuse.org/tests/375639#step/vlc/22 -> "v-lc -no-autoscale" instead of "vlc --no-autoscale" so the one "-" ended up somewhere differently.
Similar is https://openqa.opensuse.org/tests/376071#step/ooffice/7 -> "oowritre" instead of "oowriter"

krunner is using autocompletion - and it breaks often if you type manually in there too. So we need to type very slow there - but it's more a product workaround not an openqa problem.

So I experimented with waiting for screen updates after every key on malbec (which has qemu 2.8 now), but the emulated keyboard still bounces keys:
https://openqa.suse.de/tests/837199#step/snapper_cleanup/30 - so typing SLOOOW on power and aarch is the only option atm :(

https://openqa.suse.de/tests/837414 - power architecture

that test ran on the weekend when I experimented with malbec. The result of that was https://github.com/os-autoinst/os-autoinst/pull/743/files - and we haven't yet updated the machines

743 is deployed, but not used as not configured

Could you please check these tests:

https://openqa.suse.de/tests/840897#step/curl_https/7

It might be related, see difference between "Console output" and "serial output"

https://openqa.suse.de/tests/840898#step/sshd/20
https://openqa.suse.de/tests/840905#step/sshd/20
https://openqa.suse.de/tests/840899#step/curl_https/7

There's another one too: https://openqa.suse.de/tests/843506

see #17594#note-16 for more information on potential resolution helps

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: migration_zdup_offline_sle12sp2_allpatterns_fullupdate_ppc
https://openqa.suse.de/tests/878304

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: om_proxyscc_sles12sp1_smt_full_update_by_yast
https://openqa.suse.de/tests/878430