QA » openQA Project » openQA Tests

zcjia wrote:

@Oliver:
So I wonder if this setting should be added to the sle-15-desktop medium type?

I doubt this is a good idea. Also I do not think that "we type too fast". I guess the system is just loaded and a bit unresponsive just after startup. One can consider this a product bug.

I would suggest an approach similar to
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5365
which ensures with the desktop_runner module that there is a certain "cool down time" directly after login. I suggest to involve jbaier as the test module maintainer of window_system to handle this.

@Oliver, I can confirm that the system load is not high when this error happens. Also, this error can happen elsewhere every now and then, which is very annoying.

I talked to another QA, his opinion and experience is that openQA is typing too fast: 50 key strokes a second is indeed too fast, especially for a desktop environment.

@jbaier, what's your opinion on this matter?

For me it is a little bit weird. It was working quite confidently at the time of the development. Now I see those errors more and more often. Of course, I can rewrite the test (at the cost of loosing the record_info feature for pretty presentation), however this issue could come back in other tests as there are a few tests which types in the terminal.

It would be nice to know the root of the issue (load on the worker / openQA backend issue / unreliable typing short after boot).

I think problem is in broken key presses with QEMUVGA=std or QEMUGA=virtio

The reason to use QEMUVGA=virtio for wayland is that wayland doesn't support the default "cirrus", see https://progress.opensuse.org/issues/21786

Hi @zcjia, @okurz, I saw an upstream bug of qemu was reported https://bugs.launchpad.net/qemu/+bug/1802465 , it looks X is also affected but wayland is more easy to be impacted with a shorter length of trigger string. The problem is we are not sure how the issue will be escalated yet.

By the fact that there are quite some desktop testing of SLE-15-SP1 desktop were blocked by this issue, and Alpha-3 is coming:

http://openqa.suse.de/tests/overview?distri=sle&version=15-SP1&build=88.6&groupid=118

Is there a way to add an acceptable workaround to make the testing run more reliably? Thank you!

Since SLE-15-SP1-Alpha3 is coming, I will apply the workaround on OSD:

by adding "VNC_TYPING_LIMIT=10" to machine type "64bit-virtio-vga".

Let's see how it goes.

Yes, I think this is a good idea. As long as you call the ticket only done when you removed that workaround again I am fine ;)

Good evaluation. I wonder why you selected libreoffice though. IMHO libreoffice is especially prone to cause mistyping, see #43889, maybe because of "spell correction" or something. The original ticket observation was about "window_system" so maybe the results would actually be different if you would conduct the experiment in a more "simple" environment, e.g. gedit/kwrite

Thank you!
I actually tested on both gedit and libreoffice. I will proceed with gedit here after.

I found a special case about sending key streams ending with a uppercase letter, e.g. ABCDE, AbC, aaaC

If you could refer to gnome_wayland_virtio2.png, gnome_wayland_qxl2.png or kde_wayland_virtio2.png in the above attachment, you could found there's a obvious miss on lowercase or uppercase every two times.

The reason is as follows, when we send one character A, the key code sent is actually like

1. capslock pressed
2. a pressed
3. enter pressed So capslock is always used instead of shift and now capslock gets toggled.

If a lowercase letter comes after a capital letter, the capslock would be pressed again so capslock will get back to normal.
But if key stream ends with a uppercase letter, it is observed that capslock is not pressed again and therefore toggled.

Now I'm wondering which part of code actually deals with capslock, is it ps2 driver handling it or vncdotool?
What do you think?

Are you sure it's "caps lock" or just "shift"? I guess that it is either qemu or VNC, I doubt that vncdotool is involved here at all. https://build.opensuse.org/package/view_file/Virtualization/qemu/0025-Fix-tigervnc-long-press-issue.patch?expand=1 might be related? Just browsed the web quickly myself, maybe https://www.berrange.com/posts/2010/07/04/more-than-you-or-i-ever-wanted-to-know-about-virtual-keyboard-handling/ helps. Sorry, can't help more.

wow, I appreciate your fine detailed investigation notes, especially the picture can help understanding, thx.

I found the bug resides both in qemu internals and in wayland.

In qemu, it is a concurrency problem since capslock LED is handled by another thread --- so keycode streams get messed up when capslock LED is emitted.

I opened up a bug on qemu upstream

The bug regarding wayland is observed when kernel event device receives correct keycode streams, gedit still shows wrong values. It seems to be relevant to bug 1117833: Wayland: briefffffffffff unresponsivvvvvvvvvvvveness and repeated keys This bug is not observed on x11.

To check if kernel is to blame or not, I used ftrace and dynamic_debug to peek into what kernel receives. It seems kernel receives what qemu sends it and passes all recognized key streams to /dev/input/event.

Found related bug report related to fedora.

Bug 1658676 - Typing '<' more than once (with the specific keypress sequence used by os-autoinst) breaks with usb-kbd and VNC

Also,
'Less than' (<), 'more than' (>), and 'pipe' (|) can't be typed via VNC

Additionally, I will pay some attention to wayland bug, since it happens much more often comparably.

Issue raised on gnome-shell gitlab
As pointed out by zcjia, wayland typing issue only happens after gnome-shell 3.2.6.

Noted that this wayland issue is to be blame for severe typing flaws on wayland under gnome.

But typing issues on openQA involves qemu issues, and possibly other parts ---we found that create_hdd_* cases might have user name typed wrong, but I failed to reproduce this problem in installed x11 instance where I tested around 250,000 characters through vnc 10ms per char.

You could refer to gnome-gdm, where typing user name went wrong 3 out of the 10 times from 20181214 to 20190103. They fail at first_boot
Also, in this case and this case

Regarding wrong user name typing during installation: That should fail the test earlier in https://openqa.opensuse.org/tests/829175#step/user_settings/2 with the right needle. It looks like the last matched needle has vanished. Maybe you are onto this already?

This ticket is kinda more broad than my specific issue, but I did notice one very specific issue with the Firefox Wayland native backend (specifically, it does not happen with the X11 backend): typing modified characters fails very often. See https://bugzilla.redhat.com/show_bug.cgi?id=1727388 . I found that reversing the order of the 'key up' events when doing modified characters - so e.g. to type a colon we do:

1. shift down
2. ; down
3. ; up
4. shift up

instead of this:

1. shift down
2. ; down
3. shift up
4. ; up

works around that problem; https://github.com/os-autoinst/os-autoinst/pull/1174 is my PR for that.

For the record, in the Fedora tests we commonly use our type_slowly and type_very_slowly macros for typing in GNOME (and KDE), exactly because we've seen this kind of problem before. type_slowly just slows down the typing speed. type_very_slowly slows it down even more, and does a wait_screen_change between each key press.

Hi @zcjia, @zgao the issue of problems with typing characters persists as also the linked tickets describe, e.g. #43889 . What are your plans on working on this issue?

okurz wrote:

Hi @zcjia, @zgao the issue of problems with typing characters persists as also the linked tickets describe, e.g. #43889 . What are your plans on working on this issue?

Hi Oliver, (zgao was an intern and left the company), this problem is caused by multiple sources, one of them is the version of GNOME. I will give this problem another try as we are going to GNOME 3.34 in SLE15SP2.

I saw the latest problem in Tumbleweed where there is already gnome 3.34 so no need to wait :)

For the record: the original rate was 6.66 keystrokes/s from
https://github.com/os-autoinst/os-autoinst/blob/v1/bmwqemu.pm#L398
That was still using the qemu monitor interface where capital letters were one "sendkey shift-n".

And when typing longer strings, it would wait for output to catch up after 13 chars:
https://github.com/os-autoinst/os-autoinst/blob/v1/bmwqemu.pm#L423
mostly meant for isolinux and grub where BIOS keystroke buffer was limited to 15 and graphics could take a while to redraw. But I also remember cases where KDE was slow to react and slower typing helped.

This ticket was set to Normal priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.