action #39926
closedcoordination #35302: [qe-core][opensuse][functional][epic][sporadic] Various unstable tests on o3
[functional][u][sporadic][virtio] test fails in vlc - vlc is started twice, needle target_match_vlc seems wrong
Added by okurz over 5 years ago. Updated over 4 years ago.
0%
Description
Observation¶
openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-kde-wayland@64bit_virtio fails in
vlc
with vlc showing up twice. The needle check in https://openqa.opensuse.org/tests/735310#step/vlc/16 looks wrong, like looking for a string in a desktop runner but we should probably check for the VLC window to show up.
Reproducible¶
Probably sometimes, also depending on the desktop environment
Expected result¶
Last good: 20180812
For comparison:
- gnome not matching in first try, pressing esc, trying again, then matching and looking for the main window
- kde matching on the string in the desktop runner
Further details¶
Always latest result in this scenario: latest
Updated by okurz over 5 years ago
- Target version changed from Milestone 19 to Milestone 21
Updated by okurz over 5 years ago
- Status changed from New to Rejected
- Assignee set to okurz
Updated by okurz over 5 years ago
- Status changed from Rejected to Workable
- Assignee deleted (
okurz) - Target version changed from Milestone 21 to Milestone 22
problem still present and reproduced in more recent jobs as well, e.g. https://openqa.opensuse.org/tests/818067#step/vlc/22
Updated by okurz over 5 years ago
- Related to action #42674: [opensuse][functional][u][sporadic] test fails in vlc - cannot open video added
Updated by zluo over 5 years ago
- Status changed from Workable to In Progress
- Assignee set to zluo
take over and check
Updated by zluo over 5 years ago
Updated by okurz over 5 years ago
Hi zluo, this test fails for you because you are using outdated needles. Please update your needle repo. This is something that you should do always before running tests.
Could you please describe again in written form what you think is the problem with this test module?
But also I can see that even the newest needles created, e.g. vlc-url_check-20190116.json, are using the default match level so that is not exactly very robust.
Updated by zluo over 5 years ago
needles are okay. agreed that we need to change default match level for vlc-url_check.
Updated by zluo over 5 years ago
http://f40.suse.de/tests/174#step/vlc/33
fixed now.
Updated by okurz over 5 years ago
I strongly recommend that you run tests not with installation in front all the time. I consider this really a waste of time and we can work way more efficiently by scheduling only the needed test modules based on loading qcow images. An alternative which you also seem to not use are the internal snapshots. Both approaches can work.
Now, do you plan to look into the actual issue of the double startup of vlc?
Updated by zluo over 5 years ago
the first step to reproduce the issue of double startup of course.
Updated by zluo over 5 years ago
https://openqa.opensuse.org/tests/818067#downloads
doesn't show logs.
Updated by zluo over 5 years ago
http://f40.suse.de/tests/228#next_previous
200 test runs doesn't show up single failure or the problem with starting vlc.
Updated by zluo over 5 years ago
Updated by okurz over 5 years ago
Your tests are fine locally, I can see that. Please see https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation for a better way how to visualize test results with a "build" which shows up in the test overview as well. Also, the original scenario is "kde-wayland@virtio" which most certainly has an impact.
Updated by zluo over 5 years ago
- Status changed from In Progress to Feedback
https://openqa.opensuse.org/tests/818883#step/vlc/2 shows old problem with vlc started twice. xterm is typed here which is totally wrong. compared with https://openqa.opensuse.org/tests/832488#step/vlc/1
So this typing issue happened sometimes and we can hardly fix it which is more related to qemu or setup/performance.
At least this issue with vlc started twice doesn't happened for a month. Let's keep it under observation. Set is now as feedback.
Updated by okurz over 5 years ago
- Status changed from Feedback to In Progress
Again, please refrain from just mentioning single jobs. I suggested already to collect better statistical information so that we have something that is easier to compare. Regarding any "qemu or setup/performance" issues which you mentioned: They also need us to change something, be it in tests, in our test environment or in the products that we ship. Just stating the fact does not make the flaky issues "go away" though. The ticket is already 5 months old and I do not see a reason why we should keep it dangling around in "feedback" for longer when we did not apply any actual changes. If you do not want to go further then please just put the ticket back to "Workable" and unassign.
Updated by zluo over 5 years ago
- Status changed from In Progress to Workable
- Assignee deleted (
zluo)
please someone can take over, I don't see a way to fix typing issue which doesn't happen at moment on osd.
Updated by okurz over 5 years ago
- Blocked by action #42671: [opensuse][functional][u] test fails in vlc - needle vlc-done-playing does not match added
Updated by okurz over 5 years ago
- Status changed from Workable to Blocked
- Assignee set to okurz
Updated by okurz over 5 years ago
- Status changed from Blocked to Feedback
blocker resolved, VLC should be more stable now. Previously there was an error rate of 3.8% in VLC not starting to play videos in the upgrade scenarios. In all my investigations in #42674 with about 400 jobs I have not seen VLC started twice anywhere even though I scheduled the tests with all the modules in before so not like a predecessor test module would trigger something in the background. However, #42674 was visible in non-wayland/non-virtio upgrade jobs and this one here is about wayland so I will gather the statistics from the originally failing scenario:
env end=40 name=okurz_investigation_poo39926_kde-wayland openqa-clone-set https://openqa.opensuse.org/tests/842199
all of them got obsoleted by a new TW snapshot. So let's retrigger based on the latest snapshot but excluding what is unstable in there, that is ooffice,oomath,oocalc:
env end=40 name=okurz_investigation_poo39926_kde-wayland openqa-clone-set https://openqa.opensuse.org/tests/843795 EXCLUDE_MODULES=ooffice,oomath,oocalc
-> https://openqa.opensuse.org/t843961
Oh no, I did a mistake there. openqa-clone-set takes as second parameter not additional job settings but the build and test name. So deleting the wrongly created jobs again with:
for i in $(openqa_client_o3 --json-output jobs build=EXCLUDE_MODULES=ooffice,oomath,oocalc version=Tumbleweed state=scheduled latest=1 | jq --raw-output --join-output '.jobs[] | (.id," ")') ; do openqa_client_o3 jobs/$i delete ; done
env end=40 openqa-clone-set https://openqa.opensuse.org/tests/843795 okurz_investigation_poo39926_kde-wayland EXCLUDE_MODULES=ooffice,oomath,oocalc
-> https://openqa.opensuse.org/tests/overview?build=okurz_investigation_poo39926_kde-wayland
Updated by okurz about 5 years ago
These results will be interesting to investigate. 34/40 failed in one of the following modules in order of highest occurences: firefox_audio, kontact, start_wayland_plasma5, inkscape, vlc, xterm, kate, dolphin, not counting the "+1" modules which could be "amarok" as seen in one. So vlc failed 1/40 but with the original problem reproduced, vlc showing up twice. It seems we have a big problem with krunner instability in general. I guess I should try the parameters as mentioned in #35589#note-74
"firefox_audio" seems to fail a lot because of the sound control slider know being blue instead of white/grey. I checked the video for "firefox_audio" and it looks like it turns from white/grey to blue after it completely played once. Could be that this is somehow faster on x11 but I guess could still happen. I created a new needle "firefox_audio--kde-20190204" with 92% match level. Let's see how new tests fare when I trigger them with extra kernel parameters if that works:
env end=4 openqa-clone-set https://openqa.opensuse.org/tests/843795 okurz_investigation_poo39926_kde-wayland_mitigation_off EXCLUDE_MODULES=ooffice,oomath,oocalc EXTRABOOTPARAMS="nopti nospec nospectre_v2 nospec_store_bypass_disable spectre_v2_user=off"
Failed due to outdated snapshot, of course.
env end=4 openqa-clone-set https://openqa.opensuse.org/tests/846143 okurz_investigation_poo39926_kde-wayland_mitigation_off EXCLUDE_MODULES=ooffice,oomath,oocalc EXTRABOOTPARAMS="nopti nospec nospectre_v2 nospec_store_bypass_disable spectre_v2_user=off"
Updated by okurz about 5 years ago
- Status changed from Feedback to Blocked
I come to the conclusion that the performance on current Tumbleweed+virtio+wayland is really mediocre which might be just https://bugzilla.opensuse.org/show_bug.cgi?id=1112824#c143 same as observed in #35589 . I guess it is a good idea to make krunner more stable in any case, e.g. slower typing and more waiting on wayland+virtio -> Let's do that in #35589 so blocked here
Updated by okurz about 5 years ago
- Target version changed from Milestone 22 to Milestone 25
Updated by okurz almost 5 years ago
- Assignee changed from okurz to mgriessmeier
Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.
Updated by okurz almost 5 years ago
- Subject changed from [functional][u][sporadic] test fails in vlc - vlc is started twice, needle target_match_vlc seems wrong to [functional][u][sporadic][virtio] test fails in vlc - vlc is started twice, needle target_match_vlc seems wrong
Updated by mgriessmeier almost 5 years ago
- Target version changed from Milestone 25 to Milestone 26
Updated by mgriessmeier over 4 years ago
- Status changed from Blocked to New
- Assignee deleted (
mgriessmeier) - Priority changed from Normal to High
- Target version changed from Milestone 26 to Milestone 27
to be groomed
Updated by SLindoMansilla over 4 years ago
- Status changed from New to Rejected
This issue is not happening anymore.
The new sporadic issue found can be seen here: https://openqa.opensuse.org/tests/1017139#step/vlc/27
And the last good is here: https://openqa.opensuse.org/tests/1017553#step/vlc/27
This "missing key" issue will be handled in #55703
Updated by SLindoMansilla over 4 years ago
- Related to action #55703: send_keys() doesn't guarantee that the application will handle modifiers+normal_key properly added