action #39926

action #35302: [opensuse][functional][u][epic][sporadic] Various unstable tests on o3

[functional][u][sporadic][virtio] test fails in vlc - vlc is started twice, needle target_match_vlc seems wrong

Added by okurz over 1 year ago. Updated 8 months ago.

Status:RejectedStart date:17/08/2018
Priority:HighDue date:
Assignee:-% Done:

0%

Category:Bugs in existing tests
Target version:SUSE QA tests - Milestone 27
Difficulty:
Duration:

Description

Observation

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-kde-wayland@64bit_virtio fails in
vlc
with vlc showing up twice. The needle check in https://openqa.opensuse.org/tests/735310#step/vlc/16 looks wrong, like looking for a string in a desktop runner but we should probably check for the VLC window to show up.

Reproducible

Probably sometimes, also depending on the desktop environment

Expected result

Last good: 20180812

For comparison:

  • gnome not matching in first try, pressing esc, trying again, then matching and looking for the main window
  • kde matching on the string in the desktop runner

Further details

Always latest result in this scenario: latest


Related issues

Related to openQA Tests - action #42674: [opensuse][functional][u][sporadic] test fails in vlc - c... Resolved 18/10/2018
Related to openQA Project - action #55703: send_keys() doesn't guarantee that the application will h... Resolved 15/04/2019
Blocked by openQA Tests - action #42671: [opensuse][functional][u] test fails in vlc - needle vlc-... Resolved 18/10/2018

History

#1 Updated by okurz over 1 year ago

  • Target version changed from Milestone 19 to Milestone 21

#2 Updated by okurz over 1 year ago

  • Status changed from New to Rejected
  • Assignee set to okurz

#3 Updated by okurz over 1 year ago

  • Status changed from Rejected to Workable
  • Assignee deleted (okurz)
  • Target version changed from Milestone 21 to Milestone 22

problem still present and reproduced in more recent jobs as well, e.g. https://openqa.opensuse.org/tests/818067#step/vlc/22

#4 Updated by okurz about 1 year ago

  • Related to action #42674: [opensuse][functional][u][sporadic] test fails in vlc - cannot open video added

#5 Updated by zluo about 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to zluo

take over and check

#6 Updated by zluo about 1 year ago

test run on my server:

http://f40.suse.de/tests/172#step/vlc/24

95% match but failed...

#7 Updated by okurz about 1 year ago

Hi zluo, this test fails for you because you are using outdated needles. Please update your needle repo. This is something that you should do always before running tests.

Could you please describe again in written form what you think is the problem with this test module?

But also I can see that even the newest needles created, e.g. vlc-url_check-20190116.json, are using the default match level so that is not exactly very robust.

#8 Updated by zluo about 1 year ago

needles are okay. agreed that we need to change default match level for vlc-url_check.

#10 Updated by okurz about 1 year ago

I strongly recommend that you run tests not with installation in front all the time. I consider this really a waste of time and we can work way more efficiently by scheduling only the needed test modules based on loading qcow images. An alternative which you also seem to not use are the internal snapshots. Both approaches can work.

Now, do you plan to look into the actual issue of the double startup of vlc?

#11 Updated by zluo about 1 year ago

the first step to reproduce the issue of double startup of course.

#13 Updated by zluo about 1 year ago

@okurz to save time of course :)

#14 Updated by zluo about 1 year ago

http://f40.suse.de/tests/228#next_previous

200 test runs doesn't show up single failure or the problem with starting vlc.

#16 Updated by okurz about 1 year ago

Your tests are fine locally, I can see that. Please see https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation for a better way how to visualize test results with a "build" which shows up in the test overview as well. Also, the original scenario is "kde-wayland@virtio" which most certainly has an impact.

#17 Updated by zluo about 1 year ago

  • Status changed from In Progress to Feedback

https://openqa.opensuse.org/tests/818883#step/vlc/2 shows old problem with vlc started twice. xterm is typed here which is totally wrong. compared with https://openqa.opensuse.org/tests/832488#step/vlc/1

So this typing issue happened sometimes and we can hardly fix it which is more related to qemu or setup/performance.
At least this issue with vlc started twice doesn't happened for a month. Let's keep it under observation. Set is now as feedback.

#18 Updated by okurz about 1 year ago

  • Status changed from Feedback to In Progress

Again, please refrain from just mentioning single jobs. I suggested already to collect better statistical information so that we have something that is easier to compare. Regarding any "qemu or setup/performance" issues which you mentioned: They also need us to change something, be it in tests, in our test environment or in the products that we ship. Just stating the fact does not make the flaky issues "go away" though. The ticket is already 5 months old and I do not see a reason why we should keep it dangling around in "feedback" for longer when we did not apply any actual changes. If you do not want to go further then please just put the ticket back to "Workable" and unassign.

#19 Updated by zluo about 1 year ago

  • Status changed from In Progress to Workable
  • Assignee deleted (zluo)

please someone can take over, I don't see a way to fix typing issue which doesn't happen at moment on osd.

#20 Updated by okurz about 1 year ago

  • Blocked by action #42671: [opensuse][functional][u] test fails in vlc - needle vlc-done-playing does not match added

#21 Updated by okurz about 1 year ago

  • Status changed from Workable to Blocked
  • Assignee set to okurz

#22 Updated by okurz about 1 year ago

  • Status changed from Blocked to Feedback

blocker resolved, VLC should be more stable now. Previously there was an error rate of 3.8% in VLC not starting to play videos in the upgrade scenarios. In all my investigations in #42674 with about 400 jobs I have not seen VLC started twice anywhere even though I scheduled the tests with all the modules in before so not like a predecessor test module would trigger something in the background. However, #42674 was visible in non-wayland/non-virtio upgrade jobs and this one here is about wayland so I will gather the statistics from the originally failing scenario:

env end=40 name=okurz_investigation_poo39926_kde-wayland openqa-clone-set https://openqa.opensuse.org/tests/842199

-> https://openqa.opensuse.org/tests/overview?build=okurz_investigation&distri=opensuse&version=Tumbleweed

all of them got obsoleted by a new TW snapshot. So let's retrigger based on the latest snapshot but excluding what is unstable in there, that is ooffice,oomath,oocalc:

env end=40 name=okurz_investigation_poo39926_kde-wayland openqa-clone-set https://openqa.opensuse.org/tests/843795 EXCLUDE_MODULES=ooffice,oomath,oocalc

-> https://openqa.opensuse.org/t843961

Oh no, I did a mistake there. openqa-clone-set takes as second parameter not additional job settings but the build and test name. So deleting the wrongly created jobs again with:

for i in $(openqa_client_o3 --json-output jobs build=EXCLUDE_MODULES=ooffice,oomath,oocalc version=Tumbleweed state=scheduled latest=1 | jq --raw-output --join-output  '.jobs[] | (.id," ")') ; do openqa_client_o3 jobs/$i delete ; done
env end=40 openqa-clone-set https://openqa.opensuse.org/tests/843795 okurz_investigation_poo39926_kde-wayland EXCLUDE_MODULES=ooffice,oomath,oocalc

-> https://openqa.opensuse.org/tests/overview?build=okurz_investigation_poo39926_kde-wayland

#23 Updated by okurz about 1 year ago

These results will be interesting to investigate. 34/40 failed in one of the following modules in order of highest occurences: firefox_audio, kontact, start_wayland_plasma5, inkscape, vlc, xterm, kate, dolphin, not counting the "+1" modules which could be "amarok" as seen in one. So vlc failed 1/40 but with the original problem reproduced, vlc showing up twice. It seems we have a big problem with krunner instability in general. I guess I should try the parameters as mentioned in #35589#note-74

"firefox_audio" seems to fail a lot because of the sound control slider know being blue instead of white/grey. I checked the video for "firefox_audio" and it looks like it turns from white/grey to blue after it completely played once. Could be that this is somehow faster on x11 but I guess could still happen. I created a new needle "firefox_audio--kde-20190204" with 92% match level. Let's see how new tests fare when I trigger them with extra kernel parameters if that works:

env end=4 openqa-clone-set https://openqa.opensuse.org/tests/843795 okurz_investigation_poo39926_kde-wayland_mitigation_off EXCLUDE_MODULES=ooffice,oomath,oocalc EXTRABOOTPARAMS="nopti nospec nospectre_v2 nospec_store_bypass_disable spectre_v2_user=off"

-> https://openqa.opensuse.org/tests/overview?build=okurz_investigation_poo39926_kde-wayland_mitigation_off&version=Tumbleweed&distri=opensuse

Failed due to outdated snapshot, of course.

env end=4 openqa-clone-set https://openqa.opensuse.org/tests/846143 okurz_investigation_poo39926_kde-wayland_mitigation_off EXCLUDE_MODULES=ooffice,oomath,oocalc EXTRABOOTPARAMS="nopti nospec nospectre_v2 nospec_store_bypass_disable spectre_v2_user=off"

-> https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=okurz_investigation_poo39926_kde-wayland_mitigation_off

#24 Updated by okurz about 1 year ago

  • Parent task set to #35302

#25 Updated by okurz about 1 year ago

  • Status changed from Feedback to Blocked

I come to the conclusion that the performance on current Tumbleweed+virtio+wayland is really mediocre which might be just https://bugzilla.opensuse.org/show_bug.cgi?id=1112824#c143 same as observed in #35589 . I guess it is a good idea to make krunner more stable in any case, e.g. slower typing and more waiting on wayland+virtio -> Let's do that in #35589 so blocked here

#26 Updated by okurz about 1 year ago

  • Target version changed from Milestone 22 to Milestone 25

#27 Updated by okurz 11 months ago

  • Assignee changed from okurz to mgriessmeier

Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.

#28 Updated by okurz 9 months ago

  • Subject changed from [functional][u][sporadic] test fails in vlc - vlc is started twice, needle target_match_vlc seems wrong to [functional][u][sporadic][virtio] test fails in vlc - vlc is started twice, needle target_match_vlc seems wrong

#29 Updated by mgriessmeier 9 months ago

  • Target version changed from Milestone 25 to Milestone 26

#30 Updated by mgriessmeier 8 months ago

  • Status changed from Blocked to New
  • Assignee deleted (mgriessmeier)
  • Priority changed from Normal to High
  • Target version changed from Milestone 26 to Milestone 27

to be groomed

#31 Updated by SLindoMansilla 8 months ago

  • Status changed from New to Rejected

This issue is not happening anymore.

The new sporadic issue found can be seen here: https://openqa.opensuse.org/tests/1017139#step/vlc/27
And the last good is here: https://openqa.opensuse.org/tests/1017553#step/vlc/27

This "missing key" issue will be handled in #55703

#32 Updated by SLindoMansilla 8 months ago

  • Related to action #55703: send_keys() doesn't guarantee that the application will handle modifiers+normal_key properly added

Also available in: Atom PDF