action #35302: [opensuse][functional][u][epic][sporadic] Various unstable tests on o3
[functional][u][sporadic][virtio] test fails in vlc - vlc is started twice, needle target_match_vlc seems wrong
openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-kde-wayland@64bit_virtio fails in
with vlc showing up twice. The needle check in https://openqa.opensuse.org/tests/735310#step/vlc/16 looks wrong, like looking for a string in a desktop runner but we should probably check for the VLC window to show up.
Probably sometimes, also depending on the desktop environment
Last good: 20180812
- gnome not matching in first try, pressing esc, trying again, then matching and looking for the main window
- kde matching on the string in the desktop runner
Always latest result in this scenario: latest
#3 Updated by okurz over 1 year ago
- Status changed from Rejected to Workable
- Assignee deleted (
- Target version changed from Milestone 21 to Milestone 22
problem still present and reproduced in more recent jobs as well, e.g. https://openqa.opensuse.org/tests/818067#step/vlc/22
#7 Updated by okurz over 1 year ago
Hi zluo, this test fails for you because you are using outdated needles. Please update your needle repo. This is something that you should do always before running tests.
Could you please describe again in written form what you think is the problem with this test module?
But also I can see that even the newest needles created, e.g. vlc-url_check-20190116.json, are using the default match level so that is not exactly very robust.
#10 Updated by okurz over 1 year ago
I strongly recommend that you run tests not with installation in front all the time. I consider this really a waste of time and we can work way more efficiently by scheduling only the needed test modules based on loading qcow images. An alternative which you also seem to not use are the internal snapshots. Both approaches can work.
Now, do you plan to look into the actual issue of the double startup of vlc?
#12 Updated by zluo over 1 year ago
doesn't show logs.
#15 Updated by zluo over 1 year ago
#16 Updated by okurz over 1 year ago
Your tests are fine locally, I can see that. Please see https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation for a better way how to visualize test results with a "build" which shows up in the test overview as well. Also, the original scenario is "kde-wayland@virtio" which most certainly has an impact.
#17 Updated by zluo over 1 year ago
- Status changed from In Progress to Feedback
https://openqa.opensuse.org/tests/818883#step/vlc/2 shows old problem with vlc started twice. xterm is typed here which is totally wrong. compared with https://openqa.opensuse.org/tests/832488#step/vlc/1
So this typing issue happened sometimes and we can hardly fix it which is more related to qemu or setup/performance.
At least this issue with vlc started twice doesn't happened for a month. Let's keep it under observation. Set is now as feedback.
#18 Updated by okurz over 1 year ago
- Status changed from Feedback to In Progress
Again, please refrain from just mentioning single jobs. I suggested already to collect better statistical information so that we have something that is easier to compare. Regarding any "qemu or setup/performance" issues which you mentioned: They also need us to change something, be it in tests, in our test environment or in the products that we ship. Just stating the fact does not make the flaky issues "go away" though. The ticket is already 5 months old and I do not see a reason why we should keep it dangling around in "feedback" for longer when we did not apply any actual changes. If you do not want to go further then please just put the ticket back to "Workable" and unassign.
#22 Updated by okurz over 1 year ago
- Status changed from Blocked to Feedback
blocker resolved, VLC should be more stable now. Previously there was an error rate of 3.8% in VLC not starting to play videos in the upgrade scenarios. In all my investigations in #42674 with about 400 jobs I have not seen VLC started twice anywhere even though I scheduled the tests with all the modules in before so not like a predecessor test module would trigger something in the background. However, #42674 was visible in non-wayland/non-virtio upgrade jobs and this one here is about wayland so I will gather the statistics from the originally failing scenario:
env end=40 name=okurz_investigation_poo39926_kde-wayland openqa-clone-set https://openqa.opensuse.org/tests/842199
all of them got obsoleted by a new TW snapshot. So let's retrigger based on the latest snapshot but excluding what is unstable in there, that is ooffice,oomath,oocalc:
env end=40 name=okurz_investigation_poo39926_kde-wayland openqa-clone-set https://openqa.opensuse.org/tests/843795 EXCLUDE_MODULES=ooffice,oomath,oocalc
Oh no, I did a mistake there.
openqa-clone-set takes as second parameter not additional job settings but the build and test name. So deleting the wrongly created jobs again with:
for i in $(openqa_client_o3 --json-output jobs build=EXCLUDE_MODULES=ooffice,oomath,oocalc version=Tumbleweed state=scheduled latest=1 | jq --raw-output --join-output '.jobs | (.id," ")') ; do openqa_client_o3 jobs/$i delete ; done
env end=40 openqa-clone-set https://openqa.opensuse.org/tests/843795 okurz_investigation_poo39926_kde-wayland EXCLUDE_MODULES=ooffice,oomath,oocalc
#23 Updated by okurz over 1 year ago
These results will be interesting to investigate. 34/40 failed in one of the following modules in order of highest occurences: firefox_audio, kontact, start_wayland_plasma5, inkscape, vlc, xterm, kate, dolphin, not counting the "+1" modules which could be "amarok" as seen in one. So vlc failed 1/40 but with the original problem reproduced, vlc showing up twice. It seems we have a big problem with krunner instability in general. I guess I should try the parameters as mentioned in #35589#note-74
"firefox_audio" seems to fail a lot because of the sound control slider know being blue instead of white/grey. I checked the video for "firefox_audio" and it looks like it turns from white/grey to blue after it completely played once. Could be that this is somehow faster on x11 but I guess could still happen. I created a new needle "firefox_audio--kde-20190204" with 92% match level. Let's see how new tests fare when I trigger them with extra kernel parameters if that works:
env end=4 openqa-clone-set https://openqa.opensuse.org/tests/843795 okurz_investigation_poo39926_kde-wayland_mitigation_off EXCLUDE_MODULES=ooffice,oomath,oocalc EXTRABOOTPARAMS="nopti nospec nospectre_v2 nospec_store_bypass_disable spectre_v2_user=off"
Failed due to outdated snapshot, of course.
env end=4 openqa-clone-set https://openqa.opensuse.org/tests/846143 okurz_investigation_poo39926_kde-wayland_mitigation_off EXCLUDE_MODULES=ooffice,oomath,oocalc EXTRABOOTPARAMS="nopti nospec nospectre_v2 nospec_store_bypass_disable spectre_v2_user=off"
#25 Updated by okurz over 1 year ago
- Status changed from Feedback to Blocked
I come to the conclusion that the performance on current Tumbleweed+virtio+wayland is really mediocre which might be just https://bugzilla.opensuse.org/show_bug.cgi?id=1112824#c143 same as observed in #35589 . I guess it is a good idea to make krunner more stable in any case, e.g. slower typing and more waiting on wayland+virtio -> Let's do that in #35589 so blocked here
#27 Updated by okurz about 1 year ago
- Assignee changed from okurz to mgriessmeier
Move to new QSF-u PO after I moved to the "tools"-team. I mainly checked the subject line so in individual instances you might not agree to take it over completely into QSF-u. Feel free to discuss with me or reassign to me or someone else in this case. Thanks.
#31 Updated by SLindoMansilla 11 months ago
- Status changed from New to Rejected
This issue is not happening anymore.
The new sporadic issue found can be seen here: https://openqa.opensuse.org/tests/1017139#step/vlc/27
And the last good is here: https://openqa.opensuse.org/tests/1017553#step/vlc/27
This "missing key" issue will be handled in #55703