Project

General

Profile

Actions

action #132410

closed

[sporadic][ci] t/ui/18-tests-details.t fails size:M

Added by tinita over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2023-07-06
Due date:
2023-08-04
% Done:

0%

Estimated time:

Description

Observation

In this PR, the test failed, but it is unrelated:
https://app.circleci.com/pipelines/github/os-autoinst/openQA/11886/workflows/0f8caf6c-6f51-4666-940f-ed2f19b14e33/jobs/111144

[13:52:43] t/ui/18-tests-details.t .................... 1/? 
    #   Failed test 'correct tags displayed'
    #   at t/ui/18-tests-details.t line 154.
    #     Structures begin differing at:
    #          $got->{inst-bootmenu} = Does not exist
    #     $expected->{inst-bootmenu} = ARRAY(0x5645b375db90)
    # Looks like you failed 1 test of 4.
[13:52:43] t/ui/18-tests-details.t .................... 4/? 
#   Failed test 'displaying image result with candidates'
#   at t/ui/18-tests-details.t line 155.
[13:52:43] t/ui/18-tests-details.t .................... 5/?
 [13:52:43] t/ui/18-tests-details.t .................... 9/?
 [13:52:43] t/ui/18-tests-details.t .................... 10/?
 [13:52:43] t/ui/18-tests-details.t .................... 12/?
 [13:52:43] t/ui/18-tests-details.t .................... 13/?
 [13:52:43] t/ui/18-tests-details.t .................... 14/?
 [13:52:43] t/ui/18-tests-details.t .................... 15/?
 [13:52:43] t/ui/18-tests-details.t .................... 18/?
 [13:52:43] t/ui/18-tests-details.t .................... 28/?
 [13:52:43] t/ui/18-tests-details.t .................... 32/?
 [13:52:43] t/ui/18-tests-details.t .................... 35/? # Looks like you failed 1 test of 35.
[13:52:43] t/ui/18-tests-details.t .................... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/35 subtests

Acceptance criteria

  • AC1: t/ui/18-details.t is sufficiently stable with less than 1/1000 failures

Suggestions

  • Use https://github.com/okurz/scripts/blob/master/count_fail_ratio, try count_fail_ratio prove -l t/ui/18-test-details.t locally and try to reproduce and find out the current error rate
  • If not reproducible locally then try within a CI(-like) environment, otherwise if still not reproducible reject
  • Narrow down the error and fix
Actions #1

Updated by tinita over 1 year ago

  • Description updated (diff)
Actions #2

Updated by okurz over 1 year ago

  • Priority changed from Normal to High
  • Target version set to Ready
Actions #3

Updated by okurz about 1 year ago

  • Subject changed from t/ui/18-tests-details.t fails to [sporadic][ci] t/ui/18-tests-details.t fails size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #4

Updated by livdywan about 1 year ago

I seem to be getting different failures:

LANG=en_US make test TESTS=t/ui/18-tests-details.t
[...]
t/ui/18-tests-details.t .. 4/?
    #   Failed test 'name filter initially not displayed'
    #   at t/ui/18-tests-details.t line 172.

    #   Failed test 'failed filter initially not displayed'
    #   at t/ui/18-tests-details.t line 173.
    # Looks like you failed 2 tests of 17.
#   Failed test 'filtering'
#   at t/ui/18-tests-details.t line 208.
t/ui/18-tests-details.t .. 10/?
    #   Failed test 'url contains step'
    #   at t/ui/18-tests-details.t line 372.
    #                   'https://bugzilla.opensuse.org/enter_bug.cgi?product=openSUSE+Distribution&bug_file_loc=http%3A%2F%2Flocalhost
%3A35877%2Ftests%2F99946%2Fmodules%2Flogpackages%2Fsteps%2F6&comment=%23%23+Observation%0A%0AopenQA+test+in+scenario+opensuse-13.1-DVD
-i586-textmode%4032bit+fails+in%0A%5Blogpackages%5D%28http%3A%2F%2Flocalhost%3A35877%2Ftests%2F99946%2Fmodules%2Flogpackages%2Fsteps%2
F6%29%0A%0A%23%23+Test+suite+description%0A%0A%0A%0A%23%23+Reproducible%0A%0AFails+since+%28at+least%29+Build+%5B0091%5D%28http%3A%2F%
2Flocalhost%3A35877%2Ftests%2F99946%29+%28current+job%29%0A%0A%0A%23%23+Expected+result%0A%0ALast+good%3A+%5B0091%5D%28http%3A%2F%2Flo
calhost%3A35877%2Ftests%2F99945%29+%28or+more+recent%29%0A%0A%0A%23%23+Further+details%0A%0AAlways+latest+result+in+this+scenario%3A+%
5Blatest%5D%28http%3A%2F%2Flocalhost%3A35877%2Ftests%2Flatest%3Farch%3Di586%26distri%3Dopensuse%26flavor%3DDVD%26machine%3D32bit%26tes
t%3Dtextmode%26version%3D13.1%29%0A&short_desc=%5BBuild+0091%5D+openQA+test+fails+in+logpackages&cf_foundby=openQA&cf_blocker=Yes'
    #     doesn't match '(?^:#step/logpackages/6)'
    # Looks like you failed 1 test of 1.
t/ui/18-tests-details.t .. 12/?
#   Failed test 'render text results'
#   at t/ui/18-tests-details.t line 420.
findElement: no such element: Unable to locate element: {"method":"css selector","selector":".current_preview .resborder"} at ~/bau/oa/repos/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:78 at ~/bau/oa/repos/openQA/t/ui/../lib/OpenQA/SeleniumTest.p
m line 81.
Actions #5

Updated by livdywan about 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to livdywan

We discussed this in the Unblock. I'm still getting the above error. Tina couldn't reproduce this or the original error the ticket is about.
NOT_HEADLESS=1 might help me debug, and I'll upgrade my Tumbleweed container as well

Actions #6

Updated by livdywan about 1 year ago

  • Status changed from In Progress to Feedback

dup -dy wasn't enough. git clean -fd seems to have fixed the issue... so I'm guessing there is some clean-up that we are missing. I still can't reproduce the original error.

Actions #7

Updated by livdywan about 1 year ago

  • Status changed from Feedback to In Progress
Actions #8

Updated by openqa_review about 1 year ago

  • Due date set to 2023-08-04

Setting due date based on mean cycle time of SUSE QE Tools

Actions #9

Updated by livdywan about 1 year ago

  • Status changed from In Progress to Feedback
Actions #10

Updated by livdywan about 1 year ago

  • Status changed from Feedback to Resolved
for i in $(seq 1 1000); do LANG=en_US make test TESTS=t/ui/18-tests-details.t || break; done

No failures, hence resolving.

Actions

Also available in: Atom PDF