Project

General

Profile

Actions
Copy link

action #80374

closed

unstable/flaky/sporadic test t/ui/25-developer_mode.t

Added by okurz about 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
mkittler
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2020-11-25
Due date:
% Done:

0%

Estimated time:

Description

Observation

https://app.circleci.com/pipelines/github/os-autoinst/openQA/5067/workflows/10cbcdac-cd57-4c67-8cb6-0483c8a98ba4/jobs/48181/steps shows that t/ui/25-developer_mode.t failed:

[16:58:27] t/ui/25-developer_mode.t ............... 8/? 
        #   Failed test 'foo selected'
        #   at t/ui/25-developer_mode.t line 447.
        #          got: '1'
        #     expected: '0'

        #   Failed test 'option #4 present'
        #   at t/ui/25-developer_mode.t line 449.
        # Looks like you failed 2 tests of 2.

    #   Failed test 'select module to pause at'
    #   at t/ui/25-developer_mode.t line 475.
    # Looks like you failed 1 test of 19.
[16:58:27] t/ui/25-developer_mode.t ............... 12/? 
#   Failed test 'start developer session'
#   at t/ui/25-developer_mode.t line 572.
[16:58:27] t/ui/25-developer_mode.t ............... 13/? [16:58:27] t/ui/25-developer_mode.t ............... 14/? # Looks like you failed 1 test of 14.
                                                          [16:58:27] t/ui/25-developer_mode.t ............... Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/14 subtests

I am not aware of seeing this problem in before. Has anyone else seen it yet?

Acceptance criteria

  • AC1: t/ui/25-developer_mode.t is stable

Acceptance tests

  • AT1-1: make test STABILITY_TEST=1 RETRIES=50 KEEP_DB=1 TESTS=1 t/ui/25-developer_mode.t
  • AT1-2: Same as AT1-1 but in circleCI

Suggestions

  • DONE: Try to reproduce locally or within CI -> happened multiple times in circleCI
  • DONE: If reproducible add to .circleci/unstable_tests.txt -> https://github.com/os-autoinst/openQA/pull/3884
  • Consider starting with improving the error output in case of failure
  • Fix the instability in the test

Related issues 1 (0 open — 1 closed)

Related to openQA Project (public) - action #91232: Test t/ui/25-developer_mode.t failed in CIRejectedmkittler2021-04-152021-04-30

Actions
Actions
Copy link
 #1

Updated by okurz over 3 years ago

  • Related to action #91232: Test t/ui/25-developer_mode.t failed in CI added
Actions
Copy link
 #2

Updated by okurz over 3 years ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz
  • Target version changed from future to Ready

happened again: https://circleci.com/gh/os-autoinst/openQA/60185 but a slightly different error (similar code region):

[13:19:00] t/ui/25-developer_mode.t ............... 8/? 
        #   Failed test 'foo selected (Do not pause at a certain module)'
        #   at t/ui/25-developer_mode.t line 448.
        #          got: '1'
        #     expected: '0'

        #   Failed test 'option #4 present'
        #   at t/ui/25-developer_mode.t line 450.
        # Looks like you failed 2 tests of 2.

    #   Failed test 'select module to pause at'
    #   at t/ui/25-developer_mode.t line 476.
    # Looks like you failed 1 test of 19.
Actions
Copy link
 #4

Updated by okurz over 3 years ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #5

Updated by okurz over 3 years ago

  • Description updated (diff)
  • Status changed from Feedback to Workable
  • Assignee deleted (okurz)
Actions
Copy link
 #6

Updated by mkittler over 3 years ago 

[15:32:32] t/ui/25-developer_mode.t .. 11/? 
    #   Failed test 'changes submitted'
    #   at t/ui/25-developer_mode.t line 101.
    #     Structures begin differing at:
    #          $got->[0]{name} = undef
    #     $expected->[0]{name} = 'installation-bar'

        #   Failed test 'foo selected (Do not pause at a certain module)'
        #   at t/ui/25-developer_mode.t line 448.
        #          got: '1'
        #     expected: '0'

        #   Failed test 'option #4 present'
        #   at t/ui/25-developer_mode.t line 450.
        # Looks like you failed 2 tests of 2.

    #   Failed test 'select module to pause at'
    #   at t/ui/25-developer_mode.t line 476.
    # Looks like you failed 2 tests of 19.
[15:32:32] t/ui/25-developer_mode.t .. 12/? 
#   Failed test 'start developer session'
#   at t/ui/25-developer_mode.t line 573.
[15:32:32] t/ui/25-developer_mode.t .. 13/? [15:32:32] t/ui/25-developer_mode.t .. 14/? # Looks like you failed 1 test of 14.
                                             [15:32:32] t/ui/25-developer_mode.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/14 subtests 
[15:33:17]

Test Summary Report
-------------------
t/ui/25-developer_mode.t (Wstat: 256 Tests: 14 Failed: 1)
  Failed test:  12
  Non-zero exit status: 1
Files=1, Tests=14, 45.4141 wallclock secs ( 0.14 usr  0.01 sys + 24.97 cusr  0.81 csys = 25.93 CPU)
Result: FAIL
Retry 2 of 5 …
[15:33:18] t/ui/25-developer_mode.t .. 9/? Bailout called.  Further testing stopped:  clickElement: stale element reference: element is not attached to the page document at /home/squamata/project/t/ui/../lib/OpenQA/SeleniumTest.pm:93
FAILED--Further testing stopped: clickElement: stale element reference: element is not attached to the page document at /home/squamata/project/t/ui/../lib/OpenQA/SeleniumTest.pm:93
Retry 3 of 5 …
[15:33:56] t/ui/25-developer_mode.t .. 9/? 
    #   Failed test 'option #5 present'
    #   at t/ui/25-developer_mode.t line 369.
    # Looks like you failed 1 test of 2.

#   Failed test 'start developer session'
#   at t/ui/25-developer_mode.t line 573.

        #   Failed test 'selection for pausing on screen mismatch updated'
        #   at t/ui/25-developer_mode.t line 613.
        #          got: '0'
        #     expected: '1'
        # Looks like you failed 1 test of 11.

    #   Failed test 'testname and paused state updated'
    #   at t/ui/25-developer_mode.t line 614.
    # Looks like you failed 1 test of 5.
[15:33:56] t/ui/25-developer_mode.t .. 13/? 
#   Failed test 'process state changes from os-autoinst/worker'
#   at t/ui/25-developer_mode.t line 737.
[15:33:56] t/ui/25-developer_mode.t .. 14/? # Looks like you failed 2 tests of 14.
                                             [15:33:56] t/ui/25-developer_mode.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/14 subtests 
[15:34:39]

Test Summary Report
-------------------
t/ui/25-developer_mode.t (Wstat: 512 Tests: 14 Failed: 2)
  Failed tests:  12-13
  Non-zero exit status: 2
Files=1, Tests=14, 43.3515 wallclock secs ( 0.11 usr  0.02 sys + 24.46 cusr  0.79 csys = 25.38 CPU)
Result: FAIL
Retry 4 of 5 …
[15:34:40] t/ui/25-developer_mode.t .. 8/? 
    #   Failed test 'option #5 present'
    #   at t/ui/25-developer_mode.t line 369.
    # Looks like you failed 1 test of 2.

#   Failed test 'start developer session'
#   at t/ui/25-developer_mode.t line 573.

        #   Failed test 'selection for pausing on screen mismatch updated'
        #   at t/ui/25-developer_mode.t line 613.
        #          got: '0'
        #     expected: '1'
        # Looks like you failed 1 test of 11.

    #   Failed test 'testname and paused state updated'
    #   at t/ui/25-developer_mode.t line 614.
[15:34:40] t/ui/25-developer_mode.t .. 13/?     # Looks like you failed 1 test of 5.

#   Failed test 'process state changes from os-autoinst/worker'
#   at t/ui/25-developer_mode.t line 737.
[15:34:40] t/ui/25-developer_mode.t .. 14/? # Looks like you failed 2 tests of 14.
                                             [15:34:40] t/ui/25-developer_mode.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/14 subtests 
[15:35:25]

Test Summary Report
-------------------
t/ui/25-developer_mode.t (Wstat: 512 Tests: 14 Failed: 2)
  Failed tests:  12-13
  Non-zero exit status: 2
Files=1, Tests=14, 44.5828 wallclock secs ( 0.11 usr  0.00 sys + 24.68 cusr  0.82 csys = 25.61 CPU)
Result: FAIL
Retry 5 of 5 …
[15:35:26] t/ui/25-developer_mode.t .. 8/? 
    #   Failed test 'changes submitted'
    #   at t/ui/25-developer_mode.t line 101.
    #     Structures begin differing at:
    #          $got->[0]{name} = undef
    #     $expected->[0]{name} = 'installation-bar'

        #   Failed test 'foo selected (Do not pause at a certain module)'
        #   at t/ui/25-developer_mode.t line 448.
        #          got: '1'
        #     expected: '0'

        #   Failed test 'option #4 present'
        #   at t/ui/25-developer_mode.t line 450.
        # Looks like you failed 2 tests of 2.

    #   Failed test 'select module to pause at'
    #   at t/ui/25-developer_mode.t line 476.
    # Looks like you failed 2 tests of 19.
[15:35:26] t/ui/25-developer_mode.t .. 12/? 
#   Failed test 'start developer session'
#   at t/ui/25-developer_mode.t line 573.
[15:35:26] t/ui/25-developer_mode.t .. 13/? [15:35:26] t/ui/25-developer_mode.t .. 14/? # Looks like you failed 1 test of 14.
                                             [15:35:26] t/ui/25-developer_mode.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/14 subtests 
[15:36:11]

Test Summary Report
-------------------
t/ui/25-developer_mode.t (Wstat: 256 Tests: 14 Failed: 1)
  Failed test:  12
  Non-zero exit status: 1
Files=1, Tests=14, 44.9217 wallclock secs ( 0.13 usr  0.01 sys + 25.05 cusr  0.77 csys = 25.96 CPU)
Result: FAIL
make[2]: *** [Makefile:187: test-unit-and-integration] Error 1
make[2]: Leaving directory '/home/squamata/project'
make[1]: *** [Makefile:182: test-with-database] Error 2
make[1]: Leaving directory '/home/squamata/project'
make: *** [Makefile:167: test-unstable] Error 2
Actions
Copy link
 #7

Updated by mkittler over 3 years ago

  • Status changed from Workable to In Progress
  • Assignee set to mkittler
Actions
Copy link
 #8

Updated by mkittler over 3 years ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #9

Updated by mkittler over 3 years ago

  • Status changed from Feedback to Resolved

PR has been merged

Actions
Copy link

Also available in: Atom PDF