As suggested by ilausuch I am taking over for him.
Seems this does not fail in CI anymore since recent timeout bumps. I am investigating locally how long the test takes now.
Without changes in current state locally count_fail_ratio prove -l t/ui/26-jobs_restart.t
## Run 1
t/ui/26-jobs_restart.t .. 4/? Bailout called. Further testing stopped: findElement: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
FAILED--Further testing stopped: findElement: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
## count_fail_ratio: Run: 1. Fails: 1. Fail ratio 100%
## Run 2
t/ui/26-jobs_restart.t .. 5/? Bailout called. Further testing stopped: executeScript: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
FAILED--Further testing stopped: executeScript: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
## count_fail_ratio: Run: 2. Fails: 2. Fail ratio 100%
## Run 3
t/ui/26-jobs_restart.t .. 4/? Bailout called. Further testing stopped: getElementText: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
FAILED--Further testing stopped: getElementText: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
## count_fail_ratio: Run: 3. Fails: 3. Fail ratio 100%
Bumping timeout from 10s to 20s:
## Run 1
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 16 wallclock secs ( 0.11 usr 0.01 sys + 6.60 cusr 0.55 csys = 7.27 CPU)
Result: PASS
## count_fail_ratio: Run: 1. Fails: 0. Fail ratio 0%
## Run 2
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 17 wallclock secs ( 0.13 usr 0.00 sys + 6.63 cusr 0.61 csys = 7.37 CPU)
Result: PASS
## count_fail_ratio: Run: 2. Fails: 0. Fail ratio 0%
## Run 3
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 17 wallclock secs ( 0.12 usr 0.01 sys + 6.72 cusr 0.60 csys = 7.45 CPU)
Result: PASS
## count_fail_ratio: Run: 3. Fails: 0. Fail ratio 0%
## Run 4
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 16 wallclock secs ( 0.11 usr 0.02 sys + 6.73 cusr 0.51 csys = 7.37 CPU)
Result: PASS
## count_fail_ratio: Run: 4. Fails: 0. Fail ratio 0%
## Run 5
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 17 wallclock secs ( 0.12 usr 0.01 sys + 6.61 cusr 0.57 csys = 7.31 CPU)
Result: PASS
## count_fail_ratio: Run: 5. Fails: 0. Fail ratio 0%
Crosscheck on an older commit state of openQA, 80f3d69ae from 2021-06-11 reproduces the timeout problems so our codebase did not recently introduce such problems. Reproduced again on 627798bec from 2021-03-05 the last time when I was working on the same test file myself:
## Run 1
t/ui/26-jobs_restart.t .. 2/? Bailout called. Further testing stopped: getElementText: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:93
FAILED--Further testing stopped: getElementText: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:93
## count_fail_ratio: Run: 1. Fails: 1. Fail ratio 100%
## Run 2
t/ui/26-jobs_restart.t .. 6/? Bailout called. Further testing stopped: close: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:93
FAILED--Further testing stopped: close: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:93
## count_fail_ratio: Run: 2. Fails: 2. Fail ratio 100%
## Run 3
t/ui/26-jobs_restart.t .. 5/? Bailout called. Further testing stopped: switchToWindow: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:93
FAILED--Further testing stopped: switchToWindow: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:93
## count_fail_ratio: Run: 3. Fails: 3. Fail ratio 100%
## Run 4
t/ui/26-jobs_restart.t .. 6/? Bailout called. Further testing stopped: findElement: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:93
FAILED--Further testing stopped: findElement: Server returned error message Label not found for "last T2_SUBTEST_WRAPPER" at /usr/lib/perl5/vendor_perl/5.26.1/Test2/Hub/Subtest.pm line 67. at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:93
## count_fail_ratio: Run: 4. Fails: 4. Fail ratio 100%
so it's slower on CI and on other people's computers as well as mine (Leap 15.2, not quite up-to-date right now). So most likely hypothesis (not entirely serious) is that the climate change caused this.
https://github.com/os-autoinst/openQA/pull/4020
Still running a longer statistics evaluation in my local env with runs=100 count_fail_ratio prove -l t/ui/26-jobs_restart.t to decide if we can mark the test as stable again for CI. Testing for CI in https://github.com/os-autoinst/openQA/pull/4021
In some cases in between I got
## Run 36
t/ui/26-jobs_restart.t .. 2/?
# Failed test 'job is marked as restarted'
# at t/ui/26-jobs_restart.t line 125.
# got: 'minimalx@32bit'
# expected: 'minimalx@32bit (restarted)'
Bailout called. Further testing stopped: findChildElement: no such element: Unable to locate element: {"method":"xpath","selector":"./a[@title='new test']"} at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
FAILED--Further testing stopped: findChildElement: no such element: Unable to locate element: {"method":"xpath","selector":"./a[@title='new test']"} at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
## count_fail_ratio: Run: 36. Fails: 1. Fail ratio 2%
## Run 37
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 19 wallclock secs ( 0.13 usr 0.01 sys + 7.82 cusr 0.77 csys = 8.73 CPU)
Result: PASS
## count_fail_ratio: Run: 37. Fails: 1. Fail ratio 2%
## Run 38
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 18 wallclock secs ( 0.13 usr 0.01 sys + 7.87 cusr 0.67 csys = 8.68 CPU)
Result: PASS
## count_fail_ratio: Run: 38. Fails: 1. Fail ratio 2%
## Run 39
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 20 wallclock secs ( 0.12 usr 0.02 sys + 8.20 cusr 0.72 csys = 9.06 CPU)
Result: PASS
## count_fail_ratio: Run: 39. Fails: 1. Fail ratio 2%
## Run 40
t/ui/26-jobs_restart.t .. ok
All tests successful.
Files=1, Tests=10, 19 wallclock secs ( 0.12 usr 0.02 sys + 8.14 cusr 0.76 csys = 9.04 CPU)
Result: PASS
## count_fail_ratio: Run: 40. Fails: 1. Fail ratio 2%
## Run 41
t/ui/26-jobs_restart.t .. 3/?
# Failed test 'job is marked as restarted'
# at t/ui/26-jobs_restart.t line 125.
# got: 'minimalx@32bit'
# expected: 'minimalx@32bit (restarted)'
Bailout called. Further testing stopped: findChildElement: no such element: Unable to locate element: {"method":"xpath","selector":"./a[@title='new test']"} at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
FAILED--Further testing stopped: findChildElement: no such element: Unable to locate element: {"method":"xpath","selector":"./a[@title='new test']"} at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
## count_fail_ratio: Run: 41. Fails: 2. Fail ratio 4%
## Run 42
t/ui/26-jobs_restart.t .. 2/?
# Failed test 'job is marked as restarted'
# at t/ui/26-jobs_restart.t line 125.
# got: 'minimalx@32bit'
# expected: 'minimalx@32bit (restarted)'
Bailout called. Further testing stopped: findChildElement: no such element: Unable to locate element: {"method":"xpath","selector":"./a[@title='new test']"} at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
FAILED--Further testing stopped: findChildElement: no such element: Unable to locate element: {"method":"xpath","selector":"./a[@title='new test']"} at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
## count_fail_ratio: Run: 42. Fails: 3. Fail ratio 7%
## Run 43
t/ui/26-jobs_restart.t .. 2/?
# Failed test 'job is marked as restarted'
# at t/ui/26-jobs_restart.t line 125.
# got: 'minimalx@32bit'
# expected: 'minimalx@32bit (restarted)'
Bailout called. Further testing stopped: findChildElement: no such element: Unable to locate element: {"method":"xpath","selector":"./a[@title='new test']"} at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95
FAILED--Further testing stopped: findChildElement: no such element: Unable to locate element: {"method":"xpath","selector":"./a[@title='new test']"} at /home/okurz/local/os-autoinst/openQA/t/ui/../lib/OpenQA/SeleniumTest.pm:95