Project

General

Profile

Actions
Copy link

action #66718

closed

unstable/flaky/sporadic test: t/ui/27-plugin_obs_rsync_status_details.t

Added by okurz over 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Low
Assignee:
okurz
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2020-05-12
Due date:
2021-02-23
% Done:

0%

Estimated time:

Description

Observation

See https://app.circleci.com/pipelines/github/os-autoinst/openQA/2827/workflows/13672907-1ad3-48d0-a1b2-e210991ed1f8/jobs/26708/steps : 

[22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 19/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 24/? 
#   Failed test 'BatchedProj obs builds'
#   at t/ui/27-plugin_obs_rsync_status_details.t line 181.
#          got: '470.2, 469.1'
#     expected: '4704, 4703, 470.2, 469.1'
[22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 27/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 28/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. …60/? # Looks like you failed 1 test of 60.
                                                              [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. Dubious, test returned 1 (wstat 256, 0x100)

Failed 1/60 subtests

t/ui/27-plugin_obs_rsync_status_details.t failed in 1/2 runs within a stability test

Acceptance criteria

  • AC1: t/ui/27-plugin_obs_rsync_status_details.t is stable in 20/20 tries
  • AC2: t/ui/27-plugin_obs_rsync_status_details.t is removed from .circleci/unstable_tests.txt

Suggestions

  • Run make test RETRY=20 STABILITY_TEST=1 TESTS=t/ui/27-plugin_obs_rsync_status_details.t locally to reproduce
  • Identify causes of sporadic failures and fix in either production code or tests
  • Prove stability locally and in CI
  • Remove from .circleci/unstable_tests.txt again

Related issues 1 (0 open — 1 closed)

Related to openQA Project (public) - action #89935: t/ui/27-plugin_obs_rsync_status_details.t fails in circleCI, master branch evenResolvedmkittler2021-03-112021-04-30

Actions
Actions
Copy link
 #1

Updated by okurz over 4 years ago

proposed the move of the test to unstables in https://github.com/os-autoinst/openQA/pull/3081

Actions
Copy link
 #2

Updated by okurz over 4 years ago

  • Target version set to Ready
Actions
Copy link
 #3

Updated by okurz about 4 years ago

  • Priority changed from Normal to Low
Actions
Copy link
 #4

Updated by Xiaojing_liu about 4 years ago

I cannot reproduce this issue in my local.
I run make test RETRY=20 STABILITY_TEST=1 TESTS=t/ui/27-plugin_obs_rsync_status_details.t, every time passed.
I also tried for i in $(seq 1 20); do sudo docker run -it --rm -v $PWD:/opt/testing_area localtest bash -c 'make test TESTS=t/ui/27-plugin_obs_rsync_status_details.t' >> 2; done, all tests passed. I am not sure if run it in docket is the way to ensure Prove stability in CI.

BTW, I modified the timelimit in this file, because it always returns error then the timelimit is 16.

server stopped

real    0m48.786s
user    0m0.075s
sys     0m0.049s
Actions
Copy link
 #5

Updated by okurz about 4 years ago

  • Target version changed from Ready to future

Thank you for checking. If it's this hard to reproduce let's keep it like it is for now and look into this sometime later.

Actions
Copy link
 #6

Updated by livdywan almost 4 years ago

Seems like PRs are hitting this again:

not ok 24 - BatchedProj obs builds
[debug] [xl23hAJV] POST "/admin/obs_rsync/BatchedProj/obs_builds_text"
[debug] [xl23hAJV] Routing to controller "OpenQA::Shared::Controller::Session" and action "ensure_operator"
[debug] [xl23hAJV] Routing to controller "OpenQA::WebAPI::Plugin::ObsRsync::Controller::Gru" and action "update_obs_builds_text"
[debug] [xl23hAJV] 200 OK (0.024276s, 41.193/s)
[debug] [C0j4SN9_] GET "/admin/obs_rsync/BatchedProj/obs_builds_text"
[debug] [C0j4SN9_] Routing to controller "OpenQA::Shared::Controller::Session" and action "ensure_operator"
[debug] [C0j4SN9_] Routing to controller "OpenQA::WebAPI::Plugin::ObsRsync::Controller::Gru" and action "get_obs_builds_text"
[debug] [C0j4SN9_] 200 OK (0.016182s, 61.797/s)

Seen this on https://github.com/os-autoinst/openQA/pull/3694 and https://github.com/os-autoinst/openQA/pull/3635

Actions
Copy link
 #7

Updated by okurz almost 4 years ago

  • Due date set to 2021-02-23
  • Status changed from Workable to Feedback
  • Assignee set to andriinikitin
  • Target version changed from future to Ready

In recent days we saw this even more. Gladly @andriinikitin had a good idea how to address this:
https://github.com/os-autoinst/openQA/pull/3732

Actions
Copy link
 #8

Updated by okurz almost 4 years ago

  • Status changed from Feedback to In Progress
  • Assignee changed from andriinikitin to okurz

I will conduct the stability tests and see if we can remove that from the unstable list

Actions
Copy link
 #9

Updated by okurz almost 4 years ago

Did a successful local run with

env runs=100 count_fail_ratio prove -l t/ui/27-plugin_obs_rsync_status_details.t

Testing in circleCI https://github.com/os-autoinst/openQA/pull/3752

Actions
Copy link
 #10

Updated by okurz almost 4 years ago

  • Status changed from In Progress to Feedback
Actions
Copy link
 #11

Updated by okurz almost 4 years ago

  • Status changed from Feedback to Resolved
Actions
Copy link
 #12

Updated by okurz almost 4 years ago

  • Related to action #89935: t/ui/27-plugin_obs_rsync_status_details.t fails in circleCI, master branch even added
Actions
Copy link

Also available in: Atom PDF