Project

General

Profile

Actions

action #66718

closed

unstable/flaky/sporadic test: t/ui/27-plugin_obs_rsync_status_details.t

Added by okurz almost 4 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2020-05-12
Due date:
2021-02-23
% Done:

0%

Estimated time:

Description

Observation

See https://app.circleci.com/pipelines/github/os-autoinst/openQA/2827/workflows/13672907-1ad3-48d0-a1b2-e210991ed1f8/jobs/26708/steps :

[22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 19/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 24/? 
#   Failed test 'BatchedProj obs builds'
#   at t/ui/27-plugin_obs_rsync_status_details.t line 181.
#          got: '470.2, 469.1'
#     expected: '4704, 4703, 470.2, 469.1'
[22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 27/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 28/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. …60/? # Looks like you failed 1 test of 60.
                                                              [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. Dubious, test returned 1 (wstat 256, 0x100)

Failed 1/60 subtests 

t/ui/27-plugin_obs_rsync_status_details.t failed in 1/2 runs within a stability test

Acceptance criteria

  • AC1: t/ui/27-plugin_obs_rsync_status_details.t is stable in 20/20 tries
  • AC2: t/ui/27-plugin_obs_rsync_status_details.t is removed from .circleci/unstable_tests.txt

Suggestions

  • Run make test RETRY=20 STABILITY_TEST=1 TESTS=t/ui/27-plugin_obs_rsync_status_details.t locally to reproduce
  • Identify causes of sporadic failures and fix in either production code or tests
  • Prove stability locally and in CI
  • Remove from .circleci/unstable_tests.txt again

Related issues 1 (0 open1 closed)

Related to openQA Project - action #89935: t/ui/27-plugin_obs_rsync_status_details.t fails in circleCI, master branch evenResolvedmkittler2021-03-112021-04-30

Actions
Actions #1

Updated by okurz almost 4 years ago

proposed the move of the test to unstables in https://github.com/os-autoinst/openQA/pull/3081

Actions #2

Updated by okurz over 3 years ago

  • Target version set to Ready
Actions #3

Updated by okurz over 3 years ago

  • Priority changed from Normal to Low
Actions #4

Updated by Xiaojing_liu over 3 years ago

I cannot reproduce this issue in my local.
I run make test RETRY=20 STABILITY_TEST=1 TESTS=t/ui/27-plugin_obs_rsync_status_details.t, every time passed.
I also tried for i in $(seq 1 20); do sudo docker run -it --rm -v $PWD:/opt/testing_area localtest bash -c 'make test TESTS=t/ui/27-plugin_obs_rsync_status_details.t' >> 2; done, all tests passed. I am not sure if run it in docket is the way to ensure Prove stability in CI.

BTW, I modified the timelimit in this file, because it always returns error then the timelimit is 16.

server stopped

real    0m48.786s
user    0m0.075s
sys     0m0.049s
Actions #5

Updated by okurz over 3 years ago

  • Target version changed from Ready to future

Thank you for checking. If it's this hard to reproduce let's keep it like it is for now and look into this sometime later.

Actions #6

Updated by livdywan about 3 years ago

Seems like PRs are hitting this again:

not ok 24 - BatchedProj obs builds
[debug] [xl23hAJV] POST "/admin/obs_rsync/BatchedProj/obs_builds_text"
[debug] [xl23hAJV] Routing to controller "OpenQA::Shared::Controller::Session" and action "ensure_operator"
[debug] [xl23hAJV] Routing to controller "OpenQA::WebAPI::Plugin::ObsRsync::Controller::Gru" and action "update_obs_builds_text"
[debug] [xl23hAJV] 200 OK (0.024276s, 41.193/s)
[debug] [C0j4SN9_] GET "/admin/obs_rsync/BatchedProj/obs_builds_text"
[debug] [C0j4SN9_] Routing to controller "OpenQA::Shared::Controller::Session" and action "ensure_operator"
[debug] [C0j4SN9_] Routing to controller "OpenQA::WebAPI::Plugin::ObsRsync::Controller::Gru" and action "get_obs_builds_text"
[debug] [C0j4SN9_] 200 OK (0.016182s, 61.797/s)

Seen this on https://github.com/os-autoinst/openQA/pull/3694 and https://github.com/os-autoinst/openQA/pull/3635

Actions #7

Updated by okurz about 3 years ago

  • Due date set to 2021-02-23
  • Status changed from Workable to Feedback
  • Assignee set to andriinikitin
  • Target version changed from future to Ready

In recent days we saw this even more. Gladly @andriinikitin had a good idea how to address this:
https://github.com/os-autoinst/openQA/pull/3732

Actions #8

Updated by okurz about 3 years ago

  • Status changed from Feedback to In Progress
  • Assignee changed from andriinikitin to okurz

I will conduct the stability tests and see if we can remove that from the unstable list

Actions #9

Updated by okurz about 3 years ago

Did a successful local run with

env runs=100 count_fail_ratio prove -l t/ui/27-plugin_obs_rsync_status_details.t

Testing in circleCI https://github.com/os-autoinst/openQA/pull/3752

Actions #10

Updated by okurz about 3 years ago

  • Status changed from In Progress to Feedback
Actions #11

Updated by okurz about 3 years ago

  • Status changed from Feedback to Resolved
Actions #12

Updated by okurz about 3 years ago

  • Related to action #89935: t/ui/27-plugin_obs_rsync_status_details.t fails in circleCI, master branch even added
Actions

Also available in: Atom PDF