Project

General

Profile

action #66718

unstable/flaky/sporadic test: t/ui/27-plugin_obs_rsync_status_details.t

Added by okurz about 1 year ago. Updated 5 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2020-05-12
Due date:
2021-02-23
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

See https://app.circleci.com/pipelines/github/os-autoinst/openQA/2827/workflows/13672907-1ad3-48d0-a1b2-e210991ed1f8/jobs/26708/steps :

[22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 19/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 24/? 
#   Failed test 'BatchedProj obs builds'
#   at t/ui/27-plugin_obs_rsync_status_details.t line 181.
#          got: '470.2, 469.1'
#     expected: '4704, 4703, 470.2, 469.1'
[22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 27/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. 28/? [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. …60/? # Looks like you failed 1 test of 60.
                                                              [22:54:20] t/ui/27-plugin_obs_rsync_status_details.t .. Dubious, test returned 1 (wstat 256, 0x100)

Failed 1/60 subtests 

t/ui/27-plugin_obs_rsync_status_details.t failed in 1/2 runs within a stability test

Acceptance criteria

  • AC1: t/ui/27-plugin_obs_rsync_status_details.t is stable in 20/20 tries
  • AC2: t/ui/27-plugin_obs_rsync_status_details.t is removed from .circleci/unstable_tests.txt

Suggestions

  • Run make test RETRY=20 STABILITY_TEST=1 TESTS=t/ui/27-plugin_obs_rsync_status_details.t locally to reproduce
  • Identify causes of sporadic failures and fix in either production code or tests
  • Prove stability locally and in CI
  • Remove from .circleci/unstable_tests.txt again

Related issues

Related to openQA Project - action #89935: t/ui/27-plugin_obs_rsync_status_details.t fails in circleCI, master branch evenResolved2021-03-112021-04-30

History

#1 Updated by okurz about 1 year ago

proposed the move of the test to unstables in https://github.com/os-autoinst/openQA/pull/3081

#2 Updated by okurz 12 months ago

  • Target version set to Ready

#3 Updated by okurz 9 months ago

  • Priority changed from Normal to Low

#4 Updated by Xiaojing_liu 8 months ago

I cannot reproduce this issue in my local.
I run make test RETRY=20 STABILITY_TEST=1 TESTS=t/ui/27-plugin_obs_rsync_status_details.t, every time passed.
I also tried for i in $(seq 1 20); do sudo docker run -it --rm -v $PWD:/opt/testing_area localtest bash -c 'make test TESTS=t/ui/27-plugin_obs_rsync_status_details.t' >> 2; done, all tests passed. I am not sure if run it in docket is the way to ensure Prove stability in CI.

BTW, I modified the timelimit in this file, because it always returns error then the timelimit is 16.

server stopped

real    0m48.786s
user    0m0.075s
sys     0m0.049s

#5 Updated by okurz 8 months ago

  • Target version changed from Ready to future

Thank you for checking. If it's this hard to reproduce let's keep it like it is for now and look into this sometime later.

#6 Updated by cdywan 6 months ago

Seems like PRs are hitting this again:

not ok 24 - BatchedProj obs builds
[debug] [xl23hAJV] POST "/admin/obs_rsync/BatchedProj/obs_builds_text"
[debug] [xl23hAJV] Routing to controller "OpenQA::Shared::Controller::Session" and action "ensure_operator"
[debug] [xl23hAJV] Routing to controller "OpenQA::WebAPI::Plugin::ObsRsync::Controller::Gru" and action "update_obs_builds_text"
[debug] [xl23hAJV] 200 OK (0.024276s, 41.193/s)
[debug] [C0j4SN9_] GET "/admin/obs_rsync/BatchedProj/obs_builds_text"
[debug] [C0j4SN9_] Routing to controller "OpenQA::Shared::Controller::Session" and action "ensure_operator"
[debug] [C0j4SN9_] Routing to controller "OpenQA::WebAPI::Plugin::ObsRsync::Controller::Gru" and action "get_obs_builds_text"
[debug] [C0j4SN9_] 200 OK (0.016182s, 61.797/s)

Seen this on https://github.com/os-autoinst/openQA/pull/3694 and https://github.com/os-autoinst/openQA/pull/3635

#7 Updated by okurz 5 months ago

  • Due date set to 2021-02-23
  • Status changed from Workable to Feedback
  • Assignee set to andriinikitin
  • Target version changed from future to Ready

In recent days we saw this even more. Gladly andriinikitin had a good idea how to address this:
https://github.com/os-autoinst/openQA/pull/3732

#8 Updated by okurz 5 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from andriinikitin to okurz

I will conduct the stability tests and see if we can remove that from the unstable list

#9 Updated by okurz 5 months ago

Did a successful local run with

env runs=100 count_fail_ratio prove -l t/ui/27-plugin_obs_rsync_status_details.t

Testing in circleCI https://github.com/os-autoinst/openQA/pull/3752

#10 Updated by okurz 5 months ago

  • Status changed from In Progress to Feedback

#11 Updated by okurz 5 months ago

  • Status changed from Feedback to Resolved

#12 Updated by okurz 5 months ago

  • Related to action #89935: t/ui/27-plugin_obs_rsync_status_details.t fails in circleCI, master branch even added

Also available in: Atom PDF