action #59043

Fix unstable/flaky full-stack test, i.e. remove sleep

Added by okurz 5 months ago. Updated 11 days ago.

Status:WorkableStart date:04/11/2019
Priority:LowDue date:
Assignee:okurz% Done:

0%

Category:Concrete Bugs
Target version:Current Sprint
Difficulty:
Duration:


Related issues

Related to openQA Project - action #59100: Race condition in os-autoinst/commands.pm Rejected 05/11/2019
Related to openQA Project - action #62015: jobs incomplete without logs as some workers are rejected... Resolved 10/01/2020
Related to openQA Project - action #54032: flaky test: t/full-stack.t "not ok 44 - Cache tests", sub... Resolved 09/07/2019

History

#1 Updated by okurz 5 months ago

  • Related to action #59100: Race condition in os-autoinst/commands.pm added

#2 Updated by tinita 5 months ago

I'll add what I found out so far.

The failure can be reproduced locally by adding a sleep in isotovideo:

sleep 8;
($testprocess, $testfd) = autotest::start_process();

Debugging shows that the set_pause_at_test command is going through send_json before isotovideo starts to read:

# enter the main loop: process messages from autotest, command server and backend
while ($loop) {

If I add back the HTTP status call in openQA/lib/OpenQA/Worker/Job.pm _upload_results, then it is sent after isotovideo starts reading.

#3 Updated by coolo 5 months ago

  • Priority changed from Normal to Low

I see other issues as more urgent

#4 Updated by okurz 3 months ago

  • Related to action #62015: jobs incomplete without logs as some workers are rejected (was: Scheduler does not work) added

#5 Updated by okurz 2 months ago

  • Subject changed from Fix full-stack test flakyness, i.e. remove sleep to Fix unstable/flaky full-stack test, i.e. remove sleep

#6 Updated by okurz 2 months ago

  • Related to action #54032: flaky test: t/full-stack.t "not ok 44 - Cache tests", subtest: "not ok 17 - test 6 is passed" at /opt/testing_area/openqa/t/lib/OpenQA/Test/FullstackUtils.pm line 120. added

#7 Updated by okurz 2 months ago

  • Status changed from New to Feedback
  • Assignee set to okurz

#8 Updated by cdywan about 1 month ago

  • Target version set to Current Sprint

#9 Updated by okurz about 1 month ago

  • Status changed from Feedback to Workable

First https://github.com/os-autoinst/openQA/pull/2765 , then https://github.com/os-autoinst/openQA/pull/2626#issuecomment-580173772 : it seems the retry and stability tests flag are not properly exported into the sub-scope of the retry script.

#10 Updated by okurz 11 days ago

https://github.com/os-autoinst/openQA/pull/2811 is the fix for the retry script not handling "stability tests" properly. Now I can look into the test failures of https://github.com/os-autoinst/openQA/pull/2626 again.

Also available in: Atom PDF