QA » openQA Project

Observation¶

I have seen in a couple of test runs on OBS that t/api/08-jobtemplates.t is unstable. E.g. https://build.opensuse.org/package/live_build_log/openSUSE:Factory:Staging:adi:5/openQA/standard/x86_64 currently shows:

[ 1425s] #   Failed test 'posting invalid YAML template results in error'
[ 1425s] #   at ./t/api/08-jobtemplates.t line 799.
[ 1425s] #     Structures begin differing at:
[ 1425s] #          $got->{error}[0]{message} = '/anyOf/0 Expected string - got object.'
[ 1425s] #     $expected->{error}[0]{message} = '/anyOf/1 Too many properties: 2/1.'
[ 1430s] # Looks like you failed 1 test of 67.
[ 1431s] ./t/api/08-jobtemplates.t ........................ 

Problem¶

Could be that the errors are not inserted in an ordered way.

Suggestions¶

• Try to reproduce locally e.g. with make test KEEP_DB=1 CHECKSTYLE=0 PROVE_ARGS=t/api/08-jobtemplates.t RETRY=20 STABILITY_TEST=1
• Ensure test stability with either relaxation in the test or ensuring predictable results from production code