action #64454: sporadic fails of t/api/08-jobtemplates.t on "Failed test 'posting invalid YAML template results in error'" - openQA Project (public) - openSUSE Project Management Tool

Actions

Copy link

action #64454

closed

sporadic fails of t/api/08-jobtemplates.t on "Failed test 'posting invalid YAML template results in error'"

Added by okurz almost 5 years ago. Updated almost 5 years ago.

Status:

Resolved

Priority:

High

Assignee:

tinita

Category:

Regressions/Crashes

Target version:

Current Sprint

Start date:

2020-03-12

Due date:

% Done:

Estimated time:

Description

Observation¶

I have seen in a couple of test runs on OBS that t/api/08-jobtemplates.t is unstable. E.g. https://build.opensuse.org/package/live_build_log/openSUSE:Factory:Staging:adi:5/openQA/standard/x86_64 currently shows:

[ 1425s] #   Failed test 'posting invalid YAML template results in error'
[ 1425s] #   at ./t/api/08-jobtemplates.t line 799.
[ 1425s] #     Structures begin differing at:
[ 1425s] #          $got->{error}[0]{message} = '/anyOf/0 Expected string - got object.'
[ 1425s] #     $expected->{error}[0]{message} = '/anyOf/1 Too many properties: 2/1.'
[ 1430s] # Looks like you failed 1 test of 67.
[ 1431s] ./t/api/08-jobtemplates.t ........................

Problem¶

Could be that the errors are not inserted in an ordered way.

Suggestions¶

Try to reproduce locally e.g. with make test KEEP_DB=1 CHECKSTYLE=0 PROVE_ARGS=t/api/08-jobtemplates.t RETRY=20 STABILITY_TEST=1
Ensure test stability with either relaxation in the test or ensuring predictable results from production code

Actions

Copy link

Updated by okurz almost 5 years ago

I could not reproduce the error locally so far with STABILITY_TEST=1 RETRY=20, ended up with 20 passed test runs. Maybe this is only or more easily reproducible in OBS.

Actions

Copy link

Updated by livdywan almost 5 years ago

Assignee set to livdywan
Target version set to Current Sprint

I'll tentatively take this ticket. The error superficially resembles what I saw when trying to move the tests between a clean webapi instance and a manually-emptied database. I might end up finding a clue that also helps with this.

Actions

Copy link

Updated by tinita almost 5 years ago

That reminds me of the issue that JSON::Validator stopped sorting the errors at some point, but we actually fixed that by sorting the errors ourselves.

Actions

Copy link