QA » openQA Project

Motivation¶

See parent #94105 where we identified multiple users stories regarding creating tickets or identifying direct or indirect users of openQA based on openqa-investigate results. As a next step we could try to identify product issues from openqa-investigate results, in particular the step "S3: retry X, last_good_test X, last_good_build V, last_good_test+build V -> reproducible product issue => if QAM test write comment on IBS/OBS or smelt, for non-QAM report product bug" from https://progress.opensuse.org/issues/94105#Suggestions

Acceptance criteria¶

• AC1: On failed openQA jobs with openqa-investigate info "retry X, last_good_test X, last_good_build V, last_good_test+build V" (X: failed, V: passed) a comment is written pointing to a likely product regression
• AC2: No such comment is written on other jobs

Suggestions¶

• Take a look how we identify likely sporadic issues as a result of the "retry" job in https://github.com/os-autoinst/scripts/blob/master/openqa-investigate#L136=
• Then using #110176 try to fan-in on the results of multiple investigation jobs to find the jobs with the combination "retry X, last_good_test X, last_good_build V, last_good_test+build V" (X: failed, V: passed). The challenge is that job done hooks are called on a single job so one would need to identify other sibling investigation jobs. And any other job can finish sooner than the others. Maybe we just call this investigation step on the "last_good_test" and if other jobs are not finished by then, then trigger another incarnation of the same minion job with a delay (exponential back-off?). Communicate by exit code? This would also avoid the need to run job_done_hooks on passed jobs.
• https://github.com/os-autoinst/scripts/pull/170 might give good ideas how to find "related test results" and read them out
• Extend existing unit tests
• Then add an openQA comment stating the observation about a likely product regression
• Note: The bash script openqa-investigate itself must not know anything about "openQA minion jobs" or schedule any

Concrete proposal¶

• https://openqa.suse.de/tests/7190164#comments #95746
• https://openqa.suse.de/tests/10385488#comments #98862
• This one is a comment pointing to a likely product regression

When do we want to run the logic regarding product regressions?

• The simple retry is failing
• The "last good" is also failing
• The "last good build" is also failing (older image) What does "fanning in" mean here exactly?
• We need to consult relevant jobs, and we may need to consider the last/penultimate/nth job for reference
• See also https://en.wikipedia.org/wiki/Fan-in for where the term comes from

Suggestion 1¶

Create 3 investigate jobs first and save the ids, then create the simple retry job with a new setting that lists the other ids:

id1 = systemd-networkd:investigate:last_good_tests:c2c7d0f5ef0e75043509bf7fe1324a81eee077e3: http://openqa.opensuse.org/t3071082
id2 = systemd-networkd:investigate:last_good_build:369.2: http://openqa.opensuse.org/t3071083
id3 = systemd-networkd:investigate:last_good_tests_and_build:c2c7d0f5ef0e75043509bf7fe1324a81eee077e3+369.2: http://openqa.opensuse.org/t3071084

job4: systemd-networkd:investigate:retry: http://openqa.opensuse.org/t3071081 OTHER_INVESTIGATE_JOBS=id1,id2,id3

Then only in the hook script of the simple retry job we check all other 3 jobs if they are finished.

• if not: return 142, so the minion job will be run again later
• if yes: get the results of all 4 results and decide from that if is a product issue

Suggestion 2¶

Write a comment like:

* systemd-networkd:investigate:retry: http://openqa.opensuse.org/t3071081
* systemd-networkd:investigate:last_good_tests:c2c7d0f5ef0e75043509bf7fe1324a81eee077e3: http://openqa.opensuse.org/t3071082
* systemd-networkd:investigate:last_good_build:369.2: http://openqa.opensuse.org/t3071083
* systemd-networkd:investigate:last_good_tests_and_build:c2c7d0f5ef0e75043509bf7fe1324a81eee077e3+369.2: http://openqa.opensuse.org/t3071084

IDS: 3071081,3071082,3071083,3071084

And then in the hook script go to the investigate origin job, fetch comments and parse the ids out. Maybe not the best idea though.