openQA Project (public)

When we've estimated the ticket I was under the impression there's be a link to relevant tests. However, the link is only about a job group. Do you really expect us to search for the affected tests like this? Having a concrete example job plus the log excerpt that is supposed to match the regex is important. How would it otherwise reproduce/test this?

I did some digging. So https://openqa.suse.de/tests/10461612 is an example job and the log excerpt that's supposed to match is:

[2023-02-10T03:55:28.693183+01:00] [debug] [pid:12522] <<< testapi::record_info(title="screen_manage", output="Test result is the same as baseline\n[9] screen_manage PASS", result="ok")
[2023-02-10T03:55:28.693602+01:00] [debug] [pid:12522] tests/security/cc/trustedprograms.pm:51 called audit_test::compare_run_log -> lib/audit_test.pm:150 called audit_test::_parse_results_with_diff_baseline -> lib/audit_test.pm:188 called testapi::record_info
[2023-02-10T03:55:28.693796+01:00] [debug] [pid:12522] <<< testapi::record_info(title="useradd", output="Test result is NOT same as baseline \nCurrent:  [10] useradd FAIL\nBaseline: [10] useradd PASS", result="fail")
[2023-02-10T03:55:28.694071+01:00] [debug] [pid:12522] tests/security/cc/trustedprograms.pm:51 called audit_test::compare_run_log -> lib/audit_test.pm:153 called testapi::record_info
[2023-02-10T03:55:28.694239+01:00] [debug] [pid:12522] <<< testapi::record_info(title="useradd_D", output="Test result is the same as baseline\n[11] useradd_D PASS", result="ok")

I've just ran echo 'http://openqa.suse.de/tests/10461612' | scheme=http host=10.160.0.207 ./openqa-label-known-issues and it created a comment as expected. (I've been using the IP because otherwise it times out, not sure why but that's likely a local issue.)

With echo 'http://openqa.suse.de/tests/10461612' | scheme=http host=10.160.0.207 enable_force_result=true ./openqa-label-known-issues it also created a force_result label so now the job is softfailed (before it was failed).

So the openqa-label-known-issues script is fine. It has likely just not been triggered for this job.

I looked at the code of openQA and the hook script and the hook script is really just triggered like in my previous test. However, there's one exception. When a bugref could be carried over, the hook script is not triggered at all. That means auto-review and its result override are not effective.

I've checked several jobs in the scenario https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=cc_audit-test-part2&version=15-SP4. I only found jobs where a bugref could be taken over (e.g. https://openqa.suse.de/tests/10462392 - EDIT: I removed that comment now, see my next comment.) or that were older than #122215 being in the right category (e.g. https://openqa.suse.de/tests/10214182). Maybe I missed some jobs, though.

I removed the comment https://openqa.suse.de/tests/10462392 referencing poo # 122215 which has been taken over from https://openqa.suse.de/tests/10214182 so the comment with the label will be taken over the next time. This will hopefully cause the next job to be softfailed as well.

PR for 4 (and 4.1): https://github.com/os-autoinst/openQA/pull/5035

The PR has been merged.

Here's another one for 3.2: https://github.com/os-autoinst/openQA/pull/5039

Since 3.1 and 3.2 are alternative, this means all points from #123724#note-12 will be handled if that PR is merged.