QA » openQA Project

Another instance I found:

passed

• https://openqa.suse.de/tests/6865706 for https://openqa.suse.de/tests/6812495

• Verify if openQA correctly reads config for hooks (config in production looks fine)

my $key = 'job_done_hook_' . $openqa_job->result seems solid. We're not somehow setting the env variable either (grep -R OPENQA_ /etc/systemd).

• Check if OPENQA_INVESTIGATE_ORIGIN was set correctly, considering those jobs have no comment

This is set in one place, clone_settings+=("OPENQA_INVESTIGATE_ORIGIN=$host_url/t$id"), where $id is the job to clone. I can't see how that could be the wrong job.

• (Why) are comments not being posted by https://github.com/os-autoinst/scripts/blob/master/openqa-investigate ?

I manually called investigate on those jobs and it got to the comment code path just fine.

• Check gru logs

• Nothing in /var/log/openqa_gru
• Going by minion jobs I couldn't easily identify any cases. There doesn't seem to be a way to search by job state or settings.

So I'm thinking we should record the hook to be able to verify the minion job: https://github.com/os-autoinst/openQA/pull/4126

cdywan wrote:

Lowering the priority since it was found there were no extra jobs, just jobs with faulty settings.

cdywan wrote:

https://github.com/os-autoinst/scripts/pull/102

The PR was merged.

So I verified 20 investigate jobs for their OPENQA_INVESTIGATE_ORIGIN and found this: https://openqa.suse.de/tests/6947401#settings
It's incomplete, and the origin has no comment 🤔

cdywan wrote:

So I verified 20 investigate jobs for their OPENQA_INVESTIGATE_ORIGIN and found this: https://openqa.suse.de/tests/6947401#settings
It's incomplete, and the origin has no comment 🤔

$ sudo grep -E 'invest|hook|6947401|6946641' /var/log/apache2/error_log
$ sudo grep -E '6947401|6946641' /var/log/openqa{,_gru}

Nothing relevant in the logs...

cdywan wrote:

Nothing relevant in the logs...

Checking some more recent occurences:

• https://openqa.suse.de/tests/6996738#settings w/o comment on https://openqa.suse.de/tests/6994712
• https://openqa.suse.de/tests/7003101#settings w/o comment on https://openqa.suse.de/tests/7002447
• https://openqa.suse.de/tests/7003107#settings w/o comment on https://openqa.suse.de/tests/7002448

$ sudo grep --color=always -E '[^0-9](7003107|7002448)' /var/log/apache2/error_log /var/log/openqa{,_gru}

We seem to have an increasing number of investigate jobs w/o a comment on the original job, but I still can't find anything in the logs... 🤔️

Keep in mind that /var/log/openqa_gru does not go that far. I checked the first entry and it's from 2021-09-02 2300 but https://openqa.suse.de/tests/6994712 is from 2021-09-01 so will likely not show up there. You should try xzgrep --color=always -E '[^0-9](7003107|7002448)' /var/log/openqa_gru.*.xz which at least goes back to 2021-08-30. Also not that long but enough in this case. Still, the search term does not show anything.

@tinita discovered some POST /api/v1/jobs/[0-9]*/comments requests with status 404 in Apache logs. I'm wondering why we 're not seeing errors elsewhere if the post coming from investigate hooks is failing i.e.

$ openqa-cli api --osd jobs/5990783/comments
404 Not Found
{"error":"Job 5990783 does not exist","error_status":404}`

Normally this gives you output on both stderr and stdout yet we seem to get nothing?

PR for adding User-Agent to openqa-cli calls: https://github.com/os-autoinst/scripts/pull/105

Bugfix for 105: https://github.com/os-autoinst/scripts/pull/106

https://github.com/os-autoinst/scripts/pull/107

I tried to spot more instances by looking at Last 500 finished jobs with investigate typed in the filter, and could only find correct jobs with comments in there. Pondering if I should bump the due date or consider this resolved 🤔️