action #164296
closed
coordination #102915: [saga][epic] Automated classification of failures
coordination #166655: [epic] openqa-label-known-issues
openqa-label-known-issues does not look at known issues if autoinst-log.txt does not exist but reason could be looked at size:S
Added by okurz 4 months ago.
Updated 2 months ago.
Category:
Feature requests
Description
Observation¶
As observed in #162038 openQA jobs ended up with no autoinst-log.txt because the files could not be uploaded due to "timestamp mismatch". That error is actually written in the "reason" field but openqa-label-known-issues does not look at this field if autoinst-log.txt is not present, e.g. due to the reason that the job could be too old and shouldn't be looked at anyway.
Acceptance criteria¶
- AC1: openqa-label-known-issues also checks the "reason" field even if autoinst-log.txt does not exist at all
- AC2: old jobs where autoinst-log.txt is already deleted are still not considered for review
Suggestions¶
- Copied from action #162038: No HTTP Response on OSD on 10-06-2024 - auto_review:".*timestamp mismatch - check whether clocks on the local host and the web UI host are in sync":retry size:S added
- Project changed from openQA Infrastructure to openQA Project
- Category changed from Feature requests to Feature requests
- Subject changed from openqa-label-known-issues does not look at known issues if autoinst-log.txt does not exist but reason could be looked at to openqa-label-known-issues does not look at known issues if autoinst-log.txt does not exist but reason could be looked at size:S
- Description updated (diff)
- Status changed from New to Workable
- Status changed from Workable to In Progress
- Assignee set to ybonatakis
output as last change:
here with reason=null
+ autoinst-log.txt¶
❯ dry_run=1 host=openqa.suse.de ./openqa-label-known-issues https://openqa.suse.de/tests/15089626
Requesting jobs/15089626 via openqa-cli
[https://openqa.suse.de/tests/15089626](https://openqa.suse.de/tests/15089626): Unknown test issue, to be reviewed
-> [autoinst-log.txt](https://openqa.suse.de/tests/15089626/file/autoinst-log.txt)
Last lines before SUT shutdown:
# --- 8< ---
# [2024-08-05T12:10:26.164521+02:00] [debug] [pid:126110] QEMU status is not 'shutdown', it is 'running'
# [2024-08-05T12:10:26.164636+02:00] [debug] [pid:125698] backend shutdown state:
# [2024-08-05T12:10:26.164988+02:00] [info] [pid:126110] ::: OpenQA::Qemu::Proc::save_state: Saving QEMU state to qemu_state.json
# [2024-08-05T12:10:27.220847+02:00] [debug] [pid:126110] Passing remaining frames to the video encoder
# frame= 771 fps=1.9 q=0.0 Lsize= 1531kB time=00:00:32.08 bitrate= 390.9kbits/s speed=0.0777x
# video:1525kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.355806%
# [2024-08-05T12:10:27.961098+02:00] [debug] [pid:126110] Waiting for video encoder to finalize the video
# [2024-08-05T12:10:27.961202+02:00] [debug] [pid:126110] The built-in video encoder (pid 126494) terminated
# [2024-08-05T12:10:27.961265+02:00] [debug] [pid:126110] The external video encoder (pid 126492) terminated
# [2024-08-05T12:10:27.961832+02:00] [debug] [pid:126110] QEMU: qemu-system-x86_64: terminating on signal 15 from pid 126110 (/usr/bin/isotovideo: backen)
# --- >8 ---
here with reason=null
+ without autoinst-log.txt¶
❯ dry_run=1 host=openqa.suse.de ./openqa-label-known-issues https://openqa.suse.de/tests/14890572
Requesting jobs/14890572 via openqa-cli
reason null
'https://openqa.suse.de/tests/14890572' does not have autoinst-log.txt but is rather old, ignoring
[https://openqa.suse.de/tests/14890572](https://openqa.suse.de/tests/14890572): Unknown test issue, to be reviewed
-> [autoinst-log.txt](https://openqa.suse.de/tests/14890572/file/autoinst-log.txt)
Last lines before SUT shutdown:
# --- 8< ---
# --- >8 ---
here with reason=api failure
and no autoinst-log.txt¶
❯ dry_run=1 host=openqa.suse.de ./openqa-label-known-issues https://openqa.suse.de/tests/14897579
Requesting jobs/14897579 via openqa-cli
'https://openqa.suse.de/tests/14897579' does not have autoinst-log.txt but is rather old, ignoring
foo
openqa-cli api --header User-Agent: openqa-label-known-issues (https://github.com/os-autoinst/scripts) --host https://openqa.suse.de --retries=3 -X POST jobs/14897579/comments text=poo#73375 Job incompletes with reason auto_review:"(?m)api failure$" (and no further details)
- Status changed from In Progress to Feedback
Note from @titina on the PR's review
There is already handling in handle_unreachable_or_no_log that will additionally check if the job itself is deleted or not, so I think we shouldn't unconditionally go on here.
which we might want to take a look further.
- Status changed from Feedback to In Progress
Thanks to Tina and Oli I dig deeper to find a solution. I havent pushed anything yet. it is in progress but we might discuss it in the unblock. However I found https://github.com/os-autoinst/scripts/blob/master/_common#L174 which doesnt seem to be used anywhere.
Anyhow, this is how the script'soutput looks like from the latest attempts
❯ dry_run=1 host=openqa.suse.de ./openqa-label-known-issues https://openqa.suse.de/tests/14897579
Requesting jobs/14897579 via openqa-cli
curl ret 404
reason api failure
from label_on_issues_from_issue_tracker
openqa-cli api --header User-Agent: openqa-label-known-issues (https://github.com/os-autoinst/scripts) --host https://openqa.suse.de --retries=3 -X POST jobs/14897579/comments text=poo#73375 Job incompletes with reason auto_review:"(?m)api failure$" (and no further details)
'https://openqa.suse.de/tests/14897579' does not have autoinst-log.txt but is rather old, ignoring
- Due date set to 2024-08-21
Setting due date based on mean cycle time of SUSE QE Tools
- Status changed from In Progress to Feedback
Based on our discussion in the unblock my suggestion is to split the function handle_unreachable_or_no_log
into handle_unreachable
and handle_no_log
and then call like this:
if unreachable then handle_unreachable
elif label
elif handle_no_log
…
Additionally or alternatively tinita was suggesting to handle the exit code with || rc=$?
and then decide from a certain exit code to actually exit or return or go on with labelling.
- Status changed from Feedback to Resolved
Merged with some neglected improvements recommended by Tina. Close it for now
- Related to action #165716: [o3] Munin - minion hook failed - /opt/os-autoinst-scripts/openqa-label-known-issues: ERROR: line 68 size:M added
- Copied to action #166649: Rewrite openqa-label-known-issues in Python or another better maintainable language added
- Parent task set to #166655
- Due date deleted (
2024-08-21)
Also available in: Atom
PDF