action #95715
closedInvestigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" size:M
Added by tinita almost 3 years ago. Updated over 2 years ago.
Description
https://gitlab.suse.de/openqa/openqa-review/-/issues/1
https://gitlab.suse.de/openqa/openqa-review/-/jobs/500502#L235
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590756'. Could be deleted workaround needle?.
Edit: These errors are actually not fatal for the pipeline.
We should try to find out if we can fix these, or alternatively turn them into warnings.
Currently they are confusing everyone if the pipeline fails and there is no other obvious error.
Acceptance criteria¶
- AC1: No error level log message is recorded
- AC2: Report contains the error found with the job
Suggestion¶
- Don't use
log.error - Add this message to the report
Out of scope¶
- Post a comment on the job
Updated by tinita almost 3 years ago
I was able to reproduce the above errors with:
openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old|EOL)' --reminder-comment-on-issues -J https://openqa.suse.de/group_overview/161
However the exit code is 0, so maybe that isn't the cause of the pipeline failure
Updated by tinita almost 3 years ago
- Status changed from New to Feedback
I retriggered the job, and it suuceeded now.
https://gitlab.suse.de/openqa/openqa-review/-/jobs/500638
One difference which is still there is that the waiting dots are all on their own lines now, e.g.
.
.
.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
.
.
.
instead of
.....ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.....
Updated by okurz almost 3 years ago
- Due date set to 2021-08-03
- Assignee set to tinita
@tinita please only use "Feedback" with assignee so that at least the assignee should know what we wait for.
I see that currently
- reports are correctly generated
- the gitlab CI issue workaround now prints dots line-by-line instead of multiple dots per line but it does not seem to have a more severe impact and I would just accept this issue
- we can have "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" which happened in before and is still happening. My design choice was that this should not fail report generation but should be fed back into the report at the approriate location so that at least readers of the report can act on it. IMHO we should double-check this and then not call it "ERROR" but only "INFO" log level. Would this be something you could look into?
Updated by tinita almost 3 years ago
- Status changed from Feedback to Resolved
okurz wrote:
- we can have "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" which happened in before and is still happening. My design choice was that this should not fail report generation but should be fed back into the report at the approriate location so that at least readers of the report can act on it. IMHO we should double-check this and then not call it "ERROR" but only "INFO" log level. Would this be something you could look into?
Maybe, but not as part of this ticket I think.
Updated by nicksinger over 2 years ago
- Status changed from Resolved to New
I'm reopening this here because we see it again in the most recent pipeline run:
Not sure what was done here but from what I read it seems like the only relevant "ERROR" message in the output.
Updated by livdywan over 2 years ago
- Subject changed from openqa-review pipeline failed to openqa-review pipeline failed ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job
- Status changed from New to Resolved
nicksinger wrote:
I'm reopening this here because we see it again in the most recent pipeline run:
Not sure what was done here but from what I read it seems like the only relevant "ERROR" message in the output.
.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590756'. Could be deleted workaround needle?.
[...]
WARNING:openqa_review.browser:Request to https://progress.opensuse.org/issues/7270913.json failed: 404 Client Error: Not Found for url: https://progress.opensuse.org/issues/7270913.json
[...]
ERROR: Job failed: pod "runner-ydlpfvpg-project-4884-concurrent-06mnvs" status is "Failed"
Updated by livdywan over 2 years ago
- Status changed from Resolved to New
I added the error to the title to make the ticket less ambiguous. Change of status was unintended.
Updated by okurz over 2 years ago
- Due date changed from 2021-08-03 to 2021-10-26
Updated by tinita over 2 years ago
- Subject changed from openqa-review pipeline failed ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job to Investigate non-fatal openqa-review pipeline failure "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job"
- Description updated (diff)
- Due date changed from 2021-10-26 to 2021-08-03
Updated by tinita over 2 years ago
- Subject changed from Investigate non-fatal openqa-review pipeline failure "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" to Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job"
Updated by livdywan over 2 years ago
- Due date changed from 2021-08-03 to 2021-10-26
Undoing accidental undo
Updated by livdywan over 2 years ago
- Subject changed from Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" to Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" size:M
- Description updated (diff)
- Status changed from New to Workable
- Assignee deleted (
tinita)
Updated by mkittler over 2 years ago
- Status changed from Workable to Feedback
I created a PR which has been merged: https://github.com/os-autoinst/openqa_review/pull/181
I've also searched the code for further problematic log messages. Maybe the following error should also better go into the report?
Failed to find valid bug tracker URL for %s with error %s. Skipping current result` % (v, e)
If a bug reference is skipped because the bug tracker is invalid that's likely more interesting for the user might confuse us in the same way as the error message from the ticket description.
Updated by okurz over 2 years ago
yes, that would be perfect if this goes into the report rather than log error as well
Updated by mkittler over 2 years ago
PR for the change proposed in my last comment: https://github.com/os-autoinst/openqa_review/pull/185
Updated by mkittler over 2 years ago
- Status changed from Feedback to Resolved
The PR has been merged. Considering that the log messages are now completely absent in the code I'm confident they don't appear anymore. The recent pipeline failures are most likely related to #100709. So I'm considering this resolved.
Updated by tinita over 2 years ago
We now have pipeline failures for almost all of the different html pages:
Traceback (most recent call last):
101 File "/usr/bin/openqa-review", line 33, in <module>
102 sys.exit(load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')())
103 File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1549, in main
104 report = generate_report(args)
105 File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1480, in generate_report
106 return Report(browser, args, root_url, job_groups)
107 File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1429, in __init__
108 self.report[k] = self._one_report(v)
109 File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1440, in _one_report
110 return ProductReport(self.browser, job_group_url, self.root_url, self.args)
111 File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1154, in __init__
112 self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser)
113 File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 898, in __init__
114 self._search_for_bugrefs_for_softfailures(results)
115 File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 945, in _search_for_bugrefs_for_softfailures
116 match, found_actual_ref = self._get_bugref_for_softfailed_module(v, module_name)
117 File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1012, in _get_bugref_for_softfailed_module
118 match = re.search(bugref_regex, field["title"])
119KeyError: 'title'
Updated by mkittler over 2 years ago
- Status changed from Workable to In Progress
This PR should avoid the error (and similar errors): https://github.com/os-autoinst/openqa_review/pull/186
Updated by okurz over 2 years ago
please closely check the pipeline for according errors or feedback
Updated by mkittler over 2 years ago
- Status changed from In Progress to Resolved
Today's pipelines work again (apart from an issue with sending mails which is unrelated).