Project

General

Profile

action #95715

Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" size:M

Added by tinita 5 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2021-07-20
Due date:
2021-10-26
% Done:

0%

Estimated time:
Difficulty:

Description

https://gitlab.suse.de/openqa/openqa-review/-/issues/1

https://gitlab.suse.de/openqa/openqa-review/-/jobs/500502#L235

ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590756'. Could be deleted workaround needle?.

Edit: These errors are actually not fatal for the pipeline.
We should try to find out if we can fix these, or alternatively turn them into warnings.
Currently they are confusing everyone if the pipeline fails and there is no other obvious error.

Acceptance criteria

  • AC1: No error level log message is recorded
  • AC2: Report contains the error found with the job

Suggestion

  • Don't use log.error
  • Add this message to the report

Out of scope

  • Post a comment on the job

History

#1 Updated by tinita 5 months ago

  • Description updated (diff)

#2 Updated by tinita 5 months ago

I was able to reproduce the above errors with:

openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old|EOL)' --reminder-comment-on-issues -J https://openqa.suse.de/group_overview/161

However the exit code is 0, so maybe that isn't the cause of the pipeline failure

#3 Updated by tinita 5 months ago

  • Description updated (diff)

#4 Updated by tinita 5 months ago

  • Status changed from New to Feedback

I retriggered the job, and it suuceeded now.
https://gitlab.suse.de/openqa/openqa-review/-/jobs/500638

One difference which is still there is that the waiting dots are all on their own lines now, e.g.

.
.
.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
.
.
.

instead of

.....ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.....

#5 Updated by okurz 5 months ago

  • Due date set to 2021-08-03
  • Assignee set to tinita

tinita please only use "Feedback" with assignee so that at least the assignee should know what we wait for.

I see that currently

  • reports are correctly generated
  • the gitlab CI issue workaround now prints dots line-by-line instead of multiple dots per line but it does not seem to have a more severe impact and I would just accept this issue
  • we can have "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" which happened in before and is still happening. My design choice was that this should not fail report generation but should be fed back into the report at the approriate location so that at least readers of the report can act on it. IMHO we should double-check this and then not call it "ERROR" but only "INFO" log level. Would this be something you could look into?

#6 Updated by tinita 5 months ago

  • Status changed from Feedback to Resolved

okurz wrote:

  • we can have "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" which happened in before and is still happening. My design choice was that this should not fail report generation but should be fed back into the report at the approriate location so that at least readers of the report can act on it. IMHO we should double-check this and then not call it "ERROR" but only "INFO" log level. Would this be something you could look into?

Maybe, but not as part of this ticket I think.

#7 Updated by nicksinger about 2 months ago

  • Status changed from Resolved to New

I'm reopening this here because we see it again in the most recent pipeline run:

  1. https://gitlab.suse.de/openqa/openqa-review/-/jobs/635331

Not sure what was done here but from what I read it seems like the only relevant "ERROR" message in the output.

#8 Updated by cdywan about 2 months ago

  • Subject changed from openqa-review pipeline failed to openqa-review pipeline failed ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job
  • Status changed from New to Resolved

nicksinger wrote:

I'm reopening this here because we see it again in the most recent pipeline run:

  1. https://gitlab.suse.de/openqa/openqa-review/-/jobs/635331

Not sure what was done here but from what I read it seems like the only relevant "ERROR" message in the output.

.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590756'. Could be deleted workaround needle?.
[...]
WARNING:openqa_review.browser:Request to https://progress.opensuse.org/issues/7270913.json failed: 404 Client Error: Not Found for url: https://progress.opensuse.org/issues/7270913.json
[...]
ERROR: Job failed: pod "runner-ydlpfvpg-project-4884-concurrent-06mnvs" status is "Failed"

#9 Updated by cdywan about 2 months ago

  • Status changed from Resolved to New

I added the error to the title to make the ticket less ambiguous. Change of status was unintended.

#10 Updated by okurz about 2 months ago

  • Due date changed from 2021-08-03 to 2021-10-26

#11 Updated by tinita about 2 months ago

  • Subject changed from openqa-review pipeline failed ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job to Investigate non-fatal openqa-review pipeline failure "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job"
  • Description updated (diff)
  • Due date changed from 2021-10-26 to 2021-08-03

#12 Updated by tinita about 2 months ago

  • Subject changed from Investigate non-fatal openqa-review pipeline failure "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" to Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job"

#13 Updated by cdywan about 2 months ago

  • Due date changed from 2021-08-03 to 2021-10-26

Undoing accidental undo

#14 Updated by cdywan about 2 months ago

  • Subject changed from Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" to Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" size:M
  • Description updated (diff)
  • Status changed from New to Workable
  • Assignee deleted (tinita)

#15 Updated by mkittler about 2 months ago

  • Assignee set to mkittler

#16 Updated by mkittler about 2 months ago

  • Status changed from Workable to Feedback

I created a PR which has been merged: https://github.com/os-autoinst/openqa_review/pull/181

I've also searched the code for further problematic log messages. Maybe the following error should also better go into the report?

Failed to find valid bug tracker URL for %s with error %s. Skipping current result` % (v, e)

If a bug reference is skipped because the bug tracker is invalid that's likely more interesting for the user might confuse us in the same way as the error message from the ticket description.

#17 Updated by okurz about 2 months ago

yes, that would be perfect if this goes into the report rather than log error as well

#18 Updated by mkittler about 2 months ago

PR for the change proposed in my last comment: https://github.com/os-autoinst/openqa_review/pull/185

#19 Updated by mkittler about 2 months ago

  • Status changed from Feedback to Resolved

The PR has been merged. Considering that the log messages are now completely absent in the code I'm confident they don't appear anymore. The recent pipeline failures are most likely related to #100709. So I'm considering this resolved.

#20 Updated by tinita about 2 months ago

We now have pipeline failures for almost all of the different html pages:

Traceback (most recent call last):
101  File "/usr/bin/openqa-review", line 33, in <module>
102    sys.exit(load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')())
103  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1549, in main
104    report = generate_report(args)
105  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1480, in generate_report
106    return Report(browser, args, root_url, job_groups)
107  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1429, in __init__
108    self.report[k] = self._one_report(v)
109  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1440, in _one_report
110    return ProductReport(self.browser, job_group_url, self.root_url, self.args)
111  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1154, in __init__
112    self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser)
113  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 898, in __init__
114    self._search_for_bugrefs_for_softfailures(results)
115  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 945, in _search_for_bugrefs_for_softfailures
116    match, found_actual_ref = self._get_bugref_for_softfailed_module(v, module_name)
117  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1012, in _get_bugref_for_softfailed_module
118    match = re.search(bugref_regex, field["title"])
119KeyError: 'title'

https://gitlab.suse.de/openqa/openqa-review/-/jobs/650972

#21 Updated by tinita about 2 months ago

  • Status changed from Resolved to Workable

#22 Updated by mkittler about 2 months ago

  • Status changed from Workable to In Progress

This PR should avoid the error (and similar errors): https://github.com/os-autoinst/openqa_review/pull/186

#23 Updated by okurz about 2 months ago

please closely check the pipeline for according errors or feedback

#24 Updated by mkittler about 2 months ago

  • Status changed from In Progress to Resolved

Today's pipelines work again (apart from an issue with sending mails which is unrelated).

Also available in: Atom PDF