Project

General

Profile

Actions
Copy link

action #95715

closed

Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" size:M

Added by tinita almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
mkittler
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2021-07-20
Due date:
2021-10-26
% Done:

0%

Estimated time:

Description

https://gitlab.suse.de/openqa/openqa-review/-/issues/1

https://gitlab.suse.de/openqa/openqa-review/-/jobs/500502#L235

ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590756'. Could be deleted workaround needle?.

Edit: These errors are actually not fatal for the pipeline.
We should try to find out if we can fix these, or alternatively turn them into warnings.
Currently they are confusing everyone if the pipeline fails and there is no other obvious error.

Acceptance criteria

  • AC1: No error level log message is recorded
  • AC2: Report contains the error found with the job

Suggestion

  • Don't use log.error
  • Add this message to the report

Out of scope

  • Post a comment on the job
Actions
Copy link
 #1

Updated by tinita almost 3 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by tinita almost 3 years ago

I was able to reproduce the above errors with:

openqa-review --host https://openqa.suse.de -n -r -T --query-issue-status --no-empty-sections --include-softfails --running-threshold=2 --exclude-job-groups '^(Released|Development|old|EOL)' --reminder-comment-on-issues -J https://openqa.suse.de/group_overview/161

However the exit code is 0, so maybe that isn't the cause of the pipeline failure

Actions
Copy link
 #3

Updated by tinita almost 3 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by tinita almost 3 years ago

  • Status changed from New to Feedback

I retriggered the job, and it suuceeded now.
https://gitlab.suse.de/openqa/openqa-review/-/jobs/500638

One difference which is still there is that the waiting dots are all on their own lines now, e.g.

.
.
.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
.
.
.

instead of

.....ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.....
Actions
Copy link
 #5

Updated by okurz almost 3 years ago

  • Due date set to 2021-08-03
  • Assignee set to tinita

@tinita please only use "Feedback" with assignee so that at least the assignee should know what we wait for.

I see that currently

  • reports are correctly generated
  • the gitlab CI issue workaround now prints dots line-by-line instead of multiple dots per line but it does not seem to have a more severe impact and I would just accept this issue
  • we can have "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" which happened in before and is still happening. My design choice was that this should not fail report generation but should be fed back into the report at the approriate location so that at least readers of the report can act on it. IMHO we should double-check this and then not call it "ERROR" but only "INFO" log level. Would this be something you could look into?
Actions
Copy link
 #6

Updated by tinita almost 3 years ago

  • Status changed from Feedback to Resolved

okurz wrote:

  • we can have "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" which happened in before and is still happening. My design choice was that this should not fail report generation but should be fed back into the report at the approriate location so that at least readers of the report can act on it. IMHO we should double-check this and then not call it "ERROR" but only "INFO" log level. Would this be something you could look into?

Maybe, but not as part of this ticket I think.

Actions
Copy link
 #7

Updated by nicksinger over 2 years ago

  • Status changed from Resolved to New

I'm reopening this here because we see it again in the most recent pipeline run:

  1. https://gitlab.suse.de/openqa/openqa-review/-/jobs/635331

Not sure what was done here but from what I read it seems like the only relevant "ERROR" message in the output.

Actions
Copy link
 #8

Updated by livdywan over 2 years ago

  • Subject changed from openqa-review pipeline failed to openqa-review pipeline failed ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job
  • Status changed from New to Resolved

nicksinger wrote:

I'm reopening this here because we see it again in the most recent pipeline run:

  1. https://gitlab.suse.de/openqa/openqa-review/-/jobs/635331

Not sure what was done here but from what I read it seems like the only relevant "ERROR" message in the output.

.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590738'. Could be deleted workaround needle?.
.
ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job 'https://openqa.suse.de/tests/5590756'. Could be deleted workaround needle?.
[...]
WARNING:openqa_review.browser:Request to https://progress.opensuse.org/issues/7270913.json failed: 404 Client Error: Not Found for url: https://progress.opensuse.org/issues/7270913.json
[...]
ERROR: Job failed: pod "runner-ydlpfvpg-project-4884-concurrent-06mnvs" status is "Failed"
Actions
Copy link
 #9

Updated by livdywan over 2 years ago

  • Status changed from Resolved to New

I added the error to the title to make the ticket less ambiguous. Change of status was unintended.

Actions
Copy link
 #10

Updated by okurz over 2 years ago

  • Due date changed from 2021-08-03 to 2021-10-26
Actions
Copy link
 #11

Updated by tinita over 2 years ago

  • Subject changed from openqa-review pipeline failed ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job to Investigate non-fatal openqa-review pipeline failure "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job"
  • Description updated (diff)
  • Due date changed from 2021-10-26 to 2021-08-03
Actions
Copy link
 #12

Updated by tinita over 2 years ago

  • Subject changed from Investigate non-fatal openqa-review pipeline failure "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" to Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job"
Actions
Copy link
 #13

Updated by livdywan over 2 years ago

  • Due date changed from 2021-08-03 to 2021-10-26

Undoing accidental undo

Actions
Copy link
 #14

Updated by livdywan over 2 years ago

  • Subject changed from Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" to Investigate non-fatal openqa-review pipeline error "ERROR:openqa_review.openqa_review:Could not find any soft failure reference within details of soft-failed job" size:M
  • Description updated (diff)
  • Status changed from New to Workable
  • Assignee deleted (tinita)
Actions
Copy link
 #15

Updated by mkittler over 2 years ago

  • Assignee set to mkittler
Actions
Copy link
 #16

Updated by mkittler over 2 years ago

  • Status changed from Workable to Feedback

I created a PR which has been merged: https://github.com/os-autoinst/openqa_review/pull/181

I've also searched the code for further problematic log messages. Maybe the following error should also better go into the report?

Failed to find valid bug tracker URL for %s with error %s. Skipping current result` % (v, e)

If a bug reference is skipped because the bug tracker is invalid that's likely more interesting for the user might confuse us in the same way as the error message from the ticket description.

Actions
Copy link
 #17

Updated by okurz over 2 years ago

yes, that would be perfect if this goes into the report rather than log error as well

Actions
Copy link
 #18

Updated by mkittler over 2 years ago

PR for the change proposed in my last comment: https://github.com/os-autoinst/openqa_review/pull/185

Actions
Copy link
 #19

Updated by mkittler over 2 years ago

  • Status changed from Feedback to Resolved

The PR has been merged. Considering that the log messages are now completely absent in the code I'm confident they don't appear anymore. The recent pipeline failures are most likely related to #100709. So I'm considering this resolved.

Actions
Copy link
 #20

Updated by tinita over 2 years ago

We now have pipeline failures for almost all of the different html pages:

Traceback (most recent call last):
101  File "/usr/bin/openqa-review", line 33, in <module>
102    sys.exit(load_entry_point('openqa-review==0.0.0', 'console_scripts', 'openqa-review')())
103  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1549, in main
104    report = generate_report(args)
105  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1480, in generate_report
106    return Report(browser, args, root_url, job_groups)
107  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1429, in __init__
108    self.report[k] = self._one_report(v)
109  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1440, in _one_report
110    return ProductReport(self.browser, job_group_url, self.root_url, self.args)
111  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1154, in __init__
112    self.reports[arch] = ArchReport(arch, results, args, root_url, progress_browser, bugzilla_browser, browser)
113  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 898, in __init__
114    self._search_for_bugrefs_for_softfailures(results)
115  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 945, in _search_for_bugrefs_for_softfailures
116    match, found_actual_ref = self._get_bugref_for_softfailed_module(v, module_name)
117  File "/usr/lib/python3.8/site-packages/openqa_review/openqa_review.py", line 1012, in _get_bugref_for_softfailed_module
118    match = re.search(bugref_regex, field["title"])
119KeyError: 'title'

https://gitlab.suse.de/openqa/openqa-review/-/jobs/650972

Actions
Copy link
 #21

Updated by tinita over 2 years ago

  • Status changed from Resolved to Workable
Actions
Copy link
 #22

Updated by mkittler over 2 years ago

  • Status changed from Workable to In Progress

This PR should avoid the error (and similar errors): https://github.com/os-autoinst/openqa_review/pull/186

Actions
Copy link
 #23

Updated by okurz over 2 years ago

please closely check the pipeline for according errors or feedback

Actions
Copy link
 #24

Updated by mkittler over 2 years ago

  • Status changed from In Progress to Resolved

Today's pipelines work again (apart from an issue with sending mails which is unrelated).

Actions
Copy link

Also available in: Atom PDF