Project

General

Profile

action #94600

[tools][mtui] Communicate reduced visibility of openQA incident related coverage in log

Added by geor 3 months ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
Start date:
2021-06-23
Due date:
% Done:

0%

Estimated time:

Description

Motivation

The Results from openQA incidents jobs: section in a maintenance update's test log shows, as one would expect, the incident jobs related to the incident that is to be tested.
It can happen that engineers testing the incident fall under the impression that the openQA coverage shown in the log is the complete openQA test coverage for that incident.
It should thus be communicated that the openQA incident jobs section does not show the complete test coverage of the incident in openQA, but only a subset of it (the other being in aggregate runs that test the incident).

This should clarify to the engineers that the absence of failed incident jobs in the log does not mean necessarily that there are no other failed jobs related to the incident.

Acceptance criteria

  • AC1: Communicate that the jobs listed in the log of an update are not the complete set of jobs that test that update

Suggestions

History

#1 Updated by geor 3 months ago

  • Description updated (diff)

#2 Updated by okurz 3 months ago

  • Project changed from openQA Project to QA
  • Subject changed from Template Generator: Communicate reduced visibility of openQA incident related coverage in log to [tools][teregen] Template Generator: Communicate reduced visibility of openQA incident related coverage in log
  • Category deleted (Feature requests)
  • Target version set to future

I like the suggestion but this means we would rely on the data on smelt being present at a later time when the generated report just references that.

jbaier_cz do you think we could simply not have the (partially duplicate) information in the report but reference the smelt comment tab?

#4 Updated by jbaier_cz 3 months ago

Sorry, no can do here. This issue is not relevant to the template generator at all.

The section 'Results from openQA incidents jobs:' is added by mtui when the tester runs an export command, so the data inside the log are accurate to the time the tester actually does the test.

#5 Updated by geor 3 months ago

jbaier_cz wrote:

Sorry, no can do here. This issue is not relevant to the template generator at all.

The section 'Results from openQA incidents jobs:' is added by mtui when the tester runs an export command, so the data inside the log are accurate to the time the tester actually does the test.

Sorry for the mislabeling, so this issue should be mtui related.

#6 Updated by geor 3 months ago

  • Subject changed from [tools][teregen] Template Generator: Communicate reduced visibility of openQA incident related coverage in log to [tools][teregen] mtui: Communicate reduced visibility of openQA incident related coverage in log

#7 Updated by okurz 3 months ago

jbaier_cz wrote:

Sorry, no can do here. This issue is not relevant to the template generator at all.

The section 'Results from openQA incidents jobs:' is added by mtui when the tester runs an export command, so the data inside the log are accurate to the time the tester actually does the test.

I see. But still, do you think we could simply not have the (partially duplicate) information in the report but reference the smelt comment tab in what mtui generates?

#8 Updated by jbaier_cz about 1 month ago

  • Subject changed from [tools][teregen] mtui: Communicate reduced visibility of openQA incident related coverage in log to [tools][mtui] Communicate reduced visibility of openQA incident related coverage in log

I don't think plain references to external sites are good idea (specifically for test reports). Two reasons:

  1. there are things like https://progress.opensuse.org/issues/97121 which might render the link useless for some reports
  2. I always though about reports as a snapshot, so they should include all information from the time the tester did the evaluation

My proposal would be to continue with https://progress.opensuse.org/issues/94480 and for example put those possible duplicates into a separate file and link to the file (and add a hint with link to external site for up-to-date info?) inside the main log.

Also available in: Atom PDF