action #94600
open[tools][mtui] Communicate reduced visibility of openQA incident related coverage in log
0%
Description
Motivation¶
The Results from openQA incidents jobs: section in a maintenance update's test log shows, as one would expect, the incident jobs related to the incident that is to be tested.
It can happen that engineers testing the incident fall under the impression that the openQA coverage shown in the log is the complete openQA test coverage for that incident.
It should thus be communicated that the openQA incident jobs section does not show the complete test coverage of the incident in openQA, but only a subset of it (the other being in aggregate runs that test the incident).
This should clarify to the engineers that the absence of failed incident jobs in the log does not mean necessarily that there are no other failed jobs related to the incident.
Acceptance criteria¶
- AC1: Communicate that the jobs listed in the log of an update are not the complete set of jobs that test that update
Suggestions¶
- One suggestion could be to remove that section from the log and instead link the incident comments (eg https://maintenance.suse.de/incident/19067/#comments) where all jobs related to that incident are listed.
Updated by okurz over 3 years ago
- Project changed from openQA Project (public) to QA (public)
- Subject changed from Template Generator: Communicate reduced visibility of openQA incident related coverage in log to [tools][teregen] Template Generator: Communicate reduced visibility of openQA incident related coverage in log
- Category deleted (
Feature requests) - Target version set to future
I like the suggestion but this means we would rely on the data on smelt being present at a later time when the generated report just references that.
@jbaier_cz do you think we could simply not have the (partially duplicate) information in the report but reference the smelt comment tab?
Updated by jbaier_cz over 3 years ago
Sorry, no can do here. This issue is not relevant to the template generator at all.
The section 'Results from openQA incidents jobs:' is added by mtui when the tester runs an export command, so the data inside the log are accurate to the time the tester actually does the test.
Updated by geor over 3 years ago
jbaier_cz wrote:
Sorry, no can do here. This issue is not relevant to the template generator at all.
The section 'Results from openQA incidents jobs:' is added by mtui when the tester runs an
exportcommand, so the data inside the log are accurate to the time the tester actually does the test.
Sorry for the mislabeling, so this issue should be mtui related.
Updated by geor over 3 years ago
- Subject changed from [tools][teregen] Template Generator: Communicate reduced visibility of openQA incident related coverage in log to [tools][teregen] mtui: Communicate reduced visibility of openQA incident related coverage in log
Updated by okurz over 3 years ago
jbaier_cz wrote:
Sorry, no can do here. This issue is not relevant to the template generator at all.
The section 'Results from openQA incidents jobs:' is added by mtui when the tester runs an
exportcommand, so the data inside the log are accurate to the time the tester actually does the test.
I see. But still, do you think we could simply not have the (partially duplicate) information in the report but reference the smelt comment tab in what mtui generates?
Updated by jbaier_cz over 3 years ago
- Subject changed from [tools][teregen] mtui: Communicate reduced visibility of openQA incident related coverage in log to [tools][mtui] Communicate reduced visibility of openQA incident related coverage in log
I don't think plain references to external sites are good idea (specifically for test reports). Two reasons:
- there are things like https://progress.opensuse.org/issues/97121 which might render the link useless for some reports
- I always though about reports as a snapshot, so they should include all information from the time the tester did the evaluation
My proposal would be to continue with https://progress.opensuse.org/issues/94480 and for example put those possible duplicates into a separate file and link to the file (and add a hint with link to external site for up-to-date info?) inside the main log.
Updated by okurz almost 2 years ago
- Related to action #124473: [tools] Automatic regression tests export from openQA added