action #60458
closedImprove consistency of job states/results (was: /tests/overview shows Passed: 0, Failed: 0 in summary but nothing else for a build that consists of single incomplete job only)
0%
Description
Observation¶
https://github.com/os-autoinst/openqa_review tripped over https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20191128&groupid=34 which consists of a single, incomplete job. However the summary shows "Passed: 0, Failed: 0" but should show "Incomplete: 1" as for example currently https://openqa.opensuse.org/tests/overview?build=20191129&distri=opensuse&version=Tumbleweed&groupid=1 does which also has one incomplete but many others as well. https://openqa.opensuse.org/tests/overview?result=incomplete&arch=&machine=&modules=&build=20191129&distri=opensuse&version=Tumbleweed&groupid=1# does not reproduce the issue when filtering for only incomplete jobs.
Updated by mkittler over 4 years ago
- Assignee set to mkittler
which consists of a single, incomplete job
The problem here is that these jobs don't have the exact result "incomplete" but instead one of the other results considered incomplete:
use constant INCOMPLETE_RESULTS => (INCOMPLETE, SKIPPED, OBSOLETED, PARALLEL_FAILED,
PARALLEL_RESTARTED, USER_CANCELLED, USER_RESTARTED, TIMEOUT_EXCEEDED
);
I suppose the overview statistics and the filter should take all of these results into account when looking for incomplete jobs.
Updated by mkittler over 4 years ago
- Status changed from New to In Progress
- Target version set to Current Sprint
PR: https://github.com/os-autoinst/openQA/pull/2564
[...] when filtering for only incomplete jobs
Note that the PR doesn't change the filter. It still distinguishes between the different incomplete results, e.g. one can filter for timeouts specifically.
Updated by okurz over 4 years ago
merged. But I suggest to keep this ticket open for the state+result refactoring as mentioned in https://github.com/os-autoinst/openQA/pull/2564/files#r352995455
Updated by mkittler over 4 years ago
- Subject changed from /tests/overview shows Passed: 0, Failed: 0 in summary but nothing else for a build that consists of single incomplete job only to Improve consistency of job states/results (was: /tests/overview shows Passed: 0, Failed: 0 in summary but nothing else for a build that consists of single incomplete job only)
We already have "meta" results like "incomplete results" defined as constant list. It makes sense to improve the concept and use it consistently so accounting/filtering in test results overview, favicons and other CSS formatting behaves consistently and redundant code is avoided. The names of these "meta" results should also be improved (to avoid confusion with actual results).
Updated by mkittler over 4 years ago
Updated by okurz over 4 years ago
I suggest after the PR is in we can create another one to follow up on https://github.com/os-autoinst/openQA/pull/2575#discussion_r355120193 which is inconsistent naming in the UI which we do not want to change at the same time as the internal names.
Updated by okurz over 4 years ago
- Related to coordination #62420: [epic] Distinguish all types of incompletes added
Updated by mkittler over 4 years ago
WIP branch for improving the UI as well: https://github.com/Martchus/openQA/tree/meta-results-ui
However, I've ran into a few problems/questions:
- "skipped" in the UI isn't exactly the meta-result "aborted". It accounts jobs with "aborted" results and jobs without result in state "cancelled". To resolve this I would change "skipped" to "aborted" in the UI ignoring that it is not exactly the
ABORTEDmeta-result. Note that "skipped" for job modules should stay as-is. This is just about overall job results. - Then there's also "failed" which actually maps to "not ok results". But the meta-result name is not really nice and I wouldn't show it in the UI. So I prefer to keep "failed" here.
- "unfinished" would become "pending". The meaning in this context is any of "scheduled", "assigned", "setup", "running" or "uploading". So in this context there actually are further results to be expected and therefore "pending" would actually make sense.
Generally it is a bit messy that the UI mixes results, meta-results, states and meta-states here but this somehow makes sense. We sometimes just don't have a result so the state matters, some results should be finder distinguished and for others the meta-result is sufficient, …. So I wouldn't overthink/oversimplify it for the sake of consistency with our internal names.
Updated by okurz about 4 years ago
- Copied to action #63802: Further improve consistency of job states/results added
Updated by okurz about 4 years ago
- Status changed from In Progress to Resolved
Discussed this in the weekly QA Tools team meeting. We agreed that the mentioned suggestions are valid further improvements but not necessarily something that we want or can do know. But this ticket originally started out as a concrete bug that was fixed in the meantime. Hence moving the left suggestions out into a specific ticket #63802 . Whenever we work on states/results again we should keep this in mind or pick it up again at any time.