Project

General

Profile

action #60458

Improve consistency of job states/results (was: /tests/overview shows Passed: 0, Failed: 0 in summary but nothing else for a build that consists of single incomplete job only)

Added by okurz 6 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2019-11-30
Due date:
% Done:

0%

Estimated time:
Difficulty:
Duration:

Description

Observation

https://github.com/os-autoinst/openqa_review tripped over https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumbleweed&build=20191128&groupid=34 which consists of a single, incomplete job. However the summary shows "Passed: 0, Failed: 0" but should show "Incomplete: 1" as for example currently https://openqa.opensuse.org/tests/overview?build=20191129&distri=opensuse&version=Tumbleweed&groupid=1 does which also has one incomplete but many others as well. https://openqa.opensuse.org/tests/overview?result=incomplete&arch=&machine=&modules=&build=20191129&distri=opensuse&version=Tumbleweed&groupid=1# does not reproduce the issue when filtering for only incomplete jobs.


Related issues

Related to openQA Project - action #62420: [epic] Distinguish all types of incompletesBlocked2018-12-122020-06-09

Copied to openQA Project - action #63802: Further improve consistency of job states/resultsNew2020-02-25

History

#1 Updated by mkittler 6 months ago

  • Assignee set to mkittler

which consists of a single, incomplete job

The problem here is that these jobs don't have the exact result "incomplete" but instead one of the other results considered incomplete:

use constant INCOMPLETE_RESULTS => (INCOMPLETE, SKIPPED, OBSOLETED, PARALLEL_FAILED,
    PARALLEL_RESTARTED, USER_CANCELLED, USER_RESTARTED, TIMEOUT_EXCEEDED
);

I suppose the overview statistics and the filter should take all of these results into account when looking for incomplete jobs.

#2 Updated by mkittler 6 months ago

  • Status changed from New to In Progress
  • Target version set to Current Sprint

PR: https://github.com/os-autoinst/openQA/pull/2564

[...] when filtering for only incomplete jobs

Note that the PR doesn't change the filter. It still distinguishes between the different incomplete results, e.g. one can filter for timeouts specifically.

#3 Updated by okurz 6 months ago

merged. But I suggest to keep this ticket open for the state+result refactoring as mentioned in https://github.com/os-autoinst/openQA/pull/2564/files#r352995455

#4 Updated by mkittler 6 months ago

  • Subject changed from /tests/overview shows Passed: 0, Failed: 0 in summary but nothing else for a build that consists of single incomplete job only to Improve consistency of job states/results (was: /tests/overview shows Passed: 0, Failed: 0 in summary but nothing else for a build that consists of single incomplete job only)

We already have "meta" results like "incomplete results" defined as constant list. It makes sense to improve the concept and use it consistently so accounting/filtering in test results overview, favicons and other CSS formatting behaves consistently and redundant code is avoided. The names of these "meta" results should also be improved (to avoid confusion with actual results).

#6 Updated by okurz 6 months ago

I suggest after the PR is in we can create another one to follow up on https://github.com/os-autoinst/openQA/pull/2575#discussion_r355120193 which is inconsistent naming in the UI which we do not want to change at the same time as the internal names.

#7 Updated by okurz 4 months ago

  • Related to action #62420: [epic] Distinguish all types of incompletes added

#8 Updated by mkittler 4 months ago

WIP branch for improving the UI as well: https://github.com/Martchus/openQA/tree/meta-results-ui

However, I've ran into a few problems/questions:

  1. "skipped" in the UI isn't exactly the meta-result "aborted". It accounts jobs with "aborted" results and jobs without result in state "cancelled". To resolve this I would change "skipped" to "aborted" in the UI ignoring that it is not exactly the ABORTED meta-result. Note that "skipped" for job modules should stay as-is. This is just about overall job results.
  2. Then there's also "failed" which actually maps to "not ok results". But the meta-result name is not really nice and I wouldn't show it in the UI. So I prefer to keep "failed" here.
  3. "unfinished" would become "pending". The meaning in this context is any of "scheduled", "assigned", "setup", "running" or "uploading". So in this context there actually are further results to be expected and therefore "pending" would actually make sense.

Generally it is a bit messy that the UI mixes results, meta-results, states and meta-states here but this somehow makes sense. We sometimes just don't have a result so the state matters, some results should be finder distinguished and for others the meta-result is sufficient, …. So I wouldn't overthink/oversimplify it for the sake of consistency with our internal names.

#9 Updated by okurz 3 months ago

  • Copied to action #63802: Further improve consistency of job states/results added

#10 Updated by okurz 3 months ago

  • Status changed from In Progress to Resolved

Discussed this in the weekly QA Tools team meeting. We agreed that the mentioned suggestions are valid further improvements but not necessarily something that we want or can do know. But this ticket originally started out as a concrete bug that was fixed in the meantime. Hence moving the left suggestions out into a specific ticket #63802 . Whenever we work on states/results again we should keep this in mind or pick it up again at any time.

Also available in: Atom PDF