Project

General

Profile

Actions

action #165692

closed

[openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"no candidate needle.*openqa-dashboard.*matched" size:S

Added by jbaier_cz 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2024-08-22
Due date:
% Done:

0%

Estimated time:

Description

Motivation

During my work on #155413 I found out, that we are looking for state running or done, but we are also considering incomplete jobs as not problematic which can hide other issues (like in my case the AppArmor rule missing). See the attached screen. We should only allow running and passed jobs and fail otherwise.

Acceptance criteria

  • AC1: If the test scheduled is done but not passed the test should fail
  • AC2: The overview of running/finished tests is visible and we click on the test

Files

screen.png (61.2 KB) screen.png jbaier_cz, 2024-08-22 18:18
20240822_20h18m54s_grim.png (7.67 KB) 20240822_20h18m54s_grim.png jbaier_cz, 2024-08-22 18:19

Related issues 3 (1 open2 closed)

Related to openQA Project (public) - action #155413: Ensure apparmor is enforced in openQA-in-openQA tests size:MResolvedjbaier_cz

Actions
Related to openQA Project (public) - action #165408: Unreviewed issue (Group 24 openQA) test-running in openQA failing with error detecting default remote branch size:SResolvedybonatakis2024-09-06

Actions
Related to openQA Tests (public) - action #166445: [openQA-in-openQA][sporadic] test fails in tests, simple_boot incomplete auto_review:"no candidate needle.*openqa-test-details.*matched":retry size:SWorkable2024-09-06

Actions
Actions #1

Updated by jbaier_cz 4 months ago

  • Related to action #155413: Ensure apparmor is enforced in openQA-in-openQA tests size:M added
Actions #2

Updated by jbaier_cz 4 months ago

  • Related to action #165408: Unreviewed issue (Group 24 openQA) test-running in openQA failing with error detecting default remote branch size:S added
Actions #3

Updated by tinita 4 months ago

  • Category set to Feature requests

That's exactly the kind of ticket I wanted to create myself today!

I also thought it would be another improvement to go to /tests in the dashboard and click on the test, so we can see how the test looks like.
It can be much easier to see what's going on there instead of looking at a JSON file without line breaks.

Actions #4

Updated by ybonatakis 4 months ago

  • Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete to [openQA-in-openQA] test does not fail if state is done but incomplete size:S
  • Description updated (diff)
Actions #5

Updated by jbaier_cz 4 months ago

  • Status changed from New to Workable
  • Assignee set to jbaier_cz
Actions #6

Updated by jbaier_cz 4 months ago

It looks like the solution for AC1 is almost already in the code. I will try to unify the code paths.

Actions #7

Updated by jbaier_cz 4 months ago

  • Status changed from Workable to In Progress
Actions #8

Updated by openqa_review 4 months ago

  • Due date set to 2024-09-13

Setting due date based on mean cycle time of SUSE QE Tools

Actions #10

Updated by jbaier_cz 4 months ago

And I even managed to catch the issue this ticket is aiming at. No clue how to reproduce it though: https://openqa.opensuse.org/tests/4445598

Actions #11

Updated by jbaier_cz 4 months ago

  • Due date deleted (2024-09-13)
  • Status changed from In Progress to Resolved

Both changes can be seen in production:

Actions #12

Updated by jbaier_cz 4 months ago

  • Status changed from Resolved to Feedback
Actions #14

Updated by jbaier_cz 4 months ago

Hopefully last failure in https://openqa.opensuse.org/tests/4452154. Apparently we also sometimes schedule FULL_OPENSUSE_TEST=1.

Actions #15

Updated by jbaier_cz 3 months ago

  • Status changed from Feedback to In Progress
Actions #16

Updated by jbaier_cz 3 months ago

I created a new needle and adapted existing one so now should be working for both scenarios: https://github.com/os-autoinst/os-autoinst-needles-openQA/pull/28

Actions #17

Updated by openqa_review 3 months ago

  • Due date set to 2024-09-20

Setting due date based on mean cycle time of SUSE QE Tools

Actions #18

Updated by livdywan 3 months ago

  • Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete size:S to [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-test-details.*matched" size:S
Actions #19

Updated by livdywan 3 months ago

  • Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-test-details.*matched" size:S to [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-(test-details|search-results).*matched" size:S
Actions #20

Updated by jbaier_cz 3 months ago

  • Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-(test-details|search-results).*matched" size:S to [openQA-in-openQA] test does not fail if state is done but incomplete size:S
Actions #21

Updated by jbaier_cz 3 months ago

  • Related to action #166445: [openQA-in-openQA][sporadic] test fails in tests, simple_boot incomplete auto_review:"no candidate needle.*openqa-test-details.*matched":retry size:S added
Actions #22

Updated by jbaier_cz 3 months ago

https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/207 should eliminate failure in https://openqa.opensuse.org/tests/4461003#step/search/4. Sometimes the search bar is matched on the details page before the page changes and that will reset the focus so the subsequent match is broken.

Actions #23

Updated by jbaier_cz 3 months ago

  • Status changed from In Progress to Feedback
Actions #24

Updated by jbaier_cz 3 months ago

  • Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete size:S to [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"no candidate needle.*openqa-dashboard.*matched" size:S
Actions #25

Updated by jbaier_cz 3 months ago

  • Due date deleted (2024-09-20)
  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF