action #165692
closed[openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"no candidate needle.*openqa-dashboard.*matched" size:S
Description
Motivation¶
During my work on #155413 I found out, that we are looking for state running or done, but we are also considering incomplete jobs as not problematic which can hide other issues (like in my case the AppArmor rule missing). See the attached screen. We should only allow running and passed jobs and fail otherwise.
Acceptance criteria¶
- AC1: If the test scheduled is done but not passed the test should fail
- AC2: The overview of running/finished tests is visible and we click on the test
Files
| screen.png (61.2 KB) screen.png | |||
| 20240822_20h18m54s_grim.png (7.67 KB) 20240822_20h18m54s_grim.png |
Updated by jbaier_cz about 2 months ago
- Related to action #155413: Ensure apparmor is enforced in openQA-in-openQA tests size:M added
Updated by jbaier_cz about 2 months ago
- Related to action #165408: Unreviewed issue (Group 24 openQA) test-running in openQA failing with error detecting default remote branch size:S added
Updated by tinita about 2 months ago
- Category set to Feature requests
That's exactly the kind of ticket I wanted to create myself today!
I also thought it would be another improvement to go to /tests in the dashboard and click on the test, so we can see how the test looks like.
It can be much easier to see what's going on there instead of looking at a JSON file without line breaks.
Updated by ybonatakis about 1 month ago
- Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete to [openQA-in-openQA] test does not fail if state is done but incomplete size:S
- Description updated (diff)
Updated by jbaier_cz about 1 month ago
- Status changed from New to Workable
- Assignee set to jbaier_cz
Updated by jbaier_cz about 1 month ago
It looks like the solution for AC1 is almost already in the code. I will try to unify the code paths.
Updated by jbaier_cz about 1 month ago
- Status changed from Workable to In Progress
First improvement is in https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/203
Updated by openqa_review about 1 month ago
- Due date set to 2024-09-13
Setting due date based on mean cycle time of SUSE QE Tools
Updated by jbaier_cz about 1 month ago
For AC2, https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/204 should be enough.
Updated by jbaier_cz about 1 month ago
And I even managed to catch the issue this ticket is aiming at. No clue how to reproduce it though: https://openqa.opensuse.org/tests/4445598
Updated by jbaier_cz about 1 month ago
- Due date deleted (
2024-09-13) - Status changed from In Progress to Resolved
Both changes can be seen in production:
- better result matching: https://openqa.opensuse.org/tests/4448976#step/test_running/4
- test detail page: https://openqa.opensuse.org/tests/4448976#step/tests/6
Updated by jbaier_cz about 1 month ago
- Status changed from Resolved to Feedback
Updated by jbaier_cz about 1 month ago
https://github.com/os-autoinst/os-autoinst-needles-openQA/pull/27 should help with the latest failure in https://openqa.opensuse.org/tests/4449973#step/tests/7
Updated by jbaier_cz about 1 month ago
Hopefully last failure in https://openqa.opensuse.org/tests/4452154. Apparently we also sometimes schedule FULL_OPENSUSE_TEST=1.
Updated by jbaier_cz about 1 month ago
- Status changed from Feedback to In Progress
Updated by jbaier_cz about 1 month ago
I created a new needle and adapted existing one so now should be working for both scenarios: https://github.com/os-autoinst/os-autoinst-needles-openQA/pull/28
Updated by openqa_review about 1 month ago
- Due date set to 2024-09-20
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan about 1 month ago
- Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete size:S to [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-test-details.*matched" size:S
Updated by livdywan about 1 month ago
- Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-test-details.*matched" size:S to [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-(test-details|search-results).*matched" size:S
Updated by jbaier_cz about 1 month ago
- Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-(test-details|search-results).*matched" size:S to [openQA-in-openQA] test does not fail if state is done but incomplete size:S
Updated by jbaier_cz about 1 month ago
- Related to action #166445: [openQA-in-openQA][sporadic] test fails in tests, simple_boot incomplete auto_review:"no candidate needle.*openqa-test-details.*matched":retry size:M added
Updated by jbaier_cz about 1 month ago
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/207 should eliminate failure in https://openqa.opensuse.org/tests/4461003#step/search/4. Sometimes the search bar is matched on the details page before the page changes and that will reset the focus so the subsequent match is broken.
Updated by jbaier_cz about 1 month ago
- Status changed from In Progress to Feedback
Updated by jbaier_cz 25 days ago
- Due date deleted (
2024-09-20) - Status changed from Feedback to Resolved
Looking at the latest run (https://openqa.opensuse.org/tests/overview?distri=openqa&version=Tumbleweed&build=%3ATW.31375&groupid=24) all seems green again.