action #165692
closed[openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"no candidate needle.*openqa-dashboard.*matched" size:S
Description
Motivation¶
During my work on #155413 I found out, that we are looking for state running or done, but we are also considering incomplete jobs as not problematic which can hide other issues (like in my case the AppArmor rule missing). See the attached screen. We should only allow running and passed jobs and fail otherwise.
Acceptance criteria¶
- AC1: If the test scheduled is done but not passed the test should fail
- AC2: The overview of running/finished tests is visible and we click on the test
Files
| screen.png (61.2 KB) screen.png | |||
| 20240822_20h18m54s_grim.png (7.67 KB) 20240822_20h18m54s_grim.png |
Updated by jbaier_cz 3 months ago
- Related to action #155413: Ensure apparmor is enforced in openQA-in-openQA tests size:M added
Updated by jbaier_cz 3 months ago
- Related to action #165408: Unreviewed issue (Group 24 openQA) test-running in openQA failing with error detecting default remote branch size:S added
Updated by tinita 3 months ago
- Category set to Feature requests
That's exactly the kind of ticket I wanted to create myself today!
I also thought it would be another improvement to go to /tests in the dashboard and click on the test, so we can see how the test looks like.
It can be much easier to see what's going on there instead of looking at a JSON file without line breaks.
Updated by ybonatakis 3 months ago
- Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete to [openQA-in-openQA] test does not fail if state is done but incomplete size:S
- Description updated (diff)
Updated by jbaier_cz 3 months ago
- Status changed from Workable to In Progress
First improvement is in https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/203
Updated by openqa_review 3 months ago
- Due date set to 2024-09-13
Setting due date based on mean cycle time of SUSE QE Tools
Updated by jbaier_cz 3 months ago
For AC2, https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/204 should be enough.
Updated by jbaier_cz 3 months ago
And I even managed to catch the issue this ticket is aiming at. No clue how to reproduce it though: https://openqa.opensuse.org/tests/4445598
Updated by jbaier_cz 3 months ago
- Due date deleted (
2024-09-13) - Status changed from In Progress to Resolved
Both changes can be seen in production:
- better result matching: https://openqa.opensuse.org/tests/4448976#step/test_running/4
- test detail page: https://openqa.opensuse.org/tests/4448976#step/tests/6
Updated by jbaier_cz 3 months ago
https://github.com/os-autoinst/os-autoinst-needles-openQA/pull/27 should help with the latest failure in https://openqa.opensuse.org/tests/4449973#step/tests/7
Updated by jbaier_cz 3 months ago
Hopefully last failure in https://openqa.opensuse.org/tests/4452154. Apparently we also sometimes schedule FULL_OPENSUSE_TEST=1.
Updated by jbaier_cz 2 months ago
I created a new needle and adapted existing one so now should be working for both scenarios: https://github.com/os-autoinst/os-autoinst-needles-openQA/pull/28
Updated by openqa_review 2 months ago
- Due date set to 2024-09-20
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan 2 months ago
- Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-test-details.*matched" size:S to [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-(test-details|search-results).*matched" size:S
Updated by jbaier_cz 2 months ago
- Subject changed from [openQA-in-openQA] test does not fail if state is done but incomplete auto_review:"o candidate needle with tag(s) .*openqa-(test-details|search-results).*matched" size:S to [openQA-in-openQA] test does not fail if state is done but incomplete size:S
Updated by jbaier_cz 2 months ago
- Related to action #166445: [openQA-in-openQA][sporadic] test fails in tests, simple_boot incomplete auto_review:"no candidate needle.*openqa-test-details.*matched":retry size:M added
Updated by jbaier_cz 2 months ago
https://github.com/os-autoinst/os-autoinst-distri-openQA/pull/207 should eliminate failure in https://openqa.opensuse.org/tests/4461003#step/search/4. Sometimes the search bar is matched on the details page before the page changes and that will reset the focus so the subsequent match is broken.
Updated by jbaier_cz 2 months ago
- Due date deleted (
2024-09-20) - Status changed from Feedback to Resolved
Looking at the latest run (https://openqa.opensuse.org/tests/overview?distri=openqa&version=Tumbleweed&build=%3ATW.31375&groupid=24) all seems green again.