action #37592
closed
[functional][y] Record soft-failure for detected errors in logs
Added by okurz almost 6 years ago.
Updated about 5 years ago.
Category:
Bugs in existing tests
Target version:
SUSE QA - Milestone 23
Description
Observation¶
openQA test in scenario sle-15-Installer-DVD-x86_64-allmodules+allpatterns@64bit shows a failed thumbnail in
logs_from_installation_system because the logfile shows errors in the yast logs. These are of minor severity for sure and in many cases are acceptable but we should still record soft-failures pointing to according bugs to remind developers about it.
Reproducible¶
Reproducible every time
Expected result¶
A failed thumbnail should prevent the job to turn out passed but record a soft-failure pointing to a specific bug
Suggestions¶
- Understand how currently the log file is parsed from within the SUT
- Ensure we have specific bugs for the individual issues, I guess we already have them reported
- Based on regex matches add record_soft_failure's to the individual bugs observed
Further details¶
Always latest result in this scenario: latest
- Due date set to 2018-11-20
- Status changed from Workable to New
Agreed to refine it, do not want to have all jobs soft-failing.
that would probably make every job "soft-fail" and therefore defeat the purpose. I guess we can consider the following levels of severity and how we handle them:
- fail for new unknown errors that we do not know the criticality of and make the jobs fail
- soft-fail referencing existing issues that do not exist everywhere
- record_info referencing existing minor issues or issues that would popup as soft-fail in too many jobs
- ignore issues that would happen everywhere and no one will fix anyway
The option 1. might still happen too often and not provide enough value for us so maybe we would could turn that into "no red border around the thumbnail for unknown issues"?
- Due date deleted (
2018-11-20)
Let's discuss within the team. Currently YaST team gives us valuable hints what to look for in the logs and what is critical. The part we could do is to ensure that all error in the YaST logs are reported as bugs, and then after some time check what is not fixed and add soft-failures, if none were fixed - reconsider approach and communicate with YaST team.
- Target version changed from Milestone 20 to Milestone 22
- Due date set to 2019-02-12
pre-fill last sprint in M22 with all tickets within milestone not yet assigned to sprints
- Blocked by action #39815: [functional][y] logs_from_installation_system always shows a red border but we do not track it as a bug added
- Due date changed from 2019-02-12 to 2019-03-12
- Status changed from New to Blocked
- Assignee set to okurz
- Target version changed from Milestone 22 to Milestone 23
I suggest to look into #39815 first and then look back here.
- Assignee changed from okurz to riafarov
@riafarov as discussed with you in person I think you can track it best
Actually is addressed in #39815 We have accepted current errors in order to detect new ones. We will communicate our findings to the YaST team, so we can clean up logs from things which are not errors.
- Status changed from Blocked to Resolved
As per last comment, this is now addressed and we now accept >80 known errors in the logs and soft-fail those.
Also available in: Atom
PDF