action #151310
Updated by livdywan 6 months ago
## Motivation
As visible on https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&from=1700508932604&to=1700724085546&viewPanel=24
since about 2023-11-21 there is again a significant increase of multi-machine tests which should be investigated, mitigated, fixed and prevented.
## Acceptance criteria
* **AC1:** failed+parallel_failed on https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&viewPanel=24 is significantly below 20% again
## Suggestions
* Start to look into the issue early as waiting longer makes everything harder for us :)
* Lookup common failure sources and find out if it's actually not test or product regressions.
* Ask common stakeholders and/or test reviewers if they know something
* Review recent infrastructure changes which might be possibly related
* Mitigate, fix and prevent the issues you find
* Consider using the scientific method https://progress.opensuse.org/projects/openqav3/wiki/#Further-decision-steps-working-on-test-issues
* Use SQL queries to find out what failures are most common
* Consider using this opportunity to document one or two examples of how we commonly do that