action #93799
Updated by tinita over 1 year ago
## User story
There is a new recent feature in template generator.
If the automation squad disables the testcase in aggregated tests (for example in order to fix it), this is automatically detected for the testers to identify missing testcase and cover the missing tests by manual testing.
As a tester, I then occasionally encounter stuff like:
Example:
http://qam.suse.de/testreports/SUSE:Maintenance:19738:241935/log
regression tests:
-----------------
WARNING: One or more test suites are missing compared to last runs with these source rpms.
Please, check openQA manually. Missing test suites are:
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_node01 / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_node01@64bit-sap-qam
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-create_hdd_sles4sap_gnome / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-create_hdd_sles4sap_gnome@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_hana_cli / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_hana_cli@64bit-sap-qam
- Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / ppc64le
sle-12-SP3-Server-DVD-Incidents-Install-ppc64le-Build:19600:libxml2-qam-incidentinstall@ppc64le
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests2 / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests2@64bit
- Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / s390x
sle-12-SP3-Server-DVD-Incidents-Install-s390x-Build:19600:libxml2-qam-incidentinstall@zkvm
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests-zypper / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-zypper@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-incidentinstall-sap / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-Install-x86_64-Build:19600:libxml2-qam-incidentinstall-sap@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests-phub / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-phub@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_netweaver / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_netweaver@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_node02 / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_node02@64bit-sap-qam
- Maintenance: SLE 12 SP3 Incidents -> qam-allpatterns+addons / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-qam-allpatterns+addons@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-webserver / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-webserver@64bit
- Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / x86_64
sle-12-SP3-Server-DVD-Incidents-Install-x86_64-Build:19600:libxml2-qam-incidentinstall@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-incidentinstall-sap / ppc64le
sle-12-SP3-Server-DVD-SAP-Incidents-Install-ppc64le-Build:19600:libxml2-qam-incidentinstall-sap@ppc64le
- Maintenance: SLE 12 SP3 Incidents -> mau-filesystem / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-filesystem@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests1 / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests1@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-sles-robot-fw / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-sles-robot-fw@64bit-2gbram
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_saptune / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_saptune@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests-kdump / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-kdump@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_supportserver / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_supportserver@64bit
- Maintenance: SLE 12 SP3 Incidents -> mru-install-minimal-with-addons / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mru-install-minimal-with-addons@64bit
in the testreport.
Testers do not understand it and don't know how to interpret it.
## Acceptance criteria
* **AC1:** it is clear to the tester what he needs to do when he sees message like the one above
* **AC2:** ideally, information about what testing needs to be done to cover for the disabled testcase should be easily accessible
## Suggestions
* Look at https://gitlab.suse.de/qa-maintenance/teregen/-/blob/master/app/lib/TeReGen/Model/Plugin/91_CreateTemplate.pm
* Move regression tests warning details into a separate file
I gave this high priority, because there is a danger, that additional testing won't happen and updates are released with reduced test coverage with risk of regressions.