action #93799
Updated by vpelcak almost 3 years ago
## User story There is a new recent feature in template generator. If the automation squad disables the Whenever is openQA testcase in for aggregated openQA tests (for example in order to fix it), unscheduled, this information is automatically detected for shared via the testers template generator. Thing is, that its usability needs to identify missing testcase and cover be improved so that it becomes more useful for the missing tests by manual testing. Update Validation squad. As a tester, I then occasionally encounter stuff like: Example: http://qam.suse.de/testreports/SUSE:Maintenance:19738:241935/log regression tests: ----------------- WARNING: One or more test suites are missing compared to last runs with these source rpms. Please, check openQA manually. Missing test suites are: - Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_node01 / x86_64 sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_node01@64bit-sap-qam - Maintenance: SLE 12 SP3 SAP Incidents -> qam-create_hdd_sles4sap_gnome / x86_64 sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-create_hdd_sles4sap_gnome@64bit - Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_hana_cli / x86_64 sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_hana_cli@64bit-sap-qam - Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / ppc64le sle-12-SP3-Server-DVD-Incidents-Install-ppc64le-Build:19600:libxml2-qam-incidentinstall@ppc64le - Maintenance: SLE 12 SP3 Incidents -> mau-extratests2 / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests2@64bit - Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / s390x sle-12-SP3-Server-DVD-Incidents-Install-s390x-Build:19600:libxml2-qam-incidentinstall@zkvm - Maintenance: SLE 12 SP3 Incidents -> mau-extratests-zypper / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-zypper@64bit - Maintenance: SLE 12 SP3 SAP Incidents -> qam-incidentinstall-sap / x86_64 sle-12-SP3-Server-DVD-SAP-Incidents-Install-x86_64-Build:19600:libxml2-qam-incidentinstall-sap@64bit - Maintenance: SLE 12 SP3 Incidents -> mau-extratests-phub / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-phub@64bit - Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_netweaver / x86_64 sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_netweaver@64bit - Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_node02 / x86_64 sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_node02@64bit-sap-qam - Maintenance: SLE 12 SP3 Incidents -> qam-allpatterns+addons / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-qam-allpatterns+addons@64bit - Maintenance: SLE 12 SP3 Incidents -> mau-webserver / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-webserver@64bit - Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / x86_64 sle-12-SP3-Server-DVD-Incidents-Install-x86_64-Build:19600:libxml2-qam-incidentinstall@64bit - Maintenance: SLE 12 SP3 SAP Incidents -> qam-incidentinstall-sap / ppc64le sle-12-SP3-Server-DVD-SAP-Incidents-Install-ppc64le-Build:19600:libxml2-qam-incidentinstall-sap@ppc64le - Maintenance: SLE 12 SP3 Incidents -> mau-filesystem / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-filesystem@64bit - Maintenance: SLE 12 SP3 Incidents -> mau-extratests1 / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests1@64bit - Maintenance: SLE 12 SP3 Incidents -> mau-sles-robot-fw / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-sles-robot-fw@64bit-2gbram - Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_saptune / x86_64 sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_saptune@64bit - Maintenance: SLE 12 SP3 Incidents -> mau-extratests-kdump / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-kdump@64bit - Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_supportserver / x86_64 sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_supportserver@64bit - Maintenance: SLE 12 SP3 Incidents -> mru-install-minimal-with-addons / x86_64 sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mru-install-minimal-with-addons@64bit in the testreport. Testers do not understand it and don't know how Tester needs to interpret it. ## Acceptance criteria * **AC1:** it is clear be able to the tester easily figure out what is needed of him in those cases. Does he needs need to do when extra manual testing? What he sees message like the one above * **AC2:** ideally, information about what testing needs to be done to cover for the disabled testcase should be easily accessible I gave this high priority, because there is a danger, that additional testing won't happen and updates are released with reduced test coverage with risk of regressions. test?