Project

General

Profile

action #93799

Updated by vpelcak 3 months ago

## User story
There is a new recent feature in template generator.
If the automation squad disables the


Whenever is openQA
testcase in for aggregated openQA tests (for example in order to fix it), unscheduled, this information is automatically detected for shared via the testers template generator.
Thing is, that its usability needs
to identify missing testcase and cover be improved so that it becomes more useful for the missing tests by manual testing. Update Validation squad.

As a tester, I then occasionally encounter stuff like:

Example:

http://qam.suse.de/testreports/SUSE:Maintenance:19738:241935/log

regression tests:
-----------------
WARNING: One or more test suites are missing compared to last runs with these source rpms.
Please, check openQA manually. Missing test suites are:

- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_node01 / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_node01@64bit-sap-qam
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-create_hdd_sles4sap_gnome / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-create_hdd_sles4sap_gnome@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_hana_cli / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_hana_cli@64bit-sap-qam
- Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / ppc64le
sle-12-SP3-Server-DVD-Incidents-Install-ppc64le-Build:19600:libxml2-qam-incidentinstall@ppc64le
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests2 / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests2@64bit
- Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / s390x
sle-12-SP3-Server-DVD-Incidents-Install-s390x-Build:19600:libxml2-qam-incidentinstall@zkvm
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests-zypper / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-zypper@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-incidentinstall-sap / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-Install-x86_64-Build:19600:libxml2-qam-incidentinstall-sap@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests-phub / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-phub@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_netweaver / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_netweaver@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_node02 / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_node02@64bit-sap-qam
- Maintenance: SLE 12 SP3 Incidents -> qam-allpatterns+addons / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-qam-allpatterns+addons@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-webserver / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-webserver@64bit
- Maintenance: SLE 12 SP3 Incidents -> qam-incidentinstall / x86_64
sle-12-SP3-Server-DVD-Incidents-Install-x86_64-Build:19600:libxml2-qam-incidentinstall@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-incidentinstall-sap / ppc64le
sle-12-SP3-Server-DVD-SAP-Incidents-Install-ppc64le-Build:19600:libxml2-qam-incidentinstall-sap@ppc64le
- Maintenance: SLE 12 SP3 Incidents -> mau-filesystem / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-filesystem@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests1 / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests1@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-sles-robot-fw / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-sles-robot-fw@64bit-2gbram
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_scc_gnome_saptune / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_scc_gnome_saptune@64bit
- Maintenance: SLE 12 SP3 Incidents -> mau-extratests-kdump / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mau-extratests-kdump@64bit
- Maintenance: SLE 12 SP3 SAP Incidents -> qam-sles4sap_hana_supportserver / x86_64
sle-12-SP3-Server-DVD-SAP-Incidents-x86_64-Build:19600:libxml2-qam-sles4sap_hana_supportserver@64bit
- Maintenance: SLE 12 SP3 Incidents -> mru-install-minimal-with-addons / x86_64
sle-12-SP3-Server-DVD-Incidents-x86_64-Build:19600:libxml2-mru-install-minimal-with-addons@64bit

in the testreport.

Testers do not understand it and don't know how
Tester needs to interpret it.

## Acceptance criteria
* **AC1:** it is clear
be able to the tester easily figure out what is needed of him in those cases.
Does
he needs need to do when extra manual testing? What he sees message like the one above
* **AC2:** ideally, information about what testing
needs to be done to cover for the disabled testcase should be easily accessible

I gave this high priority, because there is a danger, that additional testing won't happen and updates are released with reduced test coverage with risk of regressions.
test?

Back