Project

General

Profile

Actions

coordination #88684

open

coordination #64126: [qe-core][epic] Identify packages that have automated indirect testing or that have sufficient build time test suite

[qe-core][epic][qem] Identify packages that we are indirectly testing

Added by tjyrinki_suse about 3 years ago. Updated 9 months ago.

Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
Enhancement to existing tests
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

We test some packages indirectly as part of other packages' test. The template generator (https://gitlab.suse.de/qa-maintenance/metadata/-/tree/master/template_generator) Update Squad uses does not detect these cases, and does not then tell the update tester there's automatic openQA regression testing.

For example, webkit is used in evolution - if there have been maintenance updates of webkit, was the manual testing more complex than what the evolution automatic test case would be, or not? If the evolution test tests webkit to an enough extent, webkit could be marked as automatically regression tested, and manual testing would no longer be required.

Acceptance Criteria

AC1: Identify what packages we are regression testing indirectly, and via which tests, check if it's "sufficient" for regression testing of that package: sufficient means at least similar level as in manual testing done in case of updates, see http://qam.suse.de/ database for how the packages was teded manually.
AC2: Mark sufficiently indirectly regression tested packages as automatically tested. This can be done, in the current way, with an entry at PES wiki page (https://pes.suse.de/QA_Maintenance/Automation_Progress_Tracking_in_QAM/) - the template generator parses that page to know if there has been automatic regression testing. Add a separate header for indirectly tested packages.

Actions #1

Updated by tjyrinki_suse about 3 years ago

  • Description updated (diff)
Actions #2

Updated by tjyrinki_suse almost 3 years ago

  • Description updated (diff)
  • Status changed from New to Workable
Actions #3

Updated by tjyrinki_suse about 2 years ago

  • Description updated (diff)

Changed "decide" -> "check" in AC1. GTK was offered as an example of something that is being tested indirectly, yet not documented at https://pes.suse.de/QA_Maintenance/Automation_Progress_Tracking_in_QAM/ - however in this example there is only about 1 GTK update per year (http://qam.suse.de/?report_id=&packages=gettext-its-gtk3&product=&result=&fulltext=)

Actions #4

Updated by apappas about 2 years ago

  • Assignee set to apappas
Actions #5

Updated by apappas about 2 years ago

  • Status changed from Workable to Feedback

Notes from the refinement meeting:

  1. It is not the role of the team members to decide what constitutes "enough" testing. If there is a ticket whose ACs can be fullfiled by existing tests we can certainly tell you that, but first we need to know what they are.
  2. The update validation team should be asked which are the packages that still have non-automated time consuming tests, and we should focus on them.
Actions #6

Updated by tjyrinki_suse about 2 years ago

  • Tracker changed from action to coordination
  • Subject changed from [qe-core][qem] Identify packages that we are indirectly testing to [qe-core][epic][qem] Identify packages that we are indirectly testing
  • Assignee deleted (apappas)
  1. The AC1 defines the "enough" testing, ie comparing how manual testing was done. If similar level of testing is done indirectly already in openQA, it's enough.
  2. This is a good idea, to ask if they have ideas.

All in all this is more an epic that cannot be fully considered "done" by any single group of tasks.

Actions #7

Updated by okurz 9 months ago

  • Category set to Enhancement to existing tests
Actions

Also available in: Atom PDF