Project

General

Profile

Actions

coordination #173875

open

coordination #173872: [saga][epic] Full traceability "paper trail" and test reproducability of openQA tests

[epic] Full traceability and reproducibility of openQA tests

Added by okurz about 1 month ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2024-12-06
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)

Description

This epic focuses on enhancing openQA and its related tools to provide framework and API for test traceability and reproducibility. The goal is to enable stakeholders, including QE engineers, managers, test developers, openQA users, and maybe even customers, to easily track and reproduce test executions across all software development and maintenance processes. By providing a reliable paper trail for all test activities and enabling full test reproducibility, this framework aims to improve transparency, accountability, and quality assurance.

Business value

This framework will improve customer and management confidence by providing concrete evidence of testing processes and results of any given maintenance update or software/prduct being tested. It will reduce incident resolution time by enabling precise test reproduction and enhance collaboration with contributors by streamlining test result sharing.

Key Features

1. Test catalog

  • Provide an easy to consume (manager friendly) comprenesive list of all available tests

2. Test paper trail

  • Provide a detailed record of executed tests, including:

    • Test names and exact versions.
    • Environments in which tests were executed (e.g., OS, hardware, configurations).
    • Clear results (pass/fail) with artifacts for debugging failures.
  • Accessible via an intuitive web UI and a CLI tool.

3. Full test reproducibility

  • Allow tests to be rerun under identical conditions, including:

    • Consistent environments (OS, hardware configurations).
    • Stable test inputs and configurations.
    • Fixed versions (git hash) of tests and openQA framework components.
  • Provide easy mechanisms (via UI or CLI) to rerun tests for debugging, escalation resolution, or customer inquiries.

Acceptance criteria

1. Traceability

  • All executed tests, including historical data, must be traceable with links to their configurations, source code, and artifacts.
  • Provide seamless integration with the existing openQA and SMELT for incident tracking.

2. Reproducibility demonstration

  • Demonstrate the ability to reproduce a specific test run, including pulling historical maintenance updates and executing the exact same test suite.

3. Scalability and accessibility

  • Ensure the solution works across all testing activities, not just for maintenance updates.
  • Avoid dependency on transient systems like the QAM Dashboard for data persistence.

4. Transparency

  • Clearly document the steps and requirements for test reproducibility to ensure consistency and reliability.
  • Offer demonstrations and comprehensive guides to validate reproducibility capabilities.

Subtasks 1 (1 open0 closed)

action #173878: Documentation of how to achieve full traceability "paper trail" of openQA testsNew2024-12-06

Actions
Actions #1

Updated by okurz about 1 month ago

  • Subtask #173878 added
Actions #2

Updated by bzoltan1 about 1 month ago

  • Subject changed from [epic] Full traceability "paper trail" for openQA tests to [epic] Full traceability and reproducibility of openQA tests
Actions #3

Updated by okurz about 1 month ago

sorry, I don't understand your subject change. The parent saga is about trace+reproduce. This epic is about traceability only. And I prefer to keep "paper trail" in the subject so that we would be able to find this ticket again when we search for alternative terms.

Actions #4

Updated by bzoltan1 about 1 month ago

  • Description updated (diff)
Actions

Also available in: Atom PDF