This epic focuses on enhancing openQA and its related tools to provide framework and API for test traceability and reproducibility. The goal is to enable stakeholders, including QE engineers, managers, test developers, openQA users, and maybe even customers, to easily track and reproduce test executions across all software development and maintenance processes. By providing a reliable paper trail for all test activities and enabling full test reproducibility, this framework aims to improve transparency, accountability, and quality assurance.

Business value¶

This framework will improve customer and management confidence by providing concrete evidence of testing processes and results of any given maintenance update or software/prduct being tested. It will reduce incident resolution time by enabling precise test reproduction and enhance collaboration with contributors by streamlining test result sharing.

Key Features¶

1. Test catalog¶

• Provide an easy to consume (manager friendly) comprenesive list of all available tests

2. Test paper trail¶

• Provide a detailed record of executed tests, including:

  • Test names and exact versions.
  • Environments in which tests were executed (e.g., OS, hardware, configurations).
  • Clear results (pass/fail) with artifacts for debugging failures.

• Accessible via an intuitive web UI and a CLI tool.

3. Full test reproducibility¶

• Allow tests to be rerun under identical conditions, including:

  • Consistent environments (OS, hardware configurations).
  • Stable test inputs and configurations.
  • Fixed versions (git hash) of tests and openQA framework components.

• Provide easy mechanisms (via UI or CLI) to rerun tests for debugging, escalation resolution, or customer inquiries.

Acceptance criteria¶

1. Traceability¶

• All executed tests, including historical data, must be traceable with links to their configurations, source code, and artifacts.
• Provide seamless integration with the existing openQA and SMELT for incident tracking.

2. Reproducibility demonstration¶

• Demonstrate the ability to reproduce a specific test run, including pulling historical maintenance updates and executing the exact same test suite.

3. Scalability and accessibility¶

• Ensure the solution works across all testing activities, not just for maintenance updates.
• Avoid dependency on transient systems like the QAM Dashboard for data persistence.

4. Transparency¶

• Clearly document the steps and requirements for test reproducibility to ensure consistency and reliability.
• Offer demonstrations and comprehensive guides to validate reproducibility capabilities.