QA » openQA Project » openQA Tests

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2468 is adding the "load_testdir" function which can help a bit.

One outcome of a discussion by coolo and me: Would be good if the existing tables of testsuites and the actual test plan (main.pm) reside together. Currently testsuites on o3/osd grew a lot, e.g. huge bunch of LTP and migration tests

A bit more details from the discussion. First of all, we all acknowledge the problem with main.pm and we predicted that it will become un-maintainable, which I would say already happened.
Just to write brainstorming outcomes, which was more than just what was mentioned by Oliver.

Possible solutions:

• Storing plain schedules for each test suite with possibility to inherit them. For migration part, we can evaluate schedule for each existing test suite, for each medium and each machine
  • - redundancy
  • - more efforts for maintenance, however with simple steps
  • + transparency
  • + granularity, individual test suite reflects business logic
• Storing test suites using serialization language with code injections and dependencies (yml + pre/macro processors)
  • - may get as complex as main.pm
  • - generalization which may lead to executing tests with no business value for particular setups
  • + audit and versions control for no cost using git
  • + no/less redundancy
• Parameterized test suites:
  • - conditions may get as complex as we currently have affecting transparency
  • - generalization which may lead to executing tests with no business value for particular setups
  • + no/less redundancy
  • + localized complexity for conditions (even with complex conditions, one test suite one affect conditions for another one)
• Forking main.pm per area/job group
  • + cheapest solution
  • - will only postpone current issue and introduce new challenges

Steps to mitigate consequences of un-maintainable main.pm:

• split existing test-suites to smaller ones
• reuse common images, where snapshots are supported
• reduce test coverage on bare metal and revisit business values of those by evaluating risks of package/component not to work, when tested in the different environment (e.g. firefox on ipmi)

Ideas from workshop meeting 2018-11-26:

• "Copy-pasting" happens either in "main.pm" or openQA test suites to evade the pain, we have to cover the complexity somewhere
• test suites should prescribe the workflow, main.pm "was meant to" provide a parameterized run
• test suite inheritance for code reuse?
• code reuse between work flows can be improved, e.g. have a list of settings in test suites with parameterized test suite names, e.g. switch_keyboard_%DESKTOP% with setting DESKTOP=[textmode,gnome] and START_AFTER_TEST=%DESKTOP% -> how would the specific scenario be mapped in the job groups then? Best opinion so far: Have test suites as well as job groups defined within the test distribution
• dynamic test suite definition by test code?

sure, thx. All ticket relations are still current so I can just keep he status as is.

See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html