QA » openQA Project » openQA Tests » qe-yam

As a Quality Engineer I would like to validate that all the promises that the installer does are met in the installed system.

Therefore we need a set of validations which met the following criteria:

• Single-purpose validation, enough atomic to be used in any other scenario.
• Utilize test data and have a common structure for all the scenarios.
• Use always common library for parsing the output (currently there are 2-3 implementation of the same thing which need to be unified).
• Validations are focused on the verification of the configuration that the installer does, change the system and validate it is not in scope.
• Validation are multi-architecture and product.
• Validations are focused mainly in what is visible in the stages of the installer with the following order of preference:
  • User selection/actions: test data for those should be reusable by YuiRestClient.
  • defaults displayed in the stages of the installer for a specific scenario (any control that is visible which the user didn't interact with)
• Additional validation can be focused on (the ones not visible in any stage of the installer):
  • documentation and other defaults of the test scenario.
  • any other thing the tester/developers would consider important as well.
• Validation are easy to expand adding a few more test data and perhaps some code, i.e: check a new setting in the same file should be cheap.
• Consider speed of validation, reading just once the output and doing the rest in Perl.
• When validation fails should be very easy to understand the problem (expected vs actual results)
• All scenarios in YaST group should apply those validations. Consider not to repeat the validation in similar scenarios, specially for defaults.
• Stages of the installer could be related to more than one screen, so validation modules will not match 1 to 1 with a dialog during installation, instead they will cover specific functionality of a particular stage, or in other words, all the dialogs related with some stage, i.e.: validate networking, validate software.
• Scope of this validation is huge, so main target will be partitioning, networking, software, services and additionally some others like Booting options, Timezone, etc.

Examples:

• Test case to validate stage Network Settings configuration for interface, dhcp, hostname, routing could be checked in the installed system.
• Test case to validate stage System role when selecting Text Mode we could figure out a simple validation of X11 and find out what should not be in the system comparing with a gnome installation.
• Test case to validate stage Suggested Partitioning subvolume actions could be a good candidate, clicking on 'see details' there are info that we could check.
• Test case to validate stage Installation Settings could be split in multiple modules to check every section. Clicking on each item there are more promises that the software does that could be tested individually and deserve attention even if the user does not navigate to them during the installation, but those are the promises anyway.