QA » openQA Project » openQA Tests » qe-yam

Motivation¶

In general the flow of the screen is predictable and we can fix expectation, but there are some corner cases where this is not true.

Example of this are for example these two:

Product selection is something that normally happens during installation, but if there is only one product to select, for example in s390x when no SUMA available, this screen doesn't appear. In SLE-15-SP4 GM we have no SUMA for Offline and we have it for Online.
https://openqa.suse.de/tests/9313573#step/accept_license/1
https://openqa.suse.de/tests/8751121#step/accept_license/1
https://openqa.suse.de/tests/8751247#step/install_SLES/1

Beta acceptance is also something that happen until certain point during product development
https://openqa.suse.de/tests/9313573#step/access_beta_distribution/1
https://openqa.suse.de/tests/8751121#step/accept_license/1

We have available mechanism with default+flow to achieve that.
Therefore we can say that the main goal of this ticket is to have single place to update this. Instead of focusing only in fixing the test, we should start to organize our test coverage based on defaults. Remember that 'flows' were created initially to account for all the strange variation on the test coverage that are performed and that the PRD doesn't reflect so it is up to us to make something reasonable. We can start to remove those flows and move them to default files for each flavor.

Acceptance criteria¶

AC1: The problem with no product selection is solved using yaml feature default+flows.
AC2: Solution is applied to select_modules_and_patterns+registration@s390x-kvm-sle12

Suggestions¶

As we are actually testing the final product, we can say that the default should be how it will look in GM. Then on top of that we can add our flows to modify that behavior.

A possible implementation would be to create a flow no_product_selection.yaml which will clean up the key product_selection in default.yaml making it an empty hook and add this flow to all test suite in s390x. The problem is that we need to add that flow to all the test suite in one architecture, so there is no single place.

In this case I think it would make more sense to create default_s390x.yaml. We could reuse default.yaml and use merge keys mechanism to update in one place this behavior, afir remember multiple included were messing up with the order of the key, but please verify that, if we use one include and multiple key override it might work.

In case this doesn't work, we might need to copy paste all the content in default to this new yaml. At the end we can consider single place as all the files for default inside its own folder, that might be acceptable, otherwise, if we want 'true' single place we need to implement some keyword in yaml to be able to include and overwrite multiple keys.

With this change we should fix the following two test cases besides solving the beta issues after publicbeta:
https://openqa.suse.de/tests/9313573#step/accept_license/1
https://openqa.suse.de/tests/9313575#step/accept_license/1

The beta problem is for all the architectures, so we can rely on default.yaml and update it in single place when corresponding. Nothing to do then for this case (just was an example), only to migrate test suite to this new mechanism in follow-up tickets.