QA » openQA Project » openQA Tests

My suggestion from experience: Please avoid test suites which have the machine or architecture in the name. If there needs to be architecture specific behaviour then implement according code branches within the test code as we also have e.g. for lists of modules that are expected to be auto-selected, etc.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: migration_offline_sle12sp3_sdk+allpatterns_fullupdate
https://openqa.suse.de/tests/2244637

By now openqa.suse.de even has a new feature which allows to define job templates inheriting from test suites so that it is not necessary anymore to define test suites which are product+version+architecture specific but you can define these directly as job templates within the job group. I suggest you use that feature to reduce the amount of test suites that need to be maintained. We hope you find these new features helpful to reduce maintenance effort :)

okurz wrote:

By now openqa.suse.de even has a new feature which allows to define job templates inheriting from test suites so that it is not necessary anymore to define test suites which are product+version+architecture specific but you can define these directly as job templates within the job group. I suggest you use that feature to reduce the amount of test suites that need to be maintained. We hope you find these new features helpful to reduce maintenance effort :)

Will check within team and give feedback. Thanks for notification:)

okurz wrote:

By now openqa.suse.de even has a new feature which allows to define job templates inheriting from test suites so that it is not necessary anymore to define test suites which are product+version+architecture specific but you can define these directly as job templates within the job group. I suggest you use that feature to reduce the amount of test suites that need to be maintained. We hope you find these new features helpful to reduce maintenance effort :)

Hi Oliver,
Thanks for your suggestion. Let me explain for this:
1. For product+version, That's the base system information.
For example:
offline_sles12sp2_ltss_media_sdk_all_full & offline_sles12sp3_ltss_media_sdk_all_full
The first case, upgrade from sles12sp2. The second one upgrade from sles12sp3. We need record this with cases' name.
It cannot be defined with job templates. As those cases in the same job templates.
2. For architecture, that's to define the same cases name but with different settings.
For example:
offline_sles12sp2_ltss_media_base_all_full_x86_64 & offline_sles12sp2_ltss_media_base_all_full_ppc64le
It has the some cases' name before the architecture, but it need set the different SCC_REGCODE_LTSS. So we need add architecture to add the cases to the test suits.
For this part, we add the architecture for some cases without this problem. I will update those cases' name ASAP.
I hope it can explain your question. If you have any questions or suggestion, please let me know.
BRs,

tinawang123 wrote:

Thanks for your suggestion. Let me explain for this:
1. For product+version, That's the base system information. 
   For example: 
   offline_sles12sp2_ltss_media_sdk_all_full & offline_sles12sp3_ltss_media_sdk_all_full
   The first case, upgrade from sles12sp2. The second one upgrade from sles12sp3. We need record this with cases' name. 
   It cannot be defined with job templates. As those cases in the same job templates.

I don't understand the last two sentences. "As those cases in the same job templates." is not a complete sentence. Please rephrase.

In case your example "offline_sles12sp2_ltss_media_sdk_all_full" is only used within one job group you could define that as a job template within that job group with a YAML template block, e.g.:

- offline_sles12sp2_ltss_media_sdk_all_full:
    testsuite: offline_ltss_media_sdk_all_full
    settings:
      HDDVERSION: 12-SP2

2. For architecture, that's to define the same cases name but with different settings.
   For example:
   offline_sles12sp2_ltss_media_base_all_full_x86_64 & offline_sles12sp2_ltss_media_base_all_full_ppc64le
   It has the some cases' name before the architecture, but it need set the different SCC_REGCODE_LTSS. So we need add architecture to add the cases to the test suits. 
   For this part, we add the architecture for some cases without this problem. I will update those cases' name ASAP.

Architecture specifics should definitely not be needed to encode in multiple test suites. For architecture differences you can use the product definitions in https://openqa.suse.de/admin/products and looking at the current entries it looks like there are already many products used by migration with a similar purpose. Even if this is about an SCC code for the to-be-upgraded SLE version you can put it there, e.g. with a version suffix. Probably better would be anyway to not need this code but rely on registered and properly patched base images anyway. For any other settings that differ you can do that configuration also within os-autoinst-distri-opensuse itself. Also this is (or was?) used by migration tests, e.g. see line https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/products/sle/main.pm#L213 and below.

I hope it can explain your question

Actually I did not have any questions. Did I ask any and forgot them? :)

For the first problem, I have two sides to explain:
First:
with your example:

• offline_sles12sp2_ltss_media_sdk_all_full:
  testsuite: offline_ltss_media_sdk_all_full <=========== Case name
  settings:
  HDDVERSION: 12-SP2

• offline_sles12sp3_ltss_media_sdk_all_full:
  testsuite: offline_ltss_media_sdk_all_full <=========== Same case name
  settings:
  HDDVERSION: 12-SP3
  I set YAML template:
  scenarios:
  aarch64:
  sle-12-SP5-Migration-from-SLE12-SP5-to-SLE15-SPx-aarch64:

  • offline_pscc_base_def_full: settings: HDDVERSION: 12-SP2
  • offline_pscc_base_def_full: settings: HDDVERSION: 12-SP3 Got error message: There was a problem applying the changes:

  Job template name 'offline_pscc_base_def_full' is defined more than once. Use a unique name and specify 'testsuite' to re-use test suites in multiple scenarios.

Another side:
It is easy way for us to check our tests scenario. As do upgrade to sles12sp5. We have a lot of migration path like: from sles12sp2, sles12sp3, sles12sp4. We can use the cases' name to get the base system information easily. Otherwise, we need open the cases' setting, then check the base system. This base system information is very important for us and it is very useful for us if we get the information quickly from cases' name.

This method was defined at our project prepare staging. Now, we will prepare our upgrade to sles15sp2, if you still have some concern about this. Welcome to your suggestion.

For the second problem. It is a little complex to explain. I will try my best to explain more clear.
First example:
With the same Medium types as 'sle-12-SP5-Migration-from-SLE12-SPx-x86_64':

• offline_sles12sp2_media_base_all_full_x86_64 Setting: SCC_REGCODE_LTSS=4Cxxxxxxxxxxx
• offline_sles12sp3_media_base_all_full_x86_64 Setting: SCC_REGCODE_LTSS=36xxxxxxxxxxx So, we cannot put SCC_REGCODE_LTSS to flavor setting, as 12sp3 and 12sp2 has different regcode for ltss with the same medium type. So, why we need add arch at the cases' name, let me introduce the second example: Second example: Case have same name: offline_sles12sp2_ltss_media_base_all_full_x86_64 & offline_sles12sp2_ltss_media_base_all_full_s390x except arch. But for s390x media upgrade need add setting: ADDONURL sdk ADDONURL_SDK ftp://openqa.suse.de/SLE-12-SP5-SDK-POOL-s390x-Build0245-Media1/ For x86_64 media upgrade don't need those settings. This is just an example to explain, but it has other complex settings for other cases.

I hope you can understand those information. But we will keep to simplify our cases' name according the document.
If you are interesting to our cases, when our sles15sp2 matrix document ready, you can help to review :).

I have asked xiaojing, I have already known your point.
Generally speaking, if we have same cases at different arch and with some different settings and some same settings. We can put the cases' same setting at test suits, and put the cases' different settings at YAML. Then we can reduce the length of our cases' name.
We will think about this method at our sles15sp2 migration testing.
Thanks for your advice.

tinawang123 wrote:

For the first problem, I have two sides to explain:
First:
with your example:

• offline_sles12sp2_ltss_media_sdk_all_full:
  testsuite: offline_ltss_media_sdk_all_full <=========== Case name
  settings:
  HDDVERSION: 12-SP2

• offline_sles12sp3_ltss_media_sdk_all_full:
  testsuite: offline_ltss_media_sdk_all_full <=========== Same case name
  settings:
  HDDVERSION: 12-SP3
  I set YAML template:
  scenarios:
  aarch64:
  sle-12-SP5-Migration-from-SLE12-SP5-to-SLE15-SPx-aarch64:

  • offline_pscc_base_def_full: settings: HDDVERSION: 12-SP2
  • offline_pscc_base_def_full: settings: HDDVERSION: 12-SP3 Got error message: There was a problem applying the changes:

  Job template name 'offline_pscc_base_def_full' is defined more than once. Use a unique name and specify 'testsuite' to re-use test suites in multiple scenarios.

Yes, the error message explains that pretty well. The header of each section defines the name which is either an existing testsuite or – when combined with testsuite: … you define the job template name. In my example there must be a test suite "offline_ltss_media_sdk_all_full" but the resulting scenarios in the job group have the name token "offline_sles12sp2_ltss_media_sdk_all_full" and "offline_sles12sp3_ltss_media_sdk_all_full" respectively included.

There are two problems:

1. If the test settings split to two places: test suits and YAML setting. It is difficult for us to debug the tests. As we need check two places to confirm the settings are right.
2. Add the cases' name to YAML, if input the wrong name which cases are not existed. YAML cannot return error message. It is difficult for user to check the name one by one.

tinawang123 wrote:

There are two problems:

1. If the test settings split to two places: test suits and YAML setting. It is difficult for us to debug the tests. As we need check two places to confirm the settings are right.

yes, I can understand that. There are actually many more places where settings come from. This is why I suggested initially to move much more into the test code. Now it is already splitted to medium, worker, machines, test code, test suites and now job templates. The hope is that the latter make reuse easier and eventually we can also improve on that general problem.

1. Add the cases' name to YAML, if input the wrong name which cases are not existed. YAML cannot return error message. It is difficult for user to check the name one by one.

This is a valid point. This might be adressed in the future. But please keep in mind that we need to hurry up a bit to do at least the first step, that is to manage the job group schedule in YAML, as it was planned to decommission the old way and it is better if you explicitly do the transformation otherwise eventually the tools team needs to forcefully migrate, e.g. in the coming weeks, without using the benefits.

We only changed sle15sp2 test cases' name.
For the old cases, we don't change the name as it will lost the cases' log.
Please check cases' name at:
https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=164.1&groupid=265
https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=162.1&groupid=266
https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=164.1&groupid=267
There is not arch information now.

Don't worry, there is no problem to update or delete all test suites as the existing test results are not affected by the update. QA SLE Migration needs to take responsibility for all migration test suites to ensure we have a long-term sustainable setup on osd (otherwise we will end up with an unmaintainable mess of test suites and job templates). . Also, there is no proper backup of existing older test results from osd anyway so old results can not be that important :)