Project

General

Profile

Actions

action #43499

closed

[sle][migration] test suites should not have an architecture specific suffix

Added by okurz almost 6 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Target version:
-
Start date:
2018-11-07
Due date:
% Done:

100%

Estimated time:
30.00 h
Difficulty:

Description

Observation

openQA test in scenario sle-12-SP4-Server-DVD-ppc64le-migration_offline_sle12sp2_sdk+allpatterns_fullupdate_ppc64le@ppc64le fails in
addon_products_sle

Suggestions

Please avoid test suites which have the machine or architecture in the name. If there needs to be architecture specific behaviour then implement according code branches within the test code as we also have e.g. for lists of modules that are expected to be auto-selected, etc.

Further details

Always latest result in this scenario: latest


Related issues 1 (0 open1 closed)

Related to openQA Project - coordination #55730: [epic] Move parameters from test suites into job groupsResolvedokurz2019-09-06

Actions
Actions #1

Updated by okurz almost 6 years ago

My suggestion from experience: Please avoid test suites which have the machine or architecture in the name. If there needs to be architecture specific behaviour then implement according code branches within the test code as we also have e.g. for lists of modules that are expected to be auto-selected, etc.

Actions #2

Updated by okurz almost 6 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: migration_offline_sle12sp3_sdk+allpatterns_fullupdate
https://openqa.suse.de/tests/2244637

Actions #3

Updated by okurz about 5 years ago

  • Related to coordination #55730: [epic] Move parameters from test suites into job groups added
Actions #4

Updated by okurz about 5 years ago

  • Subject changed from [sle][migration] test fails in addon_products_sle - test suite should not have an architecture specific suffix + BETA_SDK must not be set to [sle][migration] test suites should not have an architecture specific suffix
  • Description updated (diff)
  • Priority changed from Normal to High

This is now linked to #55730 . I can see that (still) many test suites used by the SLE migration job groups have an architecture specific suffix. This is something that should definitely be avoided to allow reuse. Parameterization could always been done within the test code but there is now an additional option as well, "job templates" based on YAML templates. As the old web UI job group support will be deprecated soon – as also announced in http://mailman.suse.de/mlarch/SuSE/openqa/2019/openqa.2019.08/msg00000.html – please plan and execute the migration of the job groups to YAML which is as easy as saving the existing job group schedule with the save button in the web UI :) This unlocks the new features for you to also parameterize the test scenarios within the job group which should allow you to simplify test suites again.

Actions #5

Updated by okurz about 5 years ago

By now openqa.suse.de even has a new feature which allows to define job templates inheriting from test suites so that it is not necessary anymore to define test suites which are product+version+architecture specific but you can define these directly as job templates within the job group. I suggest you use that feature to reduce the amount of test suites that need to be maintained. We hope you find these new features helpful to reduce maintenance effort :)

Actions #6

Updated by coolgw about 5 years ago

okurz wrote:

By now openqa.suse.de even has a new feature which allows to define job templates inheriting from test suites so that it is not necessary anymore to define test suites which are product+version+architecture specific but you can define these directly as job templates within the job group. I suggest you use that feature to reduce the amount of test suites that need to be maintained. We hope you find these new features helpful to reduce maintenance effort :)

Will check within team and give feedback. Thanks for notification:)

Actions #7

Updated by tinawang123 about 5 years ago

okurz wrote:

By now openqa.suse.de even has a new feature which allows to define job templates inheriting from test suites so that it is not necessary anymore to define test suites which are product+version+architecture specific but you can define these directly as job templates within the job group. I suggest you use that feature to reduce the amount of test suites that need to be maintained. We hope you find these new features helpful to reduce maintenance effort :)

Hi Oliver,
Thanks for your suggestion. Let me explain for this:
1. For product+version, That's the base system information.
For example:
offline_sles12sp2_ltss_media_sdk_all_full & offline_sles12sp3_ltss_media_sdk_all_full
The first case, upgrade from sles12sp2. The second one upgrade from sles12sp3. We need record this with cases' name.
It cannot be defined with job templates. As those cases in the same job templates.
2. For architecture, that's to define the same cases name but with different settings.
For example:
offline_sles12sp2_ltss_media_base_all_full_x86_64 & offline_sles12sp2_ltss_media_base_all_full_ppc64le
It has the some cases' name before the architecture, but it need set the different SCC_REGCODE_LTSS. So we need add architecture to add the cases to the test suits.
For this part, we add the architecture for some cases without this problem. I will update those cases' name ASAP.
I hope it can explain your question. If you have any questions or suggestion, please let me know.
BRs,

Actions #8

Updated by okurz about 5 years ago

tinawang123 wrote:

Thanks for your suggestion. Let me explain for this:
1. For product+version, That's the base system information. 
   For example: 
   offline_sles12sp2_ltss_media_sdk_all_full & offline_sles12sp3_ltss_media_sdk_all_full
   The first case, upgrade from sles12sp2. The second one upgrade from sles12sp3. We need record this with cases' name. 
   It cannot be defined with job templates. As those cases in the same job templates.

I don't understand the last two sentences. "As those cases in the same job templates." is not a complete sentence. Please rephrase.

In case your example "offline_sles12sp2_ltss_media_sdk_all_full" is only used within one job group you could define that as a job template within that job group with a YAML template block, e.g.:

- offline_sles12sp2_ltss_media_sdk_all_full:
    testsuite: offline_ltss_media_sdk_all_full
    settings:
      HDDVERSION: 12-SP2
2. For architecture, that's to define the same cases name but with different settings.
   For example:
   offline_sles12sp2_ltss_media_base_all_full_x86_64 & offline_sles12sp2_ltss_media_base_all_full_ppc64le
   It has the some cases' name before the architecture, but it need set the different SCC_REGCODE_LTSS. So we need add architecture to add the cases to the test suits. 
   For this part, we add the architecture for some cases without this problem. I will update those cases' name ASAP.

Architecture specifics should definitely not be needed to encode in multiple test suites. For architecture differences you can use the product definitions in https://openqa.suse.de/admin/products and looking at the current entries it looks like there are already many products used by migration with a similar purpose. Even if this is about an SCC code for the to-be-upgraded SLE version you can put it there, e.g. with a version suffix. Probably better would be anyway to not need this code but rely on registered and properly patched base images anyway. For any other settings that differ you can do that configuration also within os-autoinst-distri-opensuse itself. Also this is (or was?) used by migration tests, e.g. see line https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/products/sle/main.pm#L213 and below.

I hope it can explain your question

Actually I did not have any questions. Did I ask any and forgot them? :)

Actions #9

Updated by tinawang123 about 5 years ago

For the first problem, I have two sides to explain:
First:
with your example:

  • offline_sles12sp2_ltss_media_sdk_all_full:
    testsuite: offline_ltss_media_sdk_all_full <=========== Case name
    settings:
    HDDVERSION: 12-SP2

  • offline_sles12sp3_ltss_media_sdk_all_full:
    testsuite: offline_ltss_media_sdk_all_full <=========== Same case name
    settings:
    HDDVERSION: 12-SP3
    I set YAML template:
    scenarios:
    aarch64:
    sle-12-SP5-Migration-from-SLE12-SP5-to-SLE15-SPx-aarch64:

    • offline_pscc_base_def_full: settings: HDDVERSION: 12-SP2
    • offline_pscc_base_def_full: settings: HDDVERSION: 12-SP3 Got error message: There was a problem applying the changes:

    Job template name 'offline_pscc_base_def_full' is defined more than once. Use a unique name and specify 'testsuite' to re-use test suites in multiple scenarios.

Another side:
It is easy way for us to check our tests scenario. As do upgrade to sles12sp5. We have a lot of migration path like: from sles12sp2, sles12sp3, sles12sp4. We can use the cases' name to get the base system information easily. Otherwise, we need open the cases' setting, then check the base system. This base system information is very important for us and it is very useful for us if we get the information quickly from cases' name.

This method was defined at our project prepare staging. Now, we will prepare our upgrade to sles15sp2, if you still have some concern about this. Welcome to your suggestion.

Actions #10

Updated by tinawang123 about 5 years ago

For the second problem. It is a little complex to explain. I will try my best to explain more clear.
First example:
With the same Medium types as 'sle-12-SP5-Migration-from-SLE12-SPx-x86_64':

  • offline_sles12sp2_media_base_all_full_x86_64 Setting: SCC_REGCODE_LTSS=4Cxxxxxxxxxxx
  • offline_sles12sp3_media_base_all_full_x86_64 Setting: SCC_REGCODE_LTSS=36xxxxxxxxxxx So, we cannot put SCC_REGCODE_LTSS to flavor setting, as 12sp3 and 12sp2 has different regcode for ltss with the same medium type. So, why we need add arch at the cases' name, let me introduce the second example: Second example: Case have same name: offline_sles12sp2_ltss_media_base_all_full_x86_64 & offline_sles12sp2_ltss_media_base_all_full_s390x except arch. But for s390x media upgrade need add setting: ADDONURL sdk ADDONURL_SDK ftp://openqa.suse.de/SLE-12-SP5-SDK-POOL-s390x-Build0245-Media1/ For x86_64 media upgrade don't need those settings. This is just an example to explain, but it has other complex settings for other cases.

I hope you can understand those information. But we will keep to simplify our cases' name according the document.
If you are interesting to our cases, when our sles15sp2 matrix document ready, you can help to review :).

Actions #11

Updated by tinawang123 about 5 years ago

I have asked xiaojing, I have already known your point.
Generally speaking, if we have same cases at different arch and with some different settings and some same settings. We can put the cases' same setting at test suits, and put the cases' different settings at YAML. Then we can reduce the length of our cases' name.
We will think about this method at our sles15sp2 migration testing.
Thanks for your advice.

Actions #12

Updated by okurz about 5 years ago

tinawang123 wrote:

For the first problem, I have two sides to explain:
First:
with your example:

  • offline_sles12sp2_ltss_media_sdk_all_full:
    testsuite: offline_ltss_media_sdk_all_full <=========== Case name
    settings:
    HDDVERSION: 12-SP2

  • offline_sles12sp3_ltss_media_sdk_all_full:
    testsuite: offline_ltss_media_sdk_all_full <=========== Same case name
    settings:
    HDDVERSION: 12-SP3
    I set YAML template:
    scenarios:
    aarch64:
    sle-12-SP5-Migration-from-SLE12-SP5-to-SLE15-SPx-aarch64:

    • offline_pscc_base_def_full: settings: HDDVERSION: 12-SP2
    • offline_pscc_base_def_full: settings: HDDVERSION: 12-SP3 Got error message: There was a problem applying the changes:

    Job template name 'offline_pscc_base_def_full' is defined more than once. Use a unique name and specify 'testsuite' to re-use test suites in multiple scenarios.

Yes, the error message explains that pretty well. The header of each section defines the name which is either an existing testsuite or – when combined with testsuite: … you define the job template name. In my example there must be a test suite "offline_ltss_media_sdk_all_full" but the resulting scenarios in the job group have the name token "offline_sles12sp2_ltss_media_sdk_all_full" and "offline_sles12sp3_ltss_media_sdk_all_full" respectively included.

Actions #13

Updated by tinawang123 about 5 years ago

There are two problems:

  1. If the test settings split to two places: test suits and YAML setting. It is difficult for us to debug the tests. As we need check two places to confirm the settings are right.
  2. Add the cases' name to YAML, if input the wrong name which cases are not existed. YAML cannot return error message. It is difficult for user to check the name one by one.
Actions #14

Updated by okurz about 5 years ago

tinawang123 wrote:

There are two problems:

  1. If the test settings split to two places: test suits and YAML setting. It is difficult for us to debug the tests. As we need check two places to confirm the settings are right.

yes, I can understand that. There are actually many more places where settings come from. This is why I suggested initially to move much more into the test code. Now it is already splitted to medium, worker, machines, test code, test suites and now job templates. The hope is that the latter make reuse easier and eventually we can also improve on that general problem.

  1. Add the cases' name to YAML, if input the wrong name which cases are not existed. YAML cannot return error message. It is difficult for user to check the name one by one.

This is a valid point. This might be adressed in the future. But please keep in mind that we need to hurry up a bit to do at least the first step, that is to manage the job group schedule in YAML, as it was planned to decommission the old way and it is better if you explicitly do the transformation otherwise eventually the tools team needs to forcefully migrate, e.g. in the coming weeks, without using the benefits.

Actions #15

Updated by tinawang123 over 4 years ago

  • Status changed from New to Resolved
  • Assignee set to tinawang123
  • % Done changed from 0 to 100
Actions #16

Updated by okurz over 4 years ago

  • Status changed from Resolved to Workable

Sorry, but this is not done at all. Please don't just close a ticket as "Resolved" with no statement why you think it is resolved. https://openqa.suse.de/admin/test_suites easily shows already on the first page a test suite "010_offline_sles15sp1_media_basesys-srv-wsm-contm_def_full_390x" with the "s390x" suffix. So test suites should really be designed from the users point of view hence I think there should be no s390x suffix. And just looking for s390x in the name of test suites I find 63 (!) test suites that have a "s390x" suffix where the majority clearly are migration scenarios. There are less but still quite some for aarch64 and ppc64le and x86_64.

Actions #17

Updated by tinawang123 over 4 years ago

We only changed sle15sp2 test cases' name.
For the old cases, we don't change the name as it will lost the cases' log.
Please check cases' name at:
https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=164.1&groupid=265
https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=162.1&groupid=266
https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP2&build=164.1&groupid=267
There is not arch information now.

Actions #18

Updated by okurz over 4 years ago

Don't worry, there is no problem to update or delete all test suites as the existing test results are not affected by the update. QA SLE Migration needs to take responsibility for all migration test suites to ensure we have a long-term sustainable setup on osd (otherwise we will end up with an unmaintainable mess of test suites and job templates). . Also, there is no proper backup of existing older test results from osd anyway so old results can not be that important :)

Actions #19

Updated by tinawang123 over 3 years ago

  • Status changed from Workable to Resolved
  • Estimated time set to 30.00 h

Close this ticket as we don't have arch problem now.

Actions

Also available in: Atom PDF