coordination #37114
closed[functional][y][epic] Enable automated testing of "public cloud providers" (PCM) explicitly
100%
Description
Suggestions¶
- Add testsuite for google cloud
- Add testsuite for AWS
- Add testsuite for Microsoft Azure
- Ensure the other "allmodules" and "allmodules+allpatterns" tests do not enable PCM
Further details¶
As explained in comment#9 of bsc#1094879 the google-compute-engine should not be installed on regular systems but only on systems running in the google cloud.
This requires us to exclude this module from all tests, even the allmodules tests.
However, it makes sense to cover the patterns offered in SLE in our test scenarios locally so that we can detect "trivial" packaging/pattern problems early before going into real cloud.
bsc#1008988 and bsc#1008985 also describe the valid selections, that is "it's not supported to install all public cloud patterns in parallel, as this doesn't reflect any meaningful usecase.
You can install all patterns of one Cloud Service Provider, e.g. Microsoft Azure, but even that does not really make sense as the installed packages cover very different use cases. For instance one pattern is only useful on an instance running in the public cloud, etc."
Updated by okurz about 6 years ago
- Project changed from 46 to openQA Tests
- Subject changed from [functional][easy] Remove Google-Cloud-Module from allpatterns tests to [functional] Specific test covering pattern "Google-Cloud-Module" or remove? (was: Remove Google-Cloud-Module from allpatterns tests)
- Category set to Bugs in existing tests
Updated by okurz about 6 years ago
- Subject changed from [functional] Specific test covering pattern "Google-Cloud-Module" or remove? (was: Remove Google-Cloud-Module from allpatterns tests) to [kernel] Specific test covering pattern "Google-Cloud-Module" or remove? (was: Remove Google-Cloud-Module from allpatterns tests)
I understood QSK is fine to take it on their backlog.
As this is linked to currently failing tests we would appreciate if you could help us in a timely manner (within next 6 weeks?), otherwise I guess QSF would still do the first steps to ease our daily build review as well.
Updated by okurz about 6 years ago
Something like openqa_clone_job_osd --skip-chained-deps 1765305 SCC_ADDONS=pcm _GROUP="Test Development: SLE 12 SP4" TEST=allmodules+allpatterns+scc INSTALLONLY=
could be used to trigger an installation enabling the PCM and installing according patterns. Then later on the specific test plan needs to be adjusted, e.g. it does not make sense to trigger console/yast2_bootloader
on a cloud instance.
$ PERL5LIB= openqa-clone-job --skip-download --from https://openqa.suse.de --host https://openqa.suse.de --skip-chained-deps 1765305 SCC_ADDONS=pcm _GROUP="Test Development: SLE 12 SP4" TEST=pcm+allpatterns+scc INSTALLONLY= SCC_REGISTER=installation EXCLUDE_MODULES=yast2_bootloader
Created job #1765883: sle-12-SP4-Server-DVD-x86_64-Build0256-allmodules+allpatterns@64bit -> https://openqa.suse.de/t1765883
Updated by okurz about 6 years ago
- Subject changed from [kernel] Specific test covering pattern "Google-Cloud-Module" or remove? (was: Remove Google-Cloud-Module from allpatterns tests) to [kernel] Specific test covering the patterns from each of the three "public cloud provides" explicitly
- Priority changed from Normal to High
This is related to currently failing tests so bumping prio, e.g. see https://openqa.suse.de/tests/1766642#step/select_patterns_and_packages/74 . I guess asking Clemens to understand the test module "select_patterns_and_packages" and fixing this fast is too much so I will use #37531 instead "fix" the test failure in the existing test scenarios fast by not enabling the cloud module and cloud patterns in our existing "allmodules" test and then plan accordingly in this ticket to reintroduce proper test scenarios.
Updated by okurz about 6 years ago
- Related to action #37531: [sle][functional][y] PCM patterns cause package installation conflicts but these use cases are unsupported anyway added
Updated by sebchlad about 6 years ago
- Subject changed from [kernel] Specific test covering the patterns from each of the three "public cloud provides" explicitly to [functional] Specific test covering the patterns from each of the three "public cloud provides" explicitly
- Assignee set to okurz
@nsinger: agreed with the PO of the QSF:
"#3. Integration Testing
QA SLE Functional should perform regular integration testing for the Public
Cloud module in SLE12 SP4. These tests will be conducted as part of the existing
SLE functional tests based on local virtual machines without the use of any public
cloud instance to mainly ensure package, pattern and module consistency"
Please see:
https://gitlab.suse.de/sebchlad/Kernel_Network_Test_Plan/blob/master/SLE12_SP4_Level_Test_Plan_Public_Cloud
For the above reasons changing the assignment to QSF.
Updated by okurz about 6 years ago
- Subject changed from [functional] Specific test covering the patterns from each of the three "public cloud provides" explicitly to [functional][y] Specific test covering the patterns from each of the three "public cloud provides" explicitly
- Due date set to 2018-08-14
- Category changed from Bugs in existing tests to New test
- Assignee deleted (
okurz) - Target version set to Milestone 18
We will see what we can do within the next sprint. My hope is that with Clemens' (cfconrad) knowledge we could save a lot of time what scenarios in particular make sense.
With https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5281 I have for now removed the "PCM" module as it was causing invalid test failures.
Updated by cfconrad about 6 years ago
okurz wrote:
We will see what we can do within the next sprint. My hope is that with Clemens' (cfconrad) knowledge we could save a lot of time what scenarios in particular make sense.
Do I get the question right, that you asking what packages or even patterns can be installed at the same time and which not regardless the public cloud provider?
For example, to install aws-cli beside the google-cloud-sdk.
Updated by okurz about 6 years ago
Whatever makes most sense from the customer point of view. Our tests should not reflect what is technically possible but what is useful. So it could mean that we want to have a test that installs aws-cli in parallel to google-cloud-sdk, it could also mean that we need multiple scenarios for each cloud provider and only install a subset of patterns in each, e.g.
google
- pattern1
- pattern2+3
aws
- pattern4
- pattern5
- pattern6
azure
- pattern7+8+9
Updated by sebchlad about 6 years ago
In the meantime I asked Public cloud team to get some help and suggestions, so the AC1 could be met. I found them more than useful in such situation. They are really willing to help and they always provide me with professional support.
I kept all interested parties in that email thread.
I hope this model of cooperation is OK for Public Cloud team, as Marita expressed some worries that the whole communication flow might be cumbersome for them. I share this worry, so let us hope this won't be the case. But if this risk will materialize, we would need to adapt somewhat.
Basically I asked them if they could help us specifying "test scenarios for each public cloud provider" while "enabling PCM during installation and selecting the corresponding patterns in each scenario".
Updated by sebchlad about 6 years ago
"Would it be possible for you to help the team (QA SLE Functional) to have specific test scenarios for each public cloud provider in so-called "offline" mode please?"
answer provided by Robert Schweikert via email:
"Sure this is reasonably straight forward.
*-Services-Instance-Init
- Packages from these patterns should only be install tested. NO services from packages in these patterns should be enabled and the system be rebooted or the services started. It is possible to enable the services provided by these packages and check that the proper symlinks are created, but the "offline" test system should never be rebooted or the services explicitly started.
*-Services-Instance-Tools and *-Services-Tools
- All packages should be installable, packages are not expected to provide any services, and all code is expected to work on any system.
*-All
- A collection of all other patterns, thus conditions listed above apply."
Updated by okurz about 6 years ago
- Subject changed from [functional][y] Specific test covering the patterns from each of the three "public cloud provides" explicitly to [functional][y] Specific test covering the patterns from each of the three "public cloud providers" explicitly
Updated by riafarov about 6 years ago
- Subject changed from [functional][y] Specific test covering the patterns from each of the three "public cloud providers" explicitly to [functional][y][epic] Specific test covering the patterns from each of the three "public cloud providers" explicitly
- Description updated (diff)
Updated by riafarov about 6 years ago
- Subject changed from [functional][y][epic] Specific test covering the patterns from each of the three "public cloud providers" explicitly to [functional][y][epic] Enable automated testing of "public cloud providers" (PCM) explicitly
Updated by okurz about 6 years ago
- Due date changed from 2018-08-14 to 2018-08-28
due to changes in a related task
Updated by okurz about 6 years ago
- Due date changed from 2018-08-28 to 2018-10-23
due to changes in a related task
Updated by okurz about 6 years ago
- Target version changed from Milestone 18 to Milestone 20
Updated by riafarov almost 6 years ago
- Due date changed from 2018-10-23 to 2018-09-25
due to changes in a related task
Updated by okurz almost 6 years ago
- Target version changed from Milestone 20 to Milestone 22
Updated by okurz over 5 years ago
- Status changed from Workable to Resolved
- Assignee set to okurz
All subtickets resolved, reviewed the original request and I make the call to consider this ticket "Resolved".
Updated by szarate almost 4 years ago
- Tracker changed from action to coordination
Updated by szarate almost 4 years ago
See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html