Project

General

Profile

Actions

coordination #37114

closed

[functional][y][epic] Enable automated testing of "public cloud providers" (PCM) explicitly

Added by nicksinger almost 6 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
New test
Target version:
SUSE QA - Milestone 22
Start date:
2018-06-12
Due date:
2018-09-25
% Done:

100%

Estimated time:
(Total: 29.00 h)
Difficulty:

Description

Suggestions

  • Add testsuite for google cloud
  • Add testsuite for AWS
  • Add testsuite for Microsoft Azure
  • Ensure the other "allmodules" and "allmodules+allpatterns" tests do not enable PCM

Further details

As explained in comment#9 of bsc#1094879 the google-compute-engine should not be installed on regular systems but only on systems running in the google cloud.
This requires us to exclude this module from all tests, even the allmodules tests.

However, it makes sense to cover the patterns offered in SLE in our test scenarios locally so that we can detect "trivial" packaging/pattern problems early before going into real cloud.

bsc#1008988 and bsc#1008985 also describe the valid selections, that is "it's not supported to install all public cloud patterns in parallel, as this doesn't reflect any meaningful usecase.
You can install all patterns of one Cloud Service Provider, e.g. Microsoft Azure, but even that does not really make sense as the installed packages cover very different use cases. For instance one pattern is only useful on an instance running in the public cloud, etc."


Subtasks 4 (0 open4 closed)

action #38867: [functional][y] Specific test covering the patterns from each of the three "public cloud providers" explicitlyResolvedriafarov2018-06-122018-08-14

Actions
action #38870: [functional][y] Specific test covering basic functionality of the packages in Microsoft Azure pattern of PCMResolvedriafarov2018-06-122018-09-11

Actions
action #39995: [functional][y] Specific test covering basic functionality of the packages in Google Cloud pattern of PCMRejectedokurz2018-06-12

Actions
action #39992: [functional][y] Specific test covering basic functionality of the packages in Amazon Web Services pattern of PCMResolvedJERiveraMoya2018-06-122018-09-25

Actions

Related issues 1 (0 open1 closed)

Related to openQA Tests - action #37531: [sle][functional][y] PCM patterns cause package installation conflicts but these use cases are unsupported anywayResolvedokurz2018-06-20

Actions
Actions #1

Updated by okurz almost 6 years ago

  • Project changed from 46 to openQA Tests
  • Subject changed from [functional][easy] Remove Google-Cloud-Module from allpatterns tests to [functional] Specific test covering pattern "Google-Cloud-Module" or remove? (was: Remove Google-Cloud-Module from allpatterns tests)
  • Category set to Bugs in existing tests
Actions #3

Updated by okurz almost 6 years ago

  • Subject changed from [functional] Specific test covering pattern "Google-Cloud-Module" or remove? (was: Remove Google-Cloud-Module from allpatterns tests) to [kernel] Specific test covering pattern "Google-Cloud-Module" or remove? (was: Remove Google-Cloud-Module from allpatterns tests)

I understood QSK is fine to take it on their backlog.

As this is linked to currently failing tests we would appreciate if you could help us in a timely manner (within next 6 weeks?), otherwise I guess QSF would still do the first steps to ease our daily build review as well.

Actions #4

Updated by okurz almost 6 years ago

Something like openqa_clone_job_osd --skip-chained-deps 1765305 SCC_ADDONS=pcm _GROUP="Test Development: SLE 12 SP4" TEST=allmodules+allpatterns+scc INSTALLONLY= could be used to trigger an installation enabling the PCM and installing according patterns. Then later on the specific test plan needs to be adjusted, e.g. it does not make sense to trigger console/yast2_bootloader on a cloud instance.

$ PERL5LIB= openqa-clone-job --skip-download --from https://openqa.suse.de --host https://openqa.suse.de --skip-chained-deps 1765305 SCC_ADDONS=pcm _GROUP="Test Development: SLE 12 SP4" TEST=pcm+allpatterns+scc INSTALLONLY= SCC_REGISTER=installation EXCLUDE_MODULES=yast2_bootloader
Created job #1765883: sle-12-SP4-Server-DVD-x86_64-Build0256-allmodules+allpatterns@64bit -> https://openqa.suse.de/t1765883

https://openqa.suse.de/t1765883

Actions #5

Updated by okurz almost 6 years ago

  • Description updated (diff)
Actions #6

Updated by okurz almost 6 years ago

  • Description updated (diff)
Actions #7

Updated by okurz almost 6 years ago

  • Description updated (diff)
Actions #8

Updated by okurz almost 6 years ago

  • Subject changed from [kernel] Specific test covering pattern "Google-Cloud-Module" or remove? (was: Remove Google-Cloud-Module from allpatterns tests) to [kernel] Specific test covering the patterns from each of the three "public cloud provides" explicitly
  • Priority changed from Normal to High

This is related to currently failing tests so bumping prio, e.g. see https://openqa.suse.de/tests/1766642#step/select_patterns_and_packages/74 . I guess asking Clemens to understand the test module "select_patterns_and_packages" and fixing this fast is too much so I will use #37531 instead "fix" the test failure in the existing test scenarios fast by not enabling the cloud module and cloud patterns in our existing "allmodules" test and then plan accordingly in this ticket to reintroduce proper test scenarios.

Actions #9

Updated by okurz almost 6 years ago

  • Related to action #37531: [sle][functional][y] PCM patterns cause package installation conflicts but these use cases are unsupported anyway added
Actions #10

Updated by sebchlad almost 6 years ago

  • Subject changed from [kernel] Specific test covering the patterns from each of the three "public cloud provides" explicitly to [functional] Specific test covering the patterns from each of the three "public cloud provides" explicitly
  • Assignee set to okurz

@nsinger: agreed with the PO of the QSF:
"#3. Integration Testing
QA SLE Functional should perform regular integration testing for the Public
Cloud module in SLE12 SP4. These tests will be conducted as part of the existing
SLE functional tests based on local virtual machines without the use of any public
cloud instance to mainly ensure package, pattern and module consistency"

Please see:
https://gitlab.suse.de/sebchlad/Kernel_Network_Test_Plan/blob/master/SLE12_SP4_Level_Test_Plan_Public_Cloud

For the above reasons changing the assignment to QSF.

Actions #11

Updated by okurz almost 6 years ago

  • Subject changed from [functional] Specific test covering the patterns from each of the three "public cloud provides" explicitly to [functional][y] Specific test covering the patterns from each of the three "public cloud provides" explicitly
  • Due date set to 2018-08-14
  • Category changed from Bugs in existing tests to New test
  • Assignee deleted (okurz)
  • Target version set to Milestone 18

We will see what we can do within the next sprint. My hope is that with Clemens' (cfconrad) knowledge we could save a lot of time what scenarios in particular make sense.

With https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5281 I have for now removed the "PCM" module as it was causing invalid test failures.

Actions #12

Updated by cfconrad almost 6 years ago

okurz wrote:

We will see what we can do within the next sprint. My hope is that with Clemens' (cfconrad) knowledge we could save a lot of time what scenarios in particular make sense.

Do I get the question right, that you asking what packages or even patterns can be installed at the same time and which not regardless the public cloud provider?
For example, to install aws-cli beside the google-cloud-sdk.

Actions #13

Updated by okurz almost 6 years ago

Whatever makes most sense from the customer point of view. Our tests should not reflect what is technically possible but what is useful. So it could mean that we want to have a test that installs aws-cli in parallel to google-cloud-sdk, it could also mean that we need multiple scenarios for each cloud provider and only install a subset of patterns in each, e.g.

  • google

    • pattern1
    • pattern2+3
  • aws

    • pattern4
    • pattern5
    • pattern6
  • azure

    • pattern7+8+9
Actions #14

Updated by sebchlad almost 6 years ago

In the meantime I asked Public cloud team to get some help and suggestions, so the AC1 could be met. I found them more than useful in such situation. They are really willing to help and they always provide me with professional support.

I kept all interested parties in that email thread.

I hope this model of cooperation is OK for Public Cloud team, as Marita expressed some worries that the whole communication flow might be cumbersome for them. I share this worry, so let us hope this won't be the case. But if this risk will materialize, we would need to adapt somewhat.

Basically I asked them if they could help us specifying "test scenarios for each public cloud provider" while "enabling PCM during installation and selecting the corresponding patterns in each scenario".

Actions #15

Updated by sebchlad almost 6 years ago

"Would it be possible for you to help the team (QA SLE Functional) to have specific test scenarios for each public cloud provider in so-called "offline" mode please?"

answer provided by Robert Schweikert via email:

"Sure this is reasonably straight forward.

*-Services-Instance-Init

  • Packages from these patterns should only be install tested. NO services from packages in these patterns should be enabled and the system be rebooted or the services started. It is possible to enable the services provided by these packages and check that the proper symlinks are created, but the "offline" test system should never be rebooted or the services explicitly started.

*-Services-Instance-Tools and *-Services-Tools

  • All packages should be installable, packages are not expected to provide any services, and all code is expected to work on any system.

*-All

  • A collection of all other patterns, thus conditions listed above apply."
Actions #17

Updated by okurz almost 6 years ago

  • Subject changed from [functional][y] Specific test covering the patterns from each of the three "public cloud provides" explicitly to [functional][y] Specific test covering the patterns from each of the three "public cloud providers" explicitly
Actions #18

Updated by riafarov over 5 years ago

  • Subject changed from [functional][y] Specific test covering the patterns from each of the three "public cloud providers" explicitly to [functional][y][epic] Specific test covering the patterns from each of the three "public cloud providers" explicitly
  • Description updated (diff)
Actions #19

Updated by riafarov over 5 years ago

  • Subject changed from [functional][y][epic] Specific test covering the patterns from each of the three "public cloud providers" explicitly to [functional][y][epic] Enable automated testing of "public cloud providers" (PCM) explicitly
Actions #20

Updated by okurz over 5 years ago

  • Due date changed from 2018-08-14 to 2018-08-28

due to changes in a related task

Actions #21

Updated by okurz over 5 years ago

  • Status changed from New to Workable
Actions #22

Updated by okurz over 5 years ago

  • Due date changed from 2018-08-28 to 2018-10-23

due to changes in a related task

Actions #23

Updated by okurz over 5 years ago

  • Target version changed from Milestone 18 to Milestone 20
Actions #24

Updated by riafarov over 5 years ago

  • Due date changed from 2018-10-23 to 2018-09-25

due to changes in a related task

Actions #25

Updated by okurz over 5 years ago

  • Target version changed from Milestone 20 to Milestone 22
Actions #26

Updated by okurz over 5 years ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz

All subtickets resolved, reviewed the original request and I make the call to consider this ticket "Resolved".

Actions #27

Updated by szarate over 3 years ago

  • Tracker changed from action to coordination
Actions

Also available in: Atom PDF