action #17042

action #15108: [sle][functional][u][epic] Modules (Installation + migration)

[sles][functional][modules] Modules installation

Added by maritawerner about 3 years ago. Updated almost 3 years ago.

Status:ResolvedStart date:14/02/2017
Priority:HighDue date:
Assignee:dgutu% Done:

0%

Category:New test
Target version:openQA Project - Milestone 8
Difficulty:
Duration:

Description

goal

Make sure that each module installs correctly.

acceptance criteria

  • AC1: One module is tested on every SLES build using proxy-SCC
  • AC2: Multiple modules are tested

tasks

  • set SCC_ADDONS=<module_name1>[,<module_name2>] accordingly (SCC regurl should only register SLES, not point to any module)
  • after the module(s) are registered and enabled packages should be installed from the modules, e.g. by selecting the appropriate pattern(s)

further details

As SLE service packs and therefore also builds within service packs are meant to be ABI compatible modules can be simply installed over SCC while registering the latest SLES build. For all modules that are already on SCC we just install them over SCC (That's it. No need to sync anything anywhere).
Enabling modules does not install anything from it automatically. patterns or packages need to be selected, e.g. after enabling "hpcm" then "zypper search hpc" should return something. Also the repository can be explicitly selected with "zypper lr" and then all packages and patterns from that repo can be listed with "zypper search -r ".
So far module installation was never done besides textmode during online migration but these are actually "post-validation" tests as described in #15800.

canvas.png - screenshot of openQA test run showing how the repository URLs look like after rewriting (40.2 KB) okurz, 06/05/2017 07:33 am

openQA_rewritten_sle_module_paths.png (15.3 KB) okurz, 06/05/2017 09:08 am

3100
3102

History

#1 Updated by maritawerner about 3 years ago

  • Related to action #15108: [sle][functional][u][epic] Modules (Installation + migration) added

#2 Updated by maritawerner about 3 years ago

okurz: I think that that should be covered by the sle functional team? Do you agree? Any feedback is welcome.

#3 Updated by okurz about 3 years ago

  • Related to deleted (action #15108: [sle][functional][u][epic] Modules (Installation + migration))

#4 Updated by okurz about 3 years ago

  • Parent task set to #15108

#5 Updated by okurz about 3 years ago

  • Subject changed from [sles][functional] Modules Installation to [sles][functional] Modules Installation
  • Category set to New test

#6 Updated by maritawerner about 3 years ago

Modules for SP3 are not enabled in SCC yet. I wait for Anja/Frederic to get back to us.

#7 Updated by dgutu almost 3 years ago

  • Assignee set to dgutu

#8 Updated by dgutu almost 3 years ago

Modules are available while registering the system on SCC-Proxy.
Only Certifications module is missing. Will give it a try on normal SCC.

#9 Updated by dgutu almost 3 years ago

  1. Certification modules is not available for our scc key.
  2. Legacy Module is failing to download license, emailed Arteom about this.

#10 Updated by dgutu almost 3 years ago

Legacy and Web Scripting needs license to be synced, rsync should be adapted.
Other modules like, Public Cloud, ToolChain, Container, Advances System Management Modules are fine.
@Marita, what about Certification Module, is it on PRD as available one?

#11 Updated by maritawerner almost 3 years ago

Andreas Stieger, the security PJM, confirmed to me that atm there is no certifications module for SP2 and for SP3. They will develop a new module but he expects that to be available in July. So we can skip all installation/Migration scenarios with the certifications module.

#12 Updated by dgutu almost 3 years ago

Thx for the info Marita.
As per today this one is running in test development group with 4 modules from 6.
https://openqa.suse.de/tests/808525
I'm working on rsync.pl to enable the remaining 2 modules to this testmodule.

#13 Updated by dgutu almost 3 years ago

The rsync.pl syncs everything related to the latest SP and staging but modules are related to GA,
because of that I'm still working on it.

#14 Updated by maritawerner almost 3 years ago

Dumitru, could you please add more details into the ticket description or split the ticket. Please check PRD section 1.4.2.1.

#15 Updated by asmorodskyi almost 3 years ago

  • Related to action #9464: [sle][functional][u] Modules - Advanced Systems Management added

#16 Updated by asmorodskyi almost 3 years ago

  • Related to action #9466: [sle][functional][epic][medium][u]Modules - Containers: Test the SLE module for containers added

#17 Updated by asmorodskyi almost 3 years ago

  • Related to action #9470: [sle][functional][u][sle12][medium][epic]Modules - Web and Scripting added

#18 Updated by asmorodskyi almost 3 years ago

  • Related to deleted (action #9470: [sle][functional][u][sle12][medium][epic]Modules - Web and Scripting)

#19 Updated by asmorodskyi almost 3 years ago

  • Related to deleted (action #9466: [sle][functional][epic][medium][u]Modules - Containers: Test the SLE module for containers)

#20 Updated by asmorodskyi almost 3 years ago

  • Related to deleted (action #9464: [sle][functional][u] Modules - Advanced Systems Management)

#21 Updated by okurz almost 3 years ago

  • Target version set to Milestone 7

#22 Updated by dgutu almost 3 years ago

  • Subject changed from [sles][functional] Modules Installation to [sles][functional][modules] Modules installation

#23 Updated by okurz almost 3 years ago

  • Description updated (diff)

#24 Updated by okurz almost 3 years ago

What I did as an experiment:

so that works.

#25 Updated by dgutu almost 3 years ago

Thank you for the try but I found some caveats.
1. When module is selected by using registration you need to get this module as a product on
the installation settings page and respective patterns.
2. When I tried to use regurl and include the once you have used I got the module in the product list
but the installer gives a warning that the packages can't be installed.
Continue to work on that.

#26 Updated by dgutu almost 3 years ago

I found that there are 2 types sources for modules.

http://dist.suse.de/ibs/SUSE:/SLE-12:/GA/images/repo/SLE-12-Module*
Offering:
- correct structure of the folder and files so we can sync this to ftp:/openqa.suse.de

Doesn't offer:
- It contains only 4 modules from 6 (asmm, lgm, pcm and wsm - present), (tcm and contm - missing)
- very limited number of packages and old version

http://dist.suse.de/ibs/SUSE/Updates/
Offering:
- Latest and updated packages for all modules

Doesn't offer:
- Missing license files for lgm and wsm modules
- Folder and file organized like repo and not like ISO

I'm confused now...

#27 Updated by maritawerner almost 3 years ago

Answer from Stefan Behlert:

The first link is the pool channel added to the ibs for GA. At that time only 4 modules existed. The newer modules must have other channels that were added for SP1/SP2.
The second channel is the update channel for all modules were you can find all updates for all modules. Licences are not in there because they are in the pool channel.

If you want to know more here maybe ask a colleague from QAM. Mantienance are the experts around channels, etc.

#28 Updated by okurz almost 3 years ago

  • Description updated (diff)
  • Status changed from New to In Progress

I guess we got confused in before and the approach is way easier than thought. Updated description accordingly.

$ openqa_clone_job_osd 916988 GROUP=0 SCC_ADDONS=asmm TEST=gnome+sdk+proxySCC+asmm_okurz

-> sle-12-SP3-Server-DVD-x86_64-Build0367-gnome+sdk+proxy_SCC+asmm_okurz

Just in case if we would need to test modules which are not yet released over SCC or when we would need to test a specific version of modules then we would need to sync them to osd and "fake" the repositories. This has been mentioned in #12908#note-4 and it seems coolo confused me or meant something different.

#29 Updated by okurz almost 3 years ago

aha, and https://openqa.suse.de/tests/919019#step/consoletest_setup/32 failed now so … proxy-SCC issue?

#30 Updated by okurz almost 3 years ago

https://github.com/SUSE/scc-qa-proxy/issues/14 is the feature request. I asked the SCC team if someone can take a look.

#31 Updated by okurz almost 3 years ago

In case this is not feasible to do within proxy-SCC then we need to follow the approach of syncing everything to osd for testing but I am wondering about the "pool/product" and "update" part. https://openqa.suse.de/tests/919019#step/consoletest_setup/32 shows that when specifying only SLES in the regurl both "…-Pool" and "…-Updates" now point to the same directory on osd.

For the repos there are seemingly multiple versions:

I want to crosscheck how it looks like when we specify the module in the regurl as well but osd is currently down.

#32 Updated by okurz almost 3 years ago

$ openqa_clone_job_osd 916988 _GROUP=0 SCC_ADDONS=asmm TEST=gnome+sdk+proxy_SCC+asmm_okurz SCC_URL=http://Server-0367.Module-Adv-Systems-Management-20170506.proxy.scc.suse.de
Created job #920359: sle-12-SP3-Server-DVD-x86_64-Build0367-gnome+sdk+proxy_SCC_without_addon@64bit

-> https://openqa.suse.de/t920359

From the following screenshot
the screenshot
we can see how the repositories look like after rewriting from proxy-SCC when also the module is specified in SCC_URL: http://openqa.suse.de/assets/repo/SLE-12-Module-Adv-Systems-Management-POOL-x86_64-Build20170506-Media1/ but this is the same URL for both the Pool and Updates repository. I see this as another problem because when both repository URL yield the same path we can not easily sync both Pool and Updates but need to merge them. So even more messy.

As an experiment nevertheless I synced both Pool and Updates into the target path on osd:

for path in SUSE:/SLE-12:/GA/images/repo/SLE-12-Module-Adv-Systems-Management-POOL-x86_64-Build0096-Media1/ SUSE/Updates/SLE-Module-Adv-Systems-Management/12/x86_64/update/ ; do rsync -aHP rsync://dist.suse.de/ibs/$path /var/lib/openqa/share/factory/repo/SLE-12-Module-Adv-Systems-Management-POOL-x86_64-Build20170506-Media1/ ; done

yielding ftp://openqa.suse.de/SLE-12-Module-Adv-Systems-Management-POOL-x86_64-Build20170506-Media1/ , a messy merged directory. If this is the approach to go then for modules which where not existant at the time of GA we might need to find a corresponding pool directory first from a later SP or a products directory elsewhere. Retriggered above's openQA job

-> https://openqa.suse.de/tests/920360#live

to see if the repository can be at least refreshed properly.

That failed with

$ curl -s https://openqa.suse.de/tests/920360/file/start_install-y2logs.tar.bz2 | tar -xjf - YaST2/badlist -O
2 packages failed
nothing provides product_flavor(sle-module-adv-systems-management) needed by product:sle-module-adv-systems-management-12-0.x86_64
nothing provides product_flavor(sle-module-adv-systems-management) needed by product:sle-module-adv-systems-management-12-0.x86_64

so apparently the pool directory is incomplete and does not provide the product properly anymore. Do we really need to reverse engineer the update channels management infrastructure here?

#34 Updated by okurz almost 3 years ago

  • Status changed from In Progress to Feedback
  • Assignee changed from dgutu to okurz

There is a pull request open for scc-qa-proxy to only rewrite specified products. I think that is the right approach and we should wait for the PR to be be active on proxy-SCC before continuing.

#35 Updated by okurz almost 3 years ago

The PR to proxy-SCC was merged. Let's see if it already has an effect -> https://openqa.suse.de/tests/925867#live

https://openqa.suse.de/tests/925867#step/zypper_lr/6 shows that the module repositories are not rewritten and the ones from SCC are used which is what we wanted here.

Next step should be wait for https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/2848 then install packages from repo.

#36 Updated by dgutu almost 3 years ago

Modules are installalable. I have added a testsuite named gnome+allmodule_ProxySCC first for x86_64,ppc64,s390x and separate one for aarch64.
The last arch only modules are available.

#37 Updated by dgutu almost 3 years ago

  • Status changed from Feedback to In Progress
  • Assignee changed from okurz to dgutu

#38 Updated by okurz almost 3 years ago

  • Target version changed from Milestone 7 to Milestone 8

please update with current status and plan

#39 Updated by dgutu almost 3 years ago

  • Status changed from In Progress to Resolved

Since Build0374 all modules are enabled through proxy-SCC.
Sincer Build0398 I've added to be installed as a pattern, only this 2 patterns actually are available:
From ASMM - Machinery and Puppet. CFEngine has 2 entries with the same name so I've created a bug for that
FROM PCM - Google Cloud Platform and OpenStack

There are separate progress ticker which indicates what modules needs to be properly tested, not only installation of packages.

Also available in: Atom PDF