Project

General

Profile

Actions

action #20984

closed

coordination #20580: [sle][functional][epic] sle 15 setup

[sle][functional][sle15] adapt minimal_base+sdk_create_hdd test for sle 15

Added by riafarov over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
New test
Start date:
2017-08-01
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

qcow image created by minimal_base+sdk_create_hdd test suite is used in many dependent jobs and is show stopper for many teams. Therefore, their automated tests won't be able to be executed properly.
Consequently this is one of the test suites which has high priority to be adjusted for SLE 15.

As of now SCC functionality is not fully available and installation flow will probably get changed. Therefore as a first coming, we will add repo manually, as we did for SDK in case of SLE 12.


Related issues 1 (0 open1 closed)

Related to openQA Tests - coordination #23454: [sle][functional][epic][sle15] Adapt former SDK tests to something equivalent for SLE15Resolvedokurz2017-08-182018-08-14

Actions
Actions #2

Updated by rpalethorpe over 6 years ago

You may just be able to do something like zypper ar -fG http://download.suse.de/ibs/SUSE:/SLE-15:/GA/standard/SUSE:SLE-15:GA.repo after installing the minimal OS. At least that would be enough for the LTP test runner.

Actions #3

Updated by riafarov over 6 years ago

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3370
Unfortunately adding http://download.suse.de/ibs/SUSE:/SLE-15:/GA/standard/SUSE:SLE-15:GA.repo repo doesn't work if try to add it using url during installation. Whereas, with ftp we were able to add all possible modules using following directory: ftp://dist.suse.de/ibs/SUSE%3A/SLE-15%3A/GA/images/repo

Need to perform verification run, after that we can create separate test suite, as we need different set of settings and image can be used in downstream jobs.

Actions #5

Updated by rpalethorpe over 6 years ago

  • Assignee changed from okurz to riafarov

riafarov wrote:

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3370
Unfortunately adding http://download.suse.de/ibs/SUSE:/SLE-15:/GA/standard/SUSE:SLE-15:GA.repo repo doesn't work if try to add it using url during installation. Whereas, with ftp we were able to add all possible modules using following directory: ftp://dist.suse.de/ibs/SUSE%3A/SLE-15%3A/GA/images/repo

Thanks, we can add the IBS repo (or even Tumbleweed) from install_ltp if there are any missing packages.

Need to perform verification run, after that we can create separate test suite, as we need different set of settings and image can be used in downstream jobs.

Please try not to change the test suite name otherwise I may have to change all the LTP test suites.

Actions #6

Updated by okurz over 6 years ago

@rpalethorpe: Regarding the test suite rename: IMHO we should find a common test suite name and rename everything semi-automatically, e.g. using dump_templates and load_templates or by renaming all references within the osd database. No need to change each single test suite name by hand of course. But I would also be fine to just keep the name 'sdk' for now and try to get as close as necessary to that with sle 15 and module combination as needed.

Actions #7

Updated by riafarov over 6 years ago

@rpalethorpe, @okurz: keeping same test suite name won't work as we need different set of settings, which are sle15 specific. In future when we have working SCC proxy we may use same test suite as before. We should keep in mind that this is temporary solution, so I believe it will be fine to try the image we create with new job and see what is missing/not working for ltp. And in 3-4 weeks when we may have SCC proxy, we review it and act accordingly.

Actions #8

Updated by rpalethorpe over 6 years ago

If the variables are not specific to the test-product combination then we can probably avoid a rename using the Media/product variables. Or we could branch on the product name within the test as a temporary solution.

If SLE15 permanently requires an alternative definition for this test suite, then the existing test suite also needs to be renamed (to include the product version) so that we can programmatically create dependencies using variable substitution of the product version.

Actions #9

Updated by okurz over 6 years ago

  • Category set to New test

Considering that we want to support SLE12SP4 along with SLE15 and also maintenance tests on potentially four architectures I would say we basically should look for the solution of defining test suites which are valid regardless of any product/version/architecture and do all the necessary specific handling within os-autoinst-distri-opensuse, e.g. - if I understood you right, as suggested - by branching based on product/version/arch "within the test" as IMHO this is most scalable and also serves accountability (one can look into git log when and why something changed, including the need for peer-review).

Actions #10

Updated by rpalethorpe over 6 years ago

Yes, but to clarify you can also do what I have done with the LTP tests and use variable substitution like "START_AFTER_TEST=install_ltp+%DISTRI%+%FLAVOR%". Then we have multiple install_ltp jobs, but use the same run_ltp test definitions across all products and test groups (except for KOTD testing). Unfortunately I didn't include %VERSION% or whatever in the install_ltp test name.

Ideally minimal_base+sdk_create_hdd should follow the same naming pattern as install_ltp with the product version added, but changing it now is probably not worth the effort. As far as I can see, we can just ignore the settings SLE15 requires in other versions or define those settings on the media if they don't effect other tests.

Actions #11

Updated by RBrownSUSE over 6 years ago

rpalethorpe wrote:

Ideally minimal_base+sdk_create_hdd should follow the same naming pattern as install_ltp with the product version added, but changing it now is probably not worth the effort. As far as I can see, we can just ignore the settings SLE15 requires in other versions or define those settings on the media if they don't effect other tests.

I don't think that will work as well in practice as it certainly does in theory

it's worth considering that SLE 15 is fundamentally different from SLE 12

There is no SDK

There is no Server installation DVD

There is no Desktop installation DVD

So a test like install_ltp+SLE-12-Server-and-SDK makes sense in the context in SLE 12, but in SLE 15 you'd need to have something like

install_ltp+SLE-15-LeanOS-and-Server-Base-Module-and-Server-Apps-Module-and-Developer-Module

And it's just going to get worse as the full scope of the re-imagining of SLE 15 gets further developed and as packages move around all over the place.

I think Rodions new approach of START_AFTER_TEST=create_sle15_hdd,create_sle12_hdd avoids this problem by letting us set up sensible test scenarios to create the reference images we need for SLE 12 and SLE 15.

All of the logic for the differences is then put in the test, and it will be down to the scheduler to do it's thing, split the multi-value in START_AFTER_TEST, and ensure the correct START_AFTER_TEST's resolve for the correct media being scheduled.

This way we should be able to handle any variations in SLE 15 as they come about, but we should only have to change the LTP tests once to provide the additional value for START_AFTER_TEST

Actions #12

Updated by riafarov over 6 years ago

Another update regarding: START_AFTER_TEST=create_sle15_hdd,create_sle12_hdd
Actually in half of the child jobs we have ISO_1 setting with SDK iso, and this won't work. Additionally in PUBLISH_HDD we have %BUILD_SDK% variable, which is obviously not set at all in SLE15 and won't be. If first issue can be avoided if we use ftp for addons for sle12-sp3, and I'm in process of doing so. We have reverted changes only because SP2 repo was not on ftp and hence we had to revert the settings. Regarding second issue, we only apply dirty hacks. Problem with ISO_1 will remain even after we have working SCC proxy.

Actions #13

Updated by rpalethorpe over 6 years ago

After talking to Rodion I see that the ISO variable is the issue and using START_AFTER_TEST=...,... looks good. It would be useful if HDD_1 also allowed multiple choices.

So a test like install_ltp+SLE-12-Server-and-SDK makes sense in the context in SLE 12, but in SLE 15 you'd need to have something like

install_ltp+SLE-15-LeanOS-and-Server-Base-Module-and-Server-Apps-Module-and-Developer-Module

Good point, however I see an easy solution in that we add a variable like %ADDONS% which is a unique code representing the installed modules, add-ons, packages, patterns, widgets, whatever e.g. ADDONS=+SBAD. Then the test would be create_hdd+%DISTRI%+%VERSION%+%FLAVOR%%ADDONS%. If it changes all the time, then we can use a catch all term like ALL which just installs all the modules except for the weird stuff like RT or SAP.

Actions #14

Updated by riafarov over 6 years ago

@rpalethorpe
We have considered this option as well. The main drawback is that we'll need to create quite some parent test suites for these combinations (actually instead of create_hdd+%DISTRI%+%VERSION%+%FLAVOR%%ADDONS% we thought about using VERSION only). We will still have this option though if something is bad with defining multiple test suites. Using ADDON variable or actually ADDONURL is what we are going to do anyway. But at the moment it's only workaround and in SLE15 none of these will be used, as we will select product during installation.

Actions #15

Updated by riafarov over 6 years ago

  • Status changed from In Progress to Feedback

So job finally works after introducing hack here and there: https://openqa.suse.de/tests/1100501#
I've renamed job to create_sle15_hdd as per Richard's comment.
I've tried to run stress tests, they would work, but there is no QA repo for SLE15 yet. So it won't work as of now: https://openqa.suse.de/tests/1100502#step/acceptance_fs_stress/16

Actions #16

Updated by rpalethorpe over 6 years ago

We have considered this option as well. The main drawback is that we'll need to create quite some parent test suites for these combinations (actually instead of create_hdd+%DISTRI%+%VERSION%+%FLAVOR%%ADDONS% we thought about using VERSION only). We will still have this option though if something is bad with defining multiple test suites. Using ADDON variable or actually ADDONURL is what we are going to do anyway. But at the moment it's only workaround and in SLE15 none of these will be used, as we will select product during installation.

I strongly advise you reconsider, I have actually done this for install_ltp and I only regret not going further and adding VERSION plus maybe another couple of variables. It is a lot easier to create some duplicate install jobs than change or duplicate all the child jobs. It provides a lot of freedom and future proofing at relatively little expense.

Actions #17

Updated by riafarov over 6 years ago

I strongly advise you reconsider, I have actually done this for install_ltp and I only regret not going further and adding VERSION plus maybe another couple of variables. It is a lot easier to create some duplicate install jobs than change or duplicate all the child jobs. It provides a lot of freedom and future proofing at relatively little expense.

Duplicating child jobs is definitely not an option. Reconfiguration of child jobs can be done with script which we will have to do anyway, as we rename parent job for sle 12 as well. I'll talk to Oliver, Richard and Coolo on Monday. Main point here is that we don't expect any changes for sle 12 branch, so it should work for all SP, as well sle 15 parent job should be same for all SPs as well. To introduce variable in START_AFTER_TEST value, we also need to reconfigure all child jobs, which is basically same efforts.

So as of now let's try with START_AFTER_TEST=create_hdd_minimal_base+sdk,create_sle15_hdd to see how it works and if we find any other issues before applying to the all tests. But we still keep 3 options in mind:
1) One mentioned above
2) Parent jobs per version and variable expansion in START_AFTER_TEST value
3) Black magic in main.pm

Actions #18

Updated by riafarov over 6 years ago

Here is example how to trigger test:
/usr/share/openqa/script/client isos post ARCH=x86_64 DISTRI=sle FLAVOR=Leanos-DVD VERSION=15 ISO=SLE-15-Leanos-DVD-x86_64-Build102.1-Media1.iso BUILD=102.1 BUILD_SDK=116.7 TEST=create_sle15_hdd REPO_0=SLE-15-Server-DVD-x86_64-Build102.1-Media2 --host openqa.suse.de

Actions #19

Updated by okurz over 6 years ago

Media2? Shouldn't it be Media1 for binary packages?

Actions #20

Updated by riafarov over 6 years ago

Actually this variable is set when we trigger job. On ftp we don't have media1 for Server-DVD at all and I'm not really sure what is use case for it. For this testsuite we don't need correct value to be there, as it's not used.

Actions #21

Updated by rpalethorpe over 6 years ago

We still seem to be missing curl. I am going to add it to the list of dependencies for LTP, but it should probably be selected during the LeanOS installation.

Actions #22

Updated by riafarov over 6 years ago

@rpalethorpe I'll add it to out workaround test module. It's not there because I thought you use console_setup test module which prepares env and explicitly installs curl.

Actions #23

Updated by rpalethorpe over 6 years ago

If it is just LTP then it is not an issue as I added it during install_ltp.

Actions #24

Updated by riafarov over 6 years ago

I believe that your team is one of the few who pushes trying things with sle 15, so we don't have any other feedback. I've submitted PR already. And as it's only temporary workaround, we are fine to apply solutions which are not 100% correct ;)

Actions #25

Updated by okurz over 6 years ago

  • Target version set to Milestone 10
Actions #26

Updated by SLindoMansilla over 6 years ago

  • Related to coordination #23454: [sle][functional][epic][sle15] Adapt former SDK tests to something equivalent for SLE15 added
Actions #27

Updated by riafarov over 6 years ago

okurz and me decided to add boolean variable DEV_IMAGE in the test suite to have single testsuite and set ADDONURL/ADDONURL_xxx accordingly in the main.pm (it's already mess, cannot fix it with having two test suites, and for main.pm we have git log and pr review).

Actions #28

Updated by riafarov over 6 years ago

  • Status changed from Feedback to Resolved

Removed create_hdd_sle15, create_hdd_sle15_textmode and create_hdd_sle15_gnome SLE 15 specific test suite.
ALL_MODULES setting was added to original test suites, which change behavior only in sle 15. Resolve this ticket and solve other related issues in new tickets.

Actions

Also available in: Atom PDF