QA » openQA Project » openQA Tests

You may just be able to do something like zypper ar -fG http://download.suse.de/ibs/SUSE:/SLE-15:/GA/standard/SUSE:SLE-15:GA.repo after installing the minimal OS. At least that would be enough for the LTP test runner.

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/3370
Unfortunately adding http://download.suse.de/ibs/SUSE:/SLE-15:/GA/standard/SUSE:SLE-15:GA.repo repo doesn't work if try to add it using url during installation. Whereas, with ftp we were able to add all possible modules using following directory: ftp://dist.suse.de/ibs/SUSE%3A/SLE-15%3A/GA/images/repo

Need to perform verification run, after that we can create separate test suite, as we need different set of settings and image can be used in downstream jobs.

@rpalethorpe: Regarding the test suite rename: IMHO we should find a common test suite name and rename everything semi-automatically, e.g. using dump_templates and load_templates or by renaming all references within the osd database. No need to change each single test suite name by hand of course. But I would also be fine to just keep the name 'sdk' for now and try to get as close as necessary to that with sle 15 and module combination as needed.

@rpalethorpe, @okurz: keeping same test suite name won't work as we need different set of settings, which are sle15 specific. In future when we have working SCC proxy we may use same test suite as before. We should keep in mind that this is temporary solution, so I believe it will be fine to try the image we create with new job and see what is missing/not working for ltp. And in 3-4 weeks when we may have SCC proxy, we review it and act accordingly.

If the variables are not specific to the test-product combination then we can probably avoid a rename using the Media/product variables. Or we could branch on the product name within the test as a temporary solution.

If SLE15 permanently requires an alternative definition for this test suite, then the existing test suite also needs to be renamed (to include the product version) so that we can programmatically create dependencies using variable substitution of the product version.

Yes, but to clarify you can also do what I have done with the LTP tests and use variable substitution like "START_AFTER_TEST=install_ltp+%DISTRI%+%FLAVOR%". Then we have multiple install_ltp jobs, but use the same run_ltp test definitions across all products and test groups (except for KOTD testing). Unfortunately I didn't include %VERSION% or whatever in the install_ltp test name.

Ideally minimal_base+sdk_create_hdd should follow the same naming pattern as install_ltp with the product version added, but changing it now is probably not worth the effort. As far as I can see, we can just ignore the settings SLE15 requires in other versions or define those settings on the media if they don't effect other tests.

rpalethorpe wrote:

Ideally minimal_base+sdk_create_hdd should follow the same naming pattern as install_ltp with the product version added, but changing it now is probably not worth the effort. As far as I can see, we can just ignore the settings SLE15 requires in other versions or define those settings on the media if they don't effect other tests.

I don't think that will work as well in practice as it certainly does in theory

it's worth considering that SLE 15 is fundamentally different from SLE 12

There is no SDK

There is no Server installation DVD

There is no Desktop installation DVD

So a test like install_ltp+SLE-12-Server-and-SDK makes sense in the context in SLE 12, but in SLE 15 you'd need to have something like

install_ltp+SLE-15-LeanOS-and-Server-Base-Module-and-Server-Apps-Module-and-Developer-Module

And it's just going to get worse as the full scope of the re-imagining of SLE 15 gets further developed and as packages move around all over the place.

I think Rodions new approach of START_AFTER_TEST=create_sle15_hdd,create_sle12_hdd avoids this problem by letting us set up sensible test scenarios to create the reference images we need for SLE 12 and SLE 15.

All of the logic for the differences is then put in the test, and it will be down to the scheduler to do it's thing, split the multi-value in START_AFTER_TEST, and ensure the correct START_AFTER_TEST's resolve for the correct media being scheduled.

This way we should be able to handle any variations in SLE 15 as they come about, but we should only have to change the LTP tests once to provide the additional value for START_AFTER_TEST

Another update regarding: START_AFTER_TEST=create_sle15_hdd,create_sle12_hdd
Actually in half of the child jobs we have ISO_1 setting with SDK iso, and this won't work. Additionally in PUBLISH_HDD we have %BUILD_SDK% variable, which is obviously not set at all in SLE15 and won't be. If first issue can be avoided if we use ftp for addons for sle12-sp3, and I'm in process of doing so. We have reverted changes only because SP2 repo was not on ftp and hence we had to revert the settings. Regarding second issue, we only apply dirty hacks. Problem with ISO_1 will remain even after we have working SCC proxy.

After talking to Rodion I see that the ISO variable is the issue and using START_AFTER_TEST=...,... looks good. It would be useful if HDD_1 also allowed multiple choices.

So a test like install_ltp+SLE-12-Server-and-SDK makes sense in the context in SLE 12, but in SLE 15 you'd need to have something like

install_ltp+SLE-15-LeanOS-and-Server-Base-Module-and-Server-Apps-Module-and-Developer-Module

Good point, however I see an easy solution in that we add a variable like %ADDONS% which is a unique code representing the installed modules, add-ons, packages, patterns, widgets, whatever e.g. ADDONS=+SBAD. Then the test would be create_hdd+%DISTRI%+%VERSION%+%FLAVOR%%ADDONS%. If it changes all the time, then we can use a catch all term like ALL which just installs all the modules except for the weird stuff like RT or SAP.

@rpalethorpe
We have considered this option as well. The main drawback is that we'll need to create quite some parent test suites for these combinations (actually instead of create_hdd+%DISTRI%+%VERSION%+%FLAVOR%%ADDONS% we thought about using VERSION only). We will still have this option though if something is bad with defining multiple test suites. Using ADDON variable or actually ADDONURL is what we are going to do anyway. But at the moment it's only workaround and in SLE15 none of these will be used, as we will select product during installation.

We have considered this option as well. The main drawback is that we'll need to create quite some parent test suites for these combinations (actually instead of create_hdd+%DISTRI%+%VERSION%+%FLAVOR%%ADDONS% we thought about using VERSION only). We will still have this option though if something is bad with defining multiple test suites. Using ADDON variable or actually ADDONURL is what we are going to do anyway. But at the moment it's only workaround and in SLE15 none of these will be used, as we will select product during installation.

I strongly advise you reconsider, I have actually done this for install_ltp and I only regret not going further and adding VERSION plus maybe another couple of variables. It is a lot easier to create some duplicate install jobs than change or duplicate all the child jobs. It provides a lot of freedom and future proofing at relatively little expense.

I strongly advise you reconsider, I have actually done this for install_ltp and I only regret not going further and adding VERSION plus maybe another couple of variables. It is a lot easier to create some duplicate install jobs than change or duplicate all the child jobs. It provides a lot of freedom and future proofing at relatively little expense.

Duplicating child jobs is definitely not an option. Reconfiguration of child jobs can be done with script which we will have to do anyway, as we rename parent job for sle 12 as well. I'll talk to Oliver, Richard and Coolo on Monday. Main point here is that we don't expect any changes for sle 12 branch, so it should work for all SP, as well sle 15 parent job should be same for all SPs as well. To introduce variable in START_AFTER_TEST value, we also need to reconfigure all child jobs, which is basically same efforts.

So as of now let's try with START_AFTER_TEST=create_hdd_minimal_base+sdk,create_sle15_hdd to see how it works and if we find any other issues before applying to the all tests. But we still keep 3 options in mind:
1) One mentioned above
2) Parent jobs per version and variable expansion in START_AFTER_TEST value
3) Black magic in main.pm

Here is example how to trigger test:
/usr/share/openqa/script/client isos post ARCH=x86_64 DISTRI=sle FLAVOR=Leanos-DVD VERSION=15 ISO=SLE-15-Leanos-DVD-x86_64-Build102.1-Media1.iso BUILD=102.1 BUILD_SDK=116.7 TEST=create_sle15_hdd REPO_0=SLE-15-Server-DVD-x86_64-Build102.1-Media2 --host openqa.suse.de

Media2? Shouldn't it be Media1 for binary packages?

Actually this variable is set when we trigger job. On ftp we don't have media1 for Server-DVD at all and I'm not really sure what is use case for it. For this testsuite we don't need correct value to be there, as it's not used.

We still seem to be missing curl. I am going to add it to the list of dependencies for LTP, but it should probably be selected during the LeanOS installation.

@rpalethorpe I'll add it to out workaround test module. It's not there because I thought you use console_setup test module which prepares env and explicitly installs curl.

If it is just LTP then it is not an issue as I added it during install_ltp.

I believe that your team is one of the few who pushes trying things with sle 15, so we don't have any other feedback. I've submitted PR already. And as it's only temporary workaround, we are fine to apply solutions which are not 100% correct ;)

okurz and me decided to add boolean variable DEV_IMAGE in the test suite to have single testsuite and set ADDONURL/ADDONURL_xxx accordingly in the main.pm (it's already mess, cannot fix it with having two test suites, and for main.pm we have git log and pr review).