QA » openQA Project » openQA Tests

I don't hold any huge objection to this, but given ALP is a derivative of both Tumbleweed and MicroOS, the only question is how do we expect ALP to keep its test coverage aligned with its upstream if we do this?

Both MicroOS and Tumbleweed use main.pms, and whatever coverage is added to ALP would ideally exist in one of those two domains before

Yes, I agree with @RBrownSUSE here.
I would actually go even further and create a common main_microos.pm (or whatever name) for ALP, MicroOS, Leap Micro and SLE Micro, since most of the tests/schedules will be shared.

Carrying over to next sprint

Both MicroOS and Tumbleweed use main.pms, and whatever coverage is added to ALP would ideally exist in one of those two domains before
the only question is how do we expect ALP to keep its test coverage aligned with its upstream if we do this?

Hmmm, we need to replan this, to be able to guarantee that this will stay true, without overloading O3.

I would actually go even further and create a common main_microos.pm (or whatever name) for ALP, MicroOS, Leap Micro and SLE Micro, since most of the tests/schedules will be shared.

@Jose, you mean that main_microos.pm is a single file, that exists for all of the 3 distributions?

Gonna move it out of the sprint, for now we're using YAML schedule for new testsuites, all of those test modules enabled run already on TW.

szarate wrote:

Hmmm, we need to replan this, to be able to guarantee that this will stay true, without overloading O3.

In case you are concerned regarding hardware ressources rest assured that we have enough headroom. And in case we do run into limits I am convinced we will be able to improve.

Update on this.

I created this file some months ago:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/main_micro_alp.pm

The reason is to re-use the same test modules to boot the images (all of them boot similarly, but using different needles), to run common tests and specific transactional tests.
The idea is to use this main scheduler for the products that are sharing common coverage: MicroOS, SLE Micro, Leap Micro and now ALP.
Of course, there will be some if-else conditions for those products, but at least it's easier to code them in perl than in yaml, and also it's easier to check the coverage difference for each product in a single file.

jlausuch wrote:

Update on this.

I created this file some months ago:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/main_micro_alp.pm

The reason is to re-use the same test modules to boot the images (all of them boot similarly, but using different needles), to run common tests and specific transactional tests.
The idea is to use this main scheduler for the products that are sharing common coverage: MicroOS, SLE Micro, Leap Micro and now ALP.
Of course, there will be some if-else conditions for those products, but at least it's easier to code them in perl than in yaml, and also it's easier to check the coverage difference for each product in a single file.

That is how main.pm started, at the beginning the condition are simple and then goes exponentially wrong.
Even the ones that now exist are not easy to follow without diving deeper, I can see few comments of why those conditions are there.
I actually would like to have a programmatic solution instead of yaml, but we would need to think something to avoid to use conditions. It is more about exclude things with precise comments why (there is a feature in openqa to exclude test module), but have some default list of test module to run. But those exclusion might need to be done per squad in a different component, grouping all in single place by code, not in this file all together. We can talk about this some day, it depends how each squad is using it.

JERiveraMoya wrote:

That is how main.pm started, at the beginning the condition are simple and then goes exponentially wrong.
Even the ones that now exist are not easy to follow without diving deeper, I can see few comments of why those conditions are there.
I actually would like to have a programmatic solution instead of yaml, but we would need to think something to avoid to use conditions. It is more about exclude things with precise comments why (there is a feature in openqa to exclude test module), but have some default list of test module to run. But those exclusion might need to be done per squad in a different component, grouping all in single place by code, not in this file all together. We can talk about this some day, it depends how each squad is using it.

I understand the pain about main_common.pm cause I've been there too.

Let me tell you why we chose this solution.
When we added SLE Micro, we relied on yaml schedules. Then, Leap Micro came into the picture, and we reused some yamls, but of course with some conditions and hacks. We ended up in multiple yamls which were repeating the same tests with a lot of duplication, e.g. in booting test modules. I feel that by creating this common schedule in perl for specific products, everything is in a single place, instead of relaying on multiple yaml files where coverage is difficult to follow since data is scattered among multiple files. Also, you don't need to deal with EXCLUDE_MODULES variable, which is yet another place to miss information.

The same is today happening in JeOS, and guess what, I am gonna do a main_jeos.pm because currently, openSUSE JeOS uses main_common.pm and SLE JeOS uses yaml files. I found out that the coverage is not 1-1... So, by doing this, we can ensure we have also mirror coverage among openSUSE and SLE.

You can see similar things today with main_ltp, main_containers or main_publiccloud, where all the scheduling logic to a certain product is centralized in single place.

This ticket is 10 days after the due-date. Please consider closing this ticket or move the due-date accordingly.