Project

General

Profile

Actions

action #115184

open

[qe-core] Prepare for ALP - Move ALP test scheduling to YAML

Added by szarate over 1 year ago. Updated 5 months ago.

Status:
Feedback
Priority:
Normal
Assignee:
Category:
New test
Target version:
Start date:
2023-05-25
Due date:
2023-05-25 (about 9 months late)
% Done:

0%

Estimated time:
Difficulty:
Sprint:
QE-Core: October Sprint 23 (Oct 11 - Nov 08)

Description

Since we're starting with a brand new product, and it's still in development, we can use the opportunity to shy away from main.pm for scheduling tests avoiding the behemoth that we have for other products for both, SUSE and openSUSE.

As with #115181 we will need to coordinate with other contributors, but that doesn't stop us from coming up with default schedules, and a template for tests that boot from an already installed image in the future

Acceptance criteria

AC1: Pull request with a PoC/RFC for the change is created (not necessarily merged)


Related issues 1 (0 open1 closed)

Copied from openQA Tests - action #115181: [qe-core] Prepare for ALP - Enablement of ALP for QE-CoreResolveddvenkatachala2023-05-252023-05-25

Actions
Actions #1

Updated by szarate over 1 year ago

  • Copied from action #115181: [qe-core] Prepare for ALP - Enablement of ALP for QE-Core added
Actions #2

Updated by RBrownSUSE over 1 year ago

I don't hold any huge objection to this, but given ALP is a derivative of both Tumbleweed and MicroOS, the only question is how do we expect ALP to keep its test coverage aligned with its upstream if we do this?

Both MicroOS and Tumbleweed use main.pms, and whatever coverage is added to ALP would ideally exist in one of those two domains before

Actions #3

Updated by jlausuch over 1 year ago

Yes, I agree with @RBrownSUSE here.
I would actually go even further and create a common main_microos.pm (or whatever name) for ALP, MicroOS, Leap Micro and SLE Micro, since most of the tests/schedules will be shared.

Actions #4

Updated by szarate over 1 year ago

Carrying over to next sprint

Actions #5

Updated by szarate over 1 year ago

  • Sprint changed from QE-Core: August Sprint (Aug 03 - Aug 31) to QE-Core: September Sprint (Aug 31 - Sep 28)
Actions #6

Updated by apappas over 1 year ago

  • Status changed from Workable to Feedback
  • Assignee set to szarate

This ticket is unclear. There have been discussions about extending/rewording/clarifying it. So I am setting it to feedback.

Actions #7

Updated by szarate over 1 year ago

Both MicroOS and Tumbleweed use main.pms, and whatever coverage is added to ALP would ideally exist in one of those two domains before
the only question is how do we expect ALP to keep its test coverage aligned with its upstream if we do this?

Hmmm, we need to replan this, to be able to guarantee that this will stay true, without overloading O3.

I would actually go even further and create a common main_microos.pm (or whatever name) for ALP, MicroOS, Leap Micro and SLE Micro, since most of the tests/schedules will be shared.

@Jose, you mean that main_microos.pm is a single file, that exists for all of the 3 distributions?

Actions #8

Updated by szarate over 1 year ago

Gonna move it out of the sprint, for now we're using YAML schedule for new testsuites, all of those test modules enabled run already on TW.

Actions #9

Updated by szarate over 1 year ago

  • Sprint deleted (QE-Core: September Sprint (Aug 31 - Sep 28))
  • Target version deleted (QE-Core: Ready)
Actions #10

Updated by jlausuch over 1 year ago

  • Target version set to QE-Core: Ready

Maybe we can have a meeting where I go through the code to see the benefit of using a specific and separated main_xyz.pm for the products:

  • MicroOS
  • SLE Micro
  • Leap Micro
  • ALP
Actions #11

Updated by okurz over 1 year ago

szarate wrote:

Hmmm, we need to replan this, to be able to guarantee that this will stay true, without overloading O3.

In case you are concerned regarding hardware ressources rest assured that we have enough headroom. And in case we do run into limits I am convinced we will be able to improve.

Actions #12

Updated by jlausuch 11 months ago

Update on this.

I created this file some months ago:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/main_micro_alp.pm

The reason is to re-use the same test modules to boot the images (all of them boot similarly, but using different needles), to run common tests and specific transactional tests.
The idea is to use this main scheduler for the products that are sharing common coverage: MicroOS, SLE Micro, Leap Micro and now ALP.
Of course, there will be some if-else conditions for those products, but at least it's easier to code them in perl than in yaml, and also it's easier to check the coverage difference for each product in a single file.

Actions #13

Updated by JERiveraMoya 11 months ago

jlausuch wrote:

Update on this.

I created this file some months ago:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/main_micro_alp.pm

The reason is to re-use the same test modules to boot the images (all of them boot similarly, but using different needles), to run common tests and specific transactional tests.
The idea is to use this main scheduler for the products that are sharing common coverage: MicroOS, SLE Micro, Leap Micro and now ALP.
Of course, there will be some if-else conditions for those products, but at least it's easier to code them in perl than in yaml, and also it's easier to check the coverage difference for each product in a single file.

That is how main.pm started, at the beginning the condition are simple and then goes exponentially wrong.
Even the ones that now exist are not easy to follow without diving deeper, I can see few comments of why those conditions are there.
I actually would like to have a programmatic solution instead of yaml, but we would need to think something to avoid to use conditions. It is more about exclude things with precise comments why (there is a feature in openqa to exclude test module), but have some default list of test module to run. But those exclusion might need to be done per squad in a different component, grouping all in single place by code, not in this file all together. We can talk about this some day, it depends how each squad is using it.

Actions #14

Updated by jlausuch 11 months ago

JERiveraMoya wrote:

That is how main.pm started, at the beginning the condition are simple and then goes exponentially wrong.
Even the ones that now exist are not easy to follow without diving deeper, I can see few comments of why those conditions are there.
I actually would like to have a programmatic solution instead of yaml, but we would need to think something to avoid to use conditions. It is more about exclude things with precise comments why (there is a feature in openqa to exclude test module), but have some default list of test module to run. But those exclusion might need to be done per squad in a different component, grouping all in single place by code, not in this file all together. We can talk about this some day, it depends how each squad is using it.

I understand the pain about main_common.pm cause I've been there too.

Let me tell you why we chose this solution.
When we added SLE Micro, we relied on yaml schedules. Then, Leap Micro came into the picture, and we reused some yamls, but of course with some conditions and hacks. We ended up in multiple yamls which were repeating the same tests with a lot of duplication, e.g. in booting test modules. I feel that by creating this common schedule in perl for specific products, everything is in a single place, instead of relaying on multiple yaml files where coverage is difficult to follow since data is scattered among multiple files. Also, you don't need to deal with EXCLUDE_MODULES variable, which is yet another place to miss information.

The same is today happening in JeOS, and guess what, I am gonna do a main_jeos.pm because currently, openSUSE JeOS uses main_common.pm and SLE JeOS uses yaml files. I found out that the coverage is not 1-1... So, by doing this, we can ensure we have also mirror coverage among openSUSE and SLE.

You can see similar things today with main_ltp, main_containers or main_publiccloud, where all the scheduling logic to a certain product is centralized in single place.

Actions #15

Updated by szarate 5 months ago

  • Sprint set to QE-Core: October Sprint 23 (Oct 11 - Nov 08)
Actions #16

Updated by szarate 5 months ago

  • Sprint changed from QE-Core: October Sprint 23 (Oct 11 - Nov 08) to QE-Core: September Sprint 23 (Sep 06 - Oct 04)
Actions #17

Updated by szarate 5 months ago

  • Sprint changed from QE-Core: September Sprint 23 (Sep 06 - Oct 04) to QE-Core: October Sprint 23 (Oct 11 - Nov 08)
Actions

Also available in: Atom PDF