QA » openQA Project » openQA Tests

I would suggest to create a completely new parent job group and sub job groups so we know what structure is wanted. Then testers can change the job scheduling to use these news groups from now on.

Of course we can also move jobs from their old groups to the new groups using SQL queries. This shouldn't be hard to do if one knows the criteria for mapping the jobs to their new groups. We could of course do this on a staging instance first to avoid mistakes in production and to give the users a preview.

I guess it would already help if test writers would propose a job group structure like this with the relevant jobs which would go into the particular groups:

• Maintenance v2
  • Team 1: jobs matching settings FOO=bar, …
  • Team 2: jobs matching settings FOO=baz, …
  • …

okurz wrote:

• On a clean openQA instance import an OSD database dump, create a job group structure for maintenance tests similar to the "SLE15" job group and update all "maintenance" tests to show up in the new structure, e.g. with SQL updates

I would suggest to use an instance that's exposed via the VPN and can be checked out and discussed with others, if that's not already implied (to my mind clean usually suggests a test instance on an individual developer machine).

I'd use one of the staging instances for that (as they shouldn't be publicly reachable).

mkittler wrote:

[…] I guess it would already help if test writers would propose a job group structure like this with the relevant jobs which would go into the particular groups:

This is why I suggested "create a job group structure for maintenance tests similar to the "SLE15" job group and update all "maintenance" tests to show up in the new structure, e.g. with SQL updates". But it can be helpful to grab any "test writer" and work with them to try out where the test(s) that they are interested in should reside

@vpelcak any idea how target look of groups?

I brought up the topic within the weekly SUSE QE sync meeting 2021-07-21 and no reasons could be found why we have individual job groups per service pack. Some job groups like https://openqa.suse.de/parent_group_overview/23#grouped_by_build seem complicated with a job group per service pack. This means that for every service pack a new job group with new job templates and mostly copy-pasted schedule is created. Maybe at that time (years ago) the display of results was not meeting expectations but since we have test overview queries like https://openqa.suse.de/tests/overview?result=failed&arch=&flavor=&machine=&test=&modules=&groupid=366&groupid=308&groupid=232&groupid=165&groupid=280&groupid=218&groupid=108&groupid=54# that properly resolve the versions (also since years) that limitation would not be true anymore. I will ask more people.

EDIT: sent email to openqa@suse.de and qa-team@suse.de

okurz wrote:

I brought up the topic within the weekly SUSE QE sync meeting 2021-07-21 and no reasons could be found why we have individual job groups per service pack.

The reason is simple: investigation of failures. You can't filter by service pack in the "Next & previous results" tab. In some groups, there will be 100+ unrelated incidents between the current version of the package which you're investigating and the last released version that you want to compare the results to. If the per-servicepack job groups get merged into one, looking up previous job results will become nearly impossible in some groups.

It's because MUs are usually not targeted to all versions of the product, but to the specific ones only and this varies from only one of them to all.

MDoucha wrote:

okurz wrote:

I brought up the topic within the weekly SUSE QE sync meeting 2021-07-21 and no reasons could be found why we have individual job groups per service pack.

The reason is simple: investigation of failures. You can't filter by service pack in the "Next & previous results" tab. In some groups, there will be 100+ unrelated incidents between the current version of the package which you're investigating and the last released version that you want to compare the results to. If the per-servicepack job groups get merged into one, looking up previous job results will become nearly impossible in some groups.

Jobs in "next & previous" are resolved to be within the same scenario, i.e. same product, version, flavor, arch, testsuite, machine. AFAIK the job group has no impact and also jobs with differing versions do not show up as the same scenario. See in http://lord.arch/tests/2794#next_previous that there is only a single job. http://lord.arch/tests/2794 and http://lord.arch/tests/2795 only differ in version in build. Maybe you mean something different?

jkohoutek wrote:

It's because MUs are usually not targeted to all versions of the product, but to the specific ones only and this varies from only one of them to all.

Yes, that is clear. So we obviously need separate tests for each version of a product. But that should not mean that we need a separate job group per version of the product in openQA tests, right?

osukup wrote:

@vpelcak any idea how target look of groups?

Similar as for SLE 15 which looks like this:

• SLE 15 (parent group)
  • Functional -> QE-Core team
  • YaST -> YaST team
  • Migration: Regression -> Migration team
  • HA -> SHAP(?) team
  • …

where each job group shows results from e.g. SP2-Build187.1 as well as SP3-Build11.1

Instead of

• Maintenance: Single Incidents
  • Maintenance: SLE 15 SP1 Incidents
  • Maintenance: SLE 15 SP2 Incidents
  • Maintenance: SLE 15 SP2 HA Incidents
  • Maintenance: SLE 15 SP3 Incidents
  • …
• Maintenance: Test Repo
  • Maintenance: SLE 15 SP1 Updates
  • Maintenance: SLE 15 SP2 Updates
  • Maintenance: SLE 15 SP3 Updates
  • …

have something like

• Maintenance: Single Incidents
  • Functional
  • YaST
  • HA
  • …
• Maintenance: Test Repo
  • Functional
  • YaST
  • HA
  • …

where each job group can again show results for multiple versions with their builds in parallel, e.g. 15-SP1-Build20210720-2 as well as 15-SP2-Build20210720-2

+1 to this , I think it would be much better .
I wonder if in same iteration we should also drop "Maintenance" and "Updates" from group names ? or you will treat this as separate topic ?

asmorodskyi wrote:

+1 to this , I think it would be much better .
I wonder if in same iteration we should also drop "Maintenance" and "Updates" from group names ? or you will treat this as separate topic ?

Why would we want this?

jkohoutek wrote:

asmorodskyi wrote:

+1 to this , I think it would be much better .
I wonder if in same iteration we should also drop "Maintenance" and "Updates" from group names ? or you will treat this as separate topic ?

Why would we want this?

I don't see why we need it actually ? Isn't it obvious that SLE 12 SP4 is in maintenance ? this very long job group names spam a lot of places ( test reports , openQA UI , YAML config files without giving anything back ) . Yes I know that currently there is some logic in openQABot which rely on this but it can be easily done in a way to avoid it.

Discussed during the weekly QE sync 2021-11-10. We clarified some points, e.g. that the intention is not to multiply the number of job groups by squad times product times version but of course have multiple products and versions within the same job groups but use the job groups to separate by squad.

current state was presented during weekly QE sync 2021-11-24 and mgrifalconi received a lot of positive feedback from group representatives. http://d432.qam.suse.de/ shows the current state with test results moved into a different job group structure. https://gitlab.suse.de/-/snippets/1606 has some notes. With this pages like http://10.161.229.176/tests/overview?groupid=404 can be used to show a job group and hence squad specific view of all related test results regardless of build product version and/or build id.

@mgrifalconi as discussed, please provide screenshot, current state, plans for the next steps, etc. Thanks

Possible plan of action:

• Create a new job group that include all Maint.Updates SLE versions and call it QE-Core
• Create a new job group with a squad name and move out from QE-Core relevent stuff
• Repeat step 2 until done

OR

• Create a new job group with a squad name (not QE-Core) and move out relevant stuff from the existing multiple groups.
• Repeat step 1 until QE-Core stuff remains and create its group with that

I would say that latter approach has already been done with the "Containers" parent group so you could just continue there :) Also, there are already multiple job groups like "Maintenance: SLE 15 SP3 HA Incidents" and "Maintenance: SLE 15 SP3 HA Incidents" so one could go ahead with combining all these version specific job groups into one but keep the current component/squad structuring in a first step. So e.g. combine all "Maintenance: SLE $version Kernel Incidents" into one new "Maintenance: SLE Kernel Incidents", same for HA, SAP, etc.

At the same time we might want to move old test results using some SQL but it is unrelated.

True. Also I would simply not bother and keep the old test results untouched as long as they exist as they are automatically removed eventually anyway.

Decisions to take before starting

• Job Group on root dir or inside its own directory? (Dir give the advantage of having the overview button in the homepage)

I guess with "own directory" you mean "parent group"? Then the answer is parent group because as you stated you still want to see the grouping of all "maintenance" tests and such. The important part is that with squad-specific child job groups test overview links can be created that show "all tests that one squad is interested in regardless of version".

• Job Group per component or per squad (SAP/HA <<>> SAP and HA) (Container and JeOS and Public Cloud)

See my above proposal. But I consider it most important that the structure is essentially the same as for product validation.

mgrifalconi wrote:

SAP/HA is done: https://openqa.suse.de/tests/overview?groupid=405

Looks very good, but where exactly is this link accessible from in the openqa menu? I can see a SAP/HA Maintenance Updates job group, but it only displays the results for 15-SP3. The tests that are shown for 15 GA and 15-SP1 have been cancelled. What was the reason for this?

More importantly, it might be necessary to move the mau-sles-sys-param-check@64bit-2gbram test into the SLE Updates job group.

jctmichel wrote:

mgrifalconi wrote:

SAP/HA is done: https://openqa.suse.de/tests/overview?groupid=405

Looks very good, but where exactly is this link accessible from in the openqa menu? I can see a SAP/HA Maintenance Updates job group, but it only displays the results for 15-SP3. The tests that are shown for 15 GA and 15-SP1 have been cancelled. What was the reason for this?

From any page on https://openqa.suse.de from the top-down "Job Groups" selections menu one can find the parent group SAP/HA and within there the single job group SAP/HA Maintenance Updates.
This page shows builds for all the versions, it's just not obvious because only the build number is shown. We have an open feature request #53264 to also show the version when job groups have information for multiple versions

More importantly, it might be necessary to move the mau-sles-sys-param-check@64bit-2gbram test into the SLE Updates job group.

The goal is to make it explicit by the job group selections which squads take responsibility for a test scenario regardless for which product/version/flavor a scenario applies to. So if you find an agreement that another QE squad takes over then you or they can move the according job templates which are part of the schedule