action #46667

openQA Tests - action #15132: [epic] Better structure of test plans in main.pm

action #44360: [epic] Parameterize test suites within job groups

Define version-able and human readable format for job scheduling-related tables

Added by mkittler about 1 year ago. Updated 7 months ago.

Status:ResolvedStart date:25/01/2019
Priority:NormalDue date:
Assignee:cdywan% Done:

100%

Category:-
Target version:Current Sprint
Difficulty:
Duration:

Description

  • Document structure of the format.
    • So far we have a YAML document in mind.
    • Format should be version-able/diff-able.
    • Format should be human readable.
  • Relevant openQA tables which data should be in that document are:
    • test_suites
    • products
    • machines
    • job_templates
  • Display the current data (e.g. OSD database) in the format.
  • Implement a validator for that format (because it is indented for being re-imported later).
  • Keep further parameterization in mind.

Related issues

Related to openQA Tests - action #44420: [functional][y][timeboxed:6h] proof-of-concept of declara... Resolved 28/11/2018 26/02/2019

History

#1 Updated by cdywan about 1 year ago

  • Assignee set to cdywan

#2 Updated by mkittler about 1 year ago

  • Status changed from New to In Progress

I'm adding a link to the latest paste bin here to track the progress at least a little bit: http://paste.opensuse.org/view/raw/59471467

Feel encouraged to update the ticket yourself as well, maybe sharing a WIP branch and further ideas.

#3 Updated by cdywan about 1 year ago

- Maintenance: SLE 12 SP4 Incidents:
 products:
 - &sle-12-SP4-Server-DVD-Incidents
   distri: sle
   flavor: Server-DVD-Incidents
   version: 12-SP4
 - &sle-12-SP4-Server-DVD-Incidents-Minimal
   distri: sle
   flavor: Server-DVD-Incidents-Minimal
   version: 12-SP4
 architectures:
 - x86_64:
   - defaults:
     prio: 50
     machine: 64bit
   - <<: *sle-12-SP4-Server-DVD-Incidents
     - mau-extratests
       prio: 50
     - mau-filesystem
     - mau-webserver
   - <<: *sle-12-SP4-Server-DVD-Incidents-Minimal
     - qam-minimal
       machine:
       - 64bit
       - uefi
     - qam-minimal-RAID1
       machine: 64bit-smp
       prio: 50
     - qam-minimal
       prio: 50
 - aarch64:
   - defaults:
     machine: aarch64-virtio
   - <<: *sle-12-SP4-Server-DVD-Incidents-Minimal
     - qam-minimal:
       prio: 50
 - ppc64le:
   - defaults:
     machine: ppc64le
   - <<: *sle-12-SP4-Server-DVD-Incidents-Minimal
     - qam-minimal
     - qam-minimal-RAID1
     - qam-minimal-full
 - s390x:
   - defaults:
     machine: zkvm
     pro: 100
   - <<: *sle-12-SP4-Server-DVD-Incidents-Minimal
     - qam-minimal-full:
       machine: s390x-zVM-hsi-l3

#4 Updated by cdywan about 1 year ago

  • Related to action #44420: [functional][y][timeboxed:6h] proof-of-concept of declarative test schedule definition, e.g. in YAML file(s) added

#5 Updated by cdywan about 1 year ago

New branch with much simplified query and updated format https://github.com/kalikiana/openQA/tree/yaml1

#7 Updated by cdywan 11 months ago

'Maintenance: SLE 12 SP1 Updates':
  architectures:
    x86_64:
      sle-12-SP1-Server-DVD-Updates-x86_64:
      - qam-allpatterns
      - qam-minimal+base
      - qam-textmode
      - qam-gnome
  defaults:
    x86_64:
      machine: 64bit
      prio: 50
  products:
    sle-12-SP1-Server-DVD-Updates-x86_64:
      distri: sle
      flavor: Server-DVD-Updates
      version: 12-SP1

x-released SLE 11 SP4 1.Server:
  architectures:
    ppc64:
      sle-11-SP4-Alpha-Server-DVD-ppc64:
      - gnome:
          prio: 45
      - minimal_x
      - cryptlvm_minimal_x
      - mediacheck
      - ext3:
          machine: ppc64-smp
          prio: 45
      - ext3:
          prio: 45
      - RAID6
      - minimal+base:
          prio: 40
      - textmode
  defaults:
    ppc64:
      machine: ppc64
      prio: 50
  products:
    sle-11-SP4-Alpha-Server-DVD-ppc64:
      distri: sle
      flavor: Server-DVD
      version: 11-SP4-Alpha

#8 Updated by coolo 11 months ago

The feedback from xiaojing (indirectly) was that we should avoid words like 'prio' and 'distri' but use proper english

#9 Updated by cdywan 11 months ago

See below updated examples with prio and distri spelt out:

'Maintenance: SLE 12 SP1 Updates':
  architectures:
    x86_64:
      sle-12-SP1-Server-DVD-Updates-x86_64:
      - qam-allpatterns
      - qam-minimal+base
      - qam-textmode
      - qam-gnome
  defaults:
    x86_64:
      machine: 64bit
      priority: 50
  products:
    sle-12-SP1-Server-DVD-Updates-x86_64:
      distribution: sle
      flavor: Server-DVD-Updates
      version: 12-SP1

x-released SLE 11 SP4 1.Server:
  architectures:
    ppc64:
      sle-11-SP4-Alpha-Server-DVD-ppc64:
      - gnome:
          priority: 45
      - minimal_x
      - cryptlvm_minimal_x
      - mediacheck
      - ext3:
          machine: ppc64-smp
          priority: 45
      - ext3:
          priority: 45
      - RAID6
      - minimal+base:
          priority: 40
      - textmode
  defaults:
    ppc64:
      machine: ppc64
      priority: 50
  products:
    sle-11-SP4-Alpha-Server-DVD-ppc64:
      distribution: sle
      flavor: Server-DVD
      version: 11-SP4-Alpha

#10 Updated by cdywan 11 months ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

#11 Updated by coolo 11 months ago

  • Status changed from Resolved to In Progress

No no - don't close this as resolved as long as it's in experimental. The next steps (in dubio in next tickets, but in 'current sprint') are to be presented to testers.

And for that an API route is rather clumsy to use. I overlooked that fact

#12 Updated by cdywan 11 months ago

As the (experimental) API has been deployed now, here's how you try it out:

# get an export of all defined job groups in production get this URL
http://openqa.suse.de/api/v1/experimental/job_templates_scheduling

# export a group by its ID eg. 19
http://openqa.suse.de/api/v1/experimental/job_templates_scheduling/19

#13 Updated by riafarov 11 months ago

As per our discussion, I've tried it yesterday. Current format looks human readable and clear. So I have only two ideas to make it more compact by combining common parts.
First thing is that many scenarios are enabled on 3 architectures (sometimes four), so we have to add them 4 times and without additional visualization layer, it's hard to say where test suite is enabled.
Second thing is about machines, for example "gnome" scenario. It's enabled on 4 machines only on 64 bit, even though I like that we have granularity which we might need, but I guess we could also convert 'machine' node to be a list, but still allowing properties per machine. There are multiple ways to do that, and main motivation is to make it shorter but keeping it human readable.
Also, I was wondering if we should put test suite on top level in the job group, as usually question is where test suite is enabled and not which test suites are enabled for this medium. But that again can be easily solved by proper visualization.

That would be my 2 cents. And once again, good job!

#14 Updated by coolo 11 months ago

I think we can add a visualization later. If we don't edit anymore, we have way more flexibility on the way to display the matrix.

And what is possibly not known: yaml has 'macros', so we can have '3 architectures' as one macro:
https://github.com/cyklo/Bukkit-OtherBlocks/wiki/Aliases-(advanced-YAML-usage)

#15 Updated by coolo 11 months ago

ah,the aliases are already used in comment 3. But as the current examples are exported, they don't make use of this

#16 Updated by cdywan 11 months ago

riafarov wrote:

First thing is that many scenarios are enabled on 3 architectures (sometimes four), so we have to add them 4 times and without additional visualization layer, it's hard to say where test suite is enabled.

Second thing is about machines, for example "gnome" scenario. It's enabled on 4 machines only on 64 bit, even though I like that we have granularity which we might need, but I guess we could also convert 'machine' node to be a list, but still allowing properties per machine. There are multiple ways to do that, and main motivation is to make it shorter but keeping it human readable.

Sorry I'm getting back to this a bit late. I should clarify that aliases are not formally part of this proposal, but they are fully supported by the import as per #46670 - or in other words, this would be left up to the user.

So re-using one list of tests for several configurations:

architectures:
  i586:
    opensuse-13.1-DVD-i586: &tests
    - foo
    - bar
    - spam
    - eggs
  ppc64:
    opensuse-13.1-DVD-ppc64:
      *tests
  x86_64:
    sle-12-SP1-Server-DVD-Updates-x86_64:
      *tests

This is effectively the same as defining the same tests for all of the given architectures.

#17 Updated by okurz 8 months ago

  • Category set to Feature requests

#18 Updated by okurz 8 months ago

  • Category deleted (Feature requests)

#19 Updated by cdywan 7 months ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF