action #46667
closedopenQA Tests - coordination #15132: [saga][epic] Better structure of test plans in main.pm
coordination #44360: [epic] Parameterize test suites within job groups
Define version-able and human readable format for job scheduling-related tables
Added by mkittler over 5 years ago. Updated almost 5 years ago.
100%
Description
- Document structure of the format.
- So far we have a YAML document in mind.
- Format should be version-able/diff-able.
- Format should be human readable.
- Relevant openQA tables which data should be in that document are:
- test_suites
- products
- machines
- job_templates
- Display the current data (e.g. OSD database) in the format.
- Implement a validator for that format (because it is indented for being re-imported later).
- Keep further parameterization in mind.
Updated by mkittler over 5 years ago
- Status changed from New to In Progress
I'm adding a link to the latest paste bin here to track the progress at least a little bit: http://paste.opensuse.org/view/raw/59471467
Feel encouraged to update the ticket yourself as well, maybe sharing a WIP branch and further ideas.
Updated by livdywan over 5 years ago
- Maintenance: SLE 12 SP4 Incidents:
products:
- &sle-12-SP4-Server-DVD-Incidents
distri: sle
flavor: Server-DVD-Incidents
version: 12-SP4
- &sle-12-SP4-Server-DVD-Incidents-Minimal
distri: sle
flavor: Server-DVD-Incidents-Minimal
version: 12-SP4
architectures:
- x86_64:
- defaults:
prio: 50
machine: 64bit
- <<: *sle-12-SP4-Server-DVD-Incidents
- mau-extratests
prio: 50
- mau-filesystem
- mau-webserver
- <<: *sle-12-SP4-Server-DVD-Incidents-Minimal
- qam-minimal
machine:
- 64bit
- uefi
- qam-minimal-RAID1
machine: 64bit-smp
prio: 50
- qam-minimal
prio: 50
- aarch64:
- defaults:
machine: aarch64-virtio
- <<: *sle-12-SP4-Server-DVD-Incidents-Minimal
- qam-minimal:
prio: 50
- ppc64le:
- defaults:
machine: ppc64le
- <<: *sle-12-SP4-Server-DVD-Incidents-Minimal
- qam-minimal
- qam-minimal-RAID1
- qam-minimal-full
- s390x:
- defaults:
machine: zkvm
pro: 100
- <<: *sle-12-SP4-Server-DVD-Incidents-Minimal
- qam-minimal-full:
machine: s390x-zVM-hsi-l3
Updated by livdywan over 5 years ago
- Related to action #44420: [functional][y][timeboxed:6h] proof-of-concept of declarative test schedule definition, e.g. in YAML file(s) added
Updated by livdywan about 5 years ago
New branch with much simplified query and updated format https://github.com/kalikiana/openQA/tree/yaml1
Updated by livdywan about 5 years ago
Updated by livdywan about 5 years ago
'Maintenance: SLE 12 SP1 Updates':
architectures:
x86_64:
sle-12-SP1-Server-DVD-Updates-x86_64:
- qam-allpatterns
- qam-minimal+base
- qam-textmode
- qam-gnome
defaults:
x86_64:
machine: 64bit
prio: 50
products:
sle-12-SP1-Server-DVD-Updates-x86_64:
distri: sle
flavor: Server-DVD-Updates
version: 12-SP1
x-released SLE 11 SP4 1.Server:
architectures:
ppc64:
sle-11-SP4-Alpha-Server-DVD-ppc64:
- gnome:
prio: 45
- minimal_x
- cryptlvm_minimal_x
- mediacheck
- ext3:
machine: ppc64-smp
prio: 45
- ext3:
prio: 45
- RAID6
- minimal+base:
prio: 40
- textmode
defaults:
ppc64:
machine: ppc64
prio: 50
products:
sle-11-SP4-Alpha-Server-DVD-ppc64:
distri: sle
flavor: Server-DVD
version: 11-SP4-Alpha
Updated by coolo about 5 years ago
The feedback from xiaojing (indirectly) was that we should avoid words like 'prio' and 'distri' but use proper english
Updated by livdywan about 5 years ago
See below updated examples with prio and distri spelt out:
'Maintenance: SLE 12 SP1 Updates':
architectures:
x86_64:
sle-12-SP1-Server-DVD-Updates-x86_64:
- qam-allpatterns
- qam-minimal+base
- qam-textmode
- qam-gnome
defaults:
x86_64:
machine: 64bit
priority: 50
products:
sle-12-SP1-Server-DVD-Updates-x86_64:
distribution: sle
flavor: Server-DVD-Updates
version: 12-SP1
x-released SLE 11 SP4 1.Server:
architectures:
ppc64:
sle-11-SP4-Alpha-Server-DVD-ppc64:
- gnome:
priority: 45
- minimal_x
- cryptlvm_minimal_x
- mediacheck
- ext3:
machine: ppc64-smp
priority: 45
- ext3:
priority: 45
- RAID6
- minimal+base:
priority: 40
- textmode
defaults:
ppc64:
machine: ppc64
priority: 50
products:
sle-11-SP4-Alpha-Server-DVD-ppc64:
distribution: sle
flavor: Server-DVD
version: 11-SP4-Alpha
Updated by livdywan about 5 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Updated by coolo about 5 years ago
- Status changed from Resolved to In Progress
No no - don't close this as resolved as long as it's in experimental. The next steps (in dubio in next tickets, but in 'current sprint') are to be presented to testers.
And for that an API route is rather clumsy to use. I overlooked that fact
Updated by livdywan about 5 years ago
As the (experimental) API has been deployed now, here's how you try it out:
# get an export of all defined job groups in production get this URL
http://openqa.suse.de/api/v1/experimental/job_templates_scheduling
# export a group by its ID eg. 19
http://openqa.suse.de/api/v1/experimental/job_templates_scheduling/19
Updated by riafarov about 5 years ago
As per our discussion, I've tried it yesterday. Current format looks human readable and clear. So I have only two ideas to make it more compact by combining common parts.
First thing is that many scenarios are enabled on 3 architectures (sometimes four), so we have to add them 4 times and without additional visualization layer, it's hard to say where test suite is enabled.
Second thing is about machines, for example "gnome" scenario. It's enabled on 4 machines only on 64 bit, even though I like that we have granularity which we might need, but I guess we could also convert 'machine' node to be a list, but still allowing properties per machine. There are multiple ways to do that, and main motivation is to make it shorter but keeping it human readable.
Also, I was wondering if we should put test suite on top level in the job group, as usually question is where test suite is enabled and not which test suites are enabled for this medium. But that again can be easily solved by proper visualization.
That would be my 2 cents. And once again, good job!
Updated by coolo about 5 years ago
I think we can add a visualization later. If we don't edit anymore, we have way more flexibility on the way to display the matrix.
And what is possibly not known: yaml has 'macros', so we can have '3 architectures' as one macro:
https://github.com/cyklo/Bukkit-OtherBlocks/wiki/Aliases-(advanced-YAML-usage)
Updated by coolo about 5 years ago
ah,the aliases are already used in comment 3. But as the current examples are exported, they don't make use of this
Updated by livdywan about 5 years ago
riafarov wrote:
First thing is that many scenarios are enabled on 3 architectures (sometimes four), so we have to add them 4 times and without additional visualization layer, it's hard to say where test suite is enabled.
Second thing is about machines, for example "gnome" scenario. It's enabled on 4 machines only on 64 bit, even though I like that we have granularity which we might need, but I guess we could also convert 'machine' node to be a list, but still allowing properties per machine. There are multiple ways to do that, and main motivation is to make it shorter but keeping it human readable.
Sorry I'm getting back to this a bit late. I should clarify that aliases are not formally part of this proposal, but they are fully supported by the import as per #46670 - or in other words, this would be left up to the user.
So re-using one list of tests for several configurations:
architectures:
i586:
opensuse-13.1-DVD-i586: &tests
- foo
- bar
- spam
- eggs
ppc64:
opensuse-13.1-DVD-ppc64:
*tests
x86_64:
sle-12-SP1-Server-DVD-Updates-x86_64:
*tests
This is effectively the same as defining the same tests for all of the given architectures.
Updated by okurz almost 5 years ago
- Category deleted (
Feature requests)
Regarding #46667#note-8 please see https://github.com/os-autoinst/openQA/pull/2130
Updated by livdywan almost 5 years ago
- Status changed from In Progress to Resolved