action #182843
openUpstream Fedora alternate template loader (fifloader)? size:S
Description
Motivation¶
Hey folks! So I started working on a PR for this, but it turns out not to be trivial, so I figured I'd file an issue first to see if there is interest before spending too much time on the PR.
Fedora approaches template management a bit differently to how SUSE does it (AFAIK). We never use the webUI interactive editors for managing Products, Machines, JobTemplates etc. We have a canonical template file (well, actually, two). They are maintained in revision control:
- https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/templates.fif.json
- https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/templates-updates.fif.json
any changes to anything you can do via templates must be done as a git commit to those files, and we then reload the templates from scratch every time to our deployments.
If you look at those files, you'll see they're not in either of the two upstream formats (the legacy JobTemplates format, or the modern JobGroups format with in-line YAML templates). They're in a format I invented called the Fedora Intermediate Format (FIF). We maintain a loader for this format called 'fifloader':
which transforms from FIF to the legacy upstream format (and can then call openqa-load-templates, if desired).
The intermediate format includes some simple quality-of-life tweaks. It uses dicts-of-dicts with names as keys, rather than lists-of-dicts with 'name' keys, and it turns 'settings' properties from lists-of-dicts in {key: keyname, value: value} format to simple {key: value} dicts. It introduces some optional concepts for reducing boilerplate, like allowing settings at the Flavor level, and a mechanism for setting 'default' values for products.
But most importantly it kinda flips the 'design philosophy' of organizing tests. Instead of the upstream philosophy of defining Machines, Products and TestSuites and then separately defining templates that say "run this test suite on this product on this machine", the idea is you define Machines and Products, then Profiles which are combinations of machine and product (and which can be grouped to cut down on boilerplate), then you define TestSuites which define in-line which Profiles they run on. So in FIF, we can have a test suite definition like this:
"base_reboot_unmount": {
"profile_groups": {
"base-tests-atomic": 20
},
"profiles": {
"fedora-Cloud_Base-qcow2-qcow2-x86_64-*-bios": 20
},
"settings": {
"BOOTFROM": "c",
"HDD_1": "disk_%FLAVOR%_%MACHINE%.qcow2",
"POSTINSTALL": "base_reboot_unmount",
"ROOT_PASSWORD": "weakpassword",
"START_AFTER_TEST": "%DEPLOY_UPLOAD_TEST%",
"USER_LOGIN": "false"
}
},
which both defines the test suite and says "this test suite should be run on all the product/machine pairs in the 'base-tests-atomic' group, plus the 'fedora-Cloud_Base-qcow2-qcow2-x86_64-*-bios' product/machine pair".
The main benefit of this approach is to make it simpler to add new test suites, which for us is probably the most common operation we do involving this stuff. A commit that adds a test suite only needs to add a single block to the templates file, and the profile information can easily be copied from another similar test if desired. The profile grouping concept also makes it much easier to ensure consistency across similar tests - for instance, making sure all desktop tests are run on all appropriate product/machine pairs. It also makes adding and removing products and machines simpler, if you use profile groups in an organized way.
We invented this format and loader before the upstream inline-YAML format appeared, and that format achieves some of the same goals, but I think still does not achieve all of them (unless I don't fully understand it). Adding a new test suite still requires you to add new entries in multiple different groups to ensure it gets run everywhere it should, the syntax is just a bit more concise now. And it doesn't have all the convenience concepts FIF adds, so it's still substantially longer than FIF (see numbers below).
fifloader also includes JSON schemas for its own format and JSON schemas for the legacy upstream format, and checks the input data against the FIF schemas and the output data against the upstream format schemas.
Some numbers! Fedora templates in:
Upstream legacy format (after conversion by fifloader): 15546 lines
Upstream YAML format (dumped by openqa-load-templates): 6333 lines (template strings wrapped), 8846 lines (template strings unwrapped)
FIF: 3487 lines
The one obvious drawback I can think of is that job group membership is an awkward concept in this design. Previously fifloader just "magically knew" how Fedora jobs were grouped and injected that knowledge at conversion time. Recently I changed it so job group membership is a required property of Products; Products must have a group_name property, and at conversion time, each generated job template is made a member of the group specified by the product it's being generated for, then the group_names are dropped out of the Products for output in the upstream format. This is fine for how Fedora organizes tests; it may not be correct for how other folks do it. If not, we can refine the concept to let it be specified at other levels, I guess.
Another possible drawback is that, currently, fifloader can only convert to the "legacy" format, not the inline-YAML format. This isn't a practical problem so long as the legacy format works, I guess, since in the overall fifloader philosophy you're not supposed to use the web UI to edit this stuff, so it's fine if you can't (the current upstream "drawback" of the legacy format). But if we really intend to ever get rid of the API endpoints that allow loading in the legacy format, we'd have to make fifloader output to the inline-YAML format.
There are more details about FIF in the loader's docstring, if you're interested.
So: is there any interest in having the FIF loader as part of upstream openQA as a documented alternative approach to the current upstream template formats and loader?
Acceptance Criteria¶
- AC1: Alternatives that we have been working on have been proposed, in particular scenario definitions in git
- AC2: Upstream support for FIF has been discussed
- AC3: Follow-up feature requests have been filed
- AC4: Questions from Adam are answered
Updated by tinita 6 days ago
Hi Adam,
it would be interesting to know if you know about our SCENARIO_DEFINITIONS_YAML
format. We are using it for our openqa-in-openqa tests:
https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml
With that we also don't need any entries in the database. We only need some parameters when scheduling:
openqa-cli schedule --host ... \
--param-file SCENARIO_DEFINITIONS_YAML=file.yaml \
VERSION=Tumbleweed DISTRI=openqa FLAVOR=dev ARCH="..." \
HDD_1="..." BUILD="..." _GROUP="..." \
Note that the format might still evolve, but in a backwards compatible way. It would be good to know if that's also something you would be able to work with.
Updated by AdamWill 6 days ago
oh hmm! I'd seen a comment that mentioned it, I think, but didn't grok exactly what it was.
That does look very similar to the FIF setup, yeah, using neat YAML features instead of additional optional dicts. And...I guess it entirely replaces test suites with job templates, right? Just set the test suite settings at the job template level...
But...how does it work if I want to run e.g. openqa_from_git
on both common
and common_4g
, for instance? This is the main thing FIF lets us do concisely.
Updated by tinita 6 days ago
- Related to action #155218: [spike][timeboxed:30h] Use scenario definitions instead of job group templates for os-autoinst-distri-opensuse size:M added
Updated by tinita 6 days ago · Edited
- Status changed from Workable to Feedback
- Assignee set to tinita
AdamWill wrote in #note-4:
But...how does it work if I want to run e.g.
openqa_from_git
on bothcommon
andcommon_4g
, for instance? This is the main thing FIF lets us do concisely.
In #155218 I was thinking about how to improve the scenario format.
In the example I linked above I guess running on both common
and common_4g
does not make sense, but do I understand it correctly that you would want to run it on multiple machines?
This is what I came up with:
(edit: sorry, copied the wrong thing)
https://gist.github.com/perlpunk/f8967896eb7b0965dc70a686dd53ab5f#file-scenario-102-yaml-L12
job_templates:
x86_64:
leap-micro-5.5-DVD-x86_64:
microos_installation_default@(64bit,uefi):
settings:
<<: &dvd_settings
...
Updated by AdamWill 6 days ago · Edited
Both multiple machines and multiple products, yeah.
In Fedora testing (and I'm sure in SUSE testing too?) 'products' are (usually...) images, and it's common that we want to run the same test on multiple images. Usually there's just one machine per arch, but there is a fairly common 'special case': UEFI vs. BIOS. We do that with machines. So we quite commonly want to run the same test on the same product on two different machines.
in your sample above, what is x86_64
?
edit: oh, yeah, and how are job group memberships handled? When we run the same test suite on multiple products/machines in Fedora, the runs aren't all in the same group. In Fedora, job group membership happens to have a clean mapping with products - all tests on any given product are always in the same group - so in fifloader I put the group membership property at the product level. In this approach you'd probably want to have it under the job_templates
, I guess, but the syntax might get awkward...