QA (public) » openQA Project (public)

Motivation¶

Hey folks! So I started working on a PR for this, but it turns out not to be trivial, so I figured I'd file an issue first to see if there is interest before spending too much time on the PR.

Fedora approaches template management a bit differently to how SUSE does it (AFAIK). We never use the webUI interactive editors for managing Products, Machines, JobTemplates etc. We have a canonical template file (well, actually, two). They are maintained in revision control:

• https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/templates.fif.json
• https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/templates-updates.fif.json

any changes to anything you can do via templates must be done as a git commit to those files, and we then reload the templates from scratch every time to our deployments.

If you look at those files, you'll see they're not in either of the two upstream formats (the legacy JobTemplates format, or the modern JobGroups format with in-line YAML templates). They're in a format I invented called the Fedora Intermediate Format (FIF). We maintain a loader for this format called 'fifloader':

• https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/main/f/fifloader.py

which transforms from FIF to the legacy upstream format (and can then call openqa-load-templates, if desired).

The intermediate format includes some simple quality-of-life tweaks. It uses dicts-of-dicts with names as keys, rather than lists-of-dicts with 'name' keys, and it turns 'settings' properties from lists-of-dicts in {key: keyname, value: value} format to simple {key: value} dicts. It introduces some optional concepts for reducing boilerplate, like allowing settings at the Flavor level, and a mechanism for setting 'default' values for products.

But most importantly it kinda flips the 'design philosophy' of organizing tests. Instead of the upstream philosophy of defining Machines, Products and TestSuites and then separately defining templates that say "run this test suite on this product on this machine", the idea is you define Machines and Products, then Profiles which are combinations of machine and product (and which can be grouped to cut down on boilerplate), then you define TestSuites which define in-line which Profiles they run on. So in FIF, we can have a test suite definition like this:

        "base_reboot_unmount": {
            "profile_groups": {
                "base-tests-atomic": 20
            },
            "profiles": {
                "fedora-Cloud_Base-qcow2-qcow2-x86_64-*-bios": 20
            },
            "settings": {
                "BOOTFROM": "c",
                "HDD_1": "disk_%FLAVOR%_%MACHINE%.qcow2",
                "POSTINSTALL": "base_reboot_unmount",
                "ROOT_PASSWORD": "weakpassword",
                "START_AFTER_TEST": "%DEPLOY_UPLOAD_TEST%",
                "USER_LOGIN": "false"
            }
        },

which both defines the test suite and says "this test suite should be run on all the product/machine pairs in the 'base-tests-atomic' group, plus the 'fedora-Cloud_Base-qcow2-qcow2-x86_64-*-bios' product/machine pair".

The main benefit of this approach is to make it simpler to add new test suites, which for us is probably the most common operation we do involving this stuff. A commit that adds a test suite only needs to add a single block to the templates file, and the profile information can easily be copied from another similar test if desired. The profile grouping concept also makes it much easier to ensure consistency across similar tests - for instance, making sure all desktop tests are run on all appropriate product/machine pairs. It also makes adding and removing products and machines simpler, if you use profile groups in an organized way.

We invented this format and loader before the upstream inline-YAML format appeared, and that format achieves some of the same goals, but I think still does not achieve all of them (unless I don't fully understand it). Adding a new test suite still requires you to add new entries in multiple different groups to ensure it gets run everywhere it should, the syntax is just a bit more concise now. And it doesn't have all the convenience concepts FIF adds, so it's still substantially longer than FIF (see numbers below).

fifloader also includes JSON schemas for its own format and JSON schemas for the legacy upstream format, and checks the input data against the FIF schemas and the output data against the upstream format schemas.

Some numbers! Fedora templates in:

Upstream legacy format (after conversion by fifloader): 15546 lines
Upstream YAML format (dumped by openqa-load-templates): 6333 lines (template strings wrapped), 8846 lines (template strings unwrapped)
FIF: 3487 lines

The one obvious drawback I can think of is that job group membership is an awkward concept in this design. Previously fifloader just "magically knew" how Fedora jobs were grouped and injected that knowledge at conversion time. Recently I changed it so job group membership is a required property of Products; Products must have a group_name property, and at conversion time, each generated job template is made a member of the group specified by the product it's being generated for, then the group_names are dropped out of the Products for output in the upstream format. This is fine for how Fedora organizes tests; it may not be correct for how other folks do it. If not, we can refine the concept to let it be specified at other levels, I guess.

Another possible drawback is that, currently, fifloader can only convert to the "legacy" format, not the inline-YAML format. This isn't a practical problem so long as the legacy format works, I guess, since in the overall fifloader philosophy you're not supposed to use the web UI to edit this stuff, so it's fine if you can't (the current upstream "drawback" of the legacy format). But if we really intend to ever get rid of the API endpoints that allow loading in the legacy format, we'd have to make fifloader output to the inline-YAML format.

There are more details about FIF in the loader's docstring, if you're interested.

So: is there any interest in having the FIF loader as part of upstream openQA as a documented alternative approach to the current upstream template formats and loader?

Acceptance Criteria¶

• AC1: Alternatives that we have been working on have been proposed, in particular scenario definitions in git
• AC2: Upstream support for FIF has been discussed
• AC3: Follow-up feature requests have been filed
• AC4: Questions from Adam are answered