QA » openQA Project » openQA Tests

okurz wrote:

Maybe the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution?

I've cc'd the authors of the declarative scheduler. Would that be something that you could look into?

@AdamWill: if this lands upstream, this might be interesting for openqa.fedoraproject.org.

Hum, possibly...

We already by convention define test suites by editing the templates files (not using the web UI), we just always have. I invented an intermediate format earlier this year that makes it a bit more convenient, see the explanation at top of https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/fifloader.py . We sort of have some kind of primitive 'declarative scheduling' already in our loader code: if a test defines the ENTRYPOINT var, our main.pm treats it as a space-separated list of test modules to load, loads those, and doesn't load anything else. There's also a few other vars - PREUPGRADE, PREINSTALL POSTINSTALL - which just tell main.pm to add in some specified test modules at certain points in the normal loading flow.

This scheduler is an interesting idea, I'm curious what the trigger for it was. The docs seem to sort of imply it might be "oh dear, main.pm is getting complicated and no-one remembers what half of it does"...but then as you keep reading the declarative scheduler doc, it seems to be well on the way to introducing quite a lot of complexity of its own :) I suspect the key factor in how much complexity your scheduling configuration has is "how many completely different tests you're trying to run across how many different distros and platforms", not really "how your scheduler is designed". But it's a nice idea. I don't know that it'd necessarily be a clear improvement for Fedora's distri, but then I probably know our main.pm better than anyone and maybe to someone else it's a confusing mess.

AdamWill wrote:

Hum, possibly...

We already by convention define test suites by editing the templates files (not using the web UI), we just always have. I invented an intermediate format earlier this year that makes it a bit more convenient, see the explanation at top of https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/fifloader.py . We sort of have some kind of primitive 'declarative scheduling' already in our loader code: if a test defines the ENTRYPOINT var, our main.pm treats it as a space-separated list of test modules to load, loads those, and doesn't load anything else. There's also a few other vars - PREUPGRADE, PREINSTALL POSTINSTALL - which just tell main.pm to add in some specified test modules at certain points in the normal loading flow.

This scheduler is an interesting idea, I'm curious what the trigger for it was. The docs seem to sort of imply it might be "oh dear, main.pm is getting complicated and no-one remembers what half of it does"...but then as you keep reading the declarative scheduler doc, it seems to be well on the way to introducing quite a lot of complexity of its own :) I suspect the key factor in how much complexity your scheduling configuration has is "how many completely different tests you're trying to run across how many different distros and platforms", not really "how your scheduler is designed". But it's a nice idea. I don't know that it'd necessarily be a clear improvement for Fedora's distri, but then I probably know our main.pm better than anyone and maybe to someone else it's a confusing mess.

Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again. Even though still very isolated. Other things are just a support of yaml native features, like import of other files, etc. The main problem which is still there, is that we have tons of variables being still set in main.pm, and it's very clear to us that pair->key structure doesn't allow to store test data in a readable way, especially when mixed with environment settings. So I believe we are in a transitional state still.
I remember checking fedora's solution, but what you have shared wasn't yet available in February 2019, when we have decided to finally address this issue...
And as you have correctly stated, well maintained smaller entry points would not bring us to this problem at all.

I mean, it's easy enough to see the problem: the openQA loading approach is very unorthodox. It's based on a key assumption that's becoming less and less valid: that all the tests it's loading will be variations on a theme. It was designed for installer testing - it obviously works best when all the tests are basically "boot a virtual machine, run an installer, boot the installed system and check something" and the differences are only in the details. With that assumption, the variable-defined approach makes a lot of sense. The variables describe key attributes of each test within the framework of the base assumption, and the loader can do a lot of heavy lifting with just a few variables worth of information.

The further you get away from that by introducing tests that don't use the installer at all or really have much in common with each other at all, the less sense a variable-based loading system makes and the more you find yourself wanting to do stuff like, well, have variables that just straight-up tell the loader what to do :) But there still are ways the usefulness of the variable-based system show themselves, like when you want to schedule a "check X is really set to Y" for all tests that set X, stuff like that. And we both do still have our whole installer test suites that are built around the initial assumption which still makes quite a lot of sense, and it'd be a lot of spade work to redo all those, I guess.

Oh well, just thinking out loud...

AdamWill wrote:

I mean, it's easy enough to see the problem: the openQA loading approach is very unorthodox. It's based on a key assumption that's becoming less and less valid: that all the tests it's loading will be variations on a theme. It was designed for installer testing - it obviously works best when all the tests are basically "boot a virtual machine, run an installer, boot the installed system and check something" and the differences are only in the details. With that assumption, the variable-defined approach makes a lot of sense. The variables describe key attributes of each test within the framework of the base assumption, and the loader can do a lot of heavy lifting with just a few variables worth of information.

The further you get away from that by introducing tests that don't use the installer at all or really have much in common with each other at all, the less sense a variable-based loading system makes and the more you find yourself wanting to do stuff like, well, have variables that just straight-up tell the loader what to do :) But there still are ways the usefulness of the variable-based system show themselves, like when you want to schedule a "check X is really set to Y" for all tests that set X, stuff like that. And we both do still have our whole installer test suites that are built around the initial assumption which still makes quite a lot of sense, and it'd be a lot of spade work to redo all those, I guess.

Oh well, just thinking out loud...

Oh, wow. That's very close to how I have explained it to myself. It's just that we got a lot of tests, big variety of products and openQA doesn't scale well to support all of it. Next thing we plan to go is some sort of custom grouping inside of the job group, as we have 350+ scenarios in ours.
One historical detail about declarative scheduling is that it was decided to have it in the test repo, as it solved the problem we had as a test maintainers. Tools team has provided solution which allowed us to set settings per machine and therefore define corresponding schedules for the different backends and installation, where previously we would pretend that "it's the same test suite".

What still might be interesting for you is the script to generate yaml schedule from the existing job run: https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/script/yaml_generator

riafarov wrote:

Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.

Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make main.pm schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet another main.pm.

I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database. Adam achieved this for the most part via the fifloader, but I must admit that I find the yaml scheduler more user friendly (albeit it would result in more duplication).

dancermak wrote:

Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.

Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make main.pm schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet another main.pm.

riafarov only mentioned the "conditional schedule", not "declarative schedule".

The YAML test schedule certainly allows an easier introduction for people coming to os-autoinst-distri-opensuse but it can not take away the complexity of so many different test definitions while trying to cover multiple distributions, products, versions, systems and architectures. This only shifts the problem into another domain. Less pain for the individual test contributor, more pain for whoever tries to streamline the test schedules among all the different scenarios.

I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database

yes, but that did not change because main.pm was always there in the same git repo, right? What has changed a bit is instead of having test suites with defined variables within the openQA database now it is mainly (or additionally) the path to the schedule file.

okurz wrote:

dancermak wrote:

Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.

Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make main.pm schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet another main.pm.

riafarov only mentioned the "conditional schedule", not "declarative schedule".

The YAML test schedule certainly allows an easier introduction for people coming to os-autoinst-distri-opensuse but it can not take away the complexity of so many different test definitions while trying to cover multiple distributions, products, versions, systems and architectures. This only shifts the problem into another domain. Less pain for the individual test contributor, more pain for whoever tries to streamline the test schedules among all the different scenarios.

Hm, that's probably correct. I unfortunately cannot really judge that, as I'm a individual test contributor and not someone who maintains the whole test suite.

I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database

yes, but that did not change because main.pm was always there in the same git repo, right? What has changed a bit is instead of having test suites with defined variables within the openQA database now it is mainly (or additionally) the path to the schedule file.

Yes, that is what I meant: the variables and scheduled tests are in one place instead of the variables being in the openQA database and the tests being scheduled in a relatively intransparent way in main.pm.

dancermak wrote:

Yes, that is what I meant: the variables and scheduled tests are in one place instead of the variables being in the openQA database and the tests being scheduled in a relatively intransparent way in main.pm.

but this is unfortunately not true. Each YAML file describes a single scenario so you need multiple files to define multiple scenarios. The "main.pm" is the same for all scenarios so there are multiple logical paths starting within the same file. What is same for both approaches is that they describe which test modules and in which order they are started for each scenario. But both do not define the test suite either which is still in the openQA database.

This is why I linked the ticket #58184 . If you want to have more defined within the same git repo one can use either dump-templates+load-templates, e.g. Fedora are using their complete tests definitions this way, or use job template testsuite definitions, store the job templates within the same git repo and load over API, e.g. with a post-merge CI job. The latter is what more and more SUSE QE teams are doing internally but just storing the job template documents elsewhere.

just make sure it disappears from the triage list.

This ticket is now unlabled and shows up in the list of unreviewed tickets. What would be the right label then?

okurz wrote:

just make sure it disappears from the triage list.

Which of the so many?

I can add: [for-team-that-does-whatever-other-teams-are-not-doing-and-work-with-repo-maintenance-rather-than-tests-themselves] or [fttdwotandawwrmrttt], that'd take it out of most lists, as it would with the current one.

• Note I realize the silliness of the ping pong :)

maritawerner wrote:

This ticket is now unlabled and shows up in the list of unreviewed tickets. What would be the right label then?

oh. I see. I wasn't triggered by the change in priority but by the removal of the target version because "[tools]" in the subject line and no target version sets means for us that we need to at least triage the ticket. This is when it showed up on https://os-autoinst.github.io/qa-tools-backlog-assistant/ which uses the query Untriaged "tools" tagged as referenced from https://progress.opensuse.org/projects/qa/wiki/Tools#Target-numbers-or-guideline-should-be-in-priorities

Meeting: This has to be reevaluated from the side of qe-core. Most like we are going to go the "wheels" route.

apappas wrote:

Meeting: This has to be reevaluated from the side of qe-core. Most like we are going to go the "wheels" route.

Not wheels but either we'll add it to os-autoinst, or we move it to it's own cpan module, instead of keeping it in the test-distribution