action #77071
open[qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
Added by dancermak about 4 years ago. Updated 7 months ago.
0%
Description
The declarative YAML scheduler is a pretty useful thing to have, but it is currently only available in the os-autoinst-distri-opensuse repository. Would it be possible to move it to the upstream test api, so that it can be used by other test users as well?
Suggestions¶
Start by giving a look at: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/scheduler.pm
Updated by okurz about 4 years ago
- Related to coordination #58184: [saga][epic][use case] full version control awareness within openQA added
Updated by okurz about 4 years ago
- Project changed from openQA Project (public) to openQA Tests (public)
- Category changed from Feature requests to Enhancement to existing tests
I think in general this is a good idea. I see this related to our ticet #58184 . Likely the right location for this support would be os-autoinst where the main.pm is parsed and instead a YAML based schedule could be loaded. Maybe the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution?
Updated by dancermak about 4 years ago
okurz wrote:
Maybe the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution?
I've cc'd the authors of the declarative scheduler. Would that be something that you could look into?
@AdamWill: if this lands upstream, this might be interesting for openqa.fedoraproject.org.
Updated by AdamWill about 4 years ago
Hum, possibly...
We already by convention define test suites by editing the templates files (not using the web UI), we just always have. I invented an intermediate format earlier this year that makes it a bit more convenient, see the explanation at top of https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/fifloader.py . We sort of have some kind of primitive 'declarative scheduling' already in our loader code: if a test defines the ENTRYPOINT
var, our main.pm
treats it as a space-separated list of test modules to load, loads those, and doesn't load anything else. There's also a few other vars - PREUPGRADE
, PREINSTALL
POSTINSTALL
- which just tell main.pm
to add in some specified test modules at certain points in the normal loading flow.
This scheduler is an interesting idea, I'm curious what the trigger for it was. The docs seem to sort of imply it might be "oh dear, main.pm
is getting complicated and no-one remembers what half of it does"...but then as you keep reading the declarative scheduler doc, it seems to be well on the way to introducing quite a lot of complexity of its own :) I suspect the key factor in how much complexity your scheduling configuration has is "how many completely different tests you're trying to run across how many different distros and platforms", not really "how your scheduler is designed". But it's a nice idea. I don't know that it'd necessarily be a clear improvement for Fedora's distri, but then I probably know our main.pm
better than anyone and maybe to someone else it's a confusing mess.
Updated by riafarov about 4 years ago
AdamWill wrote:
Hum, possibly...
We already by convention define test suites by editing the templates files (not using the web UI), we just always have. I invented an intermediate format earlier this year that makes it a bit more convenient, see the explanation at top of https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/fifloader.py . We sort of have some kind of primitive 'declarative scheduling' already in our loader code: if a test defines the
ENTRYPOINT
var, ourmain.pm
treats it as a space-separated list of test modules to load, loads those, and doesn't load anything else. There's also a few other vars -PREUPGRADE
,PREINSTALL
POSTINSTALL
- which just tellmain.pm
to add in some specified test modules at certain points in the normal loading flow.This scheduler is an interesting idea, I'm curious what the trigger for it was. The docs seem to sort of imply it might be "oh dear,
main.pm
is getting complicated and no-one remembers what half of it does"...but then as you keep reading the declarative scheduler doc, it seems to be well on the way to introducing quite a lot of complexity of its own :) I suspect the key factor in how much complexity your scheduling configuration has is "how many completely different tests you're trying to run across how many different distros and platforms", not really "how your scheduler is designed". But it's a nice idea. I don't know that it'd necessarily be a clear improvement for Fedora's distri, but then I probably know ourmain.pm
better than anyone and maybe to someone else it's a confusing mess.
Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again. Even though still very isolated. Other things are just a support of yaml native features, like import of other files, etc. The main problem which is still there, is that we have tons of variables being still set in main.pm
, and it's very clear to us that pair->key structure doesn't allow to store test data in a readable way, especially when mixed with environment settings. So I believe we are in a transitional state still.
I remember checking fedora's solution, but what you have shared wasn't yet available in February 2019, when we have decided to finally address this issue...
And as you have correctly stated, well maintained smaller entry points would not bring us to this problem at all.
Updated by AdamWill about 4 years ago
I mean, it's easy enough to see the problem: the openQA loading approach is very unorthodox. It's based on a key assumption that's becoming less and less valid: that all the tests it's loading will be variations on a theme. It was designed for installer testing - it obviously works best when all the tests are basically "boot a virtual machine, run an installer, boot the installed system and check something" and the differences are only in the details. With that assumption, the variable-defined approach makes a lot of sense. The variables describe key attributes of each test within the framework of the base assumption, and the loader can do a lot of heavy lifting with just a few variables worth of information.
The further you get away from that by introducing tests that don't use the installer at all or really have much in common with each other at all, the less sense a variable-based loading system makes and the more you find yourself wanting to do stuff like, well, have variables that just straight-up tell the loader what to do :) But there still are ways the usefulness of the variable-based system show themselves, like when you want to schedule a "check X is really set to Y" for all tests that set X, stuff like that. And we both do still have our whole installer test suites that are built around the initial assumption which still makes quite a lot of sense, and it'd be a lot of spade work to redo all those, I guess.
Oh well, just thinking out loud...
Updated by riafarov about 4 years ago
AdamWill wrote:
I mean, it's easy enough to see the problem: the openQA loading approach is very unorthodox. It's based on a key assumption that's becoming less and less valid: that all the tests it's loading will be variations on a theme. It was designed for installer testing - it obviously works best when all the tests are basically "boot a virtual machine, run an installer, boot the installed system and check something" and the differences are only in the details. With that assumption, the variable-defined approach makes a lot of sense. The variables describe key attributes of each test within the framework of the base assumption, and the loader can do a lot of heavy lifting with just a few variables worth of information.
The further you get away from that by introducing tests that don't use the installer at all or really have much in common with each other at all, the less sense a variable-based loading system makes and the more you find yourself wanting to do stuff like, well, have variables that just straight-up tell the loader what to do :) But there still are ways the usefulness of the variable-based system show themselves, like when you want to schedule a "check X is really set to Y" for all tests that set X, stuff like that. And we both do still have our whole installer test suites that are built around the initial assumption which still makes quite a lot of sense, and it'd be a lot of spade work to redo all those, I guess.
Oh well, just thinking out loud...
Oh, wow. That's very close to how I have explained it to myself. It's just that we got a lot of tests, big variety of products and openQA doesn't scale well to support all of it. Next thing we plan to go is some sort of custom grouping inside of the job group, as we have 350+ scenarios in ours.
One historical detail about declarative scheduling is that it was decided to have it in the test repo, as it solved the problem we had as a test maintainers. Tools team has provided solution which allowed us to set settings per machine and therefore define corresponding schedules for the different backends and installation, where previously we would pretend that "it's the same test suite".
What still might be interesting for you is the script to generate yaml schedule from the existing job run: https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/script/yaml_generator
Updated by dancermak about 4 years ago
riafarov wrote:
Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.
Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make main.pm
schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet another main.pm
.
I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database. Adam achieved this for the most part via the fifloader, but I must admit that I find the yaml scheduler more user friendly (albeit it would result in more duplication).
Updated by okurz about 4 years ago
dancermak wrote:
Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.
Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make
main.pm
schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet anothermain.pm
.
riafarov only mentioned the "conditional schedule", not "declarative schedule".
The YAML test schedule certainly allows an easier introduction for people coming to os-autoinst-distri-opensuse but it can not take away the complexity of so many different test definitions while trying to cover multiple distributions, products, versions, systems and architectures. This only shifts the problem into another domain. Less pain for the individual test contributor, more pain for whoever tries to streamline the test schedules among all the different scenarios.
I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database
yes, but that did not change because main.pm was always there in the same git repo, right? What has changed a bit is instead of having test suites with defined variables within the openQA database now it is mainly (or additionally) the path to the schedule file.
Updated by dancermak about 4 years ago
okurz wrote:
dancermak wrote:
Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.
Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make
main.pm
schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet anothermain.pm
.riafarov only mentioned the "conditional schedule", not "declarative schedule".
The YAML test schedule certainly allows an easier introduction for people coming to os-autoinst-distri-opensuse but it can not take away the complexity of so many different test definitions while trying to cover multiple distributions, products, versions, systems and architectures. This only shifts the problem into another domain. Less pain for the individual test contributor, more pain for whoever tries to streamline the test schedules among all the different scenarios.
Hm, that's probably correct. I unfortunately cannot really judge that, as I'm a individual test contributor and not someone who maintains the whole test suite.
I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database
yes, but that did not change because main.pm was always there in the same git repo, right? What has changed a bit is instead of having test suites with defined variables within the openQA database now it is mainly (or additionally) the path to the schedule file.
Yes, that is what I meant: the variables and scheduled tests are in one place instead of the variables being in the openQA database and the tests being scheduled in a relatively intransparent way in main.pm.
Updated by okurz about 4 years ago
dancermak wrote:
Yes, that is what I meant: the variables and scheduled tests are in one place instead of the variables being in the openQA database and the tests being scheduled in a relatively intransparent way in main.pm.
but this is unfortunately not true. Each YAML file describes a single scenario so you need multiple files to define multiple scenarios. The "main.pm" is the same for all scenarios so there are multiple logical paths starting within the same file. What is same for both approaches is that they describe which test modules and in which order they are started for each scenario. But both do not define the test suite either which is still in the openQA database.
This is why I linked the ticket #58184 . If you want to have more defined within the same git repo one can use either dump-templates+load-templates, e.g. Fedora are using their complete tests definitions this way, or use job template testsuite definitions, store the job templates within the same git repo and load over API, e.g. with a post-merge CI job. The latter is what more and more SUSE QE teams are doing internally but just storing the job template documents elsewhere.
Updated by okurz about 4 years ago
- Subject changed from Please move the declarative/yaml scheduler to openQA upstream for os-autoinst-distri-opensuse to Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
Updated by tjyrinki_suse over 3 years ago
- Status changed from New to Feedback
- Target version set to Ready
- Start date deleted (
2020-11-06)
Updated by tjyrinki_suse over 3 years ago
- Subject changed from Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [tools] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
- Category deleted (
Enhancement to existing tests) - Target version deleted (
Ready)
Updated by tjyrinki_suse over 3 years ago
- Category set to Enhancement to existing tests
(sorry for the noise)
Updated by okurz over 3 years ago
- Subject changed from [tools] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
- Target version set to future
please consider #77071#note-2
okurz wrote:
I think in general this is a good idea. I see this related to our ticet #58184 . Likely the right location for this support would be os-autoinst where the main.pm is parsed and instead a YAML based schedule could be loaded. Maybe the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution?
Updated by szarate over 3 years ago
- Subject changed from [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
Updated by okurz over 3 years ago
- Subject changed from Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
- Priority changed from Normal to Low
Please don't just remove the team's tag without any additional comment. As suggested in #77071#note-2 I suggest that the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution.
Updated by szarate over 3 years ago
- Subject changed from [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
The team decided that this ticket does not fall within their responsabilities.
Updated by okurz over 3 years ago
just make sure it disappears from the triage list.
Updated by maritawerner over 3 years ago
This ticket is now unlabled and shows up in the list of unreviewed tickets. What would be the right label then?
Updated by szarate over 3 years ago
okurz wrote:
just make sure it disappears from the triage list.
Which of the so many?
I can add: [for-team-that-does-whatever-other-teams-are-not-doing-and-work-with-repo-maintenance-rather-than-tests-themselves] or [fttdwotandawwrmrttt], that'd take it out of most lists, as it would with the current one.
- Note I realize the silliness of the ping pong :)
maritawerner wrote:
This ticket is now unlabled and shows up in the list of unreviewed tickets. What would be the right label then?
Updated by okurz over 3 years ago
- Subject changed from Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [tools] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
- Status changed from Feedback to New
szarate wrote:
okurz wrote:
just make sure it disappears from the triage list.
Which of the so many?
I only know https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging
I can add: [for-team-that-does-whatever-other-teams-are-not-doing-and-work-with-repo-maintenance-rather-than-tests-themselves] or [fttdwotandawwrmrttt], that'd take it out of most lists, as it would with the current one.
The current one, meaning no "[" means it shows up in the todo list referenced on https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging . And that todo-list has the target to show no tickets. You can add such silly labels for sure if you think this is better than making a decision which team should do it.
Updated by szarate over 3 years ago
- Subject changed from [tools] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [tools][qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
- Description updated (diff)
- Target version changed from future to QE-Core: Ready
Updated by tjyrinki_suse almost 3 years ago
- Priority changed from Low to Normal
Updated by tjyrinki_suse almost 3 years ago
- Status changed from New to Workable
Updated by tjyrinki_suse almost 3 years ago
- Target version deleted (
QE-Core: Ready)
Updated by okurz almost 3 years ago
- Priority changed from Normal to Low
- Target version set to future
@tjyrinki_suse I suggest to provide a comment why you removed the ticket from your backlog again. With this we can only assume. I assume that you don't agree with the statements previously made and hope again that somebody else like the tools team takes over. I will set it to "future" therefore to explicitly state that I think it's a valid idea but not something we plan to do right now. I don't know what it means for you that you left "[qe-core]" in the subject though.
Updated by szarate almost 3 years ago
- Subject changed from [tools][qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
- Priority changed from Low to Normal
okurz wrote:
@tjyrinki_suse I suggest to provide a comment why you removed the ticket from your backlog again. With this we can only assume. I assume that you don't agree with the statements previously made and hope again that somebody else like the tools team takes over. I will set it to "future" therefore to explicitly state that I think it's a valid idea but not something we plan to do right now. I don't know what it means for you that you left "[qe-core]" in the subject though.
We're taking it :), this was why the priority changed
Updated by okurz almost 3 years ago
oh. I see. I wasn't triggered by the change in priority but by the removal of the target version because "[tools]" in the subject line and no target version sets means for us that we need to at least triage the ticket. This is when it showed up on https://os-autoinst.github.io/qa-tools-backlog-assistant/ which uses the query Untriaged "tools" tagged as referenced from https://progress.opensuse.org/projects/qa/wiki/Tools#Target-numbers-or-guideline-should-be-in-priorities
Updated by szarate over 2 years ago
- Target version changed from future to QE-Core: Ready
Updated by apappas over 2 years ago
Meeting: This has to be reevaluated from the side of qe-core. Most like we are going to go the "wheels" route.
Updated by szarate about 2 years ago
apappas wrote:
Meeting: This has to be reevaluated from the side of qe-core. Most like we are going to go the "wheels" route.
Not wheels but either we'll add it to os-autoinst, or we move it to it's own cpan module, instead of keeping it in the test-distribution
Updated by mgrifalconi 9 months ago
- Sprint changed from QE-Core: February Sprint 24 (Jan 31 - Feb 28) to QE-Core: March Sprint 24 (Mar 06 - Mar 28)