Project

General

Profile

action #77071

[tools][qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse

Added by dancermak 12 months ago. Updated 3 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Enhancement to existing tests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

The declarative YAML scheduler is a pretty useful thing to have, but it is currently only available in the os-autoinst-distri-opensuse repository. Would it be possible to move it to the upstream test api, so that it can be used by other test users as well?

Suggestions

Start by giving a look at: https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/scheduler.pm


Related issues

Related to openQA Project - coordination #58184: [saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches, fully versioned test schedules and configuration settingsBlocked2018-03-23

History

#1 Updated by okurz 12 months ago

  • Related to coordination #58184: [saga][epic][use case] full version control awareness within openQA, e.g. user forks and branches, fully versioned test schedules and configuration settings added

#2 Updated by okurz 12 months ago

  • Project changed from openQA Project to openQA Tests
  • Category changed from Feature requests to Enhancement to existing tests

I think in general this is a good idea. I see this related to our ticet #58184 . Likely the right location for this support would be os-autoinst where the main.pm is parsed and instead a YAML based schedule could be loaded. Maybe the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution?

#3 Updated by dancermak 12 months ago

okurz wrote:

Maybe the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution?

I've cc'd the authors of the declarative scheduler. Would that be something that you could look into?

@AdamWill: if this lands upstream, this might be interesting for openqa.fedoraproject.org.

#4 Updated by AdamWill 12 months ago

Hum, possibly...

We already by convention define test suites by editing the templates files (not using the web UI), we just always have. I invented an intermediate format earlier this year that makes it a bit more convenient, see the explanation at top of https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/fifloader.py . We sort of have some kind of primitive 'declarative scheduling' already in our loader code: if a test defines the ENTRYPOINT var, our main.pm treats it as a space-separated list of test modules to load, loads those, and doesn't load anything else. There's also a few other vars - PREUPGRADE, PREINSTALL POSTINSTALL - which just tell main.pm to add in some specified test modules at certain points in the normal loading flow.

This scheduler is an interesting idea, I'm curious what the trigger for it was. The docs seem to sort of imply it might be "oh dear, main.pm is getting complicated and no-one remembers what half of it does"...but then as you keep reading the declarative scheduler doc, it seems to be well on the way to introducing quite a lot of complexity of its own :) I suspect the key factor in how much complexity your scheduling configuration has is "how many completely different tests you're trying to run across how many different distros and platforms", not really "how your scheduler is designed". But it's a nice idea. I don't know that it'd necessarily be a clear improvement for Fedora's distri, but then I probably know our main.pm better than anyone and maybe to someone else it's a confusing mess.

#5 Updated by riafarov 12 months ago

AdamWill wrote:

Hum, possibly...

We already by convention define test suites by editing the templates files (not using the web UI), we just always have. I invented an intermediate format earlier this year that makes it a bit more convenient, see the explanation at top of https://pagure.io/fedora-qa/os-autoinst-distri-fedora/blob/master/f/fifloader.py . We sort of have some kind of primitive 'declarative scheduling' already in our loader code: if a test defines the ENTRYPOINT var, our main.pm treats it as a space-separated list of test modules to load, loads those, and doesn't load anything else. There's also a few other vars - PREUPGRADE, PREINSTALL POSTINSTALL - which just tell main.pm to add in some specified test modules at certain points in the normal loading flow.

This scheduler is an interesting idea, I'm curious what the trigger for it was. The docs seem to sort of imply it might be "oh dear, main.pm is getting complicated and no-one remembers what half of it does"...but then as you keep reading the declarative scheduler doc, it seems to be well on the way to introducing quite a lot of complexity of its own :) I suspect the key factor in how much complexity your scheduling configuration has is "how many completely different tests you're trying to run across how many different distros and platforms", not really "how your scheduler is designed". But it's a nice idea. I don't know that it'd necessarily be a clear improvement for Fedora's distri, but then I probably know our main.pm better than anyone and maybe to someone else it's a confusing mess.

Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again. Even though still very isolated. Other things are just a support of yaml native features, like import of other files, etc. The main problem which is still there, is that we have tons of variables being still set in main.pm, and it's very clear to us that pair->key structure doesn't allow to store test data in a readable way, especially when mixed with environment settings. So I believe we are in a transitional state still.
I remember checking fedora's solution, but what you have shared wasn't yet available in February 2019, when we have decided to finally address this issue...
And as you have correctly stated, well maintained smaller entry points would not bring us to this problem at all.

#6 Updated by AdamWill 12 months ago

I mean, it's easy enough to see the problem: the openQA loading approach is very unorthodox. It's based on a key assumption that's becoming less and less valid: that all the tests it's loading will be variations on a theme. It was designed for installer testing - it obviously works best when all the tests are basically "boot a virtual machine, run an installer, boot the installed system and check something" and the differences are only in the details. With that assumption, the variable-defined approach makes a lot of sense. The variables describe key attributes of each test within the framework of the base assumption, and the loader can do a lot of heavy lifting with just a few variables worth of information.

The further you get away from that by introducing tests that don't use the installer at all or really have much in common with each other at all, the less sense a variable-based loading system makes and the more you find yourself wanting to do stuff like, well, have variables that just straight-up tell the loader what to do :) But there still are ways the usefulness of the variable-based system show themselves, like when you want to schedule a "check X is really set to Y" for all tests that set X, stuff like that. And we both do still have our whole installer test suites that are built around the initial assumption which still makes quite a lot of sense, and it'd be a lot of spade work to redo all those, I guess.

Oh well, just thinking out loud...

#7 Updated by riafarov 12 months ago

AdamWill wrote:

I mean, it's easy enough to see the problem: the openQA loading approach is very unorthodox. It's based on a key assumption that's becoming less and less valid: that all the tests it's loading will be variations on a theme. It was designed for installer testing - it obviously works best when all the tests are basically "boot a virtual machine, run an installer, boot the installed system and check something" and the differences are only in the details. With that assumption, the variable-defined approach makes a lot of sense. The variables describe key attributes of each test within the framework of the base assumption, and the loader can do a lot of heavy lifting with just a few variables worth of information.

The further you get away from that by introducing tests that don't use the installer at all or really have much in common with each other at all, the less sense a variable-based loading system makes and the more you find yourself wanting to do stuff like, well, have variables that just straight-up tell the loader what to do :) But there still are ways the usefulness of the variable-based system show themselves, like when you want to schedule a "check X is really set to Y" for all tests that set X, stuff like that. And we both do still have our whole installer test suites that are built around the initial assumption which still makes quite a lot of sense, and it'd be a lot of spade work to redo all those, I guess.

Oh well, just thinking out loud...

Oh, wow. That's very close to how I have explained it to myself. It's just that we got a lot of tests, big variety of products and openQA doesn't scale well to support all of it. Next thing we plan to go is some sort of custom grouping inside of the job group, as we have 350+ scenarios in ours.
One historical detail about declarative scheduling is that it was decided to have it in the test repo, as it solved the problem we had as a test maintainers. Tools team has provided solution which allowed us to set settings per machine and therefore define corresponding schedules for the different backends and installation, where previously we would pretend that "it's the same test suite".

What still might be interesting for you is the script to generate yaml schedule from the existing job run: https://github.com/os-autoinst/os-autoinst-distri-opensuse/tree/master/script/yaml_generator

#8 Updated by dancermak 12 months ago

riafarov wrote:

Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.

Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make main.pm schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet another main.pm.

I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database. Adam achieved this for the most part via the fifloader, but I must admit that I find the yaml scheduler more user friendly (albeit it would result in more duplication).

#9 Updated by okurz 12 months ago

dancermak wrote:

Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.

Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make main.pm schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet another main.pm.

riafarov only mentioned the "conditional schedule", not "declarative schedule".

The YAML test schedule certainly allows an easier introduction for people coming to os-autoinst-distri-opensuse but it can not take away the complexity of so many different test definitions while trying to cover multiple distributions, products, versions, systems and architectures. This only shifts the problem into another domain. Less pain for the individual test contributor, more pain for whoever tries to streamline the test schedules among all the different scenarios.

I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database

yes, but that did not change because main.pm was always there in the same git repo, right? What has changed a bit is instead of having test suites with defined variables within the openQA database now it is mainly (or additionally) the path to the schedule file.

#10 Updated by dancermak 11 months ago

okurz wrote:

dancermak wrote:

Hi Adam! Seems you got it right, and I'm with you there that conditional schedule was a step in the wrong direction, which allows making it a mess again.

Oh! That's honestly a surprise to me, because I find the declarative yaml schedule much simpler to use for me as a test maintainer than trying to somehow make main.pm schedule the correct tests. Sometimes I wish that it would be a bit more powerful, but then again, I don't want it to become yet another main.pm.

riafarov only mentioned the "conditional schedule", not "declarative schedule".

The YAML test schedule certainly allows an easier introduction for people coming to os-autoinst-distri-opensuse but it can not take away the complexity of so many different test definitions while trying to cover multiple distributions, products, versions, systems and architectures. This only shifts the problem into another domain. Less pain for the individual test contributor, more pain for whoever tries to streamline the test schedules among all the different scenarios.

Hm, that's probably correct. I unfortunately cannot really judge that, as I'm a individual test contributor and not someone who maintains the whole test suite.

I see the yaml scheduler as a step in the right direction: have all information about tests in the test git repository and not parts in the repo and parts in the openQA settings database

yes, but that did not change because main.pm was always there in the same git repo, right? What has changed a bit is instead of having test suites with defined variables within the openQA database now it is mainly (or additionally) the path to the schedule file.

Yes, that is what I meant: the variables and scheduled tests are in one place instead of the variables being in the openQA database and the tests being scheduled in a relatively intransparent way in main.pm.

#11 Updated by okurz 11 months ago

dancermak wrote:

Yes, that is what I meant: the variables and scheduled tests are in one place instead of the variables being in the openQA database and the tests being scheduled in a relatively intransparent way in main.pm.

but this is unfortunately not true. Each YAML file describes a single scenario so you need multiple files to define multiple scenarios. The "main.pm" is the same for all scenarios so there are multiple logical paths starting within the same file. What is same for both approaches is that they describe which test modules and in which order they are started for each scenario. But both do not define the test suite either which is still in the openQA database.

This is why I linked the ticket #58184 . If you want to have more defined within the same git repo one can use either dump-templates+load-templates, e.g. Fedora are using their complete tests definitions this way, or use job template testsuite definitions, store the job templates within the same git repo and load over API, e.g. with a post-merge CI job. The latter is what more and more SUSE QE teams are doing internally but just storing the job template documents elsewhere.

#12 Updated by okurz 11 months ago

  • Subject changed from Please move the declarative/yaml scheduler to openQA upstream for os-autoinst-distri-opensuse to Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse

#13 Updated by tjyrinki_suse 6 months ago

  • Status changed from New to Feedback
  • Target version set to Ready
  • Start date deleted (2020-11-06)

#14 Updated by tjyrinki_suse 6 months ago

  • Subject changed from Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [tools] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
  • Category deleted (Enhancement to existing tests)
  • Target version deleted (Ready)

#15 Updated by tjyrinki_suse 6 months ago

  • Category set to Enhancement to existing tests

(sorry for the noise)

#16 Updated by okurz 6 months ago

  • Subject changed from [tools] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
  • Target version set to future

please consider #77071#note-2

okurz wrote:

I think in general this is a good idea. I see this related to our ticet #58184 . Likely the right location for this support would be os-autoinst where the main.pm is parsed and instead a YAML based schedule could be loaded. Maybe the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution?

#17 Updated by szarate 4 months ago

  • Subject changed from [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse

#18 Updated by okurz 4 months ago

  • Subject changed from Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
  • Priority changed from Normal to Low

Please don't just remove the team's tag without any additional comment. As suggested in #77071#note-2 I suggest that the expert users of os-autoinst-distri-opensuse can provide that support and try to upstream the solution.

#19 Updated by szarate 4 months ago

  • Subject changed from [qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse

The team decided that this ticket does not fall within their responsabilities.

#20 Updated by okurz 4 months ago

just make sure it disappears from the triage list.

#21 Updated by maritawerner 4 months ago

This ticket is now unlabled and shows up in the list of unreviewed tickets. What would be the right label then?

#22 Updated by szarate 4 months ago

okurz wrote:

just make sure it disappears from the triage list.

Which of the so many?

I can add: [for-team-that-does-whatever-other-teams-are-not-doing-and-work-with-repo-maintenance-rather-than-tests-themselves] or [fttdwotandawwrmrttt], that'd take it out of most lists, as it would with the current one.

  • Note I realize the silliness of the ping pong :)

maritawerner wrote:

This ticket is now unlabled and shows up in the list of unreviewed tickets. What would be the right label then?

#23 Updated by okurz 4 months ago

  • Subject changed from Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [tools] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
  • Status changed from Feedback to New

szarate wrote:

okurz wrote:

just make sure it disappears from the triage list.

Which of the so many?

I only know https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging

I can add: [for-team-that-does-whatever-other-teams-are-not-doing-and-work-with-repo-maintenance-rather-than-tests-themselves] or [fttdwotandawwrmrttt], that'd take it out of most lists, as it would with the current one.

The current one, meaning no "[" means it shows up in the todo list referenced on https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-triaging . And that todo-list has the target to show no tickets. You can add such silly labels for sure if you think this is better than making a decision which team should do it.

#24 Updated by szarate 3 months ago

  • Subject changed from [tools] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse to [tools][qe-core] Please move the declarative/yaml test flow scheduler to openQA upstream for os-autoinst-distri-opensuse
  • Description updated (diff)
  • Target version changed from future to QE-Core: Ready

Also available in: Atom PDF