action #122953
closed[qe-core] Extend openqa-cli to trigger multiple jobs (Add support for statistical investigation)
100%
Description
Too often, we need to either trigger multiple jobs to know if an intermittent test failure is gone or to try and catch a sporadic product bug or an issue somewhere in the infrastructure, so the story would go like this:
- As a developer, I would like the openQA client to trigger a list of jobs N times so I can catch a flaky testcase or reproduce a sporadic issue that doesn't affect other builds or JobGroups.
I'm looking for an extension of the openQA Client code; knowing how to extend the openQA Client will allow us to do even more in the future to support other squads.
Acceptance Criteria¶
- Solution is documented in https://open.qa/docs/ in a Section dedicated to statistical investigation
- Solution is packaged and distributed with openQA-client
- Job count is added to the name of the test
- Allow for a PR (CASEDIR) also to be triggered
Notes¶
- See https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation and Oliver's comment
- TL;DR https://github.com/okurz/scripts/blob/master/openqa-clone-set or https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation already achieve but built-in into openqa-cli?
- Initially we want a sensible amount of jobs, so we don't end up DoS'ing ourselves, so things like 1M jobs or even 1K, should be taken into consideration on how to approach that.
openqa-statistics-whatever $JOB $TIMES $JOB_SETTINGS
openqa-clone-job ... --job-count 15 ...
Updated by szarate almost 2 years ago
- Tags set to qe-core-january-sprint
- Project changed from openQA Project (public) to openQA Tests (public)
- Subject changed from [qe-core] Extend openqa-cli to trigger multiple jobs to [qe-core] Extend openqa-cli to trigger multiple jobs (Add support for statistical investigation)
- Description updated (diff)
- Category changed from Feature requests to Refactor/Code Improvements
Updated by szarate almost 2 years ago
- Sprint set to QE-Core: January Sprint (Jan 11 - Feb 08)
Updated by szarate almost 2 years ago
- The URL fragments can be cleaned up from the URL
Updated by okurz almost 2 years ago
- As a developer, I would like the openQA client to trigger a list of jobs N times so I can catch a flaky testcase or reproduce a sporadic issue that doesn't affect other builds or JobGroups.
Isn't that what
https://github.com/okurz/scripts/blob/master/openqa-clone-set or https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation already provide? We could consider moving those scripts to either https://github.com/os-autoinst/scripts/ or https://github.com/os-autoinst/openQA/ of course but I wouldn't start from script
Updated by szarate almost 2 years ago
- Description updated (diff)
okurz wrote:
- As a developer, I would like the openQA client to trigger a list of jobs N times so I can catch a flaky testcase or reproduce a sporadic issue that doesn't affect other builds or JobGroups.
Isn't that what
https://github.com/okurz/scripts/blob/master/openqa-clone-set or https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation already provide? We could consider moving those scripts to either https://github.com/os-autoinst/scripts/ or https://github.com/os-autoinst/openQA/ of course but I wouldn't start from script
It is, just not quite; I'd prefer if there's actual code extending the openQA-cli, whether the perl or python clients, rather than a bash script (I'm here mostly pushing the boundary to gain knowledge as a team on how to enhance the clients directly)
Updated by okurz almost 2 years ago
szarate wrote:
It is, just not quite; I'd prefer if there's actual code extending the openQA-cli, whether the perl or python clients, rather than a bash script (I'm here mostly pushing the boundary to gain knowledge as a team on how to enhance the clients directly)
So the actual requirement would be: Do what https://github.com/okurz/scripts/blob/master/openqa-clone-set or https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation already achieve but built-in into openqa-cli, right?
Updated by szarate almost 2 years ago
- Sprint changed from QE-Core: January Sprint (Jan 11 - Feb 08) to QE-Core: February Sprint (Feb 08 - Mar 08)
Updated by szarate almost 2 years ago
- Tags changed from qe-core-january-sprint to qe-core-january-sprint, platform-team
- Description updated (diff)
Updated by szarate almost 2 years ago
- Sprint changed from QE-Core: February Sprint (Feb 08 - Mar 08) to QE-Core: March Sprint (Mar 08 - Apr 05)
Updated by fgerling almost 2 years ago
- Status changed from Workable to In Progress
Updated by szarate over 1 year ago
- Sprint changed from QE-Core: March Sprint (Mar 08 - Apr 05) to QE-Core: April Sprint 23 (Apr 05 - May 03)
Updated by szarate over 1 year ago
- Tags changed from qe-core-january-sprint, platform-team, qe-core-february-sprint, qe-core-march-sprint to qe-core-january-sprint, platform-team, qe-core-february-sprint, qe-core-march-sprint, qe-core-october-sprint
- Status changed from In Progress to Workable
- Assignee deleted (
fgerling)
Updated by szarate about 1 year ago
- Sprint changed from QE-Core: April Sprint 23 (Apr 05 - May 03) to QE-Core: October Sprint 23 (Oct 11 - Nov 08)
- Description updated (diff)
Updated by szarate about 1 year ago
- Sprint changed from QE-Core: October Sprint 23 (Oct 11 - Nov 08) to QE-Core: September Sprint 23 (Sep 06 - Oct 04)
Updated by szarate about 1 year ago
- Sprint changed from QE-Core: September Sprint 23 (Sep 06 - Oct 04) to QE-Core: October Sprint 23 (Oct 11 - Nov 08)
Updated by amanzini about 1 year ago
my 2c:
As openqa-cli by design can be used for a number of API operations (get job status, change group, archive, restart/delete jobs, download assets and so on), I'm not sure if it can benefit from a --repeat N option. If we add it, we should allow user to specify the 'repeat' count only for some API calls, and then make the client less orthogonal.
So, while I appreciate and support the effort to explore the client codebase, to me seems like a functionality more useful to have in the openqa-clone-job wrapper.
Updated by amanzini about 1 year ago
- Status changed from Workable to In Progress
- Assignee set to amanzini
Updated by amanzini about 1 year ago
- % Done changed from 20 to 30
Updated by amanzini about 1 year ago
need some more working on unit testing and coverage
Updated by amanzini about 1 year ago
- % Done changed from 30 to 90
got interesting hints and useful advice on the PR, which should be mostly done
Updated by amanzini about 1 year ago
- Status changed from In Progress to Blocked
PR got approved by 3 of 4 engineers
Updated by amanzini about 1 year ago
- Status changed from Blocked to Closed
PR merged, closing ticket
Updated by dzedro about 1 year ago
- Status changed from Closed to Resolved
- % Done changed from 90 to 100
Updated by szarate about 1 year ago
- Tags changed from qe-core-january-sprint, platform-team, qe-core-february-sprint, qe-core-march-sprint, qe-core-october-sprint to qe-core-january-sprint, platform-team, qe-core-february-sprint, qe-core-march-sprint, qe-core-october-sprint, governance
Basically https://github.com/os-autoinst/openQA/pull/5331#discussion_r1362558824 we lost about a week arguing about a mere implementation detail on ternary operator, which spawned one other PR, making it an expensive discussion in terms of engineering hours with no added value.
As a result, our coding style guidelines should be enforced or at least reviewed by CI for all projects #96596 and #138416