Project

General

Profile

Actions

action #122953

closed

[qe-core] Extend openqa-cli to trigger multiple jobs (Add support for statistical investigation)

Added by szarate over 1 year ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Refactor/Code Improvements
Target version:
Start date:
2023-01-11
Due date:
% Done:

100%

Estimated time:
Difficulty:
Sprint:
QE-Core: October Sprint 23 (Oct 11 - Nov 08)

Description

Too often, we need to either trigger multiple jobs to know if an intermittent test failure is gone or to try and catch a sporadic product bug or an issue somewhere in the infrastructure, so the story would go like this:

  • As a developer, I would like the openQA client to trigger a list of jobs N times so I can catch a flaky testcase or reproduce a sporadic issue that doesn't affect other builds or JobGroups.

I'm looking for an extension of the openQA Client code; knowing how to extend the openQA Client will allow us to do even more in the future to support other squads.

Acceptance Criteria

  1. Solution is documented in https://open.qa/docs/ in a Section dedicated to statistical investigation
  2. Solution is packaged and distributed with openQA-client
  3. Job count is added to the name of the test
  4. Allow for a PR (CASEDIR) also to be triggered

Notes

  1. See https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation and Oliver's comment
  2. TL;DR https://github.com/okurz/scripts/blob/master/openqa-clone-set or https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation already achieve but built-in into openqa-cli?
  3. Initially we want a sensible amount of jobs, so we don't end up DoS'ing ourselves, so things like 1M jobs or even 1K, should be taken into consideration on how to approach that.

openqa-statistics-whatever $JOB $TIMES $JOB_SETTINGS
openqa-clone-job ... --job-count 15 ...

Actions #1

Updated by szarate over 1 year ago

  • Tags set to qe-core-january-sprint
  • Project changed from openQA Project to openQA Tests
  • Subject changed from [qe-core] Extend openqa-cli to trigger multiple jobs to [qe-core] Extend openqa-cli to trigger multiple jobs (Add support for statistical investigation)
  • Description updated (diff)
  • Category changed from Feature requests to Refactor/Code Improvements
Actions #2

Updated by szarate over 1 year ago

  • Sprint set to QE-Core: January Sprint (Jan 11 - Feb 08)
Actions #3

Updated by szarate over 1 year ago

  • The URL fragments can be cleaned up from the URL
Actions #4

Updated by okurz over 1 year ago

  • As a developer, I would like the openQA client to trigger a list of jobs N times so I can catch a flaky testcase or reproduce a sporadic issue that doesn't affect other builds or JobGroups.

Isn't that what
https://github.com/okurz/scripts/blob/master/openqa-clone-set or https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation already provide? We could consider moving those scripts to either https://github.com/os-autoinst/scripts/ or https://github.com/os-autoinst/openQA/ of course but I wouldn't start from script

Actions #5

Updated by szarate over 1 year ago

  • Description updated (diff)

okurz wrote:

  • As a developer, I would like the openQA client to trigger a list of jobs N times so I can catch a flaky testcase or reproduce a sporadic issue that doesn't affect other builds or JobGroups.

Isn't that what
https://github.com/okurz/scripts/blob/master/openqa-clone-set or https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation already provide? We could consider moving those scripts to either https://github.com/os-autoinst/scripts/ or https://github.com/os-autoinst/openQA/ of course but I wouldn't start from script

It is, just not quite; I'd prefer if there's actual code extending the openQA-cli, whether the perl or python clients, rather than a bash script (I'm here mostly pushing the boundary to gain knowledge as a team on how to enhance the clients directly)

Actions #6

Updated by szarate over 1 year ago

  • Description updated (diff)
Actions #7

Updated by okurz over 1 year ago

szarate wrote:

It is, just not quite; I'd prefer if there's actual code extending the openQA-cli, whether the perl or python clients, rather than a bash script (I'm here mostly pushing the boundary to gain knowledge as a team on how to enhance the clients directly)

So the actual requirement would be: Do what https://github.com/okurz/scripts/blob/master/openqa-clone-set or https://progress.opensuse.org/projects/openqatests/wiki/Wiki#Statistical-investigation already achieve but built-in into openqa-cli, right?

Actions #8

Updated by szarate over 1 year ago

  • Sprint changed from QE-Core: January Sprint (Jan 11 - Feb 08) to QE-Core: February Sprint (Feb 08 - Mar 08)
Actions #9

Updated by szarate over 1 year ago

  • Tags changed from qe-core-january-sprint to qe-core-january-sprint, platform-team
  • Description updated (diff)
Actions #10

Updated by szarate over 1 year ago

  • Status changed from New to Workable
Actions #11

Updated by szarate over 1 year ago

  • Sprint changed from QE-Core: February Sprint (Feb 08 - Mar 08) to QE-Core: March Sprint (Mar 08 - Apr 05)
Actions #12

Updated by fgerling over 1 year ago

  • Assignee set to fgerling
Actions #13

Updated by fgerling over 1 year ago

  • Status changed from Workable to In Progress
Actions #14

Updated by szarate about 1 year ago

  • Sprint changed from QE-Core: March Sprint (Mar 08 - Apr 05) to QE-Core: April Sprint 23 (Apr 05 - May 03)
Actions #15

Updated by szarate 10 months ago

  • Tags changed from qe-core-january-sprint, platform-team, qe-core-february-sprint, qe-core-march-sprint to qe-core-january-sprint, platform-team, qe-core-february-sprint, qe-core-march-sprint, qe-core-october-sprint
  • Status changed from In Progress to Workable
  • Assignee deleted (fgerling)
Actions #16

Updated by szarate 8 months ago

  • Sprint changed from QE-Core: April Sprint 23 (Apr 05 - May 03) to QE-Core: October Sprint 23 (Oct 11 - Nov 08)
  • Description updated (diff)
Actions #17

Updated by szarate 8 months ago

  • Sprint changed from QE-Core: October Sprint 23 (Oct 11 - Nov 08) to QE-Core: September Sprint 23 (Sep 06 - Oct 04)
Actions #18

Updated by szarate 8 months ago

  • Sprint changed from QE-Core: September Sprint 23 (Sep 06 - Oct 04) to QE-Core: October Sprint 23 (Oct 11 - Nov 08)
Actions #19

Updated by amanzini 8 months ago

my 2c:
As openqa-cli by design can be used for a number of API operations (get job status, change group, archive, restart/delete jobs, download assets and so on), I'm not sure if it can benefit from a --repeat N option. If we add it, we should allow user to specify the 'repeat' count only for some API calls, and then make the client less orthogonal.

So, while I appreciate and support the effort to explore the client codebase, to me seems like a functionality more useful to have in the openqa-clone-job wrapper.

Actions #20

Updated by amanzini 8 months ago

  • Status changed from Workable to In Progress
  • Assignee set to amanzini
Actions #21

Updated by amanzini 8 months ago

  • % Done changed from 0 to 20
Actions #22

Updated by amanzini 8 months ago

  • % Done changed from 20 to 30
Actions #23

Updated by amanzini 8 months ago

need some more working on unit testing and coverage

Actions #24

Updated by amanzini 8 months ago

  • % Done changed from 30 to 90

got interesting hints and useful advice on the PR, which should be mostly done

Actions #25

Updated by amanzini 8 months ago

  • Status changed from In Progress to Blocked

PR got approved by 3 of 4 engineers

Actions #26

Updated by amanzini 8 months ago

  • Status changed from Blocked to Closed

PR merged, closing ticket

Actions #27

Updated by dzedro 8 months ago

  • Status changed from Closed to Resolved
  • % Done changed from 90 to 100
Actions #28

Updated by szarate 7 months ago

  • Tags changed from qe-core-january-sprint, platform-team, qe-core-february-sprint, qe-core-march-sprint, qe-core-october-sprint to qe-core-january-sprint, platform-team, qe-core-february-sprint, qe-core-march-sprint, qe-core-october-sprint, governance

Basically https://github.com/os-autoinst/openQA/pull/5331#discussion_r1362558824 we lost about a week arguing about a mere implementation detail on ternary operator, which spawned one other PR, making it an expensive discussion in terms of engineering hours with no added value.

As a result, our coding style guidelines should be enforced or at least reviewed by CI for all projects #96596 and #138416

Actions

Also available in: Atom PDF