action #71758: [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folder - openQA Project - openSUSE Project Management Tool

Actions

Copy link

action #71758

closed

coordination #58184: [saga][epic][use case] full version control awareness within openQA

[spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folder

Added by okurz over 3 years ago. Updated almost 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Xiaojing_liu

Category:

Feature requests

Target version:

Ready

Start date:

Due date:

% Done:

Estimated time:

Tags:

spike, timeboxed, yaml

Description

Motivation¶

See #58184 . As a test maintainer using openQA I want openQA to be able to load the complete test definition from a yaml schedule in a local test distribution folder so that all the necessary configuration can come from a central place and no configuration needs to be added to the openQA database beforehand

Acceptance criteria¶

AC1: proof-of-concept exists for a complete test definition consisting of at least two scenarios (1) loadable from text files within the local test distribution folder

Suggestions¶

See https://github.com/os-autoinst/openQA/pull/2706 for a draft of the idea

References¶

Note 1 in AC1: A single job can be completely specified by test parameters on job creation time already. So the necessary challenge is that we have test definitions as commonly done by the combination of medium types, test suites, job templates, job groups (and machines and worker config) to form a test schedule consisting of multiple scenarios, i.e. at least two

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by okurz over 3 years ago

Description updated (diff)

Actions

Copy link

Updated by okurz over 3 years ago

Subject changed from [spike][timeboxed:20h] complete test definition from yaml schedule in local test distribution folder to [spike][timeboxed:20h] complete test definition from a yaml schedule in local test distribution folder

Actions

Copy link

Updated by okurz over 3 years ago

Subject changed from [spike][timeboxed:20h] complete test definition from a yaml schedule in local test distribution folder to [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folder

Actions

Copy link

Updated by livdywan over 3 years ago

Assignee set to livdywan

Actions

Copy link

Updated by livdywan over 3 years ago

Assignee deleted (~~livdywan~~)

I'll unassign myself since I rather focus on flaky CI and issues with production workers atm. If someone beats me to it, go for it.

Actions

Copy link

Updated by okurz over 3 years ago

Description updated (diff)

Added suggestion instead of just "reference".

Actions

Copy link

Updated by livdywan about 3 years ago

Some links for reference:

https://gitlab.suse.de/qsf-y/qa-sle-functional-y/-/blob/3ef279f9d4a536c3004c292442f8e50d48c024d1/JobGroups/test_suites.yaml

As discussed:

Let's assume we have nothing defined in the database
Think of how to do ISOs post from a file in a git repo
Keep in mind the user who just sets up a PR to run tests
Propose a format to load a test. We can change the format later, we have schemas for that and ways to communicate new versions supported by API routes.
Consider multi-machine tests - but the first version doesn't have to support all use cases

Actions

Copy link

Updated by Xiaojing_liu about 3 years ago

Assignee set to Xiaojing_liu

I spent some time learning this ticket, so I would like to try.

Actions

Copy link

Updated by openqa_review about 3 years ago

Due date set to 2021-04-23

Setting due date based on mean cycle time of SUSE QE Tools

Actions

Copy link

#10

Updated by okurz about 3 years ago

Status changed from Workable to In Progress

https://github.com/os-autoinst/openQA/pull/3848

Actions

Copy link

#11

Updated by Xiaojing_liu almost 3 years ago

Due date deleted (~~2021-04-23~~)
Status changed from In Progress to Workable
Assignee deleted (~~Xiaojing_liu~~)

I committed above pr to describe my thought.

we could schedule more than two jobs at one time from a file (e.g. from a yaml file), and the dependency will be created.
the format of the file needs to be confirmed with others (members of the tool team or users?)

I am not sure if the pr is in the right way.
This ticket is [spike] [timeboxed:20h], and I have spent more than 20 hours, I unassign it to wait for more feedback.

Actions

Copy link

#12

Updated by livdywan almost 3 years ago

Xiaojing_liu wrote:

I committed above pr to describe my thought.

we could schedule more than two jobs at one time from a file (e.g. from a yaml file), and the dependency will be created.

the format of the file needs to be confirmed with others (members of the tool team or users?)

I am not sure if the pr is in the right way.
This ticket is [spike] [timeboxed:20h], and I have spent more than 20 hours, I unassign it to wait for more feedback.

For the record, it's a spike, so there is no correct way. Thank you for coming up with a very nice proof of concept here!

I like how this is very similar to our existing YAML and mirrors the definitions in the database. So even without new documentation I know what it does.
Maybe as a user I'd want separate files here, for instance a separate file for the job templates so that I can re-use the machine definitions easily.
The code has a comment about cloning from a git repo. Maybe we could even allow a PR or branch name here e.g. #branchname to support a review/CI workflow.

Actions

Copy link

#13