Project

General

Profile

Actions

action #124502

closed

coordination #58184: [saga][epic][use case] full version control awareness within openQA

coordination #48641: [epic] Trigger openQA tests in pull requests of any product github pull request

coordination #124466: [epic] Put open points from okurz's hackweek 22 project into proper tickets

[spike][timeboxed:20h] complete test definition from yaml schedule in git checked out test distribution

Added by okurz over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2023-02-14
Due date:
% Done:

0%

Estimated time:

Description

Motivation

See #58184#note-23 . What I think we want in the future is: A github action workflow like in https://github.com/os-autoinst/os-autoinst-distri-example/compare/master...okurz:os-autoinst-distri-example:feature/hackweek22_trigger_openqa_in_ci can be used to trigger 1..N openQA tests from this git repo on https://openqa.opensuse.org without having anything pre-defined in the openQA database nor the filesystem.

Acceptance criteria

  • AC1: proof-of-concept exists for a complete test definition consisting of at least two scenarios (1) loadable from text files within a git test distribution folder without having anything pre-defined in the openQA database nor the local filesystem on openQA

Suggestions

Further details

I don't know why in #58184#note-23 I did not actually create a ticket right away. Maybe another ticket covering this already exists and I am unable to find it.


Related issues 3 (0 open3 closed)

Related to openQA Project - action #92311: Complete test definition from a single yaml schedule file in local test distribution folderResolvedmkittler

Actions
Copied from openQA Project - action #71758: [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folderResolvedXiaojing_liu

Actions
Copied to openQA Project - action #125720: [spike][timeboxed:20h] Add monitoring-support into openqa-cliResolvedmkittler

Actions
Actions #1

Updated by okurz over 1 year ago

  • Copied from action #71758: [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folder added
Actions #2

Updated by mkittler over 1 year ago

  • Assignee set to mkittler
Actions #3

Updated by okurz over 1 year ago

  • Related to action #92311: Complete test definition from a single yaml schedule file in local test distribution folder added
Actions #4

Updated by mkittler over 1 year ago

  • Status changed from New to In Progress

This is my current draft: https://github.com/Martchus/os-autoinst-distri-example/pull/1

It basically works to trigger a job. With https://github.com/os-autoinst/openQA/pull/5016 I was able to overcome the variable substitution problem.

Note that I'm testing this via an openQA instance hosted on my notebook that is exposed via the NGINX server in my local network to the public. I'm not sure how I'd test this otherwise (because the only public openQA instance we have is o3 but I'm not going to play around with my not-yet-merged changes for openQA in production). So I guess I'm lucky that I'm doing home office with a dedicated public IPv4. Or have we publicly exposed hosts that could be used for testing things like this?

By the way, for putting an openQA instance behind NGINX using a different root path, the Mojo plugin Mojolicious::Plugin::RequestBase (enabled via plugins = RequestBase in openqa.ini) was very useful. It doesn't seem to work for DataTables, though.

The PR for #92311 (https://github.com/os-autoinst/openQA/pull/4999) is also in good shape so the next step would be to define a meaningful scenario for os-autoinst-distri-example via YAML and use that in the GitHub action (instead of just posting a single job).

Actions #5

Updated by openqa_review over 1 year ago

  • Due date set to 2023-03-07

Setting due date based on mean cycle time of SUSE QE Tools

Actions #6

Updated by mkittler over 1 year ago

I have it working in the sense that with https://github.com/Martchus/os-autoinst-distri-example/pull/1 I am able to:

  • trigger jobs by scheduling a product.
  • waiting until all jobs are done/cancelled and returning an exit code depending on the results.
  • pass all settings as wanted, e.g. CASEDIR from Git and a relative NEEDLES_DIR within the same Git repo.
  • references to CASEDIR and NEEDLES_DIR are generally handled correctly by the openQA worker (with NEEDLES_DIR=%25%25CASEDIR%25%25/needles syntax).

An example output looks like this:

Parameters for job creation: -d CASEDIR=https://github.com/Martchus/os-autoinst-distri-example.git#1/merge -d NEEDLES_DIR=%25%25CASEDIR%25%25/needles -d BUILD=Martchus/os-autoinst-distri-example.git#1/merge -d DISTRI=example -d GROUP_ID=0 -d VERSION=0 -d FLAVOR=DVD -d ARCH=x86_64 -d TEST=simple_boot --data-urlencode SCENARIO_DEFINITIONS_YAML@scenaio-definitions.yaml ***/api/v1/isos
Posted job with IDs: 3535
Job state of job ID 3535: running, waiting…
Job ID 3535 is 'done' with result 'incomplete'
Error: Process completed with exit code 10.

This test incompleted due to https://bugzilla.opensuse.org/show_bug.cgi?id=1208248. However, even without this bug it likely would have ended up incomplete as I still have to sort out a few things. What's missing is basically:

  • that os-autoinst actually checks out the specified CASEDIR correctly as it so far doesn't checks out the PR branch. I'll either have to change how the CASEDIR is specified or how os-autoinst does the checkout.
  • that there's actually a meaningful test scenario present. However, so far it isn't clear to me where e.g. the "ISO" is supposed to come from and whether that's within the scope of this ticket.
Actions #7

Updated by mkittler over 1 year ago

This PR would fix the first missing point: https://github.com/os-autoinst/os-autoinst/pull/2263 - see its commit message for details

Actions #8

Updated by mkittler over 1 year ago

  • Status changed from In Progress to Feedback

The last two issues are resolved as well:

So I'd leave it at that for this spike/timeboxed solution.

Actions #9

Updated by mkittler about 1 year ago

The PR https://github.com/os-autoinst/os-autoinst/pull/2263 has been merged. That leaves the following open points:

  • Move script for triggering (https://github.com/Martchus/os-autoinst-distri-example/pull/1) into a separate, re-usable repo or change the openQA-API so the script could be a one-liner. I don't think the latter is not possible because the script will always have to poll and evaluate JSON and deal and do proper error handling. That's in my opinion just too much for something we'd encourage people to C&P. So I'd file a ticket for moving the script into a separate repo, making it a reusable GitHub action.
    • Another time-boxed task to build the monitoring-shell script from https://github.com/Martchus/os-autoinst-distri-example/pull/1 into openqa-cli (with Perl code).
      • Created #125720 for that.
      • The client could also validate the YAML locally. The schema is already packaged with it. I added a remark about it on #92311#note-9.
    • Provide a ready-to-use container with all dependencies to create and monitor openQA jobs from GitHub actions.
      • Created #125723 for that.
      • Alternative: Create a GitHub-action repo
  • Conduct a walk though of https://github.com/os-autoinst/openQA/pull/4999 with @okurz and others. There are already small remarks to be fixed (that I could still fix within the 20 hours). If the whole code needs to be rewritten due to some crazy ideas we need a new ticket for that, though.
    • The outcome of the walk tough is to handle remaining points as stated in #92311#note-8.
Actions #10

Updated by mkittler about 1 year ago

  • Due date changed from 2023-03-07 to 2023-03-08

We're going to walk though the code tomorrow after the daily.

Actions #11

Updated by mkittler about 1 year ago

  • Due date changed from 2023-03-08 to 2023-03-10

I hope we can have a walk though until Friday.

Actions #12

Updated by mkittler about 1 year ago

  • Parent task changed from #58184 to #124466
Actions #13

Updated by mkittler about 1 year ago

  • Copied to action #125720: [spike][timeboxed:20h] Add monitoring-support into openqa-cli added
Actions #14

Updated by mkittler about 1 year ago

  • Status changed from Feedback to Resolved

I've been updating the note with open points (#124502#note-9). All points are resolved after our discussion today and creating follow-up tickets accordingly.

Actions #15

Updated by okurz about 1 year ago

  • Due date deleted (2023-03-10)
Actions

Also available in: Atom PDF