action #124502
closedcoordination #58184: [saga][epic][use case] full version control awareness within openQA
coordination #48641: [epic] Trigger openQA tests in pull requests of any product github pull request
coordination #124466: [epic] Put open points from okurz's hackweek 22 project into proper tickets
[spike][timeboxed:20h] complete test definition from yaml schedule in git checked out test distribution
0%
Description
Motivation¶
See #58184#note-23 . What I think we want in the future is: A github action workflow like in https://github.com/os-autoinst/os-autoinst-distri-example/compare/master...okurz:os-autoinst-distri-example:feature/hackweek22_trigger_openqa_in_ci can be used to trigger 1..N openQA tests from this git repo on https://openqa.opensuse.org without having anything pre-defined in the openQA database nor the filesystem.
Acceptance criteria¶
- AC1: proof-of-concept exists for a complete test definition consisting of at least two scenarios (1) loadable from text files within a git test distribution folder without having anything pre-defined in the openQA database nor the local filesystem on openQA
Suggestions¶
- Consider initial notes from #58184#note-10 as well as #58184#note-22 regarding the use cases and #58184#note-18 regarding implementation approaches
- See https://github.com/os-autoinst/openQA/pull/2706/files
Further details¶
I don't know why in #58184#note-23 I did not actually create a ticket right away. Maybe another ticket covering this already exists and I am unable to find it.
Updated by okurz over 1 year ago
- Copied from action #71758: [spike][timeboxed:20h] complete test definition from a yaml schedule file in local test distribution folder added
Updated by okurz over 1 year ago
- Related to action #92311: Complete test definition from a single yaml schedule file in local test distribution folder added
Updated by mkittler over 1 year ago
- Status changed from New to In Progress
This is my current draft: https://github.com/Martchus/os-autoinst-distri-example/pull/1
It basically works to trigger a job. With https://github.com/os-autoinst/openQA/pull/5016 I was able to overcome the variable substitution problem.
Note that I'm testing this via an openQA instance hosted on my notebook that is exposed via the NGINX server in my local network to the public. I'm not sure how I'd test this otherwise (because the only public openQA instance we have is o3 but I'm not going to play around with my not-yet-merged changes for openQA in production). So I guess I'm lucky that I'm doing home office with a dedicated public IPv4. Or have we publicly exposed hosts that could be used for testing things like this?
By the way, for putting an openQA instance behind NGINX using a different root path, the Mojo plugin Mojolicious::Plugin::RequestBase
(enabled via plugins = RequestBase
in openqa.ini
) was very useful. It doesn't seem to work for DataTables, though.
The PR for #92311 (https://github.com/os-autoinst/openQA/pull/4999) is also in good shape so the next step would be to define a meaningful scenario for os-autoinst-distri-example via YAML and use that in the GitHub action (instead of just posting a single job).
Updated by openqa_review over 1 year ago
- Due date set to 2023-03-07
Setting due date based on mean cycle time of SUSE QE Tools
Updated by mkittler over 1 year ago
I have it working in the sense that with https://github.com/Martchus/os-autoinst-distri-example/pull/1 I am able to:
- trigger jobs by scheduling a product.
- waiting until all jobs are done/cancelled and returning an exit code depending on the results.
- pass all settings as wanted, e.g.
CASEDIR
from Git and a relativeNEEDLES_DIR
within the same Git repo. - references to
CASEDIR
andNEEDLES_DIR
are generally handled correctly by the openQA worker (withNEEDLES_DIR=%25%25CASEDIR%25%25/needles
syntax).
An example output looks like this:
Parameters for job creation: -d CASEDIR=https://github.com/Martchus/os-autoinst-distri-example.git#1/merge -d NEEDLES_DIR=%25%25CASEDIR%25%25/needles -d BUILD=Martchus/os-autoinst-distri-example.git#1/merge -d DISTRI=example -d GROUP_ID=0 -d VERSION=0 -d FLAVOR=DVD -d ARCH=x86_64 -d TEST=simple_boot --data-urlencode SCENARIO_DEFINITIONS_YAML@scenaio-definitions.yaml ***/api/v1/isos
Posted job with IDs: 3535
Job state of job ID 3535: running, waiting…
Job ID 3535 is 'done' with result 'incomplete'
Error: Process completed with exit code 10.
This test incompleted due to https://bugzilla.opensuse.org/show_bug.cgi?id=1208248. However, even without this bug it likely would have ended up incomplete as I still have to sort out a few things. What's missing is basically:
- that os-autoinst actually checks out the specified
CASEDIR
correctly as it so far doesn't checks out the PR branch. I'll either have to change how theCASEDIR
is specified or how os-autoinst does the checkout. - that there's actually a meaningful test scenario present. However, so far it isn't clear to me where e.g. the "ISO" is supposed to come from and whether that's within the scope of this ticket.
Updated by mkittler over 1 year ago
This PR would fix the first missing point: https://github.com/os-autoinst/os-autoinst/pull/2263 - see its commit message for details
Updated by mkittler over 1 year ago
- Status changed from In Progress to Feedback
The last two issues are resolved as well:
- The PR https://github.com/os-autoinst/os-autoinst/pull/2263 is in a mergable state. It works for me in conjunction with https://github.com/Martchus/os-autoinst-distri-example/pull/1 and I've also extended unit tests in a way to ensure the fallback for checking out specific commits still works.
- I've been working around https://bugzilla.opensuse.org/show_bug.cgi?id=1208248 and changed the test code so it'll conclude early if there's no bootable media. I think this is a sufficient scenario for an example distribution (considering we also have the openQA-in-openQA test for a more sophisticated example). With that I've got a green CI run on https://github.com/Martchus/os-autoinst-distri-example/pull/1. The case of a failue is also sufficiently tested by now.
So I'd leave it at that for this spike/timeboxed solution.
Updated by mkittler over 1 year ago
The PR https://github.com/os-autoinst/os-autoinst/pull/2263 has been merged. That leaves the following open points:
- Move script for triggering (https://github.com/Martchus/os-autoinst-distri-example/pull/1) into a separate, re-usable repo or change the openQA-API so the script could be a one-liner. I don't think the latter is not possible because the script will always have to poll and evaluate JSON and deal and do proper error handling. That's in my opinion just too much for something we'd encourage people to C&P. So I'd file a ticket for moving the script into a separate repo, making it a reusable GitHub action.
- Another time-boxed task to build the monitoring-shell script from https://github.com/Martchus/os-autoinst-distri-example/pull/1 into openqa-cli (with Perl code).
- Created #125720 for that.
- The client could also validate the YAML locally. The schema is already packaged with it. I added a remark about it on #92311#note-9.
- Provide a ready-to-use container with all dependencies to create and monitor openQA jobs from GitHub actions.
- Created #125723 for that.
- Alternative: Create a GitHub-action repo
- Another time-boxed task to build the monitoring-shell script from https://github.com/Martchus/os-autoinst-distri-example/pull/1 into openqa-cli (with Perl code).
- Conduct a walk though of https://github.com/os-autoinst/openQA/pull/4999 with @okurz and others. There are already small remarks to be fixed (that I could still fix within the 20 hours). If the whole code needs to be rewritten due to some crazy ideas we need a new ticket for that, though.
- The outcome of the walk tough is to handle remaining points as stated in #92311#note-8.
Updated by mkittler over 1 year ago
- Due date changed from 2023-03-07 to 2023-03-08
We're going to walk though the code tomorrow after the daily.
Updated by mkittler over 1 year ago
- Due date changed from 2023-03-08 to 2023-03-10
I hope we can have a walk though until Friday.
Updated by mkittler over 1 year ago
- Copied to action #125720: [spike][timeboxed:20h] Add monitoring-support into openqa-cli added
Updated by mkittler over 1 year ago
- Status changed from Feedback to Resolved
I've been updating the note with open points (#124502#note-9). All points are resolved after our discussion today and creating follow-up tickets accordingly.