QA » openQA Project

I tried creating a workflow that triggers when comments are created. The triggering generally works but there are a few problems:

1. The workflow isn't shown as part of the related PR (e.g. https://github.com/Martchus/os-autoinst-distri-opensuse/pull/3) but only under the "Actions" tab (e.g. https://github.com/Martchus/os-autoinst-distri-opensuse/actions). So we probably need to use the statuses API to get some in-progress/result checkmarks within the PR.
2. The event meta-data doesn't contain the PRs ref so we need to query that from our action.

Those are the changes I have so far for our GitHub action: https://github.com/os-autoinst/scripts/compare/master...Martchus:scripts:clone-on-comment

The scripts PR was merged so now the next step would be to merge https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19305.

The workflow is not triggered before merging the changes so I guess we cannot easily test this on the main repo before merging. But it worked when I tested it on my fork (where I added the changes already on the master branch). In case the token GitHub provides to workflows by default is not sufficient I will have to create a new token just for that - but it won't be a big deal as I have already tested this on my fork.

The PR was merged as well and I created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19371 to supply a token with enough permissions. I hope that'll be the last step.

Looks like it generally works now, e.g. jobs were cloned for https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19305#issuecomment-2127177246.

It failed to update the status and to clone the ref:

Unable to update status on GitHub via 'https://api.github.com/repos/Martchus/os-autoinst-distri-opensuse/statuses/acf5d851c6a6eead3605a324a14f3651117d54b0'.
{"message":"Not Found","documentation_url":"https://docs.github.com/rest/commits/statuses#create-a-commit-status"}

[2024-05-23T15:45:30.188302Z] [warn] [pid:74634] !!! main: Unable to clone Git repository 'https://github.com/https://github.com/Martchus/os-autoinst-distri-opensuse.git#acf5d851c6a6eead3605a324a14f3651117d54b0' specified via CASEDIR (see log for details) at /usr/lib/os-autoinst/OpenQA/Isotovideo/Utils.pm line 164.

Maybe the URL for updating the status needed to point to the repository under the original organization (os-autoinst) and not the fork (Martchus). And the URL of CASEDIR is just wrong in a very obvious way.

But there's another caveat: This will now spawn a GitHub action on every new/updated comment in PRs. This means setting up a container and doing some initial checks - even if no job is to be cloned. That's maybe a bit wasteful. Not sure whether this can be tweaked via the if:-condition in the workflow YAML.

mkittler wrote in #note-15:

Looks like it generally works now, e.g. jobs were cloned for https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19305#issuecomment-2127177246.

Really cool. One thing I noticed, there is no way to see the PR from the job itself or at least not easily.
This is not in the AC so just a suggestion, but it would be nice to see the PR in the settings for example.

But there's another caveat: This will now spawn a GitHub action on every new/updated comment in PRs. This means setting up a container and doing some initial checks - even if no job is to be cloned. That's maybe a bit wasteful. Not sure whether this can be tweaked via the if:-condition in the workflow YAML.

Maybe https://github.com/orgs/community/discussions/25389 (although I'd say it is fine as-is, too, if this is not straightforward to do)

I was actually able to find the contains()-syntax myself (in the official documentation). It works so I created https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19379 to use it.

All PRs merged, looks good so far: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19305#issuecomment-2133002967

EDIT: Unfortunately reporting the status back didn't work when I tested it on the open PR https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19378#issuecomment-2133222287.

The log contains the following error:

Unable to update status on GitHub via 'https://api.github.com/repos/os-autoinst/os-autoinst-distri-opensuse/statuses/32731e00569b74a04f4269c101da9d181cef8513'.
{"message":"Not Found","documentation_url":"https://docs.github.com/rest/commits/statuses#create-a-commit-status"}

I'm not sure why that's the case. The URL looks correct and the same worked on my fork.

The long build numbers are disturbing the dashboard. I think this should be handled better in general by the dashboard so I created https://github.com/os-autoinst/openQA/pull/5663 to limit the size of build labels on the dashboard.

mkittler wrote in #note-21:

The long build numbers are disturbing the dashboard. I think this should be handled better in general by the dashboard so I created https://github.com/os-autoinst/openQA/pull/5663 to limit the size of build labels on the dashboard.

That only happens though if builds are created as part of job groups which shouldn't happen for those jobs triggered as part of verification jobs. Maybe you just need to make those jobs groupless?

The jobs are intentionally in a specific job group.

Showing the status also works now after making the bot account an owner within the os-autoinst organization, see the green checkmark on https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19378. The back-reference also works, see CLONED_BY on https://openqa.opensuse.org/tests/4243617#settings.

PRs for documentation:

• https://github.com/os-autoinst/scripts/pull/325
• https://github.com/os-autoinst/openQA/pull/5670