action #127949: [spike][timeboxed:20h] Research native GitHub for running openQA tests as CI checks size:M - openQA Project (public) - openSUSE Project Management Tool

Actions

Copy link

action #127949

closed

coordination #58184: [saga][epic][use case] full version control awareness within openQA

coordination #48641: [epic] Trigger openQA tests in pull requests of any product github pull request

[spike][timeboxed:20h] Research native GitHub for running openQA tests as CI checks size:M

Added by mkittler over 1 year ago. Updated over 1 year ago.

Status:

Resolved

Priority:

Low

Assignee:

mkittler

Category:

Feature requests

Target version:

Ready

Start date:

2023-04-19

Due date:

% Done:

Estimated time:

Description

Motivation¶

With #125723 we have created an easy way to run openQA tests as CI checks. This approach is platform-agnostic, e.g. it can run on GitHub, GitLab or any platform supporting CI runners (see http://open.qa/docs/#_running_openqa_jobs_as_ci_checks). However, it would be nice to have a more tight integration for GitHub, similar to the GitHub integration provided by OBS, CircleCI and Codecov. It had the advantage that the openQA icon would show up and a "Details" link could directly point to the corresponding openQA jobs (or scheduled product page). Note that OBS also supports GitLab. Supposedly supporting additional platforms in the future should be kept in mind as well.

Acceptance criteria¶

AC1: We know what changes would be necessary to which openQA components for openQA to support a native CI integration as mentioned under "Motivation".
AC2: We know what configuration would need to be done on GitHub.

Suggestions¶

Research the GitHub integration provided by OBS.
Ponder how the requirements would fit into openQA's architecture.
- Would it make sense to create a web UI plugin for this?
- Would it make sense to create a special CLI command for e.g. creating a token?
Come up with drafts for 2. if it is not too much effort.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by mkittler over 1 year ago

Description updated (diff)

Actions

Copy link

Updated by okurz over 1 year ago

Priority changed from Normal to Low
Target version set to Ready

Actions

Copy link

Updated by mkittler over 1 year ago

Subject changed from [spike][timeboxed:20h] Research native GitHub for running openQA tests as CI checks to [spike][timeboxed:20h] Research native GitHub for running openQA tests as CI checks size:M
Status changed from New to Workable

Actions

Copy link

Updated by okurz over 1 year ago

Copied to action #128360: Supporting fork based development model size:M added

Actions

Copy link

Updated by mkittler over 1 year ago

Status changed from Workable to In Progress
Assignee set to mkittler

Actions

Copy link

Updated by mkittler over 1 year ago

We basically need to create a new "webhook" API that evaluates the payload from GitHub. The payload would look like this:

{
  "action": "opened",
  "number": 5111,
  "pull_request": {
    "url": "https://api.github.com/repos/os-autoinst/openQA/pulls/5111",
    "id": …,
    "node_id": "…",
    "html_url": "https://github.com/os-autoinst/openQA/pull/5111",
    "diff_url": "https://github.com/os-autoinst/openQA/pull/5111.diff",
    "patch_url": "https://github.com/os-autoinst/openQA/pull/5111.patch",
    "issue_url": "https://api.github.com/repos/os-autoinst/openQA/issues/5111",
    "number": 5111,
    "state": "open",
    "locked": false,
    "title": "Fix several issues with `allocate_network`",
    "user": {
      "login": "Martchus",
      "id": …,
      "node_id": "…",
      "avatar_url": "https://avatars.githubusercontent.com/u/10248953?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/Martchus",
      "html_url": "https://github.com/Martchus",
      "followers_url": "https://api.github.com/users/Martchus/followers",
      "following_url": "https://api.github.com/users/Martchus/following{/other_user}",
      "gists_url": "https://api.github.com/users/Martchus/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/Martchus/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/Martchus/subscriptions",
      "organizations_url": "https://api.github.com/users/Martchus/orgs",
      "repos_url": "https://api.github.com/users/Martchus/repos",
      "events_url": "https://api.github.com/users/Martchus/events{/privacy}",
      "received_events_url": "https://api.github.com/users/Martchus/received_events",
      "type": "User",
      "site_admin": false
    },
    "body": "* Rewrite code to no longer use `next` in `catch` block (which\r\n  does not work because this block is actually a nested function)\r\n* Use raw SQL query to simplify code\r\n    * A simple insert with `ON CONFLICT DO NOTHING` should be\r\n      sufficient (instead of using a transaction)\r\n    * Using the same query for applying the network to the whole\r\n      job cluster should avoid running into\r\n      \"duplicate key value violates unique constraint\" (see\r\n      https://progress.opensuse.org/issues/128267#note-13)\r\n* Die in case of a fatal error (instead of possibly provoking the\r\n  error again and again in an endless loop)\r\n* Improve log messages to always include the relevant job ID\r\n* Add explicit unit tests",
    "created_at": "2023-05-02T15:52:07Z",
    "updated_at": "2023-05-02T15:52:07Z",
    "closed_at": null,
    "merged_at": null,
    "merge_commit_sha": null,
    "assignee": null,
    "assignees": [

    ],
    "requested_reviewers": [

    ],
    "requested_teams": [

    ],
    "labels": [

    ],
    "milestone": null,
    "draft": false,
    "commits_url": "https://api.github.com/repos/os-autoinst/openQA/pulls/5111/commits",
    "review_comments_url": "https://api.github.com/repos/os-autoinst/openQA/pulls/5111/comments",
    "review_comment_url": "https://api.github.com/repos/os-autoinst/openQA/pulls/comments{/number}",
    "comments_url": "https://api.github.com/repos/os-autoinst/openQA/issues/5111/comments",
    "statuses_url": "https://api.github.com/repos/os-autoinst/openQA/statuses/04a3f669ea13a4aa7cbd4569f578a66f7403c43d",
    "head": {
      "label": "Martchus:allocate-network",
      "ref": "allocate-network",
      "sha": "04a3f669ea13a4aa7cbd4569f578a66f7403c43d",
      "user": {
        "login": "Martchus",
        "id": …,
        "node_id": "…",
        "avatar_url": "https://avatars.githubusercontent.com/u/10248953?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/Martchus",
        "html_url": "https://github.com/Martchus",
        "followers_url": "https://api.github.com/users/Martchus/followers",
        "following_url": "https://api.github.com/users/Martchus/following{/other_user}",
        "gists_url": "https://api.github.com/users/Martchus/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/Martchus/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/Martchus/subscriptions",
        "organizations_url": "https://api.github.com/users/Martchus/orgs",
        "repos_url": "https://api.github.com/users/Martchus/repos",
        "events_url": "https://api.github.com/users/Martchus/events{/privacy}",
        "received_events_url": "https://api.github.com/users/Martchus/received_events",
        "type": "User",
        "site_admin": false
      },
      "repo": {
        "id": …,
        "node_id": "…",
        "name": "openQA",
        "full_name": "Martchus/openQA",
        "private": false,
        "owner": {
          "login": "Martchus",
          "id": …,
          "node_id": "…",
          "avatar_url": "https://avatars.githubusercontent.com/u/10248953?v=4",
          "gravatar_id": "",
          "url": "https://api.github.com/users/Martchus",
          "html_url": "https://github.com/Martchus",
          "followers_url": "https://api.github.com/users/Martchus/followers",
          "following_url": "https://api.github.com/users/Martchus/following{/other_user}",
          "gists_url": "https://api.github.com/users/Martchus/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/Martchus/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/Martchus/subscriptions",
          "organizations_url": "https://api.github.com/users/Martchus/orgs",
          "repos_url": "https://api.github.com/users/Martchus/repos",
          "events_url": "https://api.github.com/users/Martchus/events{/privacy}",
          "received_events_url": "https://api.github.com/users/Martchus/received_events",
          "type": "User",
          "site_admin": false
        },
        "html_url": "https://github.com/Martchus/openQA",
        "description": "openQA web-frontend, scheduler and tools",
        "fork": true,
        "url": "https://api.github.com/repos/Martchus/openQA",
        "forks_url": "https://api.github.com/repos/Martchus/openQA/forks",
        "keys_url": "https://api.github.com/repos/Martchus/openQA/keys{/key_id}",
        "collaborators_url": "https://api.github.com/repos/Martchus/openQA/collaborators{/collaborator}",
        "teams_url": "https://api.github.com/repos/Martchus/openQA/teams",
        "hooks_url": "https://api.github.com/repos/Martchus/openQA/hooks",
        "issue_events_url": "https://api.github.com/repos/Martchus/openQA/issues/events{/number}",
        "events_url": "https://api.github.com/repos/Martchus/openQA/events",
        "assignees_url": "https://api.github.com/repos/Martchus/openQA/assignees{/user}",
        "branches_url": "https://api.github.com/repos/Martchus/openQA/branches{/branch}",
        "tags_url": "https://api.github.com/repos/Martchus/openQA/tags",
        "blobs_url": "https://api.github.com/repos/Martchus/openQA/git/blobs{/sha}",
        "git_tags_url": "https://api.github.com/repos/Martchus/openQA/git/tags{/sha}",
        "git_refs_url": "https://api.github.com/repos/Martchus/openQA/git/refs{/sha}",
        "trees_url": "https://api.github.com/repos/Martchus/openQA/git/trees{/sha}",
        "statuses_url": "https://api.github.com/repos/Martchus/openQA/statuses/{sha}",
        "languages_url": "https://api.github.com/repos/Martchus/openQA/languages",
        "stargazers_url": "https://api.github.com/repos/Martchus/openQA/stargazers",
        "contributors_url": "https://api.github.com/repos/Martchus/openQA/contributors",
        "subscribers_url": "https://api.github.com/repos/Martchus/openQA/subscribers",
        "subscription_url": "https://api.github.com/repos/Martchus/openQA/subscription",
        "commits_url": "https://api.github.com/repos/Martchus/openQA/commits{/sha}",
        "git_commits_url": "https://api.github.com/repos/Martchus/openQA/git/commits{/sha}",
        "comments_url": "https://api.github.com/repos/Martchus/openQA/comments{/number}",
        "issue_comment_url": "https://api.github.com/repos/Martchus/openQA/issues/comments{/number}",
        "contents_url": "https://api.github.com/repos/Martchus/openQA/contents/{+path}",
        "compare_url": "https://api.github.com/repos/Martchus/openQA/compare/{base}...{head}",
        "merges_url": "https://api.github.com/repos/Martchus/openQA/merges",
        "archive_url": "https://api.github.com/repos/Martchus/openQA/{archive_format}{/ref}",
        "downloads_url": "https://api.github.com/repos/Martchus/openQA/downloads",
        "issues_url": "https://api.github.com/repos/Martchus/openQA/issues{/number}",
        "pulls_url": "https://api.github.com/repos/Martchus/openQA/pulls{/number}",
        "milestones_url": "https://api.github.com/repos/Martchus/openQA/milestones{/number}",
        "notifications_url": "https://api.github.com/repos/Martchus/openQA/notifications{?since,all,participating}",
        "labels_url": "https://api.github.com/repos/Martchus/openQA/labels{/name}",
        "releases_url": "https://api.github.com/repos/Martchus/openQA/releases{/id}",
        "deployments_url": "https://api.github.com/repos/Martchus/openQA/deployments",
        "created_at": "2016-04-20T14:05:13Z",
        "updated_at": "2023-01-31T16:42:44Z",
        "pushed_at": "2023-05-02T15:51:21Z",
        "git_url": "git://github.com/Martchus/openQA.git",
        "ssh_url": "git@github.com:Martchus/openQA.git",
        "clone_url": "https://github.com/Martchus/openQA.git",
        "svn_url": "https://github.com/Martchus/openQA",
        "homepage": "http://openqa.opensuse.org/",
        "size": 32572,
        "stargazers_count": 0,
        "watchers_count": 0,
        "language": "Perl",
        "has_issues": false,
        "has_projects": true,
        "has_downloads": true,
        "has_wiki": true,
        "has_pages": false,
        "has_discussions": false,
        "forks_count": 0,
        "mirror_url": null,
        "archived": false,
        "disabled": false,
        "open_issues_count": 0,
        "license": {
          "key": "gpl-2.0",
          "name": "GNU General Public License v2.0",
          "spdx_id": "GPL-2.0",
          "url": "https://api.github.com/licenses/gpl-2.0",
          "node_id": "…"
        },
        "allow_forking": true,
        "is_template": false,
        "web_commit_signoff_required": false,
        "topics": [

        ],
        "visibility": "public",
        "forks": 0,
        "open_issues": 0,
        "watchers": 0,
        "default_branch": "master",
        "allow_squash_merge": true,
        "allow_merge_commit": true,
        "allow_rebase_merge": true,
        "allow_auto_merge": false,
        "delete_branch_on_merge": false,
        "allow_update_branch": false,
        "use_squash_pr_title_as_default": false,
        "squash_merge_commit_message": "COMMIT_MESSAGES",
        "squash_merge_commit_title": "COMMIT_OR_PR_TITLE",
        "merge_commit_message": "PR_TITLE",
        "merge_commit_title": "MERGE_MESSAGE"
      }
    },
    "base": {
      "label": "os-autoinst:master",
      "ref": "master",
      "sha": "135efe17598ab35c5eb859798fa8aae1661b494f",
      "user": {
        "login": "os-autoinst",
        "id": …,
        "node_id": "…",
        "avatar_url": "https://avatars.githubusercontent.com/u/6918706?v=4",
        "gravatar_id": "",
        "url": "https://api.github.com/users/os-autoinst",
        "html_url": "https://github.com/os-autoinst",
        "followers_url": "https://api.github.com/users/os-autoinst/followers",
        "following_url": "https://api.github.com/users/os-autoinst/following{/other_user}",
        "gists_url": "https://api.github.com/users/os-autoinst/gists{/gist_id}",
        "starred_url": "https://api.github.com/users/os-autoinst/starred{/owner}{/repo}",
        "subscriptions_url": "https://api.github.com/users/os-autoinst/subscriptions",
        "organizations_url": "https://api.github.com/users/os-autoinst/orgs",
        "repos_url": "https://api.github.com/users/os-autoinst/repos",
        "events_url": "https://api.github.com/users/os-autoinst/events{/privacy}",
        "received_events_url": "https://api.github.com/users/os-autoinst/received_events",
        "type": "Organization",
        "site_admin": false
      },
      "repo": {
        "id": …,
        "node_id": "…",
        "name": "openQA",
        "full_name": "os-autoinst/openQA",
        "private": false,
        "owner": {
          "login": "os-autoinst",
          "id": …,
          "node_id": "…",
          "avatar_url": "https://avatars.githubusercontent.com/u/6918706?v=4",
          "gravatar_id": "",
          "url": "https://api.github.com/users/os-autoinst",
          "html_url": "https://github.com/os-autoinst",
          "followers_url": "https://api.github.com/users/os-autoinst/followers",
          "following_url": "https://api.github.com/users/os-autoinst/following{/other_user}",
          "gists_url": "https://api.github.com/users/os-autoinst/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/os-autoinst/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/os-autoinst/subscriptions",
          "organizations_url": "https://api.github.com/users/os-autoinst/orgs",
          "repos_url": "https://api.github.com/users/os-autoinst/repos",
          "events_url": "https://api.github.com/users/os-autoinst/events{/privacy}",
          "received_events_url": "https://api.github.com/users/os-autoinst/received_events",
          "type": "Organization",
          "site_admin": false
        },
        "html_url": "https://github.com/os-autoinst/openQA",
        "description": "openQA web-frontend, scheduler and tools.",
        "fork": false,
        "url": "https://api.github.com/repos/os-autoinst/openQA",
        "forks_url": "https://api.github.com/repos/os-autoinst/openQA/forks",
        "keys_url": "https://api.github.com/repos/os-autoinst/openQA/keys{/key_id}",
        "collaborators_url": "https://api.github.com/repos/os-autoinst/openQA/collaborators{/collaborator}",
        "teams_url": "https://api.github.com/repos/os-autoinst/openQA/teams",
        "hooks_url": "https://api.github.com/repos/os-autoinst/openQA/hooks",
        "issue_events_url": "https://api.github.com/repos/os-autoinst/openQA/issues/events{/number}",
        "events_url": "https://api.github.com/repos/os-autoinst/openQA/events",
        "assignees_url": "https://api.github.com/repos/os-autoinst/openQA/assignees{/user}",
        "branches_url": "https://api.github.com/repos/os-autoinst/openQA/branches{/branch}",
        "tags_url": "https://api.github.com/repos/os-autoinst/openQA/tags",
        "blobs_url": "https://api.github.com/repos/os-autoinst/openQA/git/blobs{/sha}",
        "git_tags_url": "https://api.github.com/repos/os-autoinst/openQA/git/tags{/sha}",
        "git_refs_url": "https://api.github.com/repos/os-autoinst/openQA/git/refs{/sha}",
        "trees_url": "https://api.github.com/repos/os-autoinst/openQA/git/trees{/sha}",
        "statuses_url": "https://api.github.com/repos/os-autoinst/openQA/statuses/{sha}",
        "languages_url": "https://api.github.com/repos/os-autoinst/openQA/languages",
        "stargazers_url": "https://api.github.com/repos/os-autoinst/openQA/stargazers",
        "contributors_url": "https://api.github.com/repos/os-autoinst/openQA/contributors",
        "subscribers_url": "https://api.github.com/repos/os-autoinst/openQA/subscribers",
        "subscription_url": "https://api.github.com/repos/os-autoinst/openQA/subscription",
        "commits_url": "https://api.github.com/repos/os-autoinst/openQA/commits{/sha}",
        "git_commits_url": "https://api.github.com/repos/os-autoinst/openQA/git/commits{/sha}",
        "comments_url": "https://api.github.com/repos/os-autoinst/openQA/comments{/number}",
        "issue_comment_url": "https://api.github.com/repos/os-autoinst/openQA/issues/comments{/number}",
        "contents_url": "https://api.github.com/repos/os-autoinst/openQA/contents/{+path}",
        "compare_url": "https://api.github.com/repos/os-autoinst/openQA/compare/{base}...{head}",
        "merges_url": "https://api.github.com/repos/os-autoinst/openQA/merges",
        "archive_url": "https://api.github.com/repos/os-autoinst/openQA/{archive_format}{/ref}",
        "downloads_url": "https://api.github.com/repos/os-autoinst/openQA/downloads",
        "issues_url": "https://api.github.com/repos/os-autoinst/openQA/issues{/number}",
        "pulls_url": "https://api.github.com/repos/os-autoinst/openQA/pulls{/number}",
        "milestones_url": "https://api.github.com/repos/os-autoinst/openQA/milestones{/number}",
        "notifications_url": "https://api.github.com/repos/os-autoinst/openQA/notifications{?since,all,participating}",
        "labels_url": "https://api.github.com/repos/os-autoinst/openQA/labels{/name}",
        "releases_url": "https://api.github.com/repos/os-autoinst/openQA/releases{/id}",
        "deployments_url": "https://api.github.com/repos/os-autoinst/openQA/deployments",
        "created_at": "2014-06-16T12:23:22Z",
        "updated_at": "2023-05-02T08:58:43Z",
        "pushed_at": "2023-05-02T15:52:07Z",
        "git_url": "git://github.com/os-autoinst/openQA.git",
        "ssh_url": "git@github.com:os-autoinst/openQA.git",
        "clone_url": "https://github.com/os-autoinst/openQA.git",
        "svn_url": "https://github.com/os-autoinst/openQA",
        "homepage": "http://openqa.opensuse.org/",
        "size": 53700,
        "stargazers_count": 266,
        "watchers_count": 266,
        "language": "Perl",
        "has_issues": true,
        "has_projects": true,
        "has_downloads": true,
        "has_wiki": false,
        "has_pages": true,
        "has_discussions": true,
        "forks_count": 190,
        "mirror_url": null,
        "archived": false,
        "disabled": false,
        "open_issues_count": 29,
        "license": {
          "key": "gpl-2.0",
          "name": "GNU General Public License v2.0",
          "spdx_id": "GPL-2.0",
          "url": "https://api.github.com/licenses/gpl-2.0",
          "node_id": "…"
        },
        "allow_forking": true,
        "is_template": false,
        "web_commit_signoff_required": false,
        "topics": [
          "hacktoberfest",
          "perl",
          "test-automation",
          "test-scheduling",
          "testing"
        ],
        "visibility": "public",
        "forks": 190,
        "open_issues": 29,
        "watchers": 266,
        "default_branch": "master",
        "allow_squash_merge": true,
        "allow_merge_commit": true,
        "allow_rebase_merge": true,
        "allow_auto_merge": false,
        "delete_branch_on_merge": true,
        "allow_update_branch": false,
        "use_squash_pr_title_as_default": false,
        "squash_merge_commit_message": "COMMIT_MESSAGES",
        "squash_merge_commit_title": "COMMIT_OR_PR_TITLE",
        "merge_commit_message": "PR_TITLE",
        "merge_commit_title": "MERGE_MESSAGE"
      }
    },
    "_links": {
      "self": {
        "href": "https://api.github.com/repos/os-autoinst/openQA/pulls/5111"
      },
      "html": {
        "href": "https://github.com/os-autoinst/openQA/pull/5111"
      },
      "issue": {
        "href": "https://api.github.com/repos/os-autoinst/openQA/issues/5111"
      },
      "comments": {
        "href": "https://api.github.com/repos/os-autoinst/openQA/issues/5111/comments"
      },
      "review_comments": {
        "href": "https://api.github.com/repos/os-autoinst/openQA/pulls/5111/comments"
      },
      "review_comment": {
        "href": "https://api.github.com/repos/os-autoinst/openQA/pulls/comments{/number}"
      },
      "commits": {
        "href": "https://api.github.com/repos/os-autoinst/openQA/pulls/5111/commits"
      },
      "statuses": {
        "href": "https://api.github.com/repos/os-autoinst/openQA/statuses/04a3f669ea13a4aa7cbd4569f578a66f7403c43d"
      }
    },
    "author_association": "CONTRIBUTOR",
    "auto_merge": null,
    "active_lock_reason": null,
    "merged": false,
    "mergeable": null,
    "rebaseable": null,
    "mergeable_state": "unknown",
    "merged_by": null,
    "comments": 0,
    "review_comments": 0,
    "maintainer_can_modify": true,
    "commits": 2,
    "additions": 91,
    "deletions": 49,
    "changed_files": 2
  },
  "repository": {
    "id": …,
    "node_id": "…",
    "name": "openQA",
    "full_name": "os-autoinst/openQA",
    "private": false,
    "owner": {
      "login": "os-autoinst",
      "id": …,
      "node_id": "…",
      "avatar_url": "https://avatars.githubusercontent.com/u/6918706?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/os-autoinst",
      "html_url": "https://github.com/os-autoinst",
      "followers_url": "https://api.github.com/users/os-autoinst/followers",
      "following_url": "https://api.github.com/users/os-autoinst/following{/other_user}",
      "gists_url": "https://api.github.com/users/os-autoinst/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/os-autoinst/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/os-autoinst/subscriptions",
      "organizations_url": "https://api.github.com/users/os-autoinst/orgs",
      "repos_url": "https://api.github.com/users/os-autoinst/repos",
      "events_url": "https://api.github.com/users/os-autoinst/events{/privacy}",
      "received_events_url": "https://api.github.com/users/os-autoinst/received_events",
      "type": "Organization",
      "site_admin": false
    },
    "html_url": "https://github.com/os-autoinst/openQA",
    "description": "openQA web-frontend, scheduler and tools.",
    "fork": false,
    "url": "https://api.github.com/repos/os-autoinst/openQA",
    "forks_url": "https://api.github.com/repos/os-autoinst/openQA/forks",
    "keys_url": "https://api.github.com/repos/os-autoinst/openQA/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/os-autoinst/openQA/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/os-autoinst/openQA/teams",
    "hooks_url": "https://api.github.com/repos/os-autoinst/openQA/hooks",
    "issue_events_url": "https://api.github.com/repos/os-autoinst/openQA/issues/events{/number}",
    "events_url": "https://api.github.com/repos/os-autoinst/openQA/events",
    "assignees_url": "https://api.github.com/repos/os-autoinst/openQA/assignees{/user}",
    "branches_url": "https://api.github.com/repos/os-autoinst/openQA/branches{/branch}",
    "tags_url": "https://api.github.com/repos/os-autoinst/openQA/tags",
    "blobs_url": "https://api.github.com/repos/os-autoinst/openQA/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/os-autoinst/openQA/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/os-autoinst/openQA/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/os-autoinst/openQA/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/os-autoinst/openQA/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/os-autoinst/openQA/languages",
    "stargazers_url": "https://api.github.com/repos/os-autoinst/openQA/stargazers",
    "contributors_url": "https://api.github.com/repos/os-autoinst/openQA/contributors",
    "subscribers_url": "https://api.github.com/repos/os-autoinst/openQA/subscribers",
    "subscription_url": "https://api.github.com/repos/os-autoinst/openQA/subscription",
    "commits_url": "https://api.github.com/repos/os-autoinst/openQA/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/os-autoinst/openQA/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/os-autoinst/openQA/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/os-autoinst/openQA/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/os-autoinst/openQA/contents/{+path}",
    "compare_url": "https://api.github.com/repos/os-autoinst/openQA/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/os-autoinst/openQA/merges",
    "archive_url": "https://api.github.com/repos/os-autoinst/openQA/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/os-autoinst/openQA/downloads",
    "issues_url": "https://api.github.com/repos/os-autoinst/openQA/issues{/number}",
    "pulls_url": "https://api.github.com/repos/os-autoinst/openQA/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/os-autoinst/openQA/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/os-autoinst/openQA/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/os-autoinst/openQA/labels{/name}",
    "releases_url": "https://api.github.com/repos/os-autoinst/openQA/releases{/id}",
    "deployments_url": "https://api.github.com/repos/os-autoinst/openQA/deployments",
    "created_at": "2014-06-16T12:23:22Z",
    "updated_at": "2023-05-02T08:58:43Z",
    "pushed_at": "2023-05-02T15:52:07Z",
    "git_url": "git://github.com/os-autoinst/openQA.git",
    "ssh_url": "git@github.com:os-autoinst/openQA.git",
    "clone_url": "https://github.com/os-autoinst/openQA.git",
    "svn_url": "https://github.com/os-autoinst/openQA",
    "homepage": "http://openqa.opensuse.org/",
    "size": 53700,
    "stargazers_count": 266,
    "watchers_count": 266,
    "language": "Perl",
    "has_issues": true,
    "has_projects": true,
    "has_downloads": true,
    "has_wiki": false,
    "has_pages": true,
    "has_discussions": true,
    "forks_count": 190,
    "mirror_url": null,
    "archived": false,
    "disabled": false,
    "open_issues_count": 29,
    "license": {
      "key": "gpl-2.0",
      "name": "GNU General Public License v2.0",
      "spdx_id": "GPL-2.0",
      "url": "https://api.github.com/licenses/gpl-2.0",
      "node_id": "…"
    },
    "allow_forking": true,
    "is_template": false,
    "web_commit_signoff_required": false,
    "topics": [
      "hacktoberfest",
      "perl",
      "test-automation",
      "test-scheduling",
      "testing"
    ],
    "visibility": "public",
    "forks": 190,
    "open_issues": 29,
    "watchers": 266,
    "default_branch": "master"
  },
  "organization": {
    "login": "os-autoinst",
    "id": 6918706,
    "node_id": "…",
    "url": "https://api.github.com/orgs/os-autoinst",
    "repos_url": "https://api.github.com/orgs/os-autoinst/repos",
    "events_url": "https://api.github.com/orgs/os-autoinst/events",
    "hooks_url": "https://api.github.com/orgs/os-autoinst/hooks",
    "issues_url": "https://api.github.com/orgs/os-autoinst/issues",
    "members_url": "https://api.github.com/orgs/os-autoinst/members{/member}",
    "public_members_url": "https://api.github.com/orgs/os-autoinst/public_members{/member}",
    "avatar_url": "https://avatars.githubusercontent.com/u/6918706?v=4",
    "description": ""
  },
  "sender": {
    "login": "Martchus",
    "id": …,
    "node_id": "…",
    "avatar_url": "https://avatars.githubusercontent.com/u/10248953?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/Martchus",
    "html_url": "https://github.com/Martchus",
    "followers_url": "https://api.github.com/users/Martchus/followers",
    "following_url": "https://api.github.com/users/Martchus/following{/other_user}",
    "gists_url": "https://api.github.com/users/Martchus/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/Martchus/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/Martchus/subscriptions",
    "organizations_url": "https://api.github.com/users/Martchus/orgs",
    "repos_url": "https://api.github.com/users/Martchus/repos",
    "events_url": "https://api.github.com/users/Martchus/events{/privacy}",
    "received_events_url": "https://api.github.com/users/Martchus/received_events",
    "type": "User",
    "site_admin": false
  }
}

That is just one example from https://github.com/os-autoinst/openQA/settings/hooks/341065449.

This route would evaluate the payload and eventually just create a scheduled product like the ISOs route. So it makes sense to reuse that code. For that I've created a PR to refactor the existing code for better re-usability: https://github.com/os-autoinst/openQA/pull/5113

Supposedly we need to download the scenario definitions from GitHub on the web UI host after all.

We can use $json->{pull_request}->{head}->{sha} and $json->{pull_request}->{head}->{contents_url} to compute an URL like https://api.github.com/repos/os-autoinst/os-autoinst-distri-example/contents/scenario-definitions.yaml?ref=b9538a65ff1f2a4c6582245cbb63170501a804c7.

Likely it is better to compute a "raw.githubusercontent.com" URL directly from $json->{pull_request}->{head}->{repo}->{full_name} to avoid an intermediate download step: https://raw.githubusercontent.com/os-autoinst/os-autoinst-distri-example/b9538a65ff1f2a4c6582245cbb63170501a804c7/scenario-definitions.yaml

We'd also set the following test parameters:

_GROUP_ID="0"
BUILD="$json->{pull_request}->{head}->{repo}->{full_name}#$json->{pull_request}->{head}->{sha}"
CASEDIR="$json->{pull_request}->{head}->{repo}->{clone_url}#$json->{pull_request}->{head}->{sha}"
NEEDLES_DIR="%%CASEDIR%%/needles"

Actions

Copy link

Updated by mkittler over 1 year ago

For reporting back, we could use the checks API: https://docs.github.com/en/apps/creating-github-apps/guides/creating-ci-tests-with-the-checks-api
However, that requires authentication as GitHub app. (I got "You must authenticate via a GitHub App." when I tried it.) So that's likely not the simplest approach and definitely not what OBS is using as I don't see it configured as GitHub app anywhere in our organization.

So I'll have to check what OBS is using instead. I'm currently browsing the OBS source code and reading https://github.com/openSUSE/open-build-service/wiki/Better-SCM-CI-Integration-Overview.

Actions

Copy link

Updated by mkittler over 1 year ago

It was a good idea to check what OBS does. They use the statuses API via Oktokit. It doesn't require a GitHub app. The following has just worked:

curl -L \
  -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer …"\
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/os-autoinst/openQA/statuses/8dfa3ea4d004900a56a9b8dc572fcd878c18b88a \
  -d '{"state":"success","target_url":"https://martchus.no-ip.biz","description":"Some test status","context":"test/check"}'

The check is currently visible on https://github.com/os-autoinst/openQA/pull/5114. (It would of course vanish once I update the PR the next time.)

When invoking it again with a different state (e.g. "state":"error") the existing entry is updated. I suppose initially we'd set it to pending, then to either failure or success when all jobs have finished or error when something went wrong. That actually promises to be fairly simple :-)

Actions

Copy link

Updated by mkittler over 1 year ago

I've just been updating https://github.com/os-autoinst/openQA/pull/5114.

The PR is now in a state where I would say it is clear what changes would be necessary in openQA. I have tested at least responding to a "ping" webhook payload from GitHub and I have also tested reporting the CI status to GitHub. I have also added documentation how to configure this. So I'd say both ACs are actually fulfilled. Time-wise I guess I have at least 8 hours left for this time-boxed task and this is also useful for #128360. So I'm continuing with the real implementation.

Actions

Copy link

#10

Updated by mkittler over 1 year ago

Looks like https://github.com/os-autoinst/openQA/pull/5114 works.

What's left to do:

~~Cancellation~~/cleanup, e.g. when the PR has been updated or merged/closed
~~Handle restarted jobs correctly so they report the result back as expected (overriding the result of the original job)~~
- Also consider restarted jobs for the cleanup
Better error handling when reporting back: possibly add a retry, add an audit event in case something goes wrong
~~Better progress; e.g. we could show "2 of 5 jobs completed" while the check is pending and e.g. "4 of 5 jobs succeeded" when the check has completed.~~
Finer/better permissions. So far we require an API key/secret pair where the associated user is at least operator. That already includes many permissions. We could allow the creation of API key/secret pairs that fewer permissions.
Have an extra token as secret for signing of the webhook payload. This extra token could be generated for the used API key/secret pair so the secret for signing would not just basically be the API user/credentials again.

The crossed-out parts have been covered by https://github.com/os-autoinst/openQA/pull/5133.

Note that we've decided to just rely on openQA's normal cleanup behavior for now.

Actions

Copy link

#11

Updated by mkittler over 1 year ago

Status changed from In Progress to Resolved

https://github.com/os-autoinst/openQA/pull/5132 has been merged as well and https://github.com/os-autoinst/openQA/pull/5133 is pending.

With that I would resolve this time-boxed ticket as the time frame is nearing its end. Both ACs are fulfilled; we know what we need to change in form of merged and pending PRs and the outlook given in my last comment.

Since this looks promising and would help with #128360 I'm going to work further on the topic as part of #128360.

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

QA (public) » openQA Project (public)

Tags

Custom queries

action #127949

[spike][timeboxed:20h] Research native GitHub for running openQA tests as CI checks size:M

Motivation¶

Acceptance criteria¶

Suggestions¶

Updated by mkittler over 1 year ago

Updated by okurz over 1 year ago

Updated by mkittler over 1 year ago

Updated by okurz over 1 year ago

Updated by mkittler over 1 year ago

Updated by mkittler over 1 year ago

Updated by mkittler over 1 year ago

Updated by mkittler over 1 year ago

Updated by mkittler over 1 year ago

Updated by mkittler over 1 year ago

Updated by mkittler over 1 year ago