QA (public)

Motivation¶

As explained in the parent #180806 new maintenance workflows are planned based on Gitea. SMELT is planned to be adapted right now it is not yet there, see https://gitlab.suse.de/tools/smash/-/issues/1458.

https://gitlab.suse.de/tools/smelt/-/issues/1147, still the plan exists to have a first "dry run" of the SLE 16 maintenance workflow ready by 2025-07-08 as decided as part of #180602. As QE engineers so far rely on features like "approvable_for" comments we should find out if and how feasible an adaptation of qem-bot+qem-dashboard to such workflow is.

Acceptance criteria¶

• AC1: qem-bot+qem-dashboard is used to trigger+monitor+approve SLE 16 maintenance updates based on src.suse.de (can be separate instances)

Suggestions¶

Note that I highlighted open "TODOs" from previous spikes in bold. This doesn't mean we need to do them as long as ACs are fulfilled. I just want to highlight what next steps we should at least consider.

• DONE: Await and incorporate results from spikes #180809 and #182591 and if we can rely on SMELT https://gitlab.suse.de/tools/smash/-/issues/1458
  • We clarified in a meeting that reading data from SMELT is not wanted. We should use Gitea directly.
  • Results from #182591 have been incorporated into the subsequent suggestions.
  • The following change for the dashboard has already been merged: https://github.com/openSUSE/qem-dashboard/pull/2482
  • Subsequent suggestions refer to https://github.com/openSUSE/qem-bot/pull/195 as "draft PR" so check it out. It makes probably most sense to base further development on that PR.
    • The draft PR adds fake responses in the "responses" directory which are used by dry runs and the extended test suite. The --dry flag will prevent modifications to the dashboard. The --fake-data flag will prevent querying live data from Gitea.
    • Checkout #182591#note-15 for how the draft PR was/is tested and deployed.
• After #180809#note-14 and discussions on Slack and #182591#note-15 we established the following:
  • qem-bot should check PRs on Gitea like this dummy PR using the Gitea API to list PRs and then do the subsequent actions for each PR.
    • The draft PR already does this. It goes through all open PRs as there doesn't seem to exist an API parameter for server-side filtering of PRs by requested reviewer. The team products/qam-openqa e.g. on https://src.suse.de/products/SLFO/pulls/120 doesn't even show up in the immediate response from querying PRs. So the draft PR requests reviews for each and every open PR and only decides then whether it is relevant based on whether a review from products/qam-openqa is requested. This might also need tweaking to consider already approved PRs as such. (The approval will probably be done by a concrete user and not a team and so far the logic doesn't take that into account.)
  • So far the draft PR reads the relevant OBS projects from a comment of an OBS bot. However, the staging.config file and the PR number could also be used to determine the project on OBS instead of parsing "human readable" comments. (It is going to be :SLES instead of :SLES-15.99 as in the linked example which was from an old config.)
    • The staging project can be composed via <StagingProject>:<PR>.
    • The test productbuilds are in <StagingProject>:<PR>:<QA.Name>.
      • "IIRC you wanted to test only these (basically the binaries filtered by the list of packages which are delivered for the product)."
    • Additional remarks by Adrian:
      • The main project is also publishing, but these are the in filtered rpms.
      • IIRC you wanted to have the filtered repos for each product in addition.
      • Similar to what we had with channels before. Just that we organize it in subprojects now. This allows use to test current product source against it or even prepare changes in the product, eg. when a package split happened or packages got added which requires also an adaption in the produtcompose files.
  • The OBS-API http://api.suse.de/build/$PROJECT/_result can then be used to determine the build status and affected packages and archs.
  • If the OBS build status looks good qem-bot can trigger tests.
    • This is already done by the draft PR.
      • The draft PR also introduces the flag --allow-build-failures because at this point I haven't found a PR with OBS project where all builds are successful and all repos are published. This also means the logic for determining whether the build results on OBS look good might still need further tweaking.
      • The draft PR just schedules dummy tests (see next open point). For these dummy tests I created meta-data you can find on #182591#note-15. It also contains a remark on the versioning I'm not sure about.
    • How it knows what tests to trigger still needs clarification.
      • We could reuse the existing meta-data approach as done by the draft PR.
      • From OBS we know affected packages/archs so we could base the decision on that.
        • "yes, that's the updateinfo file, that should be enough for us to know which packages are under test, we just need to parse the xml"
  • qem-bot can approve/reject a change by creating a review on the PR.
    • DONE: The draft PR already does this. However, so far I only used my own account and reviews therefore didn't show up with the correct user. One cannot just specify a user when making the API call. Hence I created a dedicated user and https://sd.suse.com/servicedesk/customer/portal/1/SD-189112 to give it the required permissions (according to https://confluence.suse.com/display/devops/How+to+request+IBS+access) which needs to be followed-up with.
      • For account credentials, see https://gitlab.suse.de/openqa/password/-/merge_requests/23. We just need to create a token with that account on https://src.suse.de/user/settings/applications and specify that via --gitea-token when invoking the bot.
  • We should try to keep features mentioned at the end of #180809#note-11 working.
  • There will be "staging triggered tests" and "product increment triggered tests", see https://jira.suse.com/browse/MSQA-949. We need to figure out how those match with the current "incidents" and "aggregates". So this still needs clarification. So far the draft PR only cares about "incidents".
  • We could check whether the updateinfo.xml.gz file exists in the OBS repo and if not we can stop processing (probably we should still dismiss the pending review via https://src.suse.de/api/swagger#/repository/repoDismissPullReview so the PR is not stuck). (This idea came up in one of the meetings and hasn't been implemented in the draft PR.)