Project

General

Profile

action #99240

coordination #99183: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui, to openSUSE Leap 15.3

Upgrade CI container image versions to Leap 15.3 size:M

Added by okurz 4 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2021-09-24
Due date:
% Done:

0%

Estimated time:

Description

Motivation

We should run a supported version of Leap.

Acceptance criteria

  • AC1: Ensure all CI container images we use are on Leap 15.3

Suggestions

  • Find out where we use images
  • Reconfigure a container image based on 15.3
  • Don't use super powers to ignore CI test failures

Related issues

Related to QA - action #97367: Update CI image QA:Maintenance/openSUSE-Leap-Container to Leap 15.3Resolved2021-08-23

Copied to openQA Project - action #102464: Upgrade OBS package CI checks to Leap 15.3 (os-autoinst+openQA) size:MIn Progress2021-09-242022-02-11

History

#1 Updated by okurz 4 months ago

  • Related to action #97367: Update CI image QA:Maintenance/openSUSE-Leap-Container to Leap 15.3 added

#2 Updated by cdywan 4 months ago

  • Subject changed from Upgrade CI container image versions to Leap 15.3 to Upgrade CI container image versions to Leap 15.3 size:M
  • Description updated (diff)
  • Status changed from New to Workable

#3 Updated by mkittler 4 months ago

  • Assignee set to mkittler

#4 Updated by mkittler 4 months ago

  • Status changed from Workable to In Progress

Intneral GitLab repositories which need to be updated as well:

#5 Updated by openqa_review 4 months ago

  • Due date set to 2021-10-16

Setting due date based on mean cycle time of SUSE QE Tools

#6 Updated by mkittler 4 months ago

I've been adjusting the project config but two of the containers are now unresolvable with "nothing provides this-is-only-for-build-envs needed by postgresql13-devel-mini":

Not sure yet why this is a problem with Leap 15.3. I get the same error on my TW system when trying to install the package:

sudo zypper in postgresql13-devel-mini
[sudo] Passwort für root: 
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...
Paketabhängigkeiten werden aufgelöst...

Problem: nichts stellt 'this-is-only-for-build-envs' bereit, das vom zu installierenden postgresql13-devel-mini-13.4-1.2.x86_64 benötigt wird
 Lösung 1: postgresql13-devel-mini-13.4-1.2.x86_64 nicht installieren
 Lösung 2: postgresql13-devel-mini-13.4-1.2.x86_64 durch Ignorieren einiger Abhängigkeiten brechen

Wählen Sie aus den obigen Lösungen mittels Nummer oder brechen Sie (a)b [1/2/a/d/?] (a):

The package postgresql13-devel would be resolvable. However, adding Prefer: postgresql13-devel (or postgresql12-devel or -postgresql12-devel) to the project config doesn't help.

#7 Updated by mkittler 4 months ago

With help from @fvogt I could workaround the problem. The project config now looks like this:

# extension of `Substitute: build-packages:docker …` found in the project config of openSUSE:Leap:15.3 to add missing substitutes for PostgreSQL
Substitute: build-packages:docker !systemd-mini !udev-mini !krb5-mini !libsystemd0-mini !libudev-mini1 !krb5-mini !gettext-tools-mini !cmake-mini !systemd-mini-sysvinit !dummy-release !libunbound-devel-mini !gettext-runtime-mini !postgresql13-devel-mini

# choose ruby version for Leap 15.3
Prefer: ruby2.5-rubygem-rb-fsevent-0.11

I've been creating a PR to adapt our scripts: https://github.com/os-autoinst/openQA/pull/4267


I've also noticed that the container paths for the openqa_dev image are wrong and should likely be updated for Leap 15.3 as well: https://github.com/os-autoinst/openQA/pull/4268

#9 Updated by okurz 4 months ago

What's https://gitlab.suse.de/mkittler/ci/-/merge_requests/1 ?

openQA CI tests have failed now because some packages have not been found. I now did for i in perl-Furl perl-Selenium-Remote-Driver perl-Test-Warnings; do osc linkpac openSUSE:Factory $i devel:openQA:Leap:15.3; done.

#10 Updated by mkittler 4 months ago

https://gitlab.suse.de/mkittler/ci/-/merge_requests/1 should have been created in https://gitlab.suse.de/openqa/ci although I'm still not sure about it anyways.


Thanks for linking missing packages. I'll check whether they build and whether something is still missing.
EDIT: They could be built and I re-triggered tests on https://github.com/os-autoinst/openQA/pull/4266 to check whether it works now or what else is missing.

#11 Updated by mkittler 4 months ago

And I've closed https://github.com/os-autoinst/openQA/pull/4268 and switched containers to Leap 15.3 (from TW).

This breaks now https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-kvm and https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86. I'm not sure where these images are used. Maybe they should be moved to containers-tw now?

#12 Updated by okurz 4 months ago

Also os-autoinst_dev could not build anymore hence blocking os-autoinst CI jobs.

I added an additional configuration

  <repository name="containers15.3">
    <path project="devel:openQA" repository="openSUSE_Leap_15.3"/>
    <path project="openSUSE:Containers:Leap:15.3" repository="containers"/>
    <arch>x86_64</arch>
  </repository>

for when anyone wants to explicitly mention the version 15.3 in the container image URL as is currently done in os-autoinst. Now https://build.opensuse.org/package/show/devel:openQA/os-autoinst_dev is built for "containers" as well as "containers15.3" which should unblock the CI pipelines e.g. https://github.com/os-autoinst/os-autoinst/pull/1792/checks?check_run_id=3811612581

mkittler wrote:

And I've closed https://github.com/os-autoinst/openQA/pull/4268 and switched containers to Leap 15.3 (from TW).

This breaks now https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-kvm and https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86. I'm not sure where these images are used.

The images are referenced on https://github.com/os-autoinst/os-autoinst/blob/master/README.asciidoc for direct use and also they are used in
https://github.com/os-autoinst/os-autoinst-distri-example/tree/master/.github/workflows

Considering that sometimes Leap breaks, sometimes Tumbleweed we should have an easier time just building for both and referencing in README that an alternative exists if one does not work

#13 Updated by okurz 4 months ago

Understanding now that e.g. https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86 is unresolvable as it was relying on "containers" referring to Tumbleweed I have to reconsider my earlier suggestion about making "containers" latest Leap, I forgot about such use cases. I now find no Tumbleweed container definition in https://build.opensuse.org/projects/devel:openQA/meta at all. Could you please add that back as you suggested as "containers-tw" and then we can still decide what our "default" should be?

#14 Updated by mkittler 4 months ago

Yes, there's currently no TW containers repository. I can add it back as containers-tw.

#15 Updated by mkittler 4 months ago

It looks like the dependency PR works again but the documentation generation still fails with an error, see https://app.circleci.com/pipelines/github/os-autoinst/openQA/7961/workflows/2ad713ac-3932-411f-bb71-6defc4df31c3/jobs/75161.

#16 Updated by mkittler 4 months ago

Could you please add that back as you suggested as "containers-tw"

Done, none of the container "packages" in devel:openQA are unresolvable anymore.


The error for the documentation creation looked differently when I re-triggered it yesterday. Now it even fails earlier because the cache is empty:

#!/bin/bash -eo pipefail
.circleci/build_or_install_from_cache.sh

+ '[' -z 0ee7ac9e-08fe-4084-8109-9a54d80dc80e ']'
++ find /var/cache/zypp/packages/
++ grep '.rpm$'
+ packages=
+ echo 'No RPM packages found, cache is empty, aborting'
No RPM packages found, cache is empty, aborting
+ exit 1

Exited with code exit status 1

CircleCI received exit code 1

Looks like it references to a cache which is likely just cleaned up at this point:

No cache is found for key: v1-AM9kk913NN5TadgjfGzdQgkGS+jtsx2nhomGH_x_mCE=-3q6IWLmZ9Ai9ZBXW07jIuRUvGji2w5NiECwz_qDjx3Y=

#17 Updated by mkittler 4 months ago

  • Status changed from In Progress to Feedback

The latest nightly jobs worked again: https://app.circleci.com/pipelines/github/os-autoinst/openQA/7982/workflows/e96309d8-9ff6-4f36-90b0-029673117a79

I'm really wondering how I could have CircleCI to ditch the old cache instead of just waiting until it happens but at least it works now.


This still leaves https://gitlab.suse.de/openqa/ci where I'd needed a TW container with Git and SSH installed. However, I don't even know what this repository is used for and half of the code is commented-out. So I assume it is WIP and maybe not worth updating at this point.

#18 Updated by okurz 4 months ago

mkittler wrote:

This still leaves https://gitlab.suse.de/openqa/ci where I'd needed a TW container with Git and SSH installed. However, I don't even know what this repository is used for and half of the code is commented-out. So I assume it is WIP and maybe not worth updating at this point.

If you mean https://gitlab.suse.de/openqa/ci/-/blob/master/ci/ssh-deploy.yml#L24 with "commented-out" then you are misreading that. This line says "Use .setup_ssh like below:", what follows is example code. The definitions from this gitlab project are included in other projects, e.g. https://gitlab.suse.de/openqa/osd-deployment/-/blob/master/.gitlab-ci.yml#L31

#19 Updated by mkittler 4 months ago

Ok, so we need a TW image with Git and OpenSSH installed. Should I create one in my own home project? It would have the disadvantage that we're spreading container images over multiple home projects. You could also give me permission to create a "package" in your home project.

#20 Updated by okurz 4 months ago

mkittler wrote:

Ok, so we need a TW image with Git and OpenSSH installed. Should I create one in my own home project? It would have the disadvantage that we're spreading container images over multiple home projects.

yes, better not. I created https://build.opensuse.org/package/show/home:okurz:container:ca/curl-jq-ssh-git now

You could also give me permission to create a "package" in your home project.

Good idea. I have also done that

A user hit a problem regarding using container images: https://suse.slack.com/archives/C02CANHLANP/p1633702326174700

So in https://github.com/os-autoinst/openQA/commit/fcb639102491abf70c3685c04497af07e7b311f0 we have updated container definitions to 15.3 but both https://build.opensuse.org/package/view_file/devel:openQA/openQA_container_worker/_service:download_url:Dockerfile?expand=1 and https://build.opensuse.org/package/show/devel:openQA/openQA_container_webUI were not yet configured to build for 15.3. Also seems like no service was triggered to update the Dockerfile that is extracted. Looking into this …

EDIT: Manually triggering the services has properly updated the Dockerfile in both packages. I consider this a bug on our side that we should followup with.

#21 Updated by okurz 4 months ago

registry.opensuse.org/home/okurz/container/ca/containers/tumbleweed:curl-jq-ssh is live now

#22 Updated by mkittler 4 months ago

SR: https://gitlab.suse.de/openqa/ci/-/merge_requests/3

I'm looking into the containerized setup. It was likely broken by https://github.com/os-autoinst/openQA/pull/4257.

#23 Updated by mkittler 4 months ago

It looks like the containerized setup via docker-compose works. I don't know what use the example in the https://open.qa/docs/#_container_based_setup section has. Maybe it is normal that it doesn't work out of the box. I consider fixing this out of the scope of this issue because I don't even know what's supposed to happen when one is executing this command.

#24 Updated by mkittler 4 months ago

  • Status changed from Feedback to Resolved

The SR has been merged and okurz mentions that dependent pipelines work.

This was the last remaining container so I'm resolving the issue.

#25 Updated by okurz 3 months ago

  • Due date deleted (2021-10-16)

#26 Updated by okurz 2 months ago

  • Copied to action #102464: Upgrade OBS package CI checks to Leap 15.3 (os-autoinst+openQA) size:M added

Also available in: Atom PDF