https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842021-09-30T09:37:39ZopenSUSE Project Management ToolopenQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4511372021-09-30T09:37:39Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/97367">action #97367</a>: Update CI image QA:Maintenance/openSUSE-Leap-Container to Leap 15.3</i> added</li></ul> openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4511432021-09-30T09:43:38Zlivdywanliv.dywan@suse.com
<ul><li><strong>Subject</strong> changed from <i>Upgrade CI container image versions to Leap 15.3</i> to <i>Upgrade CI container image versions to Leap 15.3 size:M</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/451143/diff?detail_id=427821">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li></ul> openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4515632021-10-01T12:53:51Zmkittlermarius.kittler@suse.com
<ul><li><strong>Assignee</strong> set to <i>mkittler</i></li></ul> openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4515662021-10-01T13:07:09Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>In Progress</i></li></ul><ul>
<li>PR for openQA: <a href="https://github.com/os-autoinst/openQA/pull/4258" class="external">https://github.com/os-autoinst/openQA/pull/4258</a>
<ul>
<li>It would make sense to execute the nightly job based on this PR in some test environment before merging it.</li>
</ul></li>
<li>Additional PR for openQA to also update containers used for the containerized setup: <a href="https://github.com/os-autoinst/openQA/pull/4257" class="external">https://github.com/os-autoinst/openQA/pull/4257</a></li>
<li>os-autoinst is already on Leap 15.3: <a href="https://github.com/os-autoinst/os-autoinst/pull/1784" class="external">https://github.com/os-autoinst/os-autoinst/pull/1784</a></li>
</ul>
<hr>
<p>Intneral GitLab repositories which need to be updated as well:</p>
<ul>
<li><a href="https://gitlab.suse.de/openqa/ci/-/ci/editor" class="external">https://gitlab.suse.de/openqa/ci/-/ci/editor</a></li>
<li><a href="https://gitlab.suse.de/openqa/osd-deployment/-/ci/editor" class="external">https://gitlab.suse.de/openqa/osd-deployment/-/ci/editor</a></li>
</ul>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4516562021-10-02T04:09:43Zopenqa_reviewopenqa-review@suse.de
<ul><li><strong>Due date</strong> set to <i>2021-10-16</i></li></ul><p>Setting due date based on mean cycle time of SUSE QE Tools</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4519472021-10-04T14:43:03Zmkittlermarius.kittler@suse.com
<ul></ul><p>I've been adjusting the project config but two of the containers are now unresolvable with "nothing provides this-is-only-for-build-envs needed by postgresql13-devel-mini":</p>
<ul>
<li><a href="https://build.opensuse.org/package/show/devel:openQA:ci/base" class="external">https://build.opensuse.org/package/show/devel:openQA:ci/base</a></li>
<li><a href="https://build.opensuse.org/package/show/devel:openQA/openqa_dev" class="external">https://build.opensuse.org/package/show/devel:openQA/openqa_dev</a></li>
</ul>
<p>Not sure yet why this is a problem with Leap 15.3. I get the same error on my TW system when trying to install the package:</p>
<pre><code>sudo zypper in postgresql13-devel-mini
[sudo] Passwort für root:
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...
Paketabhängigkeiten werden aufgelöst...
Problem: nichts stellt 'this-is-only-for-build-envs' bereit, das vom zu installierenden postgresql13-devel-mini-13.4-1.2.x86_64 benötigt wird
Lösung 1: postgresql13-devel-mini-13.4-1.2.x86_64 nicht installieren
Lösung 2: postgresql13-devel-mini-13.4-1.2.x86_64 durch Ignorieren einiger Abhängigkeiten brechen
Wählen Sie aus den obigen Lösungen mittels Nummer oder brechen Sie (a)b [1/2/a/d/?] (a):
</code></pre>
<p>The package <code>postgresql13-devel</code> would be resolvable. However, adding <code>Prefer: postgresql13-devel</code> (or <code>postgresql12-devel</code> or <code>-postgresql12-devel</code>) to the project config doesn't help.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4522502021-10-05T11:51:41Zmkittlermarius.kittler@suse.com
<ul></ul><p>With help from @fvogt I could workaround the problem. The project config now looks like this:</p>
<pre><code># extension of `Substitute: build-packages:docker …` found in the project config of openSUSE:Leap:15.3 to add missing substitutes for PostgreSQL
Substitute: build-packages:docker !systemd-mini !udev-mini !krb5-mini !libsystemd0-mini !libudev-mini1 !krb5-mini !gettext-tools-mini !cmake-mini !systemd-mini-sysvinit !dummy-release !libunbound-devel-mini !gettext-runtime-mini !postgresql13-devel-mini
# choose ruby version for Leap 15.3
Prefer: ruby2.5-rubygem-rb-fsevent-0.11
</code></pre>
<hr>
<p>I've been creating a PR to adapt our scripts: <a href="https://github.com/os-autoinst/openQA/pull/4267" class="external">https://github.com/os-autoinst/openQA/pull/4267</a></p>
<hr>
<p>I've also noticed that the container paths for the <code>openqa_dev</code> image are wrong and should likely be updated for Leap 15.3 as well: <a href="https://github.com/os-autoinst/openQA/pull/4268" class="external">https://github.com/os-autoinst/openQA/pull/4268</a></p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4522592021-10-05T12:21:09Zmkittlermarius.kittler@suse.com
<ul></ul><p>Further SR for GitLab projects: <a href="https://gitlab.suse.de/openqa/osd-deployment/-/merge_requests/39" class="external">https://gitlab.suse.de/openqa/osd-deployment/-/merge_requests/39</a>, <a href="https://gitlab.suse.de/mkittler/ci/-/merge_requests/1" class="external">https://gitlab.suse.de/mkittler/ci/-/merge_requests/1</a></p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4523042021-10-05T14:18:08Zokurzokurz@suse.com
<ul></ul><p>What's <a href="https://gitlab.suse.de/mkittler/ci/-/merge_requests/1" class="external">https://gitlab.suse.de/mkittler/ci/-/merge_requests/1</a> ?</p>
<p>openQA CI tests have failed now because some packages have not been found. I now did <code>for i in perl-Furl perl-Selenium-Remote-Driver perl-Test-Warnings; do osc linkpac openSUSE:Factory $i devel:openQA:Leap:15.3; done</code>.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4523282021-10-05T15:47:08Zmkittlermarius.kittler@suse.com
<ul></ul><p><a href="https://gitlab.suse.de/mkittler/ci/-/merge_requests/1" class="external">https://gitlab.suse.de/mkittler/ci/-/merge_requests/1</a> should have been created in <a href="https://gitlab.suse.de/openqa/ci" class="external">https://gitlab.suse.de/openqa/ci</a> although I'm still not sure about it anyways.</p>
<hr>
<p>Thanks for linking missing packages. I'll check whether they build and whether something is still missing.<br>
EDIT: They could be built and I re-triggered tests on <a href="https://github.com/os-autoinst/openQA/pull/4266" class="external">https://github.com/os-autoinst/openQA/pull/4266</a> to check whether it works now or what else is missing.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4523312021-10-05T15:49:48Zmkittlermarius.kittler@suse.com
<ul></ul><p>And I've closed <a href="https://github.com/os-autoinst/openQA/pull/4268" class="external">https://github.com/os-autoinst/openQA/pull/4268</a> and switched <code>containers</code> to Leap 15.3 (from TW).</p>
<p>This breaks now <a href="https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-kvm" class="external">https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-kvm</a> and <a href="https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86" class="external">https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86</a>. I'm not sure where these images are used. Maybe they should be moved to <code>containers-tw</code> now?</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4524452021-10-06T06:46:00Zokurzokurz@suse.com
<ul></ul><p>Also os-autoinst_dev could not build anymore hence blocking os-autoinst CI jobs.</p>
<p>I added an additional configuration</p>
<pre><code> <repository name="containers15.3">
<path project="devel:openQA" repository="openSUSE_Leap_15.3"/>
<path project="openSUSE:Containers:Leap:15.3" repository="containers"/>
<arch>x86_64</arch>
</repository>
</code></pre>
<p>for when anyone wants to explicitly mention the version 15.3 in the container image URL as is currently done in os-autoinst. Now <a href="https://build.opensuse.org/package/show/devel:openQA/os-autoinst_dev" class="external">https://build.opensuse.org/package/show/devel:openQA/os-autoinst_dev</a> is built for "containers" as well as "containers15.3" which should unblock the CI pipelines e.g. <a href="https://github.com/os-autoinst/os-autoinst/pull/1792/checks?check_run_id=3811612581" class="external">https://github.com/os-autoinst/os-autoinst/pull/1792/checks?check_run_id=3811612581</a></p>
<p>mkittler wrote:</p>
<blockquote>
<p>And I've closed <a href="https://github.com/os-autoinst/openQA/pull/4268" class="external">https://github.com/os-autoinst/openQA/pull/4268</a> and switched <code>containers</code> to Leap 15.3 (from TW).</p>
<p>This breaks now <a href="https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-kvm" class="external">https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-kvm</a> and <a href="https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86" class="external">https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86</a>. I'm not sure where these images are used.</p>
</blockquote>
<p>The images are referenced on <a href="https://github.com/os-autoinst/os-autoinst/blob/master/README.asciidoc" class="external">https://github.com/os-autoinst/os-autoinst/blob/master/README.asciidoc</a> for direct use and also they are used in<br>
<a href="https://github.com/os-autoinst/os-autoinst-distri-example/tree/master/.github/workflows" class="external">https://github.com/os-autoinst/os-autoinst-distri-example/tree/master/.github/workflows</a></p>
<p>Considering that sometimes Leap breaks, sometimes Tumbleweed we should have an easier time just building for both and referencing in README that an alternative exists if one does not work</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4524482021-10-06T06:49:56Zokurzokurz@suse.com
<ul></ul><p>Understanding now that e.g. <a href="https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86" class="external">https://build.opensuse.org/package/show/devel:openQA/isotovideo-qemu-x86</a> is unresolvable as it was relying on "containers" referring to Tumbleweed I have to reconsider my earlier suggestion about making "containers" latest Leap, I forgot about such use cases. I now find no Tumbleweed container definition in <a href="https://build.opensuse.org/projects/devel:openQA/meta" class="external">https://build.opensuse.org/projects/devel:openQA/meta</a> at all. Could you please add that back as you suggested as "containers-tw" and then we can still decide what our "default" should be?</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4526252021-10-06T08:30:20Zmkittlermarius.kittler@suse.com
<ul></ul><p>Yes, there's currently no TW containers repository. I can add it back as <code>containers-tw</code>.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4526702021-10-06T09:05:09Zmkittlermarius.kittler@suse.com
<ul></ul><p>It looks like the dependency PR works again but the documentation generation still fails with an error, see <a href="https://app.circleci.com/pipelines/github/os-autoinst/openQA/7961/workflows/2ad713ac-3932-411f-bb71-6defc4df31c3/jobs/75161" class="external">https://app.circleci.com/pipelines/github/os-autoinst/openQA/7961/workflows/2ad713ac-3932-411f-bb71-6defc4df31c3/jobs/75161</a>.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4527062021-10-06T09:58:32Zmkittlermarius.kittler@suse.com
<ul></ul><blockquote>
<p>Could you please add that back as you suggested as "containers-tw"</p>
</blockquote>
<p>Done, none of the container "packages" in <code>devel:openQA</code> are unresolvable anymore.</p>
<hr>
<p>The error for the documentation creation looked differently when I re-triggered it yesterday. Now it even fails earlier because the cache is empty:</p>
<pre><code>#!/bin/bash -eo pipefail
.circleci/build_or_install_from_cache.sh
+ '[' -z 0ee7ac9e-08fe-4084-8109-9a54d80dc80e ']'
++ find /var/cache/zypp/packages/
++ grep '.rpm$'
+ packages=
+ echo 'No RPM packages found, cache is empty, aborting'
No RPM packages found, cache is empty, aborting
+ exit 1
Exited with code exit status 1
CircleCI received exit code 1
</code></pre>
<p>Looks like it references to a cache which is likely just cleaned up at this point:</p>
<pre><code>No cache is found for key: v1-AM9kk913NN5TadgjfGzdQgkGS+jtsx2nhomGH_x_mCE=-3q6IWLmZ9Ai9ZBXW07jIuRUvGji2w5NiECwz_qDjx3Y=
</code></pre> openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4530002021-10-07T08:26:38Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li></ul><p>The latest nightly jobs worked again: <a href="https://app.circleci.com/pipelines/github/os-autoinst/openQA/7982/workflows/e96309d8-9ff6-4f36-90b0-029673117a79" class="external">https://app.circleci.com/pipelines/github/os-autoinst/openQA/7982/workflows/e96309d8-9ff6-4f36-90b0-029673117a79</a></p>
<p>I'm really wondering how I could have CircleCI to ditch the old cache instead of just waiting until it happens but at least it works now.</p>
<hr>
<p>This still leaves <a href="https://gitlab.suse.de/openqa/ci" class="external">https://gitlab.suse.de/openqa/ci</a> where I'd needed a TW container with Git and SSH installed. However, I don't even know what this repository is used for and half of the code is commented-out. So I assume it is WIP and maybe not worth updating at this point.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4532222021-10-07T18:50:04Zokurzokurz@suse.com
<ul></ul><p>mkittler wrote:</p>
<blockquote>
<p>This still leaves <a href="https://gitlab.suse.de/openqa/ci" class="external">https://gitlab.suse.de/openqa/ci</a> where I'd needed a TW container with Git and SSH installed. However, I don't even know what this repository is used for and half of the code is commented-out. So I assume it is WIP and maybe not worth updating at this point.</p>
</blockquote>
<p>If you mean <a href="https://gitlab.suse.de/openqa/ci/-/blob/master/ci/ssh-deploy.yml#L24" class="external">https://gitlab.suse.de/openqa/ci/-/blob/master/ci/ssh-deploy.yml#L24</a> with "commented-out" then you are misreading that. This line says "Use .setup_ssh like below:", what follows is example code. The definitions from this gitlab project are included in other projects, e.g. <a href="https://gitlab.suse.de/openqa/osd-deployment/-/blob/master/.gitlab-ci.yml#L31" class="external">https://gitlab.suse.de/openqa/osd-deployment/-/blob/master/.gitlab-ci.yml#L31</a></p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4533662021-10-08T08:16:06Zmkittlermarius.kittler@suse.com
<ul></ul><p>Ok, so we need a TW image with Git and OpenSSH installed. Should I create one in my own home project? It would have the disadvantage that we're spreading container images over multiple home projects. You could also give me permission to create a "package" in your home project.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4534982021-10-08T14:23:05Zokurzokurz@suse.com
<ul></ul><p>mkittler wrote:</p>
<blockquote>
<p>Ok, so we need a TW image with Git and OpenSSH installed. Should I create one in my own home project? It would have the disadvantage that we're spreading container images over multiple home projects.</p>
</blockquote>
<p>yes, better not. I created <a href="https://build.opensuse.org/package/show/home:okurz:container:ca/curl-jq-ssh-git" class="external">https://build.opensuse.org/package/show/home:okurz:container:ca/curl-jq-ssh-git</a> now</p>
<blockquote>
<p>You could also give me permission to create a "package" in your home project.</p>
</blockquote>
<p>Good idea. I have also done that</p>
<p>A user hit a problem regarding using container images: <a href="https://suse.slack.com/archives/C02CANHLANP/p1633702326174700" class="external">https://suse.slack.com/archives/C02CANHLANP/p1633702326174700</a></p>
<p>So in <a href="https://github.com/os-autoinst/openQA/commit/fcb639102491abf70c3685c04497af07e7b311f0" class="external">https://github.com/os-autoinst/openQA/commit/fcb639102491abf70c3685c04497af07e7b311f0</a> we have updated container definitions to 15.3 but both <a href="https://build.opensuse.org/package/view_file/devel:openQA/openQA_container_worker/_service:download_url:Dockerfile?expand=1" class="external">https://build.opensuse.org/package/view_file/devel:openQA/openQA_container_worker/_service:download_url:Dockerfile?expand=1</a> and <a href="https://build.opensuse.org/package/show/devel:openQA/openQA_container_webUI" class="external">https://build.opensuse.org/package/show/devel:openQA/openQA_container_webUI</a> were not yet configured to build for 15.3. Also seems like no service was triggered to update the Dockerfile that is extracted. Looking into this …</p>
<p>EDIT: Manually triggering the services has properly updated the Dockerfile in both packages. I consider this a bug on our side that we should followup with.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4535042021-10-08T15:15:18Zokurzokurz@suse.com
<ul></ul><p>registry.opensuse.org/home/okurz/container/ca/containers/tumbleweed:curl-jq-ssh is live now</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4538272021-10-11T09:23:51Zmkittlermarius.kittler@suse.com
<ul></ul><p>SR: <a href="https://gitlab.suse.de/openqa/ci/-/merge_requests/3" class="external">https://gitlab.suse.de/openqa/ci/-/merge_requests/3</a></p>
<p>I'm looking into the containerized setup. It was likely broken by <a href="https://github.com/os-autoinst/openQA/pull/4257" class="external">https://github.com/os-autoinst/openQA/pull/4257</a>.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4541482021-10-11T14:45:37Zmkittlermarius.kittler@suse.com
<ul></ul><p>It looks like the containerized setup via <code>docker-compose</code> works. I don't know what use the example in the <a href="https://open.qa/docs/#_container_based_setup" class="external">https://open.qa/docs/#_container_based_setup</a> section has. Maybe it is normal that it doesn't work out of the box. I consider fixing this out of the scope of this issue because I don't even know what's supposed to happen when one is executing this command.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4550842021-10-13T13:21:57Zmkittlermarius.kittler@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>The SR has been merged and <a class="user active user-mention" href="https://progress.opensuse.org/users/17668">@okurz</a> mentions that dependent pipelines work.</p>
<p>This was the last remaining container so I'm resolving the issue.</p>
openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4605612021-11-02T09:26:52Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> deleted (<del><i>2021-10-16</i></del>)</li></ul> openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=4645492021-11-15T13:28:58Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" href="/issues/102464">action #102464</a>: Upgrade OBS package CI checks to Leap 15.3 (os-autoinst+openQA) size:M</i> added</li></ul> openQA Infrastructure - action #99240: Upgrade CI container image versions to Leap 15.3 size:Mhttps://progress.opensuse.org/issues/99240?journal_id=5245912022-05-31T18:19:55Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed child" href="/issues/111881">action #111881</a>: Upgrade CI container image versions to Leap 15.4 size:M</i> added</li></ul>