https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842022-05-31T18:19:53ZopenSUSE Project Management ToolopenQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5245582022-05-31T18:19:53Zokurzokurz@suse.com
<ul><li><strong>Copied from</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed child" href="/issues/99189">action #99189</a>: Upgrade o3 workers to openSUSE Leap 15.3 size:M</i> added</li></ul> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5246092022-05-31T18:20:38Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>Upgrade o3 workers to openSUSE Leap 15.3 size:M</i> to <i>Upgrade o3 workers to openSUSE Leap 15.4 size:M</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/524609/diff?detail_id=496256">diff</a>)</li><li><strong>Assignee</strong> deleted (<del><i>mkittler</i></del>)</li><li><strong>Target version</strong> changed from <i>Ready</i> to <i>future</i></li></ul> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5246332022-05-31T18:26:44Zokurzokurz@suse.com
<ul><li><strong>Project</strong> changed from <i>openQA Project</i> to <i>openQA Infrastructure</i></li><li><strong>Subject</strong> changed from <i>Upgrade o3 workers to openSUSE Leap 15.4 size:M</i> to <i>Upgrade o3 workers to openSUSE Leap 15.4</i></li></ul> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5341852022-07-06T11:10:44Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-5 priority-high3 closed" href="/issues/111992">action #111992</a>: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl size:M</i> added</li></ul> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5341912022-07-06T11:10:50Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Blocked</i></li><li><strong>Assignee</strong> set to <i>okurz</i></li><li><strong>Target version</strong> changed from <i>future</i> to <i>Ready</i></li></ul><p><a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl size:M (Resolved)" href="https://progress.opensuse.org/issues/111992">#111992</a></p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5363392022-07-13T13:40:41Ztinitatina.mueller+trick-redmine@suse.com
<ul><li><strong>Status</strong> changed from <i>Blocked</i> to <i>In Progress</i></li><li><strong>Assignee</strong> changed from <i>okurz</i> to <i>tinita</i></li></ul><p><a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: Deal with QEMU and OVMF default resolution being 1280x800, affecting (at least) qxl size:M (Resolved)" href="https://progress.opensuse.org/issues/111992">#111992</a> Resolved so this is workable</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5363842022-07-13T14:54:30Ztinitatina.mueller+trick-redmine@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/536384/diff?detail_id=507338">diff</a>)</li></ul> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5364802022-07-14T04:40:24Zopenqa_reviewopenqa-review@suse.de
<ul><li><strong>Due date</strong> set to <i>2022-07-28</i></li></ul><p>Setting due date based on mean cycle time of SUSE QE Tools</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5365962022-07-14T09:24:47Zmkittlermarius.kittler@suse.com
<ul><li><strong>Subject</strong> changed from <i>Upgrade o3 workers to openSUSE Leap 15.4</i> to <i>Upgrade o3 workers to openSUSE Leap 15.4 size:M</i></li></ul> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5369952022-07-14T18:10:42Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>I tried to upgrade openqaworker7 but got a problem:</p>
<pre><code>Problem: the to be installed os-autoinst-devel-4.6.1657781352.4f16c4d-lp154.1314.2.x86_64 requires 'pkgconfig(opencv)', but this requirement cannot be provided
deleted providers: opencv-devel-3.3.1-6.6.1.x86_64
not installable providers: opencv3-devel-3.4.16-150400.1.9.x86_64[repo-oss]
Solution 1: Following actions will be done:
keep obsolete opencv-devel-3.3.1-6.6.1.x86_64
keep obsolete opencv-3.3.1-6.6.1.x86_64
Solution 2: deinstallation of opencv-devel-3.3.1-6.6.1.x86_64
Solution 3: deinstallation of os-autoinst-devel-4.6.1657727584.884de90-lp153.1313.1.x86_64
Solution 4: keep obsolete opencv-devel-3.3.1-6.6.1.x86_64
Solution 5: break os-autoinst-devel-4.6.1657781352.4f16c4d-lp154.1314.2.x86_64 by ignoring some of its dependencies
</code></pre>
<p>Not sure what to do.</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5373372022-07-15T10:08:29Zokurzokurz@suse.com
<ul></ul><p>According to <a href="https://build.opensuse.org/package/show/openSUSE:Leap:15.3:Update/opencv" class="external">https://build.opensuse.org/package/show/openSUSE:Leap:15.3:Update/opencv</a> we had opencv3 in Leap15.3, in <a href="https://build.opensuse.org/package/show/openSUSE:Leap:15.4:Update/opencv" class="external">https://build.opensuse.org/package/show/openSUSE:Leap:15.4:Update/opencv</a> we have opencv4, same as in Tumbleweed. In <a href="https://github.com/os-autoinst/os-autoinst/blob/master/dist/rpm/os-autoinst.spec" class="external">https://github.com/os-autoinst/os-autoinst/blob/master/dist/rpm/os-autoinst.spec</a> line 28 and following we have:</p>
<pre><code>%if 0%{?suse_version} > 1500
# openSUSE Tumbleweed
%define opencv_require pkgconfig(opencv4)
%else
%define opencv_require pkgconfig(opencv)
%endif
</code></pre>
<p>so likely we need to update this according to <a href="https://en.opensuse.org/openSUSE:Build_Service_cross_distribution_howto#Detect_a_distribution_flavor_for_special_code" class="external">https://en.opensuse.org/openSUSE:Build_Service_cross_distribution_howto#Detect_a_distribution_flavor_for_special_code</a> to make sure users upgrading from Leap 15.3 to 15.4 have an automatic resolution.</p>
<p>I suggest to reproduce in a clean container environment, e.g. podman run --rm -it registry.opensuse.org/opensuse/leap:15.3, and see how we can upgrade. At best we would also provide a maintenance update to the vanilla version in os-autoinst. I will create a new ticket for that specifically.</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5373432022-07-15T10:10:34Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed child" href="/issues/113662">action #113662</a>: Ensure upgrade of vanilla Leap 15.3 os-autoinst can be automatically upgraded to Leap 15.4 size:S</i> added</li></ul> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5373492022-07-15T10:11:05Zlivdywanliv.dywan@suse.com
<ul></ul><ul>
<li>Ondrėj suggests to add a version requirement for opencv3-devel based on the distribution version, which was called opencv-devel on older releases, in the os-autoinst spec file.</li>
<li>Maybe it's enough to decrease <code>%if 0%{?suse_version} > 1500</code> to use a higher version? See <a href="https://github.com/os-autoinst/os-autoinst/blob/master/dist/rpm/os-autoinst.spec" class="external">https://github.com/os-autoinst/os-autoinst/blob/master/dist/rpm/os-autoinst.spec</a>
<ul>
<li><a href="https://github.com/os-autoinst/os-autoinst/pull/2125" class="external">https://github.com/os-autoinst/os-autoinst/pull/2125</a></li>
</ul></li>
</ul>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5373702022-07-15T10:18:46Zfavogtfvogt@suse.com
<ul></ul><p>I wonder what <code>os-autoinst-devel</code> is actually needed for on workers. On ow7 it can apparently be removed without breaking any dependencies.</p>
<p>ow7 was also installed with 15.4 already (not upgraded) and has <code>opencv3-devel</code> installed to provide <code>pkgconfig(opencv)</code>. So to get the same setup on ow4, solution 2 would be correct.</p>
<p>If switching os-autoinst on 15.4 to OpenCV 4 is ok, then that is probably the best option. That shouldn't require any further changes; during the upgrade it would simply stay with opencv-devel, upgrading from 3 to 4.</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5373762022-07-15T10:19:44Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>Blocked by <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Ensure upgrade of vanilla Leap 15.3 os-autoinst can be automatically upgraded to Leap 15.4 size:S (Resolved)" href="https://progress.opensuse.org/issues/113662">#113662</a></p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5374992022-07-15T13:11:32Zokurzokurz@suse.com
<ul></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/33482">@tinita</a> this ticket is <em>not</em> blocked by <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Ensure upgrade of vanilla Leap 15.3 os-autoinst can be automatically upgraded to Leap 15.4 size:S (Resolved)" href="https://progress.opensuse.org/issues/113662">#113662</a> which is only about the <em>vanilla</em> version of os-autoinst, i.e. not from devel:openQA. On o3 workers we run from devel:openQA so we can expect likely the same fix but have it available much sooner and no need for a maintenance update.</p>
<p>favogt wrote:</p>
<blockquote>
<p>I wonder what <code>os-autoinst-devel</code> is actually needed for on workers. On ow7 it can apparently be removed without breaking any dependencies.</p>
</blockquote>
<p>You are right, os-autoinst-devel is not needed for the operation of workers, just necessary to build our own. Why we have that on workers I don't know, maybe was done for experiments, maybe for the external video encoder?</p>
<p>If I would uninstall with dependencies removed as well then it would look like this:</p>
<pre><code>okurz@ariel:~> for i in aarch64 openqaworker1 openqaworker4 openqaworker7 imagetester rebel power8; do echo "## $i" && ssh root@$i "zypper rm --clean-deps --dry-run os-autoinst-devel"; done
## aarch64
Reading installed packages...
Resolving package dependencies...
The following 102 packages are going to be REMOVED:
aspell aspell-en aspell-spell cmake cmake-full cmake-man fftw3-devel gcc7-c++ gcc-c++ glu-devel kbproto-devel libenca0 libfftw3_threads3 libglvnd-devel libguess1 libICE-devel libjsoncpp19 libogg-devel libpng16-compat-devel libpng16-devel librcc0 librcd0 librhash0 libSM-devel libsndfile-devel libstdc++6-devel-gcc7 libstdc++-devel libtheora0 libtheora-devel libX11-devel libXau-devel libxcb-composite0 libxcb-damage0 libxcb-devel libxcb-dpms0 libxcb-record0 libxcb-res0 libxcb-screensaver0 libxcb-xf86dri0 libxcb-xtest0 libxcb-xv0 libxcb-xvmc0 libXext-devel Mesa-KHR-devel Mesa-libGL-devel ninja opencv opencv-devel os-autoinst-devel perl-Algorithm-Diff perl-AppConfig perl-B-Debug perl-Browser-Open perl-Class-Accessor-Lite perl-Code-TidyAll perl-Config-INI perl-Devel-Cover perl-Devel-Cover-Report-Codecov perl-Export-Attrs perl-Exporter-Lite perl-File-pushd perl-Furl perl-HTTP-Parser-XS perl-IPC-Run3 perl-JSON-MaybeXS perl-List-Compare perl-List-SomeUtils perl-List-SomeUtils-XS perl-Log-Any perl-Mixin-Linewise perl-Module-Find perl-Net-IDN-Encode perl-PerlIO-utf8_strict perl-Scope-Guard perl-Sereal-Decoder perl-Sereal-Encoder perl-Specio-Library-Path-Tiny perl-Sub-Retry perl-Sub-Uplevel perl-SUPER perl-Template-Toolkit perl-Test-Differences perl-Test-Exception perl-Test-MockModule perl-Test-MockObject perl-Test-MockRandom perl-Test-Mock-Time perl-Test-Most perl-Test-Output perl-Test-Strict perl-Test-Warn perl-Test-Warnings perl-Text-Diff perl-Time-Duration-Parse perl-UNIVERSAL-can perl-UNIVERSAL-isa pthread-stubs-devel rcc-runtime ShellCheck xextproto-devel xproto-devel zlib-devel
…
## power8
Reading installed packages...
Package 'os-autoinst-devel' not found.Resolving package dependencies...
Nothing to do.
</code></pre>
<p>So removing os-autoinst-devel and carefully removing unneeded dependencies is also fine for me.</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5375532022-07-15T15:03:30Zokurzokurz@suse.com
<ul></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/21806">@osukup</a> please observe <a href="https://github.com/os-autoinst/os-autoinst/pull/2125#issuecomment-1185632899" class="external">https://github.com/os-autoinst/os-autoinst/pull/2125#issuecomment-1185632899</a></p>
<p>Testing an according maintenance request with <code>podman run --rm -it registry.opensuse.org/opensuse/leap:15.3 /bin/sh -c 'zypper -n ar -f https://download.opensuse.org/repositories/home:/okurz:/branches:/openSUSE:/Backports:/SLE-15-SP4:/Update/standard/home:okurz:branches:openSUSE:Backports:SLE-15-SP4:Update.repo && zypper -n --gpg-auto-import-keys ref && zypper -n up && zypper -n in os-autoinst-devel && zypper --releasever=15.4 -n dup'</code></p>
<p>I ran into</p>
<pre><code>Problem: nothing provides 'pkgconfig(opencv4)' needed by the to be installed os-autoinst-devel-4.6.1639403953.ae94c4bd-bp154.4.1.x86_64
Solution 1: do not install os-autoinst-devel-4.6.1639403953.ae94c4bd-bp154.4.1.x86_64
Solution 2: break os-autoinst-devel-4.6.1639403953.ae94c4bd-bp154.4.1.x86_64 by ignoring some of its dependencies
</code></pre>
<p>so seems like something is still missing?</p>
<p>EDIT: As commented on <a href="https://build.opensuse.org/request/show/989570#comment-1653437" class="external">https://build.opensuse.org/request/show/989570#comment-1653437</a> we just need to accept the conflict and resolve manually whenever hit.</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5398392022-07-21T14:10:24Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>Upgraded aarch64 openqaworker1 openqaworker7 power8 imagetester rebel</p>
<p>Notes:</p>
<pre><code>/etc/zypp/zypp.conf
Suggestion was to add:
multiversion = provides:multiversion(kernel)
but we kept the empty value
/etc/postfix/main.cf
inet_interfaces = all
inet_protocols = ipv4
masquerade_exceptions =
vs. rpmsave
inet_interfaces = localhost
inet_protocols = all
masquerade_exceptions = root
We kept the automatic changes
On openqaworker4we changed:
/etc/zypp/zypp.conf
old:
solver.dupAllowVendorChange = true
new:
solver.dupAllowVendorChange = true
/etc/containers/storage.conf
# Default Storage Driver
driver = "btrfs"
rpmnew suggests:
driver = "overlay"
we kept the old value
</code></pre> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5398422022-07-21T14:13:12Zmkittlermarius.kittler@suse.com
<ul></ul><p>I'm adding <code>noauto,nofail,retry=30,ro,x-systemd.automount,x-systemd.device-timeout=10m,x-systemd.mount-timeout=30m</code> as config for the NFS mount on all o3 workers (in accordance with OSD workers) as the mount unit failed after rebooting aarch64 after the upgrade (because it couldn't resolve the hostname). I excluded rebel as it has no nfs mount anyways <del>and openqaworker7 as it couldn't boot at all</del>.</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5398452022-07-21T14:18:05Ztinitatina.mueller+trick-redmine@suse.com
<ul></ul><p>Leftovers:</p>
<a name="power8"></a>
<h2 >power8<a href="#power8" class="wiki-anchor">¶</a></h2>
<pre><code>power8:~ # systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION >
● systemd-hibernate-resume@dev-disk-by\x2did-scsi\x2d1IBM_IPR\x2d0_5DB3D30000000020\x2dpart2.service loaded failed failed Resume from hibernation using device /dev/disk/by-id/scsi-1IBM_IPR-0_5DB3D30000000020-p>
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed.
</code></pre>
<p>Find out about this failure.</p>
<a name="openqaworker7"></a>
<h2 >openqaworker7<a href="#openqaworker7" class="wiki-anchor">¶</a></h2>
<p>does not boot</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5398542022-07-21T14:25:10Zokurzokurz@suse.com
<ul></ul><p>tinita wrote:</p>
<blockquote>
<p>Leftovers:</p>
<a name="power8"></a>
<h2 >power8<a href="#power8" class="wiki-anchor">¶</a></h2>
<pre><code>power8:~ # systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION >
● systemd-hibernate-resume@dev-disk-by\x2did-scsi\x2d1IBM_IPR\x2d0_5DB3D30000000020\x2dpart2.service loaded failed failed Resume from hibernation using device /dev/disk/by-id/scsi-1IBM_IPR-0_5DB3D30000000020-p>
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
1 loaded units listed.
</code></pre>
<p>Find out about this failure.</p>
</blockquote>
<p>It is not a new problem but has been seen for very long. It's just that we couldn't completely mask this service so far.</p>
<p>EDIT: I tried again with masking the service and triggering a reboot. Did not work. I removed all mentions of "resume=UUID…" in /etc and /boot. Rebooted, no failed service. That seems to have helped.</p>
<blockquote>
<a name="openqaworker7"></a>
<h2 >openqaworker7<a href="#openqaworker7" class="wiki-anchor">¶</a></h2>
<p>does not boot</p>
</blockquote>
<p>it does boot just that it takes about 10 minutes due to various firmware initialization, PXE boot menu with timeout, then falling back to some other init until it reaches the grub menu where there is again a timeout until it boots the OS.</p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5398782022-07-21T15:39:57Zmkittlermarius.kittler@suse.com
<ul></ul><p>openqaworker7 is meanwhile reachable via SSH, the only failed service is <code>snapper-cleanup.service</code></p>
openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=5398992022-07-21T20:13:04Zokurzokurz@suse.com
<ul><li><strong>Due date</strong> deleted (<del><i>2022-07-28</i></del>)</li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>mkittler wrote:</p>
<blockquote>
<p>openqaworker7 is meanwhile reachable via SSH, the only failed service is <code>snapper-cleanup.service</code></p>
</blockquote>
<p>openqaworker7 has an excessive amount of btrfs snapshots, <code>for i in aarch64 openqaworker1 openqaworker4 openqaworker7 imagetester rebel power8; do echo "## $i" && ssh root@$i "snapper ls | wc -l"; done</code> yields:</p>
<pre><code>## aarch64
26
## openqaworker1
23
## openqaworker4
20
## openqaworker7
1390
## imagetester
19
## rebel
23
## power8
The config 'root' does not exist. Likely snapper is not configured.
See 'man snapper' for further instructions.
0
</code></pre>
<p>I think we had this problem already some time ago related to podman and btrfs subvolumes occupied and preventing a proper cleanup. Yep, there was <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: Failed systemd services alert: snapper-cleanup on QA-Power8-4-kvm fails size:M (Resolved)" href="https://progress.opensuse.org/issues/102942">#102942</a> and <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: aarch64.o.o fails to delete snapshots in snapper-cleanup since 2020-03-20 repeatedly (Resolved)" href="https://progress.opensuse.org/issues/65073">#65073</a>, in both cases related to some nested subvolumes that snapper fails/refuses to delete. So manual cleanup it is:</p>
<pre><code>btrfs subvolume list / | grep subvolumes | sed -n 's/^.*path @//p' | xargs btrfs subvolume delete && systemctl start snapper-cleanup
</code></pre>
<p>This is very slowly churning down the number of snapshots but seems to take some time. Anyway, overall we seem to be good:</p>
<pre><code>for i in aarch64 openqaworker1 openqaworker4 openqaworker7 imagetester rebel power8; do echo "## $i" && ssh root@$i "systemctl is-system-running; rpmconfigcheck"; done
</code></pre> openQA Infrastructure - action #111863: Upgrade o3 workers to openSUSE Leap 15.4 size:Mhttps://progress.opensuse.org/issues/111863?journal_id=6413442023-06-09T07:43:01Zokurzokurz@suse.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed child" href="/issues/130585">action #130585</a>: Upgrade o3 workers to openSUSE Leap 15.5</i> added</li></ul>