openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-15T10:13:41ZopenSUSE Project Management Tool
Redmine openQA Project - action #157333 (Closed): Log all job setting changes in autoinst-log.txthttps://progress.opensuse.org/issues/1573332024-03-15T10:13:41ZMDouchamartin.doucha@suse.com
<p>All job settings should be logged in autoinst-log.txt with source of the value (e.g. the place where <code>set_var()</code> was called or whether they were added from product/medium/worker etc.)</p>
openQA Infrastructure - action #138746 (Resolved): [tools] s390x VM randomly fails to open QCOW d...https://progress.opensuse.org/issues/1387462023-10-30T12:45:10ZMDouchamartin.doucha@suse.com
<p>s390x tests randomly fail to boot because the VM does not have permission to open the disk image. Multiple workers have the same issue. Restarting the job usually fixes the issue. Examples:</p>
<p><a href="https://openqa.suse.de/tests/12711015#step/bootloader_zkvm/31" class="external">https://openqa.suse.de/tests/12711015#step/bootloader_zkvm/31</a><br>
<a href="https://openqa.suse.de/tests/12711015/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/12711015/logfile?filename=autoinst-log.txt</a></p>
<p><a href="https://openqa.suse.de/tests/12716015#step/bootloader_zkvm/31" class="external">https://openqa.suse.de/tests/12716015#step/bootloader_zkvm/31</a><br>
<a href="https://openqa.suse.de/tests/12716015/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/12716015/logfile?filename=autoinst-log.txt</a></p>
<p><a href="https://openqa.suse.de/tests/12708886#step/bootloader_start/34" class="external">https://openqa.suse.de/tests/12708886#step/bootloader_start/34</a><br>
<a href="https://openqa.suse.de/tests/12708886/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/12708886/logfile?filename=autoinst-log.txt</a></p>
<pre><code>[2023-10-28T00:17:57.550325+02:00] [debug] [pid:56810] [run_ssh_cmd(virsh start openQA-SUT-6 2> >(tee /tmp/os-autoinst-openQA-SUT-6-stderr.log >&2))] stderr:
error: Failed to start domain 'openQA-SUT-6'
error: internal error: process exited while connecting to monitor: 2023-10-27T22:17:57.331249Z qemu-system-s390x: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images//SLES-15-SP4-s390x-mru-install-minimal-with-addons-Build20231027-1-Server-DVD-Updates-s390x-kvm.qcow2","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":true},"auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images//SLES-15-SP4-s390x-mru-install-minimal-with-addons-Build20231027-1-Server-DVD-Updates-s390x-kvm.qcow2': Permission denied
</code></pre> openQA Infrastructure - action #125798 (Resolved): Visual differences in GRUB menu on different x...https://progress.opensuse.org/issues/1257982023-03-10T17:09:54ZMDouchamartin.doucha@suse.com
<p>Here are 5 different LTP jobs booting the exact same UEFI/SecureBoot QCOW image on different workers:<br>
<a href="https://openqa.suse.de/tests/10651590" class="external">https://openqa.suse.de/tests/10651590</a> openqaworker16:14, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10658203" class="external">https://openqa.suse.de/tests/10658203</a> openqaworker16:18, pass<br>
<a href="https://openqa.suse.de/tests/10659306" class="external">https://openqa.suse.de/tests/10659306</a> openqaworker16:7, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659346" class="external">https://openqa.suse.de/tests/10659346</a> openqaworker17:12, GRUB needle mismatch<br>
<a href="https://openqa.suse.de/tests/10659359" class="external">https://openqa.suse.de/tests/10659359</a> worker9:11, pass</p>
<p>It appears that GRUB menu size changes depending on not just the worker but also specific worker slot.</p>
<p>Possibly related to <a href="https://progress.opensuse.org/issues/114523" class="external">poo#114523</a> but this time it's happening on x86_64.</p>
openQA Infrastructure - action #123960 (Resolved): s390x tests fail to log into VNC console on wo...https://progress.opensuse.org/issues/1239602023-02-06T09:49:51ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>s390x tests started randomly failing last week when they try to log into the freshly booted test system. There are multiple instances across all SLE releases both before and after applying incident patches so this is most likely an infra issue.<br>
<a href="https://openqa.suse.de/tests/10427719#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10427719#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10427825#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10427825#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10432751#step/update_kernel/20" class="external">https://openqa.suse.de/tests/10432751#step/update_kernel/20</a><br>
<a href="https://openqa.suse.de/tests/10424294#step/boot_ltp/21" class="external">https://openqa.suse.de/tests/10424294#step/boot_ltp/21</a></p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/10424537" class="external">:27598:kgraft-patch-SLE12-SP5_Update_35</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/10424525" class="external">:27599:kgraft-patch-SLE12-SP5_Update_36</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Server-DVD-Incidents-Kernel&machine=s390x-kvm-sle12&test=install_ltp%2Bsle%2BServer-DVD-Incidents-Kernel&version=12-SP5" class="external">latest</a></p>
openQA Infrastructure - action #120339 (Resolved): QEMU DNS fails to resolve openqa.suse.de via I...https://progress.opensuse.org/issues/1203392022-11-11T12:48:43ZMDouchamartin.doucha@suse.com
<a name="Observation"></a>
<h1 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h1>
<p>LTP test <code>host</code> <a href="https://openqa.suse.de/tests/9927713#step/host/8" class="external">started failing today</a>. The QEMU DNS service running at 10.0.2.3 correctly resolves hostnames to IP addresses but reverse lookup fails. <a href="https://openqa.suse.de/tests/9915478#step/host/8" class="external">Old tests</a> which passed up until yesterday are now <a href="https://openqa.suse.de/tests/9930979#step/host/8" class="external">also failing upon restart</a> so this appears to be a QEMU configuration issue. The physical worker machine can resolve IP addresses without issue.</p>
<p>This issue is confirmed on worker3, worker5, worker8 and worker13. Other workers may be affected as well. PPC64LE QEMU workers do not seem to be affected, though.</p>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Revert removal of faulty DNS</li>
</ul>
<pre><code>sudo salt --no-color --state-output=changes -C 'G@roles:worker' cmd.run 'sudo sed -i "s/\(NETCONFIG_DNS_POLICY=\)\"\"/\1\"auto\"/;s/\(NETCONFIG_DNS_STATIC_SERVERS=\)\"10.160.0.1 10.100.2.10\"/\1\"\"/" /etc/sysconfig/network/config && sudo netconfig update -f'
</code></pre> openQA Tests - action #116287 (Rejected): [qe-core][s390x] SSH serial terminal connection issues ...https://progress.opensuse.org/issues/1162872022-09-06T13:54:08ZMDouchamartin.doucha@suse.com
<p>s390x livepatch tests had a lot of installation failures this month due to SSH serial terminal connection failures. Interestingly enough, the connection failures seem to happen around the same module step. serial_terminal.txt output appears to be out of sync with the terminal because part of the commands and output is missing even though it's listed in the update_kernel module details. The dmesg output in serial0.txt often (but not always) shows some key exchange SSH error followed by output from a completely different job:</p>
<pre><code>Welcome to SUSE Linux Enterprise Server 15 SP2 (s390x) - Kernel 5.3.18-24.83-default (ttysclp0).
eth0: 10.161.145.86 fe80::5054:ff:fe84:f877
susetest login: root
Password:
Last login: Mon Sep 5 10:18:10 from 10.160.0.147
susetest:~ #�(B systemctl is-active network
active
susetest:~ #�(B systemctl is-active sshd
active
susetest:~ #�(B 2022-09-05T10:25:03.604370-04:00 susetest sshd[4272]: error: kex_exchange_identification: Connection closed by remote host
2022-09-05T10:25:04.844743-04:00 susetest sshd[4273]: error: kex_exchange_identification: Connection closed by remote host
[ 107.444474] LTP: starting DI000 (dirty)
[ 107.445525] LTP: starting DS000 (dio_sparse)
[ 107.466125] LTP: starting abort01
[ 107.758318] LTP: starting accept01
</code></pre>
<p>12-SP4: <a href="https://openqa.suse.de/tests/9438804#step/update_kernel/337" class="external">https://openqa.suse.de/tests/9438804#step/update_kernel/337</a><br>
15-SP2: <a href="https://openqa.suse.de/tests/9457752#step/update_kernel/337" class="external">https://openqa.suse.de/tests/9457752#step/update_kernel/337</a><br>
15-SP3: <a href="https://openqa.suse.de/tests/9458645#step/update_kernel/337" class="external">https://openqa.suse.de/tests/9458645#step/update_kernel/337</a><br>
15-SP4: <a href="https://openqa.suse.de/tests/9455666#step/update_kernel/199" class="external">https://openqa.suse.de/tests/9455666#step/update_kernel/199</a></p>
<p>I could not find any such connection failure on SLE-12SP5. Other SLE releases don't support s390x livepatches and KOTD tests don't show this kind of issue. This looks like a kernel bug but I'd like an s390x expert to look at this before I create a Bugzilla ticket. And of course this has exposed logging issues in OpenQA.</p>
openQA Infrastructure - action #107989 (Resolved): CPU-specific worker classeshttps://progress.opensuse.org/issues/1079892022-03-08T11:28:38ZMDouchamartin.doucha@suse.com
<p>We have a few tests which require specific CPU types, for example the CPU vulnerability mitigation tests. It'd be useful to have worker classes like <code>x86_64_amd</code> and <code>x86_64_intel</code> so that we can schedule tests on workers which have the required features or vulnerabilities.</p>
openQA Infrastructure - action #105867 (Resolved): OpenQA bot schedules jobs with incomplete INCI...https://progress.opensuse.org/issues/1058672022-02-03T10:23:54ZMDouchamartin.doucha@suse.com
<p>This week, the OpenQA bot has been scheduling kernel tests without adding the Basesystem/LTSS repository to INCIDENT_REPO. Only the livepatching repository was added. This happened on <a href="https://openqa.suse.de/tests/8085238#settings" class="external">SLE-12SP4</a>, <a href="https://openqa.suse.de/tests/8082278#settings" class="external">SLE-15SP2</a> (<a href="https://openqa.suse.de/tests/8081179#settings" class="external">twice</a>) and <a href="https://openqa.suse.de/tests/8087134#settings" class="external">SLE-15SP1</a>:</p>
<pre><code>INCIDENT_REPO=http://download.suse.de/ibs/SUSE:/Maintenance:/22660/SUSE_Updates_SLE-Module-Live-Patching_15-SP1_x86_64
</code></pre>
<p>Some of these tests have already been rescheduled with the correct settings but SLE-15SP1 is still affected. Current S:M:22660 incident data in QEM dashboard API:</p>
<pre><code>{"approved":false,"channels":["SUSE:SLE-15-SP1:Update","SUSE:Updates:SLE-Product-HA:15-SP1:x86_64","SUSE:Updates:SLE-Product-HA:15-SP1:s390x","SUSE:Updates:SLE-Product-HA:15-SP1:ppc64le","SUSE:Updates:SLE-Product-HA:15-SP1:aarch64","SUSE:Updates:Storage:6:aarch64","SUSE:Updates:Storage:6:x86_64","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:x86_64","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:s390x","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:ppc64le","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP3:aarch64","SUSE:Updates:SLE-Module-Live-Patching:15-SP1:x86_64","SUSE:Updates:SLE-Module-Live-Patching:15-SP1:ppc64le","SUSE:Updates:SUSE-CAASP:4.0:x86_64","SUSE:Updates:SLE-Product-SLES:15-SP1-BCL:x86_64","SUSE:Updates:SLE-Product-HPC:15-SP1-ESPOS:aarch64","SUSE:Updates:SLE-Product-HPC:15-SP1-ESPOS:x86_64","SUSE:Updates:SLE-Product-SLES_SAP:15-SP1:ppc64le","SUSE:Updates:SLE-Product-SLES_SAP:15-SP1:x86_64","SUSE:Updates:SLE-Product-SLES:15-SP1-LTSS:x86_64","SUSE:Updates:SLE-Product-SLES:15-SP1-LTSS:s390x","SUSE:Updates:SLE-Product-SLES:15-SP1-LTSS:ppc64le","SUSE:Updates:SLE-Product-SLES:15-SP1-LTSS:aarch64","SUSE:Updates:SLE-Product-HPC:15-SP1-LTSS:x86_64","SUSE:Updates:SLE-Product-HPC:15-SP1-LTSS:aarch64","SUSE:Updates:openSUSE-SLE:15.3","SUSE:Updates:openSUSE-SLE:15.4","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:aarch64","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:ppc64le","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:s390x","SUSE:Updates:SLE-Module-Development-Tools-OBS:15-SP4:x86_64"],"emu":false,"inReview":false,"inReviewQAM":false,"isActive":true,"number":22660,"packages":["dtb-aarch64","kernel-debug","kernel-default","kernel-docs","kernel-kvmsmall","kernel-livepatch-SLE15-SP1_Update_28","kernel-obs-build","kernel-obs-qa","kernel-source","kernel-syms","kernel-vanilla","kernel-zfcpdump"],"project":"SUSE:Maintenance:22660","rr_number":null}
</code></pre> openQA Project - action #70189 (Rejected): openQA-common package broken on Tumbleweedhttps://progress.opensuse.org/issues/701892020-08-18T11:40:34ZMDouchamartin.doucha@suse.com
<p>When I install the standard <code>openQA-common</code> package on Tumbleweed and try to access the local webUI, I get error 502 and <code>journalctl</code> shows Perl errors with traceback (see below). When I replace the <code>/usr/share/openqa</code> directory with a symlink to my local copy of OpenQA git repo and restart the OpenQA services, everything works fine.</p>
<pre><code>$ journalctl -u openqa-webuid -b0
srp 18 13:25:25 dhcp165.suse.cz systemd[1]: Started The openQA web UI.
srp 18 13:25:26 dhcp165.suse.cz openqa-webui-daemon[17382]: [2020-08-18 13:25:26.96072] [17382] [warn] Deprecated use of config key '[audit]: blacklist'. Use '[audit]: blocklist' instead
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://127.0.0.1:9526
srp 18 13:25:27 dhcp165.suse.cz openqa-webui-daemon[17382]: Web application available at http://[::1]:9526
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 14, near "include_branding 'docbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template main/index.html.ep line 17, near "include_branding 'sponsorbox'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare include_branding?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: String found where operator expected at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: (Do you need to predeclare icon_url?)
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Mojo::Reactor::Poll: I/O watcher failed: syntax error at template layouts/error.html.ep line 35, near "icon_url 'logo-16.png'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 36, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: syntax error at template layouts/error.html.ep line 43, near "icon_url 'logo.svg'"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Context:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 30: } );
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 31: % end
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 32:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 33: <link rel="icon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 34: type="image/png" sizes="16x16"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 35: href="<%= icon_url 'logo-16.png' %>">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 36: <link rel="icon" href="<%= icon_url 'logo.svg'%>" sizes="any" type="image/svg+xml">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 37:
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 38: </head>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 39: <body>
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: 40: <nav class="navbar navbar-static-top navbar-default">
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: Traceback (most recent call first):
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 163, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Template.pm", line 173, in "Mojo::Template"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPLRenderer.pm", line 40, in "Mojolicious::Plugin::EPLRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/EPRenderer.pm", line 39, in "Mojolicious::Plugin::EPRenderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 221, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 110, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 152, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Controller.pm", line 164, in "Mojolicious::Controller"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 123, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 110, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugin/DefaultHelpers.pm", line 50, in "Mojolicious::Plugin::DefaultHelpers"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Renderer.pm", line 70, in "Mojolicious::Renderer"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 203, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 15, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Plugins.pm", line 18, in "Mojolicious::Plugins"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 141, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server.pm", line 66, in "Mojo::Server"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 103, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Transaction/HTTP.pm", line 60, in "Mojo::Transaction::HTTP"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 218, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Daemon.pm", line 199, in "Mojo::Server::Daemon"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/EventEmitter.pm", line 15, in "Mojo::EventEmitter"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 109, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop/Stream.pm", line 57, in "Mojo::IOLoop::Stream"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 146, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 60, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Reactor/Poll.pm", line 103, in "Mojo::Reactor::Poll"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/IOLoop.pm", line 133, in "Mojo::IOLoop"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 152, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 93, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojo/Server/Prefork.pm", line 78, in "Mojo::Server::Prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Command/prefork.pm", line 31, in "Mojolicious::Command::prefork"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious/Commands.pm", line 57, in "Mojolicious::Commands"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/lib/perl5/vendor_perl/5.30.1/Mojolicious.pm", line 186, in "Mojolicious"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/../lib/OpenQA/WebAPI.pm", line 493, in "OpenQA::WebAPI"
srp 18 13:25:34 dhcp165.suse.cz openqa-webui-daemon[17397]: File "/usr/share/openqa/script/openqa", line 35, in "main"
</code></pre> openQA Project - action #66619 (Rejected): OpenQA jobs roll back to the wrong snapshot on hard te...https://progress.opensuse.org/issues/666192020-05-07T11:54:05ZMDouchamartin.doucha@suse.com
<p>When a job includes multiple modules that create a snapshot, VM rollback appears to always use the very first snapshot instead of the last one.</p>
<p>Example: <a href="https://openqa.suse.de/tests/4203253#step/AD044/6" class="external">https://openqa.suse.de/tests/4203253#step/AD044/6</a><br>
Module AD043 failed and triggered VM rollback. The remaining modules then fail with the following error:</p>
<pre><code>/tmp/aiodio/junkfile: No such file or directory
</code></pre>
<p>This means that the VM was rolled back all the way to <code>boot_ltp</code>. But it was supposed to use the snapshot created by <code>create_junkfile_ltp</code>.</p>
<p>This does not appear to be a new issue. The same error appears in all LTP aiodio jobs which failed since VM rollback was enabled for them by <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9264" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/9264</a><br>
Oldest known example: <a href="https://openqa.suse.de/tests/3987350#step/AD037/6" class="external">https://openqa.suse.de/tests/3987350#step/AD037/6</a></p>
openQA Infrastructure - action #63706 (Rejected): [zkvm] Connection loss between VM and host on o...https://progress.opensuse.org/issues/637062020-02-21T10:13:48ZMDouchamartin.doucha@suse.com
<p>The zkvm slots on openqaworker2 frequently lose VNC and/or SSH connection between the host and VM. The first recent appearance of this problem was on 2020-02-19 around 1AM and affects both SLE-15GA and SLE-15SP1. SLE-12* jobs use different worker class.</p>
<p><a href="https://openqa.suse.de/tests/3898309#step/install_ltp/24" class="external">https://openqa.suse.de/tests/3898309#step/install_ltp/24</a><br>
<a href="https://openqa.suse.de/tests/3898794#step/install_ltp/30" class="external">https://openqa.suse.de/tests/3898794#step/install_ltp/30</a><br>
<a href="https://openqa.suse.de/tests/3906656#step/update_kernel/30" class="external">https://openqa.suse.de/tests/3906656#step/update_kernel/30</a><br>
<a href="https://openqa.suse.de/tests/3909115#step/install_ltp/64" class="external">https://openqa.suse.de/tests/3909115#step/install_ltp/64</a><br>
<a href="https://openqa.suse.de/tests/3898244#step/update_kernel/37" class="external">https://openqa.suse.de/tests/3898244#step/update_kernel/37</a><br>
<a href="https://openqa.suse.de/tests/3906591#step/install_ltp/12" class="external">https://openqa.suse.de/tests/3906591#step/install_ltp/12</a></p>
openQA Infrastructure - action #61994 (Resolved): VNC console corruption on aarch64https://progress.opensuse.org/issues/619942020-01-10T09:46:39ZMDouchamartin.doucha@suse.com
<p>A random problem sometimes appears on aarch64 test machines where the VM screen isn't properly cleared after boot and console output gets drawn over remnants of boot splash screen. Then the job fails because needles don't match. The problem appears less than once a week and job restart usually fixes it but it might be worth investigating further.<br>
<a href="https://openqa.suse.de/tests/3773959#step/update_kernel/6" class="external">https://openqa.suse.de/tests/3773959#step/update_kernel/6</a></p>
openQA Infrastructure - action #61844 (Resolved): auto_review:"download failed: 521 - Connect tim...https://progress.opensuse.org/issues/618442020-01-07T14:21:57ZMDouchamartin.doucha@suse.com
<p>The cache service on openqaworker-arm-3 frequently fails to download assets with error 521:</p>
<pre><code>[2020-01-05T01:30:22.0405 CET] [info] [pid:49324] Downloading SLES-15-aarch64-minimal_installed_for_LTP.qcow2, request #3191 sent to Cache Service
[2020-01-05T01:30:48.0583 CET] [info] [pid:49324] Download of SLES-15-aarch64-minimal_installed_for_LTP.qcow2 processed:
[info] [#3191] Cache size of "/var/lib/openqa/cache" is 49GiB, with limit 50GiB
[info] [#3191] Downloading "SLES-15-aarch64-minimal_installed_for_LTP.qcow2" from "openqa.suse.de/tests/3754531/asset/hdd/SLES-15-aarch64-minimal_installed_for_LTP.qcow2"
[info] [#3191] Purging "/var/lib/openqa/cache/openqa.suse.de/SLES-15-aarch64-minimal_installed_for_LTP.qcow2" because the download failed: 521 - Connect timeout
</code></pre>
<p>The error may seem rare at first glance but that's most likely because of asset caching on workers. For example, of the last 10 jobs on openqaworker-arm-3:19 (at the time of writing), 2 jobs failed with connect timeout, 2 jobs downloaded at least one asset successfully and 6 jobs ran entirely from cache. It's not clear from logs whether the timeout happens during the initial connection or halfway through downloading a 2GB file.<br>
<a href="https://openqa.suse.de/admin/workers/1298" class="external">https://openqa.suse.de/admin/workers/1298</a></p>
<p>The oldest case confirmed by os-autoinst log is from 2019-12-15: <a href="https://openqa.suse.de/tests/3708066" class="external">https://openqa.suse.de/tests/3708066</a><br>
There may have been older cases but their logs have most likely been deleted by now.</p>
<p>I've also looked at 5 instances of openqaworker-arm-1 and found only 3 confirmed cases of the same error. That's low enough to be caused by chance.</p>
openQA Infrastructure - action #58945 (Resolved): OpenQA worker service not restarted after OpenQ...https://progress.opensuse.org/issues/589452019-10-31T13:12:21ZMDouchamartin.doucha@suse.com
<p>The openqa-worker service on some openqa.suse.de workers doesn't get restarted after update. This may cause version mismatch between os-autoinst and openQA-common packages.</p>
<p>One example of this mismatch are these three verification runs for <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8329" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/8329</a> below:<br>
openqaworker2: <a href="https://openqa.suse.de/tests/3541705" class="external">https://openqa.suse.de/tests/3541705</a> (openqa-worker service last restarted on 2019-10-30)<br>
openqaworker6: <a href="https://openqa.suse.de/tests/3541697" class="external">https://openqa.suse.de/tests/3541697</a> (openqa-worker service last restarted on 2019-09-18)<br>
openqaworker9: <a href="https://openqa.suse.de/tests/3544337" class="external">https://openqa.suse.de/tests/3544337</a> (openqa-worker service last restarted on 2019-09-18)</p>
<p>All three jobs ran the same test modules (see autoinst log) but all tests after intall_ltp were scheduled at runtime. Updating test schedule at runtime requires patches merged into OpenQA on 2019-09-27 so openqaworker6 and openqaworker9 didn't update test schedule due to still running openQA-common from mid-September, before the patches were merged.</p>
openQA Infrastructure - action #58805 (Resolved): [infra]Severe storage performance issue on open...https://progress.opensuse.org/issues/588052019-10-29T11:34:09ZMDouchamartin.doucha@suse.com
<p>Last week on Thursday, a handful of tests in two LTP testsuites started timing out. I've initially reported it as a kernel performance regression: <a href="https://bugzilla.suse.com/show_bug.cgi?id=1155018" class="external">https://bugzilla.suse.com/show_bug.cgi?id=1155018</a></p>
<p>However, I've tried to reproduce the problem on a released kernel version which didn't have the issue 3 weeks ago and succeeded: <a href="https://openqa.suse.de/tests/overview?build=15ga_mdoucha_bsc_1155018&version=15&distri=sle" class="external">https://openqa.suse.de/tests/overview?build=15ga_mdoucha_bsc_1155018&version=15&distri=sle</a></p>
<p>This successful reproduction on a known good kernel indicates that the problem is somewhere in OpenQA infrastructure, possibly a bug introduced during the weekly deployment on Wednesday, October 23rd. The timeout continues to appear in kernel-of-the-day LTP tests: <a href="https://openqa.suse.de/tests/3533819#step/DOR000/7" class="external">https://openqa.suse.de/tests/3533819#step/DOR000/7</a></p>
<p>Both PPC64LE and x86_64 are affected. Reproducibility on aarch64 and s390 is currently unknown because we don't run the affected testsuites on those two platforms. The failing tests mostly belong to the async & direct I/O stress testsuite.</p>