openqa-force-result #134807
open[qe-core] test fails in bootloader_start - missing assets on s390
Description
Observation¶
[2023-08-30T07:45:23.448709+02:00] [info] [pid:25289] ::: basetest::runtest: # Test died: Unable to find image SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 in /var/lib/openqa/share/factory/hdd at sle/lib/bootloader_setup.pm line 1106.
[2023-08-30T07:45:23.069386+02:00] [debug] [pid:25289] <<< backend::console_proxy::__ANON__(wrapped_call={
"wantarray" => "",
"args" => [
"find /var/lib/openqa/share/factory/hdd -name SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 | head -n1 | tr -d '\n'"
],
"function" => "get_cmd_output",
"console" => "svirt"
})
[2023-08-30T07:45:23.071318+02:00] [debug] [pid:25290] <<< backend::baseclass::run_ssh_cmd(cmd="find /var/lib/openqa/share/factory/hdd -name SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 | head -n1 | tr -d '\n'", wantarray=1, password="SECRET", keep_open=1, hostname="s390zp19.suse.de", username=undef)
[2023-08-30T07:45:23.071593+02:00] [debug] [pid:25290] <<< backend::baseclass::run_ssh(cmd="find /var/lib/openqa/share/factory/hdd -name SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 | head -n1 | tr -d '\n'", username=undef, hostname="s390zp19.suse.de", keep_open=1, wantarray=1, password="SECRET")
[2023-08-30T07:45:23.071868+02:00] [debug] [pid:25290] <<< backend::baseclass::new_ssh_connection(username=undef, hostname="s390zp19.suse.de", blocking=1, keep_open=1, wantarray=1, password="SECRET")
[2023-08-30T07:45:23.123025+02:00] [debug] [pid:25290] Use existing SSH connection (key:hostname=s390zp19.suse.de,username=root,port=22)
[2023-08-30T07:45:23.225432+02:00] [debug] [pid:25290] [run_ssh_cmd(find /var/lib/openqa/share/factory/hdd -name SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 | head -n1 | tr -d '
')] exit-code: 0
[2023-08-30T07:45:23.448709+02:00] [info] [pid:25289] ::: basetest::runtest: # Test died: Unable to find image SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 in /var/lib/openqa/share/factory/hdd at sle/lib/bootloader_setup.pm line 1106.
openQA test in scenario sle-15-SP1-Server-DVD-Updates-s390x-sssd_389ds_functional@s390x-kvm-sle12 fails in
bootloader_start
Test suite description¶
Testsuite maintained at https://gitlab.suse.de/qe-security/osd-sle15-security.
Reproducible¶
Fails since (at least) Build 20230826-1
Expected result¶
Last good: 20230825-1 (or more recent)
Acceptance criteria¶
- AC1: No openQA s390x instances in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls reference this ticket anymore (see worker class "s390-kvm_poo134807")
Further details¶
Always latest result in this scenario: latest
Updated by MDoucha about 1 year ago
A similar problem in bootloader_svirt
was alread reported: #121573. The test ignores HDD assets downloaded by the cache service and tries to download them again via NFS. Check the NFS mount on the worker machine.
Updated by szarate about 1 year ago
- Subject changed from test fails in bootloader_start - missing assets on s390 to [qe-core] test fails in bootloader_start - missing assets on s390 auto_review:"Unable to find image .*qcow2":force_result:softfailed
Updated by szarate about 1 year ago
- Tracker changed from action to openqa-force-result
- Project changed from openQA Tests to openQA auto review
- Category deleted (
Bugs in existing tests) - Start date deleted (
2023-08-30)
Updated by jlausuch about 1 year ago
@szarate this looks a lot like https://progress.opensuse.org/issues/135146
Seems that restarting the nfs server in OSD made it pass, is it the case as well for you now?
Updated by mgrifalconi about 1 year ago
- Priority changed from Normal to High
Proposing to increase prio here, since it's happening more and more.
I would also question, is this for qe-core? Can this be a misconfiguration of the test? Might sound more a topic for tools to me.
Updated by dzedro about 1 year ago
I checked few failures and it is failing "only" on worker5. Maybe tools know why.
Updated by okurz about 1 year ago
- Related to action #127337: Some s390x workers have been failing for all jobs since 11 months ago added
Updated by mgrifalconi about 1 year ago
- Subject changed from [qe-core] test fails in bootloader_start - missing assets on s390 auto_review:"Unable to find image .*qcow2":force_result:softfailed to [qe-core] test fails in bootloader_start - missing assets on s390
removing
auto_review:"Unable to find image .*qcow2":force_result:softfailed
as it is softfailing lots of broken things that should not be softfailed
Updated by okurz about 1 year ago
(Jozef Pupava) Maybe s390zp15.suse.de has some issue https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/626
(Oliver Kurz) Included in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/625 and merged. I moved the instances elsewhere and instead of commenting the section I used a special non-production worker class so that one can still test on those
Updated by okurz about 1 year ago
- Related to action #134912: Gradually phase out NUE1 based openQA workers size:M added
Updated by okurz about 1 year ago
- Copied to action #137321: [qac] s390x tests fail to access SUSE-5.5-MicroOS-POOL-s390x-GM-Media1 which was likely never provided on OSD added
Updated by dzedro about 1 year ago
- Related to action #137384: [tools][s390x] worker imagetester can't reach SUT auto_review:"backend done: Error connecting to <root@s390zl14.suse.de>: Connection timed out" size:M added
Updated by JERiveraMoya about 1 year ago
openqaworker1 has this issue, https://openqa.suse.de/tests/12456872#step/bootloader_zkvm/18
imagetester has this other one for the same test suite: https://openqa.suse.de/tests/12456861
Updated by tinita 7 months ago
This ticket was set to High priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.