Project

General

Profile

Actions

openqa-force-result #134807

open

[qe-core] test fails in bootloader_start - missing assets on s390

Added by szarate 8 months ago. Updated about 2 months ago.

Status:
New
Priority:
High
Assignee:
-
Target version:
-
Due date:

Description

Observation

[2023-08-30T07:45:23.448709+02:00] [info] [pid:25289] ::: basetest::runtest: # Test died: Unable to find image SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 in /var/lib/openqa/share/factory/hdd at sle/lib/bootloader_setup.pm line 1106.
[2023-08-30T07:45:23.069386+02:00] [debug] [pid:25289] <<< backend::console_proxy::__ANON__(wrapped_call={
    "wantarray" => "",
    "args" => [
                "find /var/lib/openqa/share/factory/hdd -name SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 | head -n1 | tr -d '\n'"
              ],
    "function" => "get_cmd_output",
    "console" => "svirt"
  })
[2023-08-30T07:45:23.071318+02:00] [debug] [pid:25290] <<< backend::baseclass::run_ssh_cmd(cmd="find /var/lib/openqa/share/factory/hdd -name SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 | head -n1 | tr -d '\n'", wantarray=1, password="SECRET", keep_open=1, hostname="s390zp19.suse.de", username=undef)
[2023-08-30T07:45:23.071593+02:00] [debug] [pid:25290] <<< backend::baseclass::run_ssh(cmd="find /var/lib/openqa/share/factory/hdd -name SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 | head -n1 | tr -d '\n'", username=undef, hostname="s390zp19.suse.de", keep_open=1, wantarray=1, password="SECRET")
[2023-08-30T07:45:23.071868+02:00] [debug] [pid:25290] <<< backend::baseclass::new_ssh_connection(username=undef, hostname="s390zp19.suse.de", blocking=1, keep_open=1, wantarray=1, password="SECRET")
[2023-08-30T07:45:23.123025+02:00] [debug] [pid:25290] Use existing SSH connection (key:hostname=s390zp19.suse.de,username=root,port=22)
[2023-08-30T07:45:23.225432+02:00] [debug] [pid:25290] [run_ssh_cmd(find /var/lib/openqa/share/factory/hdd -name SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 | head -n1 | tr -d '
  ')] exit-code: 0
[2023-08-30T07:45:23.448709+02:00] [info] [pid:25289] ::: basetest::runtest: # Test died: Unable to find image SLES-15-SP1-s390x-mru-install-minimal-with-addons-Build20230829-1-Server-DVD-Updates-s390x-kvm-sle12.qcow2 in /var/lib/openqa/share/factory/hdd at sle/lib/bootloader_setup.pm line 1106.

openQA test in scenario sle-15-SP1-Server-DVD-Updates-s390x-sssd_389ds_functional@s390x-kvm-sle12 fails in
bootloader_start

Test suite description

Testsuite maintained at https://gitlab.suse.de/qe-security/osd-sle15-security.

Reproducible

Fails since (at least) Build 20230826-1

Expected result

Last good: 20230825-1 (or more recent)

Acceptance criteria

Further details

Always latest result in this scenario: latest


Related issues 4 (0 open4 closed)

Related to openQA Infrastructure - action #127337: Some s390x workers have been failing for all jobs since 11 months agoResolvedokurz2023-04-06

Actions
Related to openQA Infrastructure - action #134912: Gradually phase out NUE1 based openQA workers size:MResolvedokurz

Actions
Related to openQA Tests - action #137384: [tools][s390x] worker imagetester can't reach SUT auto_review:"backend done: Error connecting to <root@s390zl14.suse.de>: Connection timed out" size:MResolvedokurz2023-10-04

Actions
Copied to openQA Tests - action #137321: [qac] s390x tests fail to access SUSE-5.5-MicroOS-POOL-s390x-GM-Media1 which was likely never provided on OSDResolved2023-10-02

Actions
Actions #1

Updated by MDoucha 8 months ago

A similar problem in bootloader_svirt was alread reported: #121573. The test ignores HDD assets downloaded by the cache service and tries to download them again via NFS. Check the NFS mount on the worker machine.

Actions #2

Updated by szarate 8 months ago

  • Subject changed from test fails in bootloader_start - missing assets on s390 to [qe-core] test fails in bootloader_start - missing assets on s390 auto_review:"Unable to find image .*qcow2":force_result:softfailed
Actions #3

Updated by szarate 8 months ago

  • Tracker changed from action to openqa-force-result
  • Project changed from openQA Tests to openQA auto review
  • Category deleted (Bugs in existing tests)
  • Start date deleted (2023-08-30)
Actions #5

Updated by jlausuch 8 months ago

@szarate this looks a lot like https://progress.opensuse.org/issues/135146
Seems that restarting the nfs server in OSD made it pass, is it the case as well for you now?

Actions #6

Updated by mgrifalconi 7 months ago

  • Priority changed from Normal to High

Proposing to increase prio here, since it's happening more and more.

I would also question, is this for qe-core? Can this be a misconfiguration of the test? Might sound more a topic for tools to me.

Actions #7

Updated by dzedro 7 months ago

I checked few failures and it is failing "only" on worker5. Maybe tools know why.

Actions #8

Updated by okurz 7 months ago

  • Related to action #127337: Some s390x workers have been failing for all jobs since 11 months ago added
Actions #9

Updated by mgrifalconi 7 months ago

  • Subject changed from [qe-core] test fails in bootloader_start - missing assets on s390 auto_review:"Unable to find image .*qcow2":force_result:softfailed to [qe-core] test fails in bootloader_start - missing assets on s390

removing

auto_review:"Unable to find image .*qcow2":force_result:softfailed

as it is softfailing lots of broken things that should not be softfailed

Actions #10

Updated by okurz 7 months ago

(Jozef Pupava) Maybe s390zp15.suse.de has some issue https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/626
(Oliver Kurz) Included in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/625 and merged. I moved the instances elsewhere and instead of commenting the section I used a special non-production worker class so that one can still test on those

Actions #11

Updated by okurz 7 months ago

  • Description updated (diff)
Actions #12

Updated by okurz 7 months ago

  • Related to action #134912: Gradually phase out NUE1 based openQA workers size:M added
Actions #13

Updated by okurz 7 months ago

  • Copied to action #137321: [qac] s390x tests fail to access SUSE-5.5-MicroOS-POOL-s390x-GM-Media1 which was likely never provided on OSD added
Actions #14

Updated by dzedro 7 months ago

  • Related to action #137384: [tools][s390x] worker imagetester can't reach SUT auto_review:"backend done: Error connecting to <root@s390zl14.suse.de>: Connection timed out" size:M added
Actions #15

Updated by JERiveraMoya 7 months ago

openqaworker1 has this issue, https://openqa.suse.de/tests/12456872#step/bootloader_zkvm/18
imagetester has this other one for the same test suite: https://openqa.suse.de/tests/12456861

Actions #16

Updated by tinita about 2 months ago

This ticket was set to High priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.

Actions

Also available in: Atom PDF