action #103575: [virtualization][3rd party hypervisor] Worker openqaw8-vmware.qa.suse.de is not reachable - openQA Infrastructure - openSUSE Project Management Tool

Actions

Copy link

action #103575

closed

[virtualization][3rd party hypervisor] Worker openqaw8-vmware.qa.suse.de is not reachable

Added by nanzhang over 2 years ago. Updated about 2 years ago.

Status:

Resolved

Priority:

High

Assignee:

mkittler

Category:

Target version:

openQA Project - Ready

Start date:

2021-12-07

Due date:

% Done:

Estimated time:

Description

The following OSD jobs failed at bootloader_svirt due to the worker openqaw8-vmware.qa.suse.de cannot be reached and logged in.

https://openqa.nue.suse.com/tests/7794974
https://openqa.nue.suse.com/tests/7799143
https://openqa.nue.suse.com/tests/7799141
https://openqa.nue.suse.com/tests/7799136

Actions

Copy link

Updated by jlausuch over 2 years ago

JeOS is also affected by this
15-SP3 QR2: https://openqa.suse.de/tests/overview?distri=sle&version=15-SP3&build=3.3.7&groupid=374&flavor=JeOS-for-VMware-QR
15-SP4: https://openqa.suse.de/tests/overview?distri=sle&version=15-SP4&build=150400.1.163&groupid=131&flavor=JeOS-for-VMware

Actions

Copy link

Updated by okurz over 2 years ago

Status changed from New to In Progress
Assignee set to okurz
Priority changed from Normal to High
Target version set to Ready

I think this is related to a recent IPMI firmware change done by bmwiedemann from EngInfra. I just powered on the host with ipmitool -I lanplus -H sp.openqaw8-vmware.qa.suse.de -U ADMIN -P $password power on and will check. Also https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/372 is related. IPMI SoL looks like the machine is up so I cloned https://openqa.suse.de/tests/7794713#step/bootloader_svirt/7 as https://openqa.suse.de/tests/7802844#live and will monitor.

https://openqa.suse.de/tests/7802844#step/bootloader_svirt/44 shows that we reached the SUT so I went ahead and executed

openqa-label-all --verbose --openqa-host https://openqa.suse.de --label '* bootloader_svirt: https://progress.opensuse.org/issues/103575' --module bootloader_svirt

with openqa-label-all from the package openQA-python-scripts

The complete output including all jobs that have been triggered:

{"id":452865}
{"id":452866}
{"id":452867}
{"id":452868}
{"id":452869}
{"id":452870}
{"id":452871}
{"id":452872}
{"id":452873}
{"id":452874}
{"id":452875}
{"id":452876}
{"id":452878}
{"id":452879}
{"id":452880}
{"id":452881}
{"id":452882}
{"id":452883}
{"id":452884}
{"id":452885}
{"id":452886}
{"id":452887}
{"id":452888}
{"id":452889}
{"id":452890}
{"id":452891}
{"id":452892}
{"id":452893}
{"id":452894}
{"id":452895}
{"id":452896}
{"id":452897}
{"id":452898}
{"id":452899}
{"id":452900}
{"id":452901}
{"id":452902}
{"id":452903}
{"id":452904}
{"id":452905}
{"id":452906}
{"id":452907}
{"id":452908}
{"id":452909}
{"id":452910}
{"id":452911}
{"id":452912}
{"id":452913}
{"id":452914}
{"id":452915}
{"id":452916}
{"id":452917}
{"id":452918}
{"id":452919}
{"id":452920}
{"id":452921}
{"id":452922}
{"id":452923}
{"id":452924}
{"id":452925}
{"id":452926}
{"id":452927}
{"id":452928}
{"id":452929}
{"id":452930}
{"id":452931}
{"id":452932}
{"id":452933}
{"id":452934}
{"id":452935}
{"id":452936}
{"id":452937}
{"id":452938}
{"id":452939}
{"id":452940}
{"id":452941}
{"id":452942}
{"id":452943}
{"id":452944}
{"id":452945}
{"id":452946}
{"id":452947}
{"id":452948}
{"id":452949}
{"id":452950}
{"id":452951}
{"id":452952}
{"id":452953}
{"id":452954}
{"id":452955}
{"id":452956}
{"id":452957}
{"id":452958}
{"result":[{"7772500":7802866}],"test_url":[{"7772500":"\/tests\/7802866"}]}
{"enforceable":1,"errors":["Job 7778952 misses the following mandatory assets: hdd\/SLES15-SP3-JeOS.x86_64-15.3-VMware-Build3.3.6.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7778953 misses the following mandatory assets: hdd\/SLES15-SP3-JeOS.x86_64-15.3-VMware-Build3.3.6.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7778954 misses the following mandatory assets: hdd\/SLES15-SP3-JeOS.x86_64-15.3-VMware-Build3.3.6.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7778955 misses the following mandatory assets: hdd\/SLES15-SP3-JeOS.x86_64-15.3-VMware-Build3.3.6.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7778956 misses the following mandatory assets: hdd\/SLES15-SP3-JeOS.x86_64-15.3-VMware-Build3.3.6.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7778957 misses the following mandatory assets: hdd\/SLES15-SP3-JeOS.x86_64-15.3-VMware-Build3.3.6.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7778958 misses the following mandatory assets: hdd\/SLES15-SP3-JeOS.x86_64-15.3-VMware-Build3.3.6.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7778959 misses the following mandatory assets: hdd\/SLES15-SP3-JeOS.x86_64-15.3-VMware-Build3.3.6.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"result":[{"7781404":7802867}],"test_url":[{"7781404":"\/tests\/7802867"}]}
{"result":[{"7781405":7802868}],"test_url":[{"7781405":"\/tests\/7802868"}]}
{"result":[{"7781406":7802869}],"test_url":[{"7781406":"\/tests\/7802869"}]}
{"result":[{"7781407":7802870}],"test_url":[{"7781407":"\/tests\/7802870"}]}
{"result":[{"7781408":7802871}],"test_url":[{"7781408":"\/tests\/7802871"}]}
{"result":[{"7781409":7802873}],"test_url":[{"7781409":"\/tests\/7802873"}]}
{"result":[{"7781410":7802874}],"test_url":[{"7781410":"\/tests\/7802874"}]}
{"result":[{"7781417":7802875}],"test_url":[{"7781417":"\/tests\/7802875"}]}
{"result":[{"7781418":7802877}],"test_url":[{"7781418":"\/tests\/7802877"}]}
{"result":[{"7781419":7802878}],"test_url":[{"7781419":"\/tests\/7802878"}]}
{"result":[{"7781422":7802880}],"test_url":[{"7781422":"\/tests\/7802880"}]}
{"result":[{"7781423":7802881}],"test_url":[{"7781423":"\/tests\/7802881"}]}
{"result":[{"7781424":7802883}],"test_url":[{"7781424":"\/tests\/7802883"}]}
{"result":[{"7781427":7802884}],"test_url":[{"7781427":"\/tests\/7802884"}]}
{"enforceable":1,"errors":["Job 7785181 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785182 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785183 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785184 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785185 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785186 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785187 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785189 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785190 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785191 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785196 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785197 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785198 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"enforceable":1,"errors":["Job 7785210 misses the following mandatory assets: hdd\/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.98.vmdk.xz\nEnsure to provide mandatory assets and\/or force retriggering if necessary."],"result":[],"test_url":[]}
{"result":[{"7791004":7802885}],"test_url":[{"7791004":"\/tests\/7802885"}]}
{"result":[{"7794707":7802886}],"test_url":[{"7794707":"\/tests\/7802886"}]}
{"result":[{"7794708":7802887}],"test_url":[{"7794708":"\/tests\/7802887"}]}
{"result":[{"7794709":7802888}],"test_url":[{"7794709":"\/tests\/7802888"}]}
{"result":[{"7794710":7802889}],"test_url":[{"7794710":"\/tests\/7802889"}]}
{"result":[{"7794711":7802890}],"test_url":[{"7794711":"\/tests\/7802890"}]}
{"result":[{"7794712":7802891}],"test_url":[{"7794712":"\/tests\/7802891"}]}
{"errors":["Specified job 7794713 has already been cloned as 7802844"],"result":[],"test_url":[]}
{"result":[{"7794721":7802892}],"test_url":[{"7794721":"\/tests\/7802892"}]}
{"result":[{"7794723":7802893}],"test_url":[{"7794723":"\/tests\/7802893"}]}
{"result":[{"7794724":7802894}],"test_url":[{"7794724":"\/tests\/7802894"}]}
{"result":[{"7794725":7802895}],"test_url":[{"7794725":"\/tests\/7802895"}]}
{"result":[{"7794726":7802896}],"test_url":[{"7794726":"\/tests\/7802896"}]}
{"result":[{"7794727":7802897}],"test_url":[{"7794727":"\/tests\/7802897"}]}
{"result":[{"7794728":7802898}],"test_url":[{"7794728":"\/tests\/7802898"}]}
{"result":[{"7794759":7802899}],"test_url":[{"7794759":"\/tests\/7802899"}]}
{"result":[{"7794827":7802900}],"test_url":[{"7794827":"\/tests\/7802900"}]}
{"result":[{"7794828":7802901}],"test_url":[{"7794828":"\/tests\/7802901"}]}
{"result":[{"7794830":7802902}],"test_url":[{"7794830":"\/tests\/7802902"}]}
{"result":[{"7794832":7802903}],"test_url":[{"7794832":"\/tests\/7802903"}]}
{"result":[{"7794833":7802904}],"test_url":[{"7794833":"\/tests\/7802904"}]}
{"result":[{"7794834":7802905}],"test_url":[{"7794834":"\/tests\/7802905"}]}
{"result":[{"7794835":7802906}],"test_url":[{"7794835":"\/tests\/7802906"}]}
{"result":[{"7794837":7802907}],"test_url":[{"7794837":"\/tests\/7802907"}]}
{"result":[{"7794838":7802908}],"test_url":[{"7794838":"\/tests\/7802908"}]}
{"result":[{"7794843":7802909}],"test_url":[{"7794843":"\/tests\/7802909"}]}
{"result":[{"7794881":7802910}],"test_url":[{"7794881":"\/tests\/7802910"}]}
{"result":[{"7794882":7802911}],"test_url":[{"7794882":"\/tests\/7802911"}]}
{"result":[{"7794883":7802912}],"test_url":[{"7794883":"\/tests\/7802912"}]}
{"result":[{"7794885":7802913}],"test_url":[{"7794885":"\/tests\/7802913"}]}
{"result":[{"7794886":7802914}],"test_url":[{"7794886":"\/tests\/7802914"}]}
{"result":[{"7794888":7802915}],"test_url":[{"7794888":"\/tests\/7802915"}]}
{"result":[{"7794889":7802916}],"test_url":[{"7794889":"\/tests\/7802916"}]}
{"result":[{"7794890":7802917}],"test_url":[{"7794890":"\/tests\/7802917"}]}
{"result":[{"7794974":7802918}],"test_url":[{"7794974":"\/tests\/7802918"}]}
{"result":[{"7799136":7802919}],"test_url":[{"7799136":"\/tests\/7802919"}]}
{"result":[{"7799141":7802920}],"test_url":[{"7799141":"\/tests\/7802920"}]}
{"result":[{"7799143":7802921}],"test_url":[{"7799143":"\/tests\/7802921"}]}
{"result":[{"7802336":7802922}],"test_url":[{"7802336":"\/tests\/7802922"}]}
{"result":[{"7802337":7802923}],"test_url":[{"7802337":"\/tests\/7802923"}]}
{"result":[{"7802338":7802924}],"test_url":[{"7802338":"\/tests\/7802924"}]}
{"result":[{"7802339":7802925}],"test_url":[{"7802339":"\/tests\/7802925"}]}
{"result":[{"7802340":7802927}],"test_url":[{"7802340":"\/tests\/7802927"}]}
{"result":[{"7802341":7802928}],"test_url":[{"7802341":"\/tests\/7802928"}]}
{"result":[{"7802342":7802929}],"test_url":[{"7802342":"\/tests\/7802929"}]}
{"result":[{"7802343":7802931}],"test_url":[{"7802343":"\/tests\/7802931"}]}
{"result":[{"7802344":7802932}],"test_url":[{"7802344":"\/tests\/7802932"}]}
{"result":[{"7802345":7802933}],"test_url":[{"7802345":"\/tests\/7802933"}]}
{"result":[{"7802346":7802934}],"test_url":[{"7802346":"\/tests\/7802934"}]}
{"result":[{"7802354":7802936}],"test_url":[{"7802354":"\/tests\/7802936"}]}
{"result":[{"7802355":7802937}],"test_url":[{"7802355":"\/tests\/7802937"}]}
{"result":[{"7802356":7802939}],"test_url":[{"7802356":"\/tests\/7802939"}]}
{"result":[{"7802363":7802940}],"test_url":[{"7802363":"\/tests\/7802940"}]}
{"result":[{"7802364":7802941}],"test_url":[{"7802364":"\/tests\/7802941"}]}
{"result":[{"7802615":7802942}],"test_url":[{"7802615":"\/tests\/7802942"}]}
{"result":[{"7802626":7802943}],"test_url":[{"7802626":"\/tests\/7802943"}]}

Actions

Copy link

Updated by openqa_review over 2 years ago

Due date set to 2021-12-22

Setting due date based on mean cycle time of SUSE QE Tools

Actions

Copy link

Updated by nanzhang over 2 years ago

Thank you for the fix!
The latest run looks good. - https://openqa.nue.suse.com/tests/7806054

Actions

Copy link

Updated by okurz over 2 years ago

Due date deleted (~~2021-12-22~~)
Status changed from In Progress to Resolved

All referenced jobs seem to have passed the initial step at least

Actions

Copy link

Updated by jlausuch over 2 years ago

Thanks for the help!

Actions

Copy link

Updated by openqa_review over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: jeos-fips@svirt-vmware65
https://openqa.suse.de/tests/7888369

To prevent further reminder comments one of the following options should be followed:

The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
The openQA job group is moved to "Released" or "EOL" (End-of-Life)
The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Actions

Copy link

Updated by openqa_review over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: jeos-fips@svirt-vmware65
https://openqa.suse.de/tests/7924906

To prevent further reminder comments one of the following options should be followed:

The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
The openQA job group is moved to "Released" or "EOL" (End-of-Life)
The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Actions

Copy link

Updated by openqa_review over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: jeos-fs_stress@svirt-vmware65
https://openqa.suse.de/tests/7998874

To prevent further reminder comments one of the following options should be followed:

The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
The openQA job group is moved to "Released" or "EOL" (End-of-Life)
The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Actions

Copy link

#10

Updated by okurz over 2 years ago

Status changed from Resolved to New
Assignee deleted (~~okurz~~)

We apparently need to check this again, see the openQA test references

Actions

Copy link

#11

Updated by mkittler about 2 years ago

Assignee set to mkittler

Actions

Copy link

#12

Updated by mkittler about 2 years ago

Status changed from New to Feedback

The tests which fail now look like VMWare tests but they're actually (successfully) connecting to openqaw5-xen.qa.suse.de (instead of openqaw8-vmware.qa.suse.de). Judging by its hostname, I assume openqaw5-xen.qa.suse.de only works for Xen tests so this test setup seems just wrong. So the failures haven't anything to do with openqaw8-vmware.qa.suse.de being unreachable. In fact, I can connect to that host just fine (via VPN).

Actions

Copy link

#13

Updated by mkittler about 2 years ago

Ok, it is actually using the VMWare host. However, it still seems to be reachable and responding to SSH commands:

[2022-01-26T18:33:11.275678+01:00] [debug] SSH connection to root@openqaw8-vmware.qa.suse.de established
[2022-01-26T18:33:11.364395+01:00] [debug] [run_ssh_cmd(set -x; rm -f /vmfs/volumes/datastore1/openQA/*openQA-SUT-3*)] stderr:
  + rm -f /vmfs/volumes/datastore1/openQA/SLES15-SP1-JeOS.x86_64-15.1-VMware-Build37.8.53_openQA-SUT-3_thinfile-flat.vmdk /vmfs/volumes/datastore1/openQA/SLES15-SP1-JeOS.x86_64-15.1-VMware-Build37.8.53_openQA-SUT-3_thinfile.vmdk /vmfs/volumes/datastore1/openQA/openQA-SUT-3.vmsd /vmfs/volumes/datastore1/openQA/openQA-SUT-3.vmx

[2022-01-26T18:33:11.367458+01:00] [debug] [run_ssh_cmd(set -x; rm -f /vmfs/volumes/datastore1/openQA/*openQA-SUT-3*)] exit-code: 0
…
[2022-01-26T18:33:11.831266+01:00] [debug] <<< backend::baseclass::new_ssh_connection(keep_open=1, username="root", hostname="openqaw8-vmware.qa.suse.de", password="SECRET", blocking=1, wantarray=1)
[2022-01-26T18:33:11.945655+01:00] [debug] Use existing SSH connection (key:hostname=openqaw8-vmware.qa.suse.de,username=root,port=22)
[2022-01-26T18:33:33.357052+01:00] [debug] [run_ssh_cmd(find /vmfs/volumes/openqa/hdd /vmfs/volumes/openqa/hdd/fixed -name SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz | head -n1 | awk 1 ORS='')] stdout:
  /vmfs/volumes/openqa/hdd/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz
[2022-01-26T18:33:33.359679+01:00] [debug] [run_ssh_cmd(find /vmfs/volumes/openqa/hdd /vmfs/volumes/openqa/hdd/fixed -name SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz | head -n1 | awk 1 ORS='')] exit-code: 0
[2022-01-26T18:33:33.518226+01:00] [debug] Image found: /vmfs/volumes/openqa/hdd/SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz
[2022-01-26T18:33:33.518469+01:00] [debug] tests/installation/bootloader_svirt.pm:137 called bootloader_svirt::search_image_on_svirt_host -> tests/installation/bootloader_svirt.pm:49 called testapi::enter_cmd
[2022-01-26T18:33:33.518686+01:00] [debug] <<< testapi::type_string(string="# Copying image SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz...", max_interval=250, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
[2022-01-26T18:33:36.283921+01:00] [debug] tests/installation/bootloader_svirt.pm:142 called backend::console_proxy::__ANON__
[2022-01-26T18:33:36.284202+01:00] [debug] <<< backend::console_proxy::__ANON__(wrapped_call={
    "function" => "run_cmd",
    "console" => "svirt",
    "wantarray" => "",
    "args" => [
                "test -e /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk",
                "domain",
                "sshVMwareServer"
              ]
  })
[2022-01-26T18:33:36.285235+01:00] [debug] <<< backend::baseclass::run_ssh_cmd(cmd="test -e /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk", wantarray=0, keep_open=1, username="root", password="SECRET", hostname="openqaw8-vmware.qa.suse.de")

The problem is apparently that some file exists which shouldn't:

[2022-01-26T18:34:41.051751+01:00] [debug] Use existing SSH connection (key:hostname=openqaw8-vmware.qa.suse.de,username=root,port=22)
[2022-01-26T18:34:41.064536+01:00] [debug] [run_ssh_cmd(xz --decompress --keep --verbose /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz)] stderr:
  xz: /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk: File exists

[2022-01-26T18:34:41.067476+01:00] [debug] [run_ssh_cmd(xz --decompress --keep --verbose /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz)] exit-code: 1
[2022-01-26T18:34:41.299347+01:00] [info] ::: basetest::runtest: # Test died: Image decompress in datastore failed!
   at sle/tests/installation/bootloader_svirt.pm line 146.
    bootloader_svirt::run(bootloader_svirt=HASH(0x560cf842d6e0)) called at /usr/lib/os-autoinst/basetest.pm line 360
    eval {...} called at /usr/lib/os-autoinst/basetest.pm line 354
    basetest::runtest(bootloader_svirt=HASH(0x560cf842d6e0)) called at /usr/lib/os-autoinst/autotest.pm line 372
    eval {...} called at /usr/lib/os-autoinst/autotest.pm line 372
    autotest::runalltests() called at /usr/lib/os-autoinst/autotest.pm line 242
    eval {...} called at /usr/lib/os-autoinst/autotest.pm line 242
    autotest::run_all() called at /usr/lib/os-autoinst/autotest.pm line 296
    autotest::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x560cfa2895f0)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
    eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
    Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x560cfa2895f0), CODE(0x560cfa73a5a0)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 488
    Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x560cfa2895f0)) called at /usr/lib/os-autoinst/autotest.pm line 298
    autotest::start_process() called at /usr/bin/isotovideo line 261

Actions

Copy link

#14

Updated by mkittler about 2 years ago

Maybe

[root@openqaw8-vmware:~] mv /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz.bak

helped but I cannot retry the openQA job to test that because assets are missing.

Ensuring these details within the test setup is also likely more something the test writers should handle.

I've been removing the wrong bugrefs from the jobs.

Actions

Copy link

#15

Updated by mloviska about 2 years ago

mkittler wrote:

Maybe
[root@openqaw8-vmware:~] mv /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz /vmfs/volumes/datastore1/openQA//SLES15-SP2-JeOS.x86_64-15.2-VMware-Build15.106.vmdk.xz.bak
helped but I cannot retry the openQA job to test that because assets are missing.

Ensuring these details within the test setup is also likely more something the test writers should handle.

I've been removing the wrong bugrefs from the jobs.

Seems like you are trying to use assets that we are not testing anymore. :)

Actions

Copy link

#16

Updated by mkittler about 2 years ago

Status changed from Feedback to Resolved

I am not testing anything. I am only taking care of this ticket which was reopened due to these failing jobs. However, it turns out to be unrelated so I'm resolving the ticket again. Of course it would make sense to avoid creating those jobs the they are not relevant anymore. (Maybe that's already the case. The last job is from 7 days ago.)

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

QA » openQA Project » openQA Infrastructure

Tags

Custom queries

action #103575

[virtualization][3rd party hypervisor] Worker openqaw8-vmware.qa.suse.de is not reachable

Updated by jlausuch over 2 years ago

Updated by okurz over 2 years ago

Updated by openqa_review over 2 years ago

Updated by nanzhang over 2 years ago

Updated by okurz over 2 years ago

Updated by jlausuch over 2 years ago

Updated by openqa_review over 2 years ago

Updated by openqa_review over 2 years ago

Updated by openqa_review over 2 years ago

Updated by okurz over 2 years ago

Updated by mkittler about 2 years ago

Updated by mkittler about 2 years ago

Updated by mkittler about 2 years ago

Updated by mkittler about 2 years ago

Updated by mloviska about 2 years ago

Updated by mkittler about 2 years ago