action #76951
closed
Check if new firmware for kerosene (aka. power8.o.o) exists and remove os-autoinst workarounds again when according machine settings are applied when necessary size:M
Added by okurz over 3 years ago.
Updated about 1 month ago.
Description
Motivation¶
New firmware might help to prevent qemu failing to run. If we find new firmware we could remove the parameters in os-autoinst again, see clone source ticket
Suggestions¶
- Read about context of the needed workaround #75259
- Currently https://kerosene-sp.qe.nue2.suse.org lists FW840.00. Compare to other machines like diesel+petrol to see if there is a newer ASM version?
- Look for new firmware for the machine, just search for new firmware on IBM web pages
- Check if the new firmware means we do not need https://github.com/os-autoinst/os-autoinst/pull/1554 anymore, if yes, remove again, if no, remove again but add according settings to the machine settings in openQA, this is also what "adamw" did:
[04/11/2020 17:41:52] <adamw> okurz: i don't really know what the consequences of it are, but i tend to the idea that qemu wouldn't be trying to make it the default without reason :) i can ask some virt guys if you like
[04/11/2020 17:42:09] <adamw> okurz: but on the whole, yes, it seems to be it'd be more appropriate to put it in your templates rather than hardwire it into os-autoinst.
[04/11/2020 17:42:29] <adamw> that's what i was doing when we had the problem (i was setting an older machine type in our ppc64le Machine vars)
- Copied from action #75259: 100% of powerpc tests incomplete auto_review:"(?s)Running on power8.*qemu-system-ppc64: Requested safe cache capability level not supported by kvm":retry added
- Related to action #63142: Upgrade firmware of ppc9 machine redcurrant added
- Assignee set to nicksinger
as you want to involve aeisner could you please also ask him about power8.o.o
- Subject changed from Check if new firmware for power8.o.o exists to Check if new firmware for power8.o.o exists and remove os-autoinst workarounds again when according machine settings are applied when necessary
- Description updated (diff)
- Target version changed from future to Ready
- Target version changed from Ready to future
I've created an infra ticket which you should already see on the osd-admins ML (Subject: [openQA][ppc] Please upgrade the firmware(s) for power8.opensuse.org). RT might reveal our ticket id in some minutes too…
- Assignee deleted (
nicksinger)
I think we cannot expect anybody else then us to take this task. As I'm not planning to work on this task any time soon I unassign myself. It could be worth to do some kind of mob/pair session on this topic.
- Target version changed from future to Tools - Next
- Target version changed from Tools - Next to Ready
- Subject changed from Check if new firmware for power8.o.o exists and remove os-autoinst workarounds again when according machine settings are applied when necessary to Check if new firmware for kerosene (aka. power8.o.o) exists and remove os-autoinst workarounds again when according machine settings are applied when necessary
- Description updated (diff)
- Status changed from New to Workable
- Subject changed from Check if new firmware for kerosene (aka. power8.o.o) exists and remove os-autoinst workarounds again when according machine settings are applied when necessary to Check if new firmware for kerosene (aka. power8.o.o) exists and remove os-autoinst workarounds again when according machine settings are applied when necessary size:M
- Priority changed from Low to Normal
- Status changed from Workable to In Progress
- Assignee set to okurz
- Assignee changed from okurz to nicksinger
Used https://www.ibm.com/support/fixcentral/main/selectFixes to first enter the machine type "8247-22L" and then following the wizard to receive a download link for "POWER8 System Firmware SV860_245 (FW860.B3)" and grabbed the tar.gz file.
I then followed https://www.ibm.com/support/pages/node/6985523 "Installing the Firmware" to install it which immediately rebooted the system. It took ~5 minutes before anything came back online again but we now see "FW860.B3" in the ASM web interface.
Haven't tested yet if the workaround for qemu is still needed or if we need a different firmware (e.g. for PCIe cards) for that. Will do that now.
https://github.com/os-autoinst/os-autoinst/pull/1554 mentions the previous error we had:
QEMU: qemu-system-ppc64: Requested safe cache capability level not supported by kvm, try appending -machine cap-cfpc=broken
now we have:
kerosene-8:~ # /usr/bin/qemu-system-ppc64
qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-cfpc=workaround
qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-sbbc=workaround
qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-ibs=workaround
qemu-system-ppc64: warning: TCG doesn't support requested feature, cap-ccf-assist=on
"workaround" sounds better than "broken" so maybe we already improved? I'm trying to research a little more about this to understand the impact and possible fixes (more firmwares?)
nevermind, with "-enable-kvm" everything works just fine. I hot-patched the worker to see if the mentioned options are still needed. So far it looks quite successful:
- Status changed from In Progress to Feedback
- Due date set to 2024-04-17
- Status changed from Feedback to In Progress
Got feedback about mania: https://suse.slack.com/archives/C02CANHLANP/p1712233554981339 which fails https://openqa.suse.de/tests/13943972 because of:
[2024-04-04T12:22:56.372110Z] [warn] [pid:11706] !!! : qemu-system-ppc64: Requested count cache flush assist capability level not supported by KVM
[2024-04-04T12:22:56.372192Z] [debug] [pid:11706] QEMU: Try appending -machine cap-ccf-assist=off
Machine was still running FW860.42 (from 2018), I just conducted the upgrade to FW860.B3 (from 2023).
New firmware commited onto mania after validating it works correctly again. Diesel is fine as well. Grenache didn't execute tests for a long time but given it is hmc-controlled the change should have no impact on that one except we use nedsted virt somewhere.
This leaves only petrol as PPC-worker in OSD and it indeed shows the same problem. I'm going to upgrade the firmware there as well.
petrol is our first different PowerPC platform. I looked up the product ID in http://petrol-sp.qe.nue2.suse.org -> "FRU Information" -> "FRU Device ID: 3" which is "8335-GCA" and used it to download "OP820" for it with version "OP8_v1.12_2.98". The included readme (https://ak-delivery04-mul.dhe.ibm.com/sar/CMA/SFA/08ct2/0/S822LC-8335-GCA-GTA-OpenPowerReadme_op820.30.xhtml) mentioned the necessary ipmitool commands to flash it:
ipmitool -H <BMC_IP> -U ADMIN -I lanplus -P admin -z 30000 hpm upgrade <xxxxx.hpm> component 0 force
ipmitool -H <BMC_IP> -U ADMIN -I lanplus -P admin -z 30000 hpm upgrade <xxxxx.hpm> component 1 force
# Wait for BMC to reboot (It takes about 2-5 minutes for BMC to reach ready state. The 5 minute wait is recommended)..
ipmitool -H <BMC_IP> -I lan -U ADMIN -P admin raw 0x3a 0x0a — If it returns 0x00 then BMC is at ready state otherwise it is not yet ready to continue with next step
ipmitool -H <BMC_IP> -U ADMIN -I lanplus -P admin -z 30000 hpm upgrade <xxxxx.hpm> component 2 force
I used mania to execute these commands as it is in the same network and it reduces the risk of a failed flash. After a reboot of the machine the qemu issues are gone:
https://openqa.suse.de/tests/13949433
- Status changed from In Progress to Resolved
- Copied to action #158526: Apply the latest firmware+BIOS upgrade for diesel as well size:S added
Also available in: Atom
PDF