Project

General

Profile

action #88189

Firmware upgrade of qa-power8-4.qa.suse.de

Added by nicksinger 8 months ago. Updated 5 months ago.

Status:
Rejected
Priority:
Normal
Assignee:
Target version:
Start date:
2021-01-25
Due date:
% Done:

0%

Estimated time:

Description

Motivation

We were told that we need to apply the most recent firmware on machines which might help with some problems

I upgraded qa-power8-4 to the newest firmware version available at Fix-Central. The machine is according to /proc/cpuid a 8348-21C-machine. I Flashed the hpm package according to the docs https://delivery04.dhe.ibm.com/sar/CMA/SFA/08ct3/0/8348_820.1923.20190613n.html#__RefHeading___Toc1210_1053759979 (See section 7.2). Afterwards I boot up the system again just to see:

5.83228|System shutting down with error status 0x90FF0003

Several resets of the system and BMC I don't see any message anymore. The webinterface reports errors with the "sensors" "OCC Active", "Boot Count" and "BIOS Golden Side".

History

#1 Updated by nicksinger 8 months ago

hpm check indicates that the BIOS seems to be untouched? (even though the previous "hpm upgrade" told me it updates it).

workstation ~ ยป ipmitool -I lanplus -C 3 -H qa-power8-4.qa.suse.de -U ADMIN -P admin hpm check ~/Downloads/8348_820.1923.20190613n_update.hpm

PICMG HPM.1 Upgrade Agent 1.0.9: 

Validating firmware image integrity...OK
Performing preparation stage...OK

Comparing Target & Image File version
--------------------------------------------------------------------------
|ID  | Name        |                     Versions                        |
|    |             |     Active      |     Backup      |      File       |
--------------------------------------------------------------------------
|*^ 2|BIOS         |   1.12 60000000 | ---.-- -------- |   1.12 96000000 |
|*  0|BOOT         |   2.16 51220300 | ---.-- -------- |   2.16 51220300 |
|*  1|APP          |   2.16 51220300 | ---.-- -------- |   2.16 51220300 |
--------------------------------------------------------------------------
(*) Component requires Payload Cold Reset
(^) Indicates component would be upgraded

#2 Updated by nicksinger 8 months ago

Flashing the same file resulted into a bootable system. Unfortunately the system still don't seem to find its OS in petitboot (as tracked in https://progress.opensuse.org/issues/81058). I will see if it somehow possible to update petitboot too

#3 Updated by nicksinger 8 months ago

I'm building the most recent version of openPOWER (https://github.com/open-power/op-build) with power8 support for "habanero" (the codename for the used platform in power8-4) now. This should include the newest available versions for skiboot, skiroot and petitboot (https://wiki.raptorcs.com/wiki/OpenPOWER_Firmware#Process has a nice overview how the different loaders interact with each other). Also a recommended read in that regard: https://sthbrx.github.io/blog/2016/05/13/tell-me-about-petitboot/ (how skiboot/skiroot and petitboot are linked)

#4 Updated by nicksinger 8 months ago

From "Petitboot System Information" before flashing the new openPOWER firmware:

 System type: 8348-21C
 System id:   684F75A

 Primary platform versions:
        IBM-habanero-OP8_v1.12_2.96
        op-build-v2.3-5-g98aa884
        buildroot-2019.02.1-16-ge01dcd0
        skiboot-v6.3.1
        hostboot-p8-c893515-pd6f049d
        occ-p8-a2856b7
        linux-5.0.7-openpower1-p8e31f00
        petitboot-v1.10.3
        machine-xml-c3998a5
        hostboot-binaries-hw041519a.opv23
        capp-ucode-p9-dd2-v4

BMC current side:
        Device ID: 0x20
        Device Rev: 0x1
        Firmware version: 2.16.5bff0000
        IPMI version: 2

 BMC golden side:
        Device ID: 0x20
        Device Rev: 0x1
        Firmware version: 2.08.d4470100
        IPMI version: 2

 Storage devices
 sda1:
  UUID:       f5de0a79-bfa8-41c9-ba86-c0d2df739acc
  mounted at: /var/petitboot/mnt/dev/sda1

sdb2:
  UUID:       eebe647f-e867-416e-a0fa-7a6732bfcf9d
  mounted at: /var/petitboot/mnt/dev/sdb2


 Management (BMC) interface
  MAC:  a0:42:3f:32:b4:71

#5 Updated by nicksinger 8 months ago

hrm, flashing the openPOWER firmware with pflash did not succeed as the kernel reports an IO issue as soon as you try to erase the old stuff

#6 Updated by nicksinger 8 months ago

  • Status changed from In Progress to Feedback

I've sent out a mail onto the openpower mailinglist asking for help. You can see a message of what I wrote on osd-admins@suse.de. It waits for moderators approval now.

#7 Updated by okurz 8 months ago

  • Target version set to Ready

#8 Updated by nicksinger 7 months ago

I got quite a few answers from the list. Unfortunately none of them really worked for me. The best Idea I got from it was to use the bmc system (you can ssh onto it) to flash the firmware. Unfortunately it is some old arm (ARM926EJ-S rev 5 (v5l)) with an ancient glibc (libc-2.11.3.so) and I was absolutely not able to get the pflash utility (cross-)compiled for it. Somebody in the mail thread offered me a binary download in an attachment which is a little bit too shady for me to execute.

So I'm out of ideas here. Anybody else has an idea how we could get a running version for pflash on that old system?
The code to compile pflash can be found here: https://github.com/open-power/skiboot/tree/master/external/pflash

#10 Updated by okurz 5 months ago

  • Description updated (diff)
  • Status changed from Feedback to Rejected

As discussed with nsinger we are not aware of further problems after we fixed the boot problems software side in #68053 so we can also close this ticket as rejected as we did not conduct the firmware upgrades.

Also available in: Atom PDF