QA » openQA Project » openQA Infrastructure

Take a look on unused Power8 machines in server room 2 .. probably rack 1 - Matthias will add more details

As discussed I sent an email to research@suse.de to find out what options we have here.

I've asked Hector for help to reach out to somebody from IBM: https://suse.slack.com/archives/C02CANHLANP/p1664464326700249

nicksinger wrote:

I've asked Hector for help to reach out to somebody from IBM: https://suse.slack.com/archives/C02CANHLANP/p1664464326700249

Hector is suggesting we open a bugzilla with an explanation, versions used and references to issues to be mirrored on the IBM side

https://sd.suse.com/servicedesk/customer/portal/1/SD-101953 received an update. According to aeisner we have credentials on the system for all current team members. I tried to connect systems qa-power8, qa-power8-3 and the o3 machine power8-ipmi but I can only see two machines "haldir" and "legolas". I assume that we are members of some "QA-SAP" group which would not make sense. I put a comment into the ticket and waiting for aeisner's response.

I have added power8.openqanet.opensuse.org as well as qa-power8-3.qa.suse.de in both https://powerhmc1.arch.suse.de/ as well as https://powerhmc3.arch.suse.de/ . In both cases the machine shows up under the IPv4 address 10.160.1.104 with the error message as from #116078#note-15 . Maybe the firmware update as explained in https://bugzilla.opensuse.org/show_bug.cgi?id=1202275 was actually newer than the HMC install date of hmc3, installed 2021-12. So, can we update hmc3? First we tried to change from OPAL mode to PowerVM mode in ASM. Then there was a message about needing system password so we set a "new" system password, the one from salt pillars. Then we called the action to reset the system connection. Now the machine was correctly found in hmc3. We powered on the system. Also in hmc1 we reset the connection and the system showed up fine. But we could not find where to configure IPMI or something. We could connect to the console of the "system partition" over HMC and we were in a "SMS menu" where one could select boot entries and such but there we did not find any storage volume, possibly because that would need to be a virtualized storage container anyway. So we opted to remove the machine from both HMCs again but ASM still prevented us to change to OPAL mode. First in ASM one needs to change in "Hardware Management Consoles" to reset to a non-HMC managed instance. and change the firmware mode back to OPAL mode. Still can't connect over IPMI.

I suggest to ask around to more people what we try to achieve, what has been done, what failed and ask what we should do then

We've already started a mailing list thread a while ago: https://mailman.suse.de/mlarch/SuSE/research/2022/research.2022.09/msg00047.html

The conclusion was to ask Hector: https://suse.slack.com/archives/C02CANHLANP/p1664464326700249 - This discussion switched to Power9 at some point. Hector eventually forwarded some information to IBM but it was apparently not of any help. As far I as can see, we haven't gained any new information on how to restore IPMI access of the machine.

Regarding PowerPC management, everyone should be able to login on https://powerhmc3.arch.suse.de/

asked for help in https://suse.slack.com/archives/C02B5UVEC94/p1668602502736249

Helpful pointers from Michal Suchanek:

(Michal Suchanek) doesn't openQA use network boot on Power machines already?
This is an article that shows how to use low-level VIOS commands to create and manage virtual optical media and drives
https://www.ibm.com/support/pages/how-configure-vios-media-repositoryvirtual-media-library-ex-aix-installrestore
It's also posssible to configure on HMC https://www.ibm.com/support/pages/creating-virtual-optical-drive-hmc-managed-vios-partition
ibm.comibm.com
If you have bare metal machines that cannot be connected to HMC this (likely) applies https://www.ibm.com/docs/en/linux-on-systems?topic=systems-petitboot-bootloader
System can be booted directly from a network server, https://download.suse.de/install/SLP/ should do.
In OPAL mode the last one applies. You either have PowerVM LPARS or OPAL, not both
(Oliver Kurz) Can I see somewhere if the physical machine has a physical network connection besides the one used for management?
(Michal Suchanek) On a HMC-connected machine you would see it on HMC. All machines have some sort of hardware inventory available on the web management interface but the interface varies depending on product line.
(Oliver Kurz) would you like to check the machine "qa-power8" on https://powerhmc3.arch.suse.de/dashboard/#resources/systems yourself? I failed to find > (Michal Suchanek) The petitboot bootloader is Linux so if you boot into that it should have an option to enter shell, too
(Oliver Kurz) Yes, I used that to kexec, no problem if I would be able to reach that state. But for the machine"qa-power8" I am not even sure if network is connected and for a second machine power8.openqanet.opensuse.org the thing is according to https://www.ibm.com/docs/en/linux-on-systems?topic=qsglpsls-installing-ubuntu-power-system-lc-servers-using-network-boot we would need a monitor+keyboard at least for the initial setup after we lost remote IPMI after a firmware upgrade but according to Nick Singer in https://progress.opensuse.org/issues/116078#note-28 the machine does not have any graphical output port
(Michal Suchanek) Isn't there an option to use a physical serial port?
(Oliver Kurz) I would need to ask someone to get into that server room Nbg SRV1 and check that
(Michal Suchanek) If you are unsure that network is connected you need to ask people who connect the network. In petitboot you should be able to configure network card and see what MAC address it has.
(Michal Suchanek) the option to use physical serial port would have to be configurable in the web management [ASM]

Turned out to be available over "System Service Aids", "Serial Port Setup" but only if the machine is not managed by an HMC

still the current plan to find an appointment when to do that.

Gerhard is back from vacation and we will visit SRV1 tomorrow (2023-01-10) to build out the machine so I can mount it in the TAM lab for further debugging/testing

Machine is now mounted above merckx (racktables entries pending). USB2Seral connected to seth (ttyUSB0) and ASM reachable via https://10.162.29.184/cgi-bin/cgi. Network cables connected to qanet12:
port27 == HMC1
port3 == C10-1

nicksinger wrote:

Machine is now mounted above merckx (racktables entries pending). USB2Seral connected to seth (ttyUSB0) and ASM reachable via https://10.162.29.184/cgi-bin/cgi. Network cables connected to qanet12:
port27 == HMC1
port3 == C10-1

Updated racktables but one point is not clear: Can you help me to relate C10-1 to one of the names on https://racktables.nue.suse.com/index.php?page=object&tab=ports&object_id=2012 please?

I now connected enP3p9s0f0 to qanet12-port3, same interface as was used in before, is this correct?

@nicksinger when you happen to go to the lab you can take a look for the labels of the ports and crosscheck with https://racktables.nue.suse.com/index.php?page=object&tab=ports&object_id=2012 and then just delete the present connection, add labels and not care that much about the Linux internal interface names.

nicksinger logged in as "admin" into the ASM and then in "Login Profile -> User Policy" he could enable the "celogin" account. Then we could login into the ASM with the "celogin" account and the temporary password of the day and see the option "System Service Aids -> Service Processor Command Line". nicksinger tried "iptables -F" to prevent any network blocks but we saw no effect of that, i.e. "insufficient user privileges" or something like that. But we are still running the old firmware version 840. In https://bugzilla.opensuse.org/show_bug.cgi?id=1202275#c20 mkittler mentioned

New T Image:     SV860_243
New P Image:     SV840_043

so it looks like the firmware was upgraded but only for "T". According to "Power/Restart Control -> Power On/Off System" one can select "Temporary" or "Permanent" firmware for the next boot. We selected "Temporary" and then saved settings and reset the service processor using "System Service Aids -> Reset Service Processor". https://10.168.193.46/ now shows "https://10.168.193.46/cgi-bin/cgi". Now "System Configuration" shows "Security". In there in "External Services Management" we could find a pull-down menu for IPMI which was set to "Disabled" and we set it to "Enabled" and saved. Now with ipmi we could not connect. But over "Login Profile -> Change Password" we changed for user id "IPMI", e.g. to "admin" and then tried to authenticate over ipmitool with "ADMIN" or "admin" but did not succeed though it looks like the server is responding, just that we can't authenticate.

EDIT: nicksinger succeeded in finding out what we need to do. If we just specify a password and no username on ipmitool then we can connect, e.g. ipmitool -C 3 -I lanplus -H 10.168.193.46 -P XXX sol activate. We set the default openQA worker password from https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls now. We could see the system booting up and reaching petitboot. The system does not try to boot any option by default and our PXE boot setup on qa-jump.qe.nue2.suse.org does not currently support ppc. Also we did not find any permanent storage like hdd drives. However what we can do temporarily is exit to the petitboot shell and use kexec to load a usable system so what we did for now:

cd /tmp
wget http://download.opensuse.org/ports/ppc/distribution/leap/15.4/repo/oss/boot/ppc64le/linux
wget http://download.opensuse.org/ports/ppc/distribution/leap/15.4/repo/oss/boot/ppc64le/initrd
kexec -l linux --initrd initrd --command-line 'install=http://download.opensuse.org/ports/ppc/distribution/leap/15.4/repo/oss/ sshd=1 sshpassword=XXX rootpassword=XXX console=tty0 console=hvc0 rescue=1'

The rescue system booted and we could login. Surprisingly also there no storage devices could be found.

System reachable again. I made a mistake in the reserved IPv4 address, fixed in https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/4124 . Looked into the emergency mode with nicksinger and we found that the partition /dev/sda1 used for /var/lib/openqa is corrupt and needs a manually triggered fsck.ext4 -y /dev/sda1. After that we could call mount -a successfully. But we also made the fsck for /var/lib/openqa non-critical so that the system would continue boot on problems which would be more convenient. We saw that there is incomplete console output on boot so in /etc/default/grub we added console=hvc0 to the boot line. Also network config for the currently connected "eth7" was not there so we did cd /etc/sysconfig/network && ln -s ifcfg-{enP1p9s0f1,eth7}

https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/4124 to add an DHCP/DNS entry for the currently actually connected interface

In /etc/openqa/workers.ini we changed HOST to https://openqa.opensuse.org and then added a section

[https://openqa.opensuse.org]
TESTPOOLSERVER = -e 'ssh -p 2214 -i /var/lib/openqa/ssh/id_ed25519 -o "UserKnownHostsFile /var/lib/openqa/ssh/known_hosts"' geekotest@gate.opensuse.org:/var/lib/openqa/tests

created an according ssh key which we allowed on o3. Now jobs are picked up and worked on put missing deps, e.g. SemVer. Calling zypper dup.

Apparently there is no openSUSE Leap 15.5 ppc, reported https://bugzilla.opensuse.org/show_bug.cgi?id=1215874 . We can consider to move to Tumbleweed or SLE. I would try Tumbleweed. But for now let's stay on Leap 15.4. Regarding the missing dependencies I installed os-autoinst-distri-opensuse-deps. Now zypper dup wants to delete os-autoinst and openQA and such. I don't know why. zypper in --force openQA-auto-update openQA-client openQA-common openQA-continuous-update openQA-worker os-autoinst os-autoinst-swtpm perl-DBD-Pg perl-Mojo-Pg fixed it. Restarted first worker instance. Monitoring. https://openqa.opensuse.org/tests/3608908# looks good for now.

in /etc/zypp/repos.d/repo-oss.repo replaced the URL

-baseurl=http://download.opensuse.org/ports/ppc/distribution/leap/$releasever/repo/oss/
+baseurl=http://download.opensuse.org/distribution/leap/$releasever/repo/oss/ppc64le/

From SoL

[   13.017096][ T1737] ses 0:0:8:0: Attached Enclosure device

Welcome to openSUSE Leap 15.5 - Kernel 5.14.21-150500.55.28-default (hvc0).

eth0:
eth1:
eth2:
eth3:
eth4:
eth5:
eth6:
eth7: 10.168.193.11 fe80::42f2:e9ff:fe5d:3d3f


kerosene-8 login: [  310.260509][  T101] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[  310.260565][  T101] CPU: 0 PID: 101 Comm: kworker/dying Kdump: loaded Tainted: G        W      X  N 5.14.21-150500.55.28-default #1 SLE15-SP5 55c4465f644d5be5ce7c285d13ad6540a5a1471c
[  310.260604][  T101] Call Trace:
[  310.260619][  T101] [c000000006c13ac0] [c00000000088c900] dump_stack_lvl+0x74/0xa8 (unreliable)
[  310.260646][  T101] [c000000006c13b00] [c00000000015adb4] panic+0x180/0x42c
[  310.260665][  T101] [c000000006c13b90] [c000000000e0ba24] __schedule+0x13e4/0x1440
[  310.260685][  T101] [c000000006c13ca0] [c0000000001b2b34] do_task_dead+0x64/0x70
[  310.260705][  T101] [c000000006c13cd0] [c0000000001647b4] do_exit+0x864/0xb60
[  310.260723][  T101] [c000000006c13da0] [c000000000194ba4] kthread+0x154/0x1d0
[  310.260741][  T101] [c000000006c13e10] [c00000000000cf64] ret_from_kernel_thread+0x5c/0x64

the system should not run this recent kernel version. On diesel+petrol we also run downgraded older versions.

I must have applied the downgrade in a wrong way so what I did now:

for i in 3 2 1; do zypper rl $i; done
zypper in --oldpackage http://download.opensuse.org/update/leap/15.3/sle/ppc64le/kernel-default-5.3.18-150300.59.93.1.ppc64le.rpm http://download.opensuse.org/update/leap/15.3/sle/ppc64le/util-linux-2.36.2-150300.4.23.1.ppc64le.rpm
zypper rm kernel-default-5.14.21
for i in kernel-default util-linux; do zypper al --comment "poo#119008, kernel regression boo#1202138" $i; done
reboot