QA » openQA Project » openQA Infrastructure

https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/43 for proper DHCP/DNS

ok, we learned that the reported IPv4 address 10.163.41.238 is actually my own so not helping.

I asked in #eng-testing as well as #help-it-ama

(Oliver Kurz) anyone being in or going to Nbg Maxtorhof SRV2 now/soon who could ensure that there is a network connection for QA-Power8 ? Or do you know of a better channel where to coordinate such work?

If somebody in the meantime or if there is no answer stops by one can still ensure that there is a cable connection.

acarvajal found that there is a physical connection but he did not remember on which switch port the cable is connected.

We can also try make use of the HMC control servers, e.g. "powerhmc1.arch.suse.de" or something

We added the machine to both powerhmc1.arch.suse.de and powerhmc3.arch.suse.de and in both cases we can configure the "partitions", i.e. virtual machines, on the PowerVM managed machines. The mode between OPAL – for bare-metal installations – and PowerVM can be switched in ASM, in this case https://qa-power8.qa.suse.de in the menu "Firmware Configuration". For this no HMC connection must be configured. As necessary remove the connection in the HMC and in the ASM menu "Hardware Management Consoles". The problem is that ipmi seems to be not configured to allow remote access so we would need an OS installation on the host first to call ipmitool locally to allow remote access. As alternative we should try to use one of the VMs or create a new VM and try to enable remote access ipmitool.

So the suggestion is: (Re-)install one of the LPAR VMs and try to call ipmitool to allow remote access.

What are the ASM credentials for https://qa-power8.qa.suse.de? I tried multiple combinations of standard ones we use elsewhere but nothing worked.

I checked the physical network connections. Updated racktables with entries for the two quad-ethernet access cards. There was already one connection to C10-1 (top) connector, gi30. Updated racktables accordingly. gi30 on qanet15nue.qa.suse.de is in VLAN12 so should be reachable in QA network. Connected a second connection C12-2 (second card on the right, second connector from top) to gi8, added to VLAN12 as well. And connected C10-4 (bottom) to gi2, VLAN12.

no response yet, asked again

https://www.youtube.com/@nigelargriffiths could be helpful. as a reference for "training". In the meantime mgriessmeier and me have experimented and compared the setup on qa-power8 with other machines, e.g. "redcurrant" and "blackcurrant". with the views of the "virtual network topology" it looked like on other machines there is a proper network connection from the physical network all down to the LPARs, on qa-power8 there isn't. We managed to install the VIOS instance. First we tried the internal network connection (192.168.X.X) and a VIOS version which failed to deploy, then we selected the external network connection (10.X.X.X) and select a VIOS version with "SP" in the name. This eventually got stuck. Then we connected over ssh to powerhmc1, text console to the vios instance. That showed a progress of 0% and an error message that no volume group could be found with a waiting time of 105 minutes. We just tried to do something useful in the system, typed "q", "quit", "exit" and then suddenly the installation continued and progressed successfully till the end. Then in HMC the vios instance had a "RMC" connection. Then in the LPAR we could select the "bridged" network, configure an entry in the dhcp config on qanet including PXE, and then we could boot over network from a manual grub prompt using the existing outdated PXE boot entries as template with linux+initrd specified from relative paths to what we could find on qanet. We verified that we could access the YaST installer over ssh but did not progress with the installation. I suggest to redo all the above steps, could even be done on power8.openqanet.opensuse.org, and document again with all knowledge holes filled.

Next steps:

• Connect via SSH
• Install an operating system
• Make the machine usable as a worker

We could do that in our mob session on Thursday since this should be a good learning opportunity for those not familiar with the setup, or another ad-hoc slot where there's more people involved.

We'll probably need a follow-up after that to get the network setup correctly after the machine's installed. See #119008 for reference.

I looked into this ticket again, so here a short summary:

• I was able to login on https://qa-power8.qa.suse.de. Looks like the machine is configured to be in PowerVM mode (which should also explain why IPMI access is not working).
• The system also appears to be already powered on. Not sure how to connect via SSH, though (and thus also not sure how to proceed with this ticket).
• I have access to https://powerhmc3.arch.suse.de (not https://powerhmc1.arch.suse.de). It shows the host qa-power8 and I can even open the terminal for the test VM that's present. I'm still not sure how to proceed from there to install an OS. Maybe it helps to connect to the HMC via SSH. What that meant with "Connect via SSH" in the previous comment?
• I've read #119059#note-16. So this has already been attempted but the network didn't work. After #119059#note-23 and #119059#note-26 this shouldn't be a problem anymore.
• I could theoretically try to follow-up on #119059#note-26 but the details (which are left to be documented) aren't clear to me. The comment also mentions hmc1 but I have only access to hmc3.

We could do that in our mob session on Thursday since this should be a good learning opportunity for those not familiar with the setup, or another ad-hoc slot where there's more people involved.

This would be a good idea. I feel a bit lost here.

So first: It shouldn't technically matter if we use hmc1 or hmc3 but in the future hmc3 is to be used as we have full control over it and everybody has their own account. And I recommend you try to connect to the serial terminal by connecting to the HMC over ssh and then call the mkvterm commands to connect to the terminal. Regarding "ssh" to the final system I what I did was start an installation with Linuxrc parameter ssh=1 and then connect to the SUT, not the HMC

We took another go at this story and connected with ssh to powerhmc3.arch.suse.de, rebooted the machine in SMS and selected network boot ending up in the grub menu served by qanet. We could continue from here, install something like a current Leap 15.4 and similar as for grenache-1 we could try to use nested kvm on that. But to be able to use the machine within o3 we also need an HVM there. Consider this out of scope for the current ticket, see #123712 for that.

I disconnected qa-power8 from HMC within https://powerhmc3.arch.suse.de/dashboard/#resources/systems so that we can switch to OPAL mode in the ASM and try to have IPMI and then continue with an installation of bare-metal Leap and then use as openQA kvm worker and then move to o3 network.

I disconnected qa-power8 from HMC within https://powerhmc3.arch.suse.de/dashboard/#resources/systems …

Somehow that's not recognized by the ASM. It still says "Changing the firmware type is only allowed when the system is in powered off state and not managed by a management console." despite the system being powered off for quite a while now. That also means I could not enable IPMI so far. I hope I've been using the right ASM. (I have been using https://qa-power8.qa.suse.de as mentioned in the ticket description.)

Well, then I suggest to reconnect with the HMC, ensure it's powered off and then disconnect again

I connected again via https://powerhmc3.arch.suse.de. Then I disconnected all connections to other HMCs that were still shown (there were 2 of them). Then I disconnected it from powerhmc3 itself. Unfortunately this didn't help. One can actually list the connected HMCs in the ASM and it says that the following are still connected: 10.161.26.36, 172.16.0.1
There was actually a 3rd one which I've removed. The remaining two aren't removable, though. Hence I'm not sure how to proceed. We somehow need to remove those (stale) entries.

The remaining two aren't removable, though. Hence I'm not sure how to proceed. We somehow need to remove those (stale) entries.

Let's check it in the infra call. Maybe Nick knows?

https://powerhmc1.arch.suse.de/dashboard/ is currently "Service Unavailable". The ASM shows that there is still a connection to powerhmc1.arch.suse.de by IPv4 address. mkittler could remove some connections with a checkbox but not for hmc1. mkittler has tried "soft-reset" of the service processor. We now tried the "reset", i.e. "hard reset". right now https://qa-power8.qa.suse.de is not reachable anymore. Maybe after reset and with hmc1 being broken the connection vanishes? Nope, the connection still show up. But I could still login over ssh to powerhmc1.arch.suse.de . I logged in using ssh hscroot@powerhmc1.arch.suse.de using credentials from https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls#L1198
and did

rmsysconn -m qa-power8 -o remove

After that in the ASM https://qa-power8.qa.suse.de/ in the menu "Hardware Management Consoles" I could remove the one connection using the checkbox. But there is still a connection "V467f48*7044a59" with IPv4 address 172.16.0.1 which according to nsinger is the "PowerVM HMC network" but we don't know which HMC that is. I guess the address ending with a .1 still refers to powerhmc1. That would be my guess. Right now powerhmc1.arch.suse.de is rebooting. I suggest to try again at a later time to login over ssh to powerhmc1 and if that does not help then open a ticket and ask in more channels.

I think this is actually this ticket, hence adding the bugref to https://openqa.suse.de/tests/10528653:

[2023-02-17T19:05:46.030419+01:00] [debug] [pid:575072] Could not connect to hscroot@powerhmc1.arch.suse.de, Retrying after some seconds...
[2023-02-17T19:05:56.033590+01:00] [info] [pid:575072] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
  Error connecting to <hscroot@powerhmc1.arch.suse.de>: Connection timed out

See also #124391 regarding powerhmc1.arch.suse.de being unavailable, and discussions in Slack

I logged into powerhmc2 over ssh and did a ´cat /proc/net/fib_trie´ to confirm that hmc2 is indeed 172.16.0.1. Afterwards I could do another ´rmsysconn -m qa-power8 -o remove´. The option to change firmware mode was still not enabled but the "Hardware Management Consoles" where previous connections where displayed now showed that no HMCs are connected anymore and gave a quick guide how to "Reset the server to a non-HMC managed configuration". For this you need to go to "System Service Aids", select "Factory Configuration" and use the option "Reset server firmware settings". After this go back to "Hardware Management Consoles" and it now shows a button "Reset the server to a non-HMC managed configuration" which you can click. The ASM confirms with "Operation was successful. This Server is no longer managed by an HMC". Now the "Firmware Configuration" can be changed from "PowerVM" to "OPAL"

Looks like you also already have changed the config to "OPAL" and you've also set the console type to IPMI. Unfortunately I still wasn't able to connect via IPMI, e.g. ipmitool -I lanplus -C 3 -H qa-power8.qa.suse.de -U … -P … power status didn't work. Looks like this host is in the same state as the other o3 worker power8. I've also tried to set the IPMI password again (to be identical to the ASM password) but it didn't work.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: create_hdd_textmode_hmc_ntlm
https://openqa.suse.de/tests/10695007#step/bootloader/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

blocked by #125216

okurz wrote:

blocked by #125216

This was unblocked more recently, and I'm getting started on the evaluation. Given the priority we might also re-consider e.g. pick the easiest to setup and maintain depending on how strongly stakeholders feel about the options. Just a thought I would consider anyway, but let's see what we find.