QA » openQA Project » openQA Infrastructure

@okurz well, as I understood from Antonio, he has prepared powerhmc3 already, whether and how it is running for QA power machines, I don't know.
Normally redundant powerhmc should cover this kind of failure. Long time ago it was a decision that we shutdown own QA powerhmc server.

mkittler wrote:

I've created a dedicated user for openqa tests on hmc3 and a MR to change the config accordingly: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/496

Is that enough or do I need to:

• add hosts manually on hmc3? I suppose the IPs to use here can be taken from https://racktables.suse.de/index.php?page=object&tab=default&object_id=3038 (QA-Power8-2.qa.suse.de, not sure which interface is relevant) and https://racktables.suse.de/index.php?page=object&object_id=11220 (redcurrant.arch.suse.de). Not sure what to enter as "BMC" credentials.
• remove hosts manually from hmc1? I am not sure how to do this because the credentials for hmc1 mentioned in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls#L1198 don't seem to work anymore. Maybe I could do this from hmc3, though.

@mkittler: the old powerhm1 is gone with all managed machines/lpars. We need to discovery/connect these machine again and manage these on powerhmc3. As I understood, we just use our nis account to access powerhmc3 in future. but for openQA maybe it requires a dedicated user for powerhmc to trigger connection and run installation steps afterwards.

I've got a statement from Antonio Eisner:

Okay, the explanation powerhmc1 has now been reinstalled because it did not come up after a reboot. You could migrate all machines from powerhmc3 to powerhmc1. The version is higher. At some point we update powerhmc2 and then we can turn off powerhmc3.

That means this ticket makes no sense. Those hosts should rather be re-added to hmc1.

He also shared a link to https://confluence.suse.com/display/labsprojectmanagement/Hardware+inventory but this doesn't answer my questions about FQDNs/IPs/BMC-credentials.

mkittler wrote:

I've got a statement from Antonio Eisner:

Okay, the explanation powerhmc1 has now been reinstalled because it did not come up after a reboot. You could migrate all machines from powerhmc3 to powerhmc1. The version is higher. At some point we update powerhmc2 and then we can turn off powerhmc3.

That means this ticket makes no sense. Those hosts should rather be re-added to hmc1.

I still think this ticket makes sense. As I stated there shouldn't be a problem to have machines connected to both. And we were asked to use powerhmc3 so let's do it.

He also shared a link to https://confluence.suse.com/display/labsprojectmanagement/Hardware+inventory but this doesn't answer my questions about FQDNs/IPs/BMC-credentials.

I was able to find out the IP of redcurrant as it is still connected to hmc2 (and I could guess the credentials for hmc2). I could also guess the ASM credentials of redcurrant. However, adding redcurrant to hmc3 still doesn't seem to work. It is just showing "Systeme verbinden". Maybe redcurrant is not reachable from hmc3 (as it is not reachable from the normal engineering VPN as well).

I could not find the IP of QA-Power8-2-8247-22L-SN1010D5A in a similar way as it is not connected to hmc2.

I've asked Antonio Eisner to give use the new credentials of hmc1 but I doubt this will be useful at this point.

So I have no idea how to do what @zluo is asking for.

mkittler wrote:

I was able to find out the IP of redcurrant as it is still connected to hmc2 (and I could guess the credentials for hmc2). I could also guess the ASM credentials of redcurrant. However, adding redcurrant to hmc3 still doesn't seem to work. It is just showing "Systeme verbinden". Maybe redcurrant is not reachable from hmc3 (as it is not reachable from the normal engineering VPN as well).

I could not find the IP of QA-Power8-2-8247-22L-SN1010D5A in a similar way as it is not connected to hmc2.

I've asked Antonio Eisner to give use the new credentials of hmc1 but I doubt this will be useful at this point.

So I have no idea how to do what @zluo is asking for.

redcurrant is connected to hmc2, but you cannot manage it for restart or shutdown after hmc1 is gone. And it has novalink installed like other Power machines used for openQA.
I am not sure about this: to check sub network of hmc3 and redcurrant, they need to be reachable from each other. Maybe there is an issue with gateway if both they are in different sub networks.

QA-Power8-2-8247-22L-SN1010D5A was connected to hmc1 only. So Antonio might know the differences of hmc versions, is the new hmc still able to manage Power8 machine?

I was able to add grenache on hmc3 by scanning IPs and guessing credentials. Apparently grenache-fsp1.qa.suse.de and grenache-fsp2.qa.suse.de are redundant connections to the same host. Note that this is not a big step forward because those tests are using Novalink (spvm backend) and work independently from the hmc anyways (see e.g. https://openqa.suse.de/tests/10640668 for a passing test that passed regardless of this).

@nsinger has also created an SD ticket regarding redcurrant: https://sd.suse.com/servicedesk/customer/portal/1/SD-114488

By the way, the IP of redcurrant was 172.16.222.73 (visible on hmc2).

mkittler wrote:

@nsinger has also created an SD ticket regarding redcurrant: https://sd.suse.com/servicedesk/customer/portal/1/SD-114488

By the way, the IP of redcurrant was 172.16.222.73 (visible on hmc2).

please make sure to add all such information to https://racktables.nue.suse.com/ where applicable

Note that Antonio also suggested to "create a ticket" (not mentioning where) for access via NIS credentials on HMC1. I suppose that's nevertheless something we should peruse (as it cannot hurt to have more options than HMC3).

Toni told us in the SD ticket that HMC3 will be switched of after every machine is migrated to HMC1 - so should we reconsider this task here as "move to powerhmc1"?

MR merged.

is there any connection with ppc64le-spvm now having a high rate of failures (not seen in months btw, it was very very stable until todays build 81.1) ?
https://openqa.suse.de/tests/overview?arch=&flavor=&machine=ppc64le-spvm&test=&modules=&module_re=&distri=sle&version=15-SP5&build=81.1&groupid=129

I don't think so. If you're affected by this issue then your test would just fail regardless of how often you retry. Besides, the spvm backend should be independent from the HMC.

For some reason power8-2 is already in hmc1: https://powerhmc1.arch.suse.de/dashboard/#resources/systems/9d3c4d88-a65a-3ad5-97e0-a3f9147d76fb/logical-partitions - I used the opportunity to update racktables with the according (static) IP for that interface.
I tried to verify that redcurrant is executing jobs again and found a strange needle in the most recent run (https://openqa.suse.de/tests/10720780#step/bootloader_start/4) which I don't understand where it comes from. Asked in #team-qa-tools for help

openQA test failed because hmc1 was not the primary management controller for redcurrant and couldn't control the power state for LPARs. I had to execute

chcomgmt -o setmaster -t norm -m redcurrant

Testing again with https://openqa.suse.de/tests/10747085

This bug is still referenced in a failing openQA test: online_sles15sp4_pscc_basesys-srv-desk-dev_def_full_zypp
https://openqa.suse.de/tests/10815518#step/bootloader#1/1

@zoecao This looks to be a different issue? It's novalink, not hmc?

# Test died: no candidate needle with tag(s) 'novalink-ssh' matched

$ openqa-query-for-job-label 'poo#124685'
10945318|2023-04-18 22:04:32|done|failed|select_disk||grenache-1
10945291|2023-04-18 21:21:09|done|failed|select_disk||grenache-1
10941323|2023-04-18 20:18:00|done|failed|select_disk||grenache-1
10918360|2023-04-14 08:53:05|done|failed|offline_sles15sp2_ltss_pscc_basesys-srv-phub_def_full_pre||grenache-1
10917811|2023-04-14 08:30:36|done|failed|sap_migration_online_zypper_sles4sap15sp2_pre||grenache-1
10917807|2023-04-14 08:30:35|done|failed|sap_migration_offline_scc_sles4sap15sp4_fullcd_pre||grenache-1
10917805|2023-04-14 08:25:54|done|failed|sap_migration_offline_scc_sles4sap15sp3_fullcd_pre||grenache-1
10917798|2023-04-14 08:21:34|done|failed|sap_migration_offline_dvd_sles4sap15sp3_fullcd_pre||grenache-1
10917775|2023-04-14 08:07:22|done|failed|offline_sles15sp2_ltss_pscc_basesys-srv-phub_def_full_pre||grenache-1
10912036|2023-04-13 08:56:58|done|failed|sap_migration_online_zypper_sles4sap15sp2_pre||grenache-1