QA » openQA Project » openQA Infrastructure

The first job I found was https://openqa.suse.de/tests/14779007#step/bootloader_start/20 from 1st of July. I will try to check the LPARs for which server ip is configured in petitboot. Not sure what caused it to change. The new/wrong IP points to "bare-metal4-ipmi" (https://gitlab.suse.de/OPS-Service/salt/-/blob/production/pillar/domain/oqa_prg2_suse_org/hosts.yaml#L85-88) which might misbehave in the network.

Setting due date based on mean cycle time of SUSE QE Tools

nicksinger wrote in #note-9:

I will try if I can spot something on the firewall with instructions provided in https://wiki.suse.net/index.php/OpenQA#Firewall_between_different_SUSE_network_zones and open a SD ticket if necessary.

Couldn't find anything relevant in the linked firewall log but I'm not even sure which firewall this shows. I collected a lot more details to file https://sd.suse.com/servicedesk/customer/portal/1/SD-162395

Basically I can currently say:

• .qe.prg2.suse.org (10.145.10.0/24) is affected
  • worker31 (10.145.10.4) is not affected
• .oqa.prg2.suse.org (10.145.0.0/21) is affected - this is mainly our PPC network range
• IPv4 and IPv6 behave mostly the same
• Could be related to the location PRG2-J11 but having worker31 unaffected seems to rule it out

While collecting more evidence and traces for https://sd.suse.com/servicedesk/customer/portal/1/SD-162395 I realized that the firewall was to blame on the other hosts I tried. Also on the SUT running on redcurrant-1 a firewall was active. With that disabled even redcurrant-1 was able to connect and download from qa-jump. This unfortunately brings me back to square one - that this problem only occurs from within the SMS. Next I will try out more LPARs on grenache to understand if this is happening on a machine level or only for specific LPARs. On machine level could point to an HMC or VIOS problem. Only on specific LPARs could point to some strange "corruption" of some persistent storage (we had something similar in the past already).

Made some progress while checking other redcurrant LPARs. I discovered that in every working condition I tried to reach the tftp server directly (port 69) but the SMS menu of PPC machines fail requests on port 67. So DHCP is already not working (I guess it is required for the filename? Everything else is "hardconfigured" in the SMS menu).

I checked suttner1&2 and indeed see problems there. I added this to https://sd.suse.com/servicedesk/customer/portal/1/SD-162395 and asked for help again.

No quick resolution within https://sd.suse.com/servicedesk/customer/portal/1/SD-162395 is expected.

Both suttners should be fixed again. Going to check if anything changed with with this machine.

Setting due date based on mean cycle time of SUSE QE Tools

I'm guessing this is not in progress right now in favor of more urgent tasks

the situation with the machine got worse and worse with every interaction I tried. First the LPAR didn't start (timeout in the HMC web dialogue). A reset of the ASM connection didn't change anything. Rebooting the VIOS didn't work because the HMC again timed out. After receiving a report in Slack that the machine is now offline, I tried to restart the whole hypervisor which again resulted in timeouts while trying to start the VIOS. I then tried to restart the ASM which now results in a completely unstable ASM connection:

From 10.255.255.1 icmp_seq=6742 Destination Host Unreachable
64 bytes from 10.255.255.153: icmp_seq=6759 ttl=64 time=0.263 ms
64 bytes from 10.255.255.153: icmp_seq=6774 ttl=64 time=0.387 ms
64 bytes from 10.255.255.153: icmp_seq=6789 ttl=64 time=0.238 ms
64 bytes from 10.255.255.153: icmp_seq=6790 ttl=64 time=0.282 ms
64 bytes from 10.255.255.153: icmp_seq=6791 ttl=64 time=0.256 ms
64 bytes from 10.255.255.153: icmp_seq=6805 ttl=64 time=0.238 ms
64 bytes from 10.255.255.153: icmp_seq=6806 ttl=64 time=0.284 ms
From 10.255.255.1 icmp_seq=6818 Destination Host Unreachable
From 10.255.255.1 icmp_seq=6819 Destination Host Unreachable
From 10.255.255.1 icmp_seq=6820 Destination Host Unreachable
64 bytes from 10.255.255.153: icmp_seq=6821 ttl=64 time=1024 ms
64 bytes from 10.255.255.153: icmp_seq=6822 ttl=64 time=0.363 ms
64 bytes from 10.255.255.153: icmp_seq=6836 ttl=64 time=0.284 ms
64 bytes from 10.255.255.153: icmp_seq=6837 ttl=64 time=0.300 ms
From 10.255.255.1 icmp_seq=6849 Destination Host Unreachable
From 10.255.255.1 icmp_seq=6850 Destination Host Unreachable
From 10.255.255.1 icmp_seq=6851 Destination Host Unreachable
64 bytes from 10.255.255.153: icmp_seq=6854 ttl=64 time=0.238 ms
From 10.255.255.1 icmp_seq=6880 Destination Host Unreachable
From 10.255.255.1 icmp_seq=6881 Destination Host Unreachable
From 10.255.255.1 icmp_seq=6882 Destination Host Unreachable
64 bytes from 10.255.255.153: icmp_seq=6899 ttl=64 time=0.326 ms
From 10.255.255.1 icmp_seq=6910 Destination Host Unreachable
From 10.255.255.1 icmp_seq=6911 Destination Host Unreachable

According to racktables the machine is connected to PDU-A-J11 and PDU-B-J11 which (again according to racktables) are both in the .mgmt.prg2.suse.org-domain both unreachable for me. I consider opening a SD ticket to at least reset both sockets or even request general access to these PDUs if possible.

I also want to explore the possibility if the ASM is somehow misbehaving caused by some weird network bug in the "management network" for the HMC because we observe general stability problems in this (rather simple) network.

Setting due date based on mean cycle time of SUSE QE Tools

I completely removed the machine from the HMC now. After doing a discovery scan I found a couple of IPs and found an ASM with 10.255.255.190 which is redcurrant but a different one then mentioned in https://progress.opensuse.org/issues/163529#note-24 - adding this one allowed for a super stable HMC connection again so we might had a duplicate connection again without the HMC showing the typical behavior (maybe this was "fixed"/changed by a minor HMC update conducted by me some months ago?). Anyhow, I enabled redcurrant-1 and tried network boot with the following parameters in SMS:

TFTP BOOT ---------------------------------------------------
Server IP.....................10.168.192.10
Client IP.....................10.145.10.222
Gateway IP....................10.145.10.254
Subnet Mask...................255.255.255.0
( 1  ) Filename.................ppc64le/grub2
TFTP Retries..................5
FINAL PACKET COUNT = 365 .....512 PACKET COUNT = 100

I cloned a recent job to validate how the LPAR behaves now:

# openqa-clone-job --skip-download --skip-chained-deps --within-instance https://openqa.suse.de 15403013 WORKER_CLASS=redcurrant-1 _GROUP=0 BUILD+=poo#163529
1 job has been created:
 - sle-15-SP7-Online-ppc64le-Build18.1-RAID0_test@ppc64le-hmc-4disk -> https://openqa.suse.de/tests/15427206

Currently the worker slot is waiting for a lower load on worker29 (which is controlling redcurrant-1).

https://openqa.suse.de/tests/15427206 booted fine and all worker slots are back in production. There are not more recent production jobs so if anyone encounters problems let us know.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: autoyast_create_hdd_textmode_qesec
https://openqa.suse.de/tests/15533637#step/bootloader_start/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

I did openqa-query-for-job-label poo#163529

will check

15537823|2024-09-26 09:59:10|done|failed|create_hdd_textmode_hmc_ntlm||worker29
15537542|2024-09-26 09:19:25|done|failed|create_hdd_textmode_hmc_ntlm||worker29
15537463|2024-09-26 08:39:43|done|failed|create_hdd_textmode_hmc_ntlm||worker29
15537350|2024-09-26 08:00:32|done|failed|create_hdd_textmode_hmc_ntlm||worker29
15537247|2024-09-26 07:20:22|done|failed|create_hdd_textmode_hmc_ntlm||worker29
15537112|2024-09-26 06:41:19|done|failed|create_hdd_textmode_hmc_ntlm||worker29
15536904|2024-09-26 06:00:30|done|failed|create_hdd_textmode_hmc_ntlm||worker29
15536404|2024-09-26 05:00:49|done|failed|create_hdd_textmode_hmc_ntlm||worker29
15533650|2024-09-26 02:43:17|done|failed|fips_install_lvm_full_encrypt||worker29
15533637|2024-09-26 02:16:57|done|failed|autoyast_create_hdd_textmode_qesec||worker29

On the first I added a comment "https://openqa.suse.de/tests/15537823#step/bootloader_start/29 shows that we ended up within a properly booted Linux system but fail later in the installer due to not reachable/readable installation repositories. This is not label:poo#163529 anymore" and removed the wrong taken over bugref. Equivalent for the other jobs.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: autoyast_create_hdd_gnome_qesec
https://openqa.suse.de/tests/15238123#step/bootloader_start/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

wrong reference removed

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: create_hdd_hmc_fips_ker_mode
https://openqa.suse.de/tests/15809850#step/bootloader/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Ran

workstation git/scripts ‹master› » ./openqa-query-for-job-label poo#163529
15809858|2024-10-30 10:57:32|done|failed|fips_install_lvm_full_encrypt||worker29
15809850|2024-10-30 10:38:56|done|failed|create_hdd_hmc_fips_ker_mode||worker29

and removed wrong references again.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: create_hdd_hmc_fips_env_mode
https://openqa.suse.de/tests/15927658#step/bootloader/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.