QA » openQA Project

We have two different issues here (which you normally shouldn't mix):

1. Tests using the IPMI backend are failing very soon: https://openqa.suse.de/tests/10728059#step/boot_from_pxe/11
   • Not sure yet why that is.
2. Tests failing to upload logs later on: https://openqa.suse.de/tests/10728055#step/update_package/8
   • Previous uploads work. It looks like the SUT might not be ready at the point the upload is attempted (although the SSH connection to it could be established).

Regarding openqaipmi5.qa.suse.de: It looks like https://openqa.suse.de/tests/10731111 is running now. Also previous jobs that have already finished seemed to get past the setup like https://openqa.suse.de/tests/10730149. That job then ended up running into the second issue (uploading doesn't work after reboot). Not sure whether we need to do anything regarding the setup issue considering it seems to work again.

And the uploading issue really looks like a network issue within the SUT itself, especially since the test actually was able to upload some other logs.

One important point to mention is that openqaipmi5 is located in NUE1-SRV2 so it is not just a simple problem related to the move to FC Basement

I say it is not about our setup because:

1. Other uploads as part of that job worked. So the firewall is generally not blocking traffic.
2. It happens shortly after rebooting the SUT indicating hostname resolution is just not ready at this point.

So can you check the state of the SUT, e.g. using the developer mode?

My first guess would be that one of the switches in the server room is running in hub mode. Most likely the one that IPMI SUTs are plugged into. That'll cause serious performance issues on busy networks.

mkittler wrote:

I say it is not about our setup because:

1. Other uploads as part of that job worked. So the firewall is generally not blocking traffic.
2. It happens shortly after rebooting the SUT indicating hostname resolution is just not ready at this point.

So can you check the state of the SUT, e.g. using the developer mode?

• What developer mode can do for this case ? I can not figure it out.

• And which case supports this judgement ?

  It happens shortly after rebooting the SUT indicating hostname resolution is just not ready at this point.
  

• I have been keeping an eye on test runs. Still can not have a clue.

What developer mode can do for this case ? I can not figure it out.

You could make the test pause at host_upgrade_generate_run_file and investigate the networking problems on the SUT manually.

And which case supports this judgement?

If it was a firewall issue or the command server of os-autoinst would be broken then it would likely also affect other uploads. However, other uploads (e.g. in the previous test module update_package) work fine (e.g. https://openqa.suse.de/tests/10730149#step/update_package/63). So since a general problem with the setup is unlikely this speaks for some issue within the SUT (after rebooting it, considering it is being rebooted just in the test module reboot_and_wait_up_normal before the failure). Maybe it is the simple case of hostname resolution simply not being ready or there's some other misconfiguration. (This is mainly about the case https://openqa.suse.de/tests/10730149#step/host_upgrade_generate_run_file/4. I've seen other types of failures as well. They also look like something's broken on the test side, though.)

mkittler wrote:

What developer mode can do for this case ? I can not figure it out.

You could make the test pause at host_upgrade_generate_run_file and investigate the networking problems on the SUT manually.

And which case supports this judgement?

If it was a firewall issue or the command server of os-autoinst would be broken then it would likely also affect other uploads. However, other uploads (e.g. in the previous test module update_package) work fine (e.g. https://openqa.suse.de/tests/10730149#step/update_package/63). So since a general problem with the setup is unlikely this speaks for some issue within the SUT (after rebooting it, considering it is being rebooted just in the test module reboot_and_wait_up_normal before the failure). Maybe it is the simple case of hostname resolution simply not being ready or there's some other misconfiguration. (This is mainly about the case https://openqa.suse.de/tests/10730149#step/host_upgrade_generate_run_file/4. I've seen other types of failures as well. They also look like something's broken on the test side, though.)

• I will do as you instruct
• I think we'd better use type_command => 1 with script_output API in our tests to avoid downloading it from worker machine to decrease failure.

And can not establish sol session to some machines at the moment:
waynechen:~ # ipmitool -I lanplus -C 3 -H sp.gonzo.qa.suse.de xxxxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session
waynechen:~ # ipmitool -I lanplus -C 3 -H sp.kermit.qa.suse.de xxxxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session
waynechen:~ # ipmitool -I lanplus -C 3 -H amd-zen3-gpu-sut1-sp.qa.suse.de xxxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session
waynechen:~ # ipmitool -I lanplus -C 3 -H sp.scooter.qa.suse.de xxxxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session

waynechen55 wrote:

And can not establish sol session to some machines at the moment:
waynechen:~ # ipmitool -I lanplus -C 3 -H sp.gonzo.qa.suse.de xxxxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session
waynechen:~ # ipmitool -I lanplus -C 3 -H sp.kermit.qa.suse.de xxxxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session
waynechen:~ # ipmitool -I lanplus -C 3 -H amd-zen3-gpu-sut1-sp.qa.suse.de xxxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session
waynechen:~ # ipmitool -I lanplus -C 3 -H sp.scooter.qa.suse.de xxxxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session

There have been reports of general network issues at the moment in Frankencampus office (that's also where those machines are)
I cannot ping any host in this network, but it says "Destination unreachable" - which supports my assumption that som ethings wrong with the whole network.

works again (for all but amd-zen3...)

matthi@paramore:~(:|✔) # for i in gonzo kermit scooter; do ipmitool -I lanplus -C 3 -H sp.$i.qa.suse.de -U xxx-P xxx chassis power status; done                  
Chassis Power is on
Chassis Power is on
Chassis Power is on
matthi@paramore:~(:|✔) # ipmitool -I lanplus -C 3 -H amd-zen3-gpu-sut1-sp.qa.suse.de -U xxx -P xxx chassis power status
Error: Unable to establish IPMI v2 / RMCP+ session

@mkittler

I did some investigation on three machines, scooter, ix64ph1087 and amd-zen3 with the same test suite. You can refer to this one https://openqa.suse.de/tests/10797074#step/host_upgrade_step2_run/14.

• The "Can not resolve host grenache-1.qa.suse.de" issue still happens to scooter. For the error you can refer to https://openqa.suse.de/tests/10797074#step/host_upgrade_step2_run/14.
• It passed host_upgrade_generate_run_file step, because it uses type_command => 1 in here. But it still failed in the next step host_upgrade_step2_run which needs to upload log again, so test run failed and scooter was powered off.
• In steps host_upgrade_generate_run_file and host_upgrade_step2_run, I also checked manually and I can not ping grenache-1 and openqa.suse.de from inside scooter.
• Next I powered scooter on again, and login to it manually. I still can not ping grenache-1 and openqa.suse.de form inside scooter.
• At last, I found its /etc/resolv.conf is empty as below:

### /etc/resolv.conf file autogenerated by netconfig!
#
# Before you change this file manually, consider to define the
# static DNS configuration using the following variables in the
# /etc/sysconfig/network/config file:
#     NETCONFIG_DNS_STATIC_SEARCHLIST
#     NETCONFIG_DNS_STATIC_SERVERS
#     NETCONFIG_DNS_FORWARDER
# or disable DNS configuration updates via netconfig by setting:
#     NETCONFIG_DNS_POLICY=''
#
# See also the netconfig(8) manual page and other documentation.
#
# Note: Manual change of this file disables netconfig too, but
# may get lost when this file contains comments or empty lines
# only, the netconfig settings are same with settings in this
# file and in case of a "netconfig update -f" call.
#
### Please remove (at least) this line when you modify the file!
search qe.nue2.suse.org nue2.suse.org suse.de arch.suse.de nue.suse.com suse.cz

• After adding some nameserver entries into this file, I can ping grenache-1 and openqa.suse.de successfully.
• So I think /etc/resolv.conf was emptied after reboot and it can not be populated successfully again.

• Next I powered scooter on again, and login to it manually. I still can not ping grenache-1 and openqa.suse.de form inside scooter.
• At last, I found its /etc/resolv.conf is empty as below:

### /etc/resolv.conf file autogenerated by netconfig!
#
# Before you change this file manually, consider to define the
# static DNS configuration using the following variables in the
# /etc/sysconfig/network/config file:
#     NETCONFIG_DNS_STATIC_SEARCHLIST
#     NETCONFIG_DNS_STATIC_SERVERS
#     NETCONFIG_DNS_FORWARDER
# or disable DNS configuration updates via netconfig by setting:
#     NETCONFIG_DNS_POLICY=''
#
# See also the netconfig(8) manual page and other documentation.
#
# Note: Manual change of this file disables netconfig too, but
# may get lost when this file contains comments or empty lines
# only, the netconfig settings are same with settings in this
# file and in case of a "netconfig update -f" call.
#
### Please remove (at least) this line when you modify the file!
search qe.nue2.suse.org nue2.suse.org suse.de arch.suse.de nue.suse.com suse.cz

• After adding some nameserver entries into this file, I can ping grenache-1 and openqa.suse.de successfully.
• So I think /etc/resolv.conf was emptied after reboot and it can not be populated successfully again.

Then it looks like sometimes SUT cannot get correct network dns setup from dhcp server? Can we check the dhcp server to see if anything can block it? Thanks a lot!

okurz wrote:

@waynechen55 @cachen We discussed this in our daily infra call 2023-03-30. As far as we can see the DHCP server within FC Basement works as expected. We do not currently have administrative access to the DHCP server. This is planned to be worked on in #125450 and the according SD-tickets linked in there (internal reference for myself: Specifically https://sd.suse.com/servicedesk/customer/portal/1/SD-113959). As explained by 126188 e.g. in #126188#note-10 we assume the problem is within the SUT and the test design where the SUT might not have consistent access to the network yet immediately after bootup.

We suggest you look into that from test perspective and either debug what is visible in system journals of the bootup process or adjust the test code to ensure the test execution only continues after the network initialization is finished.

With that I am reducing prio to High as we feel we did what we could provide from our side so far. Keeping mkittler assigned to support you in the investigation process and answer questions as needed if any.

Understood the challenge if you don't have access permission to dhcp server. Thank you

okurz wrote:

@waynechen55 @cachen We discussed this in our daily infra call 2023-03-30. As far as we can see the DHCP server within FC Basement works as expected. We do not currently have administrative access to the DHCP server. This is planned to be worked on in #125450 and the according SD-tickets linked in there (internal reference for myself: Specifically https://sd.suse.com/servicedesk/customer/portal/1/SD-113959). As explained by mkittler e.g. in #126188#note-10 we assume the problem is within the SUT and the test design where the SUT might not have consistent access to the network yet immediately after bootup.

We suggest you look into that from test perspective and either debug what is visible in system journals of the bootup process or adjust the test code to ensure the test execution only continues after the network initialization is finished.

With that I am reducing prio to High as we feel we did what we could provide from our side so far. Keeping mkittler assigned to support you in the investigation process and answer questions as needed if any.

Understood the challenge if you don't have access permission to dhcp server. Thank you, @okurz @mkittler and @nicksinger for taking care those machines infra stuff and network issues.
I tracked some test jobs, the disconnection between SUT and worker grenache-1.qa.suse.de not reproduce every time. Before tools team can dig into the dhcp/dns settings, just like Oliver suggested, @waynechen55 @xlai let's see if any workaround solution can be placed in test, e.g. add a step to check network setup and correct the nameserver if detecte the missing.

Again, thank you all for the co-work on those challenge tickets, many appreciated!

cachen wrote:

okurz wrote:

@waynechen55 @cachen We discussed this in our daily infra call 2023-03-30. As far as we can see the DHCP server within FC Basement works as expected. We do not currently have administrative access to the DHCP server. This is planned to be worked on in #125450 and the according SD-tickets linked in there (internal reference for myself: Specifically https://sd.suse.com/servicedesk/customer/portal/1/SD-113959). As explained by 126188 e.g. in #126188#note-10 we assume the problem is within the SUT and the test design where the SUT might not have consistent access to the network yet immediately after bootup.

We suggest you look into that from test perspective and either debug what is visible in system journals of the bootup process or adjust the test code to ensure the test execution only continues after the network initialization is finished.

With that I am reducing prio to High as we feel we did what we could provide from our side so far. Keeping mkittler assigned to support you in the investigation process and answer questions as needed if any.

Understood the challenge if you don't have access permission to dhcp server. Thank you

okurz wrote:

@waynechen55 @cachen We discussed this in our daily infra call 2023-03-30. As far as we can see the DHCP server within FC Basement works as expected. We do not currently have administrative access to the DHCP server. This is planned to be worked on in #125450 and the according SD-tickets linked in there (internal reference for myself: Specifically https://sd.suse.com/servicedesk/customer/portal/1/SD-113959). As explained by mkittler e.g. in #126188#note-10 we assume the problem is within the SUT and the test design where the SUT might not have consistent access to the network yet immediately after bootup.

We suggest you look into that from test perspective and either debug what is visible in system journals of the bootup process or adjust the test code to ensure the test execution only continues after the network initialization is finished.

With that I am reducing prio to High as we feel we did what we could provide from our side so far. Keeping mkittler assigned to support you in the investigation process and answer questions as needed if any.

Understood the challenge if you don't have access permission to dhcp server. Thank you, @okurz @mkittler and @nicksinger for taking care those machines infra stuff and network issues.
I tracked some test jobs, the disconnection between SUT and worker grenache-1.qa.suse.de not reproduce every time. Before tools team can dig into the dhcp/dns settings, just like Oliver suggested, @waynechen55 @xlai let's see if any workaround solution can be placed in test, e.g. add a step to check network setup and correct the nameserver if detecte the missing.

Again, thank you all for the co-work on those challenge tickets, many appreciated!

• I was considering this yesterday, maybe "netconfig update -f" will do the trick. But this only works if failure is not caused by network breakdown or failure is only caused by temporary dhcp glitch. If there is severe communication issue, nothing will help.
• I think the issue really worth further investigation on infra side. Although it can not be reproduced every time, it happens frequently. The latest Build88.1 test run hit the issue again, please refer to example1 and example2.

waynechen55 wrote:

• I was considering this yesterday, maybe "netconfig update -f" will do the trick. But this only works if failure is not caused by network breakdown or failure is only caused by temporary dhcp glitch. If there is severe communication issue, nothing will help.

I still doubt there is a breakdown on DHCP server side. It might behave differently and hence making such issue more apparent but I still assume it's a problem that tests try to continue too fast when the SUT is not yet ready.

• I think the issue really worth further investigation on infra side. Although it can not be reproduced every time, it happens frequently.

We would look into this issue as soon as Eng-Infra gives us access. Based on previous experiences I assume this will take more months (!). If the issue is happening frequently then I strongly recommend you follow up from test side

Considering what we've already found out the new ticket #127256 might be the same. It is very explicit about what's not working (nameserver missing from DHCP response) and in the test failures reported here have a matching sympthom. Unfortunately, this doesn't change what @okurz wrote in the last paragraph of the previous comment.

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: virt-guest-migration-developing-from-developing-to-developing-kvm-dst@virt-mm-64bit-ipmi
https://openqa.suse.de/tests/10933402#step/guest_migration_dst/1

To prevent further reminder comments one of the following options should be followed:

1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
2. The openQA job group is moved to "Released" or "EOL" (End-of-Life)
3. The bugref in the openQA scenario is removed or replaced, e.g. label:wontfix:boo1234

Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.

Related branch in progress: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/16774

https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3456 was merged and is deployed to both our DHCP servers walter1.qe.nue2.suse.org and walter2.qe.nue2.suse.org . We assume this fixes the problem.

It would be nice if you could confirm that it works now.

mkittler wrote:

I've checked the scenarios from the ticket description. Some tests pass now. I haven't seen any obvious failures due to name resolution in the recent history. Since I haven't seen the problem anymore on openqaworker1 as well I suppose it can be considered resolved.

Thanks for the fix! We will also follow the results. If reproduced again, will share info here and reopen the ticket.