QA » openQA Project » openQA Tests

The "URL shown cut off" thingy is probably not the root cause - this is just because of character limitation there.

most likely it is/was a temporary network issue:

Sending DHCP request to enc1...
no/incomplete answer.

I've restarted all jobs failing in bootloader_zkvm and will monitor them

it's not about the "cut off FTP URL" this is just a displaying issue
The reason for this is a small time frame when jobs are starting in parallel could cause duplicate IP Addresses because of the network rewrite we do in qcow images on s390 kvm

right now the workaround for this is to restart the job

Another occurence:
sle-15-Installer-DVD-s390x-Build589.1-textmode+role_textmode@s390x-kvm-sle12

That seems to be becoming more annyoing:

• https://openqa.suse.de/tests/1656731/file/serial0.txt
• https://openqa.suse.de/tests/1656787/file/serial0.txt

Can we put in a check that the network was restarted without any error output and fail otherwise?

I am not sure if the following shows the same problem but it looked interesting at least. I triggered 10 jobs with:

$ for i in {1..10} ; do openqa_clone_job_osd --skip-chained-deps 1668775 TEST=okurz_bsc#1091186_$i BUILD=598.1:bsc1091186 _GROUP="Test Development: SLE 15" ; done

Results on
https://openqa.suse.de/tests/overview?version=15&build=598.1%3Absc1091186&distri=sle&groupid=96

and soon after the triggering 4 were running, four failed in "boot_to_desktop". This looks related.

Suggestions:

• Crosscheck the IP-addresses assigned to each machine
• Crosscheck if there is output about duplicate IP adress or not
• Crosscheck timestamps

okurz wrote:

I am not sure if the following shows the same problem but it looked interesting at least. I triggered 10 jobs with:

$ for i in {1..10} ; do openqa_clone_job_osd --skip-chained-deps 1668775 TEST=okurz_bsc#1091186_$i BUILD=598.1:bsc1091186 _GROUP="Test Development: SLE 15" ; done

Results on
https://openqa.suse.de/tests/overview?version=15&build=598.1%3Absc1091186&distri=sle&groupid=96

and soon after the triggering 4 were running, four failed in "boot_to_desktop". This looks related.

Suggestions:

• Crosscheck the IP-addresses assigned to each machine
• Crosscheck if there is output about duplicate IP adress or not
• Crosscheck timestamps

3/4 are failing due to duplicate IP-Addresses
I'm currently working on a list of IP-Assignments for each and every worker and the IPs which are used by the tests
so suggesting to carry over to next Sprint

mgriessmeier wrote:

I'm currently working on a list of IP-Assignments for each and every worker and the IPs which are used by the tests

sounds like you are implementing a DHCP server ;)

Duplicate address issue over here as well. https://openqa.suse.de/tests/1682086/file/autoinst-log.txt
Interesting that it did not fail in boot_to_desktop or bootloader_zkvm but in kdump_and_crash during reboot
https://openqa.suse.de/tests/1682086#step/kdump_and_crash/35

mloviska wrote:

Duplicate address issue over here as well. https://openqa.suse.de/tests/1682086/file/autoinst-log.txt

but that message appeared during boot_to_desktop, weird that it could proceed...

Interesting that it did not fail in boot_to_desktop or bootloader_zkvm but in kdump_and_crash during reboot
https://openqa.suse.de/tests/1682086#step/kdump_and_crash/35

that seems to be a different issue to be honest

We could build a reproduce setup on s390p8. Our approach to circumvent this is following:

1. Boot the image which was created previously by a different job
2. Break in GRUB2 and add systemd.unit=rescue.target to the kernel cmdline
   • This ensures wicked does not start before we (re)configure the network interface configuration
   • Using systemd.mask=wicked did not work for us since we could not unmask the server after we're done re-configuring
3. Edit /etc/sysconfig/network/ifcfg-* to change the desired values, save, close
4. Exit and continue the boot by typing exit into the systemd rescue shell

These steps ensure that the system never acquires its configured IPs when it boots up for a second time (since we can't ensure that the initial creation-worker hasn't already started another installation with the same IP).

That sounds like we are not really testing the normal boot flow anymore. I don't understand. Can we not just use a proper DHCP setup?

okurz wrote:

Can we not just use a proper DHCP setup?

needs to be clarified with SUSE-IT

mgriessmeier wrote:

okurz wrote:

Can we not just use a proper DHCP setup?

needs to be clarified with SUSE-IT

SUSE-IT was on a training in the last two days - I will push this forward today

opened infra ticket: https://infra.nue.suse.com/Ticket/Display.html?id=113739&results=0862777de3082b5a89e00b7b04bf54f7

Next steps to take:

• test if changing "ifcfg=*=" to dhcp works
• adapt backend to write the mac address which is assigned to the worker into the xml file when generating the guest vm
• adapt workerconf:
  • remove static IP assignment
  • add MAC address for each worker

Mac address mapping as follows:

zKVM // s390pb

s390kvm003 - 10.161.145.3 - 52:54:00:9d:f3:02
s390kvm004 - 10.161.145.4 - 52:54:00:f0:7c:91
s390kvm005 - 10.161.145.5 - 52:54:00:67:03:b0


SUSE KVM // s390p8

s390kvm013 - 10.161.145.13 - 52:54:00:9f:6b:f6
s390kvm014 - 10.161.145.14 - 52:54:00:d3:26:39
s390kvm015 - 10.161.145.15 - 52:54:00:14:ee:2e
s390kvm016 - 10.161.145.16 - 52:54:00:2f:2b:94

PR created: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5150

needs testing for multiple scenarios especially upgrades and extratests/create_hdd

to add: "SetHostname=0" to Machine definitions

Unfortunately I've spotted again the black screens in boot_to_desktop when I was trying to verify my PR on Upgrade tests...
so this needs still a bit more investigation...
also the black screens (e.g. ssh to SUT not possible) appeared more frequently now on kernel tests... (https://progress.opensuse.org/issues/36745)

soo... the PR is pretty much done,
all relevant scenarios have been tested with it

but.... All the sp3 s390x qcows needs to be either created newly with the dhcp config or by manually mounting them and modifying the corresponding file with e.g.

for i in $(ls *.qcow); do qemu-nbd -c /dev/nbd0 $i; mount /dev/nbdp02 /mnt/; cp $modified_ifcfg /mnt/etc/sysconfig/network/ifcfg-eth0; umount /mnt/; done

That's the reason why I will not finish this today, because "never deploy on friday" ;)