action #150920
closedopenqaworker-arm22 is unable to join download.opensuse.org in parallel tests = tap mode size:M
Added by ggardet_arm 6 months ago. Updated 5 months ago.
0%
Description
Observation¶
openQA test in scenario microos-Tumbleweed-DVD-aarch64-remote_ssh_controller@aarch64 fails in
await_install
Test suite description¶
Maintainer: jrivera Install remote server (parallel job) with ssh.
Reproducible¶
Fails since (at least) Build 20231114 (current job)
Expected result¶
Last good: 20231109 (or more recent)
Acceptance Criteria (should be implicit)¶
- AC1: above scenario passes consistently again
Suggestions¶
- Confirm that the test is working correctly i.e. it's not a test issue, not an issue with worker arm22
- Check the parallel_failed case and not just the dependent job
- Look further into what the scenario is testing
- Check whether other MM scenarios work (that also rely on external network connectivity)
Further details¶
Always latest result in this scenario: latest
Updated by okurz 6 months ago
- Copied to action #150944: Consider showing the investigation tab for "parallel_failed" as well added
Updated by livdywan 6 months ago
- Has duplicate action #150911: remote_{vnc,ssh}_controller: unable to refresh repo download.o.o added
Updated by okurz 6 months ago
- Related to action #150845: openqaworker-arm22 broken due to packages automatically removed size:M added
Updated by okurz 6 months ago
- Tags changed from infra, o3 to o3
- Status changed from In Progress to Workable
- Assignee deleted (
okurz)
https://openqa.opensuse.org/tests/3733404#step/await_install/148 reproduces the same problem so we can continue investigation there.
Updated by osukup 6 months ago · Edited
it looks more like general issue, x86_64 show same problem -> https://openqa.opensuse.org/tests/3742085#step/await_install/75 .. so its product or infra problem ? and if I look at history it started with 20231113, but in +- same time
Updated by nicksinger 6 months ago
- Related to action #151231: package loss between o3 machines and download.opensuse.org size:M added
Updated by ggardet_arm 6 months ago
Now, the problems are wider:
- More repos unreachables: http://codecs.opensuse.org and https://openqa.opensuse.org
- Worker name is localhost: https://openqa.opensuse.org/admin/workers/1190
This breaks more parallel tests. For example ovs-client/ovs-server parallel test succeeded yesterday, but broke today: https://openqa.opensuse.org/tests/3764570#next_previous
Updated by okurz 5 months ago
- Related to action #134123: Setup new PRG2 openQA worker for o3 - two new arm workers size:M added
Updated by mkittler 5 months ago · Edited
- Status changed from Workable to In Progress
- Assignee set to mkittler
I retried the job (https://openqa.opensuse.org/tests/3791292#step/setup_multimachine/64) to see whether it wasn't just a random networking issue as the SUT has an IP and the correct MTU value.
I'll check for other possible mistakes in the MM setup on arm22.
EDIT: It failed again (https://openqa.opensuse.org/tests/3792767). Looks like no gre tunnels are setup. Maybe this is the problem but on the other hand there's no other aarch64 MM worker so this should be ok. It also looks like the firewall is blocking the port for the developer mode. This should all be fixed by running the MM setup script so I'll try that.
Updated by openqa_review 5 months ago
- Due date set to 2023-12-23
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger 5 months ago · Edited
- Assignee changed from mkittler to nicksinger
as discussed in the daily I will continue here to cover/crosscheck AC1 of #134123
Updated by nicksinger 5 months ago
I executed the script to setup multi-machine on arm21 as well as arm22. On 21 there was some previous br1 config which the script couldn't handle which I solved by removing /etc/sysconfig/network/ifcfg-br1. I later realized that $1 is used in the script to generate the gre_tunnel_preup script so I finally used this command:
instance=32 ethernet=eth0 os-autoinst-setup-multi-machine /etc/wicked/scripts/gre_tunnel_preup.sh
also I adjusted the gre-script to manually add a mutual connection between both workers. After a reboot I tested 2 MM jobs but both failed to connect to external hosts because of DNS errors so I have to now figure out if
a) The setup inside VMs is borked (I can make use of @mkittler commands to spawn a VM to investigate further)
b) How to actually fix the setup and if/what we're missing inside our setup-script
Updated by okurz 5 months ago
Regarding VM to boot manually consider http://download.opensuse.org/ports/aarch64/tumbleweed/appliances/openSUSE-Tumbleweed-Minimal-VM.aarch64-kvm-and-xen.qcow2
Updated by nicksinger 5 months ago
got a VM booting and some graphical output via VNC (needs to be tunneled via ssh because of firewalld). The command I used is:
qemu-system-aarch64 -m 2048 -enable-kvm -netdev tap,id=qanet0,ifname=tap10,script=no,downscript=no -device virtio-net,netdev=qanet0,mac=52:54:00:13:0b:4a -vnc :41,share=force-shared -cpu host -smp cores=4,threads=4 -monitor stdio -machine virt,gic-version=host -bios /usr/share/qemu/qemu-uefi-aarch64.bin -device virtio-gpu-pci,edid=on,xres=1024,yres=768 openSUSE-MicroOS.aarch64-kvm-and-xen.qcow2
compared to https://open.qa/docs/#_start_test_vms_manually I had to find a couple of additional options to make it work on aarch64. Until now I cannot type anything via VNC. I need to figure out how to fix that
Updated by okurz 5 months ago
- Due date deleted (
2023-12-23) - Status changed from In Progress to Resolved
we are trying to debug openQA multi-machine tests on openqaworker-arm21+arm22 and struggle to come up with a manual qemu command line that makes the machine boot. I tried e.g. /usr/bin/qemu-system-aarch64 -device virtio-gpu-pci,edid=on,xres=1024,yres=768 -m 2048 -machine virt,gic-version=host -cpu host -mem-prealloc -mem-path /dev/hugepages/ -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:19:34:56 -enable-kvm -no-shutdown -vnc :91,share=force-shared but when connecting over VNC I only see that the guest has not initialized the display yet. Any ideas what's missing?
openqa-clone-job --parental-inheritance --within-instance https://openqa.opensuse.org/tests/3815206 PAUSE_AT=rescuesystem_validate_131 {BUILD,TEST}+=-poo150920 _GROUP=0
rescue_system@aarch64 -> https://openqa.opensuse.org/tests/3816230
and to crosscheck with warm21
openqa-clone-job --parental-inheritance --within-instance https://openqa.opensuse.org/tests/3815206 PAUSE_AT=rescuesystem_validate_131 {BUILD,TEST}+=-poo150920 _GROUP=0 WORKER_CLASS=openqaworker-arm21
- opensuse-Tumbleweed-DVD-aarch64-Build20231214-rescue_system@aarch64 -> https://openqa.opensuse.org/tests/3816231
Both tests start up just fine but the developer mode actually can not be connected over the webUI, for both of them. Crosschecked with another machine and https://openqa.opensuse.org/tests/3816232#live for openqaworker24 is just fine. Aso both tests are not related to multi-machine and actually don't configure network in the rescue mode those are not helping towards debugging network within the hosts but the question is why the hosts do not allow to reach the developer mode. Well, in the end mkittler fixed it. Likely wrong combinations of os-autoinst-setup-multi-machine and parameters were triggered.
So let's try developer mode and the only real multi-machine scenario on aarch64
openqa-clone-job --parental-inheritance --within-instance https://openqa.opensuse.org/tests/3815268 PAUSE_AT=libzypp_config {BUILD,TEST}+=-poo150920 _GROUP=0 WORKER_CLASS=openqaworker-arm21
-> https://openqa.opensuse.org/tests/3816247
but I didn't follow up on the same day so eventually the job ran into timeout as expected.
In the end mkittler fixed the config at least on warm21 with
firewall-cmd --zone public --remove-interface=eth0
firewall-cmd --zone trusted --add-interface=eth0
I assume that nicksinger executed os-autoinst-setup-multi-machine but did not see (or ignore) error messages about the inability to add eth0 to trusted as it was already in public according to #150920-22 ?
Anyway now the developer mode works fine and so far also I assume multi-machine tests on warm21 so I added back "tap" to warm21:/etc/openqa/workers.ini and I am monitoring jobs.
Due to the new worker class there was a sudden rise of jobs and also jobs ending up incomplete with "Reason: cache failure: Cache service queue already full (5)" which is unfortunate. I now increased the cache size which is at least slightly related and might help. On both warm21+warm22 we have a 6TB NVMe from which we run most mount points including /var/lib/openqa/cache with enough free space so I set
# Limit size of CACHEDIRECTORY to the specified value in GiB (50 GiB by default)
CACHELIMIT = 1000
Multiple multi-machine tests now passed including
- https://openqa.opensuse.org/tests/3818709 remote_ssh_controller
- https://openqa.opensuse.org/tests/3818743 ovs-server
- https://openqa.opensuse.org/tests/3818735 security_389ds_sssd_client
- https://openqa.opensuse.org/tests/3818721 rsync-client
so I assume it's ok if we keep tap. Now to warm22.
openqa-clone-job --parental-inheritance --within-instance https://openqa.opensuse.org/tests/3815268 {BUILD,TEST}+=-poo150920 _GROUP=0 WORKER_CLASS=openqaworker-arm22
openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.opensuse.org/tests/3818743 {BUILD,TEST}+=-poo150920 _GROUP=0 WORKER_CLASS=openqaworker-arm22
- microos-Tumbleweed-DVD-aarch64-Build20231214-remote_ssh_controller@aarch64 -> https://openqa.opensuse.org/tests/3818773
- microos-Tumbleweed-DVD-aarch64-Build20231214-remote_ssh_target@aarch64 -> https://openqa.opensuse.org/tests/3818772
- opensuse-Tumbleweed-DVD-aarch64-Build20231215-ovs-server@aarch64 -> https://openqa.opensuse.org/tests/3818777
- opensuse-Tumbleweed-DVD-aarch64-Build20231215-ovs-client@aarch64 -> https://openqa.opensuse.org/tests/3818778
yep, also fine.
So let's check across both warm21+warm22:
openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.opensuse.org/tests/3818743 {BUILD,TEST}+=-poo150920 _GROUP=0 WORKER_CLASS:ovs-server=openqaworker-arm22 WORKER_CLASS:ovs-client=openqaworker-arm21
- opensuse-Tumbleweed-DVD-aarch64-Build20231215-ovs-server@aarch64 -> https://openqa.opensuse.org/tests/3818913
- opensuse-Tumbleweed-DVD-aarch64-Build20231215-ovs-client@aarch64 -> https://openqa.opensuse.org/tests/3818912
Both successful so enabled "tap" for production on openqaworker-arm22 again as well. With that we can resolve here.
Updated by jbaier_cz 3 months ago
- Related to action #155278: o3 aarch64 multi-machine tests on openqaworker-arm21 and 22 fail to resolve codecs.opensuse.org size:M added
Updated by okurz 2 months ago
- Related to action #157414: Network broken with multimachine on multiple workers (broken packet forwarding / NAT) size:M added