QA » openQA Project » openQA Infrastructure

livdywan wrote in #note-6:

We looked at it in the infra daily. It was estimated before the most recent discoveries and we should re-estimate it

With what we learned recently I suggest to not connect this machine to other o3 workers in PRG2 over GRE tunnels and only enable tap on specific aarch32 worker classes to not interfere with aarch64 multi-machine jobs which should run solely in PRG2.

I added PARALLEL_ONE_HOST_ONLY=1 to relevant machine definitions (those also defining WORKER_CLASS=qemu_aarch32). So far there is only one other aarch32-capable host (oss-cobbler-03) so this won't make things worse for scheduling jobs on existing worker slots.

There are no existing aarch32 jobs in the queue so I suppose I can move on with the tap setup itself.

EDIT: Ran the setup script and also took care of the wicked config of the eth interface (which is currently not covered by the script; the zone was left public). After a reboot everything seemed to be still in place so I scheduled test jobs:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org/tests/4094320 WORKER_CLASS=aarch64-o3 {BUILD,TEST}+=-poo150869
Cloning parents of opensuse-Tumbleweed-NET-arm-Build20240418-wicked_basic_ref@aarch32
Cloning children of opensuse-Tumbleweed-NET-arm-Build20240418-wicked_basic_ref@aarch32
Cloning parents of opensuse-Tumbleweed-NET-arm-Build20240418-wicked_basic_sut@aarch32
2 jobs have been created:
 - opensuse-Tumbleweed-NET-arm-Build20240418-wicked_basic_ref@aarch32 -> https://openqa.opensuse.org/tests/4101001
 - opensuse-Tumbleweed-NET-arm-Build20240418-wicked_basic_sut@aarch32 -> https://openqa.opensuse.org/tests/4101000

EDIT: The test jobs passed. I'll enable the worker class tomorrow.

I now enabled the tap worker class on the worker so production jobs can run. This is the first time we use the PARALLEL_ONE_HOST_ONLY=1 feature in production so I'll have a look at https://openqa.opensuse.org/tests/latest?arch=arm&distri=opensuse&flavor=NET&machine=aarch32&test=wicked_basic_ref&version=Tumbleweed and https://openqa.opensuse.org/tests/latest?arch=arm&distri=opensuse&flavor=NET&machine=aarch32&test=yast2_nfs_v3_server&version=Tumbleweed and https://openqa.opensuse.org/tests/latest?arch=arm&distri=opensuse&flavor=NET&machine=aarch32&test=yast2_nfs_v4_server&version=Tumbleweed after the next build. (Those seem to be the only MM scenarios we schedule for aarch32.)

Maybe the config update broke something, since some tests are failing due to network issue - see https://progress.opensuse.org/issues/159558

Just for the record, my work on this ticket was not the cause of #159558.

It doesn't look like any of the jobs of the last build were scheduled on aarch64-o3. I'll wait for the next build. (I have already created test jobs and they worked so we might also just consider this ticket resolved.)

https://openqa.opensuse.org/tests/4106737 fails with:

[2024-04-25T11:16:46.532901+02:00] [info] [pid:46959] ::: backend::baseclass::die_handler: Backend process died, backend errors are reported below in the following lines:
  Open vSwitch command 'set_vlan' with arguments 'tap13 5' failed: 'tap13' is not connected to bridge 'br1' at /usr/lib/os-autoinst/backend/qemu.pm line 127.

I've just reverted adding the tap worker class. I noticed myself that this isn't going to cut it because also 64-bit tap jobs might now run on aarch64-o3 - which of course breaks as @ggardet_arm wrote in the previous comment.

Not sure how to allow scheduling those jobs then. We would probably need #158146. Or we add PARALLEL_ONE_HOST_ONLY=1 also 64-bit arm worker classes.

The machine is up and running again. The tap setup seems still good but before enabling the tap worker class I'll run some test jobs:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org/tests/4311446 WORKER_CLASS=aarch64-o3 {BUILD,TEST}+=-poo150869
Cloning parents of opensuse-Tumbleweed-DVD-aarch64-Build20240701-wicked_basic_ref@aarch64
Cloning children of opensuse-Tumbleweed-DVD-aarch64-Build20240701-wicked_basic_ref@aarch64
Cloning parents of opensuse-Tumbleweed-DVD-aarch64-Build20240701-wicked_basic_sut@aarch64
2 jobs have been created:
 - opensuse-Tumbleweed-DVD-aarch64-Build20240701-wicked_basic_ref@aarch64 -> https://openqa.opensuse.org/tests/4315667
 - opensuse-Tumbleweed-DVD-aarch64-Build20240701-wicked_basic_sut@aarch64 -> https://openqa.opensuse.org/tests/4315666

The test jobs passed and also other production jobs that ran on aarch64-o3 seems to generally run as expected. So I enabled the tap worker class on the aarch64-o3 slots and mentioned it in the openQA chat on Matrix.

So far only one production job ran on the worker¹ but it failed on an unrelated typing issue (https://openqa.opensuse.org/tests/4318525#step/remote_controller/19).

¹

with mm_jobs as (select distinct id, result, state, t_finished, (select host from workers where id = assigned_worker_id) as worker_host from jobs left join job_dependencies on (id = child_job_id or id = parent_job_id) where dependency = 2) select concat('https://openqa.opensuse.org/tests/', id) as url, result, t_finished, worker_host from mm_jobs where state = 'done' and worker_host in ('aarch64-o3') order by id desc limit 50;