QA » openQA Project » openQA Infrastructure

just to not forget: in theory o3 setup is as far progressed as o.s.d, just the last basic test to ping the LPAR s390zl11 was not pingable from o.o.o which would need to be investigated by SUSE IT

According to mgriessmeier in the o3 network the s390x mainframe zVM interface should be reachable over 10.150.1.41.

mgriessmeier will ask gschlotter to double check the vswitch config related to that IP and if that should be reachable from o3 (10.150.1.11/24). At best find out the L2 address accordingly as well.

update:

• connection from ariel (and the whole 10.150.1.x range) to s390zl11 (10.150.1.41) is working now
• two o3 guests have been created
• modified existing worker on o3 (openqaworker23:1)
• started first job https://openqa.opensuse.org/tests/3731839#live with openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org/tests/3588513 WORKER_CLASS=s390x-zVM_poo137408 _GROUP=0 BUILD=poo137408 TEST+=-poo137408

it used the old MACHINE settings since it was a clone jobs, so I triggered another one while explicitly overwriting the MACHINE settings:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org/tests/3588513 WORKER_CLASS=s390x-zVM_poo137408 _GROUP=0 BUILD=poo137408 TEST+=-poo137408 REPO_HOST=10.150.1.11 S390_NETWORK_PARAMS="OSAMedium=eth OSAInterface=qdio InstNetDev=osa HostIP=10.150.1.150 Hostname=o3zvm001 Gateway=10.150.1.254 Nameserver=10.151.53.53 Domain=openqanet.opensuse.org PortNo= Layer2=0 ReadChannel=0.0.0A00 WriteChannel=0.0.0A01 DataChannel=0.0.0A02 OSAHWAddr="

-> https://openqa.opensuse.org/tests/3731858

we had some issues with apparmor which caused icewm (containing the xterm-console) to fail. Apparently the apparmor-profile package was missing which we fixed by just installing the apparmor pattern on worker23

the core infrastructure part is done, mainframe is able to start the installer -> https://openqa.opensuse.org/tests/3733312

This still shows an error which is most likely caused by slow network (or less likely, DNS/firewall issue, though the service is reachable from the worker) and would need more investigation

still to do:

• currently there is only one instance hardcoded as the old approach with substitution of @S390_HOST@ does not work anymore, so refactoring in the same way as on o.s.d is needed - see https://progress.opensuse.org/issues/132152#note-62 for details
• trigger enough verification runs to get statistical stability
• adapt MACHINE definition accordingly after the refactoring mentioned above

created 50 jobs with

for i in {1..50}; do openqa-clone-job --skip-chained-deps --within-instance https://openqa.opensuse.org/tests/3718373 WORKER_CLASS=s390x-zVM_poo137408 _GROUP=0 BUILD=poo137408 TEST+=-poo137408-$i REPO_HOST=10.150.1.11 S390_NETWORK_PARAMS="OSAMedium=eth OSAInterface=qdio InstNetDev=osa HostIP=10.150.1.150/24 Hostname=o3zvm001.openqanet.opensuse.org Gateway=10.150.1.254 Nameserver=10.151.1.11 Domain=openqanet.opensuse.org PortNo= Layer2=0 ReadChannel=0.0.0A00 WriteChannel=0.0.0A01 DataChannel=0.0.0A02 OSAHWAddr="; done

-> https://openqa.opensuse.org/tests/overview?build=poo137408&version=Tumbleweed&distri=opensuse

nicksinger wrote in #note-12:

we had some issues with apparmor which caused icewm (containing the xterm-console) to fail. Apparently the apparmor-profile package was missing which we fixed by just installing the apparmor pattern on worker23

hmm seems like our temporary changes to apparmor were reverted somehow/by someone -> https://openqa.opensuse.org/tests/3735119#step/bootloader_s390/33
as I understood this is not acceptable anyway and the worker should go either in a container or on a seperate host, please move it accordingly :)

However, the infrastructure to run the tests is in place, there seems to be an issue with ntp which needs to be investigated further, I assume it's a timeout but haven't verified it -> https://openqa.opensuse.org/tests/3733329#step/partitioning_finish/2

besides that, next steps would be:

• schedule a bunch of verification jobs after the worker changes have been made (ideally over the weekend)
• apply the same refactoring as to be done for the o.s.d. workers in #132152

Regarding s390 in containers please see https://progress.opensuse.org/projects/openqav3/wiki/Wiki#o3-s390-workers . As we discussed we see the main infrastructure network configuration in good shape. There might be specific individual network services still blocked. This would need to be solved on a case by case based on openQA test results.

mgriessmeier wrote in #note-14:

However, the infrastructure to run the tests is in place, there seems to be an issue with ntp which needs to be investigated further, I assume it's a timeout but haven't verified it -> https://openqa.opensuse.org/tests/3733329#step/partitioning_finish/2

For that I suggest to use the interactive mode from openQA and pause at the partitioning module and look if the connection can be established at all. I suspect firewall blocking the communication.

okurz wrote in #note-15:

Regarding s390 in containers please see https://progress.opensuse.org/projects/openqav3/wiki/Wiki#o3-s390-workers . As we discussed we see the main infrastructure network configuration in good shape. There might be specific individual network services still blocked. This would need to be solved on a case by case based on openQA test results.

Installed openqaworker23_container_101 in openqaworker23 with the following steps:

• created directory /opt/s390x_opensuse
• copied /etc/openqa/client.conf and /etc/openqa/workers.ini from openqaworker23 itself

• chmod 644 /opt/s390x_opensuse/client.conf in order to be readable by the container

• modify /opt/s390x_opensuse/workers.ini:

[global]
HOST = https://openqa.opensuse.org
WORKER_HOSTNAME = openqaworker23
CACHEDIRECTORY = /var/lib/openqa/cache
CACHESERVICEURL=http://10.150.1.26:9530/
CACHELIMIT = 400


[101]
WORKER_CLASS = s390x-zVM_poo137408
s390_HOST = '001'
ZVM_GUEST = o3zvm001
ZVM_HOST = 10.150.1.41
ZVM_PASSWORD =

• override cache service: systemctl edit openqa-worker-cacheservice.service)

# /etc/systemd/system/openqa-worker-cacheservice.service.d/override.conf
[Service]
Environment="MOJO_LISTEN=http://0.0.0.0:9530"

• podman run -d -h openqaworker23_container --name openqaworker23_container_101 -p $(python3 -c"p=101*10+20003;print(f'{p}:{p}')") -e OPENQA_WORKER_INSTANCE=101 -v /opt/s390x_opensuse:/etc/openqa -v /var/lib/openqa/share:/var/lib/openqa/share -v /var/lib/openqa/cache:/var/lib/openqa/cache registry.opensuse.org/devel/openqa/containers15.5/ openqa_worker_os_autoinst_distri_opensuse:latest

• (cd /etc/systemd/system/; podman generate systemd -f -n --new openqaworker23_container_101 --restart-policy always)

• systemctl enable container-openqaworker23_container_101.service
Created symlink /etc/systemd/system/default.target.wants/container-openqaworker23_container_101.service → /etc/systemd/system/container-openqaworker23_container_101.service.

• one presumably bug I found, openssh was missing from the container (see https://openqa.opensuse.org/tests/3736877#step/bootloader_s390/33) -> podman exec -it openqaworker23_container_101 zypper in openssh manually

https://openqa.opensuse.org/tests/3733329#step/partitioning_finish/2

For that I suggest to use the interactive mode from openQA and pause at the partitioning module and look if the connection can be established at all. I suspect firewall blocking the communication.

I tried once, but it didn't stop, however I logged in before that step to already notice that DNS resolution didn't work at all. Changed nameserver to 10.150.2.10 -> https://openqa.opensuse.org/tests/3737257

there is a job which passed the installation step: https://openqa.opensuse.org/tests/3737299#live
let's see -> https://openqa.opensuse.org/tests/overview?build=poo137408_V2&version=Tumbleweed&distri=opensuse

On w32 in /opt/s390x_opensuse I added

ZVM_HOST = s390zl11.openqanet.opensuse.org

to fix https://openqa.opensuse.org/tests/3749284/. Retriggered as https://openqa.opensuse.org/tests/3749340

There are still some old o3 jobs scheduled with old worker class "s390x-zVM-vswitch-l2", not the new one which is just "s390x-zVM". We could just ignore those. But I opted to add that back to the worker class config of all workers so we have now in w23:/opt/s390x_opensuse/workers.ini

[global]
…
WORKER_CLASS = s390x-zVM,s390x-zVM-vswitch-l2,prg,prg2,prg2-j12
ZVM_PASSWORD = …
ZVM_HOST = s390zl11.openqanet.opensuse.org

[101]
ZVM_GUEST = o3zvm001.openqanet.opensuse.org

[102]
ZVM_GUEST = o3zvm002.openqanet.opensuse.org

[103]
ZVM_GUEST = o3zvm003.openqanet.opensuse.org

[104]
ZVM_GUEST = o3zvm004.openqanet.opensuse.org

and then systemctl restart container-openqaworker23_container_10{1..4}.service

that lead to problems that reverse resolution did not work for o3zvm003.openqanet.opensuse.org and o3zvm004.openqanet.opensuse.org but working for o3zvm001 and o3zvm002 that we already had in o3:/etc/hosts. I restarted dnsmasq on o3 and restarted https://openqa.opensuse.org/tests/3751739 leading to https://openqa.opensuse.org/tests/3751741

Not clear about https://openqa.opensuse.org/tests/3751745#step/bootloader_s390/29 . Please take a look

okurz wrote in #note-22:

Not clear about https://openqa.opensuse.org/tests/3751745#step/bootloader_s390/29 . Please take a look

hmm must have missed this...

well, looks like it's not booting within 300s and there is no output after network config created... but it don't see any obvious failure