action #158985
closedcoordination #105624: [saga][epic] Reconsider how openQA handles secrets
coordination #157537: [epic] Secure setup of openQA test machines with secure network+secure authentication
openQA worker native on s390x
0%
Description
Motivation¶
In https://sd.suse.com/servicedesk/customer/portal/1/SD-150437 we are asked to handle "compromised root passwords in QA segments" including s390kvm080…099 . We might have an easier time to prevent ssh access to s390x kvm instances if we run the openQA worker instances directly on s390zl12+13. For this we should build native openQA-worker packages which seems to be feasible according to #158455.
Acceptance criteria¶
- AC1: openQA-worker packages on s390x are on-par with e.g. ppc64le
Suggestions¶
- Enable s390x in devel:openQA, devel:openQA:Leap:…
- Ensure s390x package are built and published accordingly
- Update documentation where applicable
- Manual smoke test of package on an s390x machine
Updated by okurz 8 months ago
- Copied from action #158455: [spike][timeboxed:10h] openQA worker native on s390x added
Updated by okurz 8 months ago
https://suse.slack.com/archives/C028VS8TM2B/p1713204572621339
In multiple projects I observed that despite s390x building fine and icons in OBS saying packages are "published" still I don't find an s390x repository, e.g. in https://download.opensuse.org/repositories/devel:/openQA/15.5/ based on https://build.opensuse.org/project/show/devel:openQA . Any hints?
Updated by okurz 8 months ago
As explained by adrianS in matrix/IRC:
adrianS matrix-o-o: the entire repo finished building?
adrianS matrix-o-o: if you wait for a subdirectory of s390, you can wait endless since you only build noarch packages ;)
adrianS in 15.5
adrianS hm, actually you build s390x.rpms but you also have a publishfilter defined
adrianS comes from openSUSE:Leap:15.5:Update
Added in https://build.opensuse.org/projects/devel:openQA/prjconf
# Trying to overwrite PublishFilter from https://build.opensuse.org/projects/openSUSE:Leap:15.5:Update/prjconf to enable publishing of s390x
# https://progress.opensuse.org/issues/158985
PublishFilter: none
Updated by okurz 8 months ago
and later
(adrianS) Marcus has fixed the :Update projects instead meanwhile
so I removed the setting again. Meanwhile s390x files showed up. Should test on an s390x instance. Will ask Asked mgriessmeier for a test instance.
https://suse.slack.com/archives/C02CANHLANP/p1713272574081329
(Oliver Kurz) @Matthias Griessmeier we now have openQA worker packages built on s390x and to resolve the security incident regarding s390x kvm I would like us to setup openQA qemu based workers. Can you provide us 1-2 s390x test instances we can setup as temporary or permanent openQA workers? In some weeks/months we can then give back 1-2 machines, either the new ones or zl12/13
Updated by okurz 8 months ago · Edited
root@10.145.10.49, added my key, experimenting …
zypper ar -p 95 -f 'http://download.opensuse.org/repositories/devel:openQA/$releasever' devel_openQA
zypper ar -p 90 -f 'http://download.opensuse.org/repositories/devel:openQA:Leap:$releasever/$releasever' devel_openQA_Leap
zypper -n in os-autoinst-distri-opensuse-deps openQA-worker
and manually copied client.conf content from w40 and /etc/openqa/workers.ini created with
[global]
HOST=openqa.suse.de
CACHEDIRECTORY=/var/lib/openqa/cache
LOG_LEVEL=debug
CRITICAL_LOAD_AVG_THRESHOLD=40
WORKER_CLASS=qemu_s390x,tap,region-prg,datacenter-dc7,location-prg2,s390zl19
[openqa.suse.de]
TESTPOOLSERVER = rsync://openqa.suse.de/tests
then
systemctl start openqa-worker-{cacheservice,auto-restart@1}
https://openqa.suse.de/admin/workers/3553 showed up.
openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/14005042 _GROUP=0 {TEST,BUILD}+=-okurz-poo158985 BACKEND=qemu WORKER_CLASS=qemu_s390x,s390zl19
1 job has been created:
- sle-15-SP6-Online-s390x-Build79.1-default@s390x-kvm -> https://openqa.suse.de/tests/14043454
Updated by okurz 8 months ago
- Status changed from Feedback to In Progress
https://openqa.suse.de/tests/14043454 incompleted with "no Qemu/KVM found"
-> https://github.com/os-autoinst/os-autoinst/pull/2488
alternatively I could set a path for the qemu binary.
openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/14005042 _GROUP=0 {TEST,BUILD}+=-okurz-poo158985 BACKEND=qemu WORKER_CLASS=qemu_s390x,s390zl19 QEMU=s390x
1 job has been created:
- sle-15-SP6-Online-s390x-Build79.1-default@s390x-kvm -> https://openqa.suse.de/tests/14043509
Updated by okurz 8 months ago
I could bring a test further with
sudo -u _openqa-worker isotovideo -d casedir=/var/lib/openqa/cache/openqa.suse.de/tests/sle productdir=/var/lib/openqa/cache/openqa.suse.de/tests/sle/products/sle qemu_video_device=virtio-gpu schedule=tests/installation/bootloader,tests/installation/welcome
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19128 created to fix the schedule for s390x qemu.
https://github.com/os-autoinst/os-autoinst/pull/2489 to provide a proper default display adapter.
Updated by okurz 8 months ago
https://github.com/os-autoinst/os-autoinst/pull/2489 merged. I installed manually on s390zl19. https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19128 still pending.
openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/14005042 _GROUP=0 {TEST,BUILD}+=-okurz-poo158985 BACKEND=qemu WORKER_CLASS=qemu_s390x,s390zl19 QEMU=s390x
1 job has been created:
- sle-15-SP6-Online-s390x-Build79.1-default@s390x-kvm -> https://openqa.suse.de/tests/14068509
https://openqa.suse.de/tests/14068509#step/bootloader_zkvm/2 fails due to https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19128 missing.
Updated by okurz 8 months ago
- Related to action #159063: s390x qemu backend host within SUSE networks added
Updated by okurz 8 months ago · Edited
Running
sudo -u _openqa-worker isotovideo -d casedir=https://github.com/okurz/os-autoinst-distri-opensuse#feature/s390x_qemu productdir=os-autoinst-distri-opensuse/pro
ducts/sle needles_dir=/var/lib/openqa/cache/openqa.suse.de/tests/sle/products/sle/needles schedule=tests/installation/bootloader,tests/installation/welcome iso=/var/lib/openqa/cache/openqa.suse
.de/SLE-15-SP6-Online-s390x-Build79.1-Media1.iso
now to check the code change but in VNC it just shows that the machine hasn't activated display (yet). More context to explore could be in https://github.com/os-autoinst/os-autoinst/issues/2312 and https://github.com/os-autoinst/os-autoinst/pull/2309
Haven't found anything myself. I asked in https://matrix.to/#/!dRljORKAiNJcGEDbYA:opensuse.org/$JedShJXpfMrYvLTZkTbnS2vgukbufStP9VB408z-6UA .
Updated by okurz 8 months ago
Response from lkhn in https://matrix.to/#/!dRljORKAiNJcGEDbYA:opensuse.org/$n9ddlK-QaOW261xJHGwxChlxgs1pLa0Adx0EQRfdYuo :
Hi everyone, I just started s390x test with these machine settings:
ARCH=s390x CDMODEL=scsi-cd HDDMODEL=virtio-blk MAX_JOB_TIME=36000 QEMUCPU=host QEMUCPUS=2 QEMUMACHINE=s390-ccw-virtio QEMURAM=4096 QEMU_VIDEO_DEVICE=virtio-gpu TIMEOUT_SCALE=2 WORKER_CLASS=qemu_s390x
When I created the patch on the mentioned PR, I consult the output of this command:
/usr/libexec/qemu-kvm -device ?
For the display I have these options on AlmaLinux OS 9.
Display devices:
name "virtio-gpu-ccw", bus virtual-css-bus, alias "virtio-gpu"
name "virtio-gpu-device", bus virtio-bus
What is the type is your s390x worker? Is KVM fully functioning there? 🤔 Mine is a VM on z/VM hypervisor.
https://openqa.almalinux.org/tests/7289#
actually a good point that I need to check if kvm actually works on that machine. I know it works on one where we use the zkvm/svirt backend. I need to compare those. And thanks for your openQA job URL which I can use as reference
Updated by okurz 8 months ago
- Subject changed from openQA worker native on s390x to openQA worker native on s390x - no display on VNC yet?
https://openqa.suse.de/tests/14103628/ shows no VNC based display but the VM booted the kernel and shows a prompt as visible in https://openqa.suse.de/tests/14103628/logfile?filename=serial0.txt . I also tried to start a VM on s390zl12 and same results. I don't know if we can make a graphics device work. Would be nice but not necessarily mandatory. Maybe somebody else wants to research or has an idea how to make something show up on the display.
Updated by okurz 8 months ago
- Subject changed from openQA worker native on s390x - no display on VNC yet? to openQA worker native on s390x
- Due date deleted (
2024-04-29) - Status changed from Feedback to Resolved
With https://openqa.suse.de/tests/14103628/ we know that we have generally working os-autoinst and openQA-worker packages for s390x qemu. To be continued regarding making tests work.
Updated by jbaier_cz 8 months ago
- Copied to action #159621: Make tests work with native openQA worker s390x qemu size:M added