Project

General

Profile

Actions

action #158985

closed

coordination #105624: [saga][epic] Reconsider how openQA handles secrets

coordination #157537: [epic] Secure setup of openQA test machines with secure network+secure authentication

openQA worker native on s390x

Added by okurz 8 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Motivation

In https://sd.suse.com/servicedesk/customer/portal/1/SD-150437 we are asked to handle "compromised root passwords in QA segments" including s390kvm080…099 . We might have an easier time to prevent ssh access to s390x kvm instances if we run the openQA worker instances directly on s390zl12+13. For this we should build native openQA-worker packages which seems to be feasible according to #158455.

Acceptance criteria

  • AC1: openQA-worker packages on s390x are on-par with e.g. ppc64le

Suggestions

  • Enable s390x in devel:openQA, devel:openQA:Leap:…
  • Ensure s390x package are built and published accordingly
  • Update documentation where applicable
  • Manual smoke test of package on an s390x machine

Related issues 3 (2 open1 closed)

Related to openQA Infrastructure (public) - action #159063: s390x qemu backend host within SUSE networksNew2024-04-16

Actions
Copied from openQA Project (public) - action #158455: [spike][timeboxed:10h] openQA worker native on s390xResolvedokurz2024-03-28

Actions
Copied to openQA Project (public) - action #159621: Make tests work with native openQA worker s390x qemu size:MWorkable

Actions
Actions #1

Updated by okurz 8 months ago

  • Copied from action #158455: [spike][timeboxed:10h] openQA worker native on s390x added
Actions #2

Updated by okurz 8 months ago

https://suse.slack.com/archives/C028VS8TM2B/p1713204572621339

In multiple projects I observed that despite s390x building fine and icons in OBS saying packages are "published" still I don't find an s390x repository, e.g. in https://download.opensuse.org/repositories/devel:/openQA/15.5/ based on https://build.opensuse.org/project/show/devel:openQA . Any hints?

Actions #3

Updated by okurz 8 months ago

  • Due date set to 2024-04-29
  • Status changed from In Progress to Feedback
Actions #4

Updated by okurz 8 months ago

  • Description updated (diff)
Actions #6

Updated by okurz 8 months ago

As explained by adrianS in matrix/IRC:

adrianS matrix-o-o: the entire repo finished building?
adrianS matrix-o-o: if you wait for a subdirectory of s390, you can wait endless since you only build noarch packages ;)
adrianS in 15.5
adrianS hm, actually you build s390x.rpms but you also have a publishfilter defined
adrianS comes from openSUSE:Leap:15.5:Update

Added in https://build.opensuse.org/projects/devel:openQA/prjconf

# Trying to overwrite PublishFilter from https://build.opensuse.org/projects/openSUSE:Leap:15.5:Update/prjconf to enable publishing of s390x
# https://progress.opensuse.org/issues/158985
PublishFilter: none
Actions #7

Updated by okurz 8 months ago

and later

(adrianS) Marcus has fixed the :Update projects instead meanwhile

so I removed the setting again. Meanwhile s390x files showed up. Should test on an s390x instance. Will ask Asked mgriessmeier for a test instance.

https://suse.slack.com/archives/C02CANHLANP/p1713272574081329

(Oliver Kurz) @Matthias Griessmeier we now have openQA worker packages built on s390x and to resolve the security incident regarding s390x kvm I would like us to setup openQA qemu based workers. Can you provide us 1-2 s390x test instances we can setup as temporary or permanent openQA workers? In some weeks/months we can then give back 1-2 machines, either the new ones or zl12/13

Actions #8

Updated by okurz 8 months ago · Edited

root@10.145.10.49, added my key, experimenting …

zypper ar -p 95 -f 'http://download.opensuse.org/repositories/devel:openQA/$releasever' devel_openQA
zypper ar -p 90 -f 'http://download.opensuse.org/repositories/devel:openQA:Leap:$releasever/$releasever' devel_openQA_Leap
zypper -n in os-autoinst-distri-opensuse-deps openQA-worker

and manually copied client.conf content from w40 and /etc/openqa/workers.ini created with

[global]
HOST=openqa.suse.de
CACHEDIRECTORY=/var/lib/openqa/cache
LOG_LEVEL=debug
CRITICAL_LOAD_AVG_THRESHOLD=40
WORKER_CLASS=qemu_s390x,tap,region-prg,datacenter-dc7,location-prg2,s390zl19

[openqa.suse.de]
TESTPOOLSERVER = rsync://openqa.suse.de/tests

then

systemctl start openqa-worker-{cacheservice,auto-restart@1}

https://openqa.suse.de/admin/workers/3553 showed up.

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/14005042 _GROUP=0 {TEST,BUILD}+=-okurz-poo158985 BACKEND=qemu WORKER_CLASS=qemu_s390x,s390zl19 

1 job has been created:

Actions #9

Updated by okurz 8 months ago

  • Status changed from Feedback to In Progress

https://openqa.suse.de/tests/14043454 incompleted with "no Qemu/KVM found"

-> https://github.com/os-autoinst/os-autoinst/pull/2488

alternatively I could set a path for the qemu binary.

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/14005042 _GROUP=0 {TEST,BUILD}+=-okurz-poo158985 BACKEND=qemu WORKER_CLASS=qemu_s390x,s390zl19 QEMU=s390x

1 job has been created:

Actions #10

Updated by okurz 8 months ago

That now failed with

[2024-04-16T21:41:23.336031+02:00] [warn] [pid:34224] !!! : qemu-system-s390x: -device VGA,edid=on,xres=1024,yres=768: 'VGA' is not a valid device model name

I should continue with local isotovideo runs.

Actions #11

Updated by okurz 8 months ago

I could bring a test further with

sudo -u _openqa-worker isotovideo -d casedir=/var/lib/openqa/cache/openqa.suse.de/tests/sle productdir=/var/lib/openqa/cache/openqa.suse.de/tests/sle/products/sle qemu_video_device=virtio-gpu schedule=tests/installation/bootloader,tests/installation/welcome

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19128 created to fix the schedule for s390x qemu.

https://github.com/os-autoinst/os-autoinst/pull/2489 to provide a proper default display adapter.

Actions #12

Updated by okurz 8 months ago

https://github.com/os-autoinst/os-autoinst/pull/2489 merged. I installed manually on s390zl19. https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19128 still pending.

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/14005042 _GROUP=0 {TEST,BUILD}+=-okurz-poo158985 BACKEND=qemu WORKER_CLASS=qemu_s390x,s390zl19 QEMU=s390x

1 job has been created:

https://openqa.suse.de/tests/14068509#step/bootloader_zkvm/2 fails due to https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19128 missing.

Actions #13

Updated by okurz 8 months ago

  • Status changed from In Progress to Feedback
Actions #14

Updated by okurz 8 months ago

  • Related to action #159063: s390x qemu backend host within SUSE networks added
Actions #15

Updated by okurz 8 months ago · Edited

Running

sudo -u _openqa-worker isotovideo -d casedir=https://github.com/okurz/os-autoinst-distri-opensuse#feature/s390x_qemu productdir=os-autoinst-distri-opensuse/pro
ducts/sle needles_dir=/var/lib/openqa/cache/openqa.suse.de/tests/sle/products/sle/needles schedule=tests/installation/bootloader,tests/installation/welcome iso=/var/lib/openqa/cache/openqa.suse
.de/SLE-15-SP6-Online-s390x-Build79.1-Media1.iso

now to check the code change but in VNC it just shows that the machine hasn't activated display (yet). More context to explore could be in https://github.com/os-autoinst/os-autoinst/issues/2312 and https://github.com/os-autoinst/os-autoinst/pull/2309
Haven't found anything myself. I asked in https://matrix.to/#/!dRljORKAiNJcGEDbYA:opensuse.org/$JedShJXpfMrYvLTZkTbnS2vgukbufStP9VB408z-6UA .

Actions #16

Updated by okurz 8 months ago

Response from lkhn in https://matrix.to/#/!dRljORKAiNJcGEDbYA:opensuse.org/$n9ddlK-QaOW261xJHGwxChlxgs1pLa0Adx0EQRfdYuo :

Hi everyone, I just started s390x test with these machine settings:

ARCH=s390x
CDMODEL=scsi-cd
HDDMODEL=virtio-blk
MAX_JOB_TIME=36000
QEMUCPU=host
QEMUCPUS=2
QEMUMACHINE=s390-ccw-virtio
QEMURAM=4096
QEMU_VIDEO_DEVICE=virtio-gpu
TIMEOUT_SCALE=2
WORKER_CLASS=qemu_s390x

When I created the patch on the mentioned PR, I consult the output of this command:
/usr/libexec/qemu-kvm -device ?
For the display I have these options on AlmaLinux OS 9.
Display devices:
name "virtio-gpu-ccw", bus virtual-css-bus, alias "virtio-gpu"
name "virtio-gpu-device", bus virtio-bus
What is the type is your s390x worker? Is KVM fully functioning there? 🤔 Mine is a VM on z/VM hypervisor.
https://openqa.almalinux.org/tests/7289#

actually a good point that I need to check if kvm actually works on that machine. I know it works on one where we use the zkvm/svirt backend. I need to compare those. And thanks for your openQA job URL which I can use as reference

Actions #17

Updated by okurz 8 months ago

  • Subject changed from openQA worker native on s390x to openQA worker native on s390x - no display on VNC yet?

https://openqa.suse.de/tests/14103628/ shows no VNC based display but the VM booted the kernel and shows a prompt as visible in https://openqa.suse.de/tests/14103628/logfile?filename=serial0.txt . I also tried to start a VM on s390zl12 and same results. I don't know if we can make a graphics device work. Would be nice but not necessarily mandatory. Maybe somebody else wants to research or has an idea how to make something show up on the display.

Actions #18

Updated by okurz 8 months ago

Waiting for mgriessmeier to experiment

Actions #19

Updated by okurz 8 months ago

  • Subject changed from openQA worker native on s390x - no display on VNC yet? to openQA worker native on s390x
  • Due date deleted (2024-04-29)
  • Status changed from Feedback to Resolved

With https://openqa.suse.de/tests/14103628/ we know that we have generally working os-autoinst and openQA-worker packages for s390x qemu. To be continued regarding making tests work.

Actions #20

Updated by jbaier_cz 8 months ago

  • Copied to action #159621: Make tests work with native openQA worker s390x qemu size:M added
Actions

Also available in: Atom PDF