QA » openQA Project » openQA Infrastructure

cool, thank you. For now you can track it as a "personal" task. Unless you can resolve it yourself please take care to assign it to a squad eventually. I suggest either [qe-core] or [tools].

@geor I assume you deleted the target version by mistake in an edit conflict, setting back to "Future".

On request by mgriessmeier creating a temporary worker configuration on grenache-1.qa. In /etc/openqa/workers.ini:

[66]
WORKER_CLASS=s390-kvm-sle12-poo93119-okurz,grenache-1
NETDEV=eth0
SUT_IP=10.161.145.90
VIRSH_HOSTNAME=s390zp19.suse.de
VIRSH_PASSWORD=nots3cr3t
VIRSH_GUEST=10.161.145.90
VIRSH_MAC=52:54:00:12:5c:d6
VIRSH_CMDLINE=ifcfg=dhcp
VIRSH_INSTANCE=1

and starting with systemctl start openqa-worker@66. Scheduled new job with

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/6107218 _GROUP=0 WORKER_CLASS=s390-kvm-sle12-poo93119-okurz

Created job #6111084: sle-15-SP2-Server-DVD-Updates-s390x-Build20210526-1-ltp_cve@s390x-kvm-sle12 -> https://openqa.suse.de/tests/6111084

EDIT: job is fine

new LPAR installed, https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/320 created, waiting for merge

As the new workers are in place we can now use auto-review, not only to label openQA jobs but to retrigger as well but selecting only jobs within the timeframe 2021-05-21 to 2021-05-25 to not match any potential future problems as well.

Wait. If I understand the intended change correctly we will not have any machine "zkvm" anymore at all so we would need to change all test schedules, correct?

EDIT: for the sake of being backward-compatible how about to apply the worker class "svirt" to some or all s390-kvm-sle12 instances?

@geor I think I have an idea how to extend auto-review to retrigger tests with changed settings. I hope you leave some old un-retriggered fails for me :D

ok, nevermind. I will followup with my approach elsewhere. For now you can go ahead your way to handle all failed jobs that you can find.

I have added the additional setting _MACHINE_COMMENT="As temporary workaround worker class set to s390x-kvm-sle12 instead of s390x-kvm-sle15, see https://progress.opensuse.org/issues/93119" to the MACHINE config "s390x-kvm-sle15" on https://openqa.suse.de/admin/machines

We still need to update existing, scheduled jobs though. With openqa-cli api --pretty --osd jobs state=scheduled machine=zkvm | jq '.jobs | .[] | select(.settings.WORKER_CLASS=="svirt") | .id' we can identify currently scheduled jobs that are scheduled for the worker class "svirt" which currently does not have a worker available anymore. I don't know how to update the settings over an existing job using openqa-cli though (tried openqa-cli api --pretty -X put --osd jobs/6287610 -d '{"settings[WORKER_CLASS]": "s390x-kvm-sle12"}') but we can also do it over SQL. So I did:

for i in $(openqa-cli api --pretty --osd jobs state=scheduled machine=zkvm | jq '.jobs | .[] | select(.settings.WORKER_CLASS=="svirt") | .id'); do ssh osd "sudo -u geekotest psql --command=\"update job_settings set value = 's390-kvm-sle12' where job_id = $i and key = 'WORKER_CLASS';\" openqa"; done

and the same for WORKER_CLASS "s390x-kvm-sle15". Now to check if jobs are being picked up.

EDIT: Jobs were not picked up until mkittler restarted the openQA scheduler. Likely there is some memory caching going on and the scheduler does not see the manual updates in the database.

Temporarily disabled auto_review:

auto_review:"(?s)2021-.*T.*Error connecting to <root@s390p.*.suse.de>: No route to host"

Because the regex is slow and can take over 4 minutes.
We should improve the regex. Is there a log example with lines we have to match?

See also #96713

not applicable anymore