action #28132
closed[ha] Multimachine setup for s390x
100%
Description
For HA openqa tests (QA-CSS team) we need to have multimachine setup available for ppc64, aarch64 and s390x like we have on x86_64.
Priority is:
- ppc64le
- s390x
- aarch64
But I know that s390x could be more complicated to do (aarch64 should be easier).
Updated by ldevulder about 6 years ago
- Due date changed from 2017-12-04 to 2018-02-05
- Priority changed from Urgent to High
- % Done changed from 0 to 30
aarch64 configuration done on 23/01/2018.
Updated by okurz about 6 years ago
What are you expectations how it's going on here?
Updated by ldevulder about 6 years ago
If possible, I would have ppc64le and s390x workers configured in multi-machine mode for HA tests like in x86_64. For aarch64 it's ok now, some workers have been configured by Santiago 2 weeks ago.
I don't know how to do this, I don't know if ppc64le use KVM or PowerVM, as well for s390x.
Updated by okurz about 6 years ago
- Subject changed from Multimachine setup for pp64le, aarch64 and s390x to [ha]Multimachine setup for pp64le, aarch64 and s390x
Updated by ldevulder about 6 years ago
- Subject changed from [ha]Multimachine setup for pp64le, aarch64 and s390x to [ha]Multimachine setup for pp64le and s390x
aarch64 has been done in January, was also needed for HPC tests.
Updated by szarate over 5 years ago
- Project changed from openQA Tests to openQA Project
- Subject changed from [ha]Multimachine setup for pp64le and s390x to [ha] Multimachine setup for pp64le and s390x
- Category deleted (
Infrastructure)
Since for ppc64le gives many issues with nested virtualization, this needs to be done with a machine similar to malbec or qa-power8-5. Harris offered Lpars in mania, this is not viable (neither for replacing a production worker nor for doing the tests with MM)
@okurz, @Harris do you have one for tests within your team?
Updated by okurz over 5 years ago
- Project changed from openQA Project to openQA Tests
- Category set to Infrastructure
@foursixnine, I know you favor looking at one redmine project but coolo (and others) use it differently. This is a ticket that is clearly out of scope for any outside openQA contributor. The right project for that is "openQA tests". If you do not look at the "infrastructure" component of "openQA tests", well, that is another problem ;)
What are the "many issues" with nested virtualization? Has been a long time since I last used it but it worked on shiraz-2.arch
@okurz, @Harris do you have one for tests within your team?
What exactly do you mean by that? We use the ppc64le currently enabled as production machines for osd implicitly, of course. grenache is used as a "playground" worker that (unfortunately) is also connected to osd and runs the only one critical powerVM test on osd I am aware of. Next to that e.g. zluo has machines for manual testing which IIRC he also shares on request with e.g. Bejing colleagues, etc.
Updated by szarate over 5 years ago
After looking at this on friday finally today we merged and deployed salt recipes on malbec, no changes required in the backend. After a reboot configuration took place and seems to be working fine... and picking up jobs. Waiting for Loic to take over.
Updated by szarate over 5 years ago
@okurz i meant a ppc machine that was not an lpar :) But I think that for now we're good. Let's see
Updated by ldevulder over 5 years ago
okurz wrote:
What are the "many issues" with nested virtualization? Has been a long time since I last used it but it worked on shiraz-2.arch
@okurz, I tested openQA on ppc64le with kvm-pr module (so nested virtualization in ppc64le) and the "many issues" I had were (not an exhaustive list ;-)):
- issue with VGA card: kvm-pr issue (some instruction not emulated), I had a look with Alexander Graf but no way to resolve it...
- lot of timeout, I had to change some tests, but that's not a big issue if the first one was resolved
- I was able to workaround the first issue by using virtio-console access and virtio-gpu, but after I had another issue with SLE-12-SP3 support-server that was not able to boot, and I don't know why!
- kvm-pr is not supported at all (but it's not really an issue)
So, after checking on a ppc64le worker, I was able to find that MM configuration should be possible and after discussion with @foursixnine we decided that the easiest way was to try on a ppc64le worker, and that what we did.
I'm now waiting for the results!
Updated by okurz over 5 years ago
ok, great! Thank you for sharing such a detailed explanation, something learned :)
Updated by ldevulder over 5 years ago
First tests are good!
- node01: https://openqa.suse.de/tests/2113020
- node02: https://openqa.suse.de/tests/2113021
- support-server: https://openqa.suse.de/tests/2113017
Updated by pcervinka over 5 years ago
ldevulder wrote:
- I was able to workaround the first issue by using virtio-console access and virtio-gpu, but after I had another issue with SLE-12-SP3 support-server that was not able to boot, and I don't know why!
I created fix for virtio-console on ppc64le in poo#44699.
Updated by ldevulder over 5 years ago
- Subject changed from [ha] Multimachine setup for pp64le and s390x to [ha] Multimachine setup for s390x
- Status changed from New to In Progress
- Assignee set to acarvajal
- % Done changed from 30 to 70
ppc64le configuration has been done.
Assignee change from ldevulder to acarvajal.
Updated by ldevulder about 5 years ago
poo#46349 opened for this: https://progress.opensuse.org/issues/46349.
Updated by acarvajal almost 3 years ago
- Status changed from In Progress to Resolved
- % Done changed from 70 to 100
openQA HA tests have been deployed in openqa.suse.de for all architectures since (at least) 15-SP2, perhaps even as far back as 12-SP5.
aarch64 and ppc64le tests are configured with the qemu backend and rely on support server for DHCP, DNS and iSCSI (same configuration as in x86_64); s390x tests are using the svirt backend (s390x-kvm-sle12 worker class) and rely on qanet for DNS & DHCP, on sam.qa.suse.de for iSCSI and on saphq.suse.de for NFS.
Latest results are in: https://openqa.suse.de/group_overview/143
More details regarding the s390x setup in: https://gitlab.suse.de/hsehic/qa-css-docs/-/blob/master/ha/openqa.md#information-specific-to-s390x-tests