QA » openQA Project » openQA Infrastructure

Two arm64 hw machine orders are already in pipeline.

One has been ordered as replacement for arm1 and arm2 that were used by QAM (but no longer available).
Second one for openQA.
There would be also need for this kind of validation work, with a machine more resembling what customers would use.

Heiko will talk to Andreas and what is the status of plans for orthos as well.

As stated in the meeting I suggest to not aim for "permanent reservations" as this tends to cause forgotten, unused hardware. Either ask the arch-team for a dedicated, special reservation without expiration or use automation to extend reservations. See https://github.com/okurz/scripts/blob/master/extend-orthos-reserve for a way to automate orthos reservation

@tjyrinki_suse Although it is about manual testing and I'm not sure what is purpose of it as description doesn't contain specific details, but we have one arm machine for kernel testing. We do automatic installation, basic ltp and kdump(in development). Machine is not ready and never will be ready for stable validation (hw unstability, firmware issues...already unsupported cpu by vendor), but it is good enough and maybe better than random orthos machine for our purpose.

Also I'm not sure what exactly you want to test for 15-SP3 RCx, arm stakeholders interested in bare metal testing wanted installation and simple smoke tests, which we already do.

If you want to, we can agree and you can try to test what you want. Machine usually contains latest SLE 15-SP3 build so we can start it up and you can connect.

3 new machines have arrived, 1 for manual tests two for openQA. Heiko will ask Calen for setting up the machines, FC is a problem here.

maritawerner wrote:

3 new machines have arrived, 1 for manual tests two for openQA. Heiko will ask Calen for setting up the machines, FC is a problem here.

1. can you provide more details and an updated status? I am especially interested in "two for openQA".
2. who takes care that the machines are ending up in server rooms
3. is there an EngInfra ticket that can be referenced

The "SUSE QE Tools" team can take over as soon as the according machines are accessible over IPMI+SSH

@heiko? Could you please add the info?

@nicksinger you are involved with FSP+ and something like 10G switch for ARM machines, right? I guess this ticket is about the same. If yes, can you link a ticket if you have one or take this ticket and update it?

@mgriessmeier can you help here what's the status and estimates about the ARM machines, FSP+ 10G switches, etc.?

machines are in nuremberg office / conf room london, switch space is clarified - what is need to be done is to order the FSP-Adapters, quote is already requested from Delta and a ticket to be opened to engInfra for mounting the machines

I just opened the infra ticket, so at least the machines could be already moved: https://infra.nue.suse.com/Ticket/Display.html?id=191515

mgriessmeier wrote:

"quote is already requested from Delta"

I assume that means that either you or nsinger would get a response from them and then order, right?

okurz wrote:

mgriessmeier wrote:

"quote is already requested from Delta"

I assume that means that either you or nsinger would get a response from them and then order, right?

yes,
however, meanwhile I got a response in the engInfra ticket, stating that they cannot move the servers there in the next two weeks because they are overloaded with urgent inventory works of SRV1 as well as current network issues ongoing

sorry - I forgot to update the ticket, so let me reflect current situation as of yesterday:

• due to AC failure in SRV2, the machines could not be mounted there. There is no ETA when this can happen
• Yesterday (August 28th) I was in the office with nsinger to mitigate this and we set up a temporary solution for those and potential more servers to come.
• so both machines are at the moment mounted in the big QA Lab, connected to the network and are waiting to be installed, which Nick planned to do today. As soon this is done we can connect them to openQA

I have a question. Are these machines to be used only for on-demand manual testing or we can automate jobs on them?

These two machines will be added to openQA and can than be used for all jobs. But in addition to the two machines Heiko has a third aarch64 machine that he has reserved for manual tests. I am not very familiar with manual tests in QAM but I think that there is a "pool of HW for manual tests" were this machine is/will be added.

maritawerner wrote:

These two machines will be added to openQA and can than be used for all jobs. But in addition to the two machines Heiko has a third aarch64 machine that he had reserved for manual tests.

Ok, thanks for the update. Anyway, we will keep using the qemu_aarch64 worker class, so if they are added to that pool we won't need to do anything for JeOS.

trying to boot the machine over PXE shows that the qanet PXE setup for aarch is not working and outdated. I added the following config to the grub.cfg on qanet:

submenu 'Leap 15.3' {
  menuentry 'Leap-15.3-installer' {
    echo 'Setting append...'
    set append='usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/ports/aarch64/distribution/leap/15.3/repo/oss/ console=ttyAMA0,115200n8'
    echo 'Done!'
    echo 'Loading kernel...'
    linux http,download.opensuse.org)/ports/aarch64/distribution/leap/15.3/repo/oss/boot/aarch64/linux $append
    echo 'Done!'
    echo 'Loading initrd...'
    initrd (http,download.opensuse.org)/ports/aarch64/distribution/leap/15.3/repo/oss/boot/aarch64/initrd
    echo 'Done!'
  }
}

I now downloaded the kernel+initrd on our tftp server directly and changed grub.cfg to:

    echo 'Loading kernel...'
    linux aarch64/leap15.3/linux $append
    echo 'Done!'
    echo 'Loading initrd...'
    initrd aarch64/leap15.3/initrd
    echo 'Done!'

The kernel loads successfully. However the initrd times out - most likely because it is to big. This is what grubs tells me at boot:

Setting append...
Done!
Loading kernel...
Done!
Loading initrd...
error: ../../grub-core/net/net.c:1716:timeout reading
`aarch64/leap15.3/initrd'.
Done!

looking for a way to increase that timeout now

Okay, I don't find anything where this timeout can be adjusted. I also tried several other methods of loading the initrd e.g. supplying the IP of download.opensuse.org directly but no success. I also tried to "mount" an ISO directly over the webui of the BMC but this also just failed with error messages that the ISO could not be mounted.

I'm a little bit out of ideas now what could be done to boot these systems. Maybe I contact somebody from buildops or enginfra if they can show me how they boot aarch64 machines. Another idea would be to compile iPXE for aarch64. For this I would need to have access to a functional aarch64 machine so I might hijack one of our openQA workers to do this.

I was able to cross-compile a working version of iPXE. With make ARCH=arm64 CROSS_COMPILE=aarch64-unknown-linux-gnu- -j$(nproc) bin-arm64-efi/ipxe.efi EMBED=../myscript.ipxe. myscript.ipxe contains the following:

#!ipxe
dhcp

kernel http://download.opensuse.org/ports/aarch64/distribution/leap/15.3/repo/oss/boot/aarch64/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/ports/aarch64/distribution/leap/15.3/repo/oss/ console=ttyAMA0,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/ports/aarch64/distribution/leap/15.3/repo/oss/boot/aarch64/initrd
boot
shell

This allowed me to boot and install Leap15.3 (currently running) on openqaworker-arm-5

and where did you put these files? Can you also say where the file grub.cfg on qanet resides? Is this something that could/should also be handled on salt?

everything related to these experiments here is located on qanet in the folder /srv/tftp/aarch64. And yes, a proper PXE setup for all arches should be covered by salt. However I don't think that manually tinkering with bootloaders and having years-old scripts to generate their configs is the right approach. I have yet to find a tool which can handle all this more smoothly. Maybe cobbler is an option which configs could then be managed by salt

There was an IP collision with another host making arm-4's BMC unreachable. It was resolved by https://gitlab.suse.de/qa-sle/qanet-configs/-/commit/139354ea37851407deb0570362d96ed87ef8f1d9 and over night the BMC requested another lease and is reachable again now. I started an installation on it now.
In the meantime I try to setup salt on arm-5 but currently struggling by salt detecting the host as "QA-Power8-4-kvm.qa.suse.de" (which is obviously wrong). I tried to add a proper reverse entry for those arms with https://gitlab.suse.de/qa-sle/qanet-configs/-/commit/0b12e2ab0eb025514e14612963a22e90a1ec12fd but still wrongly reported in salt-key on OSD

apparently the minion_id on the minion was wrongly generated due to some cached DNS entries. I removed the wrong key in /etc/salt/pki/master/minions_denied on OSD, deleted /etc/salt/minion_id on arm-5 and started the minion again. This time it correctly connects as openqaworker-arm-5.qa.suse.de to OSD

highstate into the worker role seems to have worked - almost. 1 issue:

----------
          ID: firewalld_zones
    Function: file.managed
        Name: /etc/firewalld/zones/trusted.xml
      Result: False
     Comment: Unable to manage file: Jinja variable 'dict object' has no attribute 'openqaworker-arm-5'
     Started: 14:19:43.186399
    Duration: 136.135 ms
     Changes:

Work has been done to fix salt in e.g. https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/572, https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/569, https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/570 and https://progress.opensuse.org/issues/98243 - this allows our highstate to fully complete again hence make telegraf+monitoring work properly on new hosts. I now look into the final deployment of openqaworker-arm-4

I added the workerconf in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/353 with a really small number of workers for now to see if these machines work as expected. Further tweaking of these instance numbers should be handled in https://progress.opensuse.org/issues/98661 #98661

I suggest to always reference other tickets with the format #<id> for better linking and preview. If you like to have both a valid URL in email updates as well as useful text on the web interface you can write both, URL and #<id>

Both workers process jobs now successfully:
https://openqa.suse.de/tests/7150683
https://openqa.suse.de/tests/7150684

Thanks a lot! That is good news!