QA » openQA Project » openQA Infrastructure

I checked ariel (the o3 host) and apparently it is the dhcp server and I see requests from this machine. I will take care of adding it.

Setting due date based on mean cycle time of SUSE QE Tools

Thanks for the suggestion, I moved the config to rebel as worker instance 5 - see: https://openqa.opensuse.org/admin/workers/382
I also added Julies ssh key from https://progress.opensuse.org/issues/93381#note-4 to all machines with

for i in aarch64 openqaworker1 openqaworker4 openqaworker7 power8 rebel imagetester; do echo $i && sshpass -p opensuse ssh root@$i "echo '$(cat .ssh/authorized_keys | grep "julie_caohe")' >> '/root/.ssh/authorized_keys'"; done

@Julie_CAO please test if the machine is working for you as expected and feel free to raise any problems and reassign back to me :)

Julie_CAO wrote:

Hi @nicksinger, thank you for adding the machine to O3 network and openQA worker pool. Is the correct way to access the machine over rebel as the jump host from SUSE? My following procedure does not seem correct as password is needed.

jcao@linux-brmq:~/.ssh> cat config
Host ariel
HostName gate.opensuse.org
Port 2213
Host *.opensuse.org
ProxyCommand ssh -q -A -x ariel -W %h:%p

jcao@linux-brmq:~/.ssh> ssh -X rebel.opensuse.org
jcao@gate.opensuse.org's password:

Yes, sorry I forgot to create your user on gate.opensuse.org and just added your key to the root account. Now it is fixed and you should be able to ssh into ariel (gate.opensuse.org). From there you can ssh into root@rebel.openqanet.opensuse.org (we don't have users on workers, just root). Please feel free to ask here or in Rocket if you have further questions how this jump host stuff is done - it can be confusing :)

Hi Nick, I am still unable to log in 'ariel'. Is the correct ssh key used?
jcao@linux-brmq:~> ssh gate.opensuse.org
Password:

jcao@linux-brmq:~> ssh jcao@gate.opensuse.org
Password:

jcao@linux-brmq:~/.ssh> pwd
/home/jcao/.ssh
jcao@linux-brmq:~/.ssh> cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDI/mD7o5XnT6X6bVN+sZgFBkfAdA7/wisXOnGYSdhzIECK6OCxf06VD4rFjDoAJuhviixBeG0h8tu4Zu49qXiXBq3BsXUJdaNlHCBxmrF0oxqPKZRprF8tCn4dCFECVOfWNj4B51/9I7A76Qq+QkSxc3qSH2hzZBj1PIXBsCnb7YyUVSq3ZLWyZDY/ts1p5G+R7xWhgwunoaDM78RfTK2HZpXUtidLy5RVhp8wZM6vyOd8nKd5oWKxlD2T8IobkPHyLb3VuoGxhOqIj2mUYIBkxDs5qXmv3e17TMT2Lj2FuQCk/a912ozuHlJwPuI+zCKv86ealFTPLEUeXlBEcxtfiMIciD/LRHYYx7iRQ8HnJ7z9ld31G1xkwWwYfXscF4w+m0k8sgVWWLa69zyCXibbAfUARNIOLG2UDOEOniwj1xMeB1zEAobHi62nNLwjwn4PIYihz8LKo3zd+Alkwx4PhLn1X7ekAvU+Rq39A6+C1JyFS4I8lKkMclgrtb6c5DWS+8q3kDbykLCdgvZRVeWzmhaoANuSjDk9RqFABSdyWh4qWYBV/hRbrhYvOVsa5YDwvD76IkUu41lhk1fy3ouy2n7rtMg5QfCcIk8HgJuCiRwW+40UT9522UgBq+l4IhcjbU7W3osdzN1Q9H9KMcjvP9IOUNhgXFqgMb0zif6qqQ== julie_caohe@163.com

Another question, is there a PXE server in O3 network? I'd like to install openSUSE Tumbleweed in each test, the command we used is:
/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210521

Yes but as far as I understand it currently only supports Leap. I will check if I can enable TW installations on there.

Thank you, Nick. We need not an actual PXE entry for TW installation. The PXE server can support TW installation with boot parameters the way we do in OSD is ok.

moving all tickets without size confirmation by the team back to "New". The team should move the tickets back after estimating and agreeing on a consistent size

@Julie_CAO I checked the key again on ariel and everything looks like it should. Please use the command ssh ariel to login into that host. If you use "gate.opensuse.org" your config is not used and therefore it does not work.

I've added a kernel and initrd from the latest tw-netinstall ISO onto the PXE server on ariel. A first try didn't seem to work despite ariel serving the required files:

Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: error 0 TFTP Aborted received from 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: failed sending /srv/tftpboot/pxelinux.0 to 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: sent /srv/tftpboot/pxelinux.0 to 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/44454c4c-3100-1059-8052-b9c04f463733 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/01-f4-02-70-b9-3a-66 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8700D not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8700 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A870 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A87 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: sent /srv/tftpboot/pxelinux.cfg/default to 192.168.112.13

I saw something hinting at syntax errors but the IPMI connection in the train is not good enough. Will check if I can make it work at home with a stable connection.

Setting due date based on mean cycle time of SUSE QE Tools

I can't reach the machine nor the ipmi interface. I created an infra ticket now to check this machine: https://infra.nue.suse.com/Ticket/Display.html?id=193192

I've fixed the PXE setup in O3 and was able to start a tw installer. Please let me know if anything is not working as expected for you.

Sorry for the long delay. I didn't notice your answer. I think you need to add the proper console parameter. Something like this should make the console show up over SOL: console=tty console=ttyS1,115200 (maybe ttyS0 depending on the machine). Please have a look what other IPMI tests on e.g. OSD supply as their command line:

[2021-07-28T20:31:29.995 CEST] [debug] <<< testapi::type_string(string="  Y2DEBUG=1 vga=791 video=1024x768 console=ttyS1,115200 linuxrc.log=/dev/ttyS1 linuxrc.core=/dev/ttyS1 linuxrc.debug=4,trace reboot_timeout=0  ", max_interval=4, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)

(from https://openqa.suse.de/tests/6599035/logfile?filename=autoinst-log.txt)

I'm resolving this now as this is an test issue and not caused by the PXE setup which apparently works for your case :) Feel free to reopen if you discover other issues

nicksinger wrote:

Sorry for the long delay. I didn't notice your answer. I think you need to add the proper console parameter. Something like this should make the console show up over SOL: console=tty console=ttyS1,115200 (maybe ttyS0 depending on the machine). Please have a look what other IPMI tests on e.g. OSD supply as their command line:

[2021-07-28T20:31:29.995 CEST] [debug] <<< testapi::type_string(string="  Y2DEBUG=1 vga=791 video=1024x768 console=ttyS1,115200 linuxrc.log=/dev/ttyS1 linuxrc.core=/dev/ttyS1 linuxrc.debug=4,trace reboot_timeout=0  ", max_interval=4, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)

(from https://openqa.suse.de/tests/6599035/logfile?filename=autoinst-log.txt)

I'm resolving this now as this is an test issue and not caused by the PXE setup which apparently works for your case :) Feel free to reopen if you discover other issues

Hi @Nick, I have to reopen this ticket as the PXE server does not work in O3 actually. Even with "console=tty console=ttyS1,115200", when I press 'tab' to input the boot command, then type 'enter', the screen went back to the pxe entry, No installation began on any console. It should not be console problem because it returns to the PXE menu.

Or let's try it as a SUT in O3, the 'boot_from_pxe' will type the correct command since the machine worked in OSD. Please help me add this SUT in O3 or point me what I shall do(it seems that I have no permission to login rebel).

    WORKER_CLASS: 64bit-ipmi,64bit-ipmi-amd
    IPMI_HOSTNAME: blackbauhinia-ipmi.openqanet.opensuse.org
    IPMI_USER: ADMIN
    IPMI_PASSWORD: ADMIN
    SUT_IP: blackbauhinia.openqanet.opensuse.org
    SUT_NETDEVICE: eno1
    IPMI_SOL_PERSISTENT_CONSOLE: 1
    IPMI_BACKEND_MC_RESET: 1

@Julie_CAO In the future, please use Feedback when re-opening tickets.

I once again tried it out manually.

First, I set the bootdev:
ipmitool -I lanplus -C 3 -H blackbauhinia-ipmi.openqanet.opensuse.org -U ADMIN -P ADMIN chassis bootdev pxe

Attaching to the machine from ariel by using IPMI:
ipmitool -I lanplus -C 3 -H blackbauhinia-ipmi.openqanet.opensuse.org -U ADMIN -P ADMIN sol activate

After that it boots into the PXE menu where I just select "install-tumbleweed" and the kernel starts to load (and give me output on SOL). Just a small example from the beginning:

Loading linux/openSUSE/tumbleweed/linux.............
Loading linux/openSUSE/tumbleweed/initrd........................................
................................................................................
................ready.

[    0.000000] Linux version 5.13.4-1-default (geeko@buildhost) (gcc (SUSE Linux) 11.1.1 20210625 [revision 62bbb113ae68a7e724255e17143520735bcb9ec9], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.36.1.20210326-4) #1 SMP Thu Jul 22 15:55:06 UTC 2021 (91a0cca)
[    0.000000] Command line: initrd=linux/openSUSE/tumbleweed/initrd vga=normal console=tty0 console=ttyS1,115200 linemode=1 ignore_loglevel unknown_nmi_panic insecure=1 install=http://download.opensuse.org/tumbleweed/repo/oss/ nameserver=192.168.112.100 BOOT_IMAGE=linux/openSUSE/tumbleweed/linux
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format.
[    0.000000] BIOS-provided physical RAM map:

So manually I can't reproduce your issue. However, as rebel is currently down (https://progress.opensuse.org/issues/97658) and oli indicated that other workers will most likely won't work due to apparmor I think we're currently blocked there to get the IPMI machine into o3 and I can't do further testing for now.

nicksinger wrote:

After that it boots into the PXE menu where I just select "install-tumbleweed" and the kernel starts to load (and give me output on SOL). Just a small example from the beginning:

Yes, the pxe entry worked for me either. But I meant that the command below(press 'tab' key to type the command the way boot_from_pxe module does in openqa) does not work:
/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830 console=ttyS1,115200

I'ed like to install the latest snapshot, such as 20210830 that day I tried. Press 'enter' after typing the command, pxe menu screen showed up, no any console output.

please take a look into the blocking ticket #97658 which we consider still "urgent" and will exceed the due-date tomorrow.

Still blocked by #97658 as I understand it, although Redmine won't show that for some reason.

Hello @Julie_CAO - sorry I was on a sick-leave yesterday and before didn't update the ticket here. I was able to boot files from download.opensuse.org directly which means you could always boot the latest snapshot. However, the architecture inside the o3 infrastructure is different. Therefore the tests would need to be adjusted accordingly. I will prepare a boot environment for you today so you can see how it would work. Afterwards you would need to adjust the tests.

Setting due date based on mean cycle time of SUSE QE Tools

So I created an IPXE config which offers several options to boot a machine. I tested it with blackbauhinia manually from ariel with the following command: ipmitool -I lanplus -C 3 -H blackbauhinia-ipmi.openqanet.opensuse.org sol activate. It always loads the most recent TW snapshot from download.opensuse.org over http since we don't have any mountpoint with the most recent release like on OSD.

I realized that the IPMI output is quite slow but still was able to start an installer. Please test it manually to see if you could adjust the tests accordingly or if anything is missing. I also attach the IPXE-script below so you can see with which options the installer gets started. Let me know if any option would need to be added.

#!ipxe
dhcp

set menu-timeout 5000

:start
menu iPXE boot menu for openqa.opensuse.org
item --gap --             ------------------------- Operating systems ------------------------------
item --key l leapS1       Boot openSUSE Leap15.3 from download.opensuse.org / ttyS1 (http)
item --key l leapS2       Boot openSUSE Leap15.3 from download.opensuse.org / ttyS2 (http)
item --key t tumbleweedS1 Boot openSUSE Tumbleweed from download.opensuse.org / ttyS1 (http)
item --key t tumbleweedS2 Boot openSUSE Tumbleweed from download.opensuse.org / ttyS2 (http)
item --gap --             ------------------------- Advanced options -------------------------------
item --key c config       Configure settings
item shell                Drop to iPXE shell
item reboot               Reboot computer
item
item --key x exit         Exit iPXE and continue BIOS boot
choose --timeout ${menu-timeout} --default exit selected || goto cancel
set menu-timeout 0
goto ${selected}

:leapS1
kernel http://download.opensuse.org/distribution/leap/15.3/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/leap/15.3/repo/oss/ console=ttyS1,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/leap/15.3/repo/oss/boot/x86_64/loader/initrd
boot

:leapS2
kernel http://download.opensuse.org/distribution/leap/15.3/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/leap/15.3/repo/oss/ console=ttyS2,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/leap/15.3/repo/oss/boot/x86_64/loader/initrd
boot

:tumbleweedS1
kernel http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/openSUSE-current/repo/oss/ console=ttyS1,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/initrd
boot

:tumbleweedS2
kernel http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/openSUSE-current/repo/oss/ console=ttyS2,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/initrd
boot

:cancel
echo You cancelled the menu, dropping you to a shell

:shell
echo Type 'exit' to get the back to the menu
shell
set menu-timeout 0
set submenu-timeout 0
goto start

:failed
echo Booting failed, dropping to shell
goto shell

:reboot
reboot

:exit
exit

:config
config
goto start

nicksinger wrote:

I realized that the IPMI output is quite slow but still was able to start an installer. Please test it manually to see if you could adjust the tests accordingly or if anything is missing. I also attach the IPXE-script below so you can see with which options the installer gets started. Let me know if any option would need to be added.

#!ipxe
dhcp

set menu-timeout 5000

HI Nick, the ipmi sol console on my side is so slow. The machine enters BIOS boot right away after the iPXE menu shows up so that I have not the chance to check these configurations. Could you set the timeout longer for me to check and for openqa to match needles?

:tumbleweedS1
kernel http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/openSUSE-current/repo/oss/ console=ttyS1,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/initrd
boot

I have not tried the command but I compared with the boot command in OSD. there are some differences:

• "usessh=1 sshpassword=linux". In OSD, it is "ssh=1 sshpassword=nots3cr3t". So 'ssh=' is SLE and 'usessh=xx' in openSUSE? And the what the default password in O3, nots3cr3t or linux? our tests typically use the openqa default password.
• we have some extra variables in the boot command in OSD, "plymouth.enable=0 Y2DEBUG=1 vga=791 video=1024x768 linuxrc.log=/dev/ttyS1 linuxrc.core=/dev/ttyS1 linuxrc.debug=4,trace reboot_timeout=0 kernel.softlockup_panic=1 vt.color=0x07", actually I am not sure if they are required for openSUSE in O3, Could you help take a look and add some of them if they are needed?

Increasing the due date after mentioning the ticket in the chat (@nicksinger is not available today).

I've bumped the timeout to 50000, changed the default PW to nots3cr3t to make it easier to run tests and also added the other parameters too. It should not hurt anyway. BTW: rebel is also back in o3 so you might be able to clone a test on it to test it. But also please test manually if everything works as you would expect it.

Regarding the slow IPMI: I recommend that you try to execute your virtualization tests with the help of nested virtualization. This would really ease the setup and require less manual work to have machines moved between OSD and o3

nicksinger wrote:

I've bumped the timeout to 50000,

thank you, Nick. 50000 seems to be 50 seconds. I can see the blue menu and move the focus up and down to choose the entry now.

changed the default PW to nots3cr3t to make it easier to run tests and also added the other parameters too. It should not hurt anyway.

for the Tumbleweed entry:

:tumbleweedS1
kernel http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/linux ...

It is the installer of OpenSUSE Leap 15.3 instead of OpenSUSE Tumbleweed. Can we get a latest tumbleweed DVD URL here?

BTW: rebel is also back in o3 so you might be able to clone a test on it to test it. But also please test manually if everything works as you would expect it.

I will continue with manual test to ensure everythings works then we add the SUT to be a worker of O3.

Regarding the slow IPMI: I recommend that you try to execute your virtualization tests with the help of nested virtualization. This would really ease the setup and require less manual work to have machines moved between OSD and o3

Because SLE virtualization product does not promise to support nexted virtualization, I think so does openSUSE. Our virtualization tests all run on barel metal machines according to our test plan.

How about a due date that lies in the future. For example end of this week, assuming you are motived to finish it this week :-)

@Julie_CAO I've updated the entries once again to the tumbleweed repos posted by @okurz - please let me know if it works :)

I've added a new entry in our /etc/dnsmasq.d/pxeboot.conf:

pxe-service=x86PC, "tw-over-ipmi", tw.ipxe.bin, 1, 192.168.112.13

This instructs the pxe server to serve this tw.ipxe.bin file only to blackbauhinia. From a first test this seems to work and the system automatically boots into the TW installer. Unfortunately this also means that only booting TW now works (no menu anymore). But I think in this specific use-case this should be fine, right?

Hi @nicksinger, It is weird, it is still the old ipxe menu when I try to boot up blackbauhinia via IPMI. The ipxe menu is attached. Could you help take a look again?

One more request, could you help remove SUT_NETDEVICE=xx in worker.ini as I'd like to set it in test suite level.

And one question, there are 3 workers listed in O3. I am a little confused which one should be used in virtualization tests.

openqaworker7:20              openqaworker7            64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia
openqaworker7_container:105   openqaworker7_container  64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia
rebel:5                       rebel               64bit-ipmi_rebel,64bit-ipmi-large-mem_rebel,64bit-ipmi-amd_rebel,blackbauhinia_rebel

Unfortunately I can't understand how I can instruct dnsmasq to only serve the tw-only pxe binary to just your host. The only documented way I found is

pxe-service=x86PC, "tw-over-ipmi", tw.ipxe.bin, 1, 192.168.112.13

but this doesn't work apparently. I now have set it that every request gets served the tw.ipxe.bin file. But this will only work until someone wants to boot something else then just TW. I can leave it for now but would suggest that you build a openQA test which can properly select the right entry. openQA needs to interact with the slow SOL anyway.

nicksinger wrote:

Unfortunately I can't understand how I can instruct dnsmasq to only serve the tw-only pxe binary to just your host. The only documented way I found is

pxe-service=x86PC, "tw-over-ipmi", tw.ipxe.bin, 1, 192.168.112.13

@nicksinger, Thank you for spending time on the ipxe configurations. Our ipxe server is running dhcp service and the ipxe configuration file is on a http server. Our tests can work with them for ipxe bootup. I know little about dnsmasq, but I may spend some time in investigation.

but this doesn't work apparently. I now have set it that every request gets served the tw.ipxe.bin file. But this will only work until someone wants to boot something else then just TW. I can leave it for now but would suggest that you build a openQA test which can properly select the right entry. openQA needs to interact with the slow SOL anyway.

I just tried and the current ipxe is reporting error, see the picture attached.

@nicksinger @Julie_CAO How about discussing this in Slack or Jitsi? That might get this moving a bit faster?

@Julie_CAO @xlai we discussed the ticket in the weekly estimation meeting of SUSE QE Tools. With the latest results it is possible to access the bare-metal SUT from within openQA tests running on o3. The dnsmasq config for PXE boot customized for the test case specific might need further adjustements in the PXE config or within openQA test code itself. We think it is more feasible for you to follow-up with that in the related ticket #92380 and we call this ticket resolved with the newly added AC1 covered.

In case you need to change the config on o3 itself you are welcome to follow
https://progress.opensuse.org/projects/openqav3/wiki/#Accessing-the-o3-infrastructure
and we will support you with that.