action #93381
closed[O3]request to add an IPMI SUT to O3 size:M
Added by Julie_CAO over 3 years ago. Updated almost 3 years ago.
0%
Description
Motivation¶
Virtualization team requested infra to move a IPMI SUT to vlan 663 from OSD for temporary use in O3. Now the infra is ready, refer to https://infra.nue.suse.com/SelfService/Display.html?id=189224.
the server has been added to racktables and was located in SRV2 not the labs.
https://racktables.suse.de/index.php?page=object&tab=default&object_id=14353
Both LAN port and BMC are untagged to Vlan662: openqa-worker.
662 f4:02:70:b9:3a:66 gi6 dynamicPlease check the mac @ on the DHCP server to get an IP@.
Now what should I do is to add the MAC of the machine to O3 dhcp server to get an IP? Is the O3 host the dhcp server? ie. request to add my ssh pub key to O3 and add the mac to /etc/dnsmasq.d/openqa.conf and /etc/hosts?(I got the idea from https://progress.opensuse.org/projects/openqav3/wiki/#Moving-worker-from-osd-to-o3) then access the IPMI machine over ssh by using the O3 server as the jump host?
Could someone from openqa infra team help? or guide me what to do. thanks!
Acceptance criteria¶
- AC1: the bare-metal SUT can be accessed from within openQA tests running on o3
Files
| ipxe_screen.png (66.2 KB) ipxe_screen.png | ipxe_screen.png | Julie_CAO, 2022-01-25 03:16 | |
| ipxe_TFTP_error.png (74.4 KB) ipxe_TFTP_error.png | Julie_CAO, 2022-02-07 12:12 |
Updated by Julie_CAO over 3 years ago
- Subject changed from [O3]add a IPMI SUT to O3 to [O3]request to add an IPMI SUT to O3
Updated by nicksinger over 3 years ago
- Status changed from New to In Progress
- Assignee set to nicksinger
I checked ariel (the o3 host) and apparently it is the dhcp server and I see requests from this machine. I will take care of adding it.
Updated by Julie_CAO over 3 years ago
Thank you nick for taking care. I need two fixed IPs for both IPMI BMC and machine NIC.
my ssh key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDI/mD7o5XnT6X6bVN+sZgFBkfAdA7/wisXOnGYSdhzIECK6OCxf06VD4rFjDoAJuhviixBeG0h8tu4Zu49qXiXBq3BsXUJdaNlHCBxmrF0oxqPKZRprF8tCn4dCFECVOfWNj4B51/9I7A76Qq+QkSxc3qSH2hzZBj1PIXBsCnb7YyUVSq3ZLWyZDY/ts1p5G+R7xWhgwunoaDM78RfTK2HZpXUtidLy5RVhp8wZM6vyOd8nKd5oWKxlD2T8IobkPHyLb3VuoGxhOqIj2mUYIBkxDs5qXmv3e17TMT2Lj2FuQCk/a912ozuHlJwPuI+zCKv86ealFTPLEUeXlBEcxtfiMIciD/LRHYYx7iRQ8HnJ7z9ld31G1xkwWwYfXscF4w+m0k8sgVWWLa69zyCXibbAfUARNIOLG2UDOEOniwj1xMeB1zEAobHi62nNLwjwn4PIYihz8LKo3zd+Alkwx4PhLn1X7ekAvU+Rq39A6+C1JyFS4I8lKkMclgrtb6c5DWS+8q3kDbykLCdgvZRVeWzmhaoANuSjDk9RqFABSdyWh4qWYBV/hRbrhYvOVsa5YDwvD76IkUu41lhk1fy3ouy2n7rtMg5QfCcIk8HgJuCiRwW+40UT9522UgBq+l4IhcjbU7W3osdzN1Q9H9KMcjvP9IOUNhgXFqgMb0zif6qqQ== julie_caohe@163.com
Let me know if any information are needed.
Updated by openqa_review over 3 years ago
- Due date set to 2021-06-17
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger over 3 years ago
I've added the machine on ariel to the dnsmasq config.
SUT can be reached under: blackbauhinia.openqanet.opensuse.org
IPMI can be reached under: blackbauhinia-ipmi.openqanet.opensuse.org
I've also added it as worker to openqaworker7 with the following config in /etc/openqa/workers.ini:
[20]
WORKER_CLASS=64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia
IPMI_HOSTNAME=blackbauhinia-ipmi.openqanet.opensuse.org
IPMI_USER=ADMIN
IPMI_PASSWORD=ADMIN
SUT_IP=blackbauhinia.openqanet.opensuse.org
SUT_NETDEVICE=em1
IPMI_SOL_PERSISTENT_CONSOLE=1
IPMI_BACKEND_MC_RESET=1
Updated by okurz over 3 years ago
I suggest to add to "rebel" which runs without apparmor validation which likely prevents the ipmi backend to work
Updated by nicksinger over 3 years ago
- Status changed from In Progress to Feedback
- Assignee changed from nicksinger to Julie_CAO
Thanks for the suggestion, I moved the config to rebel as worker instance 5 - see: https://openqa.opensuse.org/admin/workers/382
I also added Julies ssh key from https://progress.opensuse.org/issues/93381#note-4 to all machines with
for i in aarch64 openqaworker1 openqaworker4 openqaworker7 power8 rebel imagetester; do echo $i && sshpass -p opensuse ssh root@$i "echo '$(cat .ssh/authorized_keys | grep "julie_caohe")' >> '/root/.ssh/authorized_keys'"; done
@Julie_CAO please test if the machine is working for you as expected and feel free to raise any problems and reassign back to me :)
Updated by Julie_CAO over 3 years ago
Hi @nicksinger, thank you for adding the machine to O3 network and openQA worker pool. Is the correct way to access the machine over rebel as the jump host from SUSE? My following procedure does not seem correct as password is needed.
jcao@linux-brmq:~/.ssh> cat config
Host ariel
HostName gate.opensuse.org
Port 2213
Host *.opensuse.org
ProxyCommand ssh -q -A -x ariel -W %h:%p
jcao@linux-brmq:~/.ssh> ssh -X rebel.opensuse.org
jcao@gate.opensuse.org's password:
Another question, is there a PXE server in O3 network? I'd like to install openSUSE Tumbleweed in each test, the command we used is:
/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210521
Updated by nicksinger over 3 years ago
- Status changed from Feedback to In Progress
- Assignee changed from Julie_CAO to nicksinger
Julie_CAO wrote:
Hi @nicksinger, thank you for adding the machine to O3 network and openQA worker pool. Is the correct way to access the machine over rebel as the jump host from SUSE? My following procedure does not seem correct as password is needed.
jcao@linux-brmq:~/.ssh> cat config
Host ariel
HostName gate.opensuse.org
Port 2213
Host *.opensuse.org
ProxyCommand ssh -q -A -x ariel -W %h:%pjcao@linux-brmq:~/.ssh> ssh -X rebel.opensuse.org
jcao@gate.opensuse.org's password:
Yes, sorry I forgot to create your user on gate.opensuse.org and just added your key to the root account. Now it is fixed and you should be able to ssh into ariel (gate.opensuse.org). From there you can ssh into root@rebel.openqanet.opensuse.org (we don't have users on workers, just root). Please feel free to ask here or in Rocket if you have further questions how this jump host stuff is done - it can be confusing :)
Hi Nick, I am still unable to log in 'ariel'. Is the correct ssh key used?
jcao@linux-brmq:~> ssh gate.opensuse.org
Password:
jcao@linux-brmq:~> ssh jcao@gate.opensuse.org
Password:
jcao@linux-brmq:~/.ssh> pwd
/home/jcao/.ssh
jcao@linux-brmq:~/.ssh> cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDI/mD7o5XnT6X6bVN+sZgFBkfAdA7/wisXOnGYSdhzIECK6OCxf06VD4rFjDoAJuhviixBeG0h8tu4Zu49qXiXBq3BsXUJdaNlHCBxmrF0oxqPKZRprF8tCn4dCFECVOfWNj4B51/9I7A76Qq+QkSxc3qSH2hzZBj1PIXBsCnb7YyUVSq3ZLWyZDY/ts1p5G+R7xWhgwunoaDM78RfTK2HZpXUtidLy5RVhp8wZM6vyOd8nKd5oWKxlD2T8IobkPHyLb3VuoGxhOqIj2mUYIBkxDs5qXmv3e17TMT2Lj2FuQCk/a912ozuHlJwPuI+zCKv86ealFTPLEUeXlBEcxtfiMIciD/LRHYYx7iRQ8HnJ7z9ld31G1xkwWwYfXscF4w+m0k8sgVWWLa69zyCXibbAfUARNIOLG2UDOEOniwj1xMeB1zEAobHi62nNLwjwn4PIYihz8LKo3zd+Alkwx4PhLn1X7ekAvU+Rq39A6+C1JyFS4I8lKkMclgrtb6c5DWS+8q3kDbykLCdgvZRVeWzmhaoANuSjDk9RqFABSdyWh4qWYBV/hRbrhYvOVsa5YDwvD76IkUu41lhk1fy3ouy2n7rtMg5QfCcIk8HgJuCiRwW+40UT9522UgBq+l4IhcjbU7W3osdzN1Q9H9KMcjvP9IOUNhgXFqgMb0zif6qqQ== julie_caohe@163.com
Another question, is there a PXE server in O3 network? I'd like to install openSUSE Tumbleweed in each test, the command we used is:
/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210521
Yes but as far as I understand it currently only supports Leap. I will check if I can enable TW installations on there.
Thank you, Nick. We need not an actual PXE entry for TW installation. The PXE server can support TW installation with boot parameters the way we do in OSD is ok.
Updated by livdywan over 3 years ago
- Due date changed from 2021-06-17 to 2021-07-02
Updated by okurz over 3 years ago
- Status changed from Workable to New
moving all tickets without size confirmation by the team back to "New". The team should move the tickets back after estimating and agreeing on a consistent size
Updated by nicksinger over 3 years ago
- Assignee changed from nicksinger to Julie_CAO
@Julie_CAO I checked the key again on ariel and everything looks like it should. Please use the command ssh ariel to login into that host. If you use "gate.opensuse.org" your config is not used and therefore it does not work.
Updated by Julie_CAO over 3 years ago
nicksinger wrote:
@Julie_CAO I checked the key again on
arieland everything looks like it should. Please use the commandssh arielto login into that host. If you use "gate.opensuse.org" your config is not used and therefore it does not work.
Thanks, Nick. ssh ariel works for me. I successfully login blackbauhinia over ssh on ariel.
So the only remained part is the PXE support. :-)
Updated by Julie_CAO over 3 years ago
Hi Nick,
Could you kindly give a general time when the PXE server would be ready? I will have a vacation leave in August, so I'd like to arrange my tasks ahead based on different situations.
Updated by nicksinger over 3 years ago
- Status changed from New to In Progress
- Assignee changed from Julie_CAO to nicksinger
I've added a kernel and initrd from the latest tw-netinstall ISO onto the PXE server on ariel. A first try didn't seem to work despite ariel serving the required files:
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: error 0 TFTP Aborted received from 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: failed sending /srv/tftpboot/pxelinux.0 to 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: sent /srv/tftpboot/pxelinux.0 to 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/44454c4c-3100-1059-8052-b9c04f463733 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/01-f4-02-70-b9-3a-66 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8700D not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8700 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A870 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A87 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: sent /srv/tftpboot/pxelinux.cfg/default to 192.168.112.13
I saw something hinting at syntax errors but the IPMI connection in the train is not good enough. Will check if I can make it work at home with a stable connection.
Updated by openqa_review over 3 years ago
- Due date set to 2021-08-11
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger over 3 years ago
- Status changed from In Progress to Blocked
I can't reach the machine nor the ipmi interface. I created an infra ticket now to check this machine: https://infra.nue.suse.com/Ticket/Display.html?id=193192
Updated by Julie_CAO over 3 years ago
Thank you for open the infra ticket, Nick. The machine is in SERV2, it was shutdown by me days ago when AC had a water leaking. But the ipmi is inaccessible either, It may have been implicated in the cooling system accident?
Updated by nicksinger over 3 years ago
- Status changed from Blocked to In Progress
Updated by nicksinger over 3 years ago
- Status changed from In Progress to Feedback
I've fixed the PXE setup in O3 and was able to start a tw installer. Please let me know if anything is not working as expected for you.
Updated by Julie_CAO over 3 years ago
Thank you, Nick.
I just tried with new PXE server from blackbauhinia-ipmi. It finds out the PXE server and the pxe menu shows up, TW can be installed with the 'openSUSE tumbleweed' entry. but when I press 'tab' to input the boot command below, then type 'enter', the screen went back to the pxe entry, No installation began or no any error popped up. Anything wrong?
/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210804/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210804/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210804
Updated by livdywan over 3 years ago
- Due date changed from 2021-08-11 to 2021-08-31
Updated by nicksinger over 3 years ago
- Status changed from Feedback to Resolved
Sorry for the long delay. I didn't notice your answer. I think you need to add the proper console parameter. Something like this should make the console show up over SOL: console=tty console=ttyS1,115200 (maybe ttyS0 depending on the machine). Please have a look what other IPMI tests on e.g. OSD supply as their command line:
[2021-07-28T20:31:29.995 CEST] [debug] <<< testapi::type_string(string=" Y2DEBUG=1 vga=791 video=1024x768 console=ttyS1,115200 linuxrc.log=/dev/ttyS1 linuxrc.core=/dev/ttyS1 linuxrc.debug=4,trace reboot_timeout=0 ", max_interval=4, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)
(from https://openqa.suse.de/tests/6599035/logfile?filename=autoinst-log.txt)
I'm resolving this now as this is an test issue and not caused by the PXE setup which apparently works for your case :) Feel free to reopen if you discover other issues
Updated by Julie_CAO over 3 years ago
- Due date changed from 2021-08-31 to 2021-08-11
nicksinger wrote:
Sorry for the long delay. I didn't notice your answer. I think you need to add the proper console parameter. Something like this should make the console show up over SOL:
console=tty console=ttyS1,115200(maybe ttyS0 depending on the machine). Please have a look what other IPMI tests on e.g. OSD supply as their command line:[2021-07-28T20:31:29.995 CEST] [debug] <<< testapi::type_string(string=" Y2DEBUG=1 vga=791 video=1024x768 console=ttyS1,115200 linuxrc.log=/dev/ttyS1 linuxrc.core=/dev/ttyS1 linuxrc.debug=4,trace reboot_timeout=0 ", max_interval=4, wait_screen_changes=0, wait_still_screen=0, timeout=30, similarity_level=47)(from https://openqa.suse.de/tests/6599035/logfile?filename=autoinst-log.txt)
I'm resolving this now as this is an test issue and not caused by the PXE setup which apparently works for your case :) Feel free to reopen if you discover other issues
Hi @Nick, I have to reopen this ticket as the PXE server does not work in O3 actually. Even with "console=tty console=ttyS1,115200", when I press 'tab' to input the boot command, then type 'enter', the screen went back to the pxe entry, No installation began on any console. It should not be console problem because it returns to the PXE menu.
Or let's try it as a SUT in O3, the 'boot_from_pxe' will type the correct command since the machine worked in OSD. Please help me add this SUT in O3 or point me what I shall do(it seems that I have no permission to login rebel).
WORKER_CLASS: 64bit-ipmi,64bit-ipmi-amd
IPMI_HOSTNAME: blackbauhinia-ipmi.openqanet.opensuse.org
IPMI_USER: ADMIN
IPMI_PASSWORD: ADMIN
SUT_IP: blackbauhinia.openqanet.opensuse.org
SUT_NETDEVICE: eno1
IPMI_SOL_PERSISTENT_CONSOLE: 1
IPMI_BACKEND_MC_RESET: 1
Updated by livdywan over 3 years ago
- Status changed from New to Feedback
@Julie_CAO In the future, please use Feedback when re-opening tickets.
Updated by nicksinger over 3 years ago
- Due date changed from 2021-08-11 to 2021-09-11
Updated by nicksinger over 3 years ago
- Status changed from Feedback to Blocked
I once again tried it out manually.
First, I set the bootdev:
ipmitool -I lanplus -C 3 -H blackbauhinia-ipmi.openqanet.opensuse.org -U ADMIN -P ADMIN chassis bootdev pxe
Attaching to the machine from ariel by using IPMI:
ipmitool -I lanplus -C 3 -H blackbauhinia-ipmi.openqanet.opensuse.org -U ADMIN -P ADMIN sol activate
After that it boots into the PXE menu where I just select "install-tumbleweed" and the kernel starts to load (and give me output on SOL). Just a small example from the beginning:
Loading linux/openSUSE/tumbleweed/linux.............
Loading linux/openSUSE/tumbleweed/initrd........................................
................................................................................
................ready.
[ 0.000000] Linux version 5.13.4-1-default (geeko@buildhost) (gcc (SUSE Linux) 11.1.1 20210625 [revision 62bbb113ae68a7e724255e17143520735bcb9ec9], GNU ld (GNU Binutils; openSUSE Tumbleweed) 2.36.1.20210326-4) #1 SMP Thu Jul 22 15:55:06 UTC 2021 (91a0cca)
[ 0.000000] Command line: initrd=linux/openSUSE/tumbleweed/initrd vga=normal console=tty0 console=ttyS1,115200 linemode=1 ignore_loglevel unknown_nmi_panic insecure=1 install=http://download.opensuse.org/tumbleweed/repo/oss/ nameserver=192.168.112.100 BOOT_IMAGE=linux/openSUSE/tumbleweed/linux
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format.
[ 0.000000] BIOS-provided physical RAM map:
So manually I can't reproduce your issue. However, as rebel is currently down (https://progress.opensuse.org/issues/97658) and oli indicated that other workers will most likely won't work due to apparmor I think we're currently blocked there to get the IPMI machine into o3 and I can't do further testing for now.
Updated by nicksinger over 3 years ago
- Blocked by action #97658: many (maybe all) jobs on rebel within o3 run into timeout_exceeded "setup exceeded MAX_SETUP_TIME" size:M added
Updated by Julie_CAO over 3 years ago
- Due date changed from 2021-09-11 to 2021-08-11
nicksinger wrote:
After that it boots into the PXE menu where I just select "install-tumbleweed" and the kernel starts to load (and give me output on SOL). Just a small example from the beginning:
Yes, the pxe entry worked for me either. But I meant that the command below(press 'tab' key to type the command the way boot_from_pxe module does in openqa) does not work:
/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830 console=ttyS1,115200
I'ed like to install the latest snapshot, such as 20210830 that day I tried. Press 'enter' after typing the command, pxe menu screen showed up, no any console output.
Updated by Julie_CAO over 3 years ago
- Due date changed from 2021-08-11 to 2021-09-11
Updated by nicksinger over 3 years ago
Ah, now I understand. Sorry for the confusion. Of course this can not work as we don't have the "/mnt/openqa" mountpoint on o3. I asked in our team chat for some help on what we could mount instead to always have the latest snapshot of TW available to boot from.
Updated by Julie_CAO over 3 years ago
nicksinger wrote:
Ah, now I understand. Sorry for the confusion. Of course this can not work as we don't have the "/mnt/openqa" mountpoint on o3. I asked in our team chat for some help on what we could mount instead to always have the latest snapshot of TW available to boot from.
Thank you, Nick. that's exactly what I want!
Updated by okurz over 3 years ago
- Due date changed from 2021-09-11 to 2021-09-17
please take a look into the blocking ticket #97658 which we consider still "urgent" and will exceed the due-date tomorrow.
Updated by livdywan about 3 years ago
- Due date deleted (
2021-09-17)
Still blocked by #97658 as I understand it, although Redmine won't show that for some reason.
Updated by nicksinger about 3 years ago
While #97658 is still blocked I try to figure out what to mount to get the most recent TW snapshot in the opensuse network. I asked in #opensuse-factory:
14:58 <sqozz> hey all! I'm currently looking for a way to enable openqa.opensuse.org to test the most recent TW snapshot on bare hardware. For this I'd need a mountpoint to the most recent snapshot reachable from within the opensuse-network. Any hints where I could find such thing?
and in #team-buildops in slack: https://suse.slack.com/archives/C02BX1X92HM/p1632144642253300
Updated by Julie_CAO about 3 years ago
Thank you nick for asking help from outside. I saw the thread in #team-buildops but the result seems not very optimistic. I am not familiar with http boot or EFI boot. Let's see if the factory folks can give more clue.
Updated by Julie_CAO about 3 years ago
Hi Nick, do you get any positive feedback from factory folks?
Updated by Julie_CAO about 3 years ago
Hi Nick,
regarding mounting to get lastest snapshot, we always run test this way in OSD. what's the difference between O3 and OSD? it sounds there are obstacles in O3 network. I want to know what the specific problem is, and then I'll manage to find a way to get it around on my side.
boot: /mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210830
Are there extracted installer DVD directories in O3 server? for example, "openqa.opensuse.org:/share/factory/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-SnapshotXXXX/", we have this path for SLE in OSD.
If the DVD is present, what we need is to mount the repo to our pxe server, right?
If the DVD is not present, we have to extract from openSUSE-Tumbleweed-DVD-x86_64-Snapshot20211008-Media.iso in O3?
I don't know how OSD implements these requirements. we hope O3 can provide the same functions as OSD, If it is not possible on O3, it is acceptable to have workaround.
Let me know what I can help on my side.
Updated by Julie_CAO about 3 years ago
Hi @okurz, Nick seems to have been out of work these days, can we manage to mount the TW snapshot in PXE server without Nick?
Updated by nicksinger about 3 years ago
- Status changed from Blocked to In Progress
Hello @Julie_CAO - sorry I was on a sick-leave yesterday and before didn't update the ticket here. I was able to boot files from download.opensuse.org directly which means you could always boot the latest snapshot. However, the architecture inside the o3 infrastructure is different. Therefore the tests would need to be adjusted accordingly. I will prepare a boot environment for you today so you can see how it would work. Afterwards you would need to adjust the tests.
Updated by openqa_review about 3 years ago
- Due date set to 2021-10-30
Setting due date based on mean cycle time of SUSE QE Tools
Updated by nicksinger about 3 years ago
- Status changed from In Progress to Feedback
So I created an IPXE config which offers several options to boot a machine. I tested it with blackbauhinia manually from ariel with the following command: ipmitool -I lanplus -C 3 -H blackbauhinia-ipmi.openqanet.opensuse.org sol activate. It always loads the most recent TW snapshot from download.opensuse.org over http since we don't have any mountpoint with the most recent release like on OSD.
I realized that the IPMI output is quite slow but still was able to start an installer. Please test it manually to see if you could adjust the tests accordingly or if anything is missing. I also attach the IPXE-script below so you can see with which options the installer gets started. Let me know if any option would need to be added.
#!ipxe
dhcp
set menu-timeout 5000
:start
menu iPXE boot menu for openqa.opensuse.org
item --gap -- ------------------------- Operating systems ------------------------------
item --key l leapS1 Boot openSUSE Leap15.3 from download.opensuse.org / ttyS1 (http)
item --key l leapS2 Boot openSUSE Leap15.3 from download.opensuse.org / ttyS2 (http)
item --key t tumbleweedS1 Boot openSUSE Tumbleweed from download.opensuse.org / ttyS1 (http)
item --key t tumbleweedS2 Boot openSUSE Tumbleweed from download.opensuse.org / ttyS2 (http)
item --gap -- ------------------------- Advanced options -------------------------------
item --key c config Configure settings
item shell Drop to iPXE shell
item reboot Reboot computer
item
item --key x exit Exit iPXE and continue BIOS boot
choose --timeout ${menu-timeout} --default exit selected || goto cancel
set menu-timeout 0
goto ${selected}
:leapS1
kernel http://download.opensuse.org/distribution/leap/15.3/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/leap/15.3/repo/oss/ console=ttyS1,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/leap/15.3/repo/oss/boot/x86_64/loader/initrd
boot
:leapS2
kernel http://download.opensuse.org/distribution/leap/15.3/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/leap/15.3/repo/oss/ console=ttyS2,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/leap/15.3/repo/oss/boot/x86_64/loader/initrd
boot
:tumbleweedS1
kernel http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/openSUSE-current/repo/oss/ console=ttyS1,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/initrd
boot
:tumbleweedS2
kernel http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/openSUSE-current/repo/oss/ console=ttyS2,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/initrd
boot
:cancel
echo You cancelled the menu, dropping you to a shell
:shell
echo Type 'exit' to get the back to the menu
shell
set menu-timeout 0
set submenu-timeout 0
goto start
:failed
echo Booting failed, dropping to shell
goto shell
:reboot
reboot
:exit
exit
:config
config
goto start
Updated by Julie_CAO about 3 years ago
Hi Nick,
Thank you for setting it up, I'll try and adapt my test.
I reinstalled my laptop with Leap 15.3 on a new disk and the ssh keys changed. I am unable to login ariel to
access the SUT to try the new PXE now, could you help re-add my keys? sorry for bringing you the trouble.
jcao@localhost:~/.ssh> cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIvkuGQu2YkXbCMNnmtG/cyFLqUlYgrhJ0xFu0p1xjcY9gGtIlPzXZoEhNA5xxbsa0ydlW5GW/Msr+0zktNYzMksXKLQO4NQ0yAYrW5zaGpHZqrMImGCSe4JW0wG1w06Vpfj0IziqYpALdLch8sA+WEXMfe+58qW+1q+QsLL1rSvZ5ZlD18Z3PIKcMzJTgzxlSyInjqUwDzWaBnrBc8u4RfbGIZpq3USowDjX0VEGAJ+FEn6pWLbyfMkKaEQle0L5lSWLPxOPVr2XNVg1Fzy9eELfnqU23wMQdl8yGQU1WmHJrWV/jurcMbemMXo3od62bTIMO6sxooBp9pUSQFB2W9xdzmMZxJaaEQWHEe+LmVGNGogUqCbIPDstNewb8A5StavJh4iXQUIlVXsWuVJTCXzPqqAfymGR4au4UK1uggilB/rFTiYgVWM7eEO/Damb67qYAsCwh5/PTAFpWbIUnOZ/OVZ+mhMv6oXjbWe9NwSG4Kg2IAJG3Sic3gj8aFQnrrQMcUUE53QCN+akre2K61sI9DpQ8OjrbLeDNytCWCg5SpyzDbZHfpS9q2DEjRiN7BcKkijPkWzVV3cRpMPhH614CDbo4rCHs1clBKk2THuOtqGOnFVB4igNpiPdsfte/okLOBix+kbYckCmbi71JvWeyG0/R5h5KtDMPohI46w== julie_caohe@163.com
Updated by Julie_CAO about 3 years ago
- Status changed from Feedback to In Progress
nicksinger wrote:
I realized that the IPMI output is quite slow but still was able to start an installer. Please test it manually to see if you could adjust the tests accordingly or if anything is missing. I also attach the IPXE-script below so you can see with which options the installer gets started. Let me know if any option would need to be added.
#!ipxe dhcp set menu-timeout 5000
HI Nick, the ipmi sol console on my side is so slow. The machine enters BIOS boot right away after the iPXE menu shows up so that I have not the chance to check these configurations. Could you set the timeout longer for me to check and for openqa to match needles?
:tumbleweedS1
kernel http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/linux usessh=1 sshpassword=linux network=1 install=http://download.opensuse.org/distribution/openSUSE-current/repo/oss/ console=ttyS1,115200n8 root=/dev/ram0 initrd=initrd textmode=1
initrd http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/initrd
boot
I have not tried the command but I compared with the boot command in OSD. there are some differences:
- "usessh=1 sshpassword=linux". In OSD, it is "ssh=1 sshpassword=nots3cr3t". So 'ssh=' is SLE and 'usessh=xx' in openSUSE? And the what the default password in O3, nots3cr3t or linux? our tests typically use the openqa default password.
- we have some extra variables in the boot command in OSD, "plymouth.enable=0 Y2DEBUG=1 vga=791 video=1024x768 linuxrc.log=/dev/ttyS1 linuxrc.core=/dev/ttyS1 linuxrc.debug=4,trace reboot_timeout=0 kernel.softlockup_panic=1 vt.color=0x07", actually I am not sure if they are required for openSUSE in O3, Could you help take a look and add some of them if they are needed?
Updated by mkittler about 3 years ago
- Due date changed from 2021-10-30 to 2021-11-06
Increasing the due date after mentioning the ticket in the chat (@nicksinger is not available today).
Updated by nicksinger about 3 years ago
- Status changed from In Progress to Feedback
I've bumped the timeout to 50000, changed the default PW to nots3cr3t to make it easier to run tests and also added the other parameters too. It should not hurt anyway. BTW: rebel is also back in o3 so you might be able to clone a test on it to test it. But also please test manually if everything works as you would expect it.
Regarding the slow IPMI: I recommend that you try to execute your virtualization tests with the help of nested virtualization. This would really ease the setup and require less manual work to have machines moved between OSD and o3
Updated by Julie_CAO about 3 years ago
- Due date changed from 2021-11-06 to 2021-10-30
- Status changed from Feedback to In Progress
nicksinger wrote:
I've bumped the timeout to
50000,
thank you, Nick. 50000 seems to be 50 seconds. I can see the blue menu and move the focus up and down to choose the entry now.
changed the default PW to nots3cr3t to make it easier to run tests and also added the other parameters too. It should not hurt anyway.
for the Tumbleweed entry:
:tumbleweedS1
kernel http://download.opensuse.org/distribution/openSUSE-current/repo/oss/boot/x86_64/loader/linux ...
It is the installer of OpenSUSE Leap 15.3 instead of OpenSUSE Tumbleweed. Can we get a latest tumbleweed DVD URL here?
BTW: rebel is also back in o3 so you might be able to clone a test on it to test it. But also please test manually if everything works as you would expect it.
I will continue with manual test to ensure everythings works then we add the SUT to be a worker of O3.
Regarding the slow IPMI: I recommend that you try to execute your virtualization tests with the help of nested virtualization. This would really ease the setup and require less manual work to have machines moved between OSD and o3
Because SLE virtualization product does not promise to support nexted virtualization, I think so does openSUSE. Our virtualization tests all run on barel metal machines according to our test plan.
Updated by Julie_CAO about 3 years ago
- Due date changed from 2021-10-30 to 2021-11-06
Updated by livdywan about 3 years ago
- Due date changed from 2021-11-06 to 2021-11-19
How about a due date that lies in the future. For example end of this week, assuming you are motived to finish it this week :-)
Updated by Julie_CAO about 3 years ago
cdywan wrote:
How about a due date that lies in the future. For example end of this week, assuming you are motived to finish it this week :-)
I accidently change it in my comment, so I just changed it back to its original date. Your team determines the actual due date :-)
Updated by okurz about 3 years ago
We clarified that "openSUSE-Current" is actually Leap, not Tumbleweed. So please find the corresponding "Current" links from http://download.opensuse.org/tumbleweed/ for Tumbleweed.
Updated by nicksinger about 3 years ago
- Due date changed from 2021-11-19 to 2021-11-26
@Julie_CAO I've updated the entries once again to the tumbleweed repos posted by @okurz - please let me know if it works :)
Updated by nicksinger about 3 years ago
- Due date changed from 2021-11-26 to 2021-12-01
Updated by Julie_CAO about 3 years ago
nicksinger wrote:
@Julie_CAO I've updated the entries once again to the tumbleweed repos posted by @okurz - please let me know if it works :)
It works. thank you, Nick.
There is an exising ipxe install module in our openqa test git repo, which is maintained by Petr C and has been used in kernel tests. I talked with him and read through the test script. I figured out it can be used in our virtualization tests in O3 by minor changes both from iPXE server side and test script itself. So I still need your help with any of the following two options:
- option 1: allow to POST the boot commad to the ipxe configuration file in iPXE server over http. ie. the test script will write the *.ipxe in each run. The test script is like
my $url = "$http_server/v1/bootscript/script.ipxe/$ip"; //any location is ok for me. It can be set by a openqa setting
HTTP::Tiny->new->request('POST', $url, {content => $bootscript, headers => {'content-type' => 'text/plain'}});
The advantage is that the install commands can be custermized in each test. but it needs the http server support(enable POST and permissions).
- option 2: use the TW install command directly rather than the menu selection. Because the menu does not always show up with ipmi connections, to run the install command directly is more stable and easier than to move focus to choose the menu entry by needles. I had a successful installation test in my own openqa with following ipxe configuration file:
#!ipxe
echo ++++++++++++++++++++++++++++++++++++++++++
echo ++++++++++++ openQA ipxe boot ++++++++++++
echo ++++++++++++++++++++++++++++++++++++++++++
kernel http://download.opensuse.org/tumbleweed/repo/oss/boot/x86_64/loader/linux install=http://download.opensuse.org/tumbleweed/repo/oss plymouth.enable=0 Y2DEBUG=1 vga=791 video=1024x768 console=ttyS1,115200 linuxrc.log=/dev/ttyS1 linuxrc.core=/dev/ttyS1 linuxrc.debug=4,trace reboot_timeout=0 ssh=1 sshpassword=nots3cr3t kernel.softlockup_panic=1 vt.color=0x07
initrd http://download.opensuse.org/tumbleweed/repo/oss/boot/x86_64/loader/initrd
boot
Are these options capable from server side? In terms of test implementation only, I think option 2 is ok at present. but in future option 1 would be better as other tests may use ipxe install as well.
In addition, I'd like to know the URL of the ipxe config file to read the content in our test for debugging purpose.
Updated by nicksinger about 3 years ago
Julie_CAO wrote:
There is an exising ipxe install module in our openqa test git repo, which is maintained by Petr C and has been used in kernel tests. I talked with him and read through the test script. I figured out it can be used in our virtualization tests in O3 by minor changes both from iPXE server side and test script itself. So I still need your help with any of the following two options:
- option 1: allow to POST the boot commad to the ipxe configuration file in iPXE server over http. ie. the test script will write the *.ipxe in each run. The test script is like
Michi from the kernel team also wrote a similar daemon for such purpose: https://github.com/frankenmichl/baremetal_support - maybe this is something we can aim for in the future but for now I would go with option 2.
- option 2: use the TW install command directly rather than the menu selection. Because the menu does not always show up with ipmi connections, to run the install command directly is more stable and easier than to move focus to choose the menu entry by needles. I had a successful installation test in my own openqa with following ipxe configuration file:
#!ipxe echo ++++++++++++++++++++++++++++++++++++++++++ echo ++++++++++++ openQA ipxe boot ++++++++++++ echo ++++++++++++++++++++++++++++++++++++++++++ kernel http://download.opensuse.org/tumbleweed/repo/oss/boot/x86_64/loader/linux install=http://download.opensuse.org/tumbleweed/repo/oss plymouth.enable=0 Y2DEBUG=1 vga=791 video=1024x768 console=ttyS1,115200 linuxrc.log=/dev/ttyS1 linuxrc.core=/dev/ttyS1 linuxrc.debug=4,trace reboot_timeout=0 ssh=1 sshpassword=nots3cr3t kernel.softlockup_panic=1 vt.color=0x07 initrd http://download.opensuse.org/tumbleweed/repo/oss/boot/x86_64/loader/initrd bootAre these options capable from server side? In terms of test implementation only, I think option 2 is ok at present. but in future option 1 would be better as other tests may use ipxe install as well.
I've deployed a tw.ipxe.pxe on ariel which just loads your script as posted above.
In addition, I'd like to know the URL of the ipxe config file to read the content in our test for debugging purpose.
Unfortunately the scripts are build into ipxe itself and cannot be read back into clear text. Should we create some kind of git repository with these scripts?
Updated by okurz about 3 years ago
- Due date changed from 2021-12-01 to 2022-01-31
- Status changed from In Progress to Feedback
Updated by Julie_CAO about 3 years ago
nicksinger wrote:
I've deployed a tw.ipxe.pxe on ariel which just loads your script as posted above.
Thank you, Nick. Option 2 is ok. Have you configured the blackbauhinia on DHCP server side to point to tw.ipxe.pxe? I just tried a reboot, but it went to the original iPXE menu.
In addition, I'd like to know the URL of the ipxe config file to read the content in our test for debugging purpose.
Unfortunately the scripts are build into ipxe itself and cannot be read back into clear text. Should we create some kind of git repository with these scripts?
I just need to know the exact commands to install the machine in test especially in case of failures. Any location is ok for me, a git repo is sure ok but we may need it simpler, such as a file in ariel which I can view.
Updated by Julie_CAO about 3 years ago
- Status changed from Feedback to In Progress
Updated by Julie_CAO about 3 years ago
Would you please make another change to the worker.ini configuration on rebel for blackbauhinia?
remove SUT_NETDEVICE=xx as I'd like to set it in test suite level.
Updated by Julie_CAO about 3 years ago
there are 3 workers listed in O3. I am a little confused which one should be used in virtualization tests.
* openqaworker7:20 openqaworker7 64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia x86_64 Offline
* openqaworker7_container:105 openqaworker7_container 64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia x86_64 Offline
* rebel:5 rebel 64bit-ipmi_rebel,64bit-ipmi-large-mem_rebel,64bit-ipmi-amd_rebel,blackbauhinia_rebel x86_64 Idle
Updated by Julie_CAO almost 3 years ago
Hi @nicksinger, are you back from the holiday? may you had a nice vocation! Could you help fix the machine ipxe pointer and adjust the worker settings in O3?
Updated by nicksinger almost 3 years ago
- Status changed from In Progress to Feedback
I've added a new entry in our /etc/dnsmasq.d/pxeboot.conf:
pxe-service=x86PC, "tw-over-ipmi", tw.ipxe.bin, 1, 192.168.112.13
This instructs the pxe server to serve this tw.ipxe.bin file only to blackbauhinia. From a first test this seems to work and the system automatically boots into the TW installer. Unfortunately this also means that only booting TW now works (no menu anymore). But I think in this specific use-case this should be fine, right?
Updated by Julie_CAO almost 3 years ago
- File ipxe_screen.png ipxe_screen.png added
- Status changed from Feedback to In Progress
Hi @nicksinger, It is weird, it is still the old ipxe menu when I try to boot up blackbauhinia via IPMI. The ipxe menu is attached. Could you help take a look again?
One more request, could you help remove SUT_NETDEVICE=xx in worker.ini as I'd like to set it in test suite level.
And one question, there are 3 workers listed in O3. I am a little confused which one should be used in virtualization tests.
openqaworker7:20 openqaworker7 64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia
openqaworker7_container:105 openqaworker7_container 64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia
rebel:5 rebel 64bit-ipmi_rebel,64bit-ipmi-large-mem_rebel,64bit-ipmi-amd_rebel,blackbauhinia_rebel
Updated by nicksinger almost 3 years ago
Julie_CAO wrote:
Hi @nicksinger, It is weird, it is still the old ipxe menu when I try to boot up blackbauhinia via IPMI. The ipxe menu is attached. Could you help take a look again?
This is strange. I will take another look!
One more request, could you help remove
SUT_NETDEVICE=xxin worker.ini as I'd like to set it in test suite level.
Should be removed on rebel:5
And one question, there are 3 workers listed in O3. I am a little confused which one should be used in virtualization tests.
openqaworker7:20 openqaworker7 64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia openqaworker7_container:105 openqaworker7_container 64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia rebel:5 rebel 64bit-ipmi_rebel,64bit-ipmi-large-mem_rebel,64bit-ipmi-amd_rebel,blackbauhinia_rebel
Ah yes, sorry for this mess. Due to https://progress.opensuse.org/issues/97658 we setup a container replacement for that host (also see https://progress.opensuse.org/issues/97751). But rebel is back in the meantime so rebel:5 is the correct host. I will try if I can get rid of the other ones.
Updated by nicksinger almost 3 years ago
- Due date changed from 2022-01-31 to 2022-02-08
- Status changed from In Progress to Feedback
Unfortunately I can't understand how I can instruct dnsmasq to only serve the tw-only pxe binary to just your host. The only documented way I found is
pxe-service=x86PC, "tw-over-ipmi", tw.ipxe.bin, 1, 192.168.112.13
but this doesn't work apparently. I now have set it that every request gets served the tw.ipxe.bin file. But this will only work until someone wants to boot something else then just TW. I can leave it for now but would suggest that you build a openQA test which can properly select the right entry. openQA needs to interact with the slow SOL anyway.
Updated by Julie_CAO almost 3 years ago
- File ipxe_TFTP_error.png ipxe_TFTP_error.png added
- Status changed from Feedback to In Progress
nicksinger wrote:
Unfortunately I can't understand how I can instruct dnsmasq to only serve the tw-only pxe binary to just your host. The only documented way I found is
pxe-service=x86PC, "tw-over-ipmi", tw.ipxe.bin, 1, 192.168.112.13
@nicksinger, Thank you for spending time on the ipxe configurations. Our ipxe server is running dhcp service and the ipxe configuration file is on a http server. Our tests can work with them for ipxe bootup. I know little about dnsmasq, but I may spend some time in investigation.
but this doesn't work apparently. I now have set it that every request gets served the tw.ipxe.bin file. But this will only work until someone wants to boot something else then just TW. I can leave it for now but would suggest that you build a openQA test which can properly select the right entry. openQA needs to interact with the slow SOL anyway.
I just tried and the current ipxe is reporting error, see the picture attached.
Updated by livdywan almost 3 years ago
- Due date changed from 2022-02-08 to 2022-02-25
@nicksinger @Julie_CAO How about discussing this in Slack or Jitsi? That might get this moving a bit faster?
Updated by livdywan almost 3 years ago
- Subject changed from [O3]request to add an IPMI SUT to O3 to [O3]request to add an IPMI SUT to O3 size:M
- Description updated (diff)
Updated by okurz almost 3 years ago
- Due date deleted (
2022-02-25) - Status changed from In Progress to Resolved
@Julie_CAO @xlai we discussed the ticket in the weekly estimation meeting of SUSE QE Tools. With the latest results it is possible to access the bare-metal SUT from within openQA tests running on o3. The dnsmasq config for PXE boot customized for the test case specific might need further adjustements in the PXE config or within openQA test code itself. We think it is more feasible for you to follow-up with that in the related ticket #92380 and we call this ticket resolved with the newly added AC1 covered.
In case you need to change the config on o3 itself you are welcome to follow
https://progress.opensuse.org/projects/openqav3/wiki/#Accessing-the-o3-infrastructure
and we will support you with that.