Project

General

Profile

action #93381

[O3]request to add an IPMI SUT to O3

Added by Julie_CAO 2 months ago. Updated about 23 hours ago.

Status:
Blocked
Priority:
Normal
Assignee:
Target version:
Start date:
2021-06-02
Due date:
2021-08-11
% Done:

0%

Estimated time:

Description

Hi,

Virtualization team requested infra to move a IPMI SUT to vlan 663 from OSD for temporary use in O3. Now the infra is ready, refer to https://infra.nue.suse.com/SelfService/Display.html?id=189224.

the server has been added to racktables and was located in SRV2 not the labs.
https://racktables.suse.de/index.php?page=object&tab=default&object_id=14353
Both LAN port and BMC are untagged to Vlan662: openqa-worker.
662 f4:02:70:b9:3a:66 gi6 dynamic

Please check the mac @ on the DHCP server to get an IP@.

Now what should I do is to add the MAC of the machine to O3 dhcp server to get an IP? Is the O3 host the dhcp server? ie. request to add my ssh pub key to O3 and add the mac to /etc/dnsmasq.d/openqa.conf and /etc/hosts?(I got the idea from https://progress.opensuse.org/projects/openqav3/wiki/#Moving-worker-from-osd-to-o3) then access the IPMI machine over ssh by using the O3 server as the jump host?

Could someone from openqa infra team help? or guide me what to do. thanks!

History

#1 Updated by Julie_CAO 2 months ago

  • Subject changed from [O3]add a IPMI SUT to O3 to [O3]request to add an IPMI SUT to O3

#2 Updated by VANASTASIADIS 2 months ago

  • Target version set to Ready

#3 Updated by nicksinger 2 months ago

  • Status changed from New to In Progress
  • Assignee set to nicksinger

I checked ariel (the o3 host) and apparently it is the dhcp server and I see requests from this machine. I will take care of adding it.

#4 Updated by Julie_CAO 2 months ago

Thank you nick for taking care. I need two fixed IPs for both IPMI BMC and machine NIC.

my ssh key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDI/mD7o5XnT6X6bVN+sZgFBkfAdA7/wisXOnGYSdhzIECK6OCxf06VD4rFjDoAJuhviixBeG0h8tu4Zu49qXiXBq3BsXUJdaNlHCBxmrF0oxqPKZRprF8tCn4dCFECVOfWNj4B51/9I7A76Qq+QkSxc3qSH2hzZBj1PIXBsCnb7YyUVSq3ZLWyZDY/ts1p5G+R7xWhgwunoaDM78RfTK2HZpXUtidLy5RVhp8wZM6vyOd8nKd5oWKxlD2T8IobkPHyLb3VuoGxhOqIj2mUYIBkxDs5qXmv3e17TMT2Lj2FuQCk/a912ozuHlJwPuI+zCKv86ealFTPLEUeXlBEcxtfiMIciD/LRHYYx7iRQ8HnJ7z9ld31G1xkwWwYfXscF4w+m0k8sgVWWLa69zyCXibbAfUARNIOLG2UDOEOniwj1xMeB1zEAobHi62nNLwjwn4PIYihz8LKo3zd+Alkwx4PhLn1X7ekAvU+Rq39A6+C1JyFS4I8lKkMclgrtb6c5DWS+8q3kDbykLCdgvZRVeWzmhaoANuSjDk9RqFABSdyWh4qWYBV/hRbrhYvOVsa5YDwvD76IkUu41lhk1fy3ouy2n7rtMg5QfCcIk8HgJuCiRwW+40UT9522UgBq+l4IhcjbU7W3osdzN1Q9H9KMcjvP9IOUNhgXFqgMb0zif6qqQ== julie_caohe@163.com

Let me know if any information are needed.

#5 Updated by openqa_review 2 months ago

  • Due date set to 2021-06-17

Setting due date based on mean cycle time of SUSE QE Tools

#6 Updated by nicksinger about 2 months ago

I've added the machine on ariel to the dnsmasq config.
SUT can be reached under: blackbauhinia.openqanet.opensuse.org
IPMI can be reached under: blackbauhinia-ipmi.openqanet.opensuse.org

I've also added it as worker to openqaworker7 with the following config in /etc/openqa/workers.ini:

[20]
WORKER_CLASS=64bit-ipmi,64bit-ipmi-large-mem,64bit-ipmi-amd,blackbauhinia
IPMI_HOSTNAME=blackbauhinia-ipmi.openqanet.opensuse.org
IPMI_USER=ADMIN
IPMI_PASSWORD=ADMIN
SUT_IP=blackbauhinia.openqanet.opensuse.org
SUT_NETDEVICE=em1
IPMI_SOL_PERSISTENT_CONSOLE=1
IPMI_BACKEND_MC_RESET=1

#7 Updated by okurz about 2 months ago

I suggest to add to "rebel" which runs without apparmor validation which likely prevents the ipmi backend to work

#8 Updated by nicksinger about 2 months ago

  • Status changed from In Progress to Feedback
  • Assignee changed from nicksinger to Julie_CAO

Thanks for the suggestion, I moved the config to rebel as worker instance 5 - see: https://openqa.opensuse.org/admin/workers/382
I also added Julies ssh key from https://progress.opensuse.org/issues/93381#note-4 to all machines with

for i in aarch64 openqaworker1 openqaworker4 openqaworker7 power8 rebel imagetester; do echo $i && sshpass -p opensuse ssh root@$i "echo '$(cat .ssh/authorized_keys | grep "julie_caohe")' >> '/root/.ssh/authorized_keys'"; done

@Julie_CAO please test if the machine is working for you as expected and feel free to raise any problems and reassign back to me :)

#9 Updated by Julie_CAO about 2 months ago

Hi nicksinger, thank you for adding the machine to O3 network and openQA worker pool. Is the correct way to access the machine over rebel as the jump host from SUSE? My following procedure does not seem correct as password is needed.

jcao@linux-brmq:~/.ssh> cat config
Host ariel
HostName gate.opensuse.org
Port 2213
Host *.opensuse.org
ProxyCommand ssh -q -A -x ariel -W %h:%p

jcao@linux-brmq:~/.ssh> ssh -X rebel.opensuse.org
jcao@gate.opensuse.org's password:

Another question, is there a PXE server in O3 network? I'd like to install openSUSE Tumbleweed in each test, the command we used is:
/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210521

#10 Updated by nicksinger about 2 months ago

  • Status changed from Feedback to In Progress
  • Assignee changed from Julie_CAO to nicksinger

Julie_CAO wrote:

Hi nicksinger, thank you for adding the machine to O3 network and openQA worker pool. Is the correct way to access the machine over rebel as the jump host from SUSE? My following procedure does not seem correct as password is needed.

jcao@linux-brmq:~/.ssh> cat config
Host ariel
HostName gate.opensuse.org
Port 2213
Host *.opensuse.org
ProxyCommand ssh -q -A -x ariel -W %h:%p

jcao@linux-brmq:~/.ssh> ssh -X rebel.opensuse.org
jcao@gate.opensuse.org's password:

Yes, sorry I forgot to create your user on gate.opensuse.org and just added your key to the root account. Now it is fixed and you should be able to ssh into ariel (gate.opensuse.org). From there you can ssh into root@rebel.openqanet.opensuse.org (we don't have users on workers, just root). Please feel free to ask here or in Rocket if you have further questions how this jump host stuff is done - it can be confusing :)

Hi Nick, I am still unable to log in 'ariel'. Is the correct ssh key used?
jcao@linux-brmq:~> ssh gate.opensuse.org
Password:

jcao@linux-brmq:~> ssh jcao@gate.opensuse.org
Password:

jcao@linux-brmq:~/.ssh> pwd
/home/jcao/.ssh
jcao@linux-brmq:~/.ssh> cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDI/mD7o5XnT6X6bVN+sZgFBkfAdA7/wisXOnGYSdhzIECK6OCxf06VD4rFjDoAJuhviixBeG0h8tu4Zu49qXiXBq3BsXUJdaNlHCBxmrF0oxqPKZRprF8tCn4dCFECVOfWNj4B51/9I7A76Qq+QkSxc3qSH2hzZBj1PIXBsCnb7YyUVSq3ZLWyZDY/ts1p5G+R7xWhgwunoaDM78RfTK2HZpXUtidLy5RVhp8wZM6vyOd8nKd5oWKxlD2T8IobkPHyLb3VuoGxhOqIj2mUYIBkxDs5qXmv3e17TMT2Lj2FuQCk/a912ozuHlJwPuI+zCKv86ealFTPLEUeXlBEcxtfiMIciD/LRHYYx7iRQ8HnJ7z9ld31G1xkwWwYfXscF4w+m0k8sgVWWLa69zyCXibbAfUARNIOLG2UDOEOniwj1xMeB1zEAobHi62nNLwjwn4PIYihz8LKo3zd+Alkwx4PhLn1X7ekAvU+Rq39A6+C1JyFS4I8lKkMclgrtb6c5DWS+8q3kDbykLCdgvZRVeWzmhaoANuSjDk9RqFABSdyWh4qWYBV/hRbrhYvOVsa5YDwvD76IkUu41lhk1fy3ouy2n7rtMg5QfCcIk8HgJuCiRwW+40UT9522UgBq+l4IhcjbU7W3osdzN1Q9H9KMcjvP9IOUNhgXFqgMb0zif6qqQ== julie_caohe@163.com

Another question, is there a PXE server in O3 network? I'd like to install openSUSE Tumbleweed in each test, the command we used is:
/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/linux initrd=/mnt/openqa/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot2021xxxx/boot/x86_64/loader/initrd install=http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20210521

Yes but as far as I understand it currently only supports Leap. I will check if I can enable TW installations on there.

Thank you, Nick. We need not an actual PXE entry for TW installation. The PXE server can support TW installation with boot parameters the way we do in OSD is ok.

#11 Updated by okurz about 1 month ago

  • Status changed from In Progress to Workable

#13 Updated by cdywan about 1 month ago

  • Due date changed from 2021-06-17 to 2021-07-02

#14 Updated by okurz 29 days ago

  • Status changed from Workable to New

moving all tickets without size confirmation by the team back to "New". The team should move the tickets back after estimating and agreeing on a consistent size

#15 Updated by cdywan 28 days ago

  • Due date deleted (2021-07-02)

#16 Updated by nicksinger 23 days ago

  • Assignee changed from nicksinger to Julie_CAO

@Julie_CAO I checked the key again on ariel and everything looks like it should. Please use the command ssh ariel to login into that host. If you use "gate.opensuse.org" your config is not used and therefore it does not work.

#17 Updated by Julie_CAO 23 days ago

nicksinger wrote:

@Julie_CAO I checked the key again on ariel and everything looks like it should. Please use the command ssh ariel to login into that host. If you use "gate.opensuse.org" your config is not used and therefore it does not work.

Thanks, Nick. ssh ariel works for me. I successfully login blackbauhinia over ssh on ariel.

So the only remained part is the PXE support. :-)

#18 Updated by Julie_CAO 15 days ago

Hi Nick,

Could you kindly give a general time when the PXE server would be ready? I will have a vacation leave in August, so I'd like to arrange my tasks ahead based on different situations.

#19 Updated by nicksinger 8 days ago

  • Status changed from New to In Progress
  • Assignee changed from Julie_CAO to nicksinger

I've added a kernel and initrd from the latest tw-netinstall ISO onto the PXE server on ariel. A first try didn't seem to work despite ariel serving the required files:

Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: error 0 TFTP Aborted received from 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: failed sending /srv/tftpboot/pxelinux.0 to 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: sent /srv/tftpboot/pxelinux.0 to 192.168.112.13
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/44454c4c-3100-1059-8052-b9c04f463733 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/01-f4-02-70-b9-3a-66 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8700D not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8700 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A870 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A87 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A8 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0A not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C0 not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: file /srv/tftpboot/pxelinux.cfg/C not found
Jul 27 08:10:07 ariel dnsmasq-tftp[1999]: sent /srv/tftpboot/pxelinux.cfg/default to 192.168.112.13

I saw something hinting at syntax errors but the IPMI connection in the train is not good enough. Will check if I can make it work at home with a stable connection.

#20 Updated by openqa_review 7 days ago

  • Due date set to 2021-08-11

Setting due date based on mean cycle time of SUSE QE Tools

#21 Updated by nicksinger 5 days ago

  • Status changed from In Progress to Blocked

I can't reach the machine nor the ipmi interface. I created an infra ticket now to check this machine: https://infra.nue.suse.com/Ticket/Display.html?id=193192

#22 Updated by Julie_CAO about 23 hours ago

Thank you for open the infra ticket, Nick. The machine is in SERV2, it was shutdown by me days ago when AC had a water leaking. But the ipmi is inaccessible either, It may have been implicated in the cooling system accident?

Also available in: Atom PDF