QA » openQA Project » openQA Infrastructure

Improve QA related server room management, network topology and configuration

Motivation¶

Different ideas to improve, from #102650#note-25

Acceptance criteria¶

• AC1: All machines within NUE Server Room 2 : NUE-SRV2-B : Rack 1-4 have at least one port with MAC address configured (should be BMC and machine-specific interface)
• AC2: We have a best-practice documented for adding new machines / keeping existing entries up-to-date

Suggestions¶

1. Having at least the MAC address for each machine is helpful to debug. We checked holmes.qa.suse.de as the first non-production machine in NUE-SRV2-B-Rack-1. Using IPMI credentials from https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls we could login to https://sp.holmes.qa.suse.de/ (equivalent to https://holmes-sp.qa.suse.de/ , CNAME entry in DNS). Nice surprise we could find that the HMC of holmes knows (likely from SNMP) on which switch port and switch mac address it is connected. We crosschecked that by looking into the configuration ssh interface of the switch -> Do the same exercise manually or semi-automated or fully scripted for all machines and update racktables accordingly
2. https://wiki.racktables.org/index.php/RackTablesUserGuide#SNMP_Sync says that racktables can get information from network switches automatically, sounds nice and worthwhile to explore deeper. We found that SNMP lookup in racktables works nicely, e.g. on qanet15nue.qa.suse.de, v1, public. With that we can update configuration in racktables. Maybe we can allow hosts to set the switch with more SNMP stuff like port description and then we read that port description into racktables over the SNMP sync function?
3. Evaluate https://github.com/rvojcik/rt-server-client if it can be used to keep Racktables networking information up to date automatically
4. Use https://gitlab.suse.de/nicksinger/network-scripts/-/blob/main/find_mac.py to find switch ports by the mac address
5. Research what is the best approach based on the above tooling, document it, e.g. on https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs