Project

General

Profile

action #110521

openQA Project - coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

Improve QA related server room management, network topology and configuration size:M

Added by okurz about 2 months ago. Updated 27 days ago.

Status:
Workable
Priority:
Low
Assignee:
-
Target version:
Start date:
2022-05-02
Due date:
% Done:

0%

Estimated time:

Description

Improve QA related server room management, network topology and configuration

Motivation

Different ideas to improve, from #102650#note-25

Acceptance criteria

  • AC1: All machines within NUE Server Room 2 : NUE-SRV2-B : Rack 1-4 have at least one port with MAC address configured (should be BMC and machine-specific interface)
  • AC2: We have a best-practice documented for adding new machines / keeping existing entries up-to-date

Suggestions

  1. Having at least the MAC address for each machine is helpful to debug. We checked holmes.qa.suse.de as the first non-production machine in NUE-SRV2-B-Rack-1. Using IPMI credentials from https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls we could login to https://sp.holmes.qa.suse.de/ (equivalent to https://holmes-sp.qa.suse.de/ , CNAME entry in DNS). Nice surprise we could find that the HMC of holmes knows (likely from SNMP) on which switch port and switch mac address it is connected. We crosschecked that by looking into the configuration ssh interface of the switch -> Do the same exercise manually or semi-automated or fully scripted for all machines and update racktables accordingly
  2. https://wiki.racktables.org/index.php/RackTablesUserGuide#SNMP_Sync says that racktables can get information from network switches automatically, sounds nice and worthwhile to explore deeper. We found that SNMP lookup in racktables works nicely, e.g. on qanet15nue.qa.suse.de, v1, public. With that we can update configuration in racktables. Maybe we can allow hosts to set the switch with more SNMP stuff like port description and then we read that port description into racktables over the SNMP sync function?
  3. Evaluate https://github.com/rvojcik/rt-server-client if it can be used to keep Racktables networking information up to date automatically
  4. Use https://gitlab.suse.de/nicksinger/network-scripts/-/blob/main/find_mac.py to find switch ports by the mac address
  5. Research what is the best approach based on the above tooling, document it, e.g. on https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs

Related issues

Copied from openQA Infrastructure - action #109746: Improve QA related server room management, consistent naming and tagging size:MWorkable

History

#1 Updated by okurz about 2 months ago

  • Copied from action #109746: Improve QA related server room management, consistent naming and tagging size:M added

#2 Updated by okurz about 2 months ago

  • Subject changed from Improve QA related server room management, network topology and configuration to Improve QA related server room management, network topology and configuration size:M
  • Description updated (diff)
  • Status changed from New to Workable

#3 Updated by okurz 27 days ago

  • Target version changed from Ready to future

Also available in: Atom PDF