Project

General

Profile

Actions

action #110521

closed

Improve QA related server room management, network topology and configuration size:M

Added by okurz almost 2 years ago. Updated 9 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
-
Target version:
Start date:
2022-05-02
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Improve QA related server room management, network topology and configuration

Motivation

Different ideas to improve, from #102650#note-25

Acceptance criteria

  • AC1: All machines within NUE Server Room 2 : NUE-SRV2-B : Rack 1-4 have at least one port with MAC address configured (should be BMC and machine-specific interface)
  • AC2: We have a best-practice documented for adding new machines / keeping existing entries up-to-date

Suggestions

  1. Having at least the MAC address for each machine is helpful to debug. We checked holmes.qa.suse.de as the first non-production machine in NUE-SRV2-B-Rack-1. Using IPMI credentials from https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls we could login to https://sp.holmes.qa.suse.de/ (equivalent to https://holmes-sp.qa.suse.de/ , CNAME entry in DNS). Nice surprise we could find that the HMC of holmes knows (likely from SNMP) on which switch port and switch mac address it is connected. We crosschecked that by looking into the configuration ssh interface of the switch -> Do the same exercise manually or semi-automated or fully scripted for all machines and update racktables accordingly
  2. https://wiki.racktables.org/index.php/RackTablesUserGuide#SNMP_Sync says that racktables can get information from network switches automatically, sounds nice and worthwhile to explore deeper. We found that SNMP lookup in racktables works nicely, e.g. on qanet15nue.qa.suse.de, v1, public. With that we can update configuration in racktables. Maybe we can allow hosts to set the switch with more SNMP stuff like port description and then we read that port description into racktables over the SNMP sync function?
  3. Evaluate https://github.com/rvojcik/rt-server-client if it can be used to keep Racktables networking information up to date automatically
  4. Use https://gitlab.suse.de/nicksinger/network-scripts/-/blob/main/find_mac.py to find switch ports by the mac address
  5. Research what is the best approach based on the above tooling, document it, e.g. on https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs

Related issues 1 (0 open1 closed)

Copied from openQA Infrastructure - action #109746: Improve QA related server room management, consistent naming and tagging size:MResolvedokurz

Actions
Actions #1

Updated by okurz almost 2 years ago

  • Copied from action #109746: Improve QA related server room management, consistent naming and tagging size:M added
Actions #2

Updated by okurz almost 2 years ago

  • Subject changed from Improve QA related server room management, network topology and configuration to Improve QA related server room management, network topology and configuration size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #3

Updated by okurz almost 2 years ago

  • Target version changed from Ready to future
Actions #4

Updated by okurz about 1 year ago

  • Tags set to infra
Actions #5

Updated by okurz 9 months ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz
  • Target version changed from future to Ready

We have reviewed all QA related machines and made sure that there should be at least a MAC address for each where feasible to do.

I have documented all relevant ideas as best practices in https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs#Inventory_management_best_practices

Actions

Also available in: Atom PDF