QA » openQA Project » openQA Infrastructure

quick update here: we trashed several servers in the big qa lab. These machines are tagged with the "decommissioned" tag and can be seen here: https://racktables.suse.de/index.php?page=rack&rack_id=15356 and here https://racktables.suse.de/index.php?page=row&row_id=934

I got approached with the request to merge the small lab into the big lab. This will be done by me and @mgriessmeier on Monday the 1st of February. Planned move date is the 1st of March but Frankencampus will most likely be not ready at that date. Therefore a final move date is not set yet.

We need to move our big QA lab into SRV2. Our current plan is that we collect the required specs (HU’s, power sockets, network sockets) before Wed 16th. On Wed we will - together with gschlotter - take a look into SRV2 to evaluate our space and remove unnecessary old QA hardware from Rack 1 (https://racktables.suse.de/index.php?page=rack&rack_id=516). After this is done we plan to execute the final move of all machines between 22.-24. of February.

I did the "inventory" today and have them written down in my (paper) notebook. I now need to consolidate them and figure out what is really necessary to move

nicksinger wrote:

We need to move our big QA lab into SRV2. Our current plan is that we collect the required specs (HU’s, power sockets, network sockets) before Wed 16th. On Wed we will - together with gschlotter - take a look into SRV2 to evaluate our space and remove unnecessary old QA hardware from Rack 1 (https://racktables.suse.de/index.php?page=rack&rack_id=516). After this is done we plan to execute the final move of all machines between 22.-24. of February.

• @nicksinger @mgriessmeier Hello Nick and Mattias, most of virtualization ipmi test machines are located in Server room "NUE-3.2.16". Is this the one that is planned to be moved in your above plan? If yes, given that 15sp4 public beta milestone will come at 2022-02-24 and the priority of this task is low, can we ask you to postpone the lab movement to one week later? Recently, after one of our machine was moved from one lab to another, it can't work stablely any more in regard of ipmi sol console. We have to disable the worker. See poo#106056 for more details. So we really worry about our machine stability after they are moved. Is it okay to postpone the movement? @maritawerner @jstehlik @waynechen55 FYI.

xlai wrote:

nicksinger wrote:

We need to move our big QA lab into SRV2. Our current plan is that we collect the required specs (HU’s, power sockets, network sockets) before Wed 16th. On Wed we will - together with gschlotter - take a look into SRV2 to evaluate our space and remove unnecessary old QA hardware from Rack 1 (https://racktables.suse.de/index.php?page=rack&rack_id=516). After this is done we plan to execute the final move of all machines between 22.-24. of February.

• @nicksinger @mgriessmeier Hello Nick and Mattias, most of virtualization ipmi test machines are located in Server room "NUE-3.2.16". Is this the one that is planned to be moved in your above plan? If yes, given that 15sp4 public beta milestone will come at 2022-02-24 and the priority of this task is low, can we ask you to postpone the lab movement to one week later? Recently, after one of our machine was moved from one lab to another, it can't work stablely any more in regard of ipmi sol console. We have to disable the worker. See poo#106056 for more details. So we really worry about our machine stability after they are moved. Is it okay to postpone the movement? @maritawerner @jstehlik @waynechen55 FYI.

unfortunately it is not possible to postpone the move, as the physical connection to the lab will be cut off on 28th of february.
The initial plan was to do that until end of march, but recent concerns regarding common criteria, require us to move now...

We will try our best to keep the impact as low as possible

@mgriessmeier @nicksinger , thanks for the reply. Now I understand better the urgency of this move and you are aware of public beta and trying best to decrease the impact. We agree with it. Thanks a lot for the efforts!

nicksinger wrote:

I also want to use this opportunity to emphasize again that our labs where never considered stable server rooms.

I fully support this statement. I would like to use this opportunity to remind that while we are doing our best to support testing on bare-metal, virtualized infrastructure will likely always be more stable and scalable. And the according hardware for that is situated in the production-grade SRV1 server room, at least for x86_64, s390x.

Also adding the most recent update regarding the move here:

=== qsf-cluster VMs ===
running on seth & osiris - have been moved today, all VMs are running smoothly again - please let us know if you see any issues with them.

=== qamaster VMs ===
physical machine has been moved, all VMs and services are up and running

=== qanet.qa.suse.de ===
core of 10.162.x.x subnet - providing DNS/DHCP and PXE - was moved and all services are up and running

=== grenache.qa.suse.de ===
mainly working as jumphost in openQA for different backends, e.g. s390x - was moved and all openqa services are up and running

=== Virtualization testing servers ===
mainly IPMI hosts for openQA, we have moved gonzo, fozzie, scooter, kermit & quinn. all machines are reachable over BMC and should cause no problems.
I also just checked openQA and jobs are working as expected.

For tomorrow we plan to move the rest of the machines, mainly consisting of the Kernel test machines and some other leftovers. Given the experience from today, the impact on (openQA) testing is considered low.

Additionally we plan to move the remaining workstations to an office in 2nd floor. They will be connected to 10.160.x.x subnet (SUSE R&D) if you don't need them in VLAN 12 (10.162.x.x aka qanet). Please make sure to power them down before 10am, we will try to contact you in any case before we shut them down physically.

As long as racktables still show QA labs I would say we are not done here :)

Walkthrough to current information in racktables with nsinger

1. racktable entries for old QA labs still exist. After completing NUE-SRV2-B-3+NUE-SRV2-B-4 nsinger plans to revisit entries from old QA labs and move to virtual scrap racks
2. The "common name" should be including a suffix .qa.suse.de, this is what we have in most machines. We have that added in some machines rack 1 already
3. https://wiki.racktables.org/index.php/RackTablesUserGuide#SNMP_Sync says that racktables can get information from network switches automatically, sounds nice and worthwhile to explore deeper
4. There are some differing "tags", e.g. the "Usage Type" which can be "Testing"/"Development"/"Production". We suggest to avoid "Development" and use "Production" for machines that we expect to be "mostly up", e.g. qanet, qamaster, etc., or anything that is maintained by SUSE QE Tools. All bare-metal testing machines should then be "Testing"
5. We wonder about if grenache-4 and grenache-5 even still exist. Also what about grenache-2, grenache-3? We tried to access an HMC but neither powerhmc1.arch.suse.de nor powerhmc2.arch.suse.de are reachable. We tried over novalink on grenache. With command pvmctl lpar list we found grenache-1 through grenache-8. We found for some other machines that only "free slots" are used. -> Based on https://suse.slack.com/archives/C029APBKLGK/p1649246680601499?thread_ts=1649241786.564629&cid=C029APBKLGK I could make that look proper by deleting all objects except for the first within the server chassis and now we have 3 free slots showing up
6. I realized that nicksinger has permission to change types of objects, I don't. Update: Both nicksinger and okurz can change the type of some objects but not for others. Created https://sd.suse.com/servicedesk/customer/portal/1/SD-82695
7. Having at least the MAC address for each machine is helpful to debug. We checked holmes.qa.suse.de as the first non-production machine in NUE-SRV2-B-Rack-1. Using IPMI credentials from https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls we could login to https://sp.holmes.qa.suse.de/ (equivalent to https://holmes-sp.qa.suse.de/ , CNAME entry in DNS). Nice surprise we could find that the HMC of holmes knows (likely from SNMP) on which switch port and switch mac address it is connected. We crosschecked that by looking into the configuration ssh interface of the switch
8. We found that SNMP lookup in racktables works nicely, e.g. on qanet15nue.qa.suse.de, v1, public. With that we can update configuration in racktables. Maybe we can allow hosts to set the switch with more SNMP stuff like port description and then we read that port description into racktables over the SNMP sync function?

I moved all old machines from the small lab into https://racktables.suse.de/index.php?page=row&row_id=15352 and could delete "QE small" successfully. Now the same for the big lab.