action #125204: Move QA labs NUE-2.2.14-B to Frankencampus labs - non-bare-metal machines size:M - QA - openSUSE Project Management Tool

Actions

Copy link

action #125204

closed

Move QA labs NUE-2.2.14-B to Frankencampus labs - non-bare-metal machines size:M

Added by okurz about 1 year ago. Updated about 1 year ago.

Status:

Resolved

Priority:

Normal

Assignee:

okurz

Target version:

openQA Project - Ready

Start date:

2022-10-28

Due date:

% Done:

Estimated time:

Tags:

infra, frankencampus

Description

Motivation¶

Rent at Maxtorhof will end. The new location at Nbg Frankencampus will have lab rooms in the same building where the office rooms are. We must prepare and execute the move of QA equipment from the old location, mostly NUE-2.2.13 QA cold storage and NUE-2.2.14 (TAM) to an according Frankencampus room

Acceptance criteria¶

AC1: All servers from NUE-2.2.14 (TAM) are operational from the new location
AC2: racktables is up-to-date

Suggestions¶

Wait for initial setup in #119548
Organize transport of hot equipment
Setup hot equipment at new location
Ensure racktables is up-to-date

Files

IMG_20230302_153622_FC_Basement_B_5_back.jpg (923 KB) IMG_20230302_153622_FC_Basement_B_5_back.jpg

okurz, 2023-03-03 19:14

Related issues 4 (0 open — 4 closed)

Actions

Copy link

Updated by okurz about 1 year ago

Copied from action #119551: Move QA labs NUE-2.2.14-B to Frankencampus labs - bare-metal openQA workers size:M added

Actions

Copy link

Updated by okurz about 1 year ago

Due date deleted (~~2023-03-02~~)

Actions

Copy link

Updated by okurz about 1 year ago

Tags set to infra

Actions

Copy link

Updated by okurz about 1 year ago

Project changed from 46 to QA
Category deleted (~~Infrastructure~~)

Actions

Copy link

Updated by okurz about 1 year ago

Tags changed from infra to infra, next-office-day, frankencampus
Status changed from New to In Progress
Assignee set to okurz

Cabling machines in rack B5
Mounted more machines with dheidler (thanks for the help) in rack 5 but also 3, the heavy ones. Now storage_qe3_to_be_clarified_behind_row_A is empty and nothing in the way for others anymore
Created https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3246 to include multiple machines (not all) in DHCP
Created https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/54 for the corresponding remove from .qa.suse.de

Actions

Copy link

Updated by okurz about 1 year ago

Related to action #124221: Repurpose quake.qe.nue2.suse.org (formerly known as cloud4) as employee-workstation replacement size:M added

Actions

Copy link

Updated by openqa_review about 1 year ago

Due date set to 2023-03-17

Setting due date based on mean cycle time of SUSE QE Tools

Actions

Copy link

Updated by okurz about 1 year ago

Tags changed from infra, next-office-day, frankencampus to infra, frankencampus
Status changed from In Progress to Blocked

https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3246 merged, https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/54 pending.

I could reach:

I could not reach:

Unless we have a good idea for the specific machines before investing more time in investigating I would like to wait for

Actions

Copy link

Updated by okurz about 1 year ago

File IMG_20230302_153622_FC_Basement_B_5_back.jpg IMG_20230302_153622_FC_Basement_B_5_back.jpg added

Current hardware state in B:5

Actions

Copy link

#10

Updated by okurz about 1 year ago

After a bit of waiting since above I could now reach

I powered on osiris+seth over the https BMC interfaces and after some minutes of bootup I could login over root-ssh. corosync reports as failed on both, rest not failed.

I updated https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/54 big-time and IHMO we should merge that first before we continue, e.g. to update a CNAME entry for qsf-cluster and such. Anyway, I already found one issue that /etc/libvirt/storage/dist.suse.de.xml included an old IPv4 entry for dist.suse.de. I changed that manually to use the FQDN dist.suse.de and then systemctl start libvirtd was starting up fine. I could connect using virt-manager to osiris-1.qe.nue2.suse.org but this showed only "first-test-vm". I think I saw this already some time ago when the cluster thingy was not working.

With grep -R 10.162.2 /etc I could find

/etc/drbd.d/qsf-cluster.res:        address   10.162.2.95:7789;
/etc/drbd.d/qsf-cluster.res:        address   10.162.2.96:7789;
/etc/drbd.d/qsf-cluster.res.salttest:        address   10.162.2.95:7789;
/etc/drbd.d/qsf-cluster.res.salttest:        address   10.162.2.96:7789;
/etc/corosync/corosync.conf:                ring0_addr: 10.162.2.95
/etc/corosync/corosync.conf:                ring0_addr: 10.162.2.96

so I updated that using the FQDNs and restarted corosync. But libvirtd still shows only first-test-vm. Maybe something in #78206, #35506, #117697 helps

Actions

Copy link

#11

Updated by okurz about 1 year ago

Tags changed from infra, frankencampus to infra, frankencampus, next-office-day
Due date deleted (~~2023-03-17~~)

I did fibre connections to both openqaworker-arm-4+5, BMC still pending.

TODO next-office-day connect BMC for both openqaworker-arm-4+5

EDIT: 2023-04-19: BMC connections for openqaworker-arm-4+5 done

Actions

Copy link

#13

Updated by okurz about 1 year ago

Tags changed from infra, frankencampus, next-office-day to infra, frankencampus

Actions

Copy link

#14

Updated by okurz about 1 year ago

Still blocked on

Actions

Copy link

#15

Updated by okurz about 1 year ago

Copied to action #128390: Move QA labs NUE-2.2.14-B to Frankencampus labs - infrastructure management improvements added

Actions

Copy link

#16

Updated by okurz about 1 year ago

Copied to action #128393: Move QA labs NUE-2.2.14-B to Frankencampus labs - recover openQA staging test setup size:M added

Actions

Copy link

#17

Updated by okurz about 1 year ago

Status changed from Blocked to In Progress

https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 resolved, we have root access to the DHCP servers walter1/2 in FC Basement LSG QE now.

#124643 for later.

Still blocked on

https://sd.suse.com/servicedesk/customer/portal/1/SD-99379

https://sd.suse.com/servicedesk/customer/portal/1/SD-113832

https://sd.suse.com/servicedesk/customer/portal/1/SD-113814

moved to #128390

thincsus to be handled in #128393.

I think all other equipment is usable from the new location and racktables is having the information. We should walk over NUE-FC-B LSG QE to consolidate, e.g. use FQDNs for all entries.

Actions

Copy link

#18

Updated by okurz about 1 year ago

Subject changed from Move QA labs NUE-2.2.14-B to Frankencampus labs - non-bare-metal machines to Move QA labs NUE-2.2.14-B to Frankencampus labs - non-bare-metal machines size:M

Actions

Copy link

#19

Updated by okurz about 1 year ago

Status changed from In Progress to Feedback

With mkittler and nsinger we walked over complete NUE-FC-B LSG QE and used proper pseudo-FQDNs for all entries. "pseudo" because in some cases machines are not connected so they are not reachable by the FQDN as such.
https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/61 (merged) "Cleanup more entries of machines moved to FC Basement".
I also kept old CNAME entries as aliases in FQDNs after the A record FQDN entry.

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/529 to use new qe.nue2.suse.org FQDNs where applicable.

Then crosschecked properties and port entries in racktables again.

For IPv4 entries I reviewed https://racktables.nue.suse.com/index.php?id=1386&page=ipv4net&tab=default . I went over every entry corresponding to IPv4 entries in https://gitlab.suse.de/OPS-Service/salt/-/blob/production/pillar/domain/qe_nue2_suse_org/hosts.yaml by replacing the IPv4 address in the URL like
https://racktables.nue.suse.com/index.php?page=ipaddress&tab=assignment&ip=10.168.192.77
and add according entries for each address-machine-pair.

EDIT: Waiting for https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/529

Actions

Copy link

#20