action #125204
closedMove QA labs NUE-2.2.14-B to Frankencampus labs - non-bare-metal machines size:M
Added by okurz about 1 year ago. Updated about 1 year ago.
0%
Description
Motivation¶
Rent at Maxtorhof will end. The new location at Nbg Frankencampus will have lab rooms in the same building where the office rooms are. We must prepare and execute the move of QA equipment from the old location, mostly NUE-2.2.13 QA cold storage and NUE-2.2.14 (TAM) to an according Frankencampus room
Acceptance criteria¶
- AC1: All servers from NUE-2.2.14 (TAM) are operational from the new location
- AC2: racktables is up-to-date
Suggestions¶
- Wait for initial setup in #119548
- Organize transport of hot equipment
- Setup hot equipment at new location
- Ensure racktables is up-to-date
Files
| IMG_20230302_153622_FC_Basement_B_5_back.jpg (923 KB) IMG_20230302_153622_FC_Basement_B_5_back.jpg | okurz, 2023-03-03 19:14 |
Updated by okurz about 1 year ago
- Copied from action #119551: Move QA labs NUE-2.2.14-B to Frankencampus labs - bare-metal openQA workers size:M added
Updated by okurz about 1 year ago
- Project changed from 46 to QA
- Category deleted (
Infrastructure)
Updated by okurz about 1 year ago
- Tags changed from infra to infra, next-office-day, frankencampus
- Status changed from New to In Progress
- Assignee set to okurz
- Cabling machines in rack B5
- Mounted more machines with dheidler (thanks for the help) in rack 5 but also 3, the heavy ones. Now storage_qe3_to_be_clarified_behind_row_A is empty and nothing in the way for others anymore
- Created https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3246 to include multiple machines (not all) in DHCP
- Created https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/54 for the corresponding remove from .qa.suse.de
Updated by okurz about 1 year ago
- Related to action #124221: Repurpose quake.qe.nue2.suse.org (formerly known as cloud4) as employee-workstation replacement size:M added
Updated by openqa_review about 1 year ago
- Due date set to 2023-03-17
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz about 1 year ago
- Tags changed from infra, next-office-day, frankencampus to infra, frankencampus
- Status changed from In Progress to Blocked
https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3246 merged, https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/54 pending.
I could reach:
I could not reach:
- https://qanet2-sp.qe.nue2.suse.org/
- https://osiris-sp.qe.nue2.suse.org/
- https://seth-sp.qe.nue2.suse.org/
- https://fsp1-mania.qe.nue2.suse.org/
- https://thunderx21-sp.qe.nue2.suse.org/
Unless we have a good idea for the specific machines before investing more time in investigating I would like to wait for
Updated by okurz about 1 year ago
- File IMG_20230302_153622_FC_Basement_B_5_back.jpg IMG_20230302_153622_FC_Basement_B_5_back.jpg added
Current hardware state in B:5
Updated by okurz about 1 year ago
After a bit of waiting since above I could now reach
- https://qanet2-sp.qe.nue2.suse.org/
- https://osiris-sp.qe.nue2.suse.org/
- https://seth-sp.qe.nue2.suse.org/
I powered on osiris+seth over the https BMC interfaces and after some minutes of bootup I could login over root-ssh. corosync reports as failed on both, rest not failed.
I updated https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/54 big-time and IHMO we should merge that first before we continue, e.g. to update a CNAME entry for qsf-cluster and such. Anyway, I already found one issue that /etc/libvirt/storage/dist.suse.de.xml included an old IPv4 entry for dist.suse.de. I changed that manually to use the FQDN dist.suse.de and then systemctl start libvirtd was starting up fine. I could connect using virt-manager to osiris-1.qe.nue2.suse.org but this showed only "first-test-vm". I think I saw this already some time ago when the cluster thingy was not working.
With grep -R 10.162.2 /etc I could find
/etc/drbd.d/qsf-cluster.res: address 10.162.2.95:7789;
/etc/drbd.d/qsf-cluster.res: address 10.162.2.96:7789;
/etc/drbd.d/qsf-cluster.res.salttest: address 10.162.2.95:7789;
/etc/drbd.d/qsf-cluster.res.salttest: address 10.162.2.96:7789;
/etc/corosync/corosync.conf: ring0_addr: 10.162.2.95
/etc/corosync/corosync.conf: ring0_addr: 10.162.2.96
so I updated that using the FQDNs and restarted corosync. But libvirtd still shows only first-test-vm. Maybe something in #78206, #35506, #117697 helps
Updated by okurz about 1 year ago
- Tags changed from infra, frankencampus to infra, frankencampus, next-office-day
- Due date deleted (
2023-03-17)
I did fibre connections to both openqaworker-arm-4+5, BMC still pending.
- TODO next-office-day connect BMC for both openqaworker-arm-4+5
EDIT: 2023-04-19: BMC connections for openqaworker-arm-4+5 done
Updated by okurz about 1 year ago
- Tags changed from infra, frankencampus, next-office-day to infra, frankencampus
Updated by okurz about 1 year ago
Updated by okurz about 1 year ago
- Copied to action #128390: Move QA labs NUE-2.2.14-B to Frankencampus labs - infrastructure management improvements added
Updated by okurz about 1 year ago
- Copied to action #128393: Move QA labs NUE-2.2.14-B to Frankencampus labs - recover openQA staging test setup size:M added
Updated by okurz about 1 year ago
- Status changed from Blocked to In Progress
https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 resolved, we have root access to the DHCP servers walter1/2 in FC Basement LSG QE now.
#124643 for later.
Still blocked on
moved to #128390
thincsus to be handled in #128393.
I think all other equipment is usable from the new location and racktables is having the information. We should walk over NUE-FC-B LSG QE to consolidate, e.g. use FQDNs for all entries.
Updated by okurz about 1 year ago
- Subject changed from Move QA labs NUE-2.2.14-B to Frankencampus labs - non-bare-metal machines to Move QA labs NUE-2.2.14-B to Frankencampus labs - non-bare-metal machines size:M
Updated by okurz about 1 year ago
- Status changed from In Progress to Feedback
With mkittler and nsinger we walked over complete NUE-FC-B LSG QE and used proper pseudo-FQDNs for all entries. "pseudo" because in some cases machines are not connected so they are not reachable by the FQDN as such.
https://gitlab.suse.de/qa-sle/qanet-configs/-/merge_requests/61 (merged) "Cleanup more entries of machines moved to FC Basement".
I also kept old CNAME entries as aliases in FQDNs after the A record FQDN entry.
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/529 to use new qe.nue2.suse.org FQDNs where applicable.
Then crosschecked properties and port entries in racktables again.
For IPv4 entries I reviewed https://racktables.nue.suse.com/index.php?id=1386&page=ipv4net&tab=default . I went over every entry corresponding to IPv4 entries in https://gitlab.suse.de/OPS-Service/salt/-/blob/production/pillar/domain/qe_nue2_suse_org/hosts.yaml by replacing the IPv4 address in the URL like
https://racktables.nue.suse.com/index.php?page=ipaddress&tab=assignment&ip=10.168.192.77
and add according entries for each address-machine-pair.
EDIT: Waiting for https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/529
Updated by okurz about 1 year ago
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/529 merged, awaiting results from deployment.
Updated by okurz about 1 year ago
- Status changed from Feedback to Resolved
https://openqa.suse.de/tests/11025832 shows use of the new FQDNs. In the meantime no other complaints received.