Project

General

Profile

Actions
Copy link

coordination #123800

closed

coordination #121720: [saga][epic] Migration to QE setup in PRG2+NUE3 while ensuring availability

[epic] Provide SUSE QE Tools services running in PRG2 aka. Prg CoLo

Added by okurz almost 2 years ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Target version:
openQA Project (public) - Ready
Start date:
2021-10-06
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Tags:
infra

Description

Motivation

SUSE is deprecating NUE1 (Maxtorhof) and setting up a Prague Co-Location datacenter "Prg CoLo" or "DC7" as primary location in particular for serving public services. This includes what we serve so far from VM clusters managed by EngInfra and in particular the openqa.opensuse.org infrastructure, likely also openqa.suse.de. We must participate in planning and setup and accordingly a migration until we can provide our services from Prg CoLo.

Acceptance criteria

  • AC1: SUSE QE Tools services are provided out of Prg CoLo

Subtasks 42 (0 open — 42 closed)

openQA Project (public) - action #117553: multiple people can not access openqa.suse.de but can access openqa.nue.suse.com, we should clarify the difference and maybe change our wordingResolvedokurz2022-10-04

Actions
openQA Infrastructure (public) - action #132134: Setup new PRG2 multi-machine openQA worker for o3 size:MResolveddheidler2023-06-29

Actions
openQA Infrastructure (public) - action #132137: Setup new PRG2 openQA worker for osd size:MResolvedmkittler2023-06-29

Actions
openQA Infrastructure (public) - action #132143: Migration of o3 VM to PRG2 - 2023-07-19 size:MResolvednicksinger2023-06-29

Actions
action #132146: Support migration of osd VM to PRG2 - 2023-08-29 size:MResolvedmkittler2023-06-29

Actions
action #132158: Ensure that osd can work without relying on any physical machine in NUE1 size:MResolvedokurz2023-06-29

Actions
openQA Infrastructure (public) - action #132461: manage tls certificates on o3/ariel directly with dehydrated size:MResolvednicksinger2023-07-07

Actions
openQA Infrastructure (public) - action #132647: Migration of o3 VM to PRG2 - bare-metal tests size:MResolvedokurz

Actions
openQA Infrastructure (public) - action #133160: Setup a modern UEFI httpboot setup on o3 with dnsmasq size:MResolveddheidler2023-07-21

Actions
openQA Infrastructure (public) - action #133181: Migration of o3 VM to PRG2 - Fix https://openqa.opensuse.org/snapshot-changes/opensuse/Tumbleweed/Resolvedokurz

Actions
openQA Infrastructure (public) - action #133358: Migration of o3 VM to PRG2 - Ensure IPv6 is fully workingResolvedokurz

Actions
openQA Infrastructure (public) - action #133364: Migration of o3 VM to PRG2 - Decommission old-ariel in NUE1 as soon as we do not need it anymoreResolvedokurz

Actions
openQA Infrastructure (public) - action #133475: Migration of o3 VM to PRG2 - connection to rabbit.opensuse.orgResolvedmkittler

Actions
openQA Infrastructure (public) - action #133490: Migration of o3 VM to PRG2 - Fix o3 bare metal hosts iPXE booting size:MResolveddheidler

Actions
openQA Infrastructure (public) - action #134123: Setup new PRG2 openQA worker for o3 - two new arm workers size:MResolvednicksinger

Actions
openQA Infrastructure (public) - action #134822: Migration of osd VM to PRG2 - Decommission old-osd in NUE1 as soon as we do not need it anymore size:MResolvedokurz2023-08-30

Actions
openQA Project (public) - action #134837: SLE test repo not updated on OSD, cron service was not running since 2023-08-29, fetchneedles not called size:MResolvedlivdywan

Actions
openQA Infrastructure (public) - action #134879: reverse DNS resolution PTR for openqa.oqa.prg2.suse.org. yields "3(NXDOMAIN)" for PRG1 workers (NUE1+PRG2 are fine) size:MResolvedokurz2023-08-31

Actions
openQA Infrastructure (public) - action #134900: salt states fail to apply due to "Pillar openqa.oqa.prg2.suse.org.key does not exist"Resolvednicksinger2023-08-31

Actions
openQA Infrastructure (public) - action #134912: Gradually phase out NUE1 based openQA workers size:MResolvedokurz

Actions
openQA Infrastructure (public) - action #135191: Migration of o3 VM to PRG2 - Use direct zabbix connection size:MResolveddheidler

Actions
openQA Infrastructure (public) - action #137408: Support move of s390x mainframe(s) to PRG2 - o3 size:MResolvedmgriessmeier2023-06-29

Actions
coordination #137630: [epic] QE (non-openQA) setup in PRG2Resolvedokurz2023-09-20

Actions
action #138356: Migration of qam.suse.de to PRG2 size:MResolvedokurz2023-10-23

Actions
action #139130: Migration of openqa-service to PRG2 size:MResolvedokurz

Actions
action #153721: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - legolasResolvedokurz2024-01-16

Actions
action #153724: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - blackcurrantResolvedokurz2024-01-16

Actions
action #153730: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - huckleberryResolvedokurz2024-01-16

Actions
action #153796: Prepare DHCP/DNS for qe.prg2.suse.org based on former qa.suse.de entries size:MResolvednicksinger2024-01-17

Actions
action #153799: Prepare DHCP/DNS for machines coming to qe.prg2.suse.org based on former qam.suse.de entries size:MResolvedmkittler2024-01-17

Actions
action #153802: Obsolete/remove former qam.suse.de DHCP/DNS davinci configuration or references size:MResolvedybonatakis2024-01-17

Actions
action #154447: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - gollumResolvedokurz

Actions
action #159306: Fix AAAA records in qe.prg2.suse.org size:SResolvedokurz2024-01-16

Actions
openQA Infrastructure (public) - action #161756: IPMI access over IPv6 doesn't work on blackbauhinia size:SResolvedmkittler2024-04-24

Actions
action #139109: Support move of non-openQA PowerPC machines to PRG2, i.e. haldir, legolas, whale, blackcurrant, cloudberry, huckleberry, soapberry, nessberryResolvedokurz2023-06-29

Actions
action #139112: Ensure OSD openQA PowerPC machine grenache is operational from PRG2Resolvednicksinger2023-06-29

Actions
action #139115: Ensure o3 openQA PowerPC machine qa-power8-3 is operational from PRG2 size:MResolvednicksinger2023-06-29

Actions
openQA Infrastructure (public) - action #150815: unable to login over ssh to o3 (gate.opensuse.org:2214) size:MRejectedokurz2023-11-13

Actions
openQA Infrastructure (public) - action #150956: o3 cannot send e-mails via smtp relay size:MResolvedokurz2023-11-16

Actions
openQA Infrastructure (public) - action #157243: Update HMC with vMF68994Resolvedokurz2024-03-14

Actions
action #159231: Bring back worker class "hmc_ppc64le-4disk" on redcurrant or another machine size:MResolvednicksinger

Actions
openQA Infrastructure (public) - action #161318: Ensure we have a consistent racktables entry for OSDResolvedokurz2024-05-31

Actions
Actions
Copy link
 #1

Updated by okurz almost 2 years ago

  • Status changed from New to Blocked

I think we don't need to do more than the one subtask for now.

Actions
Copy link
 #2

Updated by okurz almost 2 years ago

  • Target version changed from Ready to future

I would like to track this outside our current backlog as we don't need to conduct that much work now.

Actions
Copy link
 #3

Updated by okurz over 1 year ago

  • Target version changed from future to Ready
Actions
Copy link
 #4

Updated by okurz over 1 year ago

  • Subject changed from [epic] Provide SUSE QE Tools services out of Prg CoLo to [epic] Provide SUSE QE Tools services running in PRG2 aka. Prg CoLo
Actions
Copy link
 #5

Updated by okurz over 1 year ago

  • Project changed from 46 to QA (public)
  • Category deleted (Infrastructure)
Actions
Copy link
 #6

Updated by okurz over 1 year ago

  • Subtask #134822 added
Actions
Copy link
 #7

Updated by okurz over 1 year ago

  • Subtask #134837 added
Actions
Copy link
 #8

Updated by okurz over 1 year ago

  • Subtask #134879 added
Actions
Copy link
 #9

Updated by okurz over 1 year ago

  • Subtask #134888 added
Actions
Copy link
 #10

Updated by okurz over 1 year ago

  • Subtask #134900 added
Actions
Copy link
 #11

Updated by okurz over 1 year ago

  • Subtask #134912 added
Actions
Copy link
 #12

Updated by okurz over 1 year ago

  • Subtask #135191 added
Actions
Copy link
 #13

Updated by okurz over 1 year ago

  • Subtask #134714 added
Actions
Copy link
 #14

Updated by okurz over 1 year ago

status message about datacenter migration critical tasks in https://suse.slack.com/archives/C02CANHLANP/p1695203226949509

@here I would like to inform you about the current situation of datacenter migration in particular critical pending tasks waiting for mostly SUSE-IT to be resolved before NUE1 decomissioning which is expected to happen 2023-10, only a couple of days ahead:
Current status regarding datacenter-migration related critical tasks we wait for from SUSE-IT Eng-Infra in particular especially with the pending decomissioning of NUE1:

  1. https://sd.suse.com/servicedesk/customer/portal/1/SD-129541 "Please provide the zone qa.suse.de for us"
  2. PowerPC machines that are already in PRG2 are not discoverable, no MAC addresses and IP addresses in racktables, no DNS for HMC or machines and please power on machines so that we can try to discover those from HMC https://suse.slack.com/archives/C04MDKHQE20/p1695125093587689?thread_ts=1694589332.264709&cid=C04MDKHQE20
  3. No working AAAA DNS record for openqa.suse.de https://suse.slack.com/archives/C04MDKHQE20/p1694436743403769?thread_ts=1693510248.138209&cid=C04MDKHQE20
  4. Still no s390x instances fully available for QE in PRG2, relying on NUE1-SRV1 so far https://jira.suse.com/browse/ENGINFRA-1527
  5. PowerPC machines not mounted https://jira.suse.com/browse/ENGINFRA-2501 All of the above tasks are expected to have a significant impact on LSG-QE covered SUSE product delivery if not being resolved before NUE1 decomissioning (This is a copy of my message in #dct-migration https://suse.slack.com/archives/C04MDKHQE20/p1695203039285769) Given that things will very likely break and not be working as expected in particular in the domains of s390x+PowerVM testing for the upcoming weeks
Actions
Copy link
 #15

Updated by okurz about 1 year ago

  • Subtask #137405 added
Actions
Copy link
 #16

Updated by okurz about 1 year ago

  • Subtask #137408 added
Actions
Copy link
 #17

Updated by okurz about 1 year ago

  • Subtask #137630 added
Actions
Copy link
 #18

Updated by okurz about 1 year ago

  • Subtask #139106 added
Actions
Copy link
 #19

Updated by okurz about 1 year ago

  • Subtask #139109 added
Actions
Copy link
 #20

Updated by okurz about 1 year ago

  • Subtask #139112 added
Actions
Copy link
 #21

Updated by okurz about 1 year ago

  • Subtask #139115 added
Actions
Copy link
 #22

Updated by okurz about 1 year ago

  • Subtask #139199 added
Actions
Copy link
 #23

Updated by okurz about 1 year ago

  • Subtask #150956 added
Actions
Copy link
 #24

Updated by okurz 11 months ago

  • Subtask #153664 added
Actions
Copy link
 #25

Updated by okurz 11 months ago

https://suse.slack.com/archives/C04MDKHQE20/p1705519472096829

(Oliver Kurz) @Michael Haefner as discussed I added more details for all machine-specific Jira tickets. The complete list of all tickets relevant for us at the current time is https://jira.suse.com/issues/?jql=project%20%3D%20ENGINFRA%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20QE-LSG%20ORDER%20BY%20%20priority%20DESC%2C%20updated%20DESC listing 35 tickets, 4 urgent, 5 high, 11 medium, rest low. I hope this helps further priorization, planning and execution

Actions
Copy link
 #26

Updated by okurz 11 months ago

  • Subtask #117553 added
Actions
Copy link
 #27

Updated by okurz 10 months ago

  • Subtask #157243 added
Actions
Copy link
 #28

Updated by okurz 9 months ago

  • Subtask #157528 added
Actions
Copy link
 #29

Updated by okurz 9 months ago

  • Subtask #157777 added
Actions
Copy link
 #30

Updated by okurz 8 months ago

  • Subtask #159231 added
Actions
Copy link
 #31

Updated by okurz 8 months ago

  • Subtask #159669 added
Actions
Copy link
 #32

Updated by okurz 8 months ago

  • Subtask #150815 added
Actions
Copy link
 #33

Updated by okurz 8 months ago

  • Subtask deleted (#134888)
Actions
Copy link
 #34

Updated by okurz 7 months ago

  • Subtask deleted (#153664)
Actions
Copy link
 #35

Updated by okurz 7 months ago

  • Subtask deleted (#139106)
Actions
Copy link
 #36

Updated by okurz 7 months ago

  • Subtask deleted (#157777)
Actions
Copy link
 #37

Updated by okurz 7 months ago

  • Subtask deleted (#134081)
Actions
Copy link
 #38

Updated by okurz 7 months ago

  • Subtask deleted (#157528)
Actions
Copy link
 #39

Updated by okurz 7 months ago

  • Subtask #161318 added
Actions
Copy link
 #40

Updated by okurz 7 months ago

  • Subtask deleted (#159669)
Actions
Copy link
 #41

Updated by okurz 7 months ago

  • Subtask deleted (#134126)
Actions
Copy link
 #42

Updated by okurz 6 months ago

  • Status changed from Blocked to Resolved

All remaining tasks done \o/

Actions
Copy link

Also available in: Atom PDF