Project

General

Profile

Actions
Copy link

coordination #123800

open

coordination #121720: [saga][epic] Migration to QE setup in PRG2+NUE3 while ensuring availability

[epic] Provide SUSE QE Tools services running in PRG2 aka. Prg CoLo

Added by okurz about 1 year ago. Updated about 16 hours ago.

Status:
Blocked
Priority:
Normal
Assignee:
okurz
Target version:
openQA Project - Ready
Start date:
2021-10-06
Due date:
2024-05-01 (Due in 4 days)
% Done:

64%

Estimated time:
(Total: 0.00 h)
Tags:
infra

Description

Motivation

SUSE is deprecating NUE1 (Maxtorhof) and setting up a Prague Co-Location datacenter "Prg CoLo" or "DC7" as primary location in particular for serving public services. This includes what we serve so far from VM clusters managed by EngInfra and in particular the openqa.opensuse.org infrastructure, likely also openqa.suse.de. We must participate in planning and setup and accordingly a migration until we can provide our services from Prg CoLo.

Acceptance criteria

  • AC1: SUSE QE Tools services are provided out of Prg CoLo

Subtasks 54 (19 open — 35 closed)

openQA Project - action #117553: multiple people can not access openqa.suse.de but can access openqa.nue.suse.com, we should clarify the difference and maybe change our wordingResolvedokurz2022-10-04

Actions
openQA Infrastructure - action #132134: Setup new PRG2 multi-machine openQA worker for o3 size:MResolveddheidler2023-06-29

Actions
openQA Infrastructure - action #132137: Setup new PRG2 openQA worker for osd size:MResolvedmkittler2023-06-29

Actions
action #132140: Support move of PowerPC machines to PRG2 size:MBlockedokurz2023-06-29

Actions
openQA Infrastructure - action #132143: Migration of o3 VM to PRG2 - 2023-07-19 size:MResolvednicksinger2023-06-29

Actions
action #132146: Support migration of osd VM to PRG2 - 2023-08-29 size:MResolvedmkittler2023-06-29

Actions
action #132158: Ensure that osd can work without relying on any physical machine in NUE1 size:MResolvedokurz2023-06-29

Actions
openQA Infrastructure - action #132461: manage tls certificates on o3/ariel directly with dehydrated size:MResolvednicksinger2023-07-07

Actions
openQA Infrastructure - action #132647: Migration of o3 VM to PRG2 - bare-metal tests size:MBlockedokurz

Actions
openQA Infrastructure - action #133160: Setup a modern UEFI httpboot setup on o3 with dnsmasq size:MResolveddheidler2023-07-21

Actions
openQA Infrastructure - action #133181: Migration of o3 VM to PRG2 - Fix https://openqa.opensuse.org/snapshot-changes/opensuse/Tumbleweed/Resolvedokurz

Actions
openQA Infrastructure - action #133358: Migration of o3 VM to PRG2 - Ensure IPv6 is fully workingBlockedokurz

Actions
openQA Infrastructure - action #133364: Migration of o3 VM to PRG2 - Decommission old-ariel in NUE1 as soon as we do not need it anymoreResolvedokurz

Actions
openQA Infrastructure - action #133475: Migration of o3 VM to PRG2 - connection to rabbit.opensuse.orgNew

Actions
openQA Infrastructure - action #133490: Migration of o3 VM to PRG2 - Fix o3 bare metal hosts iPXE booting size:MResolveddheidler

Actions
openQA Infrastructure - action #134081: Setup new PRG2 openQA hardware https://racktables.suse.de/index.php?page=object&object_id=23373New

Actions
openQA Infrastructure - action #134123: Setup new PRG2 openQA worker for o3 - two new arm workers size:MResolvednicksinger

Actions
openQA Infrastructure - action #134126: Setup new PRG2 openQA worker for o3 - bare-metal testing size:MBlockedokurz

Actions
openQA Infrastructure - action #134822: Migration of osd VM to PRG2 - Decommission old-osd in NUE1 as soon as we do not need it anymore size:MResolvedokurz2023-08-30

Actions
openQA Project - action #134837: SLE test repo not updated on OSD, cron service was not running since 2023-08-29, fetchneedles not called size:MResolvedlivdywan

Actions
openQA Infrastructure - action #134879: reverse DNS resolution PTR for openqa.oqa.prg2.suse.org. yields "3(NXDOMAIN)" for PRG1 workers (NUE1+PRG2 are fine) size:MResolvedokurz2023-08-31

Actions
action #134888: Ensure no job results are present in the file system for jobs that are no longer in the databaseNew

Actions
openQA Infrastructure - action #134900: salt states fail to apply due to "Pillar openqa.oqa.prg2.suse.org.key does not exist"Resolvednicksinger2023-08-31

Actions
openQA Infrastructure - action #134912: Gradually phase out NUE1 based openQA workers size:MResolvedokurz

Actions
openQA Infrastructure - action #135191: Migration of o3 VM to PRG2 - Use direct zabbix connection size:MResolveddheidler

Actions
openQA Infrastructure - action #137408: Support move of s390x mainframe(s) to PRG2 - o3 size:MResolvedmgriessmeier2023-06-29

Actions
coordination #137630: [epic] QE (non-openQA) setup in PRG2Blockedokurz2023-09-202024-05-01

Actions
action #138356: Migration of qam.suse.de to PRG2 size:MResolvedokurz2023-10-23

Actions
action #139130: Migration of openqa-service to PRG2 size:MResolvedokurz

Actions
action #153718: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - haldir size:MIn Progressnicksinger2024-01-162024-05-01

Actions
action #153721: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - legolasResolvedokurz2024-01-16

Actions
action #153724: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - blackcurrantBlockedokurz2024-01-162024-04-30

Actions
action #153727: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - cloudberry size:SWorkable2024-01-16

Actions
action #153730: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - huckleberryResolvedokurz2024-01-16

Actions
action #153733: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - soapberry size:SResolvedokurz2024-01-16

Actions
action #153736: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - nessberryBlockedokurz2024-01-16

Actions
action #153739: Move of openqa.opensuse.org machine NUE1 to PRG2 - blackbauhiniaBlockedokurz2024-01-16

Actions
action #153742: Move of OSD machine NUE1 to PRG2 - storage.qe.prg2.suse.orgResolvedokurz2024-01-16

Actions
action #153796: Prepare DHCP/DNS for qe.prg2.suse.org based on former qa.suse.de entries size:MResolvednicksinger2024-01-17

Actions
action #153799: Prepare DHCP/DNS for machines coming to qe.prg2.suse.org based on former qam.suse.de entries size:MResolvedmkittler2024-01-17

Actions
action #153802: Obsolete/remove former qam.suse.de DHCP/DNS davinci configuration or references size:MResolvedybonatakis2024-01-17

Actions
action #154447: Move of LSG QE non-openQA PowerPC machine NUE1 to PRG2 - gollumResolvedokurz

Actions
action #159306: Fix AAAA records in qe.prg2.suse.org size:SResolvedokurz2024-01-16

Actions
action #139106: Ensure a PRG2 based QE PowerPC HMC is reachable over proper FQDN and reverse PTRBlockedokurz2023-06-29

Actions
action #139109: Support move of non-openQA PowerPC machines to PRG2, i.e. haldir, legolas, whale, blackcurrant, cloudberry, huckleberry, soapberry, nessberryBlockedokurz2023-06-29

Actions
action #139112: Ensure OSD openQA PowerPC machine grenache is operational from PRG2Resolvednicksinger2023-06-29

Actions
action #139115: Ensure o3 openQA PowerPC machine qa-power8-3 is operational from PRG2 size:MResolvednicksinger2023-06-29

Actions
action #139199: Ensure OSD openQA PowerPC machine redcurrant is operational from PRG2 size:MResolvednicksinger2023-06-29

Actions
openQA Infrastructure - action #150956: o3 cannot send e-mails via smtp relay size:MResolvedokurz2023-11-16

Actions
openQA Infrastructure - action #157243: Update HMC with vMF68994Resolvedokurz2024-03-14

Actions
openQA Infrastructure - action #157528: Remove redundant ASM connections for powerPC machines size:SWorkable2024-03-19

Actions
action #157777: Provide more consistent PowerPC openQA ressources by migrating all novalink instances to hmc size:MBlockedokurz

Actions
action #159231: Bring back worker class "hmc_ppc64le-4disk" on redcurrant or another machine size:MWorkable

Actions
openQA Infrastructure - action #159669: Missing openQA data on metrics.opensuse.org since o3 migration to PRG2New2024-04-26

Actions
Actions
Copy link
 #1

Updated by okurz about 1 year ago

  • Status changed from New to Blocked

I think we don't need to do more than the one subtask for now.

Actions
Copy link
 #2

Updated by okurz about 1 year ago

  • Target version changed from Ready to future

I would like to track this outside our current backlog as we don't need to conduct that much work now.

Actions
Copy link
 #3

Updated by okurz 11 months ago

  • Target version changed from future to Ready
Actions
Copy link
 #4

Updated by okurz 10 months ago

  • Subject changed from [epic] Provide SUSE QE Tools services out of Prg CoLo to [epic] Provide SUSE QE Tools services running in PRG2 aka. Prg CoLo
Actions
Copy link
 #5

Updated by okurz 10 months ago

  • Project changed from 46 to QA
  • Category deleted (Infrastructure)
Actions
Copy link
 #6

Updated by okurz 8 months ago

  • Subtask #134822 added
Actions
Copy link
 #7

Updated by okurz 8 months ago

  • Subtask #134837 added
Actions
Copy link
 #8

Updated by okurz 8 months ago

  • Subtask #134879 added
Actions
Copy link
 #9

Updated by okurz 8 months ago

  • Subtask #134888 added
Actions
Copy link
 #10

Updated by okurz 8 months ago

  • Subtask #134900 added
Actions
Copy link
 #11

Updated by okurz 8 months ago

  • Subtask #134912 added
Actions
Copy link
 #12

Updated by okurz 8 months ago

  • Subtask #135191 added
Actions
Copy link
 #13

Updated by okurz 8 months ago

  • Subtask #134714 added
Actions
Copy link
 #14

Updated by okurz 7 months ago

status message about datacenter migration critical tasks in https://suse.slack.com/archives/C02CANHLANP/p1695203226949509

@here I would like to inform you about the current situation of datacenter migration in particular critical pending tasks waiting for mostly SUSE-IT to be resolved before NUE1 decomissioning which is expected to happen 2023-10, only a couple of days ahead:
Current status regarding datacenter-migration related critical tasks we wait for from SUSE-IT Eng-Infra in particular especially with the pending decomissioning of NUE1:

  1. https://sd.suse.com/servicedesk/customer/portal/1/SD-129541 "Please provide the zone qa.suse.de for us"
  2. PowerPC machines that are already in PRG2 are not discoverable, no MAC addresses and IP addresses in racktables, no DNS for HMC or machines and please power on machines so that we can try to discover those from HMC https://suse.slack.com/archives/C04MDKHQE20/p1695125093587689?thread_ts=1694589332.264709&cid=C04MDKHQE20
  3. No working AAAA DNS record for openqa.suse.de https://suse.slack.com/archives/C04MDKHQE20/p1694436743403769?thread_ts=1693510248.138209&cid=C04MDKHQE20
  4. Still no s390x instances fully available for QE in PRG2, relying on NUE1-SRV1 so far https://jira.suse.com/browse/ENGINFRA-1527
  5. PowerPC machines not mounted https://jira.suse.com/browse/ENGINFRA-2501 All of the above tasks are expected to have a significant impact on LSG-QE covered SUSE product delivery if not being resolved before NUE1 decomissioning (This is a copy of my message in #dct-migration https://suse.slack.com/archives/C04MDKHQE20/p1695203039285769) Given that things will very likely break and not be working as expected in particular in the domains of s390x+PowerVM testing for the upcoming weeks
Actions
Copy link
 #15

Updated by okurz 7 months ago

  • Subtask #137405 added
Actions
Copy link
 #16

Updated by okurz 7 months ago

  • Subtask #137408 added
Actions
Copy link
 #17

Updated by okurz 7 months ago

  • Subtask #137630 added
Actions
Copy link
 #18

Updated by okurz 6 months ago

  • Subtask #139106 added
Actions
Copy link
 #19

Updated by okurz 6 months ago

  • Subtask #139109 added
Actions
Copy link
 #20

Updated by okurz 6 months ago

  • Subtask #139112 added
Actions
Copy link
 #21

Updated by okurz 6 months ago

  • Subtask #139115 added
Actions
Copy link
 #22

Updated by okurz 6 months ago

  • Subtask #139199 added
Actions
Copy link
 #23

Updated by okurz 5 months ago

  • Subtask #150956 added
Actions
Copy link
 #24

Updated by okurz 3 months ago

  • Subtask #153664 added
Actions
Copy link
 #25

Updated by okurz 3 months ago

https://suse.slack.com/archives/C04MDKHQE20/p1705519472096829

(Oliver Kurz) @Michael Haefner as discussed I added more details for all machine-specific Jira tickets. The complete list of all tickets relevant for us at the current time is https://jira.suse.com/issues/?jql=project%20%3D%20ENGINFRA%20AND%20resolution%20%3D%20Unresolved%20AND%20labels%20%3D%20QE-LSG%20ORDER%20BY%20%20priority%20DESC%2C%20updated%20DESC listing 35 tickets, 4 urgent, 5 high, 11 medium, rest low. I hope this helps further priorization, planning and execution

Actions
Copy link
 #26

Updated by okurz 3 months ago

  • Subtask #117553 added
Actions
Copy link
 #27

Updated by okurz about 1 month ago

  • Subtask #157243 added
Actions
Copy link
 #28

Updated by okurz about 1 month ago

  • Subtask #157528 added
Actions
Copy link
 #29

Updated by okurz about 1 month ago

  • Subtask #157777 added
Actions
Copy link
 #30

Updated by okurz 9 days ago

  • Subtask #159231 added
Actions
Copy link
 #31

Updated by okurz about 16 hours ago

  • Subtask #159669 added
Actions
Copy link

Also available in: Atom PDF