QA

Based on the discussion in https://suse.slack.com/archives/C02CANHLANP/p1691091091576219 we made the decision to move orion.openqa.opensuse.org and andromeda.openqa.opensuse.org to PRG2e as well. Tags in netbox updated. Please accomodate them in the moving plans as well. Previously the machines were designated for NUE3 but we reconsider after NUE3 was now decided to be "cold redundancy" hence a waste for those rather new machines (from 2021). Decided for "PRG2e" because those machines are not maintained by SUSE QE Tools but by QEC, although they did not manage to setup the machines for two years already.

For openqaworker20.openqa.opensuse.org we decided to move to PRG2 next to other o3 machines for similar reasons as above. Tags in netbox updated. For openqaworker19.openqa.opensuse.org we keep the original designation to move to NUE3 so that the cold redundancy for o3 has at least some recent machine available in cold redundancy.

Additionally as discussed in an email thread started by Calen Chen and Xiaoli Ai I have now designated 6 more machines with "Move to Prague Colo2" and "Colo2 second wave" in netbox. Assuming that it is still possible please make sure that those machines get a place on the train/truck/plane/ship to PRG2e :)

mgriessmeier and me clarified in Jitsi call 2023-08-28. As commented in #132617-11 w20 should go to PRG2 (not PRG2e). w20 is tagged in netbox with "Move to Prague Colo". I created the specific ticket #134714 about the move to PRG2

1. @mgriessmeier please ensure that orion+andromeda will be prepared for move to PRG2e as they are in NUE1-SRV1 so this needs coordination with hare and Eng-Infra
2. @mgriessmeier as part of #134714 please ensure that machines planned for PRG2 (not PRG2e) are accounted for in moving plans

okurz wrote in #note-13:

1. @mgriessmeier please ensure that orion+andromeda will be prepared for move to PRG2e as they are in NUE1-SRV1 so this needs coordination with hare and Eng-Infra

orion+andromeda (+fibonacci) will be de-racked tomorrow by me and mmoese with help of gschlotter and put in the according Strohmann moving boxes provided in NUE-SRV2 and to be picked up by Strohmann on Sep 15th

Maxtorhof evacuation planned as notified by hreinecke happens 2023-11-29, Wednesday. Decommissioning will be prepared by hreinecke. We should await his notice and then put our old machine(s) likely only in a container. Crosscheck netbox for number of machines and needed rackspace.

https://suse.slack.com/archives/C04MDKHQE20/p1699003595410659

Oliver Kurz @Hannes Reinecke (CC @Matthias Griessmeier) as you asked about necessary rack space in PRG2e for LSG QE: I crosschecked https://netbox.suse.de/dcim/devices/?tag=qe-lsg&tag=move-to-prague-colo2 and I can confirm it is up-to-date and properly reflects the current plan. The query shows 12 machines. 3 are apparently already in PRG2 cold storage (first wave), 9 still in NUE1-SRV2, waiting for deracking and transport. Currently I see not more need of rackspace besides that. (edited)

okurz wrote in #note-24:

Response from hreinecke

(Hannes Reinecke) To help with the move, can you register the destination location with 'placeholder' entries (like I did for machines like 'adalid' or 'moorfoot')? I have assigned two rack to you https://netbox.suse.de/dcim/racks/1223/ and https://netbox.suse.de/dcim/racks/1224/ . If they already contain placeholder entries where machines should be mounted it will help with the process. Unless you or your team want to rack your machines ...

@mgriessmeier will you do that or should we?

feel free to do that :)

@okurz Hi Oliver, thanks for helping with the MOVE. Just now I had a check in netbox.suse.de for the latest information of all virtualization machines in NUE1. I noticed some differences with our agreement. Would you please have a look?

Agreement¶

Our final agreement in the email thread of Output from last week's IT/BCL workshop (notes and actions) is that we do not want cold redundancy, but hot redundancy , and they will be put in PRG2e.

Quote of email:

> So for now I set the tag "Move to Prague Colo2" for all the machines we
discussed here.

Yes. Thanks for that. 

BTW, I had a check in netbox.suse.de, below machines have been updated to "Prague Colo2" from previous NUE3, and gonzo stays in FC lab.

> openqaw9-hyperv.qa.suse.de -> Move to  PRG COLO2
> worker7-hyperv.oqa.suse.de -> Move to  PRG COLO2
> worker8-vmware.oqa.suse.de -> Move to  PRG COLO2
> openqaw5-xen.qa.suse.de -> Move to PRG COLO2
> fozzie -> Move to  PRG COLO2
> quinn  -> Move to PRG COLO2

Besides, the plan for the other two machines have been always

• amd-zen2-gpu-sut1.qa.suse.de -> Move to Prague Colo2(same as PRG2e), second wave
• blackbauhinia.qa.suse.de -> Move to Prague Colo2(same as PRG2e),second wave

Current status in netbox¶

I noticed that all others are as planned, except:

• worker7-hyperv.oqa.suse.de -> plan is PRG2e, but in netbox, it has been put to PRG2 / PRG2 - Cold Storage. Is it cold redundancy room? We need it hot operating.
• worker8-vmware.oqa.suse.de -> same as above
• blackbauhinia.qa.suse.de -> plan is to move to Prague Colo2,second wave, but in netbox, it is not Planned status like others of second wave(eg amd-zen2-gpu-sut1) which has placeholder for PRG2E-D rack D09 - QE. Is it forgotten, or just not yet been planned?

xlai wrote in #note-29:

@okurz Hi Oliver, thanks for helping with the MOVE. Just now I had a check in netbox.suse.de for the latest information of all virtualization machines in NUE1. I noticed some differences with our agreement. Would you please have a look?

Agreement¶

Our final agreement in the email thread of Output from last week's IT/BCL workshop (notes and actions) is that we do not want cold redundancy, but hot redundancy , and they will be put in PRG2e.

Let me clarify so that we are sure we are talking about the same wording here: The "cold redundancy" that we talk about here is a concept for providing geo-redundancy of production datacenters. So if the PRG2 datacenter including PRG2e is not available for a longer period then the cold redundancy location NUE3 aka "Marienberg DR" for "Marienberg Disaster Recovery" located in Nuremberg, Germany, takes over serving as replacement production datacenter. There is currently no plan by SUSE-IT to support hot redundant production datacenters for critical services. However, we within LSG QE, meaning as supported by LSG QE Tools team, we provide best effort hot redundancy by using lab rooms. With this we can provide hot geo-redundancy so multiple servers for the same or similar purpose being online and available at the same time but with no hardcoded agreement regarding reaction or resolution times if problems arise.

I noticed that all others are as planned, except:

• worker7-hyperv.oqa.suse.de -> plan is PRG2e, but in netbox, it has been put to PRG2 / PRG2 - Cold Storage. Is it cold redundancy room? We need it hot operating.
• worker8-vmware.oqa.suse.de -> same as above

"cold storage" is just temporary storage in the PRG2 datacenter until PRG2e is fully functional. PRG2e is expected to be functional start of 2023-12. Still, it seems you have uncovered an inconsistency which I am grateful for. I have raised that topic in
https://suse.slack.com/archives/C04MDKHQE20/p1699456686160239?thread_ts=1699003595.410659&cid=C04MDKHQE20

(Oliver Kurz) @Hannes Reinecke @John Ford @Moroni Flores apparently some device entries in netbox have lost their according tags like https://netbox.suse.de/dcim/devices/6833/ where "hare@suse.com", likely your user account running scripted, in https://netbox.suse.de/extras/changelog/247395/ removed "Move to Prague Colo2" and "Colo2 second wave". Another example would be storage.oqa.suse.de https://netbox.suse.de/dcim/devices/7233/ . Given that

1. I will need to crosscheck all "QE LSG" machines for missing tags and re-add where needed. This means I/we will need to plan for more rackspace that anticipated potentially also still needing to move machines from PRG2 to PRG2e
2. What in general is the plan for all machines in PRG2 - Cold Storage ?

Your next point:

• blackbauhinia.qa.suse.de -> plan is to move to Prague Colo2,second wave, but in netbox, it is not Planned status like others of second wave(eg amd-zen2-gpu-sut1) which has placeholder for PRG2E-D rack D09 - QE. Is it forgotten, or just not yet been planned?

Good. I overlooked this machine for planning some rack space. It would likely not have been a problem as the planned rackspace was just needed for estimation but I created a proper linked placeholder entry now:
https://netbox.suse.de/dcim/devices/10665/
Thank you.

@okurz Glad that my feedback helps and thank you for the reply in https://progress.opensuse.org/issues/132617#note-31. Moving so many servers are indeed super complex. Thank you all for the hard work.

worker7-hyperv.oqa.suse.de -> plan is PRG2e, but in netbox, it has been put to PRG2 / PRG2 - Cold Storage. Is it cold redundancy room? We need it hot operating.
worker8-vmware.oqa.suse.de -> same as above

"cold storage" is just temporary storage in the PRG2 datacenter until PRG2e is fully functional. PRG2e is expected to be functional start of 2023-12. Still, it seems you have uncovered an inconsistency which I am grateful for. I have raised that topic in
https://suse.slack.com/archives/C04MDKHQE20/p1699456686160239?thread_ts=1699003595.410659&cid=C04MDKHQE20

Thank you for the explanations. My understanding is that, for the machines that are currently in PRG2 cold storage, they will be moved to PRG2e after the lab is functional. Basically the machines will be up again in PRG2e in this December. Is this correct?

xlai wrote in #note-32:

for the machines that are currently in PRG2 cold storage, they will be moved to PRG2e after the lab is functional. Basically the machines will be up again in PRG2e in this December. Is this correct?

Not before December so my realistic estimate is that some machines might be online and usable December this year but due to limited capacity in Eng-Infra I expect that there will be problems and then some machines can only be handled 2024-01 or later

okurz wrote in #note-33:

xlai wrote in #note-32:

for the machines that are currently in PRG2 cold storage, they will be moved to PRG2e after the lab is functional. Basically the machines will be up again in PRG2e in this December. Is this correct?

Not before December so my realistic estimate is that some machines might be online and usable December this year but due to limited capacity in Eng-Infra I expect that there will be problems and then some machines can only be handled 2024-01 or later

@okurz Got it.
If it is possible, appreciate if below 2 machines can be given higher priority when bringing machines back again, to unblock O3 virt test or decrease workload pressure on one machine pair. Other VT machines can be given regular priority. Thanks!

• amd-zen2-gpu-sut1.qa.suse.de
• fozzie

okurz wrote in #note-35:

If it is possible, appreciate if below 2 machines can be given higher priority when bringing machines back again, to unblock O3 virt test or decrease workload pressure on one machine pair. Other VT machines can be given regular priority. Thanks!

• amd-zen2-gpu-sut1.qa.suse.de
• fozzie

As far as where it makes a difference, yes, it will be possible and I will keep this in mind

Cool, thanks Oliver!

I reviewed all entries of "LSG QE" machines twice, added according placeholder entries in PRG2 racks, informed jford and hreinecke in https://suse.slack.com/archives/C04MDKHQE20/p1699887233109419?thread_ts=1699003595.410659&cid=C04MDKHQE20

(Oliver Kurz) @Hannes Reinecke @John Ford @Moroni Flores apparently some device entries in netbox have lost their according tags like https://netbox.suse.de/dcim/devices/6833/ where "hare@suse.com", likely your user account running scripted, in https://netbox.suse.de/extras/changelog/247395/ removed "Move to Prague Colo2" and "Colo2 second wave". Another example would be storage.oqa.suse.de https://netbox.suse.de/dcim/devices/7233/ . Given that
I will need to crosscheck all "QE LSG" machines for missing tags and re-add where needed. This means I/we will need to plan for more rackspace that anticipated potentially also still needing to move machines from PRG2 to PRG2e
What in general is the plan for all machines in PRG2 - Cold Storage ?
(Oliver Kurz) I covered 1., this leaves 2 open.

On 2023-11-15 in the weekly DCT progress call I plan to bring up this topic. Then afterwards we will mostly wait for the physical move end of 2023-11

hreinecke+egotthold will install machines in PRG2e starting next week. Wrote in https://suse.slack.com/archives/C04MDKHQE20/p1701268766562649?thread_ts=1701263971.649939&cid=C04MDKHQE20

@Hannes Reinecke a list of QE machines to install in PRG2e by descending priority:
ada
amd-zen2-gpu-sut1.qa.suse.de
fozzie
quinn
blackbauhinia
arm4
openqaw5-xen
openqaw9-hyperv
All the rest
Our internal reference ticket https://progress.opensuse.org/issues/132617

openqaw9-hyperv.qa.suse.de PRG2E-D
openqaw5-xen.qa.suse.de PRG2E-D
quinn.qa.suse.de PRG2E-D
fibonacci.qam.suse.de PRG2E-D
sauron.qa.suse.de PRG2E-D
arm4.qe.suse.de PRG2E-D
voyager.qam.suse.de PRG2E-D
ada.qe.suse.de PRG2E-D
fozzie.qa.suse.de NUE-Scrapped_2023
orion.openqanet.opensuse.org PRG2-J
andromeda.openqanet.opensuse.org PRG2-J
amd-zen2-gpu-sut1.qa.suse.de PRG2E-D

It looks like most machines have been moved to PRG2e? However most of them have no IP, suggesting they're not usable yet?