action #169576: Recover qa-power8-3 power machine size:S - openQA Project (public) - openSUSE Project Management Tool

Actions

action #169576

closed

coordination #157969: [epic] Upgrade all our infrastructure, e.g. o3+osd workers+webui and production workloads, to openSUSE Leap 15.6

Recover qa-power8-3 power machine size:S

Added by gpathak about 1 month ago. Updated about 1 month ago.

Status:

Resolved

Priority:

Normal

Assignee:

nicksinger

Category:

Organisational

Target version:

Ready

Start date:

2024-11-08

Due date:

% Done:

Estimated time:

Tags:

infra

Description

Motivation¶

O3 workers were being upgraded to Leap 15.6 as a part of this: https://progress.opensuse.org/issues/157972
both the power machines - kerosene and qa-power8-3 were rendered useless in the upgrade process

Acceptance criteria¶

AC1: qa-power8-3 worker machines should be up and running with Leap 15.5 or Leap 15.6 whichever version works

Suggestions¶

read https://progress.opensuse.org/projects/openqav3/wiki#Distribution-upgrades
Reserve some time when the workers are only executing a few or no openQA test jobs
Keep IPMI interface ready and test that Serial-over-LAN works for potential recovery
Apply the workaround for #162296, i.e. zypper al -m "boo#1227616" *firewall*
Use the instructions from above
After upgrade reboot and check everything working as expected, if not rollback, e.g. with transactional-update rollback

Files

Download all files

pb-discover.log (11.4 KB) pb-discover.log		gpathak, 2024-10-25 09:21
clipboard-202411051428-dg60d.png (52.6 KB) clipboard-202411051428-dg60d.png		gpathak, 2024-11-05 08:58

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by gpathak about 1 month ago

Copied from action #157972: Upgrade o3 workers to openSUSE Leap 15.6 size:S added

Actions

Copy link

Updated by nicksinger about 1 month ago

Status changed from New to Workable

We broke this out into another ticket to have at least one working machine back for o3. I recovered it on Friday be explicitly selecting leap15.5 again in petitboot. Since then it conducted at least one reboot and came up automatically again. I have to conduct a proper reboot check but currently everything looks fine.

Actions

Copy link

Updated by nicksinger about 1 month ago

Status changed from Workable to In Progress

started ./reboot-stability-check qa-power8-3.openqanet.opensuse.org on ariel (I cannot ping the machine from my own machine due to the separated network).

Actions

Copy link

Updated by nicksinger about 1 month ago

Status changed from In Progress to Resolved

the script managed 27 reboots and the machine came up all the time automatically, I would consider this stable.

Actions

Copy link

Updated by okurz about 1 month ago

Status changed from Resolved to Feedback

Great news! But if I understood correctly that machine is still on Leap 15.5 . What do we need to keep in mind for a Leap 15.6 upgrade?

Actions

Copy link

Updated by nicksinger about 1 month ago

Status changed from Feedback to Blocked

okurz wrote in #note-5:

Great news! But if I understood correctly that machine is still on Leap 15.5 . What do we need to keep in mind for a Leap 15.6 upgrade?

oh, you are right I was just too focused on getting a single machine back into a stable state. Currently I would wait until a second power machine proves stables; https://progress.opensuse.org/issues/157972#note-48

Actions

Copy link