action #124877: Failing pipelines because of unreachable machine openqaworker-arm-1 - openQA Infrastructure - openSUSE Project Management Tool

Custom queries

openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (ready issues)
QE tools team - backlog (w/o infra)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (w/o infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #124877

closed

QA - coordination #121720: [saga][epic] Migration to QE setup in PRG2+NUE3 while ensuring availability

QA - coordination #116623: [epic] Migration of SUSE Nbg based openQA+QA+QAM systems to new security zones

Failing pipelines because of unreachable machine openqaworker-arm-1

Added by livdywan over 1 year ago. Updated over 1 year ago.

Status:

Resolved

Priority:

High

Assignee:

mkittler

Category:

Target version:

openQA Project - Ready

Start date:

2023-02-08

Due date:

% Done:

Estimated time:

Description

Observation¶

./ipmi-recover-worker fails in the grafana web hook actions pipeline.

> ssh openqaworker-arm-1
ssh: connect to host openqaworker-arm-1 port 22: Connection timed out                                                   
lost connection

Acceptance criteria¶

AC1:

Rollback steps¶

If the worker can be recovered
- Add it back to salt
- Remove silence for alert on automatic actions panel

Suggestions¶

System gets stuck in the GRUB rescue shell after boot
Look into vKVM session following https://progress.opensuse.org/projects/openqav3/wiki#Accessing-old-BMCs-with-Java-iKVM-Viewer-when-ipmitool-does-not-work-eg-imagetester

Related issues 1 (0 open — 1 closed)

Is duplicate of openQA Infrastructure - action #124715: Failing pipelines because of unreachable machine openqaworker-arm-1

Rejected

2023-02-08

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by livdywan over 1 year ago

Copied from action #124715: Failing pipelines because of unreachable machine openqaworker-arm-1 added

Actions

Copy link

Updated by mkittler over 1 year ago

Status changed from New to In Progress
Assignee set to mkittler

(see #124715#note-3 and further comments for what I tried before)

I gave up to early with data.jnlp. With LANG=C _JAVA_AWT_WM_NONREPARENTING=1 javaws -nosecurity -jnlp data.jnlp I could mount an ISO (openSUSE TW network installer). Maybe I can fix the boot issues from there. However, now the boot is stuck at:

Loading driver at 0x03FF9B01000 EntryPoint=0x03FF9B1F000
Load Image() Return Status = EFI_SUCCESS
WatchDog Timer Status = EFI_SUCCESS
Unknown.Entry(3FF9B1F000)
Please press 't' to show the boot menu
on this console

The Virtual Media Session shows an upload speed of 12 mbit/s at most. So maybe it is just slow (not even using the full upload bandwidth of my DSL connection).

If we're lucky then just the GRUB config is damaged (e.g. because the machine crashed when it was rewritten).

EDIT: The image download eventually worked. Now it is stuck at:

[   34.420890][  T876] RPC: Registered udp transport module.
[   34.428264][  T876] RPC: Registered tcp transport module.
[   34.435618][  T876] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   36.268941][    C3] random: crng init done
[   36.805012][  T904] No iBFT detected.

Framebuffer device detected - continuing installation on console /dev/tty1.
Use boot option 'switch_to_fb=0' to prevent this.

Loading basic drivers
Hardware detection

So I need to find a way to specify switch_to_fb=0 (I haven't seen GRUB, maybe I missed it) or gain access to the framebuffer.

Actions

Copy link

Updated by openqa_review over 1 year ago

Due date set to 2023-03-08

Setting due date based on mean cycle time of SUSE QE Tools

Actions

Copy link

Updated by mkittler over 1 year ago

One can go into GRUB by spamming the up key (or likely any other key) and then some prompt occurs where one can press t.

We could boot again after reinstalling kernel-default from a chroot env and also re-configuring GRUB. However, then the systemd service for our NVMe setup was stuck. At least another reboot still worked (so the NVMe setup doesn't override out GRUB setup or something similar). Then it even could establish the NVMe setup again. Maybe it failed on the first attempt because I still had the TW netinstall mounted.

Actions

Copy link

Updated by mkittler over 1 year ago

Status changed from In Progress to Feedback

After a long BTRFS balancing the system was now able to boot. Let's see how it behaves once a few tests are running. I've been adding it back to salt.

My theory is that the GRUB config was damaged, maybe because the worker crashed while the config was written.

Actions

Copy link