Project

General

Profile

action #116473

Add OSD PowerPC workers to automatic recovery we already have for ARM workers

Added by mkittler 3 months ago. Updated 3 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
Start date:
2022-09-12
Due date:
% Done:

0%

Estimated time:

Description

There workers are often failing similarly to the ARM workers┬╣ and at this point we should not need to manually recover them so frequently.

Suggestions

  • Note that for these workers a power cycle does not always work but power reset seems to work always. So maybe that detail needs to be adjusted for PowerPC workers.
  • I suppose all PowerPC workers controllable via IPMI should be considered (see workerconf.sls in salt pillars).

┬╣ They just randomly crash and logs just end without further clues, e.g. #114565#note-40. In addition, they sometimes also get stuck at boot.


Related issues

Related to openQA Infrastructure - action #114565: recover qa-power8-4+qa-power8-5 size:MBlocked2022-07-222023-02-10

Related to openQA Infrastructure - action #116437: Recover qa-power8-5 size:MResolved

History

#1 Updated by okurz 3 months ago

  • Target version set to future

Can you provide a little bit more context regarding "failing similarly"? Didn't you also have a bug report and there were suggestions regarding kdump and such?

#2 Updated by mkittler 3 months ago

Can you provide a little bit more context regarding "failing similarly"?

There's not much to say about it. They just randomly crash and the journal doesn't give one any clues; it just ends at some point. In addition, they sometimes also get stuck at boot.

Didn't you also have a bug report and there were suggestions regarding kdump and such?

Yes. I can link the relevant progress ticket for additional context. However, I'm not sure whether we can fix this problem anytime soon.

#3 Updated by mkittler 3 months ago

  • Description updated (diff)

#4 Updated by mkittler 3 months ago

  • Related to action #114565: recover qa-power8-4+qa-power8-5 size:M added

#5 Updated by mkittler 3 months ago

#6 Updated by mkittler 3 months ago

  • Description updated (diff)

Also available in: Atom PDF