Project

General

Profile

Actions

action #106933

closed

Use PSU capabilites to power cycle openqaworker-arm-[1-3] instead of infra tickets size:M

Added by nicksinger about 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2021-11-17
Due date:
% Done:

0%

Estimated time:

Description

Observation

Today we installed a controllable PSU (qaps06nue.qa.suse.de) into the rack for openqaworker-arm-[1-3]. We should make use of them in our automatic recovery pipeline to power cycle the BMC if it is down

Here is the mapping for each machine:

ARM1: Plug 1
ARM2: Plug 2+3
ARM3: Plug 4+5
ARM4: Plug 6
ARM6: Plug 7

Suggestions

Research how we can automate the power-cycle on the PSU side. The PSUs have a webinterface which can be scripted/scraped (no real API AFAIK) and several access options like e.g. ssh, telnet, ftp.
Ask nsinger for the password if you want to browse through the web ui. Keep in mind to choose a security sensible option (e.g. an encrypted channel).
Integrate this automation into our pipeline at https://gitlab.suse.de/openqa/grafana-webhook-actions/-/blob/master/ipmi-recover-worker. It should replace the create_ticket() function (https://gitlab.suse.de/openqa/grafana-webhook-actions/-/blob/master/ipmi-recover-worker#L26-28).

Acceptance criteria

  • AC1: Infra tickets are no longer created
  • AC2: grafana-webhook-actions uses some API of the PSU to automate the power cycle

Related issues 1 (0 open1 closed)

Related to openQA Infrastructure - action #102575: Prevent false-positive ticket reporting for openqaworker-arm-3Resolvedmkittler2021-11-172021-12-02

Actions
Actions

Also available in: Atom PDF