action #97364: openqaworker-arm-2 and openqaworker-arm-3 seem to be offline, alerts had been triggered size:S - openQA Infrastructure (public) - openSUSE Project Management Tool

Custom queries

openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (dev)
QE tools team - backlog (ready issues)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (dev)
QE tools team - non-estimated (unblocked) issues (infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #97364

closed

openqaworker-arm-2 and openqaworker-arm-3 seem to be offline, alerts had been triggered size:S

Added by okurz over 3 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

nicksinger

Category:

Target version:

openQA Project (public) - Ready

Start date:

2021-08-23

Due date:

% Done:

Estimated time:

Description

Observation¶

https://monitor.qa.suse.de/alerting/list?state=not_ok
shows

[openqa] openqaworker-arm-2 online (long-time) alert
ALERTING for 4 days
Edit alert
[openqa] openqaworker-arm-3 online (long-time) alert
ALERTING for 5 days

Related issues 2 (0 open — 2 closed)

Related to openQA Infrastructure (public) - action #97244: openqaworker-arm-3 is offline and EngInfra wants us to create JiraSD tickets instead of infra size:M

Resolved

dheidler

2021-08-19

2021-09-17

Actions

Related to openQA Infrastructure (public) - action #113561: failed pipelines for openQABot and bot-ng because of an expired cert

Resolved

okurz

2022-07-13

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by okurz over 3 years ago

Related to action #97244: openqaworker-arm-3 is offline and EngInfra wants us to create JiraSD tickets instead of infra size:M added

Actions

Copy link

Updated by nicksinger over 3 years ago

Status changed from New to In Progress
Assignee set to nicksinger

Actions

Copy link

Updated by okurz over 3 years ago

Subject changed from openqaworker-arm-2 and openqaworker-arm-3 seem to be offline, alerts had been triggered to openqaworker-arm-2 and openqaworker-arm-3 seem to be offline, alerts had been triggered size:S

discussed in daily, out of scope: changing automatic ticket creation. in scope: please take a short look into the pipeline why power cycling over gitlab did not work.

Actions

Copy link

Updated by nicksinger over 3 years ago

Status changed from In Progress to Resolved

Both workers look good again after a manual reboot and show up as "online" in grafana: https://stats.openqa-monitor.qa.suse.de/d/1bNU0StZz/automatic-actions?orgId=1

Pipeline triggered for both machine. The execution for arm-2 failed due to some CI issue which I think is something we can't really change:

ERROR: Job failed (system failure): prepare environment: error sending request: Post "https://caasp-master.suse.de:6443/api/v1/namespaces/gitlab/pods/runner-h1wecofv-project-4652-concurrent-0l8dw9/attach?container=helper&stdin=true": dial tcp: lookup caasp-master.suse.de on [2620:113:80c0:8080:10:160:2:88]:53: server misbehaving. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

(https://gitlab.suse.de/openqa/grafana-webhook-actions/-/jobs/536479#L13)

For arm-3 I created https://progress.opensuse.org/issues/97382 to fix our pipeline.

Actions

Copy link