action #97502: osd deployment failed due to openqaworker-arm-3 being down, needs to be worked around size:M - openQA Infrastructure (public) - openSUSE Project Management Tool

Custom queries

openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (dev)
QE tools team - backlog (ready issues)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (dev)
QE tools team - non-estimated (unblocked) issues (infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #97502

closed

osd deployment failed due to openqaworker-arm-3 being down, needs to be worked around size:M

Added by okurz over 3 years ago. Updated over 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

okurz

Category:

Target version:

openQA Project (public) - Ready

Start date:

2021-08-25

Due date:

% Done:

Estimated time:

Description

Observation¶

#97244 is the real issue but we need to workaround.

Steps¶

Take out openqaworker-arm-3 from production
retrigger failed step from deployment
bring back openqaworker-arm-3 into production manually or mention in another ticket as step to be done

Related issues 1 (0 open — 1 closed)

Related to openQA Infrastructure (public) - action #97244: openqaworker-arm-3 is offline and EngInfra wants us to create JiraSD tickets instead of infra size:M

Resolved

dheidler

2021-08-19

2021-09-17

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by ilausuch over 3 years ago

Status changed from Workable to In Progress
Assignee set to ilausuch

Actions

Copy link

Updated by okurz over 3 years ago

Description updated (diff)

Actions

Copy link

Updated by ilausuch over 3 years ago

Status changed from In Progress to Blocked

I did the two first suggested steps and now the pipeline passes the failed step. https://gitlab.suse.de/openqa/osd-deployment/-/pipelines

In the moment of writting this comment openqaworker-arm-3 seems down. So the last step cannot be done until thhe worker come back to life. I change the status of this ticket to blocked and write a comment in the other ticket #97244

Actions

Copy link

Updated by okurz over 3 years ago

Related to action #97244: openqaworker-arm-3 is offline and EngInfra wants us to create JiraSD tickets instead of infra size:M added

Actions

Copy link

Updated by okurz over 3 years ago

Priority changed from Urgent to Normal

thanks. This adressed the urgency hence reducing prio.

Actions

Copy link

Updated by ilausuch over 3 years ago

I checked today, and the openqaworker-arm-3 seems thhat is still down

Actions

Copy link

Updated by ilausuch over 3 years ago

Assignee deleted (~~ilausuch~~)

I removed myself from this ticket because I won't be working in this group for a period

Actions

Copy link

Updated by okurz over 3 years ago

Status changed from Blocked to In Progress
Assignee set to okurz

after #97244 resolved we can continue here. I triggered a manual explicit pipeline run for openqaworker-arm-3 in https://gitlab.suse.de/openqa/grafana-webhook-actions/-/jobs/589516 which actually recovered the machine. Now I readd the salt key and apply the current high state.

Actions

Copy link

Updated by okurz over 3 years ago

tests are correctly executed. https://monitor.qa.suse.de/d/WDopenqaworker-arm-3/worker-dashboard-openqaworker-arm-3?orgId=1&refresh=1m&from=now-24h&to=now shows some data now but https://monitor.qa.suse.de/d/1bNU0StZz/automatic-actions?viewPanel=4&orgId=1 says that there is still no data. Restarted services on monitor.qa "influxdb grafana telegraf" but no change. I assume it's actually also here (same as in the other ticket where I struggled) the telegraf service on the webUI host as that one does the actual ping to each host. So I restarted the service on osd, let's see. If this works then I should ensure that the telegraf service restarts on config file changes because I assume that is the case for both the latest postgresql related changes as well as here for the host.

EDIT: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/580 created for telegraf service. https://monitor.qa.suse.de/d/1bNU0StZz/automatic-actions?editPanel=4&viewPanel=4&orgId=1 shows data again same as https://monitor.qa.suse.de/d/4KkGdvvZk/osd-status-overview?orgId=1

Actions

Copy link

#10