⚲

Project

General

Profile

Home
Projects
Help

Search:

openQA Infrastructure (public)

All Projects

openQA Infrastructure (public)

Overview
Activity
Roadmap
Issues

Custom queries

openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (dev)
QE tools team - backlog (ready issues)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE Tools Team - Beginner
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE Tools Team - Expert
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (dev)
QE tools team - non-estimated (unblocked) issues (infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #174586

closed

Incomplete jobs (not restarted) of last 24h alert Salt

Added by jbaier_cz 3 months ago. Updated 3 months ago.

Status:

Resolved

Priority:

Urgent

Assignee:

gpathak

Category:

Target version:

openQA Project (public) - Ready

Start date:

2024-12-19

Due date:

2025-01-03

% Done:

Estimated time:

Tags:

alert, infra, reactive work

Description

Observation¶

Values
B0=365 
Labels
alertname   Incomplete jobs (not restarted) of last 24h alert
grafana_folder  Salt
rule_uid    cXo2cmBVk

https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&viewPanel=panel-17&from=2024-12-17T07:35:39.219Z&to=2024-12-19T07:38:58.850Z

The spike could be due to the CDN and SCC problems, see https://suse.slack.com/archives/C02AYV7UJSD/p1734451708127589 for context. The alert is ok right now, but there is still a bunch of incompletes in the panel, might be worth investigating.

Related issues 2 (0 open — 2 closed)

Copied from openQA Infrastructure (public) - action #154345: Incomplete jobs (not restarted) of last 24h alert Salt

Resolved

mkittler

Actions

Copied to openQA Infrastructure (public) - action #175473: OpenQA Jobs test - Incomplete jobs (not restarted) of last 24h alert Salt

Resolved

okurz

2024-12-19

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by jbaier_cz 3 months ago

Copied from action #154345: Incomplete jobs (not restarted) of last 24h alert Salt added

Actions

Copy link

Updated by okurz 3 months ago

Tags changed from reactive work, alert to reactive work, alert, infra
Priority changed from High to Urgent

Actions

Copy link

Updated by gpathak 3 months ago

Assignee set to gpathak

Actions

Copy link

Updated by gpathak 3 months ago · Edited

Status changed from New to In Progress

Seems like the issue related to SCC CDN is resolved.
The tests are passing:

https://openqa.suse.de/tests/16250335#step/zypper_ref/18
https://openqa.suse.de/tests/16252220#step/zypper_extend/92
https://openqa.suse.de/tests/16236266#step/suseconnect_scc/22
https://openqa.suse.de/tests/16252253#step/scc_registration/6 The dashboard now shows pretty low incomplete jobs:https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&viewPanel=panel-17&from=now-24h&to=now

In particular this job always fails https://openqa.suse.de/tests/16248796#line-66, not sure why the YAML schedule is missing and how we can provide the Schedule file.
And tests related to baremetal worker is failing with error Could not retrieve required variable SUT_IP: https://openqa.suse.de/tests/16251337#line-117

Actions

Copy link