action #111926: osd-deployment pipeline failed: test 481 -le 0, due to job age alert, likely just the raspberry pi based tests stuck in schedule - openQA Infrastructure (public) - openSUSE Project Management Tool

Custom queries

openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (dev)
QE tools team - backlog (ready issues)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (dev)
QE tools team - non-estimated (unblocked) issues (infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #111926

closed

osd-deployment pipeline failed: test 481 -le 0, due to job age alert, likely just the raspberry pi based tests stuck in schedule

Added by tinita over 2 years ago. Updated over 2 years ago.

Status:

Resolved

Priority:

High

Assignee:

nicksinger

Category:

Target version:

openQA Project (public) - Ready

Start date:

2022-06-01

Due date:

% Done:

Estimated time:

Tags:

reactive work, next-office-day

Description

Observation¶

https://gitlab.suse.de/openqa/osd-deployment/-/jobs/996509

++ echo '$ test ${#new} -le ${#old}'
$ test ${#new} -le ${#old}
++ test 481 -le 0
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

Related issues 2 (0 open — 2 closed)

Related to openQA Project (public) - action #111590: [alert] HPC jobs not picked up for multiple days, job age alert triggered

Resolved

livdywan

2022-05-25

2022-06-14

Actions

Related to openQA Project (public) - action #105855: [easy][beginner] openqa-worker@.service should handle NTP unavailability gracefully

Resolved

livdywan

2022-02-02

2022-07-22

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by tinita over 2 years ago

Project changed from 115 to openQA Infrastructure (public)

Actions

Copy link

Updated by tinita over 2 years ago

I retriggered and now it passes.
https://gitlab.suse.de/openqa/osd-deployment/-/jobs/997379

Actions

Copy link

Updated by okurz over 2 years ago

Related to action #111590: [alert] HPC jobs not picked up for multiple days, job age alert triggered added

Actions

Copy link

Updated by okurz over 2 years ago

Tags set to reactive work
Priority changed from Normal to High
Target version set to Ready

Actions

Copy link

Updated by livdywan over 2 years ago

The pipeline apparently finished June 1 8:18am GMT+2 which is before #111590#note-13 meaning this was just reported later than the assessment that the alert was fine.

Actions

Copy link

Updated by okurz over 2 years ago

Tags changed from reactive work to reactive work, next-office-day
Subject changed from osd-deployment pipeline failed: test 481 -le 0 to osd-deployment pipeline failed: test 481 -le 0, due to job age alert, likely just the raspberry pi based tests stuck in schedule
Assignee set to nicksinger

@nicksinger you mentioned this issue so you can look into this anyway the next day in office.

For now https://stats.openqa-monitor.qa.suse.de/d/7W06NBWGk/job-age?tab=alert&viewPanel=5&orgId=1&from=1653896632482&to=1654210000358 looks fine but if we are nearing the alert threshold and if this is still not solved until then or if we even see an additional alert at time then please pause the alert until the problem is fixed.

Actions

Copy link

Updated by nicksinger over 2 years ago

Status changed from New to Feedback

okurz wrote:

@nicksinger you mentioned this issue so you can look into this anyway the next day in office.

I asked @dheidler for support in https://suse.slack.com/archives/C02CANHLANP/p1654596394537099 - without any information about the setup there is not much I can do

Actions

Copy link

Updated by dheidler over 2 years ago

The worker host (which is a RPi 400 atm) was restarted but the openQA-worker@*.service didn't come up due to https://progress.opensuse.org/issues/105855

I fixed it for now by restarting the worker deamons.

Actions

Copy link

Updated by dheidler over 2 years ago

Related to action #105855: [easy][beginner] openqa-worker@.service should handle NTP unavailability gracefully added

Actions

Copy link

#10

Updated by nicksinger over 2 years ago

Status changed from Feedback to Resolved

I think we can resolve this again as we have a follow-up ticket about the real problem

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

openQA Project (public) » openQA Infrastructure (public)

Tags

Custom queries

action #111926

osd-deployment pipeline failed: test 481 -le 0, due to job age alert, likely just the raspberry pi based tests stuck in schedule

Observation¶

Updated by tinita over 2 years ago

Updated by tinita over 2 years ago

Updated by okurz over 2 years ago

Updated by okurz over 2 years ago

Updated by livdywan over 2 years ago

Updated by okurz over 2 years ago

Updated by nicksinger over 2 years ago

Updated by dheidler over 2 years ago

Updated by dheidler over 2 years ago

Updated by nicksinger over 2 years ago