⚲

Project

General

Profile

Home
Projects
Help

Search:

openQA Infrastructure (public)

All Projects

openQA Infrastructure (public)

Overview
Activity
Roadmap
Issues

Custom queries

openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (dev)
QE tools team - backlog (ready issues)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (dev)
QE tools team - non-estimated (unblocked) issues (infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #151597

closed

[alert] osiris-1 (osiris-1: partitions usage (%) alert Generic partitions_usage_alert_osiris-1 generic

Added by tinita about 1 year ago. Updated about 1 year ago.

Status:

Resolved

Priority:

High

Assignee:

tinita

Category:

Target version:

openQA Project (public) - Ready

Start date:

2023-11-28

Due date:

% Done:

Estimated time:

Tags:

monitoring, grafana, infra

Description

Observation¶

Alert from Grafana:

1 firing alert instance
[IMAGE]

 GROUPED BY 

hostname=osiris-1

1 firing instances

Firing [stats.openqa-monitor.qa.suse.de]
osiris-1: partitions usage (%) alert
View alert [stats.openqa-monitor.qa.suse.de]
Values
A0=96.03554068410229 
Labels
alertname
osiris-1: partitions usage (%) alert
grafana_folder
Generic
hostname
osiris-1
rule_uid
partitions_usage_alert_osiris-1
type
generic
Silence [stats.openqa-monitor.qa.suse.de]
View dashboard [stats.openqa-monitor.qa.suse.de]
View panel [stats.openqa-monitor.qa.suse.de]
Observed 32s before this notification was delivered, at 2023-11-28 11:49:00 +0100 CET

http://stats.openqa-monitor.qa.suse.de/alerting/grafana/partitions_usage_alert_osiris-1/view?orgId=1
http://stats.openqa-monitor.qa.suse.de/alerting/grafana/partitions_usage_alert_osiris-1/view?orgId=1

Related issues 1 (0 open — 1 closed)

Related to openQA Infrastructure (public) - action #138650: partition usage panels show a long list of undefined and no reasonable graphs at least for some generic machines size:M

Resolved

tinita

2023-10-27

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by tinita about 1 year ago

Related to action #138650: partition usage panels show a long list of undefined and no reasonable graphs at least for some generic machines size:M added

Actions

Copy link

Updated by tinita about 1 year ago · Edited

Status changed from New to In Progress

Since the alert doesn't really tell me which partition is problematic (see #138650) I had a look on osiris-1 and it's /var/lib/libvirt/images/dist.suse.de

Actions

Copy link

Updated by tinita about 1 year ago

The mentioned partition is of the type nfs4.
The alert is supposed to ignore nfs mounts, but checks for the exact string nfs only.

I just changed it into a regex:
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1053 Ignore all nfs partitions

Actions

Copy link

Updated by tinita about 1 year ago

It should be noted that the alert is about disk usage regarding size:

SELECT mean("used_percent") AS "used_percent" FROM "disk" WHERE ("host" = 'osiris-1' AND fstype !~ /^nfs/ AND fstype != 'udf') AND $timeFilter GROUP BY time($interval), "device", "fstype" fill(null)

while the linked panel is about diskio:

SELECT non_negative_derivative(mean(reads),1s) as "read" FROM "diskio" WHERE "host" = 'osiris-1' AND $timeFilter GROUP BY time($interval), *

I think we should have two different panels.

Actions

Copy link

Updated by tinita about 1 year ago

Tags set to infra, monitoring, grafana

Actions

Copy link

Updated by okurz about 1 year ago

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1053 merged.

Actions

Copy link

Updated by okurz about 1 year ago

The actual "alert query" still needs to be adjusted, e.g. see https://stats.openqa-monitor.qa.suse.de/alerting/grafana/partitions_usage_alert_osiris-1/view?returnTo=%2Falerting%2Flist%3Fsearch%3Dstate%3Afiring . See two occurences here https://gitlab.suse.de/search?search=fstype%20!%3D%20%27nfs%27%20AND&nav_source=navbar&project_id=743&group_id=39&search_code=true&repository_ref=master

Actions

Copy link

Updated by tinita about 1 year ago

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1054 Ignore all nfs partitions in alerts

Actions

Copy link

Updated by tinita about 1 year ago

Status changed from In Progress to Feedback

Actions

Copy link

#10

Updated by tinita about 1 year ago

Status changed from Feedback to Resolved

Alert gone

Actions

Copy link

Also available in: Atom PDF