action #164853: [alert][FIRING:1] s390zl13 (s390zl13: Memory usage alert Generic memory_usage_alert_s390zl13 generic) size:S - openQA Infrastructure (public) - openSUSE Project Management Tool

Custom queries

openQA Infrastructure Project
openqa-review - Closed tickets last updated by openqa-review, last 30 days
QA roadmap long-term
QA SLE functional
QA SLE Functional - closed in last 14 days
QA SLE Functional - High, need to be refined
QA SLE Functional - over cycle time median
QA SLE u
QA SLE y
QA tools (tag not necessary in openQA and subprojects)
QA tools tag (tag not necessary in openQA and subprojects; excluding tickets in "Ready" version as they are already on the backlog)
QAC - Backlog
QE tools team - backlog (dev)
QE tools team - backlog (ready issues)
QE tools team - backlog SLA high
QE tools team - backlog SLA immediate
QE tools team - backlog SLA no immediate/urgent in feedback/blocked
QE tools team - backlog SLA normal
QE tools team - backlog SLA urgent
QE tools team - backlog SLO high
QE tools team - backlog SLO normal
QE tools team - backlog SLO urgent
QE tools team - backlog, high-level view (epics and higher)
QE tools team - backlog, non-reactive work, needs parent
QE tools team - backlog, top-level view (all sagas)
QE tools team - closed within last 14 days
QE tools team - closed within last 60 days
QE tools team - closed yesterday
QE Tools Team - Collaborative Session
QE tools team - due date forecast
QE tools team - exceeding due-date
QE tools team - infrastructure backlog
QE tools team - next - sorted by update time
QE tools team - next issues
QE tools team - non-estimated (unblocked) issues (dev)
QE tools team - non-estimated (unblocked) issues (infra)
QE tools team - ready issues - Workable
QE tools team - ready, not assigned/blocked/low
QE tools team - SLO high forecast
QE tools team - update forecast
QE tools team - updated by priority
QE tools team - what members of the team are working on - Feedback (not-low)
QE Tools Team Backlog By Assignee
Tools Team Retrospective
Tools Team Retrospective (not estimated or assigned)

Actions

Copy link

action #164853

closed

[alert][FIRING:1] s390zl13 (s390zl13: Memory usage alert Generic memory_usage_alert_s390zl13 generic) size:S

Added by okurz 5 months ago. Updated 4 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

okurz

Category:

Regressions/Crashes

Target version:

openQA Project (public) - Ready

Start date:

2024-08-02

Due date:

% Done:

Estimated time:

Tags:

alert, memory, infra

Description

Observation¶

https://stats.openqa-monitor.qa.suse.de/d/GDs390zl13/dashboard-for-s390zl13?orgId=1&viewPanel=12054&from=1722568647371&to=1722580533085
shows high memory usage over the course of 50m. As we had repeated memory alerts especially on those s390 hosts and no problem reports by users that tests are affected I would like to loosen the alert conditions a bit.

Suggestions¶

Adjust memory thresholds
Combine an absolute and a relative threshold

Rollback actions¶

DONE Remove alert from https://monitor.qa.suse.de/alerting/silences called rule_uid=~memory_usage_.*s390

Related issues 1 (0 open — 1 closed)

Related to openQA Infrastructure (public) - action #150887: [alert] [FIRING:1] s390zl12 (s390zl12: partitions usage (%) alert Generic partitions_usage_alert_s390zl12 generic), also s390zl13 size:M

Resolved

okurz

2023-11-15

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by okurz 5 months ago

Due date set to 2024-08-16
Status changed from New to Feedback

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1241

Actions

Copy link

Updated by okurz 5 months ago

Description updated (diff)

Actions

Copy link

Updated by okurz 4 months ago

Related to action #150887: [alert] [FIRING:1] s390zl12 (s390zl12: partitions usage (%) alert Generic partitions_usage_alert_s390zl12 generic), also s390zl13 size:M added

Actions

Copy link

Updated by livdywan 4 months ago

Subject changed from [alert][FIRING:1] s390zl13 (s390zl13: Memory usage alert Generic memory_usage_alert_s390zl13 generic) to [alert][FIRING:1] s390zl13 (s390zl13: Memory usage alert Generic memory_usage_alert_s390zl13 generic) size:S
Description updated (diff)

Actions

Copy link

Updated by okurz 4 months ago

Status changed from Feedback to In Progress

As we decided we want to consider absolute values as well I did sudo salt \* cmd.run 'free -g | grep Mem:' to find lower limits. Lowest available values are:

schort-server.qe.nue2.suse.org:
    Mem:            …         901
tumblesle.qe.nue2.suse.org:
    Mem:            …         266
jenkins.qe.nue2.suse.org:
    Mem:            …        1767

so we could say anything below 200M is probably problematic.

Actions

Copy link

Updated by okurz 4 months ago

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1244

Actions

Copy link

Updated by nicksinger 4 months ago

okurz wrote in #note-5:

As we decided we want to consider absolute values as well I did sudo salt \* cmd.run 'free -g | grep Mem:' to find lower limits. Lowest available values are:
schort-server.qe.nue2.suse.org:
    Mem:            …         901
tumblesle.qe.nue2.suse.org:
    Mem:            …         266
jenkins.qe.nue2.suse.org:
    Mem:            …        1767
so we could say anything below 200M is probably problematic.

I'm not following with the conclusion here but now also better understand why you previously questioned this approach. Let me try to explain: I would expect that we have no system running with this low memory available. Reason is that a modern system commonly assumes having a little bit more available. One thing I did not take into consideration is swap which gives us headroom without needlessly blowing up RAM for a very small VM (like the ones above). So could we extend the alert to "free mem + free swap < 1G"?

Actions

Copy link