Project

General

Profile

Actions
Copy link

action #155737

closed

Salt pillars pipelines fail due to refused connection errors on telegraf

Added by livdywan 2 months ago. Updated 2 months ago.

Status:
Rejected
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
Due date:
% Done:

0%

Estimated time:
Tags:
alert, salt, infra, reactive work, pillars

Description

Observation

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2300476

++ grep ' E! ' salt_post_deploy_checks.log
    2024-02-21T09:21:58Z E! [inputs.http] Error in plugin: [url=http://localhost:9530/influxdb/minion]: Get "http://localhost:9530/influxdb/minion": dial tcp [::1]:9530: connect: connection refused
    2024-02-21T09:21:59Z E! [telegraf] Error running agent: input plugins recorded 1 errors

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2300697

monitor.qe.nue2.suse.org:
    2024-02-21T10:55:17Z E! [inputs.exec] Error in plugin: exec: command timed out for command '/etc/telegraf/scripts/maintenance_queue_monitor.py': 
    2024-02-21T10:55:17Z E! [inputs.exec] Error in plugin: exec: command timed out for command '/etc/telegraf/scripts/submission_queue_monitor.py': 
    2024-02-21T10:55:21Z E! [telegraf] Error running agent: input plugins recorded 2 errors
    telegraf errors
++ grep ' E! ' salt_post_deploy_checks.log
    2024-02-21T10:54:59Z E! [inputs.http] Error in plugin: [url=http://localhost:9530/influxdb/minion]: Get "http://localhost:9530/influxdb/minion": dial tcp [::1]:9530: connect: connection refused
    2024-02-21T10:54:59Z E! [telegraf] Error running agent: input plugins recorded 1 errors
    2024-02-21T10:55:17Z E! [inputs.exec] Error in plugin: exec: command timed out for command '/etc/telegraf/scripts/maintenance_queue_monitor.py': 
    2024-02-21T10:55:17Z E! [inputs.exec] Error in plugin: exec: command timed out for command '/etc/telegraf/scripts/submission_queue_monitor.py': 
    2024-02-21T10:55:21Z E! [telegraf] Error running agent: input plugins recorded 2 errors

Suggestions

Related issues 1 (0 open — 1 closed)

Related to openQA Project - action #155716: [alert] openqa-worker-cacheservice fails to start on worker29.oqa.prg2.suse.org with "Database has been corrupted: DBD::SQLite::db commit failed: disk I/O error" size:SResolvedmkittler2024-02-212024-03-07

Actions
Actions
Copy link
 #1

Updated by tinita 2 months ago

Maybe worth mentioning that this is failing on worker29:

worker29.oqa.prg2.suse.org:
    2024-02-21T09:21:58Z E! [inputs.http] Error in plugin: [url=http://localhost:9530/influxdb/minion]: Get "http://localhost:9530/influxdb/minion": dial tcp [::1]:9530: connect: connection refused
    2024-02-21T09:21:59Z E! [telegraf] Error running agent: input plugins recorded 1 errors

Related to #155716 ?

Actions
Copy link
 #2

Updated by mkittler 2 months ago

  • Related to action #155716: [alert] openqa-worker-cacheservice fails to start on worker29.oqa.prg2.suse.org with "Database has been corrupted: DBD::SQLite::db commit failed: disk I/O error" size:S added
Actions
Copy link
 #3

Updated by okurz 2 months ago

  • Status changed from New to Rejected
  • Assignee set to okurz
Actions
Copy link

Also available in: Atom PDF