Project

General

Profile

Actions

action #159840

closed

Munin - minion hook failed - see openqa-gru service logs for details - opensuse.org :: openqa.opensuse.org size:M

Added by livdywan 3 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Observation

Email from 2024-05-02 02.25 CEST

opensuse.org :: openqa.opensuse.org :: hook failed - see openqa-gru service logs for details
    CRITICALs: rc_failed_per_5min is 17.00 (outside range [:10]).
sudo journalctl -u openqa-gru --since '2024-05-02 0:25' --until '2024-05-02 0:30'                                                              
May 02 00:25:21 new-ariel openqa-gru[5979]:                                                                                                                           
May 02 00:25:21 new-ariel openqa-gru[5844]: Connect timeout                                                                                                           
May 02 00:25:21 new-ariel openqa-gru[5844]:                                                                                                                           
May 02 00:25:21 new-ariel openqa-gru[5843]: /opt/os-autoinst-scripts/openqa-label-known-issues: ERROR: line 129                                                       
May 02 00:25:21 new-ariel openqa-gru[5817]: /opt/os-autoinst-scripts/openqa-label-known-issues: ERROR: line 129

Suggestions

  • Check the journal e.g. sudo journalctl -u openqa-gru
  • Review the code at openqa-label-known-issues:129
  • Consider improving the scripting, error reporting, etc. e.g. reveal the URL/ API route
  • Check impact of a high timeout (and thus possibly jobs queuing up) on other time critical jobs (like saving needles)
Actions #1

Updated by okurz 3 months ago

  • Target version set to Ready
Actions #2

Updated by livdywan 3 months ago

  • Description updated (diff)
Actions #3

Updated by okurz 2 months ago

  • Target version changed from Ready to Tools - Next
Actions #4

Updated by okurz about 2 months ago

  • Target version changed from Tools - Next to Ready
Actions #5

Updated by okurz about 2 months ago

  • Subject changed from Munin - minion hook failed - see openqa-gru service logs for details - opensuse.org :: openqa.opensuse.org to Munin - minion hook failed - see openqa-gru service logs for details - opensuse.org :: openqa.opensuse.org size:M
  • Description updated (diff)
  • Category set to Feature requests
  • Status changed from New to Workable
Actions #6

Updated by dheidler about 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to dheidler
Actions #7

Updated by dheidler about 2 months ago

  • Status changed from In Progress to Feedback

Increased retry timeout to 2m and added some logging:
https://github.com/os-autoinst/scripts/pull/321

Actions #8

Updated by dheidler about 2 months ago

  • Status changed from Feedback to Resolved
Actions

Also available in: Atom PDF