QA » openQA Project

Observation¶

Date: Wed, 04 Sep 2024 12:05:07 +0000
Subject: Munin - minion hook failed - see openqa-gru service logs for details - opensuse.org :: openqa.opensuse.org                                                                                           

opensuse.org :: openqa.opensuse.org :: hook failed - see openqa-gru service logs for details
        CRITICALs: rc_failed_per_5min is 32.00 (outside range [:10]).

% journalctl -u openqa-gru --since '2024-09-04'
...
Sep 04 12:02:24 ariel openqa-gru[7200]: 404 Not Found
Sep 04 12:02:44 ariel openqa-gru[7200]: 
Sep 04 12:02:44 ariel openqa-gru[11188]: 'http://openqa.opensuse.org/tests/4454336' does not have autoinst-log.txt but is rather old, ignoring
Sep 04 12:03:28 ariel openqa-gru[18613]: 404 Not Found
Sep 04 12:03:31 ariel openqa-gru[18613]: 
Sep 04 12:03:31 ariel openqa-gru[18944]: 404 Not Found
Sep 04 12:03:36 ariel openqa-gru[18944]: 
Sep 04 12:03:36 ariel openqa-gru[20180]: 404 Not Found
Sep 04 12:03:36 ariel openqa-gru[20180]: 
Sep 04 12:03:36 ariel openqa-gru[20178]: 404 Not Found
Sep 04 12:03:42 ariel openqa-gru[20178]: 
Sep 04 12:03:42 ariel openqa-gru[21430]: 404 Not Found
Sep 04 12:03:42 ariel openqa-gru[21430]: 
Sep 04 12:03:42 ariel openqa-gru[21445]: 404 Not Found
Sep 04 12:03:49 ariel openqa-gru[21445]: 
Sep 04 12:03:49 ariel openqa-gru[22756]: 404 Not Found
Sep 04 12:03:51 ariel openqa-gru[22756]: 
Sep 04 12:03:51 ariel openqa-gru[23156]: 404 Not Found
Sep 04 12:03:54 ariel openqa-gru[23156]: 
Sep 04 12:03:54 ariel openqa-gru[23770]: 404 Not Found
Sep 04 12:03:55 ariel openqa-gru[23770]: 
Sep 04 12:03:55 ariel openqa-gru[24146]: 404 Not Found
Sep 04 12:03:56 ariel openqa-gru[24146]: 
Sep 04 12:03:56 ariel openqa-gru[24646]: 404 Not Found
Sep 04 12:04:07 ariel openqa-gru[24646]: 
...

Unfortunately we don't see a script or line number, although we are using a mechanism, e.g. in runcurl to report the caller in case of an error. Maybe this is a call that doesn't use that wrapper.

One related minion job (guessing from the timestamp) is this, I guess:
https://openqa.opensuse.org/minion/jobs?id=4272268

notes:
  hook_cmd: env from_email=o3-admins@suse.de scheme=http enable_force_result=true
    email_unreviewed=true exclude_group_regex='(Development|Open Build Service|Others|Kernel).*/.*'
    /opt/os-autoinst-scripts/openqa-label-known-issues-and-investigate-hook
  hook_rc: 1
  hook_result: ''

https://openqa.opensuse.org/tests/4454693

Acceptance Criteria¶

• AC1: Errors include line number and URL

Suggestions¶

• Search for curl calls not using runcurl
• Try to reproduce by calling the hook script on the same job
• https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues-and-investigate-hook