action #94678
Updated by okurz over 3 years ago
## Observation Seems like since some days there are no openqa-investigate jobs triggered anymore, e.g. see on openqa.suse.de/tests/ where normally we should see many now I don't see any. o3 works fine, compare https://openqa.suse.de/tests/?match=:investigate: and https://openqa.opensuse.org/tests/?match=:investigate: to see https://openqa.opensuse.org/tests/ look for jobs with ":investigate:" in the name. `journalctl -u openqa-gru` on osd shows: ``` Jun 24 15:57:01 openqa openqa-gru[843]: [2021-06-24T15:56:58.411 CEST] [debug] git fetch: remote: Total 0 (delta 0), reused 0 (delta 0), pack-r> Jun 24 15:57:01 openqa openqa-gru[843]: Jun 24 15:57:01 openqa openqa-gru[843]: Could not find '07a51bb2ffb67f4fc5bafb243e0bdcf8d2b50bed' in complete history at /usr/lib/os-autoinst/O> Jun 24 15:57:01 openqa openqa-gru[843]: --- Jun 24 15:57:01 openqa openqa-gru[843]: 1 unknown issues to be reviewed: Jun 24 15:57:01 openqa openqa-gru[843]: - https://openqa.suse.de/tests/6325089 isotovideo died: Could not find '07a51bb2ffb67f4fc Jun 24 15:57:03 openqa openqa-gru[843]: https://openqa.suse.de/tests/6325091 : Unknown issue, to be reviewed -> https://openqa.suse.de/tests/63> Jun 24 15:57:03 openqa openqa-gru[843]: Likely the error is within this log excerpt, last lines before shutdown: Jun 24 15:57:03 openqa openqa-gru[843]: --- Jun 24 15:57:03 openqa openqa-gru[843]: [2.0K blob data] Jun 24 15:57:03 openqa openqa-gru[843]: [1.2K blob data] Jun 24 15:57:03 openqa openqa-gru[843]: Jun 24 15:57:03 openqa openqa-gru[843]: [2021-06-24T15:56:59.337 CEST] [debug] git fetch: remote: Total 0 (delta 0), reused 0 (delta 0), pack-r> Jun 24 15:57:03 openqa openqa-gru[843]: Jun 24 15:57:03 openqa openqa-gru[843]: Could not find '07a51bb2ffb67f4fc5bafb243e0bdcf8d2b50bed' in complete history at /usr/lib/os-autoinst/O> Jun 24 15:57:03 openqa openqa-gru[843]: --- Jun 24 15:57:03 openqa openqa-gru[843]: 1 unknown issues to be reviewed: Jun 24 15:57:03 openqa openqa-gru[843]: - https://openqa.suse.de/tests/6325091 isotovideo died: Could not find '07a51bb2ffb67f4fc Jun 24 16:32:15 openqa openqa-gru[843]: grep: exceeded PCRE's backtracking limit Jun 24 18:59:10 openqa openqa-gru[843]: unable to clone job 6324478: it is part of a parallel or directly chained cluster (not supported) Jun 24 18:59:11 openqa openqa-gru[843]: unable to clone job 6324478: it is part of a parallel or directly chained cluster (not supported) ``` ## Acceptance criteria * **AC1:** No obvious error messages in openqa-gru logs * ~~**AC2:** openqa-investigate jobs are triggered again on osd~~ ## Suggestions * Check for failed minion jobs on https://openqa.suse.de/minion/jobs?state=failed * Try to run `openqa-investigate` scripts manually from github.com/os-autoinst/scripts/ against jobs on osd and check for errors ## Further details openqa-gru runs multiple asynchronous background processes. The ones mentioned above are from "job_post_hooks".