action #132488
closed
gitlab CI shows showing no logs or are getting stuck (was: qem-bot sync aggregates gitlab CI job times out after 2h) size:M
Added by okurz over 1 year ago.
Updated over 1 year ago.
Description
Observation¶
https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1680349 times out after 2h:
output of the bot-ng call:
++ ./qem-bot/bot-ng.py -c /etc/openqabot --token [MASKED] aggr-sync-results
++ tee bot_aggr-sync-results_0.log
2023-07-10 00:21:25 INFO Config /etc/openqabot/bot.yml does not have aggregate
…
2023-07-10 00:22:01 INFO Ignoring job '11543518' in development group 'Test Security'
and then nothing. Seems like a lot of output is happening 00:21-00:22. The job overall shows "Duration: 119 minutes 12 seconds, Finished: 3 hours ago. Queued: 1 second, Timeout: 1h (from project)" so one thing surprising is that it ran for 2h but the timeout is 1h. And second thing is that apparently there is no output anymore for more than one hour and the call is just stuck somewhere.
Rollback actions¶
- For qa-maintenance/openQABot and qa-maintenance/bot-ng: Visibility, project features, permissions -> Disable email notifications
- Subject changed from qem-bot sync aggregates gitlab CI job times out after 2h to gitlab CI shows showing no logs or are getting stuck (was: qem-bot sync aggregates gitlab CI job times out after 2h)
- Status changed from New to Blocked
- Assignee set to okurz
- Related to action #132500: NUE1-SRV2, .qa.suse.de, aarch64 workers offline due to heat-related SRV2 shutdown size:M added
Until the mentioned SD ticket is handled, I am disabling e-mail notifications for openQABot/bot-ng to prevent fatigue from the failed scheduled pipelines
- Description updated (diff)
- Due date set to 2023-07-24
- Status changed from Blocked to Feedback
- Subject changed from gitlab CI shows showing no logs or are getting stuck (was: qem-bot sync aggregates gitlab CI job times out after 2h) to gitlab CI shows showing no logs or are getting stuck (was: qem-bot sync aggregates gitlab CI job times out after 2h) size:M
okurz wrote:
@Jan Baier how did you disable email notifications in https://gitlab.suse.de/qa-maintenance/bot-ng/edit ?
See the updated description: Under settings -> Visibility, project features, permissions -> Disable email notifications
Notifications are again re-enabled.
jbaier_cz wrote:
okurz wrote:
@Jan Baier how did you disable email notifications in https://gitlab.suse.de/qa-maintenance/bot-ng/edit ?
See the updated description: Under settings -> Visibility, project features, permissions -> Disable email notifications
permission problem, fixed by making me "Maintainer->Owner", thx
- Due date deleted (
2023-07-24)
- Status changed from Feedback to Blocked
- Status changed from Blocked to Resolved
Also available in: Atom
PDF