action #132488
closedgitlab CI shows showing no logs or are getting stuck (was: qem-bot sync aggregates gitlab CI job times out after 2h) size:M
0%
Description
Observation¶
https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1680349 times out after 2h:
output of the bot-ng call:
++ ./qem-bot/bot-ng.py -c /etc/openqabot --token [MASKED] aggr-sync-results
++ tee bot_aggr-sync-results_0.log
2023-07-10 00:21:25 INFO Config /etc/openqabot/bot.yml does not have aggregate
…
2023-07-10 00:22:01 INFO Ignoring job '11543518' in development group 'Test Security'
and then nothing. Seems like a lot of output is happening 00:21-00:22. The job overall shows "Duration: 119 minutes 12 seconds, Finished: 3 hours ago. Queued: 1 second, Timeout: 1h (from project)" so one thing surprising is that it ran for 2h but the timeout is 1h. And second thing is that apparently there is no output anymore for more than one hour and the call is just stuck somewhere.
Rollback actions¶
- For qa-maintenance/openQABot and qa-maintenance/bot-ng: Visibility, project features, permissions -> Disable email notifications
Updated by okurz over 1 year ago
- Subject changed from qem-bot sync aggregates gitlab CI job times out after 2h to gitlab CI shows showing no logs or are getting stuck (was: qem-bot sync aggregates gitlab CI job times out after 2h)
- Status changed from New to Blocked
- Assignee set to okurz
Other gitlab CI jobs are also affected by stuck or missing logs. Created https://sd.suse.com/servicedesk/customer/portal/1/SD-126410
Updated by okurz over 1 year ago
- Related to action #132500: NUE1-SRV2, .qa.suse.de, aarch64 workers offline due to heat-related SRV2 shutdown size:M added
Updated by jbaier_cz over 1 year ago
Until the mentioned SD ticket is handled, I am disabling e-mail notifications for openQABot/bot-ng to prevent fatigue from the failed scheduled pipelines
Updated by okurz over 1 year ago
The SD ticket was resolved with "there is new worker on PRG2. there is stricter firewall, so if you see any jobs failing for some access problem, shoot us an SD ticket for the network access".
Most recent jobs in https://gitlab.suse.de/qa-maintenance/bot-ng/ are good, e.g. https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1682979 . Many others are "waiting", maybe there is a bigger backlog in gitlab CI. I will monitor further. https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules had all schedules disabled, I enabled all again along with email notification.
@Jan Baier how did you disable email notifications in https://gitlab.suse.de/qa-maintenance/bot-ng/edit ?
I also enabled the schedule again in https://gitlab.suse.de/qa-maintenance/openQABot/-/pipeline_schedules
I observed a problem with openqa-review based on https://gitlab.suse.de/openqa/openqa-review/-/pipeline_schedules on the new gitlab CI runners unable to reach relay.suse.de, reported https://sd.suse.com/servicedesk/customer/portal/1/SD-126756
Updated by okurz over 1 year ago
- Due date set to 2023-07-24
- Status changed from Blocked to Feedback
- https://gitlab.suse.de/openqa/openqa-review/-/jobs/1684828 is passed, openqa-review is good
- L3 https://gitlab.suse.de/qa-maintenance/openQABot/-/pipelines/727873 passed
- QR https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/1684926 passed
- bot-ng https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules all passed
- osd-deployment failed in https://gitlab.suse.de/openqa/osd-deployment/-/jobs/1684483#L38 , reported https://sd.suse.com/servicedesk/customer/portal/1/SD-126812
- monitor-o3 failed in https://gitlab.suse.de/openqa/monitor-o3/-/jobs/1684425#L41 , reported https://sd.suse.com/servicedesk/customer/portal/1/SD-126813
Updated by livdywan over 1 year ago
- Subject changed from gitlab CI shows showing no logs or are getting stuck (was: qem-bot sync aggregates gitlab CI job times out after 2h) to gitlab CI shows showing no logs or are getting stuck (was: qem-bot sync aggregates gitlab CI job times out after 2h) size:M
Updated by jbaier_cz over 1 year ago
okurz wrote:
@Jan Baier how did you disable email notifications in https://gitlab.suse.de/qa-maintenance/bot-ng/edit ?
See the updated description: Under settings -> Visibility, project features, permissions -> Disable email notifications
Notifications are again re-enabled.
Updated by okurz over 1 year ago
jbaier_cz wrote:
okurz wrote:
@Jan Baier how did you disable email notifications in https://gitlab.suse.de/qa-maintenance/bot-ng/edit ?
See the updated description: Under settings -> Visibility, project features, permissions -> Disable email notifications
permission problem, fixed by making me "Maintainer->Owner", thx
Updated by okurz over 1 year ago
- Due date deleted (
2023-07-24) - Status changed from Feedback to Blocked
Updated by okurz about 1 year ago
- Status changed from Blocked to Resolved
both SD tickets resolved. email notifications enabled again in https://gitlab.suse.de/qa-maintenance/bot-ng/edit. https://gitlab.suse.de/qa-maintenance/openQABot/edit was already on.