action #133250
closedgitlab.suse.de unusable "Failed to write to log, write /srv/www/vhosts/gitlab-ce/log/gitlab-shell.log: no space left on device" on remote operations
0%
Description
Observation¶
Changes from gitlab.suse.de can not even be fetched getting error
Failed to write to log, write /srv/www/vhosts/gitlab-ce/log/gitlab-shell.log: no space left on device
reproducibly.
Steps to reproduce¶
In any git checkout connected to gitlab.suse.de run git remote update and observe the error
Rollback steps¶
- Enable bot-ng schedule "Approve incidents" again https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules/111/edit coordinating with fniederwanger in https://suse.slack.com/archives/C02CANHLANP/p1690270323953269
Updated by okurz over 1 year ago
- Status changed from New to Blocked
- Priority changed from Normal to Urgent
Updated by okurz over 1 year ago
- Description updated (diff)
disabled the bot-ng schedule "approve incidents"
Updated by crameleon over 1 year ago
- Status changed from Blocked to Resolved
Hi,
disk space has been increased. We will work with the owner of the problematic repository to resolve the issue going forward, they currently store over 1TB of data which accumulates in their Git history by the nature of how Git works.
I understand that increasing the disk space does not solve the long term issue with our lack of alerting which could prevent such issues from turning into an outage. I filed https://jira.suse.com/browse/ENGINFRA-2474 as it seems we did not have alerting tracked albeit previous incidents having been caused by the lack of it.
Best,
Georg
Updated by okurz over 1 year ago
- Related to action #133307: mtui: Connection to svn+ssh is not possible or the "inconsistent submission" added
Updated by okurz over 1 year ago
- Status changed from Resolved to In Progress
Thank you very much. Reopening and keeping assigned to me to handle the related fallout and consequences.
Updated by openqa_review over 1 year ago
- Due date set to 2023-08-09
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz over 1 year ago
- Status changed from In Progress to Resolved
enabled the approve incidents pipeline again in https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules