action #133250
closed
gitlab.suse.de unusable "Failed to write to log, write /srv/www/vhosts/gitlab-ce/log/gitlab-shell.log: no space left on device" on remote operations
Added by okurz 10 months ago.
Updated 10 months ago.
Description
Observation¶
Changes from gitlab.suse.de can not even be fetched getting error
Failed to write to log, write /srv/www/vhosts/gitlab-ce/log/gitlab-shell.log: no space left on device
reproducibly.
Steps to reproduce¶
In any git checkout connected to gitlab.suse.de run git remote update and observe the error
Rollback steps¶
- Status changed from New to Blocked
- Priority changed from Normal to Urgent
- Description updated (diff)
disabled the bot-ng schedule "approve incidents"
- Description updated (diff)
- Status changed from Blocked to Resolved
Hi,
disk space has been increased. We will work with the owner of the problematic repository to resolve the issue going forward, they currently store over 1TB of data which accumulates in their Git history by the nature of how Git works.
I understand that increasing the disk space does not solve the long term issue with our lack of alerting which could prevent such issues from turning into an outage. I filed https://jira.suse.com/browse/ENGINFRA-2474 as it seems we did not have alerting tracked albeit previous incidents having been caused by the lack of it.
Best,
Georg
- Related to action #133307: mtui: Connection to svn+ssh is not possible or the "inconsistent submission" added
- Status changed from Resolved to In Progress
Thank you very much. Reopening and keeping assigned to me to handle the related fallout and consequences.
- Due date set to 2023-08-09
Setting due date based on mean cycle time of SUSE QE Tools
- Status changed from In Progress to Resolved
Also available in: Atom
PDF