Project

General

Profile

Actions
Copy link

action #133250

closed

gitlab.suse.de unusable "Failed to write to log, write /srv/www/vhosts/gitlab-ce/log/gitlab-shell.log: no space left on device" on remote operations

Added by okurz 10 months ago. Updated 10 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-07-25
Due date:
2023-08-09
% Done:

0%

Estimated time:
Tags:
gitlab, infra, IT

Description

Observation

Changes from gitlab.suse.de can not even be fetched getting error

Failed to write to log, write /srv/www/vhosts/gitlab-ce/log/gitlab-shell.log: no space left on device

reproducibly.

Steps to reproduce

In any git checkout connected to gitlab.suse.de run git remote update and observe the error

Rollback steps

Related issues 1 (0 open — 1 closed)

Related to QA - action #133307: mtui: Connection to svn+ssh is not possible or the "inconsistent submission"Resolvedjbaier_cz2023-07-25

Actions
Actions
Copy link
 #1

Updated by okurz 10 months ago

  • Status changed from New to Blocked
  • Priority changed from Normal to Urgent
Actions
Copy link
 #2

Updated by okurz 10 months ago

  • Description updated (diff)

disabled the bot-ng schedule "approve incidents"

Actions
Copy link
 #3

Updated by okurz 10 months ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by crameleon 10 months ago

  • Status changed from Blocked to Resolved

Hi,

disk space has been increased. We will work with the owner of the problematic repository to resolve the issue going forward, they currently store over 1TB of data which accumulates in their Git history by the nature of how Git works.
I understand that increasing the disk space does not solve the long term issue with our lack of alerting which could prevent such issues from turning into an outage. I filed https://jira.suse.com/browse/ENGINFRA-2474 as it seems we did not have alerting tracked albeit previous incidents having been caused by the lack of it.

Best,
Georg

Actions
Copy link
 #5

Updated by okurz 10 months ago

  • Related to action #133307: mtui: Connection to svn+ssh is not possible or the "inconsistent submission" added
Actions
Copy link
 #6

Updated by okurz 10 months ago

  • Status changed from Resolved to In Progress

Thank you very much. Reopening and keeping assigned to me to handle the related fallout and consequences.

Actions
Copy link
 #7

Updated by openqa_review 10 months ago

  • Due date set to 2023-08-09

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #8

Updated by okurz 10 months ago

  • Status changed from In Progress to Resolved

enabled the approve incidents pipeline again in https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules

Actions
Copy link

Also available in: Atom PDF