Project

General

Profile

Actions

action #162062

closed

No space left on device causing GitLab pipelines to fail

Added by livdywan 19 days ago. Updated 4 days ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2024-06-10
Due date:
% Done:

0%

Estimated time:

Description

Observation

Various pipelines fail either running the container or during the script because there is no space left.

https://gitlab.suse.de/openqa/openqa-review/-/jobs/2706267

WARNING: Failed to pull image with policy "always": write /var/lib/docker/tmp/GetImageBlob2814730492: no space left on device (manager.go:250:0s)
ERROR: Job failed: failed to pull image "registry.opensuse.org/home/okurz/container/ca/containers/tumbleweed:openqa-review" with specified policies [always]: write /var/lib/docker/tmp/GetImageBlob2814730492: no space left on device (manager.go:250:0s)

https://gitlab.suse.de/openqa/os-autoinst-needles-opensuse-mirror/-/jobs/2706665

 - Download (curl) error for 'http://download.opensuse.org/distribution/leap/15.5/repo/oss/repodata/d0aae74c050dca8d30fbccd949a136d8ed209eccf8fdf435ac8c1d739271d8e7-appdata.xml.gz':
   Error code: Write error
   Error message: Failure writing output to destination
[...]
Can't create metadata cache directory.

https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2706694

$ rpm --query qam-metadata-openqabot
rpmxdbOpen: No space left on device
error: cannot open Name index using unknown db - Operation not permitted (1)
rpmxdbOpen: No space left on device
error: cannot open Name index using unknown db - Operation not permitted (1)
package qam-metadata-openqabot is not installed

https://gitlab.suse.de/openqa/scripts-ci/-/jobs/2706024

ERROR: Preparation failed: adding cache volume: set volume permissions: create permission container for volume "runner-25ldi6vv-project-11950-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70": Error response from daemon: mkdir /var/lib/docker/overlay2/3efbcea02b9b1d07ff7de63478a83f5fcd6dd3e36949db2b1766d2ac77c5f8ec-init: no space left on device (linux_set.go:95:0s)

Acceptance criteria

  • AC1:

Suggestions

  • DONE File an infra SD ticket to get the GitLab runners checked
Actions #1

Updated by livdywan 19 days ago

  • Description updated (diff)
Actions #2

Updated by livdywan 19 days ago

  • Description updated (diff)
  • Status changed from New to In Progress
Actions #3

Updated by livdywan 19 days ago ยท Edited

  • Priority changed from Urgent to High

Debugging with Steven Mallindine. Pipelines running for now.

Actions #4

Updated by livdywan 19 days ago

  • Status changed from In Progress to Blocked

livdywan wrote in #note-2:

https://sd.suse.com/servicedesk/customer/portal/1/SD-159323

Let's block on this. Nothing we can do here.

Actions #5

Updated by livdywan 12 days ago

livdywan wrote in #note-4:

livdywan wrote in #note-2:

https://sd.suse.com/servicedesk/customer/portal/1/SD-159323

Let's block on this. Nothing we can do here.

Complete fix still pending.

Actions #6

Updated by livdywan 5 days ago

  • Priority changed from High to Low

SD ticket still open. I'd say it's not High for us anymore as things work for us and we've not seen any more issues.

Actions #7

Updated by livdywan 4 days ago

  • Status changed from Blocked to Resolved

Steve confirmed that the cleanup of images is working fine now. So we're good.

Actions

Also available in: Atom PDF