Project

General

Profile

action #120040

OBS Package devel:openQA/openQA_container_worker failed to build in containers15.4/x86_64 size:S

Added by okurz 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2022-11-07
Due date:
2022-11-24
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

Visit https://build.opensuse.org/package/live_build_log/devel:openQA/openQA_container_worker/containers15.4/x86_64

Check out the package for editing:
osc checkout devel:openQA openQA_container_worker

Last lines of build log:

[  227s] time="2022-11-07T17:46:40.692875053Z" level=info msg="cleaning up dead shim"
[  227s] time="2022-11-07T17:46:40.707946020Z" level=warning msg="cleanup warnings time=\"2022-11-07T17:46:40Z\" level=info msg=\"starting signal loop\" namespace=moby pid=4475 runtime=io.containerd.runc.v2\n"
[  227s] Removing intermediate container 79bc7fc77b7d
[  227s]  ---> 603ec94ce5ac
[  227s] Step 12/12 : LABEL org.openbuildservice.disturl=obs://build.opensuse.org/devel:openQA/containers15.4/3314495ae25591e4a61a7afb3d12bf92-openQA_container_worker
[  227s]  ---> Running in f039ac919cbc
[  228s] Removing intermediate container f039ac919cbc
[  228s]  ---> 4d6cbefd9cf9
[  243s] Successfully built ca299dd87497
[  243s] Successfully tagged openqa_worker:latest
[  243s] Saving image openqa_worker to openqa_worker.x86_64-11.73.tar
[  253s] time="2022-11-07T17:47:06.741546747Z" level=error msg="Handler for GET /v1.41/images/get returned error: write /var/lib/docker/tmp/docker-export-2019733357/82cfbd0b74d508c98b2f549ff60a98e67620270d46ca8881481dd527f383b539/layer.tar: no space left on device"
[  253s] Error response from daemon: write /var/lib/docker/tmp/docker-export-2019733357/82cfbd0b74d508c98b2f549ff60a98e67620270d46ca8881481dd527f383b539/layer.tar: no space left on device

Suggestions

  • Increase "constraints" to build successfully

History

#1 Updated by okurz 3 months ago

  • Description updated (diff)

#2 Updated by mkittler 3 months ago

  • Assignee set to mkittler

#3 Updated by mkittler 3 months ago

The "observation" doesn't make it clear whether this problem is affecting builds on obs (the link to the obs build shows a successful build at this time) or local builds (osc checkout command given but no clear statement on what further commands to invoke locally to reproduce the issue).

#4 Updated by mkittler 3 months ago

Looks like there are various osc build options that can affect this (--vm-type=TYPE, --vm-disk-size=DISKSIZE) so if this is about the local build then please clarify what options to use.

Likely this is about the remote build then. Constraints could be easily configured (https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.build_job_constraints.html#id-1.5.10.16.12.5.3) but it is hard to tell how much disk space we need and how "greedy" we can be.

Maybe we better discussion this in the next estimation meeting (or okurz can just leave a clarifying comment).

#6 Updated by dheidler 3 months ago

  • Status changed from New to In Progress
  • Assignee changed from mkittler to dheidler

#7 Updated by openqa_review 3 months ago

  • Due date set to 2022-11-24

Setting due date based on mean cycle time of SUSE QE Tools

#8 Updated by cdywan 3 months ago

  • Subject changed from OBS Package devel:openQA/openQA_container_worker failed to build in containers15.4/x86_64 to OBS Package devel:openQA/openQA_container_worker failed to build in containers15.4/x86_64 size:S

#9 Updated by dheidler 3 months ago

  • Status changed from In Progress to Feedback

Set the _constraints to:

<?xml version="1.0"?>
<constraints>
  <hardware>
    <disk>
      <size unit="G">16</size>
    </disk>
    <memory>
      <size unit="G">4</size>
    </memory>
    <jobs>6</jobs>
  </hardware>
</constraints>

Not sure if we would need that for the os-autoinst-distri-opensuse-deps as well, as that one is only adding an fs layer on top.

#10 Updated by dheidler 3 months ago

  • Status changed from Feedback to Resolved

Closing for now.
If something still happens again, we can increase the values.

#11 Updated by okurz 3 months ago

  • Status changed from Resolved to Feedback

#12 Updated by mkittler 3 months ago

  • Category deleted (Concrete Bugs)
  • Assignee deleted (dheidler)
  • Target version deleted (Ready)

Do we really need 4 GiB of RAM? I thought this was only about disk space. I also don't think we need to limit the number of build jobs to only 6 (as our build jobs shouldn't be very memory hungry).

#13 Updated by okurz 3 months ago

  • Category set to Concrete Bugs
  • Assignee set to dheidler
  • Target version set to Ready

#14 Updated by dheidler 3 months ago

I added the _constraints file for all containers in devel:openQA.

#15 Updated by dheidler 3 months ago

  • Status changed from Feedback to Resolved

Containers are still building, so at least nothing broke.

Also available in: Atom PDF