Project

General

Profile

Actions
Copy link

action #120040

closed

OBS Package devel:openQA/openQA_container_worker failed to build in containers15.4/x86_64 size:S

Added by okurz over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
dheidler
Category:
Regressions/Crashes
Target version:
Ready
Start date:
2022-11-07
Due date:
2022-11-24
% Done:

0%

Estimated time:

Description

Observation

Visit https://build.opensuse.org/package/live_build_log/devel:openQA/openQA_container_worker/containers15.4/x86_64

Check out the package for editing:
osc checkout devel:openQA openQA_container_worker

Last lines of build log:

[  227s] time="2022-11-07T17:46:40.692875053Z" level=info msg="cleaning up dead shim"
[  227s] time="2022-11-07T17:46:40.707946020Z" level=warning msg="cleanup warnings time=\"2022-11-07T17:46:40Z\" level=info msg=\"starting signal loop\" namespace=moby pid=4475 runtime=io.containerd.runc.v2\n"
[  227s] Removing intermediate container 79bc7fc77b7d
[  227s]  ---> 603ec94ce5ac
[  227s] Step 12/12 : LABEL org.openbuildservice.disturl=obs://build.opensuse.org/devel:openQA/containers15.4/3314495ae25591e4a61a7afb3d12bf92-openQA_container_worker
[  227s]  ---> Running in f039ac919cbc
[  228s] Removing intermediate container f039ac919cbc
[  228s]  ---> 4d6cbefd9cf9
[  243s] Successfully built ca299dd87497
[  243s] Successfully tagged openqa_worker:latest
[  243s] Saving image openqa_worker to openqa_worker.x86_64-11.73.tar
[  253s] time="2022-11-07T17:47:06.741546747Z" level=error msg="Handler for GET /v1.41/images/get returned error: write /var/lib/docker/tmp/docker-export-2019733357/82cfbd0b74d508c98b2f549ff60a98e67620270d46ca8881481dd527f383b539/layer.tar: no space left on device"
[  253s] Error response from daemon: write /var/lib/docker/tmp/docker-export-2019733357/82cfbd0b74d508c98b2f549ff60a98e67620270d46ca8881481dd527f383b539/layer.tar: no space left on device

Suggestions

  • Increase "constraints" to build successfully
Actions
Copy link
 #1

Updated by okurz over 1 year ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by mkittler over 1 year ago

  • Assignee set to mkittler
Actions
Copy link
 #3

Updated by mkittler over 1 year ago

The "observation" doesn't make it clear whether this problem is affecting builds on obs (the link to the obs build shows a successful build at this time) or local builds (osc checkout command given but no clear statement on what further commands to invoke locally to reproduce the issue).

Actions
Copy link
 #4

Updated by mkittler over 1 year ago

Looks like there are various osc build options that can affect this (--vm-type=TYPE, --vm-disk-size=DISKSIZE) so if this is about the local build then please clarify what options to use.

Likely this is about the remote build then. Constraints could be easily configured (https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.build_job_constraints.html#id-1.5.10.16.12.5.3) but it is hard to tell how much disk space we need and how "greedy" we can be.

Maybe we better discussion this in the next estimation meeting (or @okurz can just leave a clarifying comment).

Actions
Copy link
 #6

Updated by dheidler over 1 year ago

  • Status changed from New to In Progress
  • Assignee changed from mkittler to dheidler
Actions
Copy link
 #7

Updated by openqa_review over 1 year ago

  • Due date set to 2022-11-24

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #8

Updated by livdywan over 1 year ago

  • Subject changed from OBS Package devel:openQA/openQA_container_worker failed to build in containers15.4/x86_64 to OBS Package devel:openQA/openQA_container_worker failed to build in containers15.4/x86_64 size:S
Actions
Copy link
 #9

Updated by dheidler over 1 year ago

  • Status changed from In Progress to Feedback

Set the _constraints to:

<?xml version="1.0"?>
<constraints>
  <hardware>
    <disk>
      <size unit="G">16</size>
    </disk>
    <memory>
      <size unit="G">4</size>
    </memory>
    <jobs>6</jobs>
  </hardware>
</constraints>

Not sure if we would need that for the os-autoinst-distri-opensuse-deps as well, as that one is only adding an fs layer on top.

Actions
Copy link
 #10

Updated by dheidler over 1 year ago

  • Status changed from Feedback to Resolved

Closing for now.
If something still happens again, we can increase the values.

Actions
Copy link
 #11

Updated by okurz over 1 year ago

  • Status changed from Resolved to Feedback
Actions
Copy link
 #12

Updated by mkittler over 1 year ago

  • Category deleted (Regressions/Crashes)
  • Assignee deleted (dheidler)
  • Target version deleted (Ready)

Do we really need 4 GiB of RAM? I thought this was only about disk space. I also don't think we need to limit the number of build jobs to only 6 (as our build jobs shouldn't be very memory hungry).

Actions
Copy link
 #13

Updated by okurz over 1 year ago

  • Category set to Regressions/Crashes
  • Assignee set to dheidler
  • Target version set to Ready
Actions
Copy link
 #14

Updated by dheidler over 1 year ago

I added the _constraints file for all containers in devel:openQA.

Actions
Copy link
 #15

Updated by dheidler over 1 year ago

  • Status changed from Feedback to Resolved

Containers are still building, so at least nothing broke.

Actions
Copy link

Also available in: Atom PDF