Project

General

Profile

action #91097

CI: "webui-docker-compose" eventually fails building images

Added by ilausuch 6 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Concrete Bugs
Target version:
Start date:
2021-04-14
Due date:
2021-04-29
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

Eventually the image build fails on the zipper command chain.
On an initial observation it is not possible to determine witch one causes the problem

Building worker ... 
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl gzip &&     zypper in -y openQA-worker qemu-arm qemu-ppc qemu-x86 qemu-tools &&     zypper in -y kmod &&     (zypper in -y qemu-ovmf-x86_64 ||:) &&     (zypper in -y qemu-uefi-aarch64 ||:) &&     zypper clean' returned a non-zero code: 8

ERROR: for worker  (<Service: worker>, '')
Service 'worker' failed to build

Acceptance Criteria

  • AC 1: The image is built without errors on the worker service

suggestions

We don't have too much information in the logs, therefore the cause is not clear. Increase the log information could be a good first step to determine the problem.

Other examples

Building nginx         ... 
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y --force-resolution openQA nginx &&     zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 8

Related issues

Related to openQA Project - action #90614: CI test webui-docker-compose failed but PR was merged anywayResolved2021-04-012021-04-23

History

#1 Updated by ilausuch 6 months ago

  • Related to action #90614: CI test webui-docker-compose failed but PR was merged anyway added

#2 Updated by ilausuch 6 months ago

  • Description updated (diff)

#3 Updated by cdywan 6 months ago

  • Status changed from New to Workable

#4 Updated by ilausuch 6 months ago

  • Status changed from Workable to In Progress
  • Assignee set to ilausuch

#5 Updated by ilausuch 6 months ago

Interesting discovering:
I can reproduce an error over the zipper part forcing a zipper repo doesn't exist
https://github.com/os-autoinst/openQA/pull/3842/checks?check_run_id=2342341146

The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4

But in the CI we don't have information, but yes in local building using docker-compose

Repository 'devel_openQA' is invalid.
[devel_openQA|http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22] Valid metadata not found at specified URL
History:
 - [devel_openQA|http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22] Repository type can't be determined.

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'devel_openQA' because of the above error.

The reason is that we are using the flag -q (quiet) that doesn't show the output but neither the errors

NOTE: In the description we have "Building nginx" I checked that this line is not directly related and could be any of the builds that docker-compose do.

#6 Updated by ilausuch 6 months ago

Doing a test without -q shows the errors
https://github.com/os-autoinst/openQA/pull/3842/checks?check_run_id=2342449206

Now I am going to check that there aren't limitations on the number of output lines

#7 Updated by ilausuch 6 months ago

I created this PR to ensure we have all logs when the container image build fails
https://github.com/os-autoinst/openQA/pull/3842

#8 Updated by okurz 6 months ago

  • Target version set to Ready

#9 Updated by openqa_review 6 months ago

  • Due date set to 2021-04-29

Setting due date based on mean cycle time of SUSE QE Tools

#10 Updated by cdywan 6 months ago

ilausuch wrote:

I created this PR to ensure we have all logs when the container image build fails
https://github.com/os-autoinst/openQA/pull/3842
[...]

  • AC 1: The image is built without errors on the worker service

Do you have a fix in mind? As I understand the above PR only makes errors visible?

#11 Updated by ilausuch 6 months ago

Not sure there is a fix because seems are problems with the connection to the repositories. I think is external, so far from try to restart, I don't think we could do anything else here.

#12 Updated by cdywan 6 months ago

ilausuch wrote:

Not sure there is a fix because seems are problems with the connection to the repositories. I think is external, so far from try to restart, I don't think we could do anything else here.

Should we look into a work-around like re-trying in case of errors?

#13 Updated by ilausuch 6 months ago

Yes, and this PR https://github.com/os-autoinst/openQA/pull/3844 tries to solve that.
This PR is associated to #90614

#14 Updated by ilausuch 6 months ago

Now I think it can be considered done

#15 Updated by ilausuch 6 months ago

  • Status changed from In Progress to Resolved

With these PR
https://github.com/os-autoinst/openQA/pull/3842 (CI: Remove -q flag in docker-compose build in test_containers_compose)
https://github.com/os-autoinst/openQA/pull/3844 (CI: Retry if docker-compose build fails in webui-docker-compose test)

We are covering the AC. First PR ensure we have enough logs to check errors, and the second one retries if the building fails.
It works for web UI and worker builds.
Of course the failures we are seeing here are due to external problems related with the zypper repositories so the retries not ensures that the test will fail at the end.

Also available in: Atom PDF