Project

General

Profile

Actions

action #91097

closed

CI: "webui-docker-compose" eventually fails building images

Added by ilausuch almost 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
2021-04-14
Due date:
2021-04-29
% Done:

0%

Estimated time:

Description

Motivation

Eventually the image build fails on the zipper command chain.
On an initial observation it is not possible to determine witch one causes the problem

Building worker ... 
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl gzip &&     zypper in -y openQA-worker qemu-arm qemu-ppc qemu-x86 qemu-tools &&     zypper in -y kmod &&     (zypper in -y qemu-ovmf-x86_64 ||:) &&     (zypper in -y qemu-uefi-aarch64 ||:) &&     zypper clean' returned a non-zero code: 8

ERROR: for worker  (<Service: worker>, '')
Service 'worker' failed to build

Acceptance Criteria

  • AC 1: The image is built without errors on the worker service

suggestions

We don't have too much information in the logs, therefore the cause is not clear. Increase the log information could be a good first step to determine the problem.

Other examples

Building nginx         ... 
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y --force-resolution openQA nginx &&     zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 8

Related issues 1 (0 open1 closed)

Related to openQA Project - action #90614: CI test webui-docker-compose failed but PR was merged anywayResolvedilausuch2021-04-012021-04-23

Actions
Actions #1

Updated by ilausuch almost 3 years ago

  • Related to action #90614: CI test webui-docker-compose failed but PR was merged anyway added
Actions #2

Updated by ilausuch almost 3 years ago

  • Description updated (diff)
Actions #3

Updated by livdywan almost 3 years ago

  • Status changed from New to Workable
Actions #4

Updated by ilausuch almost 3 years ago

  • Status changed from Workable to In Progress
  • Assignee set to ilausuch
Actions #5

Updated by ilausuch almost 3 years ago

Interesting discovering:
I can reproduce an error over the zipper part forcing a zipper repo doesn't exist
https://github.com/os-autoinst/openQA/pull/3842/checks?check_run_id=2342341146

The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y ca-certificates-mozilla curl &&     zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m &&     zypper clean' returned a non-zero code: 4

But in the CI we don't have information, but yes in local building using docker-compose

Repository 'devel_openQA' is invalid.
[devel_openQA|http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22] Valid metadata not found at specified URL
History:
 - [devel_openQA|http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22] Repository type can't be determined.

Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'devel_openQA' because of the above error.

The reason is that we are using the flag -q (quiet) that doesn't show the output but neither the errors

NOTE: In the description we have "Building nginx" I checked that this line is not directly related and could be any of the builds that docker-compose do.

Actions #6

Updated by ilausuch almost 3 years ago

Doing a test without -q shows the errors
https://github.com/os-autoinst/openQA/pull/3842/checks?check_run_id=2342449206

Now I am going to check that there aren't limitations on the number of output lines

Actions #7

Updated by ilausuch almost 3 years ago

I created this PR to ensure we have all logs when the container image build fails
https://github.com/os-autoinst/openQA/pull/3842

Actions #8

Updated by okurz almost 3 years ago

  • Target version set to Ready
Actions #9

Updated by openqa_review almost 3 years ago

  • Due date set to 2021-04-29

Setting due date based on mean cycle time of SUSE QE Tools

Actions #10

Updated by livdywan almost 3 years ago

ilausuch wrote:

I created this PR to ensure we have all logs when the container image build fails
https://github.com/os-autoinst/openQA/pull/3842
[...]

  • AC 1: The image is built without errors on the worker service

Do you have a fix in mind? As I understand the above PR only makes errors visible?

Actions #11

Updated by ilausuch almost 3 years ago

Not sure there is a fix because seems are problems with the connection to the repositories. I think is external, so far from try to restart, I don't think we could do anything else here.

Actions #12

Updated by livdywan almost 3 years ago

ilausuch wrote:

Not sure there is a fix because seems are problems with the connection to the repositories. I think is external, so far from try to restart, I don't think we could do anything else here.

Should we look into a work-around like re-trying in case of errors?

Actions #13

Updated by ilausuch almost 3 years ago

Yes, and this PR https://github.com/os-autoinst/openQA/pull/3844 tries to solve that.
This PR is associated to #90614

Actions #14

Updated by ilausuch almost 3 years ago

Now I think it can be considered done

Actions #15

Updated by ilausuch almost 3 years ago

  • Status changed from In Progress to Resolved

With these PR
https://github.com/os-autoinst/openQA/pull/3842 (CI: Remove -q flag in docker-compose build in test_containers_compose)
https://github.com/os-autoinst/openQA/pull/3844 (CI: Retry if docker-compose build fails in webui-docker-compose test)

We are covering the AC. First PR ensure we have enough logs to check errors, and the second one retries if the building fails.
It works for web UI and worker builds.
Of course the failures we are seeing here are due to external problems related with the zypper repositories so the retries not ensures that the test will fail at the end.

Actions

Also available in: Atom PDF