action #91097
closedCI: "webui-docker-compose" eventually fails building images
Description
Motivation¶
Eventually the image build fails on the zipper command chain.
On an initial observation it is not possible to determine witch one causes the problem
Building worker ...
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl gzip && zypper in -y openQA-worker qemu-arm qemu-ppc qemu-x86 qemu-tools && zypper in -y kmod && (zypper in -y qemu-ovmf-x86_64 ||:) && (zypper in -y qemu-uefi-aarch64 ||:) && zypper clean' returned a non-zero code: 8
ERROR: for worker (<Service: worker>, '')
Service 'worker' failed to build
Acceptance Criteria¶
- AC 1: The image is built without errors on the worker service
suggestions¶
We don't have too much information in the logs, therefore the cause is not clear. Increase the log information could be a good first step to determine the problem.
Other examples¶
Building nginx ...
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y --force-resolution openQA nginx && zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 8
Updated by ilausuch over 3 years ago
- Related to action #90614: CI test webui-docker-compose failed but PR was merged anyway added
Updated by ilausuch over 3 years ago
- Status changed from Workable to In Progress
- Assignee set to ilausuch
Updated by ilausuch over 3 years ago
Interesting discovering:
I can reproduce an error over the zipper part forcing a zipper repo doesn't exist
https://github.com/os-autoinst/openQA/pull/3842/checks?check_run_id=2342341146
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
But in the CI we don't have information, but yes in local building using docker-compose
Repository 'devel_openQA' is invalid.
[devel_openQA|http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22] Valid metadata not found at specified URL
History:
- [devel_openQA|http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22] Repository type can't be determined.
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'devel_openQA' because of the above error.
The reason is that we are using the flag -q (quiet) that doesn't show the output but neither the errors
NOTE: In the description we have "Building nginx" I checked that this line is not directly related and could be any of the builds that docker-compose do.
Updated by ilausuch over 3 years ago
Doing a test without -q shows the errors
https://github.com/os-autoinst/openQA/pull/3842/checks?check_run_id=2342449206
Now I am going to check that there aren't limitations on the number of output lines
Updated by ilausuch over 3 years ago
I created this PR to ensure we have all logs when the container image build fails
https://github.com/os-autoinst/openQA/pull/3842
Updated by openqa_review over 3 years ago
- Due date set to 2021-04-29
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan over 3 years ago
ilausuch wrote:
I created this PR to ensure we have all logs when the container image build fails
https://github.com/os-autoinst/openQA/pull/3842
[...]
- AC 1: The image is built without errors on the worker service
Do you have a fix in mind? As I understand the above PR only makes errors visible?
Updated by ilausuch over 3 years ago
Not sure there is a fix because seems are problems with the connection to the repositories. I think is external, so far from try to restart, I don't think we could do anything else here.
Updated by livdywan over 3 years ago
ilausuch wrote:
Not sure there is a fix because seems are problems with the connection to the repositories. I think is external, so far from try to restart, I don't think we could do anything else here.
Should we look into a work-around like re-trying in case of errors?
Updated by ilausuch over 3 years ago
Yes, and this PR https://github.com/os-autoinst/openQA/pull/3844 tries to solve that.
This PR is associated to #90614
Updated by ilausuch over 3 years ago
- Status changed from In Progress to Resolved
With these PR
https://github.com/os-autoinst/openQA/pull/3842 (CI: Remove -q flag in docker-compose build in test_containers_compose)
https://github.com/os-autoinst/openQA/pull/3844 (CI: Retry if docker-compose build fails in webui-docker-compose test)
We are covering the AC. First PR ensure we have enough logs to check errors, and the second one retries if the building fails.
It works for web UI and worker builds.
Of course the failures we are seeing here are due to external problems related with the zypper repositories so the retries not ensures that the test will fail at the end.