action #91097
closed
CI: "webui-docker-compose" eventually fails building images
Added by ilausuch about 3 years ago.
Updated about 3 years ago.
Category:
Regressions/Crashes
Description
Motivation¶
Eventually the image build fails on the zipper command chain.
On an initial observation it is not possible to determine witch one causes the problem
Building worker ...
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl gzip && zypper in -y openQA-worker qemu-arm qemu-ppc qemu-x86 qemu-tools && zypper in -y kmod && (zypper in -y qemu-ovmf-x86_64 ||:) && (zypper in -y qemu-uefi-aarch64 ||:) && zypper clean' returned a non-zero code: 8
ERROR: for worker (<Service: worker>, '')
Service 'worker' failed to build
Acceptance Criteria¶
- AC 1: The image is built without errors on the worker service
suggestions¶
We don't have too much information in the logs, therefore the cause is not clear. Increase the log information could be a good first step to determine the problem.
Other examples¶
Building nginx ...
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y --force-resolution openQA nginx && zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 8
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.2 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 8
- Related to action #90614: CI test webui-docker-compose failed but PR was merged anyway added
- Description updated (diff)
- Status changed from New to Workable
- Status changed from Workable to In Progress
- Assignee set to ilausuch
Interesting discovering:
I can reproduce an error over the zipper part forcing a zipper repo doesn't exist
https://github.com/os-autoinst/openQA/pull/3842/checks?check_run_id=2342341146
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
The command '/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22 devel_openQA && zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.2/openSUSE_Leap_15.2 devel_openQA_Leap && zypper --gpg-auto-import-keys ref && zypper in -y ca-certificates-mozilla curl && zypper in -y --force-resolution openQA-local-db apache2 hostname which w3m && zypper clean' returned a non-zero code: 4
But in the CI we don't have information, but yes in local building using docker-compose
Repository 'devel_openQA' is invalid.
[devel_openQA|http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22] Valid metadata not found at specified URL
History:
- [devel_openQA|http://download.opensuse.org/repositories/devel:openQA/openSUSE_Leap_15.22] Repository type can't be determined.
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'devel_openQA' because of the above error.
The reason is that we are using the flag -q (quiet) that doesn't show the output but neither the errors
NOTE: In the description we have "Building nginx" I checked that this line is not directly related and could be any of the builds that docker-compose do.
- Target version set to Ready
- Due date set to 2021-04-29
Setting due date based on mean cycle time of SUSE QE Tools
ilausuch wrote:
I created this PR to ensure we have all logs when the container image build fails
https://github.com/os-autoinst/openQA/pull/3842
[...]
- AC 1: The image is built without errors on the worker service
Do you have a fix in mind? As I understand the above PR only makes errors visible?
Not sure there is a fix because seems are problems with the connection to the repositories. I think is external, so far from try to restart, I don't think we could do anything else here.
ilausuch wrote:
Not sure there is a fix because seems are problems with the connection to the repositories. I think is external, so far from try to restart, I don't think we could do anything else here.
Should we look into a work-around like re-trying in case of errors?
Now I think it can be considered done
- Status changed from In Progress to Resolved
With these PR
https://github.com/os-autoinst/openQA/pull/3842 (CI: Remove -q flag in docker-compose build in test_containers_compose)
https://github.com/os-autoinst/openQA/pull/3844 (CI: Retry if docker-compose build fails in webui-docker-compose test)
We are covering the AC. First PR ensure we have enough logs to check errors, and the second one retries if the building fails.
It works for web UI and worker builds.
Of course the failures we are seeing here are due to external problems related with the zypper repositories so the retries not ensures that the test will fail at the end.
Also available in: Atom
PDF