Project

General

Profile

Actions

action #162848

closed

webui-docker-compose tests failing on GitHub PR's size:S

Added by livdywan 4 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Regressions/Crashes
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Observation

#6 ERROR: process "/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/15.6 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.6/15.6 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -yl ca-certificates-mozilla curl gzip rsync xz &&     zypper in -yl openQA-worker os-autoinst-s390-deps os-autoinst-ipmi-deps &&     zypper in -yl qemu-arm qemu-ppc qemu-x86 qemu-tools &&     zypper in -yl qemu-hw-display-virtio-gpu qemu-hw-display-virtio-gpu-pci qemu-hw-display-virtio-vga &&     zypper in -yl kmod &&     (zypper in -yl qemu-ovmf-x86_64 ||:) &&     (zypper in -yl qemu-uefi-aarch64 ||:) &&     zypper clean &&     mkdir -p /root/qemu" did not complete successfully: exit code: 4

Acceptance criteria

  • AC1: webui-docker-compose test passes consistently

Suggestions

Out of scope

  • Only run the test depending on certain modified files

Related issues 2 (1 open1 closed)

Related to openQA Project - action #163112: test fails in openqa_webui due to repeated and reproducible errors in reading from the devel:openQA repository "repodata…filelists-ext.xml.gz not found on medium" size:SFeedbackokurz2024-07-022024-12-31

Actions
Related to openQA Tests - action #161729: [sporadic] test fails in containers/build of openqa-in-openqa probably due to temporary download.opensuse.org and zypper issuesResolvedokurz2024-06-04

Actions
Actions #1

Updated by livdywan 3 months ago

  • Subject changed from webui-docker-compose" tests failing on GitHub PR's to webui-docker-compose tests failing on GitHub PR's size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions #2

Updated by dheidler 3 months ago

  • Status changed from Workable to In Progress
  • Assignee set to dheidler
Actions #3

Updated by openqa_review 3 months ago

  • Due date set to 2024-07-19

Setting due date based on mean cycle time of SUSE QE Tools

Actions #4

Updated by dheidler 3 months ago

  • Due date deleted (2024-07-19)
  • Status changed from In Progress to Rejected

Worked fine in https://github.com/os-autoinst/openQA/pull/5748 and locally.
Unable to reproduce or even find out what actually failed.

Actions #5

Updated by okurz 3 months ago · Edited

@livdywan See? You were wrong. The issue never existed as proven by @dheidler ;)

Actions #6

Updated by okurz 3 months ago

  • Status changed from Rejected to Workable

Just some recent examples:

#6 [nginx 2/5] RUN zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/15.6 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.6/15.6 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -y openQA nginx &&     zypper clean
#6 0.300 Adding repository 'devel_openQA' [...done]
#6 0.301 Repository 'devel_openQA' successfully added
#6 0.301 
#6 0.301 URI         : http://download.opensuse.org/repositories/devel:openQA/15.6
#6 0.301 Enabled     : Yes
#6 0.301 GPG Check   : Yes
#6 0.301 Autorefresh : Yes
#6 0.301 Priority    : 95 (raised priority)
#6 0.301 
#6 0.301 Repository priorities in effect: (See 'zypper lr -P' for details)
#6 0.301       95 (raised priority)  :  1 repository
#6 0.301       99 (default priority) :  7 repositories
#6 0.388 Adding repository 'devel_openQA_Leap' [...done]
#6 0.390 Repository 'devel_openQA_Leap' successfully added
#6 0.390 
#6 0.390 URI         : http://download.opensuse.org/repositories/devel:openQA:Leap:15.6/15.6
#6 0.390 Enabled     : Yes
#6 0.390 GPG Check   : Yes
#6 0.390 Autorefresh : Yes
#6 0.390 Priority    : 90 (raised priority)
#6 0.390 
#6 0.390 Repository priorities in effect: (See 'zypper lr -P' for details)
#6 0.390       90 (raised priority)  :  1 repository
#6 0.390       95 (raised priority)  :  1 repository
#6 0.390       99 (default priority) :  7 repositories
#6 0.994 Retrieving repository 'devel_openQA' metadata [...........
#6 3.154 
#6 3.154 Automatically importing the following key:
#6 3.154 
#6 3.154   Repository:       devel_openQA
#6 3.154   Key Fingerprint:  A99A 72E3 06F2 0929 E6DE E378 5B12 1667 CBDF 5E8F
#6 3.154   Key Name:         devel:openQA OBS Project <devel:openQA@build.opensuse.org>
#6 3.154   Key Algorithm:    RSA 2048
#6 3.154   Key Created:      Tue Dec 26 20:20:51 2023
#6 3.154   Key Expires:      Thu Mar  5 20:20:51 2026
#6 3.154   Rpm Name:         gpg-pubkey-cbdf5e8f-658b35a3
#6 3.154 
#6 3.154 
#6 3.154 
#6 3.154     Note: A GPG pubkey is clearly identified by its fingerprint. Do not rely on the key's name. If
#6 3.154     you are not sure whether the presented key is authentic, ask the repository provider or check
#6 3.154     their web site. Many providers maintain a web page showing the fingerprints of the GPG keys they
#6 3.154     are using.
#6 3.246 Signature verification failed for file 'repomd.xml' from repository 'devel_openQA'.
#6 3.246 
#6 3.246     Note: Signing data enables the recipient to verify that no modifications occurred after the data
#6 3.246     were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
#6 3.246     and in extreme cases even to a system compromise.
#6 3.246 
#6 3.246     Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
#6 3.246     whole repo.
#6 3.246 
#6 3.246     Warning: This file was modified after it has been signed. This may have been a malicious change,
#6 3.246     so it might not be trustworthy anymore! You should not continue unless you know it's safe.
#6 3.246 
#6 3.246     Note: This might be a transient issue if the server is in the midst of receiving new data. The
#6 3.246     data file and its signature are two files which must fit together. In case the request hit the
#6 3.246     server in the midst of updating them, the signature verification might fail. After a few
#6 3.246     minutes, when the server has updated its data, it should work again.
#6 3.247 
#6 3.247 Signature verification failed for file 'repomd.xml' from repository 'devel_openQA'. Continue? [yes/no] (no): Cannot read input: bad stream or EOF.
#6 3.247 If you run zypper without a terminal, use '--non-interactive' global
#6 3.247 option to make zypper use default answers to prompts.
#6 3.249 error]
#6 3.249 Repository 'devel_openQA' is invalid.
#6 3.249 [devel_openQA|http://download.opensuse.org/repositories/devel:openQA/15.6] Valid metadata not found at specified URL
#6 3.249 History:
#6 3.249  - Cannot read input. Bad stream or EOF.
#6 3.249 
#6 3.249 Please check if the URIs defined for this repository are pointing to a valid repository.
File './repodata/828b3f26d27acb6f47c0dc35507d51659ab327c4387efbd04939363fdbee35cb-primary.xml.gz' not found on medium 'http://download.opensuse.org/repositories/devel:openQA/15.6'

other similar cases:

Actions #7

Updated by dheidler 3 months ago · Edited

hm - that looks similar to #163112

The repo being published while using it.

Actions #8

Updated by okurz 3 months ago

  • Related to action #163112: test fails in openqa_webui due to repeated and reproducible errors in reading from the devel:openQA repository "repodata…filelists-ext.xml.gz not found on medium" size:S added
Actions #9

Updated by okurz 3 months ago

  • Status changed from Workable to Blocked

let's block on #163112

Actions #10

Updated by dheidler 3 months ago

  • Related to deleted (action #163112: test fails in openqa_webui due to repeated and reproducible errors in reading from the devel:openQA repository "repodata…filelists-ext.xml.gz not found on medium" size:S)
Actions #11

Updated by dheidler 3 months ago

  • Blocked by action #163112: test fails in openqa_webui due to repeated and reproducible errors in reading from the devel:openQA repository "repodata…filelists-ext.xml.gz not found on medium" size:S added
Actions #12

Updated by mkittler 3 months ago · Edited

  • Status changed from Blocked to Workable

And it happened again: https://github.com/os-autoinst/openQA/actions/runs/9856943908/job/27214994886?pr=5752

You said in the daily that you block this ticket but it didn't make much sense to me so I thought I misunderstood what ticket you are talking about.

I don't think adding a retry in openQA-in-openQA tests (that's what #163112 is about) will help here. We need to add a retry on for the docker-compose check. I'm not sure whether it makes sense to add a retry directly in the line in question (as it is a RUN line within a Dockerfile). So maybe we should just retry the whole thing.

Actions #13

Updated by mkittler 3 months ago

  • Blocked by deleted (action #163112: test fails in openqa_webui due to repeated and reproducible errors in reading from the devel:openQA repository "repodata…filelists-ext.xml.gz not found on medium" size:S)
Actions #14

Updated by dheidler 3 months ago

  • Assignee deleted (dheidler)

Unassigning for now then as I'm working on another ticket atm

Actions #15

Updated by okurz 3 months ago

  • Related to action #163112: test fails in openqa_webui due to repeated and reproducible errors in reading from the devel:openQA repository "repodata…filelists-ext.xml.gz not found on medium" size:S added
Actions #16

Updated by okurz 3 months ago

@mkittler @dheidler please keep ticket relations intact! I suggest to just keep tickets with the soft "related", not blocking/blocked and just mention when there's a blocked condition

Actions #17

Updated by okurz 3 months ago

  • Related to action #161729: [sporadic] test fails in containers/build of openqa-in-openqa probably due to temporary download.opensuse.org and zypper issues added
Actions #18

Updated by tinita 3 months ago

  • Description updated (diff)
Actions #19

Updated by livdywan 3 months ago

  • Status changed from Workable to In Progress
  • Assignee set to livdywan

I'm preparing a change to use this action.

Actions #20

Updated by livdywan 3 months ago

  • Status changed from In Progress to Feedback
Actions #21

Updated by livdywan 3 months ago

livdywan wrote in #note-20:

https://github.com/os-autoinst/openQA/pull/5768

https://github.com/os-autoinst/openQA/pull/5772 is another approach retrying docker compose from within the test script itself

Actions #22

Updated by livdywan 3 months ago

  • Status changed from Feedback to Resolved

livdywan wrote in #note-20:

https://github.com/os-autoinst/openQA/pull/5768

Went with this one.

Actions #23

Updated by livdywan 3 months ago

  • Status changed from Resolved to In Progress

Looks like the default 3 retries are not enough, and we need more:

https://github.com/os-autoinst/openQA/actions/runs/10040705287/job/27747275454?pr=5784

failed to solve: process "/bin/sh -c zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/15.6 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.6/15.6 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -yl ca-certificates-mozilla curl gzip rsync xz &&     zypper in -yl openQA-worker os-autoinst-s390-deps os-autoinst-ipmi-deps &&     zypper in -yl qemu-arm qemu-ppc qemu-x86 qemu-tools &&     zypper in -yl qemu-hw-display-virtio-gpu qemu-hw-display-virtio-gpu-pci qemu-hw-display-virtio-vga &&     zypper in -yl kmod &&     (zypper in -yl qemu-ovmf-x86_64 ||:) &&     (zypper in -yl qemu-uefi-aarch64 ||:) &&     zypper clean &&     mkdir -p /root/qemu" did not complete successfully: exit code: 4
 > [worker 2/5] RUN zypper ar -p 95 -f http://download.opensuse.org/repositories/devel:openQA/15.6 devel_openQA &&     zypper ar -p 90 -f http://download.opensuse.org/repositories/devel:openQA:Leap:15.6/15.6 devel_openQA_Leap &&     zypper --gpg-auto-import-keys ref &&     zypper in -yl ca-certificates-mozilla curl gzip rsync xz &&     zypper in -yl openQA-worker os-autoinst-s390-deps os-autoinst-ipmi-deps &&     zypper in -yl qemu-arm qemu-ppc qemu-x86 qemu-tools &&     zypper in -yl qemu-hw-display-virtio-gpu qemu-hw-display-virtio-gpu-pci qemu-hw-display-virtio-vga &&     zypper in -yl kmod &&     (zypper in -yl qemu-ovmf-x86_64 ||:) &&     (zypper in -yl qemu-uefi-aarch64 ||:) &&     zypper clean &&     mkdir -p /root/qemu:
0.367 Repository priorities in effect: (See 'zypper lr -P' for details)
0.367       90 (raised priority)  :  1 repository
0.367       95 (raised priority)  :  1 repository
0.367       99 (default priority) :  7 repositories
done]
..done]
..done]
done]
42.88 Building repository 'Update Repository (Non-Oss)' cache [...done]
42.89 Some of the repositories have not been refreshed because of an error.
------
docker compose exited with non-zero code 1, showing logs:
time="2024-07-22T12:30:40Z" level=warning msg="/tmp/tmp.A9yc0PKuun/worker/docker-compose.yaml: `version` is obsolete"
time="2024-07-22T12:30:40Z" level=warning msg="/tmp/tmp.A9yc0PKuun/worker/docker-compose.yaml: `version` is obsolete"
Actions #24

Updated by openqa_review 3 months ago

  • Due date set to 2024-08-06

Setting due date based on mean cycle time of SUSE QE Tools

Actions #26

Updated by livdywan 3 months ago

  • Assignee changed from livdywan to okurz

I guess @okurz is taking over from here

Actions #27

Updated by okurz 2 months ago

  • Status changed from In Progress to Feedback
Actions #28

Updated by okurz 2 months ago

  • Priority changed from High to Normal

https://github.com/os-autoinst/openQA/pull/5789 merged. Awaiting verification from production.

Actions #29

Updated by okurz 2 months ago

  • Due date deleted (2024-08-06)
  • Status changed from Feedback to Resolved

Past days no problems showed up in CI tests.

Actions

Also available in: Atom PDF