action #170422
open[qe-core] Jobs running in NUE cannot access required repository in openqa.suse.de - Valid metadata not found at specified URL - auto_review:"Timeout exceeded when accessing.*repomd.xml"
100%
Description
Observation¶
openQA test in scenario sle-15-SP7-Online-ppc64le-extra_tests_textmode@ppc64le fails in
zypper_info
Test suite description¶
Maintainer: QE Core, asmorodskyi,dheidler. Mainly console extratest
Reproducible¶
Fails since (at least) Build 42.3
Expected result¶
Last good: 40.1 (or more recent)
Further details¶
Always latest result in this scenario: latest
Subtasks
Updated by pcervinka about 2 months ago
we have exactly same issue in our kernel job group https://openqa.suse.de/tests/16032501#step/kdump_and_crash/95. It's only on ppc64le architecture and it is network issues.
@szarate isn't it related to CC and some firewall limitation?
Updated by szarate about 2 months ago
- Blocked by action #168097: [qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de added
Updated by szarate about 2 months ago
- Assignee set to szarate
Hi, yes, this is exactly that - We're working on a solution for this, I need to sync with Jozef too
Updated by szarate about 2 months ago
- Subject changed from [qe-core][15sp7][ppc64le]test fails in zypper_info, [repo-source|ftp://openqa.suse.de/SLE-15-SP7-Module-Basesystem-POOL-ppc64le-Build43.1-Media2] Valid metadata not found at specified URL to [qe-core] Jobs running in NUE cannot access required repository in openqa.suse.de - Valid metadata not found at specified URL - auto_review:"Timeout exceeded when accessing.*repomd.xml"
- Status changed from New to Blocked
Updated by szarate about 2 months ago
Lets cross our fingers: https://openqa.suse.de/tests/16036071
openqa-clone-job --within-instance https://openqa.suse.de/tests/16032501 --skip-chained-deps QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION% MIRROR_FTP=ftp://%REPO_MIRROR_HOST%/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_HTTP=http://%REPO_MIRROR_HOST%/assets/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_HTTPS=https://%REPO_MIRROR_HOST%/assets/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_NFS=nfs://%REPO_MIRROR_HOST%/var/lib/openqa/share/factory/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_SMB=smb://%REPO_MIRROR_HOST%/inst/SLE-15-SP7-Full-aarch64-Build43.1-Media1
And failed
Updated by szarate about 2 months ago
Updated by mdati about 2 months ago
- Related to action #170491: [Containers] Some zkvm jobs can't to connect to via ftp download.suse.de for s390x added
Updated by pcervinka about 2 months ago
Still fails in https://openqa.suse.de/tests/16100855#step/kdump_and_crash/95.
024-12-04 14:19:02 <5> susetest(4309) [zypp-core] Exception.cc(log):219 MediaCurl.cc(evaluateCurlCode):958 THROW: Timeout exceeded when accessing 'ftp://mirror.nue2.suse.org/SLE-15-SP7-Module-Basesystem-POOL-ppc64le-Build45.2-Media3/repodata/repomd.xml'.
2024-12-04 14:19:02 <5> susetest(4309) [zypp-core] Exception.cc(log):219 MediaCurl.cc(doGetDoesFileExist):1084 RETHROW: Timeout exceeded when accessing 'ftp://mirror.nue2.suse.org/SLE-15-SP7-Module-Basesystem-POOL-ppc64le-Build45.2-Media3/repodata/repomd.xml'.
2024-12-04 14:19:02 <5> susetest(4309) [zypp-core] Exception.cc(log):219 MediaCurl.cc(getDoesFileExist):832 RETHROW: Timeout exceeded when accessing 'ftp://mirror.nue2.suse.org/SLE-15-SP7-Module-Basesystem-POOL-ppc64le-Build45.2-Media3/repodata/repomd.xml'.
2024-12-04 14:19:02 <5> susetest(4309) [zypp-core] Exception.cc(log):219 mediafacade.cc(provide):520 CAUGHT: Timeout exceeded when accessing 'ftp://mirror.nue2.suse.org/SLE-15-SP7-Module-Basesystem-POOL-ppc64le-Build45.2-Media3/repodata/repomd.xml'.
mirror.nue2.suse.org is defined as "REPO_MIRROR_HOST" : "mirror.nue2.suse.org" and visible in vars.json, but not visible in job settings. Where is it configured?
Updated by okurz about 2 months ago
mirror.nue2.suse.orgis defined as"REPO_MIRROR_HOST" : "mirror.nue2.suse.org"and visible invars.json, but not visible in job settings. Where is it configured?
Worker settings because it depends on the worker location, e.g. if a worker resides in NUE2 or PRG2. It might also be specified in the test code which would be wrong causing PRG2 workers failing to access the NUE2 mirror
Updated by pcervinka about 2 months ago
Testing manually from workstation in Prague within CC zone:
ftp from machine doesn't work
ftp mirror.nue2.suse.org ftp: Can't connect to `10.168.0.5:21': Connection refused ftp: Can't connect to `mirror.nue2.suse.org:ftp'http from same machine works
curl mirror.nue2.suse.org <html> <head><title>Index of /</title></head> <body> <h1>Index of /</h1><hr><pre><a href="../">../</a> <a href="ibs/">ibs/</a> 19-Sep-2024 23:00 - <a href="install/">install/</a> 05-Dec-2024 08:20 - <a href="openSUSE/">openSUSE/</a> 15-Sep-2023 14:31 - <a href="repo/">repo/</a> 17-Oct-2024 10:48 - </pre><hr></body> </html>
Also doesn't work from my laptop and I'm using NUE vpn.
Updated by okurz about 2 months ago
I can confirm that FTP is not being offered from mirror.nue2.suse.org. From sudo nmap -p 21,80,443 mirror.nue2.suse.org
Starting Nmap 7.95 ( https://nmap.org ) at 2024-12-06 15:01 CET
Nmap scan report for mirror.nue2.suse.org (10.168.0.5)
Host is up (0.00046s latency).
Other addresses for mirror.nue2.suse.org (not scanned): 2a07:de40:a100:1:10:168:0:5
PORT STATE SERVICE
21/tcp filtered ftp
80/tcp open http
443/tcp open https
Nmap done: 1 IP address (1 host up) scanned in 1.34 seconds
However the original problem was about OSD which is slightly different. What is same is that we get a timeout in both cases of trying FTP. As OSD is again accessible from NUE2 the solution might be to ensure that the firewall does not block access, at least temporarily.
And URLs like ftp://mirror.nue2.suse.org/SLE-15-SP7-Module-Basesystem-POOL-ppc64le-Build45.2-Media3/ can not work anyway as mirror.nue2.suse.org is a mirror of download.suse.de, not a mirror of openQA and a path including the build number only exists on openQA itself.
Updated by pcervinka about 1 month ago
That's very unfortunate oversight that REPO_MIRROR_HOST has such side effect. I don't know what should be done. Remove variable from worker config?
Updated by pcervinka about 1 month ago
pcervinka wrote in #note-13:
That's very unfortunate oversight that
REPO_MIRROR_HOSThas such side effect. I don't know what should be done. Remove variable from worker config?
maybe, update "broken" test to use http... which can be done independently
Updated by pcervinka about 1 month ago
pcervinka wrote in #note-14:
maybe, update "broken" test to use http... which can be done independently
kdump test was switched to use http to access debug repository on osd.
Updated by openqa_review 29 days ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: extra_tests_textmode
https://openqa.suse.de/tests/16288567#step/zypper_info/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.