action #168097
closed[qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de
Added by okurz 6 months ago. Updated 5 days ago.
100%
Description
Motivation¶
Upcoming changes to the network will have the effect that dist.suse.de is not accessible anymore for any workload running in NUE2 hence osado tests relying on that should use https://mirror.nue2.suse.org/ instead of corresponding paths in http://dist.suse.de/ along with ensuring that relevant content is available already as https://mirror.nue2.suse.org/ might only have the relevant test assets available delayed.
Acceptance criteria¶
- AC1: osado tests do not rely on http://download.suse.de when running from NUE2
- AC2: osado tests do not rely on NUE2 resources when running from other locations
- AC3.1: Mirrors are defined in workerconf.sls
Notes¶
- globally default value (mirror.nue2) for all of the workers, for workers in CC area (zone-cc) https://download.suse.de.
Out of scope¶
- AC3.2: qem-bot handles also different mirrors here
Suggestions¶
- Review occurences of
git grep dist.suse.dewithin osado and provide means to override the path based on openQA test variables - Same for download.suse.de
- Same for https://github.com/openSUSE/qem-bot/ see here
- Consider setting local mirror variables in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls
Updated by okurz 6 months ago
- Project changed from QA (public) to openQA Tests (public)
- Subject changed from Make openqa.suse.de tests work with mirrors instead of dist.suse.de to [core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de
- Description updated (diff)
- Category set to Enhancement to existing tests
- Assignee deleted (
okurz)
Updated by szarate 5 months ago · Edited
I take that http://dist.suse.de and http://download.suse.de are the same or contain the same information? is that assumption correct?
Also: https://confluence.suse.com/display/~moroni_flores/embargoed-proxy
Updated by dzedro 5 months ago · Edited
I wanted simply replace in medium types download.suse.de & dist.suse.de with mirror.nue2.suse.org, but looks like not everything works
This works
OS_TEST_REPOS=https://download.suse.de/ibs/SUSE:/ALP:/Source:/Standard:/1.0:/Staging:/C/images/repo/SL-Micro-6.0-x86_64/
This does not work
OS_TEST_REPOS=https://mirror.nue2.suse.org/ibs/SUSE:/ALP:/Source:/Standard:/1.0:/Staging:/C/images/repo/SL-Micro-6.0-x86_64/
or
works
QA_HEAD_REPO=http://dist.suse.de/ibs/QA:/Head/SLE-15-SP2/
QA_WEB_REPO=http://dist.suse.de/ibs/QA:/SLE15SP2/update/
does not work
QA_HEAD_REPO=http://mirror.nue2.suse.org/ibs/QA:/Head/SLE-15-SP2/
QA_WEB_REPO=http://mirror.nue2.suse.org/QA:/SLE15SP2/update/
Updated by mkittler 5 months ago
I've seen your qem-bot PR, see my comment on GitHub. I'll review the pillars change.
Note that I have been working on #168115 which had a similar goal. Checkout comment #168115#note-14 for my findings (or rather problems I ran into; I was not really successful in using the mirror for maintenance test).
Updated by dzedro 5 months ago · Edited
IDK what is the point of this changes when we will still use download/dist.suse.de
Also this and probably other repos should be mirrored on mirror.nue2.suse.org ?
http://dist.suse.de/ibs/Devel:/SLEnkins:/testsuites/SLE_12_SP5/ e.g. http://dist.suse.de/ibs/QA:/SLE15SP1/update/ # SLENKINS_TESTSUITES_REPO is sthis still used ?
http://dist.suse.de/ibs/QA:/VERSION/update/ e.g. http://dist.suse.de/ibs/QA:/SLE15SP2/update/ # QA_WEB_REPO
http://download.suse.de/ibs/home:/:/branches:/ e.g. http://download.suse.de/ibs/home:/pcervinka:/branches:/QA:/Head/SLE-15-SP5
http://download.suse.de/ibs/Devel:/Kernel:/VERSION/standard/ e.g. http://download.suse.de/ibs/Devel:/Kernel:/SLE15-SP3/standard/
http://download.suse.de/ibs/Devel:/PubCloud:/Stable:/CrossCloud:/Micro:/
http://download.suse.de/ibs/SUSE:/VERSION:/GA/images/ e.g. http://download.suse.de/ibs/SUSE:/SLE-15-SP6:/GA/images/SLES15-SP6.x86_64-1.0.0-EC2-Build1.26.raw.xz # PUBLIC_CLOUD_IMAGE_LOCATION
http://download.suse.de/ibs/SUSE:/VERSION:/GA/images/ e.g. https://download.suse.de/ibs/SUSE:/SLE-15-SP6:/GA/images/
Updated by szarate 5 months ago
@dzedro are those test settings?
if so, I would prefer to use variable expansion, QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION% which will have to be updated/defined at the moment of either running the test, or scheduling it, but that's for tools team to decide.
Changes to qam-bot are clearly out of scope for qe-core.
Updated by szarate 5 months ago
- Related to action #169159: Allow variable expansion incorporating worker settings size:S added
Updated by dzedro 5 months ago
szarate wrote in #note-17:
@dzedro are those test settings?
if so, I would prefer to use variable expansion,
QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION%which will have to be updated/defined at the moment of either running the test, or scheduling it, but that's for tools team to decide.Changes to qam-bot are clearly out of scope for qe-core.
I got this from db, this is mostly defined in medium types or in group job yamls
I agree REPO_MIRROR_HOST should be used, not sure where should this be defined on worker or jobs group or ?
Updated by szarate 5 months ago · Edited
Should be defined at worker level, https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/924 and on the test distribution, we need to die when it is not found (i.e check/get required var), or its empty.
Updated by waynechen55 5 months ago · Edited
Generally speaking, following resources need to be synced to https://mirror.nue2.suse.org/ for virtualization functional:
1 Installation media of SLES products. For versions have both Online and Full mediums, Full medium is required:
#SLES12
source.http.sles-12-sp3-64=http://schnell.nue.suse.com/SLE12/SLE-12-SP3-Server-GM/x86_64/DVD1/
source.http.sles-12-sp4-64=http://dist.suse.de/install/SLP/SLE-12-SP4-Server-GM/x86_64/DVD1/
source.http.sles-12-sp5-64=http://dist.suse.de/install/SLP/SLE-12-SP5-Server-GM/x86_64/DVD1/
#SLES15
source.http.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/
source.http.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/
source.http.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/
source.http.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/
source.http.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/
source.http.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/
#sle15 sp2 module repos x86_64
x86_64.source.Basesystem.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Web-Scripting/
#sle15 sp3 module repos x86_64
x86_64.source.Basesystem.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Web-Scripting/
#sle15 sp4 module repos x86_64
x86_64.source.Basesystem.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Web-Scripting/
#sle15 sp5 module repos x86_64
x86_64.source.Basesystem.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Web-Scripting/
#sle15 sp6 module repos x86_64
x86_64.source.Basesystem.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Web-Scripting/
#sle15 sp7 module repos x86_64
x86_64.source.Basesystem.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Web-Scripting/
2 virt test related repos:
https://download.suse.de/ibs/Devel:/Virt:/Tests/
3 virt devel related repos:
https://download.suse.de/ibs/Devel:/Virt:/
4 virt update related repos:
http://download.suse.de/ibs/SUSE/Updates
5 virt update milestone repos:
http://download.suse.de/ibs/SUSE:/
Updated by szarate 5 months ago
@waynechen55 please create a subticket of https://progress.opensuse.org/issues/168448 with the information from your comment above, incl where are those defined (i.e where is source.http.sles-12-sp5-64 being defined, etc)
Updated by szarate 5 months ago · Edited
lmanfredi wrote in #note-25:
For YaST , we are using the following resources that need to be synced
@lmanfredi https://progress.opensuse.org/issues/168097#note-24
Updated by mkittler 4 months ago · Edited
This also concerns baremetal tests like this secnario running on squiddlydiddly. Here the problem is more tricky because the iPXE boot script is using assets provided by openQA:
[2024-11-26T09:43:54.202432Z] [debug] [pid:63172] setting iPXE bootscript to:
#!ipxe
echo ++++++++++++++++++++++++++++++++++++++++++
echo ++++++++++++ openQA ipxe boot ++++++++++++
echo + Host: squiddlydiddly.qe.nue2.suse.org
echo ++++++++++++++++++++++++++++++++++++++++++
kernel http://openqa.suse.de/assets/repo/SLE-15-SP7-Online-aarch64-Build41.1-Media1/boot/aarch64/linux install=http://openqa.suse.de/assets/repo/SLE-15-SP7-Online-aarch64-Build41.1-Media1 regurl=http://all-41.1.proxy.scc.suse.de console=ttyAMA0,115200 root=/dev/ram0 initrd=initrd textmode=1 ssh=1 sshpassword=nots3cr3t vnc=1 VNCPassword=nots3cr3t plymouth.enable=0 Y2DEBUG=1 linuxrc.log=/dev/ttyAMA0 linuxrc.core=/dev/ttyAMA0 linuxrc.debug=4,trace reboot_timeout=0
initrd http://openqa.suse.de/assets/repo/SLE-15-SP7-Online-aarch64-Build41.1-Media1/boot/aarch64/initrd
boot
(Those are logs from https://openqa.suse.de/tests/16002467.)
EDIT: I created https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/941 to avoid running those jobs for now. This commit can be reverted when tests have been somehow adapted. Maybe it makes sense to create a separate ticket for that. We should probably coordinate with @MMoese.
Updated by okurz 4 months ago
- Copied to action #170362: [qe-core] Make openqa.suse.de SLE product validation bare-metal tests work within NUE2 added
Updated by szarate 4 months ago
- Blocks action #170422: [qe-core] Jobs running in NUE cannot access required repository in openqa.suse.de - Valid metadata not found at specified URL - auto_review:"Timeout exceeded when accessing.*repomd.xml" added
Updated by szarate 4 months ago
I cloned: https://openqa.suse.de/tests/16036071
openqa-clone-job --within-instance https://openqa.suse.de/tests/16032501 --skip-chained-deps QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION% MIRROR_FTP=ftp://%REPO_MIRROR_HOST%/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_HTTP=http://%REPO_MIRROR_HOST%/assets/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_HTTPS=https://%REPO_MIRROR_HOST%/assets/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_NFS=nfs://%REPO_MIRROR_HOST%/var/lib/openqa/share/factory/repo/SLE-15-SP7-Full-aarch64-Build43.1-Media1 MIRROR_SMB=smb://%REPO_MIRROR_HOST%/inst/SLE-15-SP7-Full-aarch64-Build43.1-Media1
And failed
due to some hardcoded stuff: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/20698
I just triggered: https://openqa.suse.de/tests/16036115
@dzedro if this works, we can move on to do the replacements
Updated by szarate 4 months ago
- Related to action #170491: [Containers] Some zkvm jobs can't to connect to via ftp download.suse.de for s390x added
Updated by szarate 4 months ago
- Related to deleted (action #170491: [Containers] Some zkvm jobs can't to connect to via ftp download.suse.de for s390x)
Updated by okurz 4 months ago
- Related to action #169918: [security] tests never get executed on tyrion and coppi added
Updated by okurz 4 months ago
Are you referring to #168097-29 ? Then I suggest you provide verification runs of what works and then one could adjust worker classes and test schedule accordingly
Updated by MMoese 4 months ago · Edited
I just manually booted a machine (sonic) from osd and installed the latest build of sp7.
sonic-1:~ # ping openqa.suse.de
PING openqa.suse.de(openqa.oqa.prg2.suse.org (2a07:de40:b203:12:0:ff:fe4f:7c2b)) 56 data bytes
64 bytes from openqa.oqa.prg2.suse.org (2a07:de40:b203:12:0:ff:fe4f:7c2b): icmp_seq=1 ttl=62 time=4.94 ms
Updated by dzedro 3 months ago
- Status changed from In Progress to Blocked
https://progress.opensuse.org/issues/169327 and other repos have to be migrated
Updated by openqa_review 3 months ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: fips_install_lvm_full_encrypt
https://openqa.suse.de/tests/16039583#step/bootloader_start/1
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The bugref in the openQA scenario is removed or replaced, e.g.
label:wontfix:boo1234
Expect the next reminder at the earliest in 28 days if nothing changes in this ticket.
Updated by mgrifalconi 3 months ago
- Sprint changed from QE-Core: November Sprint 24 (Nov 06 - Dec 04) to QE-Core: January Sprint 25 (Jan 09 - Feb 04)
Updated by dzedro 3 months ago
I created PR to add back global variable REPO_MIRROR_HOST https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/958
It is required to make expanding of REPO_MIRROR_HOST work.
Revert use of REPO_MIRROR_HOST on s390x ftp repo https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/20934
All s390x workers are in PRG and use openqa.suse.de
Updated by mgrifalconi about 2 months ago
- Sprint changed from QE-Core: January Sprint 25 (Jan 09 - Jan 31) to QE-Core: February Sprint 25 (Feb 03 - Feb 28)
Updated by dzedro about 2 months ago
- Status changed from Feedback to Blocked
Updated by nicksinger about 2 months ago
- Blocks action #177078: enable IPMI test hosts again for production after #168097 is solved added
Updated by dzedro about 2 months ago · Edited
So updates can run only on NUE2 workers, so what's the solution, maintenance scheduled only on NUE2 worker_class ?
sorry, but this proably will not work, there is no guarantee that these projects do not contain embargoed updates.
Updated by waynechen55 about 1 month ago
By the way, this repo SLES-Packages-16.0-x86_64 can not be refreshed successfully.
I opened https://sd.suse.com/servicedesk/customer/portal/1/SD-175976
Updated by livdywan about 1 month ago
I opened https://sd.suse.com/servicedesk/customer/portal/1/SD-175976
Could you share the ticket with OSD Admins, please? I can't see it.
Updated by okurz about 1 month ago
livdywan wrote in #note-58:
I opened https://sd.suse.com/servicedesk/customer/portal/1/SD-175976
Could you share the ticket with OSD Admins, please? I can't see it.
done
Updated by okurz about 1 month ago
dzedro wrote in #note-56:
So updates can run only on NUE2 workers, so what's the solution, maintenance scheduled only on NUE2 worker_class ?
What updates only run on NUE2 workers? For all tests needing direct access to IBS repositories because any non-CC mirror can not or does not provide the data please schedule with WORKER_CLASS+=zone-cc
Updated by pcervinka about 1 month ago
Is there a reason that 15-SP7 mediums settings still point to dist and not to http://mirror.nue2.suse.org/ibs/QA:/Head/SLE-15-SP7 ?
Some of our bare metal machines started to work recently. They can access OSD repositories via http, it means that we can install and validate product in development.
Last failing thing is/was installation of packaged from QA:Head repo.
If it is set to dist, it fails:
https://openqa.suse.de/tests/16844409#step/install_ltp/106
If it is set to mirror, it works.
https://openqa.suse.de/tests/16903618#
This issue as quite complex history and I'm not sure what is expected/correct these days. Not sure what is correct, fix qa head repo in medium or should we set different in test suite?
Updated by mgrifalconi about 1 month ago
- Sprint changed from QE-Core: February Sprint 25 (Feb 03 - Feb 28) to QE-Core: March Sprint 25 (Mar 03 - Mar 31)
Updated by dzedro 7 days ago
- Status changed from Blocked to Feedback
I added WORKER_CLASS=zone-cc on ppc64le workers, other workers are in zone-cc.
@pcervinka I changed all QA_HEAD_REPO in medium types to http://%REPO_MIRROR_HOST%/ibs/QA:/Head/
If there are no other issues this ticket can be resolved ?
Updated by dzedro 5 days ago
- Status changed from Feedback to Resolved
Test with QA:/Head/ repo passed. https://openqa.suse.de/tests/17170459#step/install_ltp/44