Project

General

Profile

Actions
Copy link

action #168097

open

[qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de

Added by okurz about 1 month ago. Updated 12 days ago.

Status:
In Progress
Priority:
High
Assignee:
dzedro
Category:
Enhancement to existing tests
Target version:
QA - QE-Core: Ready
Start date:
2024-10-10
Due date:
% Done:

0%

Estimated time:
Difficulty:
Sprint:
QE-Core: November Sprint 24 (Nov 06 - Dec 04)
Tags:
platform-team

Description

Motivation

Upcoming changes to the network will have the effect that dist.suse.de is not accessible anymore for any workload running in NUE2 hence osado tests relying on that should use https://mirror.nue2.suse.org/ instead of corresponding paths in http://dist.suse.de/ along with ensuring that relevant content is available already as https://mirror.nue2.suse.org/ might only have the relevant test assets available delayed.

Acceptance criteria

  • AC1: osado tests do not rely on http://download.suse.de when running from NUE2
  • AC2: osado tests do not rely on NUE2 resources when running from other locations
  • AC3.1: Mirrors are defined in workerconf.sls

Notes

  • globally default value (mirror.nue2) for all of the workers, for workers in CC area (zone-cc) https://download.suse.de.

Out of scope

  • AC3.2: qem-bot handles also different mirrors here

Suggestions

Related issues 1 (1 open — 0 closed)

Related to openQA Project - action #169159: Allow variable expansion incorporating worker settings size:SBlockedmkittler2024-10-31

Actions
Actions
Copy link
 #2

Updated by okurz about 1 month ago

  • Parent task set to #166598
Actions
Copy link
 #4

Updated by okurz about 1 month ago

  • Project changed from QA to openQA Tests
  • Subject changed from Make openqa.suse.de tests work with mirrors instead of dist.suse.de to [core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de
  • Description updated (diff)
  • Category set to Enhancement to existing tests
  • Assignee deleted (okurz)
Actions
Copy link
 #5

Updated by okurz about 1 month ago

  • Target version changed from Ready to QE-Core: Ready

@szarate moving this to "QE-Core: Ready" to bring this ticket to your attention. Feel welcome to move it elsewhere.

Actions
Copy link
 #7

Updated by szarate about 1 month ago

  • Tags set to platform-team
  • Subject changed from [core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de to [qe-core] Make openqa.suse.de tests work with mirrors instead of dist.suse.de or download.suse.de
Actions
Copy link
 #9

Updated by szarate 29 days ago

  • Description updated (diff)
Actions
Copy link
 #10

Updated by dzedro 24 days ago

  • Status changed from New to In Progress
  • Assignee set to dzedro
Actions
Copy link
 #11

Updated by szarate 24 days ago · Edited

I take that http://dist.suse.de and http://download.suse.de are the same or contain the same information? is that assumption correct?

Also: https://confluence.suse.com/display/~moroni_flores/embargoed-proxy

Actions
Copy link
 #12

Updated by dzedro 24 days ago · Edited

I wanted simply replace in medium types download.suse.de & dist.suse.de with mirror.nue2.suse.org, but looks like not everything works
This works
OS_TEST_REPOS=https://download.suse.de/ibs/SUSE:/ALP:/Source:/Standard:/1.0:/Staging:/C/images/repo/SL-Micro-6.0-x86_64/
This does not work
OS_TEST_REPOS=https://mirror.nue2.suse.org/ibs/SUSE:/ALP:/Source:/Standard:/1.0:/Staging:/C/images/repo/SL-Micro-6.0-x86_64/

or
works
QA_HEAD_REPO=http://dist.suse.de/ibs/QA:/Head/SLE-15-SP2/
QA_WEB_REPO=http://dist.suse.de/ibs/QA:/SLE15SP2/update/
does not work
QA_HEAD_REPO=http://mirror.nue2.suse.org/ibs/QA:/Head/SLE-15-SP2/
QA_WEB_REPO=http://mirror.nue2.suse.org/QA:/SLE15SP2/update/

Actions
Copy link
 #13

Updated by dzedro 24 days ago

@okurz what does this mean ? AC3.1: Mirrors are defined in workerconf.sls

Actions
Copy link
 #15

Updated by mkittler 24 days ago

I've seen your qem-bot PR, see my comment on GitHub. I'll review the pillars change.

Note that I have been working on #168115 which had a similar goal. Checkout comment #168115#note-14 for my findings (or rather problems I ran into; I was not really successful in using the mirror for maintenance test).

Actions
Copy link
 #17

Updated by szarate 23 days ago

@dzedro are those test settings?

if so, I would prefer to use variable expansion, QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION% which will have to be updated/defined at the moment of either running the test, or scheduling it, but that's for tools team to decide.

Changes to qam-bot are clearly out of scope for qe-core.

Actions
Copy link
 #18

Updated by szarate 23 days ago

  • Related to action #169159: Allow variable expansion incorporating worker settings size:S added
Actions
Copy link
 #19

Updated by szarate 23 days ago

  • Description updated (diff)
Actions
Copy link
 #20

Updated by szarate 23 days ago

Marking AC3.2 as out of scope; changes required exceed what qe-core can do.

Actions
Copy link
 #21

Updated by dzedro 22 days ago

szarate wrote in #note-17:

@dzedro are those test settings?

if so, I would prefer to use variable expansion, QA_HEAD_REPO=https://%REPO_MIRROR_HOST%/ibs/QA:/Head/SLE-%VERSION% which will have to be updated/defined at the moment of either running the test, or scheduling it, but that's for tools team to decide.

Changes to qam-bot are clearly out of scope for qe-core.

I got this from db, this is mostly defined in medium types or in group job yamls
I agree REPO_MIRROR_HOST should be used, not sure where should this be defined on worker or jobs group or ?

Actions
Copy link
 #22

Updated by szarate 22 days ago · Edited

Should be defined at worker level, https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/924 and on the test distribution, we need to die when it is not found (i.e check/get required var), or its empty.

Actions
Copy link
 #23

Updated by waynechen55 18 days ago · Edited

Generally speaking, following resources need to be synced to https://mirror.nue2.suse.org/ for virtualization functional:

1 Installation media of SLES products. For versions have both Online and Full mediums, Full medium is required:

#SLES12
source.http.sles-12-sp3-64=http://schnell.nue.suse.com/SLE12/SLE-12-SP3-Server-GM/x86_64/DVD1/
source.http.sles-12-sp4-64=http://dist.suse.de/install/SLP/SLE-12-SP4-Server-GM/x86_64/DVD1/
source.http.sles-12-sp5-64=http://dist.suse.de/install/SLP/SLE-12-SP5-Server-GM/x86_64/DVD1/
#SLES15
source.http.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/
source.http.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/
source.http.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/
source.http.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/
source.http.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/
source.http.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/

#sle15 sp2 module repos x86_64
x86_64.source.Basesystem.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp2-64=http://dist.suse.de/install/SLP/SLE-15-SP2-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp3 module repos x86_64
x86_64.source.Basesystem.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp3-64=http://dist.suse.de/install/SLP/SLE-15-SP3-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp4 module repos x86_64
x86_64.source.Basesystem.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp4-64=http://dist.suse.de/install/SLP/SLE-15-SP4-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp5 module repos x86_64
x86_64.source.Basesystem.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp5-64=http://dist.suse.de/install/SLP/SLE-15-SP5-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp6 module repos x86_64
x86_64.source.Basesystem.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp6-64=http://dist.suse.de/install/SLP/SLE-15-SP6-Full-GM/x86_64/DVD1/Module-Web-Scripting/

#sle15 sp7 module repos x86_64
x86_64.source.Basesystem.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Basesystem/
x86_64.source.Desktop-Applications.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Desktop-Applications/
x86_64.source.Server-Applications.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Server-Applications/
x86_64.source.Legacy.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Legacy/
x86_64.source.Web-Scripting.sles-15-sp7-64=http://dist.suse.de/install/SLP/SLE-15-SP7-Full-LATEST/x86_64/DVD1/Module-Web-Scripting/

2 virt test related repos:

https://download.suse.de/ibs/Devel:/Virt:/Tests/

3 virt devel related repos:

https://download.suse.de/ibs/Devel:/Virt:/

4 virt update related repos:

http://download.suse.de/ibs/SUSE/Updates

5 virt update milestone repos:

http://download.suse.de/ibs/SUSE:/
Actions
Copy link
 #24

Updated by szarate 18 days ago

@waynechen55 please create a subticket of https://progress.opensuse.org/issues/168448 with the information from your comment above, incl where are those defined (i.e where is source.http.sles-12-sp5-64 being defined, etc)

Actions
Copy link
 #27

Updated by szarate 17 days ago

  • Description updated (diff)
Actions
Copy link
 #28

Updated by szarate 12 days ago

  • Sprint set to QE-Core: November Sprint 24 (Nov 06 - Dec 04)
Actions
Copy link

Also available in: Atom PDF