Project

General

Profile

action #32038

coordination #17436: [tools][functional][y][epic] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards

[opensuse][functional][medium] Sync all repos for TW and Leap for all tested versions

Added by riafarov almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Start date:
2018-02-20
Due date:
2018-04-24
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

As we expect some problems if we disable online repos, we should sync all repos, so all used packages are available.

Acceptance criteria

  • AC1: All used repos are synced
  • AC2: Official openSUSE online repos are not used (no exceptions)

Tasks

  1. DONE: Identify all repos for all TW and Leap tested versions (including different archs)
  2. DONE: Identify if we can afford syncing all of them -> Make sure that oss, non-oss, debug is synced, avoid source and update for now
  3. DONE: Adjust rsync.pl to sync these repos as per results of task #2 (syncing oss and non-oss and oss debug repos).
  4. Enable behavior which is currently enabled by "DISABLE_ONLINE_REPOS" variable for all test suites PR#4679
  5. Flip "DISABLE_ONLINE_REPOS" to "ENABLE_ONLINE_REPOS" and activate if required at all (e.g. post-release tests) PR#4679.
  6. Avoid big source in general. Optional: Only sync the relevant parts of the source repo for corresponding tests, e.g. zypper_info on source packages or a toolchain or compile test. Alternative: Note down in tickets

Related issues

Related to openSUSE Release Process - action #20892: [perl] product-builder: adjust openQA rsync.plResolved2017-07-28

Related to openQA Tests - action #34288: [qam] test fails in zypper_info - zypper_info shouldn't be startedResolved2018-04-05

Related to openQA Tests - action #36333: [opensuse][y][medium] Invalid repositories - failing kate installResolved2018-05-172018-06-05

History

#1 Updated by okurz almost 4 years ago

  • Subject changed from [opensuse][functional] Sync all repos for TW and Leap for all tested versions to [opensuse][functional][medium] Sync all repos for TW and Leap for all tested versions
  • Description updated (diff)
  • Status changed from New to Workable

Discussed with openSUSE RMs regarding online repos: We have still the online
published repos enabled in many openQA tests.

  • Is it possible to sync all repos, including non-oss, debug, source, update? -> space should not be the problem on o3 but syncing big repos would delay the testing. Probably we already sync debug. We should make sure non-oss is synced as well. Avoid big source in general. Optional: Only sync the relevant parts of the source repo for corresponding tests, e.g. zypper_info on source packages or a toolchain or compile test
  • Is there any build validation test where we need the online published repos enabled? -> No, we could not think of any. This confirms the hypothesis by okurz that relying on them would be an invalid test by definition.

#2 Updated by okurz almost 4 years ago

  • Due date set to 2018-03-27
  • Target version set to Milestone 15

#3 Updated by riafarov over 3 years ago

  • Assignee set to riafarov

#4 Updated by riafarov over 3 years ago

  • Status changed from Workable to In Progress

#5 Updated by riafarov over 3 years ago

Waiting for the reviews and then need to request deployment: https://gitlab.suse.de/openqa/scripts/merge_requests/192
I will trigger the dry runs for all profiles we call from cron

#6 Updated by riafarov over 3 years ago

Waiting for the feedback on https://gitlab.suse.de/openqa/scripts/merge_requests/192 if we want to sync more and then need to adjust tests after rsync is deployed to o3.

#7 Updated by riafarov over 3 years ago

  • Status changed from In Progress to Feedback

#8 Updated by riafarov over 3 years ago

  • Status changed from Feedback to In Progress

#9 Updated by riafarov over 3 years ago

  • Status changed from In Progress to Blocked

Figuring out how to conduct rsync.pl deployment to o3 first.

#10 Updated by riafarov over 3 years ago

  • Status changed from Blocked to In Progress

Here are the instructions how to handle rsync.pl update:
riafarov: o3:/opt/openqa-scripts/HOW_TO_MODIFY explains how to do that. I wonder why that file is not in the repo itself. maybe you can move that file there? do you have ssh access to o3?
riafarov: http://susepaste.org/31955383 if you don't have access to the file

#11 Updated by riafarov over 3 years ago

We will sync rsync changes deployment with work DimStar does for new builder migration. https://gitlab.suse.de/openqa/scripts/merge_requests/196 is additional minor change. https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4679 changes for tests.

#12 Updated by mgriessmeier over 3 years ago

  • Due date changed from 2018-03-27 to 2018-04-10

#13 Updated by riafarov over 3 years ago

  • Related to action #20892: [perl] product-builder: adjust openQA rsync.pl added

#14 Updated by riafarov over 3 years ago

  • Description updated (diff)
  • Status changed from In Progress to Feedback

#15 Updated by michel_mno over 3 years ago

Would the rsync.pl be able to be used outside SUSE network to sync my local openQA server instance ?

#16 Updated by okurz over 3 years ago

michel_mno wrote:

Would the rsync.pl be able to be used outside SUSE network to sync my local openQA server instance ?

Well, first problem I see is that rsync.pl is not open-sourced. Next problem I see currently o3 syncs from an iternal host. Can we access the TW + leap snapshots that are under current testing from the outside?

#17 Updated by riafarov over 3 years ago

okurz wrote:
I was not able to access them using rsync from R&D network, but those work if you add them using zypper, so I guess something can be done here, but not with rsync.pl.

michel_mno wrote:

Would the rsync.pl be able to be used outside SUSE network to sync my local openQA server instance ?

Well, first problem I see is that rsync.pl is not open-sourced. Next problem I see currently o3 syncs from an iternal host. Can we access the TW + leap snapshots that are under current testing from the outside?

#18 Updated by riafarov over 3 years ago

So as of now we only have problems with source repos as we don't want to sync whole repos cause of the single package, so this is remaining part to implement it in rsync.pl and then adjust the test to add OSS source repo.

As per access from outside, we have tested that zypper is able to access snapshot repos outside of R&D network, however you cannot list directories/files over http (gets redirected to https) or ftp.
If full path is known, you should be able to access it over http:

wget http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-non-oss-i586-x86_64-Snapshot20180324/repodata/repomd.xml

#19 Updated by pcervinka over 3 years ago

  • Related to action #34288: [qam] test fails in zypper_info - zypper_info shouldn't be started added

#20 Updated by pcervinka over 3 years ago

Please update conditions for loading zypper_info, it shouldn't be run on SLE12GA. See https://openqa.suse.de/tests/overview?distri=sle&version=12&build=20180405-1&groupid=35 and https://progress.opensuse.org/issues/34288

#21 Updated by pcervinka over 3 years ago

Not only on SLE12GA, it caused also issue on SLE12SP3 https://openqa.suse.de/tests/1591046#step/zypper_info/9

#22 Updated by riafarov over 3 years ago

Another broken test suite: https://openqa.opensuse.org/tests/651007#comments we don't have update repos in the image we have created.

#23 Updated by riafarov over 3 years ago

So, after many fixes for zypper_info we have excluded following scenarios to replace repos:
create_hdd_gnome
kde-live_installation
krypton-live
kde-live

gnome-live

krypton-live-installation
krypton-live-wayland
kde-live-wayland
gnome-live_installation

create_hdd_gnome breaks yast2_control_center as no update repo is available and online update module fails.
For live images on leap we have totally different release process and as of now we may simply not schedule modules if variables are not set. This is still required for TW live, and works there.

Also in create_hdd we get problems with packagekit: https://openqa.opensuse.org/tests/651489#step/zypper_ar/10

Here are fixes for packagekit and live images:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4797

yast2 control center is fixed by adding update repos:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4793

#24 Updated by okurz over 3 years ago

Adding update repos explicitly with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4793 but apparently for openSUSE Tumbleweed we already have an update repo: https://openqa.opensuse.org/tests/652265#step/yast2_control_center/27 Do we need to retrigger the create image job?

#25 Updated by riafarov over 3 years ago

All found issues are fixed. zypper_info is still an issue, as well as yast update module in yast control center

#26 Updated by mgriessmeier over 3 years ago

  • Due date changed from 2018-04-10 to 2018-04-24

#27 Updated by riafarov over 3 years ago

  • Status changed from Feedback to Resolved

So we reached safe state, remaining tasks are mentioned in https://progress.opensuse.org/issues/34732 which is follow up.
Acceptance criteria here was too optimistic, we cannot and should not sync all and always replace. Resolving this ticket.

#28 Updated by riafarov over 3 years ago

  • Related to action #36333: [opensuse][y][medium] Invalid repositories - failing kate install added

Also available in: Atom PDF