action #32038
closedcoordination #17436: [tools][functional][y][epic] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards
[opensuse][functional][medium] Sync all repos for TW and Leap for all tested versions
0%
Description
Motivation¶
As we expect some problems if we disable online repos, we should sync all repos, so all used packages are available.
Acceptance criteria¶
- AC1: All used repos are synced
- AC2: Official openSUSE online repos are not used (no exceptions)
Tasks¶
- DONE: Identify all repos for all TW and Leap tested versions (including different archs)
- DONE: Identify if we can afford syncing all of them -> Make sure that oss, non-oss, debug is synced, avoid source and update for now
- DONE: Adjust rsync.pl to sync these repos as per results of task #2 (syncing oss and non-oss and oss debug repos).
- Enable behavior which is currently enabled by "DISABLE_ONLINE_REPOS" variable for all test suites PR#4679
- Flip "DISABLE_ONLINE_REPOS" to "ENABLE_ONLINE_REPOS" and activate if required at all (e.g. post-release tests) PR#4679.
- Avoid big source in general. Optional: Only sync the relevant parts of the source repo for corresponding tests, e.g. zypper_info on source packages or a toolchain or compile test. Alternative: Note down in tickets
Updated by okurz almost 7 years ago
- Subject changed from [opensuse][functional] Sync all repos for TW and Leap for all tested versions to [opensuse][functional][medium] Sync all repos for TW and Leap for all tested versions
- Description updated (diff)
- Status changed from New to Workable
Discussed with openSUSE RMs regarding online repos: We have still the online
published repos enabled in many openQA tests.
- Is it possible to sync all repos, including non-oss, debug, source, update? -> space should not be the problem on o3 but syncing big repos would delay the testing. Probably we already sync debug. We should make sure non-oss is synced as well. Avoid big source in general. Optional: Only sync the relevant parts of the source repo for corresponding tests, e.g. zypper_info on source packages or a toolchain or compile test
- Is there any build validation test where we need the online published repos enabled? -> No, we could not think of any. This confirms the hypothesis by okurz that relying on them would be an invalid test by definition.
Updated by okurz almost 7 years ago
- Due date set to 2018-03-27
- Target version set to Milestone 15
Updated by riafarov almost 7 years ago
- Status changed from Workable to In Progress
Updated by riafarov over 6 years ago
Waiting for the reviews and then need to request deployment: https://gitlab.suse.de/openqa/scripts/merge_requests/192
I will trigger the dry runs for all profiles we call from cron
Updated by riafarov over 6 years ago
Waiting for the feedback on https://gitlab.suse.de/openqa/scripts/merge_requests/192 if we want to sync more and then need to adjust tests after rsync is deployed to o3.
Updated by riafarov over 6 years ago
- Status changed from In Progress to Feedback
Updated by riafarov over 6 years ago
- Status changed from Feedback to In Progress
Updated by riafarov over 6 years ago
- Status changed from In Progress to Blocked
Figuring out how to conduct rsync.pl deployment to o3 first.
Updated by riafarov over 6 years ago
- Status changed from Blocked to In Progress
Here are the instructions how to handle rsync.pl update:
riafarov: o3:/opt/openqa-scripts/HOW_TO_MODIFY explains how to do that. I wonder why that file is not in the repo itself. maybe you can move that file there? do you have ssh access to o3?
riafarov: http://susepaste.org/31955383 if you don't have access to the file
Updated by riafarov over 6 years ago
We will sync rsync changes deployment with work DimStar does for new builder migration. https://gitlab.suse.de/openqa/scripts/merge_requests/196 is additional minor change. https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4679 changes for tests.
Updated by mgriessmeier over 6 years ago
- Due date changed from 2018-03-27 to 2018-04-10
Updated by riafarov over 6 years ago
- Related to action #20892: [perl] product-builder: adjust openQA rsync.pl added
Updated by riafarov over 6 years ago
- Description updated (diff)
- Status changed from In Progress to Feedback
Updated by michel_mno over 6 years ago
Would the rsync.pl be able to be used outside SUSE network to sync my local openQA server instance ?
Updated by okurz over 6 years ago
michel_mno wrote:
Would the rsync.pl be able to be used outside SUSE network to sync my local openQA server instance ?
Well, first problem I see is that rsync.pl is not open-sourced. Next problem I see currently o3 syncs from an iternal host. Can we access the TW + leap snapshots that are under current testing from the outside?
Updated by riafarov over 6 years ago
okurz wrote:
I was not able to access them using rsync from R&D network, but those work if you add them using zypper, so I guess something can be done here, but not with rsync.pl.
michel_mno wrote:
Would the rsync.pl be able to be used outside SUSE network to sync my local openQA server instance ?
Well, first problem I see is that rsync.pl is not open-sourced. Next problem I see currently o3 syncs from an iternal host. Can we access the TW + leap snapshots that are under current testing from the outside?
Updated by riafarov over 6 years ago
So as of now we only have problems with source repos as we don't want to sync whole repos cause of the single package, so this is remaining part to implement it in rsync.pl and then adjust the test to add OSS source repo.
As per access from outside, we have tested that zypper is able to access snapshot repos outside of R&D network, however you cannot list directories/files over http (gets redirected to https) or ftp.
If full path is known, you should be able to access it over http:
wget http://openqa.opensuse.org/assets/repo/openSUSE-Tumbleweed-non-oss-i586-x86_64-Snapshot20180324/repodata/repomd.xml
Updated by pcervinka over 6 years ago
- Related to action #34288: [qam] test fails in zypper_info - zypper_info shouldn't be started added
Updated by pcervinka over 6 years ago
Please update conditions for loading zypper_info, it shouldn't be run on SLE12GA. See https://openqa.suse.de/tests/overview?distri=sle&version=12&build=20180405-1&groupid=35 and https://progress.opensuse.org/issues/34288
Updated by pcervinka over 6 years ago
Not only on SLE12GA, it caused also issue on SLE12SP3 https://openqa.suse.de/tests/1591046#step/zypper_info/9
Updated by riafarov over 6 years ago
Another broken test suite: https://openqa.opensuse.org/tests/651007#comments we don't have update repos in the image we have created.
Updated by riafarov over 6 years ago
So, after many fixes for zypper_info we have excluded following scenarios to replace repos:
create_hdd_gnome
kde-live_installation
krypton-live
kde-live
gnome-live
krypton-live-installation
krypton-live-wayland
kde-live-wayland
gnome-live_installation
create_hdd_gnome breaks yast2_control_center as no update repo is available and online update module fails.
For live images on leap we have totally different release process and as of now we may simply not schedule modules if variables are not set. This is still required for TW live, and works there.
Also in create_hdd we get problems with packagekit: https://openqa.opensuse.org/tests/651489#step/zypper_ar/10
Here are fixes for packagekit and live images:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4797
yast2 control center is fixed by adding update repos:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4793
Updated by okurz over 6 years ago
Adding update repos explicitly with https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/4793 but apparently for openSUSE Tumbleweed we already have an update repo: https://openqa.opensuse.org/tests/652265#step/yast2_control_center/27 Do we need to retrigger the create image job?
Updated by riafarov over 6 years ago
All found issues are fixed. zypper_info is still an issue, as well as yast update module in yast control center
Updated by mgriessmeier over 6 years ago
- Due date changed from 2018-04-10 to 2018-04-24
Updated by riafarov over 6 years ago
- Status changed from Feedback to Resolved
So we reached safe state, remaining tasks are mentioned in https://progress.opensuse.org/issues/34732 which is follow up.
Acceptance criteria here was too optimistic, we cannot and should not sync all and always replace. Resolving this ticket.
Updated by riafarov over 6 years ago
- Related to action #36333: [opensuse][y][medium] Invalid repositories - failing kate install added