Project

General

Custom queries

Profile

Actions

coordination #17436

closed

[tools][functional][y][epic] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards

Added by okurz almost 8 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Target version:
SUSE QA (private) - Milestone 22
Start date:
2018-02-12
Due date:
2019-02-26
% Done:

100%

Estimated time:
(Total: 21.00 h)
Difficulty:

Description

acceptance criteria

  • AC1: Only repository content from openqa.opensuse.org is used for all openSUSE tests after installation
  • AC2: Same as AC1 but for the installer system itself

tasks

  • check test cases for the execution of tests/update/zypper_clear_repos.pm and see if the same approach can be adapted for a generic approach (1-2h)
  • start by selecting a standard gnome/kde test suite and any extra tests scenario, make sure the online repos not pointing to openqa.opensuse.org are disabled and check if all packages can still be installed (2-4h)
  • rewrite all occurences of "download.opensuse.org" in the SUT to the according path pointing to openqa.opensuse.org, e.g. with a test module that calls sed -i -e 's/download.opensuse.org/…/g' /etc/zypp/repos.d/* in the installed system (2-8h)
  • investigate how repos are used in the installer system, replace occurences of download.opensuse.org as described above or not add any online repos there if possible (2-16h)

further details

observation

see for example https://bugzilla.suse.com/show_bug.cgi?id=1026767. This bug was just "found" by openQA when also users reported this. The reason is probably that the tests still have published repos from download.opensuse.org enabled causing an upgrade to find packages from either the new snapshot or the old published data which can make tests pass when they shouldn't. Then the tests fail in the subsequent snapshot because the repos were updated in the meantime.

proposal

proposal by nussel in #17818
On openSUSE the 'crash' test doesn't work as the debuginfo repo is not mirrored to openqa. So whenever the kernel gets updated that test fails due to a mismatch between the installed kernel and the debug information in the debuginfo repo.

openQA should keep a local copy of all repos and intercept access to download.opensuse.org so files are downloaded from matching repos actually.

from #17926

Follow up to bug https://bugzilla.opensuse.org/show_bug.cgi?id=1030759

The problem:
whlie testing upgrade scenarios from e.g. 42.2 to TW-Next (the version being tested in openQA), the CURRENTLY published TW on download.o.o is ALSO taken into account (online repo enabled)

In case of the boo#1030759 this results in the updater NOT tagging libmutter0 is an obsolete package (it is tagged 'weakremove', which means it is ONLY marked for deletion if it is NOT found in any of the enabled repos. As TW (published) still contains the package, it is not removed - furhter blocking the update of gnome-shell * mutter, which leads to the test failures as seen in e.g. https://openqa.opensuse.org/tests/375730

We had seen similar test issues in the past, be it that stuff 'works' during the openQA test until TW-Next is published (then packages disappear, and tests would no longer pass)
In any case, having the repo published at download.o.o available to the tests is defeating openQA as it is intended totally, as many tests are not reliable this way (testing a repo combination that no user will possibly be able to have on his system)

https://openqa.opensuse.org/tests/375730#step/upgrade_select_opensuse/2 Shows that the repos are in fact enabled. https://openqa.opensuse.org/tests/375807#step/start_install/18 is an example from the 'other direction of the same problem'. With snapshot 0320, libmarco-private0 was dropped (replaced by libmarco-private1); when 0320 was being tested though, the previously published snapshot was available and thus this did not show an error in its test run at https://openqa.opensuse.org/tests/374330. As soon as snapshot 0320 has been published, the available package list changed and the test is thus invalid (and fails with one day delay). After 0322 was published, reran the 42.2-> tw update passed. before publish: https://openqa.opensuse.org/tests/375730 vs after publish and rerun: https://openqa.opensuse.org/tests/376303#

The whole openSUSE Factory snapshot should be synced to o3 and probably the same for Leap so all repository content should be there on o3.


Files

blockage_install_mc.png (41 KB) blockage_install_mc.png zypper in mc EDiGiacinto, 2017-06-14 15:09
blocks_dnsmasq.png (41.6 KB) blocks_dnsmasq.png zypper in dnsmasq EDiGiacinto, 2017-06-14 15:09
canvas2.png (26.9 KB) canvas2.png zypper in nano EDiGiacinto, 2017-06-14 15:09
packages_are_available.png (45.3 KB) packages_are_available.png zypper se mc EDiGiacinto, 2017-06-14 15:09
package_block_terminator.png (12.6 KB) package_block_terminator.png zypper in terminator EDiGiacinto, 2017-06-14 15:09

Subtasks 7 (0 open7 closed)

action #31645: [opensuse][functional] find affected test suites and disable/replace online repos thereResolvedriafarov2018-02-122018-04-24

Actions
action #32038: [opensuse][functional][medium] Sync all repos for TW and Leap for all tested versionsResolvedriafarov2018-02-202018-04-24

Actions
action #32332: [opensuse][functional][y][fast] repo bleed affecting gnuhealth installationResolvedriafarov2018-02-272018-04-10

Actions
action #34732: [functiona][y][opensuse] make zypper_info use mirrored source repo, do not remove official update reposResolvedriafarov2018-04-112019-01-15

Actions
action #38393: [functional][u] updates_packagekit_kde must not be scheduled before zypper_clear_reposResolveddheidler2018-07-132018-08-14

Actions
action #45923: [functional][opensuse][y] Enable sync of source repo for arm and ppc for TW and leapResolvedJERiveraMoya2019-01-102019-02-12

Actions
action #46127: [functional][y] test fails in firewall_enabled - we should test firewalld on upgrade scenarioResolvedriafarov2019-01-152019-02-26

Actions

Related issues 10 (0 open10 closed)

Related to openQA Tests (public) - action #18070: [opensuse][leap][kernel][functional][tools] test fails in crash because it can not find the debuginfo packageResolvedmichalnowak2017-03-28

Actions
Related to openQA Tests (public) - action #29366: [functional][easy][bsc#1073238] Add toolchain test for openSUSEResolvedSLindoMansilla2017-12-132018-02-27

Actions
Related to openQA Tests (public) - action #40286: [functional][y][opensuse] kdump_and_crash: the repo-name to refresh does not matchResolvedriafarov2018-08-282018-09-25

Actions
Related to openQA Tests (public) - action #42455: [opensuse][aarch64][functional][y] openSUSE Tumbleweed aarch64 oss repo seems to be deleted however we invest 41GB for the debuginfo repoResolvedokurz2018-10-14

Actions
Has duplicate openQA Project (public) - action #17818: mirror and redirect all repos on openqaRejected2017-03-20

Actions
Has duplicate openQA Tests (public) - action #17926: [tools]Tumbleweed upgrade tests have download.o.o repos enabledRejected2017-03-23

Actions
Has duplicate openQA Tests (public) - action #28438: [functional][opensuse] leap 15.0] package squid is missingRejectedokurz2017-11-27

Actions
Has duplicate openQA Tests (public) - action #28156: [opensuse][leap 15.0][functional] yast2-ncurses proxy server failed because of missing package squidRejectedokurz2017-11-222017-12-19

Actions
Blocks openQA Tests (public) - action #12280: [opensuse][tw] Tumbleweed Test Suite gets packages from 'last published' oss repoResolvedokurz2016-06-09

Actions
Copied from openQA Tests (public) - action #16618: [qe-core][opensuse][functional] Tumbleweed upgrade testsResolvedrfan1

Actions
#1

Updated by okurz almost 8 years ago

  • Copied from action #16618: [qe-core][opensuse][functional] Tumbleweed upgrade tests added
#3

Updated by okurz almost 8 years ago

  • Assignee deleted (okurz)
  • Priority changed from High to Normal
#4

Updated by RBrownSUSE almost 8 years ago

  • Project changed from openQA Tests (public) to openQA Project (public)
  • Subject changed from Tumbleweed tests must not have access to published repos to [tools]Tumbleweed tests must not have access to published repos
  • Category deleted (Bugs in existing tests)
#5

Updated by okurz over 7 years ago

  • Has duplicate action #17818: mirror and redirect all repos on openqa added
#6

Updated by okurz over 7 years ago

  • Project changed from openQA Project (public) to openQA Tests (public)
  • Subject changed from [tools]Tumbleweed tests must not have access to published repos to [tools] tests must not have access to published repos, e.g. Tumbleweed -> mirror all on openQA instance?
  • Description updated (diff)
#7

Updated by okurz over 7 years ago

  • Category set to Bugs in existing tests
#8

Updated by michalnowak over 7 years ago

  • Has duplicate action #18070: [opensuse][leap][kernel][functional][tools] test fails in crash because it can not find the debuginfo package added
#9

Updated by michalnowak over 7 years ago

  • Has duplicate deleted (action #18070: [opensuse][leap][kernel][functional][tools] test fails in crash because it can not find the debuginfo package)
#10

Updated by michalnowak over 7 years ago

  • Related to action #18070: [opensuse][leap][kernel][functional][tools] test fails in crash because it can not find the debuginfo package added
#12

Updated by okurz over 7 years ago

  • Has duplicate action #17926: [tools]Tumbleweed upgrade tests have download.o.o repos enabled added
#13

Updated by okurz over 7 years ago

  • Description updated (diff)
  • Priority changed from Normal to High
#17

Updated by SLindoMansilla over 7 years ago

  • Assignee set to SLindoMansilla
  • Target version set to Milestone 8
#19

Updated by RBrownSUSE over 7 years ago

  • Assignee changed from SLindoMansilla to EDiGiacinto
#20

Updated by okurz over 7 years ago

  • Status changed from New to In Progress
#23

Updated by EDiGiacinto over 7 years ago

  • Status changed from In Progress to Feedback

Updated by EDiGiacinto over 7 years ago

#32

Updated by EDiGiacinto over 7 years ago

  • Assignee deleted (EDiGiacinto)
#33

Updated by RBrownSUSE over 7 years ago

  • Subject changed from [tools] tests must not have access to published repos, e.g. Tumbleweed -> mirror all on openQA instance? to [tools][functional] tests must not have access to published repos, e.g. Tumbleweed -> mirror all on openQA instance?
  • Priority changed from High to Normal
  • Target version deleted (Milestone 8)
#34

Updated by okurz about 7 years ago

  • Subject changed from [tools][functional] tests must not have access to published repos, e.g. Tumbleweed -> mirror all on openQA instance? to [tools][functional] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards
  • Description updated (diff)
  • Status changed from Feedback to In Progress
  • Priority changed from Normal to High
#35

Updated by okurz about 7 years ago

  • Target version set to Milestone 12
#36

Updated by okurz about 7 years ago

  • Blocks action #12280: [opensuse][tw] Tumbleweed Test Suite gets packages from 'last published' oss repo added
#39

Updated by okurz about 7 years ago

  • Subject changed from [tools][functional] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards to [tools][functional][medium] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards
#40

Updated by okurz about 7 years ago

  • Has duplicate action #28438: [functional][opensuse] leap 15.0] package squid is missing added
#41

Updated by okurz about 7 years ago

  • Due date set to 2017-12-05
#42

Updated by okurz about 7 years ago

  • Due date changed from 2017-12-05 to 2017-12-19
#43

Updated by okurz about 7 years ago

  • Has duplicate action #28156: [opensuse][leap 15.0][functional] yast2-ncurses proxy server failed because of missing package squid added
#44

Updated by riafarov about 7 years ago

  • Assignee set to riafarov
#46

Updated by riafarov about 7 years ago

  • Status changed from In Progress to Feedback
#50

Updated by SLindoMansilla almost 7 years ago

  • Related to action #29366: [functional][easy][bsc#1073238] Add toolchain test for openSUSE added
#51

Updated by okurz almost 7 years ago

  • Due date changed from 2017-12-19 to 2018-01-17
#52

Updated by okurz almost 7 years ago

  • Target version changed from Milestone 12 to Milestone 13
#53

Updated by okurz almost 7 years ago

  • Due date changed from 2018-01-17 to 2018-01-30
#54

Updated by riafarov almost 7 years ago

  • Assignee changed from riafarov to okurz
#55

Updated by riafarov almost 7 years ago

  • Due date changed from 2018-01-30 to 2018-02-13
  • Target version changed from Milestone 13 to Milestone 14
#56

Updated by okurz almost 7 years ago

  • Status changed from Feedback to In Progress
#57

Updated by okurz almost 7 years ago

  • Status changed from In Progress to Feedback
  • Assignee changed from okurz to riafarov
#58

Updated by riafarov almost 7 years ago

  • Subject changed from [tools][functional][medium] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards to [tools][functional][epic] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards
#59

Updated by okurz almost 7 years ago

  • Target version changed from Milestone 14 to Milestone 16
#61

Updated by okurz almost 7 years ago

  • Due date set to 2018-03-27
#62

Updated by okurz almost 7 years ago

  • Has duplicate action #32332: [opensuse][functional][y][fast] repo bleed affecting gnuhealth installation added
#64

Updated by okurz almost 7 years ago

  • Due date changed from 2018-03-27 to 2018-04-24
#65

Updated by okurz over 6 years ago

  • Has duplicate deleted (action #32332: [opensuse][functional][y][fast] repo bleed affecting gnuhealth installation)
#66

Updated by okurz over 6 years ago

  • Blocks action #32332: [opensuse][functional][y][fast] repo bleed affecting gnuhealth installation added
#67

Updated by okurz over 6 years ago

  • Blocks deleted (action #32332: [opensuse][functional][y][fast] repo bleed affecting gnuhealth installation)
#70

Updated by okurz over 6 years ago

  • Due date changed from 2018-04-24 to 2018-06-19
#71

Updated by okurz over 6 years ago

  • Target version changed from Milestone 16 to Milestone 17
#73

Updated by riafarov over 6 years ago

  • Due date changed from 2018-06-19 to 2018-04-24
#74

Updated by okurz over 6 years ago

  • Subject changed from [tools][functional][epic] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards to [tools][functional][y][epic] tests must not have access to published repos, e.g. download.o.o on Tumbleweed -> remove all references to download.o.o from repos during installation and afterwards
#75

Updated by riafarov over 6 years ago

  • Due date changed from 2018-04-24 to 2018-12-18
#76

Updated by okurz over 6 years ago

  • Target version changed from Milestone 17 to Milestone 17
#77

Updated by okurz over 6 years ago

  • Target version changed from Milestone 17 to Milestone 22
#78

Updated by okurz over 6 years ago

  • Related to action #40286: [functional][y][opensuse] kdump_and_crash: the repo-name to refresh does not match added
#81

Updated by okurz about 6 years ago

  • Status changed from Feedback to Blocked
#82

Updated by riafarov about 6 years ago

  • Due date changed from 2018-12-18 to 2019-01-15
#83

Updated by riafarov almost 6 years ago

  • Due date changed from 2019-01-15 to 2019-02-12
#84

Updated by riafarov almost 6 years ago

  • Due date changed from 2019-02-12 to 2019-02-26
#85

Updated by riafarov almost 6 years ago

  • Status changed from Blocked to Resolved
#87

Updated by okurz almost 6 years ago

  • Related to action #42455: [opensuse][aarch64][functional][y] openSUSE Tumbleweed aarch64 oss repo seems to be deleted however we invest 41GB for the debuginfo repo added
#88

Updated by szarate about 4 years ago

  • Tracker changed from action to coordination
Actions

Also available in: Atom PDF