Project

General

Profile

action #39449

enable openQA maintenance testing

Added by lnussel almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
QA
Target version:
Start date:
2019-03-11
Due date:
2019-04-05
% Done:

100%

Estimated time:
Duration: 20

Description

openQA and the bot needs to be set up to also test maintenance updates for the new release


Related issues

Related to openQA Tests - action #49628: [qam] test fails in setup_online_repos - missing configure online repositories Rejected2019-03-25

Blocked by openQA Tests - action #50309: [qam][blue][leap][opensuse] Update online_repos behavior to activate online repositoriesResolved2019-04-11

Copied from openSUSE Leap 15.0 - action #25032: enable openQA maintenance testingResolved2018-04-012018-04-19

Copied to openSUSE Leap 15.2 - action #56330: enable openQA maintenance testingResolved2020-03-092020-04-03

History

#1 Updated by lnussel almost 2 years ago

  • Copied from action #25032: enable openQA maintenance testing added

#2 Updated by lnussel over 1 year ago

  • Due date changed from 2018-12-25 to 2019-03-15
  • Start date changed from 2018-12-07 to 2019-03-11

#3 Updated by lnussel over 1 year ago

  • Due date changed from 2019-03-15 to 2019-03-22

#4 Updated by lnussel over 1 year ago

  • Assignee set to okurz

#5 Updated by okurz over 1 year ago

  • Status changed from New to Feedback

First collected complete template dump with openqa-dump-templates --host https://openqa.opensuse.org --full > o3_templates_$(date +%F) . Then added scenarios with

old=15.0; new=15.1; for i in Incidents Updates ; do openqa-dump-templates --host https://openqa.opensuse.org --group "openSUSE Leap $old $i" --full | sed -e "s/$old/$new/g" > leap_${new}_$i && openqa-load-templates --host https://openqa.opensuse.org ./leap_${new}_$i ; done

This created the job groups visible on https://openqa.opensuse.org/admin/groups , reordered them next to the "15.0" and "42.3" update jobs by drag & drop.

Created a test maintenance request in
https://build.opensuse.org/request/show/684360

and bot adaptions in https://github.com/openSUSE/openSUSE-release-tools/pull/1903

#6 Updated by lnussel over 1 year ago

  • % Done changed from 0 to 30

#7 Updated by lnussel over 1 year ago

So before your absence can you hand this over to someone to finish please?

#8 Updated by okurz over 1 year ago

Trying :(

#9 Updated by okurz over 1 year ago

  • Assignee changed from okurz to coolo

coolo can you please take over?

#10 Updated by coolo over 1 year ago

  • Assignee deleted (coolo)

No time

#11 Updated by pluskalm over 1 year ago

QAM will take over

#12 Updated by lnussel over 1 year ago

  • Status changed from Feedback to In Progress
  • Assignee set to pluskalm

#13 Updated by lnussel over 1 year ago

  • Due date changed from 2019-03-22 to 2019-04-05

#14 Updated by pluskalm over 1 year ago

pcervinka will check this

#15 Updated by pcervinka over 1 year ago

Job groups were setup by Marcus last week with Ondrej's assistance.
I will check failures in setup_online_repos:
https://openqa.opensuse.org/tests/890548#step/setup_online_repos/2

#16 Updated by lnussel over 1 year ago

  • Assignee changed from pluskalm to pcervinka

#17 Updated by pcervinka over 1 year ago

  • Related to action #49628: [qam] test fails in setup_online_repos - missing configure online repositories added

#18 Updated by pcervinka over 1 year ago

Created bug for missing dialog for online repositories: https://bugzilla.opensuse.org/show_bug.cgi?id=1130390

#19 Updated by pcervinka over 1 year ago

So, I finished review of Leap 15.1 Updates and Incidents group. We are blocked by bsc#1130390, there are no other issues so far.

#20 Updated by pcervinka over 1 year ago

  • Status changed from In Progress to Blocked

Blocked by bsc#1130390.

#21 Updated by pcervinka over 1 year ago

  • Status changed from Blocked to In Progress

Checking results with latest iso.

#22 Updated by pcervinka over 1 year ago

Latest iso with skelcd fix was released, but tests are still failing.

Step in online_repos https://openqa.opensuse.org/tests/904506#step/online_repos/1 was never designed to be clickable on "YES" button. It just matches needle online-repos-popup and continues by pressing "alt-n". In Leap15.0 it means usual Next, but in 15.1 it means "NO" button to activate online repositories.

[2019-04-11T07:47:19.148 CEST] [debug] >>> testapi::_handle_found_needle: found online-repos-popup-20190303, similarity 1.00 @ 49/259
[2019-04-11T07:47:19.148 CEST] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/installation/online_repos.pm:67 called testapi::wait_screen_change
[2019-04-11T07:47:19.148 CEST] [debug] <<< testapi::wait_screen_change(timeout=10, similarity_level=50)
[2019-04-11T07:47:19.149 CEST] [debug] /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/tests/installation/online_repos.pm:67 called testapi::send_key
[2019-04-11T07:47:19.149 CEST] [debug] <<< testapi::send_key(key='alt-n', do_wait=0)

In Leap 15.0 repositories are activated in setup_online_repos https://openqa.opensuse.org/tests/904522#step/setup_online_repos/1, by clicking on "Configure online repositories", but this is not present in Leap15.1.

It is product behavior change. Is it expected?

lnussel what should be correct behavior and what should be done to fix it? Could be this change caught sooner on TW, if it had update repositories enabled by variables? I suppose, this scenario is not covered by usual TW runs.

If it is a test change, we will have to update behavior in online_repos and setup_online_repos. Or can we have "Configure online repositories" in the same place like in 15.0?

#23 Updated by lnussel over 1 year ago

with the switch to the role selection dialog the behavior you see in 15.1 now is correct. So the test has to be adjusted.

#24 Updated by pcervinka over 1 year ago

  • Blocked by action #50309: [qam][blue][leap][opensuse] Update online_repos behavior to activate online repositories added

#25 Updated by pcervinka over 1 year ago

  • Status changed from In Progress to Blocked

#26 Updated by pcervinka over 1 year ago

Created https://progress.opensuse.org/issues/50309, but don't have space to work on it.

#27 Updated by pcervinka over 1 year ago

Reshuffled my priorities, I will work on poo#50309.

#28 Updated by pcervinka over 1 year ago

Created bug https://bugzilla.opensuse.org/show_bug.cgi?id=1132609 for issue with repositories.
File http://download.opensuse.org/YaST/Repos/_openSUSE_Leap_15.1_Default.xml contains incorrect Leap version:

curl -s  http://download.opensuse.org/YaST/Repos/_openSUSE_Leap_15.1_Default.xml | grep download.opensuse.org
                <url>http://download.opensuse.org/distribution/leap/$releasever/repo/oss/</url>
                <url>http://download.opensuse.org/distribution/leap/$releasever/repo/non-oss/</url>
                <url>http://download.opensuse.org/source/distribution/leap/$releasever/repo/oss/</url>
                <url>http://download.opensuse.org/debug/distribution/leap/$releasever/repo/oss/</url>
                <url>http://download.opensuse.org/update/leap/$releasever/oss</url>
                <url>http://download.opensuse.org/update/leap/$releasever/non-oss/</url>
                <url>http://download.opensuse.org/debug/update/leap/$releasever/oss</url>
                <url>http://download.opensuse.org/update/leap/$releasever-test/</url>

There are just variable names.

#29 Updated by pcervinka over 1 year ago

Faulty behavior reported in bsc#1132609 was issue of build 447.1, latest build 450.2 is fine.

#30 Updated by pcervinka over 1 year ago

  • Blocked by action #50459: [qam][blue][leap] test fails in updates_packagekit_gpk - fail after update added

#31 Updated by pcervinka over 1 year ago

  • Blocked by deleted (action #50459: [qam][blue][leap] test fails in updates_packagekit_gpk - fail after update)

#32 Updated by pcervinka over 1 year ago

  • Status changed from Blocked to Feedback

Today we merged fix for setup of repositories and updated link to latest iso on ooo(thx to ludwig).

And we got first "green" Updates run of Leap 15.1:
https://openqa.opensuse.org/tests/overview?distri=opensuse&version=15.1&build=20190417-1&groupid=61

Here is the quick summary of issues:
bsc#1130390 - Bug in skelcd-control-openSUSE, missing popup about online repositories => product fixed
poo#50309 - Setup of online repositories was not ready for 15.1 => openQA test fixed
bsc#1132609 - $releasever was expanded by yast, but only issue of 447.1 build, build 450.2 is ok => bug closed as invalid
poo#50459 - failure in updates_packagekit_gpk, was only on ooo, couldn't reproduce on private instance, disappeared with latest iso => hmm/meh(?)

Single incident for hello package is green as well:
https://openqa.opensuse.org/tests/overview?distri=opensuse&version=15.1&build=%3A9859%3Ahello.1554960697&groupid=60

There is only problem with update-test-trivial, which fail at qam_verify_package_install:
https://openqa.opensuse.org/tests/overview?distri=opensuse&version=15.1&build=%3A8555%3Aupdate-test-trivial.1553528547&groupid=60
According to Marcus, we don't need to go deeper into this.

I will continue with reviewing of results for next few runs.

#33 Updated by pcervinka over 1 year ago

Removed BETA=1 from medium 15.1 DVD-Updates and DVD-Incidents, latest iso failed with https://openqa.opensuse.org/tests/915581#step/welcome/9. Leap 15.1 is in RC phase now.

#34 Updated by pcervinka over 1 year ago

  • % Done changed from 30 to 90

#35 Updated by lnussel over 1 year ago

anything missing still?

#36 Updated by pcervinka over 1 year ago

  • Status changed from Feedback to Resolved
  • % Done changed from 90 to 100

There is no pending task(rechecked with Marcus).

#37 Updated by lnussel 11 months ago

  • Copied to action #56330: enable openQA maintenance testing added

Also available in: Atom PDF