Project

General

Profile

Actions

coordination #18602

closed

[sles][functional][epic]review existing autoyast tests (was: test fails in repos because ip and tar is not installed)

Added by asmorodskyi almost 7 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Enhancement to existing tests
Start date:
2017-07-26
Due date:
2017-10-11
% Done:

100%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

Acceptance criteria

  • AC1 all autoyast tests are in a job group which has a proper review workflow (e.g. the job group does not exist anymore, all tests must be either in "SLE functional", "test development" or within the existing "autoyast" job group but properly reviewed or deleted)

Problem

Also there are two kind of autoyast tests, autoyast tests self-contained within os-autoinst-distri-opensuse (1) and a "AY framework created by APAC colleagues" (2).

The following approach is suggested:

  • DONE: find existing stable and passing jobs from job group Regression: Autoyast and move them to sles functional as long as they are of type (1)
  • DONE: review remaining failing jobs from (1), triage them and either delete if unimportant or fix and move to sles functional job group
  • DONE: delete all "aytests" as it seems there are no stakeholders in these
  • find reviewers (more than one)
  • discuss with QA PM (mawerner) how test results should be communication and also who can act as "coordinator"
  • document review workflow changes within https://wiki.microfocus.net/index.php/RD-OPS_QA/openQA_review#Functional.2Bautoyast.2Bstaging

As a reminder: The tests were added for a reason (which I don't know) and by topic they belong to sles functional (they are not about kernel, virtualization, migration, etc.) so it's in the domain of sles functional. https://wiki.microfocus.net/index.php?title=RD-OPS_QA/openQA_review#SLES.2BHA.2Bautoyast.2Bstaging also mentions the responsibility covering autoyast

Further details

Always latest result in this scenario: latest

old information

Observation

openQA test in scenario sle-12-SP3-Server-DVD-x86_64-autoyast_aytests_sles12_minimal+supportserver@64bit fails in
repos

Need to pre-install ip and tar binaries

Reproducible

Fails since (at least) Build 0333 (current job)

Expected result

Last good: 0328 (or more recent)


Subtasks 3 (0 open3 closed)

action #20798: [sle][functional] Move autoyast regression test profiles and verification scripts to aytests-tests repoResolvedriafarov2017-07-26

Actions
action #23736: [sle][functional] Adjust autoinst profiles for sle 15Resolvedriafarov2017-08-28

Actions
action #25604: [sle][functional] job group and review workflow for autoyast testsResolvedmgriessmeier2017-09-272017-10-11

Actions

Related issues 2 (0 open2 closed)

Related to openQA Tests - action #20064: [sles][functional][yast] Bring peace and order to autoyast profilesResolvedriafarov2017-06-26

Actions
Related to openQA Tests - action #23722: [sle][functional][sle15] Update autoyast profiles to be compatible with SLE15Resolvedriafarov2017-08-282017-12-19

Actions
Actions #1

Updated by asmorodskyi almost 7 years ago

  • Assignee set to asmorodskyi
Actions #2

Updated by asmorodskyi almost 7 years ago

it appears that test belongs to group of auto-yast tests created with some custom framework AY used for test auto-yast feature . But because general approach is to use regular testing of auto-yast via openqa was decided to moved this test into https://openqa.suse.de/group_overview/60

Actions #3

Updated by okurz almost 7 years ago

  • Subject changed from test fails in repos because ip and tar is not installed to [sles][functional]review existing autoyast tests (was: test fails in repos because ip and tar is not installed)
  • Description updated (diff)
  • Status changed from New to In Progress
  • Assignee changed from asmorodskyi to okurz

Actually there was a misunderstanding regarding the move of the test as this does not solve anything.

Regarding the "ip command not found" a solution might be to just delete the script lines which don't work because they also previously did not work but the test sitll continued and eventually passed.

Moved passing tests to SLES functional, now waiting for next build to provide a cleaner picture on the remaining ones.

Actions #4

Updated by asmorodskyi almost 7 years ago

I suggesting as very first step delete all "aytest" and only afterwards all other things. In this case we will not waste time on "aytests" which will be deleted in any case

Actions #5

Updated by okurz almost 7 years ago

agreed. There is https://github.com/yast/autoyast-integration-test and these tests seem to also use http://download.opensuse.org/repositories/YaST:/Head so I guess we don't need to do that.

Actions #6

Updated by okurz almost 7 years ago

seems like I forget "autoyast-supportserver" somehow? Strange. Jobs incomplete now in sles functional. Added autoyast-supportserver to sles functional and will try to trigger these tests again.

Actions #7

Updated by okurz almost 7 years ago

  • Priority changed from Normal to Low

We have quite some autoyast tests moved to the sles functional group now and we are getting a better understanding of them. I would like to track this still over the next weeks.

Actions #8

Updated by mkravec almost 7 years ago

  • Priority changed from Low to High

Setting priority to high because of duscussions on YaST call

Actions #10

Updated by mkravec almost 7 years ago

  • % Done changed from 0 to 100

Jobs restarted. Tar should be installed if "ip a" does not fail.

Actions #11

Updated by okurz almost 7 years ago

  • Related to action #20064: [sles][functional][yast] Bring peace and order to autoyast profiles added
Actions #12

Updated by mkravec almost 7 years ago

  • Status changed from In Progress to Resolved
  • Assignee changed from okurz to mkravec
Actions #13

Updated by okurz almost 7 years ago

  • Description updated (diff)
  • Status changed from Resolved to In Progress
  • Assignee changed from mkravec to okurz
  • Priority changed from High to Normal

Thanks for your work to fix the failures but there is still something missing: IMHO the task is not done. Would you mind if I reopen the ticket and assign back to me with a comment what is necessary to be done? The original issue is done but we expanded that ticket to "review autoyast group" which is not done yet. I updated acceptance criteria.

Actions #14

Updated by riafarov over 6 years ago

After couple of discussions we have following ideas:
1) Move all autoyast to separate job group. I do review of this group on daily basis. And we find another person to do meta review and take care of it when I'm not in the office.
2) Move/update all profiles in aytests repo and keep them there. Get rid of support server where possible and add precondition step to modify profiles before installation to do variables expansion, etc. We use same framework for that using same openQA test to run different verification scripts.

Actions #15

Updated by okurz over 6 years ago

  • Status changed from In Progress to Feedback

waiting for subticket

Actions #16

Updated by riafarov over 6 years ago

  • Assignee changed from okurz to riafarov
Actions #17

Updated by okurz over 6 years ago

  • Related to action #23722: [sle][functional][sle15] Update autoyast profiles to be compatible with SLE15 added
Actions #18

Updated by riafarov over 6 years ago

  • Assignee changed from riafarov to okurz
Actions #20

Updated by okurz over 6 years ago

  • Description updated (diff)
  • Category changed from Bugs in existing tests to Enhancement to existing tests

Updated description. So now it's mainly about organisational changes pending.

Actions #21

Updated by okurz over 6 years ago

  • Target version set to Milestone 12
Actions #22

Updated by okurz over 6 years ago

  • Status changed from Feedback to In Progress
  • Assignee deleted (okurz)
  • Target version changed from Milestone 12 to Milestone 11

discussed with QA SLE functional PO

Actions #23

Updated by okurz over 6 years ago

  • Subject changed from [sles][functional]review existing autoyast tests (was: test fails in repos because ip and tar is not installed) to [sles][functional][epic]review existing autoyast tests (was: test fails in repos because ip and tar is not installed)
Actions #24

Updated by okurz over 6 years ago

  • Status changed from In Progress to Resolved
  • Assignee set to okurz

So with #25604 the ACs are covered, what's coming next is covered in related tickets, e.g. cleanup of test suite names and adaptions to SLE15, e.g. see #23722

Until further notice I myself will keep the role of coordinating the efforts around autoyast and do meta-review of the review reports.

Actions #25

Updated by okurz over 6 years ago

  • Due date changed from 2017-10-11 to 2017-11-08

due to changes in a related task

Actions #26

Updated by riafarov over 6 years ago

  • Due date changed from 2017-11-08 to 2017-10-11

due to changes in a related task

Actions #27

Updated by szarate over 3 years ago

  • Tracker changed from action to coordination
Actions

Also available in: Atom PDF