Project

General

Profile

Actions

coordination #20580

closed

[sle][functional][epic] sle 15 setup

Added by okurz over 6 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
New test
Start date:
2017-07-20
Due date:
2017-09-27
% Done:

100%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

goal

New job group Functional which should be filled with a useful test coverage.

current state

riafarov and okurz have migrated all test suites from "Functional: Server" job group to "Functional" sub group of SLE 15: https://openqa.suse.de/group_overview/110
We also have triggered all test suites for x86_64 architecture in order to get an overview of common issues. After identifying most common problems and fixing them for this job group, we may migrate other job groups in similar way.

tasks

  • DONE: dumped templates for functional job group, adjusted for sle15, load
  • DONE: temporarily adjusted sync and trigger cron job to run daily only, not every 5 minutes as most new builds won't bring changes to actually fix job failures
  • DONE: incorporate review process
  • DONE: focus on former create_hdd_minimal_base+sdk scenario as a major dependency for many other job groups, e.g. create_hdd_development (clone existing sle15 job, add repos from dist.suse.de with ADDON_URL or similar, etc.)
  • DONE: fix installation only test jobs
  • DONE: fix jobs previously relying on server packages, addons, modules by implementing a workaround based on custom repos and isos
  • DONE: crosscheck other job groups, e.g. "Regression: Autoyast"
  • DONE: delete the scenarios that do not apply anymore or rename and adjust accordingly
  • optional: implement a script based solution to compare SLE15 coverage with SLE12SP3 covering multiple job groups so that we can be sure we did not forget any important scenario

further details

okurz created new job group Functional which should be filled with a useful test coverage. To make sure we don't forget anything I would

1) dump_templates the existing sle 12 sp3 functional job group and load_templates that into the new group and then sort out the useless parts and change the scenarios that need to be adapted

Alternatives are
2) same as above but move everything into the corresponding "test development" job group of SLE 15: https://openqa.suse.de/group_overview/110
and then selectively move over jobs.

3) Pick scenario by scenario whatever is considered useful and working. Challenge: We also have triggered all probably need a script to compare the existing test suites for x86_64 architecture in order to get an overview of common issues. After identifying most common problems coverage with the current coverage, evaluate the difference and fixing them for this job group, make sure we may migrate other job groups in similar way.

Attached json file was used to create job templates.


Files

sle15.json (200 KB) sle15.json riafarov, 2017-08-01 11:50

Subtasks 14 (0 open14 closed)

action #20646: [sle][functional][sle15] Evaluate the task copying the test templates from SLE 12 SP3 to SLE 15Resolvedriafarov2017-07-20

Actions
action #20648: [sle][functional][sle15] Evaluate creating template entries on template file in os-autoinst-distri-opensuse for SLE 15Resolvedriafarov2017-07-20

Actions
action #20984: [sle][functional][sle15] adapt minimal_base+sdk_create_hdd test for sle 15Resolvedriafarov2017-08-01

Actions
action #21044: [sle][functional][sle15] adapt create_hdd_textmode test suite for sle 15Resolvedriafarov2017-08-03

Actions
action #23438: [sle][functional][sle15] decide how and start using SCC using SUSEConnectResolvedriafarov2017-08-17

Actions
action #23448: [sles][functional][sle15] Introduce module to select product during installationResolvedriafarov2017-08-18

Actions
action #23460: [sles][functional] test fails in change_desktop - Missing packagesResolvedokurz2017-08-18

Actions
action #23462: [sles][functional] test fails in welcome - broken frames in ncurses in yast installationRejectedokurz2017-08-18

Actions
action #23466: [sles][functional] test fails in addon_products_sle - Missing packages for SDK moduleRejected2017-08-18

Actions
action #23492: [sle][functional]test fails in scc_registration - openQA SLE media are missing SCC_URLResolvedriafarov2017-08-202017-09-27

Actions
action #23504: [sle][functional] Adapt WE tests for SLE 15Resolvedriafarov2017-08-21

Actions
action #20824: [sles][functional] test fails in change_desktop - Failed to select default product pattern, later then fails with booting into wrong sessionResolvedokurz2017-07-27

Actions
action #23856: [sle][functional] Adjust test to the changes in welcome screenResolvedriafarov2017-09-01

Actions
action #23938: [sle][functional] Remove workarounds for broken dependencies and fail accordinglyResolvedriafarov2017-09-062017-09-27

Actions

Related issues 8 (0 open8 closed)

Related to openQA Tests - action #20582: [sle][functional] 'if check_var('VERSION, 15)" should be changed to "sle_version_at_least('15')"Resolvedriafarov2017-07-18

Actions
Related to openQA Tests - action #20630: [sles][functional] test fails in start_install - Create a workaround for missing checksums and unsigned files and perform installationResolvedSLindoMansilla2017-07-20

Actions
Related to openQA Tests - action #23402: [sles][functional] Adjust create_hdd_gnome test suite for sle15Resolvedriafarov2017-08-16

Actions
Related to openQA Tests - action #23736: [sle][functional] Adjust autoinst profiles for sle 15Resolvedriafarov2017-08-28

Actions
Related to openQA Tests - action #24704: [sles][functional] test fails in partitioning - Adapt the test to handle the new GPT as default. Boot partition should be bigger.Rejectedjorauch2017-09-062017-09-27

Actions
Related to openQA Tests - action #24706: [sles][functional] test fails in first_boot - Adapt test to workaround / accept behavior to continue with console testsResolvedSLindoMansilla2017-09-062017-10-11

Actions
Related to openQA Tests - action #25158: [sles][opensuse][functional]test fails in yast2_apparmorResolvedokurz2017-09-112017-09-27

Actions
Related to openQA Tests - coordination #23886: [sle][functional][desktop][sle15][epic] initial setupResolvedqkzhu2017-10-122017-11-08

Actions
Actions #2

Updated by SLindoMansilla over 6 years ago

  • Related to action #20582: [sle][functional] 'if check_var('VERSION, 15)" should be changed to "sle_version_at_least('15')" added
Actions #3

Updated by SLindoMansilla over 6 years ago

  • Related to action #20630: [sles][functional] test fails in start_install - Create a workaround for missing checksums and unsigned files and perform installation added
Actions #4

Updated by riafarov over 6 years ago

  • File sle15_test_development.json added

I've dumped sle 12 sp3 functional job group, replaced sle-12-sp3 settings with sle-15. Some of the repos do not exist yet, e.g. aytest_repo (http://download.suse.de/ibs/Devel:/YaST:/SLE-15/SLE_15/)
I believe second alternative fits best, as I expect that most of test suites will need tuning. Also, we will have list of jobs to be moved to sle 15 functional group, meaning chances that we miss something are low even without script. Once job group has test suites, all changes that are applied to sle 12 sp3 have to be verified with sle 15.
Please, find suggested json in the attachment.

Actions #5

Updated by riafarov over 6 years ago

  • Assignee set to riafarov
Actions #6

Updated by riafarov over 6 years ago

  • File deleted (sle15_test_development.json)
Actions #7

Updated by riafarov over 6 years ago

Attached json file was used to create job templates.

Actions #8

Updated by riafarov over 6 years ago

  • Description updated (diff)
Actions #9

Updated by okurz over 6 years ago

  • Description updated (diff)

Discussed with PO maritawerner and TL sebchlad: As full proxy-SCC enablement will take some more time we already started now to copy over the full SLE functional test matrix to its according job group to check which scenarios we can already fix, e.g. when they are relying only on the installation workflow. ~1 person will care about the review and fixing of existing failures as well as workarounds to provide prerequisities for other QA groups and other job groups than "Functional" within openQA based on available media which are not intended to be shipped to customers, e.g. the "Server"-medium as well as the module media. Either LeanOS can be extended with these media (and according repositories) or we have to revert to the full Server-DVD and add media to that to provide prerequisities for further testing. I estimated this to take about 2-4 weeks to have a complete coverage of all what we had for SLE 12 SP3. Within 2-10 days we should be able to provide an image corresponding to "create_hdd_minimal_base+sdk" which should enable further testing in other job groups based on that, e.g. kernel testing based on LTP as a first step. Other members of the Functional team will focus on existing tasks be it fixing known issues, enhancement to existing tests or new tests based on already existing requests. If necessary the ressource application can be adjusted in both directions, e.g. more or less effort on SLE15 workaround implementation. Other teams should not be blocked as there are still enough tasks to be done, e.g. for stabilization or openSUSE related.

Actions #11

Updated by riafarov over 6 years ago

  • Related to action #23402: [sles][functional] Adjust create_hdd_gnome test suite for sle15 added
Actions #12

Updated by riafarov over 6 years ago

  • Due date set to 2017-08-24

due to changes in a related task

Actions #13

Updated by riafarov over 6 years ago

  • Related to action #23736: [sle][functional] Adjust autoinst profiles for sle 15 added
Actions #14

Updated by riafarov over 6 years ago

  • Related to action #24704: [sles][functional] test fails in partitioning - Adapt the test to handle the new GPT as default. Boot partition should be bigger. added
Actions #15

Updated by SLindoMansilla over 6 years ago

  • Related to action #24706: [sles][functional] test fails in first_boot - Adapt test to workaround / accept behavior to continue with console tests added
Actions #16

Updated by riafarov over 6 years ago

  • Related to action #25158: [sles][opensuse][functional]test fails in yast2_apparmor added
Actions #17

Updated by okurz over 6 years ago

  • Due date set to 2017-09-27

due to changes in a related task

Actions #18

Updated by okurz over 6 years ago

Actions #19

Updated by okurz over 6 years ago

  • Target version changed from Milestone 10 to Milestone 11

discussed status in M10 review, blocked by subtasks

Actions #20

Updated by riafarov over 6 years ago

  • Subject changed from [sle][functional]sle 15 setup to [sle][functional][epic] sle 15 setup
Actions #21

Updated by sebchlad over 6 years ago

@okurz: I do not see the reasons for having this epic as of now. Reason: we might add all other 'functional' tasks as related or subtasks.
Let us discuss this f2f

Actions #22

Updated by okurz over 6 years ago

  • Assignee changed from riafarov to okurz

We missed the chance to talk f2f. And yes, we could just query for "[sle][functional][sle15]" instead but this ticket also shows our initial plan of what we wanted to do and we have subtickets now which were in most cases observations on what needs to be adapted for sle15 from the start already. IMHO it's good to have this ticket to show what was already known that we need to adapt for sle15 but did not yet accomplish. I will check that next week then.

Actions #23

Updated by okurz over 6 years ago

  • Description updated (diff)

updated open tasks.

The subtickets #23452, #23454 should be done first before we can close here.

Actions #24

Updated by okurz over 6 years ago

  • Description updated (diff)
  • Status changed from In Progress to Resolved

Reviewed the current test coverage again. Seems like we have no scenario still there which does not seem to make any sense at all. Updated subtickets and removed the connection for some siblings. All tasks in the description except for the optional one considered done.

Actions #25

Updated by szarate over 3 years ago

  • Tracker changed from action to coordination
Actions

Also available in: Atom PDF