coordination #20580
closed[sle][functional][epic] sle 15 setup
100%
Description
goal¶
New job group Functional which should be filled with a useful test coverage.
current state¶
riafarov and okurz have migrated all test suites from "Functional: Server" job group to "Functional" sub group of SLE 15: https://openqa.suse.de/group_overview/110
We also have triggered all test suites for x86_64 architecture in order to get an overview of common issues. After identifying most common problems and fixing them for this job group, we may migrate other job groups in similar way.
tasks¶
- DONE: dumped templates for functional job group, adjusted for sle15, load
- DONE: temporarily adjusted sync and trigger cron job to run daily only, not every 5 minutes as most new builds won't bring changes to actually fix job failures
- DONE: incorporate review process
- DONE: focus on former
create_hdd_minimal_base+sdk
scenario as a major dependency for many other job groups, e.g.create_hdd_development
(clone existing sle15 job, add repos from dist.suse.de with ADDON_URL or similar, etc.) - DONE: fix installation only test jobs
- DONE: fix jobs previously relying on server packages, addons, modules by implementing a workaround based on custom repos and isos
- DONE: crosscheck other job groups, e.g. "Regression: Autoyast"
- DONE: delete the scenarios that do not apply anymore or rename and adjust accordingly
- optional: implement a script based solution to compare SLE15 coverage with SLE12SP3 covering multiple job groups so that we can be sure we did not forget any important scenario
further details¶
okurz created new job group Functional which should be filled with a useful test coverage. To make sure we don't forget anything I would
1) dump_templates
the existing sle 12 sp3 functional job group and load_templates that into the new group and then sort out the useless parts and change the scenarios that need to be adapted
Alternatives are
2) same as above but move everything into the corresponding "test development" job group of SLE 15: https://openqa.suse.de/group_overview/110
and then selectively move over jobs.
3) Pick scenario by scenario whatever is considered useful and working. Challenge: We also have triggered all probably need a script to compare the existing test suites for x86_64 architecture in order to get an overview of common issues. After identifying most common problems coverage with the current coverage, evaluate the difference and fixing them for this job group, make sure we may migrate other job groups in similar way.
Attached json file was used to create job templates.
Files
Updated by SLindoMansilla over 7 years ago
- Related to action #20582: [sle][functional] 'if check_var('VERSION, 15)" should be changed to "sle_version_at_least('15')" added
Updated by SLindoMansilla over 7 years ago
- Related to action #20630: [sles][functional] test fails in start_install - Create a workaround for missing checksums and unsigned files and perform installation added
Updated by riafarov over 7 years ago
- File sle15_test_development.json added
I've dumped sle 12 sp3 functional job group, replaced sle-12-sp3 settings with sle-15. Some of the repos do not exist yet, e.g. aytest_repo (http://download.suse.de/ibs/Devel:/YaST:/SLE-15/SLE_15/)
I believe second alternative fits best, as I expect that most of test suites will need tuning. Also, we will have list of jobs to be moved to sle 15 functional group, meaning chances that we miss something are low even without script. Once job group has test suites, all changes that are applied to sle 12 sp3 have to be verified with sle 15.
Please, find suggested json in the attachment.
Updated by riafarov over 7 years ago
- File sle15.json sle15.json added
Attached json file was used to create job templates.
Updated by okurz over 7 years ago
- Description updated (diff)
Discussed with PO maritawerner and TL sebchlad: As full proxy-SCC enablement will take some more time we already started now to copy over the full SLE functional test matrix to its according job group to check which scenarios we can already fix, e.g. when they are relying only on the installation workflow. ~1 person will care about the review and fixing of existing failures as well as workarounds to provide prerequisities for other QA groups and other job groups than "Functional" within openQA based on available media which are not intended to be shipped to customers, e.g. the "Server"-medium as well as the module media. Either LeanOS can be extended with these media (and according repositories) or we have to revert to the full Server-DVD and add media to that to provide prerequisities for further testing. I estimated this to take about 2-4 weeks to have a complete coverage of all what we had for SLE 12 SP3. Within 2-10 days we should be able to provide an image corresponding to "create_hdd_minimal_base+sdk" which should enable further testing in other job groups based on that, e.g. kernel testing based on LTP as a first step. Other members of the Functional team will focus on existing tasks be it fixing known issues, enhancement to existing tests or new tests based on already existing requests. If necessary the ressource application can be adjusted in both directions, e.g. more or less effort on SLE15 workaround implementation. Other teams should not be blocked as there are still enough tasks to be done, e.g. for stabilization or openSUSE related.
Updated by riafarov over 7 years ago
- Related to action #23402: [sles][functional] Adjust create_hdd_gnome test suite for sle15 added
Updated by riafarov over 7 years ago
- Related to action #23736: [sle][functional] Adjust autoinst profiles for sle 15 added
Updated by riafarov over 7 years ago
- Related to action #24704: [sles][functional] test fails in partitioning - Adapt the test to handle the new GPT as default. Boot partition should be bigger. added
Updated by SLindoMansilla over 7 years ago
- Related to action #24706: [sles][functional] test fails in first_boot - Adapt test to workaround / accept behavior to continue with console tests added
Updated by riafarov over 7 years ago
- Related to action #25158: [sles][opensuse][functional]test fails in yast2_apparmor added
Updated by okurz over 7 years ago
- Related to coordination #23886: [sle][functional][desktop][sle15][epic] initial setup added
Updated by okurz over 7 years ago
- Target version changed from Milestone 10 to Milestone 11
discussed status in M10 review, blocked by subtasks
Updated by riafarov over 7 years ago
- Subject changed from [sle][functional]sle 15 setup to [sle][functional][epic] sle 15 setup
Updated by sebchlad over 7 years ago
@okurz: I do not see the reasons for having this epic as of now. Reason: we might add all other 'functional' tasks as related or subtasks.
Let us discuss this f2f
Updated by okurz over 7 years ago
- Assignee changed from riafarov to okurz
We missed the chance to talk f2f. And yes, we could just query for "[sle][functional][sle15]" instead but this ticket also shows our initial plan of what we wanted to do and we have subtickets now which were in most cases observations on what needs to be adapted for sle15 from the start already. IMHO it's good to have this ticket to show what was already known that we need to adapt for sle15 but did not yet accomplish. I will check that next week then.
Updated by okurz over 7 years ago
- Description updated (diff)
- Status changed from In Progress to Resolved
Reviewed the current test coverage again. Seems like we have no scenario still there which does not seem to make any sense at all. Updated subtickets and removed the connection for some siblings. All tasks in the description except for the optional one considered done.
Updated by szarate over 4 years ago
- Tracker changed from action to coordination
Updated by szarate over 4 years ago
See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html