Project

General

Profile

coordination #55142

[PjM][saga][migration] Extending Migration testing into functional domains and teams

Added by coolgw about 2 years ago. Updated 12 months ago.

Status:
New
Priority:
Normal
Assignee:
Category:
New test
Target version:
SUSE QA - Milestone 35+
Start date:
2019-08-06
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

[Gao Wei]:Since i delete 39812 by my mistake, we can using this to continue track.

While QA SLE department has a migration team located in Beijing (part of Sunny's line team), there is an opportunity to collaborate better and thus enhance/extend migration testing.

Why do we talk about this? During SLE15 SP0 migration was problematic area, as:

  • it was defined very late; at first we were acting on poorly communicated and not clear requirements;
  • during SLE15 SP0 some aspects (requirements) of migration were changing which was making automation nearly impossible;
  • finally our testing effort in limited migration team are focused on migration itself including testing RMT/SMT etc.

Our testing of migration is about testing matrix of various migration paths where we check if this process can be accomplished successfully.

Discussed idea with Sunny, Thorsten Kukuk, Marita and number of other people is to now ask functional teams to introduce some migration testing for some "services".
As Thorsten described it: check if certain services can survive migration.

Initial idea:

  • migration functional team is working on testing migration and maintaining all migration openQA modules;
  • each of the functional team should then consider testing some services in the way that they set them up in older system, call migration openQA module and check if configured service survives the migration.

During the talks with Thorsten following services were named as good to test:

  • apache
  • bind
  • net-snmp
  • dhcp server
  • nfs-server
  • rpcbind
  • radvd
  • cron (including all cronjobs, e.g. the in /etc/crontab)
  • apparmor
  • autofs
  • cups
  • kdump
  • ntp/chrony
  • postfix
  • firewall
  • vsftpd

Please note: this is not about asking you to do even more work within the same time. Product Owners should work with Project Manager on priorities, so perhaps some testing could be dropped in order to add some of these migration testing.

History

#1 Updated by coolgw about 2 years ago

History notes:

Wed 12/5/2018 11:56 PM
[openSUSE Tracker]
Issue #39812 has been updated by sunyan.
Assignee changed from sunyan to okurz
Sebastian, Oliver,
I had a talk with Oliver about this ticket this afternoon, we are on the same wavelength at the moment.
As I proposed the Implementation steps as below, my team is ready for step 1 and 4, and we are waiting for function team to do step 2 and 5, so that we can continue doing step 3 and 6. Based on the discussion, I move assignee to Oliver.

----------------Implementation steps ----------------------- 1. Install OS (e.g. SLE12 SP3) -- Implement by Sunny's team 2. Start important services and do some configuration for these service if it is necessary (Encapsulation module or automation script) -- Implement by Sebastian's team 3. Call result of step 2 to generate image -- Implement by Sunny's team 4. Migration (e.g. from SLE12 SP3 to SLE15 GA) -- Implement by Sunny's team 5. Check important services' status and configuration after migration (Encapsulation module or automation script) -- Implement by Sebastian's team 6. Call result of step 5 after migration -- Implement by Sunny's team

Tue 4/30/2019 3:42 PM
[openSUSE Tracker]
Issue #39812 has been updated by maritawerner.
Related to action #50576: [sle][migration][SLE12 SP5] test fails in http_srv - server function test conflict with service test added

Tue 5/7/2019 3:19 AM
[openSUSE Tracker]
Issue #39812 has been updated by sebchlad.
Certain testing for HPC cluster migration will be added. For now slurm migration.

Fri 5/17/2019 6:59 PM
[openSUSE Tracker]
Issue #39812 has been updated by mgriessmeier.
Target version changed from Milestone 25+ to Milestone 25

Wed 5/22/2019 9:30 PM
[openSUSE Tracker]
Issue #39812 has been updated by okurz.
Assignee changed from okurz to mgriessmeier
mgriessmeier not sure why you changed from M25+ to just M25. Personally I doubt we can finish this within the current milestone but I will leave the decision how to handle the ticket in particular to you. There are subtasks to follow-on with so maybe just simple tracking here for now.

Wed 5/22/2019 9:30 PM
[openSUSE Tracker]
Issue #39812 has been updated by okurz.
Assignee changed from okurz to mgriessmeier
mgriessmeier not sure why you changed from M25+ to just M25. Personally I doubt we can finish this within the current milestone but I will leave the decision how to handle the ticket in particular to you. There are subtasks to follow-on with so maybe just simple tracking here for now.

Wed 6/5/2019 4:34 PM
[openSUSE Tracker]
Issue #39812 has been updated by sunyan.
By talking to Matthias, Rodion, Jose and Marita. We plan to start from a simple example (apache service)
======================+===============================+==================
check point | Before Migration | After Migration
----------------------+-------------------------------+-----------------
----------------------+-------------------------------+--
apache_init | V | V
apache_install | V |
apache_conf_enable | V |
apache_start_service | V |
apache_check_service | V | V
apache_check_conf | V | V
apache_check_function | V | V
======================+===============================+====================

main.pm (load_patching_tests)
|
install_service.pm (use function of install_services & check_services
in service_check.pm)
|
service_check.pm (call function of "above check point (before
migration)" in http_srv.pm)
|
http_srv.pm (need to encapsulation based on each check point)


main_common.pm (load_consoletests)
|
check_upgraded_service.pm (use function of check_services
in service_check.pm)
|
service_check.pm (call function of "above check point (after migration)"
in http_srv.pm)
|
http_srv.pm (need to encapsulation based on each check point)


[Benefit]

  1. Stay in current process (here is no change for calling http_srv.pm in fresh installation process)

2. Need to encapsulate function of check point in http_srv.pm, no additional logical judgment in http_srv.pm.

[Action item]

  1. Sunny team will submit PR of http_srv.pm for wider input and discussion 2. Marita will check with Thorsten which services in this list might have higher priority than others.
  2. If there is any services (in Thorsten's list or not) which don't fit for above work flow proposal, or too complicated (different preconditions in before/after migration setups which we should split), we will discuss them case by case. It will be helpful that Matthias or Jose provide specific example, so that we can review and discuss it together.

Wed 6/5/2019 4:26 PM
[openSUSE Tracker]
Issue #39812 has been updated by sunyan.
The service list we got from Thorsten are divided by subgroup as below
Function-U:

  • apache
  • bind
  • net-snmp
  • nfs-server
  • rpcbind
  • radvd
  • cron (including all cronjobs, e.g. the in /etc/crontab)
  • apparmor
  • autofs
  • cups
  • ntp/chrony
  • postfix
  • firewall
  • vsftpd Kernel/Networking:
  • kdump
  • dhcp server

Thu 6/6/2019 6:56 PM
[openSUSE Tracker]
Issue #39812 has been updated by jlausuch.
If I take the example of DHCP, we don't have specific test cases to exercise different options in DHCP service. We use it though implicitly in some test cases (e.g. Wicked).
So, for the first part (basic sanity check after migration), I agree we can focus on basic functionality (service up with good configuration after migration) and start thinking about adding advanced specific tests for each area/service which is not covered. After SLE12-SP5 we could trigger the "more complex" tests after migration, not only the basic check.

Thu 6/6/2019 5:22 PM
[openSUSE Tracker]
Issue #39812 has been updated by maritawerner.
In the PO coordination meeting yesterday the question came up how detailed the Testcases should be. Do we need to test only some basic functionalities of the services or do we need some in-depth testing.
Thorsten told me today that for the beginning he is fine to start with testing basic functionality of the services before and after migration. Since we do not have a lot of experience her he thinks we should also wait for customer feedback and bugs reported to see what realy needs more testing. He also expressed that a lot of the services are not deeply tested during installation and that we might have to work on that first. So he and me agreed that for SLE 12 SP5 we are fine with basic tests, later for SLE 15 SP1 or even SLE 15 SP2 we will rediscuss if deeper testing is needed. And that has to be specified per service.

Thu 6/6/2019 10:20 AM
[openSUSE Tracker]
Issue #39812 has been updated by hjluo.
Due date set to 06/06/2019
due to changes in a related task
Wed 6/5/2019 5:1[openSUSE Tracker]
Issue #39812 has been updated by coolgw.
Just FYI service_check.pm is used for judge which version will check service etc..
path is /lib/service_check.pm
1 PM

Mon 7/1/2019 4:04 PM
[openSUSE Tracker]
Issue #39812 has been updated by mgriessmeier.
Target version changed from Milestone 25 to Milestone 26

Tue 7/16/2019 5:25 PM
[openSUSE Tracker]
Issue #39812 has been updated by hjluo.
File service_check_list.txt added
I've checked all the services to see if they have install, enable, start, check_service{active}, check_function in the tests/console/moduel.pm for before/after migration, please check the attached result file and correct me if I missed something.

#2 Updated by coolgw about 2 years ago

I have upload the investigation result for service module.

#4 Updated by mgriessmeier about 2 years ago

  • Target version changed from Milestone 26 to Milestone 27

#5 Updated by mgriessmeier about 2 years ago

  • Target version changed from Milestone 27 to Milestone 28

#6 Updated by okurz almost 2 years ago

  • Category set to New test

#7 Updated by mgriessmeier over 1 year ago

  • Target version changed from Milestone 28 to Milestone 31

#8 Updated by mgriessmeier over 1 year ago

  • Priority changed from High to Normal
  • Target version changed from Milestone 31 to Milestone 35+

not considering high

#9 Updated by szarate 12 months ago

  • Tracker changed from action to coordination
  • Status changed from Blocked to New

Also available in: Atom PDF