coordination #63280
closed[functional][y][epic] Reduce number of test suites used for YaST job group
Added by riafarov almost 5 years ago. Updated about 4 years ago.
100%
Description
Motivation¶
We want to reduce number of the test suites and tools team has implemented scripts to detect such scenarios.
https://gitlab.suse.de/qsf-y/qa-sle-functional-y/merge_requests/131 can be used as a base, as some scenarios are already identified.
Idea is to use Test Suites and define there variable for the environment and other things in yaml schedule or in the test data structure.
Files
data_match_75 (112 KB) data_match_75 | matching_pairs208_above_75 | ybonatakis, 2020-03-25 07:43 | |
data_groups (2.85 KB) data_groups | effort_to_group_tsuites | ybonatakis, 2020-03-25 20:09 |
Updated by JERiveraMoya almost 5 years ago
- Subject changed from [functional][y] Reduce number of test suites used for YaST job group to [functional][y][Timebox: 16h] Reduce number of test suites used for YaST job group
- Status changed from New to Workable
Updated by JERiveraMoya almost 5 years ago
- Due date changed from 2020-03-10 to 2020-03-24
no capacity
Updated by ybonatakis almost 5 years ago
- Status changed from Workable to In Progress
- Assignee set to ybonatakis
Updated by ybonatakis almost 5 years ago
So the whole idea here is you can now define settings for jobs directly in the YAML, without using a testsuite.
this would reduce the number of testsuites we have in the database.
The candidates testsuites are those that are not represented in other other jobgroups.
Tina Muller has created a script to help automate this but this will change also those which should stay intact because "they might be used elsewhere in the future" as she told me.
I tried to run the script to get some idea around it. the script requires to install YAML::LibYAML::API::XS and colordiff package on the workstation. this will let the script do its thing. Although later in the execution you might see some errors about the location of the modules (OpenQA/JobTemplates.pm).
Can't locate OpenQA/JobTemplates.pm in @INC (you may need to install the OpenQA::JobTemplates module) (@INC contains: /usr/bin/../lib /usr/lib/perl5/site_perl/5.30.1/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.30.1 /usr/lib/perl5/vendor_perl/5.30.1/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.30.1 /usr/lib/perl5/5.30.1/x86_64-linux-thread-multi /usr/lib/perl5/5.30.1 > /usr/lib/perl5/site_perl) at /usr/bin/openqa-validate-yaml line 29.
BEGIN failed--compilation aborted at /usr/bin/openqa-validate-yaml line 29.
the lib issue will be fixed in the next openQA version https://github.com/os-autoinst/openQA/pull/2829. for now a workaround is to use export PERL5LIB=/usr/share/openqa/lib
Updated by ybonatakis almost 5 years ago
So what we have to do is to find all the tests' settings in the test suites for each one in the YAST group that are not used in other jobgroups. Then for each of them we need to copy the settings in our YAST yaml(JobGroups/SLE_15_YaST.yaml).
For this we can use the Tina's script.
The question is if we have to do this work for JobGroups/SLE_12_YaST.yaml too(?)
Updated by ybonatakis almost 5 years ago
i tried to create a script to compare the yast jobgroup with the rest groups. But i bumped into a small problem with the openqa-python-client(we filed a ticket after discussion on irc[0]). i will continue on Monday trying this or another way to provide a list of those tests that we can reduce
Updated by ybonatakis almost 5 years ago
- Status changed from In Progress to Feedback
Comparing with jobgroups in OSD:
i didnt found any that is used by others.
Comparing with TW the following are used by both which if i got it right we cant remove them from the test suite:
[ RAID switch_keyboard_textmode gpt yast2_gui mediacheck crypt_no_lvm remote_ssh_controller RAID0 select_disk installer_extended yast2_nfs_v4_server lvm_thin_provisioning autoyast_multi_btrfs lvm+RAID1 autoyast_gnome yast_no_self_update RAID1 btrfs+warnings RAID10 clone_system yast2_nfs_v3_client yast2_nfs_v4_client cryptlvm remote_vnc_controller nvme autoyast_reinstal autoyast_mini lvm-encrypt-separate-boot modify_existing_partition boot_linuxrc lvm yast2_nfs_v3_server xfs switch_keyboard_gnome yast2_ncurses ]
So the following can be used from the Tina's script for now:
'RAID5',
'RAID6',
'USBinstall',
'addon-module-ftp',
'addon-module-http',
'allmodules+allpatterns',
'autologin_yast',
'autoyast-supportserver',
'autoyast_bcache',
'autoyast_btrfs',
'autoyast_bug-872532_ix64ph1069',
'autoyast_bug-876411_btrfs_h5_autoinst',
'autoyast_bug-877438_ix64ph1029',
'autoyast_bug-879147_autoinst',
'autoyast_bug-887126_autoinst',
'autoyast_bug-887653_autoinst_jy-snapshot',
'autoyast_disk_as_md_member',
'autoyast_disk_as_pv',
'autoyast_error_dialog',
'autoyast_eula_full_medium',
'autoyast_ext4',
'autoyast_invalid_default_target',
'autoyast_keyboard_layout',
'autoyast_lvm',
'autoyast_multipath',
'autoyast_nfs_share',
'autoyast_non_existing_graphical_target',
'autoyast_reinstall',
'autoyast_resize_luks2',
'autoyast_tftp',
'autoyast_y2_firstboot',
'autoyast_zkvm_sles_product_reg',
'btrfs_libstorage-ng',
'cryptlvm+activate_existing',
'cryptlvm+activate_existing+force_recompute',
'cryptlvm+activate_existing+import_users',
'cryptlvm+cancel_existing',
'detect_yast2_failures',
'dud_development_tools',
'ext4_yast',
'gnome_http',
'gnome_self_signed_https',
'gnome_smb',
'installer_extended_textmode',
'installer_extended_x11',
'iscsi_client_normal_auth_backstore_fileio',
'iscsi_client_normal_auth_backstore_hdd',
'iscsi_client_normal_auth_backstore_lvm',
'iscsi_ibft',
'iscsi_server_normal_auth_backstore_fileio',
'iscsi_server_normal_auth_backstore_hdd',
'iscsi_server_normal_auth_backstore_lvm',
'lvm+cancel_existing_cryptlvm',
'lvm+resize_root',
'lvm-full-encrypt',
'minimal+base_yast',
'minimal+role_minimal',
'msdos',
'multipath',
'nis_client',
'nis_server',
'offline_install+skip_registration',
'releasenotes_origin+unregistered',
'remote_ssh_target_ftp',
'remote_vnc_target_nfs',
'repo_inst',
'skip_registration',
'sles+sdk+proxy_SCC_via_YaST',
'yast2_cmd',
'yast2_firstboot',
'yast2_ui_devel',
'yast_hostname',
'yast_hostname+dhcp_hostname',
'yast_hostname+linuxrc_hostname',
'yast_self_update',
'zfcp'}
And this list maybe it should be compared with other groups in o3 but
I think it would be nice to have a script or something to find them automatically in the future.
Updated by riafarov almost 5 years ago
- Subject changed from [functional][y][Timebox: 16h] Reduce number of test suites used for YaST job group to [functional][y][timeboxed:10h] Reduce number of test suites used for YaST job group
- Due date changed from 2020-03-24 to 2020-04-07
- Status changed from Feedback to Workable
Updated by ybonatakis almost 5 years ago
- Status changed from Workable to In Progress
Updated by ybonatakis almost 5 years ago
- File data_match_75 data_match_75 added
I have generated this list (see attachment) which i hope it can help to do the next step.
In the file, there are 208 combinations of pairs. I "invented" a matching rate for each one.
The calculations made by comparing and matching the key:values of each pair in an additional combination of the remaining matched keys(but not values)
Obviously, the findings are not accurate, in sense that this list provides some trivial investigation but it cannot be completed.
For example: the pair (remote_ssh_target_ftp , remote_vnc_target_nfs) have 0.82 match rate because they have [INSTALLONLY, NETBOOT, NICTYPE, OVS_DEBUG, USE_SUPPORT_SERVER, WORKER_CLASS]. But as i understand, this is a case that we cannot reduce.
In additional the findings can be reduced further because the number of the the combinations is with a pair of two. As an example: RAID pairs can be combined all in one.
Updated by ybonatakis almost 5 years ago
- File data_groups data_groups added
- Status changed from In Progress to Feedback
I have attached another file which i tried to make groups of test suites, based in the previous list) that contains a set of common variables.
This turned to be harder than i thought so I spent sometime to collect some data to categorize the test_suites per the_most_used variables.
So for the first 10 most common used variables are:
[('1', 202),
('gnome', 48),
('tap', 38),
('textmode', 31),
('SLES-%VERSION%-%ARCH%-Build%BUILD%@%MACHINE%-gnome.qcow2', 15),
('0', 13),
('scsi-hd', 12),
('create_hdd_gnome', 11),
('14400', 10),
('SLES-%VERSION%-%ARCH%-Build%BUILD%@%MACHINE%-gnome-uefi-vars.qcow2', 8)]
where from them the No1, which shows the value that enables a variable, breaks down to (first 10 elements aagain):
[('INSTALLONLY', 59),
('BOOT_HDD_IMAGE', 22),
('AUTOYAST_PREPARE_PROFILE', 13),
('OVS_DEBUG', 10),
('LVM', 9),
('ENCRYPT', 8),
('EXIT_AFTER_START_INSTALL', 7),
('NETBOOT', 6),
('USE_SUPPORT_SERVER', 4),
('ENCRYPT_ACTIVATE_EXISTING', 3)]
I think from here we can think of a way to group tests more efficient. For example all the test with
Updated by riafarov over 4 years ago
After quick look on the output it's hard for me to understand how to apply it and which test suites can be reduced to one or for which we can move all the settings in the JobGroup, so I guess we will need to spend more time on research.
Also, do you have a script which checks if test suite is enabled in any other job group? As this is very important question we need to answer before acting on any of the test suites.
Updated by ybonatakis over 4 years ago
regarding the first, the answer is yes. i spent the timeboxes towards different directions. If I had more time i would have come up with a combination of the arrays above.
About the script, I made a python script "for all", just to easy some ideas during the investigation. So the answer is something between "there is something" and "need some improvements before get public or be used". I could work on it and provide something that have a more proper output
Updated by riafarov over 4 years ago
- Assignee changed from ybonatakis to riafarov
Updated by riafarov over 4 years ago
- Subject changed from [functional][y][timeboxed:10h] Reduce number of test suites used for YaST job group to [functional][y][epic] Reduce number of test suites used for YaST job group
- Due date deleted (
2020-04-07) - Target version changed from Milestone 33 to future
Converting this to epic as we will have multiple subtasks for it.
Updated by riafarov over 4 years ago
- Status changed from Feedback to In Progress
Updated by riafarov over 4 years ago
- Due date changed from 2020-04-21 to 2020-05-05
due to changes in a related task
Updated by riafarov over 4 years ago
- Due date changed from 2020-05-05 to 2020-05-19
due to changes in a related task: #65393
Updated by riafarov over 4 years ago
- Due date changed from 2020-05-19 to 2020-06-23
due to changes in a related task: #66964
Updated by riafarov over 4 years ago
- Due date changed from 2020-06-23 to 2020-06-30
due to changes in a related task: #66964
Updated by riafarov over 4 years ago
- Due date changed from 2020-06-30 to 2020-07-28
due to changes in a related task: #66967
Updated by riafarov over 4 years ago
- Due date changed from 2020-07-28 to 2020-08-11
due to changes in a related task: #66967
Updated by szarate about 4 years ago
- Tracker changed from action to coordination
Updated by szarate about 4 years ago
See for the reason of tracker change: http://mailman.suse.de/mailman/private/qa-sle/2020-October/002722.html