Project

General

Profile

Actions

coordination #63280

closed

[functional][y][epic] Reduce number of test suites used for YaST job group

Added by riafarov about 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Refactor/Code Improvements
Target version:
Start date:
2020-04-07
Due date:
2020-08-11
% Done:

100%

Estimated time:
(Total: 29.00 h)
Difficulty:

Description

Motivation

We want to reduce number of the test suites and tools team has implemented scripts to detect such scenarios.

https://gitlab.suse.de/qsf-y/qa-sle-functional-y/merge_requests/131 can be used as a base, as some scenarios are already identified.

Idea is to use Test Suites and define there variable for the environment and other things in yaml schedule or in the test data structure.


Files

data_match_75 (112 KB) data_match_75 matching_pairs208_above_75 ybonatakis, 2020-03-25 07:43
data_groups (2.85 KB) data_groups effort_to_group_tsuites ybonatakis, 2020-03-25 20:09

Subtasks 4 (0 open4 closed)

action #65393: [functional][y] Develop script to figure out where test suite is scheduledResolvedriafarov2020-04-072020-05-19

Actions
action #65927: [functional][y] Identify list of the scenarios where we can remove test suitesResolvedriafarov2020-04-212020-05-19

Actions
action #66964: [functional][y] Move test suites where we already have yaml schedule to the job groupResolvedJERiveraMoya2020-05-182020-06-30

Actions
action #66967: [functional][y] Move more scenarios to job group from openQA databaseResolvedsyrianidou_sofia2020-05-182020-08-11

Actions
Actions #1

Updated by JERiveraMoya about 4 years ago

  • Subject changed from [functional][y] Reduce number of test suites used for YaST job group to [functional][y][Timebox: 16h] Reduce number of test suites used for YaST job group
  • Status changed from New to Workable
Actions #2

Updated by JERiveraMoya about 4 years ago

  • Due date changed from 2020-03-10 to 2020-03-24

no capacity

Actions #3

Updated by ybonatakis about 4 years ago

  • Status changed from Workable to In Progress
  • Assignee set to ybonatakis
Actions #4

Updated by ybonatakis about 4 years ago

So the whole idea here is you can now define settings for jobs directly in the YAML, without using a testsuite.
this would reduce the number of testsuites we have in the database.
The candidates testsuites are those that are not represented in other other jobgroups.

Tina Muller has created a script to help automate this but this will change also those which should stay intact because "they might be used elsewhere in the future" as she told me.

I tried to run the script to get some idea around it. the script requires to install YAML::LibYAML::API::XS and colordiff package on the workstation. this will let the script do its thing. Although later in the execution you might see some errors about the location of the modules (OpenQA/JobTemplates.pm).

Can't locate OpenQA/JobTemplates.pm in @INC (you may need to install the OpenQA::JobTemplates module) (@INC contains: /usr/bin/../lib /usr/lib/perl5/site_perl/5.30.1/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.30.1 /usr/lib/perl5/vendor_perl/5.30.1/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.30.1 /usr/lib/perl5/5.30.1/x86_64-linux-thread-multi /usr/lib/perl5/5.30.1 > /usr/lib/perl5/site_perl) at /usr/bin/openqa-validate-yaml line 29.
BEGIN failed--compilation aborted at /usr/bin/openqa-validate-yaml line 29.

the lib issue will be fixed in the next openQA version https://github.com/os-autoinst/openQA/pull/2829. for now a workaround is to use export PERL5LIB=/usr/share/openqa/lib

Actions #5

Updated by ybonatakis about 4 years ago

So what we have to do is to find all the tests' settings in the test suites for each one in the YAST group that are not used in other jobgroups. Then for each of them we need to copy the settings in our YAST yaml(JobGroups/SLE_15_YaST.yaml).

For this we can use the Tina's script.

The question is if we have to do this work for JobGroups/SLE_12_YaST.yaml too(?)

Actions #6

Updated by ybonatakis about 4 years ago

i tried to create a script to compare the yast jobgroup with the rest groups. But i bumped into a small problem with the openqa-python-client(we filed a ticket after discussion on irc[0]). i will continue on Monday trying this or another way to provide a list of those tests that we can reduce

[0] https://progress.opensuse.org/issues/64496

Actions #7

Updated by ybonatakis about 4 years ago

  • Status changed from In Progress to Feedback

Comparing with jobgroups in OSD:
i didnt found any that is used by others.

Comparing with TW the following are used by both which if i got it right we cant remove them from the test suite:
[ RAID switch_keyboard_textmode gpt yast2_gui mediacheck crypt_no_lvm remote_ssh_controller RAID0 select_disk installer_extended yast2_nfs_v4_server lvm_thin_provisioning autoyast_multi_btrfs lvm+RAID1 autoyast_gnome yast_no_self_update RAID1 btrfs+warnings RAID10 clone_system yast2_nfs_v3_client yast2_nfs_v4_client cryptlvm remote_vnc_controller nvme autoyast_reinstal autoyast_mini lvm-encrypt-separate-boot modify_existing_partition boot_linuxrc lvm yast2_nfs_v3_server xfs switch_keyboard_gnome yast2_ncurses ]

So the following can be used from the Tina's script for now:
'RAID5',
'RAID6',
'USBinstall',
'addon-module-ftp',
'addon-module-http',
'allmodules+allpatterns',
'autologin_yast',
'autoyast-supportserver',
'autoyast_bcache',
'autoyast_btrfs',
'autoyast_bug-872532_ix64ph1069',
'autoyast_bug-876411_btrfs_h5_autoinst',
'autoyast_bug-877438_ix64ph1029',
'autoyast_bug-879147_autoinst',
'autoyast_bug-887126_autoinst',
'autoyast_bug-887653_autoinst_jy-snapshot',
'autoyast_disk_as_md_member',
'autoyast_disk_as_pv',
'autoyast_error_dialog',
'autoyast_eula_full_medium',
'autoyast_ext4',
'autoyast_invalid_default_target',
'autoyast_keyboard_layout',
'autoyast_lvm',
'autoyast_multipath',
'autoyast_nfs_share',
'autoyast_non_existing_graphical_target',
'autoyast_reinstall',
'autoyast_resize_luks2',
'autoyast_tftp',
'autoyast_y2_firstboot',
'autoyast_zkvm_sles_product_reg',
'btrfs_libstorage-ng',
'cryptlvm+activate_existing',
'cryptlvm+activate_existing+force_recompute',
'cryptlvm+activate_existing+import_users',
'cryptlvm+cancel_existing',
'detect_yast2_failures',
'dud_development_tools',
'ext4_yast',
'gnome_http',
'gnome_self_signed_https',
'gnome_smb',
'installer_extended_textmode',
'installer_extended_x11',
'iscsi_client_normal_auth_backstore_fileio',
'iscsi_client_normal_auth_backstore_hdd',
'iscsi_client_normal_auth_backstore_lvm',
'iscsi_ibft',
'iscsi_server_normal_auth_backstore_fileio',
'iscsi_server_normal_auth_backstore_hdd',
'iscsi_server_normal_auth_backstore_lvm',
'lvm+cancel_existing_cryptlvm',
'lvm+resize_root',
'lvm-full-encrypt',
'minimal+base_yast',
'minimal+role_minimal',
'msdos',
'multipath',
'nis_client',
'nis_server',
'offline_install+skip_registration',
'releasenotes_origin+unregistered',
'remote_ssh_target_ftp',
'remote_vnc_target_nfs',
'repo_inst',
'skip_registration',
'sles+sdk+proxy_SCC_via_YaST',
'yast2_cmd',
'yast2_firstboot',
'yast2_ui_devel',
'yast_hostname',
'yast_hostname+dhcp_hostname',
'yast_hostname+linuxrc_hostname',
'yast_self_update',
'zfcp'}

And this list maybe it should be compared with other groups in o3 but
I think it would be nice to have a script or something to find them automatically in the future.

Actions #8

Updated by riafarov about 4 years ago

  • Subject changed from [functional][y][Timebox: 16h] Reduce number of test suites used for YaST job group to [functional][y][timeboxed:10h] Reduce number of test suites used for YaST job group
  • Due date changed from 2020-03-24 to 2020-04-07
  • Status changed from Feedback to Workable
Actions #9

Updated by ybonatakis about 4 years ago

  • Status changed from Workable to In Progress
Actions #10

Updated by ybonatakis about 4 years ago

I have generated this list (see attachment) which i hope it can help to do the next step.

In the file, there are 208 combinations of pairs. I "invented" a matching rate for each one.

The calculations made by comparing and matching the key:values of each pair in an additional combination of the remaining matched keys(but not values)

Obviously, the findings are not accurate, in sense that this list provides some trivial investigation but it cannot be completed.

For example: the pair (remote_ssh_target_ftp , remote_vnc_target_nfs) have 0.82 match rate because they have [INSTALLONLY, NETBOOT, NICTYPE, OVS_DEBUG, USE_SUPPORT_SERVER, WORKER_CLASS]. But as i understand, this is a case that we cannot reduce.
In additional the findings can be reduced further because the number of the the combinations is with a pair of two. As an example: RAID pairs can be combined all in one.

Actions #11

Updated by ybonatakis about 4 years ago

I have attached another file which i tried to make groups of test suites, based in the previous list) that contains a set of common variables.
This turned to be harder than i thought so I spent sometime to collect some data to categorize the test_suites per the_most_used variables.
So for the first 10 most common used variables are:

[('1', 202),
 ('gnome', 48),
 ('tap', 38),
 ('textmode', 31),
 ('SLES-%VERSION%-%ARCH%-Build%BUILD%@%MACHINE%-gnome.qcow2', 15),
 ('0', 13),
 ('scsi-hd', 12),
 ('create_hdd_gnome', 11),
 ('14400', 10),
 ('SLES-%VERSION%-%ARCH%-Build%BUILD%@%MACHINE%-gnome-uefi-vars.qcow2', 8)]

where from them the No1, which shows the value that enables a variable, breaks down to (first 10 elements aagain):
[('INSTALLONLY', 59),
 ('BOOT_HDD_IMAGE', 22),
 ('AUTOYAST_PREPARE_PROFILE', 13),
 ('OVS_DEBUG', 10),
 ('LVM', 9),
 ('ENCRYPT', 8),
 ('EXIT_AFTER_START_INSTALL', 7),
 ('NETBOOT', 6),
 ('USE_SUPPORT_SERVER', 4),
 ('ENCRYPT_ACTIVATE_EXISTING', 3)]

I think from here we can think of a way to group tests more efficient. For example all the test with

Actions #12

Updated by riafarov almost 4 years ago

After quick look on the output it's hard for me to understand how to apply it and which test suites can be reduced to one or for which we can move all the settings in the JobGroup, so I guess we will need to spend more time on research.

Also, do you have a script which checks if test suite is enabled in any other job group? As this is very important question we need to answer before acting on any of the test suites.

Actions #13

Updated by ybonatakis almost 4 years ago

regarding the first, the answer is yes. i spent the timeboxes towards different directions. If I had more time i would have come up with a combination of the arrays above.

About the script, I made a python script "for all", just to easy some ideas during the investigation. So the answer is something between "there is something" and "need some improvements before get public or be used". I could work on it and provide something that have a more proper output

Actions #14

Updated by riafarov almost 4 years ago

  • Assignee changed from ybonatakis to riafarov
Actions #15

Updated by riafarov almost 4 years ago

  • Subject changed from [functional][y][timeboxed:10h] Reduce number of test suites used for YaST job group to [functional][y][epic] Reduce number of test suites used for YaST job group
  • Due date deleted (2020-04-07)
  • Target version changed from Milestone 33 to future

Converting this to epic as we will have multiple subtasks for it.

Actions #16

Updated by riafarov almost 4 years ago

  • Status changed from Feedback to In Progress
Actions #17

Updated by riafarov almost 4 years ago

  • Due date changed from 2020-04-21 to 2020-05-05

due to changes in a related task

Actions #18

Updated by riafarov almost 4 years ago

  • Due date changed from 2020-05-05 to 2020-05-19

due to changes in a related task: #65393

Actions #19

Updated by riafarov almost 4 years ago

  • Due date changed from 2020-05-19 to 2020-06-23

due to changes in a related task: #66964

Actions #20

Updated by riafarov almost 4 years ago

  • Due date changed from 2020-06-23 to 2020-06-30

due to changes in a related task: #66964

Actions #21

Updated by riafarov over 3 years ago

  • Due date changed from 2020-06-30 to 2020-07-28

due to changes in a related task: #66967

Actions #22

Updated by riafarov over 3 years ago

  • Due date changed from 2020-07-28 to 2020-08-11

due to changes in a related task: #66967

Actions #23

Updated by riafarov over 3 years ago

  • Status changed from In Progress to Resolved

Good job, team!

Actions #24

Updated by szarate over 3 years ago

  • Tracker changed from action to coordination
Actions

Also available in: Atom PDF