QA » openQA Project » openQA Tests » qe-yam

Created openqa-job-groups MR#32

The refactoring was done following these steps:

• CREATE_GIT_BRANCH
• SPLIT_FILE

all steps has been done using this bash script:

#!/usr/bin/env bash
set -e
echo "# issues/153005"

repo_orig="$HOME/GitLab/qam-openqa-yml"
repo_dest="$HOME/GitLab/openqa-job-groups"

function CREATE_GIT_BRANCH() {
  cd $repo_dest
  git checkout -b issues-153005
}

function SPLIT_FILE() {
  mkdir -p $repo_dest/JobGroups/yast_maint_updates
  yq '{"defaults":.defaults,"products":.products} | sort_keys(..)' $repo_orig/JobGroups/yast/maint_updates.yml > $repo_dest/JobGroups/yast_maint_updates/defaults.yaml
  yq '{"aarch64": .scenarios.aarch64 | sort_keys(..) }' $repo_orig/JobGroups/yast/maint_updates.yml > $repo_dest/JobGroups/yast_maint_updates/aarch64.yaml
  yq '{"ppc64le": .scenarios.ppc64le | sort_keys(..) }' $repo_orig/JobGroups/yast/maint_updates.yml > $repo_dest/JobGroups/yast_maint_updates/ppc64le.yaml
  yq '{"s390x":   .scenarios.s390x   | sort_keys(..) }' $repo_orig/JobGroups/yast/maint_updates.yml > $repo_dest/JobGroups/yast_maint_updates/s390x.yaml
  yq '{"x86_64":  .scenarios.x86_64  | sort_keys(..) }' $repo_orig/JobGroups/yast/maint_updates.yml > $repo_dest/JobGroups/yast_maint_updates/x86_64.yaml

  # Adapt $repo_dest/.gitlab-ci.yml
}


CREATE_GIT_BRANCH
SPLIT_FILE

I had have to created also the empty file JobGroups/yast_maint_updates/ppc64le.yaml because the CI check for all ARCHs:
scripts/generate_yaml.py#58

GitLab CI adapted by adding:

variables:
  GROUPID_YAST_MAINT_UPDATES: 421

test_yast_maint_updates:
  stage: test
  before_script:
    - git clone --depth 1 $URL_TEST_REPO
  script:
    - scripts/ci_runner_test.sh yast_maint_updates $GROUPID_YAST_MAINT_UPDATES
  only:
    refs:
      - merge_requests
    changes:
      - JobGroups/*
      - JobGroups/yast_maint_updates/*
      - .gitlab-ci.yml
  retry: 2

deploy_yast_maint_updates:
  stage: deploy
  script:
    - scripts/ci_runner_deploy.sh yast_maint_updates $GROUPID_YAST_MAINT_UPDATES
  only:
    refs:
      - master
    changes:
      - JobGroups/*
      - JobGroups/yast_maint_updates/*
      - .gitlab-ci.yml

The pipeline failure should be related with the previous empty file. Probably we need to modify/adapt the pipeline.

Modified pipeline: added check for test if file exists for each ARCH

archs = ['aarch64', 'ppc64le', 's390x', 'x86_64']
...
for arch in archs:
    yaml_path = os.path.abspath(jobgroup_dir + '/' + arch + '.yaml')
    if Path(yaml_path).is_file():
        with open(yaml_path, 'r') as fp:

Renamed:

• yast_maint_updates to yast_maintenance_updates
• GROUPID_YAST_MAINT_UPDATES to GROUPID_YAST_MAINTENANCE_UPDATES

Added TestSuites folder and related json files.
The refactoring was done following these steps:

• GET_SUITES

all steps has been done using this bash script:

#!/usr/bin/env bash
set -e
echo "# issues/153005"

repo_orig="$HOME/GitLab/qam-openqa-yml"
repo_dest="$HOME/GitLab/openqa-job-groups"

function GET_SUITES() {
  local suites=()
  for arch in $(yq '.scenarios | keys' $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
    echo "ARCH: $arch"
    for product in $(yq ".scenarios.$arch | keys" $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
      echo "  PRODUCTS: $product"
      for suite in $(yq ".scenarios.$arch.$product[] | select(. | type == \"!!str\") | ." $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
        echo "    *SUITE: $suite"
        suites+=( $suite )
      done
      for suite in $(yq ".scenarios.$arch.$product[] | select(type == \"!!map\") | keys" $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
        echo "    SUITE: $suite"
        suites+=( $suite )
      done
    done
  done

  local suites_sorted_uniq=($(echo "${suites[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))

  local description='Testsuite maintained at https://gitlab.suse.de/qe-yam/openqa-job-groups'

  mkdir -p $repo_dest/TestSuites
  for suite in "${suites_sorted_uniq[@]}"; do
    echo " -> $suite"
    [[ -e "$repo_orig/TestSuites/${suite}.json"  ]] && { 
      echo " ---> to move: TestSuites/${suite}.json" 
      jq ". | .TestSuites[0].description = \"${description}\"" "$repo_orig/TestSuites/${suite}.json" > $repo_dest/TestSuites/${suite}.json
    } || echo " *** NOT FOUND *** "
  done

}

GET_SUITES

The files to remove in origin repo qam-openqa-yml should be:

git rm ./TestSuites/autoyast_filesystem_withouthome_yast.json
git rm ./TestSuites/create_hdd_transactional_server.json
git rm ./TestSuites/create_hdd_yast_maintenance_desktop.json
git rm ./TestSuites/create_hdd_yast_maintenance_minimal.json
git rm ./TestSuites/lvm_thin_provisioning.json
git rm ./TestSuites/mau_extratests_yast2_ncurses.json
git rm ./TestSuites/mau-extratests-yast2ui.json
git rm ./TestSuites/mau_extratests_yast_cmd.json
git rm ./TestSuites/mru-iscsi_client_normal_auth_backstore_fileio.json
git rm ./TestSuites/mru-iscsi_client_normal_auth_backstore_hdd.json
git rm ./TestSuites/mru-iscsi_client_normal_auth_backstore_lvm.json
git rm ./TestSuites/mru-iscsi_server_normal_auth_backstore_fileio.json
git rm ./TestSuites/mru-iscsi_server_normal_auth_backstore_hdd.json
git rm ./TestSuites/mru-iscsi_server_normal_auth_backstore_lvm.json
git rm ./TestSuites/ncurses_interactive_installation.json
git rm ./TestSuites/qam-nfs4-client.json
git rm ./TestSuites/qam-nfs4-server.json
git rm ./TestSuites/qam-nfs-client.json
git rm ./TestSuites/qam-nfs-server.json
git rm ./TestSuites/qam-scc.json
git rm ./TestSuites/qam-yast_self_update+15.json
git rm ./TestSuites/transactional_server_helper_apps.json
git rm ./TestSuites/transactional_server_snapper.json
git rm ./TestSuites/yast-mru-install-desktop-with-addons.json
git rm ./TestSuites/yast-mru-install-minimal-with-addons.json

and also remove the section in file qam-openqa-yml/qam_jobgroups.yml

  maint_updates:
    name: 'YaST Maintenance Updates'
    id: 421

and delete the related file qam-openqa-yml/JobGroups/yast/maint_updates.yml

The list of json file to remove is given by this simply script:

#!/usr/bin/env bash
set -e
echo "# issues/153005"

repo_orig="$HOME/GitLab/qam-openqa-yml"
repo_dest="$HOME/GitLab/openqa-job-groups"

function LIST_SUITES() {
  local suites=()
  for arch in $(yq '.scenarios | keys' $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
    for product in $(yq ".scenarios.$arch | keys" $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
      for suite in $(yq ".scenarios.$arch.$product[] | select(. | type == \"!!str\") | ." $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
        suites+=( $suite )
      done
      for suite in $(yq ".scenarios.$arch.$product[] | select(type == \"!!map\") | keys" $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
        suites+=( $suite )
      done
    done
  done

  local suites_sorted_uniq=($(echo "${suites[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))

  echo "Files to remove on repo qam-openqa-yml:"
  for suite in "${suites_sorted_uniq[@]}"; do
    [[ -e "$repo_orig/TestSuites/${suite}.json"  ]] && { 
      echo "git rm ./TestSuites/${suite}.json" 
    }
  done

}

LIST_SUITES

Removed TestSuites folder and added tests inside test_suites.yaml .
The refactoring was done following these steps:

• ADD_SUITES

all steps has been done using this bash script:

#!/usr/bin/env bash
set -e
echo "# issues/153005"

repo_orig="$HOME/GitLab/qam-openqa-yml"
repo_dest="$HOME/GitLab/openqa-job-groups"

function ADD_SUITES() {
  local suites=()
  for arch in $(yq '.scenarios | keys' $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
    for product in $(yq ".scenarios.$arch | keys" $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
      for suite in $(yq ".scenarios.$arch.$product[] | select(. | type == \"!!str\") | ." $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
        suites+=( $suite )
      done
      for suite in $(yq ".scenarios.$arch.$product[] | select(type == \"!!map\") | keys" $repo_orig/JobGroups/yast/maint_updates.yml | sed -e 's/^- //'); do
        suites+=( $suite )
      done
    done
  done

  local suites_sorted_uniq=($(echo "${suites[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))

  local temp=$(mktemp -d)

  local description='Testsuite maintained at https://gitlab.suse.de/qe-yam/openqa-job-groups'

  echo "Files to convert in yaml into $temp:"
  for suite in "${suites_sorted_uniq[@]}"; do
    [[ -e "$repo_orig/TestSuites/${suite}.json"  ]] && { 
      echo "to convert $repo_orig/TestSuites/${suite}.json" 
      yq -p=json $repo_orig/TestSuites/${suite}.json | yq "{ \"test_suites\": { .TestSuites[0].name: { \"description\": \"${description}\", \"settings\": .TestSuites[0].settings | from_entries, \"testsuite\": null } } }" > $temp/${suite}.yaml
    }
  done

  yq -n '{"test_suites": null}' > $temp/test_suites.yaml

  echo "Join files in a single yaml into $temp/test_suites.yaml:"
  for suite in "${suites_sorted_uniq[@]}"; do
    [[ -e "$temp/${suite}.yaml"  ]] && { 
      echo "join $temp/${suite}.yaml" 
      yq eval-all -i 'select(fileIndex == 0) * select(fileIndex == 1)' $temp/test_suites.yaml $temp/${suite}.yaml
    }
  done

  # sort keys
  yq -i 'sort_keys(..)' $temp/test_suites.yaml

  # add anchors
  perl -i -pe 's/^\s{2}(\S+):\s*$/  \1: &\1\n/' $temp/test_suites.yaml

  # remove root key "test_suites:"
  sed -i '1d' $temp/test_suites.yaml

  # merge into repo test_suites.yaml
  cat $temp/test_suites.yaml >> $repo_dest/JobGroups/test_suites.yaml

}

ADD_SUITES

Added alias and anchors.

Aligned with last commit bcfe06c546ea92ca2ce385cb7b21836fcb5aebab of qam-openqa-yml

Created Draft qam-openqa-yml MR#701

Aligned with last commit 1da95fdb69eb3a02283637daf15f2b46e5d7375f of qam-openqa-yml

Aligned with last commit 98df8b5e70471a5cc444dc3ce3b9293dced42e90 of qam-openqa-yml

Merged openqa-job-groups MR#32

Removed Draft from qam-openqa-yml MR#701. Ready to merge.