Project

General

Profile

Actions

action #151657

closed

coordination #151984: [epic] Unify YaST and Migration CIs and job groups

Unify job group Migration Misc to use our one-and-only CI

Added by leli 3 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
2023-11-29
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Currently some job groups have all-in-one file of job template with all arches in gitlab but it is hard to maintain, we should split the job template file of Migration misc group up into files of each arch.

Refer to yast group

There will be five files in the folder: default.yaml,x86_64.yaml,s390x.yaml,ppc64le.yaml,aarch64.yaml, besides test_suites.yaml in Jobgroups contains anchors.

Acceptance criteria

AC1: Split the job template file of Migration misc group up into files of each arch.
AC2: Move CI to https://gitlab.suse.de/qe-yam/openqa-job-groups

Actions #1

Updated by rainerkoenig 3 months ago

  • Status changed from New to Workable
Actions #2

Updated by JERiveraMoya 3 months ago

  • Tags set to qe-yam-dec-sprint
  • Description updated (diff)
  • Parent task set to #151984
Actions #3

Updated by lmanfredi 3 months ago

  • Status changed from Workable to In Progress
  • Assignee set to lmanfredi
Actions #4

Updated by lmanfredi 2 months ago

Created openqa-job-groups-migrations MR#412

Actions #5

Updated by lmanfredi 2 months ago

The refactoring was done following these steps:

  • SPLIT_FILE
  • COPY_SCRIPTS
  • REMOVE_FILE

all steps has been done using this bash script:

#!/usr/bin/env bash
set -e

repo_dir="$HOME/GitLab/openqa-job-groups-migrations"

function SPLIT_FILE() {
  cd $repo_dir
  mkdir -p JobGroups/migration_miscellaneous
  cd JobGroups
  yq '{"defaults":.defaults,"products":.products} | sort_keys(..) | ... comments=""' migration_miscellaneous.yaml > migration_miscellaneous/defaults.yaml
  yq '{"aarch64": .scenarios.aarch64 | sort_keys(..) | ... comments="" }' migration_miscellaneous.yaml > migration_miscellaneous/aarch64.yaml
  yq '{"ppc64le": .scenarios.ppc64le | sort_keys(..) | ... comments="" }' migration_miscellaneous.yaml > migration_miscellaneous/ppc64le.yaml
  yq '{"s390x":   .scenarios.s390x   | sort_keys(..) | ... comments="" }' migration_miscellaneous.yaml > migration_miscellaneous/s390x.yaml
  yq '{"x86_64":  .scenarios.x86_64  | sort_keys(..) | ... comments="" }' migration_miscellaneous.yaml > migration_miscellaneous/x86_64.yaml
  yq --null-input '{"test_suites":{} } | . head_comment="this yaml contains anchors"' > test_suites.yaml
}

function COPY_SCRIPTS() {
  cd $repo_dir
  cp -r $HOME/GitLab/openqa-job-groups/scripts .
  cp $HOME/GitLab/openqa-job-groups/.yamllint .
  # Adapt by hand $HOME/GitLab/openqa-job-groups/.gitlab-ci.yml
  # cp $HOME/GitLab/openqa-job-groups/.gitlab-ci.yml .
  # Adapt by hand $HOME/GitLab/openqa-job-groups/JobGroups/header
  # cp $HOME/GitLab/openqa-job-groups/JobGroups/header JobGroups/header 
}

function REMOVE_FILE() {
  cd $repo_dir
  git rm JobGroups/migration_miscellaneous.yaml
}

SPLIT_FILE
COPY_SCRIPTS
REMOVE_FILE
Actions #6

Updated by JERiveraMoya 2 months ago

The task needs to be done in https://gitlab.suse.de/qe-yam/openqa-job-groups, splitting doesn't have much value, unifying does. Repeating those CI files in another repo is also a big mistake, we talked about that in weekly syncs, because someone else try to do the same. I see AC2 is not good enough described, the "Split" word is still very confusing. I would advice in this case always to check previous tasks related.

Nice script! btw, still could be useful for other tasks, what I'm not sure is about the sorting, perhaps there are ordered in a particular way.

Actions #7

Updated by JERiveraMoya 2 months ago

  • Tags changed from qe-yam-dec-sprint, qe-yam-jan-sprint to qe-yam-jan-sprint
Actions #8

Updated by JERiveraMoya 2 months ago

  • Subject changed from Split the job template file of Migration misc group up into files of each arch to Unify job group Migration Misc to use our one-and-only CI
Actions #9

Updated by lmanfredi 2 months ago

Closed openqa-job-groups-migrations MR#412.
The task will be done in openqa-job-groups repo, as indicated in previous #note-6

Actions #10

Updated by lmanfredi 2 months ago

Created openqa-job-groups MR#29

Actions #11

Updated by lmanfredi 2 months ago

The refactoring was done following these steps:

  • CREATE_GIT_BRANCH
  • SPLIT_FILE

all steps has been done using this bash script:

#!/usr/bin/env bash
set -e
echo "# issues/151657"

repo_orig="$HOME/GitLab/openqa-job-groups-migrations"
repo_dest="$HOME/GitLab/openqa-job-groups"

function CREATE_GIT_BRANCH() {
  cd $repo_dest
  git checkout -b issues-151657
}

function SPLIT_FILE() {
  mkdir -p $repo_dest/JobGroups/migration_miscellaneous
  yq '{"defaults":.defaults,"products":.products} | sort_keys(..) | ... comments=""' $repo_orig/JobGroups/migration_miscellaneous.yaml > $repo_dest/JobGroups/migration_miscellaneous/defaults.yaml
  yq '{"aarch64": .scenarios.aarch64 | sort_keys(..) | ... comments="" }' $repo_orig/JobGroups/migration_miscellaneous.yaml > $repo_dest/JobGroups/migration_miscellaneous/aarch64.yaml
  yq '{"ppc64le": .scenarios.ppc64le | sort_keys(..) | ... comments="" }' $repo_orig/JobGroups/migration_miscellaneous.yaml > $repo_dest/JobGroups/migration_miscellaneous/ppc64le.yaml
  yq '{"s390x":   .scenarios.s390x   | sort_keys(..) | ... comments="" }' $repo_orig/JobGroups/migration_miscellaneous.yaml > $repo_dest/JobGroups/migration_miscellaneous/s390x.yaml
  yq '{"x86_64":  .scenarios.x86_64  | sort_keys(..) | ... comments="" }' $repo_orig/JobGroups/migration_miscellaneous.yaml > $repo_dest/JobGroups/migration_miscellaneous/x86_64.yaml
  # Adapt $repo_dest/.gitlab-ci.yml
}


CREATE_GIT_BRANCH
SPLIT_FILE

GitLab CI adapted by adding:

variables:
  GROUPID_MIGRATION_MISCELLANEOUS: 478

test_migration_miscellaneous:
  stage: test
  before_script:
    - git clone --depth 1 $URL_TEST_REPO
  script:
    - scripts/ci_runner_test.sh migration_miscellaneous $GROUPID_MIGRATION_MISCELLANEOUS
  only:
    refs:
      - merge_requests
    changes:
      - JobGroups/*
      - JobGroups/migration_miscellaneous/*
      - .gitlab-ci.yml
  retry: 2

deploy_migration_miscellaneous:
  stage: deploy
  script:
    - scripts/ci_runner_deploy.sh migration_miscellaneous $GROUPID_MIGRATION_MISCELLANEOUS
  only:
    refs:
      - master
    changes:
      - JobGroups/*
      - JobGroups/migration_miscellaneous/*
      - .gitlab-ci.yml

Actions #12

Updated by lmanfredi 2 months ago

Merged openqa-job-groups MR#29

Actions #13

Updated by lmanfredi about 2 months ago

Created openqa-job-groups MR#33 to fix missing comments

Actions #14

Updated by lmanfredi about 2 months ago

Merged openqa-job-groups MR#33

Actions #15

Updated by JERiveraMoya about 2 months ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF