Project

General

Profile

Actions
Copy link

action #116269

closed

coordination #90086: [epic] Refactor container tests

Automate creation of HDDs in qac-openqa-yaml pipeline

Added by jlausuch almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
ph03nix
Target version:
-
Start date:
2022-09-06
Due date:
% Done:

100%

Estimated time:

Description

The HDDs (qcow2's) that are used by BCI tests and other container tests are created here:
https://openqa.suse.de/group_overview/377

However, there is no automation on this. I trigger them on regular basis, but this has some risks if I am on leave or vacation, because having outdated HDDs might cause some timeouts in the BCI jobs while updating the system or things like this. Besides, it takes more time to run the tests. If we have regularly-updated HDDs, the BCI jobs take less time to update the system which results in an overall time execution.

The script to be run is already created:
https://gitlab.suse.de/qac/qac-openqa-yaml/-/blob/master/containers/create_hdd_isos_post.sh

This task is about automating this script in the gitlab-ci pipeline once a week (e.g. during the weekends).

After the HDDs are created, there are few steps to be done manually in OSD after the openqa jobs need to be completed and green:

cd /var/lib/openqa/share/factory/hdd
mv centOS-Stream-8-x86_64-20220128-minimal_new.qcow2 ./fixed/centOS-Stream-8-x86_64-20220128-minimal.qcow2
mv opensuse-15.2-x86_64-695.1-textmode@64bit_new.qcow2 ./fixed/opensuse-15.2-x86_64-695.1-textmode@64bit.qcow2
mv opensuse-15.3-x86_64-20210906-4-textmode@64bit_new.qcow2 ./fixed/opensuse-15.3-x86_64-20210906-4-textmode@64bit.qcow2
mv opensuse-15.4-x86_64-20220626-1-textmode@64bit_new.qcow2 ./fixed/opensuse-15.4-x86_64-20220626-1-textmode@64bit.qcow2
mv sles-es-7.9_new.qcow2 ./fixed/sles-es-7.9.qcow2
mv sles-es-8.5_new.qcow2 ./fixed/sles-es-8.5.qcow2
mv ubuntu-22.04_new.qcow2 ./fixed/ubuntu-22.04.qcow2
mv SUSE-MicroOS.aarch64-5.1.0-Default-GM-Updated.qcow2 ./fixed/
mv SUSE-MicroOS.aarch64-5.2.0-Default-GM-Updated.qcow2 ./fixed/
mv SUSE-MicroOS.s390x-5.1.0-Default-kvm-GM-Updated.qcow2 ./fixed/
mv SUSE-MicroOS.s390x-5.2.0-Default-kvm-GM-Updated.qcow2 ./fixed/
mv SUSE-MicroOS.x86_64-5.1.0-Default-GM-Updated.qcow2 ./fixed/
mv SUSE-MicroOS.x86_64-5.2.0-Default-GM-Updated.qcow2 ./fixed/

I don't really know how to automate this, as we need to wait for jobs to be done and completed. Maybe we can have another schedule a couple of hours after first one to do these operations... Or maybe schedule the job triggering on Saturdays and these commands on Sundays? I'm open to ideas.

Acceptance criteria

  • Create a new stage/job in gitlab-ci.yml to execute the hdd creation script.
  • Create a new schedule in the pipeline that triggers the openQA jobs once a week (e.g. on Saturdays).

Files

new-hdds.png (173 KB) new-hdds.png ph03nix, 2022-10-18 13:40

Related issues 4 (0 open — 4 closed)

Related to Containers - action #116443: test fails in host_configuration on 15-SP4 BCIResolvedph03nix2022-09-12

Actions
Related to Containers - action #108102: Create autoyast installation to create HDD for containers.Rejected2022-03-10

Actions
Has duplicate Containers - action #117517: Automate the container host image generation processRejectedph03nix2022-10-04

Actions
Has duplicate Containers - action #93588: Improve trigger create hdd scriptClosed2021-06-07

Actions
Actions
Copy link
 #1

Updated by jlausuch almost 2 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by jlausuch almost 2 years ago

  • Related to action #116443: test fails in host_configuration on 15-SP4 BCI added
Actions
Copy link
 #3

Updated by jlausuch almost 2 years ago

  • Related to action #108102: Create autoyast installation to create HDD for containers. added
Actions
Copy link
 #4

Updated by jlausuch almost 2 years ago

  • Parent task set to #90086
Actions
Copy link
 #5

Updated by jlausuch almost 2 years ago

Needs further discussion to see how we can tackle the image creation.

Actions
Copy link
 #6

Updated by ph03nix over 1 year ago

  • Related to action #117517: Automate the container host image generation process added
Actions
Copy link
 #7

Updated by ph03nix over 1 year ago

  • Related to deleted (action #117517: Automate the container host image generation process)
Actions
Copy link
 #8

Updated by ph03nix over 1 year ago

  • Has duplicate action #117517: Automate the container host image generation process added
Actions
Copy link
 #9

Updated by ph03nix over 1 year ago

As Martin pointed out, it might be useful to move the host_configuration part from the test run to the image creation. This would mean that necessary tools are already pre-installed and we only need to run a simple update & reboot before running the tests.

Actions
Copy link
 #10

Updated by mloviska over 1 year ago

For fixed assets there is already a CI in place https://gitlab.suse.de/qa-sle/fixed-assets

Actions
Copy link
 #11

Updated by ph03nix over 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to ph03nix
Actions
Copy link
 #12

Updated by ph03nix over 1 year ago

mloviska wrote:

For fixed assets there is already a CI in place https://gitlab.suse.de/qa-sle/fixed-assets

AFAICS this is only for listing the assets, there is no CI for periodic job trigger in place.

Actions
Copy link
 #13

Updated by ph03nix over 1 year ago

  • Has duplicate action #93588: Improve trigger create hdd script added
Actions
Copy link
 #14

Updated by ph03nix over 1 year ago

I setup a toy project aceto-magico which schedules the creation test runs every Friday at 18:00.

Actions
Copy link
 #15

Updated by ph03nix over 1 year ago

  • % Done changed from 0 to 80
Actions
Copy link
 #16

Updated by ph03nix over 1 year ago

If the assets get created on a regular schedule we might not need to push them to fixed/ and we are already done with the task. I need to think a bit about this though.

Actions
Copy link
 #17

Updated by ph03nix over 1 year ago

POC is in place: https://gitlab.suse.de/ph03nix/aceto-magico/ - It triggers the creation of the new HDDs every Friday.

I let this run for a little bit to see, how good this works. Will also remove the corresponding assets from fixed/ and see if that works.

Actions
Copy link
 #18

Updated by ph03nix over 1 year ago

  • Status changed from In Progress to Feedback

I asked in #eng-testing and it should be fine to avoid using assets from the fixed/ folder - See https://suse.slack.com/archives/C02CANHLANP/p1665991966827409

I'm going to remove them for now and we see at the end of the week, if there are any issues with this.

Actions
Copy link
 #19

Updated by ph03nix over 1 year ago

  • % Done changed from 80 to 50

I need to move the source images to fixed/ and the produced new images to the normal folder.

Actions
Copy link
 #20

Updated by ph03nix over 1 year ago

The following assets require a change of the workflow:

centOS-Stream-8-x86_64-20220128-minimal.qcow2
centOS-Stream-8-x86_64-20220128-minimal_new.qcow2

ubuntu-22.04.qcow2
ubuntu-22.04_new.qcow2

sles-es-7.9.qcow2
sles-es-7.9_new.qcow2

opensuse-15.4-x86_64-20220626-1-textmode@64bit.qcow2
opensuse-15.4-x86_64-20220626-1-textmode@64bit_new.qcow2

opensuse-15.2-x86_64-695.1-textmode@64bit_new.qcow2
opensuse-15.2-x86_64-695.1-textmode@64bit.qcow2

opensuse-15.3-x86_64-20210906-4-textmode@64bit.qcow2
opensuse-15.3-x86_64-20210906-4-textmode@64bit_new.qcow2

sles-es-8.5.qcow2
sles-es-8.5_new.qcow2

SUSE-MicroOS.aarch64-5.2.0-Default-GM.raw.xz
SUSE-MicroOS.aarch64-5.2.0-Default-GM-Updated.qcow2

SUSE-MicroOS.s390x-5.2.0-Default-kvm-GM.qcow2
SUSE-MicroOS.s390x-5.2.0-Default-kvm-GM-Updated.qcow2

SUSE-MicroOS.x86_64-5.2.0-Default-GM.raw.xz
SUSE-MicroOS.x86_64-5.2.0-Default-GM-Updated.qcow2

SUSE-MicroOS.aarch64-5.1.0-Default-GM.raw.xz
SUSE-MicroOS.aarch64-5.1.0-Default-GM-Updated.qcow2

SUSE-MicroOS.s390x-5.1.0-Default-kvm-GM.qcow2
SUSE-MicroOS.s390x-5.1.0-Default-kvm-GM-Updated.qcow2

SUSE-MicroOS.x86_64-5.1.0-Default-GM.raw.xz
SUSE-MicroOS.x86_64-5.1.0-Default-GM-Updated.qcow2

I will rename the "original" ones to *_template.qcow2 and we should use them and release them as the actual HDDs that are being used afterwards.

Actions
Copy link
 #21

Updated by ph03nix over 1 year ago 

ph03nix@openqa:~/factory/hdd/fixed> sudo ln centOS-Stream-8-x86_64-20220128-minimal.qcow2 centOS-Stream-8-x86_64-20220128-minimal_template.qcow2
ph03nix@openqa:~/factory/hdd/fixed> sudo ln ubuntu-22.04.qcow2 ubuntu-22.04_template.qcow2
ph03nix@openqa:~/factory/hdd/fixed> sudo ln sles-es-7.9.qcow2 sles-es-7.9_template.qcow2
ph03nix@openqa:~/factory/hdd/fixed> sudo ln opensuse-15.4-x86_64-20220626-1-textmode@64bit.qcow2 opensuse-15.4-x86_64-20220626-1-textmode@64bit_template.qcow2
ph03nix@openqa:~/factory/hdd/fixed> sudo ln opensuse-15.2-x86_64-695.1-textmode@64bit.qcow2 opensuse-15.2-x86_64-695.1-textmode@64bit_template.qcow2
ph03nix@openqa:~/factory/hdd/fixed> sudo ln opensuse-15.3-x86_64-20210906-4-textmode@64bit.qcow2 opensuse-15.3-x86_64-20210906-4-textmode@64bit_template.qcow2
ph03nix@openqa:~/factory/hdd/fixed> sudo ln sles-es-8.5.qcow2 sles-es-8.5_template.qcow2
Actions
Copy link
 #23

Updated by ph03nix over 1 year ago

I will trigger the test runs today in the late afternoon, when all current maintenance test runs are completed.

Actions
Copy link
 #24

Updated by ph03nix over 1 year ago

Except for s390x (still scheduled) - the creation test runs all succeed after a new trigger: https://openqa.suse.de/group_overview/377?limit_builds=50

Actions
Copy link
 #25

Updated by ph03nix over 1 year ago

OK, last Friday's trigger was successful. I moved the aceto-magico project to QAC: https://gitlab.suse.de/qac/aceto-magico

Actions
Copy link
 #26

Updated by ph03nix over 1 year ago

  • % Done changed from 70 to 80
Actions
Copy link
 #27

Updated by ph03nix over 1 year ago

I also updated https://confluence.suse.com/pages/viewpage.action?pageId=1093763100 to point to the new repository.

Actions
Copy link
 #28

Updated by ph03nix over 1 year ago

  • Status changed from Feedback to Resolved
  • % Done changed from 80 to 100
Actions
Copy link

Also available in: Atom PDF