QA » openQA Project » openQA Tests

c250:~ # k3s kubectl get node
NAME STATUS ROLES AGE VERSION
c250 Ready control-plane,master 7m54s v1.22.6+k3s1

Connection to 10.161.59.250 closed.

susetest:~ # k3s kubectl get node
NAME STATUS ROLES AGE VERSION
susetest Ready control-plane,master 3d1h v1.22.7+k3s1

IP: 10.161.60.70

susetest:/var/lib/helm # kubectl get deployments -A
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
default rmt-app 1/1 1 1 5d16h
kube-system local-path-provisioner 1/1 1 1 9d
default rmt-db 1/1 1 1 5d16h
kube-system metrics-server 1/1 1 1 9d
kube-system traefik 1/1 1 1 9d
default rmt-front 1/1 1 1 5d16h
kube-system coredns 1/1 1 1 9d
susetest:/var/lib/helm # kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system helm-install-traefik-crd--1-9mk6l 0/1 Completed 0 9d
default mysql-1646733988-0 0/1 ContainerCreating 0 5d16h
kube-system coredns-96cc4f57d-mg4sx 1/1 Running 0 9d
kube-system traefik-7df6887f49-87jfl 1/1 Running 0 9d
kube-system local-path-provisioner-84bb864455-gwlx2 1/1 Running 0 9d
kube-system svclb-traefik-5g5jx 2/2 Running 0 9d
default rmt-app-64b4bf9d99-jmf6q 1/1 Running 0 5d15h
kube-system traefik-56c4b88c4b-xcrjw 1/1 Running 0 9d
kube-system traefik-6c74d458c9-r2d29 1/1 Running 0 9d
default rmt-db-55cddd4765-42dnx 1/1 Running 0 5d15h
kube-system metrics-server-ff9dbcb6c-g2m7q 1/1 Running 0 9d
default rmt-front-6bb4fd4c8b-z64f2 1/1 Running 0 5d15h
default rmt-app-sync-27449643--1-g8cmz 0/1 CrashLoopBackOff 685 (4m31s ago) 2d17h
default rmt-app-sync-27449716--1-rk8f6 0/1 CrashLoopBackOff 665 (4m6s ago) 2d16h
default rmt-app-sync-27449650--1-5xqmt 0/1 CrashLoopBackOff 685 (4m4s ago) 2d17h
default rmt-app-sync-27449647--1-dhnv6 0/1 CrashLoopBackOff 665 (3m50s ago) 2d16h
default rmt-app-mirror-27449730--1-874tp 0/1 CrashLoopBackOff 684 (3m46s ago) 2d17h
default rmt-app-sync-27449640--1-8q9l8 0/1 CrashLoopBackOff 686 (3m23s ago) 2d17h
default rmt-app-sync-27449645--1-2m9bc 0/1 CrashLoopBackOff 665 (97s ago) 2d16h
kube-system helm-install-traefik--1-4z7zh 0/1 CrashLoopBackOff 1144 (82s ago) 5d5h
default rmt-app-sync-27449642--1-gt8fp 0/1 CrashLoopBackOff 665 (33s ago) 2d16h

Test steps:

1. Disable ipV6
2. Fresh install sles15sp4 with container
3. Added SUSE CA # as root $ zypper ar --refresh http://download.suse.de/ibs/SUSE:/CA/SLE_15_SP4/SUSE:CA.repo $ zypper ref c254:~ # zypper ref Repository 'SLE-Module-Basesystem15-SP4-Pool' is up to date.
   Repository 'SLE-Module-Basesystem15-SP4-Updates' is up to date.
   Repository 'SLE-Module-Containers15-SP4-Pool' is up to date.
   Repository 'SLE-Module-Containers15-SP4-Updates' is up to date.
   Repository 'SLE-Module-Desktop-Applications15-SP4-Pool' is up to date.
   Repository 'SLE-Module-Desktop-Applications15-SP4-Updates' is up to date.
   Repository 'SLE-Module-DevTools15-SP4-Pool' is up to date.
   Repository 'SLE-Module-DevTools15-SP4-Updates' is up to date.
   Repository 'SLES15-SP4-15.4-0' is up to date.
   Retrieving repository 'SUSE Internal CA Certificate (SLE_15_SP4)' metadata ............................................................................................[done] Building repository 'SUSE Internal CA Certificate (SLE_15_SP4)' cache .................................................................................................[done] Repository 'SLE-Product-SLES15-SP4-Pool' is up to date.
   Repository 'SLE-Product-SLES15-SP4-Updates' is up to date.
   Repository 'SLE-Module-Server-Applications15-SP4-Pool' is up to date.
   Repository 'SLE-Module-Server-Applications15-SP4-Updates' is up to date.
   All repositories have been refreshed.

$ ssl_verify=host zypper in ca-certificates-suse p11-kit-nss-trust -mozilla-nss-certs

zypper in git
ssh-keygen -t ed25519 -C "yuwang@suse.com"

1. git clone gitlab@gitlab.suse.de:brunoleon/rmt-helm.git

2. curl -sfL https://get.k3s.io | sh -

c254:/var/lib/helm # k3s kubectl get node
NAME STATUS ROLES AGE VERSION
c254 Ready control-plane,master 60s v1.23.6+k3s1

c254:/var/lib/helm # systemctl status k3s.service (Active: active (running))

1. zypper in helm

2. export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

c254:/var/lib/helm # helm install rmt ./rmt-helm -f myvalue.yaml
NAME: rmt
LAST DEPLOYED: Thu May 5 04:29:59 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
c254:/var/lib/helm # helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
rmt default 1 2022-05-05 04:29:59.438927854 -0400 EDT deployed rmt-0.2.3 2.7.0

To register one system with RMT container:
susetest:~ # SUSEConnect --url http://susetest.local -p SLES/15.4/x86_64
Registering system to registration proxy http://susetest.local

Announcing system to http://susetest.local ...

Activating SLES 15.4 x86_64 ...
-> Adding service to system ...
-> Installing release package ...
command '/usr/bin/zypper --no-refresh --non-interactive install --no-recommends --auto-agree-with-product-licenses -t product SLES' failed
Error: zypper returned 106 with 'Error building the cache:
[SUSE_Linux_Enterprise_Server_x86_64:SLE-Product-SLES15-SP4-Pool|http://susetest.local/repo/SUSE/Products/SLE-Product-SLES/15-SP4/x86_64/product/?credentials=SUSE_Linux_Enterprise_Server_x86_64] Valid metadata not found at specified URL
History:

• [SUSE_Linux_Enterprise_Server_x86_64:SLE-Product-SLES15-SP4-Pool|http://susetest.local/repo/SUSE/Products/SLE-Product-SLES/15-SP4/x86_64/product/?credentials=SUSE_Linux_Enterprise_Server_x86_64] Repository type can't be determined.

Error building the cache:
[SUSE_Linux_Enterprise_Server_x86_64:SLE-Product-SLES15-SP4-Updates|http://susetest.local/repo/SUSE/Updates/SLE-Product-SLES/15-SP4/x86_64/update/?credentials=SUSE_Linux_Enterprise_Server_x86_64] Valid metadata not found at specified URL
History:

• [SUSE_Linux_Enterprise_Server_x86_64:SLE-Product-SLES15-SP4-Updates|http://susetest.local/repo/SUSE/Updates/SLE-Product-SLES/15-SP4/x86_64/update/?credentials=SUSE_Linux_Enterprise_Server_x86_64] Repository type can't be determined.

Some of the repositories have not been refreshed because of an error.'
susetest:~ # SUSEConnect -s
[{"identifier":"SLES","version":"15.4","arch":"x86_64","status":"Registered"}]
susetest:~ # zypper lr -u
Repository priorities are without effect. All enabled repositories share the same priority.

susetest:~ #

To check rmt-cli output:
susetest:~ # kubectl get pods
NAME READY STATUS RESTARTS AGE
rmt-front-54bfdd8c99-78tlj 1/1 Running 0 2d22h
rmt-db-5b9994d9b6-8kxxn 1/1 Running 0 2d22h
rmt-app-5996c74cd-vlmpj 1/1 Running 0 2d22h
rmt-app-mirror-27531750-h696q 0/1 Completed 0 2d1h
rmt-app-mirror-27533190-pd8x8 0/1 Completed 0 25h
rmt-app-sync-27534595-qdj4n 0/1 Completed 0 101m
rmt-app-sync-27534598-8zc8w 0/1 Completed 4 98m
rmt-app-sync-27534599-kkr4w 0/1 Completed 5 97m
rmt-app-mirror-27534630-tq4sp 0/1 Completed 0 66m
susetest:~ # kubectl exec -it rmt-app-5996c74cd-vlmpj bash
rmt-app-5996c74cd-vlmpj:/ # rmt-cli repos list
+------+----------------------------------------------------------------+---------------+---------+-------------------------+
| ID | Product | Mandatory? | Mirror? | Last mirrored |
+------+----------------------------------------------------------------+---------------+---------+-------------------------+
| 5139 | SLE-Module-Basesystem15-SP4-Pool for sle-15-x86_64 | Mandatory | Mirror | 2022-05-09 06:30:08 UTC |
| 5137 | SLE-Module-Basesystem15-SP4-Updates for sle-15-x86_64 | Mandatory | Mirror | 2022-05-09 06:30:04 UTC |
| 5179 | SLE-Module-Server-Applications15-SP4-Pool for sle-15-x86_64 | Mandatory | Mirror | 2022-05-09 06:30:10 UTC |
| 5177 | SLE-Module-Server-Applications15-SP4-Updates for sle-15-x86_64 | Mandatory | Mirror | 2022-05-09 06:30:09 UTC |
| 5103 | SLE-Product-SLES15-SP4-Pool for sle-15-x86_64 | Mandatory | Mirror | 2022-05-09 06:30:04 UTC |
| 5100 | SLE-Product-SLES15-SP4-Updates for sle-15-x86_64 | Mandatory | Mirror | 2022-05-09 06:30:02 UTC |
| 5102 | SLE15-SP4-Installer-Updates for sle-15-x86_64 | Not Mandatory | Mirror | 2022-05-09 06:30:03 UTC |
+------+----------------------------------------------------------------+---------------+---------+-------------------------+
Only enabled repositories are shown by default. Use the '--all' option to see all repositories.

Hi,

Let me put here all the info you need to automate this procedure.

We have a function to install K3s:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/containers/k8s.pm#L23
Also, a very simple one to install Helm:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/containers/k8s.pm#L47

There is a Helm test:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/tests/containers/helm.pm

Which we run against different Kubernetes, environments (Azure AKS, Amazon EKS, and locally K3S).

This is an example:
https://openqa.suse.de/tests/8759732#step/helm_K3S/1

What you would need to do is to create a single module to test RMT Container, which re-uses some of this code, and code all the manual steps you have done.

Feel free to create a job group inside /containers in OSD. Maybe call it, RMT Container (I can also do that).

Then, once we have the module merged, I will take care of enabling the automatic trigger of the tests in the container bot, where we can specify openqa variables.

Opened new ticket for automatic RMT container:
https://progress.opensuse.org/issues/111734