Project

General

Profile

Actions

action #112457

closed

coordination #112454: [epic] Kubernetes testing

[New test] Test coverage for kubernetes client (kubectl)

Added by jlausuch about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
2022-06-15
Due date:
% Done:

100%

Estimated time:

Description

Kubernetes client is part of Containers Module since 15-SP1.

From scc.suse.com:

kubernetes-client   1.17.4-4.18.1   Apache-2.0  Containers Module 15 SP1
kubernetes1.18-client   1.18.10-5.5.1   Apache-2.0  Containers Module 15 SP2
kubernetes1.18-client   1.18.10-5.5.1   Apache-2.0  Containers Module 15 SP3
kubernetes1.18-client   1.18.10-5.5.1   Apache-2.0  Containers Module 15 SP4

The goal is to have proper coverage for kubectl exercising all the command options that the CLI provides.

We could run validation of the client against several K8S environments (EKS, , similar to what we do for Helm tests.

There is already some basic smoke test for kubectl here:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/rancher/utils.pm#L39

sub kubectl_basic_test {
    assert_script_run "kubectl get pod --all-namespaces -o wide";
    assert_script_run "kubectl get service --all-namespaces -o wide";
    assert_script_run "kubectl get namespaces --all-namespaces -o wide";
    assert_script_run "kubectl get endpoints --all-namespaces -o wide";
    assert_script_run "kubectl get deployments --all-namespaces -o wide";
    assert_script_run "kubectl get replicasets --all-namespaces -o wide";
    assert_script_run "kubectl get ingresses --all-namespaces -o wide";
    assert_script_run "kubectl cluster-info";
}

This could be used as a base.

There is also a method in lib directory to install it in SLE/openSUSE:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/containers/k8s.pm#L38

Acceptance criteria:

New test module(s) for SLE 15-SP1+ and openSUSE to cover kubectl command line operations with real Kubernetes environments.

  • AC 1: Get (get) operation (namespace, deployment, replicaset, pod, job, cronjob, service, ingresses, pc, pvc, secrets, configmaps, endpoints)
  • AC 2: Describe (describe) operations (the same than in AC 1)
  • AC 3: Queries get operations
  • AC 4: Filtering over describe operations (for instance on configmaps or pod status)
  • AC 5: Creation, delete, list and change of namespaces

Files

rocket.jpg (21.1 KB) rocket.jpg jlausuch, 2022-07-19 09:44

Related issues 1 (0 open1 closed)

Related to Containers - action #114457: Run kubectl tests on SLESResolvedpdostal2022-07-21

Actions
Actions #1

Updated by jlausuch about 2 years ago

  • Subject changed from [New test] Create a test case to [New test] Test coverage for kubernetes client (kubectl)
Actions #2

Updated by ph03nix about 2 years ago

  • Status changed from Workable to In Progress
  • Assignee set to ph03nix
Actions #3

Updated by ilausuch about 2 years ago

  • Description updated (diff)
Actions #4

Updated by pdostal about 2 years ago

@ph03nix can you please get rid of rancher::kubectl_basic_test and create your new subroutines in containers::k8s instead?

Actions #5

Updated by ph03nix about 2 years ago

  • Status changed from In Progress to Blocked

Ticket blocked due to concerns about the version compatibility between client and kubernetes infrastructure until further concerns.

Actions #6

Updated by jlausuch about 2 years ago

  • Status changed from Blocked to In Progress

Unblocked.
We have a concrete requirement for kubernetes clinet version upgrade.
15-SP3: https://jira.suse.com/browse/SLE-24656
15-SP4: https://jira.suse.com/browse/SLE-24655

Expected new version is 1.23.5.

This will come as a maintenance update which will land in openQA.
Would be nice to have something in place when that happens.

We could start using just K3S for the first implementation, this way we can also enable the test in O3.
Related: https://suse.slack.com/archives/C02CGKBCGT1/p1655894730999129

Actions #7

Updated by ph03nix about 2 years ago

After discovering https://bugzilla.opensuse.org/show_bug.cgi?id=1200876 and realizing that kubeadm is broken on Tumbleweed due to outdated images:

failed to pull image "registry.opensuse.org/kubic/pause:3.7": output: E0623 14:37:13.894254   19657 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = reading manifest 3.7 in registry.opensuse.org/kubic/pause: manifest unknown" image="registry.opensuse.org/kubic/pause:3.7"
time="2022-06-23T14:37:13+02:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = reading manifest 3.7 in registry.opensuse.org/kubic/pause: manifest unknown"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

I realize it might be a good idea to follow our Factory first principle and start to establish some tests for Tumbleweed as a first step.

Actions #8

Updated by jlausuch about 2 years ago

ph03nix wrote:

After discovering https://bugzilla.opensuse.org/show_bug.cgi?id=1200876 and realizing that kubeadm is broken on Tumbleweed due to outdated images:

failed to pull image "registry.opensuse.org/kubic/pause:3.7": output: E0623 14:37:13.894254   19657 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = reading manifest 3.7 in registry.opensuse.org/kubic/pause: manifest unknown" image="registry.opensuse.org/kubic/pause:3.7"
time="2022-06-23T14:37:13+02:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = reading manifest 3.7 in registry.opensuse.org/kubic/pause: manifest unknown"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

I realize it might be a good idea to follow our Factory first principle and start to establish some tests for Tumbleweed as a first step.
+1

Actions #9

Updated by pdostal about 2 years ago

I find this approach good.

I just did not expect that Felix will pick kubeadm on Tumbleweed when we have for example EKS :smile:

But again, I find this approach very cool.

Actions #10

Updated by ph03nix about 2 years ago

https://build.opensuse.org/request/show/984715 Is for the updated kubic/pause image. Need to wait until it becomes available.

Actions #11

Updated by ph03nix almost 2 years ago

I was fiddling around with kubeadm for the whole morning and I'm making slow progress. Will still try a bit, but I'm considering to just use k3s as a kubernetes platform where we run kubectl against. Setup is easy and it provides the API that we need in order to test the kubectl package.

Actions #12

Updated by ph03nix almost 2 years ago

  • % Done changed from 0 to 40
Actions #13

Updated by ph03nix almost 2 years ago

This issue will have to wait a bit, since https://progress.opensuse.org/issues/112400 is more important now.

Actions #14

Updated by ph03nix almost 2 years ago

New PR to start fresh after a lot of changes: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15215

Actions #15

Updated by ph03nix almost 2 years ago

  • % Done changed from 40 to 60

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15215 merged. I will continue next week on scheduling some test runs.

Actions #16

Updated by ph03nix almost 2 years ago

I've added the containers_host_kubectl test run on O3 in Tumbleweed/Development:

    - containers_host_kubectl:
        testsuite: extra_tests_textmode_containers
        settings:
          CONTAINER_RUNTIME: 'kubectl'

Let's wait for tomorrow's test runs.

Actions #17

Updated by jlausuch almost 2 years ago

ph03nix wrote:

I've added the containers_host_kubectl test run on O3 in Tumbleweed/Development:

    - containers_host_kubectl:
        testsuite: extra_tests_textmode_containers
        settings:
          CONTAINER_RUNTIME: 'kubectl'

Let's wait for tomorrow's test runs.

Nice! After a few green runs, it can be moved to main group.

Actions #18

Updated by ph03nix almost 2 years ago

Let's not get overly optimistic, I'm pretty sure somewhere I screwed up and it will take still 1-2 iterations until it actually lands ;-)

Actions #19

Updated by ph03nix almost 2 years ago

HOLY MOLY, it works! https://openqa.opensuse.org/tests/2473469

Let's wait for another 1-2 days and then move it to the main Tumbleweed test runs.

Actions #20

Updated by jlausuch almost 2 years ago

ph03nix wrote:

HOLY MOLY, it works! https://openqa.opensuse.org/tests/2473469

Let's wait for another 1-2 days and then move it to the main Tumbleweed test runs.

Actions #21

Updated by ph03nix almost 2 years ago

Huh? For some reason there is no kubectl test run in the Development group today. It is still present in the job groups though.

Actions #22

Updated by jlausuch almost 2 years ago

ph03nix wrote:

Huh? For some reason there is no kubectl test run in the Development group today. It is still present in the job groups though.

I see it https://openqa.opensuse.org/tests/2475514

Actions #23

Updated by ph03nix almost 2 years ago

Huh? For some reason there is no kubectl test run in the Development group today. It is still present in the job groups though.

jlausuch wrote:

ph03nix wrote:

Huh? For some reason there is no kubectl test run in the Development group today. It is still present in the job groups though.

I see it https://openqa.opensuse.org/tests/2475514

Looks like they are being scheduled later on the day. Anyways, two successful test runs, I think I'm gonna push this to the main Tumbleweed group today.

Actions #25

Updated by ph03nix almost 2 years ago

I also added the test run to Development/Tumbleweed aarch64

Actions #26

Updated by ph03nix almost 2 years ago

Actions #27

Updated by ph03nix almost 2 years ago

  • % Done changed from 60 to 100

Job groups scheduled for Tumbleweed. Will mark the ticket as resolved when tomorrow's test runs are being scheduled correctly.

Actions #28

Updated by jlausuch almost 2 years ago

ph03nix wrote:

Job groups scheduled for Tumbleweed. Will mark the ticket as resolved when tomorrow's test runs are being scheduled correctly.

Since there has not been an new build and you are leaving on FTO, I'll take care!

Actions #29

Updated by ph03nix almost 2 years ago

  • Status changed from In Progress to Resolved

All good.

Actions

Also available in: Atom PDF