action #112457
closedcoordination #112454: [epic] Kubernetes testing
[New test] Test coverage for kubernetes client (kubectl)
100%
Description
Kubernetes client is part of Containers Module since 15-SP1.
From scc.suse.com:
kubernetes-client 1.17.4-4.18.1 Apache-2.0 Containers Module 15 SP1
kubernetes1.18-client 1.18.10-5.5.1 Apache-2.0 Containers Module 15 SP2
kubernetes1.18-client 1.18.10-5.5.1 Apache-2.0 Containers Module 15 SP3
kubernetes1.18-client 1.18.10-5.5.1 Apache-2.0 Containers Module 15 SP4
The goal is to have proper coverage for kubectl
exercising all the command options that the CLI provides.
We could run validation of the client against several K8S environments (EKS, , similar to what we do for Helm tests.
There is already some basic smoke test for kubectl
here:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/rancher/utils.pm#L39
sub kubectl_basic_test {
assert_script_run "kubectl get pod --all-namespaces -o wide";
assert_script_run "kubectl get service --all-namespaces -o wide";
assert_script_run "kubectl get namespaces --all-namespaces -o wide";
assert_script_run "kubectl get endpoints --all-namespaces -o wide";
assert_script_run "kubectl get deployments --all-namespaces -o wide";
assert_script_run "kubectl get replicasets --all-namespaces -o wide";
assert_script_run "kubectl get ingresses --all-namespaces -o wide";
assert_script_run "kubectl cluster-info";
}
This could be used as a base.
There is also a method in lib
directory to install it in SLE/openSUSE:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/containers/k8s.pm#L38
Acceptance criteria:¶
New test module(s) for SLE 15-SP1+ and openSUSE to cover kubectl
command line operations with real Kubernetes environments.
- AC 1: Get (get) operation (namespace, deployment, replicaset, pod, job, cronjob, service, ingresses, pc, pvc, secrets, configmaps, endpoints)
- AC 2: Describe (describe) operations (the same than in AC 1)
- AC 3: Queries get operations
- AC 4: Filtering over describe operations (for instance on configmaps or pod status)
- AC 5: Creation, delete, list and change of namespaces
Files
Updated by jlausuch about 2 years ago
- Subject changed from [New test] Create a test case to [New test] Test coverage for kubernetes client (kubectl)
Updated by ph03nix about 2 years ago
- Status changed from Workable to In Progress
- Assignee set to ph03nix
Updated by pdostal about 2 years ago
@ph03nix can you please get rid of rancher::kubectl_basic_test
and create your new subroutines in containers::k8s
instead?
Updated by ph03nix about 2 years ago
- Status changed from In Progress to Blocked
Ticket blocked due to concerns about the version compatibility between client and kubernetes infrastructure until further concerns.
Updated by jlausuch about 2 years ago
- Status changed from Blocked to In Progress
Unblocked.
We have a concrete requirement for kubernetes clinet version upgrade.
15-SP3: https://jira.suse.com/browse/SLE-24656
15-SP4: https://jira.suse.com/browse/SLE-24655
Expected new version is 1.23.5
.
This will come as a maintenance update which will land in openQA.
Would be nice to have something in place when that happens.
We could start using just K3S for the first implementation, this way we can also enable the test in O3.
Related: https://suse.slack.com/archives/C02CGKBCGT1/p1655894730999129
Updated by ph03nix about 2 years ago
After discovering https://bugzilla.opensuse.org/show_bug.cgi?id=1200876 and realizing that kubeadm is broken on Tumbleweed due to outdated images:
failed to pull image "registry.opensuse.org/kubic/pause:3.7": output: E0623 14:37:13.894254 19657 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = reading manifest 3.7 in registry.opensuse.org/kubic/pause: manifest unknown" image="registry.opensuse.org/kubic/pause:3.7"
time="2022-06-23T14:37:13+02:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = reading manifest 3.7 in registry.opensuse.org/kubic/pause: manifest unknown"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher
I realize it might be a good idea to follow our Factory first principle and start to establish some tests for Tumbleweed as a first step.
Updated by jlausuch about 2 years ago
ph03nix wrote:
After discovering https://bugzilla.opensuse.org/show_bug.cgi?id=1200876 and realizing that kubeadm is broken on Tumbleweed due to outdated images:
failed to pull image "registry.opensuse.org/kubic/pause:3.7": output: E0623 14:37:13.894254 19657 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = reading manifest 3.7 in registry.opensuse.org/kubic/pause: manifest unknown" image="registry.opensuse.org/kubic/pause:3.7" time="2022-06-23T14:37:13+02:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = reading manifest 3.7 in registry.opensuse.org/kubic/pause: manifest unknown" , error: exit status 1 To see the stack trace of this error execute with --v=5 or higher
I realize it might be a good idea to follow our Factory first principle and start to establish some tests for Tumbleweed as a first step.
+1
Updated by pdostal about 2 years ago
I find this approach good.
I just did not expect that Felix will pick kubeadm on Tumbleweed when we have for example EKS :smile:
But again, I find this approach very cool.
Updated by ph03nix about 2 years ago
https://build.opensuse.org/request/show/984715 Is for the updated kubic/pause
image. Need to wait until it becomes available.
Updated by ph03nix almost 2 years ago
I was fiddling around with kubeadm
for the whole morning and I'm making slow progress. Will still try a bit, but I'm considering to just use k3s as a kubernetes platform where we run kubectl
against. Setup is easy and it provides the API that we need in order to test the kubectl
package.
Updated by ph03nix almost 2 years ago
- % Done changed from 0 to 40
Updated by ph03nix almost 2 years ago
This issue will have to wait a bit, since https://progress.opensuse.org/issues/112400 is more important now.
Updated by ph03nix almost 2 years ago
New PR to start fresh after a lot of changes: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15215
Updated by ph03nix almost 2 years ago
- % Done changed from 40 to 60
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15215 merged. I will continue next week on scheduling some test runs.
Updated by ph03nix almost 2 years ago
I've added the containers_host_kubectl
test run on O3 in Tumbleweed/Development:
- containers_host_kubectl:
testsuite: extra_tests_textmode_containers
settings:
CONTAINER_RUNTIME: 'kubectl'
Let's wait for tomorrow's test runs.
Updated by jlausuch almost 2 years ago
ph03nix wrote:
I've added the
containers_host_kubectl
test run on O3 in Tumbleweed/Development:- containers_host_kubectl: testsuite: extra_tests_textmode_containers settings: CONTAINER_RUNTIME: 'kubectl'
Let's wait for tomorrow's test runs.
Nice! After a few green runs, it can be moved to main group.
Updated by ph03nix almost 2 years ago
Let's not get overly optimistic, I'm pretty sure somewhere I screwed up and it will take still 1-2 iterations until it actually lands ;-)
Updated by ph03nix almost 2 years ago
HOLY MOLY, it works! https://openqa.opensuse.org/tests/2473469
Let's wait for another 1-2 days and then move it to the main Tumbleweed test runs.
Updated by jlausuch almost 2 years ago
- File rocket.jpg rocket.jpg added
ph03nix wrote:
HOLY MOLY, it works! https://openqa.opensuse.org/tests/2473469
Let's wait for another 1-2 days and then move it to the main Tumbleweed test runs.
Updated by ph03nix almost 2 years ago
Huh? For some reason there is no kubectl test run in the Development group today. It is still present in the job groups though.
Updated by jlausuch almost 2 years ago
ph03nix wrote:
Huh? For some reason there is no kubectl test run in the Development group today. It is still present in the job groups though.
Updated by ph03nix almost 2 years ago
Huh? For some reason there is no kubectl test run in the Development group today. It is still present in the job groups though.
jlausuch wrote:
ph03nix wrote:
Huh? For some reason there is no kubectl test run in the Development group today. It is still present in the job groups though.
Looks like they are being scheduled later on the day. Anyways, two successful test runs, I think I'm gonna push this to the main Tumbleweed group today.
Updated by ph03nix almost 2 years ago
Updated by ph03nix almost 2 years ago
I also added the test run to Development/Tumbleweed aarch64
Updated by ph03nix almost 2 years ago
- Related to action #114457: Run kubectl tests on SLES added
Updated by ph03nix almost 2 years ago
- % Done changed from 60 to 100
Job groups scheduled for Tumbleweed. Will mark the ticket as resolved when tomorrow's test runs are being scheduled correctly.
Updated by jlausuch almost 2 years ago
ph03nix wrote:
Job groups scheduled for Tumbleweed. Will mark the ticket as resolved when tomorrow's test runs are being scheduled correctly.
Since there has not been an new build and you are leaving on FTO, I'll take care!