Project

General

Profile

Actions

action #116161

closed

regression in helm_K3S

Added by dimstar almost 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
Start date:
2022-09-02
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

The investigation jobs very clearly hint at this being a test regression:

containers_host_helm:investigate:retry: http://openqa.opensuse.org/t2603291
containers_host_helm:investigate:last_good_tests:33ac8322dc4e43a9d6411c07dd3d29b45652e4de: http://openqa.opensuse.org/t2603292
containers_host_helm:investigate:last_good_build:20220830: http://openqa.opensuse.org/t2603293

containers_host_helm:investigate:last_good_tests_and_build:33ac8322dc4e43a9d6411c07dd3d29b45652e4de+20220830: http://openqa.opensuse.org/t2603294

i.e. as soon as 'last_good_tests' is used, it passes, on current and past snapshot version. Last and current snapshot with latest test code fail.

Test log diff:

  • 9bf85f449 slem: switch to serial terminal and change registration flow
  • 87c87f203 Adjust autoyast variables on HPC profiles
  • 0d0ee5767 mariadb: change the order of the parameters to my_print_defaults
  • afcfd48af Move trufflehogs checks to its own branch
  • f7b493c81 Record soft failure if language isn't tranlated

  • 308f7cd55 Install SLE Micro as KVM Host

  • 89ae61f75 slem: change package installed to available in 5.2 and 5.3

  • 293f8b901 Add test to add maintenance test repos

  • 989532001 Move openssh template to /usr/lib/pam.d

  • 9186c604c fix SCC_PROXY_URL for online migration jobs

  • 0b19978d0 Add unit test for YAML schedule with reusable flows

  • f479265a9 armnn: Update path in archive used for onnx

  • 7a98a48b7 Fix timeout issue in java test

  • 2ca04bfbe Add Sles-12SP3-LTSS-Teradata repository

  • b4a33a0ae Add timeout to pkcon install command

  • e3533e2a4 Containers: Fix kubectl test

  • 061529841 Update ui-framework-documentation.md

  • 26277b9ce wicked: Adopt SAE tests for 0.6.70 final version

Off which "+ e3533e2a4 Containers: Fix kubectl test" seems to be the most relevant to this test suite

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-containers_host_helm@64bit fails in
helm_K3S

Test suite description

Maintainer: dheidler. Extra tests about CLI software in container module

Reproducible

Fails since (at least) Build 20220831

Expected result

Last good: 20220830 (or more recent)

Further details

Always latest result in this scenario: latest

Actions #1

Updated by jlausuch almost 2 years ago

  • Tags set to bug
  • Project changed from openQA Tests to Containers
  • Category deleted (Bugs in existing tests)
  • Status changed from New to Workable
  • Priority changed from Normal to High
Actions #2

Updated by ilausuch almost 2 years ago

  • Status changed from Workable to In Progress
  • Assignee set to ilausuch
Actions #3

Updated by favogt almost 2 years ago

FWICT the INSTALL_K3S_SKIP_START=true leads to the node being NotReady:

@@ -28,7 +21,7 @@
 # echo Logged into $(tty); echo 1XrKo-$?-
 Logged into /dev/hvc0
 1XrKo-0-
-# curl -sfL https://get.k3s.io | sh -s - --disable=metrics-server; echo 3BWsc-$?-
+# curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -s - --disable=metrics-server; echo Xq3DK-$?-
 [INFO]  Finding release for channel stable
 [INFO]  Using v1.24.4+k3s1 as release
 [INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.24.4+k3s1/sha256sum-amd64.txt
@@ -45,13 +38,23 @@
 [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
 [INFO]  systemd: Enabling k3s unit
 Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
-[INFO]  systemd: Starting k3s
-3BWsc-0-
+Xq3DK-0-
+# mkdir -p /etc/rancher/k3s; echo xYc23-$?-
+xYc23-0-
+# curl http://10.0.2.2:20203/v7lAp9QKSjprwjFs/data/containers/registries.yaml -o /etc/rancher/k3s/registries.yaml; echo 6vQeU-$?-
+  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
+                                 Dload  Upload   Total   Spent    Left  Speed
100    69  100    69    0     0   7296      0 --:--:-- --:--:-- --:--:--  7666
+6vQeU-0-
+# sed -E 's/REGISTRY/3.71.98.16:5000/' -i /etc/rancher/k3s/registries.yaml; echo GLL3p-$?-
+GLL3p-0-
+# systemctl --no-pager start k3s; echo CXkg~-$?-
+CXkg~-0-
 # timeout 30 test -e /etc/rancher/k3s/k3s.yaml; echo Hw9AV-$?-
 Hw9AV-0-
-# systemctl is-active k3s; echo RZeWc-$?-
+# systemctl --no-pager is-active k3s; echo pOlWT-$?-
 active
-RZeWc-0-
+pOlWT-0-
 # k3s -v; echo LudLa-$?-
 k3s version v1.24.4+k3s1 (c3f830e9)
 go version go1.18.1
@@ -60,8 +63,8 @@
 Linux susetest 5.19.2-1-default #1 SMP PREEMPT_DYNAMIC Thu Aug 18 05:06:49 UTC 2022 (6c252ef) x86_64 x86_64 x86_64 GNU/Linux
 RjRWF-0-
 # k3s kubectl get node; echo 58qXg-$?-
-NAME       STATUS   ROLES                  AGE   VERSION
-susetest   Ready    control-plane,master   20s   v1.24.4+k3s1
+NAME       STATUS     ROLES    AGE   VERSION
+susetest   NotReady   <none>   1s    v1.24.4+k3s1

I started some VRs without the commit, but there are unfortunately k3s download issues, so they fail...

Actions #4

Updated by favogt almost 2 years ago

Finally, the download worked.

Run just before the commit works: https://openqa.opensuse.org/tests/2603491
Run at the commit fails: https://openqa.opensuse.org/tests/2603377

Actions #5

Updated by ilausuch almost 2 years ago

Running locally worked

Running on opensuse.org
https://openqa.opensuse.org/tests/2603571 - ERROR

Investigating why exists is this discrepance

Actions #6

Updated by ilausuch almost 2 years ago

@favogt are you working on that? I can un-assign me

Actions #7

Updated by ilausuch almost 2 years ago

In this test I check the nodes are ready but the test is failing https://openqa.opensuse.org/tests/2603949

Actions #8

Updated by jlausuch almost 2 years ago

@pdostal and @ilausuch have proposed 2 fixes:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15468
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/15472

Both are tackling the issue in different levels, and both are valid, so I'm merging both.

Actions #9

Updated by jlausuch almost 2 years ago

  • Status changed from In Progress to Resolved
Actions

Also available in: Atom PDF