Project

General

Profile

Actions
Copy link

action #135335

closed

[tools] gitlabci salt-pillars-openqa deploy failed on imagetester and other hosts size:M

Added by osukup about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-09-07
Due date:
% Done:

0%

Estimated time:
Tags:
infra

Description

Observation

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1810907

from log:

        ID: wicked ifup br1
    Function: cmd.run
      Result: False
     Comment: Command "wicked ifup br1" run
     Started: 07:07:10.803006
    Duration: 30119.464 ms
     Changes:   
              ----------
              pid:
                  16955
              retcode:
                  157
              stderr:
              stdout:
                  br1             no-device
  Name: /etc/sysconfig/network/ifcfg-tap0 - Function: file.managed - Result: Clean Started: - 07:07:40.934647 Duration: 7.021 ms
  Name: /etc/sysconfig/network/ifcfg-tap64 - Function: file.managed - Result: Clean Started: - 07:07:40.945434 Duration: 5.152 ms
  Name: /etc/sysconfig/network/ifcfg-tap128 - Function: file.managed - Result: Clean Started: - 07:07:40.954239 Duration: 5.0 ms
  Name: /etc/sysconfig/network/ifcfg-tap1 - Function: file.managed - Result: Clean Started: - 07:07:40.962885 Duration: 5.013 ms
  Name: /etc/sysconfig/network/ifcfg-tap65 - Function: file.managed - Result: Clean Started: - 07:07:40.971581 Duration: 5.002 ms
  Name: /etc/sysconfig/network/ifcfg-tap129 - Function: file.managed - Result: Clean Started: - 07:07:40.980234 Duration: 4.984 ms
  Name: /etc/sysconfig/network/ifcfg-tap2 - Function: file.managed - Result: Clean Started: - 07:07:40.988915 Duration: 5.037 ms
  Name: /etc/sysconfig/network/ifcfg-tap66 - Function: file.managed - Result: Clean Started: - 07:07:40.997624 Duration: 4.987 ms
  Name: /etc/sysconfig/network/ifcfg-tap130 - Function: file.managed - Result: Clean Started: - 07:07:41.006362 Duration: 4.946 ms
  Name: /etc/sysconfig/network/ifcfg-tap3 - Function: file.managed - Result: Clean Started: - 07:07:41.015198 Duration: 5.266 ms
  Name: /etc/sysconfig/network/ifcfg-tap67 - Function: file.managed - Result: Clean Started: - 07:07:41.024299 Duration: 5.178 ms
  Name: /etc/sysconfig/network/ifcfg-tap131 - Function: file.managed - Result: Clean Started: - 07:07:41.033154 Duration: 4.992 ms
  Name: /etc/sysconfig/network/ifcfg-tap4 - Function: file.managed - Result: Clean Started: - 07:07:41.041806 Duration: 4.955 ms
  Name: /etc/sysconfig/network/ifcfg-tap68 - Function: file.managed - Result: Clean Started: - 07:07:41.050532 Duration: 5.287 ms
  Name: /etc/sysconfig/network/ifcfg-tap132 - Function: file.managed - Result: Clean Started: - 07:07:41.059443 Duration: 4.926 ms
  Name: /etc/sysconfig/network/ifcfg-tap5 - Function: file.managed - Result: Clean Started: - 07:07:41.068081 Duration: 4.993 ms
  Name: /etc/sysconfig/network/ifcfg-tap69 - Function: file.managed - Result: Clean Started: - 07:07:41.076758 Duration: 4.93 ms
  Name: /etc/sysconfig/network/ifcfg-tap133 - Function: file.managed - Result: Clean Started: - 07:07:41.085353 Duration: 4.942 ms
  Name: /etc/sysconfig/network/ifcfg-tap6 - Function: file.managed - Result: Clean Started: - 07:07:41.093943 Duration: 5.056 ms
  Name: /etc/sysconfig/network/ifcfg-tap70 - Function: file.managed - Result: Clean Started: - 07:07:41.102645 Duration: 4.935 ms
  Name: /etc/sysconfig/network/ifcfg-tap134 - Function: file.managed - Result: Clean Started: - 07:07:41.111287 Duration: 4.987 ms
  Name: /etc/sysconfig/network/ifcfg-tap7 - Function: file.managed - Result: Clean Started: - 07:07:41.119942 Duration: 4.913 ms
  Name: /etc/sysconfig/network/ifcfg-tap71 - Function: file.managed - Result: Clean Started: - 07:07:41.128614 Duration: 4.959 ms
  Name: /etc/sysconfig/network/ifcfg-tap135 - Function: file.managed - Result: Clean Started: - 07:07:41.137410 Duration: 4.953 ms
  Name: /etc/sysconfig/network/ifcfg-tap8 - Function: file.managed - Result: Clean Started: - 07:07:41.146176 Duration: 4.935 ms
  Name: /etc/sysconfig/network/ifcfg-tap72 - Function: file.managed - Result: Clean Started: - 07:07:41.154807 Duration: 5.035 ms
  Name: /etc/sysconfig/network/ifcfg-tap136 - Function: file.managed - Result: Clean Started: - 07:07:41.163660 Duration: 4.937 ms
  Name: /etc/sysconfig/network/ifcfg-tap9 - Function: file.managed - Result: Clean Started: - 07:07:41.172266 Duration: 4.954 ms
  Name: /etc/sysconfig/network/ifcfg-tap73 - Function: file.managed - Result: Clean Started: - 07:07:41.181001 Duration: 4.95 ms
  Name: /etc/sysconfig/network/ifcfg-tap137 - Function: file.managed - Result: Clean Started: - 07:07:41.189605 Duration: 5.503 ms

Acceptance Criteria

  • AC1: Salt states apply successfully on imageworker

Suggestions

  • Investigate if one of the service defs is missing a "requires" or similar
  • Commands were re-run - consider persistent mitigations if this is causing other pipelines to fail
  • This seems to affect imagetester, openqaworker17.qa.suse.cz, openqaworker16.qa.suse.cz and openqaworker18.qa.suse.cz so far

Related issues 1 (0 open — 1 closed)

Related to openQA Infrastructure - action #135137: Bring back imagetester size:MResolvedokurz2023-09-04

Actions
Actions
Copy link
 #1

Updated by osukup about 1 year ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by okurz about 1 year ago

Actions
Copy link
 #3

Updated by okurz about 1 year ago

  • Target version set to Ready
Actions
Copy link
 #4

Updated by livdywan about 1 year ago

  • Subject changed from [tools] gitlabci salt-pillars-openqa deploy failed on imagetester to [tools] gitlabci salt-pillars-openqa deploy failed on imagetester size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #5

Updated by tinita about 1 year ago

This also fails on other hosts, but I don't know how to find out which hosts:
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1814057/raw
I can see that message three times, but don't know where to search for the actual worker

Actions
Copy link
 #6

Updated by tinita about 1 year ago

  • Subject changed from [tools] gitlabci salt-pillars-openqa deploy failed on imagetester size:M to [tools] gitlabci salt-pillars-openqa deploy failed on imagetester and other hosts size:M
Actions
Copy link
 #7

Updated by tinita about 1 year ago

  • Priority changed from Normal to High
Actions
Copy link
 #8

Updated by livdywan about 1 year ago

tinita wrote in #note-5:

This also fails on other hosts, but I don't know how to find out which hosts:
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1814057/raw
I can see that message three times, but don't know where to search for the actual worker

Summary for openqaworker17.qa.suse.cz
--------------
Succeeded: 388 (changed=55)
Failed:      1

Summary for openqaworker16.qa.suse.cz
--------------
Succeeded: 388 (changed=55)
Failed:      1

Summary for openqaworker18.qa.suse.cz
--------------
Succeeded: 388 (changed=55)
Failed:      1
Actions
Copy link
 #9

Updated by livdywan about 1 year ago

  • Description updated (diff)
Actions
Copy link
 #10

Updated by okurz about 1 year ago

  • Status changed from Workable to Resolved
  • Assignee set to okurz

Right now the salt state could successfully applied to all salt controlled machines including the ones mentioned in the description

Actions
Copy link

Also available in: Atom PDF