action #133793
Updated by mkittler over 1 year ago
## Observation See https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1734178 running into the 2h gitlab CI timeout while applying a salt high state. There is a lot of not helpful debug output with all the lines with "Result: Clean - Started:" and a mention of hosts being down "backup.qa.suse.de" and "openqaworker1.qe.nue2.suse.org" but it's not being clear which minions in the end do not return ## Acceptance criteria * **AC1:** By default no lines with "Result: Clean - Started:": Put them in another logfile to be uploaded * **AC2:** No repeated "++ true, ++ sleep 1, ++ echo -n .": * **AC3:** We know which minions did not complete ## Suggestions * For "Result: Clean - Started:" how about just grep for those and move to a logfile? https://unix.stackexchange.com/questions/414042/how-to-split-an-output-to-two-files-with-grep can help * okurz just used ``` sudo salt --no-color --state-output=changes 'backup-qam.qe.nue2.suse.org' state.apply queue=True | awk '/Result: Clean - Started/ {print > "/tmp/salt_profiling.log"; next} 1' ``` which provides nice terse output and all the profiling information into /tmp/salt_profiling.log * Maybe don't apply the "set -x" for those commands with the dot outputting ## Out of scope * Timeout before the 2h gitlab CI timeout and write down which minions are still busy executing jobs -> #133457