QA » openQA Project » openQA Infrastructure

Having salt directly storing output into file will store it on the remote server, so in case of failure we would still need to download them. Instead, I tried to store the output form the ssh which should provide more or less the same: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/798

The whole raw log is nicely served by gitlab, for example the latest artifact right now: https://openqa.io.suse.de/-/salt-states-openqa/-/jobs/1417570/artifacts/salt_highstate.log

The reason is in the header:

The script exceeded the maximum execution time set for the job

So the post-fail hook uploading the artifact did not have chance to run.

If the long running time was expected, we should tweak the timeout (Settings -> CI/CD -> General pipelines -> Timeout); if there is some unwanted deadlock we want to investigate, we can sacrifice some verbosity (by piping the output through grep).

From what I see in the raw log, some operations took surprisingly long time like:

          ID: /etc/zypp/zypp.conf
    Function: ini.options_present
      Result: True
     Comment: No anomaly detected
     Started: 09:27:57.362744
    Duration: 19921.266 ms
     Changes:   

The ideal solution is apparently still a wanted but not implemented feature: https://gitlab.com/gitlab-org/gitlab/-/issues/284186

From salt --help there is:

    -l LOG_LEVEL, --log-level=LOG_LEVEL
                        Console logging log level. One of 'all', 'garbage',
                        'trace', 'debug', 'profile', 'info', 'warning',
                        'error', 'critical', 'quiet'. Default: 'warning'.
    --log-file=LOG_FILE
                        Log file path. Default: '/var/log/salt/master'.
    --log-file-level=LOG_LEVEL_LOGFILE
                        Logfile logging log level. One of 'all', 'garbage',
                        'trace', 'debug', 'profile', 'info', 'warning',
                        'error', 'critical', 'quiet'. Default: 'warning'.
…
    --state-output=STATE_OUTPUT, --state_output=STATE_OUTPUT
                        Override the configured state_output value for minion
                        output. One of 'full', 'terse', 'mixed', 'changes' or
                        'filter'. Default: 'none'.

so maybe we can use ssh "salt … --log-file=/dev/stdout --log-file-level=debug --state-output=changes" > log and upload log as gitlab artifact and show stderr in the gitlab console output window (or vice-versa). If piping to /dev/stdout or similar isn't possible then save to an explicit file which is copied over at the end of execution.

Maybe https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/807 can help.

I had a typo in the --state-output parameter value, https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/812 should fix that.