QA » openQA Project » openQA Infrastructure

In the past we just accepted that we need to manually retry and have not found a better way.

First we need to get grenache #158041 and openqaworker-arm-1 #157816 back

I can reproduce the problem in https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2428083#L1075

Apparently the problem was a stuck process on monitor.qe.nue2.suse.org:

root      7403  0.0  0.1 1383888 23216 ?       Sl   Mar25   0:45      \_ /usr/bin/python3 /usr/bin/salt-minion
root     17455  0.0  0.0  40468  6556 ?        S    Mar25   0:00      |   \_ systemctl start dehydrated

I terminated that process and salt continued. I realized this because I ran salt --state-output=changes 'monitor*' state.test and found:

monitor.qe.nue2.suse.org:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 7403 and was started at 2024, Mar 25 08:37:35.942378 with jid 20240325083735942378
----------
    The function "state.highstate" is running as PID 15299 and was started at 2024, Mar 26 12:49:36.904865 with jid 20240326124936904865

After some time waiting until the system settled the multiple salt calls I could call salt --state-output=changes 'monitor*' state.test again.

So salt-states-openqa is fine again. If that happens again I suggest to look for stuck processes.

I will research a bit if that is something we can improve.

Again haven't found anything. Except for getting in contact with salt developers I don't see a magic solution falling from the sky.