QA (public) » openQA Project (public) » openQA Infrastructure (public)

okurz wrote in #note-7:

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1336

merged. Interestingly in your deployment job, monitor is missing again. The associated salt_highstate.log artifact however mentions it somehow:

2025-01-14 16:41:51,723 [salt.client      :1167][DEBUG   ][12951] get_iter_returns for jid 20250114164151566706 sent to {'…-prg4.qa.suse.cz', 'grenache-1.oqa.prg2.suse.org', 'baremetal-support.qe.nue2.suse.org', 'monitor.qe.nue2.suse.org', 'storage.qe.prg2.suse.org', , …'} will timeout at 16:43:21.723622

In a following MR deployment job, monitor shows up with the expected issue. However, for some reason OSD timed out. I looked up the job (with salt-run jobs.lookup_jid 20250114193508323824) and found:

Summary for openqa.suse.de
--------------
Succeeded: 629 (changed=117)
Failed:      0
--------------
Total states run:     629
Total run time:   172.592 s
[…]
Summary for monitor.qe.nue2.suse.org
--------------
Succeeded: 456 (changed=4)
Failed:      1
--------------
Total states run:     457
Total run time:   109.676 s
[…]
Summary for backup-qam.qe.nue2.suse.org
--------------
Succeeded: 286
Failed:      0
--------------
Total states run:     286
Total run time:   161.726 s

so either your change did not apply correctly or we need to bump it even more because it includes some more overhead.

For the initial issue of the missing salt state requirement, I created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1338

I wanted to check what would be a good value to but it even more and looked into https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/3666125

$ salt-run jobs.lookup_jid 20250116085031017432
Summary for petrol.qe.nue2.suse.org
--------------
Succeeded: 423 (changed=2)
Failed:      1
--------------
Total states run:     424
Total run time:  3219.013 s
[…]
Summary for diesel.qe.nue2.suse.org
--------------
Succeeded: 422 (changed=2)
Failed:      1
--------------
Total states run:     423
Total run time:  1795.992 s

guess only raising timeouts won't cut it.

I also observed that diesel+petrol are excessively slow. Reported #175629 for this specific issue

ybonatakis suggested to look into "gather_job_timeout" as well however from https://docs.saltproject.io/en/latest/ref/configuration/master.html#gather-job-timeout this does not sound related.

I tried

sudo nice env runs=30 count-fail-ratio salt --state-output=changes -C "G@roles:worker and G@osarch:ppc64le" state.apply queue=True | grep -v 'Result.*Clean'

but that failed. nicksinger is currently looking into syntax errors for #162296. Waiting for his progress then before trying again.

merged https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1343

I will merge this now but will still test different sets of parameters after current syntax issues are fixed by @nicksinger

So far many failures, current

## count-fail-ratio: Run: 12. Fails: 12. Fail ratio 100.00±0%
## mean runtime: 3498995±415554.03 ms

that is 3,499s which are nearly 1h. looking into individual failures