QA » openQA Project

I saw errors in the salt-minion systemctl status on e.g. worker30 like

Jun 15 23:14:18 worker30 salt-minion[3796]: [WARNING ] test.ping args: []
Jun 15 23:30:36 worker30 salt-minion[3796]: [WARNING ] remote: "worker-arm1" found in workerconf.sls but not in salt mine, host currently offline?
Jun 15 23:30:36 worker30 salt-minion[3796]: [WARNING ] remote: "worker-arm2" found in workerconf.sls but not in salt mine, host currently offline?
Jun 15 23:30:36 worker30 salt-minion[3796]: [WARNING ] remote: "worker29" found in workerconf.sls but not in salt mine, host currently offline?
Jun 15 23:30:36 worker30 salt-minion[3796]: [WARNING ] remote: "worker32" found in workerconf.sls but not in salt mine, host currently offline?
Jun 15 23:30:37 worker30 salt-minion[3796]: [WARNING ] remote: "worker36" found in workerconf.sls but not in salt mine, host currently offline?
Jun 15 23:30:37 worker30 salt-minion[3796]: [WARNING ] remote: "worker37" found in workerconf.sls but not in salt mine, host currently offline?
Jun 15 23:30:37 worker30 salt-minion[3796]: [WARNING ] remote: "worker38" found in workerconf.sls but not in salt mine, host currently offline?
Jun 15 23:30:37 worker30 salt-minion[3796]: [WARNING ] remote: "worker39" found in workerconf.sls but not in salt mine, host currently offline?
Jun 15 23:30:37 worker30 salt-minion[3796]: [WARNING ] remote: "worker40" found in workerconf.sls but not in salt mine, host currently offline?

I added back worker31 to salt and applied a salt high state and then retriggered failing scripts-ci jobs and retriggered jobs like https://openqa.suse.de/tests/14634921 and they seem to work fine. But the warnings are still there.

failed_since="2024-06-14 14:00Z" result="result='parallel_failed'" host=openqa.suse.de comment="label:poo162320" openqa-advanced-retrigger-jobs

Setting due date based on mean cycle time of SUSE QE Tools

#162332

done. Many jobs scheduled. Monitoring https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1&viewPanel=24&from=now-7d&to=now and https://monitor.qa.suse.de/d/nRDab3Jiz/openqa-jobs-test?orgId=1

I disabled https://gitlab.suse.de/openqa/scripts-ci/-/pipeline_schedules/167/ as it's often running into timeout waiting for jobs to be picked up

$ openqa-query-for-job-label poo#162320
14657729|2024-06-19 04:45:41|done|failed|cc_ipsec_client||worker40
14674275|2024-06-19 02:02:44|done|failed|cc_ipsec_client||worker40
14672589|2024-06-19 01:12:49|done|failed|cc_ipsec_client||worker40
14654972|2024-06-18 14:38:30|done|failed|cc_ipsec_client||worker40
14663615|2024-06-18 09:49:09|done|failed|cc_ipsec_client||worker40
14660134|2024-06-18 08:14:01|done|failed|cc_ipsec_client||worker40
14654683|2024-06-17 23:52:41|done|failed|cc_ipsec_client||worker40
14651378|2024-06-17 14:36:22|done|failed|cc_ipsec_client||worker-arm1
14640403|2024-06-17 14:16:41|done|failed|cc_ipsec_client||worker-arm2
14641247|2024-06-17 14:12:47|done|failed|sles4sap_nw_node02||petrol

I found https://openqa.suse.de/tests/14657729#step/setup_multimachine/116 failing because ping actually does not exist. Let's make the fail_messages more specific:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19557

rollback actions conducted. jobs with label look okayish. There are other related tasks for follow-up including #162374 which would bring back more workers.