QA » openQA Project » openQA Infrastructure

And this also breaks https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2510604

but https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/789 is still open

ybonatakis wrote in #note-12:

but https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/789 is still open

Did you see the unanswered questions there?

I found the following entries with inconsistencies:

1. malbec, you did not change the password, please revert
2. imagetester, should be changed
3. storage, should be changed
4. kerosene, should be changed
5. openqaworker{20..28}, should be changed
6. openqaworker-arm{21..22}, should be changed

I addressed your question in https://suse.slack.com/archives/C02AJ1E568M/p1713963938332169?thread_ts=1713940157.475059&cid=C02AJ1E568M and deleted the stale alerts with:

sqlite3 /var/lib/grafana/grafana.db "$(for RULEID in DzAhcifVk dzA25mfVk Fk0h5iBVk Sk02ciBVk VzA2cif4zz; do echo -n "delete from alert_rule where uid = '$RULEID'; delete from alert_rule_version where rule_uid = '$RULEID'; delete from provenance_type where record_key = '$RULEID'; delete from annotation where text like '%$RULEID%';"; done)"

https://sd.suse.com/servicedesk/customer/portal/1/SD-155399

@ybonatakis could you please share the SD ticket with our "OSD Admins" group so we can see the progress? Also, can you maybe ask someone from the team to make the required changes? I don't think it is useful if you have to wait several days for a response. Especially for a "Workable" ticket with "High" Priority…

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/800
the only one machine which password havent update is kerosene

I just checked the CI jobs and there are not problems since the reopening.
It is 4 days since. The machine was restarted 3 days before

 12:59:02  up 3 days 13:54,  1 user,  load average: 1.39, 1.49, 1.31

most likely from https://gitlab.suse.de/openqa/grafana-webhook-actions/-/jobs/2596329
but I cant tell if Oliver triggered this by himself or if it was automation.
therefore I assume that the recovery works.

livdywan wrote in #note-30:

openqaworker-arm-1.qe.nue2.suse.org:
    Minion did not return. [No response]
    The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:

    salt-run jobs.lookup_jid 20240510052716652055

Is this machine broken again?

See https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2589906

ybonatakis wrote in #note-31:

I just checked the CI jobs and there are not problems since the reopening.
It is 4 days since. The machine was restarted 3 days before

 12:59:02  up 3 days 13:54,  1 user,  load average: 1.39, 1.49, 1.31

most likely from https://gitlab.suse.de/openqa/grafana-webhook-actions/-/jobs/2596329
but I cant tell if Oliver triggered this by himself or if it was automation.
therefore I assume that the recovery works.

I have not triggered a recovery manually recently so I concur, the automatic recovery works in principle. https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2589906 failed as openqaworker-arm-1 did not respond to salt commands in time. That was also what we saw in https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2589864#L326 in the pre-deploy check which isn't fatal but just informative. I am not sure how we handled the temporary outages of openqaworker-arm-[123] in the past during osd-deployment. We might have not really seen the problem in before and were possibly just lucky/unlucky? I will look into some older tickets if something comes up.

EDIT: Nothing came up from older tickets so except for "moar retries!!!1" covering the recovery periods of openqaworker-arm-1 I don't have further good ideas.

EDIT2: https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2589906 was running at 0727. There are already retries by gitlab with retry: 2 but those retries are quickly one after another. The previous jobs were https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2589904 0725 and https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2589872 at 0723 so only 4 minutes, not enough to allow openqaworker-arm-1 to be up again. We could try retries with customized retry periods. Will propose a solution.

https://gitlab.suse.de/openqa/osd-deployment/-/merge_requests/57 merged. I suggest to await at least one succesful normal osd-deployment regardless if a machine like openqaworker-arm-1 is not currently available during the deployment.