openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logs
|Target version:||Current Sprint|
- Subject changed from openQA on osd fails with empty logs to openqaworker-arm-2 is out-of-space on /was: openQA on osd fails with empty logs
- Status changed from New to In Progress
- Assignee set to okurz
- Priority changed from High to Urgent
- Target version set to Current Sprint
I stopped salt-minion and openqa-worker.target. It looks like /var/lib/openqa/pool is on the same partition as / . I don't know what changed or how it looked like in before. Probably pool should be on NVME as well.
systemctl cat openqa_nvme_prepare.service creates the pool but does not seem to do anything with it. This looks similar to #53261 only about "pool", not "cache". Could it be we deleted the "pool" symlink by mistake and should use a bind mount as well? Probably to be done properly with salt.
- change to bind mount for all,
- add that to salt
- add -3 the same and monitor all three
Done first two with https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/160
today we have some incompletes on aarch64 but seems like only openqaworker-arm-2. I disabled the worker target on the host and will retrigger incompletes. They should be picked up on openqaworker-arm-1. See e.g. https://openqa.suse.de/tests/3326179 from https://openqa.suse.de/tests/?&resultfilter=Incomplete
- Status changed from In Progress to Resolved
all problems resolved. The nvme preparation is done as available in salt and a workaround for nscd is applied, see https://gitlab.suse.de/openqa/salt-states-openqa/merge_requests/162 . The worker was able to successfully test build 0307 of SLES12SP5 so we should be good.