action #138746
closed
[tools] s390x VM randomly fails to open QCOW disk image: Permission denied
Added by MDoucha 6 months ago.
Updated 6 months ago.
- Target version set to future
We took a brief look. We weren't clear where exactly the images are stored - it's not the cache which is separate, and is being freed as can be seen in the logs. So likely it's not critical right now - but please let us know if it happens more frequently and add more details.
The path from the error message is stored on the svirt host, which is separate from the worker machine. The disk image files get rsynced from the worker cache to the svirt host via network.
- Subject changed from s390x VM randomly fails to open QCOW disk image: Permission denied to [kernel] s390x VM randomly fails to open QCOW disk image: Permission denied
- Assignee set to mkittler
@mkittler very likely related to your work on the svirt asset cache
- Status changed from New to Feedback
Then it is likely best to disable the feature again: https://github.com/os-autoinst/os-autoinst/pull/2401
Considering all the problems we've encountered so far it is probably not worth it. One can still enable it for tests where it can actually be used.
Note that the permission denied error could have a different cause at this point it likely doesn't make much sense to investigate anymore and just disable the feature. Otherwise, on every bug related to the asset copying I would have to be involved again. And probably it is in fact the feature (because maybe rsync behaves slightly different when source and destination are on different hosts?).
- Subject changed from [kernel] s390x VM randomly fails to open QCOW disk image: Permission denied to [tools] s390x VM randomly fails to open QCOW disk image: Permission denied
- Status changed from Feedback to New
- Target version changed from future to Ready
mkittler wrote in #note-5:
Then it is likely best to disable the feature again: https://github.com/os-autoinst/os-autoinst/pull/2401
Considering all the problems we've encountered so far it is probably not worth it. One can still enable it for tests where it can actually be used.
Note that the permission denied error could have a different cause at this point it likely doesn't make much sense to investigate anymore and just disable the feature. Otherwise, on every bug related to the asset copying I would have to be involved again. And probably it is in fact the feature (because maybe rsync behaves slightly different when source and destination are on different hosts?).
I would not underestimate the benefit of the feature given that for long there were various problems and performance bottlenecks in this area. I guess we will have to adopt this ticket into the scope of "[tools]" then.
okurz wrote in #note-6:
mkittler wrote in #note-5:
Note that the permission denied error could have a different cause at this point it likely doesn't make much sense to investigate anymore and just disable the feature. Otherwise, on every bug related to the asset copying I would have to be involved again. And probably it is in fact the feature (because maybe rsync behaves slightly different when source and destination are on different hosts?).
We can always re-run jobs with the setting flipped to confirm if a case is related. It could even be done in investigation jobs. Assuming the jobs are otherwise stable.
- Status changed from New to Feedback
Assuming the jobs are otherwise stable.
Unfortunately, problems with this copy command were often sporadic.
For now I'd just keep it disabled and test developers might still decide themselves whether they want to enable the optimization.
- Status changed from Feedback to New
Please link the according ticket about bringing in the svirt worker cache and make sure there is an open ticket about having a reliable efficient cache approach for svirt workers by default. I am pretty sure we have a ticket about that request originally. Then we can resolve here because the "Permission denied" problem has been "solved".
- Status changed from New to Resolved
I am pretty sure we have a ticket about that request originally.
I definitely have not worked on a ticket when coming up with https://github.com/os-autoinst/os-autoinst/pull/2357. The PR was based on an idea that came up in chat. However, it looks like this ticket is the one you're looking for: #44468
I'll update that ticket to reflect recent developments.
Also available in: Atom
PDF