action #131465
closedMake temporary files and directories created by openQA services easier to identify size:M
Description
Motivation¶
While tracking down the root cause for #131447 we encountered a lot of error messages like:
Jun 27 09:10:06 openqa openqa-gru[22285]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: No space left on device
Jun 27 09:10:37 openqa openqa-gru[22774]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: No space left on device
Jun 27 09:11:27 openqa openqa-gru[23797]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: No space left on device
Jun 27 09:13:38 openqa openqa-gru[26670]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: No space left on device
Jun 27 09:13:38 openqa openqa-gru[26684]: mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: No space left on device
These are very hard to trace back to the code that triggered them. If we simply used more unique templates like
/tmp/tmp.webui.obs.rsync.XXXXXXXXXX and the like it would be much easier.
Acceptance criteria¶
- AC1: All temporary files/directories created by openQA including use unique templates
Suggestions¶
- The underlying Perl modules usually have options for setting templates (https://metacpan.org/pod/Mojo::File#tempdir)
- Review all occurences of using temp files and directories in os-autoinst and openQA and decide where we need to add a unique name template
- Also review https://github.com/os-autoinst/scripts/
- Optional: Fix where stuff is not properly cleaned up
Updated by kraih 10 months ago
- Related to action #131447: Some jobs incomplete due to auto_review:"api failure: 400.*/tmp/.*png.*No space left on device.*Utils.pm line 285":retry but enough space visible on machines added
Updated by tinita 10 months ago
Since I was already looking at this part of the code: https://github.com/os-autoinst/openQA/pull/5225 Use a unique template for worker tempdir
Updated by tinita 10 months ago
- Status changed from New to In Progress
https://github.com/os-autoinst/openQA/pull/5225 merged and deployed.
There's still an mktemp without a template in openqa-label-known-issues that needs to be addressed.
Updated by openqa_review 10 months ago
- Due date set to 2023-07-13
Setting due date based on mean cycle time of SUSE QE Tools
Updated by tinita 10 months ago
- Related to action #131471: Leftover worker temporary directories in /tmp on OSD and O3 size:M added
Updated by tinita 10 months ago
https://github.com/os-autoinst/openQA/pull/5229 Improve tempdir template uniqueness
-> merged
Updated by tinita 10 months ago
https://github.com/os-autoinst/scripts/pull/239 Use a unique tempfile template in openqa-label-known-issues
Now I only need to check why the runcli-errorlog-XXXX tempfiles aren't cleaned up.
errorlog=$(mktemp -t runcli-errorlog-XXXX)
# shellcheck disable=SC2064
trap "rm $errorlog" EXIT
This code works in a small example script, but apparently not from openqa-investigate.
Updated by tinita 10 months ago
https://github.com/os-autoinst/scripts/pull/243 Use more unique tempfile templates (merged)
Updated by tinita 10 months ago
- Status changed from In Progress to Feedback
https://github.com/os-autoinst/scripts/pull/245 Fix error handling with trap
Updated by tinita 10 months ago
https://github.com/os-autoinst/os-autoinst/pull/2338 Improve tempfiles