Project

General

Profile

action #70774

Updated by okurz 11 months ago

## Observation ### observation

The `save_needle` Minion task fails frequently on OSD and also sometimes on o3.

This can be observed using the following query parameters: https://openqa.suse.de/minion/jobs?state=failed&offset=0&task=save_needle
I'm going to remove most of these jobs to calm down the alert but right now 24 jobs have piled up over 2 month. However, the problem actually exists longer than 2 month but the failures have been manually cleaned up so far.

The problem here is always that the Git working tree is in a state which can not be handled by the task:

1.
```
"result" => {
"error" => "<strong>Failed to save addon_products-module-dev-tools-pvm-20200805.</strong><br><pre>Unable to commit via Git: On branch master\nYour branch is up to date with 'origin/master'.\n\nnothing to commit, working tree clean</pre>"
},
```

2.
```
"result" => {
"error" => "<strong>Failed to save manually_add_profile-AppArmor-Chose-a-program-to-generate-a-profile-20200827.</strong><br><pre>Unable to reset repository to origin/master: error: cannot rebase: Your index contains uncommitted changes.\nerror: Please commit or stash them.</pre>"
},
```

## Suggestions ### suggestions

It would be useful if the task would be able to handle the problematic situations itself instead of requiring manual intervention. Note that the `delete_needle` task (which shares the same Git code) is also affected. We have likely less problems there because that task is not executed that often.

## Problematic problematic situations

1. No diff has been produced which could be committed: Maybe that's simply when there's no actual change and we can simply return early in that case.
2. The Git directory contains uncommitted changes: We could save these changes on a new branch before rebasing.
3. We can not push the new commit because in the meantime new commits have been pushed to the remote from elsewhere: Just repeat the procedure.
4. The fetch needles script is interfering.

Back