QA » openQA Project

Observation¶

The save_needle Minion task fails frequently on OSD and also sometimes on o3.

This can be observed using the following query parameters: https://openqa.suse.de/minion/jobs?state=failed&offset=0&task=save_needle
I'm going to remove most of these jobs to calm down the alert but right now 24 jobs have piled up over 2 month. However, the problem actually exists longer than 2 month but the failures have been manually cleaned up so far.

The problem here is always that the Git working tree is in a state which can not be handled by the task:

1.

  "result" => {
    "error" => "<strong>Failed to save addon_products-module-dev-tools-pvm-20200805.</strong><br><pre>Unable to commit via Git: On branch master\nYour branch is up to date with 'origin/master'.\n\nnothing to commit, working tree clean</pre>"
  },

2.

  "result" => {
    "error" => "<strong>Failed to save manually_add_profile-AppArmor-Chose-a-program-to-generate-a-profile-20200827.</strong><br><pre>Unable to reset repository to origin/master: error: cannot rebase: Your index contains uncommitted changes.\nerror: Please commit or stash them.</pre>"
  },

Suggestions¶

It would be useful if the task would be able to handle the problematic situations itself instead of requiring manual intervention. Note that the delete_needle task (which shares the same Git code) is also affected. We have likely less problems there because that task is not executed that often.

Problematic situations¶

1. No diff has been produced which could be committed: Maybe that's simply when there's no actual change and we can simply return early in that case.
2. The Git directory contains uncommitted changes: We could save these changes on a new branch before rebasing.
3. We can not push the new commit because in the meantime new commits have been pushed to the remote from elsewhere: Just repeat the procedure.
4. The fetch needles script is interfering.