Project

General

Profile

action #70774

coordination #80142: [saga][epic] Scale out: Redundant/load-balancing deployments of openQA, easy containers, containers on kubernetes

coordination #96263: [epic] Exclude certain Minion tasks from "Too many Minion job failures alert" alert

save_needle Minion tasks fail frequently

Added by mkittler over 1 year ago. Updated about 2 months ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Feature requests
Target version:
Start date:
2020-09-01
Due date:
% Done:

0%

Estimated time:
Difficulty:
Tags:

Description

Observation

The save_needle Minion task fails frequently on OSD and also sometimes on o3.

This can be observed using the following query parameters: https://openqa.suse.de/minion/jobs?state=failed&offset=0&task=save_needle
I'm going to remove most of these jobs to calm down the alert but right now 24 jobs have piled up over 2 month. However, the problem actually exists longer than 2 month but the failures have been manually cleaned up so far.

The problem here is always that the Git working tree is in a state which can not be handled by the task:

1.

  "result" => {
    "error" => "<strong>Failed to save addon_products-module-dev-tools-pvm-20200805.</strong><br><pre>Unable to commit via Git: On branch master\nYour branch is up to date with 'origin/master'.\n\nnothing to commit, working tree clean</pre>"
  },

2.

  "result" => {
    "error" => "<strong>Failed to save manually_add_profile-AppArmor-Chose-a-program-to-generate-a-profile-20200827.</strong><br><pre>Unable to reset repository to origin/master: error: cannot rebase: Your index contains uncommitted changes.\nerror: Please commit or stash them.</pre>"
  },

Suggestions

It would be useful if the task would be able to handle the problematic situations itself instead of requiring manual intervention. Note that the delete_needle task (which shares the same Git code) is also affected. We have likely less problems there because that task is not executed that often.

Problematic situations

  1. No diff has been produced which could be committed: Maybe that's simply when there's no actual change and we can simply return early in that case.
  2. The Git directory contains uncommitted changes: We could save these changes on a new branch before rebasing.
  3. We can not push the new commit because in the meantime new commits have been pushed to the remote from elsewhere: Just repeat the procedure.
  4. The fetch needles script is interfering.

Related issues

Related to openQA Project - action #33745: Improve handling external Git repositories (for needles)New2018-03-23

Related to openQA Infrastructure - action #61221: osd: unable to save needles, minion fails with "fatal: Unable to create '/var/lib/openqa/.../needles/.git/index.lock'"New2019-12-20

Related to openQA Infrastructure - action #98499: [alert] web UI: Too many Minion job failures alert size:SResolved2021-09-13

Has duplicate openQA Project - action #75070: save_needle minion task fails because "Your branch is ahead of 'origin/master'"Rejected2020-10-22

History

#1 Updated by mkittler over 1 year ago

  • Related to action #33745: Improve handling external Git repositories (for needles) added

#2 Updated by okurz over 1 year ago

  • Target version set to Ready

#3 Updated by mkittler over 1 year ago

  • Tags set to alert

#4 Updated by okurz over 1 year ago

  • Description updated (diff)
  • Category set to Feature requests

I think you once already proposed a solution to "repair" the git state. Can you reference that again so that we can think about what we can take from that?

#5 Updated by okurz over 1 year ago

  • Related to action #61221: osd: unable to save needles, minion fails with "fatal: Unable to create '/var/lib/openqa/.../needles/.git/index.lock'" added

#6 Updated by okurz over 1 year ago

  • Target version changed from Ready to future

#7 Updated by Xiaojing_liu about 1 year ago

  • Has duplicate action #75070: save_needle minion task fails because "Your branch is ahead of 'origin/master'" added

#8 Updated by mkittler 4 months ago

  • Related to coordination #96263: [epic] Exclude certain Minion tasks from "Too many Minion job failures alert" alert added

#9 Updated by okurz 4 months ago

  • Related to action #98499: [alert] web UI: Too many Minion job failures alert size:S added

#10 Updated by okurz 4 months ago

  • Parent task set to #96263

#11 Updated by cdywan about 2 months ago

From yesterday:

---
args:
- commit_message: ''
  imagedir: ''
  imagedistri: ''
  imagename: hexchat-23.png
  imageversion: ''
  job_id: 7760510
  needle_json: "{\r\n  \"area\": [\r\n    {\r\n      \"ypos\": 181,\r\n      \"type\":
    \"match\",\r\n      \"xpos\": 335,\r\n      \"click_point\": {\r\n        \"ypos\":
    18,\r\n        \"xpos\": 119\r\n      },\r\n      \"height\": 34,\r\n      \"width\":
    180\r\n    }\r\n  ],\r\n  \"properties\": [],\r\n  \"tags\": [\r\n    \"hexchat-nick-bernhard\"\r\n
    \ ]\r\n}"
  needledir: /var/lib/openqa/share/tests/sle/products/sle/needles
  needlename: hexchat-nick-bernhard--20211130
  overwrite: '1'
  user_id: 175
attempts: 1
children: []
created: 2021-11-30T11:20:59.50934Z
delayed: 2021-11-30T11:20:59.50934Z
expires: 2021-11-30T11:21:59.50934Z
finished: 2021-11-30T11:21:01.01916Z
id: 3536401
lax: 0
notes:
  gru_id: 30650350
parents: []
priority: 20
queue: default
result:
  error: "<strong>Failed to save hexchat-nick-bernhard--20211130.</strong><br><pre>Unable
    to commit via Git: On branch master\nYour branch is up to date with 'origin/master'.\n\nUntracked
    files:\n  (use \"git add <file>...\" to include in what will be committed)\n\tnautilus-1-20211102.json\n\tnautilus-1-20211102.png\n\tseahorse_sshkey-seahorse-display-sshkey-20211022.json\n\tseahorse_sshkey-seahorse-display-sshkey-20211022.png\n\nnothing
    added to commit but untracked files present (use \"git add\" to track)</pre>"
retried: ~
retries: 0
started: 2021-11-30T11:20:59.51323Z
state: failed
task: save_needle
time: 2021-12-01T10:16:52.2364Z
worker: 575

The error is this one:

Failed to save hexchat-nick-bernhard--20211130.

Unable to commit via Git: On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
(use "git add <file>..." to include in what will be committed)
    nautilus-1-20211102.json
    nautilus-1-20211102.png
    seahorse_sshkey-seahorse-display-sshkey-20211022.json
    seahorse_sshkey-seahorse-display-sshkey-20211022.png

    nothing added to commit but untracked files present (use "git add" to track)

Also available in: Atom PDF