action #179038
closedcoordination #154777: [saga][epic] Shareable os-autoinst and test distribution plugins
coordination #162131: [epic] future version control related features in openQA
Gracious handling of longer remote git clones outages size:S
Description
Motivation¶
Currently, git_clone minion jobs fail when GitLab is temporarily unreachable (see #178492), by introducing a proper error-handling mechanism, we can ensure:
- Temporary outages do not cause unnecessary job failures or alerts.
User Story¶
"As a test engineer and openQA operator,
i want openQA to handle short-lived GitLab outages without causing mass Minion job failures,
so that users do not experience unnecessary disruption
Acceptance Criteria¶
- AC1: Temporary remote git outages don't cause failing minion jobs
- AC2: An update of remote git repositories is still ensured on shorter failed requests, e.g. in range of seconds
Suggestions¶
- Damage is likely limited. If we can't sync needles nobody can edit needles.
- Jobs end up incomplete if there's an on-going issue with git_clone minion jobs
- We could decide to eventually give up and continue anyway and let jobs run
Updated by robert.richardson 2 months ago
- Related to action #178492: [alert] Many failing `git_clone` Minion jobs auto_review:"Error detecting remote default branch name":retry size:S added
Updated by okurz 2 months ago
- Copied to action #179185: Detection of long-time remote git clone outages size:S added
Updated by robert.richardson 2 months ago
- Status changed from Workable to In Progress
Updated by openqa_review 2 months ago
- Due date set to 2025-04-08
Setting due date based on mean cycle time of SUSE QE Tools
Updated by livdywan about 2 months ago
- Status changed from In Progress to Workable
Updated by livdywan about 2 months ago
- Due date set to 2025-04-11
- Status changed from Workable to In Progress
Updated by robert.richardson about 1 month ago
- Status changed from In Progress to Feedback
Updated by livdywan about 1 month ago
- Copied to action #180863: Conduct lessons learned "Five Why" analysis for "Gracious handling of longer remote git clones outages" size:S added
Updated by livdywan about 1 month ago
- Status changed from Feedback to Resolved
Updated by tinita 19 days ago
- Status changed from Resolved to Workable
- Priority changed from Normal to High
https://openqa.suse.de/minion/jobs?id=15460188
result: |
Can't open file "/var/lib/openqa/git_server_outage.gitlab.suse.de.flag": Permission denied at /usr/share/openqa/script/../lib/OpenQA/Git/ServerAvailability.pm line 33.
Updated by nicksinger 19 days ago
this also affected users as can be seen in https://suse.slack.com/archives/C02CANHLANP/p1746528651080959 - I manually fixed it by executing chown geekotest:root /var/lib/openqa/ to allow the process to write into the folder itself - according to user reports, this was enough to make it work. Either we change the path or have to ensure /var/lib/openqa/ has the correct permission and/or owner
Updated by tinita 17 days ago
- Related to action #182021: [alert] web UI: Too many Minion job failures alert added
Updated by mkittler 17 days ago
- Status changed from In Progress to Feedback
nicksinger wrote in #note-21:
this also affected users as can be seen in https://suse.slack.com/archives/C02CANHLANP/p1746528651080959 - I manually fixed it by executing
chown geekotest:root /var/lib/openqa/to allow the process to write into the folder itself - according to user reports, this was enough to make it work. Either we change the path or have to ensure/var/lib/openqa/has the correct permission and/or owner
It looks like the effect of this manual fix is gone (probably because a package update reverted the ownership):
martchus@openqa:~> sudo -u geekotest touch /var/lib/openqa/foo
touch: cannot touch '/var/lib/openqa/foo': Permission denied
I created https://github.com/os-autoinst/openQA/pull/6436 to use the webui subdir which is actually what we want and it would be writable:
martchus@openqa:~> sudo -u geekotest touch /var/lib/openqa/foo
touch: cannot touch '/var/lib/openqa/foo': Permission denied
martchus@openqa:~> sudo -u geekotest touch /var/lib/openqa/webui/foo
martchus@openqa:~> echo $?
0
martchus@openqa:~> l /var/lib/openqa/webui/foo
-rw-r--r-- 1 geekotest nogroup 0 May 8 13:11 /var/lib/openqa/webui/foo
It is also already covered by our AppArmor profile.
Updated by mkittler 13 days ago
- Status changed from Feedback to Resolved
I checked select id, state, task, priority, args, created, started, retries from minion_jobs where task = 'git_clone' and state = 'failed' order by created desc limit 100; and there were not further instances of the problem. So there's nothing to clean up besides the one job left by @tinita which I have now also deleted.