action #95105
closedcoordination #94258: [epic] deployment pipeline failed, alerts not handled
osd-deployment pipelines fail and alerts are not handled size:M
Description
Observation¶
https://gitlab.suse.de/openqa/osd-deployment/-/pipelines failed already last Wednesday and today and there was no reaction on the failed pipelines, see our alert handling processes as documented on https://progress.opensuse.org/projects/qa/wiki#Alert-handling
Acceptance criteria¶
- AC1: Team has been made aware about our alert handling process
Suggestions¶
- Make sure the team is aware about our alert handling process
- Find out why pipeline failure was not seen or reacted upon for the past days
- Do people have access to/ accounts for respective tools
- Is https://progress.opensuse.org/projects/qa/wiki#Onboarding-for-new-joiners complete? Did everyone go through the list?
- Can we have a checklist of things to go through e.g. ticket with items from the wiki per team member
Updated by livdywan about 3 years ago
- Subject changed from https://gitlab.suse.de/openqa/osd-deployment/-/pipelines fails and alerts not handled to https://gitlab.suse.de/openqa/osd-deployment/-/pipelines fails and alerts not handled size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by livdywan about 3 years ago
- Subject changed from https://gitlab.suse.de/openqa/osd-deployment/-/pipelines fails and alerts not handled size:M to osd-deployment pipelines fail and alerts are not handled size:M
- Status changed from Workable to In Progress
I'm preparing follow-up tickets, including #95188 so that this will be done in the sense that people are "aware" and tickets are on the backlog to address any remaining questions.
Updated by okurz about 3 years ago
- Related to action #95188: Document how to properly configure GitLab pipeline notifications size:M added
Updated by mkittler about 3 years ago
This is apparently a known problem, see https://stackoverflow.com/questions/46472631/notify-all-group-members-of-failed-pipelines-in-gitlab
So I've been adding myself on https://gitlab.suse.de/openqa/osd-deployment/-/services/pipelines_email/edit and monitor-o3. Let's see whether it works.
Updated by livdywan about 3 years ago
- Due date changed from 2021-07-07 to 2021-07-08
I couldn't find a way to properly copy a ticket with a checklist so honestly I've decided to wrap this up tomorrow, with a vague hope that I might be able to avoid manually adding every single step.
Updated by livdywan about 3 years ago
mkittler wrote:
This is apparently a known problem, see https://stackoverflow.com/questions/46472631/notify-all-group-members-of-failed-pipelines-in-gitlab
So I've been adding myself on https://gitlab.suse.de/openqa/osd-deployment/-/services/pipelines_email/edit and monitor-o3. Let's see whether it works.
Appreciated. But wrong ticket. I copied the comment to #95188.
Updated by livdywan about 3 years ago
- Status changed from In Progress to Feedback
Thanks to @tinita I learned that I can put all checklist item into a .txt
file and upload them with the misplaced-looking button - note the extension! I think this might be all for this. Already saw some of the tickets being taken and questions being asked.
Updated by livdywan about 3 years ago
- Status changed from Feedback to Resolved
I think we're good here, follow-ups obviously still in progress but that is expected since this is the planning ticket for it.