Project

General

Profile

Actions
Copy link

action #95105

closed

coordination #94258: [epic] deployment pipeline failed, alerts not handled

osd-deployment pipelines fail and alerts are not handled size:M

Added by okurz about 3 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
livdywan
Category:
Organisational
Target version:
Ready
Start date:
2021-07-06
Due date:
2021-07-08
% Done:

0%

Estimated time:

Description

Observation

https://gitlab.suse.de/openqa/osd-deployment/-/pipelines failed already last Wednesday and today and there was no reaction on the failed pipelines, see our alert handling processes as documented on https://progress.opensuse.org/projects/qa/wiki#Alert-handling

Acceptance criteria

  • AC1: Team has been made aware about our alert handling process

Suggestions

  • Make sure the team is aware about our alert handling process
  • Find out why pipeline failure was not seen or reacted upon for the past days
  • Do people have access to/ accounts for respective tools
  • Is https://progress.opensuse.org/projects/qa/wiki#Onboarding-for-new-joiners complete? Did everyone go through the list?
  • Can we have a checklist of things to go through e.g. ticket with items from the wiki per team member

Related issues 1 (0 open — 1 closed)

Related to openQA Project - action #95188: Document how to properly configure GitLab pipeline notifications size:MResolvedtinita2021-07-072021-09-01

Actions
Actions
Copy link
 #1

Updated by livdywan about 3 years ago

  • Subject changed from https://gitlab.suse.de/openqa/osd-deployment/-/pipelines fails and alerts not handled to https://gitlab.suse.de/openqa/osd-deployment/-/pipelines fails and alerts not handled size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #2

Updated by livdywan about 3 years ago

  • Subject changed from https://gitlab.suse.de/openqa/osd-deployment/-/pipelines fails and alerts not handled size:M to osd-deployment pipelines fail and alerts are not handled size:M
  • Status changed from Workable to In Progress

I'm preparing follow-up tickets, including #95188 so that this will be done in the sense that people are "aware" and tickets are on the backlog to address any remaining questions.

Actions
Copy link
 #3

Updated by okurz about 3 years ago

  • Related to action #95188: Document how to properly configure GitLab pipeline notifications size:M added
Actions
Copy link
 #5

Updated by mkittler about 3 years ago

Actions
Copy link
 #9

Updated by livdywan about 3 years ago

  • Due date changed from 2021-07-07 to 2021-07-08

I couldn't find a way to properly copy a ticket with a checklist so honestly I've decided to wrap this up tomorrow, with a vague hope that I might be able to avoid manually adding every single step.

Actions
Copy link
 #10

Updated by livdywan about 3 years ago

mkittler wrote:

This is apparently a known problem, see https://stackoverflow.com/questions/46472631/notify-all-group-members-of-failed-pipelines-in-gitlab
So I've been adding myself on https://gitlab.suse.de/openqa/osd-deployment/-/services/pipelines_email/edit and monitor-o3. Let's see whether it works.

Appreciated. But wrong ticket. I copied the comment to #95188.

Actions
Copy link
 #15

Updated by livdywan about 3 years ago

  • Status changed from In Progress to Feedback

Thanks to @tinita I learned that I can put all checklist item into a .txt file and upload them with the misplaced-looking button - note the extension! I think this might be all for this. Already saw some of the tickets being taken and questions being asked.

Actions
Copy link
 #16

Updated by livdywan about 3 years ago

  • Status changed from Feedback to Resolved

I think we're good here, follow-ups obviously still in progress but that is expected since this is the planning ticket for it.

Actions
Copy link

Also available in: Atom PDF