action #95188
closedDocument how to properly configure GitLab pipeline notifications size:M
Description
Observation¶
Team members don't reliably get notifications for pipelines such as osd-deployment, but also others. Some suggestions have come up in conversations but it's not clear exactly how to configure it and trivial suggestions don't seem to be fruitful.
Acceptance criteria¶
- AC1: Wiki page explains how to get reliable notifications
Out of scope¶
- Works for everyone in the team.
- Non specific kind of messages.
Suggestions¶
- Test out personal notification settings
- Test out notification settings for osd-deployment
- Optionally consider other projects and types of notifications for completeness if it's helpful
Updated by okurz over 3 years ago
I think https://stackoverflow.com/questions/46472631/notify-all-group-members-of-failed-pipelines-in-gitlab describes the problem … and a good solution
we should just configure gitlab to send emails to osd-admins@suse.de for all projects if that is possible.
Updated by okurz over 3 years ago
- Related to action #95105: osd-deployment pipelines fail and alerts are not handled size:M added
Updated by livdywan over 3 years ago
Transferred from https://progress.opensuse.org/issues/95105#note-5:
This is apparently a known problem, see https://stackoverflow.com/questions/46472631/notify-all-group-members-of-failed-pipelines-in-gitlab
So I've been adding myself on https://gitlab.suse.de/openqa/osd-deployment/-/services/pipelines_email/edit and monitor-o3. Let's see whether it works.
Updated by mkittler over 3 years ago
The previous comment is actually from me. So far I've only got a mail "Successful pipeline for master | osd-deployment | af090d3d" and that although I've checked "Notify only broken pipelines". So I am not sure yet whether it worked.
Updated by mkittler over 3 years ago
Yesterday I've received "Failed pipeline for master | osd-deployment | af090d3d" which says "Pipeline #168676 triggered by Oliver Kurz" so I guess it worked.
Updated by ilausuch over 3 years ago
- Subject changed from Document how to properly configure GitLab pipeline notifications to Document how to properly configure GitLab pipeline notifications size: M
- Description updated (diff)
Updated by okurz over 3 years ago
- Subject changed from Document how to properly configure GitLab pipeline notifications size: M to Document how to properly configure GitLab pipeline notifications size:M
Updated by tinita over 3 years ago
I added osd-admins@suse.de to https://gitlab.suse.de/openqa/osd-deployment/-/services/pipelines_email/edit
This was suggested over a month ago, but nobody did it.
Updated by tinita over 3 years ago
I also added Pipeline status emails to
https://gitlab.suse.de/openqa/openqa-review/-/services/pipelines_email/edit
Updated by tinita over 3 years ago
Updated by livdywan over 3 years ago
- Status changed from Workable to Feedback
- Assignee set to tinita
We discussed this in the daily. Let's see if either the ml, .de or .com emails work.
Updated by tinita over 3 years ago
Personal emails seem to work, but the list doesn't work because gitlab@suse.de is not allowed to write to the list.
Not sure who is the owner, so I wrote an email to osd-admins-owner@suse.de
Updated by okurz over 3 years ago
As owner of osd-admins I now added gitlab@suse.de as valid sender. You confirmed to have received according emails already.
Updated by tinita over 3 years ago
okurz wrote:
As owner of osd-admins I now added gitlab@suse.de as valid sender. You confirmed to have received according emails already.
I received an email today (friday) at 05:58:57 Failed pipeline for master | osd-deployment | 4e4ecc88
to my personal address. But nothing arrived at the list so far.
Updated by okurz over 3 years ago
From mailman:
The osd-admins@suse.de mailing list has 2 request(s) waiting for your
consideration at:
http://mailman.suse.de/mailman/admindb/osd-admins
Please attend to this at your earliest convenience. This notice of
pending requests, if any, will be sent out daily.
Pending posts:
From: gitlab@suse.de on Fri Aug 20 05:58:57 2021
Subject: Failed pipeline for master | osd-deployment | 4e4ecc88
Cause: Message has implicit destination
From: gitlab@suse.de on Mon Aug 23 05:57:27 2021
Subject: Failed pipeline for master | osd-deployment | 4e4ecc88
Cause: Message has implicit destination
so we would need to learn if we can configure mailman to accept that. gitlab has it explicitly selected to use BCC (or CC) to send the emails to.
Following https://support.tigertech.net/mailman-implicit-destination I have on https://mailman.suse.de/mailman/admin/osd-admins/privacy/recipient changed "Must posts have list named in destination (to, cc) field (or be among the acceptable alias names, specified below)?" from "Yes" to "No"
If this does not work then what I also tried is to create a proper IDP account for "osd-admins@suse.de" following the create users page from https://bugzilla.suse.com with username "osd-admins", same password as the admin password for the mailing list, and then trying to login on https://gitlab.suse.de/users/sign_in . But this failed so far. Maybe takes some time to sync. If we can login maybe "osd-admins" can take over the pipeline. I also put the password into https://gitlab.suse.de/openqa/scripts/-/commit/359f4a9a67634f70dcb35b5f429f8fe5c7fb0356
Updated by tinita over 3 years ago
Currently osd-admins@suse.de is listed as a recipient in the Pipeline status emails of the following projects:
- https://gitlab.suse.de/openqa/osd-deployment
- https://gitlab.suse.de/openqa/openqa-review
- https://gitlab.suse.de/openqa/salt-states-openqa
- https://gitlab.suse.de/openqa/monitor-o3
I added this to the Wiki:
https://progress.opensuse.org/projects/qa/wiki/Wiki#section-22
I couldn't find any other projects with scheduled pipelines.
Let me know if there are projects which should be added.
Otherwise the ticket can be resolved I think.
Updated by okurz over 3 years ago
As far as ACs this looks good because the wiki describes what to do. However there are some important projects that immediately come to mind:
- openqa/salt-pillars-openqa
- openqa/auto-review
- openqa/grafana-webhook-actions
- qa-maintenance/openQABot
- qa-maintenance/bot-ng
I might have mistyped from memory. These should be handled
Updated by tinita over 3 years ago
Ok, I added the projects suggested by @okurz and updated the wiki
Updated by okurz over 3 years ago
- Status changed from Feedback to Resolved
very nice. I think this is more than enough right now. Thanks a lot!