action #181580
closed
coordination #161414: [epic] Improved salt based infrastructure management
jenkins doesn't notify us on broken builds/runs - should there be e-mails or such? size:S
Added by nicksinger 26 days ago.
Updated 10 days ago.
Category:
Regressions/Crashes
Description
Motivation¶
During investigation of #181526 we realized that we do not get any notification of failed runs from http://jenkins.qe.nue2.suse.org/ - @nicksinger and @okurz remember that this was different in the past and we should look into why the system doesn't notify us any longer
Acceptance Criteria¶
-
AC1: Jenkins sends emails to osd-admins when builds fail
Suggestions¶
- Related to action #181526: osd-deployment fails during 'check openQA-in-openQA tests' size:S added
- Tags set to infra, reactive work, alert, email, salt
- Subject changed from jenkins doesn't notify us on broken builds/runs - should there be e-mails or such? to jenkins doesn't notify us on broken builds/runs - should there be e-mails or such? size:S
- Description updated (diff)
- Category set to Regressions/Crashes
- Status changed from New to Workable
- Target version set to Ready
- Parent task set to #161414
- Related to action #180209: Emails not sent to osd-admins for all salt-controlled hosts size:S added
- Status changed from Workable to In Progress
- Assignee set to nicksinger
I had a quick look on the system and it is perfectly fine to deliver e-mails which I tested with echo "Hello world" | mail -s "Test" nsinger@suse.de and which arrived in my inbox (after getting a "451 4.7.1 Try again later" for the first time - which is fine and accepted behavior from the upstream mail server). Checking how jenkins "does things", I quickly found https://plugins.jenkins.io/email-ext/ - not sure if this is just a e-mail plugin with more features (note the -ext in the plugin url) or if we need a plugin in the first place to send mails. Plugins are stored in /var/lib/jenkins/plugins/ and the mail-plugin looks about correct (and not disabled):
jenkins@jenkins:~> ls -lahd /var/lib/jenkins/plugins/mail*
drwxr-xr-x 1 jenkins jenkins 52 Jan 9 2017 /var/lib/jenkins/plugins/mail-watcher-plugin
drwxr-xr-x 1 jenkins jenkins 52 Oct 31 2024 /var/lib/jenkins/plugins/mailer
-rw-r--r-- 1 jenkins jenkins 134K Oct 2 2024 /var/lib/jenkins/plugins/mailer.bak
-rw-r--r-- 1 jenkins jenkins 134K Oct 31 2024 /var/lib/jenkins/plugins/mailer.jpi
so as next step it makes sense to:
- check if the plugin is actually used and enabled
- is there any config file for this plugin? It should work without any (by now)
nicksinger wrote in #note-4:
- check if the plugin is actually used and enabled
- is there any config file for this plugin? It should work without any (by now)
plugin is used and enabled as visible on http://jenkins.qe.nue2.suse.org/manage/configure#e-mail-notification - settings look sane. I only changed the "System Admin e-mail address" from previously Oliver Kurz <okurz@suse.de> to osd-admins@suse.de - this changes the sender of all jenkins mails. Using the "Test configuration by sending test e-mail" I was able to receive a test mail from jenkins -> so all fine with plugin settings and such.
This leaves only the project settings. http://jenkins.qe.nue2.suse.org/job/submit-openQA-TW-to-oS_Fctry/configure is the only project having a "Post-build Actions" of type "E-mail Notification" - which I was able to confirm from e-mail archives. I now enabled the "Send e-mail for every unstable build"-checkbox in the hopes that for some reason jenkins just decided to not inform us because it deemed the build unstable?
- Status changed from In Progress to Feedback
- Status changed from Feedback to Resolved
Also available in: Atom
PDF