action #181580
closedcoordination #161414: [epic] Improved salt based infrastructure management
jenkins doesn't notify us on broken builds/runs - should there be e-mails or such? size:S
0%
Description
Motivation¶
During investigation of #181526 we realized that we do not get any notification of failed runs from http://jenkins.qe.nue2.suse.org/ - @nicksinger and @okurz remember that this was different in the past and we should look into why the system doesn't notify us any longer
Acceptance Criteria¶
- AC1: Jenkins sends emails to osd-admins when builds fail
Suggestions¶
- Check if recent changes in the postfix setup in https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1431 from #180209 are related
- We used to get emails like this:
Build failed in Jenkins: submit-openQA-TW-to-oS_Fctry #996- Possibly last one https://mailman.suse.de/mlarch/SuSE/osd-admins/2025/osd-admins.2025.03/msg00133.html
date: Tue, 18 Mar 2025 20:53:06 +0000 (UTC)but recent failure in jenkins http://jenkins.qa.suse.de/job/monitor-openQA_in_openQA-TW/lastFailedBuild/console from 2025-04-29 and we have received no email
- Consider filing a ticket about monitoring mail-sending ability on all of our hosts and add findings of this task here
Updated by okurz 26 days ago
- Related to action #181526: osd-deployment fails during 'check openQA-in-openQA tests' size:S added
Updated by okurz 26 days ago
- Tags set to infra, reactive work, alert, email, salt
- Subject changed from jenkins doesn't notify us on broken builds/runs - should there be e-mails or such? to jenkins doesn't notify us on broken builds/runs - should there be e-mails or such? size:S
- Description updated (diff)
- Category set to Regressions/Crashes
- Status changed from New to Workable
- Target version set to Ready
- Parent task set to #161414
Updated by okurz 26 days ago
- Related to action #180209: Emails not sent to osd-admins for all salt-controlled hosts size:S added
Updated by nicksinger 11 days ago
- Status changed from Workable to In Progress
- Assignee set to nicksinger
I had a quick look on the system and it is perfectly fine to deliver e-mails which I tested with echo "Hello world" | mail -s "Test" nsinger@suse.de and which arrived in my inbox (after getting a "451 4.7.1 Try again later" for the first time - which is fine and accepted behavior from the upstream mail server). Checking how jenkins "does things", I quickly found https://plugins.jenkins.io/email-ext/ - not sure if this is just a e-mail plugin with more features (note the -ext in the plugin url) or if we need a plugin in the first place to send mails. Plugins are stored in /var/lib/jenkins/plugins/ and the mail-plugin looks about correct (and not disabled):
jenkins@jenkins:~> ls -lahd /var/lib/jenkins/plugins/mail*
drwxr-xr-x 1 jenkins jenkins 52 Jan 9 2017 /var/lib/jenkins/plugins/mail-watcher-plugin
drwxr-xr-x 1 jenkins jenkins 52 Oct 31 2024 /var/lib/jenkins/plugins/mailer
-rw-r--r-- 1 jenkins jenkins 134K Oct 2 2024 /var/lib/jenkins/plugins/mailer.bak
-rw-r--r-- 1 jenkins jenkins 134K Oct 31 2024 /var/lib/jenkins/plugins/mailer.jpi
so as next step it makes sense to:
- check if the plugin is actually used and enabled
- is there any config file for this plugin? It should work without any (by now)
Updated by nicksinger 11 days ago
nicksinger wrote in #note-4:
- check if the plugin is actually used and enabled
- is there any config file for this plugin? It should work without any (by now)
plugin is used and enabled as visible on http://jenkins.qe.nue2.suse.org/manage/configure#e-mail-notification - settings look sane. I only changed the "System Admin e-mail address" from previously Oliver Kurz <okurz@suse.de> to osd-admins@suse.de - this changes the sender of all jenkins mails. Using the "Test configuration by sending test e-mail" I was able to receive a test mail from jenkins -> so all fine with plugin settings and such.
This leaves only the project settings. http://jenkins.qe.nue2.suse.org/job/submit-openQA-TW-to-oS_Fctry/configure is the only project having a "Post-build Actions" of type "E-mail Notification" - which I was able to confirm from e-mail archives. I now enabled the "Send e-mail for every unstable build"-checkbox in the hopes that for some reason jenkins just decided to not inform us because it deemed the build unstable?
Updated by nicksinger 11 days ago
nicksinger wrote in #note-5:
that for some reason jenkins just decided to not inform us because it deemed the build unstable?
okay no. I checked http://jenkins.qe.nue2.suse.org/job/submit-openQA-TW-to-oS_Fctry/ again, entered "fail" on the build-filter on the left and I can indeed see the last failed build from 18th of March. Looking at http://jenkins.qe.nue2.suse.org/job/submit-openQA-TW-to-oS_Fctry/3108/console, at the very bottom, you can read: "Sending e-mails to: osd-admins@suse.de" - and this is indeed also what we received in https://mailman.suse.de/mlarch/SuSE/osd-admins/2025/osd-admins.2025.03/msg00133.html
talking to @okurz in the unblock and we decided to enable these mails for all projects now as we consider them as generally stable.
Updated by nicksinger 11 days ago
- Status changed from In Progress to Feedback
e-mails to "osd-admins@suse.de" now added as post-build action to all enabled jenkins projects. I also removed all IRC actions as this is not really used by us any longer (In accordance with @okurz). http://jenkins.qe.nue2.suse.org/job/monitor-openQA_in_openQA-TW/ fails quite regularly so I keep this on feedback until we get an mail or the due date reminds me that it doesn't work :)
Updated by nicksinger 10 days ago
- Status changed from Feedback to Resolved
Just today http://jenkins.qa.suse.de/job/monitor-openQA_in_openQA-TW/31060/ failed and we got a mail for it - so everything working as expected now.