Project

General

Profile

action #91605

QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

coordination #91914: [epic] Make reviewing openQA results per squad easier

notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) size:M

Added by okurz over 1 year ago. Updated 7 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

Same as #16276 but using Slack (was: Rocket.Chat) instead of IRC

Acceptance criteria

  • AC1: A failed job with no bugref or label is sent as an individual notification, e.g. using an AMQP Slack (was: Rocket.Chat) bridge in a dedicated Slack (was: Rocket.Chat) channel

Suggestions

- Try https://github.com/42wim/matterbridge

slack-email.png (30.9 KB) slack-email.png tinita, 2022-04-28 12:09
13176

Related issues

Related to openQA Project - action #16274: email notifications about new review comments availableNew2016-02-09

Related to openQA Project - action #17252: notifications to maintainer on failed modulesNew2016-02-09

Related to QA - action #98919: Improve alert handling - slack notificationsWorkable2021-09-20

Copied from openQA Project - action #16276: notifications about failed and unreviewed jobsResolved2016-02-09

Copied to openQA Infrastructure - action #110191: CI checks should ensure that apparmor changes are correctNew

History

#1 Updated by okurz over 1 year ago

  • Copied from action #16276: notifications about failed and unreviewed jobs added

#2 Updated by okurz over 1 year ago

  • Parent task set to #91467

#3 Updated by okurz over 1 year ago

  • Parent task changed from #91467 to #91914

#4 Updated by okurz over 1 year ago

  • Target version changed from Ready to future

this is the only story that is currently in the backlog as part of #91646 . To be able to focus on existing sagas and get some done we unplan for now. An additional factor is that currently SUSE is within a process to decide which chat platform to use at the moment so implementing Rocket.Chat support right now might not be long-term efficient

#5 Updated by okurz over 1 year ago

For SUSE QE maintenance tests we already have a mailing list qa-maintenance-reports@suse.de that receives comments from https://gitlab.suse.de/qa-maintenance/openQABot/ on build.suse.de whenever any incident related tests fail. The archive can be seen on
http://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/index.html

#6 Updated by okurz 11 months ago

  • Subject changed from notifications about failed and unreviewed jobs - but using Rocket.Chat to notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat)
  • Description updated (diff)

#7 Updated by okurz 9 months ago

  • Target version changed from future to Ready

I think the best solution right now is to hook into https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues-and-investigate-hook#L13 and send and email about "unknown new issues", i.e. the ones that openqa-investigate is called on to an email address that is part of the according job group's description. For example I put into https://openqa.opensuse.org/admin/job_templates/24 for the job group "openQA" on o3 the following text:

Responsible maintainer, MAILTO: o3-admins@suse.de,team-qa-tools-aaaaejigt64kk3g6p3q4rtgnl4@suse.slack.com

I suggest to parse /MAILTO: (.*)$/ and parse a comma-separated list (maybe optional after ,) and send an email to each

#8 Updated by okurz 9 months ago

  • Status changed from Workable to New

#9 Updated by cdywan 9 months ago

  • Subject changed from notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) to notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) size:M
  • Description updated (diff)
  • Status changed from New to Workable

#10 Updated by okurz 9 months ago

  • Description updated (diff)

#11 Updated by mkittler 9 months ago

  • Category deleted (Feature requests)
  • Assignee set to cwh
  • Target version deleted (Ready)

All job groups and their descriptions can be easily queried via e.g. https://openqa.opensuse.org/api/v1/job_groups.

#12 Updated by mkittler 9 months ago

  • Category set to Feature requests
  • Assignee deleted (cwh)
  • Target version set to Ready

#13 Updated by okurz 9 months ago

  • Related to action #16274: email notifications about new review comments available added

#14 Updated by okurz 9 months ago

  • Related to action #17252: notifications to maintainer on failed modules added

#15 Updated by tinita 8 months ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita

#16 Updated by tinita 8 months ago

Pushed my code so far to https://github.com/perlpunk/scripts/tree/email-to-be-reviewed

Would like to run it manually to see if it works, but on my workstation only emails to my suse.de address are delivered, not to the mailing list or slack. Will check out the code on o3 to test it there.

#17 Updated by tinita 8 months ago

https://github.com/os-autoinst/scripts/pull/145 Email notifications for unreviewed issues (merged)

So it should already be active.

Now I should configure the from_email in the hook script.

#18 Updated by tinita 8 months ago

#19 Updated by cdywan 8 months ago

  • Copied to action #110191: CI checks should ensure that apparmor changes are correct added

#20 Updated by tinita 8 months ago

https://github.com/os-autoinst/openQA/pull/4618
Apparently python3 and mutt were in the wrong section. I don't see something like a section header anywhere, so I can only guess that its correct now.

edit: merged

#21 Updated by tinita 8 months ago

The apparmor update didn't help, so I need to find out what it is trying to tell me:

/usr/bin/env: ‘python3’: Permission denied

#22 Updated by tinita 8 months ago

https://github.com/os-autoinst/openQA/pull/4621 Allow python3.6 in apparmor (merged)

https://github.com/os-autoinst/scripts/pull/148 Don't attempt to notify for empty group_id (merged)

#23 Updated by tinita 8 months ago

https://github.com/os-autoinst/scripts/pull/149 Ensure writable homedir when calling mutt (merged)

#24 Updated by tinita 8 months ago

There is still a problem when calling the script as a minion hook:
https://openqa.opensuse.org/minion/jobs?id=1178321

notes:
  gru_id: 18130689
  hook_cmd: env scheme=http exclude_group_regex='(Development|Open Build Service|Others|Kernel).*/.*'
    /opt/os-autoinst-scripts/openqa-label-known-issues-and-investigate-hook
  hook_rc: 256
  hook_result: "{\"id\":258559}\n"

But for some reason we don't get the error output of the hook command, so we never know what's actually going wrong.
When I do:

su - geekotest
env scheme=http /opt/os-autoinst-scripts/openqa-label-known-issues-hook 2317631

Then I see

http://openqa.opensuse.org/tests/2317631 : Unknown issue, to be reviewed -> http://openqa.opensuse.org/tests/2317631/file/autoinst-log.txt
Likely the error is within this log excerpt, last lines before shutdown:
  # --- 8< ---
...

1 unknown issues to be reviewed:
 - http://openqa.opensuse.org/tests/2317631 null

and I get an email.
So everything looks fine.

If we are throwing away the stderr of the hook command then I'm out of ideas. Except of course capturing the stderr of the hook command.

#25 Updated by tinita 8 months ago

What I can see is that the file /tmp/sent is created, but it is empty:

-rw------- 1 geekotest nogroup 0 Apr 26 17:25 /tmp/sent

#26 Updated by tinita 7 months ago

https://github.com/os-autoinst/scripts/pull/150 Use mailx instead of mutt (merged)
https://github.com/os-autoinst/openQA/pull/4626 Allow mailx in apparmor (merged)

#27 Updated by tinita 7 months ago

https://github.com/os-autoinst/openQA/pull/4627 Allow sendmail related things in apparmor

This is hopefully the last necessary pull request to make sending emails possible.

Sending emails to Slack still won't work unless we can decide which "From" email address we want to use which has a suse.de or suse.com domain. open.qa won't be accepted by Slack, and Oliver didn't want his email address be configured as the default address in git :)

#28 Updated by tinita 7 months ago

13176

#30 Updated by tinita 7 months ago

When using a suse.de address as From, Slack will accept it, but the o3-admins mailing list wont.
I just tried with my own email address (@suse.de), and I got an "Undelivered Mail Returned to Sender":
Sender address rejected:
We don't accept our own addresses as sender address on this host!

If I use my suse.com address, it doesn't show up in Slack, but I get a reply from o3-admins that it is waiting for approval.

Another problem is still that I don't see any output in the gru journal when an email is sent, except two warnings:

May 02 16:16:57 ariel postfix/sendmail[7411]: warning: ICU library initialization failed: U_FILE_ACCESS_ERROR
May 02 16:16:58 ariel postfix/postdrop[7413]: warning: ICU library initialization failed: U_FILE_ACCESS_ERROR

Normally we would see the log excerpt and everything, but as soon as email sending is involved, all that output is gone.

#31 Updated by okurz 7 months ago

Trying out myself. Locally I use echo "test" | mutt -s "test mutt" -e "my_hdr From: openqa-label-known-issues <my-test-okurz@suse.de>" okurz@suse.de -c okurz-test-public-aaaafv7wq7hfg7hxmanetsdnxa@suse.slack.com,o3-admins@suse.de, receiving emails to okurz@suse.de and o3-admins@suse.de but seeing no message in Slack. After some minutes I have received emails saying that the address was rejected at this point by Slack, e.g. with reasoning smtp; 550 5.7.1 IP address blacklisted by recipient when I used openqa-label-known-issues <okurz@suse.de>.

I have also asked in https://suse.slack.com/archives/C029APBKLGK/p1651569243914769 if people know how to send emails to SUSE mailing lists from a @suse.de mailing list set in the From header.

I am confused. Where does my local postfix get the address 'relay.suse.de' from? I can't find it anywhere in /etc/. And it looks like Slack banned that IPv4. Sending over authenticated imap.suse.de works, e.g. using Thunderbird, over relay.suse.de it does not.

#32 Updated by okurz 7 months ago

As we discussed today I have recorded our additional ideas in the two additional tickets

as well as extended #99741.

So this means that all three things can be considered out of scope for the current work. Also if you encounter problems that only affect one of both o3+osd then continue with just one of the two, announce the feature to users, ask them to use it and report the other issue in a separate ticket (e.g. "email sending from openqa.suse.de is rejected by Slack"). As I have not received a response in https://suse.slack.com/archives/C029APBKLGK/p1651569243914769 I have now asked in #help-slack as well with https://suse.slack.com/archives/C029GP8DCDA/p1651670821441329

#33 Updated by tinita 7 months ago

#34 Updated by tinita 7 months ago

  • Status changed from In Progress to Feedback

#35 Updated by tinita 7 months ago

I sent an email to the openqa mailing list to announce the feature.

#37 Updated by okurz 7 months ago

  • Related to action #98919: Improve alert handling - slack notifications added

#38 Updated by tinita 7 months ago

  • Status changed from Feedback to Resolved

https://github.com/os-autoinst/scripts/pull/156 Document Slack/Email notifications for unknown issues (merged)

#39 Updated by okurz 7 months ago

For the sake of completeness, the above PR added documentation which is available on https://github.com/os-autoinst/scripts#unknown-issues= now

Also available in: Atom PDF