Project

General

Profile

Actions

action #91605

closed

QA - coordination #91646: [saga][epic] SUSE Maintenance QA workflows with fully automated testing, approval and release

coordination #91914: [epic] Make reviewing openQA results per squad easier

notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) size:M

Added by okurz about 3 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Same as #16276 but using Slack (was: Rocket.Chat) instead of IRC

Acceptance criteria

  • AC1: A failed job with no bugref or label is sent as an individual notification, e.g. using an AMQP Slack (was: Rocket.Chat) bridge in a dedicated Slack (was: Rocket.Chat) channel

Suggestions

- Try https://github.com/42wim/matterbridge


Files

slack-email.png (30.9 KB) slack-email.png tinita, 2022-04-28 12:09

Related issues 5 (4 open1 closed)

Related to openQA Project - action #16274: email notifications about new review comments availableNew2016-02-09

Actions
Related to openQA Project - action #17252: notifications to maintainer on failed modulesNew2016-02-09

Actions
Related to QA - action #98919: Improve alert handling - slack notificationsWorkable2021-09-20

Actions
Copied from openQA Project - action #16276: notifications about failed and unreviewed jobsResolvedmkittler2016-02-09

Actions
Copied to openQA Infrastructure - action #110191: CI checks should ensure that apparmor changes are correctNew

Actions
Actions #1

Updated by okurz about 3 years ago

  • Copied from action #16276: notifications about failed and unreviewed jobs added
Actions #2

Updated by okurz about 3 years ago

  • Parent task set to #91467
Actions #3

Updated by okurz about 3 years ago

  • Parent task changed from #91467 to #91914
Actions #4

Updated by okurz about 3 years ago

  • Target version changed from Ready to future

this is the only story that is currently in the backlog as part of #91646 . To be able to focus on existing sagas and get some done we unplan for now. An additional factor is that currently SUSE is within a process to decide which chat platform to use at the moment so implementing Rocket.Chat support right now might not be long-term efficient

Actions #5

Updated by okurz almost 3 years ago

For SUSE QE maintenance tests we already have a mailing list qa-maintenance-reports@suse.de that receives comments from https://gitlab.suse.de/qa-maintenance/openQABot/ on build.suse.de whenever any incident related tests fail. The archive can be seen on
http://mailman.suse.de/mlarch/SuSE/qa-maintenance-reports/index.html

Actions #6

Updated by okurz over 2 years ago

  • Subject changed from notifications about failed and unreviewed jobs - but using Rocket.Chat to notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat)
  • Description updated (diff)
Actions #7

Updated by okurz about 2 years ago

  • Target version changed from future to Ready

I think the best solution right now is to hook into https://github.com/os-autoinst/scripts/blob/master/openqa-label-known-issues-and-investigate-hook#L13 and send and email about "unknown new issues", i.e. the ones that openqa-investigate is called on to an email address that is part of the according job group's description. For example I put into https://openqa.opensuse.org/admin/job_templates/24 for the job group "openQA" on o3 the following text:

Responsible maintainer, MAILTO: o3-admins@suse.de,team-qa-tools-aaaaejigt64kk3g6p3q4rtgnl4@suse.slack.com

I suggest to parse /MAILTO: (.*)$/ and parse a comma-separated list (maybe optional after ,) and send an email to each

Actions #8

Updated by okurz about 2 years ago

  • Status changed from Workable to New
Actions #9

Updated by livdywan about 2 years ago

  • Subject changed from notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) to notifications about failed and unreviewed jobs - but using Slack (was: Rocket.Chat) size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #10

Updated by okurz about 2 years ago

  • Description updated (diff)
Actions #11

Updated by mkittler about 2 years ago

  • Category deleted (Feature requests)
  • Assignee set to cwh
  • Target version deleted (Ready)

All job groups and their descriptions can be easily queried via e.g. https://openqa.opensuse.org/api/v1/job_groups.

Actions #12

Updated by mkittler about 2 years ago

  • Category set to Feature requests
  • Assignee deleted (cwh)
  • Target version set to Ready
Actions #13

Updated by okurz about 2 years ago

  • Related to action #16274: email notifications about new review comments available added
Actions #14

Updated by okurz about 2 years ago

  • Related to action #17252: notifications to maintainer on failed modules added
Actions #15

Updated by tinita about 2 years ago

  • Status changed from Workable to In Progress
  • Assignee set to tinita
Actions #16

Updated by tinita about 2 years ago

Pushed my code so far to https://github.com/perlpunk/scripts/tree/email-to-be-reviewed

Would like to run it manually to see if it works, but on my workstation only emails to my suse.de address are delivered, not to the mailing list or slack. Will check out the code on o3 to test it there.

Actions #17

Updated by tinita about 2 years ago

https://github.com/os-autoinst/scripts/pull/145 Email notifications for unreviewed issues (merged)

So it should already be active.

Now I should configure the from_email in the hook script.

Actions #18

Updated by tinita about 2 years ago

Actions #19

Updated by livdywan about 2 years ago

  • Copied to action #110191: CI checks should ensure that apparmor changes are correct added
Actions #20

Updated by tinita about 2 years ago

https://github.com/os-autoinst/openQA/pull/4618
Apparently python3 and mutt were in the wrong section. I don't see something like a section header anywhere, so I can only guess that its correct now.

edit: merged

Actions #21

Updated by tinita about 2 years ago

The apparmor update didn't help, so I need to find out what it is trying to tell me:

/usr/bin/env: ‘python3’: Permission denied
Actions #22

Updated by tinita about 2 years ago

https://github.com/os-autoinst/openQA/pull/4621 Allow python3.6 in apparmor (merged)

https://github.com/os-autoinst/scripts/pull/148 Don't attempt to notify for empty group_id (merged)

Actions #23

Updated by tinita about 2 years ago

https://github.com/os-autoinst/scripts/pull/149 Ensure writable homedir when calling mutt (merged)

Actions #24

Updated by tinita about 2 years ago

There is still a problem when calling the script as a minion hook:
https://openqa.opensuse.org/minion/jobs?id=1178321

notes:
  gru_id: 18130689
  hook_cmd: env scheme=http exclude_group_regex='(Development|Open Build Service|Others|Kernel).*/.*'
    /opt/os-autoinst-scripts/openqa-label-known-issues-and-investigate-hook
  hook_rc: 256
  hook_result: "{\"id\":258559}\n"

But for some reason we don't get the error output of the hook command, so we never know what's actually going wrong.
When I do:

su - geekotest
env scheme=http /opt/os-autoinst-scripts/openqa-label-known-issues-hook 2317631

Then I see

http://openqa.opensuse.org/tests/2317631 : Unknown issue, to be reviewed -> http://openqa.opensuse.org/tests/2317631/file/autoinst-log.txt
Likely the error is within this log excerpt, last lines before shutdown:
  # --- 8< ---
...

1 unknown issues to be reviewed:
 - http://openqa.opensuse.org/tests/2317631 null

and I get an email.
So everything looks fine.

If we are throwing away the stderr of the hook command then I'm out of ideas. Except of course capturing the stderr of the hook command.

Actions #25

Updated by tinita about 2 years ago

What I can see is that the file /tmp/sent is created, but it is empty:

-rw------- 1 geekotest nogroup 0 Apr 26 17:25 /tmp/sent
Actions #26

Updated by tinita about 2 years ago

https://github.com/os-autoinst/scripts/pull/150 Use mailx instead of mutt (merged)
https://github.com/os-autoinst/openQA/pull/4626 Allow mailx in apparmor (merged)

Actions #27

Updated by tinita about 2 years ago

https://github.com/os-autoinst/openQA/pull/4627 Allow sendmail related things in apparmor

This is hopefully the last necessary pull request to make sending emails possible.

Sending emails to Slack still won't work unless we can decide which "From" email address we want to use which has a suse.de or suse.com domain. open.qa won't be accepted by Slack, and Oliver didn't want his email address be configured as the default address in git :)

Actions #30

Updated by tinita about 2 years ago

When using a suse.de address as From, Slack will accept it, but the o3-admins mailing list wont.
I just tried with my own email address (@suse.de), and I got an "Undelivered Mail Returned to Sender":
Sender address rejected:
We don't accept our own addresses as sender address on this host!

If I use my suse.com address, it doesn't show up in Slack, but I get a reply from o3-admins that it is waiting for approval.

Another problem is still that I don't see any output in the gru journal when an email is sent, except two warnings:

May 02 16:16:57 ariel postfix/sendmail[7411]: warning: ICU library initialization failed: U_FILE_ACCESS_ERROR
May 02 16:16:58 ariel postfix/postdrop[7413]: warning: ICU library initialization failed: U_FILE_ACCESS_ERROR

Normally we would see the log excerpt and everything, but as soon as email sending is involved, all that output is gone.

Actions #31

Updated by okurz about 2 years ago

Trying out myself. Locally I use echo "test" | mutt -s "test mutt" -e "my_hdr From: openqa-label-known-issues <my-test-okurz@suse.de>" okurz@suse.de -c okurz-test-public-aaaafv7wq7hfg7hxmanetsdnxa@suse.slack.com,o3-admins@suse.de, receiving emails to okurz@suse.de and o3-admins@suse.de but seeing no message in Slack. After some minutes I have received emails saying that the address was rejected at this point by Slack, e.g. with reasoning smtp; 550 5.7.1 IP address blacklisted by recipient when I used openqa-label-known-issues <okurz@suse.de>.

I have also asked in https://suse.slack.com/archives/C029APBKLGK/p1651569243914769 if people know how to send emails to SUSE mailing lists from a @suse.de mailing list set in the From header.

I am confused. Where does my local postfix get the address 'relay.suse.de' from? I can't find it anywhere in /etc/. And it looks like Slack banned that IPv4. Sending over authenticated imap.suse.de works, e.g. using Thunderbird, over relay.suse.de it does not.

Actions #32

Updated by okurz about 2 years ago

As we discussed today I have recorded our additional ideas in the two additional tickets

as well as extended #99741.

So this means that all three things can be considered out of scope for the current work. Also if you encounter problems that only affect one of both o3+osd then continue with just one of the two, announce the feature to users, ask them to use it and report the other issue in a separate ticket (e.g. "email sending from openqa.suse.de is rejected by Slack"). As I have not received a response in https://suse.slack.com/archives/C029APBKLGK/p1651569243914769 I have now asked in #help-slack as well with https://suse.slack.com/archives/C029GP8DCDA/p1651670821441329

Actions #34

Updated by tinita about 2 years ago

  • Status changed from In Progress to Feedback
Actions #35

Updated by tinita about 2 years ago

I sent an email to the openqa mailing list to announce the feature.

Actions #37

Updated by okurz about 2 years ago

  • Related to action #98919: Improve alert handling - slack notifications added
Actions #38

Updated by tinita about 2 years ago

  • Status changed from Feedback to Resolved

https://github.com/os-autoinst/scripts/pull/156 Document Slack/Email notifications for unknown issues (merged)

Actions #39

Updated by okurz about 2 years ago

For the sake of completeness, the above PR added documentation which is available on https://github.com/os-autoinst/scripts#unknown-issues= now

Actions

Also available in: Atom PDF