Project

General

Profile

Actions

action #57239

open

Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respectively

Added by okurz about 5 years ago. Updated almost 4 years ago.

Status:
Workable
Priority:
Low
Assignee:
-
Category:
-
Target version:
QA (public, currently private due to #173521) - future
Start date:
2019-09-23
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Just as o3-admins@suse.de we have a mailing list osd-admins@suse.de now which is used for example for grafana monitoring messages. But o3:/etc/openqa/openqa.ini was changed to not write log messages to /var/log/openqa anymore but all to the journal, meaning that openqa_logwarn does not work anymore as in before.

Acceptance criteria

  • AC1: A periodic job for openqa_logwarn sends (again) emails if any unexpected errors are encountered in service logs on o3, sent to o3-admins@suse.de
  • AC2: Same as AC1 but on osd, sent to osd-admins@suse.de

Suggestions

  • Research what https://github.com/os-autoinst/openqa-logwarn is doing
  • Lookup cron job or systemd timer definitions on o3 and/or osd to find out how this should be triggered
  • Test if simple email sending from crontab works
  • Look on o3 and osd what the output of openqa_logwarn is and ensure that emails are sent if there are unexpected errors encountered in logfiles

Related issues 5 (3 open2 closed)

Related to openQA Infrastructure (public) - action #35290: [tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observedResolvedokurz2018-04-20

Actions
Related to openQA Infrastructure (public) - action #99741: Minion jobs for job hooks failed silently on o3 size:MResolveddheidler2021-10-04

Actions
Related to openQA Project (public) - action #97544: Monitoring alerts on errors in logs on osdNew2021-07-09

Actions
Related to openQA Project (public) - action #97247: Monitoring alerts on errors in logs on openQA workersNew2021-07-09

Actions
Copied to openQA Infrastructure (public) - action #137810: Add openqa_logwarn for osd sending to osd-admins@suse.de respectivelyNew2019-09-23

Actions
Actions #1

Updated by okurz about 5 years ago

  • Related to action #35290: [tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observed added
Actions #2

Updated by okurz over 4 years ago

  • Target version set to Ready
Actions #3

Updated by okurz about 4 years ago

  • Priority changed from Normal to Low
Actions #4

Updated by okurz about 4 years ago

  • Description updated (diff)
  • Status changed from New to Workable
Actions #5

Updated by okurz almost 4 years ago

  • Target version changed from Ready to future
Actions #6

Updated by okurz about 3 years ago

  • Related to action #99741: Minion jobs for job hooks failed silently on o3 size:M added
Actions #7

Updated by dheidler over 1 year ago

  • Related to action #97544: Monitoring alerts on errors in logs on osd added
Actions #8

Updated by tinita about 1 year ago

  • Related to action #97247: Monitoring alerts on errors in logs on openQA workers added
Actions #9

Updated by okurz about 1 year ago

  • Copied to action #137810: Add openqa_logwarn for osd sending to osd-admins@suse.de respectively added
Actions

Also available in: Atom PDF