Project

General

Profile

Actions

action #57239

open

Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respectively

Added by okurz about 5 years ago. Updated almost 4 years ago.

Status:
Workable
Priority:
Low
Assignee:
-
Category:
-
Target version:
Start date:
2019-09-23
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Just as o3-admins@suse.de we have a mailing list osd-admins@suse.de now which is used for example for grafana monitoring messages. But o3:/etc/openqa/openqa.ini was changed to not write log messages to /var/log/openqa anymore but all to the journal, meaning that openqa_logwarn does not work anymore as in before.

Acceptance criteria

  • AC1: A periodic job for openqa_logwarn sends (again) emails if any unexpected errors are encountered in service logs on o3, sent to o3-admins@suse.de
  • AC2: Same as AC1 but on osd, sent to osd-admins@suse.de

Suggestions

  • Research what https://github.com/os-autoinst/openqa-logwarn is doing
  • Lookup cron job or systemd timer definitions on o3 and/or osd to find out how this should be triggered
  • Test if simple email sending from crontab works
  • Look on o3 and osd what the output of openqa_logwarn is and ensure that emails are sent if there are unexpected errors encountered in logfiles

Related issues 5 (3 open2 closed)

Related to openQA Infrastructure - action #35290: [tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observedResolvedokurz2018-04-20

Actions
Related to openQA Infrastructure - action #99741: Minion jobs for job hooks failed silently on o3 size:MResolveddheidler2021-10-04

Actions
Related to openQA Project - action #97544: Monitoring alerts on errors in logs on osdNew2021-07-09

Actions
Related to openQA Project - action #97247: Monitoring alerts on errors in logs on openQA workersNew2021-07-09

Actions
Copied to openQA Infrastructure - action #137810: Add openqa_logwarn for osd sending to osd-admins@suse.de respectivelyNew2019-09-23

Actions
Actions

Also available in: Atom PDF