Project

General

Profile

action #105828

Updated by livdywan about 2 years ago

## Observation 

 Thanks to #80812 o3 can send out emails. Unfortunately now we're getting 4-7 logreport emails from openqa-monitor@ariel.suse-dmz.opensuse.org on a daily basis and we're not keeping up with handling all of them. 
 Emails are sent by a cronjob running https://github.com/os-autoinst/openqa-logwarn 

 Examples: 

 ``` 
 [2022-02-02T09:44:45.023821Z] [error] [pid:6229] Cannot read symbolic link (/opt/openqa-trigger-from-obs/openSUSE:Leap:15.4:ARM:Images:ToTest/.run_last): No such file or directory 
 [2022-02-02T08:07:52.883567Z] [warn] [pid:22053] Ignoring invalid group {"name":"38"} when creating new job 2172324 
 [2022-02-02T02:30:10.097604Z] [warn] [pid:10722] Unable to wakeup scheduler: Request timeout 
 [2022-02-02T02:30:14.810226Z] [error] [pid:13594] Publishing opensuse.openqa.job.restart failed: Connect timeout (9 attempts left) 
 [2022-02-01T15:38:12.281868Z] [warn] [pid:28556] fatal: Invalid revision range 745485c7527687dab875e0ab0f4c96f730e26dea..8f56d6708e2211a41fe189635a3bbebd2f9d0be8 
 [2022-02-01T15:38:12.282093Z] [error] [pid:28556] cmd returned 32768 
 ``` 

 ## Acceptance criteria 
 * **AC1:** All alerts from o3-admins@suse.de o3 can be handled by the team 

 ## Suggestions 
 - Team up to investigate all of the current issues 
 - Create individual tickets for the issues and blocklist them by proposing changes to https://github.com/os-autoinst/openqa-logwarn (changes are effective ~10 minutes after a merge)

Back