QA » openQA Project » openQA Infrastructure

Prioritized as "Urgent" as the alert was ignored and not handled by multiple persons for days and we are apparently suffering from alarm fatigue

Where was the alert visible? I am subscribed to osd-admins@suse.de but didn't receive an email (apart from the Re: you've just sent to the list).

Ensure you have access to https://gitlab.suse.de/OPS-Service/monitoring/ , ask in EngInfra ticket otherwise

When accessing the page I get 404. I assume this is actually 403. So I'll ask infra for access. (In the meantime someone else can pick up the ticket of course.)

mkittler wrote:

Prioritized as "Urgent" as the alert was ignored and not handled by multiple persons for days and we are apparently suffering from alarm fatigue

Where was the alert visible? I am subscribed to osd-admins@suse.de but didn't receive an email (apart from the Re: you've just sent to the list).

ah, good point. The alert was coming from nagios and is visible in the email and also in the referenced URL https://thruk.suse.de/thruk/cgi-bin/extinfo.cgi?type=2&host=openqa.suse.de&service=fs_%2Fassets
I thought you have worked with nagios alerts in the past?
Just today I added in https://progress.opensuse.org/projects/qa/wiki/Wiki#Onboarding-for-new-joiners "Ensure you have access to https://gitlab.suse.de/OPS-Service/monitoring (create EngInfra ticket otherwise) and add yourself in https://gitlab.suse.de/OPS-Service/monitoring/-/tree/master/icinga/shared/contacts to receive monitoring information".

Saw more alerts. My MR was still ignored. Created ticket as reminder: https://infra.nue.suse.com/SelfService/Display.html?id=189974

MR was merged but the change is not effective. Maybe I need to explicitly mention "assets" in a separate line: https://gitlab.suse.de/OPS-Service/monitoring/-/merge_requests/18