action #35290
closed[tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observed
0%
Description
Observation¶
dheidler was asking me about needles he created on osd but could not see in the needles gitlab repo.
Checking logfiles on osd in /var/log/openqa
[2018-04-20T14:05:09.0636 CEST] [warn] GitLab: Your account has been blocked.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
[2018-04-20T14:05:09.0636 CEST] [error] cmd returned non-zero value
[2018-04-20T14:05:09.0641 CEST] [error] failed to git commit iscsi-target-overview-service-tab-20180420 at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Step.pm line 385.
Problem¶
nsinger and me gave up to monitor the logfiles with logwarn_openqa as recent contributions to openQA over the past year have introduced the noise in the logfiles so that the alerting was seen as unfeasible for us even more considering that we are not "tools"-team members and became frustrated and I assume no one else is doing that monitoring so that problem is not found easily. Potential outcome is loss of data as needle commits are never pushed.
Updated by szarate over 6 years ago
- Related to action #18164: [devops][tools] monitoring of openqa worker instances added
Updated by okurz over 6 years ago
FYI, On 2018-03-13 I have sent the email that I removed the cronjob from osd and AFAIK have not received any answer or objection. Funny enough, on o3 everything is still in place but seems to have never worked. I guess email sending from o3 is broken.
/etc/cron.d/logwarn_openqa
SHELL=/bin/bash
MAIL=okurz@suse.de
MAILTO=okurz@suse.de,nsinger@suse.de
REPORTTO=okurz@suse.de,szarate@suse.de,nsinger@suse.de
PATH=/suse/bin:/usr/lib/ccache:/opt/bin:/usr/local/bin:/usr/bin:/bin
10 * * * * root /opt/openqa-logwarn/pretty_logwarn /var/log/openqa 2>&1
and /opt/openqa-logwarn/
is the git working copy of https://github.com/os-autoinst/openqa-logwarn.git
If anyone wants to try that again then one can just apply the same again on osd and fix the email sending from o3.
Updated by okurz over 5 years ago
- Related to action #42920: Gitlab user gets blocked: Try out gitlab deployment tokens for openQA added
Updated by okurz over 5 years ago
- Status changed from New to Feedback
- Assignee set to okurz
Updated by okurz over 5 years ago
- Project changed from openQA Project to openQA Infrastructure
- Category deleted (
168)
Updated by okurz over 5 years ago
I created a new mailing list o3-admins@suse.de . So far I can handle the load for o3 on the mailing and excluded some known warnings. Should we do the same for osd?
EDIT: We have a mailing list osd-admins@suse.de now which is used for example for grafana monitoring messages. Also o3:/etc/openqa/openqa.ini was changed to not write log messages to /var/log/openqa anymore but all to the journal, meaning that openqa_logwarn does not work anymore as in before. Will create follow-up ticket.
Updated by okurz about 5 years ago
- Related to action #57239: Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respectively added
Updated by okurz almost 4 years ago
- Related to action #75067: save_needle minion task fails because "Your account has been blocked" added