action #35290
closed
[tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observed
Added by okurz over 6 years ago.
Updated about 5 years ago.
Description
Observation¶
dheidler was asking me about needles he created on osd but could not see in the needles gitlab repo.
Checking logfiles on osd in /var/log/openqa
[2018-04-20T14:05:09.0636 CEST] [warn] GitLab: Your account has been blocked.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
[2018-04-20T14:05:09.0636 CEST] [error] cmd returned non-zero value
[2018-04-20T14:05:09.0641 CEST] [error] failed to git commit iscsi-target-overview-service-tab-20180420 at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Step.pm line 385.
Problem¶
nsinger and me gave up to monitor the logfiles with logwarn_openqa as recent contributions to openQA over the past year have introduced the noise in the logfiles so that the alerting was seen as unfeasible for us even more considering that we are not "tools"-team members and became frustrated and I assume no one else is doing that monitoring so that problem is not found easily. Potential outcome is loss of data as needle commits are never pushed.
- Related to action #18164: [devops][tools] monitoring of openqa worker instances added
FYI, On 2018-03-13 I have sent the email that I removed the cronjob from osd and AFAIK have not received any answer or objection. Funny enough, on o3 everything is still in place but seems to have never worked. I guess email sending from o3 is broken.
/etc/cron.d/logwarn_openqa
SHELL=/bin/bash
MAIL=okurz@suse.de
MAILTO=okurz@suse.de,nsinger@suse.de
REPORTTO=okurz@suse.de,szarate@suse.de,nsinger@suse.de
PATH=/suse/bin:/usr/lib/ccache:/opt/bin:/usr/local/bin:/usr/bin:/bin
10 * * * * root /opt/openqa-logwarn/pretty_logwarn /var/log/openqa 2>&1
and /opt/openqa-logwarn/ is the git working copy of https://github.com/os-autoinst/openqa-logwarn.git
If anyone wants to try that again then one can just apply the same again on osd and fix the email sending from o3.
- Related to action #42920: Gitlab user gets blocked: Try out gitlab deployment tokens for openQA added
- Status changed from New to Feedback
- Assignee set to okurz
ok, with #42920 we have a better approach for the needle pushing to gitlab.s.d . Also I joined the tools team and might be of more help in this regard. I enabled openqa-logwarn on o3 again. Also we have still #18164 for further monitoring.
Gathering feedback regarding openqa-logwarn from o3.
- Project changed from openQA Project (public) to openQA Infrastructure (public)
- Category deleted (
168)
I created a new mailing list o3-admins@suse.de . So far I can handle the load for o3 on the mailing and excluded some known warnings. Should we do the same for osd?
EDIT: We have a mailing list osd-admins@suse.de now which is used for example for grafana monitoring messages. Also o3:/etc/openqa/openqa.ini was changed to not write log messages to /var/log/openqa anymore but all to the journal, meaning that openqa_logwarn does not work anymore as in before. Will create follow-up ticket.
- Status changed from Feedback to Resolved
- Related to action #57239: Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respectively added
- Related to action #75067: save_needle minion task fails because "Your account has been blocked" added
Also available in: Atom
PDF