action #35290: [tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observed - openQA Infrastructure - openSUSE Project Management Tool

Actions

Copy link

action #35290

closed

[tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observed

Added by okurz almost 6 years ago. Updated over 4 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

okurz

Category:

Target version:

Start date:

2018-04-20

Due date:

% Done:

Estimated time:

Description

Observation¶

dheidler was asking me about needles he created on osd but could not see in the needles gitlab repo.

Checking logfiles on osd in /var/log/openqa

[2018-04-20T14:05:09.0636 CEST] [warn] GitLab: Your account has been blocked.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[2018-04-20T14:05:09.0636 CEST] [error] cmd returned non-zero value
[2018-04-20T14:05:09.0641 CEST] [error] failed to git commit iscsi-target-overview-service-tab-20180420 at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Step.pm line 385.

Problem¶

nsinger and me gave up to monitor the logfiles with logwarn_openqa as recent contributions to openQA over the past year have introduced the noise in the logfiles so that the alerting was seen as unfeasible for us even more considering that we are not "tools"-team members and became frustrated and I assume no one else is doing that monitoring so that problem is not found easily. Potential outcome is loss of data as needle commits are never pushed.

Related issues 4 (1 open — 3 closed)

Actions

Copy link

Updated by szarate almost 6 years ago

Related to action #18164: [devops][tools] monitoring of openqa worker instances added

Actions

Copy link

Updated by okurz almost 6 years ago

FYI, On 2018-03-13 I have sent the email that I removed the cronjob from osd and AFAIK have not received any answer or objection. Funny enough, on o3 everything is still in place but seems to have never worked. I guess email sending from o3 is broken.

/etc/cron.d/logwarn_openqa

SHELL=/bin/bash
MAIL=okurz@suse.de
MAILTO=okurz@suse.de,nsinger@suse.de
REPORTTO=okurz@suse.de,szarate@suse.de,nsinger@suse.de
PATH=/suse/bin:/usr/lib/ccache:/opt/bin:/usr/local/bin:/usr/bin:/bin
10 * * * * root /opt/openqa-logwarn/pretty_logwarn /var/log/openqa 2>&1

and /opt/openqa-logwarn/ is the git working copy of https://github.com/os-autoinst/openqa-logwarn.git

If anyone wants to try that again then one can just apply the same again on osd and fix the email sending from o3.

Actions

Copy link

Updated by okurz about 5 years ago

Related to action #42920: Gitlab user gets blocked: Try out gitlab deployment tokens for openQA added

Actions

Copy link

Updated by okurz almost 5 years ago

Status changed from New to Feedback
Assignee set to okurz

ok, with #42920 we have a better approach for the needle pushing to gitlab.s.d . Also I joined the tools team and might be of more help in this regard. I enabled openqa-logwarn on o3 again. Also we have still #18164 for further monitoring.

Gathering feedback regarding openqa-logwarn from o3.

Actions

Copy link

Updated by okurz almost 5 years ago

Project changed from openQA Project to openQA Infrastructure
Category deleted (~~168~~)

Actions

Copy link

Updated by okurz almost 5 years ago

I created a new mailing list o3-admins@suse.de . So far I can handle the load for o3 on the mailing and excluded some known warnings. Should we do the same for osd?

EDIT: We have a mailing list osd-admins@suse.de now which is used for example for grafana monitoring messages. Also o3:/etc/openqa/openqa.ini was changed to not write log messages to /var/log/openqa anymore but all to the journal, meaning that openqa_logwarn does not work anymore as in before. Will create follow-up ticket.

Actions

Copy link