Project

General

Profile

action #35290

[tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observed

Added by okurz over 3 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
2018-04-20
Due date:
% Done:

0%

Estimated time:

Description

Observation

dheidler was asking me about needles he created on osd but could not see in the needles gitlab repo.

Checking logfiles on osd in /var/log/openqa

[2018-04-20T14:05:09.0636 CEST] [warn] GitLab: Your account has been blocked.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[2018-04-20T14:05:09.0636 CEST] [error] cmd returned non-zero value
[2018-04-20T14:05:09.0641 CEST] [error] failed to git commit iscsi-target-overview-service-tab-20180420 at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Step.pm line 385.

Problem

nsinger and me gave up to monitor the logfiles with logwarn_openqa as recent contributions to openQA over the past year have introduced the noise in the logfiles so that the alerting was seen as unfeasible for us even more considering that we are not "tools"-team members and became frustrated and I assume no one else is doing that monitoring so that problem is not found easily. Potential outcome is loss of data as needle commits are never pushed.


Related issues

Related to openQA Infrastructure - action #18164: [devops][tools] monitoring of openqa worker instancesResolved2018-04-25

Related to openQA Infrastructure - action #42920: Gitlab user gets blocked: Try out gitlab deployment tokens for openQAResolved2018-10-25

Related to openQA Infrastructure - action #57239: Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respectivelyWorkable2019-09-23

Related to openQA Infrastructure - action #75067: save_needle minion task fails because "Your account has been blocked"Resolved2020-10-22

History

#1 Updated by szarate over 3 years ago

  • Related to action #18164: [devops][tools] monitoring of openqa worker instances added

#2 Updated by okurz over 3 years ago

FYI, On 2018-03-13 I have sent the email that I removed the cronjob from osd and AFAIK have not received any answer or objection. Funny enough, on o3 everything is still in place but seems to have never worked. I guess email sending from o3 is broken.

/etc/cron.d/logwarn_openqa

SHELL=/bin/bash
MAIL=okurz@suse.de
MAILTO=okurz@suse.de,nsinger@suse.de
REPORTTO=okurz@suse.de,szarate@suse.de,nsinger@suse.de
PATH=/suse/bin:/usr/lib/ccache:/opt/bin:/usr/local/bin:/usr/bin:/bin
10 * * * * root /opt/openqa-logwarn/pretty_logwarn /var/log/openqa 2>&1

and /opt/openqa-logwarn/ is the git working copy of https://github.com/os-autoinst/openqa-logwarn.git

If anyone wants to try that again then one can just apply the same again on osd and fix the email sending from o3.

#3 Updated by okurz over 2 years ago

  • Related to action #42920: Gitlab user gets blocked: Try out gitlab deployment tokens for openQA added

#4 Updated by okurz over 2 years ago

  • Status changed from New to Feedback
  • Assignee set to okurz

ok, with #42920 we have a better approach for the needle pushing to gitlab.s.d . Also I joined the tools team and might be of more help in this regard. I enabled openqa-logwarn on o3 again. Also we have still #18164 for further monitoring.

Gathering feedback regarding openqa-logwarn from o3.

#5 Updated by okurz over 2 years ago

  • Project changed from openQA Project to openQA Infrastructure
  • Category deleted (168)

#6 Updated by okurz about 2 years ago

I created a new mailing list o3-admins@suse.de . So far I can handle the load for o3 on the mailing and excluded some known warnings. Should we do the same for osd?

EDIT: We have a mailing list osd-admins@suse.de now which is used for example for grafana monitoring messages. Also o3:/etc/openqa/openqa.ini was changed to not write log messages to /var/log/openqa anymore but all to the journal, meaning that openqa_logwarn does not work anymore as in before. Will create follow-up ticket.

#7 Updated by okurz almost 2 years ago

  • Status changed from Feedback to Resolved

#8 Updated by okurz almost 2 years ago

  • Related to action #57239: Add/fix openqa_logwarn for o3 and osd sending to o3-admins@suse.de and osd-admins@suse.de respectively added

#9 Updated by okurz 11 months ago

  • Related to action #75067: save_needle minion task fails because "Your account has been blocked" added

Also available in: Atom PDF