[tools] again needles could not be pushed from osd to gitlab.suse.de due to "account has been blocked" and apparently no monitoring alert about this was observed
dheidler was asking me about needles he created on osd but could not see in the needles gitlab repo.
Checking logfiles on osd in /var/log/openqa
[2018-04-20T14:05:09.0636 CEST] [warn] GitLab: Your account has been blocked. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. [2018-04-20T14:05:09.0636 CEST] [error] cmd returned non-zero value [2018-04-20T14:05:09.0641 CEST] [error] failed to git commit iscsi-target-overview-service-tab-20180420 at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Step.pm line 385.
nsinger and me gave up to monitor the logfiles with logwarn_openqa as recent contributions to openQA over the past year have introduced the noise in the logfiles so that the alerting was seen as unfeasible for us even more considering that we are not "tools"-team members and became frustrated and I assume no one else is doing that monitoring so that problem is not found easily. Potential outcome is loss of data as needle commits are never pushed.
#1 Updated by szarate almost 5 years ago
- Related to action #18164: [devops][tools] monitoring of openqa worker instances added
#2 Updated by okurz almost 5 years ago
FYI, On 2018-03-13 I have sent the email that I removed the cronjob from osd and AFAIK have not received any answer or objection. Funny enough, on o3 everything is still in place but seems to have never worked. I guess email sending from o3 is broken.
SHELL=/bin/bash MAILfirstname.lastname@example.org MAILTOemail@example.com,firstname.lastname@example.org REPORTTOemail@example.com,firstname.lastname@example.org,email@example.com PATH=/suse/bin:/usr/lib/ccache:/opt/bin:/usr/local/bin:/usr/bin:/bin 10 * * * * root /opt/openqa-logwarn/pretty_logwarn /var/log/openqa 2>&1
/opt/openqa-logwarn/ is the git working copy of https://github.com/os-autoinst/openqa-logwarn.git
If anyone wants to try that again then one can just apply the same again on osd and fix the email sending from o3.
#3 Updated by okurz about 4 years ago
- Related to action #42920: Gitlab user gets blocked: Try out gitlab deployment tokens for openQA added
#4 Updated by okurz almost 4 years ago
- Status changed from New to Feedback
- Assignee set to okurz
#5 Updated by okurz almost 4 years ago
- Project changed from openQA Project to openQA Infrastructure
- Category deleted (
#6 Updated by okurz over 3 years ago
I created a new mailing list firstname.lastname@example.org . So far I can handle the load for o3 on the mailing and excluded some known warnings. Should we do the same for osd?
EDIT: We have a mailing list email@example.com now which is used for example for grafana monitoring messages. Also o3:/etc/openqa/openqa.ini was changed to not write log messages to /var/log/openqa anymore but all to the journal, meaning that openqa_logwarn does not work anymore as in before. Will create follow-up ticket.
#7 Updated by okurz over 3 years ago
- Status changed from Feedback to Resolved
#8 Updated by okurz over 3 years ago
- Related to action #57239: Add/fix openqa_logwarn for o3 and osd sending to firstname.lastname@example.org and email@example.com respectively added
#9 Updated by okurz over 2 years ago
- Related to action #75067: save_needle minion task fails because "Your account has been blocked" added