Project

General

Profile

Actions
Copy link

action #160883

closed

salt-states-openqa CI complains about "Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa.git/'" size:S

Added by jbaier_cz about 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
dheidler
Category:
-
Target version:
openQA Project - Ready
Start date:
Due date:
2024-06-13
% Done:

0%

Estimated time:
Tags:
alert, infra, reactive work

Description

Observation

CI jobs like https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2650180 complain with

+ git fetch origin
remote: HTTP Basic: Access denied. The provided password or token is incorrect or your account has 2FA enabled and you must use a personal access token instead of a password. See https://gitlab.suse.de/help/topics/git/troubleshooting_git#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa.git/'

Acceptance criteria

  • AC1: We are receiving reminders about expired GitLab tokens

Suggestions

  • DONE: Renew token
  • Check whether we got a reminder e-mail from GitLab and if not how to enable it

Files

Download all files
clipboard-202405291116-ysqbd.png (37.3 KB) clipboard-202405291116-ysqbd.png dheidler, 2024-05-29 09:16
clipboard-202405311130-kfnwv.png (47.5 KB) clipboard-202405311130-kfnwv.png dheidler, 2024-05-31 09:30

Related issues 1 (0 open — 1 closed)

Copied from openQA Infrastructure - action #116845: salt-states-openqa CI complains about "fatal: Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa/'" but no useful hint size:MResolvedmkittler2022-09-20

Actions
Actions
Copy link
 #1

Updated by jbaier_cz about 2 months ago

  • Copied from action #116845: salt-states-openqa CI complains about "fatal: Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa/'" but no useful hint size:M added
Actions
Copy link
 #2

Updated by nicksinger about 2 months ago

  • Priority changed from High to Normal

So what we're seeing here is the impact of https://about.gitlab.com/blog/2023/10/25/access-token-lifetime-limits/ .
I mitigated the urgency by generating two new tokens (could be reduced to one if we create a group scoped token), updating the pipeline variables (are they actually used?) and setting the remotes on OSD. This gives us 1y of time to come up with a proper automated solution.

Actions
Copy link
 #3

Updated by mkittler about 2 months ago

  • Tags changed from alert, reactive work to alert, reactive work, infra
  • Subject changed from salt-states-openqa CI complains about "Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa.git/'" to salt-states-openqa CI complains about "Authentication failed for 'https://gitlab.suse.de/openqa/salt-pillars-openqa.git/'" size:S
  • Description updated (diff)
  • Status changed from New to Workable
Actions
Copy link
 #4

Updated by dheidler about 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to dheidler
Actions
Copy link
 #5

Updated by dheidler about 2 months ago

https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html#when-personal-access-tokens-expire

  • GitLab runs a check at 01:00 AM UTC every day to identify personal access tokens that expire in the next seven days. The owners of these tokens are notified by email.
  • GitLab runs a check at 02:00 AM UTC every day to identify personal access tokens that expire on the current date. The owners of these tokens are notified by email.
Actions
Copy link
 #6

Updated by dheidler about 2 months ago

Let's see if and what notifications I get in the coming night:

Actions
Copy link
 #7

Updated by openqa_review about 2 months ago

  • Due date set to 2024-06-13

Setting due date based on mean cycle time of SUSE QE Tools

Actions
Copy link
 #9

Updated by okurz about 1 month ago

please try to login as qe-admins-bot in private browser window to gitlab, see https://gitlab.suse.de/openqa/password/-/blob/main/password?ref_type=heads#L22 , create token valid as long as possible and use that in the salt states pipeline that failed

Actions
Copy link
 #10

Updated by dheidler about 1 month ago

  • Status changed from In Progress to Blocked

As this account doesn't exist, I created a new one and opened https://sd.suse.com/servicedesk/customer/portal/1/SD-158524 to allow it to access gitlab.

Actions
Copy link
 #12

Updated by dheidler about 1 month ago

  • Status changed from Blocked to In Progress
Actions
Copy link
 #13

Updated by dheidler about 1 month ago

I guess all of the notifications that I expected were actually sent.
Most of them found their way into my spam folder, though.
I opened https://sd.suse.com/servicedesk/customer/portal/1/SD-159090 for that.

In the meantime https://sd.suse.com/servicedesk/customer/portal/1/SD-158524 was completed and I can login to gitlab with that account.

Actions
Copy link
 #14

Updated by dheidler about 1 month ago

  • Status changed from In Progress to Resolved

Replaced the token.
We will get a notification to the osd-admins mailing list,
when the token expires the next time.

Actions
Copy link

Also available in: Atom PDF