Project

General

Profile

Actions

action #111998

closed

Make our SLE related tooling work with upcoming changes to build.suse.de (2FA and ssh key based authentication) size:M

Added by okurz almost 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
Start date:
2022-06-03
Due date:
% Done:

0%

Estimated time:

Description

Motivation

IBS is switching to 2FA and ssh key based authentication, see https://mailman.suse.de/mlarch/SuSE/research/2022/research.2022.06/msg00003.html . We need to ensure our tools work with that.

Acceptance criteria

  • AC1: All relevant projects have been updated to support 2FA and/or ssh key based authentication

Suggestions


Related issues 1 (0 open1 closed)

Related to QA - action #138446: Ensure SUSE QE tooling always uses authenticated IBS API access size:MResolvedjbaier_cz2023-10-24

Actions
Actions #1

Updated by livdywan almost 2 years ago

The phrasing is very generic. @okurz can you narrow down what tooling needs to be adapted? Or should this be an epic with that as a research task? We can't plan work items when it's unclear what workflows need to be adapted.

Actions #2

Updated by okurz almost 2 years ago

cdywan wrote:

The phrasing is very generic. @okurz can you narrow down what tooling needs to be adapted?

No, I can't. Identification of which tooling needs adaption is part of the work.

Or should this be an epic with that as a research task? We can't plan work items when it's unclear what workflows need to be adapted.

My assumption is that the effort is limited. Of course if we put our heads together and do find out that it's more effort than workable in a single ticket then we can extend to an epic.

Actions #3

Updated by tinita almost 2 years ago

  • Subject changed from Make our SLE related tooling work with upcoming changes to build.suse.de (2FA and ssh key based authentication) to Make our SLE related tooling work with upcoming changes to build.suse.de (2FA and ssh key based authentication) size:M
  • Description updated (diff)
  • Status changed from New to Workable
Actions #4

Updated by mkittler almost 2 years ago

  • Assignee set to mkittler
Actions #5

Updated by mkittler almost 2 years ago

Ok, so that's what I've gathered so far:

  1. Add SSH key /home/teregen/.ssh/id_rsa.pub and qam_automation in /home/teregen/.ssh/authorized_keys (both files are on qam2.suse.de) to https://idp-mfa.suse.de (not https://idp-mfa-test1.suse.de/)
    1. When logging in with my personal account I can enroll a public SSH key so that looks already good. I can also enroll a totp token but that doesn't enable me to login at https://buildtest.suse.de/. It accepts my credentials but the totp is rejected with "Error : wrong otp value". Note that the totp works on https://idp-mfa-test1.suse.de/ itself instead (and can be used instead of using the mail token). I hope the SSH key will be sufficient to solve this ticket.
      1. It is still strange that the SSH key shows up wrongly when viewing it later. With that I mean the SSH key shown in the row "Info" is just wrong. The same happens when adding the key once more but a different randomly wrong key shows up. Apparently normal.
      2. When just pasting the key (without ssh-rsa and without trailing comment) I'm getting a 500 response but then nevertheless an empty sshkey is enrolled. A very well written web application…
    2. For the production setup, I suppose I need to login via user https://build.suse.de/users/qa-maintenance
    3. Who has credentials and where would e-mail with TOTP end up? asked Heiko Rommel
  2. Find out what changes in the OSC config are needed
    1. Locally it still asks me to put username and password into oscrc, despite having ssh-agent setup and despite having the SSH key deployed on https://idp-mfa-test1.suse.de/. So I'm still clueless what to do here to make it work. In the mail it sounds like no further configuration would be required but that is apparently not the case. Locally it works by providing user and password. The SSH key is determined automatically but could also be specified via sshkey = id_rsa.pub.
      1. It supposedly works to use sshkey = id_rsa.pub as that's apparently the newly introduced option (see https://github.com/openSUSE/osc/blob/0b4158590ce5323028c744cd3ded005652c455d6/NEWS#L19=) and judging by error messages one gets when just specifying e.g. sshkey = foo it is searching for files under one's .ssh directory. Still doesn't work for me, though. Maybe due to 1.1.2.
      2. Note that it also still wants a username and password if sshkey is present. At least the password should no longer be required so something must be wrong.
      3. I've been using OSC 0.179 as provided by Tumbleweed.
    2. Possibly adjust https://gitlab.suse.de/qa-maintenance/qamops/-/blob/master/ansible/roles/teregen/templates/oscrc.j2 deployed on qam2.suse.de as /home/teregen/.config/osc/oscrc.
      1. Seems unnecessary as /home/teregen/.ssh/id_rsa.pub exists on qam2.suse.de and that key should be picked up automatically according to my local testing.
  3. Ensure all steps using osc on https://gitlab.suse.de/qa-maintenance/qamops/-/blob/master/gitlab/templates.yml have ssh-agent started (in the same way as it is already started in some steps).
  4. To fix qem-bot, we might need to introduce loading a key to same ssh-agent inside the pipeline. In case some software is missing, you can also tweak the image itself: https://build.suse.de/package/view_file/QA:Maintenance/openSUSE-Leap-Container/Dockerfile
  5. Ensure OSC 0.178 or newer is installed on qam2.suse.de and all containers used in GitLab pipelines
    1. Add repo http://download.opensuse.org/repositories/openSUSE:/Tools/SLE_12_SP3/ if the necessary
    2. Supposedly https://build.suse.de/package/view_file/QA:Maintenance/openSUSE-Leap-Container/Dockerfile is covering all GitLab pipelines
Actions #6

Updated by mkittler almost 2 years ago

  • Status changed from Workable to Feedback

I've been asking Heiko but he doesn't seem to be available so far. I've also asking on the buildservice channel my remaining questions. So for now I'll just be waiting for responses. If anybody in the team has further information it would of course be appreciated as well.


I now also asked on the mailing list because I believe my questions go under on the buildservice channel.

Actions #7

Updated by toe almost 2 years ago

mkittler wrote:

(...) https://idp-mfa-test1.suse.de/ (...) I can also enroll a totp token but that doesn't enable me to login at https://buildtest.suse.de/. It accepts my credentials but the totp is rejected with "Error : wrong otp value". Note that the totp works on https://idp-mfa-test1.suse.de/ itself instead (and can be used instead of using the mail token).

Please see Adrian's mail from 2022-06-13 11:49 CET. Since yesterday, buildtest.suse.de uses the production MFA tokens which are maintained under https://idp-mfa.suse.de (VPN only)

Actions #8

Updated by mkittler almost 2 years ago

Adrian has already responded. I've also already testing it locally with my personal account and everything worked. So I'm updating my previous comment.

Actions #9

Updated by mkittler almost 2 years ago

Actions #10

Updated by mkittler almost 2 years ago

Regarding 4.: It looks like qem-bot would handle the 2FA just fine as long as oscrc is configured correctly. At least in my local tests I was able to delete a comment in my home project using the following test code:

from osc import conf
from openqabot.osclib.comments import CommentAPI

apiurl = "https://apitest.suse.de"
conf.get_config(override_apiurl=apiurl)
api = CommentAPI(apiurl)
api.delete("3876782")

I also did the opposite check. If I configure a wrong SSH key I get Couldn't load public key /home/martchus/.ssh/id_rsa.pubb: No such file or directory. So it definitely reads the SSH key.

So we only need to ensure that the SSH setup is present in all pipelines where qem-bot is running. I assume I need to adjust:

I've created SR. That means there are currently 3 pending SRs:

Actions #11

Updated by mkittler almost 2 years ago

I'm still waiting for a reply from Heiko.

Actions #12

Updated by mkittler almost 2 years ago

Heikro replied, enrolled the ssh-key for https://build.suse.de/users/qa-maintenance and I did a test on qam2.suse.de against https://apitest.suse.de and it worked.

So now I'm only waiting on feedback on the pending SRs.

Actions #14

Updated by jbaier_cz almost 2 years ago

okurz wrote:

All three https://gitlab.suse.de/qa-maintenance/qamops/-/merge_requests/17 , https://gitlab.suse.de/qa-maintenance/bot-ng/-/merge_requests/52 , https://gitlab.suse.de/qa-maintenance/openQABot/-/merge_requests/95 merged, please monitor pipelines in these three projects.

Small regression https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/1029191 caused by missing Gitlab variable SSH_KEY. I added it in the variable definitions (webui) and now the pipeline looks good.

Actions #15

Updated by mkittler almost 2 years ago

@jbaier_cz Thanks for taking care.

Actions #16

Updated by mkittler over 1 year ago

The pipelines look still good. I suppose we'll have to see whether it continues to work when 2FA actually becomes necessary.

Actions #17

Updated by okurz over 1 year ago

  • Due date deleted (2022-07-08)
  • Status changed from Feedback to Resolved

We agreed that we can resolve it as we prepared changes to the best of our knowledge. In case problems would still appear then we would get automatic notifications anyway.

Actions #18

Updated by jbaier_cz over 1 year ago

  • Status changed from Resolved to New
  • Assignee deleted (mkittler)

See: https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/1121347

  File "/usr/lib/python3.6/site-packages/osc/conf.py", line 618, in guess_keyfile
    raise oscerr.OscIOError(None, 'could not guess ssh identity keyfile')

Just a wild guess, we are using ssh-agent and the key is loaded directly from stdin, so it is not on the disk/home folder. Is that even supported by the osc?

Actions #19

Updated by jbaier_cz over 1 year ago

We tried https://gitlab.suse.de/qa-maintenance/openQABot/-/merge_requests/97 to test our theory. It is likely the case, as https://github.com/openSUSE/osc/commit/b8f76f7990eb0d158ee50843e37fe10670fc6672 is not yet included in the osc used in the CI.

Not sure, if the situation improved though: https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/1121802

ERROR: Connection error during reading from IBS: HTTP Error 401: Unauthorized
Actions #21

Updated by osukup over 1 year ago

jbaier_cz wrote:

And now also in bot-ng approve: https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1123024

of course , only part of bot-ng needing IBS:D ( + comments ..)

--> ping openqa-qam revivers approver isn't working and use file approach as in openQABot or inject patched osc into container

Actions #22

Updated by jbaier_cz over 1 year ago

  • Priority changed from High to Urgent

And also template-generator (qa-maintenance account) is unable to connect to IBS with HTTP Error 401: Unauthorized

Actions #23

Updated by osukup over 1 year ago

as @jbaier_cz mentioned in slack conversation we should change contact/user email of these accounts to tools mailing list. --> and as @coolo mentioned for this SD

Actions #25

Updated by jbaier_cz over 1 year ago

The SSH key for template generator (account qa-maintenance) is re-enrolled and seems to be working, new templates are generated.

Both openQABot and bot-ng are still in troubles as they use sle-qam-openqa account.

Actions #26

Updated by livdywan over 1 year ago

  • Status changed from New to Blocked
  • Assignee set to livdywan

This is indeed blocked by SD-97453 since sle-qam-openqa is a personal account, and we can't change it.

To avoid address alert fatigue I'm dropping the "approve incidents" step from GitLab CI.

Actions #27

Updated by livdywan over 1 year ago

cdywan wrote:

This is indeed blocked by SD-97453 since sle-qam-openqa is a personal account, and we can't change it.

To avoid address alert fatigue I'm dropping the "approve incidents" step from GitLab CI.

I now changed BOT_PARAMS to --dry in the approve incidents schedule. The MR was closed.

Actions #28

Updated by jbaier_cz over 1 year ago

  • Status changed from Blocked to Feedback
  • Assignee changed from livdywan to jbaier_cz

And we are back in business: https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1127130

I managed to confirm that login to https://idp-mfa.suse.de/ is possible (the OTP code will arrive via osd-admins). For some reason, we had to reset the password for the account sle-qam-openqa, the configuration is already updated. I also enrolled the necessary SSH key for the approval job and I also remove the dry parameter, so the bot actually does something. Let's hope that this will hold for some time.

Actions #29

Updated by jbaier_cz over 1 year ago

Some more related tweaks, this time for qam-metadata publishing pipeline which also uses IBS

Actions #30

Updated by jbaier_cz over 1 year ago

  • Status changed from Feedback to Resolved
Actions #31

Updated by jbaier_cz 5 months ago

  • Related to action #138446: Ensure SUSE QE tooling always uses authenticated IBS API access size:M added
Actions

Also available in: Atom PDF