Project

General

Profile

action #111998

Make our SLE related tooling work with upcoming changes to build.suse.de (2FA and ssh key based authentication) size:M

Added by okurz 3 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2022-06-03
Due date:
% Done:

0%

Estimated time:

Description

Motivation

IBS is switching to 2FA and ssh key based authentication, see https://mailman.suse.de/mlarch/SuSE/research/2022/research.2022.06/msg00003.html . We need to ensure our tools work with that.

Acceptance criteria

  • AC1: All relevant projects have been updated to support 2FA and/or ssh key based authentication

Suggestions

History

#1 Updated by cdywan 2 months ago

The phrasing is very generic. okurz can you narrow down what tooling needs to be adapted? Or should this be an epic with that as a research task? We can't plan work items when it's unclear what workflows need to be adapted.

#2 Updated by okurz 2 months ago

cdywan wrote:

The phrasing is very generic. okurz can you narrow down what tooling needs to be adapted?

No, I can't. Identification of which tooling needs adaption is part of the work.

Or should this be an epic with that as a research task? We can't plan work items when it's unclear what workflows need to be adapted.

My assumption is that the effort is limited. Of course if we put our heads together and do find out that it's more effort than workable in a single ticket then we can extend to an epic.

#3 Updated by tinita 2 months ago

  • Subject changed from Make our SLE related tooling work with upcoming changes to build.suse.de (2FA and ssh key based authentication) to Make our SLE related tooling work with upcoming changes to build.suse.de (2FA and ssh key based authentication) size:M
  • Description updated (diff)
  • Status changed from New to Workable

#4 Updated by mkittler 2 months ago

  • Assignee set to mkittler

#5 Updated by mkittler 2 months ago

Ok, so that's what I've gathered so far:

  1. Add SSH key /home/teregen/.ssh/id_rsa.pub and qam_automation in /home/teregen/.ssh/authorized_keys (both files are on qam2.suse.de) to https://idp-mfa.suse.de (not https://idp-mfa-test1.suse.de/)
    1. When logging in with my personal account I can enroll a public SSH key so that looks already good. I can also enroll a totp token but that doesn't enable me to login at https://buildtest.suse.de/. It accepts my credentials but the totp is rejected with "Error : wrong otp value". Note that the totp works on https://idp-mfa-test1.suse.de/ itself instead (and can be used instead of using the mail token). I hope the SSH key will be sufficient to solve this ticket.
      1. It is still strange that the SSH key shows up wrongly when viewing it later. With that I mean the SSH key shown in the row "Info" is just wrong. The same happens when adding the key once more but a different randomly wrong key shows up. Apparently normal.
      2. When just pasting the key (without ssh-rsa and without trailing comment) I'm getting a 500 response but then nevertheless an empty sshkey is enrolled. A very well written web application…
    2. For the production setup, I suppose I need to login via user https://build.suse.de/users/qa-maintenance
    3. Who has credentials and where would e-mail with TOTP end up? asked Heiko Rommel
  2. Find out what changes in the OSC config are needed
    1. Locally it still asks me to put username and password into oscrc, despite having ssh-agent setup and despite having the SSH key deployed on https://idp-mfa-test1.suse.de/. So I'm still clueless what to do here to make it work. In the mail it sounds like no further configuration would be required but that is apparently not the case. Locally it works by providing user and password. The SSH key is determined automatically but could also be specified via sshkey = id_rsa.pub.
      1. It supposedly works to use sshkey = id_rsa.pub as that's apparently the newly introduced option (see https://github.com/openSUSE/osc/blob/0b4158590ce5323028c744cd3ded005652c455d6/NEWS#L19=) and judging by error messages one gets when just specifying e.g. sshkey = foo it is searching for files under one's .ssh directory. Still doesn't work for me, though. Maybe due to 1.1.2.
      2. Note that it also still wants a username and password if sshkey is present. At least the password should no longer be required so something must be wrong.
      3. I've been using OSC 0.179 as provided by Tumbleweed.
    2. Possibly adjust https://gitlab.suse.de/qa-maintenance/qamops/-/blob/master/ansible/roles/teregen/templates/oscrc.j2 deployed on qam2.suse.de as /home/teregen/.config/osc/oscrc.
      1. Seems unnecessary as /home/teregen/.ssh/id_rsa.pub exists on qam2.suse.de and that key should be picked up automatically according to my local testing.
  3. Ensure all steps using osc on https://gitlab.suse.de/qa-maintenance/qamops/-/blob/master/gitlab/templates.yml have ssh-agent started (in the same way as it is already started in some steps).
  4. To fix qem-bot, we might need to introduce loading a key to same ssh-agent inside the pipeline. In case some software is missing, you can also tweak the image itself: https://build.suse.de/package/view_file/QA:Maintenance/openSUSE-Leap-Container/Dockerfile
  5. Ensure OSC 0.178 or newer is installed on qam2.suse.de and all containers used in GitLab pipelines
    1. Add repo http://download.opensuse.org/repositories/openSUSE:/Tools/SLE_12_SP3/ if the necessary
    2. Supposedly https://build.suse.de/package/view_file/QA:Maintenance/openSUSE-Leap-Container/Dockerfile is covering all GitLab pipelines

#6 Updated by mkittler 2 months ago

  • Status changed from Workable to Feedback

I've been asking Heiko but he doesn't seem to be available so far. I've also asking on the buildservice channel my remaining questions. So for now I'll just be waiting for responses. If anybody in the team has further information it would of course be appreciated as well.


I now also asked on the mailing list because I believe my questions go under on the buildservice channel.

#7 Updated by toe 2 months ago

mkittler wrote:

(...) https://idp-mfa-test1.suse.de/ (...) I can also enroll a totp token but that doesn't enable me to login at https://buildtest.suse.de/. It accepts my credentials but the totp is rejected with "Error : wrong otp value". Note that the totp works on https://idp-mfa-test1.suse.de/ itself instead (and can be used instead of using the mail token).

Please see Adrian's mail from 2022-06-13 11:49 CET. Since yesterday, buildtest.suse.de uses the production MFA tokens which are maintained under https://idp-mfa.suse.de (VPN only)

#8 Updated by mkittler 2 months ago

Adrian has already responded. I've also already testing it locally with my personal account and everything worked. So I'm updating my previous comment.

#9 Updated by mkittler 2 months ago

#10 Updated by mkittler 2 months ago

Regarding 4.: It looks like qem-bot would handle the 2FA just fine as long as oscrc is configured correctly. At least in my local tests I was able to delete a comment in my home project using the following test code:

from osc import conf
from openqabot.osclib.comments import CommentAPI

apiurl = "https://apitest.suse.de"
conf.get_config(override_apiurl=apiurl)
api = CommentAPI(apiurl)
api.delete("3876782")

I also did the opposite check. If I configure a wrong SSH key I get Couldn't load public key /home/martchus/.ssh/id_rsa.pubb: No such file or directory. So it definitely reads the SSH key.

So we only need to ensure that the SSH setup is present in all pipelines where qem-bot is running. I assume I need to adjust:

I've created SR. That means there are currently 3 pending SRs:

#11 Updated by mkittler about 2 months ago

I'm still waiting for a reply from Heiko.

#12 Updated by mkittler about 2 months ago

Heikro replied, enrolled the ssh-key for https://build.suse.de/users/qa-maintenance and I did a test on qam2.suse.de against https://apitest.suse.de and it worked.

So now I'm only waiting on feedback on the pending SRs.

#14 Updated by jbaier_cz about 2 months ago

okurz wrote:

All three https://gitlab.suse.de/qa-maintenance/qamops/-/merge_requests/17 , https://gitlab.suse.de/qa-maintenance/bot-ng/-/merge_requests/52 , https://gitlab.suse.de/qa-maintenance/openQABot/-/merge_requests/95 merged, please monitor pipelines in these three projects.

Small regression https://gitlab.suse.de/qa-maintenance/openQABot/-/jobs/1029191 caused by missing Gitlab variable SSH_KEY. I added it in the variable definitions (webui) and now the pipeline looks good.

#15 Updated by mkittler about 2 months ago

jbaier_cz Thanks for taking care.

#16 Updated by mkittler about 1 month ago

The pipelines look still good. I suppose we'll have to see whether it continues to work when 2FA actually becomes necessary.

#17 Updated by okurz about 1 month ago

  • Due date deleted (2022-07-08)
  • Status changed from Feedback to Resolved

We agreed that we can resolve it as we prepared changes to the best of our knowledge. In case problems would still appear then we would get automatic notifications anyway.

Also available in: Atom PDF