openSUSE admin

From SUSE R&D network. We automatically recompute some information (such as GCC testsuite coverage, for example) regularly there and (also automatically) upload it to gcc.opensuse.org via my personal VPN. But in order to keep my VPN certificate really just mine I cannot share the root password with the rest of my team, which is a bit unfortunate when I am away and something needs to be done about the machine. So I wanted to switch to using this port forwarding which was shown to me only shortly before the move.

Thanks for the explanation! And for taking care not to share your VPN credentials. :-)

For access from SUSE Engineering networks we have a jump host setup (one host in the SUSE DMZ, one counterpart in the openSUSE infrastructure) which can be used for SSH to openSUSE networks, including the one gcc.i.o.o is in.

Would using that work as an alternative? If yes, please share the SSH keys I should add!

ssh jump hosts would work just fine, yes. Add this key for me please (Martin will want his own one as well):

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCAcrttRtBsuiwIAyfXbxZQJ5iH4wetCWKDEK7olTCLtz7gFpxElU63fEgG1ENpncxXLHwfD+weCHj2ej1V7DTo= matz@knuth.suse.de

Hello,

a jump host is perfect.

my SSH key is:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK7nhm1sC6YC/B4FRXreTSdqiqB1hz3QOPhruVeW7xbv jamborm-210508R0

SSH key of the worker which uploads stuff to gcc.opensuse.org (so essentially a shared key) is:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOXfa4a7Yrx2Qjdm67TboT62/nD35bOfupgj16KJHBkb worker@tiber.arch.suse.cz

Hi,

here is a sample ssh_config(5) snippet you can use for easy shell access using ssh gcc-stats:

# openSUSE
Host jump-opensuse
        Hostname jump-opensuse.dmz-prg2.suse.org
        User jumpy

Host warp
        Hostname warp.infra.opensuse.org
        ProxyJump jump-opensuse
        User crameleon # your openSUSE Heroes username

Host gcc-stats
        Hostname gcc-stats.infra.opensuse.org
        ProxyJump warp

I added your SSH keys to the shared "jumpy" user on jump-opensuse.dmz-prg2.suse.org. The counterpart on the openSUSE side (warp.infra.opensuse.org) uses Heroes LDAP authentication.

Do you already have a Heroes account with the SSH key of your shared worker or do you use some local account on gcc-stats.i.o.o at the moment? In the latter case, I will create a service account for this purpose.

Here are the accompanying host key fingerprints:

jump-opensuse:

256 SHA256:1IboH5lftclnl1WLPnlQc68BMOpJxlExHtFWKluRQnU root@localhost (ECDSA)
256 SHA256:3hr+IP0eKuOIABlIO8AR2abV5AfzS4p+cTdEN9Yln/A root@localhost (ED25519)

warp:

256 SHA256:mq8QjcYdX8scQwW94IgPJdbx+DzFTYoH42P6j1BHeao root@warp (ECDSA)
256 SHA256:Vw4F/iZBr75V9uDaGQIeA9NLPs4yBidxzS7caJ+yrSY root@warp (ED25519)

crameleon wrote in #note-8:

Do you already have a Heroes account with the SSH key of your shared
worker or do you use some local account on gcc-stats.i.o.o at the
moment? In the latter case, I will create a service account for this
purpose.

We need a service account, we use a local account on gcc-stats (called
"gcc").

Thank you.

OK, I created a user with the same name and added the mentioned SSH key, let me know if it works. If it does, maybe you want to remove the local account to avoid confusion.

I realize, you might need to adjust the UID/GID of the home directory. I hope it doesn't cause any issues, if it does, I can also opt for a different name.

crameleon wrote in #note-8:

here is a sample ssh_config(5) snippet you can use for easy shell access using ssh gcc-stats:
...

Thanks!

I added your SSH keys to the shared "jumpy" user on jump-opensuse.dmz-prg2.suse.org. The counterpart on the openSUSE side (warp.infra.opensuse.org) uses Heroes LDAP authentication.

So, some observations: it seems routing isn't complete between our various subnets:

from my work machine in Frankencampus (10.168.5.16):
% ssh -v jumpy@jump-opensuse.dmz-prg2.suse.org
...
debug1: Connecting to jump-opensuse.dmz-prg2.suse.org [10.151.20.161] port 22.
^C
(i.e. no progress)

It does ping, though.

From wotan it works:

% ssh -v jumpy@jump-opensuse.dmz-prg2.suse.org
...
jumpy@jump-opensuse:~>

wotan has a too old ssh which doesn't support ProxyJump. But it also works from login1.prg2.suse.org,
which is recent enough.

But still something doesn't work quite right from login1. With the given config (my opensuse user is matz2):

% ssh -v gcc@gcc-stats
...
debug1: Authenticating to warp.infra.opensuse.org:22 as 'matz2'
...
debug1: Next authentication method: keyboard-interactive
Password:
Ctrl-C

So, jumping via jump-opensuse still worked, but continuing via warp doesn't anymore.
Changing the config to use 'User gcc' for warp doesn't change that:

...
debug1: Authenticating to warp.infra.opensuse.org:22 as 'gcc'
...
debug1: Next authentication method: keyboard-interactive
Password:

It's quite possible that I don't have an account on warp (at least I don't specifically remember
having ever used that host). But the gcc account should work?

Hi @matz2,

Unfortunately I'm not sure about access to SUSE DMZ hosts from SUSE office networks. You would unfortunately need to make a ticket with SUSE IT for checking whether this is possible. Through a SUSE VPN client or login1.prg2.suse.org it should however work just fine.

The warp machine uses Heroes/LDAP authentication, i.e. your same personal account you would use for any other machine in the openSUSE infrastructure (including gcc-stats.i.o.o) as well as the Heroes VPN. Upon searching for "matz{,2}" however, I do not get any results. Have you not used such an account and the Heroes VPN in the past? Then that's the next thing I'd be creating for you.

Note that SUSE IDP (aka community account) is not the same as openSUSE IPA (aka Heroes account). Sorry if that's confusing.

The gcc account I created is such an openSUSE IPA/Heroes account, and your attempts using it seem to have succeeded:

Nov 19 22:13:41 warp sshd[27743]: Accepted publickey for gcc from 2a07:de40:b280:25::1 port 58858 ssh2: ED25519

Nov 19 22:13:41 gcc-stats.opensuse.org sshd[26515]: Accepted publickey for gcc from 2a07:de40:b27e:1101::a port 50890 ssh2: ED25519

crameleon wrote in #note-14:

Unfortunately I'm not sure about access to SUSE DMZ hosts from SUSE office networks. You would unfortunately need to make a ticket with SUSE IT for checking whether this is possible. Through a SUSE VPN client or login1.prg2.suse.org it should however work just fine.

Well, then I'll defer that for now, and go via login1.prg2, I guess. One thing after another :-)

The warp machine uses Heroes/LDAP authentication, i.e. your same personal account you would use for any other machine in the openSUSE infrastructure (including gcc-stats.i.o.o) as well as the Heroes VPN. Upon searching for "matz{,2}" however, I do not get any results. Have you not used such an account and the Heroes VPN in the past? Then that's the next thing I'd be creating for you.

gcc-stats.i.o.o was created before the heroes accounts were necessary (or even existed, the gcc-stats setup is really old), we've
always used the gcc account for that machine. So, yeah, it's possible that I don't have a matz2 heroes account at all. I am on
the heroes mailing list, and (obviously) also am matz2 on this ticket system ...

Note that SUSE IDP (aka community account) is not the same as openSUSE IPA (aka Heroes account).

... but that seems to be the IDP account :-)

Sorry if that's confusing.

Nah, no different from our other account setups.

The gcc account I created is such an openSUSE IPA/Heroes account, and your attempts using it seem to have succeeded:

Nov 19 22:13:41 warp sshd[27743]: Accepted publickey for gcc from 2a07:de40:b280:25::1 port 58858 ssh2: ED25519

Nov 19 22:13:41 gcc-stats.opensuse.org sshd[26515]: Accepted publickey for gcc from 2a07:de40:b27e:1101::a port 50890 ssh2: ED25519

19th November? Nope, that wasn't me. Have you added only Martins key(s) to the authorized_keys of user gcc?
When I use these snippets in ssh_config:

Host jump-opensuse
    Hostname jump-opensuse.dmz-prg2.suse.org
    User jumpy

Host warp
    Hostname warp.infra.opensuse.org
    ProxyJump jump-opensuse
    User gcc

Host gcc-stats
    Hostname gcc-stats.infra.opensuse.org
    ProxyJump warp

then I get asked for a password from warp about user gcc.

matz2 wrote in #note-15:

crameleon wrote in #note-14:

The gcc account I created is such an openSUSE IPA/Heroes account, and your attempts using it seem to have succeeded:

Nov 19 22:13:41 warp sshd[27743]: Accepted publickey for gcc from 2a07:de40:b280:25::1 port 58858 ssh2: ED25519

Nov 19 22:13:41 gcc-stats.opensuse.org sshd[26515]: Accepted publickey for gcc from 2a07:de40:b27e:1101::a port 50890 ssh2: ED25519

19th November? Nope, that wasn't me. Have you added only Martins key(s) to the authorized_keys of user gcc?

Our buildbot uses gcc account so it was probably the bot (plus I tested it last week manually).

OK, I understand.

The gcc account only has the SSH key "worker@tiber" which was sent in https://progress.opensuse.org/issues/139244#note-6.

I now created a matz2 account as well and added your "matz@knuth" key.
If you ever feel motivated to make a separate ticket to request Heroes VPN access as well, you will be able to use the https://freeipa.infra.opensuse.org/ web GUI to manage the account.

@jamborm So that means the gcc bot is working through the new setup already? That's great news.

crameleon wrote in #note-17:

OK, I understand.

The gcc account only has the SSH key "worker@tiber" which was sent in https://progress.opensuse.org/issues/139244#note-6.

I see. Can you add the matz@knuth key as well? Hmm... I guess I can do that myself once I
figure out the login credentials for tiber again.

I now created a matz2 account as well and added your "matz@knuth" key.

It works now. I can rsync-push to gcc-stats again. Thanks!

If you ever feel motivated to make a separate ticket to request Heroes VPN access as well, you will be able to use the https://freeipa.infra.opensuse.org/ web GUI to manage the account.

Yo, thanks!

crameleon wrote in #note-18:

@jamborm So that means the gcc bot is working through the new setup already? That's great news.

Yes, it seems to work quite nicely. Thanks for setting it up.

Happy to hear the good news!

Can you add the matz@knuth key as well?

I can, though I'm not too keen to add personal keys to the service account as well. I understand this is what you used in the past, but would it be possible to separate this now?

crameleon wrote in #note-21:

Can you add the matz@knuth key as well?

I can, though I'm not too keen to add personal keys to the service account as well. I understand this is what you used in the past, but would it be possible to separate this now?

I guess I can retrieve the ssh key-pair for worker@tiber and distribute it to myself and the gcctest NIS account.
The @tiber key, right now, only sits at that single machine, as we miss real distributed home directories in prague.
The work towards gcc.opensuse.org is done from multiple machines and the real shared worker account is NIS
'gcctest' (which in turn doesn't exist in prague, which is why a former colleague set up a separate one on tiber to
start with). I need to shuffle stuff anyway to other machines, as the users-space meanwhile sits in prague as well,
and so the rsync would be abysmal slow when it's not done from prague machines as well.

Anyway, it's no matter now. The important thing (my crons pushing to gcc.opensuse.org) works again thanks to the
matz2 account on warp. We can take it from there.