openSUSE admin

Email draft:

Hello,

to further strengthen our security posture, I evaluate SSH access behavior to machines.

Historically, people tended to append their SSH keys to the root users's authorized_keys file on a machine.

Using ssh directly as root has drawbacks:
- no per-user auditing
- working with privileges often not needed for the tasks one wants to conduct
- no central management of access

Past benefits such as not being dependent on the authorization server for login are no longer valid since the migration to Kanidm. The Kanidm client running on machines caches your credentials reliably.

Already since some time our canonical way of accessing machines is by using SSH with ones Heroes username, and further to utilize either system groups for rootless tasks or Salt role [1] specific `-admins` groups (managed in our internal Heroes IDP) to elevate to root using `sudo` where needed [2].

This way has so far not been enforced, and legacy implementations were tolerated.

Starting with DD-MM-YYY, this will change, and we will no longer tolerate the practice of connecting via `ssh` to `root` users. Local root users's `authorized_keys` files will be wiped and their content will be enforced and managed by Salt.

- If you are already using your Heroes username to SSH to machines, you do not need to take any action.

- If you are currently using `root` to SSH to machines, please:

* Check if your Heroes user is already part of the `-admins` groups related to the machines you maintain:
  - every machine in our infrastructure already has sudo rule matching the IDP groups `<role>-admins` for all roles assigned to the machine in Salt.
  - members of these groups can use `sudo` to elevate to `root` on machines part of the role.
  - you can easily find out which groups you are member of by ssh-ing to any infra.opensuse.org machine with your Heroes username, and executing `groups` (simple output) or `id` (verbose output).

* If you are already part of the relevant `-admins` groups, verify you can use `sudo` to gain `root` access on the relevant machines.

* If you are missing membership to `-admins` groups for services you maintain, please create a ticket [0] to request being added.

* If machines you maintain are missing Salt roles for services running on them, please submit a merge request to our Salt repository [3] adding them.

Note that we will allow exceptions to the rule in special cases - for example for emergency access to IDP servers to avoid circular dependencies.

Thanks for cooperating!

This effort is tracked via https://progress.opensuse.org/issues/161354.

Best,
Georg

[0] https://progress.opensuse.org/projects/opensuse-admin/issues
[1] https://code.opensuse.org/heroes/salt/blob/production/f/pillar/role
[2] https://code.opensuse.org/heroes/salt/blob/production/f/pillar/common/sudo.sls#_15
[3] https://gitlab.infra.opensuse.org/infra/salt/

Operator's note:

With new requests to join -admins groups coming in, we need to verify that people requesting access are also really maintainers of a given machine or approved by one of them (by checking the "responsible" field in the pillar/id/.sls files in salt.git). If an -admins group does not yet exist, we need to create it prior (so far I only created the -admins groups in Kanidm when there was actually someone to be added).

Yep, I think this sounds good to me and is a really good security improvement :)

Your draft looks good :-)

A possible addition (not sure if it needs to be in your mail) is: if someone asks to get added to an -admin group, they should clarify if they also feel responsible enough ;-) for the machine/role to be added to pillar/id/ as responsible.

.

If an -admins group does not yet exist, we need to create it prior (so far I only created the -admins groups in Kanidm when there was actually someone to be added).

That, or adjust your mail to say

* If you are missing membership to `-admins` groups for services you maintain __or that group doesn't exist yet__, please create a ticket [0] to request being added.

so that you don't have to create several empty and useless groups.

Thank you both for the input.

--

I think we should also state the rules for being in an -admins group in the policy - it's not relevant to the task here, but will be good as -admins groups are going to be utilized more, hence summarizing here. It's only a draft to be formalized, but I think it can already be applied to new requests as it's not changing any existing process. People allowed to be in -admins groups must be ...

... in case of machines with maintainers (i.e. people listed as "responsible" in the respective pillar/id/ files of the machines assigned to the role matching the -admins group in question):

• people who are maintainers themselves
• people who are approved by maintainers

... in case of machines without designated maintainers:

• people who both, are approved by admins in the (IDM managed) wheel group, and have established trust beforehand (doing some of the various tasks which can be accomplished without root access)

--

if someone asks to get added to an -admin group, they should clarify if they also feel responsible enough ;-) for the machine/role to be added to pillar/id/ as responsible.

Of course, we can nudge people if they want to become an additional maintainer of the machines part of the role they request -admins access to - but we should note the responsibilities that come along with this as to not have an influx of new "responsible" people who then do not actually feel responsible.
I also think any existing maintainers should agree with the addition, as ideally maintainers of a machine should collaborate with each other.

--

so that you don't have to create several empty and useless groups.

We don't have to do that anyways, I would keep what I do so far, which is only creating the groups as needed. For users they are either part or not part of a group, regardless of whether it's a group that exists or doesn't, hence I deemed it an implementation detail which can be left out of the announcement.

--

I also thought of adding one more point to the drawbacks list:

- compromise of a users private SSH key yields a larger attack surface

https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/2059

Exception for Kanidm servers: https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/2061 - @firstyear to check if those keys are correct.

Exception for galera1->mybackup bootstrapping: https://gitlab.infra.opensuse.org/infra/salt/-/merge_requests/2062.

Purging applied.