Project

General

Profile

action #104673

Access to o3 workers is not well-documented and not automated

Added by cdywan 4 months ago. Updated 4 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
Due date:
2022-01-24
% Done:

0%

Estimated time:

Description

Observation

Despite having gotten access to o3 and o3 workers in the past, it's taken me multiple days now figuring out how to access the machines which jobs run on, and which I'm trying to investigate.

https://progress.opensuse.org/issues/103683#note-15
https://progress.opensuse.org/issues/103683#note-17

The Tools team wiki mentions asking for access to o3 but not workers. Individual accounts seem to confirm there's no automatic key deployment.

Acceptance criteria

  • AC1: The wiki documents how to gain access to all o3 workers
  • AC2: Bus factor > 1 in case of e.g. public holidays

Suggestions

  • Document who can deploy keys to users w/o access
  • Provide a (link to a) list of o3 workers which need to be accessible i.e. avoid the situation where it's totally unclear where a machine runs and who owns it
  • Implement deployment of SSH keys to all o3 workers

Related issues

Blocks openQA Infrastructure - action #103683: [tools][sle][x86_64][aarch64][QEMUTPM] install package "swtpm" on x86_64 and aarch64 workersResolved2021-12-082022-01-14

History

#1 Updated by cdywan 4 months ago

  • Copied from action #103683: [tools][sle][x86_64][aarch64][QEMUTPM] install package "swtpm" on x86_64 and aarch64 workers added

#2 Updated by cdywan 4 months ago

  • Copied from deleted (action #103683: [tools][sle][x86_64][aarch64][QEMUTPM] install package "swtpm" on x86_64 and aarch64 workers)

#3 Updated by cdywan 4 months ago

  • Blocks action #103683: [tools][sle][x86_64][aarch64][QEMUTPM] install package "swtpm" on x86_64 and aarch64 workers added

#4 Updated by ggardet_arm 4 months ago

For aarch64, we currently have:

  • on openSUSE Network (reachable from gate.opensuse.org):
    • openqa-aarch64:
  • Remote (owner: ggardet_arm):
    • ip-10-0-0-58
    • oss-cobbler-03
    • siodtw01 (for tests on Raspberry Pi 2,3,4)

#5 Updated by okurz 4 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz

#6 Updated by okurz 4 months ago

  • Due date set to 2022-01-24
  • Status changed from In Progress to Feedback

I extended the wiki in https://progress.opensuse.org/projects/openqav3/wiki/Wiki/diff?utf8=%E2%9C%93&version=141&version_from=140&commit=View+differences with "Extend and update o3 worker access instructions, additional hints for password and key-based authentication and list of remote machines maintained by ggardet_arm".

That should cover AC1+AC2 including "Implement deployment of SSH keys to all o3 workers" which needs to be triggered manually by each user but can be done with one simple command using the wiki-documented for-loop. Ok with that?

#7 Updated by cdywan 4 months ago

okurz wrote:

That should cover AC1+AC2 including "Implement deployment of SSH keys to all o3 workers" which needs to be triggered manually by each user but can be done with one simple command using the wiki-documented for-loop. Ok with that?

This sounds pretty nice. I do have a couple questions, though, on the specific steps.

I don't get this last part under SSH configuration:

It is suggested that you use key-based authentication. For this put your ssh keys on all the workers, e.g. using the above configuration and ssh-copy-id.

How would I execute ssh-copy-id on all workers w/o having my key on them? I do have a guess based on the next section...

To execute commands manually on all workers within the o3 infrastructure one can do for example the following:
for i in aarch64 openqaworker1 openqaworker4 openqaworker7 power8 imagetester rebel; do echo $i && ssh root@$i "(transactional-update -n dup || zypper -n dup) && reboot" ; done

This looks like one would login via root, using the password obtained from someone else. To be honest, I'd rather stick to my user and SSH. Which is seemingly even recommended above 🤔️

#8 Updated by okurz 4 months ago

  • Status changed from Feedback to Resolved

cdywan wrote:

okurz wrote:

That should cover AC1+AC2 including "Implement deployment of SSH keys to all o3 workers" which needs to be triggered manually by each user but can be done with one simple command using the wiki-documented for-loop. Ok with that?

This sounds pretty nice. I do have a couple questions, though, on the specific steps.

I don't get this last part under SSH configuration:

It is suggested that you use key-based authentication. For this put your ssh keys on all the workers, e.g. using the above configuration and ssh-copy-id.

How would I execute ssh-copy-id on all workers w/o having my key on them? I do have a guess based on the next section...

Using the password once.

To execute commands manually on all workers within the o3 infrastructure one can do for example the following:
for i in aarch64 openqaworker1 openqaworker4 openqaworker7 power8 imagetester rebel; do echo $i && ssh root@$i "(transactional-update -n dup || zypper -n dup) && reboot" ; done

This looks like one would login via root, using the password obtained from someone else. To be honest, I'd rather stick to my user and SSH. Which is seemingly even recommended above

You are free to create your personal user account on worker hosts but this is not done automatically for you.

Also available in: Atom PDF