openQA Project (public)

Indeed, this is intentional, but temporarily intentional

Given Salt's habit of exploding spectacularly when the master is not updated but the minions are, and given we patched the minions first and they now auto update, it would be suicidal to have a master running on leap 42.3 o3 talking to minions running on Leap 15.0 workers.

When o3 is also Leap 15 and updated at least as frequently as the workers, then this makes sense, thanks for tracking the item :)

I see, so I created a new ticket #43976 to cover that.

had a chat with lrupp/kl_eisbaer: He made me aware of https://build.opensuse.org/package/show/OBS:Server:Unstable/OBS-WorkerOnly which provides images that are used by OBS workers.

The images are loaded by PXE, the PXE config points to an image - and this image path is a symlink here. Allows to easily switch from one image to the other, if something is identified as broken.

[04/02/2020 10:20:36] <kl_eisbaer> the most funny part is the one that adjusts the worker after the PXE boot.
[04/02/2020 10:20:57] <kl_eisbaer> here we use a script in the init phase, that downloads files/settings from the server
[04/02/2020 10:21:22] <kl_eisbaer> with this, we can adjust the configuration of the worker (how many parallel builds, how much disk space, etc)
[04/02/2020 10:21:44] <kl_eisbaer> okurz: if you are interested, I can give you a short introduction into our setup
[04/02/2020 10:22:36] <kl_eisbaer> ...which even allows us to move workers between OBS/IBS via script since we have access to the switches :-)

My current proposal is the following:

• Ensure salt-minion on all o3 workers
• Ensure salt-master on o3
• Ensure workers are connected to o3 and salt key is accepted
• Move gitlab.suse.de/openqa/salt-states-openqa to github, e.g. in https://github.com/os-autoinst scope, and create back-mirror into salt-states repo or get rid of it completely

Anyone sees problems with this approach?

RBrownSUSE wrote:

Indeed, this is intentional, but temporarily intentional

Given Salt's habit of exploding spectacularly when the master is not updated but the minions are, and given we patched the minions first and they now auto update, it would be suicidal to have a master running on leap 42.3 o3 talking to minions running on Leap 15.0 workers.

When o3 is also Leap 15 and updated at least as frequently as the workers, then this makes sense, thanks for tracking the item :)

@okurz this still applies somewhat. While o3 is on 15.2 in the meantime it still needs manual updates. This raises a few points from my side:

• Isn't the topic "install salt" blocked by the migration of o3 onto transnational servers?
• If salt explodes that spectacularly with non-matching versions; should we maybe look into something like ansible?
• Should we at least deploy a ssh-key onto ariel which can access all workers over ssh and install something like pssh (https://linux.die.net/man/1/pssh)?

nicksinger wrote:

@okurz this still applies somewhat. While o3 is on 15.2 in the meantime it still needs manual updates. This raises a few points from my side:

• Isn't the topic "install salt" blocked by the migration of o3 onto transnational servers?

Saltstack supports transactional systems meanwhile - https://github.com/openSUSE/salt/pull/271

• If salt explodes that spectacularly with non-matching versions; should we maybe look into something like ansible?

Salt can run masterless, in which case the versions are unrelated

• Should we at least deploy a ssh-key onto ariel which can access all workers over ssh and install something like pssh (https://linux.die.net/man/1/pssh)?

RBrownSUSE wrote:

nicksinger wrote:

@okurz this still applies somewhat. While o3 is on 15.2 in the meantime it still needs manual updates. This raises a few points from my side:

• Isn't the topic "install salt" blocked by the migration of o3 onto transnational servers?

Saltstack supports transactional systems meanwhile - https://github.com/openSUSE/salt/pull/271

kind of https://github.com/saltstack/salt/pull/58520 ;)
It also doesn't solve the problem of o3 being upgraded manually so version differences can still happen. Masterless salt is an interesting point you have there. Can it cover our (current) main use-case: execute commands on multiple hosts?

I would not be concerned with version differences until I see that failing. And running salt-master (again) on o3 and a salt-minion on each worker, accepting salt keys, and then simply use it for distributed command execution, e.g. cmd.run is a good start.