openQA Project (public) » openQA Infrastructure (public)

If we do this, it should also be documented somewhere how to remove the machines from production into a staging setup, and how to return them when done with testing. Not just because i will definitely forget how to do it, but also for new team members.

Maybe worth mentioning that the staging machines tend to be left with unstable test packages installed. Dealing with that will also need a process.

Do we have instructions now on how to move workers between staging and production? I see no mention of that here.

I also think having such instructions figured out should be part of the ticket.

mkittler wrote:

I also think having such instructions figured out should be part of the ticket.

yes, this is why we have "AC1: Our wiki explains how to use the machines as "staging" workers for development or testing and how to bring them back into production" or did you mean something else?

I am roughly thinking of the following:

Use staging machines for manual testing¶

ssh osd "sudo salt-key -y -d $hostname"
ssh $hostname "sudo systemctl disable --now telegraf openqa-worker-auto-restart@\*"

Bring back staging machines into production¶

ssh osd "sudo salt-key -a -d $hostname && sudo salt --state-output=changes $hostname state.apply"

I added https://progress.opensuse.org/projects/openqav3/wiki/Wiki#Take-machines-out-of-salt-controlled-production-eg-for-investigation-or-development . openqaworker12 was unable to boot. I manually disabled the incorrect entries in /etc/fstab, brought back the machine and manually, caused alerts, informed the team, stopped broken openqa worker instances. openqaworker12 can't connect to the openQA webUI due to invalidated or expired key. Updated the key from my user account on openqa-staging-1.qa.suse.de.

Created MR for bringing openqaworker11 and openqaworker12 configuration into salt pillar that includes config for OSD access:
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/325

This is weird:

# zypper -n in openQA-worker
Loading repository data...
Reading installed packages...
'openQA-worker' is already installed.
No update candidate for 'openQA-worker-4.6.1624461400.d2e48c03e-lp152.4096.1.noarch'. The highest available version is already installed.
Resolving package dependencies...
Nothing to do.
openqaworker12:/home/okurz # rpm -q openQA-worker
package openQA-worker is not installed

https://openqa.suse.de/tests/6338980#step/suseconnect_scc/73 on openqaworker12 looks like network within multi-machine jobs does not properly work. On openqaworker12 ovs-vsctl show shows

                options: {remote_ip="10.160.0.227"}
        Port gre6
            Interface gre6
                type: gre
                options: {remote_ip="10.160.2.20"}
                error: "could not add network device gre6 to ofproto (File exists)"

Maybe that's a problem then. Calling sudo salt -l error --state-output=changes -C 'G@roles:worker' cmd.run 'ovs-vsctl show | grep -C 3 error' shows that this problem does not appear elsewhere. Also an error about openqaworker12: 'cmd.run' is not available. I disabled openQA operation on openqaworker12 with systemctl mask --now openqa-worker-cacheservice and did systemctl disable --now telegraf to prevent any false-alerts while the system is intentionally disabled. Called WORKER=openqaworker12 failed_since="2021-06-25" openqa-restart-incompletes-on-worker-instance to restart all incompletes on that worker. To cover all failed as well I did https://github.com/os-autoinst/scripts/pull/86 and called host=openqa.suse.de WORKER=openqaworker12 failed_since="2021-06-25" result=failed bash -ex ./openqa-advanced-retrigger-jobs

Created #94765 to cover openqaworker12. Let me try openqaworker11 then. Cleaned up some duplicate entries in /etc/zypp/repos.d, changed /etc/salt/minion to point to openqa.suse.de, restarted salt-minion, found in /var/log/salt/minion that the salt master pki key needs to be deleted, deleted that and restarted salt-minion again, applied high state from osd with sudo salt -l error --state-output=changes -C 'openqaworker11*' state.apply. This creates a faulty /etc/salt/repos.d/devel_openQA.repo with just a single line keeppackages=1

https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/517

Unfortunately openQA jobs immediately started to be processed on openqaworker11 but packages were not up-to-date causing incompletes like https://openqa.suse.de/tests/6344536 due to an out-of-date os-autoinst. Calling zypper dup helped. Specific verification job: https://openqa.suse.de/tests/6345082#

ovs-vsctl show reveals that openqaworker11 has the same problem as openqaworker12, see #94765 . So with this I created a new ticket for openqaworker11 as well #94783