QA » openQA Project » openQA Infrastructure

Actually the move has been conducted by mmaher and should be revertable with a simple switch update. https://racktables.nue.suse.com/index.php?page=object&object_id=4994 does not reflect this. Probably needs a manual update.

Remove from osd:

salt-key -y -d openqaworker7.suse.de

IPMI access to the machine still works as in before. I connected and rebooted the machine. At this point it still needs updated network config though. The osd worker machines are configured for DHCP. openqaworker1, openqaworker4, aarch64 and rebel have statically configured IPv4 addresses but I would prefer to use DHCP which we also use for power8 and imagetester.

• Added entry on o3 to /etc/dnsmasq.d/openqa.conf

dhcp-host=54:ab:3a:24:34:b8,openqaworker7

• Added entry to /etc/hosts which dnsmasq should pick up to give out a DHCP lease

192.168.112.12   openqaworker7.openqanet.opensuse.org openqaworker7

• Reload dnsmasq with systemctl restart dnsmasq
• Restarted network on openqaworker7 (over IMPI) using systemctl restart network and monitoring in o3:journalctl -f -u dnsmasq and adress is assigned:

Feb 29 10:48:30 ariel dnsmasq[28105]: read /etc/hosts - 30 addresses
Feb 29 10:48:54 ariel dnsmasq-dhcp[28105]: DHCPREQUEST(eth1) 10.160.1.101 54:ab:3a:24:34:b8
Feb 29 10:48:54 ariel dnsmasq-dhcp[28105]: DHCPNAK(eth1) 10.160.1.101 54:ab:3a:24:34:b8 wrong network
Feb 29 10:49:10 ariel dnsmasq-dhcp[28105]: DHCPDISCOVER(eth1) 54:ab:3a:24:34:b8
Feb 29 10:49:10 ariel dnsmasq-dhcp[28105]: DHCPOFFER(eth1) 192.168.112.12 54:ab:3a:24:34:b8
Feb 29 10:49:10 ariel dnsmasq-dhcp[28105]: DHCPREQUEST(eth1) 192.168.112.12 54:ab:3a:24:34:b8
Feb 29 10:49:10 ariel dnsmasq-dhcp[28105]: DHCPACK(eth1) 192.168.112.12 54:ab:3a:24:34:b8 openqaworker7

• Changed root password to o3 one
• Added my ssh key to openqaworker7:/root/.ssh/authorized_keys
• Updated /etc/openqa/client.conf with the same key as used on other workers for "openqa1-opensuse"
• Updated /etc/openqa/workers.ini with similar config as used on other workers, e.g. openqaworker4

# diff -Naur /etc/openqa/workers.ini{.osd,}
--- /etc/openqa/workers.ini.osd 2020-02-29 15:21:47.737998821 +0100
+++ /etc/openqa/workers.ini     2020-02-29 15:22:53.334464958 +0100
@@ -1,17 +1,10 @@
-# This file is generated by salt - don't touch
-# Hosted on https://gitlab.suse.de/openqa/salt-pillars-openqa
-# numofworkers: 10
-
 [global]
-HOST=openqa.suse.de
-CACHEDIRECTORY=/var/lib/openqa/cache
-LOG_LEVEL=debug
-WORKER_CLASS=qemu_x86_64,qemu_x86_64_staging,tap,openqaworker7
-WORKER_HOSTNAME=10.160.1.101
-
-[1]
-WORKER_CLASS=qemu_x86_64,qemu_x86_64_staging,tap,qemu_x86_64_ibft,openqaworker7
+HOST=http://openqa1-opensuse
+WORKER_HOSTNAME=192.168.112.12
+CACHEDIRECTORY = /var/lib/openqa/cache
+CACHELIMIT = 50
+WORKER_CLASS = openqaworker7,qemu_x86_64

-[openqa.suse.de]
-TESTPOOLSERVER = rsync://openqa.suse.de/tests
+[http://openqa1-opensuse]
+TESTPOOLSERVER = rsync://openqa1-opensuse/tests

• First successful o3 job: https://openqa.opensuse.org/tests/1191358

As the machine is not a transactional-server in contrast to openqaworker1 and openqaworker4 there are the following options: Keep as is and handle like power8 (also not transactional), enable transactional updates w/o root being r/o, change to root being r/o on-the-fly, reinstall as transactional. I guess I will try with option 2 first. So what I did:

• Remove OSD specifics

systemctl disable --now auto-update.timer salt-minion telegraf
for i in  NPI SUSE_CA telegraf-monitoring; do zypper rr $i; done
zypper -n dup --force-resolution --allow-vendor-change

• enable transactional updates

zypper -n in transactional-update
systemctl enable --now transactional-update.timer rebootmgr

As the latest openQA job could successfully finish I enabled more worker instances

systemctl unmask openqa-worker@{2..14} && systemctl enable --now openqa-worker@{2..14}

Will monitor if jobs are running fine and also if the nightly update works.

EDIT: Some tests were failing incomplete due to missing staging uefi images again, see #63382

EDIT: 2020-03-01: Automatic updates+reboot worked:

Mar 01 00:08:26 openqaworker7 transactional-update[10933]: Calling zypper up
…
Mar 01 00:08:51 openqaworker7 transactional-update[10933]: transactional-update finished - informed rebootmgr
Mar 01 00:08:51 openqaworker7 systemd[1]: Started Update the system.
…
Mar 01 03:30:00 openqaworker7 rebootmgrd[40760]: rebootmgr: reboot triggered now!
…
Mar 01 03:36:32 openqaworker7 systemd[1]: Reached target openQA Worker.

Keep in mind. The system is configured for multi-machine tests from OSD time but I have not tested nor enabled the worker classes with "tap". Also firewall+apparmor configuration might still be differing.

• Enable apparmor

zypper -n in apparmor-utils
systemctl unmask apparmor
systemctl enable --now apparmor

• Switch firewall from SuSEfirewall2 to firewalld

zypper -n in firewalld && zypper -n rm SuSEfirewall2
systemctl enable --now firewalld
firewall-cmd --zone=trusted --add-interface=br1
firewall-cmd --set-default-zone trusted
firewall-cmd --zone=trusted --add-masquerade

EDIT: Created new section in https://progress.opensuse.org/projects/openqav3/wiki/Wiki#Infrastructure-setup-for-o3-openqaopensuseorg with all necessary instructions for moving from osd to o3 based on above's comments.