QA

Discussed in Slack huddle 2022-10-31. We have about 20 openQA machines. We should create one openQA zone covering everything which is now in .suse.de domain, e.g. openqaworker1 through openqaworker20, also qa-power8, qa-power8-4, qa-power8-5, malbec.arch, grenache, …. And another QA zone combining everything from .qa.suse.de and .qam.suse.de. mcaj will present DNS options shortly.

EDIT: In a huddle with mcaj+lhaleplidis. mcaj can take https://gitlab.suse.de/qa-sle/qanet-configs/ as reference and build salt managed DHCP+DNS structure based on that. New DNS domain for openQA machines, CNAME entries pointing to old entries. mcaj offered he could take over maintainership for the qa domain by having control over the machine qanet and migrate the dhcp+dns config to salt control and then migrate machines to the new zone one by one. We can define a new name scheme for the network and hosts, e.g. .oqa.suse.de with hostname CNAME openqaworker11.suse.de -> A worker11.oqa.suse.de. Regarding remote control there are two ways in the separate zone. /etc/hosts or internal DNS. We prefer internal DNS.

Every employee can request to have their ssh keys added over gitlab merge requests as well as propose config changes. Same goes for automated accounts.

lhaleplidis mentioned that multiple machines miss their L2 addresses in racktables. I went over all machines in https://racktables.nue.suse.com/index.php?page=rack&rack_id=193 and completed L2 addresses using both sudo salt \* cmd.run 'ipmitool lan print | grep "MAC Address"' on OSD as well as logging into the according machines directly and ip link commands. For aarch64.openqanet.opensuse.org racktables was missing the entry for IPMI completely. ipmitool lan print on the machine did return only Set in Progress : Set Complete, not more useful information. Then from login.suse.de I retrieved an L2 address with ip=$(dig openqa-aarch64-ipmi.suse.de A +short); ping -c1 $ip && ip neighbour | grep $ip. I reviewed the complete list of all QA machines as I could and added L2 addresses as far as I managed. In cases of QAM machines I could not find useful responses and I don't know the config so I did not follow up with that but created a specific ticket.

lhaleplidis also mentioned that there is a design plan from Cyber Security to only enable SSH access to machines over a jump host as well. I doubted the RoI is worth enough for that. mcaj+lhaleplidis noted that they are merely executors here. I might need to raise that issue.

I can confirm that openqaworker11 has received a new IPv4 address 10.137.10.11 and I can't ping it from my home network. DNS resolution worker11.oqa.suse.de seems to work although CNAME redirect openqaworker11.suse.de -> worker11.oqa.suse.de does not work. openqaworker11-ipmi.suse.de is still reachable as in before.

ping -c1 worker11.oqa.suse.de works now. next step could be ssh. yes, works. DNS AAAA is missing

(Martin Caj) Can you please meanwhile check jobs on worker11 to see if all is working there fine ?

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/9825072 _GROUP=0 BUILD= TEST=salt-minion-test-migration-poo119443-w11-okurz WORKER_CLASS=openqaworker11

->
Cloning parents of sle-15-SP5-Online-x86_64-Build32.1-salt-minion@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build32.1-salt-master@64bit
Created job #9854444: sle-15-SP5-Online-x86_64-Build32.1-salt-master@64bit -> https://openqa.suse.de/t9854444
Created job #9854443: sle-15-SP5-Online-x86_64-Build32.1-salt-minion@64bit -> https://openqa.suse.de/t9854443

EDIT: It turns out OSD tries to reach worker11.oqa.suse.de over the second interface with IP

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:37:af:f4 brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    altname ens4
    inet 149.44.176.58/21 brd 149.44.183.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe37:aff4/64 scope link 
       valid_lft forever preferred_lft forever

so we agree to use a route setting

sudo ip route add 10.137.10.0/24 via 10.160.255.254
sudo ip route add 2a07:de40:a203:12::/64 via 2620:113:80c0:8080::1

I can confirm that I can ping the host over 4+6 from my home, A+AAAA works

$ for i in 4 6; do ping -c1 -$i worker11.oqa.suse.de; done
PING  (10.137.10.11) 56(84) bytes of data.
64 bytes from worker11.oqa.suse.de (10.137.10.11): icmp_seq=1 ttl=61 time=21.8 ms

---  ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 21.794/21.794/21.794/0.000 ms
PING worker11.oqa.suse.de(2a07:de40:a203:12:ec4:7aff:fe7a:7896 (2a07:de40:a203:12:ec4:7aff:fe7a:7896)) 56 data bytes
64 bytes from 2a07:de40:a203:12:ec4:7aff:fe7a:7896 (2a07:de40:a203:12:ec4:7aff:fe7a:7896): icmp_seq=1 ttl=61 time=22.0 ms

--- worker11.oqa.suse.de ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 21.989/21.989/21.989/0.000 ms

also A+AAAA lookup over the old hostname seems to work out fine:

$ nslookup openqaworker11.suse.de
Server:     127.0.0.1
Address:    127.0.0.1#53

openqaworker11.suse.de  canonical name = worker11.oqa.suse.de.
Name:   worker11.oqa.suse.de
Address: 10.137.10.11
Name:   worker11.oqa.suse.de
Address: 2a07:de40:a203:12:ec4:7aff:fe7a:7896

EDIT: lhaleplidis did further (unknown) changes. Now I can confirm ping 4&6 from openqa.suse.de to worker11.oqa.suse.de works. https://openqa.suse.de/admin/workers/2014 shows worker11 to be online and idle

Now I did a simpler test, no multi-machine test:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/9825060 _GROUP=0 BUILD= TEST=memtest-test-migration-poo119443-w11-okurz WORKER_CLASS=openqaworker11

https://openqa.suse.de/tests/9854942 scheduled and running. The test is running now

I triggered another test

https://openqa.suse.de/tests/9855171 based on rescue_system, passed, so all good for single-machine tests.

I think I understand why the multi-machine tests can't run. I only started a single worker instance so the parallel cluster can't start.

Let's try again:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/9825072 _GROUP=0 BUILD= TEST=salt-minion-test-migration-poo119443-w11-okurz WORKER_CLASS=openqaworker11,tap

-> Created job #9854940: sle-15-SP5-Online-x86_64-Build32.1-salt-minion@64bit -> https://openqa.suse.de/t9854940
-> Created job #9854941: sle-15-SP5-Online-x86_64-Build32.1-salt-master@64bit -> https://openqa.suse.de/t9854941

On worker11

for i in 2 3; do systemctl unmask openqa-worker-auto-restart@$i && systemctl start openqa-worker-auto-restart@$i; done

• DONE: The jobs failed, one example https://openqa.suse.de/tests/9855053 . I deleted the salt-master jobs again as I failed to put them outside production job groups. I asked in https://suse.slack.com/archives/C02CANHLANP/p1667394848857589 -> DONE The solution was to use TEST+= instead.

openqa-clone-job --skip-chained-deps --parental-inheritance --within-instance https://openqa.suse.de/tests/9825072 _GROUP=0 BUILD= TEST+=test-migration-poo119443-w11-okurz WORKER_CLASS=openqaworker11,tap

Cloning parents of sle-15-SP5-Online-x86_64-Build32.1-salt-minion@64bit
Cloning children of sle-15-SP5-Online-x86_64-Build32.1-salt-master@64bit
Created job #9855288: sle-15-SP5-Online-x86_64-Build32.1-salt-master@64bit -> https://openqa.suse.de/t9855288
Created job #9855289: sle-15-SP5-Online-x86_64-Build32.1-salt-minion@64bit -> https://openqa.suse.de/t9855289

but do those jobs actually wait for one another? Let's see. Both passed, all good.

• DONE: @Lazaros Haleplidis I observed multiple times that the ssh connection from my workstation to worker11.oqa.suse.de stopped working whereas a ssh connection to openqa.suse.de stays responsive. After forcefully disconnecting with ~. I can reconnect but the unstable connection is a problem. 2022-11-03 Did not happen anymore.

• TODO: memtest repeatedly fails in https://openqa.suse.de/tests/9855169#step/memtest/12 with memtest not continuing, asking for confirmation regarding multi-threaded operation?

Continuing the migration. vhaleplidis also migrated eth1 on openqaworker11. mcaj suggested for the updated network:

(Martin Caj) I was thinking about the new QA vlan / dns DHCP and how to do it without any big downtime. Since we are going to merge QA and QAM into one vlan/subnet. I have a suggestion to use new domain name. My suggestion is use: qe.suse.de
That will all me to create new DHCP/DNS server (based on your data in gitlab) and we can do host by host migration and also old machine can stay in the old domains (qam.suse.de and qa.suse.de) without any issue.

I approved that plan.

openqaworker11 currently has a 3 hop route to OSD over SUSE's NUE Nexus Core 5696Q switch. I am a bit concerned if that will stay for too long until we have OSD migrated. lhaleplidis plans to migrate a VM like OSD last.

Regarding SSH keys on the IPMI access jump host I suggested to mcaj:

I suggest to use the SSH key list from https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/sshd/users.sls . As alternative you can reference the git repo that you like to have keys in and we can each explicitly request our keys to be added as necessary.

Regarding plans for today:

(Lazaros Haleplidis) do you have a list of the machines, and their preferred order?
(Oliver Kurz) ow12 can be done immediately, no problem. After that my preferred order is that the documentation in rack tables and a generic documentation exists before touching the other production machines. That I consider crucial. For the actual order there is no strong preference so go in any order just please announce it here and update racktables with the according config. I have seen such rush often enough mentioning "we will update later" ending with "we will not document, we have forgotten the details by now". So I can't give green light to go on with production systems if we don't manage to have five lines updated in a wiki page and rather simple updates to racktables. We selected w11 as the first system to conduct the migration so I would like to see the process working there including the update. I hope you understand what I mean.

worker12 was also migrated. Next is openqaworker13, the first production machine. I followed https://progress.opensuse.org/projects/openqav3/wiki/#Take-machines-out-of-salt-controlled-production

worker11:/srv/salt # git pull --rebase origin master
fatal: unable to access 'https://gitlab.suse.de/openqa/salt-states-openqa.git/': Failed to connect to gitlab.suse.de port 443 after 129559 ms: Connection timed out

mentioned in https://suse.slack.com/archives/C0488BZNA5S/p1667488049879659

https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/2781 is adding all users from the OSD infrastructure to the Eng-Infra maintained jump hosts.

> sudo salt --no-color 'openqaworker11*' state.apply
openqaworker11.suse.de:
    Data failed to compile:
----------
    Rendering SLS 'base:openqa.worker' failed: Jinja variable list object has no element 0

same if I call salt-call locally. Not sure what this means. I already tried to use a different name "worker11" instead of "openqaworker11" in /srv/pillar/openqa/workerconf.sls assuming that maybe the host does not match anymore but that does not have an effect. Same is reproduced by sudo salt --no-color 'openqaworker11*' state.apply test=True but not reproduced for sudo salt --no-color 'openqaworker10*' state.apply test=True. But the error is also reproduced for salt-call --local --no-color state.apply test=True. So something is fishy about our machines worker11+worker12.

Reconducting tests on worker13:

ping -6 download.opensuse.org is ok now. Cloning a reference job that failed in before

openqa-clone-job --within-instance https://openqa.suse.de/tests/9887122 _GROUP=0 TEST=test-migration-worker13-okurz BUILD= WORKER_CLASS=qemu_x86_64_debug_119443

Created job #9890472: sle-12-SP5-JeOS-for-kvm-and-xen-Updates-x86_64-Build20221106-1-jeos-extratest@uefi-virtio-vga -> https://openqa.suse.de/t9890472

passed the previously failing suseconnect_scc so that part of communication works as well now.

worker10 was migrated so checking that

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/9871117 _GROUP=0 BUILD= TEST=ltp_crashme_okurz_poo119443 WORKER_CLASS=openqaworker10

Created job #9890500: sle-15-Server-DVD-Incidents-Kernel-KOTD-x86_64-Build4.12.14-150000.293.1.g99629c6-ltp_crashme@64bit -> https://openqa.suse.de/t9890500

test passed so that's good. The developer mode could not connect because the DNS entry for worker10.oqa.suse.de is not complete for both IPv4 and IPv6.

EDIT: 2022-11-07: 14:08Z this was no fixed by mcaj

Testing one more scenario mentioned in open points:

openqa-clone-job --skip-chained-deps --within-instance https://openqa.suse.de/tests/9890596 _GROUP=0 TEST=test-mau-sles-sys-param-check-migration-worker10-okurz BUILD= WORKER_CLASS=openqaworker10

Created job #9891294: sle-15-SP4-Server-DVD-Incidents-x86_64-Build:26478:samba-mau-sles-sys-param-check@64bit-2gbram -> https://openqa.suse.de/t9891294

job passed so likely the access to qa-css-hq.qa.suse.de also works now.

Discussed with Lazaros about allowing/disallowing traffic in particular from within openQA tests. All outbound traffic to the public internet is allowed. Inbound traffic is restricted and needs to be whitelisted case by case. okurz considers we are on a good track regarding critical machines first and then continue with more machines. We will restrict the outbound public internet traffic later to progress with migrating more machines. I again urged to have a dynamic, live overview of what the current rules are. Lazaros has understood and and agrees with that request but can not provide that right now but forwarded it to the cyber security team.

Temporarily over the weekend I had worker13 disabled to prevent unforeseen problems. After we have worked to whitelist multiple connections I have now enabled production use of worker13 again. Same will be done for worker10, worker2 and then the others

the fqdn as returned by salt was still worker13 so I did sudo salt --no-color --state-output=changes -C 'G@roles:worker' saltutil.refresh_grains,grains.get ,fqdn, this worked to update it. Same problem on worker2. I struggled to get the fqdn to update. Locally salt-call grains.get fqdn returned worker2.oqa.suse.de fine. Calling the refresh and get command multiple times worked. Also it seems like get is called first, then the refresh. Well, good now.

The migrated machines in salt pruned the openQA worker config as the existing entries do not match. So for now I removed openqaworker5, openqaworker6, openqaworker8, openqaworker9 from salt keys to prevent the worker config to be deleted until we have found a better way.

Removed openqaworker5 and openqaworker6 from production for now, the next candidates to migrate.

I need to make route entries persistent

ip route add 10.137.10.0/24 via 10.160.255.254
ip route add 2a07:de40:a203:12::/64 via 2620:113:80c0:8080::1
ip route add 10.136.0.0/14 via 10.160.255.254

In /etc/sysconfig/network/routes I added

10.136.0.0/14 10.160.255.254 - -
10.137.10.0/24 10.160.255.254 - -
2a07:de40:a203:12::/64 2620:113:80c0:8080::1 - -

and then did wicked ifup eth0

On openqaworker2 /etc/openqa/workers.ini was missing the entry for the FQDN hostname so tests couldn't upload logs properly. Retriggered all according incomplete and failed jobs from yesterday

WORKER=worker2 result="result='failed'" failed_since=2022-11-07 host=openqa.suse.de openqa-advanced-retrigger-jobs | tee -a worker2_restart_$(date +%F).log
WORKER=worker2 result="result='incomplete'" failed_since=2022-11-07 host=openqa.suse.de openqa-advanced-retrigger-jobs | tee -a worker2_restart_$(date +%F).log

On my openqa-review week, not specific to one squad, I observed today tests failling with networks errors to connect/get/send by curl, on s390 and x86_64. Bellow I get examples to verify

https://openqa.suse.de/tests/9902465#step/install_updates/11 - worker2 - curl timeout
https://openqa.suse.de/tests/9901371#step/iscsi_client/33 - worker10 - curl timeout to http://10.0.2.2
https://openqa.suse.de/tests/9904917#step/bind/6 - worker2 - curl timeout
https://openqa.suse.de/tests/9904919#step/prepare_test_data/9 - worker2 - curl timeout
https://openqa.suse.de/tests/9906141#step/yast2_nfs4_client/91 - worker5 - ping timeout
https://openqa.suse.de/tests/9902287#step/iscsi_client/33 - worker9 - curl timeout

Multi-machine tests need updated bridge_ip settings in salt pillars, https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/456, merged, updated on OSD, then applied the high state on all workers.

https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/457 for worker8+9

osd-deployment is now failing because the name no longer matches, so I'm proposing a trivial fix which works when I manually run this on osd.

cdywan wrote:

osd-deployment is now failing because the name no longer matches, so I'm proposing a trivial fix which works when I manually run this on osd.

thanks. merged https://gitlab.suse.de/openqa/osd-deployment/-/merge_requests/50 . I triggered a new deployment and monitored it and it went fine.

Looking at monitoring data I find worker3 was missing data on https://monitor.qa.suse.de/d/WDworker3/worker-dashboard-worker3?orgId=1 , restarted the telegraf service as the host was not rebooted nor was telegraf restarted after the migration. Now data appeared again just fine. Also the dashboards for former openqaworkers are still there and no empty as the names have changed to the format "worker*" so eventually we can delete the old dashboards. For now I have moved the according dashboards to a separate "folder" in grafana, out of the "salt" folder.