QA » openQA Project » openQA Infrastructure

Connection to openqa-livehandler is working, checking its log on OSD shows:

Jul 24 15:54:47 openqa openqa-livehandler-daemon[10625]: [debug] [pid:10625] Asking the worker 2289 to start providing livestream for job 15000414
Jul 24 15:54:47 openqa openqa-livehandler-daemon[10625]: [debug] [pid:10625] Asking worker 2289 to start providing livestream for job 15000414
Jul 24 15:54:47 openqa openqa-livehandler-daemon[10625]: [debug] attempt to open ws proxy for job sle-15-SP4-Server-DVD-Updates-x86_64-Build20240721-1-stack_tests_php@64bit (15000414) where URL to os-autoinst command server is unknown
Jul 24 15:55:00 openqa openqa-livehandler-daemon[10625]: [debug] [pid:10625] Asking worker 2289 to stop providing livestream for job 15000414
Jul 24 15:55:01 openqa openqa-livehandler-daemon[10625]: [debug] [pid:10625] Asking the worker 2289 to start providing livestream for job 15000414
Jul 24 15:55:01 openqa openqa-livehandler-daemon[10625]: [debug] [pid:10625] Asking worker 2289 to start providing livestream for job 15000414
Jul 24 15:55:27 openqa openqa-livehandler-daemon[10625]: [debug] [pid:10625] connecting to os-autoinst command server for job 15000414 at ws://openqaworker18.qa.suse.cz:20033/fIRMvu5qm_jM_N14/ws

the last message then repeats every second. I try to follow up with "attempt to open ws proxy for job sle-15-SP4-Server-DVD-Updates-x86_64-Build20240721-1-stack_tests_php@64bit (15000414) where URL to os-autoinst command server is unknown" and try to understand how to fix this keeping https://open.qa/docs/#debugdevelmode in mind

From my JS console

Received message via ws proxy: {"data":{"vnc_arg":"openqaworker18.qa.suse.cz:5993"},"type":"info","what":"connecting to os-autoinst command server at ws:\/\/openqaworker18.qa.suse.cz:20033\/fIRMvu5qm_jM_N14\/ws"} test_result.js:302:9
Received message via ws proxy: {"data":null,"type":"error","what":"unable to upgrade ws to command server"} test_result.js:302:9
Error from ws proxy: unable to upgrade ws to command server test_result.js:312:9
Connection to livehandler lost

not sure yet what this "command server" is supposed to be. Maybe spawned by the worker along side of os-autoinst (naming would indicate this). Services on other workers don't include it as long running service:

openqaworker17:~ # systemctl  | grep -i openqa | grep -vi worker
  var-lib-openqa.mount                                                                 loaded active     mounted      /var/lib/openqa
  openqa.slice                                                                         loaded active     active       Slice /openqa

worker35:~ # systemctl  | grep -i openqa | grep -vi worker
  var-lib-openqa-share.automount                                                 loaded active running   var-lib-openqa-share.automount
  var-lib-openqa-share.mount                                                     loaded active mounted   /var/lib/openqa/share
  var-lib-openqa.mount                                                           loaded active mounted   /var/lib/openqa
  openqa.slice

The documented $worker_instance_number * 10 + 20002 + 1 matches what I see in all debug messages. To test I ran nc -l -p 20033 on worker18 and tried to connect from worker17 (same network) and the connection instantly ends. Double-checking with nmap:

openqaworker17:~ # nmap -p 20033 openqaworker18.qa.suse.cz
Starting Nmap 7.92 ( https://nmap.org ) at 2024-07-24 17:23 CEST
Nmap scan report for openqaworker18.qa.suse.cz (10.100.96.76)
Host is up (0.00017s latency).

PORT      STATE    SERVICE
20033/tcp filtered unknown
MAC Address: 3C:EC:EF:93:AA:C8 (Super Micro Computer)

Nmap done: 1 IP address (1 host up) scanned in 0.18 seconds

To crosscheck, I run the same nc command on worker17 and checked from worker18 (can reach NC, port is reported "open" in nmap) as well as from engineering VPN (which also works). So the culprit has to be firewalld running on openqaworker18. I now try to find what is supposed to apply these changes (I assume our salt states) and why it did not do so on worker18.

I guess somewhere in here we do this configuration: https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/worker.sls#L221-251 - firewall-cmd --list-all-zones shows "interfaces: br1 ovs-system" but on worker18 "interfaces: br1 eth0 ovs-system". /etc/firewalld/zones/trusted.xml contains:

openqaworker18:/etc/firewalld/zones # cat trusted.xml
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <short>Trusted</short>
  <description>All network connections are accepted.</description>
  <interface name="br1"/>
  <interface name="ovs-system"/>
  <interface name="eth0"/>
  <masquerade/>
  <forward/>
</zone>

so something didn't apply our (correctly) deployed configuration. Checking salt further if I can spot where this should happen or if we maybe just completely miss it and never encountered it because a reboot would most likely fix it automatically (worker18 is up for 24 days).

while trying to come up with a proper reload command I noticed that eth0 on worker18 is in the public zone:

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: dhcpv6-client ssh
  ports: 5991-6120/tcp
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules: