QA » openQA Project » openQA Infrastructure

Take openQA svirt worker instances related to one hypervisor host, e.g. s390zl12, out of production for testing

It looks like login via ssh root@s390zl12.oqa.prg2.suse.org is already not possible using the password in workerconf.sls. The host has PermitRootLogin without-password but also PasswordAuthentication no so I think things are already configured as expected (especially considering the host is already in salt).

I'm only wondering how openQA tests can access the machine then but maybe we deployed a key on the relevant worker hosts? I couldn't find such a key and also cannot connect via ssh root@s390zl12.oqa.prg2.suse.org from worker31. Very strange, because tests seem to be able to do so: https://openqa.suse.de/tests/13924867#step/bootloader_zkvm/1

From the SD ticket:

I understand what QA machines are used for, and I understand why you need password auth, it is fair.

Do we really need password auth on the svirt hosts? When I understand correctly (probably not) the s390zl11…16 hosts are svirt hosts. So they are not SUTs themselves. We could simply deploy an SSH key there (and then wouldn't even have to worry about typing issues on the password prompt). Tests needed to be adjusted, though. (As they currently expect a password prompt, e.g. the bootloader_zkvm module that runs in e.g. https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=BCI-Updates&machine=s390x-kvm&test=bci-init_15.6_on_SLES_15-SP4-fips_podman&version=15-SP6).

My problem with the login was just that my local workerconf.sls was an old version.

So this can be easily tested by spawning a test SUT like this:

openqa-clone-job --skip-chained-deps --within-instance 'https://openqa.suse.de/tests/13913585' _GROUP=0 {BUILD,TEST}+=-test-for-poo158242 PAUSE_AT=system_prepare WORKER_CLASS=s390kvm081 (or use the latest job in the scenario https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Online&machine=s390x-kvm&test=extra_tests_bootloader&version=15-SP6)

Then it is possible to login via ssh root@s390kvm081.oqa.prg2.suse.org not only from worker31.oqa.prg2.suse.org but also my local machine (in VPN).

I'm not sure how to prevent it, though. I experimented with firewalld. The following commands do the right thing but only for the svirt host s390zl12.oqa.prg2.suse.org itself:

firewall-cmd --zone=public --remove-service=ssh
firewall-cmd --zone=work --add-source=10.145.10.0/24

So with this SSH access to s390zl12.oqa.prg2.suse.org is no longer possible from my machine but remains possible from worker31.oqa.prg2.suse.org and worker32.oqa.prg2.suse.org (and other workers in that IP range but those two are the relevant ones).

Unfortunately the SUT s390kvm081.oqa.prg2.suse.org remains accessible from everywhere and I haven't found out how to prevent the access yet. Probably because bridging is used here so the firewall cannot intercept the traffic.

To temporarily add/remove workers from production I used the following commands on worker31 and worker32:

systemctl mask --now openqa-worker-auto-restart@{1..5}.service openqa-reload-worker-auto-restart@{1..5}.{service,path}
systemctl unmask --now openqa-worker-auto-restart@{1..5}.service openqa-reload-worker-auto-restart@{1..5}.{service,path}

(I actually kept slot 2 on worker31 alive for my test job.)

Any ideas on how to block the traffic of the VMs despite the bridging? Or maybe it is possible to configure the VMs network interface differently? It is currently configured like this:

# virsh dumpxml 1
…
    <interface type='direct'>
      <mac address='52:54:00:9d:09:6a'/>
      <source dev='vlan2114' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/>
    </interface>
…

I restored the previous configuration except for keeping the source in the work zone (which shouldn't make a difference):

s390zl12:/home/martchus # firewall-cmd --get-active-zones 
docker
  interfaces: docker0
public
  interfaces: eth0 vlan0 vlan2114 docker virbr0
work
  sources: 10.145.10.0/24

The interface virbr0 is also still shown in the public zone but this didn't make any difference.

Note that the hypervisor can be accessed via https://zhmc2.suse.de (select ZL12 in the table, then click on the tiny arrow icon, then "Recovery -> Integrated ASCII console").

We haven't managed to block the SSH access to the VMs. Even iptables -P INPUT DROP and iptables -P FORWARD DROP didn't make a difference. (We also installed the non-legacy version of iptables and made sure it is being used.)

We also tried to use nft commands directly (like nft add rule ip filter FORWARD ip daddr 10.145.10.0/24 reject but also on various other chains and also ensuring via handle … that the rules are at the right place as necessary).

Useful resources:

• https://firewalld.org/documentation/zone/predefined-zones.html
• https://wiki.archlinux.org/title/Nftables
• https://developers.redhat.com/blog/2020/08/18/iptables-the-two-variants-and-their-relationship-with-nftables#using_iptables_nft
• https://stuffphilwrites.com/wp-content/uploads/2014/09/FW-IDS-iptables-Flowchart-v2019-04-30-1.png
• https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-libvirt-config-virsh.html

Link to my "test job scenario" (we can simply restart this job to continue): https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Online&machine=s390x-kvm&test=extra_tests_bootloader-test-for-poo158242&version=15-SP6

We haven't configured anything persistently so I just rebooted the machine to restore the normal state and unmasked the worker slots again.

We configured a bridge device on the svirt host and adjusted the VM settings to use regular bridge mode instead of macvtap¹. This didn't work. The DHCP server (suttner) seemed to assign the correct IP but it didn't reach back to the VM host.

(This is how the configuration would look like in openQA code: https://github.com/Martchus/os-autoinst-distri-opensuse/tree/zkvm-networking)

I asked on #eng-testing for help.

there is also https://unix.stackexchange.com/questions/499756/how-does-iptable-work-with-linux-bridge/500022#500022 to explicitly enable layer 2 traffic being brought under control of the firewall. So we can look into that.

Another alternative is to change the network config so that s390x kvm tests work like other svirt tests, like on unreal6 and openqaw5-xen, that seem to use a different network configuration. /etc/libvirt/libxl/openQA-SUT-1.xml says:

    <interface type='bridge'>
      <mac address='00:16:…'/>
      <source bridge='br0'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='18…'/>
      </virtualport>
      <model type='netfront'/>
    </interface>

@mgriessmeier mentioned the following documentation: https://documentation.suse.com/sles/15-SP2/html/SLES-all/cha-libvirt-networks.html#libvirt-networks-virtual

Maybe we can use the same setup as we use for xen hosts (e.g. https://openqa.suse.de/tests/13964449/logfile?filename=autoinst-log.txt):

      <interface type='bridge'>
        <mac address='00:16:…'/>
        <source bridge='br0'/>
        <virtualport type='openvswitch'>
          <parameters interfaceid='03…'/>
        </virtualport>
        <target dev='vif100.0'/>
        <model type='netfront'/>
      </interface>

This is similar to what we tried with the bridge on Friday but maybe the additional Open vSwitch configuration helps.

Maybe we also just needed to enable forwarding. It is enabled on openqaw5-xen.qe.prg2.suse.org where the mentioned bridge configuration is used and works (cat /proc/sys/net/ipv4/ip_forward returns 1). It is not enabled on s390zl12.oqa.prg2.suse.org (the command returns 0). @dheidler Do you think our setup would have worked with forwarding enabled?

Additional resources:

• https://wiki.libvirt.org/Net.bridge.bridge-nf-call_and_sysctl.conf.html
• https://www.ibm.com/docs/en/linux-on-systems?topic=recommendations-kvm-host-networking-configuration-choices
• https://virt.kernelnewbies.org/MacVTap
• https://askubuntu.com/questions/383082/whats-the-difference-between-tun-tap-vs-bridgevnet-vs-macvtap-for-virtualiza

I realized that ssh login to s390zl12 over IPv6 does not work anymore, IPv4 works. The command ssh s390zl12.oqa.prg2.suse.org takes very long until ssh falls back to IPv4.

Fixed after setting net.ipv6.conf.interface.accept_ra to 2. I disabled forwarding again via yast because that's supposedly what made the difference. (Otherwise we didn't change any persistent settings.)

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/19108

As visible in https://openqa.suse.de/tests/13958029#step/check_network/3 sle-15-SP6-Online-x86_64-Build79.1-default@svirt-xen-pv from the scenario https://openqa.suse.de/tests/latest?test=default&machine=svirt-xen-pv other libvirt managed VMs also have an IP in IT managed networks but dynamic DHCP leases and possibly without an SSH server enabled. So the network setup looks comparable to s390zl12+13 but the test and backend setup is different. Hence we suggest to reject this ticket and follow-up with alternatives as planned in the parent epic.