QA » openQA Project » openQA Infrastructure

Hi,
Would it also be possible to use the two machines of https://progress.opensuse.org/issues/159063 for the purpose of testing those firewalls rules instead of production hardware?
I would like to avoid a shortage of s390 production workers when GMC hits end of next week.

mgriessmeier wrote in #note-7:

Hi,
Would it also be possible to use the two machines of https://progress.opensuse.org/issues/159063 for the purpose of testing those firewalls rules instead of production hardware?
I would like to avoid a shortage of s390 production workers when GMC hits end of next week.

We discovered that only 10/20 zl12 workers were enabled. That was handled in https://progress.opensuse.org/issues/158170#note-20
As discussed in Slack we're monitoring the queue-size and see if 20 slots are good enough for now.

nicksinger wrote in #note-9:

This worked already quite well: https://openqa.suse.de/tests/overview?distri=sle&version=15-SP6&build=83.1-test-for-poo159066 - the 4 failing tests seem to fail earlier in connecting to zl12 already so I'm not sure if this is really related to my changes.

I found the issue and adjusted our sshd config for all workers: https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1179

Created https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1184 with a first draft containing the script. It makes use of our "roles" grain which needs to contain two roles at the same time:

s390zl12:~ # cat /etc/salt/grains
passwordlogin: True
roles:
 - libvirt
 - worker

openqa:~ # salt -C 'G@roles:libvirt and G@roles:worker' test.ping
s390zl12.oqa.prg2.suse.org:
    True

caused failure in https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2602994#L45

my proposal for a fix
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/807

discussed with nicksinger, I removed the "worker" role again for now and proposing reverting my https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/808. Instead we agreed to define a custom grain "external_openqa_hypervisor" with value "True" for both hosts which prevents the firewall. Or

external_openqa_hypervisor_passlist:
  - 10.145.10.0/24
  - 2a07:de40:b203:12:…/64

and iterate over each entry

I tried to set the default policy to "drop" and only explicitly allow what is needed (tried with "everything" for now to get a test working at all). I ended up with the following rules:

    nft "add table netdev filtermacvtap" #should not do anything if table is already present
    nft "add chain netdev filtermacvtap filterin_${DOMID} { type filter hook ingress device $IFACE priority filter; policy accept; }" #drop everything by default (reject is not available here)
    nft "add rule netdev filtermacvtap filterin_${DOMID} ip saddr 10.145.10.0/24 accept"
    nft "add rule netdev filtermacvtap filterin_${DOMID} ip6 saddr 2a07:de40:b203:12::0/64 accept"
    nft "add rule netdev filtermacvtap filterin_${DOMID} ip saddr 10.144.98.239/32 accept"
    nft add rule netdev filtermacvtap filterin_${DOMID} meta l4proto udp accept
    nft add rule netdev filtermacvtap filterin_${DOMID} meta l4proto icmp accept
    nft add rule netdev filtermacvtap filterin_${DOMID} meta l4proto ipv6-icmp accept
    nft add rule netdev filtermacvtap filterin_${DOMID} meta l4proto ipv6 accept
    nft add rule netdev filtermacvtap filterin_${DOMID} ether type arp accept
    nft add rule netdev filtermacvtap filterin_${DOMID} log

Unfortunately this fails because every outgoing connection of the VM causes a new incoming connection which in turn gets blocked if not explicitly whitelisted (because the used netdev table does not support stateful filtering).

As already discussed in the daily I will go ahead and just drop ports 22+59[00-99] explicitly. Not nice but should cover our use-case.

The major changes:
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1184
https://gitlab.suse.de/openqa/salt-pillars-openqa/-/merge_requests/810

Some cleanups:
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1192
https://gitlab.suse.de/openqa/salt-states-openqa/-/merge_requests/1193

all have been merged. I added the external_openqa_hypervisor-role to s390zl12 and the deploy-pipeline created /etc/libvirt/hooks/qemu.d/setup-sut-firewall.sh with the following content:

#!/bin/sh
# libvirt hook script (https://libvirt.org/hooks.html)
# receives the DOM XML via stdin and metadata from libvirt as arguments

if [ "$2" = start ] && [ "$3" = begin ]; then
    XML=$(cat /dev/stdin)
    IFACE=$(echo $XML | xmlstarlet select -t -m 'domain/devices/interface[@type="direct"]' -v 'target/@dev')
    DOMID=$(echo $XML | xmlstarlet select -t -v 'domain/@id')
    nft "add table netdev filtermacvtap" #should not do anything if table is already present
    nft "add chain netdev filtermacvtap filterin_${DOMID} { type filter hook ingress device $IFACE priority filter; policy accept; }"
    nft "add rule netdev filtermacvtap filterin_${DOMID} ip saddr != { 10.145.10.0/24 } tcp dport { 22, 5800-5899, 5900-5999 } reject comment \"reject global SUT access to specific ports\""
    nft "add rule netdev filtermacvtap filterin_${DOMID} ip6 saddr != { 2a07:de40:b203:12::0/64 } tcp dport { 22, 5800-5899, 5900-5999 } reject"
    #nft add rule netdev filtermacvtap filterin_${DOMID} log # helpful for debugging, messages can be found in `journalctl -ft kernel`
fi