QA » openQA Project » openQA Infrastructure

ip r does not show a default IPv6 route, ip -6 r does though. After I did systemctl stop firewalld the ping openqa -c 1 -6 openqa.suse.de worked.

could not find anything in logs either, even if enabling in /etc/firewalld to log all "dropped" packages. Then later ping did not work even if firewalld disabled. I did systemctl restart network and my ssh connection never recovered. over IPMI SOL I can ping the own IPV6 address of openqaworker-arm-2 but not osd:

# ping -c 1 -6 2620:113:80c0:8080:10:160:0:227
PING 2620:113:80c0:8080:10:160:0:227(2620:113:80c0:8080:10:160:0:227) 56 data bytes
64 bytes from 2620:113:80c0:8080:10:160:0:227: icmp_seq=1 ttl=64 time=0.061 ms

--- 2620:113:80c0:8080:10:160:0:227 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.061/0.061/0.061/0.000 ms
e246:~ # ping -c 1 -6 openqa.suse.de
PING openqa.suse.de(openqa.suse.de (2620:113:80c0:8080:10:160:0:207)) 56 data bytes

which resolves the right address for openqa.suse.de at least. Then after some minutes the system would not even react properly over SOL. Triggered power reset.

EDIT: Same symptoms after reboot, then sudo systemctl disable --now openqa-worker.target openqa-worker-cacheservice openqa-worker-cacheservice-minion.service

salt -l error \* cmd.run 'dig openqa.suse.de AAAA' is fine on all machines.

salt -l error \* cmd.run 'ping -c 1 -4 openqa.suse.de ; ping -c 1 -6 openqa.suse.de' is not ok on QA-Power8-4-kvm.qa.suse.de, openqaworker-arm-1.suse.de, openqaworker-arm-2.suse.de

I have applied

salt -l error -L 'QA-Power8-4-kvm.qa.suse.de,openqaworker-arm-1.suse.de,openqaworker-arm-2.suse.de' cmd.run 'echo net.ipv6.conf.all.disable_ipv6 = 1 > /etc/sysctl.d/poo73633_poo80128_debugging.conf && sysctl --load /etc/sysctl.d/poo73633_poo80128_debugging.conf && systemctl restart openqa-worker@\* openqa-worker-cacheservice openqa-worker-cacheservice-minion.service os-autoinst-openvswitch.service && systemctl mask --now postfix'

this seems to have worked. So far the machines seem to be ok with only IPv4. I don't know why IPv6 does not work but I guess we can live with that for the time being.

While working on grenache-1 I realized that there where some leftovers in sysctl on QA-Power8-4-kvm.qa.suse.de resulting in ipv6 being disabled only on some interfaces (IIRC "lo" was one of them). This resulted in these strange errors you discovered in #3 that resolving and all works but pings get stuck.
I removed your workaround file now and made sure all disable_ipv6 entries are set to 0 (on QA-Power8-4-kvm.qa.suse.de, openqaworker-arm-1.suse.de and openqaworker-arm-2.suse.de). To validate I ran the following command on osd:

openqa:~ # salt -l error --no-color -C 'G@roles:worker' cmd.run 'curl -s -6 openqa.suse.de | grep changelog'
openqaworker2.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
openqaworker5.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
openqaworker6.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
openqaworker8.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
QA-Power8-4-kvm.qa.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
openqaworker9.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
QA-Power8-5-kvm.qa.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
openqaworker13.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
openqaworker10.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
openqaworker-arm-2.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
openqaworker-arm-1.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>
grenache-1.qa.suse.de:
                          <a href="/changelog">4.6.1607440298.36d0dfbf9</a>

Please reopen if the problem still persists