action #182498
open
openqaworker25 (o3) has no working IPv6 size:S
Added by nicksinger 17 days ago.
Updated 5 days ago.
Category:
Regressions/Crashes
Description
Observation¶
Richard made us aware in Slack that IPv6 is broken: https://openqa.opensuse.org/tests/5061938#step/wget_ipv6/9
I was first fooled by the above curl_ipv6-module that IPv6 must work on this machine while in reality it is never actually explicitly tested. Checking other tests like https://openqa.opensuse.org/tests/5061495 on that machine reveal indeed a broken v6 connection inside the SUT:
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://doc.opensuse.org/release-notes/x86_64/openSUSE/Tumbleweed/RELEASE-NOTES.en.rtf'
* Host doc.opensuse.org:443 was resolved.
* IPv6: 2a07:de40:b27e:1204::10
* IPv4: 195.135.223.50
* Trying [2a07:de40:b27e:1204::10]:443...
* connect to 2a07:de40:b27e:1204::10 port 443 from fec0::5054:ff:fe12:3456 port 53614 failed: Network is unreachable
* Trying 195.135.223.50:443...
* ALPN: curl offers h2,http/1.1
openqaworker25 apparently also has (currently?) no working v6:
openqaworker25:~ # ping6 heise.de
ping6: connect: Network is unreachable
Acceptance criteria¶
-
AC1: All standard SUSE owned o3 workers can reach external systems over IPv6
-
AC2: openQA tests still work fine
Suggestions¶
- Category set to Regressions/Crashes
- Priority changed from Normal to Urgent
- Target version set to Ready
I suggest as urgency mitigation to take affected machines out of production.
Is this a good opportuntiy to switch to NetworkManager as wicked is declared deprecated?
- Assignee set to robert.richardson
- Description updated (diff)
- Priority changed from Urgent to High
okurz wrote in #note-1:
I suggest as urgency mitigation to take affected machines out of production.
I've connected to the openqaworker25 and called sudo systemctl disable --now $(systemctl list-units | grep openqa-worker-auto-restart | cut -d . -f 1 | xargs), also added according rollback steps to this ticket.
A glance at next/previous reveals that this seems to affect other workers including worker20, 22-24, and 26-27 - maybe all of them?
livdywan wrote in #note-4:
A glance at next/previous reveals that this seems to affect other workers including worker20, 22-24, and 26-27 - maybe all of them?
Yes, its actually all workers
rrichardson@ariel:~> for i in $hosts; do echo $i && ssh root@$i "ping6 heise.de"; done
openqaworker21
(root@openqaworker21) Password:
ping6: connect: Network is unreachable
openqaworker22
(root@openqaworker22) Password:
ping6: connect: Network is unreachable
openqaworker23
(root@openqaworker23) Password:
ping6: connect: Network is unreachable
openqaworker24
(root@openqaworker24) Password:
ping6: connect: Network is unreachable
openqaworker25
(root@openqaworker25) Password:
ping6: connect: Network is unreachable
openqaworker26
(root@openqaworker26) Password:
ping6: connect: Network is unreachable
openqaworker-arm21
(root@openqaworker-arm21) Password:
ping6: connect: Network is unreachable
openqaworker-arm22
(root@openqaworker-arm22) Password:
ping6: connect: Network is unreachable
qa-power8-3
(root@qa-power8-3) Password:
ping6: connect: Network is unreachable
Edit:
I reverted first mitigation attempt and removed the rollback steps from the description.
- Description updated (diff)
- Status changed from New to Feedback
- Status changed from Feedback to Blocked
Just had a quick look at the last couple runs of https://openqa.opensuse.org/tests/5061938#next_previous and although they did not fail at the wget_ipv6 step anymore, if i run wget -6 manually from a worker, it will still fail.
openqaworker21:/tmp # wget -O- -6 www3.zq1.de/test.txt
Prepended http:// to 'www3.zq1.de/test.txt'
--2025-05-26 11:27:57-- http://www3.zq1.de/test.txt
Resolving www3.zq1.de (www3.zq1.de)... 2a01:4f8:221:b52:fcfd:ff:fe00:ec0c
Connecting to www3.zq1.de (www3.zq1.de)|2a01:4f8:221:b52:fcfd:ff:fe00:ec0c|:80... failed: Network is unreachable.
Also ping6 on the individual workers will result in the same result as shown in #note-5
- Status changed from Blocked to New
For sure this is not blocked anymore as referenced. So what do you plan?
- Status changed from New to Workable
- Subject changed from openqaworker25 (o3) has no working IPv6 to openqaworker25 (o3) has no working IPv6 size:S
- Description updated (diff)
- Priority changed from High to Normal
Also available in: Atom
PDF