Project

General

Profile

action #81198

[tracker-ticket] openqaworker-arm-{1..3} have network problems (cacheservice, OSD reachability). IPv6 disabled for now

Added by nicksinger 7 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
Start date:
2020-12-18
Due date:
% Done:

0%

Estimated time:

Description

As we face repeated network problems with our arm workers (e.g. https://progress.opensuse.org/issues/81026) we decided to disable ipv6 once again completely on all our arm workers.
This ticket is to track this change to revisit it after the Christmas holidays


Related issues

Related to openQA Infrastructure - action #81026: many jobs incomplete with auto_review:"(?s)Running on openqaworker-arm-2.*failed: 521 Connect timeout.*Result: setup failure":retryResolved2020-12-14

History

#1 Updated by nicksinger 7 months ago

I disabled it on arm-2 for now with:

sysctl -a | grep disable_ipv6 | grep -v tap | cut -d= -f 1 | awk '{$1=$1;print}' | xargs -I{} echo {}=1 > /etc/sysctl.d/99-poo81198.conf

rebooting now to verify and afterwards I will apply it to arm-1 and arm-3 too

#2 Updated by Xiaojing_liu 7 months ago

On 2020-12-29, the networker problem happened on arm-3. Here is an example: https://openqa.suse.de/tests/5227454
I disabled IPv6 on arm-3 according to nicksinger comments, and reboot the machine.

#3 Updated by okurz 7 months ago

  • Target version set to future

#4 Updated by mkittler 6 months ago

Looks like IPv6 was not actually disabled on openqaworker-arm-1 today. ip addr showed IPv6 addresses. wget http://openqa.suse.de/… was using IPv6 which did not work (only wget -4 … worked). After restarting wicked it worked again (also wget -6 …). The output of ip addr for eth0 looks like before though; only the "sec" values differ slightly.

#5 Updated by mkittler 6 months ago

After restarting wicked it worked again (also wget -6 …).

And now not anymore. This didn't last very long. I applied the same command as nicksinger did for arm-2 on arm-1 and will reboot the machine.

#6 Updated by okurz 6 months ago

  • Related to action #81026: many jobs incomplete with auto_review:"(?s)Running on openqaworker-arm-2.*failed: 521 Connect timeout.*Result: setup failure":retry added

#7 Updated by okurz 6 months ago

  • Status changed from Feedback to New

#8 Updated by nicksinger 6 months ago

nicksinger wrote:

I disabled it on arm-2 for now with:

sysctl -a | grep disable_ipv6 | grep -v tap | cut -d= -f 1 | awk '{$1=$1;print}' | xargs -I{} echo {}=1 > /etc/sysctl.d/99-poo81198.conf

rebooting now to verify and afterwards I will apply it to arm-1 and arm-3 too

I've excluded the loopback interface from that file now, see https://progress.opensuse.org/issues/88225 for details why

Also available in: Atom PDF