Project

General

Profile

action #117580

web interface of qanet02nue.qa.suse.de and qanet03nue.qa.suse.de can not be reached over neither IPv4 nor IPv6 size:M

Added by okurz 2 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Start date:
2022-10-05
Due date:
2022-11-01
% Done:

0%

Estimated time:

Description

qanet02nue.qa.suse.de and qanet03nue.qa.suse.de can not be reached over neither IPv4 nor IPv6

Observation

$ ping qanet03nue.qa.suse.de
PING qanet03nue.qa.suse.de(2620:113:80c0:80a0:10:162:0:63 (2620:113:80c0:80a0:10:162:0:63)) 56 data bytes
From 2620:113:80c0:8120::5 (2620:113:80c0:8120::5) icmp_seq=1 Destination unreachable: Address unreachable
From 2620:113:80c0:8120::5 (2620:113:80c0:8120::5) icmp_seq=2 Destination unreachable: Address unreachable
^C
--- qanet03nue.qa.suse.de ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1001ms

okurz@linux-28d7:~ 1 (master) $ ping -4 qanet03nue.qa.suse.de
PING  (10.162.0.63) 56(84) bytes of data.
64 bytes from qanet03nue.qa.suse.de (10.162.0.63): icmp_seq=1 ttl=62 time=2625 ms
64 bytes from qanet03nue.qa.suse.de (10.162.0.63): icmp_seq=2 ttl=62 time=1613 ms

ssh access works, http not.

So two problems:

  1. IPv6 DNS resolution broken
  2. web interface not accessible

Acceptance criteria

Suggestions

  • Check for firmware upgrades for these switches
  • Optional: Apply the same solution to other QA switches to prevent further problems there
  • Check dhcpv6 setup on qanet to understand which v6 address these switches should have
  • If IPv6 is not feasible to use at all remove AAAA records

Workaround

If the web interface is not available a manual power cycle can help

History

#1 Updated by okurz about 2 months ago

  • Tags set to next-office-day
  • Subject changed from web interface of qanet03nue.qa.suse.de can not be reached over neither IPv4 nor IPv6 to web interface of qanet02nue.qa.suse.de and qanet03nue.qa.suse.de can not be reached over neither IPv4 nor IPv6
  • Status changed from New to In Progress
  • Assignee set to okurz

Same problem seems to appear on qanet02nue. Also tried "reload" but it seems the switch is stuck in "shutting down" for long. nicksinger stated that according to his research a process might block the shutdown.

#2 Updated by okurz about 2 months ago

After disconnecting and reconnecting the power cord both switches came back and their web configuration interface as well so this part works.

#3 Updated by openqa_review about 2 months ago

  • Due date set to 2022-10-21

Setting due date based on mean cycle time of SUSE QE Tools

#4 Updated by okurz about 2 months ago

regarding IPv6 the configuration on https://qanet03nue.qa.suse.de/ looks like IPv6 is enabled but not having the right address. From the qanet config I see that the address should be 2620:0113:80c0:80a0:10:162:0:63. We likely need to add that in config. We did with prefix 64, same as the address of qanet, but we couldn't ping it back or forth.

#5 Updated by okurz about 2 months ago

  • Due date deleted (2022-10-21)
  • Status changed from In Progress to New
  • Assignee deleted (okurz)

#6 Updated by cdywan about 2 months ago

  • Subject changed from web interface of qanet02nue.qa.suse.de and qanet03nue.qa.suse.de can not be reached over neither IPv4 nor IPv6 to web interface of qanet02nue.qa.suse.de and qanet03nue.qa.suse.de can not be reached over neither IPv4 nor IPv6 size:M
  • Description updated (diff)
  • Status changed from New to Workable

#7 Updated by okurz about 2 months ago

I just found a new firmware image on https://software.cisco.com/download/home/283019669/type/282463181/release/1.4.11.5 from 18-Jun-2020. Maybe we can try out the firmware upgrade first on switches which are currently in "QA cold storage".

#9 Updated by okurz about 2 months ago

  • Tags changed from next-office-day to reactive work
  • Parent task set to #109743

#10 Updated by nicksinger about 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to nicksinger

I will give it a try with qanet03 first

#11 Updated by nicksinger about 2 months ago

update of qanet03nue worked flawlessly. Will continue with qanet02nue after my (late) lunch break

#12 Updated by openqa_review about 2 months ago

  • Due date set to 2022-11-01

Setting due date based on mean cycle time of SUSE QE Tools

#13 Updated by nicksinger about 2 months ago

Firmware update was successful on both switches and they are now running on the most recent 1.4.11.5 version.

#14 Updated by nicksinger about 1 month ago

  • Status changed from In Progress to Resolved

I changed the "IPv6 Interfaces" settings for the "Management Interface" to be available in VLAN12 (like with v4) instead of VLAN1 (our current configuration). Afterwards added the according static entries from our dhcpd6.conf - both switches are now reachable over v4 and v6 and with the update in place the webui hopefully doesn't crash any longer.

#15 Updated by okurz about 1 month ago

Awesome, thanks :)

Also available in: Atom PDF