openSUSE admin

crameleon wrote in #note-1:

Hi Per,

there were some issues yesterday, but they should be resolved and today people connected with no issues.

• which endpoint do you use? TCP or UDP?
• do you get any other warning messages in your log output? a common scenario is a deprecation warning about --data-ciphers being ignored, it needs to be resolved by adding data-ciphers AES-256-CBC.

Cheers

Hi Georg

this is my vpn conf:

client
dev tun
proto udp
#remote scar.opensuse.org 1194
remote 195.135.221.151 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca opensuse/ca.crt
cert opensuse/pjessen.crt
key opensuse/pjessen.key
#remote-cert-tls server
tls-auth opensuse/ta.key 1
cipher AES-256-CBC
comp-lzo
# verb 4  # enable if debugging is needed
auth SHA512
auth-user-pass opensuse/creds.txt

This is the output:

2023-11-10T12:40:12+01:00 office68 systemd[1]: Starting OpenVPN tunneling daemon instance using /etc/openvpn/opensuse.conf...
2023-11-10T12:40:12+01:00 office68 openvpn[4869]: WARNING: Using --management on a TCP port WITHOUT passwords is STRONGLY discouraged and considered insecure
2023-11-10T12:40:12+01:00 office68 openvpn[4869]: OpenVPN 2.4.3 x86_64-suse-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jun 20 2017
2023-11-10T12:40:12+01:00 office68 openvpn[4869]: library versions: OpenSSL 1.1.1d  10 Sep 2019, LZO 2.10
2023-11-10T12:40:12+01:00 office68 openvpn[4870]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
2023-11-10T12:40:12+01:00 office68 systemd[1]: Started OpenVPN tunneling daemon instance using /etc/openvpn/opensuse.conf.
2023-11-10T12:40:12+01:00 office68 openvpn[4870]: TCP/UDP: Preserving recently used remote address: [AF_INET]195.135.221.151:1194
2023-11-10T12:40:12+01:00 office68 openvpn[4870]: UDP link local: (not bound)
2023-11-10T12:40:12+01:00 office68 openvpn[4870]: UDP link remote: [AF_INET]195.135.221.151:1194
2023-11-10T12:40:12+01:00 office68 openvpn[4870]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2023-11-10T12:40:12+01:00 office68 openvpn[4870]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2023-11-10T12:40:12+01:00 office68 openvpn[4870]: [scar.opensuse.org] Peer Connection Initiated with [AF_INET]195.135.221.151:1194
2023-11-10T12:40:13+01:00 office68 openvpn[4870]: AUTH: Received control message: AUTH_FAILED
2023-11-10T12:40:13+01:00 office68 openvpn[4870]: SIGTERM[soft,auth-failure] received, process exiting
2023-11-10T12:40:13+01:00 office68 systemd[1]: openvpn@opensuse.service: Succeeded.

Thanks for the information. Please use our canonical DNS name gate.opensuse.org.

crameleon wrote in #note-3:

Thanks for the information. Please use our canonical DNS name gate.opensuse.org.

Very minor improvement:

2023-11-10T13:17:49+01:00 office68 systemd[1]: Starting OpenVPN tunneling daemon instance using /etc/openvpn/opensuse.conf...
2023-11-10T13:17:49+01:00 office68 openvpn[6319]: WARNING: Using --management on a TCP port WITHOUT passwords is STRONGLY discouraged and considered insecure
2023-11-10T13:17:49+01:00 office68 openvpn[6319]: OpenVPN 2.4.3 x86_64-suse-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Jun 20 2017
2023-11-10T13:17:49+01:00 office68 openvpn[6319]: library versions: OpenSSL 1.1.1d  10 Sep 2019, LZO 2.10
2023-11-10T13:17:49+01:00 office68 openvpn[6320]: WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
2023-11-10T13:17:49+01:00 office68 systemd[1]: Started OpenVPN tunneling daemon instance using /etc/openvpn/opensuse.conf.
2023-11-10T13:17:49+01:00 office68 openvpn[6320]: TCP/UDP: Preserving recently used remote address: [AF_INET6]2a07:de40:b27e:1102::a:1194
2023-11-10T13:17:49+01:00 office68 openvpn[6320]: UDP link local: (not bound)
2023-11-10T13:17:49+01:00 office68 openvpn[6320]: UDP link remote: [AF_INET6]2a07:de40:b27e:1102::a:1194
2023-11-10T13:17:49+01:00 office68 openvpn[6320]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2023-11-10T13:17:49+01:00 office68 openvpn[6320]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2023-11-10T13:17:49+01:00 office68 openvpn[6320]: [odin.opensuse.org] Peer Connection Initiated with [AF_INET6]2a07:de40:b27e:1102::a:1194
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: Note: option tun-ipv6 is ignored because modern operating systems do not need special IPv6 tun handling anymore.
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: TUN/TAP device tun0 opened
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: add_route_ipv6(2a07:de40:b27e:1100::/64 -> 2a07:de40:b27e:5001::1 metric -1) dev tun0
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: ERROR: Linux route -6/-A inet6 add command failed: external program exited with error status: 2
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: add_route_ipv6(2a07:de40:b27e:1203::/64 -> 2a07:de40:b27e:5001::1 metric -1) dev tun0
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: ERROR: Linux route -6/-A inet6 add command failed: external program exited with error status: 2
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: add_route_ipv6(2a07:de40:b27e:1205::/64 -> 2a07:de40:b27e:5001::1 metric -1) dev tun0
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: ERROR: Linux route -6/-A inet6 add command failed: external program exited with error status: 2
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: add_route_ipv6(2a07:de40:b27e:64::/96 -> 2a07:de40:b27e:5001::1 metric -1) dev tun0
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: ERROR: Linux route -6/-A inet6 add command failed: external program exited with error status: 2
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: GID set to nobody
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: UID set to nobody
2023-11-10T13:17:50+01:00 office68 openvpn[6320]: Initialization Sequence Completed

It looks like I'm not getting assigned an IP address:

7: tun0: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 100
    link/none 

Here is a known working configuration for native OpenVPN:

client
dev tun-heroes
dev-type tun
proto udp6
remote odin.opensuse.org 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/ssl/vpn/ca-heroes.crt
cert /etc/ssl/vpn/crameleon3.crt
key /etc/ssl/vpn/crameleon3.key
remote-cert-tls server
tls-auth /etc/ssl/vpn/ta-heroes.key 1
cipher AES-256-CBC
data-ciphers AES-256-CBC
comp-lzo
auth SHA512

2023-11-10T13:17:50+01:00 office68 openvpn[6320]: Note: option tun-ipv6 is ignored because modern operating systems do not need special IPv6 tun handling anymore.

I don't find this in your pasted configuration file, are you including other files or passing custom command line options?

With the snippet I pasted, the output should be free of errors, except possibly compression and cached passphrases:

37]: WARNING: Compression for receiving enabled. Compression has been used in the past to break encryption. Sent pack>
37]: OpenVPN 2.6.4 x86_64-suse-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD]
37]: library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10
eling daemon instance using /etc/openvpn/heroes-prg-crameleon3.conf.
37]: TCP/UDP: Preserving recently used remote address: [AF_INET6]2a07:de40:b27e:1102::a:1194
37]: UDPv6 link local: (not bound)
37]: UDPv6 link remote: [AF_INET6]2a07:de40:b27e:1102::a:1194
37]: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
37]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
37]: [odin.opensuse.org] Peer Connection Initiated with [AF_INET6]2a07:de40:b27e:1102::a:1194
37]: TUN/TAP device tun-heroes opened
37]: /usr/sbin/ip link set dev tun-heroes up mtu 1500
37]: /usr/sbin/ip link set dev tun-heroes up
37]: /usr/sbin/ip -6 addr add 2a07:de40:b27e:5001::152/64 dev tun-heroes
37]: add_route_ipv6(2a07:de40:b27e:1100::/64 -> 2a07:de40:b27e:5001::1 metric -1) dev tun-heroes
37]: add_route_ipv6(2a07:de40:b27e:1203::/64 -> 2a07:de40:b27e:5001::1 metric -1) dev tun-heroes
37]: add_route_ipv6(2a07:de40:b27e:1205::/64 -> 2a07:de40:b27e:5001::1 metric -1) dev tun-heroes
37]: add_route_ipv6(2a07:de40:b27e:64::/96 -> 2a07:de40:b27e:5001::1 metric -1) dev tun-heroes
37]: GID set to nobody
37]: UID set to nobody
37]: Initialization Sequence Completed

crameleon wrote in #note-7:

2023-11-10T13:17:50+01:00 office68 openvpn[6320]: Note: option tun-ipv6 is ignored because modern operating systems do not need special IPv6 tun handling anymore.

I don't find this in your pasted configuration file, are you including other files or passing custom command line options?

The command line:
ExecStart=/usr/sbin/openvpn --daemon --suppress-timestamps --writepid /run/openvpn/%i.pid --cd /etc/openvpn/ --config %i.conf

crameleon wrote in #note-6:

Here is a known working configuration for native OpenVPN:

client
dev tun-heroes
dev-type tun
proto udp6
remote odin.opensuse.org 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca /etc/ssl/vpn/ca-heroes.crt
cert /etc/ssl/vpn/crameleon3.crt
key /etc/ssl/vpn/crameleon3.key
remote-cert-tls server
tls-auth /etc/ssl/vpn/ta-heroes.key 1
cipher AES-256-CBC
data-ciphers AES-256-CBC
comp-lzo
auth SHA512

Looks an awful lot like mine, apart from data-ciphers, dev-type tun and proto udp6. Judging by the output, the connection is working, but for some reason I am not getting an IP-address.

Can you try with the same data-ciphers and proto options, please?

crameleon wrote in #note-11:

Can you try with the same data-ciphers and proto options, please?

Yep - Options error: Unrecognized option or missing or extra parameter(s) in opensuse.conf:20: data-ciphers (2.4.3) :-)
I kept the other two: proto udp6 and dev-type tun - no change, no complaints.
It seems to me it is mostly working, otherwise why would those route options be pushed out to me?

I just had someone else connect successfully.

Can you maybe get some debug logging as to what "ERROR: Linux route -6/-A inet6 add command failed: external program exited with error status: 2" really means / if there are any issues configuring your interface?

crameleon wrote in #note-13:

I just had someone else connect successfully.

Can you maybe get some debug logging as to what "ERROR: Linux route -6/-A inet6 add command failed: external program exited with error status: 2" really means / if there are any issues configuring your interface?

I think the issue is that the interface "tun0" is down and not configured at all. Why I am not seeing the commands you see, after "TUN/TAP device tun-heroes opened" :

/usr/sbin/ip link set dev tun-heroes up mtu 1500
/usr/sbin/ip link set dev tun-heroes up
/usr/sbin/ip -6 addr add 2a07:de40:b27e:5001::152/64 dev tun-heroes

When the interface has no address, adding a route is bound to fail:
"add_route_ipv6(2a07:de40:b27e:1100::/64 -> 2a07:de40:b27e:5001::1 metric -1) dev tun0"
I'll see what sort of debug info I can get.

pjessen wrote in #note-14:

I think the issue is that the interface "tun0" is down and not configured at all.

Running openvpn in the foreground confirms:

add_route_ipv6(2a07:de40:b27e:1100::/64 -> 2a07:de40:b27e:5001::1 metric -1) dev tun0
Error: Nexthop device is not up.
ERROR: Linux route -6/-A inet6 add command failed: external program exited with error status: 2

Surely there is something in the log on the server that explains why my connection is not getting an IP address allocated ?

Nov 10 16:55:47 odin openvpn[20822]: PLUGIN AUTH-PAM: BACKGROUND: received command code: 0
Nov 10 16:55:47 odin openvpn[20822]: PLUGIN AUTH-PAM: BACKGROUND: USER: pjessen
Nov 10 16:55:47 odin openvpn[20822]: PLUGIN AUTH-PAM: BACKGROUND: my_conv[0] query='Password: ' style=1
Nov 10 16:55:47 odin openvpn[20822]: pam_sss(common-vpn:auth): authentication success; logname= uid=0 euid=0 tty= ruser= rhost= user=pjessen
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: 2a03:7520:4c68:1:de80:56c8:6846:345d PLUGIN_CALL: POST /usr/lib64/openvpn/plugins/openvpn-plugin-auth-pam.so/PLUGIN_AUTH_USER_PASS_VERIFY status=0
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: 2a03:7520:4c68:1:de80:56c8:6846:345d TLS: Username/Password authentication succeeded for username 'pjessen'
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: 2a03:7520:4c68:1:de80:56c8:6846:345d Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: 2a03:7520:4c68:1:de80:56c8:6846:345d [pjessen] Peer Connection Initiated with [AF_INET6]2a03:7520:4c68:1:de80:56c8:6846:345d:38947
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI_sva: pool returned IPv4=(Not enabled), IPv6=2a07:de40:b27e:5001::1004
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d OPTIONS IMPORT: reading client specific options from: /etc/openvpn/ccd-udp/pjessen
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d RESOLVE: Cannot resolve host address: 2a07:de40:b27e:5001::207: (Address family for hostname not supported)
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d RESOLVE: Cannot resolve host address: 2a07:de40:b27e:5001::1: (Address family for hostname not supported)
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d Options error: cannot parse --ifconfig-push addresses
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI: no dynamic or static remote--ifconfig address is available for pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI: Learn: 2a07:de40:b27e:5001::1004 -> pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI: primary virtual IPv6 for pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d: 2a07:de40:b27e:5001::1004
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d Outgoing Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d Outgoing Data Channel: Using 512 bit message hash 'SHA512' for HMAC authentication
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d Incoming Data Channel: Cipher 'AES-256-CBC' initialized with 256 bit key
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d Incoming Data Channel: Using 512 bit message hash 'SHA512' for HMAC authentication
Nov 10 16:55:48 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d PUSH: Received control message: 'PUSH_REQUEST'
Nov 10 16:55:48 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d SENT CONTROL [pjessen]: 'PUSH_REPLY,dhcp-option DNS 2a07:de40:b27e:64::c0a8:2f65,dhcp-option DNS 2a07:de40:b27e:64::c0a8:2f66,dhcp-option DOMAIN infra.opensuse.org,route-ipv6 2a07:de40:b27e:1100::/64,route-ipv6 2a07:de40:b27e:1203::/64,route-ipv6 2a07:de40:b27e:1205::/64,route-ipv6 2a07:de40:b27e:64::/96,tun-ipv6,ping 10,ping-restart 30,ifconfig-ipv6 2a07:de40:b27e:5001::1004/64 2a07:de40:b27e:5001::1,peer-id 1,cipher AES-256-CBC' (status=1)
Nov 10 17:01:21 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d [pjessen] Inactivity timeout (--ping-restart), restarting
Nov 10 17:01:21 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d SIGUSR1[soft,ping-restart] received, client-instance restarting

Your CCD file has the same syntax as mine (I batch-converted them from IPv4 to IPv6 as part of the migration):

odin (VPN Gateway):~ # more /etc/openvpn/ccd-udp/pjessen
ifconfig-push 2a07:de40:b27e:5001::207 2a07:de40:b27e:5001::1
odin (VPN Gateway):~ # more /etc/openvpn/ccd-tcp/pjessen
ifconfig-push 2a07:de40:b27e:5002::207 2a07:de40:b27e:5002::1
odin (VPN Gateway):~ # more /etc/openvpn/ccd-tcp/crameleon
ifconfig-push 2a07:de40:b27e:5002::162 2a07:de40:b27e:5002::1

The "Address family for hostname not supported" lines make it seem like somewhere it is trying to wrongly use IPv4.

2a03:7520:4c68:1:de80:56c8:6846:345d is my laptop here.
Some noteworthy error messages, I think:

Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d RESOLVE: Cannot resolve host address: 2a07:de40:b27e:5001::207: (Address family for hostname not supported)
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d RESOLVE: Cannot resolve host address: 2a07:de40:b27e:5001::1: (Address family for hostname not supported)
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d Options error: cannot parse --ifconfig-push addresses
Nov 10 16:55:47 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI: no dynamic or static remote--ifconfig address is available for pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d

Nov 10 16:55:48 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d SENT CONTROL [pjessen]: 'PUSH_REPLY,dhcp-option DNS 2a07:de40:b27e:64::c0a8:2f65,dhcp-option DNS 2a07:de40:b27e:64::c0a8:2f66,dhcp-option DOMAIN infra.opensuse.org,route-ipv6 2a07:de40:b27e:1100::/64,route-ipv6 2a07:de40:b27e:1203::/64,route-ipv6 2a07:de40:b27e:1205::/64,route-ipv6 2a07:de40:b27e:64::/96,tun-ipv6,ping 10,ping-restart 30,ifconfig-ipv6 2a07:de40:b27e:5001::1004/64 2a07:de40:b27e:5001::1,peer-id 1,cipher AES-256-CBC' (status=1)

At a glance, I think that looks good. Is there any reason why my side should not understand it?

Of course I have ipv6 locally, but surely that won't make a difference ?

It's not clear to me what'd be different from others. I assume you use a standard SUSE installation of OpenVPN. I'll have to investigate some more into how to debug such a scenario.

Of course I have ipv6 locally, but surely that won't make a difference ?

It will work either way, but having IPv6 locally as well is of course a big plus. :-)

I hate to ask this, but you don't happen to have any /etc/hosts entries or similar custom resolver setups which might try to resolve addresses in an unintended manner?

crameleon wrote in #note-20:

I hate to ask this, but you don't happen to have any /etc/hosts entries or similar custom resolver setups which might try to resolve addresses in an unintended manner?

Perfectly valid question Georg -

127.0.0.1       localhost

# special IPv6 addresses
::1             localhost ipv6-localhost ipv6-loopback

fe00::0         ipv6-localnet

ff00::0         ipv6-mcastprefix
ff02::1         ipv6-allnodes
ff02::2         ipv6-allrouters
ff02::3         ipv6-allhosts

My dnsmasq setup:

server=/local.rz2/192.168.2.254
server=/infra.opensuse.org/192.168.47.101
server=/infra.opensuse.org/192.168.47.102
server=/47.168.192.in-addr.arpa/192.168.47.101
server=/47.168.192.in-addr.arpa/192.168.47.102
server=/67.168.192.in-addr.arpa/192.168.47.101
server=/67.168.192.in-addr.arpa/192.168.47.102
server=/tebibyte.org/172.16.0.25

Thanks for the output. Not related to your addressing issue, but once you are connected you will need to use 2a07:de40:b27e:64::c0a8:2f66 and 2a07:de40:b27e:64::c0a8:2f65 as nameservers for infra.opensuse.org (these are temporarily the IPv6 mapped addresses of anna/elsa until I have the new internal forwarders in Prague ready).

I now enabled trace output. Please try to connect again (to the UDP instance) - quickly please, as I want to turn this wall off text again ASAP. :-)

crameleon wrote in #note-23:

I now enabled trace output. Please try to connect again (to the UDP instance) - quickly please, as I want to turn this wall off text again ASAP. :-)

Done!!

Thanks. Only marginally more descriptive:

Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI_sva: pool returned IPv4=(Not enabled), IPv6=2a07:de40:b27e:5001::1004
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d TEST FILE '/etc/openvpn/ccd-udp/pjessen' [1]
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d OPTIONS IMPORT: reading client specific options from: /etc/openvpn/ccd-udp/pjessen
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d GETADDRINFO flags=0x0001 ai_family=2 ai_socktype=1
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d RESOLVE: Cannot resolve host address: 2a07:de40:b27e:5001::207: (Address family for hostname not supported)
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d GETADDRINFO flags=0x0001 ai_family=2 ai_socktype=1
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d RESOLVE: Cannot resolve host address: 2a07:de40:b27e:5001::1: (Address family for hostname not supported)
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d Options error: cannot parse --ifconfig-push addresses
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI: no dynamic or static remote--ifconfig address is available for pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI: Learn: 2a07:de40:b27e:5001::1004 -> pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d
Nov 10 19:52:58 odin openvpn@heroes_udp[20821]: pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d MULTI: primary virtual IPv6 for pjessen/2a03:7520:4c68:1:de80:56c8:6846:345d: 2a07:de40:b27e:5001::1004

What's interesting as well, though it's not specific to your client, is the output about a pool address being used. Indeed, one of my clients, shows a pool address as well. Only my second, test, client uses the static one defined in its CCD file:

>INFO:OpenVPN Management Interface Version 3 -- type 'help' for more info
status
OpenVPN CLIENT LIST
Updated,2023-11-10 20:11:44
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
crameleon3,2a02:1748:f7df:9c80:c412:abff:fef2:22ae,34952,31428,2023-11-10 20:07:20
crameleon,2a02:1748:f7df:9c80:8aa4:c2ff:fed6:4038,15036077,22732786,2023-11-09 21:05:04
pjessen,2a03:7520:4c68:1:de80:56c8:6846:345d,5447,5424,2023-11-10 20:09:25
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
2a07:de40:b27e:5001::1000,crameleon,2a02:1748:f7df:9c80:8aa4:c2ff:fed6:4038,2023-11-10 20:11:27
2a07:de40:b27e:5001::152,crameleon3,2a02:1748:f7df:9c80:c412:abff:fef2:22ae,2023-11-10 20:11:44
2a07:de40:b27e:5001::1004,pjessen,2a03:7520:4c68:1:de80:56c8:6846:345d,2023-11-10 20:09:25

But given it works either way for both my clients (and for others) I guess it's probably a false lead.

I do think it is an issue on the client side, I did not find much about the "RESOLVE" step, but it seems to be the client trying to resolve things: https://openvpn-users.narkive.com/A7lVjhVI/resolve-cannot-resolve-host-address-host-not-found#post2. Unfortunately I have not yet quite gathered how I could reproduce it.

Would you mind trying to capture trace output of the connection attempt as well by using verb 9 in your configuration file (or on the command line)? It's lots of useless noise but maybe something interesting is in the client trace that didn't make it to the server one.

crameleon wrote in #note-25:

I do think it is an issue on the client side,

Given that the client config has not changed since sometime in mid-2022, that seems entirely unlikely. I mean, the client config was working only a week ago.
Everything suggests it is a server side issue.

Thank you! Comparing this with my output shows some interesting differences, for example there not being any attempt of an ip addr add call. This would happen shortly after where in your output it complains about "option tun-ipv6 is ignored". The tun-ipv6 seems to be what's sent in both of our PUSH lines. I am on Tumbleweed, so I do wonder why my OS would not be considered "modern" enough for it to ignore this option.
Besides that, I sanitized and diff'd our two connection profile outputs, they have some differences which I would contribute to different version defaults.

See my lines, which do not show the warning, and instead proceed to bring the tunnel interface up, and adding an address to it:

openvpn@heroes-prg-crameleon3[7409]: PUSH: Received control message: 'PUSH_REPLY,dhcp-option DNS 2a07:de40:b27e:64::c0a8:2f65,dhcp-option DNS 2a07:de40:b27e:64::c0a8:2f66,dhcp-option DOMAIN infra.opensuse.org,route-ipv6 2a07:de40:b27e:1100::/64,route-ipv6 2a07:de40:b27e:1203::/64,route-ipv6 2a07:de40:b27e:1205::/64,route-ipv6 2a07:de40:b27e:64::/96,tun-ipv6,ping 10,ping-restart 30,ifconfig-ipv6 2a07:de40:b27e:5001::152/64 2a07:de40:b27e:5001::1,peer-id 1,cipher AES-256-CBC'
openvpn@heroes-prg-crameleon3[7409]: OPTIONS IMPORT: timers and/or timeouts modified
openvpn@heroes-prg-crameleon3[7409]: OPTIONS IMPORT: --ifconfig/up options modified
openvpn@heroes-prg-crameleon3[7409]: OPTIONS IMPORT: route options modified
openvpn@heroes-prg-crameleon3[7409]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
openvpn@heroes-prg-crameleon3[7409]: OPTIONS IMPORT: peer-id set
openvpn@heroes-prg-crameleon3[7409]: OPTIONS IMPORT: data channel crypto options modified
openvpn@heroes-prg-crameleon3[7409]: GDG6: remote_host_ipv6=2a07:de40:b27e:1102::a
openvpn@heroes-prg-crameleon3[7409]: net_route_v6_best_gw query: dst 2a07:de40:b27e:1102::a
openvpn@heroes-prg-crameleon3[7409]: sitnl_send: checking for received messages
openvpn@heroes-prg-crameleon3[7409]: sitnl_send: rtnl: received 176 bytes
openvpn@heroes-prg-crameleon3[7409]: net_route_v6_best_gw result: via fe80::5054:ff:fe83:736d dev br0
openvpn@heroes-prg-crameleon3[7409]: ROUTE6_GATEWAY fe80::5054:ff:fe83:736d IFACE=br0
openvpn@heroes-prg-crameleon3[7409]: TUN/TAP device tun-heroes opened
openvpn@heroes-prg-crameleon3[7409]: do_ifconfig, ipv4=0, ipv6=1
openvpn@heroes-prg-crameleon3[7409]: /usr/sbin/ip link set dev tun-heroes up mtu 1500
openvpn@heroes-prg-crameleon3[7409]: /usr/sbin/ip link set dev tun-heroes up
openvpn@heroes-prg-crameleon3[7409]: /usr/sbin/ip -6 addr add 2a07:de40:b27e:5001::152/64 dev tun-heroes

Please provide some follow up details:

grep -r 'pull-filter' /etc
grep -r tun0 /etc
find /etc -name '*tun0*'
grep -r mssfix /etc
rpm -q openvpn
grep PRETTY /etc/os-release

while connected:
sysctl -a|grep tun0
ip -d a sh tun0

Then, please try to use a different tunnel interface using an interface name which you did not use before. For example dev tun-foobar, and show the last two commands with that interface name as an argument instead of tun0 as well.

Should all this not yield any more interesting news, I should at least be able to try finding setup with matching versions.

working only a week ago.

This is nice, but not necessarily meaningful given the server having changed only yesterday.

Sorry for the amount of NEEDINFO's.

I also want to point out that your OpenVPN (2.4) is quite behind mine (2.6). Even Leap 15.4 already ships with 2.5! So albeit I understand you wanting to put it on our server, I hope you understand that I will only offer limited support for a version which is not shipped with recent openSUSE Leap or Tumbleweed. Of course, I will do my best regardless.

The PUSH_REPLY looks good to me:

PUSH_REPLY,
dhcp-option DNS 2a07:de40:b27e:64::c0a8:2f65,
dhcp-option DNS 2a07:de40:b27e:64::c0a8:2f66,
dhcp-option DOMAIN infra.opensuse.org,
route-ipv6 2a07:de40:b27e:1100::/64,
route-ipv6 2a07:de40:b27e:1203::/64,
route-ipv6 2a07:de40:b27e:1205::/64,
route-ipv6 2a07:de40:b27e:64::/96,
tun-ipv6,
ping 10,
ping-restart 30,
ifconfig-ipv6 2a07:de40:b27e:5001::1004/64 2a07:de40:b27e:5001::1,
peer-id 2,
cipher AES-256-CBC

It has all of the needed information - the question is only why ifconfig-ipv6 2a07:de40:b27e:5001::1004/64 2a07:de40:b27e:5001::1 is not applied to the interface.
Using the information from the PUSH_REPLY, I have configured the interface manually:

ip addr add 2a07:de40:b27e:5001::1004/64 dev tun0
ip link set up dev tun0
ip route add 2a07:de40:b27e:1100::/64 via 2a07:de40:b27e:5001::1
ip route add 2a07:de40:b27e:1203::/64 via 2a07:de40:b27e:5001::1
ip route add 2a07:de40:b27e:1205::/64 via 2a07:de40:b27e:5001::1
ip route add 2a07:de40:b27e:64::/96 via 2a07:de40:b27e:5001::1

I have also amended my dnsmasq config with the new nameservers.

I can now connect to mailman3, but not to anna. I guess anna has yet to be migrated.

• Why does ifconfig-ip6 not work? I wonder if it ought to be ifconfig-ipv6-push. That is what I use on my own company's vpn. (albeit for ipv4)

I used ifconfig-ipv6-push as well, but both options yielded the same results for my clients:

# more /etc/openvpn/ccd-udp/crameleon3
# generated by make-config.sh, do not edit
#ifconfig-ipv6-push 2a07:de40:b27e:5001::152/64 2a07:de40:b27e:5001::1
ifconfig-push 2a07:de40:b27e:5001::152/64 2a07:de40:b27e:5001::1

I can now connect to mailman3, but not to anna. I guess anna has yet to be migrated.

No, anna/elsa will be left behind. See the list at the end of https://etherpad.opensuse.org/p/move-nue-prg.

I installed the new internal resolvers earlier today. Please now use 2a07:de40:b27e:1203::11 and 2a07:de40:b27e:1203::12 as forwarders for infra.opensuse.org. They should allow you to resolve old/NUE1 hosts such as anna/elsa as well.

Edit: updated the pushed nameserver addresses now as well.

Now that you are connected, if you want to take a look yourself you are welcome to

ssh -J thor1.infra.opensuse.org odin.infra.opensuse.org

crameleon wrote in #note-34:

Now that you are connected, if you want to take a look yourself you are welcome to

Thanks :-)

ssh -J thor1.infra.opensuse.org odin.infra.opensuse.org

Hmm, I can't reach thor1 -

per@office68:~> ip route get 2a07:de40:b27e:1100::a
2a07:de40:b27e:1100::a from :: via 2a07:de40:b27e:5001::1 dev tunk src 2a07:de40:b27e:5001::1004 metric 1024 pref medium
per@office68:~> ping6 2a07:de40:b27e:1100::a
PING 2a07:de40:b27e:1100::a(2a07:de40:b27e:1100::a) 56 data bytes
^C
--- 2a07:de40:b27e:1100::a ping statistics ---
13 packets transmitted, 0 received, 100% packet loss, time 12265ms

My routes (installed by OpenVPN) look like this:

$ ip r g 2a07:de40:b27e:1100::a
2a07:de40:b27e:1100::a from :: dev tun-heroes src 2a07:de40:b27e:5001::1002 metric 1024 pref medium

$ ip -6 r s|grep tun
2a07:de40:b27e:64::/96 dev tun-heroes metric 1024 pref medium
2a07:de40:b27e:1100::/64 dev tun-heroes metric 1024 pref medium
2a07:de40:b27e:1203::/64 dev tun-heroes metric 1024 pref medium
2a07:de40:b27e:1205::/64 dev tun-heroes metric 1024 pref medium
2a07:de40:b27e:5001::/64 dev tun-heroes proto kernel metric 256 pref medium
fe80::/64 dev tun-heroes proto kernel metric 256 pref medium

Odd why you can reach os-internal (1203) and not os-thor (1100) with your setup, I'd expect either all the routes or none of them to work.

crameleon wrote in #note-36:

Odd why you can reach os-internal (1203) and not os-thor (1100) with your setup, I'd expect either all the routes or none of them to work.

I think I screwed something up - before I could connect to mailman3, but that has also stopped. I'll be back.

Hmm, I'm not sure what has happened. This is my current setup:

23: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none 
    inet6 2a07:de40:b27e:5001::1004/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::ab7c:4f43:faa4:a455/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever

office68:~ # ip -6 route show dev tun0
2a07:de40:b27e:64::/96 via 2a07:de40:b27e:5001::1 metric 1024 pref medium
2a07:de40:b27e:1100::/64 via 2a07:de40:b27e:5001::1 metric 1024 pref medium
2a07:de40:b27e:1203::/64 via 2a07:de40:b27e:5001::1 metric 1024 pref medium
2a07:de40:b27e:1205::/64 via 2a07:de40:b27e:5001::1 metric 1024 pref medium
2a07:de40:b27e:5001::/64 proto kernel metric 256 pref medium
fe80::/64 proto kernel metric 256 pref medium

However, I cannot ping the nameservers:

office68:~ # ip route get 2a07:de40:b27e:64::c0a8:2f66
2a07:de40:b27e:64::c0a8:2f66 from :: via 2a07:de40:b27e:5001::1 dev tun0 src 2a07:de40:b27e:5001::1004 metric 1024 pref medium
office68:~ # ping 2a07:de40:b27e:64::c0a8:2f66
PING 2a07:de40:b27e:64::c0a8:2f66(2a07:de40:b27e:64::c0a8:2f66) 56 data bytes
^C
--- 2a07:de40:b27e:64::c0a8:2f66 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3049ms

The infra.o.o names resolve fine publicly though: (are they meant to?)

office68:~ # dig @8.8.8.8 mailman3.infra.opensuse.org aaaa +short
2a07:de40:b27e:1203::b46

I have to leave it for now, I'll probably resume tomorrow.

You can omit the forwarders and just use your default/public nameservers, but you will not benefit from DNS64 to resolve and access names for legacy infrastructure in Nuremberg and Provo. Hence I recommend to use the VPN advertised nameservers (2a07:de40:b27e:1203::11 and 2a07:de40:b27e:1203::12), which will ensure all i.o.o names resolve to something useful.

crameleon wrote in #note-39:

You can omit the forwarders and just use your default/public nameservers, but you will not benefit from DNS64 to resolve and access names for legacy infrastructure in Nuremberg and Provo. Hence I recommend to use the VPN advertised nameservers (2a07:de40:b27e:1203::11 and 2a07:de40:b27e:1203::12),

Above, the PUSH option said 2a07:de40:b27e:64::c0a8:2f65 and 2a07:de40:b27e:64::c0a8:2f66, so that's what I used. I see, it has changed now :-)
Got it - I didn't not expect the allocated IP address to change, mea culpa.
All/mostly good now. I can access thor1, but it doesn't look like I'm getting any further. No reply.

pjessen wrote in #note-40:

All/mostly good now. I can access thor1, but it doesn't look like I'm getting any further. No reply.

Okay, I don't know what changed, but I have access to odin now. My ccd-udp makes me curious - it is clearly static, as before:
ifconfig-push 2a07:de40:b27e:5001::207 2a07:de40:b27e:5001::1
but this is not what I see in the PUSH : ifconfig-ipv6 2a07:de40:b27e:5001::1000/64 2a07:de40:b27e:5001::1 ? Two days back I had ifconfig-ipv6 2a07:de40:b27e:5001::1004/64 2a07:de40:b27e:5001::1 - I am not quite ajour with our vpn config, but shouldn't I be seeing my tun0 configured as 2a07:de40:b27e:5001::207/64 ?
It does not explain why tun0 isn't being configured correctly on my side though.

See what I wrote in https://progress.opensuse.org/issues/139280#note-25. If you have a solution, I'm more than interested. :)

It is also causing issues with two of our machines at QSC which need to have a static address for special firewall rules.

crameleon wrote in #note-42:

See what I wrote in https://progress.opensuse.org/issues/139280#note-25. If you have a solution, I'm more than interested. :)
It is also causing issues with two of our machines at QSC which need to have a static address for special firewall rules.

Ah, sorry, I managed to skip that sentence. Okay, that is very interesting, I'll see if I can work out what's happening.

Changed ifconfig-push to ifconfig-ipv6-push for all users now, static address confirmed to work for two users, please try again on your end.

crameleon wrote in #note-44:

Changed ifconfig-push to ifconfig-ipv6-push for all users now, static address confirmed to work for two users, please try again on your end.

The PUSH looks better now: PUSH_REPLY .... ifconfig-ipv6 2a07:de40:b27e:5001::207/64 2a07:de40:b27e:5001::1.
Still doesn't get my interface configured, but that must be an issue on my end.

Thanks for acknowledging. :-)

Instead of a cheeky comment I opt for expressing my joy about this issue being resolved!