openQA Project (public) » openQA Infrastructure (public)

These are the upstream DNS servers defined in our dnsmasq config

new-ariel:/etc/dnsmasq.d # grep -ri "server="
openqa.conf:server=8.8.8.8
openqa.conf:server=/infra.opensuse.org/192.168.47.4
openqa.conf:server=/47.168.192.in-addr.arpa/192.168.47.4

only 8.8.8.8 is defined which is already discussed in other forums to cause problems similar to this: https://forum.openwrt.org/t/dnsmasq-maximum-concurrent-dns-queries-limit/164427
I've added 1.1.1.1 and 9.9.9.9 now and reloaded (trying to avoid having the issue with 8.8.8.8 or the queue size fixed by a restart) and queries seem to resolve after a short timeout:

new-ariel:/etc # dig heise.de

; <<>> DiG 9.16.43 <<>> heise.de
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1329
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;heise.de.          IN  A

;; ANSWER SECTION:
heise.de.       84524   IN  A   193.99.144.80

;; Query time: 0 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Thu Sep 21 17:23:20 UTC 2023
;; MSG SIZE  rcvd: 53

@nicksinger thank you. Seems we stepped on each other's toes

… us ourselves on eth1 anyway? Maybe I am just getting confused with the entry that is passed on to DHCP clients.

/etc/dnsmasq.d/openqa.conf says server=10.151.53.53 but 10.151.53.53 does not respond to pings.

I tried to add another entry in /etc/dnsmasq.d/openqa.conf with server=1.1.1.1 but that does not seem to help.

As workaround I added nameserver 1.1.1.1 into /etc/resolv.conf. That seems to work

okurz@new-ariel:~> nslookup github.com
Server:     1.1.1.1
Address:    1.1.1.1#53

Non-authoritative answer:
Name:   github.com
Address: 140.82.121.4

I reported https://sd.suse.com/servicedesk/customer/portal/1/SD-132971

Right, I was confused why server=10.151.53.53 was commented just right when I wanted to update the ticket :')
From the journal of dnsmasq I see that we hand out the IP of new-ariel as DNS server:

Aug 29 23:05:06 new-ariel dnsmasq-dhcp[24171]: 1804410612 sent size:  4 option:  6 dns-server  10.150.1.11

which is correct as new-ariel is a recursive server for all other hosts in the network. Adding anything else into /etc/resolv.conf shouldn't be necessary as dnsmasq should provide DNS for ariel itself too and would hide away issues for the workers. Upstream resolvers are handled by the corresponding server= entries in the config of dnsmasq. Checking dnsmasq right now with strace -p 16831 (the PID of the process listening on port 53) shows that it is stuck in trying to reach 10.151.53.53:

strace: Process 16831 attached
sendto(5, "\0-\0\1\1\0\0\1\0\0\0\0\0\1\3dns\10msftncsi\3com\0"..., 47, MSG_FASTOPEN, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("10.151.53.53")}, 16

ss -tupn | grep 10.151.53.53 confirms open connections:

tcp   SYN-SENT   0      1        10.150.2.10:53254    10.151.53.53:53    users:(("dnsmasq",pid=18926,fd=5))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
tcp   SYN-SENT   0      1        10.150.2.10:55616    10.151.53.53:53    users:(("dnsmasq",pid=22076,fd=5))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
tcp   SYN-SENT   0      1        10.150.2.10:33776    10.151.53.53:53    users:(("dnsmasq",pid=19462,fd=5))                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
tcp   SYN-SENT   0      1        10.150.2.10:56814    10.151.53.53:53    users:(("dnsmasq",pid=18922,fd=5))

I just restarted dnsmasq.service to kill these hanging connections to not cause further problems on our workers. Now 127.0.0.1 responds immediately:

/etc/dnsmasq.d # dig heise.de @127.0.0.1

; <<>> DiG 9.16.43 <<>> heise.de @127.0.0.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54140
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;heise.de.          IN  A

;; ANSWER SECTION:
heise.de.       549 IN  A   193.99.144.80

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 21 17:37:24 UTC 2023
;; MSG SIZE  rcvd: 53

So would you agree that we can remove 1.1.1.1 again from /etc/resolv.conf?