Project

General

Profile

Actions
Copy link

action #134879

closed

QA - coordination #121720: [saga][epic] Migration to QE setup in PRG2+NUE3 while ensuring availability

QA - coordination #123800: [epic] Provide SUSE QE Tools services running in PRG2 aka. Prg CoLo

reverse DNS resolution PTR for openqa.oqa.prg2.suse.org. yields "3(NXDOMAIN)" for PRG1 workers (NUE1+PRG2 are fine) size:M

Added by okurz 9 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project - Ready
Start date:
2023-08-31
Due date:
% Done:

0%

Estimated time:
Tags:
migration, infra, reactive work, DNS, prg1

Description

Observation

Based on https://suse.slack.com/archives/C02CANHLANP/p1693393323780419

@qa-tools Hello, new openqa.suse.de host does not seem to have reverse DNS entry which breaks one of our tests: https://openqa.suse.de/tests/11948174#step/host/8
$ host openqa.suse.de
openqa.suse.de is an alias for openqa.oqa.prg2.suse.org.
openqa.oqa.prg2.suse.org has address 10.145.10.207
$ host 10.145.10.207
Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)

which was mostly fixed by https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3935 . For qanet using machines nicksinger applied a fix

(Nick Singer) for some reason qanet feels authoritative for the whole 10.IN-ADDR.ARPA. zone which is wrong and I don't understand where it comes from. I have to dig deeper to understand it. ah, found it. The feature is called "automatic empty zones" (https://kb.isc.org/docs/aa-00800) and automatically handles requests which are not supposed to reach the internet even if they are not explicitly defined as master. Since we use a suse-internal DNS as upstream we can safely disable this feature which I did now […] the config has to be done in /etc/named.conf - at least I did it there on qanet

But potentially we have the same problem still for PRG1 based workers

(Oliver Kurz) […] does this explain the problem in Prague workers as well?
(Nick Singer) if the prague network runs its own downstream dns-server then yes, it would explain it. at least in the qe.nue2.suse.org-domain I can see that walter1 and walter2 are downstream dns servers. But I haven't checked if they contain the same "flaw"

From OSD salt \* cmd.run 'host 10.145.10.207'

worker33.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker31.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
backup-qam.qe.nue2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker39.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker35.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker34.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker30.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker36.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker32.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker-arm1.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker38.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker29.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker-arm2.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
sapworker2.qe.nue2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker37.oqa.prg2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
sapworker3.qe.nue2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
sapworker1.qe.nue2.suse.org:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqaworker17.qa.suse.cz:
    Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)
worker8.oqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker3.oqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker9.oqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqaworker18.qa.suse.cz:
    Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)
storage.oqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker5.oqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqaworker16.qa.suse.cz:
    Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)
qesapworker-prg7.qa.suse.cz:
    Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)
qesapworker-prg5.qa.suse.cz:
    Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)
powerqaworker-qam-1.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
qesapworker-prg4.qa.suse.cz:
    Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)
openqaworker14.qa.suse.cz:
    Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)
QA-Power8-5-kvm.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
qesapworker-prg6.qa.suse.cz:
    Host 207.10.145.10.in-addr.arpa. not found: 3(NXDOMAIN)
QA-Power8-4-kvm.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
malbec.arch.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker2.oqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker10.oqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
worker13.oqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
qamasternue.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqa-piworker.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqaworker-arm-2.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqaworker-arm-3.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
baremetal-support.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
backup.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
jenkins.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqa-monitor.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
openqaw5-xen.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
tumblesle.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.
schort-server.qa.suse.de:
    207.10.145.10.in-addr.arpa domain name pointer openqa.oqa.prg2.suse.org.

so it seems all machines except PRG1 ones are good.

Acceptance criteria

  • AC1: Reverse DNS resolution for all OSD salt controlled machines works

Suggestions

  • Ask Eng-Infra to check for the PRG1 based DNS server and propose the same solution as we applied for qanet

Workaround

Reschedule affected tests in not-PRG1 workers

Related issues 3 (0 open — 3 closed)

Related to QA - action #132146: Support migration of osd VM to PRG2 - 2023-08-29 size:MResolvedmkittler2023-06-29

Actions
Related to openQA Infrastructure - action #134912: Gradually phase out NUE1 based openQA workers size:MResolvedokurz

Actions
Related to openQA Infrastructure - action #135230: salt pillars pipelines failing due to Temporary failure in name resolutionResolvednicksinger2023-09-06

Actions
Actions
Copy link

Also available in: Atom PDF