action #137075
closedFail to login to the osd, 'Forbidden' error is returned due to DNS server change within SUSE *and* auto_review:"Bugzilla query failed: Network is unreachable":retry size:M
Added by GraceWang 8 months ago. Updated 8 months ago.
0%
Description
Observation¶
I am not able to login to the osd.
"Forbidden" error is returned.
Acceptance Criteria¶
- AC1: All machines within qe.nue2 and oqa.prg2 are able to resolve internal and external domains consistently again
Suggestions¶
- Update DNS config on openqa.suse.de according to upstream DNS changes e.g. https://sd.suse.com/servicedesk/customer/portal/1/SD-133348
- Crosscheck that all machines controlled over salt can resolve DNS
- Lookup affected openQA tests on openqa.suse.de and handle them, e.g. auto-review+retry
Steps to reproduce¶
Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
call openqa-query-for-job-label poo#137075
Updated by okurz 8 months ago
- Status changed from New to In Progress
- Assignee set to okurz
- Priority changed from High to Urgent
- Target version set to Ready
DNS issue on OSD. /etc/resolv.conf has
nameserver 10.145.10.21
nameserver 10.145.10.22
both not reachable.
on osd I added to /etc/resolv.conf
nameserver 2620:113:80c0:8080:10:160:0:1
nameserver 2620:113:80c0:8080:10:160:2:88
nameserver 10.160.0.1
nameserver 10.160.2.88
Updated by okurz 8 months ago
- Tags set to infra
- Project changed from openQA Project to openQA Infrastructure
- Category deleted (
Regressions/Crashes) - Status changed from In Progress to Blocked
- Priority changed from Urgent to High
workaround applied, ticket created https://sd.suse.com/servicedesk/customer/portal/1/SD-133348
Updated by okurz 8 months ago
- Status changed from Blocked to In Progress
I now put into /etc/sysconfig/network/config
# $ host dns1.prg2.suse.org dns2.prg2.suse.org
NETCONFIG_DNS_STATIC_SERVERS="2a07:de40:b205:7:10:144:53:53 10.144.53.53 2a07:de40:b205:7:10:144:53:54 10.144.53.54 2620:113:80c0:8080:10:160:0:1 2620:113:80c0:8080:10:160:2:88 10.160.0.1 10.160.2.88"
however only the first entries end up in /etc/resolv.conf
It seems many more jobs are affected https://suse.slack.com/archives/C02CANHLANP/p1695804358586719
(Lemon Li) Hi, there are many test failed for DNS issue, https://openqa.suse.de/tests/12319768#step/patch_sle/25 Please have a look. Thanks.
It seems related to the Eng-Infra planned issue which I can't access right now due to jira being migrated.
To handle the effect calling
for i in worker{31..40}; do env host=openqa.suse.de result="result='failed'" openqa-advanced-retrigger-jobs; done
Updated by okurz 8 months ago
https://suse.slack.com/archives/C02CANHLANP/p1695734009294569 might be related.
@qa-tools Hello, there are some network issues on petrol-1:
https://openqa.suse.de/tests/12281734#step/zram01/12[2023-09-26T13:33:41.324928+02:00] [debug] [pid:17800] Downloading bug info: http://fastzilla.suse.de/short1186059.xml [2023-09-26T13:35:51.385001+02:00] [debug] [pid:17800] Bugzilla query failed: Network is unreachable at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Transaction.pm line 54.
I found no obvious problem on petrol-1. I triggered a reboot.
Updated by okurz 8 months ago
- Related to action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker services added
Updated by okurz 8 months ago
- Subject changed from Fail to login to the osd, "Forbidden" error is returned. to Fail to login to the osd, 'Forbidden' error is returned *and* auto_review:"Bugzilla query failed: Network is unreachable":retry
- Description updated (diff)
Found #137114. But openQA jobs look fine so far, e.g. https://openqa.suse.de/tests/12285687, https://openqa.suse.de/tests/12285686, https://openqa.suse.de/tests/12285688, https://openqa.suse.de/tests/12285695 . I am not sure those problems are connected.
Updated by okurz 8 months ago
- Due date set to 2023-10-11
- Status changed from In Progress to Feedback
DNS config was updated, tests on diesel+petrol good. I could not look into the overall job result situation further today but need to rely on other parties to tell me if something else is necessary to be done.
Updated by livdywan 8 months ago
- Subject changed from Fail to login to the osd, 'Forbidden' error is returned *and* auto_review:"Bugzilla query failed: Network is unreachable":retry to Fail to login to the osd, 'Forbidden' error is returned due to DNS server change within SUSE *and* auto_review:"Bugzilla query failed: Network is unreachable":retry size:M
- Description updated (diff)
Updated by okurz 8 months ago
- Due date deleted (
2023-10-11) - Status changed from Feedback to Resolved
$ openqa-query-for-job-label poo#137075
12281734|2023-09-26 11:36:32|done|failed|ltp_kernel_misc||petrol-1
And checked from OSD salt:
sudo salt \* cmd.run 'host id.opensuse.org'
worker39.oqa.prg2.suse.org:
id.opensuse.org is an alias for login2.opensuse.org.
login2.opensuse.org has address 195.135.221.161
login2.opensuse.org has IPv6 address 2001:67c:2178:8::161
…
all look good so far.
Also
okurz@openqa:~> sudo salt \* cmd.run 'grep nameserver /etc/resolv.conf'
worker39.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
worker31.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
worker38.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
worker40.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
worker29.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
backup-qam.qe.nue2.suse.org:
nameserver 10.168.0.1
nameserver 10.168.0.2
worker30.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
worker37.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
worker-arm2.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
sapworker2.qe.nue2.suse.org:
nameserver 10.168.0.1
nameserver 10.168.0.2
sapworker3.qe.nue2.suse.org:
nameserver 10.168.0.1
nameserver 10.168.0.2
worker-arm1.oqa.prg2.suse.org:
nameserver 10.144.53.53
nameserver 10.144.53.54
sapworker1.qe.nue2.suse.org:
nameserver 10.168.0.1
nameserver 10.168.0.2
openqa.suse.de:
nameserver 2a07:de40:b205:7:10:144:53:53
nameserver 10.144.53.53
nameserver 2a07:de40:b205:7:10:144:53:54
storage.oqa.suse.de:
nameserver 10.136.53.53
nameserver 10.136.53.54
nameserver 10.100.2.10
openqaworker18.qa.suse.cz:
nameserver 10.100.96.1
nameserver 10.100.96.2
openqaworker16.qa.suse.cz:
nameserver 10.100.96.1
nameserver 10.100.96.2
openqaworker17.qa.suse.cz:
nameserver 10.100.96.1
nameserver 10.100.96.2
worker5.oqa.suse.de:
nameserver 10.136.53.53
nameserver 10.136.53.54
nameserver 10.100.2.10
qesapworker-prg4.qa.suse.cz:
nameserver 10.100.96.1
nameserver 10.100.96.2
openqaworker1.qe.nue2.suse.org:
nameserver 10.168.0.1
nameserver 10.168.0.2
openqaworker14.qa.suse.cz:
nameserver 10.100.96.1
nameserver 10.100.96.2
worker2.oqa.suse.de:
nameserver 10.136.53.53
nameserver 10.136.53.54
nameserver 10.100.2.10
qamasternue.qa.suse.de:
nameserver 10.168.0.1
nameserver 10.168.0.2
petrol.qe.nue2.suse.org:
nameserver 10.168.0.1
nameserver 10.168.0.2
powerqaworker-qam-1.qa.suse.de:
nameserver 2620:113:80c0:80a0:10:162:0:1
nameserver 10.162.0.1
jenkins.qa.suse.de:
nameserver 10.168.0.1
nameserver 10.168.0.2
qesapworker-prg5.qa.suse.cz:
nameserver 10.100.96.1
nameserver 10.100.96.2
qesapworker-prg6.qa.suse.cz:
nameserver 10.100.96.1
nameserver 10.100.96.2
qesapworker-prg7.qa.suse.cz:
nameserver 10.100.96.1
nameserver 10.100.96.2
imagetester.qe.nue2.suse.org:
nameserver 10.168.0.1
nameserver 10.168.0.2
baremetal-support.qa.suse.de:
nameserver 10.168.0.1
nameserver 10.168.0.2
openqa-monitor.qa.suse.de:
nameserver 10.168.0.1
nameserver 10.168.0.2
worker10.oqa.suse.de:
nameserver 10.136.53.53
nameserver 10.136.53.54
nameserver 10.100.2.10
diesel.qe.nue2.suse.org:
nameserver 10.168.0.1
nameserver 10.168.0.2
malbec.arch.suse.de:
nameserver 10.162.0.1
nameserver 10.160.0.1
nameserver 149.44.160.1
schort-server.qa.suse.de:
nameserver 10.168.0.1
nameserver 10.168.0.2
tumblesle.qa.suse.de:
nameserver 10.168.0.1
nameserver 10.168.0.2
openqaw5-xen.qa.suse.de:
nameserver 10.162.0.1
nameserver 2620:113:80c0:80a0:10:162:0:1
openqa-piworker.qa.suse.de:
nameserver 10.168.192.1
nameserver 10.168.0.1
nameserver 10.168.0.2
openqaworker-arm-2.suse.de:
nameserver 10.136.53.53
nameserver 10.136.53.54
nameserver 10.100.2.10
openqaworker-arm-3.suse.de:
nameserver 10.136.53.53
nameserver 10.136.53.54
nameserver 10.100.2.10
backup.qa.suse.de:
nameserver 10.168.0.1
nameserver 10.168.0.2
Commented on https://jira.suse.com/browse/ENGINFRA-2471