Project

General

Profile

Actions
Copy link

action #137075

closed

Fail to login to the osd, 'Forbidden' error is returned due to DNS server change within SUSE *and* auto_review:"Bugzilla query failed: Network is unreachable":retry size:M

Added by GraceWang about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
okurz
Category:
-
Target version:
openQA Project (public) - Ready
Start date:
2023-09-27
Due date:
% Done:

0%

Estimated time:
Tags:
infra

Description

Observation

I am not able to login to the osd.
"Forbidden" error is returned.

Acceptance Criteria

  • AC1: All machines within qe.nue2 and oqa.prg2 are able to resolve internal and external domains consistently again

Suggestions

Steps to reproduce

Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
call openqa-query-for-job-label poo#137075

Related issues 1 (1 open — 0 closed)

Related to openQA Infrastructure (public) - action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker servicesNew

Actions
Actions
Copy link
 #1

Updated by okurz about 1 year ago

  • Status changed from New to In Progress
  • Assignee set to okurz
  • Priority changed from High to Urgent
  • Target version set to Ready

DNS issue on OSD. /etc/resolv.conf has

nameserver 10.145.10.21
nameserver 10.145.10.22

both not reachable.

on osd I added to /etc/resolv.conf

nameserver 2620:113:80c0:8080:10:160:0:1
nameserver 2620:113:80c0:8080:10:160:2:88
nameserver 10.160.0.1
nameserver 10.160.2.88
Actions
Copy link
 #2

Updated by okurz about 1 year ago

  • Tags set to infra
  • Project changed from openQA Project (public) to openQA Infrastructure (public)
  • Category deleted (Regressions/Crashes)
  • Status changed from In Progress to Blocked
  • Priority changed from Urgent to High
Actions
Copy link
 #3

Updated by okurz about 1 year ago

DNS has moved to other DNS servers. Should use

nameserver 10.136.53.53
nameserver 10.136.53.54

from jnovak.

Updated /etc/sysconfig/network/config accordingly.

Actions
Copy link
 #4

Updated by okurz about 1 year ago

  • Status changed from Blocked to In Progress

I now put into /etc/sysconfig/network/config

# $ host dns1.prg2.suse.org dns2.prg2.suse.org
NETCONFIG_DNS_STATIC_SERVERS="2a07:de40:b205:7:10:144:53:53 10.144.53.53 2a07:de40:b205:7:10:144:53:54 10.144.53.54 2620:113:80c0:8080:10:160:0:1 2620:113:80c0:8080:10:160:2:88 10.160.0.1 10.160.2.88"

however only the first entries end up in /etc/resolv.conf

It seems many more jobs are affected https://suse.slack.com/archives/C02CANHLANP/p1695804358586719

(Lemon Li) Hi, there are many test failed for DNS issue, https://openqa.suse.de/tests/12319768#step/patch_sle/25 Please have a look. Thanks.

It seems related to the Eng-Infra planned issue which I can't access right now due to jira being migrated.

To handle the effect calling

for i in worker{31..40}; do env host=openqa.suse.de result="result='failed'" openqa-advanced-retrigger-jobs; done
Actions
Copy link
 #5

Updated by okurz about 1 year ago

https://suse.slack.com/archives/C02CANHLANP/p1695734009294569 might be related.

@qa-tools Hello, there are some network issues on petrol-1:
https://openqa.suse.de/tests/12281734#step/zram01/12

[2023-09-26T13:33:41.324928+02:00] [debug] [pid:17800] Downloading bug info: http://fastzilla.suse.de/short1186059.xml
[2023-09-26T13:35:51.385001+02:00] [debug] [pid:17800] Bugzilla query failed: Network is unreachable at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Transaction.pm line 54.

I found no obvious problem on petrol-1. I triggered a reboot.

Actions
Copy link
 #6

Updated by okurz about 1 year ago

  • Related to action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker services added
Actions
Copy link
 #7

Updated by okurz about 1 year ago

  • Subject changed from Fail to login to the osd, "Forbidden" error is returned. to Fail to login to the osd, 'Forbidden' error is returned *and* auto_review:"Bugzilla query failed: Network is unreachable":retry
  • Description updated (diff)
Actions
Copy link
 #8

Updated by okurz about 1 year ago

  • Due date set to 2023-10-11
  • Status changed from In Progress to Feedback

DNS config was updated, tests on diesel+petrol good. I could not look into the overall job result situation further today but need to rely on other parties to tell me if something else is necessary to be done.

Actions
Copy link
 #9

Updated by livdywan about 1 year ago

  • Subject changed from Fail to login to the osd, 'Forbidden' error is returned *and* auto_review:"Bugzilla query failed: Network is unreachable":retry to Fail to login to the osd, 'Forbidden' error is returned due to DNS server change within SUSE *and* auto_review:"Bugzilla query failed: Network is unreachable":retry size:M
  • Description updated (diff)
Actions
Copy link
 #10

Updated by okurz about 1 year ago

  • Due date deleted (2023-10-11)
  • Status changed from Feedback to Resolved
$ openqa-query-for-job-label poo#137075
12281734|2023-09-26 11:36:32|done|failed|ltp_kernel_misc||petrol-1

And checked from OSD salt:

sudo salt \* cmd.run 'host id.opensuse.org'
worker39.oqa.prg2.suse.org:
    id.opensuse.org is an alias for login2.opensuse.org.
    login2.opensuse.org has address 195.135.221.161
    login2.opensuse.org has IPv6 address 2001:67c:2178:8::161
…

all look good so far.

Also

okurz@openqa:~> sudo salt \* cmd.run 'grep nameserver /etc/resolv.conf'
worker39.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker31.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker38.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker40.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker29.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
backup-qam.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
worker30.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker37.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker-arm2.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
sapworker2.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
sapworker3.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
worker-arm1.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
sapworker1.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqa.suse.de:
    nameserver 2a07:de40:b205:7:10:144:53:53
    nameserver 10.144.53.53
    nameserver 2a07:de40:b205:7:10:144:53:54
storage.oqa.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
openqaworker18.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
openqaworker16.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
openqaworker17.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
worker5.oqa.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
qesapworker-prg4.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
openqaworker1.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqaworker14.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
worker2.oqa.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
qamasternue.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
petrol.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
powerqaworker-qam-1.qa.suse.de:
    nameserver 2620:113:80c0:80a0:10:162:0:1
    nameserver 10.162.0.1
jenkins.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
qesapworker-prg5.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
qesapworker-prg6.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
qesapworker-prg7.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
imagetester.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
baremetal-support.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqa-monitor.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
worker10.oqa.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
diesel.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
malbec.arch.suse.de:
    nameserver 10.162.0.1
    nameserver 10.160.0.1
    nameserver 149.44.160.1
schort-server.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
tumblesle.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqaw5-xen.qa.suse.de:
    nameserver 10.162.0.1
    nameserver 2620:113:80c0:80a0:10:162:0:1
openqa-piworker.qa.suse.de:
    nameserver 10.168.192.1
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqaworker-arm-2.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
openqaworker-arm-3.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
backup.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2

Commented on https://jira.suse.com/browse/ENGINFRA-2471

Actions
Copy link

Also available in: Atom PDF