Project

General

Profile

Actions

action #137075

closed

Fail to login to the osd, 'Forbidden' error is returned due to DNS server change within SUSE *and* auto_review:"Bugzilla query failed: Network is unreachable":retry size:M

Added by GraceWang 9 months ago. Updated 9 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
2023-09-27
Due date:
% Done:

0%

Estimated time:
Tags:

Description

Observation

I am not able to login to the osd.
"Forbidden" error is returned.

Acceptance Criteria

  • AC1: All machines within qe.nue2 and oqa.prg2 are able to resolve internal and external domains consistently again

Suggestions

Steps to reproduce

Find jobs referencing this ticket with the help of
https://raw.githubusercontent.com/os-autoinst/scripts/master/openqa-query-for-job-label ,
call openqa-query-for-job-label poo#137075


Related issues 1 (1 open0 closed)

Related to openQA Infrastructure - action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker servicesNew

Actions
Actions #1

Updated by okurz 9 months ago

  • Status changed from New to In Progress
  • Assignee set to okurz
  • Priority changed from High to Urgent
  • Target version set to Ready

DNS issue on OSD. /etc/resolv.conf has

nameserver 10.145.10.21
nameserver 10.145.10.22

both not reachable.

on osd I added to /etc/resolv.conf

nameserver 2620:113:80c0:8080:10:160:0:1
nameserver 2620:113:80c0:8080:10:160:2:88
nameserver 10.160.0.1
nameserver 10.160.2.88
Actions #2

Updated by okurz 9 months ago

  • Tags set to infra
  • Project changed from openQA Project to openQA Infrastructure
  • Category deleted (Regressions/Crashes)
  • Status changed from In Progress to Blocked
  • Priority changed from Urgent to High
Actions #3

Updated by okurz 9 months ago

DNS has moved to other DNS servers. Should use

nameserver 10.136.53.53
nameserver 10.136.53.54

from jnovak.

Updated /etc/sysconfig/network/config accordingly.

Actions #4

Updated by okurz 9 months ago

  • Status changed from Blocked to In Progress

I now put into /etc/sysconfig/network/config

# $ host dns1.prg2.suse.org dns2.prg2.suse.org
NETCONFIG_DNS_STATIC_SERVERS="2a07:de40:b205:7:10:144:53:53 10.144.53.53 2a07:de40:b205:7:10:144:53:54 10.144.53.54 2620:113:80c0:8080:10:160:0:1 2620:113:80c0:8080:10:160:2:88 10.160.0.1 10.160.2.88"

however only the first entries end up in /etc/resolv.conf

It seems many more jobs are affected https://suse.slack.com/archives/C02CANHLANP/p1695804358586719

(Lemon Li) Hi, there are many test failed for DNS issue, https://openqa.suse.de/tests/12319768#step/patch_sle/25 Please have a look. Thanks.

It seems related to the Eng-Infra planned issue which I can't access right now due to jira being migrated.

To handle the effect calling

for i in worker{31..40}; do env host=openqa.suse.de result="result='failed'" openqa-advanced-retrigger-jobs; done
Actions #5

Updated by okurz 9 months ago

https://suse.slack.com/archives/C02CANHLANP/p1695734009294569 might be related.

@qa-tools Hello, there are some network issues on petrol-1:
https://openqa.suse.de/tests/12281734#step/zram01/12

[2023-09-26T13:33:41.324928+02:00] [debug] [pid:17800] Downloading bug info: http://fastzilla.suse.de/short1186059.xml
[2023-09-26T13:35:51.385001+02:00] [debug] [pid:17800] Bugzilla query failed: Network is unreachable at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Transaction.pm line 54.

I found no obvious problem on petrol-1. I triggered a reboot.

Actions #6

Updated by okurz 9 months ago

  • Related to action #137114: openQA workers fail to register after bootup due to unable to resolve openqa.suse.de but manage to do so immediately when restarting worker services added
Actions #7

Updated by okurz 9 months ago

  • Subject changed from Fail to login to the osd, "Forbidden" error is returned. to Fail to login to the osd, 'Forbidden' error is returned *and* auto_review:"Bugzilla query failed: Network is unreachable":retry
  • Description updated (diff)
Actions #8

Updated by okurz 9 months ago

  • Due date set to 2023-10-11
  • Status changed from In Progress to Feedback

DNS config was updated, tests on diesel+petrol good. I could not look into the overall job result situation further today but need to rely on other parties to tell me if something else is necessary to be done.

Actions #9

Updated by livdywan 9 months ago

  • Subject changed from Fail to login to the osd, 'Forbidden' error is returned *and* auto_review:"Bugzilla query failed: Network is unreachable":retry to Fail to login to the osd, 'Forbidden' error is returned due to DNS server change within SUSE *and* auto_review:"Bugzilla query failed: Network is unreachable":retry size:M
  • Description updated (diff)
Actions #10

Updated by okurz 9 months ago

  • Due date deleted (2023-10-11)
  • Status changed from Feedback to Resolved
$ openqa-query-for-job-label poo#137075
12281734|2023-09-26 11:36:32|done|failed|ltp_kernel_misc||petrol-1

And checked from OSD salt:

sudo salt \* cmd.run 'host id.opensuse.org'
worker39.oqa.prg2.suse.org:
    id.opensuse.org is an alias for login2.opensuse.org.
    login2.opensuse.org has address 195.135.221.161
    login2.opensuse.org has IPv6 address 2001:67c:2178:8::161
…

all look good so far.

Also

okurz@openqa:~> sudo salt \* cmd.run 'grep nameserver /etc/resolv.conf'
worker39.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker31.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker38.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker40.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker29.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
backup-qam.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
worker30.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker37.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
worker-arm2.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
sapworker2.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
sapworker3.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
worker-arm1.oqa.prg2.suse.org:
    nameserver 10.144.53.53
    nameserver 10.144.53.54
sapworker1.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqa.suse.de:
    nameserver 2a07:de40:b205:7:10:144:53:53
    nameserver 10.144.53.53
    nameserver 2a07:de40:b205:7:10:144:53:54
storage.oqa.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
openqaworker18.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
openqaworker16.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
openqaworker17.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
worker5.oqa.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
qesapworker-prg4.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
openqaworker1.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqaworker14.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
worker2.oqa.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
qamasternue.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
petrol.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
powerqaworker-qam-1.qa.suse.de:
    nameserver 2620:113:80c0:80a0:10:162:0:1
    nameserver 10.162.0.1
jenkins.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
qesapworker-prg5.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
qesapworker-prg6.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
qesapworker-prg7.qa.suse.cz:
    nameserver 10.100.96.1
    nameserver 10.100.96.2
imagetester.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
baremetal-support.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqa-monitor.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
worker10.oqa.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
diesel.qe.nue2.suse.org:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
malbec.arch.suse.de:
    nameserver 10.162.0.1
    nameserver 10.160.0.1
    nameserver 149.44.160.1
schort-server.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
tumblesle.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqaw5-xen.qa.suse.de:
    nameserver 10.162.0.1
    nameserver 2620:113:80c0:80a0:10:162:0:1
openqa-piworker.qa.suse.de:
    nameserver 10.168.192.1
    nameserver 10.168.0.1
    nameserver 10.168.0.2
openqaworker-arm-2.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
openqaworker-arm-3.suse.de:
    nameserver 10.136.53.53
    nameserver 10.136.53.54
    nameserver 10.100.2.10
backup.qa.suse.de:
    nameserver 10.168.0.1
    nameserver 10.168.0.2

Commented on https://jira.suse.com/browse/ENGINFRA-2471

Actions

Also available in: Atom PDF