Project

General

Profile

Actions

action #125450

closed

coordination #125363: [epic] Improve collaboration with Eng-Infra

Improve collaboration with Eng-Infra - Firewall management access, potentially also DHCP+DNS size:M

Added by okurz about 1 year ago. Updated 9 months ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
Start date:
2023-03-06
Due date:
% Done:

0%

Estimated time:

Description

Motivation

Apparently in many cases @rwawrig can help best with issues spanning over multiple locations, e.g. firewall between NUE1 and NUE2, like in https://sd.suse.com/servicedesk/customer/portal/1/SD-113832 but the timezones diff is an obstacle. Give more people like SUSE QE Tools access to firewalls, even if it's just read-only for investigation?

Acceptance criteria

  • AC1: We can ensure that 2+ persons within EMEA timezones have access to firewalls covering multiple Nbg+Prg locations

Suggestions


Related issues 3 (1 open2 closed)

Related to QA - coordination #121726: [epic] Get management access to o3/osd and other QE related VMsBlockedokurz2022-12-08

Actions
Related to QA - coordination #116623: [epic] Migration of SUSE Nbg based openQA+QA+QAM systems to new security zonesResolvedokurz2022-09-14

Actions
Copied to QA - action #139097: Improve collaboration with Eng-Infra - Firewall management access, potentially also DHCP+DNS - take 2Resolvedokurz

Actions
Actions #1

Updated by okurz about 1 year ago

  • Status changed from New to Workable
Actions #2

Updated by okurz about 1 year ago

  • Target version changed from Ready to future
Actions #3

Updated by okurz about 1 year ago

  • Related to coordination #121726: [epic] Get management access to o3/osd and other QE related VMs added
Actions #4

Updated by okurz about 1 year ago

  • Subject changed from Improve collaboration with Eng-Infra - Firewall access, potentially also DHCP+DNS size:M to Improve collaboration with Eng-Infra - Firewall management access, potentially also DHCP+DNS size:M
  • Description updated (diff)
  • Status changed from Workable to In Progress
  • Assignee set to okurz
  • Target version changed from future to Ready

I presented the proposal in the LSG QE mgmt call within the Product Owner on the bench presentation. Tracking https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 (no update since 2023-03-03). I will discuss with runger in a follow-up next time we meet.

Actions #5

Updated by okurz about 1 year ago

  • Tags changed from infra, process, Eng-Infra, firewall, access, collaboration, admin, investigation to infra, process, Eng-Infra, firewall, access, collaboration, admin, investigation, next-office-day

I also brought up the topic in #eng-testing https://suse.slack.com/archives/C02CANHLANP/p1678783976855909

(Oliver Kurz) @Ralf Unger CC @Matthias Griessmeier et al.: https://progress.opensuse.org/issues/125363 are more ideas how to improve collaboration with the Eng-Infra domain, https://progress.opensuse.org/issues/125450 is the specific proposal to just give us more access to ease their work and ours as well

Actions #6

Updated by okurz about 1 year ago

  • Status changed from In Progress to Workable
Actions #7

Updated by okurz about 1 year ago

  • Description updated (diff)
Actions #8

Updated by okurz 12 months ago

  • Status changed from Workable to In Progress

okurz wrote:

I will discuss with runger in a follow-up next time we meet.

I had the opportunity to meet with runger. runger will discuss with mgmt contacts tomorrow. We can sync on Wednesday+ on the topic again.

In the meantime just today mcaj responded to https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 and offered root ssh access to DHCP servers for FC Basement QE. In collaboration with mcaj on this.

Actions #9

Updated by openqa_review 12 months ago

  • Due date set to 2023-05-02

Setting due date based on mean cycle time of SUSE QE Tools

Actions #10

Updated by okurz 12 months ago

I could confirm that I could now login into walter1.qe.nue2.suse.org as well as walter2. . The dhcp server logs can be accessed with journalctl -u dhcpd.

I added according instructions in https://progress.opensuse.org/projects/qa/wiki/Tools#Onboarding-for-new-joiners

Actions #11

Updated by okurz 12 months ago

  • Status changed from In Progress to Feedback

Had another talk with runger. We can go ahead with the tickets that we had planned and also we can escalate SD specific tickets as needed. mgriessmeier will talk to mflores next week sharing the good example we had with https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 being entrusted with access to walter1+walter2 for DHCP/DNS so we should apply the same approach for other areas e.g. openQA VM management, firewall access, etc.

Actions #12

Updated by okurz 12 months ago

  • Due date changed from 2023-05-02 to 2023-05-05

2023-05-01 is public holiday so bumping due date to end of next week to get feedback from mgriessmeier regarding his talk to mflores.

Actions #13

Updated by okurz 12 months ago

  • Due date changed from 2023-05-05 to 2023-05-12

mgriessmeier will ask mflores about firewall access this week, until next week. Management of VMs should be simpler with the new setup that we expect in prg2. As we hardly have problems with that setup we can wait. I see firewall access as critical. The current approach is suboptimal with little proactive problem management. So as long as we don't have at least read-only access to firewall management I strongly suggest to not further move machines into different network zones and hence also not move machines physically from Maxtorhof.

Actions #14

Updated by okurz 12 months ago

  • Tags changed from infra, process, Eng-Infra, firewall, access, collaboration, admin, investigation, next-office-day to infra, process, Eng-Infra, firewall, access, collaboration, admin, investigation
Actions #15

Updated by okurz 11 months ago

  • Due date changed from 2023-05-12 to 2023-06-02

From https://suse.slack.com/archives/C02CANHLANP/p1683545373786759?thread_ts=1683545236.034219&cid=C02CANHLANP

(Matthias Griessmeier) mflores will look into the possibility of giving you access to the current firewalls as a short term solution. in the longer run the goal is to have clear processes and permissions established
(Oliver Kurz) sounds great. When can I expect a response for the short term solution?

Actions #16

Updated by okurz 11 months ago

mgriessmeier addressed the topic with mflores. Should wait some days for details, let's see.

Actions #17

Updated by livdywan 11 months ago

  • Due date changed from 2023-06-02 to 2023-06-16

Checked with Matthias. Apparently we're still waiting for Moroni or possibly Martin to get back to us.

Actions #18

Updated by okurz 10 months ago

  • Due date changed from 2023-06-16 to 2023-07-07
  • Status changed from Feedback to Blocked
Actions #19

Updated by okurz 10 months ago

  • Status changed from Blocked to Workable
  • Priority changed from Normal to High

Update in SD ticket:

I have a progress update for you.
We made a new VM with hostname qe-debug.suse.de.
People in your list are allow to ssh there as root.
On the machine we are running syslog server and getting logs from NUE1 firewall.
you can see them in the file: /var/log/remote/gw-infra-log.suse.de.log.

Actions #20

Updated by okurz 10 months ago

  • Related to coordination #116623: [epic] Migration of SUSE Nbg based openQA+QA+QAM systems to new security zones added
Actions #21

Updated by okurz 10 months ago

  • Status changed from Workable to In Progress

wrote a message in the ticket

I tried it out now and could not login as root over ssh:
debug1: Offering public key: /home/okurz/.ssh/id_rsa RSA SHA256:sUVQI8nlrphtSJHeeieDdFEhFNgq08+BaRVcUg93n94
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /home/okurz/.ssh/id_ed25519 ED25519 SHA256:L3NVxDngm4Khys7CBQbnCEyZdspQp/Fjs80ZMnSVE5o
I guess you have the configuration for that machine already in salt, maybe https://gitlab.suse.de/OPS-Service/salt/ ? If you reference the rules in there we could crosscheck the credentials.

also asked the team to check
https://suse.slack.com/archives/C02AJ1E568M/p1687331997881979

@here can all of you try ssh root@qe-debug.suse.de and report feedback in https://sd.suse.com/servicedesk/customer/portal/1/SD-123834 please

Actions #22

Updated by okurz 10 months ago

there was a copy-paste error in the AAAA record for qe-debug, I created a fix https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3666 . As workaround I could login using ssh -4 …

Actions #23

Updated by okurz 10 months ago

I could tail the firewall log file and find relevant entries, e.g. grepping for the src MAC address of worker13.oqa.suse.de eventually some messages show up in tail -f /var/log/remote/gw-infra-log.suse.de.log | grep 0c:c4:7a:7a:78:9e:

2023-06-21T08:08:20.401607+00:00 gw-infra-log.suse.de date=2023-06-21 time=10:43:41 devname="nue1-fgfw-01" devid="FG39E6T021900088" eventtime=1687337022090011154 tz="+0200" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="dc" srcip=10.137.10.13 srcport=50850 srcintf="QE-OpenQA" srcintfrole="lan" dstip=10.162.6.237 dstport=234 dstintf="p2p-engcore" dstintfrole="lan" srccountry="Reserved" dstcountry="Reserved" sessionid=135737944 proto=6 action="deny" policyid=0 policytype="security-policy" service="tcp/234" trandisp="noop" duration=30 sentbyte=60 rcvdbyte=0 sentpkt=1 rcvdpkt=0 appcat="unscanned" mastersrcmac="0c:c4:7a:7a:78:9e" srcmac="0c:c4:7a:7a:78:9e" srcserver=0 dstdevtype="Computer" dstosname="Debian" masterdstmac="00:00:5e:00:01:33" dstmac="00:00:5e:00:01:33" dstserver=0
2023-06-21T08:08:20.451569+00:00 gw-infra-log.suse.de date=2023-06-21 time=10:43:42 devname="nue1-fgfw-01" devid="FG39E6T021900088" eventtime=1687337022140015345 tz="+0200" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="dc" srcip=10.137.10.13 srcport=50860 srcintf="QE-OpenQA" srcintfrole="lan" dstip=10.162.6.237 dstport=234 dstintf="p2p-engcore" dstintfrole="lan" srccountry="Reserved" dstcountry="Reserved" sessionid=135737968 proto=6 action="deny" policyid=0 policytype="security-policy" service="tcp/234" trandisp="noop" duration=30 sentbyte=60 rcvdbyte=0 sentpkt=1 rcvdpkt=0 appcat="unscanned" mastersrcmac="0c:c4:7a:7a:78:9e" srcmac="0c:c4:7a:7a:78:9e" srcserver=0 dstdevtype="Computer" dstosname="Debian" masterdstmac="00:00:5e:00:01:33" dstmac="00:00:5e:00:01:33" dstserver=0
Actions #24

Updated by okurz 10 months ago

  • Due date changed from 2023-07-07 to 2023-08-04
  • Status changed from In Progress to Blocked
Actions #25

Updated by okurz 9 months ago

  • Due date deleted (2023-08-04)
  • Status changed from Blocked to Resolved

MR merged

Actions #26

Updated by okurz 5 months ago

  • Copied to action #139097: Improve collaboration with Eng-Infra - Firewall management access, potentially also DHCP+DNS - take 2 added
Actions

Also available in: Atom PDF