action #125450
closedcoordination #125363: [epic] Improve collaboration with Eng-Infra
Improve collaboration with Eng-Infra - Firewall management access, potentially also DHCP+DNS size:M
Added by okurz about 1 year ago. Updated 10 months ago.
0%
Description
Motivation¶
Apparently in many cases @rwawrig can help best with issues spanning over multiple locations, e.g. firewall between NUE1 and NUE2, like in https://sd.suse.com/servicedesk/customer/portal/1/SD-113832 but the timezones diff is an obstacle. Give more people like SUSE QE Tools access to firewalls, even if it's just read-only for investigation?
Acceptance criteria¶
- AC1: We can ensure that 2+ persons within EMEA timezones have access to firewalls covering multiple Nbg+Prg locations
Suggestions¶
- See how in https://sd.suse.com/servicedesk/customer/portal/1/SD-113832 @rwawrig could help but due to the significant timezones difference the reaction time is slow in both directions
- Follow the discussion in https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 regarding DHCP and apply the same solution for firewall if applicable, e.g. create a specific ticket with specific requirements and suggestions
- Optional also try to handle #121726 in the same ticket aka. "just get it done" :)
Updated by okurz about 1 year ago
- Related to coordination #121726: [epic] Get management access to o3/osd and other QE related VMs added
Updated by okurz about 1 year ago
- Subject changed from Improve collaboration with Eng-Infra - Firewall access, potentially also DHCP+DNS size:M to Improve collaboration with Eng-Infra - Firewall management access, potentially also DHCP+DNS size:M
- Description updated (diff)
- Status changed from Workable to In Progress
- Assignee set to okurz
- Target version changed from future to Ready
I presented the proposal in the LSG QE mgmt call within the Product Owner on the bench presentation. Tracking https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 (no update since 2023-03-03). I will discuss with runger in a follow-up next time we meet.
Updated by okurz about 1 year ago
- Tags changed from infra, process, Eng-Infra, firewall, access, collaboration, admin, investigation to infra, process, Eng-Infra, firewall, access, collaboration, admin, investigation, next-office-day
I also brought up the topic in #eng-testing https://suse.slack.com/archives/C02CANHLANP/p1678783976855909
(Oliver Kurz) @Ralf Unger CC @Matthias Griessmeier et al.: https://progress.opensuse.org/issues/125363 are more ideas how to improve collaboration with the Eng-Infra domain, https://progress.opensuse.org/issues/125450 is the specific proposal to just give us more access to ease their work and ours as well
Updated by okurz about 1 year ago
- Status changed from Workable to In Progress
okurz wrote:
I will discuss with runger in a follow-up next time we meet.
I had the opportunity to meet with runger. runger will discuss with mgmt contacts tomorrow. We can sync on Wednesday+ on the topic again.
In the meantime just today mcaj responded to https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 and offered root ssh access to DHCP servers for FC Basement QE. In collaboration with mcaj on this.
Updated by openqa_review about 1 year ago
- Due date set to 2023-05-02
Setting due date based on mean cycle time of SUSE QE Tools
Updated by okurz about 1 year ago
I could confirm that I could now login into walter1.qe.nue2.suse.org as well as walter2. . The dhcp server logs can be accessed with journalctl -u dhcpd.
I added according instructions in https://progress.opensuse.org/projects/qa/wiki/Tools#Onboarding-for-new-joiners
Updated by okurz about 1 year ago
- Status changed from In Progress to Feedback
Had another talk with runger. We can go ahead with the tickets that we had planned and also we can escalate SD specific tickets as needed. mgriessmeier will talk to mflores next week sharing the good example we had with https://sd.suse.com/servicedesk/customer/portal/1/SD-113959 being entrusted with access to walter1+walter2 for DHCP/DNS so we should apply the same approach for other areas e.g. openQA VM management, firewall access, etc.
Updated by okurz about 1 year ago
- Due date changed from 2023-05-02 to 2023-05-05
2023-05-01 is public holiday so bumping due date to end of next week to get feedback from mgriessmeier regarding his talk to mflores.
Updated by okurz about 1 year ago
- Due date changed from 2023-05-05 to 2023-05-12
mgriessmeier will ask mflores about firewall access this week, until next week. Management of VMs should be simpler with the new setup that we expect in prg2. As we hardly have problems with that setup we can wait. I see firewall access as critical. The current approach is suboptimal with little proactive problem management. So as long as we don't have at least read-only access to firewall management I strongly suggest to not further move machines into different network zones and hence also not move machines physically from Maxtorhof.
Updated by okurz about 1 year ago
- Tags changed from infra, process, Eng-Infra, firewall, access, collaboration, admin, investigation, next-office-day to infra, process, Eng-Infra, firewall, access, collaboration, admin, investigation
Updated by okurz about 1 year ago
- Due date changed from 2023-05-12 to 2023-06-02
(Matthias Griessmeier) mflores will look into the possibility of giving you access to the current firewalls as a short term solution. in the longer run the goal is to have clear processes and permissions established
(Oliver Kurz) sounds great. When can I expect a response for the short term solution?
Updated by okurz about 1 year ago
mgriessmeier addressed the topic with mflores. Should wait some days for details, let's see.
Updated by okurz 11 months ago
- Status changed from Blocked to Workable
- Priority changed from Normal to High
Update in SD ticket:
I have a progress update for you.
We made a new VM with hostname qe-debug.suse.de.
People in your list are allow to ssh there as root.
On the machine we are running syslog server and getting logs from NUE1 firewall.
you can see them in the file: /var/log/remote/gw-infra-log.suse.de.log.
Updated by okurz 11 months ago
- Related to coordination #116623: [epic] Migration of SUSE Nbg based openQA+QA+QAM systems to new security zones added
Updated by okurz 11 months ago
- Status changed from Workable to In Progress
wrote a message in the ticket
I tried it out now and could not login as root over ssh:
debug1: Offering public key: /home/okurz/.ssh/id_rsa RSA SHA256:sUVQI8nlrphtSJHeeieDdFEhFNgq08+BaRVcUg93n94
debug1: Authentications that can continue: publickey,password
debug1: Offering public key: /home/okurz/.ssh/id_ed25519 ED25519 SHA256:L3NVxDngm4Khys7CBQbnCEyZdspQp/Fjs80ZMnSVE5o
I guess you have the configuration for that machine already in salt, maybe https://gitlab.suse.de/OPS-Service/salt/ ? If you reference the rules in there we could crosscheck the credentials.
also asked the team to check
https://suse.slack.com/archives/C02AJ1E568M/p1687331997881979
@here can all of you try
ssh root@qe-debug.suse.deand report feedback in https://sd.suse.com/servicedesk/customer/portal/1/SD-123834 please
Updated by okurz 11 months ago
there was a copy-paste error in the AAAA record for qe-debug, I created a fix https://gitlab.suse.de/OPS-Service/salt/-/merge_requests/3666 . As workaround I could login using ssh -4 …
Updated by okurz 11 months ago
I could tail the firewall log file and find relevant entries, e.g. grepping for the src MAC address of worker13.oqa.suse.de eventually some messages show up in tail -f /var/log/remote/gw-infra-log.suse.de.log | grep 0c:c4:7a:7a:78:9e:
2023-06-21T08:08:20.401607+00:00 gw-infra-log.suse.de date=2023-06-21 time=10:43:41 devname="nue1-fgfw-01" devid="FG39E6T021900088" eventtime=1687337022090011154 tz="+0200" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="dc" srcip=10.137.10.13 srcport=50850 srcintf="QE-OpenQA" srcintfrole="lan" dstip=10.162.6.237 dstport=234 dstintf="p2p-engcore" dstintfrole="lan" srccountry="Reserved" dstcountry="Reserved" sessionid=135737944 proto=6 action="deny" policyid=0 policytype="security-policy" service="tcp/234" trandisp="noop" duration=30 sentbyte=60 rcvdbyte=0 sentpkt=1 rcvdpkt=0 appcat="unscanned" mastersrcmac="0c:c4:7a:7a:78:9e" srcmac="0c:c4:7a:7a:78:9e" srcserver=0 dstdevtype="Computer" dstosname="Debian" masterdstmac="00:00:5e:00:01:33" dstmac="00:00:5e:00:01:33" dstserver=0
2023-06-21T08:08:20.451569+00:00 gw-infra-log.suse.de date=2023-06-21 time=10:43:42 devname="nue1-fgfw-01" devid="FG39E6T021900088" eventtime=1687337022140015345 tz="+0200" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="dc" srcip=10.137.10.13 srcport=50860 srcintf="QE-OpenQA" srcintfrole="lan" dstip=10.162.6.237 dstport=234 dstintf="p2p-engcore" dstintfrole="lan" srccountry="Reserved" dstcountry="Reserved" sessionid=135737968 proto=6 action="deny" policyid=0 policytype="security-policy" service="tcp/234" trandisp="noop" duration=30 sentbyte=60 rcvdbyte=0 sentpkt=1 rcvdpkt=0 appcat="unscanned" mastersrcmac="0c:c4:7a:7a:78:9e" srcmac="0c:c4:7a:7a:78:9e" srcserver=0 dstdevtype="Computer" dstosname="Debian" masterdstmac="00:00:5e:00:01:33" dstmac="00:00:5e:00:01:33" dstserver=0
Updated by okurz 11 months ago
- Due date changed from 2023-07-07 to 2023-08-04
- Status changed from In Progress to Blocked
I explained the setup in https://wiki.suse.net/index.php/OpenQA#Firewall_between_different_SUSE_network_zones . Waiting for resolution of
Updated by okurz 7 months ago
- Copied to action #139097: Improve collaboration with Eng-Infra - Firewall management access, potentially also DHCP+DNS - take 2 added