QA » openQA Project » openQA Infrastructure

If we don't do this soon everybody will loose memory what happened :) I suggest to use tomorrow's extensions meeting slot 0900Z to cover this. Assigning to @cdywan as discussed in daily.

I sent out invitation emails and a message in the team Slack for tomorrow 10 CET

Setting due date based on mean cycle time of SUSE QE Tools

We had the meeting 2022-02-17 and conducted a "Five Whys" analysis. We collected results on https://etherpad.opensuse.org/p/suse_qe_tools and they are as follows:

1. Why did it take so long for infra to perform a firmware upgrade?

   • Because they did not take the issue serious enough mdoucha reported ridiculous data loss marius ran iperf and reported high bandwidth nick pointed to diff between up and download we discussed performance data, broken mac address table
   • -> Make sure when we create EngInfra tickets that the severity and impact is obvious => DONE: https://progress.opensuse.org/projects/qa/wiki/Tools/diff?utf8=%E2%9C%93&version=26&version_from=25&commit=View+differences

2. Why can EngInfra state that they can not help with the issue when it's about a rack in SRV2 where members of SUSE QE Tools don't have access to without EngInfra?

   • Who else could have access? Obviously we assumed we don't
   • -> Crosscheck with relevant managers, e.g. department lead, who can drive such issues and who has access
   • -> How should we be able to work on such issues if we don't have access?
   • -> Do we need more people with access? nsinger had access (https://progress.opensuse.org/issues/102882#note-19)
   • -> Why did EngInfra not have credentials for the switches? https://racktables.nue.suse.com/index.php?page=object&tab=default&object_id=992 states that "qa-infra@suse.de" is the contact person. We can't find that mailing list if it is one? Who is that? Or is there anyone on that mailing list?

3. Why was there no follow-up in https://sd.suse.com/servicedesk/customer/portal/1/SD-67703 between 2021-11-24 11:18 and 2021-12-01 17:13 (that's 7 days from Wednesday to next Wednesday)

   • Because we tried to work on this on our own. nsinger gained access but of course due to Covid-Pandemic and remote work physical access was necessary to be planned and did not happen immediately

4. Why did nothing happen in the progress ticket for 2 weeks after 2021-12-01?

   • There was mostly information flowing in the EngInfra ticket. But for a "High" ticket in our backlog being assigned to someone and actively being worked on there should be more frequent updates
   • -> Ensure to have more frequent updates in tickets that are being worked on in our backlog. We need to have a daily sync, we are trying meanwhile with the "moderation duty". The only alternative is to have a mandatory daily meeting

5. Why did people from EngInfra repeatedly only "plug cables" instead of looking into the clear problem hypothesis we made repeatedly about "unintended traffic" and "broadcast traffic"?

   • Maybe there is the according network related technical competence missing within EngInfra. It seems like we made the hypothesis pretty clear
   • -> Raise that to department directors. Do we need to learn about maintaining Cisco switches?
   • -> Move more to other datacenters or the cloud? => We have according tickets already on our backlog. This just gives us additional motivation to work on those

I will followup with the remaining tasks, e.g. coordinating with managers regarding responsibility

EDIT: Asked mgriessmeier about the open points

Meeting scheduled with mgriessmeier tomorrow

I pinged @channel in https://suse.slack.com/archives/C02AJ1E568M/p1645197538775129 with a replication of the last message. We have all necessary points covered and either already resolved or covered in specific tickets.

I added the remaining information regarding server rooms and network infrastructure on https://confluence.suse.com/pages/diffpagesbyversion.action?pageId=192544831&selectedPageVersions=82&selectedPageVersions=81 . Now we are done here.