QA

I asked Mixtral about this:

Yes, there are industry standards and best practices that organizations can follow when deciding whether to apply mitigations as early as possible or keep a system in a broken state to ease investigation. Here are some examples:

1. ITIL (Information Technology Infrastructure Library): ITIL is a framework for managing IT services. It includes a process for incident management that recommends restoring normal service operation as quickly as possible while also minimizing the impact of the incident. This means that organizations should apply mitigations as soon as possible, while also conducting a thorough investigation to understand the root cause of the issue.

2.NIST (National Institute of Standards and Technology): NIST provides guidelines for incident response and recovery. Its guidelines recommend that organizations apply mitigations as soon as possible to reduce the impact of an incident, while also conducting a thorough investigation to understand the cause and prevent future incidents.

3. ISO/IEC 27001: This is an international standard for information security management systems (ISMS). It recommends that organizations establish processes for incident management, including the application of mitigations as soon as possible to reduce the impact of an incident.

4. COBIT (Control Objectives for Information and Related Technologies): COBIT is a framework for IT management and IT governance. It includes a process for incident management that recommends restoring normal service operation as quickly as possible while also minimizing the impact of the incident.

By following these industry standards and best practices, organizations can make informed decisions about how to balance the need to apply mitigations as early as possible with the need to conduct a thorough investigation. It's important to note that each situation is unique, and organizations should consider the specific circumstances of each incident when making a decision.```

• Draft an extension to https://progress.opensuse.org/projects/qa/wiki/Tools#Alert-handling

• If users report outages of components of our infrastructure
  • Consider teaming up and assigning individual tasks to focus on
  • Inform affected users about the impact and ETA via Slack channels, ticket updates and mailing list posts
  • Look into mitigations and short-term workarounds such as a hotpatch in production or a revert to an older release
  • Investigate a proper solution with a conservative estimate on the effort involved
  • Set a time limit to ensure either a workaround or a solution is available within 4 hours
  • Join an ad-hoc video call to discuss further steps
  • Keep a record of what was discussed and investigated to allow for a later analysis

livdywan wrote in #note-11:

• Draft an extension to https://progress.opensuse.org/projects/qa/wiki/Tools#Alert-handling

• If users report outages of components of our infrastructure […]

Very good suggestions. I like them. But the early mitigations should be applicable regardless if users report outages. So I would just add the points on the top level, not within the point "If users report …"

• Inform affected users about the impact and ETA via Slack channels, ticket updates and mailing list posts

I suggest to use "chat channels" instead of "Slack channels" to be more generic and applicable for openSUSE targeted users as well.

• Set a time limit to ensure either a workaround or a solution is available within 4 hours

Depending on the severity of the issue 4 hours might be too long or too short. I would not mention a specific time. We could say "Set a time limit to ensure either a workaround or a solution is available within a reasonable time depending on the severity"

• Set a time limit to ensure either a workaround or a solution is available within 4 hours

Depending on the severity of the issue 4 hours might be too long or too short. I would not mention a specific time. We could say "Set a time limit to ensure either a workaround or a solution is available within a reasonable time depending on the severity"

My thought here is to give examples to make it clear that this should be a short timeframe. I phrased it more explicitly now.

Updated draft:

• If users report outages of components of our infrastructure
  • Ensure there is a ticket on the backlog tracking the issue
• For any user-facing outages
  • Consider teaming up and assigning individual tasks to focus on
  • Inform affected users about the impact and ETA via chat channels, ticket updates and mailing list posts
  • Look into mitigations and short-term workarounds such as a hotpatch in production or a revert to an older release
  • Investigate a proper solution with a conservative estimate on the effort involved
  • Set a time limit to ensure either a workaround or a solution is available within a reasonable amount of time (for example 4 hours or end of working day of the person communicating the changes)
  • Join an ad-hoc video call to discuss further steps
  • Keep a record of what was discussed and investigated to allow for a later analysis

• Do a quick industry standards best practice research
• Consider reading https://sre.google/sre-book/table-of-contents/ or ask AI

I also took these into account. These are reflected in my draft. We could potentially make it even more verbose and explicitly discuss severity assessment, conduction of a root cause analysis and regular practice runs. However that would make it less of a checklist and more of a manual. I'm trying to keep it concise here.

While discussing #167335 we realized there's an important step missing here, which I'm adding to my draft now:

• If users report outages of components of our infrastructure
  • Ensure there is a ticket on the backlog tracking the issue
• For any user-facing outages
  • Consider teaming up and assigning individual tasks to focus on
  • Inform affected users about the impact and ETA via chat channels, ticket updates and mailing list posts
  • Look into mitigations and short-term workarounds such as a hotpatch in production or a revert to an older release
  • Investigate a proper solution with a conservative estimate on the effort involved
  • Set a time limit to ensure either a workaround or a solution is available within a reasonable amount of time (for example 4 hours or end of working day of the person communicating the changes)
  • Join an ad-hoc video call to discuss further steps
  • Keep a record of what was discussed and investigated to allow for a later analysis
  • Look into symptoms such as restarting incomplete jobs