QA » openQA Project » openQA Infrastructure

Conducted the lessons learned meeting with the team

What happened¶

Monitoring as well as users reported problems with "502 errors" from openqa.suse.de starting 2024-05-31 07:32Z. The initial problem seems to be that salt inconsistently wrote configuration files which left at least /etc/openqa/database.ini incorrect leading to errors in the logfiles preventing the openQA webUI service to properly restart. This was triggered by a salt-states CI pipeline merged by jlausuch. First we assumed a problem with package updates also involving glibc caused inconsistencies before we could understood that the configuration files were left invalid. jbaier+okurz decided to trigger a reboot which then left the machine unresponsive due to /etc/fstab being completely absent. okurz created an SD ticket and mentioned it also in #help-it-ama. After some time and additional personal escalation and reminders mcaj from IT picked up the SD ticket that okurz created and looked into the issue. mcaj contacted okurz over private Slack conversation. After some hours the complete problems were resolved. The most time was spent understanding the problem why the VM would not boot which in the end turned out the be an absent /etc/fstab. Second most time was spent running filesystem checks which potentially were not even necessary.

Five Whys¶

1. Why did the salt states pipelines end with success when the salt high state was never reported to be successfully applied to the openqa.suse.de salt minion (openqa.suse.de is not mentioned in the list of minions where the state was applied but the pipeline still ended)
   1.1. We do not know yet but this should help us in the future to spot errors quicker in case similar problems return. Maybe the problem is related to how we run salt over ssh from that minion openqa.suse.de and potentially the exit code from salt was never propagated but the command in bash just ended prematurely?
   1.1.1. separate ticket to research about best practices how to apply a high state from a remotely accessible master upstream and investigate this -> #161423
   1.1.2. create ticket to introduce post-deploy monitoring steps like in osd-deployment but in salt-states-openqa -> #161426
   1.1.3. create ticket to create annotations in grafana on the time of osd-deployment as well as salt-states-openqa deployments -> #161429

2. Why did salt leave files inconsistently and not abort more clearly with a fatal error or retry or revert?
   2.1. We usually see "result: false" and the run continues. Is this different? separate ticket to research about best practices how to apply a high state from a remotely accessible master upstream and investigate this -> #161423

3. Why did salt not manage to update three configuration files in the first place?
   3.1. Maybe this is due the secondary, unlikely hypothesis of "filesystem corruption" we might have had. If not we could reconsider how we write those three specific configuration files. okurz thinks at least for /etc/fstab we have two states concerning this file. Maybe it's better to combine those into one

4. Why did we need to wait on an SD ticket to IT and wait for hours for a resolution?
   4.1. We do not have hypervisor access. We already pushed for this multiple times and were denied. We should try again. See also specific proposals in https://progress.opensuse.org/issues/121726 and https://sd.suse.com/servicedesk/customer/portal/1/SD-126446 which were not followed up for multiple months now. Also the handling of the issue was sub-optimal with direct message conversation, not sharing screen, no video conference. So we did not see the boot process ourselves and could not help there. If we would have had better access, at least temporarily, at least screenshots or something, we likely would have made the mental link between "system does not boot" and "salt left /etc/fstab" absent. separate ticket to reach out to IT again to improve collaboration, give us hypervisor access, etc. -> #161324-4 and #132149

5. Why did we accept the multiple hours of waiting for filesystem checks to complete?
   5.1. We remembered the recent filesystem corruption we had in the past related to two VMs accessing the same storage and we trusted IT members to make the right choices so we accepted the waiting time. But apparently the ticket was not seen as that high priority of IT because even the already failed filesystem checks due to OOM were not realized for a longer time leading up to an hour or longer. The IT assignee did not ask us what could explain the missing /etc/fstab . same as for 4 reach to IT to improve collaboration, etc. -> #161324-4 and #132149

All relevant follow-up tasks put in separate, related tickets