Project

General

Profile

Actions
Copy link

action #162377

closed

coordination #161414: [epic] Improved salt based infrastructure management

incomplete config files on OSD due to salt - Prevent conflicting state applications on OSD "fstab" size:S

Added by okurz 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
okurz
Category:
Feature requests
Target version:
openQA Project - Ready
Start date:
2024-06-03
Due date:
% Done:

0%

Estimated time:
Tags:
salt, ssh, infra, lesson learned

Description

Motivation

See #161324 . Why did salt not manage to update three configuration files in the first place? Maybe this is due the secondary, unlikely hypothesis of "filesystem corruption" we might have had. If not we could reconsider how we write those three specific configuration files which are /etc/openqa/database.ini and /etc/openqa/openqa.ini . okurz thinks at least for /etc/fstab we have two states concerning this file. Maybe it's better to combine those into one

Acceptance criteria

  • AC1: salt state deploy pipelines do not mention recurring updates on "fstab" in no-op calls

Suggestions

  • Take a look how in https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2732256#L380 it is mentioned that a line is removed in "fstab" even though there are no related changes recently in our salt states. This change is due to fstab being deployed as a complete file by salt and then updated in another state. Probably it's better to combine those into one
  • Avoid mount.mounted and use /etc/fstab instead to avoid the issue

Related issues 1 (0 open — 1 closed)

Copied from openQA Infrastructure - action #161423: [timeboxed:10h] Incomplete config files on OSD due to salt - Improve salt state application from remotely accessible salt master size:SResolvedokurz2024-06-03

Actions
Actions
Copy link
 #1

Updated by okurz 5 months ago

  • Copied from action #161423: [timeboxed:10h] Incomplete config files on OSD due to salt - Improve salt state application from remotely accessible salt master size:S added
Actions
Copy link
 #2

Updated by okurz 5 months ago

  • Due date set to 2024-07-01
  • Status changed from New to Feedback
Actions
Copy link
 #3

Updated by okurz 5 months ago

merged, monitoring.

Actions
Copy link
 #4

Updated by okurz 5 months ago

Trying to schedule a reboot with
https://suse.slack.com/archives/C029APBKLGK/p1718784577379639?thread_ts=1718518504.603039&cid=C029APBKLGK

Hi, can someone with hypervisor access for OSD (openqa.suse.de](http://openqa.suse.de) tell us when today to schedule a reboot after the recent problems while you monitor the boot process?

Actions
Copy link
 #5

Updated by okurz 5 months ago

The file /etc/fstab was fine but we had problems in the reboot due to #162365

Actions
Copy link
 #6

Updated by livdywan 5 months ago

  • Subject changed from incomplete config files on OSD due to salt - Prevent conflicting state applications on OSD "fstab" to incomplete config files on OSD due to salt - Prevent conflicting state applications on OSD "fstab" size:S
  • Description updated (diff)
Actions
Copy link
 #7

Updated by okurz 5 months ago

  • Due date deleted (2024-07-01)
  • Status changed from Feedback to Resolved

Due to #161423 I could not find any production deploy job mentioning the recent output of state application on OSD from gitlab CI pipelines so I ran manually

salt --no-color 'openqa.suse.de' --state-output=changes state.apply | grep -v 'Result.*Clean'

and I did not see any back-and-forth changes so AC1 verified

Actions
Copy link

Also available in: Atom PDF