Project

General

Profile

Actions
Copy link

coordination #161414

open

[epic] Improved salt based infrastructure management

Added by okurz 11 months ago. Updated about 10 hours ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Feature requests
Target version:
QA (public) - future
Start date:
2020-08-07
Due date:
2025-04-24 (5 days late)
% Done:

67%

Estimated time:
(Total: 0.00 h)
Tags:
salt, infra

Subtasks 58 (19 open — 39 closed)

action #69718: harmonize timezone used for our machines size:MResolveddheidler2020-08-07

Actions
action #94492: Configure retention/downsampling policy for monitoring data stored within InfluxDB size:MResolvedmkittler2021-06-22

Actions
action #103380: Configure retention/downsampling policy for specific monitoring data stored within InfluxDBBlockedokurz2021-12-01

Actions
action #161423: [timeboxed:10h] Incomplete config files on OSD due to salt - Improve salt state application from remotely accessible salt master size:SResolvedokurz2024-06-03

Actions
action #161426: incomplete config files on OSD due to salt - introduce post-deploy monitoring steps like in osd-deployment but in salt-states-openqaNew2024-06-03

Actions
action #161429: incomplete config files on OSD due to salt - create annotations in grafana on the time of the osd deployment as well as salt-states-openqa deploymentsNew2024-06-03

Actions
action #162377: incomplete config files on OSD due to salt - Prevent conflicting state applications on OSD "fstab" size:SResolvedokurz2024-06-03

Actions
action #163790: OSD openqa.ini is corrupted, invalid characters size:MResolvedokurz2024-07-10

Actions
action #167051: https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/3109145 failed due to telegraf errors on monitor.qa.suse.de size:SResolvednicksinger2024-09-19

Actions
action #167719: No new data in monitor.qe.nue2.suse.org due to influxdb failing to write with ""error opening new segment file for wal (1): write /var/lib/influxdb/….wal: no space left on device"Resolvedokurz2024-10-02

Actions
action #167722: Efficient use of monitoring data within influxdb on monitor.qe.nue2.suse.org size:MWorkable2024-10-02

Actions
action #167728: grafana dashboard for monitor.qe.nue2.suse.org size:SResolvedgpathak2024-10-02

Actions
action #168145: implement telegraf health check and adjust according pipelinesNew

Actions
action #168148: hackweek idea: use loki to monitor our log files and explore alerting possibilites based on these size:SResolvednicksinger

Actions
action #170077: Put more storage into qamaster "to make our lives easier in general" size:MResolvedokurz2024-11-19

Actions
action #173344: Extend iPXE in qe/oqa.*.suse.org to also display on local console size:SResolvedgpathak

Actions
action #173347: Ensure we have a current backup of qamaster VMs, VM config, jenkins data, data from backup-vm itself, etc. size:SResolvedgpathak

Actions
action #173350: Migrate VMs from qamaster to modern hypervisor solutionNew2024-11-29

Actions
action #173353: physically label slots 10+11 on qamaster size:SResolvedrobert.richardson2024-11-29

Actions
action #173674: qamaster-independent backup size:SResolveddheidler2024-12-03

Actions
action #175407: salt state for machine monitor.qe.nue2.suse.org was broken for almost 2 months, nothing was alerting us size:SResolvedokurz

Actions
action #175629: diesel+petrol (possibly all ppc64le OPAL machines) often run into salt error "Not connected" or "No response" due to wireguard services failing to start on boot size:SResolvednicksinger2025-01-16

Actions
action #175686: OSD webUI ended up with "502 Bad Gateway" from nginx on 2025-01-17, needed manual restart of openqa-webuiResolvedokurz2025-01-17

Actions
action #175689: monitor.qe.nue2.suse.org "502 Bad Gateway" from nginx on 2025-01-17, missing grafana server files?Resolvednicksinger2025-01-17

Actions
action #175707: OSD backups missing since 2024-11 on backup-vm.qe.nue2.suse.org size:SResolveddheidler2025-01-17

Actions
action #175710: OSD openqa.ini is corrupted, invalid characters, again 2025-01-17Resolvedokurz2024-07-10

Actions
action #175740: [alert] deploy pipeline for salt-states-openqa failed, multiple host run into salt error "Not connected" or "No response"Resolvedokurz2025-01-16

Actions
action #175791: [alert] storage: partitions usage (%) alert size:SResolvedgpathak

Actions
action #175848: Validate sls files in salt-{states,pillars}-openqa with a best effort approachNew2025-01-20

Actions
action #175851: Prevent re-evaluation of "stop_and_disable_all_not_configured_workers" state on every run size:SResolvedjbaier_cz2025-01-20

Actions
action #176013: [alert] web UI: Too many Minion job failures alert size:SResolvedybonatakis2025-01-23

Actions
openQA Project (public) - action #176121: salt-states-openqa pipeline deploy fails on master, SaltReqTimeoutError: Message timed outResolvedokurz2025-01-24

Actions
action #176124: OSD influxdb minion route seemingly returns only a very small number of failed minion jobs, not allResolvedtinita

Actions
action #176175: [alert] Grafana failed to start due to corrupted config fileResolvedokurz2025-01-26

Actions
action #176250: file corruption in salt controlled config files size:MResolvedokurz

Actions
action #176325: sporadic: zypper related stack traces in salt pipelineNew

Actions
action #176949: [timeboxed:10h][research] understand/prevent sporadic zypper related stack traces in salt size:SWorkable

Actions
action #177066: Prevent _openqa-worker to install random packages size:SResolvedrobert.richardson2025-02-12

Actions
action #177318: 2 bare-metal machines are offline on OSDResolvedmkittler2025-02-17

Actions
action #177366: osd deployment "test.ping" check runs into gitlab CI timeoutResolvedokurz

Actions
action #177513: Proper "project" name in op-prg2New

Actions
action #177516: Proper "namespaces" in op-prg2New2025-02-19

Actions
action #177519: Ensure that there is a namepace in op-prg2 usable for non-cc-relevant resourcesNew2025-02-19

Actions
action #177522: Ensure that op-prg2 VM "obs-cpan-mirror" does not have access to CC dataNew2025-02-19

Actions
action #177748: [alert] web UI: Too many Minion job failures alert Salt liA25iB4kRejected2025-01-23

Actions
action #177766: Consider storage policy for storage.qe.prg2.suse.org size:SResolvedgpathak2025-02-24

Actions
action #178015: [false negative] Many failed systemd services but no alert has fired size:SResolvednicksinger2025-02-272025-03-18

Actions
openQA Project (public) - action #178078: salt pipeline deployment fails on master, SaltReqTimeoutError: Message timed out for petrol size:SResolvedmkittler2025-01-24

Actions
action #178576: Workers unresponsive in salt pipelines including openqa-piworker, sapworker1 and monitor size:SResolvednicksinger2025-03-07

Actions
action #179302: Better monitoring for correct MTU size limitsNew2025-03-07

Actions
action #179317: [osd] openQA does not pick up most scheduled jobs, sluggish start, multiple failures on 2025-03-21Resolvedokurz2025-03-21

Actions
action #180122: Run openqa-piworker as part of our infrastructure while still being CC compliantNew

Actions
action #180194: Workers diesel and petrol are down which blocks the salt deployment pipeline size:SResolvednicksinger2025-04-082025-04-24

Actions
action #180209: Emails not sent to osd-admins for all salt-controlled hosts size:SResolveddheidler2025-04-08

Actions
action #181151: Follow-up steps regarding backup.qa.suse.de (and backup.qe.prg2.suse.org)Newokurz

Actions
action #181250: salt-master on separate VMNew

Actions
action #181256: Easier alert handling by opting out of backup_check for individual hostsNew

Actions
action #181580: jenkins doesn't notify us on broken builds/runs - should there be e-mails or such? size:SWorkable2025-04-29

Actions
Actions
Copy link
 #1

Updated by okurz 11 months ago

  • Subtask #161423 added
Actions
Copy link
 #2

Updated by okurz 11 months ago

  • Subtask #161426 added
Actions
Copy link
 #3

Updated by okurz 11 months ago

  • Subtask #161429 added
Actions
Copy link
 #4

Updated by okurz 11 months ago

  • Subtask #162377 added
Actions
Copy link
 #5

Updated by okurz 7 months ago

  • Subtask #167719 added
Actions
Copy link
 #6

Updated by okurz 7 months ago

  • Subtask #167722 added
Actions
Copy link
 #7

Updated by okurz 7 months ago

  • Subtask #167728 added
Actions
Copy link
 #8

Updated by okurz 7 months ago

  • Subtask #103380 added
Actions
Copy link
 #9

Updated by okurz 7 months ago

  • Subtask #94492 added
Actions
Copy link
 #10

Updated by okurz 7 months ago

  • Subtask #167051 added
Actions
Copy link
 #11

Updated by nicksinger 7 months ago

  • Subtask #168145 added
Actions
Copy link
 #12

Updated by nicksinger 7 months ago

  • Subtask #168148 added
Actions
Copy link
 #13

Updated by okurz 5 months ago

  • Subtask #170077 added
Actions
Copy link
 #14

Updated by okurz 5 months ago

  • Subtask #173344 added
Actions
Copy link
 #15

Updated by okurz 5 months ago

  • Subtask #173347 added
Actions
Copy link
 #16

Updated by okurz 5 months ago

  • Subtask #173350 added
Actions
Copy link
 #17

Updated by okurz 5 months ago

  • Subtask #173353 added
Actions
Copy link
 #18

Updated by okurz 5 months ago

  • Subtask #173674 added
Actions
Copy link
 #19

Updated by okurz 3 months ago

  • Subtask #175407 added
Actions
Copy link
 #20

Updated by okurz 3 months ago

  • Subtask #175629 added
Actions
Copy link
 #21

Updated by okurz 3 months ago

  • Subtask #175686 added
Actions
Copy link
 #22

Updated by okurz 3 months ago

  • Subtask #175689 added
Actions
Copy link
 #23

Updated by okurz 3 months ago

  • Subtask #175695 added
Actions
Copy link
 #24

Updated by okurz 3 months ago

  • Subtask #163790 added
Actions
Copy link
 #25

Updated by okurz 3 months ago

  • Subtask #175707 added
Actions
Copy link
 #26

Updated by okurz 3 months ago

  • Subtask #175710 added
Actions
Copy link
 #27

Updated by jbaier_cz 3 months ago

  • Subtask #175740 added
Actions
Copy link
 #28

Updated by okurz 3 months ago

  • Subtask #175791 added
Actions
Copy link
 #29

Updated by okurz 3 months ago

  • Subtask #175851 added
Actions
Copy link
 #30

Updated by okurz 3 months ago

  • Subtask #175848 added
Actions
Copy link
 #31

Updated by okurz 3 months ago

  • Subtask #69718 added
Actions
Copy link
 #32

Updated by okurz 3 months ago

  • Subtask #176013 added
Actions
Copy link
 #33

Updated by gpuliti 3 months ago

  • Subtask deleted (#176013)
Actions
Copy link
 #34

Updated by gpuliti 3 months ago

  • Subtask #176013 added
Actions
Copy link
 #35

Updated by okurz 3 months ago

  • Subtask #176124 added
Actions
Copy link
 #36

Updated by okurz 3 months ago

  • Subtask #176121 added
Actions
Copy link
 #37

Updated by okurz 3 months ago

  • Subtask #176175 added
Actions
Copy link
 #38

Updated by okurz 3 months ago

  • Subtask #176250 added
Actions
Copy link
 #39

Updated by okurz 3 months ago

  • Subtask deleted (#175695)
Actions
Copy link
 #40

Updated by okurz 3 months ago

  • Subtask #177066 added
Actions
Copy link
 #41

Updated by okurz 2 months ago

  • Subtask #177276 added
Actions
Copy link
 #42

Updated by okurz 2 months ago

  • Subtask #177366 added
Actions
Copy link
 #43

Updated by okurz 2 months ago

  • Subtask #177513 added
Actions
Copy link
 #44

Updated by okurz 2 months ago

  • Subtask #177516 added
Actions
Copy link
 #45

Updated by okurz 2 months ago

  • Subtask #177519 added
Actions
Copy link
 #46

Updated by okurz 2 months ago

  • Subtask #177522 added
Actions
Copy link
 #47

Updated by okurz 2 months ago

  • Subtask #176421 added
Actions
Copy link
 #48

Updated by gpathak 2 months ago

  • Subtask #177748 added
Actions
Copy link
 #49

Updated by gpathak 2 months ago

  • Subtask #177766 added
Actions
Copy link
 #50

Updated by okurz 2 months ago

  • Subtask #176325 added
Actions
Copy link
 #51

Updated by okurz 2 months ago

  • Subtask #176949 added
Actions
Copy link
 #52

Updated by okurz 2 months ago

  • Subtask #178015 added
Actions
Copy link
 #53

Updated by okurz 2 months ago

  • Subtask #178078 added
Actions
Copy link
 #54

Updated by okurz 2 months ago

  • Subtask #177318 added
Actions
Copy link
 #55

Updated by gpathak about 2 months ago

  • Subtask #178567 added
Actions
Copy link
 #56

Updated by okurz about 1 month ago

  • Subtask #178576 added
Actions
Copy link
 #57

Updated by okurz about 1 month ago

  • Subtask #179302 added
Actions
Copy link
 #58

Updated by okurz about 1 month ago

  • Subtask #179317 added
Actions
Copy link
 #59

Updated by okurz about 1 month ago

  • Subtask deleted (#177276)
Actions
Copy link
 #60

Updated by okurz about 1 month ago

  • Subtask deleted (#176421)
Actions
Copy link
 #61

Updated by nicksinger 23 days ago

  • Subtask #180122 added
Actions
Copy link
 #62

Updated by okurz 22 days ago

  • Subtask #180194 added
Actions
Copy link
 #63

Updated by livdywan 12 days ago

  • Subtask #181151 added
Actions
Copy link
 #64

Updated by okurz 8 days ago

  • Subtask #181250 added
Actions
Copy link
 #65

Updated by livdywan 7 days ago

  • Subtask #181256 added
Actions
Copy link
 #66

Updated by okurz about 10 hours ago

  • Subtask #180209 added
Actions
Copy link
 #67

Updated by okurz about 10 hours ago

  • Subtask #181580 added
Actions
Copy link

Also available in: Atom PDF