openSUSE Project Management Tool: Issues
https://progress.opensuse.org/
https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?1582917784
2024-01-22T09:39:29Z
openSUSE Project Management Tool
Redmine
openQA Infrastructure - action #154018 (Resolved): [alert] Failed systemd services alert: backup-...
https://progress.opensuse.org/issues/154018
2024-01-22T09:39:29Z
tinita
tina.mueller+trick-redmine@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://stats.openqa-monitor.qa.suse.de/alerting/grafana/Uk02cifVkz/view?orgId=1" class="external">https://stats.openqa-monitor.qa.suse.de/alerting/grafana/Uk02cifVkz/view?orgId=1</a><br>
Date: Sun, 21 Jan 2024 03:56:36 +0100</p>
<pre><code>1 firing alert instance
[IMAGE]
GROUPED BY
1 firing instances
Firing [stats.openqa-monitor.qa.suse.de]
Failed systemd services alert (except openqa.suse.de)
View alert [stats.openqa-monitor.qa.suse.de]
Values
B0=1
Labels
alertname
Failed systemd services alert (except openqa.suse.de)
grafana_folder
Salt
rule_uid
Uk02cifVkz
Annotations
message
</code></pre>
<blockquote>
<p>Check failed systemd services on hosts with <code>systemctl --failed</code>. Hint: Go to parent dashboard <a href="https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services" class="external">https://stats.openqa-monitor.qa.suse.de/d/KToPYLEWz/failed-systemd-services</a> to see a list of affected hosts.</p>
</blockquote>
openQA Infrastructure - action #123082 (Resolved): backup of o3 to storage.qa.suse.de was not con...
https://progress.opensuse.org/issues/123082
2023-01-13T10:39:30Z
okurz
okurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>On storage.qa.suse.de <code>ls -ltra /storage/rsnapshot/*/openqa.opensuse.org/</code> shows:</p>
<pre><code>/storage/rsnapshot/alpha.3/openqa.opensuse.org/:
total 0
drwxr-xr-x 1 root root 6 Nov 19 2021 root
drwxr-xr-x 1 root root 8 Nov 22 2021 .
drwxr-xr-x 1 root root 38 Nov 22 2021 ..
/storage/rsnapshot/_delete.12732/openqa.opensuse.org/:
total 0
drwxr-xr-x 1 root root 6 Dec 30 2021 root
drwxr-xr-x 1 root root 8 Dec 30 2021 .
drwxr-xr-x 1 root root 66 Dec 31 2021 ..
/storage/rsnapshot/beta.2/openqa.opensuse.org/:
total 0
drwxr-xr-x 1 root root 0 Aug 25 00:00 .
drwxr-xr-x 1 root root 66 Aug 25 03:10 ..
/storage/rsnapshot/beta.1/openqa.opensuse.org/:
total 0
drwxr-xr-x 1 root root 0 Sep 22 00:00 .
drwxr-xr-x 1 root root 66 Sep 22 03:38 ..
/storage/rsnapshot/beta.0/openqa.opensuse.org/:
total 0
drwxr-xr-x 1 root root 0 Oct 24 00:00 .
drwxr-xr-x 1 root root 66 Oct 24 03:14 ..
/storage/rsnapshot/alpha.2/openqa.opensuse.org/:
total 0
drwxr-xr-x 1 root root 0 Nov 28 00:00 .
drwxr-xr-x 1 root root 66 Nov 28 03:51 ..
/storage/rsnapshot/alpha.1/openqa.opensuse.org/:
total 0
drwxr-xr-x 1 root root 0 Dec 1 00:00 .
drwxr-xr-x 1 root root 66 Dec 1 03:47 ..
/storage/rsnapshot/alpha.0/openqa.opensuse.org/:
total 0
drwxr-xr-x 1 root root 0 Jan 12 00:00 .
drwxr-xr-x 1 root root 66 Jan 12 05:55 ..
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> We are alerted if the backup can not be conducted</li>
<li><strong>AC2:</strong> o3 is backed up again</li>
</ul>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li>Add storage.qa.suse.de back to salt</li>
</ul>
openQA Infrastructure - action #105013 (New): backup o3 worker config files
https://progress.opensuse.org/issues/105013
2022-01-18T09:53:48Z
okurz
okurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>We have a backup of o3 but no automatic backup of workers.ini of o3 workers.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> workers.ini from all production o3 machines maintained by us is backed up automatically</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<p>Use config from <a href="https://progress.opensuse.org/projects/openqav3/wiki/#openQA-infrastructure-needs-o3-osd" class="external">https://progress.opensuse.org/projects/openqav3/wiki/#openQA-infrastructure-needs-o3-osd</a> so that root from o3 can access all workers automatically. Then extend backup config, e.g. <a href="https://gitlab.suse.de/qa-sle/backup-server-salt/-/blob/master/rsnapshot/rsnapshot.conf#L27" class="external">https://gitlab.suse.de/qa-sle/backup-server-salt/-/blob/master/rsnapshot/rsnapshot.conf#L27</a> , to get workers.ini from all workers automatically. Alternative: Make all workers accessible over salt and get it over that.</p>