openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842023-04-24T12:51:53ZopenSUSE Project Management Tool
Redmine openQA Infrastructure - action #128222 (New): [virtualization] The Xen specific host configuratio...https://progress.opensuse.org/issues/1282222023-04-24T12:51:53Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>With <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Consolidate the installation of openqaw5-xen with SUSE QE Tools maintained machines size:M (Resolved)" href="https://progress.opensuse.org/issues/125534">#125534</a> resolved the host openqaw5-xen.qa.suse.de is covered in salt using <a href="https://gitlab.suse.de/openqa/salt-states-openqa" class="external">https://gitlab.suse.de/openqa/salt-states-openqa</a> as a generic host. To ensure that the Xen specific host configuration can be preserved we should cover Xen specific rules in salt as well.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> A Xen host able to execute openQA Xen based jobs can be configured from <a href="https://gitlab.suse.de/openqa/salt-states-openqa" class="external">https://gitlab.suse.de/openqa/salt-states-openqa</a></li>
<li><strong>AC2:</strong> The configuration is cleanly applied on openqaw5-xen.qa.suse.de</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Collect requirements regarding Xen configuration</li>
<li>Add the necessary configuration statements to <a href="https://gitlab.suse.de/openqa/salt-states-openqa" class="external">https://gitlab.suse.de/openqa/salt-states-openqa</a> or <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa</a> correspondingly</li>
<li>Ensure the config is covered in salt CI tests, e.g. with additional explicit role</li>
<li>Ensure that the config cleanly applies on openqaw5-xen.qa.suse.de</li>
</ul>
QA - action #127907 (Resolved): jenkins package (and others) not upgraded on jenkins.qa.suse.de s...https://progress.opensuse.org/issues/1279072023-04-19T07:23:07Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="http://jenkins.qa.suse.de/manage/" class="external">http://jenkins.qa.suse.de/manage/</a> mentions that there is a new version of jenkins. In the past the jenkins package was automatically updated. <code>zypper dup</code> shows some pending changes on the host even though the "auto-update.service" is enabled. The situation should be cross-checked and fixed.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> packages on jenkins.qa.suse.de including "jenkins" are automatically upgraded</li>
<li><strong>AC2:</strong> The problem source is understood and prevented for the future and other hosts</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>jenkins.qa.suse.de should be salt-controlled from <a href="https://gitlab.suse.de/openqa/salt-states-openqa" class="external">https://gitlab.suse.de/openqa/salt-states-openqa</a> and hence should have automatic upgrades and monitoring. It should be cross-checked why this did not have the expected effect</li>
<li>Ensure that all generic workers are properly upgraded</li>
<li>Add automatic upgrade of all machines from all repos in salt</li>
</ul>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>I came across the current status on jenkins.qa.suse.de as I wanted to review the current state of the automatic submissions of openQA related packages from devel:openQA to openSUSE:Factory as we are planning to apply an equivalent approach for automatic submission to SLE for QAaaS</p>
openQA Infrastructure - action #125765 (Resolved): Make Telegraf errors visible in alert handlinghttps://progress.opensuse.org/issues/1257652023-03-10T10:56:33Zlivdywanliv.dywan@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>In the context of <a class="issue tracker-4 status-3 priority-4 priority-default closed behind-schedule" title="action: [tools][metrics] Calculate cycle + lead times for SUSE QE Tools continuously size:M (Resolved)" href="https://progress.opensuse.org/issues/121582">#121582</a> the deployed InfluxDB input wouldn't seem to be picked up by Grafana but we also saw no issues with deployment or alerts to explain that it was broken.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> The team is aware of errors in Telegraf inputs</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Run <code>sudo telegraf --test --config /etc/telegraf/telegraf.d/slo.conf</code> with the according config filename. By default only one config file will be used</li>
<li>Use logwarn (c.f. openqa logwarn)</li>
<li>Use <a href="https://grafana.com/oss/loki/" class="external">https://grafana.com/oss/loki/</a> (maybe overkill?)</li>
</ul>
openQA Infrastructure - action #125750 (Resolved): In salt-states-openqa support machines requiri...https://progress.opensuse.org/issues/1257502023-03-10T07:20:47Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>openqaw5-xen requires login of root with password over ssh for openQA tests, see <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls#L138" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls#L138</a>, hence we can not directly apply <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/sshd/sshd_config#L44" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/sshd/sshd_config#L44</a></p>
<pre><code>PermitRootLogin without-password
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> openqaw5-xen can be controlled by salt while allowing root-ssh-password login</li>
<li><strong>AC2:</strong> By default all machines in salt still prevent password authentication in salt</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>Optional:</em> We could temporarily change to allow password login over ssh</li>
<li>Find a way to allow individual machines root-ssh-password login</li>
<li><em>Optional:</em> Adapt os-autoinst backend to support ssh key login</li>
<li>Ensure by default machines still apply <code>PermitRootLogin without-password</code></li>
</ul>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li>Add openqaw5-xen back to salt and ensure a high state can be applied while still allowing password login for root on this machine</li>
</ul>
openQA Infrastructure - action #125642 (Resolved): Manage "unified alerting" via salt size:Mhttps://progress.opensuse.org/issues/1256422023-03-08T13:24:13Znicksingernsinger@suse.com
<a name="Summary"></a>
<h2 >Summary<a href="#Summary" class="wiki-anchor">¶</a></h2>
<p>Since the switch to <a href="https://stats.openqa-monitor.qa.suse.de/alerting/list" class="external">unified alerting</a>, alerts are not linked to panels any longer and therefore are not managed by our salt any longer. We should make sure to get them managed somehow.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Alerts are managed by salt</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Ensure alerts are imported/exported from/to JSON (maybe there is a built-in feature?)</li>
<li>Check if there is a similar concept as for dashboards to deploy from files</li>
<li>If it turns out to be too much for one ticket, create follow-up tickets</li>
<li>Consider using YAML (if it is easier to work with)</li>
<li><a href="https://community.grafana.com/t/ngalert-grafana-8-alert-feature-how-to-export-import-alerts-as-yml-json/51677/15" class="external">https://community.grafana.com/t/ngalert-grafana-8-alert-feature-how-to-export-import-alerts-as-yml-json/51677/15</a></li>
<li><a href="https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/" class="external">https://grafana.com/docs/grafana/latest/alerting/set-up/provision-alerting-resources/file-provisioning/</a></li>
</ul>
openQA Infrastructure - action #125534 (Resolved): Consolidate the installation of openqaw5-xen w...https://progress.opensuse.org/issues/1255342023-03-07T15:07:45Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>See #123754. We need to ensure openqaw5-xen is properly maintained. For that within the SUSE QE Tools we expect a Leap 15.4 OS and salt from <a href="https://gitlab.suse.de/openqa/salt-states-openqa" class="external">https://gitlab.suse.de/openqa/salt-states-openqa</a> which adds among other things consolidated administration, user management, backup (at least of salt-covered config), and monitoring</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> openqaw5-xen shows up in monitor.qa.suse.de</li>
<li><strong>AC2:</strong> openqaw5-xen has an updated OS</li>
<li><strong>AC3:</strong> openqaw5-xen basics are administered by SUSE QE Tools</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Conduct a basic backup, e.g. of /boot/ and /etc/ to backup.qa.suse.de</li>
<li>Look up in older tickets how we did migrate from outdated SLE to current Leap and do it</li>
<li>Add to salt following <a href="https://gitlab.suse.de/openqa/salt-states-openqa/" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/</a> instructions, apply high state and monitor and fix related problems</li>
</ul>
openQA Infrastructure - action #125531 (Resolved): salt-pillar C pipeline runs into 1h timeouthttps://progress.opensuse.org/issues/1255312023-03-07T14:57:55Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1441277" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1441277</a> fails with "The script exceeded the maximum execution time set for the job" after 1h.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> salt-pillar CI jobs take significantly less than 2h</li>
<li><strong>AC2:</strong> salt-pillar CI jobs do not commonly fail due to gitlab CI timeout</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Try to just bump the timeout</li>
<li>Look into the history of previous jobs how long it usually takes to apply the salt pillar triggered high state</li>
<li>Find out if there are specific machines that are problematic</li>
</ul>
openQA Infrastructure - action #125141 (Workable): Salt pillars deployment pipeline failed on "tu...https://progress.opensuse.org/issues/1251412023-02-28T11:17:44Zmkittlermarius.kittler@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<pre><code> ID: security-sensor.repo
Function: pkgrepo.managed
Result: False
Comment: Failed to configure repo 'security-sensor.repo': Zypper command failure: Repository 'security-sensor.repo' is invalid.
[security-sensor.repo|https://download.opensuse.org/repositories/security:/sensor/15.4] Valid metadata not found at specified URL
History:
- Signature verification failed for repomd.xml
- Can't provide /repodata/repomd.xml
Please check if the URIs defined for this repository are pointing to a valid repository.
Skipping repository 'security-sensor.repo' because of the above error.
Could not refresh the repositories because of errors.Forcing raw metadata refresh
Retrieving repository 'security-sensor.repo' metadata [..........
Warning: File 'repomd.xml' from repository 'security-sensor.repo' is unsigned.
Note: Signing data enables the recipient to verify that no modifications occurred after the data
were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
and in extreme cases even to a system compromise.
Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
whole repo.
Warning: We can't verify that no one meddled with this file, so it might not be trustworthy
anymore! You should not continue unless you know it's safe.
File 'repomd.xml' from repository 'security-sensor.repo' is unsigned, continue? [yes/no] (no): no
error]
Started: 09:39:50.917365
Duration: 9775.41 ms
Changes:
----------
ID: security-sensor.repo
Function: pkg.latest
Name: velociraptor-client
Result: False
Comment: One or more requisite failed: security_sensor.security-sensor.repo
Started: 09:40:00.699471
Duration: 0.011 ms
Changes:
…
Summary for tumblesle
--------------
Succeeded: 231 (changed=1)
Failed: 2
--------------
Total states run: 233
</code></pre>
<p>(<a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1427053/raw">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/1427053/raw</a>)</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Find out what the host "tumblesle" is -> a VM on qamaster.qa.suse.de (according to <a href="https://racktables.suse.de/index.php?page=object&tab=default&object_id=1300">https://racktables.suse.de/index.php?page=object&tab=default&object_id=1300</a>), the full domain is tumblesle.qa.suse.de</li>
<li>Check whether the problem persists -> no the repo can be refreshed (on tumblesle)</li>
<li>Check whether the error handling (retries) is in accordance with how other repos are configured -> we use <code>pkgrepo.managed: - retry: attempts: 5</code> for our own devel repos, maybe the same would make sense for <code>security:sensor</code> as well; we don't have a retry for all repos configured via <code>pkgrepo.managed</code> so far, though</li>
</ul>
<a name="Remarks"></a>
<h2 >Remarks<a href="#Remarks" class="wiki-anchor">¶</a></h2>
<ul>
<li>Likely not specific to "tumblesle".</li>
<li>Looks like a temporary signing problem of security-sensor.repo (and not like a network issue). <em>DONE</em> So maybe a one-time issue and we don't need to introduce a retry. -> It is reproducible on tumblesle.qa.suse.de with</li>
</ul>
<pre><code>for i in {001..100}; do echo "## $i" && zypper ref --force -r security-sensor.repo; done
</code></pre>
<p>after 23 runs. Directly afterwards it was working to retrieve the file.</p>
<ul>
<li><em>Optional</em> Try to reproduce the above problem in a clean container environment, at best for crosschecking both Leap and Tumbleweed</li>
<li>Based on the above report an issue to zypper on <a href="https://github.com/openSUSE/zypper/">https://github.com/openSUSE/zypper/</a> as zypper claims "File is unsigned" which is apparently not true. It's likely a temporary connection issue. Better retry</li>
<li><em>Optional:</em> Additionally report an issue with the openSUSE infrastructure with a cross-reference</li>
</ul>
openQA Infrastructure - action #106925 (Resolved): [timeboxed:10h][research] What are best practi...https://progress.opensuse.org/issues/1069252022-02-16T15:22:51Zokurzokurz@suse.comopenQA Infrastructure - action #106365 (Resolved): Improve security for OSD worker credentials br...https://progress.opensuse.org/issues/1063652022-02-09T10:25:15Zosukup
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a href="https://progress.opensuse.org/issues/105405" class="external">https://progress.opensuse.org/issues/105405</a> .. changed visibility of salt-pillars-openqa broke <code>deploy</code> stage of CI</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Working salt-states+salt-pillars pipelines in gitlab</li>
<li><strong>AC2:</strong> salt-pillars repo stays non-public</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Try out deploy tokens on OSD to fetch the git repo</li>
</ul>
openQA Project - action #90167 (New): Setup initial salt infrastructure for remote management wit...https://progress.opensuse.org/issues/901672021-03-16T12:23:55Zokurzokurz@suse.com
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> On o3 <code>salt \* test.ping</code> returns all common worker hosts as well as o3 itself</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Ensure salt-minion on all o3 workers</li>
<li>Ensure salt-master on o3</li>
<li>Ensure workers are connected to o3 and salt key is accepted</li>
</ul>
openQA Project - action #90164 (Resolved): Make gitlab.suse.de/openqa/salt-states-openqa publichttps://progress.opensuse.org/issues/901642021-03-16T12:22:20Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>To be able to use the salt recipes for o3 as well as provide it as a solution to manage any openQA infrastructure we decided to make gitlab.suse.de/openqa/salt-states-openqa public, e.g. put it on github within github.com/os-autoinst/</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> gitlab.suse.de/openqa/salt-states-openqa is available on a public, non-personal git repository</li>
<li><strong>AC2:</strong> SUSE QE Tools team member have read-write access to the repository</li>
<li><strong>AC3:</strong> Changes in the repo are still automatically applied within the OSD infrastructure</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Research about best practices to move from internal gitlab repo to external github/gitlab</li>
<li>Copy gitlab.suse.de/openqa/salt-states-openqa to github, e.g. in <a href="https://github.com/os-autoinst" class="external">https://github.com/os-autoinst</a> scope, and create back-mirror into salt-states repo or get rid of it completely</li>
</ul>
openQA Infrastructure - action #89050 (Resolved): OSD deployment blocked since two weeks - salt n...https://progress.opensuse.org/issues/890502021-02-24T09:54:30Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>The deployment of 2021-02-24 is blocked by <a href="https://gitlab.suse.de/openqa/osd-deployment/-/jobs/345595" class="external">https://gitlab.suse.de/openqa/osd-deployment/-/jobs/345595</a> "check all salt nodes are up".</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<p>All other checks of the pre-deploy step are fine but powerqaworker-qam-1 was not reachable.<br>
That can be easily circumvented by excluding the according machines from salt and marking in the according ticket</p>
openQA Infrastructure - action #75274 (Resolved): [osd-admins][alert][learning] Failed systemd se...https://progress.opensuse.org/issues/752742020-10-25T20:57:14Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://openqa.suse.de/tests/4885662" class="external">https://openqa.suse.de/tests/4885662</a> is incomplete due to</p>
<pre><code>backend died: Open vSwitch command 'set_vlan' with arguments 'tap3 1' failed: org.freedesktop.DBus.Error.ServiceUnknown: The name org.opensuse.os_autoinst.switch was not provided by any .service files
</code></pre>
<p>as the service os-autoinst-openvswitch.service aborted after 60s without network on the host due to <a class="issue tracker-4 status-3 priority-6 priority-high2 closed behind-schedule" title="action: OSD partially unresponsive, triggering 500 responses, spotty response visible in monitoring panel... (Resolved)" href="https://progress.opensuse.org/issues/73633">#73633</a></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li>AC1: DONE hpc_ALPHA_openmpi_mpi_supportserver passes</li>
<li>AC2: DONE os-autoinst-openvswitch timeout is configurable</li>
<li>AC3: A higher timeout OS_AUTOINST_OPENVSWITCH_INIT_TIMEOUT is set in salt on all osd workers</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Read <a href="https://github.com/os-autoinst/os-autoinst/pull/1555/files" class="external">https://github.com/os-autoinst/os-autoinst/pull/1555/files</a></li>
<li>Override systemd service definitions setting OS_AUTOINST_OPENVSWITCH_INIT_TIMEOUT higher than 60, e.g. 300</li>
</ul>
openQA Project - coordination #43934 (Blocked): [epic] Manage o3 infrastructure with salt againhttps://progress.opensuse.org/issues/439342018-11-17T14:37:30Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>See <a class="issue tracker-4 status-3 priority-7 priority-highest closed" title="action: o3 workers immediately incompleting all jobs, caching service can not be reached (Resolved)" href="https://progress.opensuse.org/issues/43823#note-1">#43823#note-1</a> . Previously we had a salt-minion on each worker even though no salt recipes were used, at least we used salt for structured remote execution ;)</p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>As salt was there, is the preferred system management solution, and should be extended to have full recipes we should have a salt-minion available as well on all the workers.</p>
<a name="To-be-covered-for-o3-in-system-management-eg-salt-states"></a>
<h2 >To be covered for o3 in system management, e.g. salt states<a href="#To-be-covered-for-o3-in-system-management-eg-salt-states" class="wiki-anchor">¶</a></h2>
<ul>
<li>aarch64 irqbalance workaround <a class="issue tracker-4 status-3 priority-3 priority-lowest closed" title="action: Failed service "irqbalance" on aarch64.o.o (Resolved)" href="https://progress.opensuse.org/issues/53573">#53573</a></li>
<li>hugepages workaround <a class="issue tracker-4 status-3 priority-3 priority-lowest closed" title="action: all jobs on aarch64.o.o incompleted with "Permission denied" on /dev/hugepages, "others" had no r/w (Resolved)" href="https://progress.opensuse.org/issues/53234">#53234</a></li>
<li>ppc kvm permissions <a class="issue tracker-10 status-3 priority-3 priority-lowest closed" title="tickets: openQA ppc64le workers bad kvm setup (Resolved)" href="https://progress.opensuse.org/issues/25170">#25170</a></li>
</ul>