openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-18T07:37:57ZopenSUSE Project Management Tool
Redmine openQA Tests - action #157414 (New): Network broken with multimachine on openqaworker-arm22https://progress.opensuse.org/issues/1574142024-03-18T07:37:57Zggardet_armguillaume.gardet@arm.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario microos-Tumbleweed-DVD-aarch64-remote_ssh_controller@aarch64 fails in<br>
<a href="https://openqa.opensuse.org/tests/4018286/modules/await_install/steps/5" class="external">await_install</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Maintainer: jrivera Install remote server (parallel job) with ssh.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.opensuse.org/tests/4018286" class="external">20240314</a> (current job)</p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.opensuse.org/tests/4004882" class="external">20240310</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.opensuse.org/tests/latest?arch=aarch64&distri=microos&flavor=DVD&machine=aarch64&test=remote_ssh_controller&version=Tumbleweed" class="external">latest</a></p>
openQA Project - action #157339 (New): os-autoinst t/14-isotovideo.t is again taking too long (>2...https://progress.opensuse.org/issues/1573392024-03-15T10:58:03Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>While reviewing <a href="https://github.com/os-autoinst/os-autoinst/pull/2470" class="external">https://github.com/os-autoinst/os-autoinst/pull/2470</a> and trying t/14-isotovideo.t out I found that t/14-isotovideo.t runs into timeout of 20s so taking too long. We had cases like that in the past and should try to identify the bottleneck of that test again and improve the runtime.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> t/14-isotovideo.t consistently passes well below the timeout of 20s again locally</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Run <code>prove -I. -v t/14-isotovideo.t</code> or other means to profile that test and improve the code parts taking the most time</li>
</ul>
openQA Project - action #157270 (New): [spike solution][timeboxed:20h] Run os-autoinst-distri-ope...https://progress.opensuse.org/issues/1572702024-03-14T16:36:56Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>With <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: [spike][timeboxed:10h] Run os-autoinst-distri-example directly from git and ensure candidate need... (Resolved)" href="https://progress.opensuse.org/issues/154783">#154783</a> we have proper git caching so we can run git based tests efficiently on our workers now. Now we should go the next step and migrate one "production" test distribution to use only git and not hold anything provided by admins on o3 in o3:/var/lib/openqa/share/tests for this test distribution. To try that out we can develop a spike solution.</p>
<a name="Goals"></a>
<h2 >Goals<a href="#Goals" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>G1:</strong> /var/lib/openqa/share/tests/ do not exist</li>
<li><strong>G2:</strong> openqa-in-openqa tests still pass consistently</li>
<li><strong>G3:</strong> openqa-in-openqa test details, needle candidates and source code views still show content as expected</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Change test definitions in <a href="https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml" class="external">https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml</a> in your branch to use <a href="https://github.com/os-autoinst/os-autoinst-distri-openQA" class="external">https://github.com/os-autoinst/os-autoinst-distri-openQA</a> for test code (and needles)</li>
<li>Check that tests can be triggered this way on a test instance</li>
<li>Do not put anything in /var/lib/openqa/share/tests and ensure tests still work as well as source code view and needle candidates in test details pages</li>
<li>To provide needle candidates there are multiple possibilities when and where the needle candidate data can be provided, try out one or multiple of the following:
<ol>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache (like asset downloads work or GIT_CACHE_DIR in os-autoinst and/or worker implementation), <em>When</em> tests are triggered on the side of web UI, <em>Then</em> the relevant data is git cloned, e.g. in the same steps as or similar to *_URL asset download</li>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache, <em>When</em> the worker uploads the general test structure, e.g. which modules will be executed, <em>Then</em> the relevant data is git cloned</li>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache, <em>When</em> the worker uploads individual needle check results, <em>Then</em> it also uploads as part of the JSON result files and image uploads all the necessary information to display needle candidates <em>And</em> the webUI in the receiving upload handler handles that somewhat … but does not overload when 1k workers upload in parallel or something :)</li>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache, <em>When</em> the worker uploads final results (or "finalizes" the job), <em>Then</em> the webUI triggers a download of test files and/or needle files to a local git cache dir as necessary</li>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache, <em>When</em> the first person reviews test results and selects needle candidates, <em>Then</em> the webUI triggers a download of test files and/or needle files to a local git cache dir as necessary</li>
</ol></li>
</ul>
QA - action #157204 (New): Sync openQA job removal events to qem-dashboard listening to AMQP eventshttps://progress.opensuse.org/issues/1572042024-03-14T05:33:51Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a href="https://suse.slack.com/archives/C02CLB8TZP1/p1709892527534149?thread_ts=1709883106.021479&cid=C02CLB8TZP1" class="external">https://suse.slack.com/archives/C02CLB8TZP1/p1709892527534149?thread_ts=1709883106.021479&cid=C02CLB8TZP1</a><br>
When openQA jobs are deleted then the according reference in qem-dashboard should also be removed. Listen to AMQP events to sync the removal accordingly</p>
openQA Project - action #156547 (New): A single openQA webUI search view to show all not-ok tests...https://progress.opensuse.org/issues/1565472024-03-04T11:04:30Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>From #121246-15: "We'd need to look for all the tests that are failing for a given incident, using the same TEST_ISSUES for both, Aggregates and Incidents". After <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655">#117655</a> we identified that we can use that to filter but there is a key/job id limit so we need to find out how to work with a much larger limit e.g. with a database index.</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>We can get a job for a particular incident (<a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655#note-33">#117655#note-33</a>)
<ul>
<li>openqa-cli api --o3 /job_settings/jobs key=*_TEST_ISSUES list_value=1234567</li>
<li>openqa-cli api --osd /job_settings/jobs key=LTSS_TEST_ISSUES list_value=20988</li>
<li>Note there is an implicit enforced limit of 20000 jobs here (see also the later suggestion)</li>
</ul></li>
<li>Explore removing the key/job id limit, or add a way to override it (and/or make followup ticket to finally introduce a trigram gin index for fast text searching without limits on keys)</li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>This doesn't need to be specific to squads/blocking tests (openQA itself should not know about these SUSE specific concepts)</li>
</ul>
openQA Infrastructure - action #156130 (Workable): Install Intel GPU on one of the servers in FC ...https://progress.opensuse.org/issues/1561302024-02-27T08:43:01Zszarate
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Steve Quinata provided a intel GPU ideally we should be able to connect it to one of our servers for further testing, and/or openQA enablement so we can also test hardware encoding, openCL, and other libraries that benefit from having extra hardware</p>
<p><a href="https://www.anandtech.com/show/17266/intels-arctic-soundm-server-accelerator-to-land-mid2022-with-hardware-av1-encoding" class="external">https://www.anandtech.com/show/17266/intels-arctic-soundm-server-accelerator-to-land-mid2022-with-hardware-av1-encoding</a></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> One remotely accessible computer features the GPU adapter</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Identify usable hardware from FC Basement or order if not too expensive in coordination with requester</li>
<li>Put the GPU into the suitable hardware</li>
<li>Update racktables</li>
<li>Inform szarate about the usable hardware setup</li>
</ul>
openQA Project - action #156055 (Workable): [research][timeboxed:20h] Evaluate if https://htmx.or...https://progress.opensuse.org/issues/1560552024-02-26T10:56:14Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Use less jquery, less outdated, insecure old cruft, more clean code, etc.</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Maybe we can not use htmx everywhere but on special pages where we could replace an existing outdated solution that is simple enough or it allows us to add new features where we don't have anything related yet</li>
<li>Do a little intro for the whole team about HTMX specific to what we do</li>
</ul>
openQA Project - action #155218 (New): [spike][timeboxed:10h] Use scenario definitions instead of...https://progress.opensuse.org/issues/1552182024-02-08T18:42:28Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>With <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: In openqa-in-openqa use scenario definitions instead of job group templates size:M (Resolved)" href="https://progress.opensuse.org/issues/132335">#132335</a> we showed that its feasible and not problematic to have the complete scenario definitions in the test distribution git repository. A much more complex test distribution is <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse</a> with the separate job group templates in <a href="https://github.com/os-autoinst/opensuse-jobgroups/" class="external">https://github.com/os-autoinst/opensuse-jobgroups/</a> as well as a corresponding gitlab.suse.de project for openqa.suse.de. We should elaborate what it would mean to use scenario definitions instead of the custom handling of job group templates.</p>
<a name="Goals"></a>
<h2 >Goals<a href="#Goals" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>G1:</strong> Proof of concept of scenario definitions for os-autoinst-distri-opensuse for multiple or all job group templates</li>
<li><strong>G2:</strong> Plan for migration</li>
<li><strong>G3:</strong> How to sustainably maintain all job group templates, e.g. including files, YAML anchors, aliases, etc.</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Just do it :)</li>
</ul>
QA - action #154498 (Workable): [spike][timeboxed:20h][integration] Approve/reject SLE maintenanc...https://progress.opensuse.org/issues/1544982024-01-29T17:33:42Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>One of the most important responsibilities within SLE maintenance testing is to approve/reject SLE maintenance release requests based on openQA test results. So far <a href="https://github.com/openSUSE/qem-bot" class="external">qem-bot</a> is sufficient to schedule openQA tests but merely does a mediocre job of reporting back results as test results are asynchronously polled based on a periodic schedule <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules</a> causing unnecessary delays, inefficient polling, using outdated results <a class="issue tracker-4 status-4 priority-4 priority-default child" title="action: Use live openQA test results instead of inconsistent qem-dashboard database in qem-bot approver (Feedback)" href="https://progress.opensuse.org/issues/122311">#122311</a> and not even reporting back on blocking test failures <a class="issue tracker-6 status-1 priority-5 priority-high3 child parent behind-schedule" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121">#97121</a>. Let's use a proper architecture with efficient event based triggers providing relevant information back to release requests on IBS using core openQA features rather than too much custom lacking downstream tooling: Develop a proof-of-concept of listening to yet-to-be designed "openQA product build testing finished" AMQP events and approve/reject the according release request.</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Research how common OBS checks are implemented, e.g. openQA staging test integration, legalreview, installcheck, etc. For this see <a href="https://github.com/openSUSE/opensuse-release-tools" class="external">https://github.com/openSUSE/opensuse-release-tools</a>
<ul>
<li>Read <a href="https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/rabbit-openqa.py" class="external">https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/rabbit-openqa.py</a></li>
<li>Read <a href="https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/rabbit-repoid.py" class="external">https://github.com/openSUSE/openSUSE-release-tools/blob/master/gocd/rabbit-repoid.py</a></li>
</ul></li>
<li>Follow <a class="issue tracker-4 status-3 priority-3 priority-lowest closed child" title="action: Find "last build" of a product over API size:M (Resolved)" href="https://progress.opensuse.org/issues/152939">#152939</a> and add publishing for an AMQP event for when incident "foo" finishes testing in openQA. For finding all tests related to incident "foo" see <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655">#117655</a></li>
<li>Integrate both of the above either in a new standalone application or hack into <a href="https://github.com/openSUSE/qem-bot" class="external">https://github.com/openSUSE/qem-bot</a> – as part of a spike solution so do not be afraid to break any other use case – to approve/"reject" SLE maintenance release requests. If "reject" seems to be too severe then provide only "informational" feedback, e.g. as IBS comment or checker result.</li>
<li>Optionally consider to implement this as a openQA plugin, maybe that is simpler for some cases</li>
</ul>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Also related to <a class="issue tracker-4 status-4 priority-4 priority-default child" title="action: Use live openQA test results instead of inconsistent qem-dashboard database in qem-bot approver (Feedback)" href="https://progress.opensuse.org/issues/122311">#122311</a>, <a class="issue tracker-6 status-1 priority-3 priority-lowest" title="coordination: [saga][epic] Re-combined Maintenance QA tooling covering both SLE+openSUSE (New)" href="https://progress.opensuse.org/issues/123088">#123088</a>, <a class="issue tracker-6 status-1 priority-5 priority-high3 child parent behind-schedule" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121">#97121</a>, <a class="issue tracker-6 status-1 priority-5 priority-high3 parent behind-schedule" title="coordination: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, ... (New)" href="https://progress.opensuse.org/issues/99303">#99303</a>, <a class="issue tracker-4 status-3 priority-3 priority-lowest closed child" title="action: Find "last build" of a product over API size:M (Resolved)" href="https://progress.opensuse.org/issues/152939">#152939</a>, <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: [timeboxed:6h][spike solution] a single command line or openQA webUI search view to show all test... (Resolved)" href="https://progress.opensuse.org/issues/131279">#131279</a>, <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655">#117655</a></p>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>Where to run persistently</li>
</ul>
openQA Infrastructure - action #152101 (Workable): Allow salt to properly configure non-productio...https://progress.opensuse.org/issues/1521012023-12-05T13:41:36Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>See lessons learned meeting <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Conduct "lessons learned" with Five Why analysis for "test fails in iscsi_client due to salt 'hos... (Resolved)" href="https://progress.opensuse.org/issues/139136">#139136</a>. Would diesel now work with the MTU related changes? We should ensure that diesel is treated as tap worker regardless of not being used as production tap-worker.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> non-production workers within the OSD salt config are properly configured as multi-machine openQA workers with suffixes e.g. <code>WORKER_CLASS=tap_poo1234</code> (no GRE tunnels required)</li>
<li><strong>AC2:</strong> non-production openQA workers with a special worker class can execute multi-machine jobs correctly if triggered against the special worker class</li>
<li><strong>AC3:</strong> non-production openQA workers with a special worker class do not pick up production multi-machine jobs</li>
<li><strong>AC4:</strong> The team knows how to configure non-production multi-machine workers for development/setup/debugging</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Read how <a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch.sls?ref_type=heads#L24" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/blob/master/openqa/openvswitch.sls?ref_type=heads#L24</a> matches on "tap" in WORKER_CLASS. We used to disable production tap worker classes by adding a suffix that include a ticket reference, e.g. "tap_poo1234"</li>
<li><del>Extend that to be able to match on variants of "tap"</del> It will still work with e.g. "tap_poo1234". The last part of <code>multihostclass in pillar['workerconf'][host]['workers'][wnum]['WORKER_CLASS']</code> refers to a string (and not a list) so Python does in fact just check whether the string contains <code>tap</code> at all.</li>
<li>Take a look at OSD workers that currently have "tap_$something", see <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls</a>, e.g. qesapworker-prg4 or diesel, and verify that those workers can still execute multi-machine clusters if scheduled against those specific classes</li>
<li>Document that this is how one can ensure a worker is configured for multi-machine tests but not for production jobs</li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>We don't strictly care about GRE tunnels here</li>
</ul>
QA - action #133748 (Workable): Move of openqaworker-arm-1 to FC Basement size:Mhttps://progress.opensuse.org/issues/1337482023-08-03T09:07:38Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>In #132614 openqaworker-arm-1 was moved to FC Basement so that we have one hot-redundant aarch64 OSD machine outside of PRG2. For that to be setup we need to also accomodate the automatic recovery feature.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> openqaworker-arm-1 runs OSD production jobs again</li>
<li><strong>AC2:</strong> The automatic recovery of openqaworker-arm-1 on crashes works</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Disable the automatic recovery for openqaworker-arm-1 from the old location</li>
<li>Mount the machine and connect it back into the network including DHCP/DNS in <a href="https://gitlab.suse.de/OPS-Service/salt/" class="external">https://gitlab.suse.de/OPS-Service/salt/</a></li>
<li>Remove old DHCP/DNS entries in <a href="https://gitlab.suse.de/OPS-Service/salt/" class="external">https://gitlab.suse.de/OPS-Service/salt/</a></li>
<li>Update <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/blob/master/openqa/workerconf.sls</a></li>
<li>Find on <a href="https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs" class="external">https://wiki.suse.net/index.php/SUSE-Quality_Assurance/Labs</a> how the new PDU can be used</li>
<li>Integrate the new PDU in <a href="https://gitlab.suse.de/openqa/grafana-webhook-actions" class="external">https://gitlab.suse.de/openqa/grafana-webhook-actions</a></li>
</ul>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li>Add back openqaworker-arm-1 to salt on OSD</li>
<li>after openqaworker-arm-1 is back remove silences in <a href="https://monitor.qa.suse.de/alerting/silences" class="external">https://monitor.qa.suse.de/alerting/silences</a></li>
<li>Remove the "Mute All times" in <a href="https://monitor.qa.suse.de/alerting/routes" class="external">https://monitor.qa.suse.de/alerting/routes</a> for <code>__contacts__ =~ .*"Trigger reboot of openqaworker-arm-1".*</code></li>
</ul>
openQA Project - action #130940 (New): Trigger openQA tests mentioned in github comments as part ...https://progress.opensuse.org/issues/1309402023-06-15T10:17:05Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Follow-up to <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Trigger openQA tests mentioned in github description as part of CI size:M (Resolved)" href="https://progress.opensuse.org/issues/130934">#130934</a>. After having openQA CI integration which reads github pull request description and triggers according openQA clone jobs github comments created or updated should be considered the same.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> openQA CI integration in any existing test distribution github project automatically runs openQA tests based on links to existing openQA jobs in the github comments</li>
<li><strong>AC2:</strong> Ensure openQA documentation covers that also github comments are parsed and how to use that</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Wait for <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Trigger openQA tests mentioned in github description as part of CI size:M (Resolved)" href="https://progress.opensuse.org/issues/130934">#130934</a></li>
<li>Read what was done originally in <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: Have git-clone-custom-refspec pickup tests from PR descriptions (Resolved)" href="https://progress.opensuse.org/issues/63712">#63712</a> and the according pull request to openQA <a href="https://github.com/os-autoinst/openQA/pull/2618" class="external">https://github.com/os-autoinst/openQA/pull/2618</a> and also <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Trigger openQA tests mentioned in github description as part of CI size:M (Resolved)" href="https://progress.opensuse.org/issues/130934">#130934</a></li>
<li>Check if openqa-clone-custom-git-ref with special comments also covers github comments or only the initial description</li>
<li>Extend the existing approach where necessary</li>
<li>Ensure that the same process is triggered also on comment <em>updates</em></li>
</ul>
openQA Project - action #128318 (Workable): [spike][timeboxed:20h] Current openQA+os-autoinst+dep...https://progress.opensuse.org/issues/1283182023-04-26T09:13:06Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>As the parent project about QAaaS progresses we should assess our expected efforts and risks about "openQA in SLE" ASAP for further planning. As according to hsehic openQA needs to be provided from pure SLE, i.e. no PackageHub, we should try what we can achieve in this regard by submitting packages and seeing the effect of that. Maybe some submissions can be directly accepted, maybe some need special treatment. Maybe SLE release managers tell us what processes we need to follow, which bugzilla or Jira tickets we need to create first or which contracts to sign with blood ;)</p>
<a name="Goal"></a>
<h2 >Goal<a href="#Goal" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>G1:</strong> All packages for current openQA+os-autoinst+dependencies have non-rejected (open or accepted) submit requests towards SLE</li>
<li><strong>G2:</strong> We know what process we need to follow to get openQA+os-autoinst+dependencies into SLE</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Identify all packages not yet in SLE: See the latest pipeline run within <a href="https://github.com/os-autoinst/scripts/actions/workflows/obs-list-dependencies.yml" class="external">https://github.com/os-autoinst/scripts/actions/workflows/obs-list-dependencies.yml</a></li>
<li>With a simple for-loop submit all packages from <a href="https://build.opensuse.org/project/show/devel:openQA:Leap:15.4" class="external">https://build.opensuse.org/project/show/devel:openQA:Leap:15.4</a> + <a href="https://build.opensuse.org/project/show/devel:openQA:tested" class="external">https://build.opensuse.org/project/show/devel:openQA:tested</a> (or openSUSE:Factory accordingly) to SLE (following <a href="https://en.opensuse.org/openSUSE:Maintenance_update_process" class="external">https://en.opensuse.org/openSUSE:Maintenance_update_process</a>)</li>
<li>Make sure the source is openSUSE:Factory to improve chances requests will be accepted (devel:openQA could be seen as a list of relevant packages in that sense)</li>
<li>React to comments/rejections in SRs accordingly</li>
<li>List all unattended SRs, i.e. the ones that have open review but apparently after some day receive no update</li>
<li>If you never worked with build.suse.de okurz suggests to use <a href="https://github.com/okurz/scripts/tree/master/isc" class="external">https://github.com/okurz/scripts/tree/master/isc</a></li>
<li>Submit request to <a href="https://build.suse.de/package/show/SUSE:SLE-15-SP4:GA/000product" class="external">https://build.suse.de/package/show/SUSE:SLE-15-SP4:GA/000product</a> to define a new SLE module for openQA</li>
</ul>
QA - action #119176 (Workable): Automated alerts and reminders about SLO's for openqatests size:Mhttps://progress.opensuse.org/issues/1191762022-10-21T10:12:45Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Some tickets don't see regular updates, even with automated reminders in some cases. See <a href="https://progress.opensuse.org/projects/openqatests/wiki#SLOs-service-level-objectives" class="external">https://progress.opensuse.org/projects/openqatests/wiki#SLOs-service-level-objectives</a> for the documented workflows/goals. We assume processes are followed better if there is automation that reminds people about missed targets and makes it easier to understand what updates are missing.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> <strong>DONE</strong> SLOs are reflected in automated reminders</li>
<li><strong>AC2</strong>: <strong>DONE</strong> The first reminder is implemented based on the documented workflow</li>
<li><strong>AC3</strong>: The second reminder is known to update the priority automatically</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE:</em> Wait for <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Automated alerts and reminders about SLO's for openqatests (only one reminder) size:M (Resolved)" href="https://progress.opensuse.org/issues/116545">#116545</a></li>
<li><em>DONE:</em> Same as in queries for QA tools we likely should only look at the update time of tickets with no subtasks, e.g. see the definition of <a href="https://progress.opensuse.org/issues?query_id=542" class="external">https://progress.opensuse.org/issues?query_id=542</a>, to prevent cases like <a href="https://progress.opensuse.org/issues/113749#note-13" class="external">https://progress.opensuse.org/issues/113749#note-13</a> ff which is Urgent but hasn't been updated for a long time.</li>
<li><em>DONE:</em> Only write the same comment once -> <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Automated alerts and reminders about SLO's for openqatests (only one reminder) size:M (Resolved)" href="https://progress.opensuse.org/issues/116545">#116545</a></li>
<li>Research what has been done in <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: Automated alerts and reminders about SLO's for openqatests size:M (Resolved)" href="https://progress.opensuse.org/issues/113797">#113797</a></li>
<li>Review and understand <a href="https://github.com/openSUSE/openqa-tests-backlog" class="external">https://github.com/openSUSE/openqa-tests-backlog</a> as well as <a href="https://github.com/openSUSE/backlogger" class="external">https://github.com/openSUSE/backlogger</a> and <a href="https://opensuse.github.io/openqa-tests-backlog/" class="external">https://opensuse.github.io/openqa-tests-backlog/</a></li>
<li>As the queries in <a href="https://github.com/openSUSE/openqa-tests-backlog/blob/main/queries.yaml#L5" class="external">https://github.com/openSUSE/openqa-tests-backlog/blob/main/queries.yaml#L5</a> are not named queries but defined in place we could just define a "grace period" for each query and only act automatically if not done already by users, e.g. don't remind on urgent tickets after 7 days but only 7+2 days</li>
<li>Follow the SLO about the suggestion of the "second reminder"</li>
</ul>
openQA Infrastructure - action #76951 (Workable): Check if new firmware for kerosene (aka. power8...https://progress.opensuse.org/issues/769512020-11-04T06:06:17Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>New firmware might help to prevent qemu failing to run. If we find new firmware we could remove the parameters in os-autoinst again, see clone source ticket</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Read about context of the needed workaround <a class="issue tracker-4 status-3 priority-6 priority-high2 closed" title="action: 100% of powerpc tests incomplete auto_review:"(?s)Running on power8.*qemu-system-ppc64: Requested... (Resolved)" href="https://progress.opensuse.org/issues/75259">#75259</a></li>
<li>Currently <a href="https://kerosene-sp.qe.nue2.suse.org" class="external">https://kerosene-sp.qe.nue2.suse.org</a> lists FW840.00. Compare to other machines like diesel+petrol to see if there is a newer ASM version?</li>
<li>Look for new firmware for the machine, just search for new firmware on IBM web pages</li>
<li>Check if the new firmware means we do not need <a href="https://github.com/os-autoinst/os-autoinst/pull/1554" class="external">https://github.com/os-autoinst/os-autoinst/pull/1554</a> anymore, if yes, remove again, if no, remove again but add according settings to the machine settings in openQA, this is also what "adamw" did:</li>
</ul>
<pre><code>[04/11/2020 17:41:52] <adamw> okurz: i don't really know what the consequences of it are, but i tend to the idea that qemu wouldn't be trying to make it the default without reason :) i can ask some virt guys if you like
[04/11/2020 17:42:09] <adamw> okurz: but on the whole, yes, it seems to be it'd be more appropriate to put it in your templates rather than hardwire it into os-autoinst.
[04/11/2020 17:42:29] <adamw> that's what i was doing when we had the problem (i was setting an older machine type in our ppc64le Machine vars)
</code></pre>