openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-28T00:56:46ZopenSUSE Project Management Tool
Redmine openQA Infrastructure - action #158185 (Workable): parallel job failed to get the vars from its p...https://progress.opensuse.org/issues/1581852024-03-28T00:56:46ZJulie_CAOjcao@suse.com
<a name="Observation"></a>
<h3 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h3>
<p>We have a parallel job which failed in getting the vars from its pair. Rerun still failed. Is there something wrong with the worker service?</p>
<pre><code>sub get_var_from_parent {
my ($self, $var) = @_;
my $parents = get_parents();
#Query every parent to find the var
for my $job_id (@$parents) {
my $ref = get_job_autoinst_vars($job_id);
return $ref->{$var} if defined $ref->{$var};
}
return;
}
</code></pre>
<p><a href="https://openqa.suse.de/tests/13885165/logfile?filename=autoinst-log.txt" class="external">https://openqa.suse.de/tests/13885165/logfile?filename=autoinst-log.txt</a></p>
<pre><code>[2024-03-27T15:39:25.691962Z] [debug] [pid:4639] get_job_autoinst_vars: Connection error: Can't connect: Name or service not known; URL was http://worker35:20493/wS5wkxkWNNB9LK92/vars
</code></pre> openQA Infrastructure - action #158125 (New): typing issue on ppc64 worker - only pick up (or sta...https://progress.opensuse.org/issues/1581252024-03-27T08:52:37Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>In <a class="issue tracker-4 status-4 priority-5 priority-high3 child behind-schedule" title="action: typing issue on ppc64 worker size:S (Feedback)" href="https://progress.opensuse.org/issues/158104">#158104</a> we observed typing issues due to mania being overloaded. mania was configured to run 30 openQA worker instances and that was mostly fine as proven in <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Repurpose PowerPC hardware in FC Basement - mania Power8 PowerPC size:M (Resolved)" href="https://progress.opensuse.org/issues/139271#note-24">#139271-24</a>. The recent overload was likely triggered by enabling video again as part of <a class="issue tracker-4 status-1 priority-4 priority-default" title="action: remove NOVIDEO=1 from ppc64le workers (New)" href="https://progress.opensuse.org/issues/157636">#157636</a>. I already reduced the number of worker instances. But this has the drawback that again the long test backlog takes longer to be finished. We should be more flexible in using available ressource. Here I suggest to implement a check in the worker to only pick up new jobs if CPU load is below a configured threshold.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> An openQA worker does not start an openQA job if the CPU load is higher than configured threshold</li>
<li><strong>AC2:</strong> By default worker still pick up jobs if load is not too high</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Possibly the worker code somewhere in <a href="https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/Worker.pm#L472" class="external">https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/Worker.pm#L472</a> can be extended to check the cpu load and if it exceeds a (configurable) threshold then skip picking up any next job</li>
<li>Add a sensible disabled default value in <a href="https://github.com/os-autoinst/openQA/blob/master/etc/openqa/workers.ini" class="external">https://github.com/os-autoinst/openQA/blob/master/etc/openqa/workers.ini</a> with an explanation comment</li>
</ul>
openQA Infrastructure - action #158116 (New): typing issue on ppc64 worker - crosscheck performan...https://progress.opensuse.org/issues/1581162024-03-27T08:14:10Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>In <a class="issue tracker-4 status-4 priority-5 priority-high3 child behind-schedule" title="action: typing issue on ppc64 worker size:S (Feedback)" href="https://progress.opensuse.org/issues/158104">#158104</a> system overload on ppc64le machines was found which was likely triggered by <a class="issue tracker-4 status-1 priority-4 priority-default" title="action: remove NOVIDEO=1 from ppc64le workers (New)" href="https://progress.opensuse.org/issues/157636">#157636</a>. As a snapshot the current process list output from htop looks like this:</p>
<pre><code> PID USER PRI NI VIRT RES SHR S DISK R/W CPU% MEM% TIME+ ▽Command
1541 root 20 0 320M 194M 182M S 0.00 B/s 0.0 0.0 2h29:59 /usr/lib/systemd/systemd-j
96369 root 20 0 623M 98880 14336 S 0.00 B/s 0.0 0.0 54:05.86 /usr/bin/python3 /usr/bin/
1 root 20 0 178M 25024 11776 S 0.00 B/s 0.0 0.0 48:46.08 /usr/lib/systemd/systemd n
2000 root 20 0 9728 6208 2176 S 0.00 B/s 0.0 0.0 40:44.69 /usr/sbin/haveged -w 1024
157105 _openqa-wo 20 0 427M 189M 23808 R 0.00 B/s 68.4 0.0 32:22.39 ffmpeg -y -hide_banner -no
157062 _openqa-wo 20 0 427M 193M 23808 R 0.00 B/s 42.1 0.0 32:07.83 ffmpeg -y -hide_banner -no
157107 _openqa-wo 20 0 427M 189M 23808 R 0.00 B/s 68.4 0.0 30:29.03 ffmpeg -y -hide_banner -no
157063 _openqa-wo 20 0 427M 193M 23808 R 0.00 B/s 5.3 0.0 29:30.58 ffmpeg -y -hide_banner -no
6267 _openqa-wo 20 0 427M 193M 23808 R 0.00 B/s 63.2 0.0 25:54.22 ffmpeg -y -hide_banner -no
157108 _openqa-wo 20 0 427M 189M 23808 R 0.00 B/s 63.2 0.0 25:03.79 ffmpeg -y -hide_banner -no
157064 _openqa-wo 20 0 427M 193M 23808 R 0.00 B/s 2.6 0.0 23:50.53 ffmpeg -y -hide_banner -no
156485 _openqa-wo 20 0 427M 189M 23808 R 0.00 B/s 34.2 0.0 22:18.78 ffmpeg -y -hide_banner -no
6268 _openqa-wo 20 0 427M 193M 23808 R 0.00 B/s 57.9 0.0 21:48.92 ffmpeg -y -hide_banner -no
156601 _openqa-wo 20 0 427M 193M 23808 R 0.00 B/s 10.5 0.0 20:19.58 ffmpeg -y -hide_banner -no
6269 _openqa-wo 20 0 427M 193M 23808 R 0.00 B/s 55.3 0.0 16:33.02 ffmpeg -y -hide_banner -no
5898 _openqa-wo 20 0 427M 193M 23808 R 0.00 B/s 86.8 0.0 14:48.15 ffmpeg -y -hide_banner -no
31080 _openqa-wo 20 0 5720M 758M 28416 R 0.00 B/s 57.9 0.1 12:58.63 /usr/bin/qemu-system-ppc64
15778 _openqa-wo 20 0 6767M 1779M 28480 R 0.00 B/s 81.6 0.2 12:50.94 /usr/bin/qemu-system-ppc64
15781 _openqa-wo 20 0 6767M 1779M 28480 S 0.00 B/s 0.0 0.2 10:13.25 /usr/bin/qemu-system-ppc64
156709 _openqa-wo 20 0 6762M 1766M 28288 S 0.00 B/s 13.2 0.2 10:08.67 /usr/bin/qemu-system-ppc64
33559 _openqa-wo 20 0 6756M 1724M 28416 R 0.00 B/s 86.8 0.2 10:05.56 /usr/bin/qemu-system-ppc64
35017 _openqa-wo 20 0 3946M 753M 28416 R 0.00 B/s 84.2 0.1 9:30.77 /usr/bin/qemu-system-ppc64
24085 _openqa-wo 20 0 6901M 1781M 28480 S 0.00 B/s 0.0 0.2 9:13.94 /usr/bin/qemu-system-ppc64
24092 _openqa-wo 20 0 6901M 1781M 28480 R 0.00 B/s 78.9 0.2 8:40.60 /usr/bin/qemu-system-ppc64
28718 _openqa-wo 20 0 7135M 1787M 28480 S 0.00 B/s 50.0 0.2 8:17.91 /usr/bin/qemu-system-ppc64
28720 _openqa-wo 20 0 7135M 1787M 28480 R 0.00 B/s 13.2 0.2 6:51.75 /usr/bin/qemu-system-ppc64
39280 _openqa-wo 20 0 5712M 755M 28416 R 0.00 B/s 65.8 0.1 6:41.38 /usr/bin/qemu-system-ppc64
39683 _openqa-wo 20 0 6731M 1549M 28416 R 0.00 B/s 65.8 0.2 6:24.06 /usr/bin/qemu-system-ppc64
3699 root 20 0 3968 3200 2368 S 0.00 B/s 0.0 0.0 6:04.21 /sbin/agetty -o -p -- \u -
34903 _openqa-wo 20 0 6334M 1483M 28416 R 0.00 B/s 50.0 0.2 5:29.90 /usr/bin/qemu-system-ppc64
34902 _openqa-wo 20 0 6334M 1483M 28416 S 0.00 B/s 0.0 0.2 4:40.00 /usr/bin/qemu-system-ppc64
38988 _openqa-wo 20 0 6790M 1376M 28480 R 0.00 B/s 107.9 0.2 3:52.33 /usr/bin/qemu-system-ppc64
38599 _openqa-wo 20 0 8040M 4187M 28480 R 0.00 B/s 47.4 0.5 3:41.13 /usr/bin/qemu-system-ppc64
45395 _openqa-wo 20 0 3732M 757M 28416 R 0.00 B/s 71.1 0.1 3:38.90 /usr/bin/qemu-system-ppc64
38600 _openqa-wo 20 0 8040M 4187M 28480 S 0.00 B/s 0.0 0.5 3:18.94 /usr/bin/qemu-system-ppc64
43853 _openqa-wo 20 0 5641M 1696M 28480 R 0.00 B/s 63.2 0.2 3:12.66 /usr/bin/qemu-system-ppc64
38456 _openqa-wo 20 0 9087M 4195M 28480 R 0.00 B/s 78.9 0.5 3:08.68 /usr/bin/qemu-system-ppc64
38986 _openqa-wo 20 0 6790M 1376M 28480 R 0.00 B/s 86.8 0.2 3:06.34 /usr/bin/qemu-system-ppc64
</code></pre>
<p>so ffmpeg shows significantly higher accumulated CPU time usage compared to the according qemu processes. We should investigate if ffmpeg is having a "too high" impact on machine performance, if it should be running with nice level to prevent typing issues, if ffmpeg parameters can be tweaked or if ffmpeg should be avoided at all on ppc64le.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> openQA test video compression is ensured to not significantly impacting system performance causing typing issues</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Check if ffmpeg CPU usage as visible in the above htop output is considered expected or something unusual</li>
<li>Consider introducing a nice-level for calling ffmpeg in os-autoinst</li>
<li>Crosscheck if ffmpeg can be tweaked, in particular for ppc64le qemu workers</li>
<li>Decide if ffmpeg or even complete should be completely forbidden on ppc64le, see <a class="issue tracker-4 status-1 priority-4 priority-default" title="action: remove NOVIDEO=1 from ppc64le workers (New)" href="https://progress.opensuse.org/issues/157636">#157636</a> </li>
</ul>
QA - action #157741 (Workable): Approve/reject SLE maintenance release requests on IBS synchronou...https://progress.opensuse.org/issues/1577412024-03-22T10:23:10Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>One of the most important responsibilities within SLE maintenance testing is to approve/reject SLE maintenance release requests based on openQA test results. So far <a href="https://github.com/openSUSE/qem-bot" class="external">qem-bot</a> is sufficient to schedule openQA tests but merely does a mediocre job of reporting back results as test results are asynchronously polled based on a periodic schedule <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules</a> causing unnecessary delays, inefficient polling, using outdated results <a class="issue tracker-4 status-4 priority-4 priority-default child" title="action: Use live openQA test results instead of inconsistent qem-dashboard database in qem-bot approver (Feedback)" href="https://progress.opensuse.org/issues/122311">#122311</a> and not even reporting back on blocking test failures <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121">#97121</a>. Let's use a proper architecture with efficient event based triggers providing relevant information back to release requests on IBS using core openQA features rather than too much custom lacking downstream tooling: After the PoC in <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [spike][timeboxed:20h][integration] Approve/reject SLE maintenance release requests on IBS synchr... (Resolved)" href="https://progress.opensuse.org/issues/154498#note-14">#154498-14</a> we should fully implement that to approve/reject the according release request synchronously after AMQP event listening.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> something synchronously approves based on AMQP events</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Follow-on with the PoC of <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: [spike][timeboxed:20h][integration] Approve/reject SLE maintenance release requests on IBS synchr... (Resolved)" href="https://progress.opensuse.org/issues/154498#note-14">#154498-14</a></li>
<li>Setup qem-bot or an alternative on existing or new server but make access to the logs</li>
<li>Add it as part of qem-dashbaord which already has AMQP support</li>
<li>Ensure that qem-bot runs near-continuous to be able to listen to all AMQP events accordingly, maybe back-to-back gitlab CI jobs with limits to prevent parallel execution which we already have?</li>
</ul>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Also related to <a class="issue tracker-4 status-4 priority-4 priority-default child" title="action: Use live openQA test results instead of inconsistent qem-dashboard database in qem-bot approver (Feedback)" href="https://progress.opensuse.org/issues/122311">#122311</a>, <a class="issue tracker-6 status-1 priority-3 priority-lowest" title="coordination: [saga][epic] Re-combined Maintenance QA tooling covering both SLE+openSUSE (New)" href="https://progress.opensuse.org/issues/123088">#123088</a>, <a class="issue tracker-6 status-1 priority-4 priority-default child parent" title="coordination: [epic] enable qem-bot comments on IBS (was: enable qa-maintenance/openQABot comments on smelt again) (New)" href="https://progress.opensuse.org/issues/97121">#97121</a>, <a class="issue tracker-6 status-1 priority-4 priority-default parent behind-schedule" title="coordination: [saga][epic] Future improvements for SUSE Maintenance QA workflows with fully automated testing, ... (New)" href="https://progress.opensuse.org/issues/99303">#99303</a>, <a class="issue tracker-4 status-3 priority-3 priority-lowest closed child" title="action: Find "last build" of a product over API size:M (Resolved)" href="https://progress.opensuse.org/issues/152939">#152939</a>, <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: [timeboxed:6h][spike solution] a single command line or openQA webUI search view to show all test... (Resolved)" href="https://progress.opensuse.org/issues/131279">#131279</a>, <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: Provide API to get job results for a particular incident, similar to what dashboard/qem-bot does ... (Resolved)" href="https://progress.opensuse.org/issues/117655">#117655</a></p>
openQA Infrastructure - action #157555 (Workable): [spike][timeboxed:10h] Use a different ssh roo...https://progress.opensuse.org/issues/1575552024-03-19T21:34:02Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>In <a href="https://sd.suse.com/servicedesk/customer/portal/1/SD-150437" class="external">https://sd.suse.com/servicedesk/customer/portal/1/SD-150437</a> we are asked to handle "compromised root passwords in QA segments" including s390zl11…16</p>
<a name="Goals"></a>
<h2 >Goals<a href="#Goals" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>G1:</strong> Have an s390x kvm openQA installation job with non-default password succeed as far as possible</li>
<li><strong>G2:</strong> Identify which follow-up steps need to be done to fully support non-default passwords in such scenarios</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>os-autoinst-distri-opensuse in principle supports using a different password, see <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/main_common.pm#L165" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/master/lib/main_common.pm#L165</a></li>
<li>Clone a default s390x kvm openQA installation job <a href="https://openqa.suse.de/tests/13875911" class="external">https://openqa.suse.de/tests/13875911</a> from this scenario <a href="https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Online&machine=s390x-kvm&test=default&version=15-SP6" class="external">https://openqa.suse.de/tests/latest?arch=s390x&distri=sle&flavor=Online&machine=s390x-kvm&test=default&version=15-SP6</a> but with <code>PASSWORD=<new_password></code> with <code><new_password></code> being anything you setup temporary and see how far the test can reach</li>
<li>Fix obvious small problems and identify bigger follow-up tasks</li>
<li>Actually s390x shouldn't really matter that much in this context, could also be an "svirt" job</li>
</ul>
openQA Project - action #157273 (Workable): Run os-autoinst-distri-openQA directly from git witho...https://progress.opensuse.org/issues/1572732024-03-14T16:38:04Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>With <a class="issue tracker-4 status-3 priority-4 priority-default closed child" title="action: [spike][timeboxed:10h] Run os-autoinst-distri-example directly from git and ensure candidate need... (Resolved)" href="https://progress.opensuse.org/issues/154783">#154783</a> we have proper git caching so we can run git based tests efficiently on our workers now. Now we should go the next step and migrate one "production" test distribution to use only git and not hold anything provided by admins on o3 in o3:/var/lib/openqa/share/tests for this test distribution.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> /var/lib/openqa/share/tests/open{qa,QA} do not exist</li>
<li><strong>AC2:</strong> openqa-in-openqa tests still pass consistently</li>
<li><strong>AC3:</strong> openqa-in-openqa test details, needle candidates and source code views still show content as expected</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Change test definitions in <a href="https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml" class="external">https://github.com/os-autoinst/os-autoinst-distri-openQA/blob/master/scenario-definitions.yaml</a> in your branch to use <a href="https://github.com/os-autoinst/os-autoinst-distri-openQA" class="external">https://github.com/os-autoinst/os-autoinst-distri-openQA</a> for test code (and needles)</li>
<li>Check that tests can be triggered this way on a test instance</li>
<li>Do not put anything in /var/lib/openqa/share/tests and ensure tests still work as well as source code view and needle candidates in test details pages</li>
<li>To provide needle candidates there are multiple possibilities when and where the needle candidate data can be provided, try out one or multiple of the following:
<ol>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache (like asset downloads work or GIT_CACHE_DIR in os-autoinst and/or worker implementation), <em>When</em> tests are triggered on the side of web UI, <em>Then</em> the relevant data is git cloned, e.g. in the same steps as or similar to *_URL asset download</li>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache, <em>When</em> the worker uploads the general test structure, e.g. which modules will be executed, <em>Then</em> the relevant data is git cloned</li>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache, <em>When</em> the worker uploads individual needle check results, <em>Then</em> it also uploads as part of the JSON result files and image uploads all the necessary information to display needle candidates <em>And</em> the webUI in the receiving upload handler handles that somewhat … but does not overload when 1k workers upload in parallel or something :)</li>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache, <em>When</em> the worker uploads final results (or "finalizes" the job), <em>Then</em> the webUI triggers a download of test files and/or needle files to a local git cache dir as necessary</li>
<li><em>Given</em> a test distribution/needledir does not yet exist in a local cache, <em>When</em> the first person reviews test results and selects needle candidates, <em>Then</em> the webUI triggers a download of test files and/or needle files to a local git cache dir as necessary</li>
</ol></li>
<li>If you identify any bigger feature implementation in openQA or os-autoinst itself being necessary then ensure those requirements are covered in other tickets and block on those tickets accordingly</li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>Any bigger feature implementation in openQA or os-autoinst itself.</li>
</ul>
QA - action #157204 (Workable): Sync openQA job removal events to qem-dashboard listening to AMQP...https://progress.opensuse.org/issues/1572042024-03-14T05:33:51Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a href="https://suse.slack.com/archives/C02CLB8TZP1/p1709892527534149?thread_ts=1709883106.021479&cid=C02CLB8TZP1" class="external">https://suse.slack.com/archives/C02CLB8TZP1/p1709892527534149?thread_ts=1709883106.021479&cid=C02CLB8TZP1</a><br>
When openQA jobs are deleted then the according reference in qem-dashboard should also be removed. Listen to AMQP events to sync the removal accordingly</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> SLE maintenance openQA jobs previously blocking SLE maintenance updates on <a href="http://dashboard.qam.suse.de/blocked" class="external">http://dashboard.qam.suse.de/blocked</a> do not block approval after such openQA jobs are deleted from the openQA database</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Use TDD: Extend <a href="https://github.com/openSUSE/qem-dashboard/blob/main/t/amqp.t" class="external">https://github.com/openSUSE/qem-dashboard/blob/main/t/amqp.t</a> and ensure there is a failing test first</li>
<li>Extend <a href="https://github.com/openSUSE/qem-dashboard/blob/08cea810f936faeb6af35b645270d85f6569c6b9/lib/Dashboard/Model/AMQP.pm#L33" class="external">https://github.com/openSUSE/qem-dashboard/blob/08cea810f936faeb6af35b645270d85f6569c6b9/lib/Dashboard/Model/AMQP.pm#L33</a> to update the database entry accordingly or delete, whatever is applicable</li>
<li>For all current openQA job result entries in the dashboard database crosscheck if there are entries for jobs that do not exist anymore in the openQA database. Remove accordingly.</li>
<li>Verify operation in production: E.g. create an artificial, failed openQA job in OSD for a non-critical SLE maintenance update, wait till it shows up as blocking on <a href="http://dashboard.qam.suse.de/blocked" class="external">http://dashboard.qam.suse.de/blocked</a> or in log files of the qem-bot "approve" cycle, remove the job over <code>openqa-cli -X delete jobs/$id</code> again and verify that <a href="http://dashboard.qam.suse.de/blocked" class="external">http://dashboard.qam.suse.de/blocked</a> does not show up as blocked on that job anymore</li>
</ul>
<a name="Out-of-scope"></a>
<h2 >Out of scope<a href="#Out-of-scope" class="wiki-anchor">¶</a></h2>
<ul>
<li>Regular cleanup of results when we missed or have otherwise not received according AMQP events</li>
</ul>