openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-01-18T13:58:20ZopenSUSE Project Management Tool
Redmine QA - action #153886 (New): SMELT incidents and Release Requests IDs are not unique and may interf...https://progress.opensuse.org/issues/1538862024-01-18T13:58:20Zmgrifalconi
<p>I would like to raise 2 issues (to be verified) about the current approval process of maintenance updates:</p>
<ul>
<li><p>SMELT Incidents ID can be reused for multiple Release Requests and what the process uses right now is the incident ID to tag a test that is crucial for the RR approval. Now the bot/dashboard combo uses a workaround of deleting some openqa results (from dashboard DB) to prevent issues (see <a href="https://github.com/openSUSE/qem-dashboard/pull/78/files" class="external">https://github.com/openSUSE/qem-dashboard/pull/78/files</a> ) but this makes the bot approval logic complex and shared between bot and dashboard code. Would be nice to switch from SMELT ID to IBS RR ID (or just add the RR on top) to resolve the issue at the origin.</p></li>
<li><p>RR are not unique either, but in a different way: RR can be revoked and then reopen (maybe with different content to test? to be checked). I know the bot recognize (some) changes and re-triggers incident tests, but what about aggregates? Is there a chance they could be wrongly considered for approval decision? Also incident channels could be changed while the incident/RR combo is being tested causing some confusion on bot side. If this proves to be a real issue, a solution idea would be to make sure test results related to older 'version' of a RR are not considered and the bot waits for new ones. Maybe add to SMELT-ID/RR combo, also a timestamp of smelt-incident/ibs-rr latest change?</p></li>
</ul>
<p>I can expect the valid argument that these are rare corner cases, but we should also consider that we are here to catch corner cases. Complex updates that gets modified while being tested should get enhanced attention and not reduced IMO.</p>
openQA Tests - action #137255 (Resolved): [s390x][kvm][qe-core] Make use of generic "s390-kvm" cl...https://progress.opensuse.org/issues/1372552023-09-29T12:52:48Zmgrifalconi
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>Same as <a class="issue tracker-4 status-3 priority-5 priority-high3 closed" title="action: [qe-core][s390x][kvm] Make use of generic "s390-kvm" class to prevent too long waiting for s390x ... (Resolved)" href="https://progress.opensuse.org/issues/127523">#127523</a> but for core squad. According to current plans s390-kvm-sle12 can not be provided anymore after 2023-10, see #132152 for context.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> s390x kvm tests in the qe-core scope use an openQA "machine definition" that uses the generic "s390-kvm" worker class</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Verify common openQA tests can work on "s390-kvm" worker class</li>
<li>Use "s390-kvm" in production job templates</li>
</ul>
openQA Tests - action #128933 (Resolved): test fails in acceptance_sched_stresshttps://progress.opensuse.org/issues/1289332023-05-08T14:34:27Zmgrifalconi
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>One more QA-RUN test that we should have removed long time ago. <br>
Other than finding the issue, we should convert this test to a 'normal' one instead of using qe-run</p>
<p>openQA test in scenario sle-12-SP5-Server-DVD-Updates-x86_64-mau-qa_acceptance_sched_stress@64bit fails in<br>
<a href="https://openqa.suse.de/tests/11063719/modules/acceptance_sched_stress/steps/97" class="external">acceptance_sched_stress</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Testsuite maintained at <a href="https://gitlab.suse.de/qa-maintenance/qam-openqa-yml" class="external">https://gitlab.suse.de/qa-maintenance/qam-openqa-yml</a>.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/11061611" class="external">20230507-1</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/11057394" class="external">20230506-1</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=mau-qa_acceptance_sched_stress&version=12-SP5" class="external">latest</a></p>
QA - action #123286 (Resolved): Bot and dashboard reference to wrong data and block update approv...https://progress.opensuse.org/issues/1232862023-01-18T09:19:08Zmgrifalconi
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Hello, there is some inconsistency with the dashboard data about 27130:dragonbox</p>
<p>Link of the red SLE 15 SP4 box in blocked page points to <a href="https://openqa.suse.de/tests/overview?build=%3A27130%3Afixmath&distri=sle&groupid=439" class="external">https://openqa.suse.de/tests/overview?build=%3A27130%3Afixmath&distri=sle&groupid=439</a><br>
with no failures</p>
<p>Link inside the update request page <a href="http://dashboard.qam.suse.de/incident/27130" class="external">http://dashboard.qam.suse.de/incident/27130</a> points to a different incidents results <a href="https://openqa.suse.de/tests/overview?build=%3A27130%3Alibmwaw" class="external">https://openqa.suse.de/tests/overview?build=%3A27130%3Alibmwaw</a> with this time a failure</p>
<p>Bot approval job log:</p>
<pre><code> 2023-01-17 08:05:34 INFO Found failed, not-ignored job 10166069 for incident 27130
</code></pre>
<p>Interestingly enough, I restarted the month-old job and now even that is green.<br>
But still, the bot does not like it and keeps the 'box' red.<br>
<a href="https://openqa.suse.de/tests/10166069" class="external">https://openqa.suse.de/tests/10166069</a><br>
even if its clone is green: <a href="https://openqa.suse.de/tests/10331221" class="external">https://openqa.suse.de/tests/10331221</a></p>
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>The problem here seems to be that the incident 27130 was modified multiple times and references multiple package as visible in <a href="https://smelt.suse.de/incident/27130/" class="external">https://smelt.suse.de/incident/27130/</a></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> The dashboard page and all links to openQA tests from dashboard reference the same consistent package(s) or no package at all, i.e. no "dragonbox" in dashboard but then pointing to "libmwaw" in openQA</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate if this is maybe <em>just a display issue</em> and in that case fix it</li>
<li>Ask mgrifalconi to update the ticket according to our ticket templates to help us understand what he really expects because we are not clear about that</li>
<li>Reconsider how we test maintenance requests before a release request is created while still supporting the "shift left" endeavour</li>
<li>Check if the data in the dashboard database regarding packages is consistent with SMELT (to rule out qem-bot involvement)</li>
</ul>
openQA Infrastructure - action #121816 (Resolved): Cannot access installation media on updates.su...https://progress.opensuse.org/issues/1218162022-12-12T07:51:24Zmgrifalconi
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>See <a href="http://dashboard.qam.suse.de/blocked" class="external">http://dashboard.qam.suse.de/blocked</a></p>
<p>Or a concrete example: <a href="https://openqa.suse.de/tests/10144681#step/installation/28" class="external">https://openqa.suse.de/tests/10144681#step/installation/28</a><br>
Just reproduced: <a href="https://openqa.suse.de/tests/10148791#" class="external">https://openqa.suse.de/tests/10148791#</a></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: updates.suse.com is reachable consistently</li>
<li><strong>AC2</strong>: Follow-up conversations were conducted</li>
</ul>
QA - action #117619 (Resolved): Bot approved update request with failing tests size:Mhttps://progress.opensuse.org/issues/1176192022-10-06T09:26:57Zmgrifalconi
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Incident <a href="https://smelt.suse.de/incident/25982/" class="external">https://smelt.suse.de/incident/25982/</a><br>
Request that was approved by sle-qam-openqa: <a href="https://build.suse.de/request/show/280720" class="external">https://build.suse.de/request/show/280720</a><br>
Bot job: <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1166058#L279" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1166058#L279</a><br>
<code>INFO: SUSE:Maintenance:25982:280720</code></p>
<p>Failing test: <a href="https://openqa.suse.de/tests/9642631#settings" class="external">https://openqa.suse.de/tests/9642631#settings</a><br>
Dashboard: <a href="https://dashboard.qam.suse.de/incident/25982" class="external">https://dashboard.qam.suse.de/incident/25982</a></p>
<p>Context on slack: <a href="https://suse.slack.com/archives/C02CANHLANP/p1665043765153419" class="external">https://suse.slack.com/archives/C02CANHLANP/p1665043765153419</a></p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: We know the reason why the bot approved the request and didn't see the test failure</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Run <code>./qem-bot/bot-ng.py -c /etc/openqabot --token [MASKED] inc-approve --dry</code> (see <a href="https://github.com/openSUSE/qem-bot/#usage" class="external">https://github.com/openSUSE/qem-bot/#usage</a> for more info)</li>
<li>Look into the dashboard logs on qam2.suse.de <code>journalctl -u dashboard.service</code></li>
<li>Note: The journal only goes back 3 days currently (Oct 3), so for the incident in question it's too late.
Consider increasing the journal size as a first step</li>
<li>Consider adding code that only runs the bot on a single incident</li>
</ul>
QA - action #113345 (Resolved): qem-bot does not ignore Development/Leap job groups as it should ...https://progress.opensuse.org/issues/1133452022-07-07T08:37:58Zmgrifalconi
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Bot does not ignore Development/Leap job groups and could block update approvals due to broken development tests.<br>
See <a href="http://dashboard.qam.suse.de/blocked" class="external">http://dashboard.qam.suse.de/blocked</a> and look for "leap"</p>
<a name="Problem"></a>
<h2 >Problem<a href="#Problem" class="wiki-anchor">¶</a></h2>
<p>Likely regression due to <a href="https://github.com/openSUSE/qem-bot/commit/d4d33720d183ba30b63529577e2bbad700b238cd" class="external">https://github.com/openSUSE/qem-bot/commit/d4d33720d183ba30b63529577e2bbad700b238cd</a><br>
or <a href="https://github.com/openSUSE/qem-bot/commit/c869a5cb7a56cdb5c3ba33f64e086f34c64ce5b9#diff-dbb33d499407c366ab760f232[…]e02dad0dd506c87b478b8007cf496ad" class="external">https://github.com/openSUSE/qem-bot/commit/c869a5cb7a56cdb5c3ba33f64e086f34c64ce5b9#diff-dbb33d499407c366ab760f232[…]e02dad0dd506c87b478b8007cf496ad</a></p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<p>Look hard at the above commits and try it out with "--dry-run" and fix it. We at least know that jobs are still ignored, e.g. <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1044528#L2663" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/1044528#L2663</a> shows "INFO: Ignoring job '9078831' in development group 'Maintenance: Leap 15.4 Incidents'" so there is at least <em>some</em> ignoring going on</p>
openQA Tests - action #110896 (Resolved): [sle][security][sle15sp4][fips] test fails in fips_setuphttps://progress.opensuse.org/issues/1108962022-05-11T09:14:04Zmgrifalconi
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Likely caused by <a href="https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14843" class="external">https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/14843</a> <br>
Will block update approval until fixed</p>
<p>openQA test in scenario sle-12-SP4-Server-DVD-Updates-x86_64-mau-extratests-security-fips@64bit fails in<br>
<a href="https://openqa.suse.de/tests/8733780/modules/fips_setup/steps/35" class="external">fips_setup</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>Testsuite maintained at <a href="https://gitlab.suse.de/qa-maintenance/qam-openqa-yml" class="external">https://gitlab.suse.de/qa-maintenance/qam-openqa-yml</a>.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/8732268" class="external">20220510-1</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/8726441" class="external">20220509-1</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=mau-extratests-security-fips&version=12-SP4" class="external">latest</a></p>
QA - action #107671 (Resolved): No aggregate maintenance runs scheduled today on osd size:Mhttps://progress.opensuse.org/issues/1076712022-02-28T07:13:59Zmgrifalconi
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Seems a different issue than <a class="issue tracker-4 status-3 priority-6 priority-high2 closed child" title="action: No aggregate maintenance runs scheduled today on osd - dashboard.qem.suse.de down size:S (Resolved)" href="https://progress.opensuse.org/issues/106179">#106179</a> since the dashboard is accessible this time.</p>
<p>Link to list aggregate runs of the day:</p>
<p><a href="https://openqa.suse.de/tests/overview?arch=&flavor=&machine=&test=&modules=&module_re=&groupid=366&groupid=308&groupid=232&groupid=165&groupid=280&groupid=218&groupid=108&groupid=54&groupid=405&groupid=412&groupid=411&groupid=369&groupid=352&groupid=353&groupid=357&groupid=355&groupid=354&groupid=358&groupid=370&groupid=348&groupid=349&groupid=351&groupid=356&groupid=375&groupid=376&groupid=397&groupid=414&build=20220228-1#" class="external">https://openqa.suse.de/tests/overview?arch=&flavor=&machine=&test=&modules=&module_re=&groupid=366&groupid=308&groupid=232&groupid=165&groupid=280&groupid=218&groupid=108&groupid=54&groupid=405&groupid=412&groupid=411&groupid=369&groupid=352&groupid=353&groupid=357&groupid=355&groupid=354&groupid=358&groupid=370&groupid=348&groupid=349&groupid=351&groupid=356&groupid=375&groupid=376&groupid=397&groupid=414&build=20220228-1#</a><br>
(This was showing an empty list at that point)</p>
<p>Impact: update approval blocked</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>caused by downtime of <a href="http://download.suse.de" class="external">http://download.suse.de</a></li>
<li>read suggestions from <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: openQABot pipeline failed: "ERROR:root:Something bad happended during reading MR data from SMELT/... (Resolved)" href="https://progress.opensuse.org/issues/105603">#105603</a></li>
<li>Some gitlab CI steps are failing but we allow them to fail to let other steps continue, e.g. in <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/886067" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/886067</a> "sync smelt" fails but we allow it to fail so that "sync incidents" can continue but we also don't receive an alert about it and there is not sufficient retrying. We could split the steps into separate pipelines, make each step fatal and add configurable number of retries and interval between retries customized for each step in <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/blob/master/.gitlab-ci.yml" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/blob/master/.gitlab-ci.yml</a>, e.g. for sync smelt long enough , retrying to cover the weekly SUSE IT maintenance window, less for other critical steps</li>
<li>For retrying we do not even need to change qem-bot, we could use just a wrapper in the gitlab CI job itself, e.g. <a href="https://github.com/okurz/leaky_bucket_error_count" class="external">https://github.com/okurz/leaky_bucket_error_count</a></li>
<li>Also look into gitlab CI options to either abort a previous pipeline if a new one is triggered or not start new ones as long as old ones are still running</li>
</ul>
QA - action #106179 (Resolved): No aggregate maintenance runs scheduled today on osd - dashboard....https://progress.opensuse.org/issues/1061792022-02-08T08:17:53Zmgrifalconi
<p>No aggregate runs scheduled today - dashboard.qem.suse.de down</p>
<p>Link to list aggregate runs of the day: <a href="https://openqa.suse.de/tests/overview?result=none&result=passed&result=softfailed&result=failed&result=incomplete&result=skipped&result=obsoleted&result=parallel_failed&result=parallel_restarted&result=user_cancelled&result=user_restarted&result=timeout_exceeded&state=scheduled&state=assigned&state=setup&state=running&state=uploading&state=done&state=cancelled&arch=&flavor=&machine=&test=&modules=&module_re=&groupid=366&groupid=308&groupid=232&groupid=165&groupid=280&groupid=218&groupid=108&groupid=54&groupid=405&groupid=412&groupid=411&groupid=369&groupid=352&groupid=353&groupid=357&groupid=355&groupid=354&groupid=358&groupid=370&groupid=348&groupid=349&groupid=351&groupid=356&groupid=375&groupid=376&groupid=397&groupid=414&build=20220208-1#" class="external">https://openqa.suse.de/tests/overview?result=none&result=passed&result=softfailed&result=failed&result=incomplete&result=skipped&result=obsoleted&result=parallel_failed&result=parallel_restarted&result=user_cancelled&result=user_restarted&result=timeout_exceeded&state=scheduled&state=assigned&state=setup&state=running&state=uploading&state=done&state=cancelled&arch=&flavor=&machine=&test=&modules=&module_re=&groupid=366&groupid=308&groupid=232&groupid=165&groupid=280&groupid=218&groupid=108&groupid=54&groupid=405&groupid=412&groupid=411&groupid=369&groupid=352&groupid=353&groupid=357&groupid=355&groupid=354&groupid=358&groupid=370&groupid=348&groupid=349&groupid=351&groupid=356&groupid=375&groupid=376&groupid=397&groupid=414&build=20220208-1#</a></p>
<p>This is blocking all update test/approval</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<p>Create an epic with feature requests based on this</p>
openQA Project - action #103557 (Rejected): Aggregate different versions into the same build open...https://progress.opensuse.org/issues/1035572021-12-06T13:20:45Zmgrifalconi
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>See example on newly refactored SAP/HA job group, as part of <a href="https://progress.opensuse.org/issues/92921" class="external">https://progress.opensuse.org/issues/92921</a></p>
<p><strong>Current behavior:</strong><br>
Link to group overview issue: <a href="https://openqa.suse.de/group_overview/405" class="external">https://openqa.suse.de/group_overview/405</a></p>
<p><img src="https://progress.opensuse.org/attachments/download/12282/group_overview.png" alt="group_overview.png" loading="lazy" /></p>
<p><strong>Expected behavior:</strong><br>
See one line per build number, with all test results, regardless of SLE version, Service Pack, flavor.</p>
QA - action #97274 (New): qam dashboard improvement ideashttps://progress.opensuse.org/issues/972742021-08-20T06:48:13Zmgrifalconi
<p>Hello, doing openQA review I always used smelt comments to find out which test run needs to be checked to approve an update.</p>
<p>Ideally approval is automated, but when a single test fails (out of dozens/hundreds) it still needs some manual work to decide if such failures can be ignored for that particular test.</p>
<p>I won't mention crosscheck aggregate runs with precedent days (see <a href="https://progress.opensuse.org/issues/97118" class="external">https://progress.opensuse.org/issues/97118</a>).</p>
<p>These are the current issues I found while using the dashboard for my week of review:</p>
<ul>
<li><strong>Sorting order</strong>: I like to sort on smelt the priority or due date to have an idea on the situation. Neither of which is available. Incidents are sorted by incident ID, which I do not care</li>
<li><strong>Missing Release Request ID</strong>: If I am given only a RR ID, I must go to smelt to find the incident and back to the dashboard.</li>
<li><strong>Result History</strong>: I can only see latest results, so I find more painful to crosscheck different days, but I would be happier to see such think automated (see other poo linked earlier). In the meantime though, it is just more painful than before. I also have a good overview of the situation near the end of the day, because in the morning all runs are still ongoing and cannot do review based on yesterday's results.</li>
<li><strong>Development Job Groups</strong>: such job groups are not ignored, also some test groups will fit in. This creates some confusion and time wasted.</li>
</ul>
<p>Extra thought: <br>
The dashboard and smelt might be duplicating some work. Why not having a link in smelt to the list of related tests on the dashboard? I would be using the indexing/priority/informations on smelt and then go on the dashboard to check tests, possibly with result history.<br>
What I am basically asking for is the same features as smelt comments, whichever implementation is used. </p>
QA - action #97118 (Feedback): enhance bot automatic approval: check multiple dayshttps://progress.opensuse.org/issues/971182021-08-18T12:22:41Zmgrifalconi
<p>Context: <br>
Right now the bot will approve an incident only if all tests that include such incident are green at a certain point in time.</p>
<p>Problem:<br>
Day 1 after incident creation: Test 1 to 10 are green, test 11 fails.<br>
Day 2 after incident creation: Test 1 fails, test 2 to 11 are green.<br>
Day 3 after incident creation: Test 1 is green, test 2 fails, test 3 to 11 are green.</p>
<p>You get the point.<br>
Update is likely fine since all tests were green at least once with the same update code.</p>
<p>While we always work to improve tests stability, it is tricky to find a time where "everything is green" due to test development, new updates that break aggregate runs, infra hiccups etc.</p>
<p>Right now it is slow manual work to crosscheck such results and very error prone.</p>
<p>AC:<br>
The bot can check between multiple days to make sure every test was green at least one and approve the update.</p>
<p>Implementation proposal:<br>
Same logic that checks latest run, but if a failure is found, go back in history and check if there was a green result up until incident creation.</p>
openQA Tests - action #97004 (Resolved): [yast] mru-install-minimal-with-addons boots to desktop ...https://progress.opensuse.org/issues/970042021-08-17T07:26:24Zmgrifalconi
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Multiple tests are likely affected by this. The following is one example. Please see in the comments more info about the yast problem.</p>
<p>openQA test in scenario sle-15-SP3-Server-DVD-Updates-x86_64-mau-qa_kernel_kexec@64bit fails in<br>
<a href="https://openqa.suse.de/tests/6867278/modules/kernel_kexec/steps/26" class="external">kernel_kexec</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/6838504" class="external">20210813-1</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/6654174" class="external">20210805-1</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-Updates&machine=64bit&test=mau-qa_kernel_kexec&version=15-SP3" class="external">latest</a></p>
openQA Tests - action #94171 (Rejected): [qem][sap] test fails in check_logs about 50% of timeshttps://progress.opensuse.org/issues/941712021-06-17T13:21:22Zmgrifalconi
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario sle-15-Server-DVD-SAP-Incidents-x86_64-qam-sles4sap_hana_node01@64bit-sap-qam fails in<br>
<a href="https://openqa.suse.de/tests/6279366/modules/check_logs/steps/17" class="external">check_logs</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<p>The base test suite is used for job templates defined in YAML documents. It has no settings of its own.</p>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.suse.de/tests/6274808" class="external">:19992:libnettle</a></p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.suse.de/tests/6274488" class="external">:19983:xterm</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.suse.de/tests/latest?arch=x86_64&distri=sle&flavor=Server-DVD-SAP-Incidents&machine=64bit-sap-qam&test=qam-sles4sap_hana_node01&version=15" class="external">latest</a></p>