openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-28T21:42:42ZopenSUSE Project Management Tool
Redmine openQA Tests - action #158245 (Feedback): test fails in openqa_workerhttps://progress.opensuse.org/issues/1582452024-03-28T21:42:42Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>openQA test in scenario openqa-Tumbleweed-dev-x86_64-openqa_install_nginx@64bit-2G fails in<br>
<a href="https://openqa.opensuse.org/tests/4049058/modules/openqa_worker/steps/8" class="external">openqa_worker</a></p>
<a name="Test-suite-description"></a>
<h2 >Test suite description<a href="#Test-suite-description" class="wiki-anchor">¶</a></h2>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.opensuse.org/tests/4049058" class="external">:TW.27506</a> (current job)</p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.opensuse.org/tests/4049009" class="external">:TW.27505</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=openqa&flavor=dev&machine=64bit-2G&test=openqa_install_nginx&version=Tumbleweed" class="external">latest</a></p>
openQA Infrastructure - action #158026 (Resolved): osd-deployment exceeds 2h maximum runtime duri...https://progress.opensuse.org/issues/1580262024-03-26T08:23:01Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2426666" class="external">https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2426666</a></p>
<pre><code>+ retry -r 3 -- zypper --no-refresh -n dup --replacefiles
Loading repository data...
..Reading installed packages...
.Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...
.
The following 7 packages are going to be upgraded:
openQA openQA-client openQA-common openQA-doc openQA-local-db system-user-velociraptor velociraptor
7 packages to upgrade.
Overall download size: 0 B. Already cached: 21.2 MiB. After the operation, additional 4.7 KiB will be used.
[...]
Checking for file conflicts: [..done]
(1/4) Installing: openQA-common-4.6.1711372491.18a87328-lp155.6447.1.ppc64le [...done]
(2/4) Installing: os-autoinst-distri-opensuse-deps-1.1711423505.d81d6831-lp155.14058.1.noarch [...done]
(3/4) Installing: openQA-client-4.6.1711372491.18a87328-lp155.6447.1.ppc64le [...done]
(4/4) Installing: openQA-worker-4.6.1711372491.18a87328-lp155.6447.1.ppc64le [....done]....................................................................................................................................................
[...]
............................................................................
ERROR: Job failed: execution took longer than 2h0m0s seconds
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li></li>
</ul>
openQA Infrastructure - action #158023 (Resolved): salt-states-openqa pipeline invalid arguments ...https://progress.opensuse.org/issues/1580232024-03-26T08:18:33Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2425817" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2425817</a> and also <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2422794" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2422794</a></p>
<pre><code>monitor.qe.nue2.suse.org:
Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li></li>
</ul>
openQA Infrastructure - action #158020 (Resolved): salt-states-openqa pipeline times outhttps://progress.opensuse.org/issues/1580202024-03-26T08:13:58Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2425611" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2425611</a></p>
<pre><code> ID: SUSE:SLE-15-SP6:Update:BCI
Function: cmd.run
Name: su geekotest -c 'mkdir -p SUSE:SLE-15-SP6:Update:BCI && python3 script/sctimeout: sending signal TERM to command 'ssh'
</code></pre>
<p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2425891" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2425891</a></p>
<pre><code> ID: stop_and_disable_all_not_configured_workers
Function: cmd.run
Name: services=$(systemctl list-units --all 'openqa-worker-auto-restart@*.service' | sed -e '/.*openqa-worker-auto-restart@.*\.service.*/!d' -e 's|.*openqa-worker-auto-restart@\(.*\)\.service.*|\1|' | awk '{ if($0 > 16) print "openqa-worker-auto-restart@" $0 ".service openqa-reload-worker-auto-restart@" $0 ".path" }' | tr '\n' ' '); [ -z "$services" ] || systemctl disable --ntimeout: sending signal TERM to command 'ssh'
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li></li>
</ul>
QA - action #157858 (Resolved): Repeated reminder comments about SLO's for openqatests size:Shttps://progress.opensuse.org/issues/1578582024-03-25T08:37:52Zlivdywanliv.dywan@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: No ticket reminder comments about SLO's for openqatests size:M (Resolved)" href="https://progress.opensuse.org/issues/157522">#157522</a> addressed a bug that prevented reminder comments from being sent. Unfortunately comments are added even if a comment was already present. This is especially visible in <em>immediate</em> tickets, for example #153115, which get daily reminders - as per <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Automated alerts and reminders about SLO's for openqatests (only one reminder) size:M (Resolved)" href="https://progress.opensuse.org/issues/116545">#116545</a> only one comment is supposed to be added. Maybe this is a regression or the check is not comprehensive enough.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Reminders are only added once</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>We already have the code that should handle that: Review the implementation from <a class="issue tracker-4 status-3 priority-5 priority-high3 closed child" title="action: Automated alerts and reminders about SLO's for openqatests (only one reminder) size:M (Resolved)" href="https://progress.opensuse.org/issues/116545">#116545</a> for gaps in the current logic in <a href="https://github.com/openSUSE/backlogger/blob/main/backlogger.py" class="external">https://github.com/openSUSE/backlogger/blob/main/backlogger.py</a></li>
<li>Investigate if something changed with current comments, maybe the Redmine upgrade made a difference here (complete guess)?</li>
<li>Maybe the regex needs to be adapted and/or better covered with unit testing</li>
</ul>
openQA Project - action #157774 (New): Empty scenario definitions causing Use of uninitialized va...https://progress.opensuse.org/issues/1577742024-03-22T12:22:30Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>This has not been flagged by any alerts as far as I am aware. It is visible in logs, though:</p>
<p>journal:</p>
<pre><code>Mar 21 14:49:58 openqa openqa[29349]: Use of uninitialized value $testsuite_name in hash element at /usr/share/openqa/script/../lib/OpenQA/Schema/Result/JobGroups.pm line 413.
</code></pre>
<p>accesslog:</p>
<pre><code>[21/Mar/2024:14:49:58 +0100] "POST /api/v1/job_templates_scheduling/510
[21/Mar/2024:14:49:58 +0100] "POST /api/v1/job_templates_scheduling/319
[21/Mar/2024:14:49:58 +0100] "POST /api/v1/job_templates_scheduling/456
</code></pre>
<p>This seems to be caused by this type of scenario:</p>
<pre><code>scenarios:
# ...
s390x:
sle-15-SP6-Migration-from-SLE15-SPx-s390x:
- {}
</code></pre>
<p>See <a href="https://openqa.suse.de/admin/job_templates/319" class="external">https://openqa.suse.de/admin/job_templates/319</a> at the time of this writing. Despite the errors/warnings this is valid with the current schema.</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Consider the proposed trivial fix but also notably the conversation in <a href="https://github.com/os-autoinst/openQA/pull/5540#discussion_r1534128393" class="external">https://github.com/os-autoinst/openQA/pull/5540#discussion_r1534128393</a></li>
<li>Verify the current behavior</li>
<li>Confirm what the expectation of these uses is, and how else it could be achieved if it is considered wrong
<ul>
<li>Consider changing or extending the YAML schema accordingly</li>
</ul></li>
</ul>
openQA Infrastructure - action #157726 (Blocked): osd-deployment | Failed pipeline for master (wo...https://progress.opensuse.org/issues/1577262024-03-22T08:23:53Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2415705" class="external">https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2415705</a></p>
<pre><code>worker37.oqa.prg2.suse.org:
Minion did not return. [Not connected]
worker36.oqa.prg2.suse.org:
Minion did not return. [Not connected]
worker38.oqa.prg2.suse.org:
Minion did not return. [Not connected]
worker39.oqa.prg2.suse.org:
Minion did not return. [Not connected]
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> osd-deployment passes again</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><em>DONE</em> Take machine out of production: <a href="https://progress.opensuse.org/projects/openqav3/wiki/#Take-machines-out-of-salt-controlled-production" class="external">https://progress.opensuse.org/projects/openqav3/wiki/#Take-machines-out-of-salt-controlled-production</a></li>
<li>*DONE Remove machine XYZ from production <code>ssh osd "sudo salt-key -y -d XYZ"</code></li>
<li>Retrigger failed osd deployment CI pipeline</li>
<li>Confirm if this is one or multiple, possibly already known issues</li>
<li>Fix any potential hardware issue, e.g. with hardware replacement</li>
<li>Ensure machines are back in production</li>
</ul>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li><a href="https://progress.opensuse.org/projects/openqav3/wiki/#Bring-back-machines-into-salt-controlled-production" class="external">https://progress.opensuse.org/projects/openqav3/wiki/#Bring-back-machines-into-salt-controlled-production</a> <code>for i in 36 37 38 39 ; do sudo salt-key -y -a worker$i.oqa.prg2.suse.org; done && sleep 30 && for i in 36 37 38 39 ; do sudo salt --state-output=changes "worker$i*" state.apply</code></li>
</ul>
QA - action #157522 (Resolved): No ticket reminder comments about SLO's for openqatests size:Mhttps://progress.opensuse.org/issues/1575222024-03-19T11:20:59Zlivdywanliv.dywan@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a href="https://opensuse.github.io/openqa-tests-backlog/" class="external">https://opensuse.github.io/openqa-tests-backlog/</a> contains various queries which when flagged as red should also result in comments on relevant tickets. No comments <a href="https://progress.opensuse.org/activity?from=2023-03-01&user_id=40859" class="external">have been observed</a> in several months</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Neglected tickets receive a reminder comment</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate if reminder comments are correctly enabled, e.g. by checking <a href="https://github.com/openSUSE/openqa-tests-backlog/blob/main/.github/workflows/backlogger.yml#L22" class="external">https://github.com/openSUSE/openqa-tests-backlog/blob/main/.github/workflows/backlogger.yml#L22</a>
<ul>
<li><code>--reminder-comment-on-issues</code> checks that there is no existing comment - maybe there is a flaw in the logic?</li>
<li>Comments are created via API calls - <a href="https://github.com/openSUSE/openqa-tests-backlog/actions/runs/8341719003/job/22828346106" class="external">pipelines on GitHub</a> don't show any errors, though?</li>
</ul></li>
<li>Consider adding more verbose logging to see if things work correctly in production</li>
</ul>
QA - action #157237 (Resolved): dependabot PRs for the dashboard are not getting approved and mer...https://progress.opensuse.org/issues/1572372024-03-14T11:05:52Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>See also <a href="https://suse.slack.com/archives/C02AJ1E568M/p1710360358208519" class="external">the conversation in Slack</a>.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> dependabot PR's are merged without any human interaction</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate if the <a href="https://github.com/openSUSE/qem-dashboard/blob/main/.mergify.yml#L27" class="external">mergify config</a> is effective and works as intended (hypothesis being that it doesn't)</li>
</ul>
openQA Infrastructure - action #156517 (Resolved): Can't call method "script" on an undefined val...https://progress.opensuse.org/issues/1565172024-03-04T07:57:37Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>OpenQA logreport for ariel.suse-dmz.opensuse.org at Sun, 03 Mar 2024 14:10:02 +0000:</p>
<pre><code>[2024-03-03T14:09:46.684383Z] [error] [BtWM5m48OZPz] Can't call method "script" on an undefined value at /usr/share/openqa/script/../lib/OpenQA/WebAPI/Controller/Step.pm line 366.
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate the origin of this call</li>
<li>Consider a unit test that covers this code path</li>
</ul>
openQA Infrastructure - action #156514 (Resolved): Cron <root@openqa-service> (date; fetch_openqa...https://progress.opensuse.org/issues/1565142024-03-04T07:37:00Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Cron <a href="mailto:root@openqa-service">root@openqa-service</a> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log at Mon, 4 Mar 2024 00:20:40 +0000 (UTC):</p>
<pre><code>Exception occured while fetching bsc#1158056
Traceback (most recent call last):
File "/usr/bin/fetch_openqa_bugs", line 62, in <module>
raise e
File "/usr/bin/fetch_openqa_bugs", line 48, in <module>
issue = issue_fetcher.get_issue(bugid)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 88, in get_issue
return self.prefix_table[prefix](self.conf, bugid)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 24, in __init__
self.fetch(conf)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/bugzilla_issue.py", line 28, in fetch
data = req.json()
File "/usr/lib/python3.6/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate what is causing the JSON error - likely an error string was returned instead of JSON</li>
<li>Handle the exception in bugzilla_issue.py and print a meaningful error</li>
<li>Investigate why 3 seemingly identical errors are sent in separate emails around the same time</li>
</ul>
openQA Infrastructure - action #156481 (Resolved): cron -> (fetch_openqa_bugs)> /tmp/fetch_openqa...https://progress.opensuse.org/issues/1564812024-03-01T17:06:37Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Cron <a href="mailto:root@openqa-service">root@openqa-service</a> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log:</p>
<pre><code>openqa_client.exceptions.ConnectionError: HTTPSConnectionPool(host='openqa.suse.de', port=443): Max retries exceeded with url: /api/v1/bugs?refreshable=1&delta=86400 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f91f6877080>: Failed to establish a new connection: [Errno 113] No route to host',))
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li></li>
</ul>
openQA Tests - action #156439 (New): Firefox privacy popup causing apache2_changehat to failhttps://progress.opensuse.org/issues/1564392024-03-01T08:46:32Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>The test module<br>
<a href="https://openqa.opensuse.org/tests/3964918/modules/apache2_changehat/steps/108" class="external">apache2_changehat</a> occasionally fails because a popup bar comes up between the mouse moving to the needle and the click of the button, which is no longer in the same place. The x11test base class has Firefox-specific abstractions which cover this, however this test is currently a consoletest.</p>
<p>Context: This came up in the open conversation in the QE Tools workshop, discussing recurring test issues that come up as flaky.</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Base the test on x11test</li>
<li>Replace duplicated logic to setup Firefox</li>
</ul>
<a name="Reproducible"></a>
<h2 >Reproducible<a href="#Reproducible" class="wiki-anchor">¶</a></h2>
<p>Fails since (at least) Build <a href="https://openqa.opensuse.org/tests/3964918" class="external">20240226</a> (current job)</p>
<a name="Expected-result"></a>
<h2 >Expected result<a href="#Expected-result" class="wiki-anchor">¶</a></h2>
<p>Last good: <a href="https://openqa.opensuse.org/tests/3963542" class="external">20240225</a> (or more recent)</p>
<a name="Further-details"></a>
<h2 >Further details<a href="#Further-details" class="wiki-anchor">¶</a></h2>
<p>Always latest result in this scenario: <a href="https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=apparmor_profile&version=Tumbleweed" class="external">latest</a></p>
openQA Infrastructure - action #156304 (Rejected): [salt-pillars] Pipeline failed / Passed invali...https://progress.opensuse.org/issues/1563042024-02-29T09:02:36Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2326037" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2326037</a></p>
<pre><code>++ timeout -v --kill-after=10m 90m ssh openqa.suse.de 'set -eo pipefail; rm -f salt_highstate.log; salt --log-file=salt_highstate.log --log-file-level=debug --state-output=changes --hide-timeout \* state.highstate queue=True | awk '\''/Result..Clean[- ]*Started/ {print > "salt_profiling.log"; next} 1'\'''
worker30.oqa.prg2.suse.org:
Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list
Retrieve the state data from the salt master for this minion and execute it
</code></pre>
<p>This is an error message we generally see every now and then.</p>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>DONE</strong> Restart pipelines</li>
<li><p>Investigate what's causing the invalid highstate</p>
<ul>
<li>Checkout <a href="https://progress.opensuse.org/issues/133469" class="external">https://progress.opensuse.org/issues/133469</a> - the debugging-code should be still in place on all workers</li>
</ul></li>
<li><p>As we have seen that problem in the past lookup older tickets if we have reported something similar</p></li>
<li><p>Research if this is an upstream issue</p></li>
<li><p>Try to find mitigations, workarounds, better error messages, etc.</p></li>
</ul>
openQA Infrastructure - action #156301 (Resolved): [bot-ng] Pipeline failed / KeyError: 'priority...https://progress.opensuse.org/issues/1563012024-02-29T08:54:46Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2327183" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2327183</a></p>
<pre><code>++ retry -r 30 -e -- ./qem-bot/bot-ng.py -c /etc/openqabot --token [MASKED] incidents-run
[...]
KeyError: 'priority'
Retrying up to 19 more times after sleeping 6144s …
2024-02-29 06:28:46 INFO Bot schedule starts now
Traceback (most recent call last):
File "./qem-bot/bot-ng.py", line 7, in <module>
main()
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/main.py", line 32, in main
sys.exit(cfg.func(cfg))
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/args.py", line 24, in do_incident_schedule
bot = OpenQABot(args)
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/openqabot.py", line 24, in __init__
self.incidents = get_incidents(self.token)
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/loader/qem.py", line 41, in get_incidents
xs.append(Incident(i))
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/types/incident.py", line 23, in __init__
self.priority = incident["priority"]
KeyError: 'priority'
Retrying up to 18 more times after sleeping 12288s …
ERROR: Job failed: execution took longer than 4h0m0s seconds
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>DONE</strong> Restart pipelines</li>
<li>Investigate if there is new data the bot is not handling correctly</li>
<li>Don't provoke timeouts with retrying on reproducible errors</li>
<li>Look into unit test coverage</li>
</ul>