openSUSE Project Management Tool: Issueshttps://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842024-03-26T10:10:15ZopenSUSE Project Management Tool
Redmine openQA Infrastructure - action #158041 (Resolved): grenache needs upgrade to 15.5https://progress.opensuse.org/issues/1580412024-03-26T10:10:15Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p>grenache-1 was offline for many months so it was not online when we upgraded our infrastructure to Leap 15.5 so grenache is still on 15.4 so we should upgrade that as well.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> grenache-1 runs a stable Leap 15.5</li>
<li><strong>AC2:</strong> osd-deployment and salt states deployment and alerts are good regarding grenache-1</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Conduct the distribution upgrade according to <a href="https://progress.opensuse.org/projects/openqav3/wiki/#Distribution-upgrades" class="external">https://progress.opensuse.org/projects/openqav3/wiki/#Distribution-upgrades</a></li>
<li>Apply according necessary package locks</li>
<li>Remove obsolete package locks</li>
<li>Ensure system is fully upgraded</li>
<li>Try multiple reboots</li>
<li>Ensure that there are no related alerts</li>
</ul>
<a name="Rollback-actions"></a>
<h2 >Rollback actions<a href="#Rollback-actions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Remove silence "alertname=Failed systemd services alert (except openqa.suse.de)" from <a href="https://monitor.qa.suse.de/alerting/silences" class="external">https://monitor.qa.suse.de/alerting/silences</a></li>
<li>Remove silence "alertname=grenache-1: host up alert" from <a href="https://monitor.qa.suse.de/alerting/silences" class="external">https://monitor.qa.suse.de/alerting/silences</a></li>
</ul>
openQA Infrastructure - action #158026 (Resolved): osd-deployment exceeds 2h maximum runtime duri...https://progress.opensuse.org/issues/1580262024-03-26T08:23:01Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2426666" class="external">https://gitlab.suse.de/openqa/osd-deployment/-/jobs/2426666</a></p>
<pre><code>+ retry -r 3 -- zypper --no-refresh -n dup --replacefiles
Loading repository data...
..Reading installed packages...
.Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command.
Computing distribution upgrade...
.
The following 7 packages are going to be upgraded:
openQA openQA-client openQA-common openQA-doc openQA-local-db system-user-velociraptor velociraptor
7 packages to upgrade.
Overall download size: 0 B. Already cached: 21.2 MiB. After the operation, additional 4.7 KiB will be used.
[...]
Checking for file conflicts: [..done]
(1/4) Installing: openQA-common-4.6.1711372491.18a87328-lp155.6447.1.ppc64le [...done]
(2/4) Installing: os-autoinst-distri-opensuse-deps-1.1711423505.d81d6831-lp155.14058.1.noarch [...done]
(3/4) Installing: openQA-client-4.6.1711372491.18a87328-lp155.6447.1.ppc64le [...done]
(4/4) Installing: openQA-worker-4.6.1711372491.18a87328-lp155.6447.1.ppc64le [....done]....................................................................................................................................................
[...]
............................................................................
ERROR: Job failed: execution took longer than 2h0m0s seconds
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li></li>
</ul>
openQA Infrastructure - action #158023 (Resolved): salt-states-openqa pipeline invalid arguments ...https://progress.opensuse.org/issues/1580232024-03-26T08:18:33Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2425817" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2425817</a> and also <a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2422794" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2422794</a></p>
<pre><code>monitor.qe.nue2.suse.org:
Passed invalid arguments to state.highstate: expected str, bytes or os.PathLike object, not list
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li></li>
</ul>
openQA Infrastructure - action #158020 (Resolved): salt-states-openqa pipeline times outhttps://progress.opensuse.org/issues/1580202024-03-26T08:13:58Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2425611" class="external">https://gitlab.suse.de/openqa/salt-pillars-openqa/-/jobs/2425611</a></p>
<pre><code> ID: SUSE:SLE-15-SP6:Update:BCI
Function: cmd.run
Name: su geekotest -c 'mkdir -p SUSE:SLE-15-SP6:Update:BCI && python3 script/sctimeout: sending signal TERM to command 'ssh'
</code></pre>
<p><a href="https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2425891" class="external">https://gitlab.suse.de/openqa/salt-states-openqa/-/jobs/2425891</a></p>
<pre><code> ID: stop_and_disable_all_not_configured_workers
Function: cmd.run
Name: services=$(systemctl list-units --all 'openqa-worker-auto-restart@*.service' | sed -e '/.*openqa-worker-auto-restart@.*\.service.*/!d' -e 's|.*openqa-worker-auto-restart@\(.*\)\.service.*|\1|' | awk '{ if($0 > 16) print "openqa-worker-auto-restart@" $0 ".service openqa-reload-worker-auto-restart@" $0 ".path" }' | tr '\n' ' '); [ -z "$services" ] || systemctl disable --ntimeout: sending signal TERM to command 'ssh'
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li></li>
</ul>
QA - action #157819 (Resolved): Can't login to walter1 and walter2 offlinehttps://progress.opensuse.org/issues/1578192024-03-24T18:55:53Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<pre><code>$ ssh root@walter1.qe.nue2.suse.org
Password:
</code></pre><pre><code>$ ping walter2.qe.nue2.suse.org
PING walter2.qe.nue2.suse.org(2a07:de40:a102:5:10:168:192:2 (2a07:de40:a102:5:10:168:192:2)) 56 data bytes
From 2a07:de40:a100:1:ffff:ffff:ffff:ffff (2a07:de40:a100:1:ffff:ffff:ffff:ffff) icmp_seq=3 Destination unreachable: Address unreachable
$ ping -4 root@walter2.qe.nue2.suse.org
ping: root@walter2.qe.nue2.suse.org: Name or service not known
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> SSH login for QE tools members works for both walter1+2</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Try to reproduce yourself</li>
<li>Create SUSE IT ticket <a href="https://progress.opensuse.org/projects/qa/wiki/Tools#SUSE-IT-ticket-handling" class="external">https://progress.opensuse.org/projects/qa/wiki/Tools#SUSE-IT-ticket-handling</a></li>
</ul>
QA - action #157522 (Resolved): No ticket reminder comments about SLO's for openqatests size:Mhttps://progress.opensuse.org/issues/1575222024-03-19T11:20:59Zlivdywanliv.dywan@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a href="https://opensuse.github.io/openqa-tests-backlog/" class="external">https://opensuse.github.io/openqa-tests-backlog/</a> contains various queries which when flagged as red should also result in comments on relevant tickets. No comments <a href="https://progress.opensuse.org/activity?from=2023-03-01&user_id=40859" class="external">have been observed</a> in several months</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Neglected tickets receive a reminder comment</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate if reminder comments are correctly enabled, e.g. by checking <a href="https://github.com/openSUSE/openqa-tests-backlog/blob/main/.github/workflows/backlogger.yml#L22" class="external">https://github.com/openSUSE/openqa-tests-backlog/blob/main/.github/workflows/backlogger.yml#L22</a>
<ul>
<li><code>--reminder-comment-on-issues</code> checks that there is no existing comment - maybe there is a flaw in the logic?</li>
<li>Comments are created via API calls - <a href="https://github.com/openSUSE/openqa-tests-backlog/actions/runs/8341719003/job/22828346106" class="external">pipelines on GitHub</a> don't show any errors, though?</li>
</ul></li>
<li>Consider adding more verbose logging to see if things work correctly in production</li>
</ul>
openQA Infrastructure - action #156514 (Resolved): Cron <root@openqa-service> (date; fetch_openqa...https://progress.opensuse.org/issues/1565142024-03-04T07:37:00Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Cron <a href="mailto:root@openqa-service">root@openqa-service</a> (date; fetch_openqa_bugs)> /tmp/fetch_openqa_bugs_osd.log at Mon, 4 Mar 2024 00:20:40 +0000 (UTC):</p>
<pre><code>Exception occured while fetching bsc#1158056
Traceback (most recent call last):
File "/usr/bin/fetch_openqa_bugs", line 62, in <module>
raise e
File "/usr/bin/fetch_openqa_bugs", line 48, in <module>
issue = issue_fetcher.get_issue(bugid)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 88, in get_issue
return self.prefix_table[prefix](self.conf, bugid)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/__init__.py", line 24, in __init__
self.fetch(conf)
File "/usr/lib/python3.6/site-packages/openqa_bugfetcher/issues/bugzilla_issue.py", line 28, in fetch
data = req.json()
File "/usr/lib/python3.6/site-packages/requests/models.py", line 898, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate what is causing the JSON error - likely an error string was returned instead of JSON</li>
<li>Handle the exception in bugzilla_issue.py and print a meaningful error</li>
<li>Investigate why 3 seemingly identical errors are sent in separate emails around the same time</li>
</ul>
openQA Infrastructure - action #156301 (Resolved): [bot-ng] Pipeline failed / KeyError: 'priority...https://progress.opensuse.org/issues/1563012024-02-29T08:54:46Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2327183" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2327183</a></p>
<pre><code>++ retry -r 30 -e -- ./qem-bot/bot-ng.py -c /etc/openqabot --token [MASKED] incidents-run
[...]
KeyError: 'priority'
Retrying up to 19 more times after sleeping 6144s …
2024-02-29 06:28:46 INFO Bot schedule starts now
Traceback (most recent call last):
File "./qem-bot/bot-ng.py", line 7, in <module>
main()
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/main.py", line 32, in main
sys.exit(cfg.func(cfg))
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/args.py", line 24, in do_incident_schedule
bot = OpenQABot(args)
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/openqabot.py", line 24, in __init__
self.incidents = get_incidents(self.token)
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/loader/qem.py", line 41, in get_incidents
xs.append(Incident(i))
File "/builds/qa-maintenance/bot-ng/qem-bot/openqabot/types/incident.py", line 23, in __init__
self.priority = incident["priority"]
KeyError: 'priority'
Retrying up to 18 more times after sleeping 12288s …
ERROR: Job failed: execution took longer than 4h0m0s seconds
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>DONE</strong> Restart pipelines</li>
<li>Investigate if there is new data the bot is not handling correctly</li>
<li>Don't provoke timeouts with retrying on reproducible errors</li>
<li>Look into unit test coverage</li>
</ul>
openQA Infrastructure - action #156064 (Resolved): Handle planned datacenter shutdown PRG1 2024-0...https://progress.opensuse.org/issues/1560642024-02-26T12:32:25Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a href="https://suse.slack.com/archives/C02AET1AAAD/p1708950623186539" class="external">https://suse.slack.com/archives/C02AET1AAAD/p1708950623186539</a></p>
<pre><code>(Moroni Flores)
* From February 28, 2024, starting at 18:00 CET (Feb 28 10:00 MST; Feb 29 1:00 CST)
* To February 29, 8:00 CET (Feb 29 0:00 MST; Feb 29 15:00 CST)
:round_pushpin: Location: Prague office server room (PRG1)
:wrench: Maintenance: Power outage in the PRG1 server room
:rotating_light: Impact: ALL services in PRG1 will be affectedOn February 28 evening there is planned a power outage in the Prague office building and surrounding neighbourhood.In preparation to this, all the machines in the Prague office server room (PRG1) must be shut down. The Infra team will start shutting down the local equipment from 18:00 CET.
We ask each of you who has a server in PRG1 to please shut them down before 18:00 CET.The Infra team will restart the servers and services gradually from February 29, 8:00 CET. In case a machine or service is still missing from 9:00 CET please let us know via SD ticket.Thanks,
SUSE IT Infra :suselogo:
</code></pre>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Servers maintained by QE Tools in PRG1 are operational again as in before after the shutdown</li>
<li><strong>AC2:</strong> No unused machines are powered on and wasting power</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Identify all impacted machines on <a href="https://racktables.nue.suse.com/" class="external">https://racktables.nue.suse.com/</a></li>
<li>Inform users</li>
<li>Consider shutting down all QE tools maintained machines before the shutdown in a controlled manner</li>
<li>Await shutdown to be over</li>
<li>Bring up all production machines and ensure operation</li>
<li>Ensure that no unused machines are powered on and wasting power</li>
</ul>
openQA Project - action #156052 (Resolved): [alert] Scripts CI pipeline failing after logging mu...https://progress.opensuse.org/issues/1560522024-02-26T10:26:59Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/scripts-ci/-/jobs/2315561" class="external">https://gitlab.suse.de/openqa/scripts-ci/-/jobs/2315561</a></p>
<pre><code>2 jobs have been created:
- http://openqa.suse.de/tests/13603796
- http://openqa.suse.de/tests/13603797
{"blocked_by_id":null,"id":13603796,"result":"none","state":"scheduled"}
Job state of job ID 13603796: scheduled, waiting …
{"blocked_by_id":null,"id":13603796,"result":"none","state":"running"}
Job state of job ID 13603796: running, waiting …
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Investigate what is causing the pipeline to fail
<ul>
<li>The pipeline fails.</li>
<li>The two created jobs failed.</li>
<li>There is a lot of log messages mentioning "waiting" which is not shown to be successful or unsuccessful.</li>
</ul></li>
</ul>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<p>Active pipelines on <a href="https://gitlab.suse.de/openqa/scripts-ci/-/pipeline_schedules" class="external">https://gitlab.suse.de/openqa/scripts-ci/-/pipeline_schedules</a> again</p>
QA - action #155755 (Resolved): OBS build errors in gitlinthttps://progress.opensuse.org/issues/1557552024-02-21T13:44:39Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://build.opensuse.org/package/live_build_log/devel:tools:scm/gitlint/openSUSE_Tumbleweed/x86_64" class="external">https://build.opensuse.org/package/live_build_log/devel:tools:scm/gitlint/openSUSE_Tumbleweed/x86_64</a></p>
<pre><code>[ 12s] + /usr/bin/python3.11 -mpip wheel --verbose --progress-bar off --disable-pip-version-check --use-pep517 --no-build-isolation --no-deps --wheel-dir ./build .
[ 12s] Processing /home/abuild/rpmbuild/BUILD/gitlint-core-0.18.0
…
[ 12s] ModuleNotFoundError: No module named 'setuptools'
</code></pre> openQA Infrastructure - action #155740 (Resolved): Scripts CI pipelines fail due to timeout after...https://progress.opensuse.org/issues/1557402024-02-21T11:58:03Zlivdywanliv.dywan@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://gitlab.suse.de/openqa/scripts-ci/-/jobs/2298958" class="external">https://gitlab.suse.de/openqa/scripts-ci/-/jobs/2298958</a></p>
<pre><code>{"count":2,"failed":[],"ids":[13560656,13560657],"scheduled_product_id":2058111}
2 jobs have been created:
- http://openqa.suse.de/tests/13560656
- http://openqa.suse.de/tests/13560657
{"blocked_by_id":null,"id":13560656,"result":"none","state":"scheduled"}
Job state of job ID 13560656: scheduled, waiting …
{"blocked_by_id":null,"id":13560656,"result":"none","state":"scheduled"}
[...]
Job state of job ID 13560656: scheduled, waiting …
{"blocked_by_id":null,"id":13560656,"result":"none","state":"scheduled"}
Jo
ERROR: Job failed: execution took longer than 1h0m0s seconds
</code></pre>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li></li>
</ul>
openQA Infrastructure - action #155689 (Resolved): bot-ng pipelines fails to schedule incidentshttps://progress.opensuse.org/issues/1556892024-02-20T11:05:17Znicksingernsinger@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p>Since Feb 19, 2024 6:03pm GMT+0100 our pipelines in bot-ng fail at the step "schedule incidents": <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs</a> e.g. <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2292982" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/jobs/2292982</a> which was the first I could find</p>
<a name="Acceptance-Criteria"></a>
<h2 >Acceptance Criteria<a href="#Acceptance-Criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: Pipelines do work again (complete the job "schedule incidents")</li>
</ul>
<a name="Rollback-steps"></a>
<h2 >Rollback steps<a href="#Rollback-steps" class="wiki-anchor">¶</a></h2>
<ul>
<li>activate pipelines <a href="https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules" class="external">https://gitlab.suse.de/qa-maintenance/bot-ng/-/pipeline_schedules</a></li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Check if something changed</li>
<li>Read the logs and try to understand if the issues are the same and if/how we can fix them</li>
</ul>
QA - action #155629 (Resolved): [spike][timeboxed:6h][qem-dashboard] Order blocked incidents by p...https://progress.opensuse.org/issues/1556292024-02-19T11:12:25Zokurzokurz@suse.com
<a name="Motivation"></a>
<h2 >Motivation<a href="#Motivation" class="wiki-anchor">¶</a></h2>
<p><a href="https://smelt.suse.de/overview/#testing" class="external">https://smelt.suse.de/overview/#testing</a> shows incidents that are in testing with much more details about incidents than <a href="http://dashboard.qam.suse.de/blocked" class="external">http://dashboard.qam.suse.de/blocked</a> does. As apparently openQA test reviewers are not currently able to review all blocking test failures in time maintenance coordinators asked to better focus on incidents by priority. For this to select the higher prio incidents first the entries on <a href="http://dashboard.qam.suse.de/blocked" class="external">http://dashboard.qam.suse.de/blocked</a> should reflect the incident priority, e.g. order by priority or show the priority value.</p>
<a name="Acceptance-Criteria"></a>
<h2 >Acceptance Criteria<a href="#Acceptance-Criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1:</strong> Proof-of-concept for ordering <a href="http://dashboard.qam.suse.de/blocked" class="external">http://dashboard.qam.suse.de/blocked</a> rows by incident priority</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Example of requesting incident priority for a single incident: <a href="https://smelt.suse.de/graphql/#query=%7B%0A%20%20incidents(incidentId%3A%2032579)%20%7B%0A%20%20%20%20edges%20%7B%0A%20%20%20%20%20%20node%20%7B%0A%20%20%20%20%20%20%20%20incidentpackagesSet%20%7B%0A%20%20%20%20%20%20%20%20%20%20edges%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20node%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20package%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20name%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20node%20%7B%0A%20%20%20%20%20%20%20%20priority%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A" class="external">https://smelt.suse.de/graphql/#query=%7B%0A%20%20incidents(incidentId%3A%2032579)%20%7B%0A%20%20%20%20edges%20%7B%0A%20%20%20%20%20%20node%20%7B%0A%20%20%20%20%20%20%20%20incidentpackagesSet%20%7B%0A%20%20%20%20%20%20%20%20%20%20edges%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20node%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20package%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20name%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20node%20%7B%0A%20%20%20%20%20%20%20%20priority%0A%20%20%20%20%20%20%7D%0A%20%20%20%20%7D%0A%20%20%7D%0A%7D%0A</a></li>
<li>Extend qem-bot and/or qem-dashboard as necessary to have incident priority available</li>
<li>Lookup the most recent change handling embargoed updates ("embargoed" boolean value) for an example of how to add the new data from <a href="https://github.com/openSUSE/qem-dashboard/issues?q=is%3Aclosed+-label%3Adependencies+" class="external">https://github.com/openSUSE/qem-dashboard/issues?q=is%3Aclosed+-label%3Adependencies+</a> and correspondingly from qem-bot (maybe <a href="https://github.com/openSUSE/qem-bot/pull/128" class="external">https://github.com/openSUSE/qem-bot/pull/128</a> is relevant)</li>
</ul>
QA - action #155458 (Resolved): Seemingly reproducible build failures in devel:languages:perl per...https://progress.opensuse.org/issues/1554582024-02-14T08:28:43Zokurzokurz@suse.com
<a name="Observation"></a>
<h2 >Observation<a href="#Observation" class="wiki-anchor">¶</a></h2>
<p><a href="https://build.opensuse.org/package/live_build_log/devel:languages:perl/perl-Mojo-IOLoop-ReadWriteProcess/openSUSE_Tumbleweed/x86_64" class="external">https://build.opensuse.org/package/live_build_log/devel:languages:perl/perl-Mojo-IOLoop-ReadWriteProcess/openSUSE_Tumbleweed/x86_64</a><br>
fails and I don't even see the error:</p>
<pre><code>…
[ 108s] t/12_mocked_container.t .. ok
[ 108s] t/13_shared.t ............ skipped: Skipped unless TEST_SHARED is set
[ 108s] All tests successful.
[ 108s] Files=15, Tests=69, 100 wallclock secs ( 0.08 usr 0.03 sys + 4.21 cusr 0.74 csys = 5.06 CPU)
[ 108s] Result: PASS
…
[ 108s] + RPM_EC=0
[ 108s] ++ jobs -p
[ 108s] + exit 0
[ 108s] Executing(%license): /usr/bin/bash -e /var/tmp/rpm-tmp.3qVKzU
[ 108s] + umask 022
[ 108s] + cd /home/abuild/rpmbuild/BUILD
[ 108s] + cd Mojo-IOLoop-ReadWriteProcess-0.34
[ 108s] + LICENSEDIR=/home/abuild/rpmbuild/BUILDROOT/perl-Mojo-IOLoop-ReadWriteProcess-0.340.0-38.11.x86_64/usr/share/licenses/perl-Mojo-IOLoop-ReadWriteProcess
[ 108s] + export LC_ALL=
[ 108s] + LC_ALL=
[ 108s] + export LICENSEDIR
[ 108s] + /usr/bin/mkdir -p /home/abuild/rpmbuild/BUILDROOT/perl-Mojo-IOLoop-ReadWriteProcess-0.340.0-38.11.x86_64/usr/share/licenses/perl-Mojo-IOLoop-ReadWriteProcess
[ 108s] + cp -pr /home/abuild/rpmbuild/BUILD/Mojo-IOLoop-ReadWriteProcess-0.34/LICENSE /home/abuild/rpmbuild/BUILDROOT/perl-Mojo-IOLoop-ReadWriteProcess-0.340.0-38.11.x86_64/usr/share/licenses/perl-Mojo-IOLoop-ReadWriteProcess
[ 108s] + RPM_EC=0
[ 108s] ++ jobs -p
[ 108s] + exit 0
[ 108s] Broken pipe
[ 108s] ### VM INTERACTION START ###
[ 108s] [ 105.200672][ T1] sysrq: Power Off
[ 108s] [ 105.201394][ T84] reboot: Power down
[ 108s] ### VM INTERACTION END ###
[ 108s]
[ 108s] i03-ch1c failed "build perl-Mojo-IOLoop-ReadWriteProcess.spec" at Wed Feb 14 08:02:04 UTC 2024.
</code></pre>
<p>EDIT: By now <a href="https://build.opensuse.org/package/live_build_log/devel:languages:perl/perl-Mojo-IOLoop-ReadWriteProcess/openSUSE_Tumbleweed/x86_64" class="external">https://build.opensuse.org/package/live_build_log/devel:languages:perl/perl-Mojo-IOLoop-ReadWriteProcess/openSUSE_Tumbleweed/x86_64</a> is back to "succeeded" not showing "Broken pipe". I suggest to actually just report to OBS upstream about the errors, ignorable warnings, etc.</p>
<a name="Acceptance-criteria"></a>
<h2 >Acceptance criteria<a href="#Acceptance-criteria" class="wiki-anchor">¶</a></h2>
<ul>
<li><strong>AC1</strong>: The problem has been reported to upstream OBS (either already reported or new report)</li>
</ul>
<a name="Suggestions"></a>
<h2 >Suggestions<a href="#Suggestions" class="wiki-anchor">¶</a></h2>
<ul>
<li>Lookup if there any existing issues about such confusing error messages or report a new one</li>
<li>Learn from OBS experts how to better handle such situation and share with the team</li>
</ul>