https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842020-10-14T08:02:09ZopenSUSE Project Management ToolopenQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3358662020-10-14T08:02:09ZXiaojing_liuxliu1@suse.com
<ul><li><strong>Subject</strong> changed from <i>auto_review:"setup failure: Cache service status error from API: Minion job #45813 failed: Can't use an undefined value as a HASH reference at.*"</i> to <i>auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"</i></li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3358692020-10-14T08:02:25ZXiaojing_liuxliu1@suse.com
<ul><li><strong>Subject</strong> changed from <i>auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"</i> to <i>auto_review:"setup failure: Cache service status error from API: Minion job .* failed: Can't use an undefined value as a HASH reference at.*"</i></li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3358722020-10-14T08:03:05ZXiaojing_liuxliu1@suse.com
<ul><li><strong>Subject</strong> changed from <i>auto_review:"setup failure: Cache service status error from API: Minion job .* failed: Can't use an undefined value as a HASH reference at.*"</i> to <i>auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"</i></li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3359862020-10-14T10:34:40Zokurzokurz@suse.com
<ul><li><strong>Target version</strong> set to <i>Ready</i></li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3367392020-10-17T20:04:23Zokurzokurz@suse.com
<ul><li><strong>Category</strong> set to <i>Regressions/Crashes</i></li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3375612020-10-20T13:05:43Zokurzokurz@suse.com
<ul><li><strong>Tags</strong> set to <i>cache, worker, minion</i></li><li><strong>Description</strong> updated (<a title="View differences" href="/journals/337561/diff?detail_id=332383">diff</a>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Workable</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Low</i></li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3386472020-10-21T10:50:54Zkraihsebastian.riedel@suse.com
<ul><li><strong>Assignee</strong> set to <i>kraih</i></li></ul><p>Curious, this is not something that should be possible. I'll have a closer look.</p>
openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3440202020-10-27T14:26:10Zkraihsebastian.riedel@suse.com
<ul><li><strong>Status</strong> changed from <i>Workable</i> to <i>Feedback</i></li></ul><p>The only condition where i could see this error happening would be if the SQLite database gets deleted right after the job started. Unfortunately i was too late with the investigation, so that the database had already been deleted again a few days after the error occurred. To be sure i've also double checked the <code>Minion::Backend::SQLite</code> code, and it looks fine. This was just bad timing, the SQLite file was deleted before the cache service was stopped.</p>
openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3441822020-10-27T16:51:25Zokurzokurz@suse.com
<ul></ul><p>Ok, understood. Would it be possible to just avoid the Perl warning in this case? Something that is a bit more explicit than "Can't use an undefined value"?</p>
openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3536622020-11-24T12:42:11Zkraihsebastian.riedel@suse.com
<ul></ul><p>It's not a warning but an exception that got thrown when an unexpected condition occurred in the Minion job process. It's not the best error message, but appropriate enough for what happened. Have we actually seen this more than once? Otherwise i'd just say good enough and leave it as is.</p>
openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3536642020-11-24T12:45:32Zkraihsebastian.riedel@suse.com
<ul></ul><p>Pretty sure the underlying cause for this is our ongoing fight with SQLite corruption, since that's the only case where we delete the SQLite file. Finding a solution for that will probably make this condition impossible.</p>
openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3539382020-11-25T11:58:45Zokurzokurz@suse.com
<ul><li><strong>Parent task</strong> set to <i>#62420</i></li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3541522020-11-25T20:34:38Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-6 priority-high2 closed child" href="/issues/67000">action #67000</a>: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service status error.*(database disk image is malformed|Specified job ID is invalid).*":retry</i> added</li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3541542020-11-25T20:38:56Zokurzokurz@suse.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/354154/diff?detail_id=351246">diff</a>)</li></ul><p>kraih wrote:</p>
<blockquote>
<p>Have we actually seen this more than once?</p>
</blockquote>
<p>Good question. I have added "steps to reproduce" to find any other cases where we linked openQA jobs to this ticket as we can do with all "auto_review" tickets. Did <code>for host in o3 osd; do echo "### $host" && openqa-query-for-job-label poo#73339; done</code> and found:</p>
<pre><code>### o3
1469527|2020-11-13 06:04:23|done|incomplete|gnome|setup failure: Cache service status error from API: Minion job #31656 failed: Can't use an undefined value as a HASH reference at /usr/share/openqa/script/../lib/OpenQA/CacheService/Task/Asset.pm line 30.
|openqaworker1
1469499|2020-11-13 05:51:32|done|incomplete|krypton-live-wayland|setup failure: Cache service status error from API: Minion job #31621 failed: Can't use an undefined value as a HASH reference at /usr/share/openqa/script/../lib/OpenQA/CacheService/Task/Asset.pm line 30.
|openqaworker1
1469435|2020-11-13 04:32:22|done|incomplete|upgrade_staging|setup failure: Cache service status error from API: Minion job #31546 failed: Can't use an undefined value as a HASH reference at /usr/share/openqa/script/../lib/OpenQA/CacheService/Task/Asset.pm line 30.
|openqaworker1
1469409|2020-11-13 03:34:19|done|incomplete|minimalx|setup failure: Cache service status error from API: Minion job #31529 failed: Can't use an undefined value as a HASH reference at /usr/share/openqa/script/../lib/OpenQA/CacheService/Task/Asset.pm line 30.
|openqaworker1
### osd
1469527|2020-11-13 06:04:23|done|incomplete|gnome|setup failure: Cache service status error from API: Minion job #31656 failed: Can't use an undefined value as a HASH reference at /usr/share/openqa/script/../lib/OpenQA/CacheService/Task/Asset.pm line 30.
|openqaworker1
1469499|2020-11-13 05:51:32|done|incomplete|krypton-live-wayland|setup failure: Cache service status error from API: Minion job #31621 failed: Can't use an undefined value as a HASH reference at /usr/share/openqa/script/../lib/OpenQA/CacheService/Task/Asset.pm line 30.
|openqaworker1
1469435|2020-11-13 04:32:22|done|incomplete|upgrade_staging|setup failure: Cache service status error from API: Minion job #31546 failed: Can't use an undefined value as a HASH reference at /usr/share/openqa/script/../lib/OpenQA/CacheService/Task/Asset.pm line 30.
|openqaworker1
1469409|2020-11-13 03:34:19|done|incomplete|minimalx|setup failure: Cache service status error from API: Minion job #31529 failed: Can't use an undefined value as a HASH reference at /usr/share/openqa/script/../lib/OpenQA/CacheService/Task/Asset.pm line 30.
|openqaworker1
</code></pre>
<p>kraih wrote:</p>
<blockquote>
<p>Pretty sure the underlying cause for this is our ongoing fight with SQLite corruption, since that's the only case where we delete the SQLite file. Finding a solution for that will probably make this condition impossible.</p>
</blockquote>
<p>Definitely a good idea. I have linked <a class="issue tracker-4 status-3 priority-6 priority-high2 closed child" title="action: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service... (Resolved)" href="https://progress.opensuse.org/issues/67000">#67000</a> here. Unless you plan work for this ticket in particular I recommend you set the status to "Blocked" and check the situation again as soon as we have <a class="issue tracker-4 status-3 priority-6 priority-high2 closed child" title="action: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service... (Resolved)" href="https://progress.opensuse.org/issues/67000">#67000</a> resolved.</p>
openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3555642020-11-30T11:01:43Zkraihsebastian.riedel@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Blocked</i></li></ul> openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3561862020-12-03T11:56:18Zokurzokurz@suse.com
<ul></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/23018">@kraih</a> as we consider the underlying issue <a class="issue tracker-4 status-3 priority-6 priority-high2 closed child" title="action: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service... (Resolved)" href="https://progress.opensuse.org/issues/67000">#67000</a> solved I checked <a href="https://progress.opensuse.org/issues/73339#Steps-to-reproduce" class="external">https://progress.opensuse.org/issues/73339#Steps-to-reproduce</a> and found no reference of the issue more recent than 2020-11-13 . Do you have plans to improve the error handling and e.g. prevent the perl warnings mentioned in the initial ticket description? If you see this is not feasible or useful then you can set the ticket to "Resolved".</p>
openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3791772021-01-19T13:00:21Zkraihsebastian.riedel@suse.com
<ul><li><strong>Status</strong> changed from <i>Blocked</i> to <i>Feedback</i></li></ul><p>okurz wrote:</p>
<blockquote>
<p><a class="user active user-mention" href="https://progress.opensuse.org/users/23018">@kraih</a> as we consider the underlying issue <a class="issue tracker-4 status-3 priority-6 priority-high2 closed child" title="action: Job incompletes due to malformed worker cache database disk image with auto_review:"Cache service... (Resolved)" href="https://progress.opensuse.org/issues/67000">#67000</a> solved I checked <a href="https://progress.opensuse.org/issues/73339#Steps-to-reproduce" class="external">https://progress.opensuse.org/issues/73339#Steps-to-reproduce</a> and found no reference of the issue more recent than 2020-11-13 . Do you have plans to improve the error handling and e.g. prevent the perl warnings mentioned in the initial ticket description? If you see this is not feasible or useful then you can set the ticket to "Resolved".</p>
</blockquote>
<p>Thanks, that suggests we might have resolved the issue together with the SQLite corruption (as expected). It's an exception, not a warning. And i don't think there is any need for further changes. It's a very unusual error and it was properly shown in the logs. I believe we can consider this issue resolved.</p>
openQA Project - action #73339: auto_review:"setup failure: Cache service status error from API: Minion job.* failed: Can't use an undefined value as a HASH reference at.*"https://progress.opensuse.org/issues/73339?journal_id=3792042021-01-19T14:08:39Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>agreed</p>