https://progress.opensuse.org/https://progress.opensuse.org/themes/openSUSE/favicon/favicon.ico?15829177842018-06-25T14:33:31ZopenSUSE Project Management ToolopenQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1322732018-06-25T14:33:31Znicksingernsinger@suse.com
<ul><li><strong>Subject</strong> changed from <i>[functional][s390x][medium] test fails in execute_test_run because it cannot handle broken pipes</i> to <i>[functional][medium][u] test fails in execute_test_run because it cannot handle broken pipes</i></li></ul> openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1325242018-06-26T12:40:39Zokurzokurz@suse.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-4 status-3 priority-4 priority-default closed" href="/issues/34012">action #34012</a>: [kernel] too generic test failure in "execute_test_run" for stress tests, was previously something more specific like "acceptance_fs_stress"</i> added</li></ul> openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1325302018-06-26T12:41:59Zokurzokurz@suse.com
<ul><li><strong>Subject</strong> changed from <i>[functional][medium][u] test fails in execute_test_run because it cannot handle broken pipes</i> to <i>[kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipes</i></li><li><strong>Assignee</strong> set to <i>yosun</i></li></ul><p>Hi <a class="user active user-mention" href="https://progress.opensuse.org/users/15720">@yosun</a>, as discussed in <a class="issue tracker-4 status-3 priority-4 priority-default closed" title="action: [kernel] too generic test failure in "execute_test_run" for stress tests, was previously somethin... (Resolved)" href="https://progress.opensuse.org/issues/34012">#34012</a> I assume you want to pick it up?</p>
openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1326562018-06-27T02:23:48Zyosunyosun@suse.com
<ul><li><strong>Assignee</strong> changed from <i>yosun</i> to <i>okurz</i></li></ul><p>Thanks for the info.<br>
This failed by "packet_write_wait: Connection to 10.161.145.16 port 22: Broken pipe", and checked serial log don't have any oops and crash info in it. It fails when doing "/usr/share/qa/tools/file_copy -j 4 -i 5 -s 5000", which means run in 4 parallels, iteration 5 times with 5000MB(5GB) files copy in 1 time. In this high fs stress, and this test KVM only has 912MB RAM, it's easy to cause OOM then system ramdom kill process, and then ssh service or related process being killed randomly.<br>
Log just shows it lose ssh connect, neither a kernel nor test issue but a random issue. The solution is give larger RAM or harddisk for this poor s390x KVM to avoid/reduce this kind of issue. We have reported related issue as bug to Lab team, but Lab's developer didn't take it because of resource too limited.<br>
In all, I suggest solve it in tools team to give more resources to this KVM or just reject this kind of ticket.</p>
openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1327762018-06-27T11:09:44Zokurzokurz@suse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Feedback</i></li></ul><p>yosun wrote:</p>
<blockquote>
<p>We have reported related issue as bug to Lab team, but Lab's developer didn't take it because of resource too limited.</p>
</blockquote>
<p>What "Lab's developer" are you referring to? Do you have a ticket for that?</p>
<blockquote>
<p>In all, I suggest solve it in tools team to give more resources to this KVM or just reject this kind of ticket.</p>
</blockquote>
<p>I don't think this is related to the tools team because when we talk about KVM we just configure the machine accordingly. We could do that but I want to wait for your response first.</p>
openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1332862018-06-28T19:41:39Zokurzokurz@suse.com
<ul><li><strong>Target version</strong> set to <i>future</i></li></ul> openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1356742018-07-17T06:33:12Zyosunyosun@suse.com
<ul></ul><p>I was wrong, it's not failed by OOM issue. I check test code and log in <a href="https://openqa.suse.de/tests/1777070/file/autoinst-log.txt" class="external">https://openqa.suse.de/tests/1777070/file/autoinst-log.txt</a> again, I found this test randomly fail by following line timeout in 90 second:<br>
assert_script_run("tar cjf $tarball -C /var/log/qa/ctcs2 <code>ls /var/log/qa/ctcs2/</code>");</p>
<p>It only fail after test fs_stress, and this line just after test finish. I guess after fs stress, system need more time to get enough space to create a log tarbal. I think the solution is add following lines before fail part:<br>
if (get_var("QA_TESTSUITE")=="fs_stress") {<br>
sleep 120;<br>
}</p>
openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1360792018-07-17T15:56:21Zokurzokurz@suse.com
<ul></ul><p>I guess rather than this big sleep time we should wait for what we really need, e.g. look for free space and wait until there is more free space again. Or just save to a different location, e.g. ram disk /dev/shm</p>
openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1418302018-08-10T07:50:42Zokurzokurz@suse.com
<ul><li><strong>Assignee</strong> changed from <i>okurz</i> to <i>yosun</i></li></ul><p><a class="user active user-mention" href="https://progress.opensuse.org/users/15720">@yosun</a> anything else what I could help with here?</p>
openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1420102018-08-13T02:58:40Zyosunyosun@suse.com
<ul></ul><p>It's helpful in <a class="issue tracker-5 status-5 priority-4 priority-default closed child behind-schedule" title="analysis: Decide Amount (Closed)" href="https://progress.opensuse.org/issues/8">#8</a>, thanks! But I still didn't find time to work on it. Maybe I could work on it in SLE12SP4 RC period.</p>
openQA Tests - action #37782: [kernel][functional][u][medium] test fails in execute_test_run because it cannot handle broken pipeshttps://progress.opensuse.org/issues/37782?journal_id=1643722018-11-13T09:44:19Zyosunyosun@suse.com
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Resolved</i></li></ul><p>Fixed with add a sync before tar logs. I tried some tests not reproduce this issue. Feel free to reopen it, when happen again.</p>