openQA Project (public)

working on that within the mob session.

Triggered a rerun of the "last good" to see if it still works:

openqa-clone-job --within-instance https://openqa.opensuse.org/tests/2833511 BUILD= TEST=textmode-okurz _GROUP=0

-> Created job #2851630: opensuse-Tumbleweed-DVD-s390x-Build20221024-textmode@s390x-zVM-vswitch-l2 -> https://openqa.opensuse.org/t2851630

https://openqa.opensuse.org/tests/2842380#step/textinfo/10 shows that the curl command does not finish within a reasonable time. But later curl commands for smaller log files do succeed. Using the SSH config that is visible in https://openqa.opensuse.org/tests/2842380#step/bootloader_s390/17 we interactively logged into the test https://openqa.opensuse.org/tests/2851630# while it was running. The test by the way reproduced the problem so very likely the problem was introduced in the past days when dheidler and mkittler worked on forwarding the cache service into the s390x worker containers. Following the autoinst-log.txt file while we tried uploads we found:

Mojo::Reactor::Poll: I/O watcher failed: Error in tempfile() using template /var/lib/openqa/cache/tmp/mojo.tmp.XXXXXXXXXXXXXXXX: Could not create temp file /var/lib/openqa/cache/tmp/mojo.tmp.mbmTdPv6NyxTOBTy: Permission denied at /usr/lib/perl5/5.26.1/File/Temp.pm line 1102.
        File::Temp::tempfile("mojo.tmp.XXXXXXXXXXXXXXXX", "DIR", "/var/lib/openqa/cache/tmp") called at /usr/lib/perl5/5.26.1/File/Temp.pm l
ine 898
        File::Temp::new("File::Temp", "DIR", "/var/lib/openqa/cache/tmp", "TEMPLATE", "mojo.tmp.XXXXXXXXXXXXXXXX", "UNLINK", 0) called at /u
sr/lib/perl5/vendor_perl/5.26.1/Mojo/File.pm line 153
        Mojo::File::tempfile("DIR", "/var/lib/openqa/cache/tmp", "TEMPLATE", "mojo.tmp.XXXXXXXXXXXXXXXX", "UNLINK", 0) called at /usr/lib/pe
rl5/vendor_perl/5.26.1/Mojo/Asset/File.pm line 23
        Mojo::Asset::File::__ANON__(Mojo::Asset::File=HASH(0x562be3256808)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Base.pm line 78
        Mojo::Asset::File::handle(Mojo::Asset::File=HASH(0x562be3256808)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Asset/File.pm line 41

so the problem is unable to create temp files. Taking a look into the original job's log file we find
https://openqa.opensuse.org/tests/2842380/logfile?filename=autoinst-log.txt

Mojo::Reactor::Poll: I/O watcher failed: Error in tempfile() using template /var/lib/openqa/cache/tmp/mojo.tmp.XXXXXXXXXXXXXXXX: Could not create temp file /var/lib/openqa/cache/tmp/mojo.tmp.56k5Q5xGI4tiAW9g: Permission denied at /usr/lib/perl5/5.26.1/File/Temp.pm line 1102.
    File::Temp::tempfile("mojo.tmp.XXXXXXXXXXXXXXXX", "DIR", "/var/lib/openqa/cache/tmp") called at /usr/lib/perl5/5.26.1/File/Temp.pm line 898
    File::Temp::new("File::Temp", "DIR", "/var/lib/openqa/cache/tmp", "TEMPLATE", "mojo.tmp.XXXXXXXXXXXXXXXX", "UNLINK", 0) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/File.pm line 153
    Mojo::File::tempfile("DIR", "/var/lib/openqa/cache/tmp", "TEMPLATE", "mojo.tmp.XXXXXXXXXXXXXXXX", "UNLINK", 0) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Asset/File.pm line 23

so the exact same error

Error message we see on the worker:

openqaworker1_container:/var/lib/openqa/pool/104 # tail -f autoinst-log.txt                                                              

        autotest::__ANON__(Mojo::IOLoop::ReadWriteProcess=HASH(0x562be31010a8)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/R
eadWriteProcess.pm line 326
        eval {...} called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 326
        Mojo::IOLoop::ReadWriteProcess::_fork(Mojo::IOLoop::ReadWriteProcess=HASH(0x562be31010a8), CODE(0x562be31d27c8)) called at /usr/l
ib/perl5/vendor_perl/5.26.1/Mojo/IOLoop/ReadWriteProcess.pm line 488
        Mojo::IOLoop::ReadWriteProcess::start(Mojo::IOLoop::ReadWriteProcess=HASH(0x562be31010a8)) called at /usr/lib/os-autoinst/autotes
t.pm line 296
        autotest::start_process() called at /usr/bin/isotovideo line 273

[2022-11-03T13:31:36.497328Z] [debug] ||| finished logs_from_installation_system installation (runtime: 450 s)
[2022-11-03T13:31:36.497458Z] [debug] ||| post fail hooks runtime: 326 s
[2022-11-03T13:31:36.502084Z] [debug] ||| starting reboot_after_installation tests/installation/reboot_after_installation.pm
[2022-11-03T13:31:36.504036Z] [debug] isotovideo: pausing test execution of installation-reboot_after_installation because we're supposed
 to pause at this test module
Mojo::Reactor::Poll: I/O watcher failed: Error in tempfile() using template /var/lib/openqa/cache/tmp/mojo.tmp.XXXXXXXXXXXXXXXX: Could no
t create temp file /var/lib/openqa/cache/tmp/mojo.tmp.mbmTdPv6NyxTOBTy: Permission denied at /usr/lib/perl5/5.26.1/File/Temp.pm line 1102
.
        File::Temp::tempfile("mojo.tmp.XXXXXXXXXXXXXXXX", "DIR", "/var/lib/openqa/cache/tmp") called at /usr/lib/perl5/5.26.1/File/Temp.pm line 898
        File::Temp::new("File::Temp", "DIR", "/var/lib/openqa/cache/tmp", "TEMPLATE", "mojo.tmp.XXXXXXXXXXXXXXXX", "UNLINK", 0) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/File.pm line 153
        Mojo::File::tempfile("DIR", "/var/lib/openqa/cache/tmp", "TEMPLATE", "mojo.tmp.XXXXXXXXXXXXXXXX", "UNLINK", 0) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Asset/File.pm line 23
        Mojo::Asset::File::__ANON__(Mojo::Asset::File=HASH(0x562be3256808)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Base.pm line 78
        Mojo::Asset::File::handle(Mojo::Asset::File=HASH(0x562be3256808)) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Asset/File.pm line 41
        Mojo::Asset::File::add_chunk(Mojo::Asset::File=HASH(0x562be3256808), "BZh91AY\x{26}SY!\x{ac}1\"\x{3}\x{19}\x{14}\x{7f}\x{ff}\x{ff
}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{f
f}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{ff}\x{e6}\x{ed}\x{df}\x{2}\x{dd}\x{b7}{}\x{cf}3"...) called at /usr/lib/perl5/vendor_perl/
5.26.1/Mojo/Asset/Memory.pm line 18
        Mojo::Asset::Memory::add_chunk(Mojo::Asset::Memory=HASH(0x562be325cb18), "I\x{81}\x{b0}-\x{1c}\x{fd}\x{a7}r\x{f2}*\x{88}\x{be}\x{
a1}\x{dc}\x{8f}\x{e3}m\x{aa}t\x{cd}\x{81}\x{bc}V\x{c}\x{97}\x{c5}\x{f4}u\x{9b}0\x{a2}o\"\x{d4}G)\x{1b}\x{8a}J\x{ea}N\x{fa}cj\x{88}8a\x{df
}V{\x{1c}\x{83},\x{ed}[\x{9c}h\x{8e}\x{85}\x{a4}\x{be}"...) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/Content/Single.pm line 32
        Mojo::Content::Single::__ANON__(Mojo::Content::Single=HASH(0x562be2eb3b80), "I\x{81}\x{b0}-\x{1c}\x{fd}\x{a7}r\x{f2}*\x{88}\x{be}
\x{a1}\x{dc}\x{8f}\x{e3}m\x{aa}t\x{cd}\x{81}\x{bc}V\x{c}\x{97}\x{c5}\x{f4}u\x{9b}0\x{a2}o\"\x{d4}G)\x{1b}\x{8a}J\x{ea}N\x{fa}cj\x{88}8a\x
{df}V{\x{1c}\x{83},\x{ed}[\x{9c}h\x{8e}\x{85}\x{a4}\x{be}"...) called at /usr/lib/perl5/vendor_perl/5.26.1/Mojo/EventEmitter.pm line 15
        Mojo::EventEmitter::emit(Mojo::Content::Single=HASH(0x562be2eb3b80), "read", "I\x{81}\x{b0}-\x{1c}\x{fd}\x{a7}r\x{f2}*\x{88}\x{be

https://github.com/os-autoinst/openQA/pull/4884

https://openqa.opensuse.org/tests/2851985#live is a verification run to show the code in hot-patched production.

https://openqa.opensuse.org/tests/2851985#step/logs_from_installation_system/12 shows the test passing the upload attempt.

We should still ensure that the current cache service implementation does the efficient approach using a temporary directory on the same filesystem where final assets end up.

Actually we verified that https://github.com/os-autoinst/openQA/blob/master/lib/OpenQA/CacheService/Model/Cache.pm#L113 does that with

my $downloader = $self->downloader->log($log)->tmpdir($self->_realpath->child('tmp')->to_string);

so all good.

PR merged. https://openqa.opensuse.org/tests/2864552 is passed