action #19436
closed[tools]cache download suddently interrupts within the output of a log message
0%
Description
observation¶
job https://openqa.suse.de/tests/967456/file/autoinst-log.txt is incomplete. Message:
Downloading SLES-12-gnome-x86_64-gpt.img from http://openqa.suse.de/tests/967456/asset/hdd/SLES-12-gnome-x86_64-gpt.img
CACHE: Size of Mojo::Asset::Memory=HASH(0x1002b187028) differs, Expected: 245760 / Downloaded:
+++ worker notes +++
I checked the hard disk space on qa-power8-5-kvm.qa and it's not full.
Checked journalctl -u openqa-worker@5 for the job related output
May 30 09:54:35 QA-Power8-5-kvm worker[118633]: [INFO] got job 967456: 00967456-sle-12-SP3-Server-DVD-ppc64le-Build0405-gpt@ppc64le
May 30 09:54:35 QA-Power8-5-kvm worker[118633]: [DEBUG] CACHE: Health: Real size: 52576958976, Configured limit: 53687091200
May 30 09:54:35 QA-Power8-5-kvm worker[118633]: [INFO] OpenQA::Worker::Cache: Initialized with http://openqa.suse.de at /var/lib/openqa/cache, current size is
May 30 09:54:35 QA-Power8-5-kvm worker[118633]: [DEBUG] Found HDD_1, caching SLES-12-gnome-x86_64-gpt.img
May 30 09:54:35 QA-Power8-5-kvm worker[118633]: [DEBUG] CACHE: Aquiring lock for /var/lib/openqa/cache/SLES-12-gnome-x86_64-gpt.img in the database
May 30 09:54:54 QA-Power8-5-kvm worker[118633]: Use of uninitialized value $size in numeric eq (==) at /usr/share/openqa/script/../lib/OpenQA/Worker/Cache.pm
May 30 09:54:54 QA-Power8-5-kvm worker[118633]: Use of uninitialized value $size in concatenation (.) or string at /usr/share/openqa/script/../lib/OpenQA/Work
May 30 09:54:54 QA-Power8-5-kvm worker[118633]: [WARN] job is missing files, releasing job
May 30 09:54:56 QA-Power8-5-kvm worker[118633]: [INFO] cleaning up 00967456-sle-12-SP3-Server-DVD-ppc64le-Build0405-gpt@ppc64le
problem¶
first time seen. Remind me: Why on earth is "use of uninitialized value in numeric eq" only a warning again in perl?
Problem is already in the lines above that when it says "current size is" and the nothing
suggestion¶
- Make checking for variables defined stricter
- Find out what the real error is
Updated by EDiGiacinto almost 7 years ago
Not sure if strictly related to this issue, or should be a separate bug, but i got this on a local instance:
May 30 15:12:29 dhcp88.suse.cz worker[13639]: Use of uninitialized value in subroutine entry at /usr/share/openqa/script/../lib/OpenQA/Worker.pm line 80.
(wild guess) It could be that stats calls are failing because can't find the actual asset.
Mojo::URL will silently return undef anyway if the supplied argument doesn't look like an url, opened a PR for it in such case: https://github.com/os-autoinst/openQA/pull/1352
Updated by okurz almost 7 years ago
- Status changed from New to In Progress
https://openqa.suse.de/tests/971554/file/autoinst-log.txt
the mentioned PR is not yet deployed on that worker
Updated by nicksinger almost 7 years ago
- Copied to action #19526: [tools] Tests incomplete if asset download fails even the first time added
Updated by okurz almost 7 years ago
Updated by szarate almost 7 years ago
Messages with:
CACHE: Size of Mojo::Asset::Memory=HASH(0x10044440988) differs, Expected: 245760 / Downloaded:
Should be addressed by this pr
Updated by RBrownSUSE almost 7 years ago
- Project changed from openQA Project to openQA Tests
- Category deleted (
Regressions/Crashes)
File SLES-12-gnome-x86_64-gpt.img is currently 240kB
Looks like whatever created that image is broken and the image needs to be repaired - moving the test to "openQA tests" as the Tool appears to be doing it's job in this case (phew)
Updated by asmorodskyi almost 7 years ago
- Category set to Infrastructure
last good: https://openqa.suse.de/tests/937937
Updated by okurz almost 7 years ago
- Project changed from openQA Tests to openQA Project
- Subject changed from cache download suddently interrupts within the output of a log message to [tools]cache download suddently interrupts within the output of a log message
- Category changed from Infrastructure to Regressions/Crashes
- Assignee set to szarate
crosschecked with asmorodskyi and discussed with rbrown, szarate: Apparently the disk image was never changed and is fine. It's a sparse qcow image containing a GPT partitioning table that is used for the purpose of testing that installation on GPT can run fine on ppc64le so the initial report was ok as well. The test is not scheduled on x86_64 but for easier reproduction I suggest to try exactly this and then see if the same error happens there what I assume will happen.
Updated by szarate almost 7 years ago
I can't really pinpoint 100% of the problem, but seems to be related to the way mojolicious handles downloads when the filesize is under 256K, changing it into:
my $tmpasset = $tx->res->content->asset->move_to($asset);
my $file = Mojo::Asset::File->new(path => $asset);
my $size = (stat $file->path)[7];
Seems to fix the problem. I have a chat in the #mojo channel to figure out possible solutions (Another one is changing the MOJO_MAX_MEMORY_SIZE)
Updated by asmorodskyi almost 7 years ago
As temporary solution I create another img for 17 TB which now has size 464 KB so according to @szarate it suppose to workaround issue until it will be really fixed
Updated by szarate almost 7 years ago
PR: https://github.com/os-autoinst/openQA/pull/1359 will address this issue
Updated by szarate almost 7 years ago
- Status changed from In Progress to Resolved