QA » openQA Project

the sysread error is from the commands thread which is shut down. I don't think this is related at all to corrupt images

Of course the checksum in db is calculated on webui side:
1) Worker uploads the file
2) Worker starts to calculate the local checksum.
3) After WebUI receives the file, webUI starts to calculate the checksum and store to db
4) Worker queries webUI for asset info and compares the checksum.

If checksum mismatch .. fail the job, or invalidate asset and trigger reupload.

this clearly has the advantage that we can reuse the checksum also if we want to replace NFS one point in the future :)

I would say calculation should be done by the worker because it's a "number crunching" task. So yet another data entry that should be added to some json file and uploaded.

Do you mean something like this?

Job A:
1) worker calculates checksum
2) worker uploads image + expected checksum

Job B:
1) worker calculates checksum of required asset
2) in case of mismatch abort the job

At the end it does the same effect as using wrong asset IMO.

Or wait until the asset is back-propagated through shared storage and check the checksum there?

The load on the webUI is indeed a concern, esp. with higher worker count. We have about 6 mojo processes thanks to prefork => at max 6 concurrent large image uploads (md5 takes ~45s for 5G image on i7-3820@3.6GHz).

Or let GRU calculate checksum, in that case however I don't see how we can prevent running chained jobs with wrong asset.

well, the job is only done when the checksum is calculated - but getting a GRU slot can take a while. I want the job to fail when the upload failed - and not the chained jobs.

So let's redefine this a bit:

• worker uploads assets_public/FOO.qcow2
• openQA puts it with temporary filename in /var/lib/openqa/share and tells worker resulting filename
• worker calculates crc32 (md5 is really the worst choice when speed wins over security) of assets_public/FOO.qcow2 and /var/lib/openqa/FOO-temp.qcow2
• if it matches, tells openqa to rename
• if it mismatches, job fails with tons of debug :)

Yes, just let the workers calculate the checksum synchronously at the end of job. It does not take ages, it only happens for publishing jobs. The worker should do it because it scales better this way. If we have more jobs and/or more workers then it will still work w/o overloading the webui host (which should be just webui, at best).

Ok, what about the plan to obsolete NFS at some point?

Right, md5 is slow. cksum gives ~14s, sha1sum ~16s for the same image.

if we obsolete NFS, we will have to redownload the asset to validate it - we do the same now basically

I had the hope that having unique file names would make it less likely, but it's not the case ;(

https://openqa.opensuse.org/tests/237472/file/autoinst-log.txt
says:
sha1sum 1addfc7b73ae49f69dbd6eb3941586a10ffbdd06 *assets_public/opensuse-42.1-x86_64-Updates-20160806-2-cryptlvm@uefi.qcow2
but after upload it's
sha1sum 8576693e5032177c1d26ac9fde21af465fe26363 opensuse-42.1-x86_64-Updates-20160806-2-cryptlvm@uefi.qcow2

https://github.com/os-autoinst/openQA/pull/826 implements what I described in #12

uploading SLES-12-SP2-ppc64le-ha.qcow2
ERROR SLES-12-SP2-ppc64le-ha.qcow2: Connection error: Premature connection close

in https://openqa.suse.de/tests/523391

https://openqa.suse.de/tests/522879
https://openqa.suse.de/tests/522943
https://openqa.suse.de/tests/523463
https://openqa.suse.de/tests/522868

latest failing example:
https://openqa.suse.de/tests/528417

(wrong ticket referenced)

yup, after updating the Fedora staging deployment to current git, I immediately got a couple of jobs with failed disk image uploads:

https://openqa.stg.fedoraproject.org/tests/33982
https://openqa.stg.fedoraproject.org/tests/34004

Note, we are still running on Mojolicious 6. I will try reverting f2547e9bcc0a166f993426bceeacd00179116716 , I guess.

https://github.com/os-autoinst/openQA/commit/a50e86ac6654bea9ec0fd32b3b132fb162a79c39 was not sufficient to fix this for our staging deployment (again, remember we're still running on Mojo 6), but along with a further patch that completes the reversion of https://github.com/os-autoinst/openQA/commit/f2547e9bcc0a166f993426bceeacd00179116716 (restoring the submission of the request via $OpenQA::Worker::Common::ua->start() and the recursive IOLoop entry bit), our uploads seem to be working properly again.

Per http://mojolicious.org/perldoc/Mojolicious/Guides/FAQ#What-does-Inactivity-timeout-mean , 'premature connection close' could be caused by Mojo's default inactivity timeouts. I'm testing to see if this helps:

From 735d487fb0cb61cd8dbe9715cd6838254e9165d4 Mon Sep 17 00:00:00 2001
From: Adam Williamson <awilliam@redhat.com>
Date: Wed, 31 Aug 2016 08:59:15 -0700
Subject: [PATCH] go back to blocking post for upload, extend inactivity
 timeout

---
 lib/OpenQA/Worker/Jobs.pm | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/lib/OpenQA/Worker/Jobs.pm b/lib/OpenQA/Worker/Jobs.pm
index 2364945..0a43045 100644
--- a/lib/OpenQA/Worker/Jobs.pm
+++ b/lib/OpenQA/Worker/Jobs.pm
@@ -138,15 +138,12 @@ sub upload {
     my $ua_url = $OpenQA::Worker::Common::url->clone;
     $ua_url->path("jobs/$job_id/artefact");

-    my $tx = $OpenQA::Worker::Common::ua->build_tx(POST => $ua_url => form => $form);
-    # override the default boundary calculation - it reads whole file
-    # and it can cause various timeouts
-    my $ct = $tx->req->headers->content_type;
-    my $boundary = encode_base64 join('', map chr(rand 256), 1 .. 32);
-    $boundary =~ s/\W/X/g;
-    $tx->req->headers->content_type("$ct; boundary=$boundary");
-
-    my $res = $OpenQA::Worker::Common::ua->start($tx);
+    # Uploading multi-GB files takes time: extend the inactivity timeout
+    my $origto = $OpenQA::Worker::Common::ua->inactivity_timeout;
+    $OpenQA::Worker::Common::ua->inactivity_timeout(300);
+    my $res = $OpenQA::Worker::Common::ua->post($ua_url => form => $form);
+    # reset inactivity timeout
+    $OpenQA::Worker::Common::ua->inactivity_timeout($origto);

     if (my $err = $res->error) {
         my $msg;
-- 
2.9.3

note you're actually getting a lot of "Premature connection close" incompletions on openqa.opensuse.org , but a lot of them get auto-duplicated so maybe it's not totally obvious. But you can query for it:

https://openqa.opensuse.org/api/v1/jobs?result=incomplete&maxage=36000

is just in the last 10 hours. A lot of them seem to be hitting 'Premature connection close' on fairly innocuous-looking file uploads...they often seem to fail on consoletest_setup-loadavg_consoletest_setup.txt , which is like 20 bytes.

https://openqa.suse.de/tests/569766/file/autoinst-log.txt

I don't think any of the fixes are deployed