QA » openQA Project

It could be helpful to post more links to tests with the same behaviour (could be that there is a pattern)

tinita wrote:

It could be helpful to post more links to tests with the same behaviour (could be that there is a pattern)

I would post more links if I had any but this is maybe the third time I'm seeing this bug and the last one I've seen before happened months ago. But you could probably find a few dozen cases by scanning existing test results with a script. The details-*.json file will have text keys but no text_data in the details array.

Maybe this should be self-evident, but can you clarify that https://openqa.suse.de/tests/6308588#step/cn_pec_sh/1 is expected to pass even without the missing file? What is the expected result?

From the logs it seems the upload is completely missing so my first thought would be that the file wasn't created. But from a single result it's hard to tell. I guess that's also what Tina was wondering.

To put the information into structured form I suggest to follow the ticket template from https://progress.opensuse.org/projects/openqav3/wiki/#Defects

cdywan wrote:

Maybe this should be self-evident, but can you clarify that https://openqa.suse.de/tests/6308588#step/cn_pec_sh/1 is expected to pass even without the missing file? What is the expected result?

Yes, this is expected to pass. The command result (pass/fail) is reported to OpenQA database first and the console output is uploaded much later. It is expected to see a green box with "Unable to read foo-123.txt" while the test is still running but not after it finishes.

From the logs it seems the upload is completely missing so my first thought would be that the file wasn't created. But from a single result it's hard to tell. I guess that's also what Tina was wondering.

Like I said, the easiest way to find more examples is to scan existing test results with a script. Valid details-*.json file looks like this:

{"dents":0,"details":[{"result":"ok","text":"boot_ltp-4.txt","text_data":"# wait_serial expected: qr\/login:\\s*$\/ui\n\n# Result:\n\r\nsusetest login: \n","title":"wait_serial"},...],...}

But when it's affected by this issue, the text key will be present but the text_data key will be completely missing:

{"dents":0,"details":[{"result":"ok","text":"boot_ltp-4.txt","title":"wait_serial"},...],...}

More thoughts:

lib/OpenQA/Task/Job/FinalizeResults.pm#L26 uploads results in a minion job.
Maybe errors are eaten here but should appear on https://openqa.suse.de/minion

cdywan wrote:

More thoughts:

lib/OpenQA/Task/Job/FinalizeResults.pm#L26 uploads results in a minion job.
Maybe errors are eaten here but should appear on https://openqa.suse.de/minion

No, this is a minion task which collects the already uploaded text files and inlines them into details-*.json file for faster webpage rendering. It does not upload anything but it can detect this problem and send an alert somehow. The uploading is done by the OpenQA worker which actually runs the test job.

MDoucha wrote:

cdywan wrote:

More thoughts:

lib/OpenQA/Task/Job/FinalizeResults.pm#L26 uploads results in a minion job.
Maybe errors are eaten here but should appear on https://openqa.suse.de/minion

No, this is a minion task which collects the already uploaded text files and inlines them into details-*.json file for faster webpage rendering. It does not upload anything but it can detect this problem and send an alert somehow. The uploading is done by the OpenQA worker which actually runs the test job.

Thank you for the correction!

FYI @kraih helping out by improving related code: https://github.com/os-autoinst/openQA/pull/3982 Not a fix but could contribute to errors not being propagated

This sounds like a bug in the uploading code of the openQA worker (code in lib/OpenQA/Worker/Job.pm). Hard to tell what's going on exactly (the VPN is down so I cannot even access the referenced jobs right now). (It could be that the results are already not generated correctly by os-autoinst, though. However, that's not that likely.)

It would of course be useful to see the pool directory of such a job. Is the test order of such jobs changed at some point during the test execution? Are the test details created in a special way (e.g. parsed from an external testsuite)?

Additional observations:

1. Some jobs (at least https://openqa.suse.de/tests/6288225) also miss some of the screenshots.
2. Previous and next step results are often nevertheless present; not the whole test module is affected.
   1. This is not true for the job from the ticket description, though. Here the steps for the whole modules are gone.
3. Worker logs show the upload problems at least in some jobs, e.g. [2021-06-18T15:20:50.0791 CEST] [debug] [pid:16668] Uploading artefact gpg-4.txt [2021-06-18T15:20:50.0797 CEST] [error] [pid:16668] Uploading artefact gpg-4.txt failed: Service Unavailable but apparently no further upload attempts have been tried (in this example these are the only log lines containing gpg-4.txt).
   1. The job from the ticket description does not show the upload problem within the worker log, though. Here it looks like the upload hasn't even been attempted once. Maybe we're seeing two different problems here.

mkittler wrote:

1. The job from the ticket description does not show the upload problem within the worker log, though. Here it looks like the upload hasn't even been attempted once. Maybe we're seeing two different problems here.

I also strongly suspect that these are two different bugs. But fixing the lack of retries first will still help us diagnose the other bug (if it exists).

Worker log for the job in the ticket description:

…
[2021-06-22T17:51:33.0494 CEST] [debug] [pid:5432] Uploading artefact install_ltp-205.txt
[2021-06-22T17:51:33.0719 CEST] [debug] [pid:5368] Uploading artefact install_ltp-215.txt
[2021-06-22T17:51:33.0899 CEST] [info] [pid:5436] Uploading connectors
[2021-06-22T17:51:33.0899 CEST] [info] [pid:5436] Uploading connectors using multiple chunks
[2021-06-22T17:51:33.0900 CEST] [info] [pid:5436] connectors: 1 chunks
[2021-06-22T17:51:33.0900 CEST] [info] [pid:5436] connectors: chunks of 1000000 bytes each
[2021-06-22T17:51:34.0091 CEST] [debug] [pid:5432] Uploading artefact install_ltp-144.txt
[2021-06-22T17:51:34.0307 CEST] [debug] [pid:5368] Uploading artefact install_ltp-145.txt
[2021-06-22T17:51:34.0513 CEST] [info] [pid:5436] connectors: Processing chunk 1/1, avg. speed ~0.056 KiB/s
[2021-06-22T17:51:34.0520 CEST] [info] [pid:5436] Uploading vars.json
[2021-06-22T17:51:34.0599 CEST] [debug] [pid:5432] Uploading artefact install_ltp-152.txt
[2021-06-22T17:51:34.0692 CEST] [debug] [pid:5368] Uploading artefact install_ltp-14.txt
[2021-06-22T17:51:34.0911 CEST] [info] [pid:5436] Uploading autoinst-log.txt
[2021-06-22T17:51:34.0961 CEST] [debug] [pid:5432] Uploading artefact install_ltp-215.txt
[2021-06-22T17:51:35.0075 CEST] [debug] [pid:5368] Uploading artefact install_ltp-100.txt
[2021-06-22T17:51:35.0305 CEST] [info] [pid:5436] Uploading worker-log.txt

This is how it is supposed to work (judging by the code and comparing to log files):

1. Job starts
   1. Upload results (= step details) concurrently and continuously
2. Job stops
   1. The upload from 1. might still be running; that's ok
   2. Set job status to uploading
   3. Upload logs/assets (including autoinst-log.txt and worker-log.txt)
3. Logs/assets (2.3) have been uploaded
   1. Wait until possibly ongoing results upload (1.1) has concluded
   2. Start a "final" results upload
4. Final result upload has been concluded
   1. Final job state/result is set

• Point 2.1 explains the weird interleaving between Uploading [some logfile] lines and Uploading artefact [some step file] lines.
• Point 2.3 explains why log lines for certain artefacts can be missing completely: worker-log.txt is simply uploaded before these lines would be logged in step 3 and 4. (This is plausible because the test modules with missing artefacts are indeed at the end.)

So I would conclude that both cases (log lines for artefacts completely missing within the log file and errors logged for artefacts) are the same after all. It is just confusing (and hence problematic) that log messages from steps 3 and 4 are not part of worker-log.txt.

As a first step I'd just try to improve the error handling. It would make sense to retry artefact uploads in the same way other API calls to the web UI are retired. If they fail nevertheless I assume the test be marked as incomplete (with the relevant error message as reason).

As a second step I can try to move uploading autoinst-log.txt after the 4.

mkittler wrote:

As a first step I'd just try to improve the error handling. It would make sense to retry artefact uploads in the same way other API calls to the web UI are retired. If they fail nevertheless I assume the test be marked as incomplete (with the relevant error message as reason).

I'd recommend a different approach for artefact uploads: If it fails even after retry, put it back in upload queue and pause further uploads until the next trigger from the worker loop. If the network connection recovers, everything will be uploaded. If it doesn't recover, the main OpenQA server will terminate the job due to timeout either way. Only the final upload step would kill the job after several failed upload retries.

Makes sense; I'll make it only a fatal error on the final upload. Unfortunately the upload is not written using a queue (or something similar)¹. So I cannot simply put it back. I guess rewriting the code to use a queue is a bit out of scope here so I simply resort to non-fatal retries during the execution plus the final upload to catch everything which has possibly be missed.

¹ although it would make a lot of sense of course

By the way, I'm not even sure whether it'll be a big win to mark such tests as incomplete. Maybe it is sufficient to add the upload error (after all attempts have been depleted during the final upload) as reason (unless there is a more relevant reason of course)?

Moving the upload of the worker log to the end would also help with that (without considering the test as incomplete).

mkittler wrote:

By the way, I'm not even sure whether it'll be a big win to mark such tests as incomplete. Maybe it is sufficient to add the upload error (after all attempts have been depleted during the final upload) as reason (unless there is a more relevant reason of course)?

You can certainly try to send more descriptive error info to the central OpenQA server. But since you'll be doing it while the network is so horribly broken that asset uploads have exceeded retry limit, I don't think you have a very high chance of success.

My current PR adds a retry mechanism for the artefact upload similar to what we already have for regular API calls. For regular API calls it already seems to improve the situation a lot, e.g. most of the jobs you've found have their final state/result set as expected (which means the API call to send this information eventually succeeded). Hence the PR should make it more robust. That means the situations when the upload ultimately fails but we are still able to set the final state/result (where also the reason is passed) are indeed very unlikely. So likely it doesn't make much sense putting further effort (and complicated code) into this (like changing the job result to incomplete or trying to upload the worker log later).

By the way, to avoid interference with https://github.com/os-autoinst/openQA/pull/4062 I haven't tried to streamline the retry-behavior with regular API calls yet. I will do that after it has been merged if it make sense.

PR for streamlining the retry-behavior with regular API calls (also for having less redundant/inconsistent code): https://github.com/os-autoinst/openQA/pull/4067

mkittler wrote:

All PRs have been merged. So now uploading those artefacts have the same re-try behavior as regular API calls and therefore should be equally reliable.

Great. Setting my alarm clock to August 31st. When it goes off, I'll run another scan for jobs with missing console output.
Note to self: OSD is currently at job ID 6596263.

Sounds like a plan. (Especially because adding an automatic alert within our monitoring for this problem would be quite some work.)

You're right, it definitely looks still broken - as if it doesn't actually give it another try after logging Upload attempts remaining: 59/60 for….

PR for improving the related logging: https://github.com/os-autoinst/openQA/pull/4174

However, I still don't know what's the problem here. Judging by our code and unit tests it actually re-attempts the upload. Maybe the error isn't visible on the worker-side? The error handling correctly checks for both, connection and HTTP errors, though.

Since not all changes of https://github.com/os-autoinst/openQA/pull/4179 were covered I've been creating https://github.com/os-autoinst/openQA/pull/4184. However, that means the code path which would return a 200 response despite an error was most likely dead code and didn't cause the problem.

I also ran Martin's script again on OSD but couldn't find any further occurrences since the logging changes have been deployed.

In the first case when the upload is logged it looks like an additional attempt is actually made but then no further error handling (e.g. yet another attempt) happens:

[2021-09-29T07:37:44.0757 CEST] [warn] [pid:439939] Error uploading pread01_64-6.txt: 502 response: Proxy Error
[2021-09-29T07:37:44.0987 CEST] [warn] [pid:427545] Websocket connection to http://openqa.suse.de/api/v1/ws/1333 finished by remote side with code 1006, no reason - trying again in 10 seconds
[2021-09-29T07:37:54.0990 CEST] [info] [pid:427545] Registering with openQA openqa.suse.de
[2021-09-29T07:37:55.0010 CEST] [warn] [pid:427545] Failed to register at openqa.suse.de - 503 response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
<p>Additionally, a 503 Service Unavailable
error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache/2.4.43 (Linux/SUSE) Server at openqa.suse.de Port 80</address>
</body></html>
 - trying again in 10 seconds
[2021-09-29T07:38:05.0020 CEST] [info] [pid:427545] Registering with openQA openqa.suse.de
[2021-09-29T07:38:06.0187 CEST] [info] [pid:427545] Establishing ws connection via ws://openqa.suse.de/api/v1/ws/1333
[2021-09-29T07:38:08.0558 CEST] [info] [pid:427545] Registered and connected via websockets with openQA host openqa.suse.de and worker ID 1333
[2021-09-29T07:38:22.0255 CEST] [info] [pid:427545] Received signal HUP
[2021-09-29T07:38:44.0758 CEST] [warn] [pid:439939] Uploading artefact pread01_64-6.txt (attempts remaining: 59/60)
[2021-09-29T07:38:44.0791 CEST] [debug] [pid:439939] Uploading artefact pread03_64-6.txt

It also looks like the web UI was overloaded at the time, e.g. even the re-registration didn't work.

In the other case it indeed looks like the worker upload completely ignored the file (without any error message) or the file hasn't been created in the first place (and we don't show errors in that case).

I'll have yet another look at the code (and tests). I also noticed that I ran your script incorrectly when checking my self the last time.

mkittler wrote:

In the first case when upload is logged it looks like an additional attempt is actually made but then no further error handling (e.g. yet another attempt) happens

so if I got you right then we conduct one retry but ignore if that succeeds? In that case I would go for the same approach that we usually do on retrying, e.g. a for loop with $configurable_retries // 7 with sleep periods in between and die with explicit error if we don't succeed even after the retries.

I'll have yet another look at the code (and tests).

Maybe just try to improve the logging in error cases for a start and collect more experience from production runs.

mkittler wrote:

In the other case it indeed looks like the worker upload completely ignored the file (without any error message) or the file hasn't been created in the first place (and we don't show errors in that case).

There are dozens of udp6_ipsec_vti*.txt result files generated by that job. Almost all of them were correctly uploaded as you can see in the webUI but none of them is mentioned in the worker log. I thought that worker-log.txt should be the last uploaded file now?

I guess it is safe to exclude that the sub process we spawn to do the upload is terminated prematurely (without proper error handling). Even if the main worker process receives SIGHUP the upload process (in this case PID 25077) continues to run and only one single file is missing:

[2021-09-24T15:23:23.0472 CEST] [debug] [pid:25077] Uploading artefact sshd-402.txt
[2021-09-24T15:23:24.0709 CEST] [debug] [pid:25077] Uploading artefact sshd-111.txt
[2021-09-24T15:23:25.0219 CEST] [warn] [pid:15314] Websocket connection to http://openqa.suse.de/api/v1/ws/1313 finished by remote side with code 1006, no reason - trying again in 10 seconds
[2021-09-24T15:23:25.0730 CEST] [warn] [pid:25077] Error uploading sshd-111.txt: 503 response: Service Unavailable
[2021-09-24T15:23:35.0230 CEST] [info] [pid:15314] Registering with openQA openqa.suse.de
[2021-09-24T15:23:38.0396 CEST] [info] [pid:15314] Establishing ws connection via ws://openqa.suse.de/api/v1/ws/1313
[2021-09-24T15:23:46.0746 CEST] [info] [pid:15314] Registered and connected via websockets with openQA host openqa.suse.de and worker ID 1313
[2021-09-24T15:24:04.0266 CEST] [info] [pid:15314] Received signal HUP
[2021-09-24T15:24:25.0731 CEST] [warn] [pid:25077] Uploading artefact sshd-111.txt (attempts remaining: 59/60)
[2021-09-24T15:24:27.0952 CEST] [debug] [pid:25077] Uploading artefact sshd-366.txt
[2021-09-24T15:24:29.0234 CEST] [debug] [pid:25077] Uploading artefact sshd-68.txt
[2021-09-24T15:24:31.0449 CEST] [debug] [pid:25077] Uploading artefact sshd-257.txt
[2021-09-24T15:24:32.0816 CEST] [debug] [pid:25077] Uploading artefact sshd-31.txt
[2021-09-24T15:24:34.0249 CEST] [debug] [pid:25077] Uploading artefact sshd-216.txt
[2021-09-24T15:24:35.0559 CEST] [debug] [pid:25077] Uploading artefact sshd-163.txt
[2021-09-24T15:24:36.0617 CEST] [debug] [pid:25077] Uploading artefact sshd-58.txt
[2021-09-24T15:24:38.0132 CEST] [debug] [pid:25077] Uploading artefact sshd-48.txt
[2021-09-24T15:24:39.0793 CEST] [debug] [pid:25077] Uploading artefact sshd-90.txt

so if I got you right then we conduct one retry but ignore if that succeeds? In that case I would go for the same approach that we usually do on retrying, e.g. a for loop with $configurable_retries // 7 with sleep periods in between and die with explicit error if we don't succeed even after the retries.

We have a loop which is supposed to work like this. The question is why this loop sometimes doesn't work. I was just describing the observation of the actual behavior which goes against what is supposed to happen as the code looks right now.

Maybe just try to improve the logging in error cases for a start and collect more experience from production runs.

That's actually what I've already did and the reason we see Error uploading pread01_64-6.txt: 502 response: Proxy Error. That we don't see a further error message after the next logged attempt actually means that no error occurred (from the perspective of the upload code). In reality files are missing so either the server did not tell the worker that an error happened (although it did) or the worker's error handling is somehow wrong or the file has been deleted for some reason afterwards¹.

¹ The code for finalizing jobs looks good and shouldn't lead to this problem even though it is interrupted in the middle.

There are dozens of udp6_ipsec_vti*.txt result files generated by that job. Almost all of them were correctly uploaded as you can see in the webUI but none of them is mentioned in the worker log. I thought that worker-log.txt should be the last uploaded file now?

That is a good point and it also bugs me. If these files were normal files under "ulogs" or normal files mentioned in a module result I suppose they should show up in the logs. Apparently they don't so maybe these particular files are treated differently as they are "External results" and still lack proper error handling with retries. However, your script also shows completely normal jobs like https://openqa.suse.de/tests/7220766#step/sshd/111 which are affected. So I guess the code for "External results" and the code for normal "ulogs" files are both still problematic.

mkittler wrote:

There are dozens of udp6_ipsec_vti*.txt result files generated by that job. Almost all of them were correctly uploaded as you can see in the webUI but none of them is mentioned in the worker log. I thought that worker-log.txt should be the last uploaded file now?

That is a good point and it also bugs me. If these files were normal files under "ulogs" or normal files mentioned in a module result I suppose they should show up in the logs. Apparently they don't so maybe these particular files are treated differently as they are "External results" and still lack proper error handling with retries. However, your script also shows completely normal jobs like https://openqa.suse.de/tests/7220766#step/sshd/111 which are affected. So I guess the code for "External results" and the code for normal "ulogs" files are both still problematic.

No, those udp6_ipsec_vti*.txt files missing in the log are not external results or ulogs or anything of that sort. Those files are regular console output snippets captured automatically by wait_serial().

@mkittler, @cdywan, @kraih to meet tomorrow 13.00 CEST

Some notes from the conversation (no verdict or result here):

• Job.pm upload_log_file
  • evaluate_error
  • Why don't we see "all attempts failed" in https://openqa.suse.de/tests/7220766/logfile?filename=worker-log.txt? Maybe the very last attempt succeeded?
• tested in 24-worker-jobs.t

Unable to read sshd-111.txt. is this in the log:

[2021-09-24T15:23:24.0709 CEST] [debug] [pid:25077] Uploading artefact sshd-111.txt
[2021-09-24T15:23:25.0219 CEST] [warn] [pid:15314] Websocket connection to http://openqa.suse.de/api/v1/ws/1313 finished by remote side with code 1006, no reason - trying again in 10 seconds
[2021-09-24T15:23:25.0730 CEST] [warn] [pid:25077] Error uploading sshd-111.txt: 503 response: Service Unavailable
[2021-09-24T15:23:35.0230 CEST] [info] [pid:15314] Registering with openQA openqa.suse.de
[2021-09-24T15:23:38.0396 CEST] [info] [pid:15314] Establishing ws connection via ws://openqa.suse.de/api/v1/ws/1313
[2021-09-24T15:23:46.0746 CEST] [info] [pid:15314] Registered and connected via websockets with openQA host openqa.suse.de and worker ID 1313
[2021-09-24T15:24:04.0266 CEST] [info] [pid:15314] Received signal HUP
[2021-09-24T15:24:25.0731 CEST] [warn] [pid:25077] Uploading artefact sshd-111.txt (attempts remaining: 59/60)

• why HUP after 30s?
  • only main process receives the signal, meaning it shouldn't be relevant here?
• Uploading artefact sshd-111.txt is expected to be delayed because of 503 and should have succeeded since there's no error afterwards
• Not seen in the journal?
• Can we checksum results instead of logging every step of artifact creation
• Semi-related, but we see a lot of extra log messages due to "debug" level. A newer mojolicious version might get rid of some messages like e.g. 200 OK now at "strace" level?
  • kraih: This is a regression to be fixed in the next release

I ran the script again and we checked more jobs. There are different types of jobs which miss txt results but showing slightly different logging behavior:

1. Some jobs have no traces of txt result uploads in the worker log at all. It isn't clear why that's the case. At least for the successfully uploaded txt results there should be a corresponding line in the log file. Note that the corresponding "Create artifact:" messages still show up on the server log, though (except for the few actually missing txt results).
   • e.g. https://openqa.suse.de/tests/7492641
2. In some jobs the upload is logged but only for a file with a slightly different name. In this case the log message for the relevant upload is likely just completely absent because the present log name actually belongs to a different result which has a very similar name.
   • e.g. in https://openqa.suse.de/tests/7473737#step/sigsuspend01/1
3. In some jobs a failed upload attempt is actually logged but then it looks like the next attempt succeeded and hence no further retry comes into play. So retrying generally works. The problem is more that sometimes an upload which didn't work is still considered a success.
   • e.g. https://openqa.suse.de/tests/7220766#step/sshd/111
4. In some jobs the upload is logged as if it has worked on the first try.
   • e.g. https://openqa.suse.de/tests/7446866#step/hostname/14

• We have logging problems, at least concerning the (client-side) worker log.
• 3. and 4. show that the error handling isn't reliable and sometimes failed uploads are treated as success (and hence the retry isn't effective).
• 1. also shows that the finalize job is unlikely do cause the problem because it shows that the "Create artifact: …" log line was actually missing for the missing result.

Unfortunately we still couldn't find the source of these problems.

mkittler wrote:

1. In some jobs the upload is logged but only for a file with a slightly different name. It isn't clear where the file with the slightly different name belongs to. The missing log line with the expected file name could be the same as the previous case although in this case there were log messages for the successfully uploaded txt results.
   • e.g. in https://openqa.suse.de/tests/7473737#step/sigsuspend01/1 the log contains Uploading artefact rt_sigsuspend01-1.txt but not Uploading artefact sigsuspend01-1.txt which would actually be the expected name of the file

Unsurprisingly, rt_sigsuspend01-1.txt belongs to the rt_sigsuspend01 test: https://openqa.suse.de/tests/7473737#step/rt_sigsuspend01/1

Ah, makes sense. I must have made a typo when searching. I'll update my comment.

dheidler found another issue, maybe it's actually the same. Please see #102786 , I added some fancy screenshots there. Maybe that helps :)

okurz wrote:

dheidler found another issue, maybe it's actually the same. Please see #102786 , I added some fancy screenshots there. Maybe that helps :)

I see that a lot but simply waiting for the job to finish uploading and then reloading the page will fix it. My best guess is that the upload thread has unnecessary delays. That would be a design problem, not really a bug.

As @MDoucha already stated, #102786 is completely unrelated. This is about artifacts missing persistently (on finished tests).

Maybe https://github.com/os-autoinst/openQA/pull/4412 and https://github.com/os-autoinst/openQA/pull/4427 for #102332 helped with that as well. I'm running @MDoucha's script again to check that.

The script is still running without output. I suppose that's not really the most efficient way to check for problematic jobs.

mkittler wrote:

The script is still running without output. I suppose that's not really the most efficient way to check for problematic jobs.

That's because you're running the wrong script. scan_details.py is the main scanner script that reads job result data. filter_details.py reads the scan results from standard input and merely filters out bogus jobs by checking OpenQA web API for "valid" causes of missing result files. Run both scripts like this and you can kill them when the counter printed to console drops under 8000:

scan_details.py | filter_details.py >results.txt

Yes, some of the jobs in the beginning of the list will be false positives because they were still uploading results while they got scanned but they fully finished before they got filtered.

One of my verification runs seems to have skipped the upload phase entirely:
https://openqa.suse.de/tests/9582368