QA » openQA Project

The code which parses the settings variables is the same for the ISO post as for the single jobs post (parse_assets_from_settings method). The code for enqueuing the download jobs is also the same in both cases (enqueue_download_jobs method). Hence it must be the way enqueue_download_jobs is called when scheduling the ISO is buggy. But both use create_downloads_list for this. So I'm not sure what makes the difference here.

Cloning the job using the Web UI results in the same error 100% reproducible.

I assume you mean clicking the "restart" button on the web UI. Restarting the job via the web UI does not help because openQA's job duplication code simply does not create a new download task. That is a known limitation, see https://progress.opensuse.org/issues/57782#note-2 and https://progress.opensuse.org/issues/57782#note-5. I've also created a draft to prevent jobs with missing assets from being restarted in the first place which goes in the opposite direction. Maybe we should clarify whether we want to retrigger downloads or not. I've been messing the the related code recently so I would agree with @coolo's statement from the other issue:

I wouldn't restart the download from retriggering jobs. Everyone who knows the retriggering jobs code will agree :)

How is this a feature request? It's quite clearly a bug.

I am following what we defined on https://progress.opensuse.org/projects/openqav3/wiki#ticket-categories . Categorizing it does not have a direct impact on severity or our priority of the issue. To my understanding this never worked. Maybe I am misreading your observation and this really a regression? In this case could you help us to find any "last good"?

@mkittler so do you know since when this regression was introduced then?

Any news here? This is also needed by MicroOS tests now.

rbrown added a download cron job to workaround this, but that's not great for multiple reasons.

After reading the ticket description again I'm not sure anymore what this ticket is about. Is it about

1. restarting a job within the web UI or API? That's now actually prevented if there are missing assets (with a force option) as requested by #34783.
2. posting "an ISO" via the API? (Likely that's not the case because the job mentioned in the description has not been created by posting an ISO.)
3. posting a single job via the API?

If it is option 1. that still leaves the question how the job has been created initially and the the asset download failed in the first place.

If there are more recent examples, can you provide some links?

The job was created by posting an ISO, through the obs_rsync scripts.

Both 1 and 3 should be fine AFAIK, though I haven't tried that again.

As okurz removed the test for some reason, I don't have any recent example. I'll try to create a minimal PoC locally.

favogt wrote:

As okurz removed the test for some reason, I don't have any recent example. I'll try to create a minimal PoC locally.

Done and copied to o3: https://openqa.opensuse.org/tests/1293416#details

To reproduce: openqa-client --host http://openqa.opensuse.org isos post DISTRI=opensuse VERSION=1 FLAVOR=poo62159 ARCH=x86_64 BUILD=1
The created job incompletes, but when cloning it with openqa-clone-job it creates a GRU task and works.

The difference in creating download asset between 'isos post' and 'job create' is that: 'isos post' uses the arguments to create the download list, and 'job create' (such as openqa-clone-job) uses the job's setting to create the download list. Maybe this can explain why 'openqa-clone-job' works, but 'isos post' does not works.

I am not sure if we should call the create_downloads_list for every job when doing 'isos post', because there may be many jobs will be created.

@favogt
workaround:
openqa-client --host http://openqa.opensuse.org isos post DISTRI=opensuse VERSION=1 FLAVOR=poo62159 ARCH=x86_64 BUILD=1 HDD_1=openSUSE-Tumbleweed-JeOS.x86_64-kvm-and-xen.qcow2 HDD_1_URL=http://download.opensuse.org/tumbleweed/appliances/openSUSE-Tumbleweed-JeOS.x86_64-kvm-and-xen.qcow2?foobar=1

Xiaojing_liu wrote:

I am not sure if we should call the create_downloads_list for every job when doing 'isos post', because there may be many jobs will be created.

I think this should be the same as calling jobs post … HDD_1_URL=http://download.opensuse.org/my/same/asset.qcow2 10 times. I suggest to just try this out and see what happens.

I recommend to just try out what happens when we call create_downloads_list.

@favogt
workaround:
openqa-client --host http://openqa.opensuse.org isos post DISTRI=opensuse VERSION=1 FLAVOR=poo62159 ARCH=x86_64 BUILD=1 HDD_1=openSUSE-Tumbleweed-JeOS.x86_64-kvm-and-xen.qcow2 HDD_1_URL= http://download.opensuse.org/tumbleweed/appliances/openSUSE-Tumbleweed-JeOS.x86_64-kvm-and-xen.qcow2?foobar=1

I think there are spaces between HDD_1_URL= and its arguments which should not be there.

Ok, so that would create multiple asset downloads. Judging by the code in openQA/lib/OpenQA/Task/Asset/Download.pm nothing bad will happen in that case. There's a lock to prevent concurrently downloading the same asset and a check to prevent downloading an existing asset again.

I suppose it would nevertheless be a good idea to de-duplicate the download lists for the jobs created by isos post by the download destination to produce less overhead. (E.g. enqueue_download_jobs would accept multiple download lists and skips "visited" download destinations.)

mkittler wrote:

@fvogt We came to the conclusion that this issue is actually: Add support for triggering GRU asset downloads via "isos post" when the relevant _URL parameters are not directly provided but only pulled from e.g. the test suites table

Is that right? I'm just wondering because in the ticket description this feature request is mixed up with restarting jobs and errors reported from the worker's asset cache.

Yes. I don't see how there's any mixup, the errors and observations with job restarting are directly related.

There's just one caveat but for now I wouldn't over-optimize it.

favogt wrote:

Unfortunately there is an issue with the way this is implemented.

I added a testsuite ("jeos2twnext") which defines HDD_1_URL and linked them to the JeOS medium in the "Development Tumbleweed" group.
When the next snapshot scheduled the JeOS product, all unrelated tests in the main group also failed with the GRU error:
https://openqa.opensuse.org/tests/1375735#step/GRU/1

So it appears like the download jobs are attached to all scheduled jobs and not just the ones which actually need them?

yes, when using isos post, the download jobs are attached to all scheduled jobs, even the unrelated jobs. This pr does not fix this, I could create a new ticket to record this feature.

@Xiaojing_liu I guess by now we know that your change works and all changes left to be done are tracked in #70687 , right? If this as true then please resolve this ticket.