Project

General

Profile

action #43511

[functional][u] Asset cache does not support ASSET_ assets

Added by szarate about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
SUSE QA - Milestone 23
Start date:
2018-11-07
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Observation

In job this job there's an extra asset being defined, and Gru has already code to handle "external downloads" however the caching does not handle these, causing the job to die.

ASSET_1_URL     https://w3.nue.suse.com/~slindomansilla/tmp/bootloader_poo40670.pm
SCHEDULE    bootloader_poo40670,tests/installation/welcome
error on bootloader_poo40670.pm: Can't locate ../../../pool/1/bootloader_poo40670.pm in @INC (you may need to install the ..::..::..::pool::1::bootloader_poo40670 module) (@INC contains: ../../../pool/1 /var/lib/openqa/cache/tests/sle/lib /var/lib/openqa/pool/1/blib/arch /var/lib/openqa/pool/1/blib/lib /usr/lib/os-autoinst /usr/lib/perl5/site_perl/5.18.2/ppc64le-linux-thread-multi /usr/lib/perl5/site_perl/5.18.2 /usr/lib/perl5/vendor_perl/5.18.2/ppc64le-linux-thread-multi /usr/lib/perl5/vendor_perl/5.18.2 /usr/lib/perl5/5.18.2/ppc64le-linux-thread-multi /usr/lib/perl5/5.18.2 /usr/lib/perl5/site_perl .) at (eval 739) line 1.

In a more optimal scenario, the webUI shows a list of assets to download, and the cache just takes care of said list, except the repos.

Acceptance criteria

  • AC1: A job scheduled with a downloadable override test module can run successfully on osd

Suggestions

  • Hook into the detect_asset_keys method of OpenQA::Worker::Engines::isotovideo and add handling for the ASSET_*_URL.

Related issues

Related to openQA Tests - action #45038: [functional][y][leap] test fails in installation - autoyast profile can not be retrieved, no network?Rejected2018-12-122019-01-29

Blocks openQA Tests - action #46094: [sle][functional][u] test fails in installation - missing assets or wrong linkResolved2019-01-142019-01-29

Blocks openQA Project - action #36994: Dynamic test flow definition + overrideResolved2018-06-08

History

#1 Updated by okurz about 3 years ago

  • Subject changed from [y][u] Asset cache does not support 'other' assets to [functional][u] Asset cache does not support 'other' assets
  • Description updated (diff)
  • Assignee set to szarate
  • Target version set to Milestone 21

szarate please clarify the suggestions to make it workable

#2 Updated by szarate about 3 years ago

  • Description updated (diff)

#3 Updated by szarate about 3 years ago

  • Status changed from New to In Progress

After looking at changes to the cache, I think this can be done independantly. as we already support KERNEL and INITRD. and the handling for asset urls didn't change...

Working on this today.

#4 Updated by szarate about 3 years ago

  • Status changed from In Progress to Feedback

#5 Updated by okurz about 3 years ago

PR merged, we should be able to test this tomorrow on o3 assuming that the auto-deployment of workers does its job.

#6 Updated by AdamWill about 3 years ago

I think the attempted fix here was wrong and would not have worked. I think https://github.com/os-autoinst/openQA/pull/1907 should fix it. If someone can check this out, that'd be great (I don't have worker caching enabled on the Fedora instances ATM, so I can't test it out Most Interesting Man In The World-style):

You know, like this

#7 Updated by AdamWill about 3 years ago

  • Subject changed from [functional][u] Asset cache does not support 'other' assets to [functional][u] Asset cache does not support ASSET_ assets

#8 Updated by szarate almost 3 years ago

  • Assignee deleted (szarate)

More information can be found on: https://github.com/os-autoinst/openQA/pull/1907#issuecomment-444394975

Long story short: loadtests needs to support loading test scripts out of tree (out of $CASEDIR), for this either the find_script or the load_tests itself needs to be modified in the os_autoinst project, by altering the behaviour if the SCHEDULE backend variable is set.

#9 Updated by okurz almost 3 years ago

  • Status changed from Feedback to Workable

Also as agreed we set back to "Workable", no use to have it in "Feedback" without an assignee.

szarate wrote:

More information can be found on: https://github.com/os-autoinst/openQA/pull/1907#issuecomment-444394975

Long story short: loadtests needs to support loading test scripts out of tree (out of $CASEDIR), for this either the find_script or the load_tests itself needs to be modified in the os_autoinst project, by altering the behaviour if the SCHEDULE backend variable is set.

But that is already working, see https://github.com/os-autoinst/os-autoinst/pull/1019 . The problem is only with cache it seems.

#10 Updated by szarate almost 3 years ago

  • Assignee set to szarate

I m taking this one back for the time being :)

#11 Updated by szarate almost 3 years ago

  • Status changed from Workable to In Progress

I'm still looking at bits and pieces, but the asset isn't really downloaded for the time being to the pool nor is linked in the test. Initially cache wouldn't look at other assets except for kernel and initrd, downloadable assets were never meant to be part of cache (it came after)...

So for the time being I guess it's matter of registering a downloaded asset as part of a job (which somehow already is), and ensure the cache properly links it. Perhaps in the meantime, explain a bit better how it's all connected

#12 Updated by okurz almost 3 years ago

  • Target version changed from Milestone 21 to Milestone 22

#13 Updated by szarate almost 3 years ago

  • Status changed from In Progress to Workable

I will drop this one for now. I'm taking too long figuring out how to add unit tests for the download part in gru (Note that I mean both things together ;) not the unit tests that are in 14-grutasks.t)

#14 Updated by szarate almost 3 years ago

  • Related to action #45038: [functional][y][leap] test fails in installation - autoyast profile can not be retrieved, no network? added

#15 Updated by szarate almost 3 years ago

  • Status changed from Workable to In Progress

#16 Updated by szarate almost 3 years ago

  • Related to action #46094: [sle][functional][u] test fails in installation - missing assets or wrong link added

#17 Updated by szarate almost 3 years ago

  • Related to deleted (action #46094: [sle][functional][u] test fails in installation - missing assets or wrong link)

#18 Updated by szarate almost 3 years ago

  • Blocks action #46094: [sle][functional][u] test fails in installation - missing assets or wrong link added

#19 Updated by szarate almost 3 years ago

  • Status changed from In Progress to Feedback

I'm disabling for the time being changes done by https://github.com/os-autoinst/openQA/pull/1907 and https://github.com/os-autoinst/openQA/pull/1853 since it looks like the dynamic scheduling of test modules conflicts a bit with jobs that depend on ASSET_1_URL.

#20 Updated by okurz almost 3 years ago

  • Target version changed from Milestone 22 to Milestone 23

#21 Updated by okurz almost 3 years ago

  • Blocks action #36994: Dynamic test flow definition + override added

#22 Updated by okurz almost 3 years ago

  • Status changed from Feedback to Resolved

PR was deployed to osd and looks good, e.g. see https://openqa.suse.de/tests/2506696 . I think we are done here. Followup work can be done in #36994 . Thanks for your work.

Also available in: Atom PDF