action #70333
closedproviding a git branch for NEEDLES_DIR does not appear to work on o3
0%
Description
I have tried to launch a test with a custom set of needles from a git branch on github by using the NEEDLES_DIR variable. Unfortunately, this does not appear to work reliably. For example these two tests: https://openqa.opensuse.org/tests/1367310 and https://openqa.opensuse.org/tests/1367262 appear to clone the git branch, but then the test fails as it cannot find one of the new needles. The same thing happened for a completely unrelated test as well: https://openqa.opensuse.org/tests/1365359.
Updated by okurz over 3 years ago
- Category set to Feature requests
- Priority changed from Normal to Low
- Target version set to future
I am afraid we might have the issue reported already (see other tickets about needles). Likely this is something that never worked this way but with the current description I am not sure. Would you please be so kind to provide clear expectations according to the format in https://progress.opensuse.org/projects/openqav3/wiki#ticket-templates
Updated by mkittler over 3 years ago
- Related to action #56789: New needles from git repository not working with openqa-clone-custom-git-refspec added
Updated by mkittler over 3 years ago
I've linked another needle issue which is about web UI limitations. This comment explains what's still missing on the web UI side: https://progress.opensuse.org/issues/56789#note-20
However, matching needles from a custom checkout specified via NEEDLES_DIR within the os-autoinst backend is actually supposed to work. At least when I remember it correctly from my tests for the web UI issue it worked. Does it work locally for you? (And I mean locally with the NEEDLES_DIR variable and your regular needles dir not containing the required needle as well.)
The test log looks like this:
[2020-08-20T16:37:29.653 CEST] [debug] Checking out git refspec/branch 'rstudio_profiler_notebook_needles'
[2020-08-20T16:37:29.653 CEST] [debug] Cloning git URL 'https://github.com/dcermak/os-autoinst-needles-opensuse' to use as test distribution
[2020-08-20T16:38:17.830 CEST] [debug] Cloning into 'os-autoinst-needles-opensuse'...
[2020-08-20T16:38:18.042 CEST] [debug] git hash in /var/lib/openqa/pool/10/os-autoinst-needles-opensuse: 237d3899e26f02e6f07122dcb285fc560620dcd2
[2020-08-20T16:38:18.042 CEST] [debug] init needles from /var/lib/openqa/pool/10/os-autoinst-needles-opensuse
[2020-08-20T16:38:18.836 CEST] [debug] inst-console-20200224 contains inst-console twice
[2020-08-20T16:38:19.076 CEST] [debug] loaded 8181 needles
It looks like it would clone the Git repo as expected and also loads the needles from there. Is the hash from the log correct?
Updated by dancermak over 3 years ago
mkittler wrote:
I've linked another needle issue which is about web UI limitations. This comment explains what's still missing on the web UI side: https://progress.opensuse.org/issues/56789#note-20
However, matching needles from a custom checkout specified via
NEEDLES_DIRwithin the os-autoinst backend is actually supposed to work. At least when I remember it correctly from my tests for the web UI issue it worked. Does it work locally for you? (And I mean locally with theNEEDLES_DIRvariable and your regular needles dir not containing the required needle as well.)
The test log looks like this:
[2020-08-20T16:37:29.653 CEST] [debug] Checking out git refspec/branch 'rstudio_profiler_notebook_needles' [2020-08-20T16:37:29.653 CEST] [debug] Cloning git URL 'https://github.com/dcermak/os-autoinst-needles-opensuse' to use as test distribution [2020-08-20T16:38:17.830 CEST] [debug] Cloning into 'os-autoinst-needles-opensuse'... [2020-08-20T16:38:18.042 CEST] [debug] git hash in /var/lib/openqa/pool/10/os-autoinst-needles-opensuse: 237d3899e26f02e6f07122dcb285fc560620dcd2 [2020-08-20T16:38:18.042 CEST] [debug] init needles from /var/lib/openqa/pool/10/os-autoinst-needles-opensuse [2020-08-20T16:38:18.836 CEST] [debug] inst-console-20200224 contains inst-console twice [2020-08-20T16:38:19.076 CEST] [debug] loaded 8181 needlesIt looks like it would clone the Git repo as expected and also loads the needles from there. Is the hash from the log correct?
Interesting, I get this one: git hash in /var/lib/openqa/share/tests/opensuse: 384bebc213e7d6a24267972111878f1b407562b0
However, the rstudio test executes successfully on my machine when overriding the NEEDLES_DIR variable (tried it twice just to be sure).
Updated by mkittler over 3 years ago
You obviously might get a different Git hash when the branch has been altered. What I was actually wanted to ask: Was the Git hash 237d3899e26f02e6f07122dcb285fc560620dcd2 the expected hash at the time the test ran on o3?
However, the rstudio test executes successfully on my machine when overriding the NEEDLES_DIR variable (tried it twice just to be sure).
Did you also ensure that the regular needle directory does not contain the needle anymore? (E.g. just checkout the master branch in the regular checkout.)
Updated by dancermak over 3 years ago
mkittler wrote:
You obviously might get a different Git hash when the branch has been altered. What I was actually wanted to ask: Was the Git hash
237d3899e26f02e6f07122dcb285fc560620dcd2the expected hash at the time the test ran on o3?
That is the correct hash indeed.
However, the rstudio test executes successfully on my machine when overriding the NEEDLES_DIR variable (tried it twice just to be sure).
Did you also ensure that the regular needle directory does not contain the needle anymore? (E.g. just checkout the master branch in the regular checkout.)
Yes, I switched to the master branch and restarted openQA.
Updated by dancermak over 3 years ago
I have retried the problematic tests on o3 and took a closer look at the failure and I must apologize: openqa actually does pickup the needles. However, the webui does not pick up the needles and because new part that I was testing took longer on o3 than I thought, it made it appear as if openqa was completely oblivious to the new needles.
So I guess this is a bug in the webui instead?
Updated by mkittler over 3 years ago
- Status changed from New to Resolved
- Assignee set to mkittler
So I guess this is a bug in the webui instead?
Yes. However, I wouldn't call it a bug because displaying custom/versioned needles is simply something which has never been implemented so far and the implementation is also not trivial (unlike enabling os-autoinst to use custom needles).
I'm closing this issue in favor of #56789 then (where I've already added a comment how a possible implementation could work).