Project

General

Profile

action #70333

providing a git branch for NEEDLES_DIR does not appear to work on o3

Added by dancermak about 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Low
Assignee:
Category:
Feature requests
Target version:
Start date:
2020-08-20
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

I have tried to launch a test with a custom set of needles from a git branch on github by using the NEEDLES_DIR variable. Unfortunately, this does not appear to work reliably. For example these two tests: https://openqa.opensuse.org/tests/1367310 and https://openqa.opensuse.org/tests/1367262 appear to clone the git branch, but then the test fails as it cannot find one of the new needles. The same thing happened for a completely unrelated test as well: https://openqa.opensuse.org/tests/1365359.


Related issues

Related to openQA Project - action #56789: New needles from git repository not working with openqa-clone-custom-git-refspecNew2019-09-11

History

#1 Updated by okurz about 1 year ago

  • Category set to Feature requests
  • Priority changed from Normal to Low
  • Target version set to future

I am afraid we might have the issue reported already (see other tickets about needles). Likely this is something that never worked this way but with the current description I am not sure. Would you please be so kind to provide clear expectations according to the format in https://progress.opensuse.org/projects/openqav3/wiki#ticket-templates

#2 Updated by mkittler about 1 year ago

  • Related to action #56789: New needles from git repository not working with openqa-clone-custom-git-refspec added

#3 Updated by mkittler about 1 year ago

I've linked another needle issue which is about web UI limitations. This comment explains what's still missing on the web UI side: https://progress.opensuse.org/issues/56789#note-20

However, matching needles from a custom checkout specified via NEEDLES_DIR within the os-autoinst backend is actually supposed to work. At least when I remember it correctly from my tests for the web UI issue it worked. Does it work locally for you? (And I mean locally with the NEEDLES_DIR variable and your regular needles dir not containing the required needle as well.)


The test log looks like this:

[2020-08-20T16:37:29.653 CEST] [debug] Checking out git refspec/branch 'rstudio_profiler_notebook_needles'
[2020-08-20T16:37:29.653 CEST] [debug] Cloning git URL 'https://github.com/dcermak/os-autoinst-needles-opensuse' to use as test distribution
[2020-08-20T16:38:17.830 CEST] [debug] Cloning into 'os-autoinst-needles-opensuse'...
[2020-08-20T16:38:18.042 CEST] [debug] git hash in /var/lib/openqa/pool/10/os-autoinst-needles-opensuse: 237d3899e26f02e6f07122dcb285fc560620dcd2
[2020-08-20T16:38:18.042 CEST] [debug] init needles from /var/lib/openqa/pool/10/os-autoinst-needles-opensuse
[2020-08-20T16:38:18.836 CEST] [debug] inst-console-20200224 contains inst-console twice
[2020-08-20T16:38:19.076 CEST] [debug] loaded 8181 needles

It looks like it would clone the Git repo as expected and also loads the needles from there. Is the hash from the log correct?

#4 Updated by dancermak about 1 year ago

mkittler wrote:

I've linked another needle issue which is about web UI limitations. This comment explains what's still missing on the web UI side: https://progress.opensuse.org/issues/56789#note-20

However, matching needles from a custom checkout specified via NEEDLES_DIR within the os-autoinst backend is actually supposed to work. At least when I remember it correctly from my tests for the web UI issue it worked. Does it work locally for you? (And I mean locally with the NEEDLES_DIR variable and your regular needles dir not containing the required needle as well.)


The test log looks like this:

[2020-08-20T16:37:29.653 CEST] [debug] Checking out git refspec/branch 'rstudio_profiler_notebook_needles'
[2020-08-20T16:37:29.653 CEST] [debug] Cloning git URL 'https://github.com/dcermak/os-autoinst-needles-opensuse' to use as test distribution
[2020-08-20T16:38:17.830 CEST] [debug] Cloning into 'os-autoinst-needles-opensuse'...
[2020-08-20T16:38:18.042 CEST] [debug] git hash in /var/lib/openqa/pool/10/os-autoinst-needles-opensuse: 237d3899e26f02e6f07122dcb285fc560620dcd2
[2020-08-20T16:38:18.042 CEST] [debug] init needles from /var/lib/openqa/pool/10/os-autoinst-needles-opensuse
[2020-08-20T16:38:18.836 CEST] [debug] inst-console-20200224 contains inst-console twice
[2020-08-20T16:38:19.076 CEST] [debug] loaded 8181 needles

It looks like it would clone the Git repo as expected and also loads the needles from there. Is the hash from the log correct?

Interesting, I get this one: git hash in /var/lib/openqa/share/tests/opensuse: 384bebc213e7d6a24267972111878f1b407562b0
However, the rstudio test executes successfully on my machine when overriding the NEEDLES_DIR variable (tried it twice just to be sure).

#5 Updated by mkittler about 1 year ago

You obviously might get a different Git hash when the branch has been altered. What I was actually wanted to ask: Was the Git hash 237d3899e26f02e6f07122dcb285fc560620dcd2 the expected hash at the time the test ran on o3?

However, the rstudio test executes successfully on my machine when overriding the NEEDLES_DIR variable (tried it twice just to be sure).

Did you also ensure that the regular needle directory does not contain the needle anymore? (E.g. just checkout the master branch in the regular checkout.)

#6 Updated by dancermak about 1 year ago

mkittler wrote:

You obviously might get a different Git hash when the branch has been altered. What I was actually wanted to ask: Was the Git hash 237d3899e26f02e6f07122dcb285fc560620dcd2 the expected hash at the time the test ran on o3?

That is the correct hash indeed.

However, the rstudio test executes successfully on my machine when overriding the NEEDLES_DIR variable (tried it twice just to be sure).

Did you also ensure that the regular needle directory does not contain the needle anymore? (E.g. just checkout the master branch in the regular checkout.)

Yes, I switched to the master branch and restarted openQA.

#7 Updated by dancermak about 1 year ago

I have retried the problematic tests on o3 and took a closer look at the failure and I must apologize: openqa actually does pickup the needles. However, the webui does not pick up the needles and because new part that I was testing took longer on o3 than I thought, it made it appear as if openqa was completely oblivious to the new needles.

So I guess this is a bug in the webui instead?

#8 Updated by mkittler about 1 year ago

  • Status changed from New to Resolved
  • Assignee set to mkittler

So I guess this is a bug in the webui instead?

Yes. However, I wouldn't call it a bug because displaying custom/versioned needles is simply something which has never been implemented so far and the implementation is also not trivial (unlike enabling os-autoinst to use custom needles).

I'm closing this issue in favor of #56789 then (where I've already added a comment how a possible implementation could work).

Also available in: Atom PDF