QA » openQA Project

I reproduced this issue in my local environment. Because the cloned source job's PRODUCTDIR is /var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/ (remote worker and enable cache), so the openqa-clone-custom-git-refspec set the new job's NEEDLES_DIR as '/var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles' in this line: https://github.com/os-autoinst/openQA/blob/master/script/openqa-clone-custom-git-refspec#L126

In os-autoinst, when the needles_dir does not exist, it will change the needles_dir to CASEDIR+NEEDLES_DIR, https://github.com/os-autoinst/os-autoinst/blob/master/needle.pm#L329.
So in this situation, the needle_dir is /var/lib/openqa/pool/2/os-autoinst-distri-opensuse/var/lib/openqa/cache/openqa1-opensuse/tests/opensuse/products/opensuse/needles

I am wondering, do we need to give the arguments NEEDLES_DIR and PRODUCTDIR when calling openqa-clone-job in openqa-clone-custom-git-refspec?
I have not got a good idea to fix this ticket.
A workaround:
use openqa-clone-job ... CASEDIR=https://github.com/ggardet/os-autoinst-distri-opensuse.git#add_ai_ml _GROUP="0"

I reproduced it in my local environment, so both source job and cloned job are local jobs. Here are the link:
source job: http://10.67.19.103/tests/158 (worker: barry:1 is a remote worker)
clone job: http://10.67.19.103/tests/159

okurz wrote:

@ggardet_arm can you please check again if this works now after https://github.com/os-autoinst/openQA/pull/3348 has been merged?

No, it does not work.
See: https://openqa.opensuse.org/tests/1384426#details created by:

./openqa-clone-custom-git-refspec https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/10944 https://openqa.opensuse.org/tests/1384351 WORKER_CLASS=aws

That's a good and correct explanation and a good workaround. For now I don't think we should do more than document this for the special case of test distributions with extra needle repo cloned from caching worker to non-caching worker. How about we document within the openQA docs or the help of openqa-clone-custom-git-refspec? For a better intelligence I think we should work on other tickets first, e.g. the full version control awareness

Thanks for the workaround. The problem is when jobs are cloned and WORKER_CLASS not overwritten, any aarch64 worker could be picked up and could fail due to this problem.
Maybe we should setup all workers to use the same path, as done in osd?

Yes. I guess for this you just need to ensure two things on the remote worker machines that you maintain:

• Enabling caching
• Ensure the main o3 webui server can be reached by the hostname "openqa1-opensuse"

To show the relevant parts from the worker config

$ ssh root@aarch64 cat /etc/openqa/workers.ini
[global]
HOST = http://openqa1-opensuse
WORKER_HOSTNAME = 192.168.112.3
CACHEDIRECTORY = /var/lib/openqa/cache
WORKER_CLASS = qemu_aarch64,qemu_aarch32,heavyload

[http://openqa1-opensuse]
TESTPOOLSERVER = rsync://openqa1-opensuse/tests

Is rsync available from Internet?

And I think it would be better to rename openqa1-opensuse to openqa.opensuse.org.

ggardet_arm wrote:

Is rsync available from Internet?

It looks like it is not:

rsync: [Receiver] failed to connect to openqa.opensuse.org (2001:67c:2178:8::16): Connection timed out (110)
rsync: [Receiver] failed to connect to openqa.opensuse.org (195.135.221.140): Connection timed out (110)
rsync error: error in socket IO (code 10) at clientserver.c(137) [Receiver=3.2.3]

So, we cannot go this way.

May I set this ticket as 'Resolve'?

Hm, I don't think we have actually resolved the issue yet. In the case of test distributions where needles are in a separate directory and we have differing workers config, e.g. some with caching, some without, we still run into problems.

@Xiaojing_liu do you think the acceptance criteria as currently specified are fulfilled?

I think we can improve the documentation to mention the limitations. Also we can try to handle this situation better, e.g. look in multiple alternative paths when looking for needles.

PR: https://github.com/os-autoinst/openQA/pull/3578
https://github.com/os-autoinst/os-autoinst/pull/1569

https://github.com/os-autoinst/openQA/pull/3604
https://github.com/os-autoinst/os-autoinst/pull/1577

Today multiple persons met and we discussed the ticket and various points around it. Important points to mention:

• https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc describes the current test variables that are read and written by isotovideo
• https://github.com/os-autoinst/openQA/blob/master/docs/Installing.asciidoc#terms-and-variables-for-certain-directories-used-by-openqa-and-isotovideo has an overview of path related variables from openQA side
• openQA-in-openQA tests like https://openqa.opensuse.org/tests/1506935# using https://github.com/os-autoinst/os-autoinst-distri-openQA/ show an example of a test distribution that have main.pm in the root folder so we can not strictly set productdir=$casedir/products/$distri
• We need to keep in mind that openQA has job settings stored in the database, which are also used by the clone script. The worker updates vars.json, overwrites some variables. os-autoinst uses vars.json and also overwrites some variables and adds more variables to vars.json. vars.json is also used by openqa-clone-custom-git-refspec

Ideas:

• In every step overwrite less or no variables but instead only define new ones instead of overwriting input parameters
• Use relative paths where possible, e.g. absolute PRJDIR or SHAREDIR but CASEDIR, PRODUCTDIR, NEEDLESDIR relative to PRJDIR or SHAREDIR

With https://github.com/os-autoinst/os-autoinst/pull/1606 merged which superseded previous PRs, is there more to be done here? @Xiaojing_liu

cdywan wrote:

With https://github.com/os-autoinst/os-autoinst/pull/1606 merged which superseded previous PRs, is there more to be done here? @Xiaojing_liu

https://github.com/os-autoinst/openQA/pull/3604 There is an open pr which is waiting for feedback.

https://github.com/os-autoinst/openQA/pull/3604 was merged and reverted again in https://github.com/os-autoinst/openQA/pull/3693 after an error report https://github.com/os-autoinst/openQA/pull/3604#issuecomment-765888161

Another pr to fix that issue: https://github.com/os-autoinst/openQA/pull/3696

We decided to merge the PR after we added additional unit tests and not finding easy cases to test pre-merge. We observed openQA "last_good_tests" openQA investigation jobs incompleted due to being unable to find the main schedule files, e.g.
openqa.opensuse.org/tests/1609706
On openqa.opensuse.org/tests?match=last_good_test I also saw the suspicion confirmed that any "last_good_tests" investigation job since this day (since the new version is deployed) are incomplete except of the ones on the old aarch64 workers which still have the old version.

Further notes from chat about the reason and our choices:

@Xiaojing_liu
15:02
Oliver Kurz I checked others incomplete, have the same reason. Actually I confuse about the scenario. If we supply a git url as CASEDIR, and use the old PRODUCTDIR value, the main.pm in productdir will not make sense. right? I guess it's the reason why we supply RPODUCTDIR in openqa-clone-custom-git-refspec together.

okurz
Oliver Kurz @okurz
15:04
the reason why I supplied PRODUCTDIR in openqa-clone-custom-git-refspec initially was a workaround because it is needed. Let me check the openQA code what is expected and what should be done for the future

okurz
Oliver Kurz @okurz
15:07
right. https://github.com/os-autoinst/openQA/blob/1998b3cde8c85be95d8ddab6933d7877c9437670/lib/OpenQA/Utils.pm#L127 explains it: Normally the user should not need to specify PRODUCTDIR because we have the line return $dir . "/products/$distri" if -e "$dir/products/$distri"; in https://github.com/os-autoinst/openQA/blob/1998b3cde8c85be95d8ddab6933d7877c9437670/lib/OpenQA/Utils.pm#L142 . And if someone wants to use CASEDIR= and keep DISTRI in before there should be no need to specify PRODUCTDIR, only if one wants to rename "DISTRI" to a value that would not correspond to "$casedir/products/$distri/main.pm" then PRODUCTDIR would need to be set

Please help to confirm those points:

1. If users only supply CASEDIR=https://github.com/os-autoinst/os-autoinst-distri-opensuse.git#6f7510c5fc8d0c3c46a1d2154f5fc54296c22bd5, we should define PRODUCTDOR=/var/lib/openqa/share/tests/opensuse/products/opensuse? This is the behavior for now. In this scenario, if the push includes the modification for products/opensuse/main.pm, it doesn't work, right? Because the PRODUCTDIR is not under the CASEDIR.
2. There is also an issue of openqa-clone-custom-git-refspec. If use this script to trigger a new job (job's CASEDIE=opensuse, PRODUCTDIR=opensuse/products/opensuse), the NEEDLES_DIR will be set as opensuse/products/opensuse/needles. it's not correct. we could modify this script to set this value as needles, or modify the Worker code to Compatible with it.

According note#44, commit new pr: https://github.com/os-autoinst/openQA/pull/3712

Regression found with #89990 . I will handle the rollback in #89990 . Can you try to understand the specific test requirements and include in your changes as well?

Further ideas for improvements:

• Instead of enabling the new functionality by default, introduce an experimental feature switch so that we do not have to revert the source code, only the switch configuration
• We should have an alert for the broken jobs caused by the change in behaviour -> Check the existing alert for "new incompletes"

From the weekly:

• no alerts due to incomplete alerts not firing, this was expected
• use feature switch approach similar to faster localhost uploads
• enable via job setting (could be machine, job group, etc)

PR for this regression issue: https://github.com/os-autoinst/os-autoinst/pull/1634

Yes, we should bring back the PR but only enable the changed behaviour based on a feature switch, ok?

As discussed with @cdywan, we agreed with using a job setting to enable or disable this feature. Will commit a new pull request later.

pr: https://github.com/os-autoinst/openQA/pull/3799