action #93101

investigation jobs scheduled against specific git hashes load wrong product dir files

Added by okurz 2 months ago. Updated about 1 month ago.

Concrete Bugs
Target version:
Start date:
Due date:
% Done:


Estimated time:


Observation shows investigation jobs, e.g. textmode:investigate:last_good_tests:f4a80a14155f874200833e833d30edac634ad9a1: showing the error:

[2021-05-25T14:38:16.121 CEST] [debug] git hash in /var/lib/openqa/pool/1/os-autoinst-distri-opensuse: f4a80a14155f874200833e833d30edac634ad9a1
Bareword "load_pynfs_tests" not allowed while "strict subs" in use at /var/lib/openqa/share/tests/opensuse/products/opensuse/ line 273.
Compilation failed in require at /usr/lib/os-autoinst/OpenQA/Isotovideo/ line 211.

As the path /var/lib/openqa/share/tests/opensuse/products/opensuse/ is mentioned it looks like the default path is preferred when instead products/opensuse/ should be read from the local git checkout directory. This might be a regression from work on #80372 or a subticket, e.g. #67723 or #90302

Acceptance criteria

  • AC1: openQA investigate tests against "last good tests" should read test code from the same git hash, i.e. from the local git checkout directory


  • check if this is a regression introduced
  • this likely can be solved within os-autoinst only


#1 Updated by mkittler 2 months ago

  • Assignee set to mkittler

#2 Updated by mkittler 2 months ago

  • Status changed from Workable to In Progress

I've just created a custom branch which prints Running custom version at the beginning of and it works as expected. However, in the same log a Perl warning appears similarly to the one mentioned in the ticket description:

Use of uninitialized value in pattern match (m//) at /hdd/openqa-devel/openqa/pool/1/os-autoinst-distri-opensuse/lib/ line 697.
    main_common::remove_common_needles() called at /hdd/openqa-devel/openqa/share/tests/opensuse/products/opensuse/ line 36
    OpenQA::Isotovideo::Utils::cleanup_needles() called at /hdd/openqa-devel/repos/os-autoinst/ line 329
    needle::init() called at /hdd/openqa-devel/repos/os-autoinst/isotovideo line 270

The code sets unshift @INC, '.'; before the require. Apparently it is only half-effective.

#3 Updated by mkittler 2 months ago

The problem is actually that PRODUCTDIR is set to the default by the worker using an absolute path. So os-autoinst actually behaves as expected (test libraries are used from Git, also loaded from default products directory).

Draft to fix this on the worker-side:

#4 Updated by openqa_review 2 months ago

  • Due date set to 2021-06-10

Setting due date based on mean cycle time of SUSE QE Tools

#5 Updated by mkittler about 2 months ago

  • Status changed from In Progress to Feedback

#6 Updated by okurz about 2 months ago

  • Due date changed from 2021-06-10 to 2021-06-15 merged. This needs validation in production

#7 Updated by mkittler about 1 month ago

  • Status changed from Feedback to Resolved

It looks like variables are now set as expected in production, in particular PRODUCT_DIR is now a relative path (e.g. This should now work. Of course it is hard to tell from the logs which files have actually been loaded unless there's an error caused by loading the wrong file. I've checked some recent incompletes and didn't find such errors (only a lot with e.g. Could not find '24eddcb8631f572d64236898aaac3dcfe5a14b06' in complete history at /usr/lib/os-autoinst/OpenQA/Isotovideo/ line 96..).

Also available in: Atom PDF