QA (public) » openQA Project (public) » openQA Tests (public) » Containers and images

Test cases examples:¶

• xfail case: test test_tensorf, Tensorflow install ko for pip version:
  https://openqa.suse.de/tests/9114118#external;
  See also the serial_terminal.txt log in https://openqa.suse.de/tests/9114118/logfile?filename=serial_terminal.txt,
  then find in the page (ctrl-F): xfail

• Also possible to check other similar xfail events in the tests on BCI language python 3.10 or 3.9 in the group https://openqa.suse.de/group_overview/444

In java we have XFAILs too

This ticket was set to High priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.

The BCI tests are written in python and tested with 'pytest' using 'tox'. The openqa container tests start with the Perl module bci_test.pm that run in a bash session the tox command composed with proper parameters based on the test settings.

The pytest exit codes are 5 specific values but where xfail occurrence is not visible, that is i.e., if only xfail occurr in the test execution, we have no evidence of it and zero is returned as all pass.

Therefore the simplest way to detect xfail in the openqa bci_test.pm module seems to be,after tox ran with the pytest '-rxX' option added, to execute a pytest logs inspection searching for XFAIL string and manage the result in the Perl module, in order to assign a Soft Failure.

So, to verify the validity of this method, I am cloning some tests using a temporary modifyed branch of the BCI test with additional XFAIL tests in test_all.py, that should appear in the log and proof that behaviour.

After the BCI test code updated, the proof cloning some tests of the Containers/BCI Base Images 15-SP3 group are ok and report Soft Failure where XFAIL is present.

But depending on the test function input parameters, if needed to select all the xfail also where the MARKER is missing,it could be useful to add to the MARKER parameter an "or" expression, like:
BCI_IMAGE_MARKER = $BCI_IMAGE_MARKER + " or XFAIL" .

I.e.: BCI_IMAGE_MARKER="bci/bci-init_15.3 or XFAIL"

In this way the pytest -k filter will also select all the tests resulting XFAIL.

This ticket was set to High priority but was not updated within the SLO period. Please consider picking up this ticket or just set the ticket to the next lower priority.