action #48389

action #37958: [functional][y][epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule

[tools] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo

Added by okurz 12 months ago. Updated 3 months ago.

Status:WorkableStart date:25/02/2019
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:Enhancement to existing tests
Target version:QA - future
Difficulty:
Duration:

Description

Motivation

see parent ticket #37958

Acceptance criteria

  • AC1: travis CI tests on PRs fail if a simple "openQA test" can not succeed

Suggestions

  • Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
  • Experiment how we could use a stable staging medium to test
  • We can start isotovideo with vars.json using EXIT_AFTER_SCHEDULE, which will already help
  • We should check if including modules helps to try
  • We aim just to catch more errors as we will have some code executed

Related issues

Related to openQA Project - action #10784: test framework for openQA test distributions New 17/02/2016

History

#1 Updated by riafarov 9 months ago

  • Due date set to 16/07/2019

Simplest thing we can try is just to trigger isotovideo to evaluate some modules and see if it fails with some compilation error. We need to check if we can include many modules to have this covered.

#2 Updated by okurz 8 months ago

  • Related to action #10784: test framework for openQA test distributions added

#3 Updated by riafarov 8 months ago

  • Description updated (diff)

#4 Updated by riafarov 8 months ago

  • Estimated time set to 8.00

#5 Updated by oorlov 8 months ago

  • Assignee set to oorlov

#6 Updated by oorlov 8 months ago

  • Status changed from Workable to In Progress

#7 Updated by oorlov 8 months ago

  • Status changed from In Progress to Rejected

I've started openQA tests with the minimal vars.json file and isotovideo.

{
   "ARCH" : "x86_64",
   "ASSETDIR" : "/var/lib/openqa/share/factory",
   "BACKEND" : "qemu",
   "CASEDIR" : "/path/to/os-autoinst-distri-opensuse",
   "DESKTOP" : "gnome",
   "DISTRI" : "sle",
   "FLAVOR" : "Server-DVD",
   "ISO" : "/var/lib/openqa/factory/iso/SLE-12-SP5-Server-DVD-x86_64-Build0201-Media1.iso",
   "JOBTOKEN" : "hUpWvBMnjC",
   "NEEDLES_DIR" : "/path/to/os-autoinst-distri-opensuse/products/sle/needles",
   "PRJDIR" : "/path/to/os-autoinst-distri-opensuse",
   "PRODUCTDIR" : "/path/to/os-autoinst-distri-opensuse/products/sle",
   "QEMUPORT" : 15222,
   "TEST" : "test",
   "VERSION" : "12-SP5",
   "VNC" : "91",
   "_EXIT_AFTER_SCHEDULE" : 1
}

It doesn't give any valuable advantages. For example, if remove imports (e.g. remove use registration; from bootloader.pm) and run isotovideo, the will be no compilation or runtime errors.

The other problem is that running isotovideo with "_EXIT_AFTER_SCHEDULE" : 1 exits with "1" in console, which means negative result of execution, though it was executed correctly and without errors. So, it does not allow to rely on the result of execution.

After discussing with @riafarov it was decided to close the ticket as it cannot be resolved with the initial plan.

We need to re-think what we need to test and how it should be done.

#8 Updated by okurz 4 months ago

  • Start date set to 25/02/2019

due to changes in a related task

#9 Updated by okurz 4 months ago

  • Subject changed from [functional][y] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo to [tools] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo
  • Due date deleted (16/07/2019)
  • Status changed from Rejected to In Progress
  • Assignee changed from oorlov to okurz
  • Start date deleted (25/02/2019)
  • Estimated time deleted (8.00)

In light of the recent epic #58184 we brought up within the tools team I will give this another try.

#10 Updated by okurz 3 months ago

So I started with this in branch "feature/isotovideo_ci" , see https://github.com/okurz/os-autoinst-distri-opensuse/tree/feature/isotovideo_ci . Wanna see travis CI results first, should be triggered.

#11 Updated by okurz 3 months ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

https://travis-ci.org/okurz/os-autoinst-distri-opensuse/builds/613699612 shows the latest travis CI tests. What I tried in https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...okurz:feature/isotovideo_ci is to start isotovideo evaluating the schedule and comparing that against a reference but so far isotovideo failed to startup due to missing dependencies as visible in https://travis-ci.org/okurz/os-autoinst-distri-opensuse/jobs/613699616#L18288 for IPC::System::Simple . https://travis-ci.org/okurz/os-autoinst-distri-opensuse/jobs/613699613 should show all cpan dependencies. Tried to patch this but so far with no success.

Also available in: Atom PDF