Project

General

Profile

action #48389

coordination #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule

self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo

Added by okurz almost 3 years ago. Updated 27 days ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Feature requests
Target version:
Start date:
2019-02-25
Due date:
% Done:

0%

Estimated time:
Difficulty:

Description

Motivation

see parent ticket #37958

Acceptance criteria

  • AC1: travis CI tests on PRs fail if a simple "openQA test" can not succeed

Suggestions

  • Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
  • Experiment how we could use a stable staging medium to test
  • We can start isotovideo with vars.json using EXIT_AFTER_SCHEDULE, which will already help
  • We should check if including modules helps to try
  • We aim just to catch more errors as we will have some code executed

Related issues

Related to openQA Project - action #10784: test framework for openQA test distributionsWorkable2016-02-17

History

#1 Updated by riafarov over 2 years ago

  • Due date set to 2019-07-16

Simplest thing we can try is just to trigger isotovideo to evaluate some modules and see if it fails with some compilation error. We need to check if we can include many modules to have this covered.

#2 Updated by okurz over 2 years ago

  • Related to action #10784: test framework for openQA test distributions added

#3 Updated by riafarov over 2 years ago

  • Description updated (diff)

#4 Updated by riafarov over 2 years ago

  • Estimated time set to 8.00 h

#5 Updated by oorlov over 2 years ago

  • Assignee set to oorlov

#6 Updated by oorlov over 2 years ago

  • Status changed from Workable to In Progress

#7 Updated by oorlov over 2 years ago

  • Status changed from In Progress to Rejected

I've started openQA tests with the minimal vars.json file and isotovideo.

{
   "ARCH" : "x86_64",
   "ASSETDIR" : "/var/lib/openqa/share/factory",
   "BACKEND" : "qemu",
   "CASEDIR" : "/path/to/os-autoinst-distri-opensuse",
   "DESKTOP" : "gnome",
   "DISTRI" : "sle",
   "FLAVOR" : "Server-DVD",
   "ISO" : "/var/lib/openqa/factory/iso/SLE-12-SP5-Server-DVD-x86_64-Build0201-Media1.iso",
   "JOBTOKEN" : "hUpWvBMnjC",
   "NEEDLES_DIR" : "/path/to/os-autoinst-distri-opensuse/products/sle/needles",
   "PRJDIR" : "/path/to/os-autoinst-distri-opensuse",
   "PRODUCTDIR" : "/path/to/os-autoinst-distri-opensuse/products/sle",
   "QEMUPORT" : 15222,
   "TEST" : "test",
   "VERSION" : "12-SP5",
   "VNC" : "91",
   "_EXIT_AFTER_SCHEDULE" : 1
}

It doesn't give any valuable advantages. For example, if remove imports (e.g. remove use registration; from bootloader.pm) and run isotovideo, the will be no compilation or runtime errors.

The other problem is that running isotovideo with "_EXIT_AFTER_SCHEDULE" : 1 exits with "1" in console, which means negative result of execution, though it was executed correctly and without errors. So, it does not allow to rely on the result of execution.

After discussing with riafarov it was decided to close the ticket as it cannot be resolved with the initial plan.

We need to re-think what we need to test and how it should be done.

#8 Updated by okurz about 2 years ago

  • Start date set to 2019-02-25

due to changes in a related task

#9 Updated by okurz about 2 years ago

  • Subject changed from [functional][y] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo to [tools] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo
  • Due date deleted (2019-07-16)
  • Status changed from Rejected to In Progress
  • Assignee changed from oorlov to okurz
  • Start date deleted (2019-02-25)
  • Estimated time deleted (8.00 h)

In light of the recent epic #58184 we brought up within the tools team I will give this another try.

#10 Updated by okurz about 2 years ago

So I started with this in branch "feature/isotovideo_ci" , see https://github.com/okurz/os-autoinst-distri-opensuse/tree/feature/isotovideo_ci . Wanna see travis CI results first, should be triggered.

#11 Updated by okurz about 2 years ago

  • Status changed from In Progress to Workable
  • Assignee deleted (okurz)

https://travis-ci.org/okurz/os-autoinst-distri-opensuse/builds/613699612 shows the latest travis CI tests. What I tried in https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...okurz:feature/isotovideo_ci is to start isotovideo evaluating the schedule and comparing that against a reference but so far isotovideo failed to startup due to missing dependencies as visible in https://travis-ci.org/okurz/os-autoinst-distri-opensuse/jobs/613699616#L18288 for IPC::System::Simple . https://travis-ci.org/okurz/os-autoinst-distri-opensuse/jobs/613699613 should show all cpan dependencies. Tried to patch this but so far with no success.

#12 Updated by okurz over 1 year ago

  • Category set to Feature requests

#13 Updated by okurz over 1 year ago

  • Subject changed from [tools] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo to self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo

#14 Updated by okurz about 2 months ago

  • Status changed from Workable to In Progress
  • Assignee set to okurz
  • Target version changed from future to Ready

#15 Updated by okurz about 2 months ago

  • Due date set to 2021-10-27
  • Status changed from In Progress to Feedback

PR fully tested and ready. After merge we just need to update the URL for the Dockerfile from which the specified container is built

#16 Updated by okurz about 1 month ago

  • Due date changed from 2021-10-27 to 2021-11-10

the container image path was updated. After 2 weeks with no feedback I now merged the PR myself. With this being part of a test that runs for every pull request I would like to monitor the impact over the next days.

#17 Updated by okurz 27 days ago

  • Due date deleted (2021-11-10)
  • Status changed from Feedback to Resolved

Also available in: Atom PDF