action #48389
closedcoordination #37958: [epic] self-tests in os-autoinst-distri-opensuse for impact on staging test schedule
self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo
Description
Motivation¶
see parent ticket #37958
Acceptance criteria¶
- AC1: travis CI tests on PRs fail if a simple "openQA test" can not succeed
Suggestions¶
- Add a fitting openSUSE staging test vars.json to the repo for test purposes (could be copied from existing ones and stripped down to bare minimum what is needed or newly created)
- Experiment how we could use a stable staging medium to test
- We can start isotovideo with vars.json using EXIT_AFTER_SCHEDULE, which will already help
- We should check if including modules helps to try
- We aim just to catch more errors as we will have some code executed
Updated by riafarov over 5 years ago
- Due date set to 2019-07-16
Simplest thing we can try is just to trigger isotovideo to evaluate some modules and see if it fails with some compilation error. We need to check if we can include many modules to have this covered.
Updated by okurz over 5 years ago
- Related to action #10784: test framework for openQA test distributions added
Updated by oorlov over 5 years ago
- Status changed from In Progress to Rejected
I've started openQA tests with the minimal vars.json file and isotovideo.
{
"ARCH" : "x86_64",
"ASSETDIR" : "/var/lib/openqa/share/factory",
"BACKEND" : "qemu",
"CASEDIR" : "/path/to/os-autoinst-distri-opensuse",
"DESKTOP" : "gnome",
"DISTRI" : "sle",
"FLAVOR" : "Server-DVD",
"ISO" : "/var/lib/openqa/factory/iso/SLE-12-SP5-Server-DVD-x86_64-Build0201-Media1.iso",
"JOBTOKEN" : "hUpWvBMnjC",
"NEEDLES_DIR" : "/path/to/os-autoinst-distri-opensuse/products/sle/needles",
"PRJDIR" : "/path/to/os-autoinst-distri-opensuse",
"PRODUCTDIR" : "/path/to/os-autoinst-distri-opensuse/products/sle",
"QEMUPORT" : 15222,
"TEST" : "test",
"VERSION" : "12-SP5",
"VNC" : "91",
"_EXIT_AFTER_SCHEDULE" : 1
}
It doesn't give any valuable advantages. For example, if remove imports (e.g. remove use registration; from bootloader.pm) and run isotovideo, the will be no compilation or runtime errors.
The other problem is that running isotovideo with "_EXIT_AFTER_SCHEDULE" : 1 exits with "1" in console, which means negative result of execution, though it was executed correctly and without errors. So, it does not allow to rely on the result of execution.
After discussing with @riafarov it was decided to close the ticket as it cannot be resolved with the initial plan.
We need to re-think what we need to test and how it should be done.
Updated by okurz about 5 years ago
- Subject changed from [functional][y] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo to [tools] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo
- Due date deleted (
2019-07-16) - Status changed from Rejected to In Progress
- Assignee changed from oorlov to okurz
- Start date deleted (
2019-02-25) - Estimated time deleted (
8.00 h)
In light of the recent epic #58184 we brought up within the tools team I will give this another try.
Updated by okurz almost 5 years ago
So I started with this in branch "feature/isotovideo_ci" , see https://github.com/okurz/os-autoinst-distri-opensuse/tree/feature/isotovideo_ci . Wanna see travis CI results first, should be triggered.
Updated by okurz almost 5 years ago
- Status changed from In Progress to Workable
- Assignee deleted (
okurz)
https://travis-ci.org/okurz/os-autoinst-distri-opensuse/builds/613699612 shows the latest travis CI tests. What I tried in https://github.com/os-autoinst/os-autoinst-distri-opensuse/compare/master...okurz:feature/isotovideo_ci is to start isotovideo evaluating the schedule and comparing that against a reference but so far isotovideo failed to startup due to missing dependencies as visible in https://travis-ci.org/okurz/os-autoinst-distri-opensuse/jobs/613699616#L18288 for IPC::System::Simple . https://travis-ci.org/okurz/os-autoinst-distri-opensuse/jobs/613699613 should show all cpan dependencies. Tried to patch this but so far with no success.
Updated by okurz over 4 years ago
- Subject changed from [tools] self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo to self-tests in os-autoinst-distri-opensuse executing a simple (staging) test using isotovideo
Updated by okurz about 3 years ago
- Status changed from Workable to In Progress
- Assignee set to okurz
- Target version changed from future to Ready
Updated by okurz about 3 years ago
- Due date set to 2021-10-27
- Status changed from In Progress to Feedback
PR fully tested and ready. After merge we just need to update the URL for the Dockerfile from which the specified container is built
Updated by okurz about 3 years ago
- Due date changed from 2021-10-27 to 2021-11-10
the container image path was updated. After 2 weeks with no feedback I now merged the PR myself. With this being part of a test that runs for every pull request I would like to monitor the impact over the next days.
Updated by okurz almost 3 years ago
- Due date deleted (
2021-11-10) - Status changed from Feedback to Resolved
Everything looks good so far. Some example pull request tests:
- https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13648 showing all good with a passed job https://github.com/os-autoinst/os-autoinst-distri-opensuse/runs/4152111008?check_suite_focus=true
- https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13645 with currently broken changes that rightfully cause a test failure https://github.com/os-autoinst/os-autoinst-distri-opensuse/runs/4152798500?check_suite_focus=true