action #20206
closed[functional][y][yast][easy] Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)
Added by lnussel over 7 years ago. Updated about 4 years ago.
0%
Description
WAS: please remove yast2_cmdline from the 42.3 test plan. It fails all the time and doesn't seem to serve a useful purpose.
Motivation¶
We would like to run yast command line tests (see http://yastgithubio.readthedocs.io/en/latest/how-to-write-tests/) against packages.
Test module was abandoned for a while, but this part is not covered anywhere at the moment.
Acceptance criteria¶
- yast2_cmdline test module is working or failing on a valid bug
- yast2_cmdline test module is executed in job group which is reviewed (not development job groups)
Suggestions¶
- Learn how to run tests and adjust code
- For openSUSE it's tricky as we don't sync source repo, but still can get tests from official source repo, elaborate with riafarov or okurz
- Adjust schedule to run test module as a part of extra tests in textmode or as a separate test suite
Adding extra source package to be synced should be added here: https://gitlab.suse.de/openqa/scripts/blob/master/rsync_opensuse.pm#L459
Updated by okurz over 7 years ago
- Status changed from New to In Progress
yast2_cmdline also fails consistently on openSUSE Tumbleweed. There it's marked with a bug but not regarded as high priority. On SLES currently we don't run this module (waiting for zluo here). I will try to exclude yast2_cmdline by default as it highly depends on external test packages by developers so I will split the scenarios.
Updated by okurz over 7 years ago
Updated by okurz over 7 years ago
PR merged, test module does no harm anymore for the yast2_ui scenario. Created new test suite "yast2_ui_devel" triggered after yast2_ui and added to development job groups for openSUSE Tumbleweed as well as openSUSE Leap.
Should check how the test module behaves for SLE
Updated by okurz over 7 years ago
SLE tests fails in http://lord.arch/tests/latest#step/yast2_cmdline/6 with the source repo not being found. Would need to add iso2
Updated by okurz over 7 years ago
- Status changed from In Progress to Feedback
- Priority changed from High to Low
waiting for SLE15 to progress further until I would continue to try to add it back to the SLE test plan acordingly
Updated by okurz about 7 years ago
- Subject changed from please remove yast2_cmdline from 42.3 test plan to Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)
Added "yast2_ui_devel" to SLE15 test plan for now in the development group. Let's gather some testing results before evaluation.
Updated by okurz almost 7 years ago
- Target version set to Milestone 16
yast2_ui_devel was not executed for SLE15 ever since yast2_ncurses always fails. Let's see how it handles the RC phase of SLE15.
Updated by okurz almost 7 years ago
- Subject changed from Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan) to [functional]Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)
Updated by okurz almost 7 years ago
- Subject changed from [functional]Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan) to [functional][y][yast]Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)
- Status changed from Feedback to New
- Assignee deleted (
okurz)
Over to [y]-team
Updated by okurz over 6 years ago
- Subject changed from [functional][y][yast]Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan) to [functional][y][yast] Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)
- Due date set to 2018-06-05
@riafarov I think this is something you could clarify with the yast team
Updated by riafarov over 6 years ago
- Description updated (diff)
- Status changed from New to Workable
Updated by mloviska over 6 years ago
- Status changed from Workable to In Progress
- Assignee set to mloviska
Updated by mloviska over 6 years ago
Issue with hdd_1 variable.
[2018-05-31T04:55:48.0972 CEST] [debug] CACHE: Download of /var/lib/openqa/cache/SLES-15-x86_64-657.1-gnome@64bit-gnome.qcow2 failed with: 404 - Not Found
[2018-05-31T04:55:48.0973 CEST] [debug] CACHE: removed /var/lib/openqa/cache/SLES-15-x86_64-657.1-gnome@64bit-gnome.qcow2
Updated by mloviska over 6 years ago
qcow2 image is missing "Module development tools", hence we are getting missing package error.
Possible fix using SUSEConnect -p sle-module-development-tools/15/x86_64
Updated by mloviska over 6 years ago
Successful run:
http://dhcp151.suse.cz/tests/3380#step/yast2_cmdline/16
Tested modules:
- yast2-network
- yast2-http-server
OpenQA variables:
- DESKTOP=textmode
- HDD_1=sle-15-x86_64-661.1@64bit-minimal_with_sdk661.1_installed.qcow2
- Y2UITEST_DEVEl=1
- Y2UITEST_NCURSES=1
Updated by riafarov over 6 years ago
- Due date changed from 2018-06-05 to 2018-06-19
Updated by riafarov over 6 years ago
- Target version changed from Milestone 16 to Milestone 17
Updated by mgriessmeier over 6 years ago
- Subject changed from [functional][y][yast] Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan) to [functional][y][yast][easy] Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)
Updated by mloviska over 6 years ago
PR to make it work on sle
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/5171
Updated by mloviska over 6 years ago
- Status changed from In Progress to Feedback
Updated by okurz over 6 years ago
- Target version changed from Milestone 17 to Milestone 17
Updated by mloviska over 6 years ago
test suite yast2_ui_devel added to sle12sp4's functional group.
Updated by riafarov over 6 years ago
Once we have run on osd, please, set as blocked with https://progress.opensuse.org/issues/34732
Updated by mloviska over 6 years ago
- Status changed from Feedback to Blocked
Updated by mloviska over 6 years ago
- Has duplicate action #34732: [functiona][y][opensuse] make zypper_info use mirrored source repo, do not remove official update repos added
Updated by mloviska over 6 years ago
- Has duplicate deleted (action #34732: [functiona][y][opensuse] make zypper_info use mirrored source repo, do not remove official update repos)
Updated by mloviska over 6 years ago
- Blocked by action #34732: [functiona][y][opensuse] make zypper_info use mirrored source repo, do not remove official update repos added
Updated by okurz over 6 years ago
mloviska wrote:
test suite yast2_ui_devel added to sle12sp4's functional group.
I think in this case it is better to track the scenario in "test development". We should not add scenarios that never worked and we don't know if they will ever work
Updated by mloviska over 6 years ago
This one is already working for sle15, however I haven't made it in time so we can have runs in OSD. I am afraid that test case will be forgotten by moving it to dev group. Considering that the job is currently block by missing dependency it might be actually easier to track the bug in functional group than in dev.
Updated by okurz over 6 years ago
mloviska wrote:
I am afraid that test case will be forgotten by moving it to dev group
Yes, I share your concerns. This is why we still have this progress ticket. Based on that the ticket is still open and blocked we now that the test can not work. The job within openQA should also be labeled by this. The validation job groups in openQA should be reserved for tests that we expect to work. I know we did not apply this cleanly during the past months and review turned out to miss some failures which never went away. Btw, this has been mentioned as a concern by RMs as well so the suggestion is to apply more strictly our best practices we followed in before: Whenever we know that a test will not be fixed very soon we track it in "test development" based on progress tickets. It's easier for the progress ticket assignee to review a single job and determine if the job failure reason is still the same in contrast to the (overwhelmed) daily reviewer trying to get a hold of all failures.
Updated by mloviska over 6 years ago
Valid point, yast2_ui_devel is currently in dev group of sle12sp4.
Updated by riafarov over 6 years ago
- Due date changed from 2018-06-19 to 2018-07-03
Updated by okurz over 6 years ago
- Due date changed from 2018-07-03 to 2018-08-28
- Target version changed from Milestone 17 to Milestone 18
Considering blocked now by the mentioned bug. I do not expect the resolution within the timeframe of the next sprint so let's take a look at the latest again in S24. Of course if the bug is resolved in before feel free to bring it up again.
Updated by mloviska over 6 years ago
Runs fine on sle15 sle-15-Installer-DVD-x86_64-Build665.2-yast2_ui_devel@64bit
Updated by riafarov over 6 years ago
- Due date changed from 2018-08-28 to 2018-09-25
I do not expect this to get unblocked soon, @mloviska, what do you think? Maybe we should put it back to the backlog instead of checking state.
Updated by mloviska over 6 years ago
- Due date deleted (
2018-09-25)
I am going to be informed by mail anyway, thus it should be fine to put this one back to backlog again. Nevertheless we need to wait what is going to happen with Bug 1096302 - rspec from BSK cannot be installed first. I will remove the due date for now.
Updated by okurz over 6 years ago
- Target version changed from Milestone 18 to Milestone 26
right, let's review later again
Updated by okurz about 6 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: yast2_ui_devel
https://openqa.suse.de/tests/2096097
Updated by mloviska about 6 years ago
In case of test case bug as this one. We can update the test suite by ourselves
Updated by mloviska about 6 years ago
PR to fix type for yast2 lan module
https://github.com/yast/yast-network/pull/686
Updated by mloviska over 5 years ago
Maybe we can decommission this module and related test suites since we test CLI only for 2 yast2 modules and QAM is working on their own way how to test CLI for yast2 modules anyway.
QAM test plan
Updated by riafarov over 5 years ago
- Due date set to 2019-08-27
- Status changed from Blocked to Workable
- Target version changed from Milestone 26 to Milestone 27
Not sure why it was blocked. In recent runs we miss some source packages, which should be easy to add even in case we don't sync them.
Might be that there are more issues hidden by the first one, but let's try.
Updated by riafarov over 5 years ago
- Description updated (diff)
- Estimated time set to 5.00 h
Updated by riafarov over 5 years ago
- Due date changed from 2019-08-27 to 2019-09-10
Updated by JERiveraMoya over 5 years ago
- Status changed from Workable to In Progress
- Assignee set to JERiveraMoya
Updated by JERiveraMoya over 5 years ago
Updated by JERiveraMoya over 5 years ago
- Status changed from In Progress to Feedback
Updated by JERiveraMoya over 5 years ago
Updated by JERiveraMoya over 5 years ago
Now the package is present. Created bug: Broken yast command line tests
Updated by JERiveraMoya over 5 years ago
- Status changed from Feedback to Blocked
Updated by mgriessmeier over 5 years ago
- Target version changed from Milestone 27 to Milestone 28
Updated by JERiveraMoya over 5 years ago
- Due date changed from 2019-09-10 to 2019-10-08
Updated by riafarov about 5 years ago
- Due date changed from 2019-10-08 to 2019-11-05
- Target version changed from Milestone 28 to Milestone 29
Looks like we can work on this one soon, once package reaches the repo. Moving to the sprint after next one.
Updated by riafarov about 5 years ago
- Due date changed from 2019-11-05 to 2020-02-11
- Target version changed from Milestone 29 to Milestone 30+
https://openqa.opensuse.org/tests/1077392 still shows same issue.
Updated by mgriessmeier almost 5 years ago
- Target version changed from Milestone 30+ to Milestone 30
bulk moved to M30 for revisiting
Updated by riafarov almost 5 years ago
- Due date deleted (
2020-02-11) - Target version changed from Milestone 30 to future
Same issue: https://openqa.opensuse.org/tests/1166336#
Updated by riafarov about 4 years ago
- Project changed from openQA Tests (public) to qe-yam
- Category deleted (
Bugs in existing tests) - Status changed from Blocked to Resolved
Isn't it done now? I see https://openqa.opensuse.org/tests/1441560# being green for a months now. Feel free to reopen in case you believe I miss something.