Project

General

Profile

action #20206

[functional][y][yast][easy] Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)

Added by lnussel almost 4 years ago. Updated 6 months ago.

Status:
Resolved
Priority:
Low
Assignee:
Target version:
Start date:
2017-07-02
Due date:
% Done:

0%

Estimated time:
5.00 h

Description

WAS: please remove yast2_cmdline from the 42.3 test plan. It fails all the time and doesn't seem to serve a useful purpose.

Motivation

We would like to run yast command line tests (see http://yastgithubio.readthedocs.io/en/latest/how-to-write-tests/) against packages.
Test module was abandoned for a while, but this part is not covered anywhere at the moment.

Acceptance criteria

  1. yast2_cmdline test module is working or failing on a valid bug
  2. yast2_cmdline test module is executed in job group which is reviewed (not development job groups)

Suggestions

  1. Learn how to run tests and adjust code
  2. For openSUSE it's tricky as we don't sync source repo, but still can get tests from official source repo, elaborate with riafarov or okurz
  3. Adjust schedule to run test module as a part of extra tests in textmode or as a separate test suite

Here https://openqa.opensuse.org/tests/latest?arch=x86_64&distri=opensuse&flavor=DVD&machine=64bit&test=yast2_ui_devel&version=Tumbleweed

Adding extra source package to be synced should be added here: https://gitlab.suse.de/openqa/scripts/blob/master/rsync_opensuse.pm#L459


Related issues

Blocked by openQA Tests - action #34732: [functiona][y][opensuse] make zypper_info use mirrored source repo, do not remove official update reposResolved2018-04-112019-01-15

History

#1 Updated by okurz almost 4 years ago

  • Status changed from New to In Progress

yast2_cmdline also fails consistently on openSUSE Tumbleweed. There it's marked with a bug but not regarded as high priority. On SLES currently we don't run this module (waiting for zluo here). I will try to exclude yast2_cmdline by default as it highly depends on external test packages by developers so I will split the scenarios.

#3 Updated by okurz almost 4 years ago

PR merged, test module does no harm anymore for the yast2_ui scenario. Created new test suite "yast2_ui_devel" triggered after yast2_ui and added to development job groups for openSUSE Tumbleweed as well as openSUSE Leap.

Should check how the test module behaves for SLE

#4 Updated by okurz almost 4 years ago

SLE tests fails in http://lord.arch/tests/latest#step/yast2_cmdline/6 with the source repo not being found. Would need to add iso2

#5 Updated by okurz over 3 years ago

  • Status changed from In Progress to Feedback
  • Priority changed from High to Low

waiting for SLE15 to progress further until I would continue to try to add it back to the SLE test plan acordingly

#6 Updated by tampakrap over 3 years ago

foo

#7 Updated by tampakrap over 3 years ago

foo

#8 Updated by tampakrap over 3 years ago

foo

#9 Updated by okurz over 3 years ago

foo

#10 Updated by tampakrap over 3 years ago

foo

#11 Updated by okurz over 3 years ago

  • Subject changed from please remove yast2_cmdline from 42.3 test plan to Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)

Added "yast2_ui_devel" to SLE15 test plan for now in the development group. Let's gather some testing results before evaluation.

#12 Updated by okurz about 3 years ago

  • Target version set to Milestone 16

yast2_ui_devel was not executed for SLE15 ever since yast2_ncurses always fails. Let's see how it handles the RC phase of SLE15.

#13 Updated by okurz about 3 years ago

  • Subject changed from Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan) to [functional]Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)

#14 Updated by okurz about 3 years ago

  • Subject changed from [functional]Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan) to [functional][y][yast]Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)
  • Status changed from Feedback to New
  • Assignee deleted (okurz)

Over to [y]-team

#15 Updated by okurz almost 3 years ago

  • Subject changed from [functional][y][yast]Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan) to [functional][y][yast] Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)
  • Due date set to 2018-06-05

riafarov I think this is something you could clarify with the yast team

#16 Updated by riafarov almost 3 years ago

  • Description updated (diff)
  • Status changed from New to Workable

#17 Updated by mloviska almost 3 years ago

  • Status changed from Workable to In Progress
  • Assignee set to mloviska

#18 Updated by mloviska almost 3 years ago

Issue with hdd_1 variable.

[2018-05-31T04:55:48.0972 CEST] [debug] CACHE: Download of /var/lib/openqa/cache/SLES-15-x86_64-657.1-gnome@64bit-gnome.qcow2 failed with: 404 - Not Found
[2018-05-31T04:55:48.0973 CEST] [debug] CACHE: removed /var/lib/openqa/cache/SLES-15-x86_64-657.1-gnome@64bit-gnome.qcow2

#19 Updated by mloviska almost 3 years ago

qcow2 image is missing "Module development tools", hence we are getting missing package error.

Possible fix using SUSEConnect -p sle-module-development-tools/15/x86_64

#20 Updated by mloviska almost 3 years ago

Successful run:

http://dhcp151.suse.cz/tests/3380#step/yast2_cmdline/16

Tested modules:

  • yast2-network
  • yast2-http-server

OpenQA variables:

#21 Updated by riafarov almost 3 years ago

  • Due date changed from 2018-06-05 to 2018-06-19

#22 Updated by riafarov almost 3 years ago

  • Target version changed from Milestone 16 to Milestone 17

#23 Updated by mgriessmeier almost 3 years ago

  • Subject changed from [functional][y][yast] Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan) to [functional][y][yast][easy] Add yast2_ui_devel scenario (was: please remove yast2_cmdline from 42.3 test plan)

#25 Updated by mloviska almost 3 years ago

  • Status changed from In Progress to Feedback

#26 Updated by mloviska almost 3 years ago

pr merged

#27 Updated by okurz almost 3 years ago

  • Target version changed from Milestone 17 to Milestone 17

#28 Updated by mloviska almost 3 years ago

test suite yast2_ui_devel added to sle12sp4's functional group.

#29 Updated by riafarov almost 3 years ago

Once we have run on osd, please, set as blocked with https://progress.opensuse.org/issues/34732

#30 Updated by mloviska almost 3 years ago

  • Status changed from Feedback to Blocked

#31 Updated by mloviska almost 3 years ago

  • Has duplicate action #34732: [functiona][y][opensuse] make zypper_info use mirrored source repo, do not remove official update repos added

#32 Updated by mloviska almost 3 years ago

  • Has duplicate deleted (action #34732: [functiona][y][opensuse] make zypper_info use mirrored source repo, do not remove official update repos)

#33 Updated by mloviska almost 3 years ago

  • Blocked by action #34732: [functiona][y][opensuse] make zypper_info use mirrored source repo, do not remove official update repos added

#34 Updated by okurz almost 3 years ago

mloviska wrote:

test suite yast2_ui_devel added to sle12sp4's functional group.

I think in this case it is better to track the scenario in "test development". We should not add scenarios that never worked and we don't know if they will ever work

#35 Updated by mloviska almost 3 years ago

This one is already working for sle15, however I haven't made it in time so we can have runs in OSD. I am afraid that test case will be forgotten by moving it to dev group. Considering that the job is currently block by missing dependency it might be actually easier to track the bug in functional group than in dev.

#36 Updated by okurz almost 3 years ago

mloviska wrote:

I am afraid that test case will be forgotten by moving it to dev group

Yes, I share your concerns. This is why we still have this progress ticket. Based on that the ticket is still open and blocked we now that the test can not work. The job within openQA should also be labeled by this. The validation job groups in openQA should be reserved for tests that we expect to work. I know we did not apply this cleanly during the past months and review turned out to miss some failures which never went away. Btw, this has been mentioned as a concern by RMs as well so the suggestion is to apply more strictly our best practices we followed in before: Whenever we know that a test will not be fixed very soon we track it in "test development" based on progress tickets. It's easier for the progress ticket assignee to review a single job and determine if the job failure reason is still the same in contrast to the (overwhelmed) daily reviewer trying to get a hold of all failures.

#37 Updated by mloviska almost 3 years ago

Valid point, yast2_ui_devel is currently in dev group of sle12sp4.

#38 Updated by riafarov almost 3 years ago

  • Due date changed from 2018-06-19 to 2018-07-03

#39 Updated by okurz almost 3 years ago

  • Due date changed from 2018-07-03 to 2018-08-28
  • Target version changed from Milestone 17 to Milestone 18

Considering blocked now by the mentioned bug. I do not expect the resolution within the timeframe of the next sprint so let's take a look at the latest again in S24. Of course if the bug is resolved in before feel free to bring it up again.

#41 Updated by riafarov over 2 years ago

  • Due date changed from 2018-08-28 to 2018-09-25

I do not expect this to get unblocked soon, mloviska, what do you think? Maybe we should put it back to the backlog instead of checking state.

#42 Updated by mloviska over 2 years ago

  • Due date deleted (2018-09-25)

I am going to be informed by mail anyway, thus it should be fine to put this one back to backlog again. Nevertheless we need to wait what is going to happen with Bug 1096302 - rspec from BSK cannot be installed first. I will remove the due date for now.

#43 Updated by okurz over 2 years ago

  • Target version changed from Milestone 18 to Milestone 26

right, let's review later again

#44 Updated by okurz over 2 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: yast2_ui_devel
https://openqa.suse.de/tests/2096097

#45 Updated by mloviska over 2 years ago

In case of test case bug as this one. We can update the test suite by ourselves

#46 Updated by mloviska over 2 years ago

PR to fix type for yast2 lan module
https://github.com/yast/yast-network/pull/686

#47 Updated by mloviska almost 2 years ago

Maybe we can decommission this module and related test suites since we test CLI only for 2 yast2 modules and QAM is working on their own way how to test CLI for yast2 modules anyway.
QAM test plan

#48 Updated by mloviska almost 2 years ago

  • Assignee deleted (mloviska)

#49 Updated by riafarov over 1 year ago

  • Due date set to 2019-08-27
  • Status changed from Blocked to Workable
  • Target version changed from Milestone 26 to Milestone 27

Not sure why it was blocked. In recent runs we miss some source packages, which should be easy to add even in case we don't sync them.
Might be that there are more issues hidden by the first one, but let's try.

#50 Updated by riafarov over 1 year ago

  • Description updated (diff)
  • Estimated time set to 5.00 h

#51 Updated by riafarov over 1 year ago

  • Due date changed from 2019-08-27 to 2019-09-10

#52 Updated by JERiveraMoya over 1 year ago

  • Status changed from Workable to In Progress
  • Assignee set to JERiveraMoya

#54 Updated by JERiveraMoya over 1 year ago

  • Status changed from In Progress to Feedback

#56 Updated by JERiveraMoya over 1 year ago

Now the package is present. Created bug: Broken yast command line tests

#57 Updated by JERiveraMoya over 1 year ago

  • Status changed from Feedback to Blocked

#58 Updated by mgriessmeier over 1 year ago

  • Target version changed from Milestone 27 to Milestone 28

#59 Updated by JERiveraMoya over 1 year ago

  • Due date changed from 2019-09-10 to 2019-10-08

#60 Updated by riafarov over 1 year ago

  • Due date changed from 2019-10-08 to 2019-11-05
  • Target version changed from Milestone 28 to Milestone 29

Looks like we can work on this one soon, once package reaches the repo. Moving to the sprint after next one.

#61 Updated by riafarov over 1 year ago

  • Due date changed from 2019-11-05 to 2020-02-11
  • Target version changed from Milestone 29 to Milestone 30+

#62 Updated by mgriessmeier over 1 year ago

  • Target version changed from Milestone 30+ to Milestone 30

bulk moved to M30 for revisiting

#63 Updated by riafarov about 1 year ago

  • Due date deleted (2020-02-11)
  • Target version changed from Milestone 30 to future

#64 Updated by riafarov 6 months ago

  • Project changed from openQA Tests to qe-yast
  • Category deleted (Bugs in existing tests)
  • Status changed from Blocked to Resolved

Isn't it done now? I see https://openqa.opensuse.org/tests/1441560# being green for a months now. Feel free to reopen in case you believe I miss something.

Also available in: Atom PDF