Project

General

Profile

Actions

action #47123

closed

[functional][u][tw] extra_tests_in_textmode fails in zbar

Added by coolo almost 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Bugs in existing tests
Target version:
SUSE QA (private) - Milestone 23
Start date:
2019-02-04
Due date:
% Done:

100%

Estimated time:
Difficulty:

Description

The console seems broken - the test doesn't even start

Observation

openQA test in scenario opensuse-Tumbleweed-DVD-x86_64-extra_tests_in_textmode@64bit fails in
zbar

Test suite description

Maintainer: okurz@suse.de

Mainly console extratest.

Reproducible

Fails since (at least) Build 20180714 -> okurz: Not true ;)

Expected result

Last good: https://openqa.opensuse.org/tests/841664

Further details

Always latest result in this scenario: latest


Related issues 4 (0 open4 closed)

Related to openQA Tests (public) - action #55115: [qe-core][functional] test fails in sssd - Test fails switching to serial terminalResolved2019-08-05

Actions
Has duplicate openQA Tests (public) - action #47930: test fails in zbar - no permission to write to /dev/ttyS0Rejectedokurz2019-02-15

Actions
Has duplicate openQA Tests (public) - action #48065: test fails in zbar - permission denied on /dev/ttyAMA0Rejectedokurz2019-02-18

Actions
Blocks openQA Tests (public) - action #47417: [functional][u][svirt-xen] test fails in dracut - no serial output even though command returns 0Resolvedzluo2019-02-12

Actions
Actions #1

Updated by okurz almost 6 years ago

  • Subject changed from [tw] extra_tests_in_textmode fails in zbar to [functional][u][tw] extra_tests_in_textmode fails in zbar
  • Status changed from New to In Progress
  • Assignee set to okurz

@coolo I suggest you don't use the prefix "[tw]" in the subject. QSF looks for all tickets without tags according to https://progress.opensuse.org/projects/openqatests/wiki#ticket-backlog-refinement and tries to assign to teams. But if you add a tag you basically rely on yourself and me to monitor all tickets and act accordingly. Or is there someone watching the tag "[tw]"?

Regarding the test failure: The user-console is logged in however the test assumes the console to be not logged in (anymore). https://openqa.opensuse.org/tests/843816/file/autoinst-log.txt shows that the last console that uses the user_console before zbar is "mutt". So I assume that either the needle "text-logged-in-bernhard" needs to be adapted or some module in between is resetting consoles.

Actions #2

Updated by okurz almost 6 years ago

  • Description updated (diff)
  • Assignee changed from okurz to pstivanin
  • Priority changed from Normal to High

coolo wrote:

Expected result

Last good: 20180714 (or more recent)

Obviously that is of not much help ;)

The actual "last good" is https://openqa.opensuse.org/tests/841664 so we can compare the git log of tests:

$ git log1 --no-merges 1c0d1bb2f36aed440f6f49bc04b51b973c9fc18b..cf5e80bf718a764d06d115692829590854e2907f
e5bd56cb1 Fix paths for the modules for ay installation with all packages DVD
2a48fd53a Fix broken textmode installation on IPMI
d3b4107dd Add kiwi test for kiwi images generated at buildsystem.
7a339b9ad yast2_virtualization: Fix inversed script_run exit code check
9772253b5 Fix typo in used API
d15629629 Add procps test
c400d814b Create autoyast profile for ppc64le to upgrade on SCC
26c0984bb Do not erase disks on IPMI textmode
141d966e6 Dont run docker tests on older SLE than 12SP3
0a501ff94 (asdil12/nested) Warn about unavailable nested virt
1a1ca3c3b (rwx788/y2logs) Extract common test for yast2 lan in ncurses and x11
f32bdc060 publiccloud: Use vaultproject.io to create credentials
9303f3e7b Fix dracut on SLE 15.1 and aarch64
ea94ef676 Add timezone regression tests
14346b58f Increase wait time for dovecot start on SLE12
6447e1912 Add softfail for bsc#1123699
116fc7c87 Update clamav test for JeOS
32d690a5f New test for Yast module: yast2-rmt
7f7012bf8 Explicitly enable sshd service in qam SLE15 ay profile
0bb2edffb Add regex in yast log detection
82f153a9b Fix scheduling of docker test for SLE12-SP3 and SLE12-SP4
04ef4fd09 Unschedule cron test on JeOS
a03f17d4a Add rpcbind regression test
73cfc7dfd Assure focused password textbox
742b7f29d Increase timeout for btrfs maintainance tasks
9b1ff31c1 Revert "Assure focused password textbox"
91c46c8e9 Fix SMT URL field selection for SLE12 registration
7d56b3b71 Assure focused password textbox
744289f29 (sysrich/new_image_tests2) Add multi-arch to docker_image, introduce podman_image
fed63ae0d (asdil12/bootstrap_logs) Upload openqa_bootstrap autoinst-log.txt
748477134 Fix dracut test on SLE 12.1
9cff9b88f Close opening file pop-up
765222e12 Add soft-fail bsc#1092088 in yast2 logs
49644485c Increase timeout for downloading container
5301ba063 (asdil12/vino) Select x11 console for vino test
57b2137b8 Make the Xen bare metal test compatible with SLE 15
c4362d4f8 Update password prompt for DE language on JeOS
744c28c7a Increase default wait time for boot
2177d13c7 Add zypper_docker testmodule
c06c5e122 Fix issue that aa-genprof does not parse audit log
52c1faa53 Add dracut test
0b4d668b1 Virtualization: remove duplicate zypper option
7ac22ae8d Fix wrong hotkey used on icewm screen unlock check
c82482f6c (okurz/fix/vlc) Detect hanging VLC and kill if necessary
234fd94d7 Log out root user in window_system.pm (poo#46655)
54d220d84 Add an always_rollback flag
890b372f5 Small changes for readability
1c8603100 Try to recover network before saving logs
446d4b686 Add a function to check if /etc/hosts is properly updated by yast2 lan
fb6c599f0 Adjust tests for changes in yast2 lan module
428405930 Try to trim timeout
c58de09bc (mnowaksuse/explicit_enter_on_boot_from_hdd) Explicitly confirm boot from disk
481906d1c Take mount point into consideration when call efiboomgr

I have a suspicion for 52c1faa53, adding dracut, including the reboot.

https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6651 is the according PR which does not mention that openSUSE Tumbleweed was tested even though obviously the test module was added to the schedule of openSUSE distributions. The mentioned verification run http://d502.qam.suse.de/tests/274# also shows a lot of failed modules. I do not know why the PR was merged nevertheless. Seems like QAM are speeding up their review as in "quick, let's merge before the annoying QA SLE / openSUSE reviewers realize" ;)

@pstivanin could you please crosscheck your added test module in the test scenario "extra_tests_in_textmode" in openSUSE Tumbleweed and openSUSE Leap? I have a suspicion that you broke at least the test module "zbar", potentially more. Please reassign if I am wrong.

Actions #3

Updated by pstivanin almost 6 years ago

Hello,
@okurz I'm fairly new to openQA, so first thing first sorry for this and the past issues caused in prod by my limited knowledge about the topic :(

1) about https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6651 : yes, I didn't think about testing it also on Leap and TW. Next time I'll test the code really on every platform :)

2) about http://d502.qam.suse.de/tests/274. I didn't pay attention to the other tests because I thought they were a separate thing. I thought that openQA would treat each test individually and that they weren't related with each other. I didn't know that a test could cause a failure on another unrelated test.

3) I'm looking into it!

Actions #4

Updated by okurz almost 6 years ago

pstivanin wrote:

1) about https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6651 : yes, I didn't think about testing it also on Leap and TW. Next time I'll test the code really on every platform :)

Well, I guess testing on "every platform" is a bit too much. A more diligent PR review and also inviting reviewers outside QAM might be helpful. This goes much more to the existing reviewers than to you as new contributor and reviewee.

2) about http://d502.qam.suse.de/tests/274. I didn't pay attention to the other tests because I thought they were a separate thing. I thought that openQA would treat each test individually and that they weren't related with each other. I didn't know that a test could cause a failure on another unrelated test.

In principle, if a test module passes, the next modules just continue within the same session. Think of it as just checkpoints within the same shell session. These checkpoints are only used to fall back to in case of failures but because your test module did not fail it did continue. There is also the option to just run less tests within one scenario. You simply added your test module to the "catch-it-all" scenario :)

Actions #5

Updated by pstivanin almost 6 years ago

Ok, thanks for the explanation :)

  • by scenario you mean the parent node? (eg "x11/evolution", "sysauth", "console") ?

  • if I have correctly understood, the problem is that 8 tests are executed before zbar and dracut is the only one rebooting the instance (therefore resetting the console). And because the zbar test is expecting the console to be logged in, then the tests fails. Am I correct?

Actions #6

Updated by pstivanin almost 6 years ago

the zbar test is executing 'select_console' as first thing (line 21). From my understanding (http://open.qa/api/testapi/#_select_console) this method should do also the login if not yet done.
So how can a reboot affect this test?

Actions #7

Updated by pstivanin almost 6 years ago

As you can see from http://d502.qam.suse.de/tests/387 (oS 15.1) and http://d502.qam.suse.de/tests/388 (oS TW), disabling the reboot code in the dracut test made the zbar test green.

@okurz was right, rebooting the instance (therefore resetting the console) is the issue because zbar is expecting a different state.

Now I'll look deeper into zbar (but it will take a bit because I'm not yet really familiar with how to deal with needles), maybe it's enough to update the needle there.

Actions #8

Updated by pstivanin almost 6 years ago

  • % Done changed from 0 to 100

Together with Petr we were able to come up with a fix (tested only on TW for now http://d502.qam.suse.de/tests/408). Now I'm going to test it everywhere before creating the PR

Actions #9

Updated by okurz almost 6 years ago

pstivanin wrote:

  • by scenario you mean the parent node? (eg "x11/evolution", "sysauth", "console") ?

Actually no. I am using the same wording as in http://open.qa/docs/#concepts

I think all the other open points you have found out yourself already or we have discussed in jangouts :)

Actions #10

Updated by pstivanin almost 6 years ago

  • % Done changed from 100 to 50

As you can see from the following links (both are using the fixed-dracut), zbar still fails when running in the full test suite. So it seems not to be 100% related to dracut

opensuse 15.1 with only dracut and zbar: http://d502.qam.suse.de/tests/414
opensuse 15.1 complete test suite: http://d502.qam.suse.de/tests/410

I'm now executing a stripped down version of the full test suite, maybe I'll be able to pinpoint the culprit. Let's see

Actions #11

Updated by pstivanin almost 6 years ago

stripped down test suite: http://d502.qam.suse.de/tests/416

  • all green besides rail (which was already red).
Actions #12

Updated by pstivanin almost 6 years ago

Now I'm really puzzled:

zbar green:

zbar red:

Actions #13

Updated by pcervinka almost 6 years ago

@okurz what do you think about idea to create extratest for dracut? as it does reboot during its tests, basically in similar way like it is for kdump_and_crash (to isolate it from others)

Actions #14

Updated by pstivanin almost 6 years ago

Petr's idea would work: http://d502.qam.suse.de/tests/428

Actions #15

Updated by pstivanin almost 6 years ago

To summarize, I think we have 3 possibilities here:

  1. create an extratest for dracut, as suggested by Petr
  2. move dracut with kdump
  3. decrease dracut's coverage by removing the reboot part
Actions #16

Updated by pstivanin almost 6 years ago

  • Status changed from In Progress to Feedback
Actions #17

Updated by okurz almost 6 years ago

pstivanin wrote:

To summarize, I think we have 3 possibilities here:

  1. create an extratest for dracut, as suggested by Petr
  2. move dracut with kdump
  3. decrease dracut's coverage by removing the reboot part

All of these can work, I prefer point 1.

Actions #18

Updated by pstivanin almost 6 years ago

latest TW build is green for zbar https://openqa.opensuse.org/tests/848123 because the needle was updated

Actions #19

Updated by pstivanin almost 6 years ago

Both oS 15.1 and TW are now green thanks to the needle update :)

What shall we do now? Should we still split dracut or not?

Actions #20

Updated by pstivanin almost 6 years ago

  • Priority changed from High to Low

I reduced prio since the testing is working now.

Follow-up question is: do we still want to split dracut?

Actions #21

Updated by okurz almost 6 years ago

  • Priority changed from Low to High

Sorry, I don't know what you mean with "testing is working now". Leap 15.1 is not good: https://openqa.opensuse.org/tests/844196#step/zbar/2 is the latest job and it is failing and lnussel is already loosing trust in the scenario. Please fix that as soon as possible.

Regarding splitting: Yes, please do so. The scenario has way too many modules and any logical separation should be used to split into specific scenarios.

Actions #22

Updated by pstivanin almost 6 years ago

  • Status changed from Feedback to In Progress

@okurz sorry, typo. I meant that the zbar test is green both on Leap[1] and TW[2] because the needles were updated.

[1] https://openqa.opensuse.org/tests/850963#step/zbar/1
[2] https://openqa.opensuse.org/tests/850148#step/zbar/1

PS: Ok, working on splitting dracut right now!

Actions #23

Updated by pstivanin almost 6 years ago

PR: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6745

Test runs are still running (SLE 12.1, 12.4, 15.0 and oS Leap and TW)

Actions #24

Updated by pstivanin almost 6 years ago

Copying & pasting the email exchange Oliver and I had.

Me:

Hello Oliver,
let's take this link as an example: https://openqa.suse.de/tests/overvi
ew?distri=sle&version=15-SP1&build=161.1&groupid=110

What we want to achieve is to make dracut callable only when
EXTRATEST=dracut is set (like I did here https://github.com/os-autoinst
/os-autoinst-distri-opensuse/pull/6745), but we also want to "add
another row" on the table you see in the first link, right? So we would
have a new row in that table called "dracut" where only the dracut test
is executed, right?
Or do we want to keep dracut in mau-extratests and
extra_tests_in_textmode?

Oliver:

Given that the daily test reviewers, e.g. openSUSE release managers, are
frustated by the instability of the existing extra_tests_in_textmode I
think a dedicated scenario is the right approach at least for the build
validation job groups.
For this you need to go to
http://openqa.opensuse.org/admin/test_suites
add another test suite based on the example of extra_tests_in_textmode
which we already have and then add the according scenario everywhere
where we have extra_tests_in_textmode already scheduled, e.g.
go to https://openqa.opensuse.org/admin/groups
open at best all the groups one by one, take a look for
"extra_tests_in_textmode" and make sure we have the extra_tests_dracut
testsuite scheduled for all combinations where previously
extra_tests_in_textmode is in. For "mau-extratests" I suggest to decide
in coordination with the daily test result reviewers of openSUSE/SLE
tests. Sorry, I do not know who that would be at least not for openSUSE.
I have seen mainly astieger (Andreas Stieger) do the reviewer and he
promised me that QAM would pick up the test review according to a proper
schedule but I am not aware if that happened – bluntly, seems like not :)

I am also following the tickets so feel free to ask the same questions
in tickets instead of personal emails which I struggle with to answer in
time. This has the added benefit of more information documented which is
also easily accessible for others then. Feel free to copy the
information from this email in your ticket as well.

@okurz: ok, thanks!
Before editing stuff in ooo I guess we need to review and merge the aforementioned PR so we don't break ooo, right?
To recap:

  • in ooo I can edit myself info for textmode in order to add dracut
  • in ooo I need to find the responsible for mau before being able to add dracut
  • in osd I need to find the responsible for textmode and mau before being able to add dracut
Actions #25

Updated by okurz almost 6 years ago

  • Has duplicate action #47930: test fails in zbar - no permission to write to /dev/ttyS0 added
Actions #26

Updated by okurz almost 6 years ago

  • Has duplicate action #48065: test fails in zbar - permission denied on /dev/ttyAMA0 added
Actions #27

Updated by okurz almost 6 years ago

  • Target version set to Milestone 23
Actions #28

Updated by pstivanin almost 6 years ago

The PR is ready to be merged: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/6745

After that we can proceed with editing the openqa instances.

Actions #29

Updated by pcervinka almost 6 years ago

  • Status changed from In Progress to Feedback

Paolo's PR was merged, i created mau-extratests-dracut and assigned to qam update jobs:

BOOT_HDD_IMAGE=1
DESKTOP=textmode
EXTRATEST=dracut
HDDSIZEGB=22
HDD_1=SLES-%VERSION%-%ARCH%-mru-install-minimal-with-addons-Build%BUILD%.qcow2
NUMDISKS=2
SCC_ADDONS=%REQUIRED_ADDON%,%MRU_ADDONS%
START_AFTER_TEST=mru-install-minimal-with-addons

I created extra_tests_dracut for usage within product qa jobs, not included yet into product qa jobs:

BOOT_HDD_IMAGE=1
DESKTOP=textmode
EXTRATEST=dracut
HDD_1=%DISTRI%-%VERSION%-%ARCH%-%BUILD%-%DESKTOP%@%MACHINE%.qcow2
START_AFTER_TEST=create_hdd_textmode
UEFI_PFLASH_VARS=%DISTRI%-%VERSION%-%ARCH%-%BUILD%-%DESKTOP%@%MACHINE%-uefi-vars.qcow2

Let's wait for few qam job runs, than we can define it non QAM jobs.

Actions #30

Updated by pstivanin almost 6 years ago

  • Status changed from Feedback to In Progress
  • % Done changed from 50 to 100
Actions #31

Updated by okurz almost 6 years ago

great news! Please make sure we have both o3 and osd covered. Of course it would be fine to delegate the specific test suite and scenario adjustements to other people.

Actions #32

Updated by okurz almost 6 years ago

@pstivanin any progress?

In
https://openqa.opensuse.org/tests/latest?machine=64bit&version=Tumbleweed&flavor=DVD&arch=x86_64&distri=opensuse&test=extra_tests_in_textmode#dependencies

I can see that there is no scenario "dracut" yet so can you enable the according scenarios?

Actions #33

Updated by pstivanin almost 6 years ago

@okurz I don't have permissions to edit job definition. We have to wait for @pcervinka I guess

Actions #34

Updated by okurz almost 6 years ago

I see, I have no problem to make you "admin" on both openQA instances to do the change. However I would ask you to join during normal working ours on the IRC channels #opensuse-factory and internal #testing

Actions #35

Updated by pstivanin almost 6 years ago

@okurz done!

Actions #36

Updated by mgriessmeier almost 6 years ago

  • Blocks action #47417: [functional][u][svirt-xen] test fails in dracut - no serial output even though command returns 0 added
Actions #37

Updated by pstivanin almost 6 years ago

@okurz any news about giving me permissions to edit o3?

Actions #38

Updated by okurz almost 6 years ago

ping me on #opensuse-factory IRC and I will add you :)

Actions #39

Updated by pstivanin almost 6 years ago

For now I have enabled it only for TW 64bit and Leap 64bit.
If everything goes well I will proceed to add the test suite to all other job groups.

Actions #40

Updated by pstivanin over 5 years ago

  • Status changed from In Progress to Resolved

The 'extra_tests_dracut' has been added to all scenario :)

Thank you all for your help!!

Actions #41

Updated by pcervinka over 5 years ago

  • Related to action #55115: [qe-core][functional] test fails in sssd - Test fails switching to serial terminal added
Actions

Also available in: Atom PDF