Project

General

Profile

action #13598

[sles][functional]Test module sles+extratests has inconsistent results

Added by mkravec about 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Start date:
2017-01-16
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

Results of job sles+extratests are very inconsistent. Usually 2-8 modules fail, this number is different for every job run. This issue is parent for separate failures which should help spread work.

Failures are usually caused by test modules expecting openqa not under load - or opposite. Typical failure is yast test modules pressing keys before screen is loaded. (missing or wrong assert_screen)

Every failing task should have bug or progress issue assigned, we should avoid restarting jobs for better results.


Subtasks

action #16020: split "extratests" in same structure as in products/opensuse/main.pmResolvedasmorodskyi

action #16054: Test fails in yast2_nis after job restartResolvedmkravec

action #16028: test fails in java - Build0209Resolvedasemen

action #16024: test fails in snapper_cleanup Build0209(ppc64le & x86-64)Resolveddgutu

action #16056: Test fails in yast2_tftpResolvedmichalnowak

action #16026: test fails in yast2_apparmor - Build0209Resolvedzluo

action #16098: yast2_lan_restart test has soft_failures referencing fixed bugsResolvedmkravec

action #16110: Network restart detection in yast2_lan_restart is not reliabeResolveddzedro

action #16208: Test results are inconsistent in yast2_httpResolvedzluo

action #16210: Test results are inconsistent in yast2_sambaResolvedmichalnowak

action #16212: Test fails in vnc_two_passwordsResolvedmkravec

action #16540: yast2_lan_hostname has failure status for quite long timeResolved

History

#1 Updated by mkravec about 6 years ago

  • Description updated (diff)

#2 Updated by mkravec about 6 years ago

  • Description updated (diff)
  • Assignee set to mkravec

#3 Updated by mkravec about 6 years ago

  • Description updated (diff)

#4 Updated by mkravec about 6 years ago

  • Description updated (diff)

#5 Updated by mkravec about 6 years ago

  • Description updated (diff)

#6 Updated by mkravec about 6 years ago

  • Description updated (diff)

#7 Updated by mkravec about 6 years ago

  • Description updated (diff)
  • Status changed from New to In Progress

#8 Updated by mkravec about 6 years ago

  • Description updated (diff)

#9 Updated by mkravec about 6 years ago

  • Description updated (diff)

#10 Updated by mkravec about 6 years ago

  • Description updated (diff)

#11 Updated by mkravec about 6 years ago

  • Description updated (diff)

#12 Updated by mkravec about 6 years ago

  • Description updated (diff)

#13 Updated by mkravec about 6 years ago

  • Description updated (diff)

#14 Updated by mkravec about 6 years ago

  • Description updated (diff)

#15 Updated by mkravec about 6 years ago

  • Description updated (diff)

#16 Updated by mkravec about 6 years ago

  • Description updated (diff)

#17 Updated by mkravec about 6 years ago

  • Description updated (diff)

#18 Updated by mkravec about 6 years ago

  • Description updated (diff)

#19 Updated by mkravec about 6 years ago

  • Description updated (diff)

#20 Updated by mkravec about 6 years ago

  • Description updated (diff)

#21 Updated by mkravec about 6 years ago

  • Description updated (diff)

#22 Updated by mkravec about 6 years ago

  • Description updated (diff)

#23 Updated by mkravec about 6 years ago

  • Description updated (diff)

#24 Updated by mkravec about 6 years ago

  • Description updated (diff)

#25 Updated by mkravec about 6 years ago

  • Description updated (diff)

#26 Updated by mkravec about 6 years ago

  • Description updated (diff)

#27 Updated by zluo about 6 years ago

because yast2_lan_restart fails due to the issue (Bug 993666), all extra tests afterwards go to fail or don't start at all.

Please see my local openQA server: http://e13.suse.de/tests/1257

#28 Updated by mkravec about 6 years ago

In your local openQA server yast2_lan_restart is last test scheduled. If it fails it has no effect on previous tests.

You modified main.pm, that is why your local openQA does not schedule other tests.

See: https://openqa.suse.de/tests/621222

#29 Updated by zluo about 6 years ago

yes, I modified main.pm for support server and forgot this to change it back...

#30 Updated by mkravec about 6 years ago

  • Assignee changed from mkravec to zluo

Reassigning so Zaoliang can continue to work on OpenSUSE tests.

#31 Updated by zluo about 6 years ago

yast2_http, yast2_tftp got fixed finally.

#32 Updated by okurz about 6 years ago

  • Category set to Bugs in existing tests

#33 Updated by okurz about 6 years ago

mkravec: https://openqa.suse.de/tests/627414 now show failing "snapper_cleanup" and "btrfs_qgroups". care to take a look?

#34 Updated by mkravec about 6 years ago

btrfs_qgroups: - bsc#993841 is still open, patch not applied yet - this test is failing by intention
# rm: cannot remove 'e/file': Disk quota exceeded
# command 'rm e/file' failed: bsc#993841 at /usr/lib/os-autoinst/testapi.pm line 626

snapper_cleanup: - this is question for dgutu, but I think answer will be simmilar

yast2_ftp: failed by intention, bsc#975538

yast2_tftp: seems that test module / needles are broken

#35 Updated by okurz about 6 years ago

ok, I get it. But we should be careful to not overlook new failures in the job. Can you turn this into soft fails with bugrefs?

#36 Updated by mkravec about 6 years ago

Are you sure we want this? Extratests module is already overlooked, I think if we change bugs to softfails it will be ignored even more (now it's bugging at least you & me) :)

#37 Updated by maritawerner about 6 years ago

  • Target version set to Milestone 4

#38 Updated by zluo almost 6 years ago

all my extra tests look good now for TW:
https://openqa.opensuse.org/tests/311354

#39 Updated by zluo almost 6 years ago

All my extratests works fine now for sles 12 sp3:

https://openqa.suse.de/tests/647250

#40 Updated by zluo almost 6 years ago

  • Assignee deleted (zluo)

I think all my yast module tests work. mkravec, can you take over?

#41 Updated by mkravec almost 6 years ago

Thanks, extra tests look greener - but I see yast2_nis still failing:

#42 Updated by zluo almost 6 years ago

yes, checking yast2_nis now.

#43 Updated by zluo almost 6 years ago

yast2_nis test got fixed now:
https://openqa.suse.de/tests/650154#live

#45 Updated by okurz almost 6 years ago

subticket #16020 created. Also as a reminder: openQA now supports more than one bugref per job so no problem to label all individual failures.

#46 Updated by okurz almost 6 years ago

unfortunately could not be finished within M4 but we discussed it properly. Enhancement is planned first in subticket #16020 which should help ourselves with better feedback, then we will see about the generic status.

#47 Updated by okurz almost 6 years ago

  • Target version changed from Milestone 4 to Milestone 5

(see last comment)

#48 Updated by mkravec almost 6 years ago

  • Subject changed from sles+extratests have many occasionally failing tests to Test module sles+extratests has inconsistent results
  • Description updated (diff)

#49 Updated by zluo almost 6 years ago

  • Assignee set to zluo

#50 Updated by okurz over 5 years ago

  • Target version changed from Milestone 5 to Milestone 6

#51 Updated by okurz over 5 years ago

  • Subject changed from Test module sles+extratests has inconsistent results to [sles][functional]Test module sles+extratests has inconsistent results

#52 Updated by okurz over 5 years ago

  • Target version changed from Milestone 6 to Milestone 7

zluo: please update

#53 Updated by zluo over 5 years ago

  • Status changed from In Progress to Resolved

finally resolved...

#54 Updated by okurz over 5 years ago

  • Status changed from Resolved to In Progress

please give more information how the ticket is resolved

#55 Updated by zluo over 5 years ago

Each test got fixed or resolved now. Are you expecting something else than this?
If all issues related to this ticket, it means to me it is resolved...

#56 Updated by zluo over 5 years ago

  • Status changed from In Progress to Resolved

@Oliver set it as resolved. Please check sub ticket for details.

#57 Updated by okurz over 5 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: sles+extratests
http://openqa.suse.de/tests/627414

Also available in: Atom PDF