Project

General

Profile

Actions

action #13598

closed

[sles][functional]Test module sles+extratests has inconsistent results

Added by mkravec almost 8 years ago. Updated almost 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Bugs in existing tests
Start date:
2017-01-16
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Difficulty:

Description

Results of job sles+extratests are very inconsistent. Usually 2-8 modules fail, this number is different for every job run. This issue is parent for separate failures which should help spread work.

Failures are usually caused by test modules expecting openqa not under load - or opposite. Typical failure is yast test modules pressing keys before screen is loaded. (missing or wrong assert_screen)

Every failing task should have bug or progress issue assigned, we should avoid restarting jobs for better results.


Subtasks 12 (0 open12 closed)

action #16020: split "extratests" in same structure as in products/opensuse/main.pmResolvedasmorodskyi2017-01-16

Actions
action #16054: Test fails in yast2_nis after job restartResolvedmkravec2017-01-17

Actions
action #16028: test fails in java - Build0209Resolvedasemen2017-01-16

Actions
action #16024: test fails in snapper_cleanup Build0209(ppc64le & x86-64)Resolveddgutu2017-01-16

Actions
action #16056: Test fails in yast2_tftpResolvedmichalnowak2017-01-17

Actions
action #16026: test fails in yast2_apparmor - Build0209Resolvedzluo2017-01-16

Actions
action #16098: yast2_lan_restart test has soft_failures referencing fixed bugsResolvedmkravec2017-01-19

Actions
action #16110: Network restart detection in yast2_lan_restart is not reliabeResolveddzedro2017-01-19

Actions
action #16208: Test results are inconsistent in yast2_httpResolvedzluo2017-01-24

Actions
action #16210: Test results are inconsistent in yast2_sambaResolvedmichalnowak2017-01-24

Actions
action #16212: Test fails in vnc_two_passwordsResolvedmkravec2017-01-24

Actions
action #16540: yast2_lan_hostname has failure status for quite long timeResolved2017-02-07

Actions
Actions #1

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #2

Updated by mkravec almost 8 years ago

  • Description updated (diff)
  • Assignee set to mkravec
Actions #3

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #4

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #5

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #6

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #7

Updated by mkravec almost 8 years ago

  • Description updated (diff)
  • Status changed from New to In Progress
Actions #8

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #9

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #10

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #11

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #12

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #13

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #14

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #15

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #16

Updated by mkravec almost 8 years ago

  • Description updated (diff)
Actions #17

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #18

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #19

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #20

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #21

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #22

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #23

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #24

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #25

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #26

Updated by mkravec over 7 years ago

  • Description updated (diff)
Actions #27

Updated by zluo over 7 years ago

because yast2_lan_restart fails due to the issue (Bug 993666), all extra tests afterwards go to fail or don't start at all.

Please see my local openQA server: http://e13.suse.de/tests/1257

Actions #28

Updated by mkravec over 7 years ago

In your local openQA server yast2_lan_restart is last test scheduled. If it fails it has no effect on previous tests.

You modified main.pm, that is why your local openQA does not schedule other tests.

See: https://openqa.suse.de/tests/621222

Actions #29

Updated by zluo over 7 years ago

yes, I modified main.pm for support server and forgot this to change it back...

Actions #30

Updated by mkravec over 7 years ago

  • Assignee changed from mkravec to zluo

Reassigning so Zaoliang can continue to work on OpenSUSE tests.

Actions #31

Updated by zluo over 7 years ago

yast2_http, yast2_tftp got fixed finally.

Actions #32

Updated by okurz over 7 years ago

  • Category set to Bugs in existing tests
Actions #33

Updated by okurz over 7 years ago

@mkravec: https://openqa.suse.de/tests/627414 now show failing "snapper_cleanup" and "btrfs_qgroups". care to take a look?

Actions #34

Updated by mkravec over 7 years ago

btrfs_qgroups: - bsc#993841 is still open, patch not applied yet - this test is failing by intention
# rm: cannot remove 'e/file': Disk quota exceeded
# command 'rm e/file' failed: bsc#993841 at /usr/lib/os-autoinst/testapi.pm line 626

snapper_cleanup: - this is question for @dgutu, but I think answer will be simmilar

yast2_ftp: failed by intention, bsc#975538

yast2_tftp: seems that test module / needles are broken

Actions #35

Updated by okurz over 7 years ago

ok, I get it. But we should be careful to not overlook new failures in the job. Can you turn this into soft fails with bugrefs?

Actions #36

Updated by mkravec over 7 years ago

Are you sure we want this? Extratests module is already overlooked, I think if we change bugs to softfails it will be ignored even more (now it's bugging at least you & me) :)

Actions #37

Updated by maritawerner over 7 years ago

  • Target version set to Milestone 4
Actions #38

Updated by zluo over 7 years ago

all my extra tests look good now for TW:
https://openqa.opensuse.org/tests/311354

Actions #39

Updated by zluo over 7 years ago

All my extratests works fine now for sles 12 sp3:

https://openqa.suse.de/tests/647250

Actions #40

Updated by zluo over 7 years ago

  • Assignee deleted (zluo)

I think all my yast module tests work. mkravec, can you take over?

Actions #41

Updated by mkravec over 7 years ago

Thanks, extra tests look greener - but I see yast2_nis still failing:

Actions #42

Updated by zluo over 7 years ago

yes, checking yast2_nis now.

Actions #43

Updated by zluo over 7 years ago

yast2_nis test got fixed now:
https://openqa.suse.de/tests/650154#live

Actions #45

Updated by okurz over 7 years ago

subticket #16020 created. Also as a reminder: openQA now supports more than one bugref per job so no problem to label all individual failures.

Actions #46

Updated by okurz over 7 years ago

unfortunately could not be finished within M4 but we discussed it properly. Enhancement is planned first in subticket #16020 which should help ourselves with better feedback, then we will see about the generic status.

Actions #47

Updated by okurz over 7 years ago

  • Target version changed from Milestone 4 to Milestone 5

(see last comment)

Actions #48

Updated by mkravec over 7 years ago

  • Subject changed from sles+extratests have many occasionally failing tests to Test module sles+extratests has inconsistent results
  • Description updated (diff)
Actions #49

Updated by zluo over 7 years ago

  • Assignee set to zluo
Actions #50

Updated by okurz over 7 years ago

  • Target version changed from Milestone 5 to Milestone 6
Actions #51

Updated by okurz about 7 years ago

  • Subject changed from Test module sles+extratests has inconsistent results to [sles][functional]Test module sles+extratests has inconsistent results
Actions #52

Updated by okurz about 7 years ago

  • Target version changed from Milestone 6 to Milestone 7

@zluo: please update

Actions #53

Updated by zluo about 7 years ago

  • Status changed from In Progress to Resolved

finally resolved...

Actions #54

Updated by okurz about 7 years ago

  • Status changed from Resolved to In Progress

please give more information how the ticket is resolved

Actions #55

Updated by zluo about 7 years ago

Each test got fixed or resolved now. Are you expecting something else than this?
If all issues related to this ticket, it means to me it is resolved...

Actions #56

Updated by zluo about 7 years ago

  • Status changed from In Progress to Resolved

@Oliver set it as resolved. Please check sub ticket for details.

Actions #57

Updated by okurz almost 7 years ago

This is an autogenerated message for openQA integration by the openqa_review script:

This bug is still referenced in a failing openQA test: sles+extratests
http://openqa.suse.de/tests/627414

Actions

Also available in: Atom PDF