action #13598
closed[sles][functional]Test module sles+extratests has inconsistent results
100%
Description
Results of job sles+extratests are very inconsistent. Usually 2-8 modules fail, this number is different for every job run. This issue is parent for separate failures which should help spread work.
Failures are usually caused by test modules expecting openqa not under load - or opposite. Typical failure is yast test modules pressing keys before screen is loaded. (missing or wrong assert_screen)
Every failing task should have bug or progress issue assigned, we should avoid restarting jobs for better results.
Updated by mkravec over 7 years ago
- Description updated (diff)
- Assignee set to mkravec
Updated by mkravec over 7 years ago
- Description updated (diff)
- Status changed from New to In Progress
Updated by zluo over 7 years ago
because yast2_lan_restart fails due to the issue (Bug 993666), all extra tests afterwards go to fail or don't start at all.
Please see my local openQA server: http://e13.suse.de/tests/1257
Updated by mkravec over 7 years ago
In your local openQA server yast2_lan_restart is last test scheduled. If it fails it has no effect on previous tests.
You modified main.pm, that is why your local openQA does not schedule other tests.
Updated by zluo over 7 years ago
yes, I modified main.pm for support server and forgot this to change it back...
Updated by mkravec over 7 years ago
- Assignee changed from mkravec to zluo
Reassigning so Zaoliang can continue to work on OpenSUSE tests.
Updated by okurz over 7 years ago
@mkravec: https://openqa.suse.de/tests/627414 now show failing "snapper_cleanup" and "btrfs_qgroups". care to take a look?
Updated by mkravec over 7 years ago
btrfs_qgroups: - bsc#993841 is still open, patch not applied yet - this test is failing by intention
# rm: cannot remove 'e/file': Disk quota exceeded
# command 'rm e/file' failed: bsc#993841 at /usr/lib/os-autoinst/testapi.pm line 626
snapper_cleanup: - this is question for @dgutu, but I think answer will be simmilar
yast2_ftp: failed by intention, bsc#975538
yast2_tftp: seems that test module / needles are broken
Updated by okurz over 7 years ago
ok, I get it. But we should be careful to not overlook new failures in the job. Can you turn this into soft fails with bugrefs?
Updated by mkravec over 7 years ago
Are you sure we want this? Extratests module is already overlooked, I think if we change bugs to softfails it will be ignored even more (now it's bugging at least you & me) :)
Updated by zluo over 7 years ago
all my extra tests look good now for TW:
https://openqa.opensuse.org/tests/311354
Updated by zluo over 7 years ago
All my extratests works fine now for sles 12 sp3:
Updated by zluo over 7 years ago
- Assignee deleted (
zluo)
I think all my yast module tests work. mkravec, can you take over?
Updated by mkravec over 7 years ago
Thanks, extra tests look greener - but I see yast2_nis still failing:
Updated by zluo over 7 years ago
yast2_nis test got fixed now:
https://openqa.suse.de/tests/650154#live
Updated by mkravec over 7 years ago
On PPC 1-5 test modules are always failing - does not look consistent:
https://openqa.suse.de/tests/703676#previous
cnf: https://openqa.suse.de/tests/703676#step/command_not_found/4
yast2proxy: https://openqa.suse.de/tests/703676#step/yast2_proxy/72
yast2_lan_hostname: https://openqa.suse.de/tests/703676#step/yast2_lan_hostname/10
...
Updated by okurz over 7 years ago
subticket #16020 created. Also as a reminder: openQA now supports more than one bugref per job so no problem to label all individual failures.
Updated by okurz over 7 years ago
unfortunately could not be finished within M4 but we discussed it properly. Enhancement is planned first in subticket #16020 which should help ourselves with better feedback, then we will see about the generic status.
Updated by okurz over 7 years ago
- Target version changed from Milestone 4 to Milestone 5
(see last comment)
Updated by mkravec over 7 years ago
- Subject changed from sles+extratests have many occasionally failing tests to Test module sles+extratests has inconsistent results
- Description updated (diff)
Updated by okurz about 7 years ago
- Target version changed from Milestone 5 to Milestone 6
Updated by okurz about 7 years ago
- Subject changed from Test module sles+extratests has inconsistent results to [sles][functional]Test module sles+extratests has inconsistent results
Updated by okurz about 7 years ago
- Target version changed from Milestone 6 to Milestone 7
@zluo: please update
Updated by okurz about 7 years ago
- Status changed from Resolved to In Progress
please give more information how the ticket is resolved
Updated by zluo about 7 years ago
Each test got fixed or resolved now. Are you expecting something else than this?
If all issues related to this ticket, it means to me it is resolved...
Updated by zluo about 7 years ago
- Status changed from In Progress to Resolved
@Oliver set it as resolved. Please check sub ticket for details.
Updated by okurz over 6 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: sles+extratests
http://openqa.suse.de/tests/627414