action #99591
closed[qe-core] many test called with qa_run.pm fail
100%
Description
Observation¶
This is one example, almost all tests on SLE-12 are failing for the same issue after a 3h timeout.
openQA test in scenario sle-12-SP3-Server-DVD-Updates-x86_64-mau-qa_kernel_lynis@64bit fails in
kernel_lynis
Test suite description¶
Testsuite maintained at https://gitlab.suse.de/qa-maintenance/qam-openqa-yml.
Reproducible¶
Fails since (at least) Build 20211001-1 (current job)
Expected result¶
Last good: 20210930-1 (or more recent)
Further details¶
Always latest result in this scenario: latest
Files
Updated by tjyrinki_suse about 3 years ago
- Status changed from New to Workable
- Target version set to QE-Core: Ready
- Start date deleted (
2021-10-01)
Updated by tjyrinki_suse about 3 years ago
From Michael to lessen the impact on openQA: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13376/
If qa_run is still to be used in the future, it's useful to file a new ticket before closing this one to change it back possibly, but so far that's only a beginning and we do not have a fix for the test failures.
Updated by mgrifalconi about 3 years ago
Cloning a test that worked yesterday still fails https://openqa.suse.de/tests/7290761#
More info on the situation https://suse.slack.com/archives/C02D16TCP99/p1633075991136300
Updated by dzedro about 3 years ago
I created ticket https://sd.suse.com/servicedesk/customer/portal/1/SD-62274
Looks like dashboard.qa2.suse.asia is unreachable.
# nc -zv dashboard.qa2.suse.asia 8888
nc: connect to dashboard.qa2.suse.asia port 8888 (tcp) failed: Connection timed out
Updated by kgw_suse about 3 years ago
- Related to action #99714: [qem] test fails in kernel_multipath added
Updated by maritawerner about 3 years ago
@dzedro China is on Public Holiday and they had to switch of the servers due to a power switch off in the building. The team will be back on Thursday.
Updated by vpelcak about 3 years ago
Do we have then some action plan for the future?
Like having tests geo-distributed or maybe even removed?
Updated by dzedro about 3 years ago
- Status changed from Workable to Blocked
- Assignee set to dzedro
Updated by pcervinka about 3 years ago
Maybe it is right time to drop (at least part of them) these old ctcs2 based tests. Their outcome was questionable all the time and some of of those tests were not touched for couple of years.
Updated by mgrifalconi about 3 years ago
@dzedro That is great! Thanks Jozef!
@vpelcak @pcervinka Yes it is planned to get rid of it https://progress.opensuse.org/issues/99657
Updated by dzedro about 3 years ago
- Status changed from Blocked to Resolved
- % Done changed from 0 to 100
The issue is resolved, there is another ticket to handle the old tests.
Updated by cachen about 3 years ago
Hello Guys, like Marita mentioned dashboard.qa2.suse.asia was shutdown due to power down of the office building. There is a mistake in code probably, dashboard.qa2.suse.asia is using by qe-performance squad and message_queue to fetching its status is internal service shouldn't affect your testing. @Jerry Tang will take a look and see how to fix the influence.
Updated by jerrytang about 3 years ago
dzedro wrote:
I created ticket https://sd.suse.com/servicedesk/customer/portal/1/SD-62274
Looks like dashboard.qa2.suse.asia is unreachable.# nc -zv dashboard.qa2.suse.asia 8888 nc: connect to dashboard.qa2.suse.asia port 8888 (tcp) failed: Connection timed out
Yes, root cause is dashboard.qa2.suse.asia power off.
Automation test will update progress of test by message queue API.
When server is down,it will increase running time because of connection hung.
I'll submit a PR to set it off by default.
Updated by jerrytang about 3 years ago
MR submit waiting for review and merge
https://gitlab.suse.de/qa-testsuites/sleperf/-/merge_requests/72
Updated by jerrytang about 3 years ago
qa-testsuite commit id : eb83983dbd6c5734540f822424c977c61ec0e813
"set message queue api disable by default"