Project

General

Profile

action #99591

[qe-core] many test called with qa_run.pm fail

Added by mgrifalconi about 2 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Immediate
Assignee:
dzedro
Category:
Bugs in existing tests
Target version:
QA - QE-Core: Ready
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:

Description

Observation

This is one example, almost all tests on SLE-12 are failing for the same issue after a 3h timeout.

openQA test in scenario sle-12-SP3-Server-DVD-Updates-x86_64-mau-qa_kernel_lynis@64bit fails in
kernel_lynis

Test suite description

Testsuite maintained at https://gitlab.suse.de/qa-maintenance/qam-openqa-yml.

Reproducible

Fails since (at least) Build 20211001-1 (current job)

Expected result

Last good: 20210930-1 (or more recent)

Further details

Always latest result in this scenario: latest

Screenshot_2021-10-04_15-49-28.png (9.55 KB) Screenshot_2021-10-04_15-49-28.png dzedro, 2021-10-04 14:01
11994

Related issues

Related to openQA Tests - action #99714: [qem] test fails in kernel_multipathClosed2021-10-04

History

#1 Updated by tjyrinki_suse about 2 months ago

  • Status changed from New to Workable
  • Target version set to QE-Core: Ready
  • Start date deleted (2021-10-01)

#2 Updated by tjyrinki_suse about 2 months ago

From Michael to lessen the impact on openQA: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/13376/

If qa_run is still to be used in the future, it's useful to file a new ticket before closing this one to change it back possibly, but so far that's only a beginning and we do not have a fix for the test failures.

#3 Updated by mgrifalconi about 2 months ago

Cloning a test that worked yesterday still fails https://openqa.suse.de/tests/7290761#

More info on the situation https://suse.slack.com/archives/C02D16TCP99/p1633075991136300

#4 Updated by dzedro about 2 months ago

11994

I created ticket https://sd.suse.com/servicedesk/customer/portal/1/SD-62274
Looks like dashboard.qa2.suse.asia is unreachable.

# nc -zv dashboard.qa2.suse.asia 8888
nc: connect to dashboard.qa2.suse.asia port 8888 (tcp) failed: Connection timed out

#5 Updated by kgw_suse about 2 months ago

  • Related to action #99714: [qem] test fails in kernel_multipath added

#6 Updated by maritawerner about 2 months ago

dzedro China is on Public Holiday and they had to switch of the servers due to a power switch off in the building. The team will be back on Thursday.

#7 Updated by vpelcak about 2 months ago

Do we have then some action plan for the future?

Like having tests geo-distributed or maybe even removed?

#8 Updated by dzedro about 2 months ago

  • Status changed from Workable to Blocked
  • Assignee set to dzedro

#9 Updated by pcervinka about 2 months ago

Maybe it is right time to drop (at least part of them) these old ctcs2 based tests. Their outcome was questionable all the time and some of of those tests were not touched for couple of years.

#10 Updated by mgrifalconi about 2 months ago

dzedro That is great! Thanks Jozef!
vpelcak pcervinka Yes it is planned to get rid of it https://progress.opensuse.org/issues/99657

#11 Updated by dzedro about 2 months ago

  • Status changed from Blocked to Resolved
  • % Done changed from 0 to 100

The issue is resolved, there is another ticket to handle the old tests.

#12 Updated by cachen about 2 months ago

Hello Guys, like Marita mentioned dashboard.qa2.suse.asia was shutdown due to power down of the office building. There is a mistake in code probably, dashboard.qa2.suse.asia is using by qe-performance squad and message_queue to fetching its status is internal service shouldn't affect your testing. @Jerry Tang will take a look and see how to fix the influence.

#13 Updated by jerrytang about 2 months ago

dzedro wrote:

I created ticket https://sd.suse.com/servicedesk/customer/portal/1/SD-62274
Looks like dashboard.qa2.suse.asia is unreachable.

# nc -zv dashboard.qa2.suse.asia 8888
nc: connect to dashboard.qa2.suse.asia port 8888 (tcp) failed: Connection timed out

Yes, root cause is dashboard.qa2.suse.asia power off.
Automation test will update progress of test by message queue API.
When server is down,it will increase running time because of connection hung.

I'll submit a PR to set it off by default.

#15 Updated by jerrytang about 2 months ago

qa-testsuite commit id : eb83983dbd6c5734540f822424c977c61ec0e813

"set message queue api disable by default"

Also available in: Atom PDF