action #72118
closed[qe-core][qem][rmt] test fails in rmt_feature
100%
Description
Observation¶
Test is failing every day, this looks like SCC issue.
Please open bug and workaround it or fix the test.
openQA test in scenario sle-15-Server-DVD-Updates-x86_64-qam_rmt@64bit fails in
rmt_feature
Test suite description¶
Meant for scheduling rmt tests on QAM Updates https://progress.opensuse.org/issues/55784
Reproducible¶
Fails since (at least) Build 20200930-1 (current job)
Expected result¶
Last good: 20200929-1 (or more recent)
Further details¶
Always latest result in this scenario: latest
Updated by mgrifalconi over 3 years ago
Thanks for the finding Jozef!
I see some of 504 on SCC even on successful runs: https://openqa.suse.de/tests/4713655#step/rmt_feature/36
Seems like the 3 times retry with 30s delay is not enough https://github.com/os-autoinst/os-autoinst-distri-opensuse/blob/5cf18bfa356f00b4a42baac72107480382579b7d/lib/repo_tools.pm#L277
Updated by dzedro over 3 years ago
The problem is that looks like this is happening always on night run and at that time there is something happening or broken with SCC.
I assume 504 is gateway timeout.
Updated by mgrifalconi over 3 years ago
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/11116 was merged, job failed again:
https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/11116
For almost 1 hour, SCC replied with 504 Gateway Time-Out.
https://openqa.suse.de/tests/4758701#step/rmt_feature/36
Updated by dzedro over 3 years ago
Still failed https://openqa.suse.de/tests/4758701#step/rmt_feature/47
The thing is that there are 9 modules. This test is running approx. 1h30m
Wouldn't be 1-3 modules enough ?
Is there any log or verbose arg to know what repo update is exactly failing ?
Updated by mgrifalconi over 3 years ago
Updated by mgrifalconi over 3 years ago
- Status changed from Workable to Blocked
Waiting for feedback on bsc
Updated by dzedro over 3 years ago
It is always failing on product 1797 https://openqa.suse.de/tests/4882338#step/rmt_feature/36
Deactivate syncing of this product as workaround.
Updated by mgrifalconi over 3 years ago
Doesn't 1797
mean only the timeout in seconds?
timeout 1797 rmt-cli sync;
No idea why it is 3 seconds short of the configured timeout though: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/11116/files#diff-df032fc1142ccdd3287c7dbc03ae0e11238efd02ddb8ed1ec7bed6e7ccc6cd2fR277
script_retry 'rmt-cli sync', delay => 60, retry => 6, timeout => 1800;
Updated by dzedro over 3 years ago
mgrifalconi wrote:
Doesn't
1797
mean only the timeout in seconds?
timeout 1797 rmt-cli sync;
No idea why it is 3 seconds short of the configured timeout though: https://github.com/os-autoinst/os-autoinst-distri-opensuse/pull/11116/files#diff-df032fc1142ccdd3287c7dbc03ae0e11238efd02ddb8ed1ec7bed6e7ccc6cd2fR277
script_retry 'rmt-cli sync', delay => 60, retry => 6, timeout => 1800;
hahah rofl I'm dumb, I have been looking on the code and have seen product numbers enabled and disabled and automatically thought this number is also product number .... ignore
Updated by dzedro over 3 years ago
Unscheduled from SLE 15 & 15-SP1 as this versions don't have any other SLE failures now.
Scheduled on 15-SP2 where are another failures blocking automatic approval.
Updated by tjyrinki_suse over 3 years ago
- Subject changed from [qam] [rmt] test fails in rmt_feature to [qe-core][qam] [rmt] test fails in rmt_feature
Updated by dzedro over 3 years ago
I enabled rmt test on SLE 15 & 15-SP1 as it was passing on 15-SP2, no idea what changed ...
Updated by tjyrinki_suse over 3 years ago
- Subject changed from [qe-core][qam] [rmt] test fails in rmt_feature to [qe-core][qem][rmt] test fails in rmt_feature
Updated by mgrifalconi about 3 years ago
- Status changed from Blocked to Resolved
- % Done changed from 0 to 100
Hello, considering that:
- tests are enabled and green right now
- there was no response on the bug
I would close both this issue and the bug.
Updated by openqa_review over 2 years ago
This is an autogenerated message for openQA integration by the openqa_review script:
This bug is still referenced in a failing openQA test: qam_rmt
https://openqa.suse.de/tests/6859780
To prevent further reminder comments one of the following options should be followed:
- The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted
- The openQA job group is moved to "Released" or "EOL" (End-of-Life)
- The label in the openQA scenario is removed