action #106017
closed
[ipmi backend]VNC stalled, no update for 5.38 seconds
Added by Julie_CAO about 2 years ago.
Updated about 2 years ago.
Description
Most of virtualization tests failed with this issue when . Could you please investigate?
job groups:
https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP4&build=91.2&groupid=263 https://openqa.nue.suse.com/tests/overview?distri=sle&version=15-SP4&build=91.2&groupid=264
scc_registration failed in "send_key 'alt-s' if check_screen('license-insert-disc-issue', 30);" statement.
https://openqa.nue.suse.com/tests/8113373/logfile?filename=autoinst-log.txt
[2022-02-07T02:47:39.684525+01:00] [debug] no match: -0.1s, best candidate: license-insert-disc-issue-bsc1153326-20191114 (0.00)
[2022-02-07T02:47:40.039376+01:00] [debug] >>> testapi::_check_backend_response: match=license-insert-disc-issue timed out after 90 (check_screen)
[2022-02-07T02:47:40.070779+01:00] [debug] tests/installation/scc_registration.pm:25 called registration::fill_in_registration_data -> lib/registration.pm:572 called registration::process_modules -> lib/registration.pm:957 called registration::process_scc_register_addons -> lib/registration.pm:479 called testapi::check_screen
[2022-02-07T02:47:40.070984+01:00] [debug] <<< testapi::check_screen(mustmatch=[
"import-untrusted-gpg-key"
], timeout=60)
[2022-02-07T02:47:41.115512+01:00] [warn] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 0.79 seconds for 44 candidate needles - make your needles more specific
[2022-02-07T02:47:41.115638+01:00] [debug] no match: 179.7s, best candidate: add_update_test_repo-import-untrusted-gpg-key-20210321 (0.00)
[2022-02-07T02:47:41.322902+01:00] [debug] no change: 178.7s
[2022-02-07T02:47:41.731741+01:00] [debug] no match: 178.7s, best candidate: add_update_test_repo-import-untrusted-gpg-key-20210321 (0.00)
[2022-02-07T02:47:42.323396+01:00] [debug] no change: 177.7s
[2022-02-07T02:47:42.722764+01:00] [debug] no match: 177.7s, best candidate: add_update_test_repo-import-untrusted-gpg-key-20210321 (0.00)
[2022-02-07T02:47:43.327645+01:00] [debug] no change: 176.7s
[2022-02-07T02:47:43.723201+01:00] [debug] no match: 176.7s, best candidate: add_update_test_repo-import-untrusted-gpg-key-20210321 (0.00)
[2022-02-07T02:47:49.614396+01:00] [warn] !!! backend::baseclass::check_asserted_screen: check_asserted_screen took 5.29 seconds for 44 candidate needles - make your needles more specific
[2022-02-07T02:47:49.614546+01:00] [debug] no match: 175.7s, best candidate: add_update_test_repo-import-untrusted-gpg-key-20210321 (0.00)
[2022-02-07T02:47:49.615018+01:00] [debug] considering VNC stalled, no update for 5.38 seconds
XIO: fatal IO error 11 (Resource temporarily unavailable) on X server ":44401"
after 28811 requests (28654 known processed) with 0 events remaining.
xterm: fatal IO error 11 (Resource temporarily unavailable) or KillClient on X server ":44401"
- Priority changed from High to Urgent
- Target version set to Ready
So, ... not do the fullscreen refresh on exactly where it might have helped most, on the IPMI backend?
- Related to action #105429: openQA's fullstack test fails in `shutdown` module added
@okurz commented out the most relevant line¹ in /usr/lib/os-autoinst/backend/baseclass.pm on openqaworker2. The restarted job shows the same problem. I'm now commenting out more code to see whether the 2nd line makes a difference.
¹
elsif ($n % FULL_UPDATE_REQUEST_FREQUENCY == 0) {
#$self->request_screen_update({incremental => 0});
}
- Related to action #105882: Test using svirt backend fails with auto_review:"Error connecting to VNC server.*localhost.*Connection refused" added
- Status changed from New to In Progress
fails with what looks like the same issue. I wonder what's the "last good" here. Within the history of the job mentioned in the description I can find the last "passed" job https://openqa.nue.suse.com/tests/8030431 which is from 12 days ago, which is a lot.
The complete git log of os-autoinst between the last good and the current is from git log1 --no-merges d319802b..origin/master
a7103b16 Fix sporadic failures in openQA's fullstack test
45d6a872 Adapt configure_repositories step in OBS workflow
bc6eac58 Add unit test for requesting full screen updates when checking screen
f5d175d0 Speed up `t/23-baseclass.t` using `Time::Mock::Time`
c49e2ec5 Avoid division by zero when computing stopwatch data
79950eeb Force full screen update in intervals similar to full screen search
f2399bcc Force full screen update shortly before check screen would fail
0225b12b Simplify initialization of variables in `check_asserted_screen`
7608a54f (okurz/fix/svirt) Revert "svirt: Implement do_extract_assets"
531bbaf0 Revert "svirt: Store vmname early for use after test run"
26d946b7 (okurz/fix/tidy_include_rules) Fix include of extension-less perl files in tidy config
62576fe1 Tidy tools/check_coverage according to new os-autoinst rules
8c883703 git subrepo pull (merge) external/os-autoinst-common
772201f0 (kalikiana/svirt_vmname) svirt: Store vmname early for use after test run
ca120328 Run openQA full-stack test as part of os-autoinst CI tests
a2d9c194 Add new variables in backend/generalhw.pm
34c83a3c (okurz/feature/ikvm) Add simple test for backend::ikvm
aa99694f (kalikiana/svirt_do_extract_assets) svirt: Implement do_extract_assets
aa936732 (okurz/feature/amt_tests) Mark backend::amt as deprecated as it seems to be unused
95b8d8d9 Ensure 100% statement coverage for backend::amt
40ba8c80 backend::amt: Remove unsupported call to remove crash file
I made a mistake. The job I triggered was not actually running on openqaworker2 as I missed to bind it to one host. So again with openqa-clone-job --within-instance https://openqa.suse.de/8114218 TEST=prj2_host_upgrade_sles15sp3_to_developing_kvm_poo106017_investigation_okurz BUILD=X _GROUP=0 WORKER_CLASS=virt-mm-64bit-ipmi,64bit-ipmi,64bit-ipmi-large-mem,openqaworker2
Created https://openqa.suse.de/tests/8114316
https://openqa.suse.de/tests/8114316 has passed the step where tests previously failed.
@mkittler unless you have better ideas, how about for now we skip requesting full screen updates in the case of consoles that involve localXvnc?
- Description updated (diff)
- Due date set to 2022-02-22
Setting due date based on mean cycle time of SUSE QE Tools
I saw the pr has been merged to github master. Can it be deployed in OSD ASAP? the target date Feb 22 is too late, our tests will be blocked for several builds.
- Status changed from In Progress to Feedback
As mentioned in the chat it has already been deployed. Let me know whether it works.
I've been restarting remaining jobs via failed_since='2022-02-03' host=openqa.suse.de additional_filters="reason like '%Error connecting to VNC server%localhost%Connection refused%'" ./openqa-advanced-retrigger-jobs.
- Status changed from Feedback to Resolved
the verify test passed. your restarting tool did not seem to work, but it does not matter, I've rerun all our failing tests. Thank @mkittler & @okurz for your quick fix.
- Related to deleted (action #105429: openQA's fullstack test fails in `shutdown` module)
Also available in: Atom
PDF