action #113201
closedcoordination #109668: [saga][epic] Stable and updated non-qemu backends for SLE validation
coordination #100688: [epic][virtualization][3rd party hypervisor] Add svirt backend compatibility for vmware 7.0
Integrate spike solution for accessing VMWare's VNC-over-websockets into os-autoinst's VNC console size:M
Description
Motivation¶
- Ticket for spike solution: #107032
- PR of spike solution: https://github.com/os-autoinst/os-autoinst/pull/2091
- See parent ticket for further context
Acceptance criteria¶
- AC1: It is possible to connect to a VMWare virtual machine via VMWare's VNC interface (e.g. by specifying credentials via some test variables, possibly also requiring to use a special/new VNC-console-API from test code)
- AC2: The connection is attempted to be restored if it breaks (which involves requesting the websocket URL from scratch and invoking dewebsockify again)
- AC3: Documentation for AC1 exists, e.g. under
os-autoinst/doc/backends.md
- AC4: Behavior is tested against a real VMWare instance
- AC5: Relevant code has unit tests
Suggestions¶
- See commit message of https://github.com/os-autoinst/os-autoinst/pull/2091 and build your changes on top of that PR
- See https://progress.opensuse.org/issues/100688#note-25 for VMWare credentials for running a test against the real VMWare instance
- e.g. invoke the test like this (password redacted, use URL encoding when inserting the password):
OS_AUTOINST_TEST_AGAINST_REAL_VMWARE_INSTANCE='https://root:…@vh002.qa2.suse.asia/2' OS_AUTOINST_DEWEBSOCKIFY_PORT=5900 prove -vI. t/27-consoles-vmware.t
- e.g. invoke the test like this (password redacted, use URL encoding when inserting the password):
- Extend unit tests as suggested in the mentioned commit message
Out of scope¶
- Deal with poor performance; there's likely not much we can do
- Implement JPEG compression in our VNC module (it might help with performance but likely also cause problems with the pattern matching)
Updated by okurz over 2 years ago
- Priority changed from Normal to High
- Target version set to Ready
Updated by okurz over 2 years ago
- Subject changed from Integrate spike solution for accessing VMWare's VNC-over-websockets into os-autoinst's VNC console to Integrate spike solution for accessing VMWare's VNC-over-websockets into os-autoinst's VNC console size:M
- Description updated (diff)
- Status changed from New to Workable
Updated by mkittler over 2 years ago
I've been trying to run a real svirt/VMWare test locally. Unfortunately I've set the wrong password. Seems like that were too many login attempts via SSH with the wrong password. At least I now cannot login anymore, not even via the web app on https://vh002.qa2.suse.asia (using credentials from https://progress.opensuse.org/issues/100688#note-25).
Could someone reset the password? Maybe something like faillock --user … --reset
would help.
Updated by mkittler over 2 years ago
- Status changed from Workable to In Progress
Draft to configure VNC over WebSockets usage via test distribution: https://github.com/Martchus/os-autoinst-distri-opensuse/pull/new/svirt-vmware-vnc-cfg
Maybe it makes more sense to configure this "under the hood" within backend code, though.
Updated by okurz over 2 years ago
mkittler wrote:
Draft to configure VNC over WebSockets usage via test distribution: https://github.com/Martchus/os-autoinst-distri-opensuse/pull/new/svirt-vmware-vnc-cfg
Maybe it makes more sense to configure this "under the hood" within backend code, though.
You can consider the test distribution the "rapid prototyping" platform for os-autoinst for now :)
Updated by mkittler over 2 years ago
Updated https://github.com/os-autoinst/os-autoinst/pull/2091, see additional commit
Updated by openqa_review over 2 years ago
- Due date set to 2022-07-26
Setting due date based on mean cycle time of SUSE QE Tools
Updated by mkittler over 2 years ago
I've now managed to test this with a real job but I've ran into
error: internal error: HTTP response code 500 for call to 'RegisterVM_Task'. Fault: ServerFaultCode - Current license or ESXi version prohibits execution of the requested operation.
so I've resorted to use https://openqaw8-vmware.qa.suse.de where I've got one step further:
[2022-07-12T12:32:04.055634+02:00] [info] ::: basetest::runtest: # Test died: VMWare web socket URL request failed: The object 'vim.VirtualMachine:10' has already been deleted or has not been completely created
So I looks like I need to implement a retry after errors like this because likely it is just the "or has not been completely created" part.
I will also extend the svrit backend documentation to explain how I set things up.
Updated by mkittler over 2 years ago
One step further, now the auth fails (despite correct credentials):
[2022-07-12T14:14:13.360057+02:00] [debug] VMWare web socket URL request failed: The object 'vim.VirtualMachine:10' has already been deleted or has not been completely created, trying 11 more times
[2022-07-12T14:14:22.379136+02:00] [debug] VMWare auth request failed: Cannot complete login due to an incorrect user name or password., trying 10 more times
[2022-07-12T14:14:32.397369+02:00] [debug] VMWare auth request failed: Cannot complete login due to an incorrect user name or password., trying 9 more times
Updated by mkittler over 2 years ago
Looks like the auth issue is because one is already logged in at this point and cannot login again, see VMWare's event log:
Benutzer kann sich nicht anmelden, weil er bereits angemeldet ist Dienstag, 12. Juli 2022, 16:41:53 +0200 Info Keine
Benutzer root@10.163.28.162 als Mojolicious (Perl) angemeldet Dienstag, 12. Juli 2022, 16:41:48 +0200 Info Keine
So I'm either clever about reusing a session or just change the user agent (one can apparently login multiple times with different user agents).
Clearing the cookies is sufficient to get past the auth error. And The object 'vim.VirtualMachine:10' has already been deleted
was because the VM-ID was wrong. However, now I'm maybe as far as it gets with https://openqaw8-vmware.qa.suse.de because creating a VNC over WebSockets "ticket" like on https://vh002.qa2.suse.asia (which already has the newer version) doesn't work.
Updated by okurz over 2 years ago
At any time feel welcome to ask users to test themselves
Updated by mkittler over 2 years ago
- The license problem has been resolved.
- I've been able to get a VNC image within an openQA test (see videos on https://w3.suse.de/~mkittler/vnc-over-ws). The needle
bootloader_svirt-vmware_bios_frontpage-20190306
isn't matching in my setup so the test fails quite early, though. - So I've been updating the PR, see its updated description: https://github.com/os-autoinst/os-autoinst/pull/2091#issue-1274995579=
Updated by mkittler over 2 years ago
- Status changed from In Progress to Feedback
The PR https://github.com/os-autoinst/os-autoinst/pull/2091 should now be ready to be merged. The diff coverage is slightly below our target but it should be ok for now as the most important parts are covered (and some covered lines are just not tracked). Before we invest any more effort into improving this it would make sense to merge the PR to get more user feedback.
Updated by mkittler over 2 years ago
Now the changes to use VNC over WebSockets have been merged. It should fulfill all acceptance criteria. To use it, one still needs to set the test variable VMWARE_VNC_OVER_WS=1
(and possibly VMWARE_VNC_OVER_WS_INSECURE=1
). I've mentioned that in #eng-testing and will hopefully get some feedback.
Updated by okurz over 2 years ago
- Due date deleted (
2022-07-26) - Status changed from Feedback to Resolved
With the above I agree that all ACs have been fulfilled. mkittler also mentioned and explained the current state in chat to important stakeholders so we can consider this done.